Share My Creation Handy Median Filter With Side Salads - Version 2

Hi,

I am making a Shortt Free Pendulum clock and record the time error. When making changes to the tuning of the pendulum you must stop it from moving. This creates artifacts in the time data which are a nuisance when plotting trends. To get rid of these artifacts I made a median filter class with a variable window. I use a list as the window as this makes sorting a breeze. However a median filter is a blunt instrument so I added a couple of side salads.

The first of these applies exception limits to the data and whilst the data is within these limits the data is smoothed with a moving average of the window size.

The second one simply outputs the input when the data is within the exception limits, but if the data goes out side the limits the median filter is sued to produce the output.

I hope someone finds this useful.

Best regards
Rob

PS

Kim Studio found a nasty bug which I have fixed along with some other equally nasty bugs.

I am not happy with the data last end case for the median with moving average and median with limits. It works OK for a single exceptio, but witjh a positive and negative exception in close proximity; not so good. I will have another look at this when I have time.

Any ideas would be appreciated.

I have updated the attached file.

A library class is included in the zip.

 

[kimstudio]

Hi Rob, thanks for sharing this!

A small suggestion could be taking account of filter delay: filtered_sig(2) = median(raw_sig(0:4)) if windows size = 5. Current implementation may have half window size beforehand delay.

The circular buffer may have a bug by reading the code, correct me if I am wrong:

        iw = 0
        For id = 1 To (nD - nW - 1)
            ' Handle window as circ buffer
            If iw = nW Then iw = 0
            w.Set(iw, dataBuffer(id))  ' bug? should be dataBuffer(id + nW - 1) ?
            iw = iw + 1
           
            w.Sort(True)
            dBuf(id) = w.Get(wMidIndex)
        Next

 

[rgarnett1955]

Hi Kim,

Thanks

You are correct. When I started looking at it again I saw other problems which I will fix up and repost as V2.

I have added in some public functions to set parameters as well.

Best regards
Rob

 

[kimstudio]

If the aim is to remove these artifacts then another way is to make a peak detector with values of peak width/height/distance/rising edge slope/etc. for each peak and remove noise spikes that fulfill certain conditions accordingly, then interpolate values in between.

 

[rgarnett1955]

If the aim is to remove these artifacts then another way is to make a peak detector with values of peak width/height/distance/rising edge slope/etc. for each peak and remove noise spikes that fulfill certain conditions accordingly, then interpolate values in between.

Hi Kim,

Thanks for those ideas.

There are plenty of other ways of identifying and removing artifacts. I chose the median filter as it is very good at removing short large amplitude artifacts only a few samples long. It is also quite fast. The 2D median filter is used in image processing to remove speckle artifacts. Viz:

Because I am not doing a streaming filter the end data handling is always an issue. I want to estimate the last data point in some way as it is the latest data and usually that data is the most relevant. I was thinking of linear interpolation or even a spline to handle the last data points, with the invalid (exception limit) data being filled with the weighted mean of points either side. It's tricky because spikes can occur together in the window or spaced apart. Luckily, because median windows are usually short there is not a lot of data to process.

I don't have the time to improve on the filter at the moment, but I will get to it again soon and will take on your suggestions.

Best regards
Rob