2

I have data captured by a wireless sensor that is noisy. It randomly jumps in value frequently, and I want to know what this signal will look like without these jumps. I am looking for an elegant signal processing technique to do this, if one exists. Below is the time-series signal:

Time-series signal from sensor, captured at 8 Hz

I initially thought to do a Fourier Transform to see whether there is some frequency I can filter out. The FT looks like this: Fourier Transform of the same signal

Applying a LPF with a cutoff frequency of 2.5 Hz to try getting rid of the 3 Hz signal doesn't yield what I want. It just smooths out the signal and I lose most of the important underlying information that I care about. Using a 10th order Butterworth LPF, with fc = 2 Hz, I get the following signal: Filtering out 3 Hz signal to try recovering underlying signal

As you can tell, I'm not very well-versed in signal processing, which is why I've come to you.

How can I denoise this signal and get rid of the random spikes?

stochasticlover1
  • 23
  • 3
  • 3
    Can you post a picture of a "clean" signal as a reference. It's hard to tell what is the actual signal and what's the noise. – Hilmar Feb 27 '24 at 04:45
  • Denoising is a relative thing - what is the noise and what is your desired signal? Best would be to have a precise mathematical signal model, but even a description of the characteristics you would like to see would be better than nothing. – mateC Feb 27 '24 at 14:58
  • If you set the peaks to 0 and IFFT what do you get? – Baddioes Feb 27 '24 at 21:35

2 Answers2

0

If you want to remove the spikes from the signal, you can use a median filter. There are built-in function for it in MATLAB medfilt1. You can also implement in yourself by setting a threshold to detect the spikes and then replacing those spikes with a median value of a a window of some length around that spike.

Rafae
  • 36
  • 1
  • Median filtering worked well as well. Instead of implementing the slope thresholding, I tried implementing median filtering, and, for the most part, it gets rid of the periodic and aperiodic spikes. Only thing is that when a spike is very large, the median filter doesn't get rid of it. Because I wanted a quick and easy solution, I didn't take the time to implement slope thresholding because it would have been a manual implementation. – stochasticlover1 Mar 01 '24 at 08:01
0

Note that it appears that the interference is isolated to 1 Hz and it's higher harmonics. We can implement a multiband notch filter easily to reject those specific frequencies while minimizing impact to the desired signal.

A harmonic notch filter is simplified (elegant) when the sampling rate is also a harmonic (integer multiple of 1 Hz in this case) and given by the transfer function:

$$H(z) = \frac{1+\alpha^N}{2}\frac{1-z^{-N}}{1-\alpha^Nz^{-N}}$$

Where the closer $\alpha$ is to $1$, the higher the "Q" of the filter meaning tighter notches. This will produce periodic notches at $f_s/N$ where $f_s$ is the sampling rate. A possible implementation is shown below, where $z^{-N}$ indicates a delay of $N$ samples.

Implementation

Note that for $\alpha$ close to $1$, $(1+\alpha)/2 \approx 1$ and the first multiplier can be eliminated with minimum consequence. (That is not the case with the second multiplier, and has implications on the importance of precision used in this "leaky accumulator" section.)

Here is an example frequency response for $\alpha=0.99$, $f_s=10$, and $N=10$:

Notch Filter Frequency Response

For sampling rates that are not conveniently a multiple of the notch frequency, the integer sampling delays given by $z^{-N}$ with $N$ as an integer can be replaced with fractional delay all-pass filter elements (so $z^{-\tau}$ with $\tau$ as any positive real number).

I detail the derivation of the harmonic notch filter at this other post with further details and other options on harmonic rejection filters:

https://dsp.stackexchange.com/a/52728/21048

Dan Boschen
  • 50,942
  • 2
  • 57
  • 135
  • This is an awesome suggestion. I implemented it, and the periodic spikes go away. The random aperiodic ones don't go away, though, but I'll accept this answer as it did what I asked. – stochasticlover1 Mar 01 '24 at 07:58
  • @stochasticlover1 That's great and thanks for letting me know it worked. The solutions for arbitrary sampling rates using the fractional delay all-pass elements instead of unit sampled delays are even more interesting (and complicated in comparison). (Feel free to upvote the answer as well if it was indeed "awesome"! Also not sure what the actual source of the noise is (always best to address it there if determined) but it appears you may have secondary features at 0.5 Hz increments. You could try an implemention with notches every 0.5 Hz to see if it gets rid of your remaining spikes too. – Dan Boschen Mar 01 '24 at 13:56
  • 1
    I am unable to upvote answers as I don't have 15 reputation, unfortunately. I am still unaware of what is causing this noise in my signal; it's my main concern right now, but yes I agree that understanding the source of the noise is important. It looks as if I have secondary features at 0.5 Hz and 1.5 Hz, though removing the main periodic spikes gives me a good enough signal to analyze. – stochasticlover1 Mar 06 '24 at 04:03