What is the best method to filter a signal where baseline and the signal of interest have overlapping frequency range?

Question

I am reading out the movement of a motor arm using a Hall sensor and a magnet pair. The hall sensor measures the distance between the sensor and the magnet. The motor arm is being moved with a band-limited Gaussian white-noise signal (0-300 Hz). Due to the movement being very small, the stimulus being a white-noise, and the inherent noise of the Hall sensor, I have a terrible Signal to Noise Ratio. I am trying to improve the SNR by filtering. But the problem is that, the frequencies in the sensor output in the absence of any movement of the motor (baseline noise, in blue) hugely overlap with the frequencies of the actual movement (0-600 Hz) and noise (stimulus, in orange).

The figure below shows the generated white-noise signal that I use to actuate the motor (in black). Grey background marks the presence of a movement stimulus to the motor arm (this signal is in orange in rest of the figures). Notice the 0 baseline outside the grey region in subplot 1. The subplot below it shows the unfiltered hall sensor output. Notice the high baseline noise in the absence of any actual movement. So, this "baseline noise" is the inherent noise of the Hall sensor.

Without filtering, the baseline and stimulus are indistinguishable.

But, the powers are slightly different for frequencies below 150 Hz. But the noise and the signal have the same power after 150 Hz. I do need better SNR in this range.

I tried filtering the signal with 10th order Butterworth with cut-off at 600 Hz (Because I need 300 Hz to be represented properly). I can now distinguish baseline from stimulus but SNR is still very bad, as visible in the PSD.

I want to use the noise in the baseline to denoise the stimulus. How should I do it?

Answer 1

This would be a good application for using the Cross Spectral Density between the input and output. The Cross Spectral density (which is Coherence when normalized) provides the relative magnitude and phase for the system transfer function based on the correlated components between the input noise-like signal and the resulting output noise-like signal while attenuating the independent noise components present regardless of input. Please see these other posts for further explanation of this as well as further details in computing the CSD:

https://dsp.stackexchange.com/a/85712/21048

Intuitive explanation of coherence

The result will have best fidelity where the input signal has energy in the frequency domain, so for this purpose a white noise input is ideal. The stronger the input can be while still in the linear range of the system, the better the resulting estimate will be (in terms of SNR).

Answer 2

You can technically use a white noise stimulus to perform a system identification, I would not do it for the following reasons

1. The white noise stimulus seems to have a lower amplitude than the baseline white noise, making it hard to distinguish what your actual stimulus is. Maybe you can increase the amplitude of your stimulus ?
2. Even if you can increase the amplitude of the stimulus, you basically input a wideband signal and try to deduce the frequency response and transfer function. It's gonna be hard to identify the amplitude and phase at higher frequencies, especially if you have a 20 dB or 40 dB/decade rolloff. Even if you had an infinite-precision ADC, the high frequency signals will be buried in white noise.

I recommand performing a sine sweep for system identification. Basically, you input a sine with an amplitude that will not saturate your sensor or your ADC.

$ x = Asin(\omega t) $

Wait for the transient to die out, then measure the amplitude of the output signal, it should look like this $y = Bsin(\omega t + \phi)$

Perform a synchronous demodulation, to extract the gain and phase at that specific frequency. You can average multiple cycles to increase the SNR. This can be useful if you want to measure low amplitudes. If you see a resonance, I recommand that you take more data points around the resonance frequency as resonances can greatly impact the design a control loop.