How to estimate the local trend in a signal?

Question

I need to remove trend from my time-series which looks like the following images.

However, I want to estimate the trend before removing it. Hence directly removing it won't do it.

The simple polynomial models didn't work, because — I think — the signal is long and the trend is changing. Also, filtering the signal with low pass removed too much of the trend as well.

---

---

---

The files are available here.

Answer 1

Start simple: just use a 1-D median filter of an appropriate length.

If I do that with a length of 100 samples, I get the following for your first signal.

The top plot shows the original signal (blue) and the median filtered signal (red). The bottom plot shows just the zoomed-in median filtered signal.

Another possibility is to use a DC Blocker.

---

Code Below

load signals_samples.mat
clf
for k=1:10
figure(k)
clf
subplot(211)
plot(signals_samples{k})
hold on
plot(medfilt1(signals_samples{k},100),'r')
subplot(212)
plot(medfilt1(signals_samples{k},100))


end

Answer 2

In A Self Supervised Learning for a Signal Denoising I wrote:

You may have a look at the method called JOT: A Variational Signal Decomposition Into Jump, Oscillation and Trend (You may access it in A Two Stage Signal Decomposition into Jump, Oscillation and Trend Using ADMM).

This method basically does what you're after, it decomposes the signal into 3 signals:

You may look on the results of a signal similar to yours:

The method is quite simple if you know ADMM.
In any way, they supply code.

You signals looks very long, so it might stress the solver for memory.
Still worth a try.

Answer 3

Similar to the JOT method suggested by @Royi, the BEADS algorithm decomposes a signal in three components:

• a trend,
• a sparse signal,
• a residual (which can be used as quality control).

I did a quick test on your first signal. BEADS is not (yet) good at the beginning and end, but it is quite fast: 0.25 seconds for the first signal. It scales almost linearly with signal size. Its parameters allow to control the smoothness of the trend. Here are the results. Parameters follow.

data1 = [signals_samples{1}];
% Filter parameters
fc = 0.0035;     % fc : cut-off frequency (cycles/sample)
d = 1;          % d : filter order parameter (d = 1 or 2)
% Positivity bias (peaks are positive)
r = 1;          % r : asymmetry parameter
% Regularization parameters
% amp = 0.8;

amp = 0.1;

lam0 = 1amp;
lam1 = 5amp;
lam2 = 2*amp;
 [data1Filt, dataBaseline, cost] = beads(data1, d, fc, r, lam0, lam1, lam2);