Detrending or removing linear trend from 2D data

Question

I have some data that when plotted I get an inclined graph, as described:

data = Import["p02q.dat", "Table"];
xdata = data[[All, 1]];
ydata = data[[All, 2]];
dataplot = Transpose[{xdata, ydata}];
plot2q = ListLinePlot[ dataplot, AspectRatio -> 1/GoldenRatio, 
  ImageSize -> 800, PlotRange -> All, GridLines -> Automatic, 
  InterpolationOrder -> 2, PlotMarkers -> {Automatic, 2}]

My intention is to get a leveled graph out of this one, like the one bellow:

Could anyone suggest how to solve this, please? Data file can be downloaded here.

Find a linear best fit, use this to define an angle to the x-axis, generate a rotation matrix, and apply that to your data — b3m2a1, Dec 21 '20 at 19:36
This looks like a phase unwrap problem. This answer may help. — Hugh, Dec 21 '20 at 20:01
Find the best-fit linear function, $f(t)$, then subtract that from the data. (Don't rotate! What a mess! It means you'll have a non-function.) — David G. Stork, Dec 21 '20 at 20:51

Vitaliy Kaurov · Answer 1 · 2020-12-23T16:18:47.840

3

Removing linear trend, as suggested in comments, is almost a one-liner (check out, Fit, FindFit, LinearModelFit):

data=Import["https://pastebin.com/raw/aWYk1Jba"];
lm=LinearModelFit[data,x,x];
ListLinePlot[Transpose[{data[[All,1]],lm["FitResiduals"]}],
PlotLabel->lm["BestFit"],PlotTheme->"Detailed"]

edited Dec 23 '20 at 16:18

answered Dec 23 '20 at 05:07

Vitaliy Kaurov

73,078
9
204
355

Thank you, Vitaliy. – Florin Dec 23 '20 at 08:56

Anton Antonov · Answer 2 · 2020-12-24T16:58:10.487

I different version of the workflow from Vitaliy's answer using both Quantile Regression and Least Squares fits. Note that the de-trending results are slightly different.

Get the QRMon package:

Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MonadicQuantileRegression.m"]

Get data:

data = Import["https://pastebin.com/raw/aWYk1Jba"];

Construct a QRMon workflow with data summarization, Quantile Regression and Least Squares fitting, and error plots:

qrObj = 
   QRMonUnit[data]⟹
    QRMonEchoDataSummary⟹
    QRMonQuantileRegression[1, 0.5, InterpolationOrder -> 1, Method -> {LinearProgramming, Method -> "InteriorPoint"}]⟹
    QRMonFit[{1, x}]⟹
    QRMonPlot⟹
    QRMonErrorPlots["RelativeErrors" -> False, PlotRange -> {-0.06, 0.16}];

Get the regression functions:

aFuncs = qrObj⟹QRMonTakeRegressionFunctions;
Map[Simplify[#[x]] &, aFuncs]

Get the errors, i.e., the signal de-trendings:

aErrors = qrObj⟹QRMonErrors["RelativeErrors" -> False]⟹QRMonTakeValue;

ListLinePlot /@ aErrors

@Florin Welcome, good luck! – Anton Antonov Dec 25 '20 at 17:38 — Anton Antonov, Dec 25 '20 at 17:38

Detrending or removing linear trend from 2D data

2 Answers2