1

I am attempting to use the least squares method to fit a polynomial to a gapped dataset. Regularization was performed assuming variances defined by the function:

$$v_{\epsilon_n} = 0.01 + 0.001(t_n + t_0).$$

There is a large period of time where the recording instrument was offline and no data were recorded, which seems to decrease the quality of my fit.

Adding weights slightly improves my result; however, the result is still poor. Looking at the shape of the data, it would seem a 5th-order polynomial would be more than sufficient. This leads me to think that I am not handing this gap correctly (currently I'm doing nothing about it ...).

My first thought was to split the dataset and produce two separate fits that I could then combine, but this seems unnecessary and messy. Is there an elegant solution to handling this gap?

Data (grey dots), 5th-order polynomial fit (blue), and 5th-order polynomial fit with weights (orange)

For reference, here is how I've computed these solutions:

data = Import["...~/filepath/practical.dat"];
t = data[[All, 1]];
d = data[[All, 2]];
ts = t/(Last[t] - t[[1]]) - 15.3;
k = 6;
n = Length[t];

a = Table[ts[[i]]^(j - 1), {i, n}, {j, k}];
c = DiagonalMatrix[Array[(1/(0.01 + (t[[#]] - t[[1]])*.001)) &, n]];
cs = c/(Last[t] - t[[1]]);
m = PseudoInverse[a\[Transpose].a].a\[Transpose].d;
mw = PseudoInverse[a\[Transpose].c.a].a\[Transpose].c.d;
soln = MapThread[{#1, #2} &, {t, a.m}];
wsoln = MapThread[{#1, #2} &, {t, a.mw}];

Using Fit, I can get the same unweighted result (blue curve), so at least I know my least squares soln is correct ...

https://drive.google.com/drive/folders/0B4OUmLXw4ZJ7MVBaZWZXX1NGU28?usp=sharing

J. M.'s missing motivation
  • 124,525
  • 11
  • 401
  • 574
acquisto2
  • 75
  • 3
  • 2
    The gap is not your problem. It is attempting to fit a polynomial to the data. (Although you can do much better with k=9.) You'll get a much better fit (although the change in variance is not accounted for) if you use smoothing splines (https://mathematica.stackexchange.com/questions/33206/implementation-of-smoothing-splines-function/85310#85310) or Anton Antonov's quantile regression package (https://mathematicaforprediction.wordpress.com/2014/04/19/find-fit-for-non-linear-data/). – JimB Oct 18 '17 at 20:35
  • 1
    It looks a lot like a slowly growing sinusoid... why not use that instead of a polynomial? – bill s Oct 18 '17 at 21:30
  • This is just an exercise where I am required to fit a 5th-order polynomial using least squares. Basically, I was under the impression that something was wrong because multiple people (professors actually) told me that this fit wasn't good enough... – acquisto2 Oct 19 '17 at 06:01

0 Answers0