unevenly spaced data

Question

Suppose I have a (very large) noisy data set of points $(x_i, y_i)$ and I want to smooth it. Mathematica seems to have a number of smoothing schemes (EstimatedBackground[], ListConvolve[], etc) but they all seem to have the underlying assumption that the data is sampled at regular intervals (and thus, usually takes a flat list of numbers as input), which is not my case (nor, I suspect, is it the case particularly frequently). Is there some more or less standard way to deal with this?

But you need some assumption about the underlying relation and the noise. Otherwise your perceived "noise" could be your signal. — Dr. belisarius, Oct 06 '14 at 16:17
@Belisarius Of course. In my case I am doing a computational experiment where I am computing spectral invariants of random (in a certain sense) matrices (so, I have a list of pairs of the form (eigenvalue, some_function_of_eigenvector) of my matrices, which, to make things more annoying, are not precisely the same size. If I do the experiment 1000 times, and compute the mean of my function, and plot it vs the ordinal number of eigenvector, all is well, and the curve is smooth. If I want to plot it against the corresponding eigenvalue, lots of noise. Moving average does sort of work, but... — Igor Rivin, Oct 06 '14 at 16:23
... is not clearly the right thing. In any case, there is the actual mathematical/scientific problem, and then there is the functionality provided by mathematica, which seems to not be quite what is needed (of course, I can write my own Gauss convolver, or whatever, but this will be actual work and run slowly, too boot :() — Igor Rivin, Oct 06 '14 at 16:24
@IgorRivin Can you supply a sample data set or should I experiment on one of my own ? — Sektor, Oct 06 '14 at 16:35
@Sektor here you go (warning, large): https://dl.dropboxusercontent.com/u/5188175/valnorms.m — Igor Rivin, Oct 06 '14 at 16:39
There are generalizations of frequency methods for un-even data eg, http://en.wikipedia.org/wiki/Non-uniform_discrete_Fourier_transform — alancalvitti, Oct 06 '14 at 16:50
@alancalvitti Thanks! That is very cool (though the concerns I expressed before [implementing this is work, and in Mathematica at least will probably be quite slow, when a lot of data is involved] still stand, alas). — Igor Rivin, Oct 06 '14 at 16:54
@IgorRivin Thanks Igor ! Already downloaded it and will try a few things on the data :) — Sektor, Oct 06 '14 at 17:03
@IgorRivin, at least some special types of generalization of Fourier analysis like chirp-z transforms (off the unit disc) can be made fast like FFT. — alancalvitti, Oct 06 '14 at 17:25
@alancalvitti No doubt, but not in Mathematica :( I am perfectly happy to find/prduce implementations (and again, your suggestions are very welcome), but since WRI is making a very big deal of its data handling capabilities, one might hope that there are tools already available. Of course, one might be wrong... — Igor Rivin, Oct 06 '14 at 17:28
@belisarius, sooner or later there might be. Maybe fractal in frequency plane like http://mathworld.wolfram.com/DevilsStaircase.html — alancalvitti, Oct 07 '14 at 18:40

score 4 · Answer 1 · answered Oct 06 '14 at 18:56

4

Mathematica 10 has new interesting functions for irregularly spaced data like MovingMap

data = RandomReal[1, {1000, 2}];

ListLinePlot[MovingMap[Mean, data, {{0.1}}]]

enter image description here

answered Oct 06 '14 at 18:56

ybeltukov

43,673
5
108
212

Aha! That could be what I am looking for, I will check it out and report! – Igor Rivin Oct 06 '14 at 19:07
But wait, this is just a generalized MovingAverage[] – Igor Rivin Oct 06 '14 at 19:09
@IgorRivin, what functionality were you expecting? – alancalvitti Oct 24 '14 at 19:11

score 2 · Answer 2 · answered Oct 06 '14 at 20:07

2

Show[
   ListPlot[#,
    DataRange -> {0, 6 Pi},
    PlotTheme -> "Detailed",
    PlotStyle -> PointSize[Tiny]],

   ListLinePlot[
    MovingMap[Mean, #, {{250}, Center}, 0],
    PlotStyle -> {Thick, Blue},
    DataRange -> {0, 6 Pi}]] & [Table[Cos[x] + RandomReal[{-1, 1}], {x, 0, 6 Pi, 0.01}]]

enter image description here

answered Oct 06 '14 at 20:07

eldo

67,911
5
60
168

I will try with my data, and report. – Igor Rivin Oct 06 '14 at 20:19
On second thought, your data is actually regularly spaced (and so, just like the built-in stuff, you are not using the abscissas...) – Igor Rivin Oct 06 '14 at 21:04

unevenly spaced data

2 Answers2