5

I use a PC to record time series of some physical property. The problem is that, for some reason, I did not record the time series as a whole, rather I record first segment, then second, third, etc. Each segment is 500,000 points, representing ~30 min. However, they are not strictly consecutive: between each segment and the other there will be a 'lag', or missing points, equal to a few seconds.

Since the number of missed points is too small compared to the total number of samples for each segment, can I simply concatenate the segments and treat them as a single time series? At the end I will be interested in doing FFTs, histograms, etc.

Royi
  • 19,608
  • 4
  • 197
  • 238
student1
  • 384
  • 6
  • 16
  • 1
    A measurement length of 500000 might result in too small frequency resolution, so it might be better to combine the FFTs of all measurements instead of combining the measurements and then take the FFT. A single measurement might even be to long, so you could use Welch's method. – fibonatic Apr 23 '15 at 02:35

3 Answers3

2

Given $ \left\{ x \left[ n \right] \right\}_{n \in M} $ where $ M $ is the set of indices given for the samples of $ x \left[ n \right] $.

The trivial solution (Which it would be great to have a faster more efficient solution is what I'm looking for) would be:

$$ \arg \min_{y} \frac{1}{2} \left\| \hat{F}^{T} y - x \right\|_{2}^{2} $$

Where $ \hat{F} $ is formed by subset of columns of the DFT Matrix $ F $ matching the given indices of the samples, $ x $ is the vector of the given samples and $ y $ is the vector of the estimated DFT of the full data of $ x \left[ n \right] $.

The solution is then given by the Pseudo Inverse (Least Squares Solution):

$$ y = { ( \hat{F} \hat{F}^{T} ) }^{-1} \hat{F} x $$

In practice, the matrix will be very poorly conditioned hence solution must be generated using the LS Solution using the SVD.

The code is available at my StackExchange Codes Signal Processing Q17734 GitHub Repository (Look at the SignalProcessing\17734 folder).

Result of the code:

enter image description here

Remark: Borrowed from Estimate the Discrete Fourier Transform / Series of a Signal with Missing Samples.

Royi
  • 19,608
  • 4
  • 197
  • 238
  • So for any $x$, let $\hat x$ be the "expanded" version in which all the absent samples from $x$ are set equal to $0$. Won't $y$ always be the Fourier transform of $\hat x$? i.e., the "interpolation" you're doing will set all of the absent samples to 0. – Mike Battaglia Jan 16 '21 at 20:00
  • @MikeBattaglia, You're right. It is better not to decimate the Matrix but build it differently as in https://dsp.stackexchange.com/questions/54569. – Royi Apr 03 '21 at 13:10
1

Spectral analysis, FFT, wavelets requires that sequences are of equal duration and that the series is consecutive thus that no missing data points are present. I would suggest to fill the voids with extrapolated values based on the last few instances from last sequence or another solution would be to mirror the last few instances missing values based on inverted pass values or another solution would be to use the last data points and repeat it to fill the void.

Barnaby
  • 333
  • 1
  • 8
  • What about simply concatenating one to another? I don't have absolute time axis, only relative one (i.e. a point of start and a point of end), and my data is random. – student1 Apr 23 '15 at 19:23
  • In FFT and Spectral analysis you have two coordinates (X is time in Spectral analysis and Y is frequency in FFT X is time and Y is power) You require that each elements is of the same extent and that no emply spaces exist otherwise you will have a resulting FFT with limited value as per interpretation. Seeing examples of what you are trying to do will help. (spectrogram and FFT) – Barnaby Apr 24 '15 at 00:18
1

Depending on what you're tryinh to do, you may not have a problem at all. If you merely want to examine FFT spectral magnitude results, then just perform an FFT on each of your separate blocks of time samples. You might want to average the multiple FFT magnitude result sequences to obtain an "averaged FFT magnitude" display which will have a reduced noise variance. And that's often useful.

If you're implementing some sort of 2-dimensional, real-time, spectral analysis (a "waterfall" display) then I suggest you merely concatenate your multiple blocks of time samples and then perform your spectral analysis to see what happens.

By the way, when you say 500,000 samples, do you really mean 2^19 = 524,288 samples (an integer power of two)?

Richard Lyons
  • 5,777
  • 13
  • 25