Compressing plotted data with variable sample rate?

Question

I'm currently plotting high sample rate data by down-sampling by a factor and averaging. Most of the time this is working ok, but I am aware that at very low zoom levels (zoomed right out) the averaging is hiding vast amounts of data.

I have a timestamp for every data point, so are there any algorithms out there which will remove the pointless and 'less useful' points? I've done a bit of Googling and found papers retlated to time-series data compression, but that's not really what I'm after. I'm looking for something which will take advantage of the particular properties of plotting. ie. if you have 10 consecutive points with the same value, the intermediate ones can effectively be removed without losing any detail in the visual representation.

score 3 · Answer 1 · answered Feb 24 '12 at 21:46

Here is what I did. The basic idea is to mimic what you would see if you were plotting every single sample, just do it much faster

Calculate roughly how many data points will fall in one horizontal pixel. There is no point in plotting more points than the are pixels on the screen.
Divvy up the data in blocks so that you roughly get one block per pixel
Calculate the min and the max for each block
For each block just add the max and the min in the correct order to your plot data

This way you get a plot array that has about twice the number of data points than there are pixels which is about the highest number that makes sense. You have to recalculate whenever you zoom, change the data range or re-size the window, but that tends to be fairly quick. If you don't want to handle window resizing, you can simply just calculate for the worst case (full screen).

Here is a code snippet that shows that in Matlab (it omits the ordering for brevity). The hi-res and the low-res plot look almost the same.

%% reduced sample plot example
n = 200000; % lots of samples
fs = 44100;
% noise with exponential decay
x = randn(n,1).*exp(-(0:n-1)'/fs/3);
% block approach
npix = 2000; % emulate an HD display
x1 = reshape(x,n/npix,npix); % chop into blocks
x2 = [min(x1); max(x1)];  % cal min and max
x2 = x2(:); % interleave
figure(1); plot(x); % plot with 200000 data points
figure(2); plot(x2); % plot with 2000 data points

score 1 · Answer 2 · answered Feb 25 '12 at 08:11

Multi-scale representation is what you are really looking for!

Probably one of the best example is the Google Earth (and similar class of) data. If you had only one scale to represent all the data the challenge is that when you are doing significant zoom out of extremely detailed data over a large range makes it very cumbersome to handle where as if zoom-in deep enough an extensively sub-sampled data looks all flat. This is a classical problem in signal processing : "multi-scale representation of data".

As a very simple and intuitive way, you can just sub-sample them in /2, /4, /8, /16 of sampling rate. Stop at appropriate level. When you are presenting the data, when you zoom out,pick data from coarser resolution; when you zoom in back, you can pull out respective segment from the finer resolution content.

You can also look at wavelets which provides a theoretically most elegant way to do it.

Compressing plotted data with variable sample rate?

2 Answers2