20

Possible Duplicate:
How to select a finite number of samples from the file when plotting using pgfplot

Compiling my document with many PGF Plots, each containing thousands of data points from a CSV file takes too long (minutes). I realize that it does not make sense to have this many data points -- were there, say, a tenth, it would be enough to recreate the same plot on paper.

One of my files, for example, consists of 216000 lines of such a format:
0.000000000000000000e+00 0.000000000000000000e+00 1.388888888888888888e-04 -2.182787284255027771e-11
Of course, that is far too much. I then read the file as:

\documentclass{scrartcl}

\usepackage{pgfplots}
\pgfplotsset{compat=1.5.1}

\begin{document}

\begin{tikzpicture}
    \begin{loglogaxis}[
        \addplot[mark=*, color=red] file {Data/plotXYZ.dat};
    \end{loglogaxis}
\end{tikzpicture}

\end{document}

Is it possible for the package to take only every x-th line to speed up the compilation process? Or are there other ways to make it (significantly) faster?

I am running Ubuntu and a friend of mine recommended me to write an awk script that would go through a .csv file and retain only every x-th line. However, I have no clue how I would do that, as I have never used awk before. If this would turn out to be the option of choice, could someone help me setting up such a script?

Community
  • 1
Ingo
  • 20,035
  • We seem not to use the tag [tag:speed]. We use [tag:compiling] for questions concerning compiling time. – yo' Aug 24 '12 at 12:24
  • 1
    Take a look at http://tex.stackexchange.com/questions/47787/how-to-select-a-finite-number-of-samples-from-the-file-when-plotting-using-pgfpl, for example – Joseph Wright Aug 24 '12 at 12:48
  • Thanks, that was exactly what I was looking for. Feel free to close this one. Maybe we can make the other question more discoverable by adding tags or keywords or so? – Ingo Aug 24 '12 at 13:33

2 Answers2

31

pgfplots offers the filter each nth point=100.

Since coordinate filtering has different use-cases, you may want to set 

each nth point=100, filter discard warning=false, unbounded coords=discard

this combination will not generate warnings for every discarded point. Furthermore, it will silently drop the 99 points - just as if you did not write them into the input stream at all. Note that the alternative choice unbounded coords=jump would explicitly interrupt your plot (which is not what you want here).

This approach has been report to work fairly well: in fact, the time was not much larger as if some external tool had been used to filter the data file (has been reported recently by some power user per mail).

Christian Feuersänger
  • 55,840
  • I would suggest choosing a "nice" number for each nth point. Imagine that the orinigal sampling is on a grid 600x600 points. Then the points you leave there are aligned in one axis direction. If the number is e.g. 107, then the points will be spread better. – yo' Aug 24 '12 at 21:56
  • Good idea. However, note that each nth point is applicable for one-dimensional data - it is unsuitable for grid data. – Christian Feuersänger Aug 25 '12 at 09:25
  • Christian, thanks a lot for this first-hand answer. This is a really fantastic approach for my work-flow, since I can have the CSV file changing when running simulations and do not need to write a post-processing script whatsoever to edit the data afterwards. – Ingo Aug 25 '12 at 10:35
3

Although the pgfplots filter each nth point might be a solution at some degree, it certainly isn't one that can be used in general. For example, choosing each nth point=100 might distort some areas of the plot that really need the extra points in order to be illustrated correctly. On the other hand setting each nth point to a lower number might not be "economic" in terms of the number of points being used.

In short, a cleverer downsampling method is needed. That method shoulb be able to dynamically adjust the number of points kept, depending on the complexity of the plot at each area, so that we have the best visual result using only the minimum required number of points (something like latex being able to automatically adjust each nth point=? several times during the "pass" of a diagram).

Take a look at this answer.

In case you have matlab or octave you can use the modified version of matlab2tikz provided there, which implements an effiecient downsampling algorithm.

On the other hand, if you use neither matlab nor octave, you can still translate that downsampling algorithm (it's in function pointReduction) to the language you want and use it to downsample your figures and then export them to CSV.

Community
  • 1
niels
  • 3,295
  • Note that @ChristianFeuersänger 's answer can be applied to different data regions. So one can manipulate the data in batches and save it as a new data file at each step. Via this, MATLAB functionality can be achieved which is possible through pgfplotstable. – percusse Aug 24 '12 at 22:41
  • That's true, but choosing the regions manually for each one of the diagrams of an entire book (it is not guaranteed in general that the regions can be chosen in batch, that depends on the diagrams themselves), is just loosing precious time. The whole process of choosing the points to keep can be done automatically by a simple algorithm, in a much more efficient way. – niels Aug 25 '12 at 08:00
  • What you propose is to use adaptive function compression - which is a reasonable idea, although with its own challenges and the requirement to use an external tool. Note that Ingo asked for a solution for "taking each nth line" explicitly. – Christian Feuersänger Aug 25 '12 at 09:16
  • @ChristianFeuersänger, oh I merely think its a better idea to do the job outside latex with an external tool, the same one that was used to produce the "heavyweight" plots at first place. You can have far better results than just keeping every nth point. – niels Aug 25 '12 at 09:28
  • @niels Agreed, thanks for the clarification. – Christian Feuersänger Aug 25 '12 at 09:56
  • Thanks for your proposal niels. One approach I might take in the end is to first plainly reduce the number of data points per plot, just to speed up the compilation. For the final version, last compilation, I could probably live with a long compilation time, so I would increase the number of points again. – Ingo Aug 25 '12 at 10:38