4

We have the need to save signal samples measured with an arbitary sampling rate. The samples are usually about one second in duration. Currently we save them in a text file format which consumes a lot of disk space and is very slow. Also this type of format doesn't suit for streaming applications.

I have been searching for a binary file format for this kind of data, but I'm not sure what factors I need to consider when choosing a format.

The sampling rate of our data varies from 1kHz to 152kHz and currently 16 bits would be enough to represent the data (varies roughly from -1000 to 1000). This is vibration data measured from different sensors if it makes a difference. Also we would need to save pre-calculated FFT's in the same format. Is this possible?

Just to point out, I have no education in signal processing. I have the task to implement the system to be used in the analysis.

Currently the best option seems to be WavPack, which is basically a compressed WAV file and supports sampling rates from 1 to 4.3GHz and the compression is lossless.

WavPack is released in 1998 so I'm thinking are there any newer or more efficient formats available today?

kor_
  • 143
  • 5
  • 2
    I'd agree, for any significant amount of data, as well for accuracy reasons, a text format is a terrible choice for samples. how many samples are there per recording? and: what's the numeric format of the original samples? Real? Complex? Number of bits? – Marcus Müller Apr 20 '17 at 07:06
  • 8 bit makes no sense if you need to store 2000 different values. – Marcus Müller Apr 20 '17 at 09:41
  • Marcus, true, my mistake. – kor_ Apr 20 '17 at 10:48
  • I store all my sampled signals in WAV files. They're supported by everything on every platform, are easily compressible (if needed) with standard tools, can store an arbitrary number of channels with essentially arbitrary number format, and it's a well documented standard. It's the ideal container for generic sampled data. – MBaz Apr 20 '17 at 13:52

2 Answers2

3

Let's write down a quick spec:

  • The samples are usually about one second in duration,
    i.e. let's say the maximum length is 4s
  • The sampling rate of our data varies from 1kHz to 152kHz,
    i.e. max rate 152 kS/s
  • varies roughly from -1000 to 1000,
    i.e. more than 2048, so let's head straight for 16 bit integers

That makes a maximum recording raw data size of

$$4\,\text s\cdot 152\,\frac{\text{kS}}{\text s}\cdot 2\,\frac{\text B}{\text S} = 1216\,\text{kB.}$$

In other words, by all modern standards for permanent storage and RAM of machines that you'd use to store recordings... nearly no data at all, and that's the maximum case. Even when you incorporate the pre-calculated FFTs (why?? Storage access of data in this length is usually much slower than computing them from the data when you need it), that's still not much data.

So, I would recommend not to use any specific compression at all, but simply write the data as raw int16 into files. Every, language, in this world has methods of reading raw data from files.

Regarding metadata like sampling rate, time of capture, and something like that:

That really depends on your usage scenario. If you know you'll keep the data with the metadata database, I'd quickly put up a separate table (be it in CSV file, be it in a SQLite file, or in a proper relational database).

If you know you'll distribute these files somewhere else, check whether you'd rather be using a custom header format (which usually boils down to a packed C struct in one way or another) is really advantageous, or whether a simple standard scientific data file format like HDF5 with well-defined structures within wouldn't be better – that would allow you to skip writing your own header parser in any target language, and also, you could easily put your raw samples, and an arbitrary amount of secondary data into it. Drawback is the fact that you can't just start streaming at the first byte of the data stream without loading the file to a large extent into memory. However, again, with your very small files, that isn't really a problem. I've not worked on a PC with less than 4 GB of RAM in the last decade, and neither should someone who does signal analysis. If of those 4GB of RAM, 2GB are available, you could completely have around 2000 of your maximum size data files in RAM at once.

Marcus Müller
  • 30,525
  • 4
  • 34
  • 58
1

If the samples are going to be very short (in the order of 1 or 2 seconds in duration), why not store the raw uncompressed binary data directly in a file? You could invent your own header for the binary file that will have maybe some metadata explaining the sample rate, the total number of samples, byte format or something like this, and then the stream of numbers directly. If uncompressed becomes to costly in terms of storage space, you could simply grab the stream of numbers, and store the first one directly and then encode the difference of the next number to the previous with Golomb Coding.

Hope it helps.

bone
  • 1,241
  • 11
  • 15
  • Thanks but I try to avoid DIY solutions because if someone has already done this better then it saves a lot of time and effort from us. Especially if we had to implement error checking, streaming etc. capabilities. – kor_ Apr 20 '17 at 10:45
  • error checking is a feature of the device and file system that you store these files on. Shouldn't be part of your file format. Streaming is a non-feature for files of less than 2MB of size... – Marcus Müller Apr 20 '17 at 12:17