Simple approach to importing data and reading headers

Question

Admittedly, this will be a very naive question, but I'm really curious to learn how one can achieve the following in a simple manner:

Suppose we have a text file formatted like so:

#name input
#x = 500
#y = 150
20 53.2 60 20.1 100 200
2 93.3 65 22.2 400 300
.... (continues with more data similarly)

So all headers are on top of the file, and the numerical data is a mix of integers and floats, and in this example we have 6 columns. I'm trying to learn how one can in a straightforward manner extract each column and map to a variable, moreover, to also extract certain info from the header, for instance we want to know what x is (so we want to extract 500 from the header).

The reason I ask for simple approaches is because most solutions to these kinds of questions I've come across here on SE are quite compact and hard to decipher for newcomers, e.g. here.

If I were to do this task in Python, my approach would go as follows:

import numpy as np

fname = 'sample'
data = np.genfromtxt(fname,comments='#')
firstcol = data[:,0]
secondcol = data[:,1] #and similarly for other columns

x = 0
#now we find where the header starting with #x is and save its value in x:
with open(fname,'r') as f:
    lines = f.readlines()
    for el in lines:
        if el.startswith('#x'):
            x = int(el[el.rindex(' ')+1:len(el)])

• In principle, would there be the possibility of going about this task in Mathematica similar to the above approach? I admit it's not the most elegant approach, but I would learn a lot by the direct comparison of the approaches.

Answer 1

For me, simple means one step at a time. First, we read the entire file and split it into lines

fname = "sample.txt";
whole = StringSplit[
   Import[fname, "Plaintext", Path -> NotebookDirectory[]],
   "\n"];

Next, we split each line using the comment, if any, as the separator pattern, like this

raw = Flatten[StringSplit[#, "#" ~~ ___] & /@ whole /. {} -> Nothing];

We extract the numbers from the raw strings like this

data = ToExpression@StringSplit /@ raw;

Of course, we could have combined those last two steps into one, but two steps is simpler to explain. Likewise, we could have combined the last three steps, but that would not be so simple, to me.

We extract the comments from whole like this

comments = StringCases[#, "#" ~~ ___] & /@ whole;

We can parse out the value after "#x =" from the comments like this

Flatten[StringCases[#, "#x =" ~~ z___ :> ToExpression[z]] & /@ 
  comments]

Alternatives that use less shorthand

Sometimes in Mathematica it is better to spell things out than to use the shorthand as in the above expression. Here is an alternative way of extracting the raw data from the whole list of strings:

rough = Table[
 StringSplit[w, StringExpression["#", ___]],
 {w, whole}];
raw = Flatten[rough];

If the above statement was written in Python, it might be rendered as rough = [ StringSplit[...] for w in whole]. When we see Table in Mathematica, some of us are reminded of list comprehensions in Python.

Almost every expression is spelled out in the above, except for the triple blank ___ . We could spell it out, too, but its name is too long. It means "anything or nothing" and we just get used to it.

The above gives us the same raw list of non-comment strings, which still must be processed to get the numbers. We can use Table again, as

data = Table[
 ToExpression[StringSplit[r]],
 {r, raw}];

Here we are using StringSplit to split the raw strings at the whitespace. Then we apply ToExpression to convert from string to numeric expressions.

Good ways to look at these lists are with expressions like Column[data], or Grid[data] or Dataset[data]. Personally, I prefer the postfix notation data // Dataset, which means the same thing.

The longhand way of extracting the comments is to use

comments = Table[
   StringCases[w, StringExpression["#", ___]],
   {w, whole}];
comments = Flatten[comments];

A longhand way to parse out the value of $x$ in the comments is with

xStr = Table[StringCases[c,
   StringExpression["#x = ", z__ ] :> z],
  {c, comments}]
x = ToExpression[First[Flatten[xStr]]]

We use a double blank in the above expression to match all the characters that follow. A single blank would match only one character and a triple blank would match no characters. The triple blank would work in the sample.txt file we are considering, but a double blank makes more sense.

Here is a link that might be helpful in deciphering some of the shorthand that is used:

• What the @#%^&*?! do all those funny signs mean?