How to redefine characters as alignment tabs in a table

Question

I have a pretty good understanding of LaTex, but I've spent a while now digging up information on catcodes, but because they're lesser known, I haven't seen any applicable examples that will help with my problem.

For those of you familiar with any experimental work with data tables, you'll probably be aware that the standard way to store data as numbers in is a table, either tab or space delimited, with enters for a new line. This is how many data processing programs read and write data.

As a result, if I want to enter data into LaTex, I have to reformat my data table to include ampersands between columns and a double-backslash to end lines. Is there a straight forward way to create a new environment, which redefines spaces and tabs and double-backslashes, inserts a tabular environment, and then undoes the redefinitions so that they do not have any effect on the rest of the code, something like

\newenvironment{datatable}[2]  
\catcode{10}=\catcode{4} 
% catcode 10 is spaces and tabs, catcode 4 is an ampersand 

\catcode{5}=\\ % catcode 5 is the return character
% I haven't a clue how to define it as a double-backslash   

\begin{tabular}{#1}  
   \#2  
\end{tabular}  
% I have no idea how I would undo the definitions from above, because if I said   
\catcode{10}=\catcode{10} 
% I'm not sure it would make any sense
% catcode 10 already points to catcode 4
% and I haven't seen anyone use an undo command for catcodes yet 

\catcode{10}=\catcode{4} 
% catcode 10 is spaces and tabs, catcode 4 is an ampersand 

\catcode{5}=\\ 
% catcode 5 is the return character, I haven't a clue how to define it
% as a double-backslash   

\begin{tabular}{#1}  
   #2  
\end{tabular}  
% I have no idea how I would undo the definitions from above
% because if I said   
\catcode{10}=\catcode{10} 
% I'm not sure it would make any sense
% catcode 10 already points to catcode 4
% and I haven't seen anyone use an undo command for catcodes yet 

Hopefully it's clear what I want to do, but effectively I just want to get a better understanding of redefining catcodes, and then undoing redefinitions. Hopefully this question also helps some other Tex users as well. Thanks for your help!

Edit:
The solution does not require the use of catcodes. In my case, I require the ability to implement the solution on overleaf.com (which is just an online latex editor and compiler), but other than that, I have no restrictions.

Answer 1

I think you're complicating your life, what you actually need is "read a text file and produce a table".

There are many packages for doing that. In the following MWE, you can find some examples with csvsimple and pgfplotstable, but there are others: datatool is more powerful, even if perhaps not for beginners. I recommend to you to read their documentation and leave \catcodes to egreg & Co.

I've made my example on Overleaf, with the usual pdfLaTeX compiler.

\documentclass{article}

\usepackage{booktabs}
\usepackage{siunitx}
\usepackage{csvsimple}
\usepackage{pgfplotstable}
\pgfplotsset{compat=1.14}
% you can set some option for all your table if you write this 
\pgfplotstableset{string type, 
    every head row/.style={before row=\toprule,after row=\midrule},
    every last row/.style={after row=\bottomrule}}

\usepackage{filecontents}% only needed to create the csv in this example
% the following filecontents* environments are needed only to create the text files, you don't need the if you already have yourfile....dat
\begin{filecontents*}{yourfilecommasep.dat}
Name,Surname,Length,Gender
Paulinho,van Duck,.56,M
Paulette,de la Quack,.52,F
\end{filecontents*}
\begin{filecontents*}{yourfilespacesep.dat}
Name Surname Length Gender
Paulinho {van Duck} .56 M
Paulette {de la Quack} .52 F
\end{filecontents*}

\begin{document}
\verb|pgfplotstable| accepts space, tab, comma, semicolon, colon, braces, and ampersand as 
separators.

\begin{center}
\pgfplotstabletypeset[
    col sep=space, 
    ]
    {yourfilespacesep.dat}
\end{center}

With \verb|\pgfplotstabletypeset|, you don't even need a file, you can type your data directly.

\begin{center}
\pgfplotstabletypeset[
    col sep=space, 
    ]
    {
    Name Surname Length Gender
    Paulinho {van Duck} .56 M
    Paulette {de la Quack} .52 F
    }
\end{center}

\verb|csvsimple| accepts comma (the default), semicolon, pipe, and tab as separators.

Here I also merged the two colums "Name" and "Surname", and inverted the two colums "Gender" and "Length":
\begin{center}
\begin{tabular}{
    lcS[table-format=1.2,round-mode=places]}
    \toprule
    Ducks and drakes & Gender & {Length} \\
    & & {(\si{\metre})} \\
    \midrule
    \csvreader[head to column names,
        late after line=\\]{yourfilecommasep.dat}{}%
        {\Name\ \Surname & \Gender & \Length}
    \bottomrule
\end{tabular}
\end{center}

\end{document}

Answer 2

Changing catcodes is always a bit tricky especially with space, and especially if here input may contained braced items with spaces too.

So I am changing only catcode of end-of-line.

I am illustrating here inline input, if data is in external file, small variant is needed. But anyhow, there are packages for this, of course method here is only a few lines of macros.

\documentclass{article}

\usepackage{array}
\usepackage{booktabs}

\makeatletter
\def\mytabular@end{\end{mytabular}}
\def\mytabular@parse#1 %
{%
  \if\relax\detokenize{#1}\relax
    \g@addto@macro\mytabular@line{\\}%
    \xdef\mytabular@tabular{%
       \unexpanded\expandafter{\mytabular@tabular}%
       \unexpanded\expandafter\expandafter\expandafter{\mytabular@line}%
       }%
    \expandafter\mytabular@again
  \else
    \g@addto@macro\mytabular@line{&#1}%
    \expandafter\mytabular@parse
  \fi
}%
% some complications to insert toprule but then midrule only once
% after header line
\def\mytabular@newline{%
    \let\mytabular@newline\mytabular@@newline
    \gdef\mytabular@line{\mytabular@atstart\toprule}%
}%
\def\mytabular@@newline{%
    \let\mytabular@newline\mytabular@@@newline
    \gdef\mytabular@line{\mytabular@atstart\midrule}%
}%
\def\mytabular@@@newline{%
    \gdef\mytabular@line{\mytabular@atstart{}}%
}%
\def\mytabular@atstart#1&{#1}%
\def\mytabular@finish{%
    \g@addto@macro\mytabular@tabular{\bottomrule\end{tabular}}%
    \end{mytabular}\mytabular@tabular
}%
\begingroup\catcode`\^^M\active
  \gdef\mytabular@again{\expandafter^^M}%
\endgroup
\def\zzzzzzzzzzzzzzzzzzzzz#1{%
\newenvironment{mytabular}[1]%
  {%
   \gdef\mytabular@tabular{\begin{tabular}{##1}}%
   \obeylines
   \begingroup\lccode`~`\^^M 
   \lowercase{\endgroup\def~####1~}{%
     \def\mytabular@tmp{####1}%
     \ifx\mytabular@tmp\mytabular@end
       \expandafter\mytabular@finish
     \else
       \mytabular@newline%
       \mytabular@parse####1 #1%
     \fi}%
  }%
  {}%
}\zzzzzzzzzzzzzzzzzzzzz{ }%
\makeatother

\begin{document}

\begin{mytabular}{cccc}
    Name Surname Length Gender
    Paulinho {van Duck} .56 M
    Paulette {de la Quack} .52 F
\end{mytabular}

\end{document}

Answer 3

TeX maintains a catcode array; this array has length 256 (indexed from 0) in 8-bit engines and 0x110000 in Unicode engines (XeTeX and LuaTeX). Each entry in the array should be a 4-bit number.

You assign an entry by a declaration of the form

\catcode<number> = <4-bit number>

(the = and the spaces around it are optional); the first <number> must be in the allowed range, that is, 0–255 for 8-bit engines (Knuth TeX, pdftex) or 0–1114111 for Unicode engines.

If used in a non assignment context, \catcode<number> returns the corresponding entry in the array. For instance, \the\catcode`a returns 11 (under standard setup).

Numbers can be input in one of the standard TeX ways: an integer in its decimal representation, an octal number, a hexadecimal number, or by character code:

\catcode 97 = 11
\catcode '141 = 11
\catcode "61 = 11
\catcode `a = 11
\catcode `\a = 11

are all equivalent and assign character a the catcode 11; more technically, they assign the number 11 to the entry in the catcode array indexed by 97. The <number> can also be anything that returns a number in the context (a counter's value, a \chardef token, a \mathchardef token, \numexpr, an internal array entry). Also the right-hand side in the assignment can be expressed in different formats. For instance

\catcode`a=\catcode`b

will assign a the same category code as b (whatever is the current category code of b. A common idiom is \catcode<number>=\active, where \active is a \chardef token whose value is 13.

Thus your \catcode{10}=\catcode{4} is syntactically wrong. Note that there's no way of telling “change all characters of category code 10 into category code 4” without looping through the array; in an 8-bit engine

\count255=0
\loop\ifnum\count255<256
  \ifnum\catcode\count255=10 \catcode\count255=4 \fi
  \advance\count255 by 1
\repeat

would change the characters of category code 10 into characters of category code 4.

When TeX is in the tokenization phase, interpreting input and forming tokens from it, it attaches to character tokens the corresponding category code.

Let's attack your problem. You want to input a file, but assigning different category codes to some characters, namely tab and spaces, but also interpreting the end-of-line as \\. Let's use a syntax such as

 \maketabularfromfile{<table specs>}{<filename>}

Here's a possible code:

\begin{filecontents*}{\jobname.dat}
a   b c
1   2 3
4   5 6
\end{filecontents*}

\documentclass{article}

\makeatletter
\newcommand{\maketabularfromfile}[2]{%
  % confine the changes
  \begingroup
  \catcode`\ =4   % space is column separator
  \catcode`\^^I=4 % tab is column separator
  \catcode`\^^M=\active % end-of-line is active
  \begingroup\lccode`~=`\^^M \lowercase{\endgroup\def~}{\\}%
  \begin{tabular}{#1}
  \@@input #2 % use the primitive \input
  \end{tabular}
  \endgroup
}
\makeatother

\begin{document}

\maketabularfromfile{ccc}{\jobname.dat}

\end{document}

I used <TAB>s as separators for the second column, just by way of example (the site will most likely convert them to spaces).

This is very crude, of course. But since there are packages such as datatool or cvssimple that do a much better job, I don't think it's worthwhile to reinvent the wheel.