Parsing rows (`\\`) and tab characters (`&`)

Question

I have a use for parsing what would look like tabular input data, except that it isn't part of a tabular environment, but rather the argument to a macro. So I developed two macros, \readTABrow which will read a & separated list into different tokens, and \readMANYrows which will read a \\ separated list into many tokens.

Each of these two macros behaves as expected, when I pass it arguments that would otherwise appear in a tabular "row" or "table". The problem comes when I take the output from \readMANYrows, which gives me one token per row, each populated with &-separated data, and I pass that token to \readTABrow in an effort to further parse a tabular row into individual cell data.

I'm suspecting that somehow, the "nature" of the & characters change when being parsed by readMANYrows and that they no longer look like tab separators by the time they reach \readTABrow.

My question is how to preserve the tab "nature" of the & characters during a \readMANYrows, so that I can pass the token containing them straight to \readTABrow without a problem.

Here is my MWE:

\documentclass{article}
\usepackage{ifthen}

\makeatletter

% FOR PROCESSING A &-SEPARATED ROW 
\newcounter{TABcellindex@}

\newcommand\readTABrow[2]{%
  \def\doneTABread{F}%
  \def\postTAB{#2}%
  \setcounter{TABcellindex@}{0}%
  \whiledo{\equal{\doneTABread}{F}}{%
    \stepcounter{TABcellindex@}%
    \expandafter\processTAB\postTAB&\\%
    \ifthenelse{\equal{\preTAB}{}}{%
      \addtocounter{TABcellindex@}{-1}%
      \def\doneTABread{T}%
    }{%
      \expandafter\protected@edef\csname #1X\roman{TABcellindex@}\endcsname{%
        \preTAB}%
    }%
  }%
% \#1TABcells GIVES HOW MANY TAB COLUMNS WERE PROCESSED
  \expandafter\xdef\csname #1TABcells\endcsname{\arabic{TABcellindex@}}%
}

\def\processTAB#1&#2\\{%
  \protected@edef\preTAB{#1}%
  \protected@edef\postTAB{#2}%
}

% FOR PROCESSING A \\-SEPARATED GROUP OF ROWS
\newcounter{ROWcellindex@}

\newcommand\readMANYrows[2]{%
  \def\doneROWread{F}%
  \def\postROW{#2\\}%
  \setcounter{ROWcellindex@}{0}%
  \whiledo{\equal{\doneROWread}{F}}{%
    \stepcounter{ROWcellindex@}%
    \expandafter\processROW\postROW||%
    \ifthenelse{\equal{\postROW}{}}{%
      \def\doneROWread{T}%
    }{}%
    \expandafter\protected@edef\csname #1X\roman{ROWcellindex@}\endcsname{%
      \preROW}%
  }%
% \#1ROWs GIVES HOW MANY ROWS WERE PROCESSED
  \expandafter\xdef\csname #1ROWs\endcsname{\arabic{ROWcellindex@}}%
}

\def\processROW#1\\#2||{%
  \protected@edef\preROW{#1}%
  \protected@edef\postROW{#2}%
}

\makeatother

\begin{document}
\verb|\readTABrow| Input: \verb|{a&b&c&d}|
\readTABrow{myrow}{a&b&c&d}\par
.\myrowXi.\par
.\myrowXii.\par
.\myrowXiii.\par
.\myrowXiv.\par
COLUMNS: \myrowTABcells\par
---

\verb|\readMANYrows| Input: \verb|{a&b&c&d\\b\\\\ c}|
\readMANYrows{mydata}{a&b&c&d\\b\\\\ c}\par
.\detokenize\expandafter{\mydataXi}.(detokenized)\par
.\mydataXii.\par
.\mydataXiii.\par
.\mydataXiv.\par
ROWS: \mydataROWs\par
---

\verb|\readTABrow| Input: \verb|\mydataXi|
\readTABrow{mydataXi}{\mydataXi}\par
.\detokenize\expandafter{\mydataXiXi}.(detokenized)\par
COLUMNS:\mydataXiTABcells\par

\end{document}

The output, given below, shows individual cases of \readTABrow and \readMANYrows working as I expected, while the last example shows that \readTABrow, when passed a token produced by \readMANYrows, erroneously determines the token to be & free, composed of a single cell, rather than being composed of 4 cells

Note that the use of \detokenize is merely as a debugging aid to display the contents of various macros that contain & characters. No \detokenized data is passed to \readTABrow.

On a related side note, is it traditional when parsing & and \\ markers to ignore the spaces following them or not? As you see from my MWE, I do not ignore them.

UPDATE:

David Carlisle solved the problem, though he still isn't quite sure what I want with it. So, let me show it. If the last lines of the MWE are replaced with

\verb|\readTABrow| Input: \verb|\mydataXi|
\def\foo{\readTABrow{mydataXi}}
\expandafter\foo\expandafter{\mydataXi}\par
.\mydataXiXi.\par
.\mydataXiXii.\par
.\mydataXiXiii.\par
.\mydataXiXiv.\par
COLUMNS:\mydataXiTABcells\par

the final output becomes

Thus, with David's help, I can use my two routines to take a tabular like argument like

{a&b&c\\d&e&f\\g&h&i}

and store the table components as if I had typed elements of an array:

\edef\mydataXiXi{a}
\edef\mydataXiXii{b}
\edef\mydataXiXiii{c}
\edef\mydataXiiXi{d}
\edef\mydataXiiXii{e}
\edef\mydataXiiXiii{f}
\edef\mydataXiiiXi{g}
\edef\mydataXiiiXii{h}
\edef\mydataXiiiXiii{i}

which is amenable to accessing via \csname mydataX\roman{row}X\roman{col}\endcsname

This will be useful when the output associated with the argument must be split up (horizontally across pages) or spliced (merged with other data), or if individual cells need to be later recalled.

Answer 1

I'm still unclear what output you need, but in the last call you are not passing a & separated list, you are passing a macro that expands to such a list, so you need to expand it first.

\def\foo{\readTABrow{mydataXi}}
\expandafter\foo\expandafter{\mydataXi}\par
.\detokenize\expandafter{\mydataXiXi}.(detokenized)\par
COLUMNS:\mydataXiTABcells\par

produces COLUMNS:4 rather than COLUMNS:1

As for the related question, if you are trying to emulate the normal TeX \halign behaviour, spaces are not dropped by the parser, so space tokens are generated, but LaTeX always avoids typesetting any white space resulting from such as each cell preamble starts with \ignorespaces and ends with \unskip if it's in hmode.

Answer 2

I don't like to reinvent the wheel; with LaTeX3 macros it's quite easy. Note the *-variant for expanding the argument if given as a control sequence expanding to the required token list. It could be dispensed with if you have full control over the tokens that can appear in your input, that is, if they are only unexpandable tokens (but one has to be careful with a leading \\, of course).

There's some code duplication, so the macros could be simplified.

\documentclass{article}
\usepackage{xparse}

\ExplSyntaxOn
\NewDocumentCommand{\readTABrow}{ s m m }
 {
  \IfBooleanTF{#1}
   { \segle_read_tab_row:no { #2 } { #3 } }
   { \segle_read_tab_row:nn { #2 } { #3 } }
 }
\NewDocumentCommand{\readMANYrows}{ s m m }
 {
  \IfBooleanTF{#1}
   { \segle_read_many_rows:no { #2 } { #3 } }
   { \segle_read_many_rows:nn { #2 } { #3 } }
 }

\seq_new:N \l_segle_row_seq
\seq_new:N \l_segle_manyrows_seq
\int_new:N \l_segle_row_int
\int_new:N \l_segle_manyrows_int
\cs_new_protected:Npn \segle_read_tab_row:nn #1 #2
 {
  \seq_set_split:Nnn \l_segle_row_seq { & } { #2 }
  \int_zero:N \l_segle_row_int
  \seq_map_inline:Nn \l_segle_row_seq
   {
    \int_incr:N \l_segle_row_int
    \cs_set:cpn { #1 X \int_to_roman:n { \l_segle_row_int } } { ##1 }
   }
  \cs_set:cpx { #1 TABcells } { \int_to_arabic:n { \l_segle_row_int } }
 }
\cs_generate_variant:Nn \segle_read_tab_row:nn { no }

\cs_new_protected:Npn \segle_read_many_rows:nn #1 #2
 {
  \seq_set_split:Nnn \l_segle_manyrows_seq { \\ } { #2 }
  \int_zero:N \l_segle_manyrows_int
  \seq_map_inline:Nn \l_segle_manyrows_seq
   {
    \int_incr:N \l_segle_manyrows_int
    \cs_set:cpn { #1 X \int_to_roman:n { \l_segle_manyrows_int } } { ##1 }
   }
  \cs_set:cpx { #1 ROWs } { \int_to_arabic:n { \l_segle_manyrows_int } }
 }
\cs_generate_variant:Nn \segle_read_many_rows:nn { no }

\ExplSyntaxOff

\begin{document}
\verb|\readTABrow| Input: \verb|{a&b&c&d}|
\readTABrow{myrow}{a&b&c&d}\par
.\myrowXi.\par
.\myrowXii.\par
.\myrowXiii.\par
.\myrowXiv.\par
COLUMNS: \myrowTABcells\par

\verb|\readMANYrows| Input: \verb|{a&b&c&d\\b\\\\ c}|
\readMANYrows{mydata}{a&b&c&d\\b\\\\ c}\par
.\detokenize\expandafter{\mydataXi}.\par
.\mydataXii.\par
.\mydataXiii.\par
.\mydataXiv.\par
ROWS: \mydataROWs\par

---

\verb|\readTABrow| Input: \verb|\mydataXi|
\readTABrow*{mydataXi}{\mydataXi}\par
.\detokenize\expandafter{\mydataXiXi}.(detokenized)\par
COLUMNS:\mydataXiTABcells\par
\end{document}