How can I read the whole file content verbatim into a (expl3) string variable?

Question

Problem Description

As in the title.

MWE:

\documentclass{article}
\begin{document}
% first write some file content for demo...
\begin{filecontents}{test test.txt}
line 1  !?#\xyz
line 2
\end{filecontents}
% what to do here? I want to read the whole content of the file into variable \result
\ExplSyntaxOn
\str_show:N \result  % → the whole content of the file.
\ExplSyntaxOff
\end{document}

Answer 1

One option is to use \file_get:nnN. There are other options as well.

%! TEX program = pdflatex
\documentclass{article}
\begin{document}
\begin{filecontents}{test test.txt}
line 1  !?#\xyz
line 2
\end{filecontents}
\ExplSyntaxOn
\file_get:nnN {test~test.txt} {
    % alternatively (although the effects the different on LuaTeX)
    %   \int_step_inline:nnn {0} {255} {\char_set_catcode_other:n{#1}}
    \cctab_select:N \c_other_cctab
    \endlinechar=10~
    % ↑ this must be done after the cctab line
    %   because cctab changes the value of endlinechar as well
} \result
\str_set:NV \result \result
\str_show:N \result  % → the whole content of the file.
\ExplSyntaxOff
\end{document}

This saves the whole content of the file into variable \result, each "new line character" in the file will be represented with a character 10.

Explanation

\file_get:nnN {test~test.txt} {

Use the command, as explained. Note that since this is expl3 environment the space need to be written as ~.

    \cctab_select:N \c_other_cctab

Set the catcode. On XeTeX it's impractical to set for all the characters in 0..1114111, so there exists corner cases where one of the characters in the file has catcode 1/2 and make the content unbalanced thus cause an error.

Note that this command, in addition to setting the catcode table as its name suggests, it also sets the endlinechar (in this catcode table endlinechar = -1), as such we want char with charcode 10 to separate the lines we need to explicitly set it below.

    \endlinechar=10~

Set endlinechar to 10, so each new line character in the file will be represented with a character 10 _{to be precise, a token with char code 10 and catcode 12} in the resulting string.

This is expl3 environment it's good practice to specify the space explicitly to terminate the number (or use \scan_stop: / \relax but I don't like that name it's longer. There's also \int_set:Nn \endlinechar {10} but that... relies on implementation details...?)

} \result
\str_set:NV \result \result

Detokenize the result. This step is important, because in order to get the result as a string, tokens with charcode 32 (space) should have catcode 10 (space), while it would have catcode 12 (other) as above.

Limitations

• First (for XeTeX/LuaTeX engine only), if there happens to be some character with char code ≥ 256 and some unusual catcode (e.g. catcode 1 -- open brace) it might break. Most of the time the detokenization can handle it however.
• If the file to be read is a TeX file, synctex data on that file might be "lost". (the details is complex.)
• Trailing spaces on each line are stripped (and maybe trailing tabs as well).

Related questions

• related code using \readline: File copy without expansion, Read arbitrary lines from file
• read file in normal catcode regime/read using Lua: Read a file and store its contents in a variable
• pdffiledump solution: Read file contents to variable and iterate over each character in file (hexdump)
• "structured"/database-style files: File input and output

Answer 2

I propose to use \ior_str_map_variable:NNn:

\begin{filecontents*}{test.txt}
line 1  !?#\xyz
line 2
\end{filecontents*}
\documentclass{article}
\ExplSyntaxOn
\ior_new:N \g__my_ior
\str_new:N \l__my_str
\cs_new_protected:Npn \my_verb_read_into:Nn #1#2
  {
    \str_clear_new:N #1   % define #1 if necessary, then clear it
\ior_open:Nn \g__my_ior {#2}
\ior_str_map_variable:NNn \g__my_ior \l__my_str
  {
    % Precaution in case the LINE FEED char ^^J might be active
    \str_put_right:Nx #1 { \l__my_str \char_generate:nn { 10 } { 12 } }
  }
\ior_close:N \g__my_ior

}
\NewDocumentCommand \verbReadInto { m m }
  {
    \my_verb_read_into:Nn #1 {#2}
  }
\ExplSyntaxOff
\begin{document}
\verbReadInto{\result}{test.txt}
\show\result
\end{document}

which shows:

\result=macro:
->line 1  !?#\xyz
line 2
.

All lines, including the last one, are terminated by a LINE FEED character (10) with catcode 12.

Of course, one could use \g_tmpa_ior and \l_tmpa_str in \my_verb_read_into:Nn (and spare the declarations of \g__my_ior and \l__my_str), but be careful not to shoot yourself in the foot when the code grows with new functions that use the same scratch variables.

Actually, if it were not for space reasons in this tight space for the code block, I'd have included most of the function name in the names of these two variables, for the same reason.