Read file contents to variable and iterate over each character in file (hexdump)

Question

I need to read file content as string (binary file) to variable, then iterate over each character in that string, convert each character to its hexadecimal representation, use comma (or any other distinguishable character) as character separator and embed the result in my actual LaTeX file where I'm doing all that stuff.

In python I have something like this (4 lines!):

fileContent = ""
with open("binary.so", "rb") as f:
    fileContent = f.read()
print( ",".join( map(lambda x: str(ord(x)), fileContent) ) )

Now the hard part:

• no extra packages = only tex primitives
• defined as tex macro: \readfile{ input_file_here }
• must work with pdflatex
• file might be empty
• newlines and other special characters must be preserved (it is binary!)
• I need a check if it is really a file and not a directory and if I'm allowed to read such files
• should work on Linux/Unix and Windows (actually I need only for Linux, but more general solution would be nice)
• no extra definitions in preamble
• no luatex, no immediate, so no shell and no extra tools

Because this is a binary file (I need to treat all files like this) we have character values between 0 (0x00) and 255 (0xFF). Numbers might be HEX or just decimal. base64 encoding would be OK too, but without external tools and extra flags like --shell-escape.

\readfile{binary.so} % will return something like "D,E,A,D,B,E,E,F,1,3,3,7" or "0x0D,0x0E,..." or "13,14,..."

---

I tried something like this and it didn't work: (example taken from here)

\newread\makerfile
\openin\makerfile=binary.so
\ifeof\makerfile
\else
    % it looks promising, but it shouldn't read by newlines
    \read\makerfile to\makerline
    \closein\makerfile
    % iterate over each character
    % ...
\fi

E.g. this works, but I cannot iterate over characters, because I don't know how

\makeatletter
\newcommand\saferead[1] {
  \bgroup
  \let\do\@makeother
  \dospecials
  \catcode`\ =10 % spaces
  \catcode`\^^M=\active % newlines
  \input{#1}
  \egroup 
}
\makeatother

I googled like crazy and found only some useful bits but I'm still far away from what I want.

Answer 1

As Heiko Oberdiek points out in this answer, pdfTeX defines a new, expandable primitive \pdffiledump which can be used to read files in binary mode. The syntax of the command is

\pdffiledump offset 0 length <length>{<filename>}

where for <length> we can use another primitive \pdffilesize{<filename>}. The result is a sequence of pairs XX, where XX is the hex representation of each character in the input file. The rest of the processing is similar to the answer below, beside we don't need the extra hex conversion.

\documentclass{article}

\makeatletter

\def\showbinary#1{%
    \begingroup
    \xdef\@temp{\pdffiledump offset 0 length \pdffilesize{#1}{#1}}%
    \expandafter\analyze\expandafter{\@temp}%
    \endgroup
}

\def\analyze#1{%
    \count@=0
    \if\relax\detokenize{#1}\relax\else
        \expandafter\analyze@#1\@end
    \fi
}
\def\analyze@#1#2#3\@end{%
    #1#2
    \advance\count@ by 1
    \ifnum\count@>15
        \count@=0
        \par
    \fi
%
    \let\@next=\relax
    \if\relax\detokenize{#3}\relax\else
        \def\@next{\analyze@#3\@end}%
    \fi
    \@next
}

\makeatother

\begin{document}
\ttfamily
\showbinary{ascii.txt}
\end{document}

outputs

---

Old answer

Not a complete answer, but this is the best I could come up with for reading a binary file:

\documentclass{article}

\makeatletter

\def\showbinary#1{%
    \begingroup
    \count@=0
    \loop
        \catcode\count@=12
        \advance\count@ by 1
    \ifnum\count@<256
    \repeat
%
    \endlinechar=-1
    \everyeof{\noexpand}%
    \xdef\@temp{\@@input #1 }%
%
    \analyze\@temp
    \endgroup
}

\def\analyze#1{%
    \expandafter\analyze@#1\@end
}
\def\analyze@#1#2\@end{%
    \count@=`#1\relax
    \expandafter\hex\expandafter{\the\count@}
    \let\@next=\relax
    \if\relax\detokenize{#2}\relax\else
        \def\@next{\analyze@#2\@end}%
    \fi
    \@next
}

\def\hex#1{%
    \begingroup
    \count@=#1\relax
    \divide\count@ by 16
    \hexchar\count@
%
    \multiply\count@ by 16
    \advance\count@ by -#1\relax
    \multiply\count@ by -1
    \hexchar\count@
    \ifnum\count@=15\par\fi
    \endgroup
}
\def\hexchar#1{%
    \ifcase#10\or1\or2\or3\or4\or5\or6\or7\or8\or9\or A\or B\or C\or D\or E\or F\else x\fi
}

\makeatother

\begin{document}
\ttfamily
\showbinary{ascii.txt}
\end{document}

outputs

ascii.txt is a binary file including all characters from 0x00 to 0xFF. First all those characters are set to catcode 12 (other), then the file is \input'ed and its contents stored in a macro \@temp. Afterwards we iterate over each character in \@temp to output its hex representation.

As you can see, three characters are missing: 0x09 (\t), 0x0A (\n) and 0x0D (\r). The latter two are likely because TeX files are read in text mode and not in binary mode. Not sure if something can be done about that. The tab character is missing in this particular test file because TeX treats tabs like spaces when they occur at the end of a line (the \t is immediately followed by \n) and thus removes it from the input line.

Answer 2

You're using LaTeX, not Plain, so it doesn't make much sense to not use packages. With a bit of expl3 code you can make yourself a proper hexdump of a file.

My previous answer (see edit history) used a rather simple expl3 code to read in the file and make a hexdump out of it. However the code was rather slow (it took about 60 seconds to produce 7 pages of hexdump of a 6 kB file).

I made a slightly optimized version (takes about half a second to process the same file :-) with a few more niceties: it's faster, it has some key-val properties to control the output, it's faster, it uses \pdf@filedump from pdftexcmds to avoid losing line feeds and spaces, and it's much faster :-)

Here it is:

\documentclass{article}
\usepackage{pdftexcmds}
\usepackage{xparse}
\ExplSyntaxOn
\cs_new_eq:Nc \__hexdump_filedump:nnn { pdf@filedump }
\cs_new_eq:Nc \__hexdump_filesize:n { pdf@filesize }
\int_new:N \l__hexdump_begin_int
\int_new:N \l__hexdump_bytes_int
\int_new:N \l__hexdump_filesize_int
\int_new:N \l__hexdump_byte_int
\int_new:N \l__hexdump_byte_ptr_int
\int_new:N \l__hexdump_word_int
\int_new:N \l__hexdump_word_ptr_int
\int_new:N \l__hexdump_column_int
\int_new:N \l__hexdump_column_ptr_int
\int_new:N \l__hexdump_line_length_int
\int_new:N \l__hexdump_address_size_int
\int_new:N \l__hexdump_address_int
\bool_new:N \l__hexdump_address_bool
\tl_new:N \l__hexdump_dump_tl
\tl_new:N \l__hexdump_font_tl
\tl_new:N \l__hexdump_visible_tl
\clist_new:N \l__hexdump_cols_clist
\seq_new:N \l__hexdump_cols_seq
\cs_generate_variant:Nn \str_count:n { f }
\keys_define:nn { hexdump }
  {
    , begin   .int_set:N   = \l__hexdump_begin_int
    , begin   .initial:n   = { 0 }
    , length  .int_set:N   = \l__hexdump_bytes_int
    , length  .initial:n   = { -1 }
    , byte    .int_set:N   = \l__hexdump_byte_int
    , byte    .initial:n   = { 2 }
    , columns .clist_set:N = \l__hexdump_cols_clist
    , columns .initial:n   = { 4, 4 }
    , font    .tl_set:N    = \l__hexdump_font_tl
    , font    .initial:n   = \ttfamily
  }
\NewDocumentCommand \hexdump { o m }
  {
    \group_begin:
      \IfValueT {#1} { \keys_set:nn { hexdump } {#1} }
      \hexdump:n {#2}
    \group_end:
  }
\cs_new_protected:Npn \hexdump:n #1
  {
    \file_if_exist:nTF {#1}
      { \__hexdump_read:n {#1} }
      { \msg_error:nnn { hexdump } { file-not-found } {#1} }
  }
\cs_new_protected:Npn \__hexdump_read:n #1
  {
    \int_set:Nn \l__hexdump_filesize_int { \__hexdump_filesize:n {#1} }
    \__hexdump_assert_int:Nnn \l__hexdump_begin_int
      { \c_zero_int } { \l__hexdump_filesize_int }
    \int_compare:nNnT { \l__hexdump_bytes_int } = { -1 }
      { \int_set:Nn \l__hexdump_bytes_int { \l__hexdump_filesize_int } }
      {
        \__hexdump_assert_int:Nnn \l__hexdump_bytes_int
          { \c_zero_int } { \l__hexdump_filesize_int }
      }
    \tl_set:Nx \l__hexdump_dump_tl
      {
        \__hexdump_filedump:nnn
          { \l__hexdump_begin_int } { \l__hexdump_bytes_int }
          {#1}
      }
    \tl_map_function:nN { \. \? \! \: \; \, } \__hexdump_french_spacing:N
    \tl_use:N \l__hexdump_font_tl
    \__hexdump:N \l__hexdump_dump_tl
  }
\cs_new_protected:Npn \__hexdump_french_spacing:N #1
  { \char_set_sfcode:nn { `#1 } { 1000 } }
\cs_new_protected:Npn \__hexdump_assert_int:Nnn #1 #2 #3
  { \int_set:Nn #1 { \int_min:nn { \int_max:nn { #1 } { #2 } } { #3 } } }
\msg_new:nnn { hexdump } { file-not-found }
  { File~`#1'~not~found. }
\cs_new_protected:Npn \__hexdump:N #1
  {
    \__hexdump_initialise:
    \exp_last_unbraced:NV \__hexdump:NNw #1
      \q_recursion_tail \q_recursion_tail \q_recursion_stop
  }
\cs_new_protected:Npn \__hexdump_initialise:
  {
    \seq_set_from_clist:NN \l__hexdump_cols_seq \l__hexdump_cols_clist
    \int_set:Nn \l__hexdump_word_int { \seq_item:Nn \l__hexdump_cols_seq { 1 } }
    \int_set:Nn \l__hexdump_column_int { \seq_count:N \l__hexdump_cols_seq }
    \int_set:Nn \l__hexdump_address_size_int
      { \str_count:f { \int_to_hex:n { \l__hexdump_bytes_int } } }
    \int_set_eq:NN \l__hexdump_address_int \l__hexdump_begin_int
    \int_set:Nn \l__hexdump_line_length_int
      { \l__hexdump_byte_int * ( \seq_use:Nn \l__hexdump_cols_seq { + } ) }
    \exp_args:NNf \seq_put_right:Nn \l__hexdump_cols_seq
      { \seq_item:Nn \l__hexdump_cols_seq { 1 } }
    \bool_set_true:N \l__hexdump_address_bool
    \int_zero:N \l__hexdump_byte_ptr_int
    \int_zero:N \l__hexdump_word_ptr_int
    \int_zero:N \l__hexdump_column_ptr_int
  }
\cs_new_protected:Npn \__hexdump:NNw #1 #2
  {
    \quark_if_recursion_tail_stop_do:Nn #1
      { \__hexdump_end: }
    \bool_if:NT \l__hexdump_address_bool { \__hexdump_address: }
    #1 #2
    \tl_put_right:Nx \l__hexdump_visible_tl
      {
        \__hexdump_if_visible_ascii:nTF { "#1#2 }
          { \char_generate:nn { "#1#2 } { 12 } }
          { . }
      }
    \__hexdump_ptr_check:
    \__hexdump:NNw
  }
\cs_new_protected:Npn \__hexdump_ptr_check:
  {
    \__hexdump_ptr_step:nn { byte }
      {
        \c_space_tl
        \__hexdump_ptr_step:nn { word }
          {
            \int_set:Nn \l__hexdump_word_int
              {
                \seq_item:Nn \l__hexdump_cols_seq
                  { \l__hexdump_column_ptr_int + 2 }
              }
            \c_space_tl
            \__hexdump_ptr_step:nn { column }
              { \tex_unskip:D \__hexdump_dump_visible: }
          }
      }
  }
\cs_new_protected:Npn \__hexdump_ptr_step:nn #1 #2
  {
    \int_incr:c { l__hexdump_#1_ptr_int }
    \int_compare:nNnT
        { \int_use:c { l__hexdump_#1_ptr_int } }
          =
        { \int_use:c { l__hexdump_#1_int } }
      {
        \int_zero:c { l__hexdump_#1_ptr_int }
        #2
      }
  }
\prg_new_protected_conditional:Npnn \__hexdump_if_visible_ascii:n #1 { TF }
  {
    \int_compare:nNnTF {#1} > {31}
      {
        \int_compare:nNnTF {#1} < {127}
          { \prg_return_true: }
          { \prg_return_false: }
      }
      { \prg_return_false: }
  }
\cs_new_protected:Npn \__hexdump_address:
  {
    \bool_set_false:N \l__hexdump_address_bool
    \exp_args:Nf \__hexdump_address:nn
      { \str_count:f { \int_to_hex:n { \l__hexdump_address_int } } }
      { \l__hexdump_address_size_int }
    \int_add:Nn \l__hexdump_address_int { \l__hexdump_line_length_int }
  }
\cs_new_protected:Npn \__hexdump_address:nn #1 #2
  {
    \prg_replicate:nn { #2 - #1 } { 0 }
    \int_to_hex:n { \l__hexdump_address_int } : ~
  }
\cs_new_protected:Npn \__hexdump_dump_visible:
  {
    | \tl_use:N \l__hexdump_visible_tl |
    \tl_clear:N \l__hexdump_visible_tl
    \bool_set_true:N \l__hexdump_address_bool
    \tex_par:D
  }
\cs_new_protected:Npn \__hexdump_end:
  {
    \bool_if:NF \l__hexdump_address_bool
      {
        \c_space_tl \c_space_tl
        \tl_put_right:Nn \l__hexdump_visible_tl { ~ }
        \__hexdump_ptr_check:
        \__hexdump_end:
      }
  }
\ExplSyntaxOff
\begin{document}
\hexdump{somebinary.file}
\end{document}

Visible bytes (ASCII 32 – 126) are printed, and everything else is represented by a . in the right pane:

Answer 3

Here's an expandable version (using two “forbidden functions”), using ideas of siracusa.

\documentclass{article}
\usepackage{xparse}

\ExplSyntaxOn

\NewExpandableDocumentCommand{\hexdump}{O{~}m}
 {
  \awa_hexdump:ne {#1} { \tex_filedump:D~offset~0~length~\tex_filesize:D{#2}{#2} }
 }
% there's not yet an official interface to \pdffiledump and \filesize

\cs_new:Nn \awa_hexdump:nn
 {
  \__awa_hexdump_read_byte:nNNN {#1} #2 \q_nil \q_stop
 }
\cs_generate_variant:Nn \awa_hexdump:nn { ne }

\cs_new:Nn \__awa_hexdump_read_byte:nNNN
 {
  \quark_if_nil:nTF { #4 }
   % true: print the last two digits and ignores the trailer
   { #2#3 \use_none:n }
   % false: print two digits, a comma and some space
   { #2#3#1 \__awa_hexdump_read_byte:nNNN { #1 } #3 }
 }

\ExplSyntaxOff

\begin{document}

\raggedright\ttfamily
\hexdump{cmr10.tfm}

\hexdump[,\hspace{0pt plus 1fill}]{\jobname.tex}

\end{document}

I used a copy of the standard cmr10.tfm file. The optional argument (default a space) is for controlling the delimiter between two bytes.

The picture shows the last two lines from the first call and the first two lines from the second call.

A check for the existence of the file can be easily added.