Why does \detokenize # sometimes fail inside of environments?

Question

When I use the pound sign/number sign/hash sign # inside of commands and environments, this refers to arguments. However, shouldn't \detokenize{#} make it NOT an argument?

Situation

The pound sign makes for nice dividers in log files.

Example

Try uncommenting the second \detokenize to get an error.

\documentclass{article}
\usepackage{fontspec}% compile with xelatex
\newenvironment{detokenizetest}[1]
{% firstoftwo
 \detokenize{##############################################################################}%
 %\detokenize{##BEGIN#######################################################################}% uncommenting this causes a compile error
}%
{% secondoftwo
}%
\begin{document}
\null
\end{document}

Error Output

This error only occurs when uncommenting the line containing BEGIN.

! Illegal parameter number in definition of \detokenizetest.
<to be read again>
                   }
l.9 }
     %
?

\newenvironment doesn't execute the code and so \detokenize can do nothing at definition time. You get an error with the second \detokenize as it has an odd number of hashes. — Ulrike Fischer, Jun 10 '16 at 09:13
@UlrikeFischer I purposely did not call the environment to demonstrate that this is a syntax problem while parsing the preamble. Thanks for the hint. — Jonathan Komar, Jun 10 '16 at 09:16

egreg · Accepted Answer · 2016-06-10T11:20:29.597

Since # has category code 6, it has a very special meaning in TeX, because it is used for denoting parameters in macro definitions.

The rules of TeX tell that when you want to store a # in the replacement text of a macro, you need to double it.

When processing the replacement text for a macro definition, a single # has to be followed by a digit (1 to 9), denoting a pointer to the corresponding argument, whereas ## will be stored as #. This is the trick that allows something like

\def\foo{\def\baz##1{-##1-}}

so the replacement text of \foo is \def\baz#1{-#1-} and a call of \foo will define \baz as a single argument macro.

The primitive \detokenize in the replacement text will do its work when the macro is called and expanded, not at definition time, unless the definition is done with \edef, which will first perform full expansion before storing the replacement text in memory.

Note that when you do \meaning\foo (assuming the above definition), you will be presented with

macro:->\def \baz ##1{-##1-}

and the same will happen with \detokenize that uses the same mechanism. Thus something like

\edef\hashmarks{\detokenize{####}}

will actually see two # in the replacement text, but the final result will be four of them. The standard mechanism when processing \edef will first half the number of # and then \detokenize will double them back. In particular, you can't produce this way an odd number of #.

If you want to produce delimiter lines, it's much better to use an indirect method, making in advance the required token list consisting of # with category code 12. In the example I use three of them, just because it's an odd number.

\documentclass{article}

\begingroup\lccode`?=`# \lowercase{\endgroup
  \newcommand{\lineofhashsigns}{???}
}

\newenvironment{delimitedtext}
  {\par\lineofhashsigns\par}
  {\par\lineofhashsigns\par}

\begin{document}

\begin{delimitedtext}
abc
\end{delimitedtext}

\end{document}

A different definition for \lineofhashsigns that doesn't require \lowercase is by changing the category code:

\catcode`#=12
\newcommand{\lineofhashsigns}{###}
\catcode`#=6

You can add whatever token in the replacement text, so long as you don't want the macro to have arguments.

Great answer! My only problem with your TeXBook trick is that BEGIN would also be lowercase if I add \newcommand{\lineofhashsignsB}{??BEGIN???}. I am sure there is a fix for that. It is not a show stopper either way. Also, I thought groups cannot overlap each other like that. I need to reread that part in the TeXBook. I found your explanation here: http://tex.stackexchange.com/a/38012/13552 for future reference. — Jonathan Komar, Jun 10 '16 at 11:15

score 6 · Answer 2 · answered Jun 10 '16 at 09:16

The \detokenize primitive will indeed mean that # is not treated as a parameter, but one when expanded. Remember that in a normal assignment (\newcommand in LaTeX, \def as a TeX primitive) no expansion takes place. Here, \newenvironment is performing exactly the same 'just store' the tokens as \newcommand. This means that we have the usual TeX rule that on creation of a macro, we have to have the matching parameters for each # present or we have to have the # tokens doubled.

As \newenvironment ultimately resolves to a \def, we can do what you want using \edef:

\edef\detokenizetest#1{%
 \detokenize{##############################################################################}%
 \detokenize{##BEGIN#######################################################################}%
}
\def\enddetokenizetest{}% As \newenvironment will do this
\show\detokenizetest

Why does \detokenize # sometimes fail inside of environments?

Situation

Example

Error Output

2 Answers2