47

How does \detokenize work?

The e-TeX Short Reference Manual states:

When followed by a <general text>, expands to yield a sequence of character tokens of \catcode 10 (space) or 12 (other) corresponding to a decomposition of the tokens of the <balanced text> of the unexpanded <general text>; c.f. \showtokens. The effect is rather as if \scantokens applied to the <general text> within a régime in which only \catcodes 10 and 12 existed. Note that in order to preserve the boundaries between control words and any following letter, a space is yielded after each control word including the last.

What does that mean in plain English? I want to understand this snippet of code:

\def\task#1{\@task#1::\@nil}
\def\@task#1:#2:#3\@nil{%
  \if\relax\detokenize{#2}\relax
  ...

I think that \task is defined as a macro taking one argument which calls \@task. I don't know what \@nil means -- it is probably used to end the macro call. I also don't know what \relax and \detokenize do, but detokenize takes the second part of the argument for the task macro (the parts are separated with semicolons).

einpoklum
  • 12,311
ipavlic
  • 8,091

2 Answers2

45

\task takes one argument, passing it to \@task which is defined in such a way that its arguments are delimited; if the call is

\@task xyz:AB:cde:u\@nil

the first argument is xyz, the second is AB and the third is cde:u. Here \@nil doesn't mean anything, it's just required by the syntax of \@task and TeX throws it away.

\relax is a primitive of TeX, its function is "do nothing". The test

\if\relax\detokenize{#2}\relax

is a safe way to determine if the argument #2 is empty. If it is, \detokenize{#2} expands to nothing, so \if compares the tokens \relax and \relax, which are indeed equal, so the "true" branch is followed, which starts immediately after the second \relax, up to and excluding \else. If #2 is not empty, say it's 30, \if will compare \relax with 3 which are different, so the "false" branch is followed, which starts after \else up to and excluding \fi.[1]

The similar construct \if\relax#2\relax does not work in all cases, because "all control sequences are equal as far as \if is concerned" [2]. It would not work if #2 was \relax (or any other control sequence, possibly followed by other tokens)! So we use \detokenize that, as explained also by Joseph, splits everything into a string. So, even in the weird case that #2 is \relax, \if would compare the token \relax with the character \, which are different.

[1] This is not strictly true, but it's an approximation of the truth sufficient for the purpose of this description.

[2] TeX by Topic (section 13.2.1)

egreg
  • 1,121,712
  • 1
    Thank you again! I think that I understand it now. There are two more points that are not that clear to me:
    1. Why are arguments for \@task split like that? xyz+AB+cde or xyz+AB:cde+u or xyz+AB+cde:u all seem to satisfy ::.
    2. Will \detokenize split \relax into \ r e l a x?
     – ipavlic Jun 06 '11 at 14:50
  • 1
    @ipavlic 1. TeX takes always the shortest token list that satisfies the requests (as long as the braces are balanced) so the first argument is anything up to the first colon, the second is anything from the first to the second colon and the third is anything from the second colon to \@nil. 2. Yes. – egreg Jun 06 '11 at 14:57
  • Doesn't then the third argument has to be cde:u instead of u as written in the answer? This should not be nitpicking, I am really not sure enough to edit it. If I am wrong, please explain why. – Patrick Häcker Oct 09 '12 at 17:19
  • @MMM You're right. – egreg Oct 09 '12 at 17:28
30

The \detokenize primitive converts all of its input into category code 12 ('other') tokens, except for spaces which stay as category code 10 ('space'). It is used to convert tokens into strings. For example, if we have

\detokenize{\foo}

then \foo would be a control word, i.e. one token, which is converted into 5 string characters (\, f, o, o, ), all which are 'other' characters other than the space, which is a 'space'. Note the additional space at the end here: \detokenize adds a space after each 'control word' (see below).

I'll take your example one step at a time. First, \@nil is being used as a delimiter here. The definition of \@task means that the argument must contain :, then : again and finally \@nil. This is used to 'tidy up' awkward input

\task{stuff:more stuff}

will be turned into

 \@task stuff:more stuff::\@nil

which means #1 is stuff, #2 is more stuff and #3 is :. Thus it ensures that everything is 'used up': this is a common requirement.

To see \detokenize in action, try something like

\documentclass{article}
\usepackage[T1]{fontenc}
\edef\test{\detokenize{\test}}
\show\test
\begin{document}
\detokenize{\test}
\end{document}

A bit more on the addition of spaces by \detokenize, and indeed more generally. Whenever TeX writes something that can be one or more tokens as a 'string', it always inserts a space after each 'control word' (escape character followed by one or more 'letters') to avoid confusion. For example, with

\catcode`\X=12 % 'Other'
\def\foo{\barX}
\meaning\foo

you get 

macro:-> \bar X

where the space is not there in the original but tells us that \bar is a control sequence followed by X, rather than printing just \barX, which looks like it's a different thing. This makes sense as TeX skips spaces after control words, so the two versions are equivalent as far as it is concerned.

This 'insert spaces' behaviour is respected by \detokenize, meaning that you get a space after each control word. The only place TeX doesn't insert these spaces is with \string, which can only be used on a single token and so in places where there is no possible 'confusion'.

Joseph Wright
  • 259,911
  • 34
  • 706
  • 1,036
  • Let me see if I understand it correctly. #1 from task expands to stuff:more stuff in the command \@task stuff:more stuff::\@nil. The purpose of the added :: is to enable input without semicolons, e.g. \task{1}. Now the semicolons will be concatenated to 1 to form 1:: which will satisfy the two-semicolons requirement of \@task. – ipavlic Jun 06 '11 at 14:43
  • @ipavlic: #1 is never expanded, rather it is replaced by the input, but the result is as you say \@task stuff:more stuff::\@nil is executed. You are also right about the :: part - without this, you'd be in trouble with non-conforming input. – Joseph Wright Jun 06 '11 at 14:47
  • 1
    @ipavlic: \detokenize leaves the result in the input stream. Thus you end up with \detokenize{foo} => foo wherever it was used. In the current case, this is as part of a test, which will throw away any excess tokens. On the other hand, if you use it in a typesetting context then the material will just be typeset. See my edit – Joseph Wright Jun 06 '11 at 14:49
  • @Joseph: I guess that the word expand has a special meaning in LaTeX, so "replace" would be more fitting. If I understand correctly, depending on the context, the commands behave a bit differently -- sometimes the parts of the content are replaced, and sometimes they are typeset "as they are". I think I am beginning to understand the concept of how LaTeX works. Thank you! – ipavlic Jun 06 '11 at 14:57
  • @ipavlic: This is all TeX, not LaTeX per se. Expansion has a very precise meaning in TeX programming, and for low-level TeX questions like this one I think it's important to be careful about the terminology. – Joseph Wright Jun 06 '11 at 15:02
  • @ipavlic: On the insertion of tokens, this always happens in the same way, but what TeX then does with the tokens depends on the situation. So the resulting tokens are typeset in a 'typesetting context'. – Joseph Wright Jun 06 '11 at 15:04
  • @Joseph: I agree with you on the importance of using proper terminology. – ipavlic Jun 06 '11 at 15:23
  • 1
    So why does \detokenize{\something}\bye comes out as '"something' in the typeset output? I'd expect '\something'. – AlexG May 30 '13 at 08:07
  • 1
    @AlexG Font issue: the OT1 (Knuth original) font encoding doesn't have a \ in the appropriate slot, it has ". Assuming LaTeX, using \usepackage[T1]{fontenc} will solve this as in T1 encoding the slot is correct. – Joseph Wright May 30 '13 at 09:22
  • Thanks!!! And what is the difference between \detokenize and \string (apart from \string only working on a single token that follows)? They seem to fulfil a similar task. – AlexG May 30 '13 at 09:33
  • @AlexG Well the difference in number of tokens is pretty significant! \detokenize works in the same way as say \meaning or writing to a toks to file: spaces are inserted by TeX after each control word. On the other hand, string does not do this. Try for example \detokenize{\foo\bar\baz} and \string\foo\string\bar\string\baz to see this. The two primitives are both useful. – Joseph Wright May 30 '13 at 09:43
  • Thanks again! But why was \detokenize designed in this way? This gives rise to new problems. – AlexG May 30 '13 at 10:17
  • @AlexG You'd have to ask the NTS team for a definitive answer, but if you read the docs it's clear that \detokenize and \unexpanded are supposed to provide expandable access to things that otherwise require assignments. The way they are described, they are supposed to work the same as going via a toks, and if you do that the TeX behaviour is to add spaces. The point of that is with multiple tokens, something like \catcode`\a=12\detokenize{\fooa} would leave \fooa (looks like one control word), rather than \foo a (one control word and one token). – Joseph Wright May 30 '13 at 10:20
  • Thank you for your explanation and for your extended answer! It makes things clearer now. – AlexG May 30 '13 at 10:58