How are tokenized arguments of a macros?

Question

In the following pdftex example we make verbatim environment inside "...", i.e., all special characters get catcode 12. The problem is with {:

\def\setverb{\def\do##1{\catcode`##1=12}\dospecials}

\catcode`\"=\active
\def"#1"{%
  \leavevmode\hbox{%
    \setverb
    \tt
    \endlinechar=-1 \scantokens{#1}%
  }%
}

hello "{" world
\bye

Why it gives this error?

(./test.tex
Runaway argument?
{" world 
! Forbidden control sequence found while scanning use of ".
<inserted text> 
                \par 
<to be read again> 
                   \bye 
l.13 \bye

?

Should not it tokenize arguments until it encounters the second "?

How to make \def"#1"{...} work in all cases?

Well braces can still group arguments, e.g. \def"#1#2"{arg1: #1 arg2: #2} and "{abc}d" will give arg1: abc arg2: d. So braces must be balanced. Check the definition of \verb in latex.ltx for a solution. — Ulrike Fischer, Aug 10 '16 at 07:48

egreg · Accepted Answer · 2016-08-10T17:00:09.687

When TeX finds ", it looks for the next " at the same group level, which it doesn't find, because { still has its usual category code 1. Since " is not \long, the first \par token triggers the error.

You have to split processing in two parts: first set the category codes, then absorb the argument.

The parameter text for a macro can define both delimited or undelimited arguments, but, in any case, TeX keeps track of <left brace> and <right brace> (meaning explicit character tokens of category code 1 and 2 respectively).

So, for instance,

\def\foo (#1,#2){#1-#2}

defines a macro that must be followed by (₁₂, with the first argument delimited by ,₁₂ and the second argument delimited by )₁₂.

However, the call

\foo({(a,b)},c)

would absorb #1 as {(a,b)} (and the braces would be removed), and #2 as c.

In your case, only balanced braces would take you home.

Henri Menke · Answer 2 · 2016-08-10T11:29:22.903

3

I'd not use \scantokens and not even make " scan for an argument at all but set the appropriate catcodes in a group and let " to \endgroup (also works with Knuth TeX).

\def\setverb{\def\do##1{\catcode`##1=12}\dospecials}

\catcode`\"=\active
\def"{\begingroup\setverb\tt\let"=\endgroup}

hello "{}%~&^_$#\" world
\bye

If you want to have it in a box, use some two-stage method.

\def\setverb{\def\do##1{\catcode`##1=12}\dospecials}

\catcode`\"=\active
\def"{\begingroup\setverb\tt\doverb}
\def\doverb#1"{\hbox{#1}\endgroup}

hello "{}%~&^_$#\" world
\bye

Knuth uses something similar in manmac.tex.

\chardef\other=12
\def\ttverbatim{\begingroup
  \catcode`\\=\other
  \catcode`\{=\other
  \catcode`\}=\other
  \catcode`\$=\other
  \catcode`\&=\other
  \catcode`\#=\other
  \catcode`\%=\other
  \catcode`\~=\other
  \catcode`\_=\other
  \catcode`\^=\other
  \obeyspaces \obeylines \tt}

\catcode`\|=\active
{\obeylines \gdef|{\ttverbatim \spaceskip\ttglue \let^^M=\  \let|=\endgroup}}

edited Aug 10 '16 at 11:29

answered Aug 10 '16 at 08:34

Henri Menke

109,596

2

Which is a simplified version of what \verb does in LaTeX. – egreg Aug 10 '16 at 08:46
The reason that I use \scantokens is to be able to use active quotes in arguments to a macro. See this question http://tex.stackexchange.com/questions/317879/how-is-catcode-handled Your solution will not work in such case. – Igor Liferenko Aug 10 '16 at 08:49
@IgorLiferenko Okay, but then you can't use it to typeset verbatim content in the argument of a macro. I think that's just not possible. The unmatched brace is kind of harmless, but try forwarding %. – Henri Menke Aug 10 '16 at 08:55
@IgorLiferenko You could typeset the verbatim material at global scope and save it in a box. Then you can pass the box to a macro without having to worry. – Henri Menke Aug 10 '16 at 08:56

How are tokenized arguments of a macros?

2 Answers2