Command delimited by return or blank line

Question

I have this document

\documentclass{scrartcl}
\long\def\quotenextparagraph#1
{\begin{quote}#1\end{quote}}
\begin{document}
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat
non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
\quotenextparagraph
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat
non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat
non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
\end{document}

The command \quotenextparagraph puts its argument (which is delimited by a blank line) in a quote environment. One may suggest using

\long\def\quotenextparagraph#1\par{\begin{quote}#1\end{quote}}

but with that definition you loose the hability to include paragraph breaks (without little tricks) inside the argument. With the blank line in the definition you can use

\quotenextparagraph
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
consequat.\par
Duis aute irure dolor in reprehenderit in voluptate velit esse
cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat
non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

However… you need to leave no space before the \par token. If you leave a space, then LaTeX understands that the argument is delimited there.

First question: why is that?

And now the second part of the question. I'm interested in doing the same thing with only one return (I don't know the exact word; I mean “blank line” = two returns).

\def\quotethisline#1
{\begin{quote}#1\end{quote}}

so that

\quotethisline Lorem ipsum dolor sit amet.
Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut
aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in
voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint
occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit
anim id est laborum.

gives only Lorem ipsum dolor sit amet. in the quote. As I expected, it only gives Lorem (until the first space).

Second question: why does this happen?

Third question: how can I achieve the argument delimited by only one return? (preferably without touching catcodes, if possible)

PS: I might know why, but I think it's worth to have an answer explaining it.

egreg · Accepted Answer · 2014-07-13T14:29:58.797

In the first definition, the argument is delimited by a space and a \par token:

\long\def\testA#1

{\begin{quote}#1\end{quote}}

\long\def\testB#1 \par{\begin{quote}#1\end{quote}}

\ifx\testA\testB\message{EQUAL}\else\message{DIFFERENT}\fi

prints

EQUAL

The first end-of-line in \testA is converted to a space token; the second one is converted to a \par token, because it is scanned when TeX is in its “skipping blanks” state.

The definition

\def\quotethisline#1
{\begin{quote}#1\end{quote}}

is fully equivalent to

\def\quotethisline#1 {\begin{quote}#1\end{quote}}

because the tokenization converts the end-of-line to a space token.

You can't delimit a string by just an end-of-line without category code juggling, because tokenization is performed before macro expansion.

Also, TeX reads a full line and replaces the operating system end-of-record separator with \endlinechar before tokenization, which gives even more problems.

One can do like this, which has minimal impact on category codes:

\documentclass{scrartcl}

\catcode`\^^L=\active
\def^^L{\end{quote}\endlinechar=13 }
\def\quotenextline{%
  \begin{quote}
  \endlinechar`\^^L
}

\begin{document}

Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut
aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in
voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint
occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit
anim id est laborum.
\quotenextline
Lorem ipsum dolor sit amet.
Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut
aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in
voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint
occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit
anim id est laborum.

\end{document}

However, this is just a plaything: never use the end of line as delimiter, unless you're in a very controlled environment (verbatim mode, for instance).

This is another solution, copied from section 11.9.4 of TeX by Topic:

\documentclass{scrartcl}

\def\quotethisline{\begingroup\catcode`\^^M=12 \xquotethisline}
{\catcode`\^^M=12 %
 \gdef\xquotethisline#1^^M{\begin{quote}#1\end{quote}\endgroup}%
}

\begin{document}

Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut
aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in
voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint
occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit
anim id est laborum.
\quotethisline Lorem ipsum dolor sit amet.
Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut
aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in
voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint
occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit
anim id est laborum.

\end{document}

In the “More later.” could you include a little about why are those end of lines converted before being swallowed by the command as a delimiter? — Manuel, Jul 13 '14 at 12:22
@Manuel I did; basically you can't do it without category code changes. And it's surely something I wouldn't recommend, because it's bound to fail more often than you think: just opening the file with a different editor might insert end-of-lines at a fixed column. — egreg, Jul 13 '14 at 12:24
So there is no clear way of adding a solution even with category code changing involved? — Manuel, Jul 13 '14 at 12:26
I just saw this solution. However it's just too localized (I mean, you define the end of line as the end of the quote). I thought more like introducing a special token in the end of line, recognizable by a delimiter argument but letting it be almost the same (in a similar way of \*DOTS from amsmath, which does nothing but is there so other commands can see it). — Manuel, Jul 13 '14 at 13:08
That last one is great :) I will leave this open a little while in case other people come here with another solution. — Manuel, Jul 13 '14 at 17:29

wipet · Answer 2 · 2014-07-15T04:36:23.813

TeX processes its input using several processors:

input processor -> token processor -> expand processor -> main processor

A gastro-anatomical terminology is mentioned in the TeXbook for this but I never use this terminology.

The Input processor reads lines from file, appends \endlinechar and (maybe) re-encodes the line and outputs each line as a buffer to the token processor. The token processor reads the buffers and generates sequence of tokens. The main reason of the token processor is (roughly speaking):

to pack up each \csname as one token.
to merge double (or more) spaces to one space token.
to replace end of buffer (i.e. end of line) to space token.
to generate \par token instead of each (visual) empty line.
to assign a category (catcode) to each generated token.

There are two types of catcodes: first type is designated to the token processor itself (0 \, 5 endline, 9 ignored char, 14 comment char, 15 forbidden char) and second: the others catcodes. The first type of catcodes never occurs at the output side of the token processor.

The expand processor expands macros. It means that macros treat only with sequences of tokens from the token processor, the original structure of lines is invisible from this point of view (by default settings of catcodes and \endlinechar). The task to scan endline by macros is comparable to the task to scan doublespaces between words: this is impossible because double spaces don't exist at this level of processing.

The main processor takes unexpandable primitive commands (as an output from expand processor) and does: assignments or typessetting.

The processing is controlled from main processor: when main processor needs the command (or command parameter), it asks to the expand procesor. When expand processor needs the new token from the input stream, it asks to the token processor. When token processor needs the new buffer, it asks to the input processor. The minimal data is stored at each level of processors. Example:

\catcode`x=14x aaa
\catcode`X=14 X bbb

The first line sets x to the comment character (like %) but the letter x immediatelly after 14 has catcode 11 (letter) because it is touched by token processor when the parameter of the \catcode primitive was scanned. This was a time where catcode of x was 11. The second line sets X to the comment character. The parameter scanning is terminated by space, the \catcode primitive does assigmnent after this and the next token X is touched by token processor when X has catcode 14. Thus the aaa is visible but bbb is invisible (commented out).

You can set \endlinechar=something but you have to take into account that this endlinechar is appended firstly to the next line after the line already read. This is the reason why the first solution by egreg places the \quotenextline before the scanned line. His second solution utilizes the fact that default \endlinechar is 13 (i.e. ^^M character). Its cactode is set by default to 5 (i.e. replace it to space token and stop the reading of the current linebuffer) and it is replaced to the active character which does some work and terminates the group.

You can use similar technique for \quotenextparagraph too:

\def\quotethisparagraph {\bgroup 
    ... do some starting work ...
    \def\par{\endgraf ... do some ending work ... \egroup}%
}

This technique is better beause it doesn't scan text to the parameter. Thus catcode changing (like \verb|...|) will work inside this environment. You can tell to the user (in the documentation about \quotethisparagraph) that if more paragraphs are expected inside this environemt then the \endgraf instead \par or instead empty line have to be written.

Nice explanation. It comes to my mind a different (and unrelated) question: why (or why not) use \bgroup or \begingroup. We have a question here, but the answer (which, to my surprise, has 60 upvotes) is not enough (in my opinion). In case you want to answer that :) — Manuel, Jul 14 '14 at 14:48

score 3 · Answer 3 · answered Jul 14 '14 at 14:57

Here's just a “semi-generalization” (with expl3) of the solution provided by egreg. I tried exactly that before asking this question (just touching catcodes of ^^M and using an auxiliar command, inspired by the definition of \obeylines) but, unfortunately for me, it didn't work (I need more training with catcodes and groups and where to change catcodes and where not to).

It's not quite important, but just automatises this process a little bit.

\documentclass{scrartcl}
\usepackage{xparse}

\ExplSyntaxOn
\group_begin:
  \char_set_catcode_other:N \^^M %
  \cs_new:Npn \cs_new_delimited_by_newline:Nn #1 #2 %
    { %
      \cs_new:Npn #1 %
        { %
          \group_begin: %
            \char_set_catcode_other:N \^^M %
            \use:c { __ \cs_to_str:N #1 } %
        } %
      \cs_new:cpn { __ \cs_to_str:N #1 } ##1 ^^M %
        { %
            #2 %
          \group_end: %
        } %
    } %
\group_end:
\cs_new_delimited_by_newline:Nn \quotethisline
  {
    \begin{quote} ``\emph{#1}'' \end{quote}
  }
\ExplSyntaxOff


\begin{document}
Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut
aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in
voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint
occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit
anim id est laborum.
\quotethisline Lorem ipsum dolore sit amet.
Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut
aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in
voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint
occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit
anim id est laborum.
\end{document}

enter image description here

Command delimited by return or blank line

First question: why is that?

Second question: why does this happen?

Third question: how can I achieve the argument delimited by only one return? (preferably without touching catcodes, if possible)

3 Answers3

Linked