Your smurf-environment (without changing the category-code-régime) looks for an optional argument/looks for the presence of [12 and then—after "fetching" the optional argument if present—calls \verbatim.
\verbatim in turn switches to verbatim-category-code-régime and relies on following tokens getting/being tokenized under verbatim-category-code-régime.
But with your third example the opening curly brace { of {2} is not tokenized according to verbatim-category-code-régime as it already gets tokenized under normal/unchanged category-code-régime at the time of looking for the optional argument, before \verbatim gets called.
Therefore at the time of executing \verbatim you already have a {-character-token of category code 1(begin group), but due to \verbatim switching to verbatim-category-code-régime subsequent closing curly braces won't get tokenized as category-code-2(end group)-character-tokens but will get tokenized as category-code-12(other)-character-tokens. Therefore when \verbatim@start gathers its argument, it finds a {-character-token of category code 1(begin group) but it does not find a matching }-character-token of category code 2(end group). This leads to a Runaway argument?-error.
As the category code of [ under normal category-code-régime is the same as under verbatim-category-code-régime, you can implement a mechanism which—after switching to verbatim-category-code-régime (via \let\do\@makeother\dospecials\@vobeyspaces\obeylines)—does via \kernel@ifnextchar "look" for the presence of an [ indicating the presence on an optional argument.
Be aware that with this approach you cannot separate the optional argument from \begin{smurf} by using combinations of a newline and spaces any more.
I.e.,
With
\begin{smurf}[optional]
the phrase [optional] is not taken for a part of the verbatim-listing but is taken for an optional argument.
With
\begin{smurf} [optional]
the phrase ⟨space⟩[optional] is not taken for an optional argument but is taken for a part of the verbatim-listing.
With
\begin{smurf}
[optional]
the phrase [optional] is not taken for an optional argument but is taken for a part of the verbatim-listing.
\documentclass{article}
\usepackage{verbatim}
\newcommand\FetchoptionalArgAndCallVerbatim[1][]{%
The optional argument---in parentheses---is: (#1)%
\verbatim
}%
\makeatletter
\newenvironment{smurf}{%
\begingroup
\let\do\@makeother
\dospecials
\@vobeyspaces
\obeylines
\kernel@ifnextchar[%
{\endgroup\FetchoptionalArgAndCallVerbatim}%
{\endgroup\FetchoptionalArgAndCallVerbatim[]}%
}{%
\endverbatim
}
\makeatother
\begin{document}
This is okay:
\begin{smurf}
x x x x x x
\end{smurf}
bla bla bla
This is okay, too:
\begin{smurf}[2]
x x x x x x
\end{smurf}
bla bla bla
This is okay, also:
\begin{smurf}[{\LaTeX[]}]
x x x x x x
\end{smurf}
bla bla bla
This no longer causes a strange error:
\begin{smurf}{2}
x x x x x x
\end{smurf}
bla bla bla
\end{document}
The following example lets you separate the optional argument from \begin{smurf} with spaces.
But separating via a line-break is not implemented because this would remove the possibility of the first line of the verbatim-listing beginning with an opening square bracket [/this would remove the possibility of the first line of the verbatim-listing beginning with something nested in square brackets [⟨something⟩].
I.e.,
With
\begin{smurf}[optional]
the phrase [optional] is not taken for a part of the verbatim-listing but is taken for an optional argument.
With
\begin{smurf} [optional]
the phrase ⟨space⟩[optional] is not taken for a part of the verbatim-listing but [optional] is taken for an optional argument.
With
\begin{smurf}
[optional]
the phrase [optional] is not taken for an optional argument but is taken for a part of the verbatim-listing.
With
\begin{smurf}⟨space⟩
[optional]
the phrase [optional] is not taken for an optional argument but is taken for a part of the verbatim-listing.
With
\begin{smurf}
⟨space⟩[optional]
the phrase ⟨space⟩[optional] is not taken for an optional argument but is taken for a part of the verbatim-listing.
\documentclass{article}
\usepackage{verbatim}
\newcommand\FetchoptionalArgAndCallVerbatim[1][]{%
The optional argument---in parentheses---is: (#1)%
\verbatim
}%
\makeatletter
\newenvironment{smurf}{%
\begingroup
\let\do\@makeother
\dospecials
\@vobeyspaces
\obeylines
\lookaheadloop{}%
}{%
\endverbatim
}
\begingroup
% As catcode of space will be changed, don't indent the following lines:
\@vobeyspaces%
\obeylines%
\@firstofone{% brace-level 1
\endgroup%
\newcommand\lookaheadloop[1]{% brace-level 2
\kernel@ifnextchar{ }{\gatherspace{#1}}{% brace-level 3
\kernel@ifnextchar[%
{\endgroup\FetchoptionalArgAndCallVerbatim}%
{\endgroup\FetchoptionalArgAndCallVerbatim[]#1}%
}% brace-level 3
}% brace-level 2
\@ifdefinable\gatherspace{\long\def\gatherspace#1 {\lookaheadloop{#1 }}}%
}% brace-level 1
% Now the catcode of space isn't changed any more.
\makeatother
\begin{document}
Example 1:
\begin{smurf}
x x x x x x
\end{smurf}
End of example 1.
Example 2:
\begin{smurf}[2]
x x x x x x
\end{smurf}
End of example 2.
Example 3:
\begin{smurf} [{\LaTeX[]}]
x x x x x x
\end{smurf}
End of example 3.
Example 4:
\begin{smurf}[{\LaTeX[]}]
x x x x x x
\end{smurf}
End of example 4.
Example 5:
\begin{smurf}
[{\LaTeX[]}]
x x x x x x
\end{smurf}
End of example 5.
Example 6:
\begin{smurf}
[{\LaTeX[]}]
x x x x x x
\end{smurf}
End of example 6.
Example 7:
\begin{smurf}
[{\LaTeX[]}]
x x x x x x
\end{smurf}
End of example 7.
Example 8:
\begin{smurf}{2}
x x x x x x
\end{smurf}
End of example 8.
Example 9:
\begin{smurf} {2}
x x x x x x
\end{smurf}
End of example 9.
Example 10:
\begin{smurf}
{2}
x x x x x x
\end{smurf}
End of example 10.
\end{document}
