Why \NewDocumentEnvironment within \ExplSyntaxOn/Off removes spaces from content passed to it outside of \ExplSyntaxOn/Off?

Question

I thought turning syntax on/off at appropriate places would make everything work, but I can't use \ExplSyntaxOn/Off inside \NewDocumentEnvironment so the following code sample doesn't work:

\documentclass[varwidth]{standalone}
\usepackage{expl3}
\usepackage{xparse}
\begin{document}
  \NewDocumentEnvironment{myPar}{}
    {
      \parbox{40mm}
        {% START PARBOX
          \ExplSyntaxOn
          \dim_new:N\myDim % DOESN'T WORK
          \ExplSyntaxOff
    }
    {
        } % END PARBOX
    }
  \begin{myPar}
    Wikipedia is hosted by the Wikimedia Foundation, a non-profit organization that also hosts a range of other projects.
  \end{myPar}
\end{document}

So I have to enclose the environment within syntax switches, as follows, but then anything passed to the environment loses its spaces:

\documentclass[varwidth]{standalone}
\usepackage{expl3}
\usepackage{xparse}
\begin{document}
\ExplSyntaxOn % BROUGHT OUTSIDE
  \NewDocumentEnvironment{myPar}{}
    {
      \parbox{40mm}
        {% START PARBOX
          \dim_new:N\myDim % WORKS (BUT NO SPACES REMAIN)
    }
    {
        } % END PARBOX
    }
\ExplSyntaxOff % BROUGHT OUTSIDE
  % SPACES OF THE FOLLOWING CONTENT ARE STRIPPED
  \begin{myPar}
    Wikipedia is hosted by the Wikimedia Foundation, a non-profit organization that also hosts a range of other projects.
  \end{myPar}
\end{document}

How do I retain spaces of the passed content without replacing them with ~ symbols?

By the way, following doesn't strip spaces (but lacks \parbox as I need it):

\documentclass[varwidth]{standalone}
\usepackage{expl3}
\usepackage{xparse}
\begin{document}
\ExplSyntaxOn % BROUGHT OUTSIDE
  \NewDocumentEnvironment{myPar}{}
    {
      \dim_new:N\myDim % WORKS (NO SPACES STRIPPED BUT NO PARBOX)
    } {}
\ExplSyntaxOff % BROUGHT OUTSIDE
  % NO SPACES STRIPPED FROM THE FOLLOWING CONTENT (BUT NO PARBOX)
  \begin{myPar}
    Wikipedia is hosted by the Wikimedia Foundation, a non-profit organization that also hosts a range of other projects.
  \end{myPar}
\end{document}

Answer 1

You are tricking yourself with improper indentation. Remember that TeX does not care about indentation; any closing brace will end the group started by the last opening brace. Let's look at the environment definition in your second code snippet with the correct indentation (i.e. indentation that reflects how TeX reads your code).

\ExplSyntaxOn % BROUGHT OUTSIDE
    \NewDocumentEnvironment{myPar}{}
        {
            \parbox{40mm}
            {% START PARBOX
                \dim_new:N\myDim % WORKS (BUT NO SPACES REMAIN)
            }
            {
            } % END PARBOX
        }
        \ExplSyntaxOff % BROUGHT OUTSIDE

As you can see, your \parbox ends right after the \dim_new:N line, then there is an empty group and then the group of the environment start code ends. Importantly there is no environment end code before \ExplSyntaxOff, which means that \ExplSyntaxOff itself is taken as the environment end code.

Thus, expl3 syntax stays on, which causes all the spaces to be gobbled (among other havoc). This is not isolated to your environment. Add any text outside of \begin{myPar} .. \end{mypar} and you will see the same problems. This is even true after the environment because the \ExplSyntaxOff at the end of the environment is confined to the environment group and has no effect outside of it.

As Ulrike Fischer pointed out in her comment, using \dim_new:N\myDim inside the environment is also problematic. You will get an error message telling you that \myDim is already defined once you use the environment a second time. Define \myDim once in the preamble and then only use it inside the environment.

You can avoid your grouping problem by using the b-type argument as demonstrated in Henri Menke's answer or just using a minipage instead of a \parbox:

\ExplSyntaxOn
    \NewDocumentEnvironment{myPar}{}{
        \begin{minipage}{40mm}
            % Don't do \dim_new:N\myDim here!
    }{
        \end{minipage}
    }
\ExplSyntaxOff

Answer 2

1. Unbalanced token lists

Here's what you're trying to do is, in traditional terms,

\newenvironment{myPar}
  {%
   \parbox{40mm}{% start \parbox
  }
  {%
   }% end \parbox
  }

This is wrong code to begin with. Written in a different way to better see what happens

\newenvironment{myPar}{\parbox{40mm}{}{}}

This is unfinished, because it misses the third mandatory argument to \newenvironment.

It is not possible to have a brace unbalanced token list in the argument to a macro. The } that you mean to balance the { before \parbox is instead matching the } after \parbox. Chaos ensues.

2. Internal syntax markers

Whoever wants to delve into LaTeX programming should know about tokenization. When input is absorbed by TeX, it is transformed into tokens: symbolic tokens (vulgo, commands) and character tokens. The latter have their category code permanently attached. So if you say

\newcommand{\foo}{\makeatletter @\makeatother}

you get @ to have the category code it has when \newcommand is performed. So if you do

\newcommand{\foo}{\makeatletter\dimen@=2\p@\makeatother}

upon execution of \foo you'll receive puzzling error messages, unless the definition is surrounded by \makeatletter and \makeatother itself. But then the ones inside the definition do nothing at all. See What do \makeatletter and \makeatother do? for more information.

It is exactly the same with \ExplSyntaxOn and \ExplSyntaxOff.

What happens in your second attempt, when bringing them outside the code for myPar is that, due to the mistake described in chapter 1, \ExplSyntaxOff is taken as the “end part” of the myPar environment, so it is not executed and \ExplSyntaxOn remains in force when reading the text following \begin{myPar}.

3. Miscellaneous

Doing \dim_new:N as part of the environment is wrong no matter what. Doing \newlength inside an environment would be wrong as well, although it doesn't raise errors. Let's see with an example

\newenvironment{foo}{\newlength{\foolen}}{}

will dutifully create and waste a new skip register at each call of \begin{foo}. This is because the name assignment is local, but the register assignment is global as LaTeX doesn't want to take the risk of referring to the same register in different ways.

With expl3 the things are a bit different: with

\NewDocumentEnvironment{foo}{}
 {\dim_new:N \myDim}
 {}

(properly surrounded by \ExplSyntaxOn and \ExplSyntaxOff), also the name assignment is global. Thus upon calling \begin{foo} a second time LaTeX will raise an error.

4. How to make a parbox with an environment?

Use minipage, of course.

Answer 3

Use the new b-type argument which captures the body of an environment for use like a regular argument #1. The +b allows multiple paragraphs.

\documentclass[varwidth]{standalone}
\usepackage{xparse}

\ExplSyntaxOn

\NewDocumentEnvironment { myPar } { +b }
  {
      \parbox { 40mm }
        {
          \dim_new:N \myDim
          #1
        }
  } { }

\ExplSyntaxOff

\begin{document}
\begin{myPar}
  Wikipedia is hosted by the Wikimedia Foundation, a non-profit organization that also hosts a range of other projects.
\end{myPar}
\end{document}