vspace mysteriously being inserted after nested custom environment

Question

I have created my own environment that creates a table using tabu and then puts some vertical space after it, unless my environment is being nested, in which case I want it to skip the vertical space. However, the vertical space seems to be getting inserted even after a nested use of my environment. What am I doing wrong?

MWE (though I wish it was more M):

\documentclass{article}

\usepackage{tabu}
\usepackage{environ}
\usepackage{xparse}

\makeatletter

\newcommand{\lxtabu@table}[1]{
  \begin{tabu}{X}
    #1
  \end{tabu}%
}

\newcounter{lxtabu@counter}

\NewDocumentEnvironment {lxtabu} {}
{%
  \stepcounter{lxtabu@counter}
  \Collect@Body\lxtabu@table
}{%
  \typeout{ending lxtabu}%
  \addtocounter{lxtabu@counter}{-1}%
  % Only output vspace if we're not in a nested lxtabu.
  \ifnum\value{lxtabu@counter} = 0 %
    % If you comment this out the extra vertical space between lines 2
    % and 3 goes away.  (But so does the space between lines 3 and 4,
    % which is bad.)
    \typeout{inserting space}\vspace{\baselineskip}%
  \fi
}

\makeatother

\begin{document}

\begin{lxtabu}
  Line 1 \\
    \begin{lxtabu}
      Line 2 \\
    \end{lxtabu}
  \\
  Line 3, should not have big empty space above it \\
\end{lxtabu}

Line 4, should have \verb|\baselineskip| space above it.

\end{document}

Result:

My desire is that the spacing between lines 2 and 3 should be the same as between lines 1 and 2.

I'm not positive that the \vspace is being "executed" when it shouldn't be, but if I change it to \vspace{0pt} the unwanted space between lines 2 and 3 goes away. The \typeout commands I added produce the expected result, with "inserting space" only appearing once, at the end.

A few more notes:

• I'm using LuaLaTex from MacTeX 2014, which is presumably built upon TeX Live 2014.

• I'm using \Collect@Body because \begin{tabu} and \end{tabu} give errors when used inside the beginning and end, respectively, of \NewDocumentEnvironment. Same problem with just \tabu and \endtabu.

• Though not shown in the MWE, I need \NewDocumentEnvironment for its argument parsing. However, I'd accept a solution that doesn't involve \NewDocumentEnvironment if I can still declare an environment that can parse optional arguments after mandatory ones, i.e. \begin{lxtabu}{Foo}[Bar].

• My lxtabu environment needs to make sure there is space between its bottom and whatever comes next, which could be text, another lxtabu environment, or perhaps something more exotic. I don't want to have to move the \vspace outside of my lxtabu environment.

Answer 1

The mystery is readily solved: tabu processes its input twice or more. It disables some commands during the trial processing, but it forgets to do it for \stepcounter and \addtocounter. One of the commands it disables is \write, with the consequence that the \vspace is added but the message is not issued.

\documentclass{article}

\usepackage{tabu}
\usepackage{environ}
\usepackage{xparse}
\usepackage{xpatch}
\usepackage{showframe}

\makeatletter

\patchcmd{\tabu@setstrategy}
  {\let\color}
  {\let\stepcounter\@gobble\let\addtocounter\@gobbletwo\let\color}
  {}{}

\newcommand{\lxtabu@table}[1]{%
  \begin{tabu}{|X|}
    #1
  \end{tabu}%
}

\newcounter{lxtabu@counter}

\NewDocumentEnvironment {lxtabu} {}
{%
  \stepcounter{lxtabu@counter}%
  \Collect@Body\lxtabu@table
}{%
  \typeout{ending lxtabu}%
  \addtocounter{lxtabu@counter}{-1}%
  % Only output vspace if we're not in a nested lxtabu.
  \ifnum\value{lxtabu@counter} = 0 %
    % If you comment this out the extra vertical space between lines 2
    % and 3 goes away.  (But so does the space between lines 3 and 4,
    % which is bad.)
    \typeout{inserting space}\vspace{1\baselineskip}%
  \fi
  \ignorespacesafterend
}

\makeatother

\begin{document}

\begin{lxtabu}
  Line 1 \\
    \begin{lxtabu}
      Line 2 \\
    \end{lxtabu}
  \\
  Line 3, should not have big empty space above it \\
\end{lxtabu}

Line 4, should have \verb|\baselineskip| space above it.

\end{document}

I don't know whether this patch can have other consequences; I never use tabu because it has several problems and its author announced that the next version will be largely incompatible with the present one.

I added the showframe package just to show the page frames and also vertical rules to tabu for making clearer what environment is concerned. The \ignorespacesafterend is better, although possibly not needed.

Answer 2

I get the desired result by removing some of your '\\', specifically the ones after line 2 and the one on the line before line 3:

\begin{document}

\begin{lxtabu}
  Line 1\\ 
    \begin{lxtabu}
      Line 2 %\\
    \end{lxtabu}
%\\
Line 3, should not have big empty space above it \\
\end{lxtabu}

Line 4, should have \verb|\baselineskip| space above it.

\end{document}