algorithm2e - `\small` interferes with `\nonl` (line number removal)

Question

I use a custom command \nonl, which removes the line number for a certain line.

• When I simply use \nonl, it works.
• When I use {\small \nonl ...}, it works.
• When I use \nonl {\small ...}, it also removes the line number for the next line.

What's the reason? I kind of expected the opposite (that {\small \nonl ...} won't work).

\documentclass{article}
\usepackage[ruled,vlined,linesnumbered]{algorithm2e}
\let\oldnl\nl
\newcommand{\nonl}{\renewcommand{\nl}{\let\nl\oldnl}}
\begin{document}
\begin{algorithm}
    Line \
    \nonl Nonl here \
    Line \
    {\small \nonl Nonl inside small here} \
    Line \textbf{where number didn't disappear} \
    \nonl {\small Nonl outside small here} \
    Line \textbf{where number disappeared} \
    Line \
\end{algorithm}
\end{document}