Get \string-ification of first opening brace in argument?/Get \string-ification of first opening brace's matching closing brace in argument?

Question

Can you write a LaTeX macro where the output is the result of applying \string to the argument's first character token with category code 1 which is not a control-sequence-token and not an active character?

For example,

\catcode`\Y=1
\macro{ text text Y z{ww} z} n n nb}

should return result of \stringY, i.e. catcode-12-Y.

For example,

\catcode`\~=10
\catcode`\ =1~%
\macro{~text~text~ ~z{ww}~z}~n~n~nb}%

should return result of \string<catcode-1-space>, i.e. catcode-10-space.

Can you write a LaTeX macro where the output is the result of applying \string to character token with category code 2 which is not a control-sequence-token and not an active character and which matches the argument's first character token with category code 1 which is not a control-sequence-token and not an active character?

For example,

\catcode`\Y=2
\macro{ text text { z{ww} zY n n nb}

should return result of \stringY, i.e. catcode-12-Y.

For example,

\catcode`\~=10
\catcode`\ =2~%
\macro{~text~text~{~z{ww}~z ~n~n~nb}

should return result of \string<catcode-1-space>, i.e. catcode-10-space.

(I admit I can't.)

Answer 1

Honestly, I don't see any practical use in stringifying explicit category-2-character-tokens while preserving character-code—in the following, I do answer the question, but I assume that it is something rather "academic". The code provided by me also is not really high-performing, because iterating over many tokens of the argument takes place.

About TeX-jargon:

• TeX's eyes translate characters to TeX's internal character-representation-scheme (which with traditional TeX is ASCII (8bit) and with XeTeX/LuaTeX is unicode) and pass them to TeX's mouth where tokenization takes place. The internal character-representation-scheme can be seen as a function where the characters are the domain and the code-point-numbers are the range/codomain.
  Category codes refer to characters before tokenization, not to character-tokens. The category-code-régime can be seen as a function where the characters (determined by their code-point-number in the TeX-engine's internal character-representation-scheme) are the domain and the category codes are the codomain. Category codes in turn determine the action that shall be triggered when, in the stage of tokenization, a character of the respective category code is encountered.
• Thus, strictly spoken, character tokens don't have category codes. They have categories. And character codes. "Category" and "character code" are properties of a specific character token that got inserted into the token-stream. The values of these properties are determined in the course of tokenizing in TeX's mouth. During tokenization the category is determined by applying the catcode-régime-function to the character and "seeing" what action the character shall trigger. In many cases the action to be triggered is creating and appending to the token-stream a character-token of the respective category and with a character-code which is determined by applying the internal character-representation-scheme-function to the character. The token-stream in turn goes down TeX's gullet where in a process of regurgitation expansion of expandable tokens takes place. After that unexpandable tokens (and expandable tokens whose expansion was suppressed) reach the stomach for further processing, e.g., performing assignments, creating boxes, ... The category of a character token determines the treatment of that character token and the actions it triggers in the stage of expansion and in later stages.
• There is this "expansion-thingie" in TeX: Expanding a macro-token "returns" the macro-token's toplevel-expansion, not the final result of the expansion-cascade.
  Thus when describing desired output, please be specific about what you mean by "return", i.e., tell
  • whether the result to be "returned" is to be a sequence of tokens or to be a page of a .dvi/.pdf-output-file or to be a message on the terminal or ...
  • in case of wishing to obtain a sequence of tokens: whether expandability is required (and if so: about the amount of expansion-steps to be triggered for obtaining the tokens that form the result—with complex macro-mechanisms based on \romannumeral-expansion you need to trigger at least two expansion-steps for obtaining the result) or "side-effects" like defining temporary macros/scratch-macros are allowed.

---

Macro arguments hold explicit character tokens of category 1(begin group) in the same amount as explicit character tokens of category 2(end group).

---

"Off-the-cuff"-outline:

Both the routine for finding/stringifying the first explicit category-1-character-token and the routine for finding/stringifying the matching explicit category-2-character-token iterate and remove the first component from the argument (which either is an explicit space-token of category 10 and character code 32 or is an undelimited argument itself) until either the argument is empty or the argument has a leading explicit character token of category 1.

In case of emptiness the routines are done without returning any tokens as there are no explicit category 1/2 tokens in the argument.

In case of having reduced the argument to something that has a leading explicit character token of category 1,

• the routine for finding/stringifying the first explicit category-1-character-token hits that token with \string, then prepends another opening-brace and then extracts the first component of the first component of the argument, using an extraction-routine which takes explicit space tokens into account.

• the routine for finding/stringifying the matching explicit category-2-character-token does a loop where both the routine for finding/stringifying the first explicit category-1-character-token and the routine for extracting the first component of the argument, which takes explicit space tokens into account, are used:

  "Lay hands" on the first component of the argument (which is known not to be a space, but something nested inside a brace-group) for checking if that first component is empty.

  If the first component of the argument is empty, there is something like {}etc etc. In this case stringify the opening brace, remove it (taking into account that stringification might have yielded a space-token), hit the closing brace with \string and extract the first component from the result, using the extraction-routine which takes explicit space tokens into account. Terminate, returnig the result of the extraction.

  If the first component of the argument is not empty, there is something like {<stuff>}etc etc. In this situation you get four cases by two checks:
  Check 1: Use the routine for finding/stringifying the first explicit category-1-character-token for checking if stringification of the argument's first token yields an explicit space-token.
  Check 2: Check if the first component of the argument, which is the content of the brace-group {<stuff>}, has a leading space.
  Depending on the result of these checks: Stringify the opening brace and remove it either as a space-token or as an undelimited argument, then remove the following component either as space-token or as undelimited argument and then prepend an opening-brace. With the result do the loop again.

Here comes the code. It's a nice \expandafter-orgy. I did this off the cuff, fiddling something together quickly. The code can surely be shortened.

The names of the macros \UD@ExtractFirstOpeningBraceStringified and \UD@ExtractFirstOpeningBracesMatchingClosingBraceStringified are self-explanatory.
Due to \romannumeral-expansion the result is deliverded by triggering two expansion-steps on \UD@ExtractFirstOpeningBraceStringified/\UD@ExtractFirstOpeningBracesMatchingClosingBraceStringified.
Running the code does not require ε-TeX-extensions or Lua-extensions or the like.
The implementation does without any \if..\else..\fi-thingies (except for the "already-defined"-checks before defining macros).
There are no "sentinel-tokens" or the like whose usage in the argument would be forbidden.

\makeatletter
\errorcontextlines=10000
%%=============================================================================
%% PARAPHERNALIA:
%% \UD@firstoftwo, \UD@secondoftwo, \UD@PassFirstToSecond, \UD@Exchange,
%% \UD@removespace, \UD@stopromannumeral, \UD@CheckWhetherNull,
%% \UD@CheckWhetherLeadingExplicitSpace, \UD@CheckWhetherSpace
%%=============================================================================
\newcommand\UD@firstoftwo[2]{#1}%
\newcommand\UD@secondoftwo[2]{#2}%
\newcommand\UD@PassFirstToSecond[2]{#2{#1}}%
\newcommand\UD@Exchange[2]{#2#1}%
\@ifdefinable\UD@removespace{\UD@Exchange{ }{\def\UD@removespace}{}}%
\@ifdefinable\UD@stopromannumeral{\chardef\UD@stopromannumeral=`\^^00}%
%%-----------------------------------------------------------------------------
%% Check whether argument is empty:
%%.............................................................................
%% \UD@CheckWhetherNull{<Argument which is to be checked>}%
%%                     {<Tokens to be delivered in case that argument
%%                       which is to be checked is empty>}%
%%                     {<Tokens to be delivered in case that argument
%%                       which is to be checked is not empty>}%
%%
%% The gist of this macro comes from Robert R. Schneck's \ifempty-macro:
%% <https://groups.google.com/forum/#!original/comp.text.tex/kuOEIQIrElc/lUg37FmhA74J>
\newcommand\UD@CheckWhetherNull[1]{%
  \romannumeral\expandafter\UD@secondoftwo\string{\expandafter
  \UD@secondoftwo\expandafter{\expandafter{\string#1}\expandafter
  \UD@secondoftwo\string}\expandafter\UD@firstoftwo\expandafter{\expandafter
  \UD@secondoftwo\string}\expandafter\UD@stopromannumeral\UD@secondoftwo}{%
  \expandafter\UD@stopromannumeral\UD@firstoftwo}%
}%
%%-----------------------------------------------------------------------------
%% Check whether argument's first token is an explicit character of
%% category 1:
%%.............................................................................
%% \UD@CheckWhetherBrace{<Argument which is to be checked>}%
%%                      {<Tokens to be delivered in case that argument
%%                        which is to be checked has a leading
%%                        explicit catcode-1-character-token>}%
%%                      {<Tokens to be delivered in case that argument
%%                        which is to be checked does not have a
%%                        leading explicit catcode-1-character-token>}%
\newcommand\UD@CheckWhetherBrace[1]{%
  \romannumeral\expandafter\UD@secondoftwo\expandafter{\expandafter{%
  \string#1.}\expandafter\UD@firstoftwo\expandafter{\expandafter
  \UD@secondoftwo\string}\expandafter\UD@stopromannumeral\UD@firstoftwo}{%
  \expandafter\UD@stopromannumeral\UD@secondoftwo}%
}%
%%-----------------------------------------------------------------------------
%% Check whether brace-balanced argument starts with an explicit space-token:
%%.............................................................................
%% \UD@CheckWhetherLeadingExplicitSpace{<Argument which is to be checked>}%
%%                                     {<Tokens to be delivered in case <argument
%%                                       which is to be checked> does have a
%%                                       leading explicit space-token>}%
%%                                     {<Tokens to be delivered in case <argument
%%                                       which is to be checked> does not have a
%%                                       a leading explicit space-token>}%
\newcommand\UD@CheckWhetherLeadingExplicitSpace[1]{%
  \romannumeral\UD@CheckWhetherNull{#1}%
  {\expandafter\UD@stopromannumeral\UD@secondoftwo}%
  {%
    % Let's nest things into \UD@firstoftwo{...}{} to make sure they are nested in braces
    % and thus do not disturb when the test is carried out within \halign/\valign:
    \expandafter\UD@firstoftwo\expandafter{%
      \expandafter\expandafter\expandafter\UD@stopromannumeral
      \romannumeral\expandafter\UD@secondoftwo
      \string{\UD@CheckWhetherLeadingExplicitSpaceB.#1 }{}%
    }{}%
  }%
}%
\@ifdefinable\UD@CheckWhetherLeadingExplicitSpaceB{%
  \long\def\UD@CheckWhetherLeadingExplicitSpaceB#1 {%
    \expandafter\UD@CheckWhetherNull\expandafter{\UD@firstoftwo{}#1}%
    {\UD@Exchange{\UD@firstoftwo}}{\UD@Exchange{\UD@secondoftwo}}%
    {\expandafter\expandafter\expandafter\UD@stopromannumeral
     \expandafter\expandafter\expandafter}%
     \expandafter\UD@secondoftwo\expandafter{\string}%
  }%
}%
%%-----------------------------------------------------------------------------
%% Check whether brace-balanced argument is an explicit space-token:
%%.............................................................................
\newcommand\UD@CheckWhetherSpace[1]{%
  \romannumeral\expandafter\UD@CheckWhetherNull
               \expandafter{\UD@GobbleToExclam#1!}{%
    \expandafter\UD@firstoftwo\expandafter{%
      \UD@SpaceFork!#1!{\UD@firstoftwo}! !{\UD@secondoftwo}!!!!%
    }{}%
  }{\expandafter\UD@stopromannumeral\UD@secondoftwo}%
}%
\@ifdefinable\UD@SpaceFork{%
  \long\def\UD@SpaceFork#1! !#2#3!!!!{\expandafter\UD@stopromannumeral#2}%
}%
\@ifdefinable\UD@GobbleToExclam{\long\def\UD@GobbleToExclam#1!{}}%
%%=============================================================================
%% Extract first inner undelimited argument:
%%
%%   \UD@ExtractFirstArg{ABCDE} yields  A
%%
%%   \UD@ExtractFirstArg{{AB}CDE} yields  AB
%%
%% Due to \romannumeral-expansion the result is delivered after two 
%% expansion-steps/after "hitting" \UD@ExtractFirstArg with \expandafter
%% twice.
%%
%% \UD@ExtractFirstArg's argument must not be blank.
%% This case can be cranked out via \UD@CheckWhetherBlank before calling
%% \UD@ExtractFirstArg.
%%
%% Use frozen-\relax as delimiter for speeding things up.
%% I chose frozen-\relax because David Carlisle pointed out in
%% <https://tex.stackexchange.com/a/578877>
%% that frozen-\relax cannot be (re)defined in terms of \outer and cannot be
%% affected by \uppercase/\lowercase.
%%
%% \UD@ExtractFirstArg's argument may contain frozen-\relax:
%% The only effect is that internally more iterations are needed for
%% obtaining the result.
%%
%%.............................................................................
\@ifdefinable\UD@RemoveTillFrozenrelax{%
  \expandafter\expandafter\expandafter\UD@Exchange
  \expandafter\expandafter\expandafter{%
  \expandafter\expandafter\ifnum0=0\fi}%
  {\long\def\UD@RemoveTillFrozenrelax#1#2}{{#1}}%
}%
\expandafter\UD@PassFirstToSecond\expandafter{%
  \expandafter\romannumeral\expandafter\UD@ExtractFirstArgLoop
  \expandafter{\expandafter#\expandafter1%
  \romannumeral\expandafter\expandafter\expandafter\UD@stopromannumeral
  \expandafter\expandafter\ifnum0=0\fi}%
}{\newcommand\UD@ExtractFirstArg[1]}%
\newcommand\UD@ExtractFirstArgLoop[1]{%
  \expandafter\UD@CheckWhetherNull\expandafter{\UD@firstoftwo{}#1}%
  {\expandafter\UD@stopromannumeral\UD@secondoftwo{}#1}%
  {\expandafter\UD@ExtractFirstArgLoop\expandafter{\UD@RemoveTillFrozenrelax#1}}%
}%
%%=============================================================================
%% Extract first inner component, either being a space or being an undelimited
%% argument:
%%
%%   \romannumeral\UD@Romannumeral@ExtractFirstComponent{ABCDE} yields  A
%%
%%   \romannumeral\UD@Romannumeral@ExtractFirstComponent{{AB}CDE} yields  AB
%%
%%   \romannumeral\UD@Romannumeral@ExtractFirstComponent{ ABCDE} yields  <explicit space token>
%%
%%   \romannumeral\UD@Romannumeral@ExtractFirstComponent{ {AB}CDE} yields  <explicit space token>
%%
%% Due to \romannumeral-expansion the result is delivered after two 
%% expansion-steps/after "hitting" \UD@ExtractFirstArg with \expandafter
%% twice.
%%
%% \UD@Romannumeral@ExtractFirstComponent's argument must not be empty.
%% This case can be cranked out via \UD@CheckWhetherNull before calling
%% \UD@Romannumeral@ExtractFirstComponent.
%%.............................................................................
\newcommand\UD@Romannumeral@ExtractFirstComponent[1]{%
  \UD@CheckWhetherLeadingExplicitSpace{#1}{%
    \UD@firstoftwo{\UD@stopromannumeral}{} %
  }{%
    \expandafter\expandafter\expandafter\UD@stopromannumeral
    \UD@ExtractFirstArg{#1}%
  }%
}%
%%=============================================================================
%% \UD@ExtractFirstOpeningBraceStringified{<tokens>}
%%
%% Obtain \string-representation of argument's first explicit category-1-
%% character token.
%%
%% Due to \romannumeral-expansion the result is delivered after two 
%% expansion-steps/after "hitting" \UD@ExtractFirstOpeningBraceStringified
%% with \expandafter twice.
%% If the argument does not have an opening brace you get emptiness.
%%.............................................................................
\newcommand\UD@ExtractFirstOpeningBraceStringified[1]{%
  \romannumeral\UD@ExtractFirstOpeningbraceStringifiedloop{#1}%
}%
\newcommand\UD@ExtractFirstOpeningbraceStringifiedloop[1]{%
  \UD@CheckWhetherNull{#1}{\UD@stopromannumeral}{%
    \UD@CheckWhetherBrace{#1}{%
      \expandafter\UD@Romannumeral@ExtractFirstComponent
      \expandafter{%
        \romannumeral\expandafter\UD@Romannumeral@ExtractFirstComponent
        \expandafter{%
        \romannumeral\expandafter\expandafter\expandafter\UD@stopromannumeral
        \expandafter\UD@firstoftwo\expandafter{\expandafter}%
        \romannumeral\expandafter\expandafter\expandafter\UD@stopromannumeral
        \expandafter\string\expandafter}%
        \string#1%
      }%
    }{%
      \UD@CheckWhetherLeadingExplicitSpace{#1}{%
        \expandafter\UD@ExtractFirstOpeningbraceStringifiedloop
        \expandafter{\UD@removespace#1}%
      }{%
        \expandafter\UD@ExtractFirstOpeningbraceStringifiedloop
        \expandafter{\UD@firstoftwo{}#1}%
      }%
    }%
  }%
}%
%%=============================================================================
%% \UD@ExtractFirstOpeningBracesMatchingClosingBraceStringified{<tokens>}
%%
%% Obtain \string-representation of argument's explicit category-2-
%% character token that matches argument's first explicit category-1-
%% character token.
%%
%% Due to \romannumeral-expansion the result is delivered after two 
%% expansion-steps/after "hitting" \UD@ExtractFirstClosingBraceStringified
%% with \expandafter twice.
%% If the argument does not have a closing brace you get emptiness.
%%.............................................................................
\newcommand\UD@ExtractFirstOpeningBracesMatchingClosingBraceStringified[1]{%
  \romannumeral
  \UD@ExtractFirstClosingbraceStringifiedloop{#1}%
}%
\newcommand\UD@ExtractFirstClosingbraceStringifiedloop[1]{%
  \UD@CheckWhetherNull{#1}{%
    \UD@stopromannumeral
  }{%
    \UD@CheckWhetherBrace{#1}{%
       \UD@ExtractFirstClosingbraceStringifiedloopB{#1}%
    }{%
      \UD@CheckWhetherLeadingExplicitSpace{#1}{%
        \expandafter\UD@ExtractFirstClosingbraceStringifiedloop
        \expandafter{\UD@removespace#1}%
      }{%
        \expandafter\UD@ExtractFirstClosingbraceStringifiedloop
        \expandafter{\UD@firstoftwo{}#1}%
      }%
    }%
  }%
}%
\newcommand\UD@mergeargs[3]{%
  \expandafter#1\expandafter{%
    \romannumeral\expandafter\UD@CheckWhetherSpace
    \expandafter{\romannumeral\UD@ExtractFirstOpeningbraceStringifiedloop{#3}}{%
      \UD@Exchange{\expandafter\UD@removespace}%
    }{%
      \UD@Exchange{\expandafter\UD@firstoftwo\expandafter{\expandafter}}%
    }%
    {\expandafter#2\romannumeral\expandafter\expandafter\expandafter\UD@stopromannumeral}%
    \string#3%
  }%
}%
\newcommand\UD@ExtractFirstClosingbraceStringifiedloopB[1]{%
  \expandafter\UD@CheckWhetherNull
  \expandafter{\romannumeral\UD@Romannumeral@ExtractFirstComponent{#1}}{%
    \UD@mergeargs{\UD@Romannumeral@ExtractFirstComponent}{%
      \expandafter\expandafter\UD@stopromannumeral\expandafter\string
    }%
  }{%
    \UD@mergeargs{\UD@ExtractFirstClosingbraceStringifiedloopB}{%
      \UD@CheckWhetherLeadingExplicitSpace
      \expandafter{\romannumeral\UD@Romannumeral@ExtractFirstComponent{#1}}{%
        \UD@Exchange{\expandafter\UD@removespace}%
      }{%
        \UD@Exchange{\expandafter\UD@firstoftwo\expandafter{\expandafter}}%
      }%
      {%
        \expandafter\UD@stopromannumeral\expandafter{\romannumeral
        \expandafter\UD@firstoftwo\expandafter{\expandafter}\string}%
        \expandafter\expandafter\expandafter\UD@stopromannumeral
      }%
    }%
  }{#1}%
}%
\message{%
  ^^JResult:  |\UD@ExtractFirstOpeningBraceStringified{ n A { X{m } j}{}jh}|%
}%
\begingroup
\catcode`\Y=1
\message{%
  ^^JResult:  |\UD@ExtractFirstOpeningBraceStringified{ n A Y X{m } j}{}jh}|%
}%
\endgroup
\begingroup
\catcode\~=10 \catcode\ =1~%
\message{%
^^JResult:~~|\UD@ExtractFirstOpeningBraceStringified{~n~A~ ~X{m~}~j}{}jh}|%
}%
\endgroup
\message{%
  ^^JResult:  |\UD@ExtractFirstOpeningBraceStringified{ n A  Xm  jjh}|%
}%
\begingroup
\catcode`\Y=1
\expandafter\expandafter\expandafter\def
\expandafter\expandafter\expandafter\test
\expandafter\expandafter\expandafter{%
  \UD@ExtractFirstOpeningBraceStringified{ n A Y X{m } j}{}jh}%
}%
\message{%
  ^^JResult:  |\meaning\test|%
}%
\endgroup
\message{%
  ^^J------------------------------------------------------%
}%
\message{%
  ^^JResult:  |\UD@ExtractFirstOpeningBracesMatchingClosingBraceStringified{ n A { X{m } j}{}jh}|%
}%
\begingroup
\catcode`\Y=2
\message{%
  ^^JResult:  |\UD@ExtractFirstOpeningBracesMatchingClosingBraceStringified{ n A { X{m } jY{}jh}|%
}%
\endgroup
\begingroup
\catcode\~=10 \catcode\ =2~%
\message{%
^^JResult:|\UD@ExtractFirstOpeningBracesMatchingClosingBraceStringified{~n~A~{~X{m~}~j {}jh}|%
}%
\endgroup
\message{%
  ^^JResult:  |\UD@ExtractFirstOpeningBracesMatchingClosingBraceStringified{ n A  Xm  jjh}|%
}%
\begingroup
\catcode`\Y=2
\expandafter\expandafter\expandafter\def
\expandafter\expandafter\expandafter\test
\expandafter\expandafter\expandafter{%
  \UD@ExtractFirstOpeningBracesMatchingClosingBraceStringified{ n A { X{m } jY{}jh}%
}%
\message{%
  ^^JResult:  |\meaning\test|%
}%
\endgroup
\stop

Terminal-output:

$ pdflatex test.tex
This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020) (preloaded format=pdflatex)
 restricted \write18 enabled.
entering extended mode
(./test.tex
LaTeX2e <2020-10-01> patch level 4
L3 programming layer <2021-02-18> 
Result: |{| 
Result: |Y| 
Result: | | 
Result: || 
Result: |macro:->Y| 
------------------------------------------------------ 
Result: |}| 
Result: |Y| 
Result: | | 
Result: || 
Result: |macro:->Y| )
No pages of output.
Transcript written on test.log.

Answer 2

Ulrich Diez's answer already cover the "how to do it" part. In the spirit of "one hundred percent correct", the following algorithm will always takes at most O(N log N) time (and O(N) time if the delimiter token is not present in the token list).

Currently there's no implementation, but the idea should work, and we are relatively sure that the (asymptotic) time complexity is computed correctly.

The algorithm.

The description of the algorithm follows.

1. First, use #{ to gobble until the first {} (needless to say, add a trailing {} to avoid the case there's no braced group in the token list)
   • then deal with empty token list case accordingly.
2. Then, count the number of items in the head. (this can be done without explicitly take the head, by putting them forward in the token list, then after counting skip through the count itself to add a brace and remove the tail part)
3. Use a function to remove exactly that many items.
4. Deal with any remaining space characters. This part is not hard.
5. Stringify and get the result. Recall that the stringfication of the } is a single token, as such it's possible in linear time.

Step 2 is the bottleneck here with O(N log N) time complexity. The rest takes linear time. (at the cost of linear \romannumeral recursion depth)

---

Old answer

(uses the approach identical to Ulrich Diez's answer)

Since I think writing complex code such as the above is way too complex, I decide to write a "compiler" that compiles easy-to-understand imperative-style code such as this... _{that's the definition of the macro that extracts the string of the first }}

\rdeflinenumbered \firstegroup #x {}!
    \while {}{ \ifnotempty {#x} } {
        \conditional {\ifbrace {#x}} {
            % found the first group.
        % first make sure the open brace has charcode `{` (anything not a space will do.).
        \assignoperate #x {\string #x} {
            \expandonce
            \rcall{\putnextbgroup}
        }

        % then empty out that group.
        \while {
            \assignr #\firstcomponent {\firstarg{#x}}
        } {
            \ifnotempty {#\firstcomponent}
        }{
            % firstcomponent is still nonempty. Pick one item
            \conditional{\ifspace {#\firstcomponent}} {
                \assignoperate #x {#x} {
                    % following in the input stream:  { <space> ... }  ...
                    \putnext{\string} \expandonce
                    % following in the input stream:  '{' <space> ... }  ...  where the initial { is stringified and is definitely not a space
                    \matchrm{#1 ~}
                    % following in the input stream:    ... }  ...
                    \rcall{\putnextbgroup}
                }
            } {
                \assignoperate #x {#x} {
                    % following in the input stream:  { <item> ... }  ...
                    \putnext{\string} \expandonce
                    % following in the input stream:  '{' <item> ... }  ...  where the initial { is stringified and is definitely not a space
                    \matchrm{#1 #2}
                    % following in the input stream:    ... }  ...
                    \rcall{\putnextbgroup}
                }
            }
            % now firstcomponent is shorter.
        }

        % finally firstcomponent is empty now. (and the opening brace is guaranteed to be non-space)
        \assignoperate #x {#x} {
            % following in the input stream: { } ...
            \putnext{\string} \expandonce
            % following in the input stream: '{' } ...
            \matchrm{#1}
            % following in the input stream:     } ...
            \putnext{\string} \expandonce
            % following in the input stream:     '}' ...
        }

        % finally done
        \assignr #x{\firstcomponent{#x}}
        \return{#x}

    } {
        \conditional {\ifspace {#x}} {
            \assignr #x{\removespace {#x}}
        } {
            \assignr #x{\dropfirst {#x}}
        }
    }
}
\return {}

!

to a set of TeX macros.

Currently there's little to no documentation, but hopefully you can roughly understand what the code does (for instance...

• while does a while loop
• conditional does a conditional ("if" statement)
• assign "assigns" values to token lists
• assignr "computes" result of a "function" then assign that to token lists
• matchrm matches a pattern forward in the input stream then remove that, assign matched value to token lists
• putnext puts tokens forward in the input stream
• rcall calls a "subroutine"
• etc.

Currently the compiler is quite slow and generates bloated code, this will be improved later. Also it's not on CTAN. If you're interested you can try to find some ways to run it from source.

The compiler is implemented in LuaTeX, but the generated code can be used in any compiler.

For a demo, try running the following code

%! TEX program = pdflatex
\documentclass{article}
\usepackage{filecontentsdef}
\begin{document}
\ExplSyntaxOn
% ======== some auxiliary macros ========
\def__process_char #1 #2 {
    %\prettye:n{\expandafter \expandafter \expandafter \noexpand \char_generate:nn {#2} {"#1}} \expandafter \expandafter \expandafter \noexpand \char_generate:nn {#2} {"#1}
    __process_s
}
\def__process_space_other_cat #1 {
    \expandafter \expandafter \expandafter \noexpand \char_generate:nn {32} {"#1}
    __process_s
}
\def__process_cs #1 / {
    \expandafter \noexpand \csname #1 \endcsname
    __process_s
}
\def__process_s#1{
    \token_if_eq_charcode:NNTF #1 0 { % 0 <name> / → the control sequence
        __process_cs
    } {
        \token_if_eq_charcode:NNTF #1 s { ~   __process_s
        } {
            \token_if_eq_charcode:NNTF #1 S { % S <cat> → a space
                __process_space_other_cat
            } {
                \token_if_eq_charcode:NNF #1 . { % . → end
                    __process_char #1
                }
            }
        }
    }
}
% main handler function, will exec the resulting token list.
\def__process_all#1{
    \begingroup \exp_last_unbraced:Nx \endgroup {__process_s #1}
}
\ExplSyntaxOff
\begin{filecontentsdefmacro}{\data}
0def/0stzz241/s1{0exp_end:/2}0def/0removespace/6#6#C11{0expandafter/0stzz584/0expandafter/1{0exp:w/0stzz241/6#6#C12}2}0def/0ifempty/6#6#C11{0iffalse/1{0fi/0expandafter/0use_none:n/0expandafter/1{0expandafter/1{0string/6#6#C12}0@ifempty@casei/2}0@ifempty@caseii/2}0use_i:nn/2}0def/0@ifempty@casei/1{0exp:w/0removenextegroup/2}0def/0@ifempty@caseii/1{0expandafter/0stzz309/0exp:w/0removenextegroup/2}0def/0stzz309/6#6#C11{0use_ii:nn/2}0def/0ifbrace/6#6#C11{0iffalse/1{0fi/0expandafter/0use_none:n/0expandafter/1{0expandafter/1{0string/6#6#C12}0@ifbrace@casei/2}0@ifbrace@caseii/2}0use_ii:nn/2}0def/0@ifbrace@casei/1{0expandafter/0stzz353/0exp:w/0removenextegroup/2}0def/0stzz353/1{0expandafter/0stzz354/0exp:w/0removeuntilegroup/2}0def/0stzz354/6#6#C11{0use_i:nn/2}0def/0@ifbrace@caseii/1{0exp:w/0removenextegroup/2}0def/0removeuntilegroup/1{0expandafter/0stzz400/0expandafter/1{0iffalse/2}0fi/2}0def/0putnextbgroup/1{0expandafter/0exp_end:/0expandafter/1{0iffalse/2}0fi/2}0def/0putnextegroup/1{0expandafter/0exp_end:/0iffalse/1{0fi/2}2}0def/0stzz400/6#6#C11{0exp_end:/2}0def/0dropfirst/6#6#C11{0expandafter/0stzz584/0expandafter/1{0exp:w/0stzz400/6#6#C12}2}0def/0stzz411/6#6#C1s1{0expandafter/0stzz425/0expandafter/1{0exp:w/0dropfirst/1{6#6#C12}2}2}0def/0stzz425/6#6#C11{0ifempty/1{6#6#C12}0stzz426/0stzz429/2}0def/0stzz429/1{0expandafter/0stzz434/0expandafter/0use_ii:nn/0exp:w/0removeuntilegroup/2}0def/0stzz426/1{0expandafter/0stzz434/0expandafter/0use_i:nn/0exp:w/0removeuntilegroup/2}0def/0stzz434/6#6#C11{0expandafter/0stzz584/0expandafter/6#6#C10exp:w/0putnextegroup/2}0def/0ifspace/6#6#C11{0expandafter/0stzz439/0expandafter/1{0exp:w/0stzz411/C.6#6#C1s2}2}0def/0stzz439/6#6#C11{6#6#C12}0def/0ifnotempty/6#6#C11{0ifempty/1{6#6#C12}0use_ii:nn/0use_i:nn/2}0def/0stzz464/6#6#C16#6#C20relax/1{0exp_end:/1{6#6#C12}2}0def/0firstarg/6#6#C11{0expandafter/0stzz463c/0expandafter/1{0exp:w/0dropfirst/1{6#6#C10relax/2}2}1{6#6#C10relax/2}2}0def/0stzz463a/6#6#C11{0expandafter/0stzz463b/0expandafter/1{0exp:w/0stzz464/6#6#C12}2}0def/0stzz463b/6#6#C11{0expandafter/0stzz463c/0expandafter/1{0exp:w/0dropfirst/1{6#6#C12}2}1{6#6#C12}2}0def/0stzz463c/6#6#C16#6#C21{0ifnotempty/1{6#6#C12}0stzz463a/0stzz472/1{6#6#C22}2}0def/0stzz472/6#6#C11{0expandafter/0stzz584/0expandafter/1{0use:n/6#6#C12}2}0def/0firstargsingletoken/6#6#C11{0expandafter/0stzz484/0removebgroup/1{6#6#C12}2}0def/0stzz484/6#6#C11{0stzz486/6#6#C12}0def/0stzz486/6#6#C11{0expandafter/0stzz487/0expandafter/6#6#C10exp:w/0putnextbgroup/2}0def/0stzz487/6#6#C16#6#C21{0exp_end:/6#6#C12}0def/0firstcomponent/6#6#C11{0ifspace/1{6#6#C12}0stzz495/1{0stzz497/1{6#6#C12}2}2}0def/0stzz497/6#6#C11{0expandafter/0stzz584/0expandafter/1{0exp:w/0firstarg/1{6#6#C12}2}2}0def/0stzz495/1{0exp_end:/s2}0def/0stzz509/1{0expandafter/0putnextbgroup/2}0def/0firstbgroup/6#6#C11{0ifnotempty/1{6#6#C12}1{0stzz505a/1{6#6#C12}2}0exp_end:/2}0def/0stzz505a/6#6#C11{0ifbrace/1{6#6#C12}0stzz520c/0stzz519/1{6#6#C12}2}0def/0stzz519/6#6#C11{0ifspace/1{6#6#C12}0stzz522b/0stzz522/1{6#6#C12}2}0def/0stzz522/6#6#C11{0expandafter/0stzz520a/0expandafter/1{0exp:w/0dropfirst/1{6#6#C12}2}2}0def/0stzz522b/6#6#C11{0expandafter/0stzz520a/0expandafter/1{0exp:w/0removespace/1{6#6#C12}2}2}0def/0stzz520a/6#6#C11{0ifnotempty/1{6#6#C12}1{0stzz505a/1{6#6#C12}2}0exp_end:/2}0def/0stzz520c/6#6#C11{0expandafter/0stzz515/0expandafter/1{0exp:w/0stzz509/0string/6#6#C12}2}0def/0stzz515/6#6#C11{0expandafter/0stzz583/0expandafter/1{0exp:w/0firstarg/1{6#6#C12}2}2}0def/0stzz537/1{0expandafter/0putnextbgroup/2}0def/0stzz559/1{0expandafter/0stzz563/0string/2}0def/0stzz563/6#6#C16#6#C21{0putnextbgroup/2}0def/0stzz550/1{0expandafter/0stzz554/0string/2}0def/0stzz554/6#6#C1s1{0putnextbgroup/2}0def/0stzz572/1{0expandafter/0stzz576/0string/2}0def/0stzz576/6#6#C11{0expandafter/0exp_end:/0string/2}0def/0firstegroup/6#6#C11{0ifnotempty/1{6#6#C12}1{0stzz532a/1{6#6#C12}2}0exp_end:/2}0def/0stzz532a/6#6#C11{0ifbrace/1{6#6#C12}0stzz588c/0stzz587/1{6#6#C12}2}0def/0stzz587/6#6#C11{0ifspace/1{6#6#C12}0stzz590b/0stzz590/1{6#6#C12}2}0def/0stzz590/6#6#C11{0expandafter/0stzz588a/0expandafter/1{0exp:w/0dropfirst/1{6#6#C12}2}2}0def/0stzz590b/6#6#C11{0expandafter/0stzz588a/0expandafter/1{0exp:w/0removespace/1{6#6#C12}2}2}0def/0stzz588a/6#6#C11{0ifnotempty/1{6#6#C12}1{0stzz532a/1{6#6#C12}2}0exp_end:/2}0def/0stzz588c/6#6#C11{0expandafter/0stzz550b/0expandafter/1{0exp:w/0stzz537/0string/6#6#C12}2}0def/0stzz543a/6#6#C16#6#C21{0ifspace/1{6#6#C22}0stzz559c/0stzz559a/1{6#6#C12}2}0def/0stzz559a/6#6#C11{0expandafter/0stzz550b/0expandafter/1{0exp:w/0stzz559/6#6#C12}2}0def/0stzz559c/6#6#C11{0expandafter/0stzz550b/0expandafter/1{0exp:w/0stzz550/6#6#C12}2}0def/0stzz550b/6#6#C11{0expandafter/0stzz544a/0expandafter/1{0exp:w/0firstarg/1{6#6#C12}2}1{6#6#C12}2}0def/0stzz544a/6#6#C16#6#C21{0ifnotempty/1{6#6#C12}1{0stzz543a/1{6#6#C22}1{6#6#C12}2}1{0stzz572a/1{6#6#C22}2}2}0def/0stzz572a/6#6#C11{0expandafter/0stzz583/0expandafter/1{0exp:w/0stzz572/6#6#C12}2}0def/0stzz583/6#6#C11{0expandafter/0stzz584/0expandafter/1{0exp:w/0firstcomponent/1{6#6#C12}2}2}0def/0stzz584/6#6#C11{0exp_end:/6#6#C12}.
\end{filecontentsdefmacro}
\ExplSyntaxOn
\exp_args:NV __process_all \data
\ExplSyntaxOff
\let\removenextegroup\removeuntilegroup
\message{%
  ^^JResult:  |\romannumeral\firstbgroup{ n A { X{m } j}{}jh}|%
}%
\begingroup
\catcode`\Y=1
\message{%
  ^^JResult:  |\romannumeral\firstbgroup{ n A Y X{m } j}{}jh}|%
}%
\endgroup
\begingroup
\catcode\~=10 \catcode\ =1~%
\message{%
^^JResult:~~|\romannumeral\firstbgroup{~n~A~ ~X{m~}~j}{}jh}|%
}%
\endgroup
\message{%
  ^^JResult:  |\romannumeral\firstbgroup{ n A  Xm  jjh}|%
}%
\begingroup
\catcode`\Y=1
\expandafter\expandafter\expandafter\def
\expandafter\expandafter\expandafter\test
\expandafter\expandafter\expandafter{%
  \romannumeral\firstbgroup{ n A Y X{m } j}{}jh}%
}%
\message{%
  ^^JResult:  |\meaning\test|%
}%
\endgroup
\message{%
  ^^J------------------------------------------------------%
}%
\message{%
  ^^JResult:  |\romannumeral\firstegroup{ n A { X{m } j}{}jh}|%
}%
\begingroup
\catcode`\Y=2
\message{%
  ^^JResult:  |\romannumeral\firstegroup{ n A { X{m } jY{}jh}|%
}%
\endgroup
\begingroup
\catcode\~=10 \catcode\ =2~%
\message{%
^^JResult:|\romannumeral\firstegroup{~n~A~{~X{m~}~j {}jh}|%
}%
\endgroup
\message{%
  ^^JResult:  |\romannumeral\firstegroup{ n A  Xm  jjh}|%
}%
\begingroup
\catcode`\Y=2
\expandafter\expandafter\expandafter\def
\expandafter\expandafter\expandafter\test
\expandafter\expandafter\expandafter{%
  \romannumeral\firstegroup{ n A { X{m } jY{}jh}%
}%
\message{%
  ^^JResult:  |\meaning\test|%
}%
\endgroup
\end{document}

The demo part is copied from Ulrich Diez's answer.

The algorithm used is mostly identical to that answer as well.

For the source code, see https://github.com/user202729/TeXlib/blob/main/test_imperative.tex#L487 etc. if you're interested. (that file is runnable in LuaLaTeX with appropriate libraries, nevertheless the output is shown in the HTML file, nowadays I prefer using my prettytok package to display the output)

Because of some limitations, \relax is used in place of (frozen relax) token, which will make the algorithm slower in certain cases.

Regarding the compiled code -- the output is some expandable macros e.g., but I make the program generate tokens that does not tokenize well in the normal catcode, thus the auxiliary function to reconstruct the token list.

Example generated code (as I've mentioned before, the current code is very bloated. Will be fixed later)

Answer 3

Use \tl_analysis_map_inline:nn. The inner code can use three arguments: one is a representation of the token, then the character code if the item is a character, and finally the category code.

You can use the second argument to generate the corresponding character with category code 12.

Note: this won't work with \bgroup, but you didn't ask for this case.

\documentclass{article}
\ExplSyntaxOn
\NewDocumentCommand{\macro}{m}
 {
  \bool_set_true:N \l__egreg_leftbrace_notfound_bool
  \tl_analysis_map_inline:nn { #1 } { __egreg_leftbrace:nnn { ##1 } { ##2 } { ##3 } }
  \str_show:N \l_tmpa_str
 }
\bool_new:N \l__egreg_leftbrace_notfound_bool
\cs_new_protected:Nn __egreg_leftbrace:nnn
 {
  \bool_lazy_and:nnT { \l__egreg_leftbrace_notfound_bool } { \int_compare_p:n { "#3 = 1 } }
   {
    \str_set:Nx \l_tmpa_str { \char_generate:nn { #2 } { 12 } }
    \bool_set_false:N \l__egreg_leftbrace_notfound_bool
   }
 }
\ExplSyntaxOff
\begin{document}
\begingroup
\catcode`Y=1
\macro{ text text Y z{ww} z} n n nb}
\endgroup
\macro{abc}
\macro{a{b}c}
\end{document}

The console output:

> \l_tmpa_str=Y.
<recently read> }
l.28 \macro{ text text Y z{ww} z} n n nb}
?
> \l_tmpa_str=.
<recently read> }
l.31 \macro{abc}
?
> \l_tmpa_str={.
<recently read> }
l.33 \macro{a{b}c}
?