How can I create a comma-separated list of characters from a text string?

Question

How can I split the string Hello World! - stored into a macro \def\mystring{Hello Word!} - into a comma-separated list of characters (including spaces) which can be used as argument of a \foreach loop (\foreach \char in {\myCSlist}) in order to loop over each character (as with \foreach \char in {H,e,l,l,o,\space,W,o,r,l,d,!})?

EDIT 1 (Mar 21): Why I want that?

At first, I didn't explain why I want the solution generates a comma-separated list of characters (e.g. \myCSlist) which can be used as argument of a \foreach. It's because I want to create a Tikz \node for each character using a pic. Something like this (from here):

\newcommand{\hsp}{.5}
\tikzset{symbols/.pic={%
    \foreach \s[count=\n from 0] in {\myCSlist}{%
    \pgfmathsetmacro{\myangle}{360*rnd}%
    \node[rotate=\myangle] at (\hsp*\n,0){\s};%
}}}%

First attempt

I tried with the \markletters macro of egreg (using xparse and expl3) but the results seems to be not suitable for looping with foreach.

I changed the result print - of the \markletters macro - from (##1) to ##1,, in order to obtain a comma-separated list, but this also results in a unwanted comma , at the end of the comma-separated list.

Here a M(Non)WE:

\documentclass{article}
\usepackage{tikz}% For the \foreach loop
\usepackage{xparse}
% egreg's \markletters macro : https://tex.stackexchange.com/a/359204/262081
\ExplSyntaxOn
\NewDocumentCommand{\markletters}{om}
 {
  \IfNoValueTF{#1}
   {
    \kessels_markletters:nn { #2 } { \tl_use:N \l_kessels_marked_letters_tl }
   }
   {
    \kessels_markletters:nn { #2 } { \tl_set_eq:NN #1 \l_kessels_marked_letters_tl }
   }
 }
\tl_new:N \l_kessels_unmarked_letters_tl
\tl_new:N \l_kessels_marked_letters_tl
\cs_new_protected:Nn \kessels_markletters:nn
 {
  \tl_set:Nn \l_kessels_unmarked_letters_tl { #1 }
  \tl_replace_all:Nnn \l_kessels_unmarked_letters_tl { ~ } { \textvisiblespace }
  \tl_clear:N \l_kessels_marked_letters_tl
  \tl_map_inline:Nn \l_kessels_unmarked_letters_tl
   {
    \tl_put_right:Nn \l_kessels_marked_letters_tl { ##1, }
   }
  #2
 }
\ExplSyntaxOff
\def\mystring{Hello World!}
\begin{document}
\markletters[\foo]{\mystring}%
\foreach \char in {\foo}{%
    <\char>%
}%
\end{document}

This gives the following result:

But I would prefer this result (note the presence of the space between 'Hello' and 'World'):

Answer 1

Here's a LuaLaTeX-based solution. It's fully utf8-code aware, i.e., the input string may contain utf8-encoded, as opposed to just ascii-encoded, characters.

The code consists of a Lua function named DoString and a LaTeX utility macros called \DoString, which takes 1 argument -- a character string -- and passes it to the Lua function for further processing.

% !TEX TS-program = lualatex
\documentclass{article}
\directlua{
function DoString ( s )
  for i = 1,string.utflength(s) do 
    tex.sprint ( '<' .. unicode.utf8.sub ( s , i , i ) .. '>' )
  end
end
}
%% Define a LaTeX utility macro:
\newcommand\DoString[1]{\directlua{DoString("#1")}}
\begin{document}
\ttfamily
\DoString{Hello World!}
\DoString{äöü ÄÖÜ // ß}
\end{document}

---

Addendum to address the OP's follow-up question: To create tikz nodes along the lines suggested in the comment, I suggest you (a) load the luacode package (for the \luaexec macro) and (b) change the \directlua{...} chunk to

\luaexec{
function DoString ( s )
  for i = 1,string.utflength(s) do 
    tex.sprint ( '\\node{' .. unicode.utf8.sub ( s , i , i ) .. '};' )
  end
end
}

Answer 2

Here I use a token cycle. No mention was made of macros or groups appearing in the input (only characters and spaces), and so those should be excluded from the input, unless the OP advises on how they should be handled.

If one wishes to then use this list in a for-like loop, one can use \foreachitem from the listofitems package, after reading the comma-separated \mystring into a list.

\documentclass{article}
\usepackage{tokcycle,listofitems}
\Characterdirective{\addcytoks{,#1}}
\Spacedirective{\addcytoks{,#1}}
\makeatletter
\newcommand\markletters[1]{%
  \tokcyclexpress{#1}%
  \expandafter\expandafter\expandafter\def
  \expandafter\expandafter\expandafter\mystring
  \expandafter\expandafter\expandafter
  {\expandafter\@gobble\the\cytoks}%
}
\makeatother
\begin{document}
\markletters{Hello World!}
(\detokenize\expandafter{\mystring})
\readlist\mylist\mystring
\foreachitem\z\in\mylist{$<$\z$>$}
\end{document}

Of course, the whole process could be done more simply inside the token cycle, such that listofitems would not even be needed. And, much to my surprise, if I compile under lualatex, it properly handles UTF-8 input

\documentclass{article}
\usepackage{tokcycle}
\newcommand\markletters[1]{%
  \tokencycle{$<$##1$>$}{}{}{$<$##1$>$}#1\endtokencycle%
}
\begin{document}
\markletters{Hello World! äöü ÄÖÜ // ß}
\end{document}

SUPPLEMENT

Based on OP comments, here is a version (based on the first approach) that handles macros and groups. However, macros requiring arguments would have to be handled carefully, since the comma is being inserted after each token/group.

In this implementation, group content is merely echoed as a single entity. If one wanted group content, subdivided by token, tokcycle can do that, but it is not done in this implementation.

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage{tokcycle,listofitems}
\makeatletter
\stripgroupingtrue
\newcommand\markletters[1]{%
  \tokcycle{\addcytoks{,##1}}{\addcytoks{,}\groupedcytoks{##1}}%
    {\addcytoks{,##1}}{\addcytoks{,##1}}{#1}%
  \expandafter\expandafter\expandafter\def
  \expandafter\expandafter\expandafter\mystring
  \expandafter\expandafter\expandafter
  {\expandafter\@gobble\the\cytoks}%
}
\makeatother
\begin{document}
\markletters{Hello World! \today{} is a great day}
(\detokenize\expandafter{\mystring})
\readlist\mylist\mystring
\foreachitem\z\in\mylist{$<$\z$>$}
\mylist[7] is the 7th token
\detokenize\expandafter\expandafter\expandafter{\mylist[14]}
 is the 14th token
\end{document}

Answer 3

This can be done using the package xstring. The code also uses \foreach, which is part of tikz but can be loaded with pgffor if your document doesn't use tikz.

\documentclass{article}
\usepackage[T1]{fontenc} % to use accented characters and others
\usepackage{xstring}
\usepackage{pgffor} % in case you're not using tikz
\newcommand{\markletters}[1]{\StrLen{#1}[\strlen]\foreach \chr in {1,...,\strlen}{$<$\StrChar{#1}{\chr}$>$}}
\begin{document}
\markletters{Hello World!}
\markletters{äöü ÄÖÜ // ß}
\end{document}

Answer 4

Here's a \stringforeach command, with the same syntax as \foreach.

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage{geometry}
\usepackage{pgffor}
\makeatletter
\newcommand{\stringforeach}{}
\def\stringforeach#1#2in{%
  \def@foreachvariable{#1}%
  \def@foreachoptions{#2}%
  @ifnextchar\bgroup{@foreachexplicit}{@foreachimplicit}%
}
\def@foreachexplicit#1{@stringforeachdo{#1}}
\def@foreachimplicit#1{\expandafter@stringforeachdo\expandafter{#1}}
\def@stringforeachdo#1{%
  @transformstringintoclist{#1}%
  \begingroup\edef\x{%
    \endgroup
    \noexpand\foreach
    \unexpanded\expandafter{@foreachvariable}%
    \unexpanded\expandafter{@foreachoptions}%
    in
    \noexpand@transformedstring
  }\x
}
\makeatother
\ExplSyntaxOn
\cs_new_protected:cpn {@transformstringintoclist} #1
 {
  \tl_set:Nn \l_tmpa_tl { #1 }
  \tl_replace_all:Nnn \l_tmpa_tl { ~ } { \space }
  \seq_set_split:NnV \l_tmpa_seq { } \l_tmpa_tl
  \tl_set:cx { @transformedstring } { \seq_use:Nn \l_tmpa_seq { , } }
 }
\ExplSyntaxOff
\def\mystring{Hello World!}
\begin{document}
\stringforeach \char in \mystring {<\char>}
\stringforeach \char in {Hello World!} {<\char>}
\stringforeach \char [count=\x] in \mystring {<\x=\char>}
\stringforeach \char [count=\x] in {Hello World!} {<\x=\char>}
\end{document}

If you don't need all the \foreach features, it's simpler.

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage{geometry}
\ExplSyntaxOn
\NewDocumentCommand{\processtokenlist}{smm}
 {
  \IfBooleanTF{#1}
   {% we have a macro
    \sylvain_processtl:Vn #2 { #3 }
   }
   {% we have an explicit token list
    \sylvain_processtl:nn { #2 } { #3 }
   }
 }
\tl_new:N \l__sylvain_processtl_tl
\cs_new_protected:Nn \sylvain_processtl:nn
 {
  \tl_set:Nn \l__sylvain_processtl_tl { #1 }
  \tl_replace_all:Nnn \l__sylvain_processtl_tl { ~ } { \c_space_tl }
  \cs_set_protected:Nn __sylvain_processtl_item:n { #2 }
  \tl_map_function:NN \l__sylvain_processtl_tl __sylvain_processtl_item:n
 }
\cs_generate_variant:Nn \sylvain_processtl:nn { V }
\ExplSyntaxOff
\def\mystring{Hello World!}
\begin{document}
\processtokenlist*{\mystring}{<#1>}
\processtokenlist{Hello World!}{<#1>}
\end{document}

Answer 5

Because we cannot absorb single space by unseparated macro parameter, we must pre-process the given macro by \replspaces macro first. Then you can use \insertcommas which inserts commas between token but not after the last one. Or \insertangles which replaces each token by <token>.

\def\afterfi#1#2\fi{\fi#1}
\def\replspaces#1{\edef#1{\expandafter\replspacesA#1 \end}}
\def\replspacesA #1 #2{#1\ifx\end#2\else { }\afterfi{\replspacesA#2}\fi}
\def\insertcommas#1{\edef#1{\expandafter\insertcommasA#1\end}}
\def\insertcommasA#1#2{#1\ifx\end#2\else,\afterfi{\insertcommasA{#2}}\fi}
\def\insertangles#1{\edef#1{\expandafter\insertanglesA#1\end}}
\def\insertanglesA#1{\ifx\end#1\else<#1>\expandafter\insertanglesA\fi}
% test:
\def\mystring{Hello World!}
\replspaces\mystring
\insertcommas\mystring
\meaning\mystring  % macro:-> H,e,l,l,o, ,W,o,r,l,d,!
\def\mystring{Hello World!}
\replspaces\mystring
\insertangles\mystring
\meaning\mystring  % macro:-> <H><e><l><l><o>< ><W><o><r><l><d><!>
\bye

Edit Your first ask was: how to create a macro H,e,l,l,o, ,w,o,r,d,! from the macro Hello World!. I answered correctly.

Now, your second ask is: how to use such macro in \foreach from Tikz. I don't know how to set effectively the expanded parameter to a somewhat impractical \foreach from Tikz (you can use \expanded{\unexpanded {\foreach ...}{\mystring}} but it seems horrible). It should be better not to use \foreach from Tikz at all and define your own \myforeach:

\def\replspaces#1{\edef#1{\expandafter\replspacesA#1 \end}}
\def\replspacesA #1 #2{#1\ifx\end#2\else { }\afterfi{\replspacesA#2}\fi}
\def\myforeach#1{\expandafter\myforeachA#1\end}
\def\myforeachA#1{\ifx\end#1\else\body{#1}\expandafter\myforeachA\fi}
\def\body#1{<#1>}
\def\mystring{Hello World!}
\replspaces\mystring 
\myforeach\mystring % prints: <H><e><l><l><o>< ><W><o><r><l><d><!>
\bye

Answer 6

Since you're already using PGF/TikZ you might as well use the parser module that comes with it.

We'll define a parser char parser that does something (/char parser/do) with each letter, for this we need three actions:

1. the one for the space (it's special),
2. the one for every other character and
3. for the end of the parsing.

I'm using a @-protected macro \charparser@stop to end the parsing since any other letter (like ;) could show up in the text that shall be parsed.

The action for every other character is done by defining an action for an unknown letter – unknown except for the space or for the ending. The parser would actually raise an error in that case but we can set char parser/silent=true.

And then it's just a question about what you want to do with every letter.

We can place a node for it:

\tikz[
  /char parser/do/.code={\node[rotate=360*rnd] at (\charparsercount,0) {#1};},
  /char parser/parse={Hello World! äöüßÄÖÜ}
]{}

Yes, the parser increments a macro (not a counter) \charparsercount. The key do will be used with the found character.

We can also make a comma-separated list and use it with \foreach:

\tikz[
  /char parser/list=\myList,
  /char parser/parse={Hello World! äöüßÄÖÜ}%
]
\foreach \s[count=\n from 0] in \myList
  \node[rotate=360*rnd] at (\n,0) {\s};

If you have the to-be-parsed text in a macro just use /.expand once:

\newcommand*\myText{Hello World! äöüßÄÖ}
/char parser/parse/.expand once=\myText

Code

% use lualatex
\documentclass[tikz]{standalone}
\newcommand*\charparserset{\pgfqkeys{/char parser}}
\usepgfmodule{parser}
\pgfparserdefunknown{char parser}{initial}{%
  \pgfkeysalso{/char parser/do/.expand once=\pgfparserletter,/char parser/next}}
\pgfparserdef{char parser}{initial}{blank space}{%
  \pgfkeysalso{/char parser/do=\space,/char parser/next}}
\pgfparserset{char parser/silent=true}
\makeatletter
\pgfparserdef{char parser}{initial}\charparser@stop{%
  \pgfkeysalso{/char parser/final/.try}\pgfparserswitch{final}}
\charparserset{parse/.code={%
    \charparserset{init}%
    \pgfparserparse{char parser}#1\charparser@stop}}
\makeatother
\charparserset{
  init/.code=\def\charparsercount{0},
  next/.code=\edef\charparsercount{\the\numexpr\charparsercount+1\relax},
  list/.style={
    /char parser/final/.append code=\pgfkeysgetvalue{/char parser/@list}#1,
    /char parser/@list/.initial=,
    /char parser/do/.style={
      /char parser/@list/.append={##1},
      /char parser/do/.style={
        /char parser/@list/.append={,####1}}}},
  do/.code={#1}}
\begin{document}
\tikz[
  x=5mm,
  /char parser/do/.code={\node[rotate=360*rnd] at (\charparsercount,0) {#1};},
  /char parser/parse={Hello World! äöüßÄÖÜ}
]{}
\tikz[
  x=5mm,
  /char parser/list=\myList,
  /char parser/parse={Hello World! äöüßÄÖÜ}%
]
\foreach \s[count=\n from 0] in \myList
  \node[rotate=360*rnd] at (\n,0) {\s};
\end{document}

Output