latex3 — issues with cyrillic

Question

For some reason the following code

\documentclass{article}
\usepackage[english,bulgarian]{babel}
\usepackage[utf8]{inputenc}
\usepackage[T2A]{fontenc}
\begin{document}
Здравей
\ExplSyntaxOn
\tl_set:Nx \l_tmpa_tl {Здравей}
\tl_map_inline:Nn \l_tmpa_tl { #1 ~ }
\ExplSyntaxOff
\end{document}

leads to bunch of errors

Looks like it's the problem with \tl_map_inline, since storing and using token list with cyrillic works fine.

---

Note that I'm using TeXLive 2022 while on 2019 it the code above works perfectly.

How can I fix the issues right now on 2022 setup?

Answer 1

If you use

\documentclass{article}
\usepackage[english,bulgarian]{babel}
\usepackage[utf8]{inputenc}
\usepackage[T2A]{fontenc}
\begin{document}
Здравей
\ExplSyntaxOn
\tl_set:Nx \l_tmpa_tl {Здравей}
\show\l_tmpa_tl
\tl_map_inline:Nn \l_tmpa_tl { #1 ~ }
\ExplSyntaxOff
\end{document}

You will see the current release defines

> \l_tmpa_tl=macro:
->Здравей.

older releases

> \l_tmpa_tl=macro:
->\T2A\CYRZ \T2A\cyrd \T2A\cyrr \T2A\cyra \T2A\cyrv \T2A\cyre \T2A\cyrishrt .

Usually the new version is to be preferred, what is the actual use case here, there is probably a way to achieve it?

---

A quick fix would be this which keeps each two-byte UTF-8 pair in a group:

> \l_tmpa_tl=macro:
->{З}{д}{р}{а}{в}{е}{й}.

\documentclass{article}
\usepackage[english,bulgarian]{babel}
\usepackage[utf8]{inputenc}
\usepackage[T2A]{fontenc}
\begin{document}
Здравей
\ExplSyntaxOn
\def\uviii#1#2{\ifx\relax#1\else{#1#2}\expandafter\uviii\fi}
\tl_set:Nx \l_tmpa_tl {\uviii Здравей\relax\relax}
%\show\l_tmpa_tl
\tl_map_inline:Nn \l_tmpa_tl { #1 ~ }
\ExplSyntaxOff
\end{document}

\documentclass{article}
\usepackage[english,bulgarian]{babel}
\usepackage[utf8]{inputenc}
\usepackage[T2A]{fontenc}

\begin{document}

Здравей

\ExplSyntaxOn
\def\uviii#1{\ifx\relax#1\else
\ifnum\expandafter`\string#1<128~
\expandafter\expandafter\expandafter\uviiia\else
\expandafter\expandafter\expandafter\uviiib
\fi
\fi
#1}
\def\uviiia#1{#1\uviii}
\def\uviiib#1#2{{#1#2}\uviii}
\tl_set:Nx \l_tmpa_tl {\uviii Здравейabc\relax}
%\show\l_tmpa_tl
\tl_map_inline:Nn \l_tmpa_tl { #1 ~ }
\ExplSyntaxOff

\end{document}

Answer 2

The other answers cover why this is happening. It is likely that the team will add a 'text mapping' function soon - as a model for the present:

\documentclass{article}
\usepackage[T2A]{fontenc}
\ExplSyntaxOn
\cs_new:Npn \text_map_function:nN #1#2
  { \exp_args:Ne \__text_map_function:nN { \text_expand:n {#1} } #2 }
\cs_new:Npn \__text_map_function:nN #1#2
  {
    \__text_map_function:Nw #2 #1
      \q__text_recursion_tail \q__text_recursion_stop
    \prg_break_point:Nn \text_map_break: { }
  }
\bool_lazy_or:nnTF
  { \sys_if_engine_luatex_p: }
  { \sys_if_engine_xetex_p: }
  {
    \cs_new:Npn \__text_map_function:Nw #1#2
      {
        \__text_if_recursion_tail_stop_do:Nn #2 { \text_map_break: }
        #1 {#2}
        \__text_map_function:Nw #1
      }
  }
  {
    \cs_new:Npn \__text_map_function:Nw #1#2
      {
        \__text_if_recursion_tail_stop_do:Nn #2 { \text_map_break: }
        \bool_lazy_and:nnTF
          { \tl_if_single_token_p:n {#2} }
          { ! \token_if_cs_p:N #2 }
          {
            \int_compare:nNnTF { `#2 } > { "80 }
              {
                \int_compare:nNnTF { `#2 } < { "E0 }
                  { \__text_map_function:NNN }
                  {
                     \int_compare:nNnTF { `#2 } < { "F0 }
                       { \__text_map_function:NNNN }
                       { \__text_map_function:NNNNN }
                  }
              }
              { \__text_map_function:Nn }
                #1 #2
          }
          { \__text_map_function:Nn #1 {#2} }
      }
    \cs_new:Npn \__text_map_function:NNN #1#2#3
      { \__text_map_function:Nn #1 {#2#3} }
    \cs_new:Npn \__text_map_function:NNNN #1#2#3#4
      { \__text_map_function:Nn #1 {#2#3#4} }
    \cs_new:Npn \__text_map_function:NNNNN #1#2#3#4#5
      { \__text_map_function:Nn #1 {#2#3#4#5} }
    \cs_new:Npn \__text_map_function:Nn #1#2
      {
        #1 {#2}
        \__text_map_function:Nw #1
      }
  }
\cs_new:Npn \text_map_break:
  { \prg_map_break:Nn \text_map_break: { } }
\cs_new:Npn \text_map_break:n
  { \prg_map_break:Nn \text_map_break: }
\cs_new_protected:Npn \text_map_inline:nn #1#2
  {
    \int_gincr:N \g__kernel_prg_map_int
    \cs_gset_protected:cpn
      { __text_map_ \int_use:N \g__kernel_prg_map_int :w } ##1 {#2}
    \exp_args:Nnc \text_map_function:nN {#1}
      { __text_map_ \int_use:N \g__kernel_prg_map_int :w }
    \prg_break_point:Nn \text_map_break:
      { \int_gdecr:N \g__kernel_prg_map_int }
  }
\ExplSyntaxOff
\begin{document}
\ExplSyntaxOn
\text_map_inline:nn { Здравей } { #1 ~ }
\ExplSyntaxOff
\end{document}

Answer 3

hyperref has to unprotect the utf8 chars locally to write the bookmarks. You can copy its code:

\documentclass{article}
\usepackage[T2A]{fontenc}
\begin{document}
\makeatletter
\def\antshar@expand@utfvii{%
    \count@"C2
    @tempcnta"F5
    \def\UTFviii@tmp{\expandafter\def\expandafter~\expandafter{~}}%
    \UTFviii@loop
  }
\ExplSyntaxOn
\group_begin:
\antshar@expand@utfvii
\tl_set:Nx \l_tmpa_tl {Здравей}
\tl_map_inline:Nn \l_tmpa_tl { #1 ~ }
\group_end:
\ExplSyntaxOff
\end{document}