Expandable macro that extracts the first character of UTF-8/cyrillic string without additional packages

Question

I would like to have an expandable macro that extracts the first (and sometimes the second) character of UTF-8/Cyrillic text strings without using additional packages. No simple solutions from TeX or LaTeX work with UTF-8/Cyrillic strings.

I give below an example of a working macro, which is partially taken from Get the first and second character of a macro argument :

\documentclass{article}
\usepackage[T2A]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage[russian]{babel}
\makeatletter
\newcommand{\firstof}[1]{@car#1@nil}
\makeatother
\begin{document}
\firstof{Vladimir}
\end{document}

Unfortunately, this example fails with the error Error: Invalid UTF-8 byte sequence (Ð\par) using Cyrillic strings like \firstof{Владимир}.

I roughly understand that by default TeX is not adapted to manipulating strings with multibyte characters, but this problem is solved in some packages. However, I do not want to use other packages for such a simple problem (as it seems at first glance) and I will be grateful to the community for help and tips.

Ideally, I would like to have an expandable macro like \newcommand{\firstof}[2][1]{.....}, which by default for UTF-8/Cyrillic strings returns the value of the first character, for example, in the case of \firstof{Владимир} returns В, and for \firstof[2]{Владимир} returns Вл, and these chars could be used in /ifx to compare with others and written to a file using \write.

Answer 1

This should probably work. I defined \headof and \tailof (using code stolen from here and here), which do what their names promise. If you use \headof* (or \tailof*), it will expand its argument, so you can effortlessly get the nth character out of a sequence by nesting \headof* and \tailof* (pretty much like car and cdr in lisp, if you're into that). For example, the fourth character of Владимир can be extracted with

\headof*{\tailof*{\tailof*{\tailof{Владимир}}}}

Annoyingly easy :)

You can then compare strings using the primitive \pdfstrcmp or a higher level \str_if_eq:eeTF. \ifx won't work because it compares two tokens, and л (for example) is two tokens by itself (assuming you're using pdfTeX, of course).

If the argument is empty, the result is also empty. If the head is a group of tokens (within {...}), the group is treated as a single thing and returned without the outer braces.

\documentclass{article}
\usepackage[T2A]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage[russian]{babel}
\ExplSyntaxOn
\NewExpandableDocumentCommand \headof { s +m }
  { \IfBooleanT {#1} { \exp_args:Ne } \crosfield_text_head:n {#2} }
\NewExpandableDocumentCommand \tailof { s +m }
  { \IfBooleanT {#1} { \exp_args:Ne } \crosfield_text_tail:n {#2} }
\cs_new:Npn \crosfield_text_head:n #1
  {
    \tl_if_head_is_N_type:nTF {#1}
      { __crosfield_get_head:Nw #1 \q_stop }
      {
        \tl_if_head_is_group:nTF {#1}
          { \exp_not:o { \use_i_delimit_by_q_stop:nw #1 \q_stop } }
          { \tl_if_empty:nTF {#1} { } { ~ } }
      }
  }
\cs_new:Npn \crosfield_text_tail:n #1
  {
    \tl_if_head_is_N_type:nTF {#1}
      { __crosfield_get_tail:Nw #1 \q_stop }
      {
        \tl_if_head_is_group:nTF {#1}
          { \exp_not:o { \use_none:n #1 } }
          { \tl_if_empty:nTF {#1} { } { \exp_not:o { \exp:w \exp_end_continue_f:w #1 } } }
      }
  }
\bool_lazy_or:nnTF
    { \sys_if_engine_luatex_p: }
    { \sys_if_engine_xetex_p: }
  {
    \cs_new:Npn __crosfield_get_head:Nw #1 #2 \q_stop { \exp_not:N #1 }
    \cs_new:Npn __crosfield_get_tail:Nw #1 #2 \q_stop { \exp_not:n {#2} }
    \use_none:n
  }
  { \makeatletter \use:n }
  {
    \makeatother
    \cs_new:Npn __crosfield_get_head:Nw
      { __crosfield_head_tail:NNw \use_i:nn }
    \cs_new:Npn __crosfield_get_tail:Nw
      { __crosfield_head_tail:NNw \use_ii:nn }
    \cs_new:Npn __crosfield_head_tail:NNw #1 #2 #3 \q_stop
      {
        \use:e
          {
            \exp_not:N __crosfield_head_tail:w
              \exp_not:o { \token_to_meaning:N #2 }
              \tl_to_str:n { UTFviii@ one @octets } ~
          }   \q_stop { #2 #3 } #1
      }
    \use:e
      {
        \cs_new:Npn \exp_not:N __crosfield_head_tail:w
          #1 \tl_to_str:n { UTFviii@ } #2 \tl_to_str:n { @octets } ~ #3
          \exp_not:N \q_stop #4 #5
      }
      {
        \str_case:nnTF {#2}
          {
            { one   } { __crosfield_head_or_tail:NNw #5 }
            { two   } { __crosfield_head_or_tail:NNNw #5 }
            { three } { __crosfield_head_or_tail:NNNNw #5 }
            { four  } { __crosfield_head_or_tail:NNNNNw #5 }
          }
          { #4 \q_stop }
          { \ERROR? }
      }
    \cs_new:Npn __crosfield_head_or_tail:NNw #1 #2 #3 \q_stop
      { \exp_not:o { #1 {#2} {#3} } }
    \cs_new:Npn __crosfield_head_or_tail:NNNw #1 #2#3 #4 \q_stop
      { \exp_not:o { #1 {#2#3} {#4} } }
    \cs_new:Npn __crosfield_head_or_tail:NNNNw #1 #2#3#4 #5 \q_stop
      { \exp_not:o { #1 {#2#3#4} {#5} } }
    \cs_new:Npn __crosfield_head_or_tail:NNNNNw #1 #2#3#4#5 #6 \q_stop
      { \exp_not:o { #1 {#2#3#4#5} {#6} } }
  }
\cs_new_eq:NN \StrCompare \str_if_eq:eeTF
\ExplSyntaxOff
\begin{document}
\StrCompare
  {\headof*{\tailof{Владимир}}}{\detokenize{л}}
  {\true}{\false}
\end{document}

Answer 2

Each byte of the UTF-8 encoding is a separate token in pdflatex, however you can recognise the leading token which tells you how many bytes are needed. This version covers the one and two byte cases.

\documentclass{article}
\usepackage[T2A]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage[russian]{babel}
\makeatletter
\newcommand{\firstof}[1]{\expandafter\checkfirst#1@nil}
\def\checkfirst#1{%
  \ifx\UTFviii@two@octets#1%
  \expandafter\gettwooctets
  \else
  \expandafter@car\expandafter#1%
  \fi
}
\def\gettwooctets#1#2#3@nil{\UTFviii@two@octets#1#2}
\makeatother
\begin{document}
\firstof{Vladimir}
\firstof{Владимир}
\end{document}

---

If you want to handle the rest of the input as opposed to discarding everything after the first letter, you can make a small change so that you pass in a command to appy to the remaning text. If you pass in \gobble it extracts as before. If you pass in \firstofx\gobble then it exctracts the first letter of the remaining text so you get two letters:

\documentclass{article}
\usepackage[T2A]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage[russian]{babel}
\makeatletter
\newcommand{\firstofx}[2]{\expandafter\checkfirst#2@nil{#1}}
\def\checkfirst#1{%
  \ifx\UTFviii@two@octets#1%
  \expandafter\gettwooctetsx
  \else
  \expandafter\getasciix\expandafter#1%
  \fi
}
\def\getasciix#1#2@nil#3{#1#3{#2}}
\def\gettwooctetsx#1#2#3@nil#4{\UTFviii@two@octets#1#2#4{#3}}
\newcommand\gobble[1]{}
\makeatother
\begin{document}
\firstofx\gobble{Vladimir}
\firstofx{\firstofx\gobble}{Vladimir}
\firstofx\gobble{Владимир}
\firstofx{\firstofx\gobble}{Владимир}
\end{document}

Answer 3

Here's a LuaLaTeX-based solution. The solution doesn't employ the standard string.sub Lua function, at it isn't unicode-aware. Instead, it employs the unicode.utf8.sub function to extract the first n (by default, n=1) characters of the string argument. The macro \firstof is expandable.

% !TEX TS-program = lualatex
\documentclass{article}
\usepackage[russian]{babel}
\usepackage{fontspec}
\setmainfont{Noto Serif} % select a suitable opentype text font
\newcommand\firstof[2][1]{\directlua{%
    tex.sprint ( unicode.utf8.sub ( "#2" , 1 , #1 ) ) }}
\newcommand\zzz{Владимир}
\begin{document}
\firstof{\zzz}, \firstof[2]{\zzz}, \firstof[4]{\zzz}
\end{document}