How to make an array of tokens?

Question

I'm trying to code a function with the help of some characters in Devanagari script. I've replaced the characters with some other characters of a totally different script. This is the MWE to do it.

\documentclass{article}
\usepackage{fontspec}
\setmainfont{Sanskrit 2003} %Any Devanagari font.
\newfontfamily{\doulo}{Doulos SIL}
\setlength{\parindent}{0cm}
\makeatletter
\catcode`\क=\active
\protected\def क{\doulo k}
\catcode`\ख=\active
\protected\def ख{\doulo kʰ}
\catcode`\ड=\active
\protected\def ड{\doulo ɖ}
\makeatother

\begin{document}
    कखड
\end{document}

Doulos SIL font is necessary for the characters to appear and can be obtained from here

This produces the following -

Now I want 'ə' this character to be added conditionally. It should only be added when another diacritic of Devanagari i.e. '्' is absent. (Which in a way means whenever '्' comes 'ə' symbol should not be added and for every character which is not followed by the same 'ə' should be added.) For this conditional setting, I've coded this.

\documentclass{article}
\usepackage{fontspec}
\setmainfont{Sanskrit 2003} %Any Devanagari font.
\newfontfamily{\doulo}{Doulos SIL}    
\setlength{\parindent}{0cm}
\makeatletter
\catcode`\क=\active
\protected\def क{\bgroup\doulo k\futurelet\tmp\viramacheck}
\catcode`\ख=\active
\protected\def ख{\bgroup\doulo kʰ\futurelet\tmp\viramacheck}
\catcode`\ड=\active
\protected\def ड{\bgroup\doulo ɖ\futurelet\tmp\viramacheck}
\protected\def\viramacheck{\ifx\tmp ्\egroup\expandafter\@gobble\else ə\egroup\fi}
\makeatother

\begin{document}
    क ख ड\\
    क् ख् ड्
\end{document}

This provides -

This much is clear to me. Now what I wanted to know is, instead of having one conditional antecedent i.e. '्' I want to have an array of antecedents.

eg. To put it into a logical language, if x is followed by ् and 'a' and 'b' and 'c' do not add 'ə'

\documentclass{article}
\usepackage{fontspec}
\setmainfont{Sanskrit 2003} %Any Devanagari font.
\newfontfamily{\doulo}{Doulos SIL}    
\setlength{\parindent}{0cm}
\makeatletter
\catcode`\क=\active
\protected\def क{\bgroup\doulo k\futurelet\tmp\viramacheck}
\catcode`\ख=\active
\protected\def ख{\bgroup\doulo kʰ\futurelet\tmp\viramacheck}
\catcode`\ड=\active
\protected\def ड{\bgroup\doulo ɖ\futurelet\tmp\viramacheck}
\protected\def\viramacheck{\ifx\tmp ्\egroup\expandafter\@gobble\else ə\egroup\fi}
\makeatother

\begin{document}
    कaखbडc
\end{document}

The output is -

and what I want is -

How to make an array of these conditionals i.e. '्', 'a', 'b' & 'c'?

As suggested by @siracusa I've edited the code in the following way. This is the final and fully ready code.

\documentclass{article}
\usepackage{fontspec}
\setmainfont{Sanskrit 2003} %Any Devanagari font.
\newfontfamily{\doulo}{Doulos SIL}    
\setlength{\parindent}{0cm}
\makeatletter
\def\ifischar#1{%
    \ifx\tmp#1%
    \expandafter\@firstoftwo
    \else
    \expandafter\@secondoftwo
    \fi
}
\catcode`\क=\active
\protected\def क{\bgroup\doulo k\futurelet\tmp\viramacheck}
\catcode`\ख=\active
\protected\def ख{\bgroup\doulo kʰ\futurelet\tmp\viramacheck}
\catcode`\ड=\active
\protected\def ड{\bgroup\doulo ɖ\futurelet\tmp\viramacheck}
\catcode`\अ=\active
\protected\def अ{\bgroup\doulo ə}
\catcode`\ा=\active
\protected\def ा{\doulo a}
\catcode`\आ=\active
\protected\def आ{\doulo a}
\catcode`\ि=\active
\protected\def ि{\doulo i}
\catcode`\इ=\active
\protected\def इ{\doulo i}
\catcode`\ी=\active
\protected\def ी{\doulo iː}
\catcode`\ई=\active
\protected\def ई{\doulo iː}
\catcode`\ु=\active
\protected\def ु{\doulo u}
\catcode`\उ=\active
\protected\def उ{\doulo u}
\catcode`\ू=\active
\protected\def ू{\doulo uː}
\catcode`\ऊ=\active
\protected\def ऊ{\doulo uː}
\catcode`\े=\active
\protected\def े{\doulo e}
\catcode`\ए=\active
\protected\def ए{\doulo e}
\catcode`\ै=\active
\protected\def ै{\doulo ə͡i}
\catcode`\ऐ=\active
\protected\def ऐ{\doulo ə͡i}
\catcode`\ो=\active
\protected\def ो{\doulo o}
\catcode`\ओ=\active
\protected\def ओ{\doulo o}
\catcode`\ौ=\active
\protected\def ौ{\doulo ə͡u}
\catcode`\औ=\active
\protected\def औ{\doulo ə͡u}
\catcode`\ं=\active
\protected\def ं{\doulo ũ}
\protected\def\viramacheck{%
    \ifischar{्}{\egroup\@gobble}{%
        \ifischar{ा}{\egroup\@gobble}{%
            \ifischar{ी}{\egroup\@gobble}{%
                \ifischar{ू}{\egroup\@gobble}{%
                    \ifischar{ि}{\egroup\@gobble}{%
                        \ifischar{ु}{\egroup\@gobble}{%
                            \ifischar{े}{\egroup\@gobble}{%
                                \ifischar{ै}{\egroup\@gobble}{%
                                    \ifischar{ो}{\egroup\@gobble}{%
                                        \ifischar{ौ}{\egroup\@gobble}{%
                                            \ifischar{अ}{\egroup\@gobble}{%
                                                \ifischar{आ}{\egroup\@gobble}{%
                                                    \ifischar{ई}{\egroup\@gobble}{%
                                                        \ifischar{ई}{\egroup\@gobble}{%
                                                            \ifischar{उ}{\egroup\@gobble}{%
                                                                \ifischar{ऊ}{\egroup\@gobble}{%
                                                                    \ifischar{ए}{\egroup\@gobble}{%
                                                                        \ifischar{ऐ}{\egroup\@gobble}{%
                                                                            \ifischar{ओ}{\egroup\@gobble}{%
                                                                                \ifischar{औ}{\egroup\@gobble}{%
                                                                                    ə\egroup
}}}}}}}}}}}}}}}}}}}}%
}
\makeatother

\begin{document}
    काखीडूअए
\end{document}

The output -

This solved the problem of adding character 'ə', but if we look at line number 23, 31 & 39 ा, ी & ू are defined as a, iː & uː respectively. They are not coming in the output. On the contrary if we look at line number 21 & 45 अ & ए are defined as ə & e respectively and they are coming in the output. Probably there is something wrong with the diacritics. Can we solve this?

You could nest several \ifx tests to check for different characters. See also Check if a string contains a given character — siracusa, Jul 11 '19 at 09:10
You mean to say I'll have to add \protected\def\viramacheck{\ifx\tmp a\egroup\expandafter@gobble\else ə\egroup\fi}? — Niranjan, Jul 11 '19 at 09:49
but I tried doing this and then the previous definition does not work. — Niranjan, Jul 11 '19 at 09:51
What do you mean by your last paragraph? How are a, i: and u: related to the input? If you want a lot of different, multi-character replacements, the l3regex engine may be a better option. — siracusa, Jul 11 '19 at 15:27
ा, ी & ू are the characters which are defined as a, i: and u: In the input I've used काखीडू which are composed of क + ा, ख + ी, ड + ू — Niranjan, Jul 11 '19 at 15:37
I've noticed that diacritics on their own follow the definition, but when combined with other characters, they do not follow the same. — Niranjan, Jul 11 '19 at 15:40
I think the problem is that you gobble these diacritic characters away. For example, the line \ifischar{ा}{\egroup\@gobble}{% shouldn't have the \@gobble in it because you actually want your next (active) character to be exectuted. Something with the group nesting also seems to be wrong. — siracusa, Jul 11 '19 at 16:24

score 2 · Answer 1 · answered Jul 11 '19 at 13:21

As I don't have the right fonts available on my system, this is kind of a shot in the dark. The idea is to have nested \ifx checks, like the following structure:

\ifx\tmp X
\else
  \ifx\tmp Y
  \else
  \fi
\fi

Because you are using \@gobble to remove the following token from the input stream, you have to be a bit more careful, because TeX leaves parts of the \else/\fi structure in the token stream. So the \@gobble would remove these tokens instead of the following character (this is why you have \expandafter in your test). For nested ifs putting the right number of \expandafters is too tedious.

What you can use instead is a macro like

% use as \ifischar{X}{then part}{else part}
\def\ifischar#1{%
    \ifx\tmp#1%
        \expandafter\@firstoftwo
    \else
        \expandafter\@secondoftwo
    \fi
}

which performs the same check but takes care no spurious tokens are left in the token stream. Your \viramacheck macro would then become

\protected\def\viramacheck{%
    \ifischar{्}{\egroup\@gobble}{%
    \ifischar{a}{\egroup\@gobble}{%
    \ifischar{b}{\egroup\@gobble}{%
    \ifischar{c}{\egroup\@gobble}{%
        ə\egroup
    }}}}%
}

I think this gives the correct result, but I can't tell for sure.

Thanks for the effort, but probably something weird happened to the code. Please see the edit. — Niranjan, Jul 11 '19 at 14:38
Should I make one MWE with English characters? The only risk with English characters is if I make them active some commands may go wrong. If you can guide me how to avoid that, I'll make one example in English for this problem. — Niranjan, Jul 11 '19 at 14:40
Oh! I did not remove it, but it was in a wrong place. Now when I put it back everything is fine. I even worked it out with my script and it was fine there too. Just one last difficulty. Please see the edit. — Niranjan, Jul 11 '19 at 15:17

Niranjan · Accepted Answer · 2019-07-12T11:46:46.640

@siracusa suggested the right way of formatting my code and now I've this, which is fully ready.

\documentclass{article}
\usepackage{fontspec}
\setmainfont{Sanskrit 2003} %Any Devanagari font.
\newfontfamily{\doulo}{Doulos SIL}    
\setlength{\parindent}{0cm}
\makeatletter
\def\ifischar#1{%
    \ifx\tmp#1%
    \expandafter\@firstoftwo
    \else
    \expandafter\@secondoftwo
    \fi
}
\catcode`\क=\active
\protected\def क{\bgroup\doulo k\futurelet\tmp\viramacheck}
\catcode`\ख=\active
\protected\def ख{\bgroup\doulo kʰ\futurelet\tmp\viramacheck}
\catcode`\ग=\active
\protected\def ग{\bgroup\doulo g\futurelet\tmp\viramacheck}
\catcode`\घ=\active
\protected\def घ{\bgroup\doulo gʱ\futurelet\tmp\viramacheck}
\catcode`\ङ=\active
\protected\def ङ{\bgroup\doulo ŋ\futurelet\tmp\viramacheck}
\catcode`\च=\active
\protected\def च{\bgroup\doulo ʧ\futurelet\tmp\viramacheck}
\catcode`\छ=\active
\protected\def छ{\bgroup\doulo ʧʰ\futurelet\tmp\viramacheck}
\catcode`\ज=\active
\protected\def ज{\bgroup\doulo ʤ\futurelet\tmp\viramacheck}
\catcode`\झ=\active
\protected\def झ{\bgroup\doulo ʤʱ\futurelet\tmp\viramacheck}
\catcode`\ञ=\active
\protected\def ञ{\bgroup\doulo ɲ\futurelet\tmp\viramacheck}
\catcode`\ट=\active
\protected\def ट{\bgroup\doulo ʈ\futurelet\tmp\viramacheck}
\catcode`\ठ=\active
\protected\def ठ{\bgroup\doulo ʈʰ\futurelet\tmp\viramacheck}
\catcode`\ड=\active
\protected\def ड{\bgroup\doulo ɖ\futurelet\tmp\viramacheck}
\catcode`\ढ=\active
\protected\def ढ{\bgroup\doulo ɖʱ\futurelet\tmp\viramacheck}
\catcode`\ण=\active
\protected\def ण{\bgroup\doulo ɳ\futurelet\tmp\viramacheck}
\catcode`\त=\active
\protected\def त{\bgroup\doulo t̪\futurelet\tmp\viramacheck}
\catcode`\थ=\active
\protected\def थ{\bgroup\doulo t̪ʰ\futurelet\tmp\viramacheck}
\catcode`\द=\active
\protected\def द{\bgroup\doulo d̪\futurelet\tmp\viramacheck}
\catcode`\ध=\active
\protected\def ध{\bgroup\doulo d̪ʱ\futurelet\tmp\viramacheck}
\catcode`\न=\active
\protected\def न{\bgroup\doulo n̪\futurelet\tmp\viramacheck}
\catcode`\प=\active
\protected\def प{\bgroup\doulo p\futurelet\tmp\viramacheck}
\catcode`\फ=\active
\protected\def फ{\bgroup\doulo pʰ\futurelet\tmp\viramacheck}
\catcode`\ब=\active
\protected\def ब{\bgroup\doulo b\futurelet\tmp\viramacheck}
\catcode`\भ=\active
\protected\def भ{\bgroup\doulo bʱ\futurelet\tmp\viramacheck}
\catcode`\म=\active
\protected\def म{\bgroup\doulo m\futurelet\tmp\viramacheck}
\catcode`\य=\active
\protected\def य{\bgroup\doulo j\futurelet\tmp\viramacheck}
\catcode`\र=\active
\protected\def र{\bgroup\doulo ɾ\futurelet\tmp\viramacheck}
\catcode`\ऱ=\active
\protected\def ऱ{\bgroup\doulo ɾ\futurelet\tmp\viramacheck}
\catcode`\ल=\active
\protected\def ल{\bgroup\doulo l\futurelet\tmp\viramacheck}
\catcode`\व=\active
\protected\def व{\bgroup\doulo w\futurelet\tmp\viramacheck}
\catcode`\श=\active
\protected\def श{\bgroup\doulo ʃ\futurelet\tmp\viramacheck}
\catcode`\ष=\active
\protected\def ष{\bgroup\doulo ʂ\futurelet\tmp\viramacheck}
\catcode`\स=\active
\protected\def स{\bgroup\doulo s\futurelet\tmp\viramacheck}
\catcode`\ह=\active
\protected\def ह{\bgroup\doulo h\futurelet\tmp\viramacheck}
\catcode`\ळ=\active
\protected\def ळ{\bgroup\doulo ɭ\futurelet\tmp\viramacheck}
\catcode`\अ=\active
\protected\def अ{\bgroup\doulo ə}
\catcode`\ा=\active
\protected\def ा{\doulo a}
\catcode`\आ=\active
\protected\def आ{\doulo a}
\catcode`\ि=\active
\protected\def ि{\doulo i}
\catcode`\इ=\active
\protected\def इ{\doulo i}
\catcode`\ी=\active
\protected\def ी{\doulo iː}
\catcode`\ई=\active
\protected\def ई{\doulo iː}
\catcode`\ु=\active
\protected\def ु{\doulo u}
\catcode`\उ=\active
\protected\def उ{\doulo u}
\catcode`\ू=\active
\protected\def ू{\doulo uː}
\catcode`\ऊ=\active
\protected\def ऊ{\doulo uː}
\catcode`\े=\active
\protected\def े{\doulo e}
\catcode`\ए=\active
\protected\def ए{\doulo e}
\catcode`\ै=\active
\protected\def ै{\doulo ə͡i}
\catcode`\ऐ=\active
\protected\def ऐ{\doulo ə͡i}
\catcode`\ो=\active
\protected\def ो{\doulo o}
\catcode`\ओ=\active
\protected\def ओ{\doulo o}
\catcode`\ौ=\active
\protected\def ौ{\doulo ə͡u}
\catcode`\औ=\active
\protected\def औ{\doulo ə͡u}
\catcode`\ं=\active
\protected\def ं{\doulo ũ}
\protected\def\viramacheck{%
    \ifischar{्}{\egroup\@gobble}{%
        \ifischar{ा}{\egroup}{%
            \ifischar{ी}{\egroup}{%
                \ifischar{ू}{\egroup}{%
                    \ifischar{ि}{\egroup}{%
                        \ifischar{ु}{\egroup}{%
                            \ifischar{े}{\egroup}{%
                                \ifischar{ै}{\egroup}{%
                                    \ifischar{ो}{\egroup}{%
                                        \ifischar{ौ}{\egroup}{%
                                            \ifischar{अ}{\egroup}{%
                                                \ifischar{आ}{\egroup}{%
                                                    \ifischar{ई}{\egroup}{%
                                                        \ifischar{ई}{\egroup}{%
                                                            \ifischar{उ}{\egroup}{%
                                                                \ifischar{ऊ}{\egroup}{%
                                                                    \ifischar{ए}{\egroup}{%
                                                                        \ifischar{ऐ}{\egroup}{%
                                                                            \ifischar{ओ}{\egroup}{%
                                                                                \ifischar{औ}{\egroup}{%
                                                                                    ə\egroup
}}}}}}}}}}}}}}}}}}}}%
}
\makeatother

\begin{document}
काखीडूअए क की ा क्\\
क् क का कि की कु कू के कै को कौ कं\\
ढ ढा ढि ढी ढु ढू ढे ढै ढो ढौ ढह\\
\end{document}

You don't need the \blank command here, I think, as it just expands to an empty replacement text. — siracusa, Jul 12 '19 at 09:40

How to make an array of tokens?

2 Answers2