How can I filter math with Lua toks?

Question

Related to a previous question (How can I avoid TeX environment's when token filtering in LuaLaTeX?), I have been playing with creating a package using Lua to censor text (though it's at an early stage and will need quite a bit of work before it can be trusted with sensitive information).

However, I have been having difficulty getting the package to filter TeX math commands, as Math mode seems to totally oust the otherwise redoubtable toks filter. Is there a way to overcome this problem seen below?

\documentclass{article}
\usepackage{luacode}
\newcommand{\onething}{ • }
\begin{luacode}
--fulsome thanks to TeX.SE users Henri Menke and David Carlisle, without whom none of this would be possible 
local function rndstring()
local toks = token.scan_toks(s)
        local on = true
for n, t in ipairs(toks) do
if on and t.cmdname ==  "letter" or t.cmdname == "other_char" then
               local letter = token.create'onething'
 toks[n] = letter
    end end
 --Drop the token in and move on
        token.put_next(toks)
end
local lft = lua.get_functions_table()
--make a global command
lft[#lft + 1] = rndstring
token.set_lua("rndstring", #lft, "global")
\end{luacode}
\begin{document}
    \rndstring{Test of a thing 1234}
$\epsilon \gamma \delta \div 1234$

\end{document}

Steven B. Segletes · Answer 1 · 2022-02-18T00:20:47.903

This answer does not answer the crux of your question, "How can I filter math with Lua toks?". However, it does address the issue of censoring text, to include math.

My censor package (which can run in pdflatex) has recently been revamped [2022-02-09] to process the arguments of \blackout and \xblackout with token cycles (see tokcycle package for details). This rework provides a capability to address the possibility of macros in the stream of tokens to be censored. Nonetheless, no consideration is given for processing math directly, though several workarounds might be found.

Here, in this answer, however, I expand on the capabilities of the revamped \blackout to try to address the censoring of math in a more automatic manner. Several caveats should be noted: sub/super-scripted arguments must be located in explicit braces for the censoring to work; text that is not to be censored (for example, environment names) must be escaped using the tokcycle method of enclosing the escaped text between | tokens (by default); because censoring is essentially text-based, censored sub/superscripts will be presented in the text size (not script size), which could affect spacing.

I will consider incorporating changes along these lines into a future release of the censor package.

\documentclass{article}
\usepackage{censor}[2022-02-09]
\makeatletter
\xtokcycleenvironment\blackoutenv
  {\ifx.##1\@dump\censored@word\addcytoks[1]{\censordot}\else
    \ifx$##1\@dump\censored@word\addcytoks{##1}\else
    \ifx^##1\@dump\censored@word\addcytoks{##1}\else 
    \ifx_##1\@dump\censored@word\addcytoks{##1}\else
    \@append\censored@word{##1}%
    \tcpeek\@next\ifx\@next\@tcEscapeptr\@dump\censored@word
    \fi\fi\fi\fi\fi}
  {\tctestifcon\ifexpandarg{\expandafter\processtoks\expandafter
    {\expanded{##1}}\@dump\censored@word}{\groupedcytoks{\processtoks{##1}%
    \@dump\censored@word}}\expandargfalse}
  {\tctestifx{~##1}{\@append\censored@word{##1}}%
    {\tctestifx{\expanded##1}{\@dump\censored@word\expandargtrue}%
    {\test@chars{##1}\ifchar\ifmathgreek
    \@append\censored@word{\ensuremath{##1}}\else
    \@append\censored@word{##1}\fi\else
    \test@accents{##1}\ifaccent\@append\censored@word{##1}\tcpop\tc@popped
    \expandafter\@append\expandafter\censored@word\expandafter{\tc@popped}%
    \else\@dump\censored@word\addcytoks{##1}\fi\fi}}}
  {\@dump\censored@word\addcytoks{##1}}
  {\stripgroupingtrue\def\censored@word{}}
  {\@dump\censored@word}
\renewcommand\test@chars[1]{\charfalse\mathgreekfalse
  \ifx$#1\chartrue\else
  \ifx&amp;#1\chartrue\else
  \ifx##1\chartrue\else
  \ifx%#1\chartrue\else
  \ifx_#1\chartrue\else
  \ifx\o#1\chartrue\else
  \ifx\O#1\chartrue\else
  \ifx\oe#1\chartrue\else
  \ifx\OE#1\chartrue\else
  \ifx\aa#1\chartrue\else
  \ifx\AA#1\chartrue\else
  \ifx\ae#1\chartrue\else
  \ifx\AE#1\chartrue\else
  \ifx\l#1\chartrue\else
  \ifx\L#1\chartrue\else
  \ifcensormathgreek\testmathgreek{#1}\ifmathgreek\chartrue\fi\fi%
  \fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi
}
\newif\ifmathgreek
\newcommand\testmathgreek[1]{%
  \ifx\alpha#1\mathgreektrue\else
  \ifx\beta#1\mathgreektrue\else
  \ifx\gamma#1\mathgreektrue\else
  \ifx\delta#1\mathgreektrue\else
  \ifx\epsilon#1\mathgreektrue\else
  \ifx\varepsilon#1\mathgreektrue\else
  \ifx\zeta#1\mathgreektrue\else
  \ifx\eta#1\mathgreektrue\else
  \ifx\theta#1\mathgreektrue\else
  \ifx\vartheta#1\mathgreektrue\else
  \ifx\iota#1\mathgreektrue\else
  \ifx\kappa#1\mathgreektrue\else
  \ifx\lambda#1\mathgreektrue\else
  \ifx\mu#1\mathgreektrue\else
  \ifx\nu#1\mathgreektrue\else
  \ifx\xi#1\mathgreektrue\else
  \ifx\pi#1\mathgreektrue\else
  \ifx\varpi#1\mathgreektrue\else
  \ifx\rho#1\mathgreektrue\else
  \ifx\varrho#1\mathgreektrue\else
  \ifx\sigma#1\mathgreektrue\else
  \ifx\varsigma#1\mathgreektrue\else
  \ifx\tau#1\mathgreektrue\else
  \ifx\upsilon#1\mathgreektrue\else
  \ifx\phi#1\mathgreektrue\else
  \ifx\varphi#1\mathgreektrue\else
  \ifx\chi#1\mathgreektrue\else
  \ifx\psi#1\mathgreektrue\else
  \ifx\omega#1\mathgreektrue\else
  \ifx\Gamma#1\mathgreektrue\else
  \ifx\Delta#1\mathgreektrue\else
  \ifx\Theta#1\mathgreektrue\else
  \ifx\Lambda#1\mathgreektrue\else
  \ifx\Xi#1\mathgreektrue\else
  \ifx\Pi#1\mathgreektrue\else
  \ifx\Sigma#1\mathgreektrue\else
  \ifx\Upsilon#1\mathgreektrue\else
  \ifx\Phi#1\mathgreektrue\else
  \ifx\Psi#1\mathgreektrue\else
  \ifx\Omega#1\mathgreektrue
  \fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi
  \fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi
}
\newif\ifcensormathgreek
\censormathgreektrue
\makeatother
\begin{document}
%\censormathgreekfalse
%\StopCensoring
\blackoutenv
    Test of a thing 1234
$\epsilon \gamma \delta \div 1234$

[
  y \ne mx + \zeta b
]
\begin{|equation|}
  y \ne \frac{m_{1}}{m_{2}}x +  b^{3}
\end{|equation|}
\endblackoutenv
\end{document}

With the \StopCensoring macro uncommented, the result becomes

On the other hand, with censoring in place, but \censormathgreekfalse uncommented, one can save a little on the computation time, if one doesn't mind greek characters remaining uncensored:

How can I filter math with Lua toks?

1 Answers1