Automatically formating all-caps passages with lualatex

Question

I want to apply specific formatting to all all-caps passages in a longer text, specifically tracking/letterspacing using fontspec's mechanism, so it can be adjusted for different font-styles. I have a macro that does this nicely but I really want the macro to be applied automatically on the text. So I have tried to adapt the accepted answer about colourising specific words which works most of the time:

% !TEX TS-program = lualatex
\documentclass[12pt]{book}
\usepackage{fontspec}
\newcommand{\TEST}{Write out “TEST”!}
\newcommand{\test}{Write out “TEST”!}
%use underline for testing as it makes it easier to see where the command actually is applied 
\usepackage{ulem}
\usepackage{luacode} % for "luacode" environment and "\luastring" macro
%% Lua-side code
\begin{luacode}
function letterspace ( buff )
   buff = string.gsub ( buff, [[([ “])([A-Z][A-Z]+)([”,.;:?%!- ])]],  "\emph{%1%2%3}" )
   return buff
end
\end{luacode}
%% TeX-side code
\newcommand\letterspaceOn{\directlua{luatexbase.add_to_callback 
   ( "process_input_buffer" , letterspace , "letterspace" )}}
\newcommand\letterspaceOff{\directlua{luatexbase.remove_from_callback 
   ( "process_input_buffer" , "letterspace" )}}
\AtBeginDocument{\letterspaceOn} % turn Lua function on by default
\begin{document}
\chapter{The quick brown fox jumps OFF OF A cliff}
\section{Zorro juMPs over the lazy dog}
(EVERY
EVERY
“EVERY”
EVERY day, foo's foo doesn't foo.)
THE QUICK BROWN FOX JUMPS OVER THE LAZY DOG.
quick THE quick A! brown fox JUMPS OVER the lazy dog.
Another “quick” brown fox
\test
%Problematic lines:
%THE “QUICK”, BROWN FOX! JUMPS OVER? THE LAZY DOG.
%\TEST
%{\LARGE test TEST}
\letterspaceOff  % switch off the Lua function
The quick brown fox JUMPS! over the lazy dog.
THE QUICK, BROWN FOX! JUMPS OVER? THE LAZY DOG.
THE “QUICK”, BROWN FOX! JUMPS OVER? THE LAZY DOG.
\TEST
{\LARGE test TEST}
\letterspaceOn   % switch Lua function back on
The quick brown fox JUMPS! over the lazy dog.
\end{document}

There are still several issues with this (lines commented out in the MWE):

1. LaTeX-commands containing all-caps have be excluded from the match. RegEx offers negative lookbehinds that could be used but I am not familiar with lua's pattern matching and couldn't find a fitting equivalent.

2. I don't understand why I get a “String contains an invalid utf-8 sequence”-error for the quotation marks around "“QUICK”".

3. A perfect solution would also underline (letterspace) the word “TEST” written by the macro \test but this would only be the icing on the cake.

Answer 1

LuaTeX captures are not REGEX, they are very limited and hence one needs to be careful. The LPEG module is better but has in my opinion a difficult syntax and a steep learning curve.

Mounting callbacks is dangerous, as TeX would read a full line and then hand over to Lua. Hence you will be underlining commands such as {\LARGE \TEST}. It is always best to delimit the capture with a marker. In the example below I use two backticks (``). Also preferable to use unicode.utf8.gsub rather than string.gsub

\documentclass[12pt]{book}
\usepackage{fontspec,textcase} 
\usepackage{luacode} 
\newcommand\textsmallcaps[1]{\textsc{\MakeTextLowercase{#1}}}
\begin{luacode}
function letterspace ( buff )
  buff = unicode.utf8.gsub (buff, '][[`][`]',   function(w)
               if w ~=nil then 
                   return '\textsmallcaps{'..w..'}' 
               else
                 return buff
               end

             end

           ) 
  return buff

end
\end{luacode}
\newcommand\letterspaceOn{\directlua{luatexbase.add_to_callback 
   ( "process_input_buffer" , letterspace , "letterspace" )}}
\newcommand\letterspaceOff{\directlua{luatexbase.remove_from_callback 
   ( "process_input_buffer" , "letterspace" )}}
\AtBeginDocument{\letterspaceOn}
\begin{document}
The quick brown fox JUMPS ! over the lazy dog.
THE “QUICK”, BROWN FOX! JUMPS OVER? THE LAZY. {\LARGE DOG}.
\end{document}