8

This seems to have come up a couple times, but I cannot glean an answer to the generic question of how one might protect an arbitrary string in Lua to ensure its characters are printed.

In other words, given some code in a .lua file:

local i_can_be_anything = '&\\%*_'
context(i_can_be_anything)

This balks for any number of reasons, because the characters have special meaning in TeX when ConTeXt pushes them out to the TeX compiler.

The most apparent option to me would be to convert the string in Lua so that it contains the proper escaping or substitutions that can be passed to TeX to achieve the desired outcome, which would be in this case along the lines of:

\&\textbackslash{}\%\&\_

I believe the full list of replacements one would be looking for would be along the lines of (and this is from some Python I wrote):

tex_replacements = [
    (u'{', ur'\{'),
    (u'}', ur'\}'),
    (u'[', ur'{[}'),
    (u']', ur'{]}'),
    (u'\\', ur'\textbackslash{}'),
    (u'$', ur'\$'),
    (u'%', ur'\%'),
    (u'&', ur'\&'),
    (u'#', ur'\#'),
    (u'_', ur'\_'),
    (u'~', ur'\textasciitilde{}'),
    (u'^', ur'\textasciicircum{}'),
]

tex_mapping =  {ord(char):rep for char, rep in tex_replacements}
def escape(s):
    return unicode(s).translate(tex_mapping)

Is there no function in Lua or LuaTeX to achieve this set of substitutions (or otherwise) so that an arbitrary string could be printed?

A straightforward iteration over the characters with a substitution along the lines of the tex_replacements above could do the trick, but I would expect that someone had already written a solution to this problem.

EDIT

It is worth noting that @phg posed a plausible solution of changing the catcodes to vrbcatcode (i.e. \verbatim, or context.sprint(catcodes.numbers.vrbcatcodes, i_can_be_anything)). This does perform the escaping desired, however it seems to have serious issues with line breaking that make it essentially unusable.

Community
  • 1
Brian M. Hunt
  • 14,277
  • 1
    The solution you are looking for is switching the catcode table. vrbcatcodes should suffice for most literal printing, but if you like you can skim the list for a more appropriate one: http://git.contextgarden.net/context/context/blob/master/tex/context/base/catc-def.mkiv (Btw. there is no such thing as a “TeX compiler”.) – Philipp Gesang Jul 23 '14 at 21:21
  • Thanks @phg. I am afraid it is not apparent from the link how one employs said catcodes. Incidentally, and aside, if there is no such thing as a TeX compiler, what is the mechanism by which the TeX source is converted to machine instructions? – Brian M. Hunt Jul 23 '14 at 21:30
  • You can use \pushcatcodetable \setcatcodetable \somecatcodetable to set the table and \popcatcodetable to drop it again (that’s context.pushcatcodetable() context.setcatcodetable(catcodes.numbers.vrbcatcodes) / context.popcatcodetable() at the Lua end, or use context.sprint(n, ...) with the catcode table as the first argument). As for the terminology: TeX is interpreted by a TeX engine. – Philipp Gesang Jul 23 '14 at 21:52
  • (Untested) context(catcodes.numbers.vrbcatcodes, i_can_be_anything) should also work. – Aditya Jul 24 '14 at 04:14
  • 1
    @Aditya Are you sure? It calls tex.sprint() explicitly with the currently active catcode table. There is context.verbatim(), though, which sets the vrbcatcodes table. – Philipp Gesang Jul 24 '14 at 05:24
  • It looks like the proper catcode might be \notcatcodes from catc-def.mkiv – Brian M. Hunt Jul 25 '14 at 10:42
  • @phg: Clearly, I was mistaken. See my answer for a "proper" solution. – Aditya Jul 29 '14 at 04:29

2 Answers2

5

This answer is based on an email from Hans Hagen. One way to achieve the desired output is using formatters. For example:

\starttext
\startluacode
  local text='&\%*_'
  context("%!tex!", text)
\stopluacode
\stoptext

The %!tex! is plug to the formatters and tex is one of the pre-defined formatters. Other pre-defined formatters are lua and xml (which escape strings for Lua and XML, respectively).

The above code is equivalent to 

context(string.formatters["%!tex!"](text))

which is equivalent to the rather low-level

context(lpeg.match(lpeg.patterns.texescape,text))

The corresponding lpeg patterns for lua and xml formatters are lpeg.patterns.luaescape and lpeg.patterns.xmlescape. These are defined in util-str.lua as:

patterns.xmlescape = Cs((P("<")/"&lt;" + P(">")/"&gt;" + P("&")/"&amp;" + P('"')/"&quot;" + P(1))^0)
patterns.texescape = Cs((C(S("#$%\\{}"))/"\\%1" + P(1))^0)
patterns.luaescape = Cs(((1-S('"\n'))^1 + P('"')/'\\"' + P('\n')/'\\n"')^0) -- maybe also \0
patterns.luaquoted = Cs(Cc('"') * ((1-S('"\n'))^1 + P('"')/'\\"' + P('\n')/'\\n"')^0 * Cc('"'))

The luaquoted pattern is not used in the core.

Aditya
  • 62,301
  • Thanks Aditya - why do you think one would prefer these over the discussed catcodes? other than Hans saying so, which ought to be compelling enough :-) —just curious. – Brian M. Hunt Jul 29 '14 at 10:36
  • One note, context("%!tex!", text) seems to handle "\" in an unintuitive way. For example, the string "\escape" results in "<LINEBREAK>escape". This may not be the desired behaviour. The catcodes solution does not have this particular caveat. – Brian M. Hunt Jul 29 '14 at 13:12
  • It converts \ to \\, which I guess is mapped to \crlf. – Aditya Jul 29 '14 at 14:26
1

I am summarizing the comments here, but the gist is:

@phg suggested that one can escape by using catcodes. There is the verbatim catcode, but it has issues with linebreaks.

An alternative, albeit undocumented but seemingly preferable (i.e. escapes without wrapping issues), catcode is \notcatcodes.

One can employ this in Lua to escape any string with the following, for example:

context.sprint(catcodes.numbers.notcatcodes, str)
-- or --
context.pushcatcodetable()
context.setcatcodetable(catcodes.numbers.vrbcatcodes)
context(str)
context.popcatcodetable()
Brian M. Hunt
  • 14,277