Hyphenating at active characters

Question

Let's say I want to make certain letters active and do some devious things with them, like

\catcode`y=13
\defy{\leavevmode\raise.1ex\hbox{\char`y}}

(I know this is terrible in that it prohibits using control sequences containing a "y".)
Now I would like TeX to still consider hyphenating at the "y" normally (poly-ethy-lene). I know that TeX won't break a word containing an \hbox, but is it possible to trick TeX into thinking that there is just an innocent letter? Why I think it might be possible:

• TeX doesn't care per se whether or not the characters in a word are active when looking for a suitable breakpoint;
• at the time the active char occurs, it still knows which letter should be there.

A solution as general as possible is encouraged (i.e., regardless of what the active character does), but if that's not feasible, here is what I might want to make such an active character do:

• print itself normally
• insert itself into an \hbox and \raise it
• add some kerns before or after it
• manipulate its appearance with \pdfliterals

Answer 1

Using LuaTeX, it's possible to avoid making the letters active and just manipulate them after hyphenation, as Ulrike Fischer pointed out in the comments.

Following is an implementation of this approach, inspired by the chickenize package. Since this is the first time I'm writing code in Lua, any suggestions are welcome.

transform.lua

First, I define a function iterating through the glyph nodes in a list and checking if the character has an entry in a table called chartbl, in which case it calls a function transform_char that uses the values in the table to manipulate the glyph node. This function is then registered as a post_linebreak_filter, so that it will be applied to a paragraph list after it was broken into lines (therefore obeying hyphenation patterns):

transform_chars = function(head)
  for l in node.traverse_id(node.id("hhead"),head) do
    for n in node.traverse_id(node.id("glyph"),l.head) do
      chr = n.char
      if chartbl[chr] ~= nil then
        transformed = transform_char(n)
        l.head = node.insert_before(l.head,n,node.copy(transformed))
        node.remove(l.head,n)
      end
    end
  end
  return head
end

callback.register("post_linebreak_filter",transform_chars)

Now transform_char(n) can be adapted to the specific needs. In this case, we add a kern and a pdfliteral before and after the character and virtually shift the character:

transform_char = function(c)
  kbfn = node.new(node.id("kern")) -- additional kern before char
  pdfbfn = node.new(node.id("whatsit"),node.subtype("pdf_literal")) -- pdf literal before
  cn = node.new(node.id("glyph")) -- char
  cn = node.copy(c)
  pdfan = node.new(node.id("whatsit"),node.subtype("pdf_literal")) -- pdf literal after
  kan = node.new(node.id("kern")) -- additional kern after char

  tbl = chartbl[c.char]

  kbfn.kern = tex.sp(tbl["kbf"])
  pdfbfn.data = tbl["pdfbf"]
  cn.xoffset = tex.sp(tbl["xoff"])
  cn.yoffset = tex.sp(tbl["yoff"])
  pdfan.data = tbl["pdfa"]
  kan.kern = tex.sp(tbl["ka"])

  kbfn.next = pdfbfn
  pdfbfn.next = cn
  cn.next = pdfan
  pdfan.next = kan
  t = node.hpack(kbfn)
  return t
end

The values for each of the operations are stored in chartbl:

chartbl = {
  [string.byte("y")] = {
    ["kbf"] = "-0.1em",
    ["pdfbf"] = "-1 0 0 1 5.5 0 cm",
    ["xoff"] = "0ex",
    ["yoff"] = "0.5ex",
    ["pdfa"] = "-1 0 0 1 -5.5 0 cm",
    ["ka"] = "0.2em"
  }
}

Example:

\directlua{dofile("transform.lua")}
\hsize1cm
\noindent polyethylene

---

For documentation: In principle, this approach is a general solution as, as far as I understood, everything TeX might want to do with a character can also be done in Lua, with potential changes to the transform_char function. However, for more complex tasks, this seems to be more difficult to do than having TeX typeset it.

So what I initially tried to do was having the post_linebreak_filter call a TeX macro on every character which puts the result of the desired transformation in a \box register and then having Lua replace the node by that box.

I think this is impossible. Any code called by tex.print is only executed after the Lua code, and the approaches to a concurrent interaction discussed in this question are, as it seems to me, not applicable to this situation:

1. Putting the code into a coroutine co and having it call

tex.print("\\directlua{coroutine.resume(co)}")
coroutine.yield()

whenever TeX should execute some macro before continuing works for 14 characters, after which I get
! TeX capacity exceeded, sorry [text input levels=15]

2. In the answer to the question linked above, a \loop is used in the TeX code to repeatedly resume the coroutine, which obviously doesn't work if the Lua code is to be used in a callback.

As a last resort, I only see the possibility of making the characters active, saving the paragraph hlist and temporarily replacing the boxes built by the active character with a glyph node containing the original character in pre_linebreak_filter, and changing them back in the post_linebreak_filter using the saved list. But that's a task for another day ...