Why doesn't luatex recognise lyaml, a manually installed library (when lua itself does)?

Question

I want to parse a yaml file containing my CV data and then write content programatically, in LaTeX, from that data. I have installed the relevant library lyaml and it works, but lua(la?)tex doesn't detect it. Why not, and how can I change that?

Further details

In case these help anyone to diagnose my problem...

The following program works:

-- requirements
local io    = require "io"
local lyaml = require "lyaml"
-- get yaml data as a string
local yaml_file   = "cv.yaml"
local f           = assert(io.open(yaml_file, 'r'))
local yaml_string = f:read('a')
f:close()
local cv_table = {}
-- convert yaml string to lua table
cv_table.t = lyaml.load(yaml_string)
for key, value in pairs(cv_table.t) do
    print(key)

end

It opens the file, reads its contents as a string, closes the file, parses the string as yaml, and then prints each top-level key of the resulting lua table to stdout.

I'm on Debian. In particular, this works because I've installe lua-yaml. dpkg --listfiles lua-yaml gives me this:

/.
/usr
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/liblua5.1-yaml.so.0.0.0
/usr/lib/x86_64-linux-gnu/liblua5.2-yaml.so.0.0.0
/usr/lib/x86_64-linux-gnu/liblua5.3-yaml.so.0.0.0
/usr/lib/x86_64-linux-gnu/lua
/usr/lib/x86_64-linux-gnu/lua/5.1
/usr/lib/x86_64-linux-gnu/lua/5.2
/usr/lib/x86_64-linux-gnu/lua/5.3
/usr/share
/usr/share/doc
/usr/share/doc/lua-yaml
/usr/share/doc/lua-yaml/changelog.Debian.gz
/usr/share/doc/lua-yaml/changelog.gz
/usr/share/doc/lua-yaml/copyright
/usr/share/lintian
/usr/share/lintian/overrides
/usr/share/lintian/overrides/lua-yaml
/usr/share/lua
/usr/share/lua/5.1
/usr/share/lua/5.1/lyaml
/usr/share/lua/5.1/lyaml/explicit.lua
/usr/share/lua/5.1/lyaml/functional.lua
/usr/share/lua/5.1/lyaml/implicit.lua
/usr/share/lua/5.1/lyaml.lua
/usr/share/lua/5.2
/usr/share/lua/5.2/lyaml
/usr/share/lua/5.3
/usr/share/lua/5.3/lyaml
/usr/lib/x86_64-linux-gnu/liblua5.1-yaml.so.0
/usr/lib/x86_64-linux-gnu/liblua5.2-yaml.so.0
/usr/lib/x86_64-linux-gnu/liblua5.3-yaml.so.0
/usr/lib/x86_64-linux-gnu/lua/5.1/yaml.so
/usr/lib/x86_64-linux-gnu/lua/5.2/yaml.so
/usr/lib/x86_64-linux-gnu/lua/5.3/yaml.so
/usr/share/lua/5.2/lyaml/explicit.lua
/usr/share/lua/5.2/lyaml/functional.lua
/usr/share/lua/5.2/lyaml/implicit.lua
/usr/share/lua/5.2/lyaml.lua
/usr/share/lua/5.3/lyaml/explicit.lua
/usr/share/lua/5.3/lyaml/functional.lua
/usr/share/lua/5.3/lyaml/implicit.lua
/usr/share/lua/5.3/lyaml.lua``shell

However, this does not even compile:

\documentclass{article}
\usepackage{luacode}
\begin{document}
\begin{luacode*}
-- requirements
local io    = require "io"
local lyaml = require "lyaml"

-- get yaml data as a string
local yaml_file   = "cv.yaml"
local f           = assert(io.open(yaml_file, 'r'))
local yaml_string = f:read('a')
f:close()

local cv_table = {}

-- convert yaml string to lua table
cv_table.t = lyaml.load(yaml_string)

for key, value in pairs(cv_table.t) do
    tex.print(key)  
end


\end{luacode*}
\end{document}

The tex file is just the lua program, wrapped in the appropriate latex, and with tex.print replacing print. It is in the same directory.

Trying to compile with lualatex test.tex gives the following errors:

(/usr/local/texlive/2020/texmf-dist/tex/latex/base/ts1cmr.fd)...0/texmf-dist/te
x/luatex/lualibs/lualibs-basic-merged.lua:381: attempt to call a nil value (fie
ld 'cpath specification')
stack traceback:
        ...0/texmf-dist/tex/luatex/lualibs/lualibs-basic-merged.lua:381: in field '?'
        ...0/texmf-dist/tex/luatex/lualibs/lualibs-basic-merged.lua:403: in function <
...0/texmf-dist/tex/luatex/lualibs/lualibs-basic-merged.lua:395>
        [C]: in function 'require'
        [\directlua]:3: in main chunk.
\luacode@dbg@exec ...code@maybe@printdbg {#1} #1 }
l.28 \end{luacode}

The same thing happens if I use \directlua or the(unstarred luacode environment. Same errors, so that's not the problem. The errors concern the lyaml library, and go away as soon as I remove it from the code (to be replaced by errors complaining that I'm referencing a library I don't import). As I understand it, these all use \directlua under the hood anyway, so they will all suffer the same problems.

So the problem is with the lyaml library. Clearly it's installed just fine, so why can't lua(la?)tex see it?

Things I have tried

I have read this question. I'm on debian so the comments weren't directly relevant to me, but I was inspired to try symlinking /usr/share/lua/5.3/lyaml.lua to /usr/local/texlive/2020/texmf-dist/tex/luatex/lualibs/lualibs-lyaml.lua. This didn't make any difference. I then tried what the comment suggested and ran sudo fmtutil-sys --byfmt=lualatex but that also had no effect.

Following this question I have also tried symlinking the relevant files straight into the working dir, but this didn't help either.

Answer 1

I finally managed to get this to work. There were three components I needed:

1. \usepackage{luacode}. I already had this.
2. \usepackage{luapackageloader} (thanks @likethevegetable for the suggestion) From CTAN: "This package allows LuaTeX to load packages from the default package.path and package.cpath locations. This could be useful to load external Lua modules, including modules installed via LuaRocks".
3. Specify the paths in lua. For me, on debian, this was right, but of course others might need different paths:

\begin{luacode*}
    local my_path = "/usr/share/lua/5.3"
    local my_cpath ="/usr/lib/x86_64-linux-gnu/lua/5.3/"
    package.path  = package.path  .. string.format(";%s/?.lua;%s/?/init.lua", my_path, my_path)
    package.cpath = package.cpath .. string.format(";%s/?.so;%s/?/init.so", my_cpath, my_cpath)
\end{luacode*}