4

I was originally following the idea from Garrick Peschke To import all .tex files inside a folder, but I want my Lua function to also search inside subfolders.

Unfortunately I can't make the Lua function work on subfolders. It is worth noting that TeX doesn't recognize this as an error, any .tex file will be properly processed while folders will not.


function inputAll(dir)
    for content in lfs.dir(dir) do
        if ("." ~= content) and (".." ~= content)  then
            fullpath = dir .."/".. content
                contentType = lfs.attributes(fullpath, "mode")
            if isTexFile(content, contentType) then
                    tex.sprint("\\input{" .. fullpath .. "}")
            elseif ("directory" == contentType) then
                return inputAll(fullpath)
            end
        end
    end
end
function isTexFile(dirOrFolder, contentType)
    return ("file" == contentType) and (string.find(dirOrFolder, ".+%.tex"))
end

Following the idea proposed by Skeen to use an inner function, the result is the same: it works on files but not on folders.


function inputAll(dir)
    local function inner(m)
        for content in lfs.dir(m) do
            if ("." ~= content) and (".." ~= content)  then
                fullpath = m .."/".. content
                    contentType = lfs.attributes(fullpath, "mode")
                if isTexFile(content, contentType) then
                        tex.sprint("\\input{" .. fullpath .. "}")
                elseif ("directory" == contentType) then
                    return inner(fullpath);
                end
            end
        end
    end
    inner(dir)
end
function isTexFile(dirOrFolder, contentType)
    return ("file" == contentType) and (string.find(dirOrFolder, ".+%.tex"))
end

following are some system details:

Mac Os X: 10.13.6
TeX 3.141592653 (TeX Live 2022)
LuaTeX, Version 1.15.0 (TeX Live 2022)
|_Compiled with lua version 5.3.6
TeXShop: 4.44 (4.44)

Conclusion

As David Carlisle, pointed out, LuaTeX from TexLive 2022 is actually able to recurse when the latest instruction does not use return.

With TexLive 2018 it was not working with or without return. Unfortunately I haven't saved any evidence so you'll have to take my words on it.
It is still unclear to me if provided a suitable proper tail recursive algoritm LuaTex will work or not:
AFAIK Lua needs the return statement to execute tail recursion properly. LuaTex I don't really know.


Final words:

I've been messing around with lua recursion in some other tex works of mine, and it seems that using a suitable algorithm will and using only pure lua (not sure whether it's needed or not) will allow the use of return statement no problem.

By pure lua I mean not messing with tex table. E.g.: tex.sprint( ... )

I did not had enough time to understand all the lfs library code to understand if there is another way to get the content as list (without me preparing the data first, essentially creating it myself).

The following code should be enough to prove what I meant

function excludeDependencyRecursive(csvListOfGroupIdColonArtifactId, result, textDecorator)  
  if (csvListOfGroupIdColonArtifactId) then
    local beginIndex, endIndex = string.find(csvListOfGroupIdColonArtifactId,",");
texio.write_nl( "term and log", "beginIndex : " .. (beginIndex or "nil") .. " endIndex : " .. (endIndex or "nil"))

local first = string.sub(csvListOfGroupIdColonArtifactId, 1, getEndIndex(endIndex, csvListOfGroupIdColonArtifactId))
--[[ first is always a valid string ]]
local groupId, artifactId = getGroupAndArtifact(first)

result = result .. textDecorator("<dependency>") .. "\\+ \\\\"
result = result .. textDecorator("<groupId>" .. groupId .. "</groupId>") .. "\\\\"
result = result .. textDecorator("<artifactId>" .. artifactId .. "</artifactId>") .. "\\\\"
result = result .. textDecorator("\\dots") .. "\\- \\\\"
result = result .. textDecorator("</dependency>") .. "\\\\"

--[[ lua's shortcircut for IF endIndex THEN string.sub ELSE nil ]]
local rest  = endIndex and string.sub(csvListOfGroupIdColonArtifactId, 1+endIndex, #csvListOfGroupIdColonArtifactId)
texio.write_nl( "term and log", "first : " .. first .. " rest : " .. (rest or "nil"))

return excludeDependencyRecursive(rest, result, textDecorator)

end return result end


--[[ can throw an error if the groupAndArtifact is not a string of the form A:B ]]
function getGroupAndArtifact(groupAndArtifact)
  --[[ both are local ]]
  local beginIndex, endIndex = string.find(groupAndArtifact,":");

local first = string.sub(groupAndArtifact, 1, endIndex-1) local rest = string.sub(groupAndArtifact, 1+endIndex, #groupAndArtifact) return first, rest end


function getEndIndex(index, string)
  --[[ lua version of C if then else ( A ? B : C) ]]
    return index and (index - 1) or #string
end

What you have seen is a not so elegant decorator which you can call with

excludeDependencyRecursive(csvListOfGroupIdColonArtifactId, "", orangeBackgroudDecorator) 

where

greenBackgroudDecorator = function (x) return "\\colorbox[HTML]{".. colorGreen .."}{" .. x .. "}" end

For those wondering about \\, \+, \-, excludeDependencyRecursive's caller was using latex's tabbing environment


Thank you all very much for the help, I mean it.

Paul
  • 43
  • POSIX only requires a minimum of 19 file descriptors. After you subtract stdin, stdout, and stderr, you're only left with 16 files. macOS almost certainly supports more, but LuaTeX may not have been compiled with a higher limit. You could perhaps request that they set a higher limit on the LuaTeX mailing list, but it would probably be easiest to remove the recursion. – Max Chernoff Aug 10 '22 at 00:31
  • 1
    Does the problem persist after updating your TeX distribution from MacTeX2018 to MacTeX2022? For that matter, any chance of updating your MacOS distribution to something more recent than 10.13 "High Sierra"? – Mico Aug 10 '22 at 00:44
  • @Mico : I can try updating MacTeX, but I won't change the OS for this. It's clearly something not related to architecture. – Paul Aug 10 '22 at 07:57
  • @MaxChernoff : I noticed I wasn't using inner function variable so with the second attempt now the results are the same without the inner function : it works on files but not on folders I'll update the question accordingly – Paul Aug 10 '22 at 08:15
  • @Mico I've updated to Tex to 2022, nothing new. I'm going to update the spec info on the question – Paul Aug 11 '22 at 14:20
  • Why are you only processing folders with dots in their name? – Taco Hoekwater Aug 12 '22 at 07:15
  • 1
    @Paul you probably should not be 'return'-ing after the inner inputAll() (sorry about the earlier comment, that was wrong) – Taco Hoekwater Aug 12 '22 at 07:28
  • @TacoHoekwater hi Taco, yes my answer overlapped your comment:-) – David Carlisle Aug 12 '22 at 07:36

1 Answers1

4

If I start from a directory

$ ls -lR
.:
total 15
-rwxrw-r--+ 1 davidc davidc  652 Aug 12 08:30 aa.log
-rwxrw-r--+ 1 davidc davidc  596 Aug 12 08:30 aa.lua
-rwxrw-r--+ 1 davidc davidc 9639 Aug 12 08:30 aa.pdf
-rwxrw-r--+ 1 davidc davidc   52 Aug 12 08:22 aa.tex
drwxrwxr-x+ 1 davidc davidc    0 Aug 12 08:26 dirtop

./dirtop: total 2 -rwxrw-r--+ 1 davidc davidc 3 Aug 12 08:17 a.tex -rwxrw-r--+ 1 davidc davidc 3 Aug 12 08:17 b.tex drwxrwxr-x+ 1 davidc davidc 0 Aug 12 08:17 dir1 drwxrwxr-x+ 1 davidc davidc 0 Aug 12 08:25 dir2

./dirtop/dir1: total 2 -rwxrw-r--+ 1 davidc davidc 3 Aug 12 08:17 1a.tex -rwxrw-r--+ 1 davidc davidc 3 Aug 12 08:17 1b.tex

./dirtop/dir2: total 2 -rwxrw-r--+ 1 davidc davidc 3 Aug 12 08:25 2a.tex -rwxrw-r--+ 1 davidc davidc 3 Aug 12 08:25 2b.tex

with aa.tex

\directlua{
require("aa")
inputAll("dirtop")
}

\bye

and aa.lua

function inputAll(dir)
    for content in lfs.dir(dir) do
         if ("." ~= content) and (".." ~= content)  then
            fullpath = dir .."/".. content
                contentType = lfs.attributes(fullpath, "mode")
            if isTexFile(content, contentType) then
                    tex.sprint("\\input{" .. fullpath .. "}")
            elseif ("directory" == contentType) then
                return inputAll(fullpath)
            end
        end
    end
end
function isTexFile(dirOrFolder, contentType)
    return ("file" == contentType) and (string.find(dirOrFolder, ".+%.tex"))
end

then I get

$ luatex aa
This is LuaTeX, Version 1.15.1 (TeX Live 2023/dev) 
 restricted system commands enabled.
(./aa.tex (./dirtop/a.tex) (./dirtop/b.tex) (./dirtop/dir1/1a.tex)
(./dirtop/dir1/1b.tex) [1{/usr/local/texlive/2022/texmf-var/fonts/map/pdftex/up
dmap/pdftex.map}])</usr/local/texlive/2022/texmf-dist/fonts/type1/public/amsfon
ts/cm/cmr10.pfb>
Output written on aa.pdf (1 page, 9639 bytes).
Transcript written on aa.log.

showing luatex stopped at the first directory. The reason is the return so deleting that:

function inputAll(dir)
    for content in lfs.dir(dir) do
         if ("." ~= content) and (".." ~= content)  then
            fullpath = dir .."/".. content
                contentType = lfs.attributes(fullpath, "mode")
            if isTexFile(content, contentType) then
                    tex.sprint("\\input{" .. fullpath .. "}")
            elseif ("directory" == contentType) then
                inputAll(fullpath)
            end
        end
    end
end
function isTexFile(dirOrFolder, contentType)
    return ("file" == contentType) and (string.find(dirOrFolder, ".+%.tex"))
end

gives

$ luatex aa
This is LuaTeX, Version 1.15.1 (TeX Live 2023/dev) 
 restricted system commands enabled.
(./aa.tex (./dirtop/a.tex) (./dirtop/b.tex) (./dirtop/dir1/1a.tex)
(./dirtop/dir1/1b.tex) (./dirtop/dir2/2a.tex) (./dirtop/dir2/2b.tex) [1{/usr/lo
cal/texlive/2022/texmf-var/fonts/map/pdftex/updmap/pdftex.map}])</usr/local/tex
live/2022/texmf-dist/fonts/type1/public/amsfonts/cm/cmr10.pfb>
Output written on aa.pdf (1 page, 9640 bytes).
Transcript written on aa.log.

showing a full recursive pass over the directory tree

David Carlisle
  • 757,742
  • Ok nice, Effectively I tried both with and WITHOUT the return statement with TeX 2018, but forgot to try without return statement on TeX 2022.

    Without a return statement it properly works. But this raises another concerns of Mine:

    AFAIK Lua is only able to perform Tail Recursion when using the "return" I (https://stackoverflow.com/questions/13303347/tail-call-optimization-in-lua )

    Am I losing this language capability ? If so is there a way to prevent it ?

    – Paul Aug 12 '22 at 11:39
  • Clearly tail recursion elimination can only happen on tail recursve functions and if you recurse down a tree rather than a list that is not tail recursive. @Paul – David Carlisle Aug 12 '22 at 12:55
  • Clearly this algoritm is not suited for tail recursion, but the removing of the return concerns me since it will prevent me from using the feature. I guess I'll have to try it out to further investigate. – Paul Aug 12 '22 at 19:07
  • 1
    @Paul you could only use tail recursion elemination if there was a single recursive call and in that case return would be no issue. It's a problem here as after the first return the function stops so you do not see the second subdirectory, so if you restructure the iteration to only have one call, you could put return back. – David Carlisle Aug 12 '22 at 20:11