If I want to store a lookup table in LaTeX3, what is a better way to do so than an l3prop?

Question

As described in How can I correctly expand the result of \prop_item:Nn to work with my own \DoSomething function? I am currently working on a topic that requires a lookup function in LaTeX. Basically I have an information which can be accessed through three levels and I want to give the user the option to use each level to access the information. Until now I had three functions, one allowed direct access to level 1. One allowed access through level 2 and looked up the corresponding information on level 1. The latter did the same for level 3.

However now I want multiple functions to access the information and hence I separated data from function, this is where the problem described in the other question came up. But since my prop is now 2200 items long, the lookup in the prop takes about 3 times longer than using a case compare function, even though I am using a const l3prop:

prop_const_from_keyval:Nn \l_tobisbs_lookup_prop
{
    {key1} = {value1},
    {key2} = {value2},
}

So I guess the comment in the documentation, that this is not the proper datatype for this purpose is true, but what would be a good datatype or general solution to store this type of information and use it in multiple functions in LaTeX3?

Answer 1

I don't know if this would be faster, but if the database is really very huge, you could probably organize it in a MySQL- or MariaDB-server.

Then you could use LuaLaTeX with "--shell-escape" and via assert call a commandline-SQL-client for retrieving values as strings from the database. The strings in turn can be written to TeX-level. If the strings contain only things that shall be tokenized as character-tokens or if you don't mind relying on backslash having category code 0, you can probably use tex.sprint for this:

\documentclass{article}
\usepackage{verbatim}
\makeatletter
\let\percentchar=@percentchar
\makeatother
\newcommand\CallExterrnal[1]{%
  \directlua{ 
    function runcommand(cmd) 
    local fout = assert(io.popen(cmd, 'r')) 
    local str = assert(fout:read('*a')) 
    fout:close() 
    return str 
    end 
    tex.sprint (runcommand("#1"))
  }%
}%
\begin{document}
% Instead of the echo-command a commandline-SQL/MariaDB-client
% could be called for retrieving values of a database.
Let's catch the output of \verb|echo \LaTeX| between parentheses:
(\CallExterrnal{echo \string\\string\LaTeX\percentchar})
Let's catch the output of \verb|echo \LaTeX| into a macro definition:
\expandafter\expandafter\expandafter\def
\expandafter\expandafter\expandafter\test
\expandafter\expandafter\expandafter{%
  \CallExterrnal{echo \string\\string\LaTeX\percentchar}%
}
\texttt{(\string\test=\meaning\test)}
\end{document}

You may need to add the commandline-SQL-client to shell_escape_commands = in texmf.cnf.

---

Probably another approach might be having knitr create the .tex-file for you from an .rnw-file or .Rtex-file where R-code and TeX-code is merged and where you have R-code-chunks which serve the purpose of calling the local commandline-SQL-client for retrieving values from the database and placing them into the .tex-file. (knitr works out of the box with overleaf if the file is named .Rtex.)

<<templates, include=FALSE, cache=FALSE, echo=FALSE, results='asis'>>=
knitr::opts_template$set(
  CallExternalApp = list(include=TRUE, cache=FALSE, echo=FALSE, results='asis')
)
@
<<CallExternal, include=TRUE, cache=FALSE, echo=FALSE, results='asis', >>=
CallExternalApp <- function(A) {
    return(cat(system(A, intern = TRUE), sep="", fill=FALSE))
}
@
\documentclass{article}
\begin{document}
Instead of calling \verb|echo| there could probably be a call to a commandline-SQL -client.
\bigskip
In Linux with the \verb|echo|-command you need to escape backslash, i.e., you need to type
two backslashes to get one. Same for R-code-chunks. Thus if you wish R to send an
\verb|echo|-command that produces a backslash, you need to type four backslashes.
\bigskip
Let's catch the output of \verb|echo \LaTeX| between parentheses:
(%
<<, opts.label='CallExternalApp' >>= 
<<CallExternal>>
CallExternalApp("echo \\LaTeX\%")
@
)
\bigskip
Let's catch the output of \verb|echo \LaTeX| into a macro definition:
\def\test{%
<<, opts.label='CallExternalApp' >>= 
<<CallExternal>>
CallExternalApp("echo \\LaTeX\%")
@
}
\texttt{(\string\test=\meaning\test)}
\end{document}