Using the package memoize with a specified --output-directory

Question

I'm trying to configure the package memoize, but am running into issues when trying to pass an --output-directory option to LuaLatex. From looking at the output files and how they end up, it looks as though memoize is mistakenly writing to the output directory, but not reading from it. With the MWE below:

\documentclass{article}
\usepackage[memo dir]{memoize}
\usepackage[linguistics]{forest}
\usepackage{fontspec}
\setmainfont{Charis SIL}
\begin{document}
A test document.
\begin{forest}
[S
    [NP]
    [VP]
]
\end{forest}
\end{document}

The package works correctly with the simple command lualatex untitled.tex, however fails with lualatex --output-directory=output untitled.tex with the generic error.

I've also tried adding \mmzset{prefix=output/} to no avail.

Answer 1

Good news: Memoize actually supports --output-directory, or more precisely, it will support it once TeXLive 2024 is released. Let me explain.

Memoize's extraction scripts respect the value of environment variable TEXMF_OUTPUT_DIRECTORY which will be introduced by TeXLive 2024 (so, very soon!). (The new variable was in fact conceived as a solution to Memoize's problem with --output-directory, and I'm very grateful to Karl Berry for figuring out the solution and implementing it!) In TeXLive 2024, (a) setting TEXMF_OUTPUT_DIRECTORY will be an alternative to specifying --output-directory, and (b) however the output directory is set, its value will be reflected by the environment variable TEXMF_OUTPUT_DIRECTORY received by the embedded jobs (like Memoize extraction script). Bottomline: in 11 days, things will work out of the box.

Meanwhile, the workaround is to set TEXMF_OUTPUT_DIRECTORY manually:

TEXMF_OUTPUT_DIRECTORY=output lualatex --output-directory=output untitled.tex

Now the bad news: while this was tested and works with the pdftex engine, it appears it does not work with luatex. Or rather, things work fine insofar as the externs are correctly produced and extracted, but utilizing the externs fails, as lualatex does not search for them in the output directory (unlike pdflatex). I will investigate asap.

A workaround could be to soft-link the memo directory from the output directory into the current directory, but I realize that the OP's setup might not allow for this. From the directory containing untitled.tex run:

ln -s output/untitled.memo.dir

Answer 2

I would add to Sašo Živanović's answer that it is possible to configure latexmk to use memoize with an output directory.

Here's what goes in a latexmkrc file. (NOTE that an improved version of this configuration will be in the next version of latexmk (v. 4.84). The solution here works only with TeXLive on Unix-type OSs (linux and macOS), and only when latexmk uses lualatex for pdflatex for the compilation of the .tex file.)

$pdf_mode = 4;  # Use lualatex.  Set to 1 for pdflatex, 5 for xelatex.
You can have separate build and output directories, or they can be
the same, e.g., by $out_dir = $aux_dir = 'output';
$out_dir = 'output';
$aux_dir = 'build';
$emulate_aux = 1;
Get latexmk -C to delete memoize-generated files, at least
when the default setting in memoize for the prefix of .memo
and .pdf files is used.
push @generated_exts, 'mmz', 'mmz.log', '%R..memo', '%R..pdf';
&set_tex_cmds( 'internal latex_memoize %C %B %S -synctex=1 %O' );
sub latex_memoize {
  print "============= I am latex_memoize \n";
Ensure TEX_OUTPUT_DIRECTORY is set. (In v. >=4.84 of latexmk, this
will not be necessary, since latexmk will make the setting itself.)
local %ENV = %ENV;
  foreach ( 'TEXMF_OUTPUT_DIRECTORY' ) {
    $ENV{$} = $aux_dir;
    print "latex_memoize: ENV{$} = '$ENV{$_}'\n";
  }
my ($cmd, $base, $source, @args) = @_;
  my $tex_log = "$aux_dir1$base.log";
  my $mmz_file = "$aux_dir1$base.mmz";
  my @tex_cmd = ($cmd, @args, $source );
  my @memo_cmd = ("memoize-extract.pl", '-F', 'latex', $mmz_file );
print "latex_memoize: Running\n  '@tex_cmd'\n";
  my $ret = system @tex_cmd;
if (! -e $mmz_file) {
    print "latex_memoize: No mmz file '$mmz_file', so memoize is not being used.\n";
    return $ret;
  }
Use (not-currently-documented) latexmk subroutine to determine
whether mmz file was generated in current run:
if ( ! test_gen_file_time( $mmz_file) ) {
    warn "latex_memoize: Mmz file '$mmz_file' exists, but wasn't generated\n",
         "  in this run so memoize is not currently being used.\n";
    return $ret;
  }
Fix up dependency information in log file:
my $mmz_fh = undef;
  my $tex_log_fh = undef;
  if (! open( $mmz_fh, '<', $mmz_file ) ) {
    print "latex_memoize: Cannot open mmz file '$mmz_file':\n $!\n";
    return $ret;
  }
  if (! open( $tex_log_fh, '>>', $tex_log ) ) {
    warn "latex_memoize: Cannot open log file '$tex_log': $!\n";
    close $mmz_fh;
    return $ret;
  }
  while ( <$mmz_fh> ) {
    s/\s$//;
    my $source = undef;
    if ( /^\mmzNewExtern\s+{([^}]+)}/ ) {
      # We have a new memo item without a corresponding pdf file
      # Put a suitable message in the log file, so that latexmk will
      # know to run latex again, when it analyzes the results of 
      # the current run.
      $file = "$aux_dir1$1";
      print "latex_memoize: new extern for memoize: '$file'\n";
      print $tex_log_fh "No file $file\n";
    }
  }
  close $tex_log_fh;
  close $mmz_fh;
Run memoize-extract.pl now.  Then missing pdf files are
created, which will provoke latexmk into rerunning *latex.
It also handles the case that the memoize package is used
with the external=no option.  (But gives no problem when
the option is not used.)
It also allows the use of separate aux and output directories
which would otherwise give a problem, since in that situation
latexmk will move the pdf file from the aux directory to the
output directory, where it is no longer accessible to
memoize-extract.pl.
print "Running\n @memo_cmd\n";
  my $ret2 = system @memo_cmd;
  if ($ret2) { return $ret2; }
return $ret;
}

This has been tested on TeXLive 2023 and 2024, and with all of pdflatex, lualatex, and xelatex. (On Windows, there may be some issues when files have non-ASCII characters in their names, but I haven't yet checked.)

As mentioned in comments to the question, there are some complications in working with an output directory. Ways to get things working are already programmed into latexmk; that was done many years ago and was quite simple, but not something you'd want to do regularly from the command line. I've found that the use of latexmk insulates me against the difficulties that were mentioned.