Testing Framework API for LaTeX

Question

As part of an online exam application, we need to automate the evaluation of LaTeX scripts written by the students. Essentially, the script should be evaluated against a set of test cases and if it passes all of them should return passed (true) or should throw the error message back so that the student can retry the problem.

We came across qstest, but I wish to know are there any better solutions out there which can be employed in the web app easily and efficiently. The web app has been developed using Django.

Answer 1

Testing in many languages is done using assertions and test cases as you mentioned in your question. An assertion in LaTeX can be of the general form:

      \def\test#1{\def\res{#1}\ifx\foo\res\else\ERROR\fi}

or just return "Passed" or "Failed". I lean for the latter for long test cases, and if you colour code the results is easy to pick-up the errors.

Code can be simple for true false based on meaning

\newcommand\assert[2][]{%
   \ttfamily
   \def\result{#2}
   \ifx\foo\result\textcolor{green}{Passed}%
     \else \textcolor{red}{Failed}\fi
   \space Test:\,\stepcounter{tst}\thetst
   \detokenize{#1}
\par}

or checking if things have been defined (sometimes tricky).

% Checks if defined can be unreliable
\newcommand\assertdef[2][]{
 \ttfamily
  \ifdefined#2\textcolor{green}{Passed}%
     \else \textcolor{red}{Failed}\fi
   \space Test:\,\stepcounter{tst}\thetst
   \detokenize{#1}
\par
}

The MWE example below tests some fixed point arithmetic and some Lisp relics from the LaTeX kernel and produced the output for the image above:

\documentclass{article}
\usepackage{xcolor,fp}
\parindent0pt
\makeatletter
\newcounter{tst}
\setcounter{tst}{9}
% asserts true on meaning
\newcommand\assert[2][]{%
   \ttfamily
   \def\result{#2}
   \ifx\test\result\textcolor{green}{Passed}%
     \else \textcolor{red}{Failed}\fi
   \space Test:\,\stepcounter{tst}\thetst
   \detokenize{#1}
   \par}
% Checks if defined can be unreliable
\newcommand\assertdef[2][]{
 \ttfamily
  \ifdefined#2\textcolor{green}{Passed}%
     \else \textcolor{red}{Failed}\fi
   \space Test:\,\stepcounter{tst}\thetst
   \detokenize{#1}
\par
}
\begin{document}
\section{\jobname\\ \today}
\edef\test{\@car 123\@nil}\assert[\@car 123\@nil==1]{1}
\edef\test{\@car{1}23\@nil} \assert[\@car {1}23\@nil==1]{1}
\edef\test{\@car {123}{456}{7}\@nil} \assert{123}
\edef\test{\@carcube1234567\@nil}\assert{123}
\edef\test{\@cdr 123\@nil} \assert{23}
\edef\test{\@cdr {134}{x}\@nil}\assert{x}
\edef\test{\@cdr {134}{{x}}\@nil}\assert{{x}}
\let\test\@nnil\assert{\@nil}
\toks@={abc\test}\addto@hook\toks@{x\bar}
\expandafter\def\expandafter\test\expandafter{\the\toks@} \assert{abc\test x\bar}
\g@addto@macro\test{y\gee} \assert{abc\test x\bar y\gee}
\def\xx{456}
\def\test{123}\@cons\test{\xx78}\assert{123\@elt45678}
\@cons\test{\xx780}\assert{123\@elt45678}
% assert if defined
\assertdef\assert
\assertdef[\@@par is defined]\@@par
% asserts for fp values
\FPadd\test{1}{1}\assert[\FPadd\test{1}{1}==2.000000000000000000]{2.000000000000000000}
\FPadd\test{1}{1}\assert[\FPadd\test{1}{1}==2.00000000000000000]{2.000000000000000000}
\end{document}

One can get more creative and create additional assertions such as \assertfalse, \assertcat for category code checks etc. The LaTeX3 Team I understand does a lot of testing; however applying the above to a learning environment I am not too sure how feasible or desirable it is to do so.

In a tutorial you will be testing mostly mastering usage of commands. For an online application I would have gone for a Moodle type of installation with build-in structures for exams and quizzes, but then you went with Django:)

Answer 2

Here's an assert routine that I used to help write an update of the LaTeX lawtex package.

\documentclass{article}
\usepackage{xstring} % IfEq, StrCompare
\usepackage{xcolor}

\newcounter{tst}

% \assert{test name}{string value1}{string value 2}- will test that
% the second argument matches the third argument and print in green if
% it passed or in red if it fails.
\long\def\assert#1#2#3{{% add a block to prevent namespace pollution
    % Note that you cannot use the macro on macros with optional
    % arguments because the macros won't expand properly in an edef.
    % see https://tex.stackexchange.com/questions/21514/why-isnt-a-command-defined-by-newcommand-with-an-optional-argument-expandable
    \edef\assertA{#2}\edef\assertB{#3}%
    \def\name{\if\relax\detokenize{#1}\relax\detokenize{#2} = \detokenize{#3}\else #1\fi}%
    \def\desc{\space Test:\,\stepcounter{tst}\thetst~(\name)\par}%
    \comparestrict%
    \IfEq{\assertA}{\assertB}{%
      \textcolor{green}{Passed} \desc%
    }{%
      \ifcsname EnforceAssertions\endcsname%
      \errmessage{Assertion \thetst (#1) failed. (#2) $\ne$ (#3)}%
      \else%
      \fi%
      \textcolor{red}{Failed} \desc%
      \par~~~~~~~~(\assertA)(\detokenize{#2}) $\ne$ \par~~~~~~~~(\assertB)(\detokenize{#3})\par%
      First difference at position %
      \comparestrict\StrCompare{\assertA}{\assertB}.\par%
    }%
}}

\begin{document}

\def\hello#1{hello #1}

\assert{hello world}{\hello{world}}{hello world}

\assert{failing hello world}{\hello{world}}{hello cruel world}

\end{document}

If you call it with the macro \EnforceAssertions defined, then assertion errors will print an error message to the log file. From the commandline, I can test using a bash script like this:

#!/bin/bash
function dotest {
    echo testing $1
    pdflatex -halt-on-error "\def\EnforceAssertions{defined} \input{$1}" &> /dev/null && echo Test $1 passed || echo Test $1 FAILED.
}
export -f dotest
echo test.tex | parallel --timeout 1000% dotest 

Here's how it's called:

bash-4.4$ ./temptestall.sh
testing test.tex
Test test.tex FAILED.

This may not be the best way to test using the web, but I put this answer here for other people who may be looking for a testing framework to help when programming TeX/LaTeX.