20

I'm getting started with unit testing of Mathematica code and looking for information on fitting this into an automated testing environment.

For example with Node.js I write some code and tests and then hook it up with a testing pipeline. So when I check in my changes, Jenkins gets the changes, runs the tests and publishes the results (and possibly other things like build artifacts or build documentation, but this is the simplest case).

From the Mathematica documentation I'm having a hard time figuring out how to do two important steps here.

First, everything I see about testing in Mathematica talks about testing and seeing the results in a notebook. For automated testing by a build server the tests would have to be run without a notebook with just a command line interface.

Second, the test report would need to be in some type of format that could be consumed by Jenkins (or some other build server). This could be one of various different formats but I don't see how to export the test results.

Sean Lynch
  • 1,079
  • 5
  • 16
  • I doubt you would get a simple answer. Mathematica does not have the same facilities as say Visual Studio. There is some integration with Eclipse but it is far from good (IMO). Search for Wolfram Workbench. – Hector Jan 17 '18 at 19:39
  • @Hector I don't think it should need any integrations with an IDE or something like that. Most build servers just need a way to run all the tests from the shell and output the report in some format. – Sean Lynch Jan 17 '18 at 19:47
  • In the comparison with Visual Studio, I had their testing tools in mind. You might want to check this answer. As for writing output to files, You can use Export. – Hector Jan 17 '18 at 19:57
  • AFAIK there is nothing ready to use but it should certainly be possible to build something. I would suggest to save the testing-notebooks as .wlt files (or directly write your tests in that format). You could then use a simple Mathematica script like Print[TestReport["path/to/testfile.wlt"]@"AllTestsSucceeded"] and run that from the command line. Once you have that running it shouldn't be too difficult to start that script from Jenkins or comparable software... – Albert Retey Jan 17 '18 at 22:31
  • @AlbertRetey I was looking into this more today and started to think along the lines of your idea. This should be put into an answer. – Sean Lynch Jan 17 '18 at 23:30
  • For local testing at least there is Wolfram Workbench, which is based on Eclipse and supports unit testing. Maybe this is a starting point to get an idea how to write unit tests for Mathematica and then look into a commandline based test runner that can be run by jenkins. – Thies Heidecke Jan 18 '18 at 01:47
  • @ThiesHeidecke I would except I think we can all agree that Eclipse just a terrible IDE. :) – Sean Lynch Jan 18 '18 at 01:49
  • @SeanLynch Yeah, i can't say i'm a fan of Eclipse either ;) At the end of the page there is a link to Building large software systems in Mathematica which mentions MUnit, which seems to be Wolframs implementation of a unit testing framework. This MUnit question gives details. – Thies Heidecke Jan 18 '18 at 01:57
  • Also on an interesting side note, since Version 10.0 there is a testing framework for the frontend, too (documented here), which is also able to export .wlt files. Would be interesting to check if those are just MUnit .mt files in disguise. – Thies Heidecke Jan 18 '18 at 02:03
  • @SeanLynch: As you suggested I made an answer from my comment. If you have any success feel free to fill in more details. If you have any further questions, feel free to ask. I have no time to do such a setup myself but would be very willing to help when you encounter problems... – Albert Retey Jan 18 '18 at 08:44
  • @SeanLynch Here I described how I integrated Mathematica tests with Git. Your question (and answer) about automated testing was really helpful to me. – Pinti Jul 04 '18 at 09:58
  • @SeanLynch - your edit to Albert's answer included a link to a github repository, but I can't find anything there. I removed the link, feel free to revert if the repo is reestablished. – Jason B. Jul 15 '19 at 21:47

1 Answers1

11

I am not aware of something ready to use for fully automated testing of Mathematica code, but the building blocks for that are available in newer Mathematica versions.

The following is just a rough description of the steps necessary for such a setup. As I think it would be valuable to have a more detailed recipe for such a setup I made this a community wiki, so everybody is welcomed to fill in the details...

Here are the steps I think would be necessary:

Create testfiles

You can either convert testing-notebooks to *.wlt files (using the "More" -> "Save as .wlt" menu in the Testing-Notebook docked cells), write .wlt files directly or also write "old-style" .mt files for MUnit (for the latter you would also need to ensure the package is available). Here is a very minimal example of .wlt file content.

BeginTestSection["AutomatedTests"]
VerificationTest[Plus[1, 1],2]
EndTestSection[]

Create a "driver" script

Create a .m file or probably better a .wls script which runs the tests and prints or returns the outcome of the tests. A very simplified first step would be something like: 

Print[TestReport["/path/to/testfile.wlt"]@"AllTestsSucceeded"]

Start driver script from command line

Ensure you can run the above file from the command line. Depending on your preferences and OS you might want to write a shell-script to do that. Using a .wls script and Exit[returncode] you should be able to do everything you need from within the wolfram script, though...

Configure a test automation tool

For the automation I would suggest to install the test automation tool of your choice and configure it to run the above script on the given events (e.g. a checkin or push to a version control system). You might need to adopt the driver script so that the tool can understand the outcome of it.

Jason B.
  • 68,381
  • 3
  • 139
  • 286
Albert Retey
  • 23,585
  • 60
  • 104
  • Where can I find the documentation for BeginTestSection and more generally the overall acceptable structure of .wlt files? Or did you just save a testing notebook and extract this? – Alan Aug 03 '20 at 22:29
  • 1
    Honestly I don't remember but most probably that was just what I got from saving a testing notebook as a wlt file. It could also be some guessing based on the knowledge of the older MUnit package which is still existing in the workbench AFAIK and for which you will find documentation online... – Albert Retey Aug 05 '20 at 11:18
  • Do you know if BeginTestSection is currently a no-op? E.g., it does not change the context or the context path, and the section name does not appear to be part of the test report. – Alan Aug 05 '20 at 13:17
  • No, I have no idea. As it is undocumented it is pure speculation but I could well imagine that there is functionality at Wolfram or even included in the product which lets you run several .wlt file in one go and then uses BeginTestSection for bookkeeping purposes and thus is not a complete no-op. But for the documented part of the framework it looks very much like a no-op.Have you tried to remove the BeginTestSection EndTestSection from a wlt file and see whether it still can be run? – Albert Retey Aug 06 '20 at 16:06