How to prevent tests from evaluating with TestCreate?

Question

When I am creating a test for example with TestCreate, I have input and expected output. How can I prevent the expected input from evaluating the output? This was one of my favorite areas of new functionality expanded in 13.3. For example, if I have a function ResistanceMatrix, how can I test that it produces the same output as GraphData's "ResistanceMatrix" for the "PetersenGraph"?

Answer 1

I discovered how to do this after reading Stephen Wolfram's release announcement for 13.3 section Symbolic Testing Framework. The solution is to use the scoping construct With in Table, which effectively uses the scoping construct Block to do something similar to Map with pure functions over a list. There might also be a way to make it work with Unevaluated.

tests = Table[
  With[{i = i, j = i + 1}, 
   TestCreate[i + 1, j, TestID -> Automatic]], {i, 4}]

Then run

TestReport[tests]

and I get

TestReportObject[<|"Title" -> Automatic, "Aborted" -> False, 
  "TestResults" -> <|3890776590901293718 -> TestObject[\
<|"MetaInformation" -> None, "AbsoluteTime" -> 3.897225378977161*^9, 
       "SameTest" -> SameQ, "SameMessages" -> Testing`MessageMatchQ, 
       "MemoryConstraint" -> DirectedInfinity[1], 
       "TimeConstraint" -> DirectedInfinity[1], 
       "CreationID" -> "7326579c-2d13-4115-9260-e18d1f282685", 
       "TestID" -> "65cf6dc6-313b-4a30-aea7-56478682fef1", 
       "TestFileName" -> "", 
       "EvaluationID" -> "e5b3960e-021c-411a-bd96-0be9d44821fa", 
       "Input" -> HoldForm[1 + 1], "ExpectedOutput" -> HoldForm[2], 
       "ExpectedMessages" -> HoldForm[{}], 
       "ActualOutput" -> HoldForm[2], "ActualMessages" -> {}, 
       "AbsoluteTimeUsed" -> 0``7.150514997831988, 
       "CPUTimeUsed" -> 0., "MemoryUsed" -> 2496, 
       "Outcome" -> "Success"|>], 
    1792103122768780638 -> TestObject[<|"MetaInformation" -> None, 
       "AbsoluteTime" -> 3.897225378977161*^9, "SameTest" -> SameQ, 
       "SameMessages" -> Testing`MessageMatchQ, 
       "MemoryConstraint" -> DirectedInfinity[1], 
       "TimeConstraint" -> DirectedInfinity[1], 
       "CreationID" -> "a22068e2-9c71-4a71-9008-f0d99a773acd", 
       "TestID" -> "579065fe-1adc-4b27-a0b8-9593b8799da1", 
       "TestFileName" -> "", 
       "EvaluationID" -> "25f33fd3-48d3-4c91-9411-055c8801a08d", 
       "Input" -> HoldForm[2 + 1], "ExpectedOutput" -> HoldForm[3], 
       "ExpectedMessages" -> HoldForm[{}], 
       "ActualOutput" -> HoldForm[3], "ActualMessages" -> {}, 
       "AbsoluteTimeUsed" -> 0``7.150514997831988, 
       "CPUTimeUsed" -> 0., "MemoryUsed" -> 2128, 
       "Outcome" -> "Success"|>], 
    4195592839575175517 -> TestObject[<|"MetaInformation" -> None, 
       "AbsoluteTime" -> 3.897225378977161*^9, "SameTest" -> SameQ, 
       "SameMessages" -> Testing`MessageMatchQ, 
       "MemoryConstraint" -> DirectedInfinity[1], 
       "TimeConstraint" -> DirectedInfinity[1], 
       "CreationID" -> "32d336ce-ff2f-4b5d-94c6-04e8caa6ec0d", 
       "TestID" -> "4acc8da8-24c3-4ac8-a06b-1f62d806f4ad", 
       "TestFileName" -> "", 
       "EvaluationID" -> "356aa29f-3c40-4827-badb-0e81421ca2db", 
       "Input" -> HoldForm[3 + 1], "ExpectedOutput" -> HoldForm[4], 
       "ExpectedMessages" -> HoldForm[{}], 
       "ActualOutput" -> HoldForm[4], "ActualMessages" -> {}, 
       "AbsoluteTimeUsed" -> 0``7.150514997831988, 
       "CPUTimeUsed" -> 0., "MemoryUsed" -> 2128, 
       "Outcome" -> "Success"|>], 
    2729069003583258653 -> TestObject[<|"MetaInformation" -> None, 
       "AbsoluteTime" -> 3.8972253789781694`*^9, "SameTest" -> SameQ, 
       "SameMessages" -> Testing`MessageMatchQ, 
       "MemoryConstraint" -> DirectedInfinity[1], 
       "TimeConstraint" -> DirectedInfinity[1], 
       "CreationID" -> "4b329be3-7805-4728-a822-93404350b0ff", 
       "TestID" -> "d1b274ad-cc2c-453b-a15f-a15ffb5008d7", 
       "TestFileName" -> "", 
       "EvaluationID" -> "c23dc937-e49e-41a5-b926-412c94c64174", 
       "Input" -> HoldForm[4 + 1], "ExpectedOutput" -> HoldForm[5], 
       "ExpectedMessages" -> HoldForm[{}], 
       "ActualOutput" -> HoldForm[5], "ActualMessages" -> {}, 
       "AbsoluteTimeUsed" -> 0``7.150514997831988, 
       "CPUTimeUsed" -> 0., "MemoryUsed" -> 2128, 
       "Outcome" -> "Success"|>]|>, "FailureResults" -> <||>, 
  "TestsNotEvaluatedKeys" -> {}, "TestsFailedWrongResultsKeys" -> {}, 
  "TestsFailedWithMessagesKeys" -> {}, "TestsFailedWithErrorsKeys" -> {},
   "TestsSucceededKeys" -> {3890776590901293718, 1792103122768780638, 
   4195592839575175517, 2729069003583258653}|>]

To add echos showing the individual tests, do

TestReport[tests, HandlerFunctions -> <|"TestEvaluated" -> Echo|>]

Here's another example where I want to test that my function ResistanceMatrix produces the same output as GraphData's "ResistanceMatrix" property for the Petersen graph implemented as "PetersenGraph" with GraphData.

tests = Table[
  With[{input = input, 
    output = GraphData[input, "ResistanceMatrix"]},
   TestCreate[ResistanceMatrix[GraphData[input]], output, 
    TestID -> Automatic]], {input, {"PetersenGraph"}}]

Then I run the tests.

TestReport[tests]

I can also echo.

TestReport[tests, HandlerFunctions -> <|"TestEvaluated" -> Echo|>]