Test runs
A Test run asks a draft or a published version every case in the Test sets you choose, and keeps every answer it gave. Open Evaluations in a Decision and choose Run tests. You pick:
- Run: the draft or any published version
- Cases: confirmed cases only, or proposals too
- Test sets: the sets to include (Core is ticked for you)
- Cases per request: 1, as in production, or several cases asked together to finish sooner
A Test run takes up to 200 cases. Run tests is available once a Test set has cases; see Examples, Test sets and the Sandbox.
From the CLI, typefuse evaluate <org/decision> runs the tests and prints the Evaluation. Through
MCP, an agent uses run_tests.
Test run evidence
A Test run's page lists every answer beside what each case expected, with the result, the returned answer and its confidence (the probabilities show on hover). Its summary shows how many answers matched, differed or failed, how long it took, and the model and plan that answered.
Run again repeats it, or runs only the failed cases, or the current draft, depending on what the run was. Export JSON downloads the whole record. A Test run is never changed after it finishes.
Reading an Evaluation
Every Test run gets an Evaluation as soon as it finishes: an analysis of the recorded answers against the expected ones. It reads the Test run and asks the model nothing, so evaluating the same run always gives the same result.
An Evaluation shows:
- how many cases agree, and the percentage
- agreement for each question
- exact matches, differences and errors
- which questions no case checks, so you can see gaps in your evidence
- the trend in agreement over recent runs
Answers match only when they are exactly equal: the same noul side, the same choice or the same
score level. Per-item answers are compared item by item.
A difference is a prompt to look again at the expected answer and the Definition, not an automatic failure. An Evaluation says where the Decision and your expectations disagree; it doesn't decide which one is right. There is no pass mark or threshold.
Comparing two Evaluations
To see what changed between two runs, tick two in the list and choose Compare. The earlier run is always "Before". The comparison shows the change in agreement and why it changed (a different version, cases added or removed, or the same cases answered differently), then agreement per question, the cases whose result changed and the cases that were only in one run.
Compare runs over the same Test sets when you want to judge a change to the Definition; otherwise the difference in evidence explains part of the result.
To compare two versions on real production inputs rather than Test cases, use a Comparison.