# Test runs and Evaluations

Run your Test sets against a draft or a version, read the evidence and compare two runs.

## Test runs

A Test run asks a draft or a published version every case in the Test sets you choose, and keeps
every answer it gave. Open **Evaluations** in a Decision and choose **Run tests**. You pick:

- **Run**: the draft or any published version
- **Cases**: confirmed cases only, or proposals too
- **Test sets**: the sets to include (Core is ticked for you)
- **Cases per request**: 1, as in production, or several cases asked together to finish sooner

A Test run takes up to 200 cases. **Run tests** is available once a Test set has cases; see
[Examples, Test sets and the Sandbox](https://typefuse.dev/guides/examples-and-test-sets).

From the CLI, `typefuse evaluate <org/decision>` runs the tests and prints the Evaluation. Through
MCP, an agent uses `run_tests`.

## Test run evidence

A Test run's page lists every answer beside what each case expected, with the result, the returned
answer and its confidence (the probabilities show on hover). Its summary shows how many answers
matched, differed or failed, how long it took, and the model and plan that answered.

**Run again** repeats it, or runs only the failed cases, or the current draft, depending on what the
run was. **Export JSON** downloads the whole record. A Test run is never changed after it finishes.

## Reading an Evaluation

Every Test run gets an Evaluation as soon as it finishes: an analysis of the recorded answers
against the expected ones. It reads the Test run and asks the model nothing, so evaluating the same
run always gives the same result.

An Evaluation shows:

- how many cases agree, and the percentage
- agreement for each question
- exact matches, differences and errors
- which questions no case checks, so you can see gaps in your evidence
- the trend in agreement over recent runs

Answers match only when they are exactly equal: the same `noul` side, the same `choice` or the same
`score` level. Per-item answers are compared item by item.

A difference is a prompt to look again at the expected answer and the Definition, not an automatic
failure. An Evaluation says where the Decision and your expectations disagree; it doesn't decide
which one is right. There is no pass mark or threshold.

## Comparing two Evaluations

To see what changed between two runs, tick two in the list and choose **Compare**. The earlier run
is always "Before". The comparison shows the change in agreement and why it changed (a different
version, cases added or removed, or the same cases answered differently), then agreement per question,
the cases whose result changed and the cases that were only in one run.

Compare runs over the same Test sets when you want to judge a change to the Definition; otherwise the
difference in evidence explains part of the result.

To compare two versions on real production inputs rather than Test cases, use a
[Comparison](https://typefuse.dev/guides/comparisons).