What a Comparison is for
A Comparison answers "what would change?" before you ship. It runs two targets over the same inputs and shows, question by question, how their answers differ. Unlike a Test run, it needs no expected answers, so it works on real traffic.
Each side, A and B, can be:
- Recorded production answers: what production actually said (side A, for replayed calls)
- Live in production: the release deployed there now
- Draft: the current draft
- any published version
Datasets
A Dataset is a fixed set of inputs with no expected answers. Open Evaluations, then Comparisons, then Datasets, and make one of two kinds:
- Upload JSONL: a file or pasted rows, one JSON object per line. Rows that aren't JSON objects are refused and listed by line number.
- Snapshot production: the inputs of recent production calls, over the days you choose, up to a number of calls. Owners and admins can make one, and the snapshot is recorded in the audit trail. Only calls whose Decision kept their inputs can be replayed. Relayed calls are left out, and calls whose input wasn't kept or is no longer stored (see How long records are kept) are skipped and counted.
A Dataset never changes once made. Deleting it keeps the Comparisons made from it.
Comparisons
Choose Start a Comparison. The same dialog opens from What would change? in the Definition editor and Compare with production on Versions. Pick a Dataset (by default, Replay recent production calls), how many rows to compare, and sides A and B.
Comparisons run in the background, and the page updates as they progress; you can cancel one.
The result shows:
- how many rows answer differently, and how many were compared, not accepted or failed
- for each question, the spread of A's and B's answers, and what changed from A to B
- the rows themselves, filtered to those that differ, failed or weren't accepted
- how each side performed: speed, tokens and spend, over the rows both answered
When A and B disagree on a row, you can decide which answer is right and add the row to Core as a Test case, so the difference becomes evidence.
Replaying production on your plan
Your plan sets how many rows a replay of production compares; past that, it compares a sample. With your own TypeSafe key saved for the session (on your Account page), a replay compares every row. See Plans and usage.
From the CLI, typefuse compare starts a Comparison and typefuse comparison <id> shows one; through
MCP an agent uses compare and get_comparison.