# Datasets and Comparisons

Run two releases over the same real inputs and see where their answers differ before you ship.

## What a Comparison is for

A Comparison answers "what would change?" before you ship. It runs two targets over the same inputs
and shows, question by question, how their answers differ. Unlike a Test run, it needs no expected
answers, so it works on real traffic.

Each side, A and B, can be:

- **Recorded production answers**: what production actually said (side A, for replayed calls)
- **Live in production**: the release deployed there now
- **Draft**: the current draft
- any published version

## Datasets

A Dataset is a fixed set of inputs with no expected answers. Open **Evaluations**, then
**Comparisons**, then **Datasets**, and make one of two kinds:

- **Upload JSONL**: a file or pasted rows, one JSON object per line. Rows that aren't JSON objects
  are refused and listed by line number.
- **Snapshot production**: the inputs of recent production calls, over the days you choose, up to a
  number of calls. Owners and admins can make one, and the snapshot is recorded in the audit trail.
  Only calls whose Decision kept their inputs can be replayed. Relayed calls are left out, and
  calls whose input wasn't kept or is no longer stored (see
  [How long records are kept](https://typefuse.dev/guides/decision-log#how-long-records-are-kept)) are skipped and
  counted.

A Dataset never changes once made. Deleting it keeps the Comparisons made from it.

## Comparisons

Choose **Start a Comparison**. The same dialog opens from **What would change?** in the Definition
editor and **Compare with production** on Versions. Pick a Dataset (by default, **Replay recent
production calls**), how many rows to compare, and sides A and B.

Comparisons run in the background, and the page updates as they progress; you can cancel one.

The result shows:

- how many rows answer differently, and how many were compared, not accepted or failed
- for each question, the spread of A's and B's answers, and what changed from A to B
- the rows themselves, filtered to those that differ, failed or weren't accepted
- how each side performed: speed, tokens and spend, over the rows both answered

When A and B disagree on a row, you can decide which answer is right and add the row to Core as a Test
case, so the difference becomes evidence.

## Replaying production on your plan

Your plan sets how many rows a replay of production compares; past that, it compares a sample. With
your own TypeSafe key saved for the session (on your Account page), a replay compares every row. See
[Plans and usage](https://typefuse.dev/guides/plans).

From the CLI, `typefuse compare` starts a Comparison and `typefuse comparison <id>` shows one; through
MCP an agent uses `compare` and `get_comparison`.