A Decision is only as good as the cases it has been checked against. Proving it means agreeing on the right answer for real requests, then running every change against all of them.
Examples you’re still unsure about stay separate, so they never count as evidence by accident.
- Who
- The people who know the right answers
- Where
- Examples, Test sets and Evaluations in the app, or the CLI in CI
- You end up with
- A release with a measured result on cases you trust
Start from real requests.
A Worked Example is a request with the answer it should get. Save them while you build, from your own records or from real calls, correcting the answer where it was wrong.
- Kept with the Decision, beside the draft
- Separate from the Test set until you confirm them
- Keep any call as an example, correcting its answers first
acme/refunds| Request | Should answer | Status |
|---|---|---|
| £84 · 12 days · damaged | review | In Test set |
| £19 · 4 days · new | approve | In Test set |
| £230 · 31 days · damaged | review | Example only |
| £52 · 29 days · used | approve | In Test set |
Only confirmed cases count.
Promote the examples you trust into the Test set. Those are the only cases that count as evidence, and by default a release reaches Production only when it agrees on every one.
- Every Evaluation runs the whole set
- Each failure is named: which case, what was expected, what came back
- Fix and re-run until it agrees
| Order | Since delivery | Condition | Expected | Answered |
|---|---|---|---|---|
| £84 | 12 days | damaged | review | review✓ |
| £19 | 4 days | new | approve | approve✓ |
| £640 | 45 days | used | decline | decline✓ |
| £230 | 31 days | damaged | review | review✓ |
| £52 | 29 days | used | approve | approve✓ |
A score you can compare.
An Evaluation runs a release against the Test set and keeps the result with it: how many cases agree, and how well its confidence matches how often it is right.
- Agreement case by case
- Brier score and calibration error for the confidence
- Kept with the release, and in its evidence pack on paid plans
- Agreement
- 42 of 42 cases
- Brier score
- 0.06
- Calibration error
- 0.03
- Ran on
- Test set, 42 confirmed cases
illustrative figures
Compare on real traffic.
A Comparison runs two releases, or a release and your draft, over the same inputs, such as last week’s calls, and shows where their answers differ. Inputs with no agreed answer are fine here: the question is what would change.
- The rows that disagree, as a list to review
- The mix of answers on each side
- Cost and response time for each side
| 1.4.0 | Draft | |
|---|---|---|
| approve | 75% | 73% |
| review | 21% | 24% |
| decline | 4% | 3% |
| Disagree | – | 37 of 1,780 |
| p95 | 160 ms | 164 ms |
Replayed from last week’s production calls. illustrative figures
Try a new model on your cases.
Before you move to a new model version, run your Test set and a Comparison against it. Nothing changes for callers until you publish a release that uses it.
- Your own cases: the Test set you already trust
- Side by side with the release in use
- A new release when you choose to move
acme/refunds| Model | Test set | p95 |
|---|---|---|
jev-1.13.0 | 42/42 | 160 ms |
jev-1.14.0 | 42/42 | 150 ms |
illustrative figures
Where it happens.
In the app
- Worked Examples and Test sets on each Decision
- Evaluations and Comparisons side by side
- Keep a call as an example from its record
Your agent, over MCP
- Adds examples and runs the tests
- Reads which cases failed and why
- Proposes a fix to the draft
Your code
typefuse evaluatein CI- Upload a dataset as JSON lines
- Compare a draft against it
Private preview
Own your next decision.
Keep the judgement calls your agents and apps make under your control, with a record of every answer.
- Your criteriaThe people who own a decision set its rules, and change them when the business changes.The people who own a decision set its rules.
- Your evidenceEvery release is tested against real cases before your application ever sees it.Every release is tested on real cases first.
- Your recordEvery answer is traced and sealed, ready for an auditor or the customer who asks why.Every answer is traced and sealed.