Skip to content

Prove.

How you know a Decision is right before anyone calls it.

Collect real requests as Worked Examples while you build. Confirm the answer for the ones you trust and they become the Test set, which every draft and release is measured against.

A Decision is only as good as the cases it has been checked against. Proving it means agreeing on the right answer for real requests, then running every change against all of them.

Examples you’re still unsure about stay separate, so they never count as evidence by accident.

Who
The people who know the right answers
Where
Examples, Test sets and Evaluations in the app, or the CLI in CI
You end up with
A release with a measured result on cases you trust

Start from real requests.

A Worked Example is a request with the answer it should get. Save them while you build, from your own records or from real calls, correcting the answer where it was wrong.

  • Kept with the Decision, beside the draft
  • Separate from the Test set until you confirm them
  • Keep any call as an example, correcting its answers first

Only confirmed cases count.

Promote the examples you trust into the Test set. Those are the only cases that count as evidence, and by default a release reaches Production only when it agrees on every one.

  • Every Evaluation runs the whole set
  • Each failure is named: which case, what was expected, what came back
  • Fix and re-run until it agrees

A score you can compare.

An Evaluation runs a release against the Test set and keeps the result with it: how many cases agree, and how well its confidence matches how often it is right.

  • Agreement case by case
  • Brier score and calibration error for the confidence
  • Kept with the release, and in its evidence pack on paid plans

Compare on real traffic.

A Comparison runs two releases, or a release and your draft, over the same inputs, such as last week’s calls, and shows where their answers differ. Inputs with no agreed answer are fine here: the question is what would change.

  • The rows that disagree, as a list to review
  • The mix of answers on each side
  • Cost and response time for each side

Try a new model on your cases.

Before you move to a new model version, run your Test set and a Comparison against it. Nothing changes for callers until you publish a release that uses it.

  • Your own cases: the Test set you already trust
  • Side by side with the release in use
  • A new release when you choose to move

Where it happens.

In the app

  • Worked Examples and Test sets on each Decision
  • Evaluations and Comparisons side by side
  • Keep a call as an example from its record

Your agent, over MCP

  • Adds examples and runs the tests
  • Reads which cases failed and why
  • Proposes a fix to the draft

Your code

  • typefuse evaluate in CI
  • Upload a dataset as JSON lines
  • Compare a draft against it

Private preview

Own your next decision.

Keep the judgement calls your agents and apps make under your control, with a record of every answer.

  • Your criteriaThe people who own a decision set its rules, and change them when the business changes.The people who own a decision set its rules.
  • Your evidenceEvery release is tested against real cases before your application ever sees it.Every release is tested on real cases first.
  • Your recordEvery answer is traced and sealed, ready for an auditor or the customer who asks why.Every answer is traced and sealed.