# API: Examples, Test sets and Evaluations

Worked Examples, Test set cases, Test runs and the Evaluations that read them.

Part of the API reference: requests carry a personal access token as `Authorization: Bearer <token>`, and bodies and answers are JSON.

- [API reference](https://typefuse.dev/guides/api-reference)
- [Try it in the API console](https://api.typefuse.dev/docs)

## Examples

### GET /v1/organisations/{org}/decisions/{slug}/examples

**List Examples.**

The Decision's Examples; with `full=true`, each whole (its input and expected answers); with `limit` (1 to 100), the first that many.

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |

**Query**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `full` | `boolean` | Optional |  |
| `limit` | `integer` | Optional |  |

It answers with no body.

```sh
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/examples" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"
```

### POST /v1/organisations/{org}/decisions/{slug}/examples

**Add an Example.**

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |

**Body**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `name` | `string` | Required |  |
| `input` | `string or object or array` | Required | Dynamic TypeSafe state: a string, object, or array, checked by state_schema. |
| `expectedAnswers` | `object` | Required |  |
| `note` | `string or null` | Optional |  |

**Answer (200)**

| Name | Type | Description |
| --- | --- | --- |
| `id` | `string` |  |
| `name` | `string` |  |
| `inputKind` | `string`, one of `text`, `record`, `array` |  |
| `inputHash` | `string` |  |
| `inputPreview` | `string` |  |
| `expectedAnswers` | `object` |  |
| `note` | `string or null` |  |
| `status` | `string`, one of `proposed`, `approved` |  |
| `createdAt` | `string` |  |
| `updatedAt` | `string` |  |
| `input` | `string or object or array` | Dynamic TypeSafe state: a string, object, or array, checked by state_schema. |

```sh
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/examples" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"Refunds","input":"<input>","expectedAnswers":{}}'
```

### POST /v1/organisations/{org}/decisions/{slug}/examples/propose-updates

**Propose Example updates.**

Proposes an update for every Example that no longer fits the draft; a proposal takes effect only once accepted. 402 FAIR_USE_REACHED when the Definition assistant is needed and can't be asked for now.

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |

**Answer (200)**

| Name | Type | Description |
| --- | --- | --- |
| `kind` | `string`, one of `cases`, `examples` |  |
| `proposals` | `array` |  |
| `unresolved` | `array of { "targetId": string, "name": string, "reason": string }` |  |
| `model` | `string or null` |  |

```sh
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/examples/propose-updates" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"
```

### GET /v1/organisations/{org}/decisions/{slug}/examples/{exampleId}

**Get an Example.**

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |
| `exampleId` | `string` | Required |  |

**Answer (200)**

| Name | Type | Description |
| --- | --- | --- |
| `id` | `string` |  |
| `name` | `string` |  |
| `inputKind` | `string`, one of `text`, `record`, `array` |  |
| `inputHash` | `string` |  |
| `inputPreview` | `string` |  |
| `expectedAnswers` | `object` |  |
| `note` | `string or null` |  |
| `status` | `string`, one of `proposed`, `approved` |  |
| `createdAt` | `string` |  |
| `updatedAt` | `string` |  |
| `input` | `string or object or array` | Dynamic TypeSafe state: a string, object, or array, checked by state_schema. |

```sh
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/examples/<exampleId>" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"
```

### PATCH /v1/organisations/{org}/decisions/{slug}/examples/{exampleId}

**Update an Example.**

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |
| `exampleId` | `string` | Required |  |

**Body**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `name` | `string` | Optional |  |
| `input` | `string or object or array` | Optional | Dynamic TypeSafe state: a string, object, or array, checked by state_schema. |
| `expectedAnswers` | `object` | Optional |  |
| `note` | `string or null` | Optional |  |
| `status` | `string`, one of `approved` | Optional | Generated proposals become ordinary worked examples only after approval. |
| `proposalId` | `string` | Optional | Accepts an update from propose-updates: saves exactly what was proposed and records it. |

**Answer (200)**

| Name | Type | Description |
| --- | --- | --- |
| `id` | `string` |  |
| `name` | `string` |  |
| `inputKind` | `string`, one of `text`, `record`, `array` |  |
| `inputHash` | `string` |  |
| `inputPreview` | `string` |  |
| `expectedAnswers` | `object` |  |
| `note` | `string or null` |  |
| `status` | `string`, one of `proposed`, `approved` |  |
| `createdAt` | `string` |  |
| `updatedAt` | `string` |  |
| `input` | `string or object or array` | Dynamic TypeSafe state: a string, object, or array, checked by state_schema. |

```sh
curl -X PATCH "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/examples/<exampleId>" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{}'
```

### DELETE /v1/organisations/{org}/decisions/{slug}/examples/{exampleId}

**Delete an Example.**

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |
| `exampleId` | `string` | Required |  |

It answers with no body.

```sh
curl -X DELETE "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/examples/<exampleId>" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"
```

### POST /v1/organisations/{org}/decisions/{slug}/examples/{exampleId}/add-to-test-set

**Add an Example to Test sets.**

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |
| `exampleId` | `string` | Required |  |

**Body**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `testSetIds` | `array of string` | Optional | Empty or omitted selects the protected Core Test set. |

**Answer (200)**

| Name | Type | Description |
| --- | --- | --- |
| `caseId` | `string` |  |
| `testSetIds` | `array of string` |  |
| `created` | `boolean` |  |

```sh
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/examples/<exampleId>/add-to-test-set" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{}'
```

## Test cases

### GET /v1/organisations/{org}/decisions/{slug}/cases

**List test cases.**

The Decision's cases; with `full`, each case whole (as GET /cases/{caseId}), in one response.

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |

**Query**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `testSetId` | `string` | Optional |  |
| `full` | `boolean` | Optional |  |

It answers with no body.

```sh
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/cases" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"
```

### POST /v1/organisations/{org}/decisions/{slug}/cases

**Add a test case.**

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |

**Body**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `name` | `string or null` | Optional |  |
| `input` | `string or object or array` | Required | Dynamic TypeSafe state: a string, object, or array, checked by state_schema. |
| `expectedAnswers` | `object` | Required |  |
| `note` | `string or null` | Optional |  |
| `testSetIds` | `array of string` | Optional | Empty or omitted adds the case to the protected Core Test set. |

**Answer (200)**

| Name | Type | Description |
| --- | --- | --- |
| `id` | `string` |  |
| `name` | `string or null` |  |
| `inputKind` | `string`, one of `text`, `record`, `array` |  |
| `inputHash` | `string` |  |
| `inputPreview` | `string` | One line of the state, for lists. |
| `expectedAnswers` | `object` |  |
| `note` | `string or null` |  |
| `origin` | `string`, one of `manual`, `example`, `run`, `synthetic` | Where a case came from; synthetic cases are the authoring model's and start unconfirmed. |
| `sourceRunId` | `string or null` |  |
| `confirmed` | `boolean` |  |
| `testSetIds` | `array of string` | Named Test sets that currently include this reusable case. |
| `createdAt` | `string` |  |
| `updatedAt` | `string` |  |
| `input` | `string or object or array` | The full state, read from the payload store. |

```sh
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/cases" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"input":"<input>","expectedAnswers":{}}'
```

### POST /v1/organisations/{org}/decisions/{slug}/cases/import-examples

**Import Examples as test cases.**

The draft's examples become cases, once each.

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |

**Answer (200)**

| Name | Type | Description |
| --- | --- | --- |
| `created` | `array` |  |
| `skipped` | `integer` | Inputs whose state already existed as a case. |
| `invalid` | `integer` |  |

```sh
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/cases/import-examples" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"
```

### POST /v1/organisations/{org}/decisions/{slug}/cases/propose-updates

**Propose test case updates.**

Proposes an update for every case that no longer fits the draft; a proposal takes effect only once accepted. 402 FAIR_USE_REACHED when the Definition assistant is needed and can't be asked for now.

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |

**Answer (200)**

| Name | Type | Description |
| --- | --- | --- |
| `kind` | `string`, one of `cases`, `examples` |  |
| `proposals` | `array` |  |
| `unresolved` | `array of { "targetId": string, "name": string, "reason": string }` |  |
| `model` | `string or null` |  |

```sh
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/cases/propose-updates" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"
```

### GET /v1/organisations/{org}/decisions/{slug}/cases/{caseId}

**Get a test case.**

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |
| `caseId` | `string` | Required |  |

**Answer (200)**

| Name | Type | Description |
| --- | --- | --- |
| `id` | `string` |  |
| `name` | `string or null` |  |
| `inputKind` | `string`, one of `text`, `record`, `array` |  |
| `inputHash` | `string` |  |
| `inputPreview` | `string` | One line of the state, for lists. |
| `expectedAnswers` | `object` |  |
| `note` | `string or null` |  |
| `origin` | `string`, one of `manual`, `example`, `run`, `synthetic` | Where a case came from; synthetic cases are the authoring model's and start unconfirmed. |
| `sourceRunId` | `string or null` |  |
| `confirmed` | `boolean` |  |
| `testSetIds` | `array of string` | Named Test sets that currently include this reusable case. |
| `createdAt` | `string` |  |
| `updatedAt` | `string` |  |
| `input` | `string or object or array` | The full state, read from the payload store. |

```sh
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/cases/<caseId>" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"
```

### PATCH /v1/organisations/{org}/decisions/{slug}/cases/{caseId}

**Update a test case.**

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |
| `caseId` | `string` | Required |  |

**Body**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `name` | `string or null` | Optional |  |
| `input` | `string or object or array` | Optional | Dynamic TypeSafe state: a string, object, or array, checked by state_schema. |
| `expectedAnswers` | `object` | Optional |  |
| `note` | `string or null` | Optional |  |
| `confirmed` | `boolean` | Optional |  |
| `proposalId` | `string` | Optional | Accepts an update from propose-updates: saves exactly what was proposed, keeps a confirmed case confirmed and records that the assistant proposed it and the person accepted it. |

**Answer (200)**

| Name | Type | Description |
| --- | --- | --- |
| `id` | `string` |  |
| `name` | `string or null` |  |
| `inputKind` | `string`, one of `text`, `record`, `array` |  |
| `inputHash` | `string` |  |
| `inputPreview` | `string` | One line of the state, for lists. |
| `expectedAnswers` | `object` |  |
| `note` | `string or null` |  |
| `origin` | `string`, one of `manual`, `example`, `run`, `synthetic` | Where a case came from; synthetic cases are the authoring model's and start unconfirmed. |
| `sourceRunId` | `string or null` |  |
| `confirmed` | `boolean` |  |
| `testSetIds` | `array of string` | Named Test sets that currently include this reusable case. |
| `createdAt` | `string` |  |
| `updatedAt` | `string` |  |
| `input` | `string or object or array` | The full state, read from the payload store. |

```sh
curl -X PATCH "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/cases/<caseId>" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{}'
```

### DELETE /v1/organisations/{org}/decisions/{slug}/cases/{caseId}

**Delete a test case.**

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |
| `caseId` | `string` | Required |  |

It answers with no body.

```sh
curl -X DELETE "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/cases/<caseId>" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"
```

### POST /v1/organisations/{org}/decisions/{slug}/cases/generate

**Generate test cases.**

Generates test cases with the Definition assistant; they are created unconfirmed. 402 FAIR_USE_REACHED when the assistant can't be asked for now.

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |

**Body**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `count` | `integer` | Required |  |
| `mode` | `string`, one of `cover`, `boundary`, `variations` | Required |  |
| `caseId` | `string` | Optional | For variations: the case to vary. |
| `testSetIds` | `array of string` | Optional | Empty or omitted adds generated cases to the protected Core Test set. |

**Answer (200)**

| Name | Type | Description |
| --- | --- | --- |
| `created` | `array` |  |
| `skipped` | `integer` | Inputs whose state already existed as a case. |
| `invalid` | `integer` |  |

```sh
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/cases/generate" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"count":1,"mode":"cover"}'
```

## Test sets

### GET /v1/organisations/{org}/decisions/{slug}/test-sets

**List Test sets.**

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |

**Answer (200)**

| Name | Type | Description |
| --- | --- | --- |
| `testSets` | `array` |  |

```sh
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-sets" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"
```

### POST /v1/organisations/{org}/decisions/{slug}/test-sets

**Create a Test set.**

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |

**Body**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `name` | `string` | Required |  |
| `description` | `string or null` | Optional |  |

**Answer (200)**

| Name | Type | Description |
| --- | --- | --- |
| `id` | `string` |  |
| `name` | `string` |  |
| `description` | `string or null` |  |
| `isDefault` | `boolean` | Core is the protected fallback used when no Test set is specified. |
| `caseCount` | `integer` |  |
| `confirmedCount` | `integer` |  |
| `createdAt` | `string` |  |
| `updatedAt` | `string` |  |

```sh
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-sets" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"Refunds"}'
```

### PATCH /v1/organisations/{org}/decisions/{slug}/test-sets/{testSetId}

**Update a Test set.**

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |
| `testSetId` | `string` | Required |  |

**Body**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `name` | `string` | Required |  |
| `description` | `string or null` | Optional |  |

**Answer (200)**

| Name | Type | Description |
| --- | --- | --- |
| `id` | `string` |  |
| `name` | `string` |  |
| `description` | `string or null` |  |
| `isDefault` | `boolean` | Core is the protected fallback used when no Test set is specified. |
| `caseCount` | `integer` |  |
| `confirmedCount` | `integer` |  |
| `createdAt` | `string` |  |
| `updatedAt` | `string` |  |

```sh
curl -X PATCH "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-sets/<testSetId>" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"Refunds"}'
```

### DELETE /v1/organisations/{org}/decisions/{slug}/test-sets/{testSetId}

**Delete a Test set.**

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |
| `testSetId` | `string` | Required |  |

It answers with no body.

```sh
curl -X DELETE "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-sets/<testSetId>" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"
```

### POST /v1/organisations/{org}/decisions/{slug}/test-sets/{testSetId}/cases

**Add test cases to a Test set.**

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |
| `testSetId` | `string` | Required |  |

**Body**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `caseIds` | `array of string` | Required |  |

**Answer (200)**

| Name | Type | Description |
| --- | --- | --- |
| `testSetId` | `string` |  |
| `caseIds` | `array of string` |  |

```sh
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-sets/<testSetId>/cases" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"caseIds":[]}'
```

### DELETE /v1/organisations/{org}/decisions/{slug}/test-sets/{testSetId}/cases/{caseId}

**Take a test case out of a Test set.**

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |
| `testSetId` | `string` | Required |  |
| `caseId` | `string` | Required |  |

It answers with no body.

```sh
curl -X DELETE "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-sets/<testSetId>/cases/<caseId>" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"
```

## Test runs

### GET /v1/organisations/{org}/decisions/{slug}/test-runs

**List Test runs.**

The Decision's last 50 Test runs; with `recent`, the newest that many whole as well.

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |

**Query**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `recent` | `integer` | Optional |  |

**Answer (200)**

| Name | Type | Description |
| --- | --- | --- |
| `testRuns` | `array` |  |
| `recent` | `array` | With `recent=N` (1 to 10): the newest N runs by when they started, whole, as GET …/test-runs/{testRunId} answers each. |

```sh
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-runs" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"
```

### POST /v1/organisations/{org}/decisions/{slug}/test-runs

**Start a Test run.**

Runs the chosen Test set cases and records their raw results, without judging them; an Evaluation reads them. With X-TypeSafe-Key it runs on your own TypeSafe key.

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |

**Headers**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `X-TypeSafe-Key` | `string` | Optional |  |

**Body**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `versionNumber` | `integer or null` | Optional | A published version to run, or null for the current draft. |
| `include` | `string`, one of `confirmed`, `all` | Optional | Confirmed cases only (default), or proposals as well. |
| `batchSize` | `integer` | Optional | Cases per provider call, 1–10; 1 matches production. A Decision with per-item questions runs one case per call. |
| `expansion` | `string`, one of `repeat`, `shared` | Optional | Draft runs only: how per-item (`each`) questions are sent. `repeat` (the default) repeats each question per item; `shared` sends its wording once. Use it to evaluate `shared` before it becomes the default. |
| `testSetIds` | `array of string` | Optional | Test sets whose deduplicated cases should run; defaults to Core. |
| `caseIds` | `array of string` | Optional | An optional subset, used when running one case from a Test set. |

**Answer (200)**

| Name | Type | Description |
| --- | --- | --- |
| `id` | `string` |  |
| `evaluationId` | `string or null` | The deterministic Evaluation derived from this run, when one exists. |
| `createdBy` | `integer` | The user who started this historical run. |
| `createdByName` | `string` |  |
| `versionId` | `string or null` | Null when the draft was run; see draftHash. |
| `versionNumber` | `integer or null` |  |
| `draftHash` | `string or null` |  |
| `planHash` | `string` |  |
| `provider` | `string or null` | The provider that answered, such as "typesafe"; null on runs recorded before this was kept. |
| `providerModel` | `string or null` | The exact model that answered; null on runs recorded before this was kept. |
| `testSetIds` | `array of string` | Exact Test sets selected when this historical run started. |
| `testSetNames` | `array of string` |  |
| `include` | `string`, one of `confirmed`, `all` |  |
| `batchSize` | `integer` |  |
| `status` | `string`, one of `completed`, `completed_with_errors` |  |
| `caseCount` | `integer` |  |
| `succeededCount` | `integer` |  |
| `errorCount` | `integer` |  |
| `inputTokens` | `integer` |  |
| `latencyMs` | `integer` |  |
| `startedAt` | `string` |  |
| `completedAt` | `string` |  |
| `items` | `array` |  |

```sh
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-runs" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{}'
```

### GET /v1/organisations/{org}/decisions/{slug}/test-runs/{testRunId}

**Get a Test run.**

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |
| `testRunId` | `string` | Required |  |

**Answer (200)**

| Name | Type | Description |
| --- | --- | --- |
| `id` | `string` |  |
| `evaluationId` | `string or null` | The deterministic Evaluation derived from this run, when one exists. |
| `createdBy` | `integer` | The user who started this historical run. |
| `createdByName` | `string` |  |
| `versionId` | `string or null` | Null when the draft was run; see draftHash. |
| `versionNumber` | `integer or null` |  |
| `draftHash` | `string or null` |  |
| `planHash` | `string` |  |
| `provider` | `string or null` | The provider that answered, such as "typesafe"; null on runs recorded before this was kept. |
| `providerModel` | `string or null` | The exact model that answered; null on runs recorded before this was kept. |
| `testSetIds` | `array of string` | Exact Test sets selected when this historical run started. |
| `testSetNames` | `array of string` |  |
| `include` | `string`, one of `confirmed`, `all` |  |
| `batchSize` | `integer` |  |
| `status` | `string`, one of `completed`, `completed_with_errors` |  |
| `caseCount` | `integer` |  |
| `succeededCount` | `integer` |  |
| `errorCount` | `integer` |  |
| `inputTokens` | `integer` |  |
| `latencyMs` | `integer` |  |
| `startedAt` | `string` |  |
| `completedAt` | `string` |  |
| `items` | `array` |  |

```sh
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-runs/<testRunId>" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"
```

## Evaluations

### GET /v1/organisations/{org}/decisions/{slug}/evaluations

**List Evaluations.**

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |

**Answer (200)**

| Name | Type | Description |
| --- | --- | --- |
| `evaluations` | `array` |  |

```sh
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/evaluations" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"
```

### POST /v1/organisations/{org}/decisions/{slug}/evaluations

**Evaluate a Test run.**

Deterministically analyses one immutable Test run without another provider call.

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |

**Body**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `testRunId` | `string` | Required | The immutable Test run to analyse. Evaluating never calls the provider again. |

**Answer (200)**

| Name | Type | Description |
| --- | --- | --- |
| `id` | `string` |  |
| `testRunId` | `string or null` | The immutable Test run analysed by this Evaluation; null only for historical Evaluations. |
| `versionId` | `string or null` | Null when the draft was evaluated; see draftHash. |
| `versionNumber` | `integer or null` |  |
| `draftHash` | `string or null` |  |
| `planHash` | `string` |  |
| `batchSize` | `integer` |  |
| `repeatOf` | `string or null` | The evaluation this one repeated on the same cases and plan, if any. |
| `testSetIds` | `array of string` | Exact Test sets selected when this historical evaluation was started. |
| `testSetNames` | `array of string` |  |
| `status` | `string`, one of `done`, `error` |  |
| `error` | `string or null` |  |
| `caseCount` | `integer` |  |
| `agreedCount` | `integer` |  |
| `perQuestion` | `object` |  |
| `byOrigin` | `object` |  |
| `inputTokens` | `integer` |  |
| `latencyMs` | `integer` |  |
| `startedAt` | `string` |  |
| `completedAt` | `string` |  |
| `coverage` | `object or null` | Null when no case returned answers, so the Decision's questions are unknown. |
| `coverage.questions` | `array of string` | Every question the Decision answered in the Test run. |
| `coverage.checked` | `array of string` | Questions at least one case sets an expectation for; see perQuestion for their agreement. |
| `coverage.unchecked` | `array of string` | Questions no case checks: their answers were never compared. |
| `calibration` | `object or null` | Null when no case expects a yes/no or choice answer. |
| `calibration.status` | `string`, one of `measured`, `not_available` | `not_available` when the model gave no probability for each possible answer (only a single confidence, or nothing). |
| `calibration.answers` | `integer` | Labelled yes/no and choice answers with a distribution: the answers measured. |
| `calibration.agreed` | `integer` | Of those, how many matched the expected answer. |
| `calibration.withoutProbabilities` | `integer` | Labelled yes/no and choice answers left out because the model gave no probability for each answer. |
| `calibration.brier` | `number or null` | Mean Brier score, 0 (certain and always right) to 1 (certain and always wrong); a yes/no answer scores (1 − p)² for the probability p it gave the expected answer. |
| `calibration.expectedCalibrationError` | `number or null` | Expected calibration error over ten equal bands: the count-weighted gap between stated probability and how often the answers were right. |
| `calibration.rough` | `boolean` | Fewer than 50 answers were measured: read the figures as rough. |
| `calibration.bins` | `array` | Ten equal bands of the probability each answer gave the answer it chose. |
| `items` | `array` |  |

```sh
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/evaluations" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"testRunId":"<testRunId>"}'
```

### GET /v1/organisations/{org}/decisions/{slug}/evaluations/{evaluationId}

**Get an Evaluation.**

**Path**

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `org` | `string` | Required |  |
| `slug` | `string` | Required |  |
| `evaluationId` | `string` | Required |  |

**Answer (200)**

| Name | Type | Description |
| --- | --- | --- |
| `id` | `string` |  |
| `testRunId` | `string or null` | The immutable Test run analysed by this Evaluation; null only for historical Evaluations. |
| `versionId` | `string or null` | Null when the draft was evaluated; see draftHash. |
| `versionNumber` | `integer or null` |  |
| `draftHash` | `string or null` |  |
| `planHash` | `string` |  |
| `batchSize` | `integer` |  |
| `repeatOf` | `string or null` | The evaluation this one repeated on the same cases and plan, if any. |
| `testSetIds` | `array of string` | Exact Test sets selected when this historical evaluation was started. |
| `testSetNames` | `array of string` |  |
| `status` | `string`, one of `done`, `error` |  |
| `error` | `string or null` |  |
| `caseCount` | `integer` |  |
| `agreedCount` | `integer` |  |
| `perQuestion` | `object` |  |
| `byOrigin` | `object` |  |
| `inputTokens` | `integer` |  |
| `latencyMs` | `integer` |  |
| `startedAt` | `string` |  |
| `completedAt` | `string` |  |
| `coverage` | `object or null` | Null when no case returned answers, so the Decision's questions are unknown. |
| `coverage.questions` | `array of string` | Every question the Decision answered in the Test run. |
| `coverage.checked` | `array of string` | Questions at least one case sets an expectation for; see perQuestion for their agreement. |
| `coverage.unchecked` | `array of string` | Questions no case checks: their answers were never compared. |
| `calibration` | `object or null` | Null when no case expects a yes/no or choice answer. |
| `calibration.status` | `string`, one of `measured`, `not_available` | `not_available` when the model gave no probability for each possible answer (only a single confidence, or nothing). |
| `calibration.answers` | `integer` | Labelled yes/no and choice answers with a distribution: the answers measured. |
| `calibration.agreed` | `integer` | Of those, how many matched the expected answer. |
| `calibration.withoutProbabilities` | `integer` | Labelled yes/no and choice answers left out because the model gave no probability for each answer. |
| `calibration.brier` | `number or null` | Mean Brier score, 0 (certain and always right) to 1 (certain and always wrong); a yes/no answer scores (1 − p)² for the probability p it gave the expected answer. |
| `calibration.expectedCalibrationError` | `number or null` | Expected calibration error over ten equal bands: the count-weighted gap between stated probability and how often the answers were right. |
| `calibration.rough` | `boolean` | Fewer than 50 answers were measured: read the figures as rough. |
| `calibration.bins` | `array` | Ten equal bands of the probability each answer gave the answer it chose. |
| `items` | `array` |  |

```sh
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/evaluations/<evaluationId>" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"
```
