Skip to content

Reference

API: Examples, Test sets and Evaluations

Worked Examples, Test set cases, Test runs and the Evaluations that read them.

Part of the API reference: requests carry a personal access token as Authorization: Bearer <token>, and bodies and answers are JSON.

Examples

GET /v1/organisations/{org}/decisions/{slug}/examples

List Examples

The Decision's Examples; with full=true, each whole (its input and expected answers); with limit (1 to 100), the first that many.

Path

  • orgRequiredstring
  • slugRequiredstring

Query

  • fullOptionalboolean
  • limitOptionalinteger

Answer (200)

It answers with no body.

Terminal
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/examples" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"

POST /v1/organisations/{org}/decisions/{slug}/examples

Add an Example

Path

  • orgRequiredstring
  • slugRequiredstring

Body

  • nameRequiredstring
  • inputRequiredstring or object or array
    Dynamic TypeSafe state: a string, object, or array, checked by state_schema.
  • expectedAnswersRequiredobject
  • noteOptionalstring or null

Answer (200)

  • idstring
  • namestring
  • inputKindstring
    One of text, record, array
  • inputHashstring
  • inputPreviewstring
  • expectedAnswersobject
  • notestring or null
  • statusstring
    One of proposed, approved
  • createdAtstring
  • updatedAtstring
  • inputstring or object or array
    Dynamic TypeSafe state: a string, object, or array, checked by state_schema.
Terminal
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/examples" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"Refunds","input":"<input>","expectedAnswers":{}}'

POST /v1/organisations/{org}/decisions/{slug}/examples/propose-updates

Propose Example updates

Proposes an update for every Example that no longer fits the draft; a proposal takes effect only once accepted. 402 FAIR_USE_REACHED when the Definition assistant is needed and can't be asked for now.

Path

  • orgRequiredstring
  • slugRequiredstring

Answer (200)

  • kindstring
    One of cases, examples
  • proposalsarray
  • unresolvedarray of { "targetId": string, "name": string, "reason": string }
  • modelstring or null
Terminal
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/examples/propose-updates" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"

GET /v1/organisations/{org}/decisions/{slug}/examples/{exampleId}

Get an Example

Path

  • orgRequiredstring
  • slugRequiredstring
  • exampleIdRequiredstring

Answer (200)

  • idstring
  • namestring
  • inputKindstring
    One of text, record, array
  • inputHashstring
  • inputPreviewstring
  • expectedAnswersobject
  • notestring or null
  • statusstring
    One of proposed, approved
  • createdAtstring
  • updatedAtstring
  • inputstring or object or array
    Dynamic TypeSafe state: a string, object, or array, checked by state_schema.
Terminal
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/examples/<exampleId>" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"

PATCH /v1/organisations/{org}/decisions/{slug}/examples/{exampleId}

Update an Example

Path

  • orgRequiredstring
  • slugRequiredstring
  • exampleIdRequiredstring

Body

  • nameOptionalstring
  • inputOptionalstring or object or array
    Dynamic TypeSafe state: a string, object, or array, checked by state_schema.
  • expectedAnswersOptionalobject
  • noteOptionalstring or null
  • statusOptionalstring
    Generated proposals become ordinary worked examples only after approval.One of approved
  • proposalIdOptionalstring
    Accepts an update from propose-updates: saves exactly what was proposed and records it.

Answer (200)

  • idstring
  • namestring
  • inputKindstring
    One of text, record, array
  • inputHashstring
  • inputPreviewstring
  • expectedAnswersobject
  • notestring or null
  • statusstring
    One of proposed, approved
  • createdAtstring
  • updatedAtstring
  • inputstring or object or array
    Dynamic TypeSafe state: a string, object, or array, checked by state_schema.
Terminal
curl -X PATCH "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/examples/<exampleId>" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{}'

DELETE /v1/organisations/{org}/decisions/{slug}/examples/{exampleId}

Delete an Example

Path

  • orgRequiredstring
  • slugRequiredstring
  • exampleIdRequiredstring

Answer (204)

It answers with no body.

Terminal
curl -X DELETE "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/examples/<exampleId>" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"

POST /v1/organisations/{org}/decisions/{slug}/examples/{exampleId}/add-to-test-set

Add an Example to Test sets

Path

  • orgRequiredstring
  • slugRequiredstring
  • exampleIdRequiredstring

Body

  • testSetIdsOptionalarray of string
    Empty or omitted selects the protected Core Test set.

Answer (200)

  • caseIdstring
  • testSetIdsarray of string
  • createdboolean
Terminal
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/examples/<exampleId>/add-to-test-set" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{}'

Test cases

GET /v1/organisations/{org}/decisions/{slug}/cases

List test cases

The Decision's cases; with full, each case whole (as GET /cases/{caseId}), in one response.

Path

  • orgRequiredstring
  • slugRequiredstring

Query

  • testSetIdOptionalstring
  • fullOptionalboolean

Answer (200)

It answers with no body.

Terminal
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/cases" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"

POST /v1/organisations/{org}/decisions/{slug}/cases

Add a test case

Path

  • orgRequiredstring
  • slugRequiredstring

Body

  • nameOptionalstring or null
  • inputRequiredstring or object or array
    Dynamic TypeSafe state: a string, object, or array, checked by state_schema.
  • expectedAnswersRequiredobject
  • noteOptionalstring or null
  • testSetIdsOptionalarray of string
    Empty or omitted adds the case to the protected Core Test set.

Answer (200)

  • idstring
  • namestring or null
  • inputKindstring
    One of text, record, array
  • inputHashstring
  • inputPreviewstring
    One line of the state, for lists.
  • expectedAnswersobject
  • notestring or null
  • originstring
    Where a case came from; synthetic cases are the authoring model's and start unconfirmed.One of manual, example, run, synthetic
  • sourceRunIdstring or null
  • confirmedboolean
  • testSetIdsarray of string
    Named Test sets that currently include this reusable case.
  • createdAtstring
  • updatedAtstring
  • inputstring or object or array
    The full state, read from the payload store.
Terminal
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/cases" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"input":"<input>","expectedAnswers":{}}'

POST /v1/organisations/{org}/decisions/{slug}/cases/import-examples

Import Examples as test cases

The draft's examples become cases, once each.

Path

  • orgRequiredstring
  • slugRequiredstring

Answer (200)

  • createdarray
  • skippedinteger
    Inputs whose state already existed as a case.
  • invalidinteger
Terminal
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/cases/import-examples" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"

POST /v1/organisations/{org}/decisions/{slug}/cases/propose-updates

Propose test case updates

Proposes an update for every case that no longer fits the draft; a proposal takes effect only once accepted. 402 FAIR_USE_REACHED when the Definition assistant is needed and can't be asked for now.

Path

  • orgRequiredstring
  • slugRequiredstring

Answer (200)

  • kindstring
    One of cases, examples
  • proposalsarray
  • unresolvedarray of { "targetId": string, "name": string, "reason": string }
  • modelstring or null
Terminal
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/cases/propose-updates" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"

GET /v1/organisations/{org}/decisions/{slug}/cases/{caseId}

Get a test case

Path

  • orgRequiredstring
  • slugRequiredstring
  • caseIdRequiredstring

Answer (200)

  • idstring
  • namestring or null
  • inputKindstring
    One of text, record, array
  • inputHashstring
  • inputPreviewstring
    One line of the state, for lists.
  • expectedAnswersobject
  • notestring or null
  • originstring
    Where a case came from; synthetic cases are the authoring model's and start unconfirmed.One of manual, example, run, synthetic
  • sourceRunIdstring or null
  • confirmedboolean
  • testSetIdsarray of string
    Named Test sets that currently include this reusable case.
  • createdAtstring
  • updatedAtstring
  • inputstring or object or array
    The full state, read from the payload store.
Terminal
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/cases/<caseId>" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"

PATCH /v1/organisations/{org}/decisions/{slug}/cases/{caseId}

Update a test case

Path

  • orgRequiredstring
  • slugRequiredstring
  • caseIdRequiredstring

Body

  • nameOptionalstring or null
  • inputOptionalstring or object or array
    Dynamic TypeSafe state: a string, object, or array, checked by state_schema.
  • expectedAnswersOptionalobject
  • noteOptionalstring or null
  • confirmedOptionalboolean
  • proposalIdOptionalstring
    Accepts an update from propose-updates: saves exactly what was proposed, keeps a confirmed case confirmed and records that the assistant proposed it and the person accepted it.

Answer (200)

  • idstring
  • namestring or null
  • inputKindstring
    One of text, record, array
  • inputHashstring
  • inputPreviewstring
    One line of the state, for lists.
  • expectedAnswersobject
  • notestring or null
  • originstring
    Where a case came from; synthetic cases are the authoring model's and start unconfirmed.One of manual, example, run, synthetic
  • sourceRunIdstring or null
  • confirmedboolean
  • testSetIdsarray of string
    Named Test sets that currently include this reusable case.
  • createdAtstring
  • updatedAtstring
  • inputstring or object or array
    The full state, read from the payload store.
Terminal
curl -X PATCH "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/cases/<caseId>" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{}'

DELETE /v1/organisations/{org}/decisions/{slug}/cases/{caseId}

Delete a test case

Path

  • orgRequiredstring
  • slugRequiredstring
  • caseIdRequiredstring

Answer (204)

It answers with no body.

Terminal
curl -X DELETE "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/cases/<caseId>" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"

POST /v1/organisations/{org}/decisions/{slug}/cases/generate

Generate test cases

Generates test cases with the Definition assistant; they are created unconfirmed. 402 FAIR_USE_REACHED when the assistant can't be asked for now.

Path

  • orgRequiredstring
  • slugRequiredstring

Body

  • countRequiredinteger
  • modeRequiredstring
    One of cover, boundary, variations
  • caseIdOptionalstring
    For variations: the case to vary.
  • testSetIdsOptionalarray of string
    Empty or omitted adds generated cases to the protected Core Test set.

Answer (200)

  • createdarray
  • skippedinteger
    Inputs whose state already existed as a case.
  • invalidinteger
Terminal
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/cases/generate" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"count":1,"mode":"cover"}'

Test sets

GET /v1/organisations/{org}/decisions/{slug}/test-sets

List Test sets

Path

  • orgRequiredstring
  • slugRequiredstring

Answer (200)

  • testSetsarray
Terminal
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-sets" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"

POST /v1/organisations/{org}/decisions/{slug}/test-sets

Create a Test set

Path

  • orgRequiredstring
  • slugRequiredstring

Body

  • nameRequiredstring
  • descriptionOptionalstring or null

Answer (200)

  • idstring
  • namestring
  • descriptionstring or null
  • isDefaultboolean
    Core is the protected fallback used when no Test set is specified.
  • caseCountinteger
  • confirmedCountinteger
  • createdAtstring
  • updatedAtstring
Terminal
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-sets" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"Refunds"}'

PATCH /v1/organisations/{org}/decisions/{slug}/test-sets/{testSetId}

Update a Test set

Path

  • orgRequiredstring
  • slugRequiredstring
  • testSetIdRequiredstring

Body

  • nameRequiredstring
  • descriptionOptionalstring or null

Answer (200)

  • idstring
  • namestring
  • descriptionstring or null
  • isDefaultboolean
    Core is the protected fallback used when no Test set is specified.
  • caseCountinteger
  • confirmedCountinteger
  • createdAtstring
  • updatedAtstring
Terminal
curl -X PATCH "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-sets/<testSetId>" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"Refunds"}'

DELETE /v1/organisations/{org}/decisions/{slug}/test-sets/{testSetId}

Delete a Test set

Path

  • orgRequiredstring
  • slugRequiredstring
  • testSetIdRequiredstring

Answer (204)

It answers with no body.

Terminal
curl -X DELETE "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-sets/<testSetId>" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"

POST /v1/organisations/{org}/decisions/{slug}/test-sets/{testSetId}/cases

Add test cases to a Test set

Path

  • orgRequiredstring
  • slugRequiredstring
  • testSetIdRequiredstring

Body

  • caseIdsRequiredarray of string

Answer (200)

  • testSetIdstring
  • caseIdsarray of string
Terminal
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-sets/<testSetId>/cases" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"caseIds":[]}'

DELETE /v1/organisations/{org}/decisions/{slug}/test-sets/{testSetId}/cases/{caseId}

Take a test case out of a Test set

Path

  • orgRequiredstring
  • slugRequiredstring
  • testSetIdRequiredstring
  • caseIdRequiredstring

Answer (204)

It answers with no body.

Terminal
curl -X DELETE "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-sets/<testSetId>/cases/<caseId>" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"

Test runs

GET /v1/organisations/{org}/decisions/{slug}/test-runs

List Test runs

The Decision's last 50 Test runs; with recent, the newest that many whole as well.

Path

  • orgRequiredstring
  • slugRequiredstring

Query

  • recentOptionalinteger

Answer (200)

  • testRunsarray
  • recentarray
    With recent=N (1 to 10): the newest N runs by when they started, whole, as GET …/test-runs/{testRunId} answers each.
Terminal
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-runs" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"

POST /v1/organisations/{org}/decisions/{slug}/test-runs

Start a Test run

Runs the chosen Test set cases and records their raw results, without judging them; an Evaluation reads them. With X-TypeSafe-Key it runs on your own TypeSafe key.

Path

  • orgRequiredstring
  • slugRequiredstring

Headers

  • X-TypeSafe-KeyOptionalstring

Body

  • versionNumberOptionalinteger or null
    A published version to run, or null for the current draft.
  • includeOptionalstring
    Confirmed cases only (default), or proposals as well.One of confirmed, all
  • batchSizeOptionalinteger
    Cases per provider call, 1–10; 1 matches production. A Decision with per-item questions runs one case per call.
  • expansionOptionalstring
    Draft runs only: how per-item (each) questions are sent. repeat (the default) repeats each question per item; shared sends its wording once. Use it to evaluate shared before it becomes the default.One of repeat, shared
  • testSetIdsOptionalarray of string
    Test sets whose deduplicated cases should run; defaults to Core.
  • caseIdsOptionalarray of string
    An optional subset, used when running one case from a Test set.

Answer (200)

  • idstring
  • evaluationIdstring or null
    The deterministic Evaluation derived from this run, when one exists.
  • createdByinteger
    The user who started this historical run.
  • createdByNamestring
  • versionIdstring or null
    Null when the draft was run; see draftHash.
  • versionNumberinteger or null
  • draftHashstring or null
  • planHashstring
  • providerstring or null
    The provider that answered, such as "typesafe"; null on runs recorded before this was kept.
  • providerModelstring or null
    The exact model that answered; null on runs recorded before this was kept.
  • testSetIdsarray of string
    Exact Test sets selected when this historical run started.
  • testSetNamesarray of string
  • includestring
    One of confirmed, all
  • batchSizeinteger
  • statusstring
    One of completed, completed_with_errors
  • caseCountinteger
  • succeededCountinteger
  • errorCountinteger
  • inputTokensinteger
  • latencyMsinteger
  • startedAtstring
  • completedAtstring
  • itemsarray
Terminal
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-runs" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{}'

GET /v1/organisations/{org}/decisions/{slug}/test-runs/{testRunId}

Get a Test run

Path

  • orgRequiredstring
  • slugRequiredstring
  • testRunIdRequiredstring

Answer (200)

  • idstring
  • evaluationIdstring or null
    The deterministic Evaluation derived from this run, when one exists.
  • createdByinteger
    The user who started this historical run.
  • createdByNamestring
  • versionIdstring or null
    Null when the draft was run; see draftHash.
  • versionNumberinteger or null
  • draftHashstring or null
  • planHashstring
  • providerstring or null
    The provider that answered, such as "typesafe"; null on runs recorded before this was kept.
  • providerModelstring or null
    The exact model that answered; null on runs recorded before this was kept.
  • testSetIdsarray of string
    Exact Test sets selected when this historical run started.
  • testSetNamesarray of string
  • includestring
    One of confirmed, all
  • batchSizeinteger
  • statusstring
    One of completed, completed_with_errors
  • caseCountinteger
  • succeededCountinteger
  • errorCountinteger
  • inputTokensinteger
  • latencyMsinteger
  • startedAtstring
  • completedAtstring
  • itemsarray
Terminal
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-runs/<testRunId>" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"

Evaluations

GET /v1/organisations/{org}/decisions/{slug}/evaluations

List Evaluations

Path

  • orgRequiredstring
  • slugRequiredstring

Answer (200)

  • evaluationsarray
Terminal
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/evaluations" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"

POST /v1/organisations/{org}/decisions/{slug}/evaluations

Evaluate a Test run

Deterministically analyses one immutable Test run without another provider call.

Path

  • orgRequiredstring
  • slugRequiredstring

Body

  • testRunIdRequiredstring
    The immutable Test run to analyse. Evaluating never calls the provider again.

Answer (200)

  • idstring
  • testRunIdstring or null
    The immutable Test run analysed by this Evaluation; null only for historical Evaluations.
  • versionIdstring or null
    Null when the draft was evaluated; see draftHash.
  • versionNumberinteger or null
  • draftHashstring or null
  • planHashstring
  • batchSizeinteger
  • repeatOfstring or null
    The evaluation this one repeated on the same cases and plan, if any.
  • testSetIdsarray of string
    Exact Test sets selected when this historical evaluation was started.
  • testSetNamesarray of string
  • statusstring
    One of done, error
  • errorstring or null
  • caseCountinteger
  • agreedCountinteger
  • perQuestionobject
  • byOriginobject
  • inputTokensinteger
  • latencyMsinteger
  • startedAtstring
  • completedAtstring
  • coverageobject or null
    Null when no case returned answers, so the Decision's questions are unknown.
  • coverage.questionsarray of string
    Every question the Decision answered in the Test run.
  • coverage.checkedarray of string
    Questions at least one case sets an expectation for; see perQuestion for their agreement.
  • coverage.uncheckedarray of string
    Questions no case checks: their answers were never compared.
  • calibrationobject or null
    Null when no case expects a yes/no or choice answer.
  • calibration.statusstring
    not_available when the model gave no probability for each possible answer (only a single confidence, or nothing).One of measured, not_available
  • calibration.answersinteger
    Labelled yes/no and choice answers with a distribution: the answers measured.
  • calibration.agreedinteger
    Of those, how many matched the expected answer.
  • calibration.withoutProbabilitiesinteger
    Labelled yes/no and choice answers left out because the model gave no probability for each answer.
  • calibration.briernumber or null
    Mean Brier score, 0 (certain and always right) to 1 (certain and always wrong); a yes/no answer scores (1 − p)² for the probability p it gave the expected answer.
  • calibration.expectedCalibrationErrornumber or null
    Expected calibration error over ten equal bands: the count-weighted gap between stated probability and how often the answers were right.
  • calibration.roughboolean
    Fewer than 50 answers were measured: read the figures as rough.
  • calibration.binsarray
    Ten equal bands of the probability each answer gave the answer it chose.
  • itemsarray
Terminal
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/evaluations" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"testRunId":"<testRunId>"}'

GET /v1/organisations/{org}/decisions/{slug}/evaluations/{evaluationId}

Get an Evaluation

Path

  • orgRequiredstring
  • slugRequiredstring
  • evaluationIdRequiredstring

Answer (200)

  • idstring
  • testRunIdstring or null
    The immutable Test run analysed by this Evaluation; null only for historical Evaluations.
  • versionIdstring or null
    Null when the draft was evaluated; see draftHash.
  • versionNumberinteger or null
  • draftHashstring or null
  • planHashstring
  • batchSizeinteger
  • repeatOfstring or null
    The evaluation this one repeated on the same cases and plan, if any.
  • testSetIdsarray of string
    Exact Test sets selected when this historical evaluation was started.
  • testSetNamesarray of string
  • statusstring
    One of done, error
  • errorstring or null
  • caseCountinteger
  • agreedCountinteger
  • perQuestionobject
  • byOriginobject
  • inputTokensinteger
  • latencyMsinteger
  • startedAtstring
  • completedAtstring
  • coverageobject or null
    Null when no case returned answers, so the Decision's questions are unknown.
  • coverage.questionsarray of string
    Every question the Decision answered in the Test run.
  • coverage.checkedarray of string
    Questions at least one case sets an expectation for; see perQuestion for their agreement.
  • coverage.uncheckedarray of string
    Questions no case checks: their answers were never compared.
  • calibrationobject or null
    Null when no case expects a yes/no or choice answer.
  • calibration.statusstring
    not_available when the model gave no probability for each possible answer (only a single confidence, or nothing).One of measured, not_available
  • calibration.answersinteger
    Labelled yes/no and choice answers with a distribution: the answers measured.
  • calibration.agreedinteger
    Of those, how many matched the expected answer.
  • calibration.withoutProbabilitiesinteger
    Labelled yes/no and choice answers left out because the model gave no probability for each answer.
  • calibration.briernumber or null
    Mean Brier score, 0 (certain and always right) to 1 (certain and always wrong); a yes/no answer scores (1 − p)² for the probability p it gave the expected answer.
  • calibration.expectedCalibrationErrornumber or null
    Expected calibration error over ten equal bands: the count-weighted gap between stated probability and how often the answers were right.
  • calibration.roughboolean
    Fewer than 50 answers were measured: read the figures as rough.
  • calibration.binsarray
    Ten equal bands of the probability each answer gave the answer it chose.
  • itemsarray
Terminal
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/evaluations/<evaluationId>" \
  -H "Authorization: Bearer $TYPEFUSE_TOKEN"