Reference
API: Examples, Test sets and Evaluations
Worked Examples, Test set cases, Test runs and the Evaluations that read them.
Part of the API reference: requests carry a personal access token as Authorization: Bearer <token>, and bodies and answers are JSON.
Examples
GET /v1/organisations/{org}/decisions/{slug}/examples
List Examples
The Decision's Examples; with full=true, each whole (its input and expected answers); with limit (1 to 100), the first that many.
Path
orgRequiredstringslugRequiredstring
Query
fullOptionalbooleanlimitOptionalinteger
Answer (200)
It answers with no body.
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/examples" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN"POST /v1/organisations/{org}/decisions/{slug}/examples
Add an Example
Path
orgRequiredstringslugRequiredstring
Body
nameRequiredstringinputRequiredstring or object or arrayDynamic TypeSafe state: a string, object, or array, checked by state_schema.expectedAnswersRequiredobjectnoteOptionalstring or null
Answer (200)
idstringnamestringinputKindstringOne oftext,record,arrayinputHashstringinputPreviewstringexpectedAnswersobjectnotestring or nullstatusstringOne ofproposed,approvedcreatedAtstringupdatedAtstringinputstring or object or arrayDynamic TypeSafe state: a string, object, or array, checked by state_schema.
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/examples" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN" \
-H "Content-Type: application/json" \
-d '{"name":"Refunds","input":"<input>","expectedAnswers":{}}'POST /v1/organisations/{org}/decisions/{slug}/examples/propose-updates
Propose Example updates
Proposes an update for every Example that no longer fits the draft; a proposal takes effect only once accepted. 402 FAIR_USE_REACHED when the Definition assistant is needed and can't be asked for now.
Path
orgRequiredstringslugRequiredstring
Answer (200)
kindstringOne ofcases,examplesproposalsarrayunresolvedarray of { "targetId": string, "name": string, "reason": string }modelstring or null
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/examples/propose-updates" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN"GET /v1/organisations/{org}/decisions/{slug}/examples/{exampleId}
Get an Example
Path
orgRequiredstringslugRequiredstringexampleIdRequiredstring
Answer (200)
idstringnamestringinputKindstringOne oftext,record,arrayinputHashstringinputPreviewstringexpectedAnswersobjectnotestring or nullstatusstringOne ofproposed,approvedcreatedAtstringupdatedAtstringinputstring or object or arrayDynamic TypeSafe state: a string, object, or array, checked by state_schema.
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/examples/<exampleId>" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN"PATCH /v1/organisations/{org}/decisions/{slug}/examples/{exampleId}
Update an Example
Path
orgRequiredstringslugRequiredstringexampleIdRequiredstring
Body
nameOptionalstringinputOptionalstring or object or arrayDynamic TypeSafe state: a string, object, or array, checked by state_schema.expectedAnswersOptionalobjectnoteOptionalstring or nullstatusOptionalstringGenerated proposals become ordinary worked examples only after approval.One ofapprovedproposalIdOptionalstringAccepts an update from propose-updates: saves exactly what was proposed and records it.
Answer (200)
idstringnamestringinputKindstringOne oftext,record,arrayinputHashstringinputPreviewstringexpectedAnswersobjectnotestring or nullstatusstringOne ofproposed,approvedcreatedAtstringupdatedAtstringinputstring or object or arrayDynamic TypeSafe state: a string, object, or array, checked by state_schema.
curl -X PATCH "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/examples/<exampleId>" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN" \
-H "Content-Type: application/json" \
-d '{}'DELETE /v1/organisations/{org}/decisions/{slug}/examples/{exampleId}
Delete an Example
Path
orgRequiredstringslugRequiredstringexampleIdRequiredstring
Answer (204)
It answers with no body.
curl -X DELETE "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/examples/<exampleId>" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN"POST /v1/organisations/{org}/decisions/{slug}/examples/{exampleId}/add-to-test-set
Add an Example to Test sets
Path
orgRequiredstringslugRequiredstringexampleIdRequiredstring
Body
testSetIdsOptionalarray of stringEmpty or omitted selects the protected Core Test set.
Answer (200)
caseIdstringtestSetIdsarray of stringcreatedboolean
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/examples/<exampleId>/add-to-test-set" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN" \
-H "Content-Type: application/json" \
-d '{}'Test cases
GET /v1/organisations/{org}/decisions/{slug}/cases
List test cases
The Decision's cases; with full, each case whole (as GET /cases/{caseId}), in one response.
Path
orgRequiredstringslugRequiredstring
Query
testSetIdOptionalstringfullOptionalboolean
Answer (200)
It answers with no body.
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/cases" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN"POST /v1/organisations/{org}/decisions/{slug}/cases
Add a test case
Path
orgRequiredstringslugRequiredstring
Body
nameOptionalstring or nullinputRequiredstring or object or arrayDynamic TypeSafe state: a string, object, or array, checked by state_schema.expectedAnswersRequiredobjectnoteOptionalstring or nulltestSetIdsOptionalarray of stringEmpty or omitted adds the case to the protected Core Test set.
Answer (200)
idstringnamestring or nullinputKindstringOne oftext,record,arrayinputHashstringinputPreviewstringOne line of the state, for lists.expectedAnswersobjectnotestring or nulloriginstringWhere a case came from; synthetic cases are the authoring model's and start unconfirmed.One ofmanual,example,run,syntheticsourceRunIdstring or nullconfirmedbooleantestSetIdsarray of stringNamed Test sets that currently include this reusable case.createdAtstringupdatedAtstringinputstring or object or arrayThe full state, read from the payload store.
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/cases" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN" \
-H "Content-Type: application/json" \
-d '{"input":"<input>","expectedAnswers":{}}'POST /v1/organisations/{org}/decisions/{slug}/cases/import-examples
Import Examples as test cases
The draft's examples become cases, once each.
Path
orgRequiredstringslugRequiredstring
Answer (200)
createdarrayskippedintegerInputs whose state already existed as a case.invalidinteger
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/cases/import-examples" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN"POST /v1/organisations/{org}/decisions/{slug}/cases/propose-updates
Propose test case updates
Proposes an update for every case that no longer fits the draft; a proposal takes effect only once accepted. 402 FAIR_USE_REACHED when the Definition assistant is needed and can't be asked for now.
Path
orgRequiredstringslugRequiredstring
Answer (200)
kindstringOne ofcases,examplesproposalsarrayunresolvedarray of { "targetId": string, "name": string, "reason": string }modelstring or null
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/cases/propose-updates" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN"GET /v1/organisations/{org}/decisions/{slug}/cases/{caseId}
Get a test case
Path
orgRequiredstringslugRequiredstringcaseIdRequiredstring
Answer (200)
idstringnamestring or nullinputKindstringOne oftext,record,arrayinputHashstringinputPreviewstringOne line of the state, for lists.expectedAnswersobjectnotestring or nulloriginstringWhere a case came from; synthetic cases are the authoring model's and start unconfirmed.One ofmanual,example,run,syntheticsourceRunIdstring or nullconfirmedbooleantestSetIdsarray of stringNamed Test sets that currently include this reusable case.createdAtstringupdatedAtstringinputstring or object or arrayThe full state, read from the payload store.
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/cases/<caseId>" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN"PATCH /v1/organisations/{org}/decisions/{slug}/cases/{caseId}
Update a test case
Path
orgRequiredstringslugRequiredstringcaseIdRequiredstring
Body
nameOptionalstring or nullinputOptionalstring or object or arrayDynamic TypeSafe state: a string, object, or array, checked by state_schema.expectedAnswersOptionalobjectnoteOptionalstring or nullconfirmedOptionalbooleanproposalIdOptionalstringAccepts an update from propose-updates: saves exactly what was proposed, keeps a confirmed case confirmed and records that the assistant proposed it and the person accepted it.
Answer (200)
idstringnamestring or nullinputKindstringOne oftext,record,arrayinputHashstringinputPreviewstringOne line of the state, for lists.expectedAnswersobjectnotestring or nulloriginstringWhere a case came from; synthetic cases are the authoring model's and start unconfirmed.One ofmanual,example,run,syntheticsourceRunIdstring or nullconfirmedbooleantestSetIdsarray of stringNamed Test sets that currently include this reusable case.createdAtstringupdatedAtstringinputstring or object or arrayThe full state, read from the payload store.
curl -X PATCH "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/cases/<caseId>" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN" \
-H "Content-Type: application/json" \
-d '{}'DELETE /v1/organisations/{org}/decisions/{slug}/cases/{caseId}
Delete a test case
Path
orgRequiredstringslugRequiredstringcaseIdRequiredstring
Answer (204)
It answers with no body.
curl -X DELETE "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/cases/<caseId>" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN"POST /v1/organisations/{org}/decisions/{slug}/cases/generate
Generate test cases
Generates test cases with the Definition assistant; they are created unconfirmed. 402 FAIR_USE_REACHED when the assistant can't be asked for now.
Path
orgRequiredstringslugRequiredstring
Body
countRequiredintegermodeRequiredstringOne ofcover,boundary,variationscaseIdOptionalstringFor variations: the case to vary.testSetIdsOptionalarray of stringEmpty or omitted adds generated cases to the protected Core Test set.
Answer (200)
createdarrayskippedintegerInputs whose state already existed as a case.invalidinteger
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/cases/generate" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN" \
-H "Content-Type: application/json" \
-d '{"count":1,"mode":"cover"}'Test sets
GET /v1/organisations/{org}/decisions/{slug}/test-sets
List Test sets
Path
orgRequiredstringslugRequiredstring
Answer (200)
testSetsarray
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-sets" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN"POST /v1/organisations/{org}/decisions/{slug}/test-sets
Create a Test set
Path
orgRequiredstringslugRequiredstring
Body
nameRequiredstringdescriptionOptionalstring or null
Answer (200)
idstringnamestringdescriptionstring or nullisDefaultbooleanCore is the protected fallback used when no Test set is specified.caseCountintegerconfirmedCountintegercreatedAtstringupdatedAtstring
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-sets" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN" \
-H "Content-Type: application/json" \
-d '{"name":"Refunds"}'PATCH /v1/organisations/{org}/decisions/{slug}/test-sets/{testSetId}
Update a Test set
Path
orgRequiredstringslugRequiredstringtestSetIdRequiredstring
Body
nameRequiredstringdescriptionOptionalstring or null
Answer (200)
idstringnamestringdescriptionstring or nullisDefaultbooleanCore is the protected fallback used when no Test set is specified.caseCountintegerconfirmedCountintegercreatedAtstringupdatedAtstring
curl -X PATCH "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-sets/<testSetId>" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN" \
-H "Content-Type: application/json" \
-d '{"name":"Refunds"}'DELETE /v1/organisations/{org}/decisions/{slug}/test-sets/{testSetId}
Delete a Test set
Path
orgRequiredstringslugRequiredstringtestSetIdRequiredstring
Answer (204)
It answers with no body.
curl -X DELETE "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-sets/<testSetId>" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN"POST /v1/organisations/{org}/decisions/{slug}/test-sets/{testSetId}/cases
Add test cases to a Test set
Path
orgRequiredstringslugRequiredstringtestSetIdRequiredstring
Body
caseIdsRequiredarray of string
Answer (200)
testSetIdstringcaseIdsarray of string
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-sets/<testSetId>/cases" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN" \
-H "Content-Type: application/json" \
-d '{"caseIds":[]}'DELETE /v1/organisations/{org}/decisions/{slug}/test-sets/{testSetId}/cases/{caseId}
Take a test case out of a Test set
Path
orgRequiredstringslugRequiredstringtestSetIdRequiredstringcaseIdRequiredstring
Answer (204)
It answers with no body.
curl -X DELETE "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-sets/<testSetId>/cases/<caseId>" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN"Test runs
GET /v1/organisations/{org}/decisions/{slug}/test-runs
List Test runs
The Decision's last 50 Test runs; with recent, the newest that many whole as well.
Path
orgRequiredstringslugRequiredstring
Query
recentOptionalinteger
Answer (200)
testRunsarrayrecentarrayWithrecent=N(1 to 10): the newest N runs by when they started, whole, as GET …/test-runs/{testRunId} answers each.
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-runs" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN"POST /v1/organisations/{org}/decisions/{slug}/test-runs
Start a Test run
Runs the chosen Test set cases and records their raw results, without judging them; an Evaluation reads them. With X-TypeSafe-Key it runs on your own TypeSafe key.
Path
orgRequiredstringslugRequiredstring
Headers
X-TypeSafe-KeyOptionalstring
Body
versionNumberOptionalinteger or nullA published version to run, or null for the current draft.includeOptionalstringConfirmed cases only (default), or proposals as well.One ofconfirmed,allbatchSizeOptionalintegerCases per provider call, 1–10; 1 matches production. A Decision with per-item questions runs one case per call.expansionOptionalstringDraft runs only: how per-item (each) questions are sent.repeat(the default) repeats each question per item;sharedsends its wording once. Use it to evaluatesharedbefore it becomes the default.One ofrepeat,sharedtestSetIdsOptionalarray of stringTest sets whose deduplicated cases should run; defaults to Core.caseIdsOptionalarray of stringAn optional subset, used when running one case from a Test set.
Answer (200)
idstringevaluationIdstring or nullThe deterministic Evaluation derived from this run, when one exists.createdByintegerThe user who started this historical run.createdByNamestringversionIdstring or nullNull when the draft was run; see draftHash.versionNumberinteger or nulldraftHashstring or nullplanHashstringproviderstring or nullThe provider that answered, such as "typesafe"; null on runs recorded before this was kept.providerModelstring or nullThe exact model that answered; null on runs recorded before this was kept.testSetIdsarray of stringExact Test sets selected when this historical run started.testSetNamesarray of stringincludestringOne ofconfirmed,allbatchSizeintegerstatusstringOne ofcompleted,completed_with_errorscaseCountintegersucceededCountintegererrorCountintegerinputTokensintegerlatencyMsintegerstartedAtstringcompletedAtstringitemsarray
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-runs" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN" \
-H "Content-Type: application/json" \
-d '{}'GET /v1/organisations/{org}/decisions/{slug}/test-runs/{testRunId}
Get a Test run
Path
orgRequiredstringslugRequiredstringtestRunIdRequiredstring
Answer (200)
idstringevaluationIdstring or nullThe deterministic Evaluation derived from this run, when one exists.createdByintegerThe user who started this historical run.createdByNamestringversionIdstring or nullNull when the draft was run; see draftHash.versionNumberinteger or nulldraftHashstring or nullplanHashstringproviderstring or nullThe provider that answered, such as "typesafe"; null on runs recorded before this was kept.providerModelstring or nullThe exact model that answered; null on runs recorded before this was kept.testSetIdsarray of stringExact Test sets selected when this historical run started.testSetNamesarray of stringincludestringOne ofconfirmed,allbatchSizeintegerstatusstringOne ofcompleted,completed_with_errorscaseCountintegersucceededCountintegererrorCountintegerinputTokensintegerlatencyMsintegerstartedAtstringcompletedAtstringitemsarray
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/test-runs/<testRunId>" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN"Evaluations
GET /v1/organisations/{org}/decisions/{slug}/evaluations
List Evaluations
Path
orgRequiredstringslugRequiredstring
Answer (200)
evaluationsarray
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/evaluations" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN"POST /v1/organisations/{org}/decisions/{slug}/evaluations
Evaluate a Test run
Deterministically analyses one immutable Test run without another provider call.
Path
orgRequiredstringslugRequiredstring
Body
testRunIdRequiredstringThe immutable Test run to analyse. Evaluating never calls the provider again.
Answer (200)
idstringtestRunIdstring or nullThe immutable Test run analysed by this Evaluation; null only for historical Evaluations.versionIdstring or nullNull when the draft was evaluated; see draftHash.versionNumberinteger or nulldraftHashstring or nullplanHashstringbatchSizeintegerrepeatOfstring or nullThe evaluation this one repeated on the same cases and plan, if any.testSetIdsarray of stringExact Test sets selected when this historical evaluation was started.testSetNamesarray of stringstatusstringOne ofdone,errorerrorstring or nullcaseCountintegeragreedCountintegerperQuestionobjectbyOriginobjectinputTokensintegerlatencyMsintegerstartedAtstringcompletedAtstringcoverageobject or nullNull when no case returned answers, so the Decision's questions are unknown.coverage.questionsarray of stringEvery question the Decision answered in the Test run.coverage.checkedarray of stringQuestions at least one case sets an expectation for; see perQuestion for their agreement.coverage.uncheckedarray of stringQuestions no case checks: their answers were never compared.calibrationobject or nullNull when no case expects a yes/no or choice answer.calibration.statusstringnot_availablewhen the model gave no probability for each possible answer (only a single confidence, or nothing).One ofmeasured,not_availablecalibration.answersintegerLabelled yes/no and choice answers with a distribution: the answers measured.calibration.agreedintegerOf those, how many matched the expected answer.calibration.withoutProbabilitiesintegerLabelled yes/no and choice answers left out because the model gave no probability for each answer.calibration.briernumber or nullMean Brier score, 0 (certain and always right) to 1 (certain and always wrong); a yes/no answer scores (1 − p)² for the probability p it gave the expected answer.calibration.expectedCalibrationErrornumber or nullExpected calibration error over ten equal bands: the count-weighted gap between stated probability and how often the answers were right.calibration.roughbooleanFewer than 50 answers were measured: read the figures as rough.calibration.binsarrayTen equal bands of the probability each answer gave the answer it chose.itemsarray
curl -X POST "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/evaluations" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN" \
-H "Content-Type: application/json" \
-d '{"testRunId":"<testRunId>"}'GET /v1/organisations/{org}/decisions/{slug}/evaluations/{evaluationId}
Get an Evaluation
Path
orgRequiredstringslugRequiredstringevaluationIdRequiredstring
Answer (200)
idstringtestRunIdstring or nullThe immutable Test run analysed by this Evaluation; null only for historical Evaluations.versionIdstring or nullNull when the draft was evaluated; see draftHash.versionNumberinteger or nulldraftHashstring or nullplanHashstringbatchSizeintegerrepeatOfstring or nullThe evaluation this one repeated on the same cases and plan, if any.testSetIdsarray of stringExact Test sets selected when this historical evaluation was started.testSetNamesarray of stringstatusstringOne ofdone,errorerrorstring or nullcaseCountintegeragreedCountintegerperQuestionobjectbyOriginobjectinputTokensintegerlatencyMsintegerstartedAtstringcompletedAtstringcoverageobject or nullNull when no case returned answers, so the Decision's questions are unknown.coverage.questionsarray of stringEvery question the Decision answered in the Test run.coverage.checkedarray of stringQuestions at least one case sets an expectation for; see perQuestion for their agreement.coverage.uncheckedarray of stringQuestions no case checks: their answers were never compared.calibrationobject or nullNull when no case expects a yes/no or choice answer.calibration.statusstringnot_availablewhen the model gave no probability for each possible answer (only a single confidence, or nothing).One ofmeasured,not_availablecalibration.answersintegerLabelled yes/no and choice answers with a distribution: the answers measured.calibration.agreedintegerOf those, how many matched the expected answer.calibration.withoutProbabilitiesintegerLabelled yes/no and choice answers left out because the model gave no probability for each answer.calibration.briernumber or nullMean Brier score, 0 (certain and always right) to 1 (certain and always wrong); a yes/no answer scores (1 − p)² for the probability p it gave the expected answer.calibration.expectedCalibrationErrornumber or nullExpected calibration error over ten equal bands: the count-weighted gap between stated probability and how often the answers were right.calibration.roughbooleanFewer than 50 answers were measured: read the figures as rough.calibration.binsarrayTen equal bands of the probability each answer gave the answer it chose.itemsarray
curl "https://api.typefuse.dev/v1/organisations/acme/decisions/refunds/evaluations/<evaluationId>" \
-H "Authorization: Bearer $TYPEFUSE_TOKEN"