Skip to main content

Evals API

The Evals API does everything the console does with Agent evals: manage suites and cases, import and export them as JSON or YAML, start and cancel runs, read results and their files, and set an agent's publish gate.

Access​

All paths start with https://<region host>/api/projects/{project_id}/evals. See Hosts and data regions.

RequirementIf it is not met
The caller is a member of the project. Reads (GET) need view access, every other method needs edit access, the same as for agents.403
The organization's plan includes Agent evals.403 with code feature_not_in_plan
Evals are switched on for the organization. See When evals are switched off.403 with code evals_disabled

Authentication. Send a signed-in project member's access token as Authorization: Bearer <token>, the same token the console uses after you sign in at app.voisx.ai. The access rules above are checked against that member. Project API keys (vois_pk_..., see Authentication) are not accepted on these routes and return 403. In the examples, $TOKEN is that access token.

The table applies to every endpoint on this page, so any of them can return these 403 codes even where its responses below do not list them.

Every ID in a path or body (suite, case, run, result, agent) must belong to the project in the path. An ID from another project returns 404, never 403. A case ID must also belong to the suite in the path.

Endpoints​

MethodPathWhat it does
GET/suites?agent_id=List suites, with case_count and last_run
POST/suitesCreate a suite, with its cases
POST/suites/importImport a suite from JSON or YAML
GET/suites/{suite_id}A suite with its cases
GET/suites/{suite_id}/export?format=yaml|jsonExport a suite as a document
PATCH/suites/{suite_id}Change a suite's name, description, agent or defaults
DELETE/suites/{suite_id}Delete a suite with its runs and their files
POST/suites/{suite_id}/casesAdd a case
PATCH/suites/{suite_id}/cases/{case_id}Change a case
DELETE/suites/{suite_id}/cases/{case_id}Delete a case. Past results keep it as case_id: null.
POST/suites/{suite_id}/cases/reorderReorder cases. Send the full list: {"case_ids": [...]}.
POST/suites/{suite_id}/cases/from-sessionAdd a conversation as a case
POST/suites/{suite_id}/runsStart a run
GET/runs?suite_id=&agent_id=&status=&trigger=&limit=50&before_id=List runs, newest first
GET/runs/{run_id}A run
POST/runs/{run_id}/cancelCancel a run. Does nothing to a finished run.
GET/runs/{run_id}/results?status=&case_id=A run's results
GET/results/{result_id}One result
GET/results/{result_id}/artifacts/{name}Download a result's file
GET/scorersList checks
GET/agents/{agent_id}/eval-on-publishAn agent's publish gate, or null
PUT/agents/{agent_id}/eval-on-publishSet the publish gate

Case fields​

A suite's defaults and each case use the same fields. Every field is optional on its own, but each case needs a target and, except on replay, a persona once the suite defaults are applied. A field the case sets replaces the suite's value for that field. Unknown fields are refused with 422, so a typo fails when you save.

FieldTypeDescription
targetstringchat, phone, widget or replay
carrierstringplivo, twilio, tryvox or sip. Phone only. Required on phone unless matrix.carriers is set.
personaobjectThe simulated caller: {"type": "scripted", "prompt", "language", "steps": [...]} or {"type": "llm", "prompt", "language", "max_turns"}. See Personas. Not allowed on replay.
goalstringWhat the caller wants. The judge scores it as task success.
checksarray[{"name": "latency", "params": {...}}]. Names come from GET /scorers.
rubricarray[{"criterion", "weight", "threshold"}]. weight above 0, default 1. threshold 0 to 1, default 0.7.
expected_extractionobjectField name to {"value", "match"}. match is exact, normalized (default) or judge.
expected_toolsarray[{"name", "args_match", "order"}]. args_match checks only the keys you give. order is the 1-based position, unique.
forbidden_toolsarrayTool names the agent must not call. A tool cannot be both expected and forbidden.
tools_modestringmock (default) or live
tool_mocksobjectTool name to the answer the agent gets in mock mode. Default {"ok": true}.
ragobject{"expect_docs": [...], "grounded": true}. Document IDs or file names.
memoryobject{"seed_session": [{"role", "text"}], "recall_assertions": [...]}. Not allowed on replay.
replayobject{"session_id"}. Replay only. Defaults to the case's source conversation.
matrixobject{"voice_types": [], "models": [], "carriers": []}. At most 24 combinations. See Matrix.
limitsobject{"max_call_s": 10..900, "max_cost_usd": up to 20}. Defaults 240 and 1.0.
tagsarrayYour own labels.

A case also has name (required, up to 200 characters), position, and source (manual or session).

Create a suite​

POST/api/projects/{project_id}/evals/suites

Creates a suite and, optionally, its cases in one request.

Body​

namestringrequired
Up to 200 characters.
descriptionstringoptional
Up to 4,000 characters.
agent_idintegeroptional
Pins the suite to an agent. Runs use this agent unless you pass another.
defaultsobjectoptional
The suite's default case fields.
casesarrayoptional
Cases: each a name plus case fields. Up to 200.

Responses​

201
The suite, as SuiteOut.
403
feature_not_in_plan, evals_disabled, or no edit access.
404
The project or agent was not found.
422
eval_spec_invalid: a field is wrong, a case does not resolve, or a check does not exist or does not support the case's target. errors lists each problem.

Import a suite​

POST/api/projects/{project_id}/evals/suites/import

Creates a suite from a JSON or YAML document, the same shape GET /suites/{suite_id}/export writes: name, description, agent_id, defaults and cases. Send the document as the body with Content-Type application/json, application/yaml or text/yaml, or as a multipart file. Up to 1 MB.

Responses​

201
The suite, as SuiteOut.
422
eval_spec_invalid: the document is not valid. errors lists each problem.

Add a conversation as a case​

POST/api/projects/{project_id}/evals/suites/{suite_id}/cases/from-session

Turns a stored conversation into a case, added at the end of the suite.

Body​

session_idstringrequired
The conversation. It must belong to the project.
namestringoptional
The case name.
modestringoptional
replay (default) replays the caller's turns as text. llm writes an AI caller persona from the conversation's summary, intent and language, with the agent's goal.
include_extractionbooleanoptional
Default true. Copies what the agent extracted in that conversation into expected_extraction, with match: exact.

Responses​

201
The case, as CaseOut, with source: "session".
404
The suite or conversation was not found in this project.

Start a run​

POST/api/projects/{project_id}/evals/suites/{suite_id}/runs

Queues a run of the suite against one agent. The run happens in the background: poll GET /runs/{run_id} until its status is final.

Body​

agent_idintegeroptional
The agent to test. Defaults to the suite's agent. Required when the suite has none.
case_idsarrayoptional
Only these cases. Defaults to every case.
matrixobjectoptional
One matrix for every case, replacing each case's own.

Responses​

202
The run, as RunOut, with status: "queued".
404
The suite or agent was not found in this project.
409
eval_run_active: this suite already has a queued or running run for this agent.
422
eval_run_too_large (more than 500 results), or no agent to test.

Run status​

statusMeaning
queuedWaiting to start.
runningIn progress.
passedEvery counted result passed, or, on a publish run, pass_rate reached min_pass_rate.
failedNot enough results passed.
cancelledCancelled by a user, or evals were switched off during the run.
errorThe run could not be evaluated, for example the agent was deleted or every result had an error or was skipped. error says why.

trigger is manual for runs you start, and publish for publish gate runs.

List runs​

GET/api/projects/{project_id}/evals/runs

Lists runs, newest first.

Query parameters​

suite_idintegeroptional
Only this suite.
agent_idintegeroptional
Only this agent.
statusstringoptional
Only runs with this status.
triggerstringoptional
manual or publish.
limitintegeroptional
Default 50.
before_idintegeroptional
For the next page, pass next_before_id from the previous response.

Responses​

200
{"items": RunOut[], "next_before_id": int | null}. next_before_id is null on the last page.

Get a run's results​

GET/api/projects/{project_id}/evals/runs/{run_id}/results

One result per case and matrix cell.

Query parameters​

statusstringoptional
queued, running, passed, failed, cancelled, error or skipped.
case_idintegeroptional
Only this case.

Result fields​

FieldDescription
cell{voice_type, model, carrier}. null means the agent's own setting.
statusThe result's status. skipped means the cell cannot run here, for example the carrier is switched off.
passWhether the result passed.
scoreOverall score, 0 to 1.
scoresOne entry per check: name, score, pass and detail.
judgeThe judge's model and one entry per criterion: criterion, score, reason and evidence_turns (0-based transcript turns).
artifactsWhich files the result has. Download them with the artifacts route.
session_idThe simulated conversation's session ID. It starts with eval-.
cost_usdThe conversation's own cost plus the Agent evals usage for this result.
timingsSeconds spent running, scoring and judging.
errorWhy the result is error or skipped.

Download a result's file​

GET/api/projects/{project_id}/evals/results/{result_id}/artifacts/{name}

Redirects to a download link that works for 10 minutes. If a download link cannot be made, the response is the file itself instead of a redirect, so your client must handle both.

Path parameters​

namestringrequired
nameFile
transcriptThe conversation, with personal data masked
timelineVoice only: turn timings
audioVoice only: the recording, stereo WAV (left the caller, right the agent)
judgeThe judge's full request and answer, masked
toolsTool calls with their arguments and results

Responses​

302
Follow the Location header to download the file.
200
The file itself, when a download link cannot be made.
404
The result or file was not found.

List checks​

GET/api/projects/{project_id}/evals/scorers

Lists every check you can use in checks, with the targets it supports and its parameters. kind is deterministic for rule-based checks and judge for checks the judge scores. An empty targets list means every target.

Responses​

200
[{name, kind, targets, description, params_schema}]

Set the publish gate​

PUT/api/projects/{project_id}/evals/agents/{agent_id}/eval-on-publish

Makes a suite the agent's publish gate. Send null to remove the gate.

Body​

suite_idintegerrequired
A suite in the same project.
min_pass_ratenumberrequired
0 to 1. The publish run must reach this pass rate.

Responses​

200
The stored gate.
403
feature_not_in_plan, evals_disabled, or no edit access.
404
The agent was not found in this project.
422
The suite is not in this project.

What publishing returns​

When the gate is set and any change moves the agent from Draft to Live (an agent update, a bulk status change, or creating an agent from a template as Live), the update is saved but the agent stays Draft, and the response is 202:

{"code": "eval_gate_pending", "run_id": 81, "message": "..."}

The agent goes Live when run 81 passes, the agent has not changed since the run started, and it is still Draft. Otherwise it stays Draft and the audit log records agent.publish_blocked_by_eval with a reason: failed, error, cancelled, stale_version (the agent changed during the run) or suite_missing.

ResponseWhen
202 eval_gate_pendingA publish run started.
409 eval_run_activeA publish run for this agent is already going.
409 eval_gate_suite_missingThe gate's suite no longer exists. Point the gate at another suite or remove it.
"eval_gate": "skipped" in a normal responseEvals are not in the plan or are switched off. The agent went Live at once.

Errors​

Errors use the platform's error shape with an object in detail: {"detail": {"code": "...", "message": "..."}}.

CodeStatusMeaning
feature_not_in_plan403The plan does not include Agent evals.
evals_disabled403Evals are switched off for the organization.
eval_spec_invalid422A suite or case is not valid. errors lists the problems.
eval_run_active409A run of this suite for this agent is already queued or running.
eval_run_too_large422The run would have more than 500 results.
eval_suite_in_use409The suite is an agent's publish gate, so it cannot be deleted.
eval_gate_suite_missing409Publishing is blocked because the gate's suite no longer exists.
eval_gate_pending202Publishing waits for a gate run.
Was this page helpful?