Evals API
The Evals API does everything the console does with Agent evals: manage suites and cases, import and export them as JSON or YAML, start and cancel runs, read results and their files, and set an agent's publish gate.
Access
All paths start with https://<region host>/api/projects/{project_id}/evals. See Hosts and data regions.
| Requirement | If it is not met |
|---|---|
The caller is a member of the project. Reads (GET) need view access, every other method needs edit access, the same as for agents. | 403 |
| The organization's plan includes Agent evals. | 403 with code feature_not_in_plan |
| Evals are switched on for the organization. See When evals are switched off. | 403 with code evals_disabled |
Authentication. Send a signed-in project member's access token as Authorization: Bearer <token>, the same token the console uses after you sign in at app.voisx.ai. The access rules above are checked against that member. Project API keys (vois_pk_..., see Authentication) are not accepted on these routes and return 403. In the examples, $TOKEN is that access token.
The table applies to every endpoint on this page, so any of them can return these 403 codes even where its responses below do not list them.
Every ID in a path or body (suite, case, run, result, agent) must belong to the project in the path. An ID from another project returns 404, never 403. A case ID must also belong to the suite in the path.
Endpoints
| Method | Path | What it does |
|---|---|---|
GET | /suites?agent_id= | List suites, with case_count and last_run |
POST | /suites | Create a suite, with its cases |
POST | /suites/import | Import a suite from JSON or YAML |
GET | /suites/{suite_id} | A suite with its cases |
GET | /suites/{suite_id}/export?format=yaml|json | Export a suite as a document |
PATCH | /suites/{suite_id} | Change a suite's name, description, agent or defaults |
DELETE | /suites/{suite_id} | Delete a suite with its runs and their files |
POST | /suites/{suite_id}/cases | Add a case |
PATCH | /suites/{suite_id}/cases/{case_id} | Change a case |
DELETE | /suites/{suite_id}/cases/{case_id} | Delete a case. Past results keep it as case_id: null. |
POST | /suites/{suite_id}/cases/reorder | Reorder cases. Send the full list: {"case_ids": [...]}. |
POST | /suites/{suite_id}/cases/from-session | Add a conversation as a case |
POST | /suites/{suite_id}/runs | Start a run |
GET | /runs?suite_id=&agent_id=&status=&trigger=&limit=50&before_id= | List runs, newest first |
GET | /runs/{run_id} | A run |
POST | /runs/{run_id}/cancel | Cancel a run. Does nothing to a finished run. |
GET | /runs/{run_id}/results?status=&case_id= | A run's results |
GET | /results/{result_id} | One result |
GET | /results/{result_id}/artifacts/{name} | Download a result's file |
GET | /scorers | List checks |
GET | /agents/{agent_id}/eval-on-publish | An agent's publish gate, or null |
PUT | /agents/{agent_id}/eval-on-publish | Set the publish gate |
Case fields
A suite's defaults and each case use the same fields. Every field is optional on its own, but each case needs a target and, except on replay, a persona once the suite defaults are applied. A field the case sets replaces the suite's value for that field. Unknown fields are refused with 422, so a typo fails when you save.
| Field | Type | Description |
|---|---|---|
target | string | chat, phone, widget or replay |
carrier | string | plivo, twilio, tryvox or sip. Phone only. Required on phone unless matrix.carriers is set. |
persona | object | The simulated caller: {"type": "scripted", "prompt", "language", "steps": [...]} or {"type": "llm", "prompt", "language", "max_turns"}. See Personas. Not allowed on replay. |
goal | string | What the caller wants. The judge scores it as task success. |
checks | array | [{"name": "latency", "params": {...}}]. Names come from GET /scorers. |
rubric | array | [{"criterion", "weight", "threshold"}]. weight above 0, default 1. threshold 0 to 1, default 0.7. |
expected_extraction | object | Field name to {"value", "match"}. match is exact, normalized (default) or judge. |
expected_tools | array | [{"name", "args_match", "order"}]. args_match checks only the keys you give. order is the 1-based position, unique. |
forbidden_tools | array | Tool names the agent must not call. A tool cannot be both expected and forbidden. |
tools_mode | string | mock (default) or live |
tool_mocks | object | Tool name to the answer the agent gets in mock mode. Default {"ok": true}. |
rag | object | {"expect_docs": [...], "grounded": true}. Document IDs or file names. |
memory | object | {"seed_session": [{"role", "text"}], "recall_assertions": [...]}. Not allowed on replay. |
replay | object | {"session_id"}. Replay only. Defaults to the case's source conversation. |
matrix | object | {"voice_types": [], "models": [], "carriers": []}. At most 24 combinations. See Matrix. |
limits | object | {"max_call_s": 10..900, "max_cost_usd": up to 20}. Defaults 240 and 1.0. |
tags | array | Your own labels. |
A case also has name (required, up to 200 characters), position, and source (manual or session).
Create a suite
/api/projects/{project_id}/evals/suitesCreates a suite and, optionally, its cases in one request.
Body
namestringrequireddescriptionstringoptionalagent_idintegeroptionaldefaultsobjectoptionalcasesarrayoptionalname plus case fields. Up to 200.Responses
SuiteOut.feature_not_in_plan, evals_disabled, or no edit access.eval_spec_invalid: a field is wrong, a case does not resolve, or a check does not exist or does not support the case's target. errors lists each problem.Import a suite
/api/projects/{project_id}/evals/suites/importCreates a suite from a JSON or YAML document, the same shape GET /suites/{suite_id}/export writes: name, description, agent_id, defaults and cases. Send the document as the body with Content-Type application/json, application/yaml or text/yaml, or as a multipart file. Up to 1 MB.
Responses
SuiteOut.eval_spec_invalid: the document is not valid. errors lists each problem.Add a conversation as a case
/api/projects/{project_id}/evals/suites/{suite_id}/cases/from-sessionTurns a stored conversation into a case, added at the end of the suite.
Body
session_idstringrequirednamestringoptionalmodestringoptionalreplay (default) replays the caller's turns as text. llm writes an AI caller persona from the conversation's summary, intent and language, with the agent's goal.include_extractionbooleanoptionaltrue. Copies what the agent extracted in that conversation into expected_extraction, with match: exact.Responses
CaseOut, with source: "session".Start a run
/api/projects/{project_id}/evals/suites/{suite_id}/runsQueues a run of the suite against one agent. The run happens in the background: poll GET /runs/{run_id} until its status is final.
Body
agent_idintegeroptionalcase_idsarrayoptionalmatrixobjectoptionalResponses
RunOut, with status: "queued".eval_run_active: this suite already has a queued or running run for this agent.eval_run_too_large (more than 500 results), or no agent to test.Run status
status | Meaning |
|---|---|
queued | Waiting to start. |
running | In progress. |
passed | Every counted result passed, or, on a publish run, pass_rate reached min_pass_rate. |
failed | Not enough results passed. |
cancelled | Cancelled by a user, or evals were switched off during the run. |
error | The run could not be evaluated, for example the agent was deleted or every result had an error or was skipped. error says why. |
trigger is manual for runs you start, and publish for publish gate runs.
List runs
/api/projects/{project_id}/evals/runsLists runs, newest first.
Query parameters
suite_idintegeroptionalagent_idintegeroptionalstatusstringoptionaltriggerstringoptionalmanual or publish.limitintegeroptionalbefore_idintegeroptionalnext_before_id from the previous response.Responses
{"items": RunOut[], "next_before_id": int | null}. next_before_id is null on the last page.Get a run's results
/api/projects/{project_id}/evals/runs/{run_id}/resultsOne result per case and matrix cell.
Query parameters
statusstringoptionalqueued, running, passed, failed, cancelled, error or skipped.case_idintegeroptionalResult fields
| Field | Description |
|---|---|
cell | {voice_type, model, carrier}. null means the agent's own setting. |
status | The result's status. skipped means the cell cannot run here, for example the carrier is switched off. |
pass | Whether the result passed. |
score | Overall score, 0 to 1. |
scores | One entry per check: name, score, pass and detail. |
judge | The judge's model and one entry per criterion: criterion, score, reason and evidence_turns (0-based transcript turns). |
artifacts | Which files the result has. Download them with the artifacts route. |
session_id | The simulated conversation's session ID. It starts with eval-. |
cost_usd | The conversation's own cost plus the Agent evals usage for this result. |
timings | Seconds spent running, scoring and judging. |
error | Why the result is error or skipped. |
Download a result's file
/api/projects/{project_id}/evals/results/{result_id}/artifacts/{name}Redirects to a download link that works for 10 minutes. If a download link cannot be made, the response is the file itself instead of a redirect, so your client must handle both.
Path parameters
namestringrequiredname | File |
|---|---|
transcript | The conversation, with personal data masked |
timeline | Voice only: turn timings |
audio | Voice only: the recording, stereo WAV (left the caller, right the agent) |
judge | The judge's full request and answer, masked |
tools | Tool calls with their arguments and results |
Responses
Location header to download the file.List checks
/api/projects/{project_id}/evals/scorersLists every check you can use in checks, with the targets it supports and its parameters. kind is deterministic for rule-based checks and judge for checks the judge scores. An empty targets list means every target.
Responses
[{name, kind, targets, description, params_schema}]Set the publish gate
/api/projects/{project_id}/evals/agents/{agent_id}/eval-on-publishMakes a suite the agent's publish gate. Send null to remove the gate.
Body
suite_idintegerrequiredmin_pass_ratenumberrequiredResponses
feature_not_in_plan, evals_disabled, or no edit access.What publishing returns
When the gate is set and any change moves the agent from Draft to Live (an agent update, a bulk status change, or creating an agent from a template as Live), the update is saved but the agent stays Draft, and the response is 202:
{"code": "eval_gate_pending", "run_id": 81, "message": "..."}
The agent goes Live when run 81 passes, the agent has not changed since the run started, and it is still Draft. Otherwise it stays Draft and the audit log records agent.publish_blocked_by_eval with a reason: failed, error, cancelled, stale_version (the agent changed during the run) or suite_missing.
| Response | When |
|---|---|
202 eval_gate_pending | A publish run started. |
409 eval_run_active | A publish run for this agent is already going. |
409 eval_gate_suite_missing | The gate's suite no longer exists. Point the gate at another suite or remove it. |
"eval_gate": "skipped" in a normal response | Evals are not in the plan or are switched off. The agent went Live at once. |
Errors
Errors use the platform's error shape with an object in detail: {"detail": {"code": "...", "message": "..."}}.
| Code | Status | Meaning |
|---|---|---|
feature_not_in_plan | 403 | The plan does not include Agent evals. |
evals_disabled | 403 | Evals are switched off for the organization. |
eval_spec_invalid | 422 | A suite or case is not valid. errors lists the problems. |
eval_run_active | 409 | A run of this suite for this agent is already queued or running. |
eval_run_too_large | 422 | The run would have more than 500 results. |
eval_suite_in_use | 409 | The suite is an agent's publish gate, so it cannot be deleted. |
eval_gate_suite_missing | 409 | Publishing is blocked because the gate's suite no longer exists. |
eval_gate_pending | 202 | Publishing waits for a gate run. |