Evals
These endpoints need a Keycloak access token. Reading needs any role on the suite's project. Creating, changing and running need the developer or admin role. See Evals for how suites work.
Cases and checks
A case is an object:
| Field | Required | What it is |
|---|---|---|
input | Yes | The user's message. |
checks | Yes | One or more checks. All must pass. |
history | No | Earlier turns, each with role (user or assistant) and content. |
id | No | Orchestrate assigns one. Keep it when you edit a case, so runs can be compared. |
The checks:
type | Fields | Passes when |
|---|---|---|
contains | value | The reply contains value, ignoring case. |
not_contains | value | The reply does not contain value, ignoring case. |
calls_tool | tool, optional arguments_contain | The agent called tool. With arguments_contain, at least one call's arguments contain each given text, by argument name. |
not_calls_tool | tool | The agent never called tool. |
judge | rubric, optional model | The judge model decides the reply meets the rubric. Without model, the suite's judge_model is used. |
Create a suite
/eval-suitesBody
project_idstringrequirednamestringrequireddatasetarrayrequiredagent_namestringoptionalpass_thresholdnumberoptionaljudge_modelstringoptionalResponse fields
idstringoptionaldatasetarrayoptionalid.run_blocked_reasonstring or nulloptionalnull when it can run.Responses
detail names the problem.Read and change suites
/eval-suitesThe project's suites, each with its ten most recent runs. Query parameter: project_id.
/eval-suites/{suite_id}One suite.
/eval-suites/{suite_id}Changes the suite's settings: name, pass_threshold, agent_name or judge_model. Fields you leave out are not changed. Send null or "" to clear agent_name or judge_model. The dataset is changed through the case endpoints below.
/eval-suites/{suite_id}Deletes the suite and its runs. Returns 409 while a run is in progress or while the project's release gate requires the suite, and 503 if Orchestrate cannot check for a running run.
Change cases
/eval-suites/{suite_id}/casesAdds one case. If the suite already has a case with the same input and history, that case is updated instead and keeps its id. The response says added or updated, with the case_id.
/eval-suites/{suite_id}/cases/{case_id}Replaces one case's input, checks and history. Its id stays the same.
/eval-suites/{suite_id}/cases/{case_id}Removes one case. The last case of a suite cannot be removed.
Past runs keep the cases as they were when they ran.
Run a suite
/eval-suites/{suite_id}/runsRuns every case against one agent version and returns at once with status pending. Poll GET /eval-runs/{run_id}. Write tools are never executed during an eval run.
Body
target_typestringrequiredagent.target_idstringrequireddefinition_id. It must be in the suite's project and have one step./eval-suites/{suite_id}/agent-versionsThe versions of the suite's agent, newest first, each with runnable and, if not, the reason.
Responses
Get a run
/eval-runs/{run_id}A run's live progress and results. Reading a finished run also saves its result. GET /eval-suites/{suite_id}/runs lists a suite's runs as summaries.
Response fields
stagestringoptionalrunning, passed, failed or errored. errored means no case could be graded.cases_completedintegeroptionaltotal_casesintegeroptionalscorenumber or nulloptionalresultsarrayoptionalcase_id, input, status (passed, failed or error), reply, checks with passed and detail, tool_calls, latency_ms, tokens, judge_tokens and model.totalsobjectoptionalagent_tokens and judge_tokens, each with prompt and completion, or null when no case reported them.runobjectoptionalCompare two runs
/eval-runs/{run_id}/compareWhat changed between a finished run and an earlier one of the same suite.
Query parameters
baselinestringoptionalResponse fields
baselineobject or nulloptionalnull for a suite's first graded run, with a message saying so.comparison.cases[]objectoptionalchange: got_worse, fixed, still_passing, still_failing, errored, new or removed, with both statuses and whether its checks or input changed.comparison.cases[].flakybooleanoptionaltrue when the case has flipped both ways across recent runs of the same version, so it is not measuring the agent reliably.comparison.countsobjectoptionalcomparison.scoreobjectoptionalcurrent, baseline and change. latency_ms and tokens have the same shape.Responses
Code agent versions
POST /eval-suites/{suite_id}/runs also accepts {"target_type": "code_agent_version", "target_id": "<version id>"} to run a suite against a code agent version. The version needs a ready deployment, or the call answers 409. See Code agents.
Online evals and feedback
See Online evals and feedback for how these work.
| Method and path | Who | What it does |
|---|---|---|
GET /projects/{project_id}/agents/{agent_name}/online-evals | viewer | The agent's online eval settings. |
PUT /projects/{project_id}/agents/{agent_name}/online-evals | developer or admin | {"enabled", "sample_rate", "judge_model", "checks"}. sample_rate is above 0 and at most 1. Checks are judge, contains or not_contains. Audited. |
GET /projects/{project_id}/agents/{agent_name}/online-evals/results | viewer | Graded samples, newest first. Filter by status and version; page with limit and before. |
GET /projects/{project_id}/agents/{agent_name}/quality | viewer | Per version and per day: production runs, feedback and online verdicts, over days (default 30). |
GET /projects/{project_id}/feedback | viewer | Feedback, newest first, each with the run it is about. Filter by agent_name, version, rating and with_comment. |
GET /projects/{project_id}/production-runs | viewer | One agent's completed production runs, with feedback and online verdicts. Filter by version, feedback (up, down, any, none), online_eval (passed, failed, error), days and q. |
POST /projects/{project_id}/production-runs/eval-cases | developer or admin | Save runs as eval cases: agent_name, run_ids, checks, reference_answer, and suite_id or new_suite. Audited. |
People rate answers through the Agent API, the hosted chat and channels.