Skip to main content

Evals

These endpoints need a Keycloak access token. Reading needs any role on the suite's project. Creating, changing and running need the developer or admin role. See Evals for how suites work.

Cases and checks​

A case is an object:

FieldRequiredWhat it is
inputYesThe user's message.
checksYesOne or more checks. All must pass.
historyNoEarlier turns, each with role (user or assistant) and content.
idNoOrchestrate assigns one. Keep it when you edit a case, so runs can be compared.

The checks:

typeFieldsPasses when
containsvalueThe reply contains value, ignoring case.
not_containsvalueThe reply does not contain value, ignoring case.
calls_tooltool, optional arguments_containThe agent called tool. With arguments_contain, at least one call's arguments contain each given text, by argument name.
not_calls_tooltoolThe agent never called tool.
judgerubric, optional modelThe judge model decides the reply meets the rubric. Without model, the suite's judge_model is used.

Create a suite​

POST/eval-suites

Body​

project_idstringrequired
The project.
namestringrequired
The suite's name.
datasetarrayrequired
At least one case.
agent_namestringoptional
The agent the suite tests.
pass_thresholdnumberoptional
The share of cases that must pass, from 0 to 1. Default 0.8.
judge_modelstringoptional
The model reference of a chat model in the catalog, for judge checks.

Response fields​

idstringoptional
The suite id.
datasetarrayoptional
The cases as stored, each with its id.
run_blocked_reasonstring or nulloptional
Why the suite cannot run yet, for example judge checks with no judge model. null when it can run.

Responses​

200
Created.
400
A case is invalid, or the judge model is not a chat model in the catalog. detail names the problem.

Read and change suites​

GET/eval-suites

The project's suites, each with its ten most recent runs. Query parameter: project_id.

GET/eval-suites/{suite_id}

One suite.

PATCH/eval-suites/{suite_id}

Changes the suite's settings: name, pass_threshold, agent_name or judge_model. Fields you leave out are not changed. Send null or "" to clear agent_name or judge_model. The dataset is changed through the case endpoints below.

DELETE/eval-suites/{suite_id}

Deletes the suite and its runs. Returns 409 while a run is in progress or while the project's release gate requires the suite, and 503 if Orchestrate cannot check for a running run.

Change cases​

POST/eval-suites/{suite_id}/cases

Adds one case. If the suite already has a case with the same input and history, that case is updated instead and keeps its id. The response says added or updated, with the case_id.

PUT/eval-suites/{suite_id}/cases/{case_id}

Replaces one case's input, checks and history. Its id stays the same.

DELETE/eval-suites/{suite_id}/cases/{case_id}

Removes one case. The last case of a suite cannot be removed.

Past runs keep the cases as they were when they ran.

Run a suite​

POST/eval-suites/{suite_id}/runs

Runs every case against one agent version and returns at once with status pending. Poll GET /eval-runs/{run_id}. Write tools are never executed during an eval run.

Body​

target_typestringrequired
agent.
target_idstringrequired
The agent version's definition_id. It must be in the suite's project and have one step.
GET/eval-suites/{suite_id}/agent-versions

The versions of the suite's agent, newest first, each with runnable and, if not, the reason.

Responses​

200
The run started.
400
Unknown target, an agent with more than one step, or a judge check with no judge model.

Get a run​

GET/eval-runs/{run_id}

A run's live progress and results. Reading a finished run also saves its result. GET /eval-suites/{suite_id}/runs lists a suite's runs as summaries.

Response fields​

stagestringoptional
running, passed, failed or errored. errored means no case could be graded.
cases_completedintegeroptional
Cases finished so far.
total_casesintegeroptional
Cases in the run.
scorenumber or nulloptional
Passed cases divided by all cases.
resultsarrayoptional
One entry per case: case_id, input, status (passed, failed or error), reply, checks with passed and detail, tool_calls, latency_ms, tokens, judge_tokens and model.
totalsobjectoptional
agent_tokens and judge_tokens, each with prompt and completion, or null when no case reported them.
runobjectoptional
The run record: suite, agent name and version, who started it and when.

Compare two runs​

GET/eval-runs/{run_id}/compare

What changed between a finished run and an earlier one of the same suite.

Query parameters​

baselinestringoptional
The run to compare with. Defaults to the suite's previous graded run.

Response fields​

baselineobject or nulloptional
The run compared with. null for a suite's first graded run, with a message saying so.
comparison.cases[]objectoptional
Each case's change: got_worse, fixed, still_passing, still_failing, errored, new or removed, with both statuses and whether its checks or input changed.
comparison.cases[].flakybooleanoptional
true when the case has flipped both ways across recent runs of the same version, so it is not measuring the agent reliably.
comparison.countsobjectoptional
How many cases fell into each kind of change.
comparison.scoreobjectoptional
current, baseline and change. latency_ms and tokens have the same shape.

Responses​

200
The comparison.
400
The baseline is not a run of this suite, or is the same run.
409
The run or the baseline has not finished.

Code agent versions​

POST /eval-suites/{suite_id}/runs also accepts {"target_type": "code_agent_version", "target_id": "<version id>"} to run a suite against a code agent version. The version needs a ready deployment, or the call answers 409. See Code agents.

Online evals and feedback​

See Online evals and feedback for how these work.

Method and pathWhoWhat it does
GET /projects/{project_id}/agents/{agent_name}/online-evalsviewerThe agent's online eval settings.
PUT /projects/{project_id}/agents/{agent_name}/online-evalsdeveloper or admin{"enabled", "sample_rate", "judge_model", "checks"}. sample_rate is above 0 and at most 1. Checks are judge, contains or not_contains. Audited.
GET /projects/{project_id}/agents/{agent_name}/online-evals/resultsviewerGraded samples, newest first. Filter by status and version; page with limit and before.
GET /projects/{project_id}/agents/{agent_name}/qualityviewerPer version and per day: production runs, feedback and online verdicts, over days (default 30).
GET /projects/{project_id}/feedbackviewerFeedback, newest first, each with the run it is about. Filter by agent_name, version, rating and with_comment.
GET /projects/{project_id}/production-runsviewerOne agent's completed production runs, with feedback and online verdicts. Filter by version, feedback (up, down, any, none), online_eval (passed, failed, error), days and q.
POST /projects/{project_id}/production-runs/eval-casesdeveloper or adminSave runs as eval cases: agent_name, run_ids, checks, reference_answer, and suite_id or new_suite. Audited.

People rate answers through the Agent API, the hosted chat and channels.

Was this page helpful?