Evals
An eval suite is a set of test cases for one agent. Each case sends a message to a specific agent version and checks what comes back: the reply and the tool calls behind it. Run a suite whenever you change an agent, and compare the result with the last run.
A project can require a passing eval before a new version goes live. See Live versions and the release gate.
To see how an agent does after it goes live, use online evals and feedback: graded samples of real answers, thumbs from the people using it, and real runs saved as cases.
You need the developer or admin role on the project to create, change or run suites. Viewers can read them.
Create a suite
- Open Evals and choose New suite.
- Name it, pick the agent it tests, and write its first case.
- Choose Create suite.
In the suite's Settings you can change:
- the agent it tests. A suite tests one agent, by name, so it works for every version.
- the pass threshold: the share of cases that must pass for the run to pass. The default is 80%.
- the judge model, for judge checks. See Judge checks.
Write cases
A case has:
- a Message: what the user says
- an optional History: earlier user and assistant turns, so you can test the middle of a conversation
- one or more Checks. A case passes only if every check passes.
| Check | Passes when |
|---|---|
| Reply contains | The reply contains this text, ignoring case. |
| Reply doesn't contain | The text appears nowhere in the reply, ignoring case. |
| Calls tool | The agent called this tool. You can also require arguments: each named argument must contain the given text. |
| Doesn't call tool | The agent never called this tool. |
| Judge | A judge model decides the reply meets a rubric you write. |
| Reply matches pattern | A regular expression matches anywhere in the reply, with or without case. Patterns that could stall the check, such as nested repeats like (a+)+ or back-references, are refused. |
| Reply is JSON matching a schema | The reply is JSON, bare or in a code fence, that matches the JSON Schema you give. |
| Reply is similar to | The reply's wording overlaps a reference answer by at least a minimum score from 0 to 1, 0.8 by default. This is word overlap, not meaning: use a judge for meaning. |
| Answers within (ms) | The agent answered within this many milliseconds. |
| Uses at most (tokens) | The agent's model calls used at most this many tokens, prompt and answer together. |
A latency or token check that couldn't be measured, for example because the model reported no token usage, is not graded. It never passes by default.
Keep text checks to the one fact that matters, such as a figure or a name. A model rarely repeats a whole sentence word for word.
You can add, edit, duplicate and delete cases at any time. A suite must keep at least one case. Past runs keep each case as it was when they ran.
Import cases
To add many cases at once, choose Import on the suite and pick a CSV or JSONL file, up to 1 MiB:
- CSV: a header row, then one case per row.
inputis required. Each check column adds one check when its cell is filled; repeat a column for two checks of the same kind. Acheckscolumn takes a JSON list of further checks, andhistorya JSON list of earlier turns. The dialog shows the columns and an example. - JSONL: one case per line, as the Evals API takes it.
Choose whether to Add the cases to the suite's current ones or Replace them. Every line is checked first: if one is wrong, nothing is imported, and every bad line is named. Past runs keep the cases they ran. Through the API, the same import is POST /eval-suites/{id}/cases/import.
You can also save a real exchange from the Playground as a case. See Playground.
Judge checks
A judge check sends the conversation, the user's message, the agent's reply and your rubric to a chat model, and asks it for a pass or fail with a reason.
- Choose the judge model yourself, for the suite or for one check. There is no default. A suite with judge checks and no judge model cannot run, and says why.
- The judge does not see the agent's system prompt. It grades the reply as the user would see it.
- Judge calls skip the response cache and use temperature 0, so the same reply gets the same treatment.
- If the judge does not answer in the expected form, Orchestrate asks once more, then marks the check as not graded.
- The judge prompt is versioned, and each graded check records the version and the model that judged it.
If you pick an external model as the judge, the case and the reply leave your cluster. The console warns you.
Write rubrics that a careful reviewer could apply, for example: "Answers politely, gives a forecast for the right city, and does not invent a temperature."
Run a suite
- Open the suite.
- Pick the Agent version to run. A run always tests one exact version.
- Choose Run.
Each case runs through the agent's real run: its prompt, model, tools and knowledge base. Eval runs are safe for agents with write tools:
- Tools that only read run for real.
- Tools that change data never run. The agent is told the call was not executed, and the case can still check that it tried, with Calls tool.
- A step whose approval mode is
neveris treated aspolicyduring evals.
Eval suites run against single-step agents and against code agent versions. A code agent's staging version is evaluated automatically against the suites its orchestrate.yaml names. See Code agents. Console-made agents with several steps are listed but can't be picked yet.
When an agent calls another agent as a tool during an eval, the called agent runs in eval mode too, so its write tools don't run either.
Run on a schedule
A suite can run by itself, against the agent's live version, so you notice when a model, a tool or a knowledge base changes under it. On the suite's page, under Schedule:
- Choose the suite's agent in Settings first.
- Turn on Run on a schedule, and enter a cron and a time zone, as for a workflow schedule.
- Choose Save schedule.
A suite runs at most hourly by default. Each scheduled run is a full suite run and is billed as one. The card shows the next run and the last one, with a link to it. A run that couldn't start, for example because the agent has no live version, is shown on the card with the reason. Scheduled runs work for console-made agents and code agents.
Read the results
| Result | Meaning |
|---|---|
| Case Passed | Every check passed. |
| Case Failed | At least one check ran and did not pass. |
| Case not graded | Nothing failed, but a check could not be graded, for example the judge did not answer. It counts as not passed. |
| Run Passed | The share of passed cases reached the threshold. |
| Run Failed | It did not. |
| Run Couldn't run | No case could be run or graded, for example because the model was down. This says nothing about the agent. |
The score is the number of passed cases divided by the number of cases. Each case shows the reply, the tool calls, every check with its detail, the time taken and the token counts. The agent's tokens and the judge's tokens are counted separately.
Compare runs
A run's page shows what changed Since the previous run of the same suite: cases that got worse, cases that got fixed, and cases that were added or removed. A case that has flipped between passing and failing across recent runs of the same version is marked as flaky: it is not measuring the agent reliably, for example because the model answers differently each time or a tool returns live data. Through the API you can compare with any finished run of the suite. See Evals API.
Delete a suite
Choose Delete suite… at the foot of the suite's page, then confirm. This also deletes its run history. Orchestrate refuses while a run of the suite is in progress, and while the project's release gate requires the suite. Change the release settings first.