Skip to main content

Evals

An eval suite is a set of test cases for one agent. Each case sends a message to a specific agent version and checks what comes back: the reply and the tool calls behind it. Run a suite whenever you change an agent, and compare the result with the last run.

A project can require a passing eval before a new version goes live. See Live versions and the release gate.

To see how an agent does after it goes live, use online evals and feedback: graded samples of real answers, thumbs from the people using it, and real runs saved as cases.

You need the developer or admin role on the project to create, change or run suites. Viewers can read them.

Create a suite​

  1. Open Evals and choose New suite.
  2. Name it, pick the agent it tests, and write its first case.
  3. Choose Create suite.

In the suite's Settings you can change:

  • the agent it tests. A suite tests one agent, by name, so it works for every version.
  • the pass threshold: the share of cases that must pass for the run to pass. The default is 80%.
  • the judge model, for judge checks. See Judge checks.

Write cases​

A case has:

  • a Message: what the user says
  • an optional History: earlier user and assistant turns, so you can test the middle of a conversation
  • one or more Checks. A case passes only if every check passes.
CheckPasses when
Reply containsThe reply contains this text, ignoring case.
Reply doesn't containThe text appears nowhere in the reply, ignoring case.
Calls toolThe agent called this tool. You can also require arguments: each named argument must contain the given text.
Doesn't call toolThe agent never called this tool.
JudgeA judge model decides the reply meets a rubric you write.
Reply matches patternA regular expression matches anywhere in the reply, with or without case. Patterns that could stall the check, such as nested repeats like (a+)+ or back-references, are refused.
Reply is JSON matching a schemaThe reply is JSON, bare or in a code fence, that matches the JSON Schema you give.
Reply is similar toThe reply's wording overlaps a reference answer by at least a minimum score from 0 to 1, 0.8 by default. This is word overlap, not meaning: use a judge for meaning.
Answers within (ms)The agent answered within this many milliseconds.
Uses at most (tokens)The agent's model calls used at most this many tokens, prompt and answer together.

A latency or token check that couldn't be measured, for example because the model reported no token usage, is not graded. It never passes by default.

Keep text checks to the one fact that matters, such as a figure or a name. A model rarely repeats a whole sentence word for word.

You can add, edit, duplicate and delete cases at any time. A suite must keep at least one case. Past runs keep each case as it was when they ran.

Import cases​

To add many cases at once, choose Import on the suite and pick a CSV or JSONL file, up to 1 MiB:

  • CSV: a header row, then one case per row. input is required. Each check column adds one check when its cell is filled; repeat a column for two checks of the same kind. A checks column takes a JSON list of further checks, and history a JSON list of earlier turns. The dialog shows the columns and an example.
  • JSONL: one case per line, as the Evals API takes it.

Choose whether to Add the cases to the suite's current ones or Replace them. Every line is checked first: if one is wrong, nothing is imported, and every bad line is named. Past runs keep the cases they ran. Through the API, the same import is POST /eval-suites/{id}/cases/import.

You can also save a real exchange from the Playground as a case. See Playground.

Judge checks​

A judge check sends the conversation, the user's message, the agent's reply and your rubric to a chat model, and asks it for a pass or fail with a reason.

  • Choose the judge model yourself, for the suite or for one check. There is no default. A suite with judge checks and no judge model cannot run, and says why.
  • The judge does not see the agent's system prompt. It grades the reply as the user would see it.
  • Judge calls skip the response cache and use temperature 0, so the same reply gets the same treatment.
  • If the judge does not answer in the expected form, Orchestrate asks once more, then marks the check as not graded.
  • The judge prompt is versioned, and each graded check records the version and the model that judged it.

If you pick an external model as the judge, the case and the reply leave your cluster. The console warns you.

Write rubrics that a careful reviewer could apply, for example: "Answers politely, gives a forecast for the right city, and does not invent a temperature."

Run a suite​

  1. Open the suite.
  2. Pick the Agent version to run. A run always tests one exact version.
  3. Choose Run.

Each case runs through the agent's real run: its prompt, model, tools and knowledge base. Eval runs are safe for agents with write tools:

  • Tools that only read run for real.
  • Tools that change data never run. The agent is told the call was not executed, and the case can still check that it tried, with Calls tool.
  • A step whose approval mode is never is treated as policy during evals.

Eval suites run against single-step agents and against code agent versions. A code agent's staging version is evaluated automatically against the suites its orchestrate.yaml names. See Code agents. Console-made agents with several steps are listed but can't be picked yet.

When an agent calls another agent as a tool during an eval, the called agent runs in eval mode too, so its write tools don't run either.

Run on a schedule​

A suite can run by itself, against the agent's live version, so you notice when a model, a tool or a knowledge base changes under it. On the suite's page, under Schedule:

  1. Choose the suite's agent in Settings first.
  2. Turn on Run on a schedule, and enter a cron and a time zone, as for a workflow schedule.
  3. Choose Save schedule.

A suite runs at most hourly by default. Each scheduled run is a full suite run and is billed as one. The card shows the next run and the last one, with a link to it. A run that couldn't start, for example because the agent has no live version, is shown on the card with the reason. Scheduled runs work for console-made agents and code agents.

Read the results​

ResultMeaning
Case PassedEvery check passed.
Case FailedAt least one check ran and did not pass.
Case not gradedNothing failed, but a check could not be graded, for example the judge did not answer. It counts as not passed.
Run PassedThe share of passed cases reached the threshold.
Run FailedIt did not.
Run Couldn't runNo case could be run or graded, for example because the model was down. This says nothing about the agent.

The score is the number of passed cases divided by the number of cases. Each case shows the reply, the tool calls, every check with its detail, the time taken and the token counts. The agent's tokens and the judge's tokens are counted separately.

Compare runs​

A run's page shows what changed Since the previous run of the same suite: cases that got worse, cases that got fixed, and cases that were added or removed. A case that has flipped between passing and failing across recent runs of the same version is marked as flaky: it is not measuring the agent reliably, for example because the model answers differently each time or a tool returns live data. Through the API you can compare with any finished run of the suite. See Evals API.

Delete a suite​

Choose Delete suite… at the foot of the suite's page, then confirm. This also deletes its run history. Orchestrate refuses while a run of the suite is in progress, and while the project's release gate requires the suite. Change the release settings first.

Was this page helpful?
On this page
Talk to our teamQuestions about licensing or sizing VoisX Orchestrate for your site? We can help.Talk to sales →