Skip to main content

Online evals and feedback

Eval suites test an agent before it goes live. Online evals and feedback tell you how it does afterwards, on real conversations:

  • Feedback: the people using the agent rate answers with a thumb up or down and an optional comment.
  • Online evals: Orchestrate grades a sample of the agent's production answers with checks you choose, including a judge model.
  • Dataset from production: you pick real runs, for example the ones rated down, and save them as eval cases.

Open Evals > Online evals & feedback and pick an agent. The page has Overview, Feedback, Samples, Settings and Dataset from production tabs, over the last 7, 30 or 90 days.

Feedback​

People can rate an answer wherever they use the agent:

SurfaceHow
Hosted chat and widgetThumbs on each answer, with an optional comment. Only the person whose conversation it is can rate it.
Agent APIPOST /v1/runs/{run_id}/feedback with rating (up or down), an optional comment, and an optional user_id, your application's id for the person. DELETE takes it back. See Agent API.
Teams, Slack, WhatsAppA message that starts with a thumb rates the last answer. A native reaction rates the answer it is on. See Channels.

A run can be rated only once it has ended. Each person has one verdict per answer, and rating again replaces it. Comments are redacted for personal data before they are stored.

Feedback is shown on the run's page in Runs & traces, and per agent version on the Overview tab, so you can see how each version is received.

Online evals​

On the Settings tab, a developer or admin switches online evals on for the agent and sets:

  • the sample rate: the share of completed production runs to grade, from just above 0 up to all of them
  • the checks: Judge (a rubric a judge model applies), Reply contains, Reply doesn't contain, Reply matches pattern, Reply is JSON matching a schema and Reply is similar to. See Write cases. Tool, latency and token checks aren't offered, because an online eval reads only the answer.
  • the judge model, for judge checks

Orchestrate then picks runs as they complete. The choice is a fixed function of each run's id, so it is repeatable and a change of rate affects only later runs. The eval worker grades each picked run's exchange with the checks. Nothing is run again and no tool is called: the judge reads what the agent already answered. A run whose exchange can't be read is recorded as Skipped, with the reason.

Production text reaches the judge model through the Model Gateway, with its PII guardrails. If you pick an external judge model, the exchange leaves your cluster.

Results appear on the Samples tab, newest first, and on the Overview tab per version and per day:

VerdictMeaning
Waiting to be judged, JudgingPicked, not graded yet.
Online eval passedEvery check passed.
Online eval failedAt least one check failed.
Could not be judgedThe judge didn't answer in the expected form.
SkippedThe run's exchange couldn't be read.

A run's online verdict also shows on its page in Runs & traces.

How long they are kept​

Feedback and finished online eval verdicts are deleted after 90 days by default. Your operator can change the period with the API setting FEEDBACK_RETENTION_DAYS; 0 keeps them. The Feedback tab says how long they are kept.

Dataset from production​

The Dataset from production tab turns real runs into eval cases:

  1. Filter the agent's runs by feedback (thumbs down, thumbs up, rated, not rated), online verdict, version, age and text.
  2. Pick the runs to keep.
  3. Choose the checks to add to every case.
  4. Optionally keep the agent's production answer as a reference for a judge check, so the case asks for "the same facts as this answer". This is how a thumbs-up becomes a regression test.
  5. Save into an existing suite or a new one.

A saved case is the user's message and the conversation before it, as stored, redacted. It runs like any other case, and its source records the run, the agent version and the feedback it had. Saving the same run again updates the case. Developers and admins can save, and each save is audited.

API​

See the Evals API for the endpoints behind these pages.

Was this page helpful?
On this page
Talk to our teamQuestions about licensing or sizing VoisX Orchestrate for your site? We can help.Talk to sales →