Online evals and feedback
Eval suites test an agent before it goes live. Online evals and feedback tell you how it does afterwards, on real conversations:
- Feedback: the people using the agent rate answers with a thumb up or down and an optional comment.
- Online evals: Orchestrate grades a sample of the agent's production answers with checks you choose, including a judge model.
- Dataset from production: you pick real runs, for example the ones rated down, and save them as eval cases.
Open Evals > Online evals & feedback and pick an agent. The page has Overview, Feedback, Samples, Settings and Dataset from production tabs, over the last 7, 30 or 90 days.
Feedback
People can rate an answer wherever they use the agent:
| Surface | How |
|---|---|
| Hosted chat and widget | Thumbs on each answer, with an optional comment. Only the person whose conversation it is can rate it. |
| Agent API | POST /v1/runs/{run_id}/feedback with rating (up or down), an optional comment, and an optional user_id, your application's id for the person. DELETE takes it back. See Agent API. |
| Teams, Slack, WhatsApp | A message that starts with a thumb rates the last answer. A native reaction rates the answer it is on. See Channels. |
A run can be rated only once it has ended. Each person has one verdict per answer, and rating again replaces it. Comments are redacted for personal data before they are stored.
Feedback is shown on the run's page in Runs & traces, and per agent version on the Overview tab, so you can see how each version is received.
Online evals
On the Settings tab, a developer or admin switches online evals on for the agent and sets:
- the sample rate: the share of completed production runs to grade, from just above 0 up to all of them
- the checks: Judge (a rubric a judge model applies), Reply contains, Reply doesn't contain, Reply matches pattern, Reply is JSON matching a schema and Reply is similar to. See Write cases. Tool, latency and token checks aren't offered, because an online eval reads only the answer.
- the judge model, for judge checks
Orchestrate then picks runs as they complete. The choice is a fixed function of each run's id, so it is repeatable and a change of rate affects only later runs. The eval worker grades each picked run's exchange with the checks. Nothing is run again and no tool is called: the judge reads what the agent already answered. A run whose exchange can't be read is recorded as Skipped, with the reason.
Production text reaches the judge model through the Model Gateway, with its PII guardrails. If you pick an external judge model, the exchange leaves your cluster.
Results appear on the Samples tab, newest first, and on the Overview tab per version and per day:
| Verdict | Meaning |
|---|---|
| Waiting to be judged, Judging | Picked, not graded yet. |
| Online eval passed | Every check passed. |
| Online eval failed | At least one check failed. |
| Could not be judged | The judge didn't answer in the expected form. |
| Skipped | The run's exchange couldn't be read. |
A run's online verdict also shows on its page in Runs & traces.
How long they are kept
Feedback and finished online eval verdicts are deleted after 90 days by default. Your operator can change the period with the API setting FEEDBACK_RETENTION_DAYS; 0 keeps them. The Feedback tab says how long they are kept.
Dataset from production
The Dataset from production tab turns real runs into eval cases:
- Filter the agent's runs by feedback (thumbs down, thumbs up, rated, not rated), online verdict, version, age and text.
- Pick the runs to keep.
- Choose the checks to add to every case.
- Optionally keep the agent's production answer as a reference for a judge check, so the case asks for "the same facts as this answer". This is how a thumbs-up becomes a regression test.
- Save into an existing suite or a new one.
A saved case is the user's message and the conversation before it, as stored, redacted. It runs like any other case, and its source records the run, the agent version and the feedback it had. Saving the same run again updates the case. Developers and admins can save, and each save is audited.
API
See the Evals API for the endpoints behind these pages.