Agent evals
Agent evals test your own agents the way your customers use them. A simulated caller talks to the agent, by phone, through the website widget or in chat. VoisX then scores the conversation with automatic checks and an AI judge, and tells you which cases passed.
Use evals to:
- catch a regression before it reaches callers, for example after you change the prompt, a tool or the voice
- compare voices, models and phone carriers on the same conversations
- turn a real call that went wrong into a test, so it stays fixed
- stop an agent from going Live when its tests fail
Suites, cases and runs
| Term | What it is |
|---|---|
| Suite | A named set of cases, for example "Booking agent smoke". A suite can be pinned to one agent. Its defaults apply to every case. |
| Case | One conversation to test: who calls, what they want, and what a good result looks like. A case can override any default of its suite. |
| Run | One execution of a suite's cases against one agent. A run starts when you select Run, or when the agent is published (see Publish gate). |
| Result | One case in one matrix cell, with its scores, the judge's reasons, the transcript and, for voice, the recording. |
A suite holds up to 200 cases. When a case sets a list, such as its checks or expected tools, that list replaces the suite's list instead of adding to it.
You can import and export a suite as JSON or YAML, so you can keep suites in your own repository:
name: Booking agent smoke
description: Core booking paths
agent_id: 42
defaults:
target: chat
persona: {type: llm, prompt: "A polite customer", language: en}
checks: [{name: not_contains, params: {text: Question}}]
cases:
- name: books a table
goal: Books a table for 4 tomorrow at 8pm
expected_tools: [{name: create_booking}]
- name: phone smoke
target: phone
carrier: plivo
checks: [{name: answered}]
Targets
Each case has a target, the way the simulated caller reaches the agent:
| Target | How the caller talks to the agent |
|---|---|
chat | Text, through the same path as the chat API. |
phone | A real phone call through a carrier: plivo, twilio, tryvox or sip (your own SIP trunk). |
widget | Browser voice, through the website widget. |
replay | The caller's turns from a stored conversation, sent again as text to the current agent. See Turn a real call into a case. |
Personas
The persona is the simulated caller. There are two kinds.
Scripted personas follow steps you write. Each step either says fixed text (say) or gives the caller's AI an instruction (llm), which answers what the agent just said:
{"type": "scripted",
"prompt": "You are Ravi, a regular customer, polite but in a hurry.",
"language": "en-IN",
"steps": [
{"say": "Hi, I want to book a table", "lang": "en-IN"},
{"llm": "answer the agent's question briefly", "lang": "en-IN", "repeat": 2},
{"say": ["my name is", "Ravi Kumar"], "lang": "en-IN", "style": "pause_mid"},
{"say": "thanks, bye", "lang": "en-IN", "goodbye": true}
]}
On voice targets, a step's style shapes how it is spoken:
| Style | The caller |
|---|---|
normal | Speaks the line. |
pause_mid | Pauses in the middle of a sentence. Tests that the agent waits instead of jumping in. |
backchannel | Says short "mm-hm" style words while the agent talks. |
barge_in | Talks over the agent. Tests that the agent stops and listens. |
one_word | Answers with a single word. |
Steps can also set gap_s, after_s, wait_s, repeat (1 to 20), goodbye and expect_agent_lang. lang is any language tag, such as hi-IN or ar-AE. Text targets ignore the spoken styles and timings.
LLM personas improvise from a description until the goal is reached, the agent ends the conversation, or max_turns (1 to 50, default 10) is reached:
{"type": "llm", "prompt": "You are a frustrated customer whose order is late.", "language": "hi", "max_turns": 10}
Checks
Checks are automatic scores. Each check gives a score from 0 to 1 and passes or fails. Add them to a case or to the suite defaults by name, with optional parameters:
"checks": [
{"name": "answered"},
{"name": "latency", "params": {"p50_max": 2.0, "p95_max": 3.5}}
]
The console lists every check with its parameters. The API lists them at GET /scorers.
Voice checks
For phone and widget cases:
| What it tests | Check | What it measures |
|---|---|---|
| Turn taking | answered | Every caller turn got a reply. |
pause_mid | The agent waited when the caller paused in the middle of a sentence, instead of cutting in. | |
repeats | The agent did not repeat itself. | |
| Barge-in | barge_in, interruption_handling | The agent stopped speaking when the caller talked over it, and answered what the caller said. |
| Latency | latency | Time from the end of the caller's turn to the agent's reply, against your limits (for example p50 and p95). |
| Language follow | language | The agent answered in the language you expected, including when the caller switched language. |
| Hang-up | hangup | The call ended cleanly when the conversation was over, and the agent did not hang up early. |
| Audio | audio_quality | The agent's audio was clear and complete. |
Conversation checks
For every target:
| What it tests | Check | What it measures |
|---|---|---|
| Tools | tool_calls | The agent called the tools in expected_tools, with the arguments and in the order you set. Added automatically when the case has expected_tools. |
forbidden_tools | The agent did not call any tool in forbidden_tools. | |
| Knowledge base (RAG) | rag_docs | The documents in rag.expect_docs were among those the agent retrieved. |
grounded | Every fact the agent stated is supported by what it retrieved. Scored by the judge. | |
| Memory | memory_recall | The agent remembered facts from an earlier conversation. Scored by the judge. Not available on replay. |
| Personal data | pii_leak | The agent did not reveal personal data it should not. |
| Extraction accuracy | extraction_accuracy | The data the agent extracted matches expected_extraction. Added automatically when the case has expected_extraction. |
| Task success | task_success | The caller's goal was achieved. Scored by the judge. Added automatically when the case has a goal. |
| Text | contains, not_contains, regex | The agent's replies contain, or do not contain, the text or pattern you set. |
| Length | max_turns | The conversation finished within a number of turns. |
| Limits | cost_limit, duration_limit | The result stayed within its cost and time limits. |
Style rules
Some things callers notice are best tested with a text check or a rubric line:
| What it tests | How |
|---|---|
| No "Question N" labels | A not_contains or regex check, for example the pattern (?i)question\s+\d+. Catches an agent that reads numbered questions from its prompt aloud ("Question 2: what is your date of birth?"). |
| Simple language | A rubric criterion, for example "Uses short, simple sentences a caller can follow on the phone, with no jargon". |
Expected extraction
expected_extraction lists the values the agent should extract from the conversation, and how to compare them:
"expected_extraction": {
"party_size": {"value": 4, "match": "exact"},
"name": {"value": "Ravi Kumar", "match": "normalized"},
"notes": {"value": "window seat", "match": "judge"}
}
match | Passes when |
|---|---|
exact | The values are equal. |
normalized (default) | They are equal ignoring case, spaces and punctuation. Numbers are compared as numbers. |
judge | The judge decides they mean the same thing. |
Expected tools
"expected_tools": [
{"name": "check_availability", "args_match": {"party_size": 4}, "order": 1},
{"name": "create_booking", "order": 2}
],
"forbidden_tools": ["cancel_booking"],
"tools_mode": "mock",
"tool_mocks": {"check_availability": {"available": true}}
By default (tools_mode: mock) tool calls are not sent to your systems. The agent gets the answer you put in tool_mocks, or {"ok": true}. With tools_mode: live, tools really run, so a test can create real bookings, tickets or messages. Use live only against test systems.
The judge
The judge is an AI model (by default gpt-4.1, set by the platform) that reads the conversation and scores what a fixed rule cannot. It scores, in one pass:
- task success, from the case's
goal - every rubric criterion you write
- grounded answers and memory recall, when the case asks for them
- extraction fields with
match: judge
"goal": "The caller books a table for 4 at 8pm tomorrow",
"rubric": [
{"criterion": "Confirms date, time and party size before booking", "weight": 2, "threshold": 0.8}
]
Each criterion gets a score from 0 to 1, a short reason and the turns it based that on. A criterion passes when its score reaches its threshold (default 0.7). weight (default 1) sets how much it counts towards the result's overall score.
The judge sees the transcript with personal data masked, the same way Insights does.
When a result passes
A result passes when every check passes and every rubric criterion reaches its threshold. A result with no scores at all fails. Checks and rubric criteria give scores, and so do fields that add their own check: goal, expected_extraction and expected_tools.
A result can also end as:
| Status | Meaning |
|---|---|
error | The conversation could not be run or scored, for example the call did not connect. Counts as failed. |
skipped | The cell cannot run here, for example the carrier is switched off or memory is off for the agent. Not counted. |
cancelled | The run was cancelled first. Not counted. |
The run's pass rate is passed results divided by counted results. A manual run passes only when every counted result passes. A publish run passes when the pass rate reaches the gate's minimum.
Test across voices, models and carriers
A case's matrix runs the same conversation several times, once per combination:
"matrix": {
"voice_types": ["openai", "gemini", "elevenlabs", "sarvam"],
"models": [],
"carriers": ["plivo", "twilio"]
}
| Axis | Values | Applies to |
|---|---|---|
voice_types | The agent's voice provider: openai (OpenAI Realtime), gemini (Gemini Live), elevenlabs, sarvam. See Models and voices. | phone, widget |
models | Model IDs. On voice targets, the voice model, for example gpt-live-1 to test GPT-Live. On chat and replay, the chat model. | All |
carriers | plivo, twilio, tryvox, sip | phone |
An empty list keeps the agent's own setting. A case can have up to 24 combinations, and a run up to 500 results. When you start a run you can also pass one matrix for every case.
Each result shows its cell, so you can compare, for example, latency on Plivo and Twilio, or task success on two models.
Turn a real call into a case
When a real conversation goes wrong, add it to a suite as a case, by its session ID (POST .../cases/from-session). The conversation must belong to the same project. There are two ways:
- Replay (
mode: replay) makes areplaycase. The caller's exact words are sent again, as text, to the current version of the agent. - AI caller (
mode: llm) makes an LLM persona from the conversation's summary, intent and language, with the agent's goal. Use it when the agent's new answers would take the conversation somewhere else.
By default the values the agent extracted in that conversation become the case's expected_extraction. Check them before you save: if the agent extracted something wrong, fix the expected value.
Publish gate
You can make a suite a publish gate for an agent: set the suite and a minimum pass rate (from 0 to 1) on the agent (PUT .../eval-on-publish).
When someone changes the agent from Draft to Live:
- The agent stays Draft. The rest of the change is saved. A publish run starts.
- When the run passes (pass rate at or above the minimum), the agent goes Live.
- When the run fails, has an error or is cancelled, the agent stays Draft. The audit log records why.
If the agent changes again while the run is going, the result does not count: the agent stays Draft and you publish again. A second publish while a gate run is going is refused.
The gate does not run for edits to an agent that is already Live, or when you move an agent back to Draft.
If the gate's suite no longer exists, publishing is blocked, not skipped. Pick another suite or remove the gate to publish.
Billing
Evals bill in two parts:
- The simulated conversations are real conversations of your agent. They bill on the normal meters, the same as any call or chat: voice minutes, chat messages, phone carrier minutes and so on. See Usage and billing. They also show in Logs and Insights, with a session ID that starts with
eval-. - Agent evals covers the judge, the simulated caller's AI and the caller's Sarvam speech. It is billed on cost: what the model and speech providers charge us, times your plan's multiplier (1.5 by default). It shows as Agent evals in your usage details.
Each result and run shows its cost. To keep costs in check, each case has limits:
| Limit | Range | Default |
|---|---|---|
max_call_s | 10 to 900 seconds | 240 |
max_cost_usd | Above 0, up to 20 USD per result | 1.0 |
A conversation that reaches a limit is stopped.
Plan availability
Agent evals are a plan feature. When your plan does not include it, the Evals pages and API answer with feature_not_in_plan, and publishing never waits for a gate. Contact us to add it to your plan.
When evals are switched off
VoisX can switch evals off for everyone, or for one organization, for example during an incident with a provider. While evals are off:
- New runs cannot start, and the API answers
403with the codeevals_disabled. - Runs that are going stop: results not yet finished become
cancelled. - Publishing is not blocked. Agents with a publish gate go Live at once, and the audit log records that the gate was skipped.
Suites, cases and past results are kept. Everything works again when evals are switched back on.
Data and retention
Transcripts, recordings, timelines and judge notes for each result are stored in your project's data region. Transcripts are masked before they are stored. Runs and their files are deleted after your Transcripts retention period, or when you delete the suite.
API
Everything on this page is available through the Evals API.