Voice agents
Every agent can hold a voice conversation: one-step agents, multi-step workflows and code agents. The browser connects to a LiveKit media server in your cluster. A voice worker joins the call, turns speech into text, gets the agent's reply and speaks it back. Speech-to-text, the chat model and text-to-speech all run in your cluster and go through the Model Gateway, so every leg is policy-checked, metered to the agent and audited.
Phone calls use the same path. See Phone calls (SIP).
What a voice session needs
- The licence includes voice (
realtime). - A speech-to-text model. Add a model of kind Speech to text with the Whisper runtime, for example
tiny.enfor English or Whispersmallfor many languages, and deploy it. - A text-to-speech model. Add a model of kind Text to speech with the Piper runtime, for example
en_US-lessac-medium, and deploy it. For more voices, or a cloned voice, see Voices and speech models. - An agent with a chat model.
- A route from the browser to LiveKit. Your operator sets the address browsers use for LiveKit. See Configuration.
When neither the agent nor the call names them, Orchestrate uses tiny.en for speech-to-text and en_US-lessac-medium for the voice. A new install seeds these two into the catalog. See Models and serving to add and deploy more.
Talk to an agent
- Open Playground and pick the agent. For a code agent you can also pick the version.
- Choose Switch to voice.
- Choose the transcriber, the voice and the language, or keep the agent's settings.
- Start the call and allow microphone access when your browser asks.
- Speak. The transcript appears as you talk, and the spoken turns are saved to the Playground session.
You can interrupt the agent while it is speaking.
Voice settings
Each agent has a Voice section (on a code agent, in its Connect tab). Saving it needs the developer or admin role.
| Setting | What it does |
|---|---|
| Voice, Speech-to-text model | What a call uses unless the caller picks another. |
| Pipeline | How a turn is answered. See below. |
| Talker model | Direct pipeline only. A small, fast model that holds the conversation instead of the step's own model. Same prompt and tools. |
| Reasoning model (handoff) | Direct pipeline only. A stronger model the talker can consult for hard questions. |
| Filler | What the agent says while it works on a slow turn. Default "One moment, let me think about that." |
| Approval wait | How long a call waits for a person to approve a tool call: 60 to 1800 seconds, default 120. |
| Caller's language | The language callers speak, or Detect automatically. |
| Voice per language | A voice for each language, so the reply switches voice when the caller switches language. |
| Auto-translate | A translation model and the agent's language. See Auto-translate. |
| Turn detection | When the caller started and stopped speaking. See Turn detection. |
| Speaker labels on recordings | A diarization model for the agent's call recordings. See Speaker labels. |
Both models are checked against the catalog and the project's model access when you save and again when a call starts.
How a call runs the agent
| Pipeline | Used for | What a turn is |
|---|---|---|
| Direct | One-step agents, by default | The model streams its reply in the voice worker, and the agent's tools run through the tool execution service. Fastest: the first audio leaves after one model's time to first token. Only the agent's first step exists on this path. |
| Whole workflow | Multi-step agents, by default | Each turn is one whole agent run: every step, every tool and the approval gate, recorded in Runs & traces. The answer arrives whole, after the last step, so the agent says its filler while it works. |
| Code agent | Code agents, always | Each turn is a streamed call to the code agent, spoken as it streams. The call stays on the version it started with. |
Auto, the default, picks Whole workflow for an agent with more than one step and Direct otherwise.
Fast talker and reasoning handoff
On the direct pipeline, a talker model answers quickly and a reasoning model takes the hard questions. The talker gets one extra tool, consult_reasoning_model, and is told to use it when a question is hard. While the reasoning model works, the caller hears the filler. The reasoning model gets the agent's instructions, the last 12 messages and the question, and has no tools. The talker then speaks its answer. A failure is spoken as words, never silence.
Answers from a knowledge base
An agent with a knowledge base can answer callers from it. On the direct pipeline the voice worker adds three things:
- It searches before the model answers. Small, fast talker models often answer from memory instead of calling the search tool. So when the caller finishes a turn of three words or more, the worker searches the knowledge base itself, for the top 3 chunks, and hands the model the result as if it had searched. One model call then answers from the documents. The search tool stays available for a different question.
- A short filler. If the search hasn't come back within 0.8 seconds, the caller hears "Let me check that." once. A fast search says nothing.
- Grounding. The agent is told to answer only from what the search returns, and to say so when it finds nothing, rather than answer from memory in a voice that sounds just as sure.
Each search is recorded on the session: the question, the passages and their scores, and how long the embedding, the search and the round trip took. The Playground's voice panel and the session's page in Realtime show it live. On the whole workflow pipeline the search runs inside the agent run and shows in Runs & traces instead.
The search results are a tool result, so the gateway's prompt-injection scan reads them like any other.
Auto-translate
An agent that works in one language can serve callers in others. In the agent's Voice settings, under Auto-translate, pick a Translation model (a catalog model of kind Translation, such as TranslateGemma) and the Agent's language. On each turn, what the caller says is translated into the agent's language before the agent runs, and the answer is translated back into the caller's language before it is spoken.
Auto-translate needs the whole workflow pipeline, which Auto picks when it is on. Each translation goes through the Model Gateway like any model call, with the guardrails, budgets, model access and cost ledger of the agent's project. Making a version live checks the project may use the translation model.
Agents can also translate on purpose: each ready translation model is offered as a tool, translate:<model>, in an agent's tools.
Approvals on a call
A tool call that needs a person waits on the line. The caller hears, for example, "That needs a person's approval before I can create ticket. I've asked for it, and I'll wait on the line." and a reminder every 30 seconds. A person approves or rejects it in the console, as for any run. Then the caller hears the result, or that a person declined and their reason, or that nobody approved in time.
The wait is the agent's Approval wait. When the call ends, any approval still open is expired, so a late approval can't act for a caller who is gone.
See Human approval.
Languages
A call can be in any language your speech models handle:
- Caller's language tells Whisper which language to expect. With Detect automatically, Whisper detects it for each utterance and the agent replies in the caller's language.
- With Voice per language set, the reply voice switches when the caller switches language. A Piper voice speaks the language its name starts with, for example
ar_JO-kareem-mediumspeaks Arabic. - A model whose name ends in
.en, such astiny.en, transcribes English only. Pick a multilingual Whisper model for other languages or for detection. - A single call can override the language. The Playground's Language picker does this, and so does
languageonPOST /realtime-sessions. - For an agent that should work in one language whatever the caller speaks, use Auto-translate.
See Languages.
Session history
Open Realtime to see the project's voice sessions and phone calls. Each row shows the room, the model, the voice, the speech-to-text model, the status, and when it started and ended. Open a session to see its event log: what was said, the language heard, each tool call and how the session ended.
A session whose room disappears without the worker reporting an end is closed as abandoned. Session records are deleted 30 days after the call ends by default.
You need the developer or admin role on the project to start or view voice sessions.
Capacity
A platform admin sets three numbers for the install under Settings > Voice capacity. An empty field follows the install's default. A change applies to calls started after it.
| Setting | Default | Range | What it does |
|---|---|---|---|
| Simultaneous calls | 2 | 1 to 200 | Browser and phone calls together. When all are in use, starting another returns 503 with Retry-After: 30, before anything starts. |
| End a silent call after | 300 seconds | 30 to 3600, or 0 for never | Ends a call when nobody has spoken and the agent has neither spoken nor worked for this long. |
| Longest call | 3600 seconds | 60 to 14400, or 0 for never | Ends any call this long. |
When a limit ends a call, the agent says why, the session records it, and the slot is free at once. Under the Playground's call button, a line says how many calls are in use out of the limit.
The default of two is what a CPU-only install with one small chat model serves without the agent stalling mid-sentence. Past that, calls don't fail; they slow down, which is why calls are refused at the start instead. An install with faster models or more replicas can take more. Measure the model time per turn in the session's turn metrics, or contact us for sizing.
Speech-to-text and text-to-speech calls count against the project's and organization's budgets and rate limits too, so a project past its monthly cap stops transcribing and speaking as it stops chatting.
Start a session from your own app
- Call
POST /realtime-sessionswith the project and the agent name. Orchestrate creates a LiveKit room, sends the voice worker into it, and returns atokenand thelivekit_url. - Join the room from your app with a LiveKit client SDK, using that URL and token.
The agent's live version runs, on the pipeline its voice settings choose.
Network notes
- By default, Orchestrate gives browsers no usable public STUN server, so media uses LiveKit's own node address. Your operator can set internal STUN or TURN servers for clients behind a NAT. See Configuration.
- The browser needs to reach LiveKit's signalling address over a long-lived WebSocket. A proxy with a short idle timeout drops calls.
Not available yet
- Workflow turns are spoken whole, after the last step. Token streaming from a multi-step run isn't available yet.
- A code agent's voice call isn't saved into a Playground session.
- Interrupting a code agent ends that turn.