Skip to main content

Voices and speech models

Whisper and Piper cover speech to text and a voice per language. The speech runtime adds the rest of the Hugging Face speech models: voices you clone from a short recording, multi-voice text-to-speech, larger speech-to-text models, voice activity detection and speaker diarization. Each runs in your cluster, one model per deployment, and every call goes through the Model Gateway, with policy, model access, budgets, the cost ledger and the audit trail.

Speech models in the library​

ModelWhat it doesLicenceRuns on
OmniVoiceText to speech with zero-shot voice cloning, and voices designed from a descriptionNon-commercial (CC-BY-NC-4.0 weights, plus its tokenizer's community licence); code Apache-2.0CPU, slowly. A GPU for live calls.
Kokoro 82MText to speech with 54 built-in voices in several languagesApache-2.0CPU
Whisper large v3 turboSpeech to text, through transformersMITGPU
Silero VAD v5Voice activity detection: where speech starts and stopsMITCPU
Speaker diarizationWho spoke when (pyannote segmentation 3.0 with 3D-Speaker CAM++)MIT and Apache-2.0CPU

Every library source is pinned to an exact Hugging Face commit. Any other Hugging Face text-to-speech or speech-to-text repository builds too, at the commit its main branch had when it was built. Only safetensors and ONNX weights are accepted; a model with pickle files is refused, because pickle can carry code.

Licences are shown in the model library, the Add model panel and the Voices page. A model under a non-commercial licence is added only once you tick that you have read its licence, and is never put in a release model pack.

On an air-gapped install, download a speech model into the local model store on a connected install and carry it over. A running speech model never downloads anything.

Requires a GPU build for GPU models

The speech runtime's default base image is CPU-only, and a GPU size on it stops at start-up and says so. For Whisper large v3 turbo, and for OmniVoice on live calls, your operator builds a GPU variant of the speech runtime's base image and points the install at it. Contact us for the steps. Kokoro, voice activity detection and diarization run on CPU either way.

The Voices page​

Open Voices to see every voice the project's agents can speak with:

  • Built-in voices: the voices of the text-to-speech models in the catalog, such as Kokoro's, with their language and licence.
  • Cloned voices: voices cloned for this project.

Choose Preview to hear any voice with your own text. A voice whose model isn't built is shown, with the reason no call can use it.

Clone a voice​

You need the developer or admin role on the project, and a voice cloning model such as OmniVoice in the catalog.

  1. Choose Clone a voice.
  2. Give it a Name and pick the Cloning model.
  3. Upload a Reference clip: 3 to 10 seconds of one person speaking clearly, as WAV, FLAC or OGG, at most 10 MB.
  4. Type What the clip says, exactly, or leave it empty and pick a speech-to-text model to transcribe it.
  5. Pick the Language.
  6. Tick that the speaker agreed to having their voice cloned for this use. Who recorded the consent is shown with the voice.
  7. Choose Clone voice.

The model keeps an encoding of the clip, never the recording itself. A cloned voice belongs to its project, and only that project's calls can use it. A voice can't be deleted while an agent uses it.

Pick a voice for an agent​

An agent's Voice setting takes any of:

  • a Piper voice, such as en_US-lessac-medium
  • a built-in voice of a speech model, such as one of Kokoro's
  • a cloned voice

Choose Preview voice to hear it before you save. See Voice agents.

Turn detection​

On a live call, the voice worker decides when the caller started and stopped speaking with Silero VAD v5, on every audio frame. The defaults suit most calls. To tune them, open the agent's Voice settings and turn on Tune turn detection:

SettingEffect
Speech thresholdHigher ignores more background noise.
Silence that ends a turn (ms)Longer waits out pauses in the middle of a sentence.
Shortest speech (ms)Shorter sounds, such as a cough, don't interrupt the agent.

Speaker labels on recordings​

Pick a diarization model under Speaker labels on recordings in the agent's Voice settings. Each stored call recording is then split by speaker, and each turn transcribed with the call's speech-to-text model. The call's transcript under Telephony says who spoke when. To label a recording on demand, choose Label speakers on the call.

Recordings are made only with the organization's consent. See Phone calls (SIP).

From your own code​

The Model Gateway serves these models on OpenAI-shaped routes:

RouteWhat it does
POST /v1/audio/speechSpeak text. Name a built-in or cloned voice as <model reference>#<voice>.
POST /v1/audio/transcriptionsTranscribe audio.
POST /v1/audio/vadFind the speech segments in a recording.
POST /v1/audio/diarizationWho spoke when in a recording.

Each is its own policy action, so a policy can allow speaking but not cloning, for example. See Call the Model Gateway.

Was this page helpful?
On this page
Talk to our teamQuestions about licensing or sizing VoisX Orchestrate for your site? We can help.Talk to sales →