Call the Model Gateway
The Model Gateway is the one endpoint your applications use for every model in Orchestrate. It takes requests in the OpenAI API format for chat, embeddings, speech-to-text and text-to-speech.
Every call goes through the same checks:
- The gateway checks your API key or token.
- For chat and text-to-speech, it scans the text for personal data. See PII guardrails.
- It asks the policy engine whether the call is allowed, and records the decision in the audit log. See Policy.
- For a chat call, it looks in its response cache.
- It sends the call to the model and returns the answer. A transcript from speech-to-text is scanned on the way back.
For every field and status code, see the Model Gateway API reference.
Reach the gateway
The gateway's external address is a switch your operator sets at install, and it is off by default. When it is on, the gateway has a hostname of its own, often one that resolves only on your internal network. The console shows it on API keys > Agent keys. See The Model Gateway address.
When it is off, the console says that no external gateway address is set. From a machine with cluster access, you can forward the gateway's port instead. Use the namespace Orchestrate is installed in:
kubectl port-forward -n vois-orchestrate svc/model-gateway 8001:8001
export ORCHESTRATE_GATEWAY=http://localhost:8001
Authenticate
Send an API key as a bearer token:
Authorization: Bearer vcpk_...
To create a key, open API Keys in the console. Under New API key, enter an optional label and choose Create key. Copy the key straight away. The console shows it only once, and Orchestrate stores only a hash of it.
- A key belongs to you. You see and revoke only your own keys. A platform admin can revoke anyone's key.
- A revoked key stops working on the next request.
- A key can be rotated with an overlap, so the old secret keeps working while you switch. See Usage controls.
- The gateway also accepts a Keycloak access token in the same header. The console uses this.
Agent keys
An agent that runs outside Orchestrate, or a code agent, calls the gateway with an agent key (vcag_) instead. An agent key is bound to one project and one agent, so the gateway knows which agent is calling without any header: usage is metered to the agent, the agent's model policy applies, and its cached answers are kept apart. Point the framework's ordinary OpenAI client at the gateway with the key:
from openai import OpenAI
client = OpenAI(base_url="https://orchestrate-gateway.example.com/v1", api_key="vcag_...")
A project admin creates agent keys on API keys > Agent keys. Code agents that Orchestrate deploys get a key of their own, minted and rotated with each deployment. A project API key (vcpj_) is refused here: it is for the Agent API.
How it compares with the OpenAI API
- Requests use the OpenAI format:
messages,model,tools,tool_choice,streamandtemperaturefor chat. Fields the gateway does not list are ignored. - Streaming chat responses are OpenAI
chat.completion.chunkevents. - Non-streaming chat and embeddings responses use Orchestrate's own shape, shown below. Read the answer from
content, not fromchoices. - There is no
/v1/modelsendpoint. To list models, use the Models API.
Chat
Set model to the model reference of a deployed chat model.
- cURL
- Python
curl -X POST "$ORCHESTRATE_GATEWAY/v1/chat/completions" \
-H "Authorization: Bearer $ORCHESTRATE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen2.5:1.5b",
"messages": [
{ "role": "system", "content": "You answer in one sentence." },
{ "role": "user", "content": "What is a vector database?" }
],
"temperature": 0
}'
import os
import requests
res = requests.post(
f"{os.environ['ORCHESTRATE_GATEWAY']}/v1/chat/completions",
headers={"Authorization": f"Bearer {os.environ['ORCHESTRATE_API_KEY']}"},
json={
"model": "qwen2.5:1.5b",
"messages": [
{"role": "system", "content": "You answer in one sentence."},
{"role": "user", "content": "What is a vector database?"},
],
"temperature": 0,
},
timeout=120,
)
res.raise_for_status()
print(res.json()["content"])
The response:
| Field | What it is |
|---|---|
content | The model's answer. null when the model only called tools. |
tool_calls | Tool calls the model wants to make, in the OpenAI format, or null. |
provider | Who served the call: self_hosted, or the name of an external provider. |
model | The model that served it. |
usage | Token counts as the model reported them, or null. |
cached | exact or semantic if the answer came from the cache, otherwise null. |
guardrail | The PII scan report for the prompt, with a completion report for the answer. |
Self-hosted models on CPU can be slow. Use a client timeout of a minute or more, or stream.
Stream a chat
Add "stream": true. The gateway answers with server-sent events. Each data: line is an OpenAI chat.completion.chunk, and the stream ends with data: [DONE].
curl -N -X POST "$ORCHESTRATE_GATEWAY/v1/chat/completions" \
-H "Authorization: Bearer $ORCHESTRATE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen2.5:1.5b",
"messages": [{ "role": "user", "content": "Count to five." }],
"stream": true
}'
data: {"id":"...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":"One"}}]}
data: {"id":"...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":", two"}}]}
data: [DONE]
- For a model in your cluster, the last chunk before
[DONE]carriesusage. - The prompt's PII report is in the
X-Vois-Guardrailresponse header, as JSON. - The streamed answer is scanned for personal data as it streams, and the report comes in the final chunk. Streamed answers are never cached.
Use tools
Send tool definitions in the OpenAI function format. When the model wants a tool, the response has tool_calls. Run the tool yourself, then send the result back as a tool message with the matching tool_call_id.
{
"model": "qwen2.5:1.5b",
"messages": [{ "role": "user", "content": "What time is it in Dubai?" }],
"tools": [
{
"type": "function",
"function": {
"name": "get_time",
"description": "Get the current time in a city.",
"parameters": {
"type": "object",
"properties": { "city": { "type": "string" } },
"required": ["city"]
}
}
}
]
}
Small models are often unreliable at calling tools. If you want Orchestrate to run the tools, with policy checks and human approval, build an agent instead.
Embeddings
curl -X POST "$ORCHESTRATE_GATEWAY/v1/embeddings" \
-H "Authorization: Bearer $ORCHESTRATE_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "model": "all-minilm", "input": ["first text", "second text"] }'
input is a string or a list of strings. The response has one vector per input, in order:
{
"provider": "self_hosted",
"model": "all-minilm",
"data": [[0.021, -0.113, 0.067], [0.004, 0.089, -0.052]],
"usage": { "prompt_tokens": 6, "total_tokens": 6 }
}
Vectors are shortened here. Embeddings always go to the model you name. There is no fallback.
Speech to text
Send the audio as a multipart form, with a speech-to-text model.
curl -X POST "$ORCHESTRATE_GATEWAY/v1/audio/transcriptions" \
-H "Authorization: Bearer $ORCHESTRATE_API_KEY" \
-F "file=@./question.wav" \
-F "model=tiny.en" \
-F "language=en"
{ "provider": "self_hosted", "model": "tiny.en", "text": "What are your opening hours?", "guardrail": { "action": "observe" } }
Text to speech
curl -X POST "$ORCHESTRATE_GATEWAY/v1/audio/speech" \
-H "Authorization: Bearer $ORCHESTRATE_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "model": "en_US-lessac-medium", "input": "We open at nine.", "response_format": "wav" }' \
--output reply.wav
Piper voices produce wav (the default) or pcm. pcm is raw 16-bit mono audio, streamed as it is made. Its sample rate is in the X-Sample-Rate response header.
More routes
The gateway also serves the other model kinds. Each is governed like a chat call: key scope, budgets and rate limits, policy (each route is its own policy action), model access, guardrails on the text, the cost ledger and the audit log.
| Route | Kind | Shape |
|---|---|---|
POST /v1/rerank | Reranker | Cohere's rerank request: model, query, documents, optional top_n. |
POST /v1/ocr | OCR | A multipart page image and model. Charged per page. |
POST /v1/moderations | Moderation | OpenAI's moderation request. model is required. See Content moderation. |
POST /v1/translations | Translation | model, input (text or a list), target_language, and an optional source_language; without it, the language is detected and returned. |
POST /v1/audio/voices/encode, /v1/audio/vad, /v1/audio/diarization | Speech runtime | Voice cloning, speech segments and speaker labels. See Voices and speech models. |
A chat request to a vision model can carry images as OpenAI image_url parts. See Images.
The response cache
The gateway caches non-streaming chat answers in Redis.
- Exact match. A request with the same model, messages and tools gets the stored answer. Answers are kept for an hour by default.
- Similar match. If your operator sets an embedding model for the cache, a request whose text is very close to a cached one also gets the stored answer. Requests with tools never use this tier.
- The policy check and the audit record happen on every call, cached or not.
To skip the cache for one request, send "cache": false. Do this when you need a fresh answer, such as when you grade replies with a model.
Each caller has their own cache. A person's console session and the API keys they created share one. An answer cached for one person is never returned to another. Calls the platform makes on someone's behalf, such as agent runs, voice sessions and evals, are not cached.
Errors
Error bodies have a detail field that says what went wrong.
| Status | When |
|---|---|
400 | The model cannot be reached or the request is wrong. For example, no deployment serves the model, or the audio file is empty. detail names the model and the reason. |
401 | The key or token is missing, invalid or revoked. |
403 | Denied by policy: the policy engine refused the call, or could not be reached. Orchestrate fails closed. Also a model the project, organization, agent or key may not use, with a code such as model_not_in_key_scope. See Model access. |
422 | A guardrail refused the call. detail.code is guardrail_refused for personal data or prompt_injection_refused for a prompt injection, with the report. A 422 without detail.code is a malformed request body. |
429 | A budget or rate limit was reached: budget_exceeded, token_budget_exceeded, rate_limit_exceeded or token_rate_limit_exceeded, with Retry-After. See Usage controls. |
429, 5xx from a model | The model's runtime answered with this status, for example a speech runtime that is at capacity. The gateway passes the runtime's own status and message through. |
See the API reference for the full request and response fields.