External providers
Every model call in Orchestrate is served inside your cluster. Calls to an AI provider outside the cluster are off by default. This is what keeps prompts, documents and transcripts on your infrastructure.
The switches
Two switches decide whether calls may leave the cluster:
- The ceiling, set by your operator in the Helm values:
externalProviders.enabled, off by default. While it is off, no model call leaves the cluster, whatever the console says. This is the setting an auditor reads. - The console switch, under Settings > External providers. With the ceiling on, a platform admin can switch external providers off, and on again, without a redeploy. The console can never turn them on past the ceiling: trying answers
409and names the Helm value.
An organization can also keep its own projects off external providers, and list the only models they may use. See Model access.
While external providers are off:
- The External providers page says so, and the API refuses to register an external model. The error names the setting.
- The Model Gateway refuses to call an external model, even one registered while they were on, and leaves it out of
GET /v1/models. Turning them off stops those calls without deleting anything. - The fallback provider keys below are not used.
Connect a provider
This section applies only to an install where external providers are on. Open Models > External providers.
| Who | What they can connect |
|---|---|
| A platform admin | Providers for the whole install, including an OpenAI-compatible server on your own network, such as your own vLLM. |
| An organization admin | Providers for their organization only. The endpoint must be on the internet: loopback, private, link-local and cluster-internal addresses are left to a platform admin, because the connection test runs from inside the cluster. The host is checked by its DNS answer, again before every test and model list. |
- Choose Connect a provider and pick the Provider: OpenAI, Anthropic, Gemini, Azure OpenAI, Mistral, or any OpenAI-compatible endpoint.
- Enter a Name, the API base URL where the provider needs one (for Azure OpenAI, with your resource name), and the API key. The key goes to Vault and is never shown again. A public provider is reached over HTTPS.
- Choose Connect, then Test connection. The list shows each provider's key, last test, models and state.
Enable models
Choose Models on a provider. Orchestrate lists the provider's models, read live from the provider, or a curated list when that can't be read. For Azure OpenAI, type the deployment names you created. For each model you want:
- Set its Kind: chat, embedding, speech to text or text to speech. Kinds the gateway doesn't route aren't offered.
- Set its prices, in US dollars per million input and output tokens. They go to the cost ledger. A model can stay unpriced.
- Choose Enable.
An enabled model joins the catalog and every model picker as <provider id>/<model>, the name callers send to the Model Gateway. That name is also what prices and model access rules use. In Models it shows as External, with the warning that calls to it leave the cluster. There is nothing to build or deploy.
Switch off stops a provider's models without removing them. Remove deletes the provider, every model enabled from it and its key; agents that use those models stop working.
Error messages from a provider are cleaned of any part of a key before they are shown or logged.
Through the API
A platform admin can still register a single external model with POST /admin/models, naming provider, api_base, model_ref and an optional api_key. See the Models API.
What stays in control
- PII guardrails. Before a prompt leaves for an external provider, the gateway scans it. By default it refuses the call if it finds a payment card number, an IBAN, an Emirates ID, a US Social Security number or a secret token. See PII guardrails.
- Policy. The policy engine sees which provider serves every call. An admin can write a rule that denies, for example, chat calls to one provider. See Policy.
- Audit. The audit event for each call names the provider, so you can show which calls left the cluster. See Audit log.
- Responses say who served them. Every Model Gateway response has a
providerfield. It isself_hostedfor a model in your cluster.
The fallback provider keys
The Model Gateway also accepts an Anthropic API key and an OpenAI API key as install settings, modelGateway.anthropicApiKey and modelGateway.openaiApiKey. They are empty by default. When external providers are on and either key is set, a chat request that names no model, or names a model nothing in your cluster serves, is sent to that provider's default model instead.
While external providers are off, these keys are never used. Leave them empty unless you want this fallback, and always name a model in your requests.