Models and serving
Orchestrate serves models inside your cluster. You add a model to the catalog, Orchestrate builds a serving image for it, and then you deploy it. Once it is running, anyone with access can call it through the Model Gateway by its model reference.
The catalog
Open Models to see the catalog. It is shared by every project. Each entry shows its kind, runtime, build status and where it is deployed.
Every model has two names:
| Name | Example | What it is for |
|---|---|---|
| Catalog id | qwen2-5-1-5b | Names the model's build job and deployment in Kubernetes. Lowercase letters, digits, - and . only. |
| Model reference | qwen2.5:1.5b | The model's real name. You send it as model to the Model Gateway, and agents store it. |
A new install comes with a few small Ollama models in the catalog, including an embedding model for knowledge bases. Orchestrate starts building them when the platform first starts. Those builds download the models, so they need internet access.
Kinds and runtimes
| Kind | Used for | Runtimes |
|---|---|---|
| Chat | Agents, the Playground and eval judges | Ollama, vLLM, llama.cpp, SGLang, Triton |
| Vision | A chat model that also reads images. Offered wherever a chat model is. See Images. | Ollama, vLLM |
| Embedding | Knowledge bases and the semantic response cache | Ollama, vLLM, Triton, Infinity |
| Reranker | Reordering a knowledge base's search results. See Knowledge bases. | Infinity |
| OCR | Reading scanned pages, photos and forms into a knowledge base | RapidOCR |
| Moderation | A classifier that scores messages for a guardrail profile, such as Granite Guardian or Llama Guard. See Guardrail profiles. | Ollama |
| Translation | Translating text, as an agent tool or around every turn. See Voice agents. | Ollama, vLLM |
| Speech to text | Transcribing speech in voice sessions | Whisper, speech runtime |
| Text to speech | Speaking agent replies in voice sessions | Piper, speech runtime |
| Voice activity, diarization | Turn detection, and who spoke when in a recording | Speech runtime |
The curated library has models of every kind. A runtime that serves only some kinds, such as Infinity or RapidOCR, refuses the others when you add the model, so a model can't build and then fail every call.
What you type as the model reference depends on the runtime:
| Runtime | Model reference | Default size |
|---|---|---|
| Ollama | An Ollama model tag, such as qwen2.5:1.5b | Small |
| vLLM | A Hugging Face repository id, such as Qwen/Qwen2.5-0.5B-Instruct. The weights are baked into the image at build time. | Large |
| Triton | A direct HTTPS URL to an .onnx file. ONNX is the only format this runtime serves. | Small |
| Whisper | A Whisper size, such as tiny.en or base.en, or a Hugging Face repository id | Medium |
| Piper | A Piper voice name, such as en_US-lessac-medium, or a direct HTTPS URL to your own .onnx voice | Medium |
| llama.cpp | A GGUF model, from Hugging Face or the local store. Runs on CPU, on amd64 and arm64. | Medium |
| SGLang | A Hugging Face repository id. GPU only. | 1 GPU |
| Speech runtime | A Hugging Face repository id of a speech model, such as k2-fsa/OmniVoice. See Voices and speech models. | Medium |
| Infinity | A Hugging Face repository id of a reranker or embedding model | Medium |
| RapidOCR | An OCR pipeline from the library | Medium |
| Your own container | Any image that serves the OpenAI API. See Your own models and containers. | Your choice |
Any runtime can also use weights kept in your install's local model store, with a reference like artifact://my-model.
Add a model
You need the platform admin role.
- Open Models and choose Add model.
- Search the model picker, for example for
qwen,llama 8b,whisperorarabic voice, and pick a model. See The model picker. - The picker fills in the Kind, Runtime, Model reference, Display name, Catalog id and Default size. Check them and change anything you want. People who deploy the model can still pick another size.
- Optionally add a description, then choose Add & build.
To add a model that the picker doesn't list, skip the picker and fill in the form yourself: the kind, the runtime, a display name (the catalog id is filled in from it) and the model reference.
The model picker
The picker lists models from up to four sources:
| Source | When | What it lists |
|---|---|---|
| Curated | Always, also offline | About 40 popular models that come with Orchestrate: Qwen, Llama, Gemma, Mistral, Phi and DeepSeek chat models in common sizes, embedding models, Whisper sizes and Piper voices. |
| Ollama library | When the install can reach the internet | A live search of the Ollama library. A full name:tag is also checked against the Ollama registry. |
| Hugging Face | When the install can reach the internet | A live search of Hugging Face, matched to a runtime: GGUF repositories to Ollama, text-generation and embedding repositories to vLLM, and CTranslate2 Whisper repositories to Whisper. |
| Local store | Always | Every model in your install's local model store. |
Each entry shows its runtime, parameter count, size on disk, licence and whether it needs a CPU or a GPU. It also tells you two things about this install:
-
Where it comes from. Local store means the weights are already in your install and the build doesn't download them. Internet means the build fetches them.
-
Whether it runs here. One of:
Label Meaning Runs here This install can build and serve it. Needs a GPU The model needs more GPUs than this install has. Needs a GPU (unknown) means Orchestrate couldn't read how many GPUs the install has. Not available offline The install has no internet access and the model isn't in the local store. Gated Hugging Face requires you to accept the model's licence first. The build can't fetch gated weights, so download them into the local store with a Hugging Face token set on the install. Can't run here Orchestrate has no build recipe for it, or it is too big for any CPU size. Fits a shared GPU, Fits MIG No whole GPU here is big enough, but a shared GPU or a MIG slice is. Deploy it with a custom size on that device. See GPU sharing. Needs runtime It needs a runtime this install doesn't serve yet, which the label names. Not supported yet No runtime serves it, with the reason, such as an audio-in chat model. Models the install can't deploy are still listed, so you can find them and see why, but they can't be added. Speech repositories without safetensors or ONNX weights aren't offered, because their build would refuse them.
-
Its licence. Shown on every entry. A model under a non-commercial licence, such as OmniVoice, is added only once you tick that you have read its licence.
Live search uses short timeouts. With no internet, the picker shows the curated list and the local store, and says that the live sources are unavailable. Nothing fails.
The build runs in your cluster. It renders the runtime's build recipe, builds the image with Kaniko and pushes it to your model registry. Its status moves through Validating recipe, Submitting build and Building image. Orchestrate gives up on a build after 45 minutes.
Orchestrate checks the catalog id before it starts. If you type a name like smollm2:360m, it refuses and suggests smollm2-360m as the id, with smollm2:360m as the model reference.
A build fetches the runtime's base image and, unless you import the weights, downloads the model weights too. Your cluster also pulls the build tool images (Kaniko, BusyBox and the MinIO client) from their public registries. On a network with no internet access, builds fail even with imported weights. Contact us about getting models into an air-gapped cluster. See Requirements.
The model registry, where built images are pushed and pulled from, is set by your operator. If it is not set, every build fails with a message naming the setting. See Configuration.
Deploy a model
You need the developer or admin platform role, and the developer or admin role on the project you deploy into.
-
Open Models and choose Deploy on a built model.
-
Choose a Size:
Size CPU (request / limit) Memory (request / limit) Replicas Small 0.5 / 1 1 GiB / 2 GiB 1 Medium 1 / 2 2 GiB / 4 GiB 1 Large 2 / 4 4 GiB / 8 GiB 2 1 GPU 4 / 8 24 GiB / 32 GiB 1 2 GPUs 8 / 16 48 GiB / 64 GiB 1 4 GPUs 16 / 32 96 GiB / 128 GiB 1 GPU sizes are offered for runtimes with a GPU path, such as vLLM and SGLang, and need GPU nodes in the cluster. See Requirements.
-
Under Who can use it, choose Shared (one deployment every project can call) or the current project only (private to this project).
-
Under Cluster, keep this cluster, or pick a cluster your admin connected.
-
Choose Deploy to the cluster.
Besides the named sizes, Custom sets exact CPU, memory and GPUs, a GPU type, and autoscaling. See Sizes and GPUs.
The deploy moves through Verifying integrity, Submitting to cluster and Waiting for Ready to Running. It fails if the model is not ready within about 10 minutes, and the model page shows why.
vLLM runs out of memory on the small size, even for small models. Use large for vLLM on CPU, and a GPU size for 7B models and larger. A GPU size reserves whole GPUs, and the model is split across them. Its first start pulls a large image and loads the weights onto the GPU, which can take many minutes.
Ollama, llama.cpp, Triton, Whisper, Piper, Infinity and RapidOCR run on CPU only. SGLang needs a GPU. The speech runtime runs on CPU, and on GPU with a GPU build of its base image. vLLM runs on GPU, or on CPU when your operator builds it for CPU. See vLLM without a GPU. A model deploy is refused if the project may not use the model. See Who can use a model.
These sizes are what Orchestrate asks Kubernetes for. They are not a guide to how much hardware a model needs. For sizing advice, contact us.
The integrity check
Before every deploy, Orchestrate resolves the model's image to its digest and compares it with the digest it recorded when it built the image. The result is one of:
| Verdict | Meaning | What happens |
|---|---|---|
| Verified | The image is exactly what Orchestrate built. | The deploy goes ahead. |
| Unattested | Orchestrate has no record of building this image. | Allowed by default. A platform admin can refuse these with a custom policy. |
| Mismatch | The image has changed since Orchestrate built it. | The deploy is always refused. |
Orchestrate then deploys the image by digest, so a pod that restarts later runs the same bytes. Each verdict is written to the audit log.
This check proves the image is the one Orchestrate built. It is not a malware or vulnerability scan, and it does not prove the weights are what their publisher released.
Images
A vision model reads images as well as text, as OpenAI image_url parts in a chat request. Use one in an agent step, as the Playground target, or as a voice agent's model. People can then attach, paste or drop images:
- in the Playground, for a vision model, an external provider's model, or an agent whose first step uses a vision model
- in the hosted chat and widget, and in pictures sent on Teams, Slack or WhatsApp, when the agent's first step uses a model that reads images
- through the Agent API, as
image_urlparts in the user's message
Up to 4 images per message (PNG, JPEG, WebP or GIF), up to about 1.5 MB in total. Large photos are scaled down in the browser first. Images travel inline: a self-hosted model can't fetch an image from a web address, so a remote image URL is accepted only for an external provider's model. A model that can't read images is refused before the run starts, with the agent's name in the message.
An image turn gets its own model-call time limit, 5 minutes by default, because reading an image on CPU is slow.
Who can use a model
By default every project may use every model. A platform admin can restrict a model to chosen projects:
- Open Models and select the model.
- Under Project access, turn on Restrict to specific projects.
- Pick the projects that may use it, and choose Save access.
Everyone can see a model's access setting. Only a platform admin can change it.
Once a model is restricted, Orchestrate refuses it for every other project:
- The Model Gateway refuses calls to it from any other project, and leaves it out of that project's model list.
- An agent that names the model can't be saved, made live or deployed in another project.
- A knowledge base in another project can't use it as its embedding model.
- It can't be deployed into another project.
Each refusal names the model and says that a platform admin can change its access on the Models page. The Projects page shows which models each project may use.
Call a deployed model
Send the model reference as model to the Model Gateway. The gateway finds the shared deployment in this cluster on its own. A newly deployed model can take up to a minute to become callable. See Call the Model Gateway.
The gateway finds shared deployments in the cluster it runs in. It does not route to a deployment that is private to one project, or to a deployment on another cluster, unless your operator adds a route for it. See Configuration.
Deploy to another cluster
A platform admin can connect more Kubernetes clusters on the Clusters page:
- Under Add cluster, enter a Slug (id), a Name, the Kubernetes type (Vanilla or OpenShift) and the cluster's kubeconfig.
- Choose Add cluster.
- Choose Test connectivity on the new cluster.
Orchestrate stores the kubeconfig in Vault, never in its database. Once a cluster is connected, people can choose it under Cluster when they deploy a model. Through the API, an admin can also run a model's build on that cluster. See Models API.
Remove a model
A platform admin can choose Delete from catalog on a model. This removes the catalog entry only. Deployments that are already running keep running.
The console and the API do not stop a running deployment yet. To stop one, someone with access to the cluster deletes its KServe InferenceService. A shared deployment's InferenceService is named after the model's catalog id. One that is private to a project is named <project id>-<catalog id>.
kubectl -n vois-orchestrate get inferenceservices
kubectl -n vois-orchestrate delete inferenceservice qwen2-5-1-5b
Use the namespace Orchestrate is installed in.