Skip to main content

Models and serving

Orchestrate serves models inside your cluster. You add a model to the catalog, Orchestrate builds a serving image for it, and then you deploy it. Once it is running, anyone with access can call it through the Model Gateway by its model reference.

The catalog​

Open Models to see the catalog. It is shared by every project. Each entry shows its kind, runtime, build status and where it is deployed.

Every model has two names:

NameExampleWhat it is for
Catalog idqwen2-5-1-5bNames the model's build job and deployment in Kubernetes. Lowercase letters, digits, - and . only.
Model referenceqwen2.5:1.5bThe model's real name. You send it as model to the Model Gateway, and agents store it.

A new install comes with a few small Ollama models in the catalog, including an embedding model for knowledge bases. Orchestrate starts building them when the platform first starts. Those builds download the models, so they need internet access.

Kinds and runtimes​

KindUsed forRuntimes
ChatAgents, the Playground and eval judgesOllama, vLLM, llama.cpp, SGLang, Triton
VisionA chat model that also reads images. Offered wherever a chat model is. See Images.Ollama, vLLM
EmbeddingKnowledge bases and the semantic response cacheOllama, vLLM, Triton, Infinity
RerankerReordering a knowledge base's search results. See Knowledge bases.Infinity
OCRReading scanned pages, photos and forms into a knowledge baseRapidOCR
ModerationA classifier that scores messages for a guardrail profile, such as Granite Guardian or Llama Guard. See Guardrail profiles.Ollama
TranslationTranslating text, as an agent tool or around every turn. See Voice agents.Ollama, vLLM
Speech to textTranscribing speech in voice sessionsWhisper, speech runtime
Text to speechSpeaking agent replies in voice sessionsPiper, speech runtime
Voice activity, diarizationTurn detection, and who spoke when in a recordingSpeech runtime

The curated library has models of every kind. A runtime that serves only some kinds, such as Infinity or RapidOCR, refuses the others when you add the model, so a model can't build and then fail every call.

What you type as the model reference depends on the runtime:

RuntimeModel referenceDefault size
OllamaAn Ollama model tag, such as qwen2.5:1.5bSmall
vLLMA Hugging Face repository id, such as Qwen/Qwen2.5-0.5B-Instruct. The weights are baked into the image at build time.Large
TritonA direct HTTPS URL to an .onnx file. ONNX is the only format this runtime serves.Small
WhisperA Whisper size, such as tiny.en or base.en, or a Hugging Face repository idMedium
PiperA Piper voice name, such as en_US-lessac-medium, or a direct HTTPS URL to your own .onnx voiceMedium
llama.cppA GGUF model, from Hugging Face or the local store. Runs on CPU, on amd64 and arm64.Medium
SGLangA Hugging Face repository id. GPU only.1 GPU
Speech runtimeA Hugging Face repository id of a speech model, such as k2-fsa/OmniVoice. See Voices and speech models.Medium
InfinityA Hugging Face repository id of a reranker or embedding modelMedium
RapidOCRAn OCR pipeline from the libraryMedium
Your own containerAny image that serves the OpenAI API. See Your own models and containers.Your choice

Any runtime can also use weights kept in your install's local model store, with a reference like artifact://my-model.

Add a model​

You need the platform admin role.

  1. Open Models and choose Add model.
  2. Search the model picker, for example for qwen, llama 8b, whisper or arabic voice, and pick a model. See The model picker.
  3. The picker fills in the Kind, Runtime, Model reference, Display name, Catalog id and Default size. Check them and change anything you want. People who deploy the model can still pick another size.
  4. Optionally add a description, then choose Add & build.

To add a model that the picker doesn't list, skip the picker and fill in the form yourself: the kind, the runtime, a display name (the catalog id is filled in from it) and the model reference.

The model picker​

The picker lists models from up to four sources:

SourceWhenWhat it lists
CuratedAlways, also offlineAbout 40 popular models that come with Orchestrate: Qwen, Llama, Gemma, Mistral, Phi and DeepSeek chat models in common sizes, embedding models, Whisper sizes and Piper voices.
Ollama libraryWhen the install can reach the internetA live search of the Ollama library. A full name:tag is also checked against the Ollama registry.
Hugging FaceWhen the install can reach the internetA live search of Hugging Face, matched to a runtime: GGUF repositories to Ollama, text-generation and embedding repositories to vLLM, and CTranslate2 Whisper repositories to Whisper.
Local storeAlwaysEvery model in your install's local model store.

Each entry shows its runtime, parameter count, size on disk, licence and whether it needs a CPU or a GPU. It also tells you two things about this install:

  • Where it comes from. Local store means the weights are already in your install and the build doesn't download them. Internet means the build fetches them.

  • Whether it runs here. One of:

    LabelMeaning
    Runs hereThis install can build and serve it.
    Needs a GPUThe model needs more GPUs than this install has. Needs a GPU (unknown) means Orchestrate couldn't read how many GPUs the install has.
    Not available offlineThe install has no internet access and the model isn't in the local store.
    GatedHugging Face requires you to accept the model's licence first. The build can't fetch gated weights, so download them into the local store with a Hugging Face token set on the install.
    Can't run hereOrchestrate has no build recipe for it, or it is too big for any CPU size.
    Fits a shared GPU, Fits MIGNo whole GPU here is big enough, but a shared GPU or a MIG slice is. Deploy it with a custom size on that device. See GPU sharing.
    Needs runtimeIt needs a runtime this install doesn't serve yet, which the label names.
    Not supported yetNo runtime serves it, with the reason, such as an audio-in chat model.

    Models the install can't deploy are still listed, so you can find them and see why, but they can't be added. Speech repositories without safetensors or ONNX weights aren't offered, because their build would refuse them.

  • Its licence. Shown on every entry. A model under a non-commercial licence, such as OmniVoice, is added only once you tick that you have read its licence.

Live search uses short timeouts. With no internet, the picker shows the curated list and the local store, and says that the live sources are unavailable. Nothing fails.

The build runs in your cluster. It renders the runtime's build recipe, builds the image with Kaniko and pushes it to your model registry. Its status moves through Validating recipe, Submitting build and Building image. Orchestrate gives up on a build after 45 minutes.

Orchestrate checks the catalog id before it starts. If you type a name like smollm2:360m, it refuses and suggests smollm2-360m as the id, with smollm2:360m as the model reference.

note

A build fetches the runtime's base image and, unless you import the weights, downloads the model weights too. Your cluster also pulls the build tool images (Kaniko, BusyBox and the MinIO client) from their public registries. On a network with no internet access, builds fail even with imported weights. Contact us about getting models into an air-gapped cluster. See Requirements.

The model registry, where built images are pushed and pulled from, is set by your operator. If it is not set, every build fails with a message naming the setting. See Configuration.

Deploy a model​

You need the developer or admin platform role, and the developer or admin role on the project you deploy into.

  1. Open Models and choose Deploy on a built model.

  2. Choose a Size:

    SizeCPU (request / limit)Memory (request / limit)Replicas
    Small0.5 / 11 GiB / 2 GiB1
    Medium1 / 22 GiB / 4 GiB1
    Large2 / 44 GiB / 8 GiB2
    1 GPU4 / 824 GiB / 32 GiB1
    2 GPUs8 / 1648 GiB / 64 GiB1
    4 GPUs16 / 3296 GiB / 128 GiB1

    GPU sizes are offered for runtimes with a GPU path, such as vLLM and SGLang, and need GPU nodes in the cluster. See Requirements.

  3. Under Who can use it, choose Shared (one deployment every project can call) or the current project only (private to this project).

  4. Under Cluster, keep this cluster, or pick a cluster your admin connected.

  5. Choose Deploy to the cluster.

Besides the named sizes, Custom sets exact CPU, memory and GPUs, a GPU type, and autoscaling. See Sizes and GPUs.

The deploy moves through Verifying integrity, Submitting to cluster and Waiting for Ready to Running. It fails if the model is not ready within about 10 minutes, and the model page shows why.

vLLM runs out of memory on the small size, even for small models. Use large for vLLM on CPU, and a GPU size for 7B models and larger. A GPU size reserves whole GPUs, and the model is split across them. Its first start pulls a large image and loads the weights onto the GPU, which can take many minutes.

note

Ollama, llama.cpp, Triton, Whisper, Piper, Infinity and RapidOCR run on CPU only. SGLang needs a GPU. The speech runtime runs on CPU, and on GPU with a GPU build of its base image. vLLM runs on GPU, or on CPU when your operator builds it for CPU. See vLLM without a GPU. A model deploy is refused if the project may not use the model. See Who can use a model.

These sizes are what Orchestrate asks Kubernetes for. They are not a guide to how much hardware a model needs. For sizing advice, contact us.

The integrity check​

Before every deploy, Orchestrate resolves the model's image to its digest and compares it with the digest it recorded when it built the image. The result is one of:

VerdictMeaningWhat happens
VerifiedThe image is exactly what Orchestrate built.The deploy goes ahead.
UnattestedOrchestrate has no record of building this image.Allowed by default. A platform admin can refuse these with a custom policy.
MismatchThe image has changed since Orchestrate built it.The deploy is always refused.

Orchestrate then deploys the image by digest, so a pod that restarts later runs the same bytes. Each verdict is written to the audit log.

This check proves the image is the one Orchestrate built. It is not a malware or vulnerability scan, and it does not prove the weights are what their publisher released.

Images​

A vision model reads images as well as text, as OpenAI image_url parts in a chat request. Use one in an agent step, as the Playground target, or as a voice agent's model. People can then attach, paste or drop images:

  • in the Playground, for a vision model, an external provider's model, or an agent whose first step uses a vision model
  • in the hosted chat and widget, and in pictures sent on Teams, Slack or WhatsApp, when the agent's first step uses a model that reads images
  • through the Agent API, as image_url parts in the user's message

Up to 4 images per message (PNG, JPEG, WebP or GIF), up to about 1.5 MB in total. Large photos are scaled down in the browser first. Images travel inline: a self-hosted model can't fetch an image from a web address, so a remote image URL is accepted only for an external provider's model. A model that can't read images is refused before the run starts, with the agent's name in the message.

An image turn gets its own model-call time limit, 5 minutes by default, because reading an image on CPU is slow.

Who can use a model​

By default every project may use every model. A platform admin can restrict a model to chosen projects:

  1. Open Models and select the model.
  2. Under Project access, turn on Restrict to specific projects.
  3. Pick the projects that may use it, and choose Save access.

Everyone can see a model's access setting. Only a platform admin can change it.

Once a model is restricted, Orchestrate refuses it for every other project:

  • The Model Gateway refuses calls to it from any other project, and leaves it out of that project's model list.
  • An agent that names the model can't be saved, made live or deployed in another project.
  • A knowledge base in another project can't use it as its embedding model.
  • It can't be deployed into another project.

Each refusal names the model and says that a platform admin can change its access on the Models page. The Projects page shows which models each project may use.

Call a deployed model​

Send the model reference as model to the Model Gateway. The gateway finds the shared deployment in this cluster on its own. A newly deployed model can take up to a minute to become callable. See Call the Model Gateway.

note

The gateway finds shared deployments in the cluster it runs in. It does not route to a deployment that is private to one project, or to a deployment on another cluster, unless your operator adds a route for it. See Configuration.

Deploy to another cluster​

A platform admin can connect more Kubernetes clusters on the Clusters page:

  1. Under Add cluster, enter a Slug (id), a Name, the Kubernetes type (Vanilla or OpenShift) and the cluster's kubeconfig.
  2. Choose Add cluster.
  3. Choose Test connectivity on the new cluster.

Orchestrate stores the kubeconfig in Vault, never in its database. Once a cluster is connected, people can choose it under Cluster when they deploy a model. Through the API, an admin can also run a model's build on that cluster. See Models API.

Remove a model​

A platform admin can choose Delete from catalog on a model. This removes the catalog entry only. Deployments that are already running keep running.

note

The console and the API do not stop a running deployment yet. To stop one, someone with access to the cluster deletes its KServe InferenceService. A shared deployment's InferenceService is named after the model's catalog id. One that is private to a project is named <project id>-<catalog id>.

kubectl -n vois-orchestrate get inferenceservices
kubectl -n vois-orchestrate delete inferenceservice qwen2-5-1-5b

Use the namespace Orchestrate is installed in.

Next steps​

Was this page helpful?