Skip to main content

Models

These endpoints are on the API. They need a Keycloak access token. See Authentication.

A model's catalog id (such as qwen2-5-1-5b) names it in these paths. Its model reference (such as qwen2.5:1.5b) is what you send to the Model Gateway. A catalog id can contain a /, and the paths accept it as is.

List models​

GET/models

Lists the catalog with each model's build status and its deployments across every connected cluster, including deploys still in progress. Any signed-in user can call it.

Query parameters​

kindstringoptional
Only models of this kind: chat, embedding, stt or tts. Any other value returns 400.

Response fields​

modelsarrayoptional
The catalog entries.
models[].model_idstringoptional
The catalog id.
models[].model_refstringoptional
The model reference to send to the Model Gateway.
models[].kindstringoptional
chat, embedding, stt or tts.
models[].runtime_typestringoptional
ollama, vllm, triton, whisper or piper.
models[].providerstring or nulloptional
The external provider, or null for a model served in your cluster.
models[].build_statusstringoptional
pending, building, ready or failed.
models[].build_workflow_idstring or nulloptional
models[].default_resource_profilestringoptional
small, medium or large.
models[].deploymentsarrayoptional
One entry per deployment: scope (shared or isolated), project_id, cluster_id, deployed, ready, url and active_deployment_workflow_id.
live_statebooleanoptional
false when no cluster could be read. Then an empty deployments list means "unknown", not "not deployed".

Deploy a model​

POST/models/{model_id}/deploy

Starts a deploy and returns at once. Poll GET /models/deployments/{workflow_id} for progress. A second deploy of the same model, project, scope and cluster while one is running returns the running one.

Needs the developer or admin platform role and the developer or admin role on project_id. The policy engine is asked too.

Path parameters​

model_idstringrequired
The catalog id.

Body​

project_idstringrequired
The project the deploy belongs to.
resource_profilestringoptional
small (the default), medium or large.
isolationstringoptional
shared (the default) for one deployment every project can call, or isolated for one private to project_id.
cluster_idstringoptional
The cluster to deploy on. Defaults to local, the cluster Orchestrate runs in.

Responses​

200
The deploy started. Returns workflow_id and model_id.
400
The model's runtime has no known port.
403
Your roles do not allow this, or policy denied it.
404
No such model.
409
The model is not built yet, it is an external model with nothing to deploy, or no model registry is configured for the cluster.

Get deploy status​

GET/models/deployments/{workflow_id}

Live progress of a deploy. Any signed-in user can call it.

Response fields​

stagestringoptional
verifying_integrity, submitting, polling_ready, ready or failed.
detailstring or nulloptional
Why it failed, when it failed.
poll_attemptintegeroptional
How many readiness checks have run.
max_pollsintegeroptional
How many it makes before giving up.
endpointstring or nulloptional
The deployment's in-cluster URL once it is ready.
integrityobject or nulloptional
The integrity verdict and the digest deployed, once checked.

Responses​

200
The status.
404
No such deploy.

Add a model to the catalog​

POST/admin/models

Adds a model and starts building its image. Returns at once. Poll GET /models/builds/{workflow_id} for progress. Needs the admin platform role.

To register an external model instead, send provider and api_base. Nothing is built, and the model is ready at once. This needs external providers to be turned on. See External providers.

Body​

idstringrequired
The catalog id. Lowercase letters, digits, - and ., starting and ending with a letter or digit, at most 57 characters.
display_namestringrequired
The name shown in the console.
runtime_typestringrequired
ollama, vllm, triton, whisper or piper.
model_refstringrequired
The model reference. Its format depends on the runtime. See Kinds and runtimes. Use artifact://<name> for imported weights.
kindstringoptional
chat (the default), embedding, stt or tts.
descriptionstringoptional
What teams should use it for.
default_resource_profilestringoptional
small, medium or large. Leave it out to use the runtime's default.
cluster_idstringoptional
The cluster that runs the build. Defaults to local.
providerstringoptional
For an external model: the provider's name.
api_basestringoptional
For an external model: the provider's OpenAI-compatible base URL. Required with provider.
api_keystringoptional
For an external model: the provider's key. Stored in Vault, never returned.

Responses​

200
Created. Returns id, build_workflow_id and build_status, or external: true for an external model.
400
The id is not a valid name (the message suggests one), the runtime or kind is unknown, or an imported artifact is missing files.
403
Not a platform admin, or external providers are off.
409
A model with this id exists.
502
The external model's key could not be stored, so nothing was created.

Get build status​

GET/models/builds/{workflow_id}

Live progress of a model build. The model's build_status is updated when the build finishes.

Response fields​

stagestringoptional
validating_recipe, submitting, polling_build, ready or failed.
detailstring or nulloptional
Why it failed, when it failed.
poll_attemptintegeroptional
How many checks have run.
max_pollsintegeroptional
How many it makes before giving up.
imagestring or nulloptional
The image pushed, once ready.
digeststring or nulloptional
The image's digest, recorded for the integrity check.

Responses​

200
The status.
404
No such build.

List catalog entries​

GET/admin/models

The catalog rows as stored, without deployment state. Needs the admin platform role. Each entry has id, display_name, description, runtime_type, model_ref, default_resource_profile, image, build_status, build_workflow_id, kind, provider and api_base.

Remove a model from the catalog​

DELETE/admin/models/{model_id}

Removes the catalog entry. Running deployments of the model are not stopped. For an external model, its stored key is deleted too. Needs the admin platform role.

Responses​

200
Removed. Returns {"removed": "<model_id>"}.
404
No such model.

Model artifacts​

Imported model weights. All artifact endpoints need the admin platform role. See Local model store for which files each runtime needs.

POST/admin/model-artifacts/{name}/{filename}

Uploads one file of an artifact, as multipart/form-data with the file in file. Large files are streamed.

Path parameters​

namestringrequired
The artifact name. Lowercase letters, digits and hyphens, at most 63 characters.
filenamestringrequired
One of model.gguf, Modelfile, model.tar.gz, model.onnx, voice.onnx or voice.onnx.json.

Response fields​

namestringoptional
The artifact name.
filenamestringoptional
The file stored.
sizeintegeroptional
Its size in bytes, as stored.
model_refstringoptional
The reference to use in the catalog, artifact://<name>.

Responses​

200
Stored.
400
The name is not valid, or no runtime uses this filename.
GET/admin/model-artifacts

Lists every artifact with its files and their sizes.

GET/admin/model-artifacts/{name}

One artifact's files and sizes. 404 if nothing is stored under the name.

DELETE/admin/model-artifacts/{name}

Deletes every file of the artifact. Catalog entries and built images are not touched.

Clusters​

Other Kubernetes clusters that models can be built and deployed on. The cluster Orchestrate runs in is always present as local and cannot be removed.

GET/admin/clusters

Lists clusters with id, name, kubernetes_type and registry_url. Any signed-in user can call it. Credentials are never returned.

POST/admin/clusters

Connects a cluster. Needs the admin platform role. The kubeconfig is stored in Vault.

Body​

idstringrequired
A slug for the cluster.
namestringrequired
Its display name.
kubeconfigstringrequired
The kubeconfig, as YAML text.
kubernetes_typestringoptional
vanilla (the default) or openshift.
registry_urlstringoptional
The model registry this cluster's nodes pull from, if it differs from the install's default.
POST/admin/clusters/{cluster_id}/test

Checks that Orchestrate can reach the cluster with its stored kubeconfig. Returns ok and a detail with the server version or the error. Needs the admin platform role.

DELETE/admin/clusters/{cluster_id}

Disconnects a cluster and deletes its kubeconfig from Vault. Needs the admin platform role.

Responses​

200
Done.
400
You tried to remove local, or the registry URL is not valid.
409
A cluster with this id exists.
502
The kubeconfig could not be stored in Vault.
Was this page helpful?