Models
These endpoints are on the API. They need a Keycloak access token. See Authentication.
A model's catalog id (such as qwen2-5-1-5b) names it in these paths. Its model reference (such as qwen2.5:1.5b) is what you send to the Model Gateway. A catalog id can contain a /, and the paths accept it as is.
List models
/modelsLists the catalog with each model's build status and its deployments across every connected cluster, including deploys still in progress. Any signed-in user can call it.
Query parameters
kindstringoptionalchat, embedding, stt or tts. Any other value returns 400.Response fields
modelsarrayoptionalmodels[].model_idstringoptionalmodels[].model_refstringoptionalmodels[].kindstringoptionalchat, embedding, stt or tts.models[].runtime_typestringoptionalollama, vllm, triton, whisper or piper.models[].providerstring or nulloptionalnull for a model served in your cluster.models[].build_statusstringoptionalpending, building, ready or failed.models[].build_workflow_idstring or nulloptionalGET /models/builds/{workflow_id}.models[].default_resource_profilestringoptionalsmall, medium or large.models[].deploymentsarrayoptionalscope (shared or isolated), project_id, cluster_id, deployed, ready, url and active_deployment_workflow_id.live_statebooleanoptionalfalse when no cluster could be read. Then an empty deployments list means "unknown", not "not deployed".Deploy a model
/models/{model_id}/deployStarts a deploy and returns at once. Poll GET /models/deployments/{workflow_id} for progress. A second deploy of the same model, project, scope and cluster while one is running returns the running one.
Needs the developer or admin platform role and the developer or admin role on project_id. The policy engine is asked too.
Path parameters
model_idstringrequiredBody
project_idstringrequiredresource_profilestringoptionalsmall (the default), medium or large.isolationstringoptionalshared (the default) for one deployment every project can call, or isolated for one private to project_id.cluster_idstringoptionallocal, the cluster Orchestrate runs in.Responses
workflow_id and model_id.Get deploy status
/models/deployments/{workflow_id}Live progress of a deploy. Any signed-in user can call it.
Response fields
stagestringoptionalverifying_integrity, submitting, polling_ready, ready or failed.detailstring or nulloptionalpoll_attemptintegeroptionalmax_pollsintegeroptionalendpointstring or nulloptionalintegrityobject or nulloptionalResponses
Add a model to the catalog
/admin/modelsAdds a model and starts building its image. Returns at once. Poll GET /models/builds/{workflow_id} for progress. Needs the admin platform role.
To register an external model instead, send provider and api_base. Nothing is built, and the model is ready at once. This needs external providers to be turned on. See External providers.
Body
idstringrequired- and ., starting and ending with a letter or digit, at most 57 characters.display_namestringrequiredruntime_typestringrequiredollama, vllm, triton, whisper or piper.model_refstringrequiredartifact://<name> for imported weights.kindstringoptionalchat (the default), embedding, stt or tts.descriptionstringoptionaldefault_resource_profilestringoptionalsmall, medium or large. Leave it out to use the runtime's default.cluster_idstringoptionallocal.providerstringoptionalapi_basestringoptionalprovider.api_keystringoptionalResponses
id, build_workflow_id and build_status, or external: true for an external model.Get build status
/models/builds/{workflow_id}Live progress of a model build. The model's build_status is updated when the build finishes.
Response fields
stagestringoptionalvalidating_recipe, submitting, polling_build, ready or failed.detailstring or nulloptionalpoll_attemptintegeroptionalmax_pollsintegeroptionalimagestring or nulloptionaldigeststring or nulloptionalResponses
List catalog entries
/admin/modelsThe catalog rows as stored, without deployment state. Needs the admin platform role. Each entry has id, display_name, description, runtime_type, model_ref, default_resource_profile, image, build_status, build_workflow_id, kind, provider and api_base.
Remove a model from the catalog
/admin/models/{model_id}Removes the catalog entry. Running deployments of the model are not stopped. For an external model, its stored key is deleted too. Needs the admin platform role.
Responses
{"removed": "<model_id>"}.Model artifacts
Imported model weights. All artifact endpoints need the admin platform role. See Local model store for which files each runtime needs.
/admin/model-artifacts/{name}/{filename}Uploads one file of an artifact, as multipart/form-data with the file in file. Large files are streamed.
Path parameters
namestringrequiredfilenamestringrequiredmodel.gguf, Modelfile, model.tar.gz, model.onnx, voice.onnx or voice.onnx.json.Response fields
namestringoptionalfilenamestringoptionalsizeintegeroptionalmodel_refstringoptionalartifact://<name>.Responses
/admin/model-artifactsLists every artifact with its files and their sizes.
/admin/model-artifacts/{name}One artifact's files and sizes. 404 if nothing is stored under the name.
/admin/model-artifacts/{name}Deletes every file of the artifact. Catalog entries and built images are not touched.
Clusters
Other Kubernetes clusters that models can be built and deployed on. The cluster Orchestrate runs in is always present as local and cannot be removed.
/admin/clustersLists clusters with id, name, kubernetes_type and registry_url. Any signed-in user can call it. Credentials are never returned.
/admin/clustersConnects a cluster. Needs the admin platform role. The kubeconfig is stored in Vault.
Body
idstringrequirednamestringrequiredkubeconfigstringrequiredkubernetes_typestringoptionalvanilla (the default) or openshift.registry_urlstringoptional/admin/clusters/{cluster_id}/testChecks that Orchestrate can reach the cluster with its stored kubeconfig. Returns ok and a detail with the server version or the error. Needs the admin platform role.
/admin/clusters/{cluster_id}Disconnects a cluster and deletes its kubeconfig from Vault. Needs the admin platform role.
Responses
local, or the registry URL is not valid.