Skip to main content

Local model store

The local model store keeps model weights inside your install, in its own object store. A model built from the store gets its weights from there, not from Ollama, Hugging Face or another website. This is how you add models to an install that can't, or shouldn't, download them.

Each stored model is called an artifact. A catalog entry points at it with a model reference like artifact://qwen-offline.

You need the platform admin role to see and change the store.

Open the store​

Open Models and choose Local model store. The page lists every stored artifact with its files and sizes, and which runtimes can build from it. An artifact that doesn't yet have a full set of files for any runtime is marked.

Fill the store​

Upload weights you already have​

Under Upload, enter an Artifact name, pick the Runtime, and choose the files. Each runtime expects fixed filenames, and Orchestrate refuses any other name:

RuntimeFilesWhat they are
Ollamamodel.gguf and ModelfileThe GGUF weights, and the model's Modelfile. Get the Modelfile on a connected machine with ollama show --modelfile <model> > Modelfile.
vLLMmodel.tar.gzA Hugging Face snapshot directory, as huggingface-cli download --local-dir produces it.
Tritonmodel.onnxThe ONNX model.
Whispermodel.tar.gzA CTranslate2 model directory, as faster-whisper downloads it.
Pipervoice.onnx and voice.onnx.jsonThe voice and its config file.
llama.cppmodel.ggufOne GGUF file. Merge a split model first with llama-gguf-split.
SGLangmodel.tar.gzA Hugging Face snapshot directory, the same as vLLM.
Speech (hf-speech)model.tar.gzThe speech model's files, safetensors or ONNX only, with a vois-speech.json that names the engine. See Voices and speech models.
Reranker (infinity)model.tar.gzThe reranker's Hugging Face snapshot, in safetensors.
OCR (rapidocr)det.onnx, rec.onnx and cls.onnxThe OCR pipeline's detector, recognizer and orientation classifier.

An Ollama vision model can also carry its image projector as projector.gguf. Downloading an Ollama vision model into the store keeps it, so vision models build offline too.

An artifact name uses lowercase letters, digits and hyphens, starts and ends with a letter or digit, and is at most 63 characters.

Tar the contents, not the folder

A model.tar.gz must unpack with the files at the top level, not inside a folder. For vLLM, config.json must be at the top. For Whisper, model.bin must be at the top. The build stops with an error if it is not.

cd my-model-dir && tar -czf ../model.tar.gz .

The Modelfile is required for Ollama. Without it, an instruct model never gets its chat template and answers with unrelated text.

Download into the store​

While your install can reach the internet, it can fetch a model's weights straight into the store, already in the right layout. Under Download into local store, search for the model with the same picker as Add model, check the Store as name, and choose Download into store. The weights come from:

RuntimeDownloaded from
OllamaThe Ollama registry. The weights are checked against their published SHA-256 digest, and the Modelfile is composed from the model's template and parameters.
vLLM and WhisperHugging Face. A bare Whisper size, such as small, means the matching faster-whisper model.
PiperThe Piper voice repository, or a URL.
TritonA URL to the .onnx file.

Progress shows under Downloads. A download that an API restart cut off shows as failed. Start it again. Downloading under a name that already holds a complete artifact replaces it.

Gated Hugging Face models, whose licence you must accept first, need a Hugging Face token on the install. Contact us to set one.

Delete an artifact​

Delete a stored artifact from its row in the list. This removes its files only. Catalog entries that point at it, and images already built from it, keep working, because a built image carries its own copy of the weights. New builds from it fail.

Build a model from the store​

In Models, choose Add model. Stored models show in the picker as Local store. Pick one, and the form fills in artifact://<name> as the model reference. You can also type artifact://<name> yourself, with the runtime that matches the files.

Before it creates the entry, Orchestrate checks that the artifact has every file the runtime needs. If one is missing, it tells you which. Call the model through the Model Gateway with model set to artifact://<name>.

What still needs internet​

Nothing, for the CPU runtimes. On an install from the release bundle, model builds start from the prebuilt runtime images in the bundle, with every software package already inside, and copy in the weights from the store. No base image is pulled and no package is installed from the internet, so a model whose weights are in the store builds and deploys on a network with no internet access.

The baseline speech models come prebuilt in the bundle, so voice works offline from the first install.

GPU runtimes (vLLM for GPU, Triton, SGLang) are not in the current bundle, so those models can't be built offline yet. See Requirements.

Coming: model packs

Signed model packs with ready-to-use catalog models (a CPU starter pack, a speech pack, an embeddings pack and a GPU pack) are planned, so that adding those models needs no weights of your own. voisctl can already import them; the packs are not available to download yet.

Bring models into an air-gapped install​

Model weights reach an air-gapped install in one of three ways.

From a model pack​

A model pack is a signed file of catalog models from us. Import it with voisctl models import-pack --pack <file> --add: it checks the signature and every file, uploads the weights into the store and adds each model, ready to deploy. See Model packs.

From a connected staging install​

  1. On a connected install, download the models into its store, add them from the picker, and check that they build and answer.

  2. Copy them out with voisctl:

    export VOIS_API_URL=https://<staging-api-host>
    export VOIS_ADMIN_TOKEN=<admin access token>
    ./voisctl models pull --out ./models qwen2-5-1-5b whisper-base-en

    The names are the artifact names the Local model store page lists. You get ./models/<name>/<file> for each one, plus models.json, which records the SHA-256 of every file.

  3. Carry the ./models directory into your network.

  4. At the site, push them into the store:

    export VOIS_API_URL=https://<site-api-host>
    export VOIS_ADMIN_TOKEN=<admin access token>
    ./voisctl models push ./models
    ./voisctl models list

    push checks every file against models.json before it uploads anything.

In the release bundle​

We can include stored models in your release bundle. Their SHA-256 hashes are then in the bundle's signed manifest, and voisctl verify checks them with everything else. After the install or upgrade, push them into the store:

./voisctl models push --bundle ./bundle --keys ./keys

push checks every file against the signed manifest before the first upload. Pass the extracted bundle directory to avoid unpacking the .tar again. Add --only <name> to push just one artifact.

In both cases the models then show in the site's picker as Local store.

The token is an access token for a platform admin. A personal API key isn't enough. See API authentication.

Use the API​

The store has API endpoints too, for scripts. Upload one file per request, with the artifact name and filename in the path:

curl -X POST "$ORCHESTRATE_API/admin/model-artifacts/qwen-offline/model.gguf" \
-H "Authorization: Bearer $ORCHESTRATE_TOKEN" \
-F "file=@./model.gguf"

List an artifact's files with GET /admin/model-artifacts/qwen-offline, and delete it with DELETE /admin/model-artifacts/qwen-offline. See the Models API for every endpoint.

Was this page helpful?