Skip to main content

Sizes and GPUs

Every model deployment has a size: how much CPU, memory and GPU each replica gets, and how many replicas run. You pick a named size, or set an exact custom size.

Custom sizes​

When you deploy a model, choose Custom under Size:

FieldWhat to enter
CPUExact cores, such as 4, 1.5 or 500m. From 0.1 to 128.
MemoryWith a unit, such as 16Gi or 512Mi. From 0.25 GiB to 2 TiB.
GPUNone, Whole GPUs (1 to 8), a Shared GPU or a MIG slice. See GPU sharing.
GPU typeAny type, or one of the GPU models the cluster has, such as NVIDIA-L4.

A custom size sets requests equal to limits, so the scheduler reserves exactly what the console shows.

If you deploy without choosing a size, the model's catalog entry decides, and then the runtime's default.

Replicas and autoscaling​

Min and max replicas apply to every size. With max above min, the deployment autoscales: replicas are added when average use passes the target and removed when it falls. Scale on CPU or Memory, at a target from 1 to 100%. The default is CPU at 80%.

Each running deployment shows the size and scaling it actually runs with, read back from the cluster.

What is checked before a deploy​

Before anything is scheduled, Orchestrate refuses a deploy, with a sentence that says what to change, when:

  • the size is outside the limits above, or asks for GPUs on a runtime with no GPU path
  • the model's weights don't fit the GPU memory (whole cards, the share or the MIG slice), or the memory of a CPU size
  • the cluster has no GPU of that type, kind or count, or no single node has as many GPUs as one replica asks for
  • the CPU or memory is more than any node has
  • min replicas times the size is more than the namespace's ResourceQuota allows
  • the project would hold more GPUs than the install allows one project, 8 by default, counted at max replicas
  • autoscaling has max not above min, or a target outside 1 to 100
  • a vLLM image is built for the wrong kind of hardware. See vLLM without a GPU.

These checks read the cluster's nodes and quota. If your operator turned that off (modelServing.gpu.nodeInventory: false), the console says sizes aren't checked, and the scheduler decides.

GPU types​

The console offers only the GPU types the cluster has. It reads them from a node label, nvidia.com/gpu.product by default, which NVIDIA GPU Feature Discovery writes. The labels managed Kubernetes services use are read too. A deploy that picks a type is scheduled only on nodes with that type.

GPU sharing​

GPU sharing is off by default. Your operator turns it on with gpuSharing.mode: time-slicing or mig.

Requires the NVIDIA device plugin

The NVIDIA device plugin, or the GPU Operator, does the sharing. The chart doesn't install it, but it can render the plugin's sharing configuration as a ConfigMap (gpuSharing.devicePluginConfig.create) for the plugin to load. MIG needs A100, A30, H100, H200 or B200-class cards, and the GPU Operator's MIG manager chooses each node's slice layout. On an air-gapped install, mirror the plugin's images into your registry like any other cluster add-on.

Whole GPUTime-slicingMIG
HardwareAny NVIDIAAny NVIDIAA100, A30, H100, H200, B200 class
Memory isolationFullNone: every share sees the whole cardFull, per slice
Fault isolationFullNone: one model's crash or out-of-memory can take the others downFull
PerformanceThe full cardShares take turns; latency rises with every busy neighbourA fixed slice, predictable
Memory per modelThe cardThe card divided by the shares, a budget rather than a wallThe slice, such as 10 GB
Models across several GPUsYesNo: one share per replicaNo: one slice per replica
Use forProduction and large modelsDevelopment, demos, many small models, embeddingsProduction on shared cards, where workloads must not affect each other

What Orchestrate does about each:

  • Time-slicing. A time-sliced node is shown as shared, never counted as whole cards: four shares of one card are not four GPUs, and a model that needs four GPUs is never placed on one card. A vLLM model on a share is told to use only its part of the card's memory, so two models on one card don't each claim 90% of it.
  • MIG. Each slice is offered by name, such as 1g.10gb, with its memory, and the weights are checked against the slice.
  • The model picker says when a GPU model has no whole card on this install but fits a shared GPU or a MIG slice, and names the device to deploy it on with a custom size.

A rule of thumb: time-slicing for development and many small models where an occasional slow reply is fine, MIG when tenants or workloads must not affect each other, and whole GPUs for anything large or latency-critical.

vLLM without a GPU​

vLLM's usual image needs a GPU: on a CPU it starts, reports ready, and then fails every request. An install with no GPU nodes builds its vLLM models from vLLM's CPU image instead, when your operator sets modelServing.vllm.variant: cpu.

  • The CPU base image is in the release bundle like the other runtime bases, so it works offline.
  • A model image is built for one or the other. Orchestrate refuses a CPU-built image on a GPU size and a GPU-built one on a CPU size, and names the fix. Existing vLLM models need a rebuild after the variant changes; your operator can have that done automatically with modelCatalog.rebuildOnRuntimeChange.
  • It is slow but it works: a 0.5B to 1.5B model answers in seconds, a 7B model in tens of seconds. Give it the Large size or a custom size with room for the weights: a quarter of the memory limit goes to vLLM's cache.

Next steps​

Was this page helpful?
On this page
Talk to our teamQuestions about licensing or sizing VoisX Orchestrate for your site? We can help.Talk to sales →