Sizes and GPUs
Every model deployment has a size: how much CPU, memory and GPU each replica gets, and how many replicas run. You pick a named size, or set an exact custom size.
Custom sizes
When you deploy a model, choose Custom under Size:
| Field | What to enter |
|---|---|
| CPU | Exact cores, such as 4, 1.5 or 500m. From 0.1 to 128. |
| Memory | With a unit, such as 16Gi or 512Mi. From 0.25 GiB to 2 TiB. |
| GPU | None, Whole GPUs (1 to 8), a Shared GPU or a MIG slice. See GPU sharing. |
| GPU type | Any type, or one of the GPU models the cluster has, such as NVIDIA-L4. |
A custom size sets requests equal to limits, so the scheduler reserves exactly what the console shows.
If you deploy without choosing a size, the model's catalog entry decides, and then the runtime's default.
Replicas and autoscaling
Min and max replicas apply to every size. With max above min, the deployment autoscales: replicas are added when average use passes the target and removed when it falls. Scale on CPU or Memory, at a target from 1 to 100%. The default is CPU at 80%.
Each running deployment shows the size and scaling it actually runs with, read back from the cluster.
What is checked before a deploy
Before anything is scheduled, Orchestrate refuses a deploy, with a sentence that says what to change, when:
- the size is outside the limits above, or asks for GPUs on a runtime with no GPU path
- the model's weights don't fit the GPU memory (whole cards, the share or the MIG slice), or the memory of a CPU size
- the cluster has no GPU of that type, kind or count, or no single node has as many GPUs as one replica asks for
- the CPU or memory is more than any node has
- min replicas times the size is more than the namespace's ResourceQuota allows
- the project would hold more GPUs than the install allows one project, 8 by default, counted at max replicas
- autoscaling has max not above min, or a target outside 1 to 100
- a vLLM image is built for the wrong kind of hardware. See vLLM without a GPU.
These checks read the cluster's nodes and quota. If your operator turned that off (modelServing.gpu.nodeInventory: false), the console says sizes aren't checked, and the scheduler decides.
GPU types
The console offers only the GPU types the cluster has. It reads them from a node label, nvidia.com/gpu.product by default, which NVIDIA GPU Feature Discovery writes. The labels managed Kubernetes services use are read too. A deploy that picks a type is scheduled only on nodes with that type.
GPU sharing
GPU sharing is off by default. Your operator turns it on with gpuSharing.mode: time-slicing or mig.
The NVIDIA device plugin, or the GPU Operator, does the sharing. The chart doesn't install it, but it can render the plugin's sharing configuration as a ConfigMap (gpuSharing.devicePluginConfig.create) for the plugin to load. MIG needs A100, A30, H100, H200 or B200-class cards, and the GPU Operator's MIG manager chooses each node's slice layout. On an air-gapped install, mirror the plugin's images into your registry like any other cluster add-on.
| Whole GPU | Time-slicing | MIG | |
|---|---|---|---|
| Hardware | Any NVIDIA | Any NVIDIA | A100, A30, H100, H200, B200 class |
| Memory isolation | Full | None: every share sees the whole card | Full, per slice |
| Fault isolation | Full | None: one model's crash or out-of-memory can take the others down | Full |
| Performance | The full card | Shares take turns; latency rises with every busy neighbour | A fixed slice, predictable |
| Memory per model | The card | The card divided by the shares, a budget rather than a wall | The slice, such as 10 GB |
| Models across several GPUs | Yes | No: one share per replica | No: one slice per replica |
| Use for | Production and large models | Development, demos, many small models, embeddings | Production on shared cards, where workloads must not affect each other |
What Orchestrate does about each:
- Time-slicing. A time-sliced node is shown as shared, never counted as whole cards: four shares of one card are not four GPUs, and a model that needs four GPUs is never placed on one card. A vLLM model on a share is told to use only its part of the card's memory, so two models on one card don't each claim 90% of it.
- MIG. Each slice is offered by name, such as
1g.10gb, with its memory, and the weights are checked against the slice. - The model picker says when a GPU model has no whole card on this install but fits a shared GPU or a MIG slice, and names the device to deploy it on with a custom size.
A rule of thumb: time-slicing for development and many small models where an occasional slow reply is fine, MIG when tenants or workloads must not affect each other, and whole GPUs for anything large or latency-critical.
vLLM without a GPU
vLLM's usual image needs a GPU: on a CPU it starts, reports ready, and then fails every request. An install with no GPU nodes builds its vLLM models from vLLM's CPU image instead, when your operator sets modelServing.vllm.variant: cpu.
- The CPU base image is in the release bundle like the other runtime bases, so it works offline.
- A model image is built for one or the other. Orchestrate refuses a CPU-built image on a GPU size and a GPU-built one on a CPU size, and names the fix. Existing vLLM models need a rebuild after the variant changes; your operator can have that done automatically with
modelCatalog.rebuildOnRuntimeChange. - It is slow but it works: a 0.5B to 1.5B model answers in seconds, a 7B model in tens of seconds. Give it the Large size or a custom size with room for the weights: a quarter of the memory limit goes to vLLM's cache.