Skip to main content

Operations

This page covers day-to-day running of a VoisX Orchestrate install: checking health, reading logs, backing up and restoring, scaling and fixing common problems.

Every command here names the cluster with --context. Pass it on every command so nothing runs against the wrong cluster.

Check health​

Start with the pods:

kubectl --context <your-context> -n vois-orchestrate get pods

Every pod should be Running and Ready. To see why a pod isn't, describe it and read its events:

kubectl --context <your-context> -n vois-orchestrate describe pod <pod-name>

Health endpoints​

These are the checks Kubernetes runs to decide whether each component is ready. You can call them yourself from inside the cluster or through a port-forward.

ComponentService and portCheck
APIapi:8000GET /health. GET /version returns the installed version.
Model Gatewaymodel-gateway:8001GET /health
License Servicelicense-service:8002GET /health. GET /version and GET /license/status too.
Audit serviceaudit:8003GET /health
Knowledge-base ingestionrag-ingestion:8004GET /health
Consoleweb:3000GET /
Keycloakkeycloak:8080GET /realms/vois-orchestrate
Open Policy Agentopa:8181GET /health?bundles. Ready only once it has loaded the custom policies from the API.
MinIOminio:9000GET /minio/health/ready
Qdrantqdrant:6333GET /readyz
LiveKitlivekit:7880GET /
PostgreSQLpostgres:5432pg_isready inside the pod
Redisredis:6379redis-cli ping inside the pod
Agent edge (code agents on)agent-edge:8020GET /health, GET /health/ready

The orchestration worker, tool execution, deployment controller, eval worker, realtime voice worker and Temporal have no HTTP health check. For these, watch the pod status and the logs, or the System page.

GET /health and GET /version on the API need no sign-in. For example:

kubectl --context <your-context> -n vois-orchestrate exec deploy/api -- \
python3 -c "import urllib.request; print(urllib.request.urlopen('http://127.0.0.1:8000/version').read().decode())"

The System page​

Platform admins can open System in the console for one read-only answer to "what is this install and is it working": the platform version, the health of every component, the licence in force, whether NetworkPolicy is enforced for code agents, and the facts an offline upgrade needs (registry, namespace, database revision). The workers that have no HTTP port are checked by whether the orchestration engine sees them polling for work. Each check has its own short timeout, so one hung service never hides the others. Components the install doesn't run are shown as off.

The Vulnerability scanning card shows your registry's image scanner and how current its vulnerability database is: Database current with its date, Database stale (older than modelRegistry.scanner.maxDbAgeDays, 30 days by default), No database, Scanner unreachable or Not configured. A stale database, or none, is shown as an error: a scan with it reports nothing published since, so don't rely on its results. The card also says how model weights are checked: every vLLM build scans its weights for pickle payloads, and for the other runtimes the card says why no scan applies. An air-gapped install receives a fresh scanner database with each release bundle. See Requirements.

Metrics and alerts​

With observability.metrics.enabled, the API and the Model Gateway export Prometheus metrics on port 9464: requests and latency by route, model calls, latency and tokens by model, guardrail refusals and policy denials, and channel messages. The port is a separate listener that is never exposed. Pods carry the usual scrape annotations, and observability.metrics.serviceMonitor.enabled adds ServiceMonitors for the Prometheus operator.

With PostgreSQL on CloudNativePG, postgresHA.monitoring.podMonitor and postgresHA.monitoring.prometheusRule add a PodMonitor and alerts. See Database alerts.

For traces, set observability.tracing.otlpEndpoint to an OpenTelemetry collector on your network.

The chart sets readiness checks but no liveness checks, so Kubernetes stops sending traffic to a stuck pod but doesn't restart it. Restart a stuck component yourself, as in Restart a component.

Logs​

Every component logs to standard output. Read a component's log with kubectl logs:

kubectl --context <your-context> -n vois-orchestrate logs deployment/api --since=1h
kubectl --context <your-context> -n vois-orchestrate logs deployment/model-gateway -f

Use the Deployment name: api, model-gateway, web, license-service, audit, opa, keycloak, orchestration-worker, tool-execution, deployment-controller, eval, rag-ingestion, realtime-agent-worker, livekit, temporal or temporal-ui, and, when those features are on, agent-edge, sip-gateway and site-gateway. For the data stores, use the pod: postgres-0 (or postgres-ha-1, postgres-ha-2 and so on with high availability), qdrant-0, minio-0 or redis-0.

Logs aren't kept after a pod is replaced. To keep them, collect them with your cluster's log collector.

If an internalAuth value is empty, the audit service and the knowledge-base search service log a warning at startup that their endpoints accept calls without a credential. Set the matching value from Configuration.

Workflow runs​

Agent runs, model builds and deployments, ingestion and evals run as Temporal workflows. To look at one in detail, open the Temporal web UI through a port-forward:

kubectl --context <your-context> -n vois-orchestrate port-forward svc/temporal-ui 8080:8080

Then open http://localhost:8080.

The Temporal web UI has no sign-in

Anyone who reaches the Temporal web UI can read every workflow, including its inputs and results, and can act on it. Only reach it through a port-forward from an operator's machine. Never publish it.

Audit log​

Who did what in Orchestrate is recorded in a signed, hash-chained audit log. Platform administrators can read and verify it on the Audit log page of the console, or with voisctl audit verify, and export it to a SIEM. See Audit log.

Back up and restore​

What to back up​

DataWhere it livesHow to back it up
Application data: organizations, projects, agents, runs, the model catalog, policies, keys, settingsPostgreSQL, the Orchestrate databaseContinuous backups with high availability, or pg_dump
Users, passwords, roles and identity providersPostgreSQL, Keycloak's databaseThe same
Workflow historyPostgreSQL, Temporal's databasesThe same
Audit logPostgreSQL, the audit databaseThe same, plus your SIEM copies
Uploaded documents, model files, agent sources and call recordingsMinIOThe scheduled MinIO copy, a volume snapshot, or an S3 tool
Knowledge-base vectorsQdrantThe scheduled Qdrant snapshots, or a volume snapshot. You can also rebuild them by ingesting the documents again.
Response cacheRedisOptional. Losing it costs cache hits, not data.
A licence uploaded in the consoleThe License Service's volumeThe scheduled licence copy
Connector, channel and code agent credentials, users' tokens, registered clustersOpenBao (or your own Vault)Back up its data the way your setup requires, and keep the unseal keys with it: the data is useless without them.
Model and agent imagesYour registryYour registry's own backup.
Generated keys: audit keys, the Temporal payload key, the site CAKubernetes Secrets in the namespaceKeep copies somewhere safe. Losing the payload key makes running workflows unreadable.
Your configurationYour values file, the vois-orchestrate-install-values Secret, and the Secrets you created: licence, TLS, pull secrets, registry and backup credentialsKeep copies somewhere safe. They contain secrets.

Scheduled backups to S3​

Orchestrate can back up every store to one S3-compatible target: AWS S3, MinIO, Ceph, Wasabi, R2 or any other. Create a Secret with the access key pair once, then set backups.s3:

orchestrate-values.yaml
backups:
s3:
endpointURL: https://s3.eu-west-1.amazonaws.com
bucket: orchestrate-backups
credentialsSecret: { name: backup-s3 } # keys ACCESS_KEY_ID and ACCESS_SECRET_KEY
# caSecret: { name: backup-ca } # an endpoint with a private CA
postgres: { enabled: true, schedule: "0 0 2 * * *", retention: 30d }
qdrant: { enabled: true, schedule: "30 2 * * *", keep: 7 }
minio: { enabled: true, schedule: "0 3 * * *" }
redis: { enabled: false }
license: { enabled: true, schedule: "45 2 * * *", keep: 30 }
kubectl --context <your-context> -n vois-orchestrate create secret generic backup-s3 \
--from-literal=ACCESS_KEY_ID=<key> --from-literal=ACCESS_SECRET_KEY=<secret>
StoreWhat is copiedRecovery point
PostgreSQLWith high availability: a base backup every night and continuous WAL archiving, kept for retention. Point-in-time recovery reaches back that far.At most about 5 minutes
QdrantA snapshot of every collection each night, keeping keep sets24 hours
MinIONew and changed objects each night. Nothing is ever deleted at the target. Needs a target outside this install.24 hours
RedisA snapshot each night, keeping keep sets24 hours
LicenceThe uploaded licence each night24 hours

backups.s3.inClusterMinio: true sends backups to the install's own MinIO. That proves the pipeline but isn't disaster recovery, because the backups share the cluster with the data. Use an endpoint outside the cluster for anything that matters.

Back up PostgreSQL without high availability​

With postgresHA.mode: off, the chart's default, PostgreSQL is a single pod, and you back it up with pg_dump. Production installs use high availability instead; this applies to installs that haven't moved to it yet. Run this from the admin workstation every day, and right before every upgrade:

#!/usr/bin/env bash
set -euo pipefail
CTX=<your-context>
NS=vois-orchestrate
STAMP=$(date +%Y%m%d-%H%M)

for db in $(kubectl --context "$CTX" -n "$NS" exec postgres-0 -- sh -c \
'psql -U "$POSTGRES_USER" -d postgres -Atc "SELECT datname FROM pg_database WHERE NOT datistemplate"'); do
[ "$db" = postgres ] && continue
kubectl --context "$CTX" -n "$NS" exec postgres-0 -- sh -c \
'pg_dump -U "$POSTGRES_USER" -Fc "$1"' _ "$db" > "orchestrate-$db-$STAMP.dump"
done

kubectl --context "$CTX" -n "$NS" exec postgres-0 -- sh -c \
'pg_dumpall -U "$POSTGRES_USER" --globals-only' > "orchestrate-globals-$STAMP.sql"

You get one .dump file per database and a orchestrate-globals-*.sql file with the database roles. Store them off the cluster, and protect them: they hold all your Orchestrate data.

To restore, stop the Orchestrate services so nothing writes, run pg_restore --clean --if-exists for each dump into the database of the same name inside postgres-0, then start the services again. If you are rolling back an upgrade, roll back the Helm release first. See Roll back.

A restore replaces current data

Restoring a database replaces everything in it with the backup's contents. Anything written since the backup is lost.

Test a restore from time to time, on a separate install, so you know the backups work.

High availability​

PostgreSQL on CloudNativePG​

Production installs run PostgreSQL on CloudNativePG, as in Install: a primary and streaming replicas with automatic failover, and continuous backups to your S3 target. It needs the CloudNativePG operator and its Barman Cloud plugin in the cluster. See Requirements.

SituationRecovery
A node or pod is lostThe most advanced replica is promoted, in about 30 to 60 seconds. With synchronous replication no acknowledged write is lost; without it, at most the last few transactions in flight. Every service keeps connecting to the same postgres Service, which follows the new primary.
The data is lost or damagedRecover from the base backup and the archived WAL, to any moment inside the retention window.

A fresh install sets postgresHA.mode: cnpg, postgresHA.bootstrap: initdb and a backups.s3 target. No single PostgreSQL pod is ever created.

An existing install moves over in two upgrades, so nothing written can be lost:

  1. postgresHA.mode: import. The existing databases become read-only: reads keep working, writes fail. A CloudNativePG cluster is created by importing every database, and its first backup is taken. Services still use the old pod.
  2. postgresHA.mode: cutover. The deploy refuses unless the new cluster is ready with every instance, WAL archiving is running, a base backup has completed, the old databases are still read-only, and every table has the same row count on both sides. Only then does the postgres Service switch to the new primary. Sessions still open on the old pod are ended.

Keep the old pod for a few days as the way back. postgresHA.mode: rollback makes it writable and points the Service at it again. Writes made on the new cluster since the cutover are not copied back. Then set postgresHA.mode: cnpg to remove the old pod from the release. Its volume stays until you delete it.

Roles that exist on the old database but own nothing, such as a read-only role you made by hand, must be listed in postgresHA.extraRoles, or the import fails.

Contact us before you move a production install, and drill a restore first.

Restore drill​

Restore the backup into a temporary cluster regularly, at least after every cutover and every operator upgrade. The drill creates a one-instance cluster from the backup, optionally at a point in time, waits until it is ready, checks every database and table against the live primary, reports how long it took, and deletes it. The drill cluster never writes into the live backup archive.

Coming

Recovering in place into the live cluster name is a manual step in this release.

Synchronous replication​

By default replication is asynchronous: the primary confirms a commit once its own log is flushed, and a replica receives it a moment later. If the primary dies in that moment, the promoted replica lacks the last few confirmed transactions.

With postgresHA.synchronous.enabled: true, a commit returns only once a set number of standbys have flushed it, so the replica that is promoted holds every write a client was told succeeded.

SettingMeaning
method: anyQuorum: any number of the standbys. The usual choice.
method: firstPriority: the first number in the list.
numberStandbys each commit waits for, from 1 to instances - 1.
dataDurability: requiredNever confirm a commit with fewer than number standbys. Writes block until enough are back.
dataDurability: preferredOnly healthy standbys count, down to none. Writes never block on a missing standby, and the guarantee is suspended until one streams again.
failoverQuorum: trueBefore a failover, wait rather than promote a replica that may lack a synchronous commit. Matters from 3 instances.

The trade-off:

  • Latency. Every write waits for one network round trip to a standby and its flush, typically 1 to 3 ms inside one region. Reads are unchanged.
  • Availability. With required and no spare standby (2 instances with number: 1, or 3 with 2), every rolling update, drain or restart of a replica stops all writes. The chart refuses that combination unless you set allowWriteStalls: true. With preferred, those same events silently suspend the guarantee until the replica streams again; the PostgresReplicaNotStreaming alert is the signal.
  • Failover speed is unchanged. Commits that were waiting when the primary died fail on the client and may or may not have landed. What changes is that no confirmed commit can be missing.
InstancesRecommendedSurvives
2any, 1, preferredA replica outage without stalling writes, and a primary loss with no data loss while the replica was streaming.
3 (default)any, 1, required, with failoverQuorumAny one instance with no data loss and no write stall.
5any, 2, required, with failoverQuorumAny two instances.

Turning it on or off changes only the replication setting: PostgreSQL reloads it, with no restart and no new pod.

Failover drill​

A failover drill proves that a failover loses no confirmed write. It checks the cluster is ready with a standby in sync, writes numbered rows through the primary from inside a replica, deletes the primary's pod (by default as abruptly as a crash), waits for the new primary, and checks that every confirmed row is on it and, once the old primary rejoins, on every instance. It reports how long writes were interrupted, and fails on any lost confirmed write. Its table lives in the postgres database only and is dropped afterwards.

Every client loses its connections during the drill, so run it in a maintenance window, after you turn synchronous replication on or change it, and after operator upgrades. Contact us for the drill script and its options.

Database alerts​

With postgresHA.monitoring.prometheusRule, the chart adds these alerts:

AlertFires whenSeverity
PostgresInstanceDownFewer instances answer than configured, for 5 minutescritical
PostgresNoPrimaryNot exactly one primary, for 2 minutescritical
PostgresReplicaNotStreamingThe primary streams to fewer replicas than expected, for 10 minuteswarning
PostgresReplicationLagA replica is more than 60 seconds behind, for 5 minuteswarning
PostgresWalArchivingFailingWAL archiving is failing, for 10 minutescritical
PostgresWalArchivingStalledThe primary archived nothing for an hourwarning
PostgresConnectionsNearLimitConnections above 80% of the maximum, for 10 minuteswarning
PostgresVolumeNearlyFullA volume under 15% free, for 15 minuteswarning

Both the PodMonitor and the alerts need the Prometheus operator's CRDs.

Replicas of the other services​

availability.replicas runs the stateless services more than once: api, web, modelGateway, toolExecution, ragIngestion, audit, opa, orchestrationWorker, realtimeAgentWorker and eval. Each with two or more replicas gets a PodDisruptionBudget, so a node drain takes them down one pod at a time. Keycloak, Temporal, LiveKit, the deployment controller and the data stores other than PostgreSQL run as one replica.

Two replicas double each service's resource requests. A few per-user rate limits, such as those of channels and anonymous chat, are counted per API replica.

Scaling​

To give Orchestrate more room:

  • Add replicas of the stateless services, as above.
  • Raise a resource tier. Each component uses one of four tiers, resources.small, resources.medium, resources.jvm and resources.ml_inference. See Configuration.
  • Size models separately. Each model deployment has its own resource profile, which sets its CPU, memory and replica count.
  • Grow storage. Resize a data volume's PersistentVolumeClaim directly, if your StorageClass allows expansion. See Configuration.
  • Add nodes. Make sure your nodes have room for the platform, every model and every code agent you deploy.

Contact us for help sizing a larger deployment.

After a restart: unseal OpenBao​

The bundled OpenBao keeps its encryption key only in memory. After any restart of its pod, such as a node drain, a cluster upgrade, a node reboot or an upgrade that moves OpenBao to a new version, it starts sealed, unless it unseals itself from your KMS or HSM (auto-unseal, which we recommend for production; see Requirements). Orchestrate keeps running, but connector tool calls, cluster registration, code agents, channels and external model providers fail until someone unseals it.

Check it after every node drain, cluster upgrade or reboot, or better, from your monitoring:

./voisctl openbao status --kube-context <your-context>

It says whether OpenBao is sealed, and exits with code 2 when it is, so you can use it in a monitoring check.

To unseal, use the keys file from the install. Bring it back from your safe, unseal, and put it away again:

./voisctl openbao unseal --kube-context <your-context> --keys ./openbao-init.json

If the keys are held by several people, each types their own key instead, until 3 have been entered:

./voisctl openbao unseal --kube-context <your-context> --prompt

voisctl is in the release bundle's tools/ directory. With auto-unseal from a key management service or HSM, OpenBao unseals itself and none of this is needed. See Install.

Restart a component​

kubectl --context <your-context> -n vois-orchestrate rollout restart deployment/<name>

A few things to know:

  • LiveKit is replaced, not rolled, so voice is down for everyone until the new pod is ready. After LiveKit restarts or moves to another node, restart realtime-agent-worker too.
  • The License Service reads the licence again every 10 minutes, so a renewed licence doesn't need a restart.
  • Keycloak keeps users and sessions in PostgreSQL, so a restart doesn't sign people out.

Troubleshooting​

Install and images​

SymptomLikely causeWhat to do
Pods stuck in ImagePullBackOff, with an error that names your registryThe registry project is private and the pod has no pull credential, or the nodes don't trust your registry's certificate.Create the pull Secret and list it in imagePullSecrets. For model predictors, use modelRegistry.imagePullSecrets. Make sure every node trusts your registry's CA.
A pod tries to pull an image from a public registryOne of the third-party image values still has its public name.voisctl points every image at your registry by digest: run voisctl upgrade --force with the bundle you run. If you installed with plain Helm, set every third-party image value yourself first.
voisctl install stops at the post-install step with Permission denied on the CA fileThe --registry-ca file can't be read by the user running voisctl.Pass a readable copy of the CA and run the same command again. It carries on where it stopped.
voisctl install stops part-way for any other reasonSee the error it prints.Fix the cause and run the same command again. See If the install fails.
Model deployments stay in ImagePullBackOff, naming the Harbor hostThe nodes don't trust Harbor's certificate, or Harbor got a new one in an upgrade.Repeat Let the nodes trust Harbor.
voisctl verify reports a signature or image errorThe bundle is damaged or isn't the one we signed.Don't install it. Ask us for another copy.
voisctl upgrade refuses to downgradeThe bundle is older than the installed version.Use the right bundle. To go back a version, see Roll back.
helm upgrade fails after you changed a storageSizeKubernetes doesn't allow a StatefulSet's volume size to change.Put the old size back in your values file, and resize the PersistentVolumeClaim directly.
The chart refuses to render, saying it won't expose a componentYou listed a component that can never be exposed.Remove it from exposure.components. See Configuration.

Sign-in and console​

SymptomLikely causeWhat to do
Console pages load but every panel is emptyThe browser can't reach the API, or the API rejects the console's origin.Expose api along with web, and make sure the hostnames in exposure.components are the ones people's browsers use. If you set web.apiUrl or web.nextAuthUrl, check they match.
Everyone is signed out after each upgradeweb.nextAuthSecret is empty, so a new one is generated on every upgrade.Set web.nextAuthSecret in your values file.
Sign-in fails for a new userThe user's password is marked temporary, or another required action is pending in Keycloak.In Keycloak, set the password with Temporary off, and clear the user's required actions.
Open Policy Agent isn't readyIt waits for the API, because it loads custom policies from it.Check that the API is ready. Then check kubectl logs deployment/opa.
A model call fails with Denied by policyA policy denied the call.Check the rules under Governance in the console.
A model call fails with HTTP 422 and the code guardrail_refusedThe guardrails refused the prompt, for example because it held data they block.This is by design. The response explains what was refused.
A call fails with HTTP 422 and the code prompt_injection_refusedThe prompt-injection detector blocked it. Block is the default.Check the report. To flag instead of block for an agent, use a guardrail profile. See Guardrail profiles.
Sign-in goes to localhost:8180Keycloak isn't exposed, so the browser is sent to the port-forward address.Expose keycloak with the console. See Configuration.
Sign-in fails for everyone after an upgradeThe realm wasn't brought up to date, or keycloak.adminUser and keycloak.adminPassword no longer match Keycloak's administrator.Check the keycloak-realm-sync job's log.
A user is refused with a message about two-factor sign-inTwo-factor sign-in is required for them and they have no authenticator.They follow the console's banner to set one up.
Calls fail with 429 and budget_exceeded or rate_limit_exceededA budget or rate limit was reached.See Usage controls.

Models​

SymptomLikely causeWhat to do
No route for model ...The model isn't a static route, and there is no ready catalog model with that name.Deploy the model from the Models page, or add a static route in modelGateway.selfHostedModels. Check the name matches exactly.
Could not reach ... the model is in the catalog but nothing is serving itThe model's predictor isn't running.Check the model's InferenceService: kubectl --context <your-context> -n vois-orchestrate get inferenceservices.
A request that names no model fails with a message about external providersRequests must name a model. External providers are off.Send a model in every request.
Model builds fail with MODEL_REGISTRY_URL is not setmodelRegistry.url is empty.Set it. See Configuration.
A model build job never startsThe harbor-push-credentials or harbor-ca-cert Secret is missing.Create both. See Requirements.
A model build fails while downloadingThe build needs internet access for base images and packages.See Air-gapped networks.
A model deployment stops while verifying the image, with a TLS errorThe deployment controller can't verify your registry's certificate.Set modelRegistry.apiUrl. See Configuration.
Workers fail to call the Model Gateway with an authentication errorA worker apiKey value doesn't start with vcpk_.Give it the vcpk_ prefix and upgrade. The API rotates the stored key on start-up. See Configuration.

Connectors and voice​

SymptomLikely causeWhat to do
Connector tool calls or cluster registration fail with an error that names VaultOpenBao is sealed after a restart, or, with your own Vault, it isn't at the expected address or its roles are missing.Run voisctl openbao status and unseal it. With your own Vault, set it up as in Requirements.
The browser can't start a voice sessionLiveKit isn't exposed, or browsers can't reach its media ports.Expose LiveKit on its own hostname, and open 7881/TCP and 7882/UDP to the LiveKit node. See Configuration.
Voice sessions drop after a quiet minuteYour ingress closes idle connections.Set long proxy timeouts on LiveKit's Ingress. The chart sets them for ingress-nginx.
Voice stops working after LiveKit moved to another nodeThe voice worker lost its connection.Restart realtime-agent-worker.
A channel can't be connected, with a message naming CHANNELS_OUTBOUND_HOSTSThe vendor's hosts aren't allowed.Add them to channels.outboundHosts. See Channels.
A code agent deploy is refused, saying NetworkPolicy isn't enforcedagents.networkPolicy.enforced isn't set.Install a NetworkPolicy engine and set it. See Code agents.
A code agent build fails at the install stage, naming a packageThe package isn't in the configured package index, or the index can't be reached.Add it to your mirror or with voisctl wheelhouse add, or list the index under agents.build.egress.
A webhook URL is refusedIts host isn't in agentApi.webhookAllowedHosts.Add the host.
A carrier's calls don't arriveThe gateway's SIP and RTP ports aren't reachable from the carrier, or the source isn't on a trunk.Check sipGateway settings and your firewall. See Phone calls (SIP).

Licence​

SymptomLikely causeWhat to do
The Licence page shows unlicensed after an upgradeNo licence is uploaded or configured.Upload it, or set license.secretName. See Licence.
The Licence page shows invalid or expiredThe file is wrong, or the licence has expired past its grace period.Read detail, and see Licence.
Admin actions fail with 402The licence has lapsed, or a seat or code-agent limit is reached.Renew the licence, or stay under the limit.
An action fails with 403 naming a licence featureThe licence doesn't include it.Contact us.

If you can't solve a problem, contact us with the output of kubectl get pods, the logs of the failing component, and the version from GET /version.

Was this page helpful?