Skip to main content

Upgrade VoisX Orchestrate

New releases of VoisX Orchestrate appear on the VoisX portal as signed release bundles, the same kind of bundle you installed from. You download the bundle, carry it into your network, check it, and apply it with voisctl. No step needs internet access. Sites that install online upgrade from the VoisX registry instead: see Upgrade online. See How you receive VoisX Orchestrate.

An upgrade keeps your data, your settings and your licence: projects, agents, conversations, models and their deployments are all still there afterwards. You don't need a new licence file for a new release. Upgrades only go forward: voisctl refuses to apply an older version.

voisctl needs only Python, Helm and kubectl. It doesn't need Docker or any other image tool. See Requirements.

Before you start​

  1. Check what you run now.

    helm list --kube-context <your-context> -n vois-orchestrate

    The APP VERSION column shows the installed version.

  2. Back up the databases. Take a fresh backup right before the upgrade, and keep it until you are happy with the new version. See Back up and restore. Without it you can't go back.

  3. Save your current values. The file contains every secret in clear text, so keep it somewhere safe:

    umask 077
    helm get values vois-orchestrate --kube-context <your-context> -n vois-orchestrate -o yaml > orchestrate-values-before-upgrade.yaml
  4. Keep the previous release's images in your registry until the upgrade is done. You need them if you roll back.

  5. Plan a maintenance window. Components restart during the upgrade. Voice sessions drop while LiveKit restarts.

1. Check the bundle​

Download the new release from the portal's Downloads: the bundle, SHA256SUMS and the release notes. Check the files and compare the release key fingerprint in the notes with the one we sent you separately:

sha256sum -c SHA256SUMS --ignore-missing

Copy the bundle into your network, then extract it once:

mkdir bundle-new
tar -xf vois-orchestrate-<version>-base-linux-amd64.tar -C bundle-new

Use the voisctl from the new bundle, at bundle-new/tools/voisctl. Each release's voisctl knows that release's bundle format. Check the signature, against the VoisX release key built into voisctl, and every image:

./voisctl verify --bundle ./bundle-new

Don't go further unless it ends with OK. A bundle that fails verification isn't the bundle we signed. Ask us for another copy.

To see what is inside without trusting it yet, run ./voisctl inspect --bundle ./bundle-new. inspect reads the unsigned copy of the manifest and doesn't check anything.

2. Import the images ahead of time (optional)​

Pushing several gigabytes of images is the slowest step. You can do it during the day and keep the maintenance window short:

./voisctl import --bundle ./bundle-new \
--registry registry.example.internal:5000 --project vois-orchestrate \
--registry-user <push-account> --registry-password <password>

import checks the bundle, pushes every image, and reads each digest back from your registry. Nothing in the cluster changes.

3. Rehearse the upgrade​

A dry run checks the bundle and the downgrade rule, and prints what Helm would do, without applying it. Nothing is pushed or changed:

./voisctl upgrade --bundle ./bundle-new \
--registry registry.example.internal:5000 \
--registry-user <push-account> --registry-password <password> \
--registry-ca ./site-ca.crt \
--kube-context <your-context> \
--with-prerequisites \
--model-registry-url harbor.example.internal \
--openbao-keys ./openbao-init.json \
--dry-run

Use the same registry, --profile (for example openshift, or single-node on an evaluation server) and --model-registry-url you installed with. --openbao-keys is the keys file from the install: bring it back from your safe for the upgrade, then put it away again. Without it the upgrade still runs, but can't check OpenBao's roles.

4. Upgrade​

Run the same command without --dry-run. voisctl stops at the first problem. In order, it:

  1. Checks the signature before it reads anything from the bundle as instructions.
  2. Re-hashes every image layer and the chart, so the files on disk are the ones that were signed.
  3. Refuses a downgrade. If the installed version is newer than the bundle, it stops before changing anything.
  4. Pushes every image into your registry and reads each digest back. If you already ran import, this goes quickly.
  5. Moves the add-ons forward, with --with-prerequisites: each add-on voisctl installed goes to the bundle's version. Add-ons you brought yourself are left alone.
  6. Runs the setup steps again: OpenBao unsealed and its roles checked, Harbor's projects, accounts and Secrets checked, the CA bundle rewritten. Anything already right is left as it is; a new release may add a role.
  7. Upgrades Orchestrate with the bundle's chart. It keeps the values your install already has, including the credentials voisctl install generated, adds any new defaults the release introduces, applies your -f files on top, pins every image to its digest, and waits for the rollout.
  8. Checks the result. It confirms each Orchestrate Deployment runs the signed digest, the database reached the schema revision the release expects, and the API reports the new version.

It ends with Upgraded to <version>.

The API applies database migrations itself when it starts. There is no separate database step.

Change settings in the same step​

Pass a values file with -f to change settings while you upgrade. Only what the file sets changes; every other value, including those from the install, stays as it was. To apply a settings change without a new release, run the upgrade with the bundle you already run and --force.

After the upgrade: Harbor's certificate​

If the upgrade moved Harbor to a new version, Harbor has a new certificate. voisctl writes it to ./harbor-ca.crt (or your --harbor-ca-out) again: repeat Let the nodes trust Harbor, or model deployments can't pull new images.

Options​

OptionDefaultWhat it does
--registryrequiredYour registry host, or host:port.
--projectvois-orchestrateYour registry project.
--registry-user, --registry-passwordnoneAn account that can push to the project.
--plain-httpoffTalk to the registry over plain HTTP.
--insecure-registryoffUse HTTPS without checking the registry's certificate. Every image is still checked against its digest.
--kube-contextnoneThe kubeconfig context for your cluster. Always set it.
--namespacevois-orchestrateThe release namespace.
--releasevois-orchestrateThe Helm release name.
-f, --valuesnoneA values file to apply in the same upgrade. Repeat for more files.
--timeout10mHow long Helm waits for the rollout.
--dry-runoffCheck and render only.
--forceoffRe-apply a version that is already installed, for example to apply a configuration change.
--no-reuse-valuesoffDon't carry your existing values forward. Use it only if -f supplies your entire configuration, or your secrets are reset.
--allow-downgradeoffSkip the downgrade check. See Roll back.
--no-wheelhouseoffDon't turn on the bundle's offline Python package index for code-agent builds. Use it if your builds use your own package mirror.
--with-prerequisitesoffMove the add-ons voisctl installed forward, and run the setup steps again.
--profilenoneThe profile you installed with, if any: openshift, eks, gke, or single-node on an evaluation server.
--model-registry-urlnoneThe address the nodes reach Harbor at.
--openbao-keysnoneThe OpenBao keys file from the install.
--registry-canoneYour registry's CA, if it is your own.
--onlineoffDon't apply the offline settings. Only for a site that reaches the internet.

Check the upgrade​

After voisctl finishes:

  1. Check every pod is Running and Ready:

    kubectl --context <your-context> -n vois-orchestrate get pods
  2. Sign in to the console and open a few pages.

  3. Open the Licence page and check the status. See Licence.

  4. Check that OpenBao is unsealed: ./voisctl openbao status --kube-context <your-context>. See After a reboot: unseal OpenBao.

Upgrade online from the VoisX registry​

If you installed from the VoisX registry, upgrade the same way, with the new version number and your registry credential file:

./voisctl upgrade --from-registry <registry> --version <version> --pull-secret ./<credential-file>.json \
--kube-context <your-context> --dry-run
./voisctl upgrade --from-registry <registry> --version <version> --pull-secret ./<credential-file>.json \
--kube-context <your-context>

If you mirror into your own registry, add --registry and your push account, as for the install. voisctl runs the same checks as from a bundle, and replaces the pull Secret with the credential you pass, so a renewed credential takes effect at the next upgrade.

Coming: online upgrades

The VoisX release registry is not open yet. Until it is, upgrade from the bundle as above.

If the upgrade fails​

voisctl names the step that failed.

  • Before the Helm upgrade (signature, image check, downgrade check, push): nothing in the cluster has changed. Fix the cause and run the command again.

  • During or after the Helm upgrade: look at the pods first.

    kubectl --context <your-context> -n vois-orchestrate get pods
    kubectl --context <your-context> -n vois-orchestrate logs deployment/api

    The API runs its migrations before it starts serving, so a migration problem shows in its log. If you can't fix the cause, roll back, and contact us with the voisctl output and the logs.

Roll back​

Coming: rollback with voisctl

A later release will take a database backup automatically before an upgrade and give you one command to go back. Until then, roll back by hand as below.

There is no automatic way back to an older version. Once the new API has started, it has migrated the database, and the older version can't run against the migrated database. That is why voisctl refuses downgrades.

To go back, restore the database backup you took before the upgrade, then roll back the Helm release.

Restoring loses recent changes

A restore puts every database back to the moment of the backup. Agents, runs, users, approvals and audit entries created after the backup are lost. Stop all Orchestrate services first, so nothing writes during the restore.

  1. Find the revision to go back to. It is the last one before the upgrade:

    helm history vois-orchestrate --kube-context <your-context> -n vois-orchestrate
  2. Stop the Orchestrate services. PostgreSQL keeps running:

    kubectl --context <your-context> -n vois-orchestrate scale deployment --replicas=0 \
    api audit deployment-controller eval keycloak license-service livekit model-gateway opa \
    orchestration-worker rag-ingestion realtime-agent-worker temporal temporal-ui tool-execution web
  3. Restore the databases from your pre-upgrade backup. See Restore PostgreSQL.

  4. Roll back the release:

    helm rollback vois-orchestrate <revision> --kube-context <your-context> -n vois-orchestrate --wait --timeout 15m

    The rollback brings back the previous chart, values and image digests.

  5. Start the services again:

    kubectl --context <your-context> -n vois-orchestrate scale deployment --replicas=1 \
    api audit deployment-controller eval keycloak license-service livekit model-gateway opa \
    orchestration-worker rag-ingestion realtime-agent-worker temporal temporal-ui tool-execution web
  6. Check the version and the pods:

    kubectl --context <your-context> -n vois-orchestrate get pods
    kubectl --context <your-context> -n vois-orchestrate exec deploy/api -- \
    python3 -c "import urllib.request; print(urllib.request.urlopen('http://127.0.0.1:8000/version').read().decode())"

Don't use --allow-downgrade to go back. It skips the safety check, but it can't make an older version understand a newer database.

Disk space​

voisctl accepts the bundle either as the .tar file or as an extracted directory. Given the .tar, it extracts it into a temporary directory first, so you need room for a second copy. Extract the bundle once and pass the directory, as above. Delete old extracted bundles once you no longer need them for a rollback.

Was this page helpful?