Litmus Unify Upgrade Procedure: 2.0.2 to 2.0.7
Use this procedure to upgrade Litmus Unify on Kubernetes from 2.0.2 to 2.0.7. Your NATS and PostgreSQL data are preserved.
MQTT clients are offline from step 1 through step 6, so schedule a maintenance window before you start.
Before you begin
- Schedule a maintenance window and notify the owners of your MQTT clients.
- Confirm shell access with kubectl and helm. On K3s, set KUBECONFIG=/etc/rancher/k3s/k3s.yaml.
- Check that the pull secret litmusautomation-uns-centralportal-pull-secret exists in your namespace.
- Make sure you can reach quay.io/litmusautomation/charts/litmus-uns.
- Take a current cluster backup or volume snapshot.
- Wait until the NATS stream pending message count is at or near 0.
To record the baseline:
Run the following commands, and keep the output to compare after the upgrade:
helm list -n <namespace> # expect 2.0.2
kubectl get pods -n <namespace> # all healthy
kubectl get pvc -n <namespace> # must be unchanged after the upgradeUpgrade Litmus Unify
Complete all steps in order.
To upgrade Litmus Unify from 2.0.2 to 2.0.7:
- Disable every MQTT account. See Accounts. This disconnects clients cleanly and prevents them from reconnecting during the upgrade.
- Stop the MQTT broker:
kubectl scale statefulset mqtt -n <namespace> --replicas=0Wait until no mqtt-* pods remain. The brokers shut down gracefully and flush their in-flight state to NATS.
- Delete the MQTT StatefulSet, and keep its data:
kubectl -n <namespace> delete statefulset mqtt --cascade=orphanVersion 2.0.7 changes StatefulSet fields that Helm can't update in place. The --cascade=orphan option removes only the StatefulSet object. MQTT has no persistent volume claims (PVCs), and all data is in NATS and PostgreSQL, so no data is lost.
- Upgrade to 2.0.7:
helm upgrade uns oci://quay.io/litmusautomation/charts/litmus-uns --version=2.0.7 --namespace <namespace> --set "global.image.pullSecretNames[0]=litmusautomation-uns-centralportal-pull-secret"- Wait until all three NATS pods are Running and 2/2 ready, with stable restart counts:
kubectl get pods -n <namespace> -w | grep nats- Wait until mqtt-0, mqtt-1, and mqtt-2 are Running and 1/1 ready, with stable restart counts:
kubectl get pods -n <namespace> -wThe upgrade updates all components and recreates the MQTT StatefulSet, which starts the brokers on 2.0.7. The brokers need a complete NATS JetStream cluster.
Note: When the MQTT pods start, they migrate the NATS database. The migration must run with no incoming messages. This is why you drain the stream before you start and disable MQTT accounts in step 1.
- When all NATS and MQTT pods are healthy, run the key-value replica fix:
./fix-kv-replica.shThis corrects the NATS key-value replica configuration before client traffic resumes.
- Enable the MQTT accounts again. Clients can connect once the broker cluster is healthy.
Important: After you enable the accounts, keep monitoring client traffic and the NATS stream pending count. Don't close the maintenance window until the client reconnect burst has cleared and traffic is back to normal levels.
Verify the upgrade
Run the following commands, and compare the output with your baseline:
helm list -n <namespace>
kubectl get pods -n <namespace>
kubectl get statefulset mqtt -n <namespace>
kubectl get pvc -n <namespace>The upgrade is complete when:
- The release is at 2.0.7, with the status deployed.
- All pods in the namespace are Running, with no restart loops.
- The mqtt StatefulSet is ready.
- The NATS and PostgreSQL PVCs match the baseline, and all are Bound.
- Clients have reconnected, and data flows end to end.
Troubleshooting
- Don't delete the NATS or PostgreSQL PVCs.
- If step 4 fails, keep the MQTT accounts disabled. Capture the output of helm history uns -n <namespace>, and kubectl describe pod and the logs for the failing pod. Then contact Litmus Support.