Upgrade, rollback, and backup
Day-to-day operations after installation.
Start with this table: four kinds of change, four paths
The Bundle is not a "source of truth for everything." It does one thing: converge the cluster to "these releases all exist." So do not assume that "edit the YAML and apply it" works for every kind of change:
| What you want to change | Where to do it | Why nothing else works |
|---|---|---|
| Swap the image (within the same major version, retag and push) | Console "minor version upgrade" | Changing only the Bundle version and reapplying is a no-op; see below |
| Cross a major version (v3 → v4) | Bundle: change releaseVersion and reapply |
A major version must create a new Deployment; the operator only recognizes this Bundle path |
| A minor upgrade fails; return to the previous ReplicaSet | Console "rollback" | Only takes effect within the same release |
| A major upgrade fails; route traffic back to the old release | Console "downgrade" | The Bundle refuses to go backwards; see below |
| Change an environment variable / mounted file | Console "reconfigure" | The Bundle writes these two only on first install and major version upgrades |
"Rollback" and "downgrade" are two different things; do not mix them up: the former swaps the image inside the same Deployment, the latter swaps the entire Deployment (major version). Look at whether the Deployment name changed and you can tell.
Each of the last two has its own section below. kubectl can do the same thing through another
entry point (kubectl set image, etc.), but kubectl bypasses the record; the console writes
every change into change-cause.
Version identity and image tag
A Bundle pack has two different fields; do not mix them up when upgrading:
versionis the image tag and may contain dots, for examplev3.1.releaseVersionis the version identity, shaped likev3, and determines the Deployment name (<app>-v3), theversion/releaselabels, and the Istio subset.
You only need to write releaseVersion when the two differ; when it is empty it is inferred by
truncating at the first dot (v3.1 → v3). Dots cannot go into an Istio subset, so the identity
must be a DNS label. If an existing Bundle lacks this field, the import infers it with the rule
above; you do not need to add it by hand.
For a major version upgrade (v1 → v2), write the image tag as v2 or v2.x directly. Do not
use a value like v1.1 that "looks like a minor version" to represent a new major version -
identity and tag are two different things.
Upgrade: swap the image within the same major version
Rebuild and push the tag for the same major version (for example push v3.1 to v3.1 again, or
push v3.2), then run "minor version upgrade" for each service in the console.
- This is an in-place rolling update: only the container image is replaced; the Deployment
name,
versionlabel, and Istio subset all stay put. Old and new Pods therefore share the same subset during the rollout, traffic transitions naturally, and no full release is needed. - You can upgrade only one service at a time, in dependency order: Themis → Rhea → Thia → the rest.
⚠️
kubectl apply -f bundle.yamldoes not do this. The operator's convergence rule is "act only when the target release's Deployment does not exist." The Deployment for the same major version is always there, so the whole thing is skipped - apply succeeds, nothing changes, and the logs report no error. Changing onlyversionand reapplying falls in this category.This is deliberate: the Bundle manages "which versions are running," not "which image is running inside a given version."
Acceptance check: kubectl get deploy <app>-v3 -o jsonpath='{.spec.template.spec.containers[0].image}'
shows the new tag; kubectl rollout status deploy/<app>-v3 converges.
Upgrade: across major versions
Change this pack in the Bundle to:
- app: thia-web version: v4 # image tag releaseVersion: v4 # version identity (optional, inferred from the tag)kubectl apply -f bundle.yaml.The operator does three things in order: create Deployment
<app>-v4→ wait until it is ready → full release (Promote).A full release converges the Service, VirtualService, and DestinationRule onto the single
v4subset, then scales the old version<app>-v3to 0 replicas.Observe:
kubectl get deploy -n <your namespace> -l app=thia-web kubectl get pods -n <your namespace> -l version=v4
When reading the Deployment list, note: the old version does not disappear; it is 0/0.
This is not leftover debris; it is the vehicle for rollback (see the next section). Seeing both
thia-web-v3 and thia-web-v4 in kubectl get deploy is normal.
⚠️ A major version upgrade rewrites environment variables and mounted files from the values in the Bundle. If this service had its env changed in production through the console's "reconfigure," those changes are not in the Bundle and will be overwritten by the Bundle values pushed up during the upgrade. Before upgrading, copy the production values back into the Bundle, or merge once more in the console after upgrading.
The acceptance check is the same as the "first-start acceptance check" of the first installation; run at least through item 4 (you can send one conversation), and once it is fine, run "delete version" on the old version - that clears the Deployment and the routing rules together, and once deleted, the downgrade path is gone.
Rolling back
Two paths; choose by the scope of the failure.
When a minor upgrade goes wrong: rollback
In the console's version list, run "rollback" on that release; the default target is the previous version (you can also pick a specific revision).
- It swaps the Deployment's Pod template back to the target ReplicaSet's (image, env, and mounts go back together);
- The Deployment name does not change, so traffic does not need to switch -
<app>-v3is still that subset.
This is the equivalent of kubectl rollout undo. It only works within the same major
version: after you switch major versions, the previous ReplicaSet belongs to another
Deployment, so use the downgrade below.
When a major upgrade goes wrong: downgrade
In the version list, run "downgrade" on the old release (v3). What happens:
- The replica count of
v3's Deployment is raised to the current active version's replica count; - The Service's
versionlabel, the VirtualService, and the DestinationRule point back tov3; - The replaced
v4is scaled to 0 replicas (the Deployment is still kept).
In other words, a downgrade does not rebuild, pull images, or touch the database - the old version is still in the cluster, so it takes seconds.
⚠️ You cannot roll back by lowering
releaseVersionand reapplying the Bundle. Neither outcome is a rollback: if the old version's Deployment still exists (0/0), the operator simply skips it; if the old version has already been deleted from the cluster, the operator rejects it withUPGRADE_MAJOR_INVALID_VERSION- it requires the new identity to be greater than the current one. This prevents "apply an old Bundle," which looks harmless but actually mixes the semantics of "rollback" and "deploy" together.
After a downgrade, reapplying the same Bundle is a no-op (<app>-v4 already exists), and
traffic will not bounce back to the new version by itself; to return to the new version, run
"full release" on v4 in the console again. This means the Bundle file and the cluster's actual
routing may be temporarily inconsistent; do not use the Bundle for other changes while rolling
back.
Database rollback
An image rollback is not a schema rollback. Migration scripts only go forward: roll a service back to v3 and it faces the table structures that v4 changed.
There is only one criterion - did v4 run a destructive migration (drop column / change type / rename):
- No (only added tables, columns, indexes): the rollback is safe; old code ignores the new structures;
- Yes: you cannot roll back the application alone; you must restore the database together with the backup taken before the change. First take a current backup as described in the next section, then restore from the backup.
Backup
Only three things need backing up, and the first two cannot be "rebuilt":
| Object | Contents | What happens if lost |
|---|---|---|
| PostgreSQL | Users, credits ledger, sessions and messages (including attachments as base64), Agent definitions, publish records | The data is gone |
| Secret | JWT key pair, session signing key, SMTP and payment credentials, DATABASE_URL |
The data is still there, but everyone is signed out and external services must be reconfigured |
| Bundle file | Which version is running, which env are configured | The cluster is still running, but "rebuild exactly as it was" has no reference |
No backup needed: images (they are in the image registry), the cluster structure (the Bundle
describes it), user-uploaded attachments (stored in the database as base64 with the messages, so
they are already in PostgreSQL), and tethys-shipping's SQLite (without a PVC it lives and dies
with the Pod anyway; see the item in manual deployment).
Database
Multiple services share one database, so dump the whole database once:
pg_dump -Fc -h <database host> -U <username> -d <database name> -f backup-$(date +%Y%m%d).dump
Restore:
pg_restore -h <host> -U <username> -d <database name> --clean --if-exists backup-YYYYMMDD.dump
Stop the services before restoring, otherwise running services will keep writing to a
half-restored database. The smallest outage scope is to scale all services that write to the
database to 0 - thia-agent (sessions and messages, writes continuously), themis-auth
(signup and sign-in), rhea-payment (ledger); scale them back after the restore. Read-only
services can keep running.
Secret
kubectl get secret -n <your namespace> -o json > secrets-backup-$(date +%Y%m%d).json
⚠️ This file is plaintext (base64 is not encryption) and is equivalent to the cluster's keys. Store it in a password manager or encrypted volume; do not commit it to git and do not send it by email.
Restore order
At first installation it is "Secret before Bundle"; the same applies to restore:
- Secret - the Secrets referenced by the Bundle must exist, otherwise Pods get stuck in
CreateContainerConfigError; - Database - restore before the applications come up, so old and new versions do not interleave writes;
- Bundle -
kubectl apply; the operator creates the Deployment / Service back.
If any step fails, go to the corresponding symptom in troubleshooting.