Upgrade and backup
Upgrade
- Saptiva AI publishes the new version (images and configuration) and agrees on a window with the operator.
- Full backup before any major or minor upgrade. No exceptions.
- The pinned version is changed in the deployment repository; the cluster agent applies the change layer by layer. A layer that fails partway through rolls back on its own.
- Smoke tests.
- If something fails, rollback: the repository goes back to the previous version and the cluster reapplies it. If the upgrade already migrated data, the rollback uses the backup from step 2.
Studio upgrades (the console) do not interrupt labs or running jobs. Lab platform upgrades may require stopping notebooks; this is announced in advance.
What is backed up
Four classes of state, with a maximum loss objective (RPO) of 24 hours:
| Class | What it contains | How |
|---|---|---|
| Platform databases | Clients, members, agents, workflows, runs, pipeline metadata | Daily base backup at 02:00 (cluster time) plus continuous transaction archiving, with 30 days of retention. At installation, a first backup is taken immediately, not the following night. |
| Artifact store | Pipeline artifacts and results | Daily copy of the object store |
| Declarative configuration | Labs, quotas, queues, policies | Lives in the deployment repository; the repository is the backup |
| User volumes | Notebooks, datasets, models of each lab | Snapshots of the Client's storage, as often as its system allows. It is the class that depends most on the hardware: agree on it at installation. |
Secrets (provider keys, certificates) are stored in the secrets store, outside the cluster backup, and are restored from there. The Secret not synced alert (Observability) is the one that warns you when that restoration is not happening.
What is not backed up
- A notebook's temporary disk: only the volumes. That is the reason for The notebook lost its files on restart.
- The platform images: they are mirrored again from the version package.
- Models downloaded to the catalog: they are downloaded again or loaded from the physical media. In air-gapped installations, keep the media.
Objectives
| Objective | Value | What it means for the Client |
|---|---|---|
| RPO (maximum data loss) | 24 hours | In the worst case, the last day's work that is not on a backed-up volume is lost. |
| RTO (recovery time) | 3 business days | For total loss of the cluster. A partial failure (a component, a node) is resolved in hours. |
If the Client needs a lower RPO for user volumes, a higher snapshot frequency is agreed on; the cost is storage space.
Recovery
Recovery time objective (RTO): 3 business days for total loss of the cluster. Order:
- Rebuild the base cluster and the requirements.
- Reapply the declarative configuration from the repository.
- Restore databases and the artifact store to the same point in time.
- Restore user volumes from snapshots.
- Smoke tests and a test lab.
The recovery drill is part of the handover. A backup whose restoration has not been tested does not count as disaster recovery. Saptiva AI runs the drill before go-live and repeats it with each major upgrade.