Backup and restore

r5d.chat state spans a database, two secrets, the shared control-plane artifact volume, and one private workspace volume per user. They must be treated as a single recovery set, because restoring one without the others produces a broken installation.

The recovery set

required

Users, sessions, conversations, messages, encrypted provider and MCP configuration, workspace records, branding, and audit history.

required

Decrypts provider API keys, custom headers, MCP headers, OIDC client secrets, and workspace identity tokens. Without it, those rows are unreadable ciphertext.

required

Signs sessions. Restoring without it forces every user to sign in again, which is survivable but disruptive.

required

Remote project/chat artifact bytes. PostgreSQL stores their metadata and authorization but not their contents.

Each user’s private home directory. Not recoverable from the database.

The GHCR pull secret and any AWR_* values needed to bring the installation back up.

A database backup on its own is not sufficient. Restoring it with a different SETTINGS_ENCRYPTION_KEY leaves every user’s provider configuration permanently unusable, and they must re-enter their credentials.

Back up the database

For the bundled PostgreSQL, take a logical dump:

POD="$(kubectl -n r5d-chat get pod \
  -l app.kubernetes.io/name=r5d-chat-postgresql \
  -o jsonpath='{.items[0].metadata.name}')"

kubectl -n r5d-chat exec "$POD" -- \
  pg_dump -U r5d_chat -d r5d_chat --format=custom \
  > "r5d-chat-$(date -u +%Y%m%d%H%M%S).dump"

Verify the dump is non-trivial and readable before trusting it:

pg_restore --list r5d-chat-*.dump | head

For an external managed database, use the provider’s snapshot mechanism and its documented restore procedure.

Back up the secrets

Export the Secret and store it in a secrets manager, not in Git:

kubectl -n r5d-chat get secret r5d-chat-secrets -o yaml \
  > r5d-chat-secrets.yaml

This file contains base64-encoded plaintext credentials. It is as sensitive as the database itself. Encrypt it at rest.

Back up artifact storage

The chart’s retained artifact claim is named <release>-r5d-chat-artifacts, uses ReadWriteOnce, and mounts whole at /var/lib/r5d-chat/artifacts. Snapshot it at the same recovery point as the PostgreSQL dump. If CSI snapshots are unavailable, copy the mounted directory from the control-plane pod:

POD="$(kubectl -n r5d-chat get pod \
  -l app.kubernetes.io/name=r5d-chat \
  -o jsonpath='{.items[0].metadata.name}')"
kubectl -n r5d-chat exec "$POD" -- \
  tar czf - -C /var/lib/r5d-chat/artifacts . > artifacts-backup.tgz

A database backup alone is insufficient: mismatched metadata and bytes can produce missing blobs or opaque unreferenced files.

Back up workspace volumes

Use your CSI driver’s snapshot support. With a VolumeSnapshotClass available:

kubectl -n r5d-chat get pvc \
  -l app.kubernetes.io/name=r5d-chat-workspace \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' |
while read -r pvc; do
  cat <<YAML | kubectl apply -f -
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: ${pvc}-$(date -u +%Y%m%d)
  namespace: r5d-chat
spec:
  volumeSnapshotClassName: csi-rbdplugin-snapclass
  source:
    persistentVolumeClaimName: ${pvc}
YAML
done

Snapshots of a live ReadWriteOnce volume are crash-consistent. For a consistent point-in-time copy, scale the workspace to zero first. Alternatively, copy files out of a running workspace:

kubectl -n r5d-chat exec "$POD" -- tar czf - -C /home/r5d . > home-backup.tgz

Restore

Prepare an empty namespace

Restore into a clean namespace rather than on top of a damaged one, so you can compare before cutting over.

kubectl create namespace r5d-chat-restore

Restore the original secrets first

The encryption key must be in place before the application reads any encrypted row.

sed 's/namespace: r5d-chat/namespace: r5d-chat-restore/' \
  r5d-chat-secrets.yaml | kubectl apply -f -

Restore the database

Install the chart so the database exists, then load the dump:

kubectl -n r5d-chat-restore exec -i "$PG_POD" -- \
  pg_restore -U r5d_chat -d r5d_chat --clean --if-exists \
  < r5d-chat-backup.dump

Restore artifact storage

Recreate the artifact PVC from its snapshot using the release-derived name before starting the control plane. If restoring a tar archive, use a temporary helper pod that mounts the whole claim at a dedicated directory; never use subPath.

Restore workspace volumes

Recreate each PVC from its snapshot using the same name as the original. Names are derived from the user ID hash, so a renamed PVC will not be found.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: r5d-<hash>
  namespace: r5d-chat-restore
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: rook-ceph-block
  resources:
    requests:
      storage: 10Gi
  dataSource:
    name: r5d-<hash>-20250219
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

Deploy the matching application version

Use the chart version that matches the backup. A newer version may expect a schema the restored database does not have; run the upgrade afterwards so migrations apply in order.

Validate the restore

Do not declare success on a green rollout alone.

Authentication

Decryption

Open Settings → Providers. Providers should list correctly and Test connection should succeed. Failure here means the encryption key does not match the database.

Conversation history

Open an older conversation and confirm messages and tool results render.

Workspace identity and private files

Start a workspace and browse the home directory. Private files should match the snapshot. Recoll indexes can be rebuilt from the files if search is stale.

Artifacts

Browse and download an older artifact through the application, then run r5dchat artifacts list from a workspace shell in an authorized chat.

A real agent task

Run a task that writes a file and returns a download link, end to end.

Retention and deletion

helm uninstall is not a data purge. It removes chart-managed compute and routing, while these persist:

The bundled database’s StatefulSet PVC
The control-plane artifact PVC, which carries keep semantics
Every workspace PVC, which carries keep semantics
Database rows, encrypted credentials, and conversations if the database is external

Before uninstalling, take an inventory and a final backup:

kubectl -n r5d-chat get pvc
helm uninstall r5d-chat --namespace r5d-chat
kubectl -n r5d-chat get all,pvc,secret

Deleting a namespace destroys everything still namespaced in it, including retained PVCs. Never use namespace deletion as a shortcut, and never as an accidental purge.

Test your restore procedure on a schedule. An untested backup is a hypothesis.

Get started

Install

Configure

Operate

Reference

Backup and restore

The recovery set

Back up the database

Back up the secrets

Back up artifact storage

Back up workspace volumes

Restore

Validate the restore

Retention and deletion

​The recovery set

​Back up the database

​Back up the secrets

​Back up artifact storage

​Back up workspace volumes

​Restore

​Validate the restore

​Retention and deletion

The recovery set

Back up the database

Back up the secrets

Back up artifact storage

Back up workspace volumes

Restore

Validate the restore

Retention and deletion