Offsite backup (restic)

Offsite backups use modules/backup/restic-gdrive.nix on lea and vanessa. The repository is rclone:gdrive:backups/restic/<hostname>. Local borg stays until a restore test from this repo passes.

What is backed up

restic paths are the union of clan.core.state.<name>.folders.

• lea adds clan.core.state.home = /home/luxus.
• vanessa backs up the service states already declared (postgres dump, homebox, grocy, bambuddy, obico, hindsight). It does not add /home.
• HAOS disk /var/lib/libvirt/images/haos.qcow2 is not under any of those folders. Home Assistant keeps its own backups. The module adds that path to exclude only if a state folder covers it.

Live /home/luxus, not the newest btrbk snapshot

btrbk on lea and vanessa (services.btrbk.instances.home, daily, snapshot_dir = /home) writes sibling subvolumes /home/home.<timestamp>. restic does not follow symlinks, so a stable “latest” symlink would store the link and not the snapshot. Resolving the snapshot in backupPrepareCommand would mean a bind mount. That mount would also be the folder borg reads, because borg uses the same clan.core.state folders, and the mount is empty except while restic is running. /home/luxus is not the snapshot parent, so a live backup does not walk /home/home.*.

Database consistency is preBackupCommand (clan-core’s wrapper around preBackupScript). Postgres runs pg_dumpall into /var/backup/postgresql before the restic backup.

lea excludes, matched against the absolute path:

• /home/luxus/.local/share/Steam and /home/luxus/.steam
• /home/luxus/.config/superpowers/worktrees
• /home/luxus/.grok/worktrees
• /home/luxus/**/target, /home/luxus/**/node_modules, /home/luxus/.cache, /home/luxus/**/.cache, /home/luxus/**/.direnv, /home/luxus/**/result

--exclude-caches is also set. Declaring clan.core.state.home makes local borg back up /home/luxus too, and borg does not use these excludes. After the switch, stop borgbackup-job-vanessa.timer on lea if that extra data is not wanted. The client role is unchanged.

Schedule

Unit	When
restic-backups-gdrive.timer	daily, Persistent=true, RandomizedDelaySec=1h
restic-gdrive-prune.timer	weekly (Monday), same persistence and delay. forget --prune --keep-daily 7 --keep-weekly 5 --keep-monthly 12 --keep-yearly 2
restic-gdrive-check.timer	Thursday 00:00, same persistence and delay. restic check --read-data-subset=5%

Prune is its own unit because services.restic.backups.*.pruneOpts would prune after every backup. Check is its own unit for the same reason. Thursday avoids the Monday prune slot so the two jobs do not fight over the restic lock.

rclone options on every job: tpslimit=10, drive-chunk-size=64M, drive-use-trash=false.

A failed backup, prune, or check starts restic-gdrive-failure@%n.service, which writes one journal line. There is no other failure-notify mechanism in this repo.

1. Set the clan vars

No secret values belong in git. Declare them, then generate on the machine.

The rclone file is one shared var (share = true). The password is generated per host (share = false), same xkcdpass -n 6 -d - shape as clan’s borg repokey.

Create rclone.conf somewhere outside the repo. The remote name must be gdrive:

[gdrive]
type = drive
scope = drive
token = <paste from `rclone authorize "drive"`>

Then, from the repo root, with the clan admin key:

clan vars set lea rclone-gdrive-conf          # paste the whole file when prompted
clan vars generate lea --generator restic-gdrive-password
clan vars generate vanessa --generator restic-gdrive-password

rclone-gdrive-conf is shared, so setting it on lea is enough for vanessa. Clan writes age-encrypted files under vars/. Commit those the same way as the other vars. Do not commit the plaintext rclone.conf or the password.

Deploy with just update-lea and just update-vanessa (those evaluate the git index; stage first).

2. First snapshot

On the host, after the generation that contains this module is running:

sudo systemctl start restic-backups-gdrive.service
systemctl status restic-backups-gdrive.service
sudo restic-gdrive snapshots

initialize = true runs restic init when the repo has no config yet. The shell wrapper restic-gdrive has the same repository, password file, and rclone config as the unit.

3. restic check

sudo systemctl start restic-gdrive-check.service
systemctl status restic-gdrive-check.service

That is restic check --read-data-subset=5%. The weekly timer runs the same command.

4. Restore one file

Restore into an empty directory, then copy the file into place. Do not restore onto the live path.

sudo mkdir -p /tmp/restic-restore
sudo restic-gdrive restore latest --target /tmp/restic-restore --include /home/luxus/path/to/file
sudo ls -l /tmp/restic-restore/home/luxus/path/to/file

Snapshot paths are the absolute paths restic read. On vanessa, use a path from sudo restic-gdrive snapshots / sudo restic-gdrive ls latest.

5. Restore a directory

sudo mkdir -p /tmp/restic-restore-dir
sudo restic-gdrive restore latest --target /tmp/restic-restore-dir --include /home/luxus/projects/some-dir

For a service dir, stop that service first, restore beside it, and swap only after the files look right. Postgres state in the repo is the pg_dumpall file /var/backup/postgresql/dumpall.sql, not the live cluster directory.

6. Forced-failure test

Confirm the template unit fires without touching the real repository:

sudo systemd-run --collect --unit=restic-gdrive-failtest \
  --property=OnFailure=restic-gdrive-failure@%n.service \
  /bin/false
systemctl status restic-gdrive-failure@restic-gdrive-failtest.service
journalctl -u restic-gdrive-failure@restic-gdrive-failtest.service --no-pager

The status should show the oneshot and a journal line restic-gdrive: restic-gdrive-failtest.service failed.

A real job failure (for example starting restic-gdrive-check.service before the first snapshot) triggers restic-gdrive-failure@restic-gdrive-check.service the same way.

7. Planned borg removal

borgbackup stays in modules/clan/services.nix: vanessa is the server, tags.nixos (lea and vanessa) are clients. Do not remove that role until all of the following are true:

1. A restic snapshot exists for lea and for vanessa (restic-gdrive snapshots).
2. A single-file restore and a directory restore from those snapshots have been checked (sections 4 and 5).
3. restic check --read-data-subset=5% has succeeded on both hosts.

Then delete the borgbackup instance, drop the local repos under vanessa /var/lib/borgbackup, and remove the “local-only, deprecated” notes in inventory.md and configuration-matrix.md.