Rollback

How to recover when a rebuild breaks a machine. Covers NixOS generation management, fix-forward recovery, and redeploying a known-good commit to a remote Clan machine.

For boot-order / dual-boot questions (NixOS vs Windows, rEFInd, EFI entries) see Boot Management. This guide is about NixOS system generations, not OS selection.

Background

Every nh os switch / clan machines update creates a new system generation under /nix/var/nix/profiles/system. Old generations stay on disk (and in the boot menu) until garbage-collected, so a bad config is almost always recoverable.

• lea and vanessa boot with systemd-boot (modules/nixos/systemd-boot.nix); lea also exposes rEFInd as the top-level menu for dual-boot.
• Darwin hosts (emily, zoe) keep generations too, managed via darwin-rebuild.

NixOS: list / switch / rollback generations

List system generations (root profile):

sudo nix-env --list-generations --profile /nix/var/nix/profiles/system

Roll back to the previous generation and activate it now:

## Preferred on this machine: nh wraps nixos-rebuild
nh os switch . -- --rollback

## Equivalent raw command if nh is unavailable
sudo nixos-rebuild switch --rollback

Activate a previous generation only on next boot (safer — current system keeps running):

nh os boot . -- --rollback

Switch to a specific generation number (e.g. 142):

sudo nixos-rebuild switch --switch-generation 142

Roll back from the boot menu (when the system won’t boot)

1. Reboot. On lea, pick NixOS in rEFInd, then the systemd-boot menu appears.
2. systemd-boot lists the current generation plus older ones (NixOS … Generation N). Pick a known-good older generation and boot it.
3. Once booted, make it the default by fix-forwarding (below) or rolling back.

Fix-forward (preferred recovery)

Rolling back is a stopgap; the real fix is to correct the config and rebuild.

1. Boot a working generation (boot menu or --rollback).
2. Fix the offending Nix code in the repo.
3. Validate before switching (see Testing Strategy):

   just check                       # deadnix, statix, nixfmt --check, nix flake check
   nh os build .                    # build lea locally without activating

4. Apply:

   nh os switch .                   # local (lea)
   clan machines update <machine>   # remote (vanessa, emily, zoe)

Clan: redeploy a known-good commit to a remote machine

clan machines update builds from the flake’s current working tree, so to roll a remote machine back to a known-good state, check that state out first.

## Find a good commit
git log --oneline

## Option A: deploy a past commit without moving your branch
git stash                      # if you have local changes
git checkout <good-commit>
clan machines update vanessa   # builds + activates that commit on the target
git checkout -               # return to your branch when done

## Option B: revert the bad change properly, then deploy
git revert <bad-commit>
clan machines update vanessa

Notes:

• clan machines update <name> uses deploy.targetHost from flake.nix (root@vanessa.local, root@lea.local, luxus@emily.local, luxus@zoe.local).
• Equivalent to just update <name>.
• For a NixOS target you can also build/activate over SSH with nh: nh os switch '.#nixosConfigurations.vanessa' --target-host root@vanessa.local.
• Clan activates remotely with rollback safety, but if a deploy leaves a remote host unbootable, use the host’s boot menu (above) or out-of-band console.

Cleaning up generations

Free disk space by deleting old generations. This also removes boot entries.

## nh wrapper (runs collect-garbage across profiles)
nh clean all                   # delete-generations + gc, default keep policy
nh clean all --keep 5 --keep-since 7d

## Manual equivalents
sudo nix-env --delete-generations old --profile /nix/var/nix/profiles/system
sudo nix-env --delete-generations 30d --profile /nix/var/nix/profiles/system
sudo nix-collect-garbage -d    # delete ALL old generations + gc (aggressive)

After deleting system generations, refresh the boot menu:

sudo /run/current-system/bin/switch-to-configuration boot

Warning: never nix-collect-garbage -d the generation you may need to roll back to. Keep at least one or two known-good generations until the new one is proven.

Darwin (emily, zoe)

darwin-rebuild --list-generations
darwin-rebuild --rollback                 # previous generation
darwin-rebuild --switch-generation <N>

## Redeploy / fix-forward via Clan or nh
clan machines update emily
nh darwin switch '.#darwinConfigurations.emily'

Emergency checklist

Symptom	Action
New config boots but is broken	nh os boot . -- --rollback then fix-forward
System won’t boot	Boot an older generation from systemd-boot menu
Need a specific past state	nixos-rebuild switch --switch-generation <N>
Remote machine broken after deploy	git checkout <good-commit> + clan machines update <name>
Disk full, can’t rebuild	nh clean all --keep 3 then retry the build
Darwin broke after switch	darwin-rebuild --rollback

Common failure scenarios

• Boot loops / kernel panic after switch — boot the previous generation, then bisect the change (commonly kernel/NVIDIA changes on lea/vanessa).
• Secrets missing after deploy — see Troubleshooting; fix-forward with clan vars generate --machine <name> then redeploy.
• nix flake check passes but switch fails on target — usually a runtime/activation issue (services, mounts); roll back and inspect journalctl -b -1 / failed units, then fix-forward.