Troubleshooting

lea crashes / stability

Two failure classes showed up in journals + /var/log/crash-telemetry/telemetry.log. Do not treat them as one bug.

Class	What it looks like	Typical last telemetry
GPU / compositor reset	kwin graphics reset, noctalia guilty-context-reset, kwin SEGV with both libvulkan_radeon and libnvidia-eglcore	Cool CPU, GPU not at 4090 TDP, then dirty reboot
Silent power-cut under CPU	Journal just stops (no System Reboot). Often while nix is compiling.	load high, Tctl well under 90°C, GPU idle

Isolation suite (does not change config):

just lea-stability              # history: boots + telemetry gaps + unfinished tests
just lea-stability snapshot     # current CPU/RAM/GPUs/kwin maps
just lea-stability cpu 5        # CPU only
just lea-stability ram          # stress-ng --vm, 16G × 10 min (not memtester; user memlock is 8MB)
just lea-stability compile 5    # loop gcc -O2 (no nix-daemon)
just lea-stability nix-compile 5
just lea-stability gpu 5        # vkcube on NVIDIA ICD only
just lea-stability combo 5      # CPU+RAM (EXPO/IMC)
just lea-stability power 5      # CPU+NVIDIA (PSU)
just lea-stability igpu 2       # vkcube on Raphael iGPU — expected fail while amdgpu is blacklisted
just lea-stability memtest-hint

Run one load probe at a time. Each writes a fsynced TEST START to ~/.local/share/lea-stability/tests.log. If the box dies, reboot and just lea-stability history — an unfinished START is the probe that killed it.

Existing 1 Hz sensor logger: tail -40 /var/log/crash-telemetry/telemetry.log (Tctl ramp to ~90–95°C ⇒ thermal; flat then gap ⇒ electrical).

Hardware context: 9800X3D + RTX 4090, Corsair CMH64GX5M2B6000Z30 2×32G, ASUS ProArt X670E-CREATOR BIOS 3702, CachyOS kernel, nix max-jobs=8 cores=8. Raphael iGPU (amdgpu) is blacklisted. Memtest86+ is already a systemd-boot entry.

BIOS / cliff: lea stability (BIOS). Stop A/B. CPB-off + 4800 j16 dirty ~4 min (did not reproduce the 2026-08-22 pass). Chrome x.com SIGILL is separate (renderer Media, not a power-cut). Nix cores=8 is the compile seatbelt (j8 passed; j12+ dirty).

lea reboot: /oldroot/nix/store EBUSY (~1–2 min)

Normal systemctl reboot reaches exitrd, then fails to unmount the nested NixOS read-only store bind (boot.readOnlyNixStore, default on) at /oldroot/nix/store. systemd parks it under /run/shutdown/mounts/<random>, retries, logs “Cannot finalize remaining file systems, continuing”, then reboots. Data path is already remounted RO / synced. Watchdog “did not stop” is expected with the reboot watchdog armed.

Root shape (not a userspace process after kill-all): lea’s /nix is a btrfs @nix subvolume; /nix/store is an extra RO bind on top. Something still holds that leaf (almost certainly a kernel/driver mapping, NVIDIA being the remaining suspect). App services were ruled out. See #133.

Do not flip boot.readOnlyNixStore = false or add a lazy-umount helper from CI — that needs a reboot A/B on lea. Revisit if the delay grows past a few minutes, the FS looks dirty, or reboot needs a power cut.

vanessa HAOS “restarted but UI never comes back”

This is usually not a libvirt autostart miss. Check the host first.

What you see	Meaning
virsh list shows haos running, ping 192.168.178.186 works, :8123 refused	Guest kernel up, Home Assistant Core container dead. ha core start over SSH
mount on vanessa shows / ro	Host btrfs aborted a transaction (forced readonly) — qemu cannot write the qcow2. Reboot vanessa
Guest docker inspect homeassistant → Dead=true Exit=137 overlay2 read-only file system	Supervisor killed Core to restart it; overlay cleanup failed because /mnt/data is emergency_ro
Ping works, :8123 accepts TCP and returns no bytes (often :4357 and SSH :22222 too)	Guest wedge. virsh reboot does nothing if ACPI is ignored. virsh reset haos

Proven host sequence (2026-08-28): host BTRFS error … write time tree block corruption at 20:10 → / remounted ro → guest I/O error, dev vda → Core restart left the container dead. Live mount -o remount,rw / fails while btrfs is in state EA. Reboot vanessa only for that read-only host. Then ha core start if :8123 is still down (ssh -p 22222 root@192.168.178.186).

Proven guest wedges, host / still rw, corruption_errs unchanged at 674:

• 2026-10-10: kernel 6.18.52-haos RCU stall from 15:52. runc stuck in futex_wait on a spinlock, both vCPUs at ~100%. The kernel was tainted since a 2026-10-06 general protection fault in the Recorder ftruncate path. virsh reset recovered Core.
• 2026-10-18: logging stopped at 16:11 while the Supervisor updated add-on ha-mcp-dev (8.6.0.dev3016 → 8.7.0.dev3080) and unmounted its overlay. No oops in the journal. vCPUs busy, not pinned. SSH never finished the banner. virsh reset recovered the UI and add-ons.

Longer notes: docs/agents/operator-notes.md.

qcow2 on btrfs must be NOCOW (chattr +C on /var/lib/libvirt/images, then copy haos.qcow2 while the guest is stopped). services.btrfs.autoScrub on vanessa is weekly.

Deployment Issues

clan machines deploy fails with SSH errors

Error: Connection refused / Permission denied

• Verify the machine is reachable: ping <machine>.local
• Check SSH key is in lib/ssh-keys.nix and deployed via sshd instance
• Verify deploy.targetHost in flake.nix is correct
• For Darwin: ensure luxus@<machine>.local (not root)
• For NixOS: ensure root@<machine>.local

nix-daemon: not found when deploying Darwin from another host

remote-program=bash -lc 'exec nix-daemon --stdio'
--stdio: line 1: exec: nix-daemon: not found
Failed to upload sources

Clan uploads flake sources to the Darwin target before building. That SSH session runs nix-daemon by name, but Determinate Nix keeps it under /nix/var/nix/profiles/default/bin, which is often missing from non-interactive login shells.

Permanent fix: modules/darwin/nix-clan-deploy.nix adds that path via environment.systemPath. Apply it once on the Mac (chicken-and-egg: remote deploy cannot work until this is active):

## on zoe/emily, from a checkout of this flake
darwin-rebuild switch --flake .#zoe

One-time unblock (if you cannot rebuild yet): replace stale shell symlinks (e.g. from prior HM use) and put the daemon on PATH for bash -lc:

rm -f ~/.bash_profile ~/.profile ~/.bashrc
printf '%s\n' 'export PATH="/nix/var/nix/profiles/default/bin:$PATH"' > ~/.bash_profile

Then retry clan machines update zoe from lea.

Nix evaluation errors

Error: attribute 'X' missing

• Run nix flake check --no-build to identify the issue
• Check that all referenced modules exist (run scripts/validate-inventory.sh)
• Verify flake inputs are up to date: nix flake update

Secret generation prompts loop forever

clan vars generate --machine <machine>

• Check that the secret’s promptType is correct in lib/secret-registry.nix
• For hidden type: input is masked; press Enter after typing
• For line type: input is visible
• If a secret was already set, it won’t re-prompt unless the var is deleted first

MicroVM won’t start

• Check host prerequisites: systemctl status microvm@<vm>
• Verify prep service ran: systemctl status agent-vm-prep-<vm>
• Check secret bundle: ls /var/lib/microvm-bundles/<hostname>/secrets/
• Check project mount exists: ls /srv/agent-projects/<name>/
• Run scripts/verify-agent-vms.sh from emily; see Agent VM Architecture

Hjem Issues

Profile import conflicts

error: The option 'rum.programs.X' in '/path/to/A.nix' is already declared in '/path/to/B.nix'

• Don’t import overlapping app modules twice (e.g. git in both minimal-darwin.nix and a separate git.nix)
• Choose one base profile per machine, then layer extras on top
• Profile hierarchy: minimal-darwin → base → development → platform-specific

Package build failures on Darwin

• Some packages don’t support aarch64-darwin — check lib.optionals (!pkgs.stdenv.hostPlatform.isDarwin)
• Use nixpkgs-review to test package builds locally

Clan Vars / Secrets

Secret not available at deploy time

• Ensure the secret is in lib/machine-env-matrix.nix for that machine
• Run clan vars generate --machine <machine> to populate
• Check that lib/vars-generators.nix is imported in the machine config

How to reset a secret

## List all vars for a machine
clan vars list <machine>

## Delete and regenerate a specific secret
clan vars delete <machine> <generator-name>
clan vars generate --machine <machine>

Build Performance

Evaluation is slow

• Documentation is disabled in modules/nixos/common.nix (documentation.enable = false)
• Use binary caches listed in flake.nix nixConfig
• Run cachix use luxus to enable the project cache

Large closure size

• Use nix path-info -rsSh .#nixosConfigurations.<machine>.config.system.build.toplevel to inspect
• Consider using minimal-*.nix profiles for machines that don’t need dev tools
• The packages-llm.nix and packages-core.nix add significant packages

Common Commands

Task	Command
Check flake	nix flake check --no-build
Deploy	just update-<machine> or clan machines update <machine> (see justfile)
Generate secrets	clan vars generate --machine <machine>
List secrets	clan vars list <machine>
Format code	nix fmt
Validate inventory	scripts/validate-inventory.sh
Compare machines	scripts/diff-machines.sh <m1> <m2>
Evaluate config	nix eval .#nixosConfigurations.<machine>.config.system.build.toplevel.drvPath --no-write-lock-file

Resolved Cleanups (2026)

• SDDM greeter idle blanking on primary (#85) is obsolete: lea now uses greetd + noctalia-greeter by default (see services/desktop/default.nix, modules/hosts/lea/default.nix).
• kmonad removed entirely from lea (#110): EurKEY XKB layout is used instead. No kmonad references remain in the tree.
• Per-app QT_QUICK_CONTROLS_STYLE=Basic workarounds eliminated (#111): EasyEffects and similar now use the consolidated global QML/Union environment from lib/kde-settings-qml.nix + hjem/profiles/kwin-noctalia.nix.
• Hjem migration children (#87–#94, #96) closed: full fleet migrated off Home Manager (see modules/clan/hjem-users.nix, hjem/, flake.nix has no home-manager input).

See inventory.md, configuration-matrix.md and services/README.md for current state.