edit: everything sucks. i just mounted usb for /home and /var, "classic" sysv-style. boom done.

* overlay is a PITA; could probably be made to work but zomg is complex. Better if you don't actually want to preserve anything (like, put upperdir on tmpfs)

* exotic filesystems don't really help

* with non-SD stuff (home, var) off-SD, one could easily-enough just remount ro, and remount rw during upgrades/updates.

problem: raspberry pi needs an SD card to boot* and SD cards aren't great for repeated writes. Eventually they die and it sucks.

Raspberry Pi OS now has an overlay function which can (sort of) let you use a read-only root on the sd card. I'm not using rpiOS tho, because ZFS is happier on vanilla 64-bit Ubuntu.

Ubuntu has an overlay option, but it's kinda a one-time deal; breaking out of it is *hard*.

What I really need:

  • Something VERY easy to manage. Like if I forget it's there it should still work
  • Something that limits or eliminates writing to the DS card
  • but I should be able to write to the card e.g. for updates

So here's what *I* and doing:

  • remount /boot/firmware as read-only (a FAT FS for the kernel and initramfs)
  • boot as usual on the rw root fs, mounted on the SD card
  • in runlevel3, remount high-usage dirs as overlay; use the local USB storage for the "upper"
  • provide a "sync" option which can rsync stuff from upper -> lower, e.g. following an apt upgrade
    • implied: /lower is remounted (mount --bind) in another part of the VFS

Gory details / notes for future self:

  • mount -t overlay overlay -o lowerdir=/home,upperdir=/mnt/overlay/home/upper,workdir=/mnt/overlay/home/work /home
  • # mount --bind /home /mnt/overlay/home/lower # does not work; or, only works if /home is a filesystem (not a subdir). If home is just part of a larger fs, use mount --bind / /mnt/overlay/lower/root and be clever about which of those are mucked-with
  • rsync -axXp /mnt/overlay/home/upper /mnt/overlay/lower/root/home # note that if you delete a file in the overlay it is NOT deleted in the lower; therefore lower could get ugly (leftover files)
  • overlay doesn't work with zfs (no upper or work); ext4 works OK. I use LVM + 4 smallish partitions to host raid1 volumes for /var, /home
  • /tmp can be tmpfs; 1GB is *plenty* on an 8GB rpi
  • most of the rest of the VFS is relatively static