edit: everything sucks. i just mounted usb for /home and /var, "classic" sysv-style. boom done.

* overlay is a PITA; could probably be made to work but zomg is complex. Better if you don't actually want to preserve anything (like, put upperdir on tmpfs)

* exotic filesystems don't really help

* with non-SD stuff (home, var) off-SD, one could easily-enough just remount ro, and remount rw during upgrades/updates.

problem: raspberry pi needs an SD card to boot* and SD cards aren't great for repeated writes. Eventually they die and it sucks.

Raspberry Pi OS now has an overlay function which can (sort of) let you use a read-only root on the sd card. I'm not using rpiOS tho, because ZFS is happier on vanilla 64-bit Ubuntu.

Ubuntu has an overlay option, but it's kinda a one-time deal; breaking out of it is *hard*.

What I really need:

• Something VERY easy to manage. Like if I forget it's there it should still work
• Something that limits or eliminates writing to the DS card
• but I should be able to write to the card e.g. for updates

So here's what *I* and doing:

• remount /boot/firmware as read-only (a FAT FS for the kernel and initramfs)
• boot as usual on the rw root fs, mounted on the SD card
• in runlevel3, remount high-usage dirs as overlay; use the local USB storage for the "upper"
• provide a "sync" option which can rsync stuff from upper -> lower, e.g. following an apt upgrade
  • implied: /lower is remounted (mount --bind) in another part of the VFS

Gory details / notes for future self:

• mount -t overlay overlay -o lowerdir=/home,upperdir=/mnt/overlay/home/upper,workdir=/mnt/overlay/home/work /home
• # mount --bind /home /mnt/overlay/home/lower # does not work; or, only works if /home is a filesystem (not a subdir). If home is just part of a larger fs, use mount --bind / /mnt/overlay/lower/root and be clever about which of those are mucked-with
• rsync -axXp /mnt/overlay/home/upper /mnt/overlay/lower/root/home # note that if you delete a file in the overlay it is NOT deleted in the lower; therefore lower could get ugly (leftover files)
• overlay doesn't work with zfs (no upper or work); ext4 works OK. I use LVM + 4 smallish partitions to host raid1 volumes for /var, /home
• /tmp can be tmpfs; 1GB is *plenty* on an 8GB rpi
• most of the rest of the VFS is relatively static