edit: everything sucks. i just mounted usb for /home and /var, "classic" sysv-style. boom done.
* overlay is a PITA; could probably be made to work but zomg is complex. Better if you don't actually want to preserve anything (like, put upperdir on tmpfs)
* exotic filesystems don't really help
* with non-SD stuff (home, var) off-SD, one could easily-enough just remount ro, and remount rw during upgrades/updates.
problem: raspberry pi needs an SD card to boot* and SD cards aren't great for repeated writes. Eventually they die and it sucks.
Raspberry Pi OS now has an overlay function which can (sort of) let you use a read-only root on the sd card. I'm not using rpiOS tho, because ZFS is happier on vanilla 64-bit Ubuntu.
Ubuntu has an overlay option, but it's kinda a one-time deal; breaking out of it is *hard*.
What I really need:
- Something VERY easy to manage. Like if I forget it's there it should still work
- Something that limits or eliminates writing to the DS card
- but I should be able to write to the card e.g. for updates
So here's what *I* and doing:
- remount /boot/firmware as read-only (a FAT FS for the kernel and initramfs)
- boot as usual on the rw root fs, mounted on the SD card
- in runlevel3, remount high-usage dirs as overlay; use the local USB storage for the "upper"
- provide a "sync" option which can rsync stuff from upper -> lower, e.g. following an apt upgrade
- implied: /lower is remounted (mount --bind) in another part of the VFS
Gory details / notes for future self:
mount -t overlay overlay -o lowerdir=/home,upperdir=/mnt/overlay/home/upper,workdir=/mnt/overlay/home/work /home
mount --bind /home /mnt/overlay/home/lower# does not work; or, only works if /home is a filesystem (not a subdir). If home is just part of a larger fs, use
mount --bind / /mnt/overlay/lower/rootand be clever about which of those are mucked-with
rsync -axXp /mnt/overlay/home/upper /mnt/overlay/lower/root/home #note that if you delete a file in the overlay it is NOT deleted in the lower; therefore lower could get ugly (leftover files)
- overlay doesn't work with zfs (no upper or work); ext4 works OK. I use LVM + 4 smallish partitions to host raid1 volumes for /var, /home
- /tmp can be tmpfs; 1GB is *plenty* on an 8GB rpi
- most of the rest of the VFS is relatively static