Mountains

Mountains

Wednesday, January 10, 2024

My Ubuntu ZFS on Root Gripe List

I sort of love Ubuntu ZFS on root and sort of don't.

I have a lot of complaints.

(1) Almost completely missing documentation. The entire knowledge base is centered around Didrocks blog.

  • How do you add datasets to zsys?
  • How do you backup/restore this?
  • zfs complains that the pools can be updated to use new features but it's not clear if this is safe.
  • Unclear about how zfs volumes were setup. How frequently does this scrub and trim? (not very)

(2) Poor handling of file systems with lots of changes. This sounds like an edge case, but if you use steam to manage a games library and do some video or photo work on the side, you're going to find a lot of snapshot induced storage bloat. I've had to reduce the snapshots held in zsys.conf. 

(3) bpool size is too small for the kernel image rotations. I end up having to manually remove old kernels and snapshots. Complaints of this are extremely common. You can, with some pain, replace the bpool with a larger one.

(4) Unclear behavior in recovery mode. If I go back to an early snapshot, is the current dataset still fine? What happens if I make a change? -Confused-

(5)  Snapshot names are cryptic. I would have hoped for something like an ISO timestamp and a simple iterator.

(6) No configuration options at setup. Going back to the above, having a sane default is/would be great, but adding ZFS options and setting the sizes of bpool and dealing with complex zfs pools would be useful in the installer.

Overall, zsys seems like requires an understanding of ZFS to run with confidence that you won't suffer some catastrophe in an edge case. It so opaque that it's a big tradeoff to commit to zfs on root, as you need to really understand ZFS to use it, but at the same time, if you're that into ZFS, you'll find it frustrating.

Ubuntu 22.04 LTS Starts In Emergency Mode: no network drives, snapd apparmor failure, and can't mount /boot

 I sort of love Ubuntu ZFS on root and sort of don't. A main gripe have been almost completely missing documentation. The entire knowledge base is centered around Didrocks blog.

Yesterday, I started my computer from a cold start, logged in, realized I meant to turn on virtualization in the bios to use virtual box, restarted within maybe 60 seconds or less of logging in, tweaked the bios, and restarted again. Instead of a clean boot, the system started into emergency mode. I guessed that something about virtualization extentions was causing problems, and restarted, set the bios back, and restarted again, but the system still would only get into emergency mode.

journalctl -xd errors noted

(1) Snapd.apparmor errors for firefox

(2) Couldn't mount network samba shares

(3) Couldn't mount /boot

(4) bpool was missing from zfs list  

Disabling app-armor in grub did not resolve the issue (some people have had corrupted app-armor installs). Apparmor failure maybe were a symptom?

Disabling network shares did not resolve. This was a symptom of the network stack not working.

I kept focusing on fstab. I tried mounting the vfat partition labeled /boot on my ssd. This did not work. But I recalled that zsys does some kind of rotation with bpool and /boot.

Maybe something was keeping zsys from mounting boot? Did it "forget"?

I am still not sure how to guess how to mount the correct bpool to /boot.

I found this reddit discussion where someone converted a /dev/ to a uuid fstab:

https://www.reddit.com/r/zfs/comments/n26j2f/has_anyone_been_able_to_convert_ubuntus_zsys/

 Based on that, after entering emergency mode and finding that 

mount -a

could not mount boot from fstab, and that bpool was not in

zfs list

zpool list

I ran

zpool import -d /dev/disk/by-id bpool

Which correctly mounted bpool to /boot. I then exited emergency mode and the computer started normally.

I have no idea what happened...