My Ubuntu ZFS on Root Gripe List

I sort of love Ubuntu ZFS on root and sort of don't.

I have a lot of complaints.

(1) Almost completely missing documentation. The entire knowledge base is centered around Didrocks blog.

• How do you add datasets to zsys?
• How do you backup/restore this?
• zfs complains that the pools can be updated to use new features but it's not clear if this is safe.
• Unclear about how zfs volumes were setup. How frequently does this scrub and trim? (not very)

(2) Poor handling of file systems with lots of changes. This sounds like an edge case, but if you use steam to manage a games library and do some video or photo work on the side, you're going to find a lot of snapshot induced storage bloat. I've had to reduce the snapshots held in zsys.conf. 

(3) bpool size is too small for the kernel image rotations. I end up having to manually remove old kernels and snapshots. Complaints of this are extremely common. You can, with some pain, replace the bpool with a larger one.

(4) Unclear behavior in recovery mode. If I go back to an early snapshot, is the current dataset still fine? What happens if I make a change? -Confused-

(5)  Snapshot names are cryptic. I would have hoped for something like an ISO timestamp and a simple iterator.

(6) No configuration options at setup. Going back to the above, having a sane default is/would be great, but adding ZFS options and setting the sizes of bpool and dealing with complex zfs pools would be useful in the installer.

Overall, zsys seems like requires an understanding of ZFS to run with confidence that you won't suffer some catastrophe in an edge case. It so opaque that it's a big tradeoff to commit to zfs on root, as you need to really understand ZFS to use it, but at the same time, if you're that into ZFS, you'll find it frustrating.

Ubuntu 22.04 LTS Starts In Emergency Mode: no network drives, snapd apparmor failure, and can't mount /boot

 I sort of love Ubuntu ZFS on root and sort of don't. A main gripe have been almost completely missing documentation. The entire knowledge base is centered around Didrocks blog.

Yesterday, I started my computer from a cold start, logged in, realized I meant to turn on virtualization in the bios to use virtual box, restarted within maybe 60 seconds or less of logging in, tweaked the bios, and restarted again. Instead of a clean boot, the system started into emergency mode. I guessed that something about virtualization extentions was causing problems, and restarted, set the bios back, and restarted again, but the system still would only get into emergency mode.

journalctl -xd errors noted

(1) Snapd.apparmor errors for firefox

(2) Couldn't mount network samba shares

(3) Couldn't mount /boot

(4) bpool was missing from zfs list  

Disabling app-armor in grub did not resolve the issue (some people have had corrupted app-armor installs). Apparmor failure maybe were a symptom?

Disabling network shares did not resolve. This was a symptom of the network stack not working.

I kept focusing on fstab. I tried mounting the vfat partition labeled /boot on my ssd. This did not work. But I recalled that zsys does some kind of rotation with bpool and /boot.

Maybe something was keeping zsys from mounting boot? Did it "forget"?

I am still not sure how to guess how to mount the correct bpool to /boot.

I found this reddit discussion where someone converted a /dev/ to a uuid fstab:

https://www.reddit.com/r/zfs/comments/n26j2f/has_anyone_been_able_to_convert_ubuntus_zsys/

 Based on that, after entering emergency mode and finding that 

mount -a

could not mount boot from fstab, and that bpool was not in

zfs list

zpool list

I ran

zpool import -d /dev/disk/by-id bpool

Which correctly mounted bpool to /boot. I then exited emergency mode and the computer started normally.

I have no idea what happened...

Folding at Home on Ubuntu 19.10 with AMD GPU

For future reference, there's a global pandemic on at the time I'm writing this.

I found out that Folding@Home was taking on various Covid-19 research projects, and though, since I had some computer laying around not doing much, so I installed the client and let it go. However, I'm mainly a Linux user, and the my RX 460 was not getting used. I read several pieces of advice on via google. Generally, linux, opencl, and AMD gpus have to exist in certain combinations in order for folding at home to work correctly.

I've been running FAH on a workstation with Ubuntu 19.10 and a AMD RX 460 graphics card.

To get things to work, I had to
(1) Remove opencl-mesa and rcom
(2) I have ocl-icd-opencl-dev installed.

(3) Install AMD's opencl from the amdgpupro package. (amdgpupro IS NOT REQUIRED)
I used this script to do it, though I had to modify the version numbers as is described in the comments:
https://gist.github.com/kytulendu/3351b5d0b4f947e19df36b1ea3c95cbe
(4) Add <slot id='1' type='GPU'/> to the config.xml file (like this)

Bonus
(5) Disable FAHClient as a service in /etc/init.d (https://foldingforum.org/viewtopic.php?f=61&t=25328)
(6) Run FAHClient as root

The machine runs great while folding, and seems as productive in linux as it does in windows. By running from the command line, this allows the computer to quickly freed for heavy lifting, or rebooted to other operating systems.

Lenovo x120e 802.11n Wireless Data Corruption with RTL 8188CE 1x1 NIC

I've known that this wireless card was toxic for a long while.

However, laziness has meant that our household has been trapped in 802.11g land for years and years, even as I gave 802.11ac routers to family members as gifts.

Because the shoemakers children have no shoes or something.

Anyhow, I knew the Thinkpads* Realtek RTL 8188CE wireless card was bad news when the other machines started experiencing network instability whenever it was active on the network. The wifes Netflix stream would glitch. Copying files requires 100% of the CPU, and similarly, network intensive multiplayer games took it in the chin as the CPU was soaked by the NIC, detracting from rendering.

But, I recently got the joy of trying out one of the 802.11ac routers that I had been gifting (Asus rt-ac68u), and was pleasantly surprised by the speed (the adapter reported speed of up to 150 mbps) and unpleasantly surprised by how all the data that came through the pipe was more or less ruined by the experience. Installers claimed checkup errors, binaries wouldn't run right, and webpages and other internet content looked a little...weird.

An example corrupted page from this blog

The network interface showed itself as a possible candidate when other machines in the house didn't have the same issues, and I found that downloading the same file several times yielded different checksums each time. I was particularly ticked off when my linux virtualmachine a) had trouble getting updates b) once updated couldn't stay booted for long, presumably because of data transfer errors. At first, I thought this was due to an old version of virtualbox, but then when I tried to update virtualbox, it complained of .cab checksum errors. Running hashmyfiles on windows 7 showed that each time the virtualbox installer was downloaded, I got a different checksum. Of course, I was suspicious of that too, but it was all I had to work with.

Crazy.

Sadly, There is little information avaiable about this situation and what can be done about it. After a day of fiddling, I determined that the RTL 8188CE can be forced into 802.11g or 802.11b mode, which home use had more-or-less shown to be less error prone. To force 802.11b or 802.11g, activate Taskbar->Wireless Icon->(Your Wireless Network)->Status->Properties->Configure->Advanced->Wireless Mode->802.11b/g (or 802.11b). Setting "Auto" will allow 802.11n, and thus data corruption, at least in my usage scenario. Corruption was greatly reduced (i.e. I could download files and view websites normally) with 802.11g forced (by selecting 802.11b/g mode).

I had read in a few places references to errors generated by the TCP/IP checksum offloading feature of the card, though it seems possible that the checksum maybe fine and in fact the card is not moving data correctly.

Things to meditate on:
1) Checksum are valuable to making sure data makes it from point a to point b without getting fubared.
2) All the advice about hacking the bios to remove the pcie whitelist to allow other mini pcie NICs is probably well founded.
3) The RTL 8188CE is just painful on this machine. Neither realtek nor lenovo have fixed the drivers, and data corruption is evil. I have to ask myself if other issues i'm experiencing on the machine are a product of running windows update through a leaky wireless pipe.
4) Curiously, the RTL 8188CE is not mentioned in the list of 802.11n adapters provided by Lenovo, but Broadcom 43225 and Intel 6200 (6250 as well) are. They would seem like rational upgrades.
5) The 2x2 abgn adapter chipset/part number relationships reported online are all over the map. The Lenovo FRU 60Y3253 corresponds to both Intel and Broadcom chips.

*More like Stinkpad amiright?

The Deep End of the Bathtub (Curve) with Dead Albatross

The dual g4 Quicksilver I use for general productivity at home was made in 2002. It has lived at 10 different addresses in 4 different states over that period of time. I used it to writing my B.S. thesis project and heavily used it in my Ph.D. work. While some of my friends have had lemon macs, I got my moneys worth and then some out of it.

To wit: the thing's been used. And I think it wants to retire, as much as a computer can tell you it wants to retire, but having trouble running a current operating system, having vanishingly small amounts of new code targeted to it, and most recently, killing all the hard drives I put in it.

I don't think there's anything physically wrong with the system; the problem is that every ATA-66/100/133 drive I have that is large enough to be useful is 5-7 years old and already has a lot of on-time. Statistically, they're already end-of-life.

One drive kicked the bucket in March. This past week, the drive I replaced it with, a barely used 160 gig drive I used to use for backups (back when 160 gigs was a lot) started spinning up and down, first at random, then all the time. It's motor is clearly gone. Now I get to pull out the screwdriver and put in another drive. I think I'll put in the newest drive.

Thank goodness for rsync. There is script that synchronizes the mac's data with the linux server over the LAN. Used weekly, it has saved every byte worth saving from depolarization.

Backups rock.

Keeping stuff working forever rocks.

But this is getting ridiculous. I have real stuff to do. Every computer in the house is newer and better in most ways. Time to kick this thing to the curb and find a way to do business with something else.

EDIT:
It turns out, the 160 gig drive is fine, and it's one of the other disks that have not raised a S.M.A.R.T. error, that's dying. Insanity.

The Sickness is Revealed

The Sickness: Why my T42 randomly hangs.

This is what is wrong with the Thinkpad. The T42 is hanging because the linux kernel does not manage the video memory correctly on it's Radeon 9600.

I can now only sit here and wonder how long the patches will take to be merged into the main linux kernel, then into the ubuntu system. Of course, I suppose I could try compiling a patched kernel myself...

... But it has been pointed out that I have spent a great deal of free time, perhaps a couple of weekends and all the free evening I can muster over the past 2 weeks attempting to solve this little thinkpad crash faux-pax. I think my interests are better served by just being patient, at this point. This is one of those things I would like to -work- so that I can get other things done. I am somewhat dissenterested in crossing all the bridges it takes to effectively debug kernels when I would really like to be thinking about my photography or looking for exciting new ways to analyze geographic data. Or writing...

I keep thinking I will stop using the mac due to it's age and incredible milage, however, it keeps turning on when other machines go up in smoke. It's not hard to really like a computer that works this well.

Icarus Pheonix Hybrid

You would forgive me for finding this network of interactions the least bit hard to wrap my head around.

Damn near impossible after a couple of beers, really.

Worse, it's missing parts!

Don't believe me?

Find the computer's BIOS in there.

I think it's hidden in the blue box, but I'm not sure.

Most laptops BIOS provide some mechanism to handle performance and power consumption if the OS does not take the lead. There have been some arguements (including from chipset makers) that the powermanagement would be -best- handled by the BIOS... but that's another side track we don't want to derail on).

Fortunately, some poor bloke (I think on thinkwiki.org.... but I forgot) mentioned that linux seems happier if you disable all the energy control settings in the BIOS.

After a weekend of constant on time, the computer has not crashed, yet. Though posting to blogger has been my favorite place for it to crash.

A BIOS powermanagement conflict would explain why it never crashed while plugged in....

Every Rose Has It's Thorns

The stinkpad managed to survive grad school on it's original windows installation. Somewere in 2007 it began to get a little cranky, and by 2009, the hard drive was full to the gills and it had developed the characteristic unresponsiveness that plagues a well used windows boot disk. Then, 2 weeks before my defence, it caught a rootkit. While I was able to extract part of the virus, it also became somewhat less stable... the wireless won't connect reliably and there a few blue screens to deal with.

Did I mention I wrote a dissertation on it? Yea, the keys are pretty polished too.

One of the hangovers from graduate school is that my dissertation and resulting papers are written under Word and Endnote. In fact, my entire reference library is in Endnote. I hope to write my new work in LaTex, but it is not expedient at all to convert old things.

So, simply pulling the drive and replacing it with a new one is still out of the question while I get my papers submitted (Endnote is on loan from my former research group, they'll want it back).

In the mean time, i've been trying to install Ubuntu on the expansion bay hard drive. You'd think that getting this working would be easy! Just run the Ubuntu install disc and go! Right?

Wrong.

Something about the Thinkpad causes Ubuntu to hang randomly, without leaving a trace in any log file. I thought there might be an error on the drive or the install disc. Trying a new install disk made things worse, because the newest ubuntu installer does not control where it writes grub well (POOR IMPULSE CONTROL), and ended up nuking the windows MBR on the main drive.


That wasn't inconvenient at all!

When it is not randomly hanging, I like it Xubuntu alot. It boots fast, it's clean and tiny, and it's unix. These things are goey happy joy to me. But then it randomly hangs and requires a reboot.

*sigh*

This is why I still have the mac...