After one year and a half at DxO working on the DxO ONE, a diminutive, yet highly capable camera that fits in your pocket and connects to an iPhone, I’ve decided to move on and join Red Hat to work on Spice, focusing on 3D acceleration for virtual machines. Free software, virtualization and 3D together, and working again with old colleagues from the HPVM era, Karen Noel and Denise Dumas. What’s not to love?
Back to Linux, and not impressed
I’ve been using MacOSX as my primary desktop since the days of Rhapsody. But I always had Linux virtual machines running on occasions. A dwindling number of machines at home were running Linux, most notably a couple of Raspberry Pi and a Synology Diskstation. And when I installed Linux, I usually went for Ubuntu, which did a good job polishing the user experience. The build ring for Tao3D includes a number of virtual machines running several major distros for testing purpose, but it’s been quite inactive for a while, and repairing it is on my short-term to-do list.
Working for Red Hat, I thought I had to use Fedora as my primary desktop. And the experience has been a bit underwhelming so far, unfortunately. In just three days, I managed to render a Mac Book Pro unbootable in OSX (update: it worked the second time I tried), had several different issues with skippy or laggy mouse cursors and even non-responsive keyboards, had a driver crash attempting to access my home Wi-Fi, found out the hard way that NFS performance is just horrible, and had to use Google for trivial things way too often.
I complained several times on this blog about what I perceived as a degradation of OSX software quality since 10.6, but this experience with Linux puts all this in some serious perspective.
You never have enough storage
Before I describe my woes with Linux, I must first describe the storage I wanted to manage and the applications I want to run. Mostly, my objective is to be able to test and run software on a large number of virtual machine configurations, without running out of disk space every other day. I also would like to setup Redmine and Jenkins to automate the build of various open source projects I maintain on a variety of platforms. Lately, there has been issues reported on GitHub regarding the installation of Tao3D, for example.
The primary file server at home is a Synology Diskstation 412+ with 8TB of storage in RAID5 (roughly 6TB available). It has been working like a charm for several years, and I find the built-in software really easy to use, with tons of features. It is connected to the rest of the house with a dual-1Gb bond over a 24-port switch. It uses the BTRFS filesystem, although for the moment I’m not doing much use of replication or snapshotting (I hope to setup BTRFS replication someday).
I also have a Terra-Master D5-300 which I initially used in JBOD configuration to host a variety of old disks of various sizes to back-up the Synology NAS and other machines in the home. Unfortunately, a rather severe cleaning lady incident damaged two disks, and one more died of old age. BTRFS is a good file system, but there is not much it can do when 3 of its 5 disks die within one week. I’m lucky these were only backups.
So I ended up reprovisioning it with 3TB disks in order to be able to activate the hardware RAID-5 in the box. You have to go through a MacOSX or Windows software to activate hardware RAID.
The OSX software is the worst instance of fugly cartoon-sci-fi user interface I’ve seen since, I’d say, the Commodore Amiga days:
Bizarro shaped windows, round dialog boxes with OK button in diagonal, and an attempt at competing in the “most egregious waste of pixel space in the history of mankind” category. All that design effort, yet the application does not even have an icon, so it shows the default application icon! Seriously?
But what really matters is that once you enable hardware RAID with this utility, the settings stick, and you can use your RAID array from Linux, as one big 12TB (out of 15TB) disk. So far, it works like a charm. So if you need some high-capacity storage for home, that works OK with Linux, OSX or Windows, this is recommended.
I’m a bit concerned about what happens when a disk goes bad, though. There are LEDs at the front that will tell me. But I would have liked to get an e-mail. There is an option for that in the OSX software, but I’m pretty sure that part of the settings does not stick when you connect the array to a Synology Diskstation…
Converting a Mac Mini Server to Fedora Server
The first thing I tried to do was converting an old Mac Mini Server (2009 vintage, 2 CPUs, 4G of RAM) to run things like Redmine or Jenkins and act as a backup file server. We used to run that from OSX using Bitnami stacks, but I was really tempted to try Docker instead.
The “Done” button should be at the bottom right, folks
The Fedora installer is globally allright, except for one thing that really annoys me. For some reason, all their setup pages have a big blue Done button sitting in the… top left of the window. This is the last place I would search for it.
Believe it or not, I spent something like 2 minutes searching for that Done button all over the place, my eye carefully avoiding the one place where I assumed it could not be. Until finally a side glance made me realize this button was not “Back”, but “Done”.
There are some standards about the placement of buttons on a page. On the left, you have “back”. On the right, you have “forward” or “done”. I’ll probably file a usability bug about this one.
Oh, and by the way, when there are things that happen after you click on the button, it should be labelled “Next” and not “Done”. Otherwise, you keep looking for the things that are actually in the next page of the configuration process.
Attempt #1: Boot from RAID array
My initial intent was to install Linux on the Terra Master array, and to boot from it, hoping that I would be able to boot other machines from it if I wanted to. The Mac Mini is old, it may die on me at any time, even if it’s Apple hardware which usually lasts a while. For some reason, installing like this never quite worked.
Since the Mac Mini Server has no DVD, I installed from a USB stick. So during installation, the USB stick was
/dev/sdc and the RAID array was
/dev/sdd. But after installation, you remove the USB stick, and then the array becomes
/dev/sdc. That somehow confuses the GRUB bootloader, which renders the system unbootable. You need the USB stick to boot from an external disk, which is rather annoying…
From what I can tell, the GRUB startup scripts have some code that tries to identify disks by ID. And from what I can tell, that code does not work at all, at least in this configuration.
Attempt #2: Boot from internal disk
The Mac Mini Server has two internal disks, and I thought that it was a rather risk free operation to dedicate one to Linux. They were initially in RAID1 setup, so I split them, and I started installing.
I then rationalized that it was a better choice any way. The RAID disk is capable of more than 100MB/s, but the Mac Mini Server only has USB 2 ports (5 of them, which is quite a lot on such a tiny machine). So that meant not using the full bandwidth of my RAID. I figured it would be better to connect the RAID disk to the Synology, and access it over a 1GB network would get me a better bandwidth. Also, it would mean that the Synology would be able to do its backups at full speed.
In the end, I had a fully functioning Fedora 25 server.
Regular keyboard and mouse issues
This second installation refused to accept any keyboard input. That’s when I realized that I had changed keyboards, thinking that the old PC keyboard I had used was dead, because keys would stick or not work. You had to type any letter two or three times to get it, and then sometimes it would stick. So to type “
Hello“, you had to type something like “
He e e e DEL DEL DEL l l l l o o o“, all the
DEL being when the
e was stuck and you saw
eeeee on screeen. Rather horrible. But actually a software glitch.
Don’t ask me to explain why, but it turns out that every other boot, the Fedora installer will have real trouble with USB mice or keyboards. The symptoms vary from unresponsive keyboard to skippy mouse to keys that repeat… I’ve tried swapping devices, changing USB ports, to no avail. What is funny is that for some reason, the installation only worked with an Apple-branded keyboard, but after installation, it really did not like it much anymore, and wanted the old PC keyboard… Go figure.
<sarcasm>It’s a good thing that you practically never need a keyboard or a mouse. Plus it’s easy to excuse Linux for not yet fully supporting new-fangled technologies such as USB-1</sarcasm>.
It’s not the only input device glitch, but a mile. When booting Fedora Workstation on a Mac Book Pro, the trackpad also sticks, in a way that is just annoying enough that you notice, but not clear enough that you can describe what happens exactly. It’s as if the trackpad was somehow slow to react, but I can’t explain exactly why. I just know that it is way less comfortable than under OSX, and rather terribly frustrating, because you keep missing your target on screen.
This kind of user experience glitch is really annoying.
DNF: How many software packaging tools do you need?
I remember when
rpm, the Red Hat package manager, was the de-facto standard in Linux installation tools. There was one big alternative, of course, which was Debian’s
apt-get. It turns out
apt-get got a big boost since then as a consequence of being integrated in Ubuntu. But back then, when mice were mice and used Linux-supported PS/2 connectors, Ubuntu was not on the radar yet, and frankly,
apt-get install, that was a lot of typing relative to
rpm -i. In short, I was quite content with
rpm at the time.
rpm was replaced with
yum for reasons that remain, at least to me, a mystery. Why couldn’t
rpm evolve to incorporate the goodies in
yum? But oh well, it’s still quite a bit shorter than
apt-get. And apparently, you can still use
rpm, although I’m not sure it’s a good idea.
I was barely getting used to
yum, and now they replace with with
dnf. Come on, seriously? Apparently, this stands for “Dandyfied Yum”. And the reason for the switch was to repair a broken dependency algorithm? Is that worth changing the name? In any case, I still have to get used to typing
dnf. Quite often, my finger autopilot types something else, a true word like dns.
Hawaii desktop: probably beautiful, needs working keyboard
The server edition does not come up with a GUI. So I thought I would install it. There is a nice notion of “group” in
dnf, so I opted for the Hawaii destkop. Everything installed fine. In the process, I learned how to enable runlevel 5 on a systemd desktop, and quite frankly, I liked it.
Just one thing. For some reason, the login screen of Hawaii does not accept keyboard input. Now, that’s annoying. What’s up with Linux and keyboards? I’m not a keyboard-obsessed person, but I like to have it working on occasions.
Cockpit: it does almost nothing, but does it well
One of the advertised features of Fedora server is Cockpit. Based on my experience with Synology DSM, I was clearly expecting too much from Cockpit. It actually does not do much, but the little it does, it seems to do relatively well.
The only issue I have there is with HTTPS certificates for the server. Apparently, Cockpit does not work with Safari in secure mode. I wanted to use free certificates from LetsEncrypt, but that requires a publicly-available server. For some reason, with IPv6 addresses,
certbot won’t connect to my server:
Failed authorization procedure. my-server.com (tls-sni-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Failed to connect to [ip-v6-address]:443 for TLS-SNI-01 challenge
For now, I don’t use Cockpit much anyway, so having to fire up Chrome for it is OK.
NFS is not as good as I remember it
Since the Mac Mini was now accessing the hard disks remotely, I figured that I would we able to connect remote directories on the Synology using NFS. It actually works somewhat, but with several issues:
- I could not get user ID mapping to work correctly, despite much fiddling with
/etc/idmapd.conf. Apparently, it’s a known thing that it does not work without Kerberos, and the bug is recent, it’s only 4 years old. So I ended up having to change my user ID on the Mac Mini Server to match the one on the Synology.
- It’s has a rather high latency. I put
/homeon NFS, as we used to do on lab workstations back when I was in engineering school (in 1988, that says something…). Back then, it worked OK. Nowadays, not so sure. My desktop sometimes pauses for several seconds, presumably waiting for NFS to respond. I’m on a local 1GB network, with less than one meter of cable between the NFS server and client. I expected good performance. I reserve judgement, though, because the Synology is still doing some major first-time backup, which may impact its response time.
- Some key features of BTRFS are not transparent over the network, notably snapshotting. Docker apparently is smart enough to use BTRFS snapshots internally if they are available, from what I see in
/var/lib/docker. So that means I probably will have to use BTRFS replication to backup this stuff.
Creating a VM with Boxes: poof!
Since I’m going to work on virtual machines, I wanted to try and create a couple of small VMs for testing purpose. Typing “virtualization” in Software showed up two products: Boxes and Virtual Machines Manager. Time to try.
The first time I ran Boxes, it asked for an ISO file. I had these ISO files on my NFS sever, so I tried to go there and grab one. I did not have time to select the ISO file that Boxes had gone poof. The second attempt was slightly more successful, fortunately. But software that goes poof is irritating.
Mac Book Pro installation – Near disaster
Red Hat provided me with a 15″ Mac Book Pro. Since this was one of their choices of machine for developers, I assumed it was well supported by Fedora. And the couple of installs on the Mac Mini Server had given me some confidence in the installation process. I was wrong.
I resized the primary OSX volume on the hard disk, making room for a Fedora install, I put my USB stick in the machine, and started the installer. I immediately noticed that the trackpad was not behaving OK, but that was only a minor nuisance. And frankly, at that stage, I was happy the built-in keyboard was working.
Why doesn’t it recognize my EFI partition?
The first thing that caused problem was that the installer kept telling me that it that the existing EFI partition was not good. I kept getting a message that I needed an EFI partition on a GPT-formatted disk mounted at
/boot/efi … which was exactly what I thought I had.
In the end, I gave up. I thought that wasting 200MB of disk space for a second EFI partition was no big deal. The installation proceeded OK, and after installation, I had a working Fedora Workstation.
OSX won’t boot anymore
Time to reboot in OSX. Uh oh… the MacOSX menu entry in the Fedora menu does not work. It complains about various xnu commands not being found. Apparently, this too is an old known bug.
Well, I though it was not a major problem. After all, the Macs have their own (graphical) boot menu, and since Fedora forced me to create some additional EFI partition, it could not possibly have messed up with that, right?
Wrong. Booting while holding Alt down showed me only the Fedora entry, which brought me to the non-functioning ugly text-mode GRUB menu. The expected OSX entry was gone. The Mac OSX recovery partition was gone too. Aaaaargh. Fedora really went out of its way to make this machine unbootable under OSX… If I did not know better, I would think that they played a Microsoft on me.
Wi-Fi driver crash
If, at least, Fedora was fully usable as an OS, I’d shrug and setup things like a mail client, etc. The problem is that I have Wi-Fi access only in half of the house. If I’m in the living room, the best Wi-Fi access point is a FreeBox Revolution, and it always rejects my password.
dmesg shows something a bit frightening: the Wi-Fi driver simply crashes every time I try to enter my password:
[ 1008.652836] brcmfmac: brcmf_cfg80211_reg_notifier: not a ISO3166 code (0x30 0x30) [ 1011.548069] ------------[ cut here ]------------ [ 1011.548099] WARNING: CPU: 2 PID: 2262 at net/wireless/sme.c:724 __cfg80211_connect_result+0x3ab/0x410 [cfg80211] [ 1011.548100] Modules linked in: rfcomm fuse nf_conntrack_netbios_ns nf_conntrack_broadcast xt_CT ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_raw ip6table_security iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_raw iptable_security ebtable_filter ebtables ip6table_filter ip6_tables cmac bnep nls_utf8 hfsplus intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel btusb snd_hda_codec_cirrus snd_hda_codec_generic snd_hda_codec_hdmi brcmfmac kvm btrtl btbcm btintel joydev bluetooth brcmutil snd_hda_intel snd_hda_codec applesmc cfg80211 iTCO_wdt irqbypass iTCO_vendor_support input_polldev [ 1011.548128] snd_hda_core crct10dif_pclmul mmc_core crc32_pclmul snd_hwdep snd_seq ghash_clmulni_intel intel_cstate intel_uncore mei_me intel_rapl_perf bcm5974 snd_seq_device rfkill mei i2c_i801 snd_pcm i2c_smbus lpc_ich thunderbolt snd_timer snd soundcore shpchp sbs sbshc acpi_als apple_gmux kfifo_buf industrialio video tpm_tis tpm_tis_core tpm apple_bl nfsd auth_rpcgss nfs_acl lockd grace sunrpc uas usb_storage hid_apple btrfs xor amdkfd amd_iommu_v2 radeon raid6_pq i2c_algo_bit drm_kms_helper ttm crc32c_intel drm fjes [ 1011.548154] CPU: 2 PID: 2262 Comm: kworker/u16:9 Tainted: G W 4.8.14-300.fc25.x86_64 #1 [ 1011.548155] Hardware name: Apple Inc. MacBookPro11,5/Mac-06F11F11946D27C5, BIOS MBP114.88Z.0172.B09.1602151732 02/15/2016 [ 1011.548164] Workqueue: cfg80211 cfg80211_event_work [cfg80211] [ 1011.548165] 0000000000000286 00000000650b1753 ffff92b31fe0fc98 ffffffff823e5ded [ 1011.548167] 0000000000000000 0000000000000000 ffff92b31fe0fcd8 ffffffff820a0e8b [ 1011.548169] 000002d400000008 ffff92b36aa9c008 0000000000000000 ffff92b36aa9b000 [ 1011.548171] Call Trace: [ 1011.548175]  dump_stack+0x63/0x86 [ 1011.548178]  __warn+0xcb/0xf0 [ 1011.548180]  warn_slowpath_null+0x1d/0x20 [ 1011.548190]  __cfg80211_connect_result+0x3ab/0x410 [cfg80211] [ 1011.548199]  cfg80211_process_wdev_events+0x129/0x1a0 [cfg80211] [ 1011.548207]  cfg80211_process_rdev_events+0x38/0x70 [cfg80211] [ 1011.548215]  cfg80211_event_work+0x1e/0x30 [cfg80211] [ 1011.548217]  process_one_work+0x184/0x430 [ 1011.548218]  worker_thread+0x4e/0x480 [ 1011.548219]  ? process_one_work+0x430/0x430 [ 1011.548220]  ? process_one_work+0x430/0x430 [ 1011.548222]  kthread+0xd8/0xf0 [ 1011.548226]  ret_from_fork+0x1f/0x40 [ 1011.548228]  ? kthread_worker_fn+0x180/0x180 [ 1011.548229] ---[ end trace 022b2d25dbb9a245 ]---
Apparently, once more, this is a known old bug, so I added a comment on the kernel bugzilla and one on the Ubuntu forum that deals with the issue. At least, it gives me a hint about how to fix things, which is that I should not use the 5GHz band. That would be a good hint if I was using 5GHz, but I’m not. Shrug.
Disconnect a screen, have to reboot
Another funny thing happened. I connected an external display to the Mac Book Pro. I had to use Google to understand how to use the Displays control panel to setup that external display as my primary. So much for ease of use, although it is quite simple as soon as you understand that the big icons on the left do not represent displays, but actions on displays.
Having set that up, I took the laptop with me, disconnecting the external display without so much as a second thought. You see, Mac laptops have always behaved correctly when you do that since around 1998. So I sort of expected that Linux in late 2016 would do something equally smart in that case. Which it did not.
What I got was a big secondary screen, with no obvious way to do anything. Sometimes, the screen would dim, so I knew it was popping up some dialog box on the other display. You know, that other display that was no longer connected to the machine. <sarcasm>I guess Linux is ahead of its time and was trying to connect to that other display over Wi-Fi, using the Wi-Fi driver that had crashed</sarcasm>.
I tried various things, including the old keyboard shortcuts that used to give you a text console (apparently, that’s not how it’s configured on Fedora 25) and killing the X server, which worked, but brought me back another X server with the same display setup as before.
In the end, I had to force-reboot the Mac and contemplate once more the lack of MacOSX display option at boot. Sigh.
This is so double plus annoying
Overall, I spent less than three days with Linux, and I already have been plagued with several severe bugs, none of them in the area where I’m supposed to work:
- The Fedora installer making OSX non bootable
- The Wi-Fi driver crash in the kernel
- The USB keyboard with sticky or non responsive keys in the installer
- The USB keyboard not working at all in Hawaii Desktop on server edition
- The USB mouse skipping every other boot in Wayland on the Mac Mini
- The built-in trackpad not tracking “intuitively” on Mac Book Pro
- The lack of detection of disconnected displays
- The low NFS performance
- The non-working NFS ID remapping
- Boxes going poof while selecting an ISO file
- The installer Done button in the upper-left corner of the screen
I’m probably forgetting a few, but that’s already a fair number of issues in such a short period of time.
The good news is: there is apparently some work to do :-). I love Linux.