Introduction
I have been running FreeBSD on my server and really enjoying it. ZFS, Jails, and the amazing package availability have been incredible to work with.
Except… the lack of Docker, and some modern Linux-focused software.
About Docker
I never really used Docker all that much. I’m still very much a fan of the “pets” not “cattle” approach for my small homelab server. This approach has been easy to understand, realiable, and low-maintenance for me.
Jails on FreeBSD are exactly what I wanted from a “lightweight OS virtualization” feature. It feels almost exactly like a full virtual machine once logged in to a Jail, and everything behaves as you expect it to.
What about Linux? Linux has LXC, their version of “system containers” - a similar idea to FreeBSD’s Jails - running a close-to-full Linux system with the init process, package manager, and more, as part of a base Linux distribution such as Debian or Alpine Linux. You can log in, use the package manager to install software, configure it, and enable the init services for it to run on “boot”. Docker is more focused on “application containers” - usually containing a minimal base and focused on running only the application processes. This makes Docker more of a glorified application package manager like Flatpak, bundling the environment and dependencies with the application for consistent behavior wherever you run the container.
I wasn’t particularly interested in Docker, so why Linux?
Why?
It boils down to: performance, support, and Immich.
Immich
Immich is a self-hosted photo library management server with photo backup, a mobile app, facial detection, and other cool features. It feels almost like a self-hosted replacement for Apple iCloud Photos or Google Photos.
My interest was piqued, especially after testing it out on my desktop machine and seeing how nicely it worked. I had several thousand unorganized photos dumped into almost a hundred disorganized directories on a file share. Not very helpful when you want to easily browse them on any device. I wanted one massive photo library that was easily searchable, without paying Apple for hundreds of gigabytes of iCloud Storage, and without wanting my personal data to be in other hands in the first place.
Time to install Immich on my FreeBSD server then, right?
Immich and FreeBSD
This was fun. I’m smiling so much right now (I’m not). Turns out Immich is only really easy to deploy using Docker, on Linux, of course. I decided that I would try running Docker using the Linuxulator that FreeBSD has, but it didn’t work at all. The Linuxulator simply wasn’t good enough to get Docker to play nicely. So the next step was: native installation!
Immich looks to be written mainly in web frameworks and many of the dependencies are already available on FreeBSD. In addition, some people have already made scripts that install Immich natively without Docker on Linux systems, although these solutions were far from ideal in terms of hackiness/robustness against updates. Using one of these scripts as a base, I tried to get Immich up and running on FreeBSD, but, soon realized that it just wasn’t meant to be.
From missing Node npm FreeBSD-built packages, to compilation failures, to libva errors, I gave up. In the end, I managed to get Immich “running”, if “running” counts as loading the web interface and uploading images at a whopping 10 KB/s for some reason, coupled with none of the machine learning features working, and several random errors and other weirdness. Oh, and don’t forget the black-magic rain-dance I had to do just to get Immich to start up at all. I decided this wasn’t going to work and that I’d have to simply run Linux somehow if I wanted to use this software.
Virtualization
Why not run Linux in a virtual machine? I actually wanted to do this - it’s ideal in my case. Except my aging server has a CPU that doesn’t support Unrestricted Guest, an instruction set extension that is required for bhyve to use more than 1 vCPU, and/or the UEFI firmware loader. Bummer. What about VirtualBox? Compiling kernel modules, Guru Meditations, absolutely atrocious performance (even though the CPU supports virtualization through VT-x), and a weird bug that caused my VM to get stuck using 100% CPU on boot and hang quickly put an end to that path.
It became clear that there was only one path forward - Linux.
ZFS on Linux
I wanted to continue using ZFS since that was one of the main reasons I went with FreeBSD in the first place. But ZFS on Linux is, well, not very “welcome” due to licensing differences. Ubuntu ships with it, but the rest usually don’t. What distribution do I choose?
Linux Distribution Adventures
Well, time to try them all, I guess! I decided to try out Fedora Server, Alpine Linux, Arch Linux (yes!), and finally Debian. Why not Ubuntu? Well, the last time I used it, it randomly killed my LXC containers when snap updated. Oh snap. Not to mention that when I typed in lxc it just started installing a snap for LXD! That’s not what I wanted, and don’t do that automatically without asking me for permission first! It also took a few seconds to log in, felt quite heavy with tons of preinstalled stuff, and was generally annoying with so much stuff going on, it reminded me a little bit of Windows with its intrusiveness and so I just developed a slight disliking for it. Still a great distribution for many people though, just not me personally. It’s nice that it ships with ZFS - a major advantage for those who want to use it!
Debian
Out of all the distributions I tried, I ended up settling with Debian. I should’ve just started with this and called it a day. I don’t have much to say besides “it works” and “it works well”. Installing ZFS from the backports archive was easy enough, and AppArmor works well out of the box. Debian even has a neat script that parses your Samba shares and automatically generates AppArmor rules for them, no manual AppArmor tweaking necessary, so SMB just worked!
There are a couple things that I wasn’t particularly fond of though, such as APT defaulting to installing recommended packages, and also keeping recommended packages around, not allowing autoremove/autopurge to remove them if ANY other package also happens to have them listed as a recommended package. Enabling and starting services by default after installing new software is also maybe not the greatest thing for minimalism and control, but it also does have the benefit of “just working”. I changed the APT behavior with some config file drop-ins, made a few other tweaks, and now I’m happy.
Debian is peaceful to use. Peace and quiet. It’s nice!
Performance
Linux seems to be running faster than FreeBSD, at least on my server setup. Encryption and decryption is faster (SMB transfer speeds increased from ~60 MB/s to 90 MB/s), and Nextcloud seems to load pages a bit faster, especially on cold boot.
Running any sort of stress test, or even just the yes command shows increased power consumption and faster throughput compared to FreeBSD. Idle power consumption is the same. I am not sure why this is the case other than a lack of optimization in FreeBSD, or a lack of hardware driver support for power management and related things. I even had the CPU frequency set to the max (turbo mode) in FreeBSD, tried powerd, and turned on C-states. It seems like Linux is able to push the system harder and extract more performance out of it. I can even hear the fans run faster and a louder electrical whining noise on Linux than FreeBSD when the system is under stress.
As this system is aging, I’m happy to squeeze every last drop of performance out of it.
Linux Containers
Of course, I now have Immich up and running without issue. I even installed it using Docker inside an unprivileged LXC container! All I did was enable AppArmor as well as seccomp nesting for that container. It seems perfectly happy, reports that it is using native overlay diffs, and runs great.
So, was it worth it?
I’d say yes! While I’m sad to be leaving FreeBSD, Linux is working pretty well.
I’d like to investigate returning to FreeBSD in the future, on newer hardware with Unrestricted Guest support so I can run Linux in a bhyve virtual machine. But I may just end up sticking with Linux since it’s working well.
Farewell FreeBSD (for now). Thank you for your service. FreeBSD really does have the power to serve! And I really hope that more people consider adding FreeBSD support to their projects. Immich? ;)