Addendum: during the event, I was interviewed by Jason Tubnor for the BSD Now Podcast, where I provided further information about the talk and the BSD Cafe project. Here is the link to the episode
This is the text underlying my presentation at EuroBSDCon 2024, on 21 September 2024, in Dublin, Ireland.
The slides can be downloaded here
The video, not yet separated from the live stream, can be viewed here - At first, I was a bit tense, then I relaxed.
Happy reading!
EuroBSDCon Dublin - 21 September 2024 - Why (and how) we're migrating many of our servers from Linux to the BSDs
"I'm Stefano Marinelli, I solve problems."
I’m the founder and Barista of the BSD Cafe, a community of *BSD enthusiasts.
I work in my company, called Prodottoinrete - a container of ideas and solutions.
I’m passionate about technology and computing, and I’ve made my passion my profession. Every morning, when I sit in front of the computer, a new world opens up for me to explore, and I try to share this passion with my clients. Sometimes, I succeed.
I've been a Linux user since 1996, before I turned 17. Back then, I used Fidonet and would read about alternative operating systems. Curiosity got the best of me, and I bought a set of CDs with various Linux distributions at the first opportunity. I tried it, I liked it, but at the time, I didn’t find it advantageous, so I continued using it for secondary tasks while Windows remained my daily driver.
Things changed in late 1997. I decided to go deeper into Linux and, with the purchase of a new, more powerful computer, I realized that aside from gaming, Linux could be my everyday system. This came in handy when I started university in 1998, where the computer science department was very oriented towards Open Source solutions. I was one of the few students who already understood the concepts of Open Source and knew how to use Linux. I was also one of the few who wasn’t baffled when we found Solaris machines in the lab. After all, there were similarities.
Over the years, I became one of the administrators of that lab, which was almost entirely Linux-based.
In 2000, I was fortunate to have Professor Özalp Babaoğlu as my lecturer, which pushed me to explore other operating systems like the BSDs. However, all I had at the time was an old Compaq laptop (486/25 MHz and 4 MB RAM) and no fast internet connection. So, apart from some theoretical studies, I postponed hands-on experiments until I had better resources. By 2002, thanks to a broadband connection and a new computer, I began exploring BSD systems. I started with FreeBSD, largely thanks to its fantastic handbook. I asked my parents to buy a laser printer so I could "print academic material" — but really, I wanted to print all the documentation I could find, including the BSD handbook. And it was incredibly helpful.
Before long, FreeBSD became my daily driver — entire nights spent compiling KDE while I slept a meter away from my laptop’s wildly spinning fans—but FreeBSD ran much, much better on that machine than Linux did. Unfortunately, OpenBSD was too slow to be usable with any graphical interface.
In 2003, my final thesis focused on virtualization on Open Source systems, NetBSD/Xen was one of the best and most efficient solutions I tested.
Shortly after, I found a job at a company that was just beginning to offer Linux-based solutions, mainly for web and mail servers, but I was criticized because I completed tasks that didn’t require constant interventions — this hurt the company’s billing. That’s when I set my future guidelines:
- I would work for myself, following my own philosophy.
- I would not be tied to any particular vendor. I love exploring and learning, so I would always study solutions in depth and recommend the one I thought best suited for the client.
- I solve problems — I don’t sell boxes.
- I would use and promote Open Source solutions whenever possible.
- I would use a BSD whenever feasible and Linux where a BSD was not suitable.
In every technological decision, my priority is solving my clients’ specific problems, not selling a predefined solution. I solve problems.
I was often told that Open Source systems were “toys for universities” and that the real world ran on something else (mostly meaning Windows). I pressed on. In some cases, I offered to be paid only if the results were achieved, showing how much they’d save compared to traditional licensing costs. It worked, but...
- They accepted Linux, albeit reluctantly, but rejected BSDs because they didn’t know them. In some cases, I managed to convince them. In others, sadly, I didn’t.
Result: I decided to use the BSDs where clients didn’t have direct access — like email servers or web hosting — while using Linux where they specifically requested it.
- NetBSD/Xen as the virtualization base, with excellent longevity and reliability.
- OpenBSD as network/firewall entry points.
- FreeBSD (especially with the introduction of ZFS) for various services, including backup servers.
It worked. What surprised clients most was the stability and the reduced need for maintenance. They saw more uptime and were happy.
Problem: "If nothing is working, what am I paying you for? If everything’s working, what am I paying you for?" So, I selected clients and situations that understood that if everything works, it’s because of the work behind the scenes. It’s better to pay for everything to work than to pay to fix problems. The problem to solve, in this case, is not stopping the client’s work. And I solve problems.
I managed about 65% *BSD machines and 35% Linux. But as Linux’s popularity grew, so did client demand. It became necessary on multiple occasions to replace or implement Linux instead of a BSD, for specific requests.
At a certain point, Linux virtualization solutions matured, and many of my hosts transitioned from NetBSD/Xen to OpenNebula (often with MooseFS) and then to Proxmox (often with Ceph). This happened because my clients needed to manage their VM lifecycles autonomously, change configurations, migrate hosts, etc.
Proxmox showed excellent reliability and stability. The VMs were often Linux-based (especially if the client needed direct management) or FreeBSD, but in smaller quantities. My own infrastructure, however, remained primarily BSD-based.
Over time, I gradually started to reflect and 'take stock'.
No *BSD has ever caused me to lose data to the point of having to restore from backup. On Linux, I’ve lost data with ext4, XFS, and btrfs. The most catastrophic case was with XFS — just a few files, backed up and restored, but the client was highly demanding and didn’t take it well. The largest failure was with btrfs — after a reboot, a 50 TB filesystem (in mirror, for backups) simply stopped working. No more mounting possible. Data was lost, but I had further backups. The client was informed and understood the situation. Within a few days, the server was rebuilt from scratch on FreeBSD with ZFS — since then, I haven’t lost a single bit.
In 2018, I started introducing Docker and Podman to the developers with specific needs, especially to help them with their development. Many developers began incessantly asking for Docker, unfortunately some of them only wanted to bypass many limitations that traditional setups imposed. Unfortunately, in many cases, those "limitations" were just bad practices — running outdated versions of software and libraries, or not wanting to bother with keeping the stack updated. There are similar problems with other solutions, but at least I can block them. They're great for quick deployment and increased security, but they aren’t always the best choice. Plus, they’re not the only option, despite what many people think these days.
Linux has had major development over the past years, but this has shifted towards specific players’ interests (mainly cloud providers) rather than technical reasons. Maybe not in the kernel, but many Linux distributions and communities now seem focused on the constant push to replace "the old with the new" with solid theoretical reasons but sometimes seemingly without practical benefit.
For me, computing should solve problems and provide opportunities to those who use it. I solve problems. Every change or variation will solve one problem but create new ones. It’s crucial to be mindful not to create worse problems than the ones you’re solving, and today’s enterprise world often seems to overlook that.
It’s common to see software distributed solely via Docker Compose files. Sometimes I use it as an installation tutorial, but I realize they’re just specific pieces precariously glued together (patched files, specific dependency versions, or nothing works).
The trend is to rush, to simplify deployments as much as possible, sweeping structural problems under the rug. The goal is to "innovate", not necessarily improve — just as long as it’s "new" or "how everyone does it, nowadays."
A massive business has grown around Linux: certifications, training, pentesting, certified platforms — the community is losing decision-making power. This is a far cry from the early days.
I solve problems. And these fast-changing technologies risk creating more problems than they solve, at least in certain situations. The workloads I manage often stay up for years, requiring a more stable, upgradable, and consistent approach.
If a client’s problem is to have an e-commerce site, they don’t really care if it runs on Docker, Podman, a FreeBSD jail, or a Raspberry Pi cluster — as long as their problem is solved. In fact, clients are happy when their solution is stable, upgradable, and secure.
There isn’t a single solution to all problems, but many solutions for each problem. My job is to give clients the best solution to solve their specific problem, not the most fashionable one.
That’s why I decided, several years ago, to reverse the proportion and implement and the BSDs for all possible workloads.
Each *BSD has its characteristics and target audience. Sometimes these targets overlap, but it’s generally not difficult to choose the most appropriate solution.
The goal of the migration was to create stable, coherent, upgradable, and secure systems.
By implementing an OpenBSD system, I often don’t need to install any additional packages. When it’s time to upgrade, it’s simple and secure. When a vulnerability emerges, I’m often fortunate to read "OpenBSD is excluded from this issue because it eliminated this risk X years ago..."
By implementing a NetBSD system, I know I’m installing a system that isn’t in a rush to release new versions and will likely run for years with only a few package updates and security patches, when necessary. And it’s quite rare for a patch to be required.
By implementing a FreeBSD system, I know I’ll have ZFS at my disposal, a fast and efficient hypervisor like bhyve, and a native, mature jail system that will ensure service and setup separation coherently and precisely. Not as “add-ons” but built into the system itself.
Moving to the BSDs means, in my experience, reaching systems that are more stable, more easily upgradable, and more consistent in their parts. They don’t chase the hype of the moment, much like the early days of Linux. In the case of FreeBSD, this also means moving to native ZFS and boot environments, giving me greater peace of mind when it comes to upgrades.
The initial strategy was to migrate what would soon need updates anyway, what would be moved (thus requiring a new setup), and what was causing problems and deserved a deeper dive.
First, I decided to migrate to FreeBSD the hypervisors not directly accessed by clients, especially on leased servers. The approach was to create a twin machine (to have an objective performance and stability comparison — it wouldn’t make sense to compare it to a different machine), install FreeBSD, bridge it with the production server, install vm-bhyve, and start copying VMs from Proxmox, reconfiguring the main parameters in the configuration file. In some cases, I already used ZFS on Proxmox, so the first part of the migration was a simple zfs-send/receive. In other cases (e.g., when using Ceph), I made an intermediate move by live-migrating the storage to ZFS and then proceeding as in the first case. The first noticeable effect was a reduction in resources used by the host to handle VM traffic — as expected, only FreeBSD’s basic processes and bhyve were running — but at the same time, there was a significant increase in I/O performance, further enhanced by switching the virtual disk driver from virtio to NVMe. This allowed for in-depth testing and revealed that FreeBSD suffers less under heavy I/O (the VMs on Linux tended to block their I/O, an effect I didn’t notice on FreeBSD) and showed significantly lower loads. In short, it handled the load better.
As an experiment, I decided to migrate two hosts (each with about 10 VMs) of a client — where I had full control—without telling them, over a weekend. By Tuesday, they called me, concerned: they had noticed a massive performance boost and were worried I had upgraded their hardware without approval, thinking that, "given the performance boost," it would cost them a lot. After explaining what I had done, they asked me to run further tests and progressively continue migrating everything. 20 hosts, all based on (sometimes slightly older) versions of Proxmox. And so I did, but taking advantage of their open-mindedness, I went a step further.
Many of their VMs handled simple workloads — PHP websites, or Java-based management systems (running on Tomcat), device monitoring software for industrial machinery, etc. The decision to use VMs had been made by their highly competent internal IT staff to separate environments and dependencies. It was a perfect use case for jails. I decided to try, VM by VM, to replicate the setups inside FreeBSD jails. In some cases, for convenience, I managed to run everything directly in Linux jails (with Linuxulator); in others, it was impossible, so I recreated the setups in individual jails. They immediately noticed an effect: faster operations. It didn’t surprise me. We avoided double buffering (VM and OS), so all the saved RAM could be used by ZFS for its cache and by the host to run other services. Some VMs remained as VMs (e.g., Zimbra). By the end of the operation, I had drastically reduced the number of VMs, replaced by jails, and consequently, the number of hosts. From 20 down to 11 — with a significant monthly cost saving.
The road was now clear, and I progressively continued down this path. One of the most interesting anecdotes: a client told me that they used to start an operation before taking a coffee break, around 15 minutes, to find the task almost done by the time they returned. After the migration, they shared that they launched the process, grabbed their things, and the task was already complete. An estimated reduction from about 18 minutes to 6 minutes on average. I didn’t investigate too much, but I suspect a combination of factors, with the predominant one being bhyve’s NVMe driver.
The main challenge I often face is ideological. Some people are used to thinking that the ideal solution is X — and believe that X is the only solution for their problems. Often, X is the hype of the moment (a few years ago, I fought to convince people that VMware wasn’t necessary and that Proxmox would be a great solution; today, Proxmox is on everyone’s lips — but it’s not the only solution). Often, X is a "cloud" cluster with Kubernetes - running WordPress on it. Even for hosting a law firm’s website, which will be updated every five years.
When I ask, "Okay, but why? Who will manage it? Where will your data really be, and who will safeguard it?", I get blank faces. They hadn’t considered these questions. No one had even mentioned them. "But everyone I spoke to proposed this type of solution... It’s like at the beginning with Windows, when I proposed *BSD or Linux. Or later with VMware, when I proposed Proxmox. Or now with Kubernetes, when I propose the BSDs.
But the simplest solutions are the easiest to maintain and manage over time. My experience has taught me that setting something up is often the easiest part. The hardest part is returning to it after 1, 5, 10 years. Keeping it running, updating it, stabilizing it. For many, IT isn’t their business, but a tool to achieve their goals. A Kubernetes cluster is fantastic, but it requires maintenance. Or it’s external — so it’s no longer ours. We’ve lost control of the data. For many, it’s unnecessary to complicate things. And with every additional layer, we’re creating more problems.
No BSD system has ever surprised me during an update or a simple reboot. I’ve never encountered, for example, a network interface renaming from enx3e3300c9e14e to enp10s0f0np0* on Linux, effectively locking me out of a server. ix0 will remain ix0.
I’ve never had to recompile ZFS on FreeBSD, only to find that the module wouldn’t load, blocking the filesystem from mounting after reboot.
Many of the developers I work with have embraced the challenge. Most of them are passionate about technology, and learning a new operating method has been very interesting. Almost all of them, after experiments, were positive and, in fact, began explicitly requesting "jails" instead of Docker hosts. They started using BastilleBSD to clone "template" jails and deploy them. They learned to access ZFS automatic snapshots and recover lost files. They learned to manage the resources at their disposal without repeating the usual mantra of "we need mooooar powaaaar!" every time there’s a problem, a slowdown, or a storage overload. They’ve returned to trying to understand what’s happening, rather than just assembling pieces, libraries, and containers without considering the effects.
Others, however, struggled, but remained positive nonetheless. And that’s okay. I solve problems and can’t force my solutions on everyone.
In other cases, the challenge wasn’t technical but "commercial." Often, decision-makers have little to no technical knowledge. Linux sells well. "Cloud" sells even better. A "NetBSD-based solution," unfortunately, has less commercial appeal today. So, they want what they can sell, without focusing too much on the advantages of alternative solutions.
Linux today is subject to many compliance requirements — I’m often asked which version of OpenSSH I’m running — and they complain when the version (the latest from OpenBSD) isn’t considered "secure" because it doesn’t match their procedures (e.g., OpenSSH_9.2p1 Debian-2+deb12u3). They don’t understand when I explain that the "Debian" part refers to the Linux distribution, not a release. Those who prepare these documents are often, sadly, unaware of what they’re asking for. They just have a checklist.
The transition is ongoing, and as I see opportunities, I’m migrating from Linux to the BSDs whenever possible. Today, I can say that 78% of the hypervisors I manage run on FreeBSD, and 66% of the workloads (VPS, jails, hosts, etc.) are running on one of the BSDs — including solutions like OPNsense. None of the clients have experienced major issues. No one has complained about performance or reliability. All feedback has been positive. Many appreciate moving away from IT monocultures, especially when problems arise — following the recent severe SSH vulnerability, many clients contacted me, worried. It was nice to tell some of them that the exposed SSH service was running OpenBSD’s version, and that OpenBSD wasn’t affected as they had developed a secure mechanism back in 2001. They appreciated it. They want more setups based on OpenBSD.
I'm Stefano Marinelli, I solve problems. And I love solving problems using BSD systems.