IT Notes - bhyve

FreeBSD vs. SmartOS: Who's Faster for Jails, Zones, and bhyve VMs?

Stefano Marinelli — Fri, 19 Sep 2025 10:50:00 +0200

Disclaimer
These benchmarks were performed on my specific hardware and tuned for the workloads I expect to run.
They should not be taken as absolute or universally applicable results.
Different CPUs, storage, networking setups, or workload profiles could produce very different outcomes.
What I’m sharing here is a faithful snapshot of my test environment and use case - a guidepost, not a final verdict.

Years ago, I installed a PCEngines APU at a client's site. It dutifully ran Proxmox with a few small VMs inside. It wasn't a speed demon, but it got the job done. Tasked with running in a closed, uncooled, and unsupervised server closet, it soldiered on for about seven years.

Then, while I was at BSDCan, I got the call. A series of power outages and surges had finally taken their toll, and the APU was dead. It was probably just the power supply, but given its age, we decided it was time for a replacement. I set up a remote bypass to keep them running, but I knew I'd need to install something more powerful soon.

I ordered a modern MiniPC-based on the low-power Intel Processor N150 platform, but with 16GB of RAM and more than enough performance to serve as a decent workstation. I have a similar one in my office running openSUSE Tumbleweed, and it works beautifully.

This time, however, I decided to replace Proxmox with a different virtualization system. This decision wasn't made in a vacuum. In the past, I've put bhyve head-to-head with Proxmox, and my findings were clear: bhyve on FreeBSD is an extremely efficient hypervisor, often outperforming KVM on Proxmox in my tests.

This positive experience is what made FreeBSD with bhyve a top contender. The other path was a KVM-style approach (which would require fewer changes to the VMs), where my options would be NetBSD or an illumos-based OS like SmartOS. Since I had the new hardware on hand, I decided to run some tests to see how these different technologies stacked up against each other, and against the bare metal itself.

The Lineup: What I Put on the Test Bench

My goal was to test every reasonable option on this Intel N150 hardware. The final lineup covered the entire spectrum:

The Baseline:
- FreeBSD 14.3-RELEASE Bare Metal: The ground truth for performance on this hardware.
OS-Level Virtualization (Containers):
- SmartOS Native Zone: The baseline native container on SmartOS.
- SmartOS LX Zone: Running Ubuntu 24.04 and Alpine Linux.
- FreeBSD Native Jail: The baseline native container on FreeBSD.
- FreeBSD Jail with Linux: A jail running a Ubuntu 22.04 userland.
Full Hardware Virtualization (HVM):
- SmartOS bhyve Zone: A FreeBSD guest inside the bhyve hypervisor on a SmartOS host.
- SmartOS KVM Zone: A FreeBSD guest inside the KVM hypervisor on a SmartOS host.
- FreeBSD bhyve VM: A FreeBSD guest inside the bhyve hypervisor on a FreeBSD host.

The Benchmark: My `sysbench` Commands

To keep the comparison fair and simple, I used two core sysbench commands. To ensure consistency, I even compiled sysbench from scratch on the SmartOS native zone to match the versions and compile options on the other systems as closely as possible.

The commands I used in each environment were:

For CPU performance: sysbench --test=cpu --cpu-max-prime=20000 run
For memory performance: sysbench --test=memory run

First Look: CPU and Memory on the Intel N150

My initial tests on the Intel N150 hardware immediately revealed some interesting trends. The sysbench CPU results from any native FreeBSD environment (bare metal or jail) were on a completely different scale from the Linux and SmartOS guests, making a direct comparison meaningless.

However, by excluding the incompatible FreeBSD-native results, we get a very clear picture of the overhead between the various container technologies.

Valid CPU Performance Comparison (Single Thread, Intel N150)

Host OS	Container Tech	Guest OS	CPU Performance (Events/sec)
FreeBSD	Jail (OS-level)	Ubuntu 22.04	1108.18
SmartOS	LX Zone (OS-level)	Ubuntu 24.04	1107.13
SmartOS	Native Zone (OS-level)	SmartOS	1107.04
SmartOS	LX Zone (OS-level)	Alpine Linux	1022.81

The takeaway here was clear: for CPU work, the overhead from these containers is basically a rounding error. For CPU-bound tasks, neither SmartOS Zones nor FreeBSD Jails will be a bottleneck.

The memory results, which were consistent across all platforms, were far more revealing.

Overall Memory Performance Comparison (Intel Processor N150)

Host OS	Virtualization Tech	Guest OS	Memory Performance (Transfer Rate)
SmartOS	LX Zone (OS-level)	Ubuntu 24.04	4970.54 MiB/sec
SmartOS	Native Zone (OS-level)	SmartOS (Native)	4549.97 MiB/sec
FreeBSD	Jail (OS-level)	Ubuntu 22.04	4348.32 MiB/sec
FreeBSD	Bare Metal	FreeBSD (Native)	4005.08 MiB/sec
FreeBSD	Native Jail (OS-level)	FreeBSD (Native)	3990.13 MiB/sec
SmartOS	LX Zone (OS-level)	Alpine Linux	3803.72 MiB/sec
FreeBSD	bhyve VM (Full HVM)	FreeBSD	3636.01 MiB/sec
SmartOS	bhyve Zone (Full HVM)	FreeBSD	3020.15 MiB/sec
SmartOS	KVM Zone (Full HVM)	FreeBSD	205.18 MiB/sec

These initial numbers led to a few conclusions: a virtual layer could be a performance boost, the userland matters, and bhyve clearly outclassed the legacy KVM on SmartOS. However, one result was nagging at me: the performance gap between FreeBSD bare metal (4005.08 MiB/sec) and a native bhyve VM (3636.01 MiB/sec) was about 9%. This was a larger drop than I expected. It prompted a new question: was this overhead inherent to bhyve, or was it a quirk of the new N150 hardware?

Going deeper: Testing on an Intel i7-7500U

To see if more mature, better-supported hardware would tell a different story, I replicated the FreeBSD tests on an older Qotom Mini-PC powered by an Intel i7-7500U. The results were illuminating and dramatically changed the narrative.

CPU Performance Comparison (Intel i7-7500U)

Once again, the CPU tests produced strange results. The native FreeBSD environments all reported incredibly high numbers in the millions of events/sec, while the Ubuntu Linuxulator jail's result was on a completely different, incompatible scale. Frankly, given the massive discrepancy between FreeBSD-native and Linux-based environments, I'm unsure that the sysbench CPU figures can be considered totally reliable in absolute terms.

However, what is useful is comparing the native FreeBSD results against each other. This tells us about relative overhead.

Platform	CPU Performance (Events/sec)	Overhead vs. Bare Metal
FreeBSD Bare Metal	6,377,778	Baseline
FreeBSD Native Jail	6,379,271	~0.0%
FreeBSD bhyve VM	6,346,852	-0.48%

Even if we're skeptical of the absolute numbers, the relative comparison is crystal clear: the CPU overhead of bhyve is less than half a percent. This is the key takeaway.

Memory Performance Comparison (Intel i7-7500U)

The memory benchmarks, in contrast, were consistent and highly informative.

Platform	Memory Performance (Transfer Rate)	Overhead vs. Bare Metal
Ubuntu 22.04 Jail	4856.23 MiB/sec	+7.55%
FreeBSD Native Jail	4517.73 MiB/sec	+0.05%
FreeBSD Bare Metal	4515.24 MiB/sec	Baseline
FreeBSD bhyve VM	4491.60 MiB/sec	-0.52%

This is where the real story is. The memory performance of a bhyve VM was a mere 0.52% slower than bare metal. This is the kind of near-native performance one hopes for from a top-tier hypervisor and stands in stark contrast to the 9% drop seen on the newer N150.

Breaking Down the Results: What I Learned From Both Tests

This comprehensive two-platform analysis paints a much clearer picture.

1. Hardware Really Matters Performance is not an absolute. The difference between the two platforms was stark: on the mature i7-7500U, bhyve’s overhead was less than 1%, while on the newer, budget N150, it was a more significant 9%. This suggests the performance dip is likely due to missing optimizations for that specific CPU architecture, rather than a fundamental flaw in bhyve itself.

2. bhyve's True Potential is Near-Native Speed The i7 tests prove that bhyve is an exceptionally efficient hypervisor on well-supported hardware. The relative CPU overhead was a negligible -0.48%, and more importantly, the reliable memory benchmarks showed a performance drop of just 0.52% compared to bare metal. This is the gold standard for virtualization.

3. FreeBSD Jails are Feather-Light On both platforms, native FreeBSD jails demonstrated almost zero performance overhead. On the i7, both CPU and memory performance were virtually identical to bare metal (a 0.05% difference). The N150 CPU tests further showed that FreeBSD's container implementation is so efficient that running a Linux userland inside a jail delivered the best CPU scores of the entire lineup.

4. SmartOS Zones Are Also Extremely Efficient Just like Jails, SmartOS's native Zones proved to be remarkably lightweight. The N150 CPU tests confirm this, showing that native and LX zones have virtually identical, top-tier performance. On the memory front, the native Zone delivered performance over 13% faster than the FreeBSD bare-metal baseline, pointing to the high efficiency of the illumos kernel.

5. The Linux Userland Excels at Throughput A clear pattern emerged on both testbeds: the Ubuntu userland consistently delivered excellent benchmark results. On the CPU front, Ubuntu on both FreeBSD and SmartOS delivered the highest, and nearly identical, performance scores on the N150. For memory, the story was even more dramatic: the Ubuntu LX Zone on SmartOS was the top performer, beating bare-metal FreeBSD by nearly 25%, while the Ubuntu jail on the i7 also surpassed its host by over 7%.

Final Thoughts: The Verdict for My Client's New Server

So, what's the bottom line for my client's new MiniPC? This benchmarking journey has made the path forward much clearer.

At the beginning of this process, my main question was whether to stick with a KVM-based setup or make the switch to bhyve. The performance data answers that decisively. The legacy KVM on SmartOS showed a crippling performance penalty, making it a non-starter. Given that, the extra effort to migrate the existing VMs to a bhyve-compatible format is absolutely worth it. The performance gain is just too significant to ignore.

The final question, then, is which host OS to use for bhyve: SmartOS or FreeBSD? This is a much tougher call, as both platforms demonstrated incredible strengths.

SmartOS, powered by the illumos kernel, was a true surprise. It delivered astonishing performance on the target N150 hardware. Its key advantage is the raw speed of its containerization for both CPU and memory tasks. The Ubuntu LX Zone not only ran flawlessly but delivered top-tier CPU scores and outperformed the bare-metal FreeBSD baseline in memory by a massive 25% margin. This points to a highly efficient kernel and offers the tantalizing prospect of running ultra-fast Linux containers alongside performant bhyve VMs on the same host.

On the other hand, FreeBSD proved its mastery of bhyve virtualization. The tests on the i7 hardware showed its implementation to be the gold standard, offering virtually zero performance overhead for full hardware virtualization. Its native Jails are equally efficient, and its Linux compatibility layer is so effective that an Ubuntu jail delivered the fastest CPU performance of all containers tested on FreeBSD. For workloads that must live in a full VM, FreeBSD offers the most performant and native bhyve experience, with the reasonable expectation that its support for newer hardware like the N150 will only improve over time.

Ultimately, the choice comes down to the primary workload. It's a decision between the raw container speed and Linux flexibility of SmartOS versus the pure, uncompromising HVM performance of FreeBSD.

But one thing is certain: thanks to this deep dive, the path forward is much clearer, and it's paved by bhyve.

Introducing the illumos Cafe: Another Cozy Corner for OS Diversity

Stefano Marinelli — Mon, 18 Aug 2025 09:04:00 +0200

Introducing the illumos Cafe: Another Cozy Corner for OS Diversity

From the BSD Cafe to illumos Cafe

The idea for this new project was born from the success of the BSD Cafe, an initiative I introduced to the world in July 2023, which received an incredibly positive response. Far more than I ever anticipated. The BSD community already had its well-established hubs: in the Fediverse, places like bsd.network, exquisite.social, and others were already thriving, not to mention all the forums, channels, and Reddit communities.

But in my vision, something was still missing: a hub of services with a positive spirit, built exclusively with open-source tools, where people could come to share, learn, and experience technology with a positive mindset. The BSD Cafe is therefore not just an instance, but a true Cafe - I’ll be speaking more about the BSD Cafe in detail at the next EuroBSDCon.

Why Another Cafe?

In a world increasingly dominated by centralized services under the control (or lack thereof) of the usual big players, it has become essential to create free, independent communities, devoid of the algorithmic and commercial controls that influence our overall experience. From day one, the BSD Cafe has embodied this spirit.

Linux is a good kernel, and there are excellent distributions based on it (some using the GNU userland, others only partially, like Alpine Linux), but it cannot and should not become a monoculture. The alternatives are extremely capable, and for many use cases - in my opinion and experience - they are even more suitable. BSD systems have served me exceptionally well for over 20 years, providing stability and security. At the same time, many other operating systems are renowned for their robustness, reliability, and the quality of their design and implementation.

Why illumos?

illumos is one of them. As the open-source descendant of OpenSolaris, it is an operating system known for its enterprise-grade stability and innovative technologies like ZFS, DTrace, and "zones". It was born from the solid foundations of Solaris and has evolved over time while remaining true to many of its core principles. I have always seen illumos and its distributions as kindred spirits to the BSDs, despite their differences. The philosophy is one of evolution without revolution, of guaranteeing long-term continuity and reliability rather than chasing the latest hype. This is precisely why, for some time now (and thanks in part to the inspiring posts by Joel Carnat, which further sparked my curiosity), I have been running OmniOS and SmartOS alongside my BSD-based setups for certain workloads.

However, there is very little information online about services running on them. So, a few months ago, I began to consider a new project: the illumos Cafe.

The illumos Cafe Project

The illumos Cafe is a project similar to the BSD Cafe (though perhaps less complex, at least initially). It shares the same spirit of positivity and inclusivity and aims to provide services running on illumos-based operating systems to demonstrate that there are no reasons not to use them. Just like with the BSD Cafe, diversifying the operating systems we use - even while using the same platforms - is fundamental to improving the reliability and resilience of the Internet. The Internet was born as a decentralized network, but for most people, it has sadly become just a tool to access the services of big players.

Community and Philosophy

But we want to connect. We want relationships with people, between people. We don't want algorithms. We don't want our data to be monetized by "us and our 65535 partners". We want a network that serves us, an OS that serves us - not an OS that just serves as a vehicle to store our data in "someone else's house". The illumos Cafe, therefore, aims to be a home for anyone interested in developing, using, or who is simply curious about illumos-based operating systems.

Technical Setup

As with the BSD Cafe, the entire setup will be documented. For now, it is very simple: there is a VM (running on FreeBSD and bhyve, on hardware I manage) where I have installed SmartOS. The physical host also runs the reverse proxy (in a jail). Inside the SmartOS VM, there are a series of zones:

Zone 1: nginx (Web Server) - Currently serving the project's homepage.
Zone 2: Mastodon (Social) - Hosting the Mastodon instance and its dependencies at https://mastodon.illumos.cafe.
Zone 3: PostgreSQL (Database) - The Mastodon database, on a dedicated zone.
Zone 4: Redis (Cache) - The Mastodon cache, on a dedicated zone.
Zone 5: snac (LX Zone) - Currently in an LX zone (Alpine) as I ran into some issues getting it to work in a native zone. It will be moved to a native zone as soon as I resolve them. It's serving the snac instance at https://snac.illumos.cafe

Media files are stored on an external physical server (running FreeBSD, the same one as the BSD Cafe, but in a dedicated jail) with SeaweedFS. I was able to compile and run SeaweedFS on illumos without any problems, but at the moment, I don't have a host with enough storage space for the media.

Available Services

More services will arrive over time. For now, two gateways to the Fediverse are already available:

Mastodon - https://mastodon.illumos.cafe
snac - https://snac.illumos.cafe

Both instances share the same rules as the BSD Cafe. Positivity. Supporters, not haters. I want them to be places of enjoyment, not venting. Of friendship, not hate.

Registrations and Logo

Registrations for the Mastodon instance are now open, and the available themes are the default ones plus the colorful TangerineUI - whose orange hue echoes the illumos logo.

The project's logo was not generated by an AI. I made it myself by hastily sticking the illumos SVG onto a coffee cup. Basic, perhaps. But authentic.

Looking Ahead

The BSD Cafe will, of course, remain my primary home. But I want to bring illumos into the Fediverse and provide a home for anyone who wishes to share their interest in this excellent OS.

I will document the entire process, just as I did with Mastodon on FreeBSD, as it is a bit more intricate. Because in my dreams, I see Fediverse statistics showing instances spread fairly evenly across the major open-source operating systems. Because relying on a single OS, even if it's open-source, and ceasing to support the others is also a single point of failure.

Make Your Own Backup System – Part 2: Forging the FreeBSD Backup Stronghold

Stefano Marinelli — Tue, 29 Jul 2025 08:00:00 +0200

With the primary backup strategies and methodologies introduced, we've reached the point where we can get specific: the Backup Server configuration.

When choosing the type of backup server to use, I tend to favor specific setups: either I trust a professional backup service provider (like Colin Percival's Tarsnap), or I want full control over the disks where the backups will be hosted. In both cases, for the past twenty years, my operating system of choice for backup servers has been FreeBSD. With a few rare exceptions for clients with special requests, it covers all my needs. When I require Linux-based solutions, such as the Proxmox Backup Server, I create a VM and manage it within.

I typically use both IPv4 and IPv6. For IPv4, I "play" with NAT and port forwarding. For IPv6, I tend to assign a public IPv6 address to each jail or VM, which is then filtered by the physical server's firewall. Unfortunately, every provider, server, and setup has a different approach to IPv6, making it impossible to cover them all in this article. When a provider allows for routed setups, I use this approach: Make your own VPN: FreeBSD, WireGuard, IPv6, and ad-blocking included - assigning a /72 to the bridge for the jails and VMs.

In my opinion, FreeBSD is a perfect all-rounder for backups, thanks to its ability to completely partition services. You can separate backup services (or specific servers/clients) into different jails or even VMs. Furthermore, using ZFS greatly enhances both flexibility and the range of tools you can use.

The main distinction is usually between local backup servers (physically accessible, though not always attended, and in locations deemed secure) and remote ones, such as leased external servers. I personally use a combination of both. If the services I need to back up are external, in a datacenter, and need to be quickly restorable, I prefer to always have a copy on another server in a different datacenter with good outbound connectivity. This guarantees good bandwidth for restores, which isn't always available from a local connection to the outside world. However, an internal, nearby, and accessible backup server (even a Raspberry Pi or a mini PC) ensures physical access to the data. Whenever possible, I maintain both an external and an internal copy - and they are autonomous, meaning the internal copy is not a replica of the external one, but an additional, independent backup. This ensures that if a problem occurs with the external backup, it won't automatically propagate to the internal one. In any case, the backup must always be in a different datacenter from the one containing the production data. When the fire at the OVH datacenter in Strasbourg caused the entire complex to shut down, many people found themselves in trouble because their backups were in the same, now unreachable, location. I had a copy with another provider, in a different datacenter and country, as well as a local copy.

Despite it being "just" a backup server, I almost always use some form of disk redundancy. If I have two disks, I set up a mirror. With three or more, I use RaidZ1 or RaidZ2. This is because, in my view, backups are nearly as important as production data. The inability to recover data from a backup means it's lost forever. And it happens often, very often, that someone contacts me to recover a file (or a database, etc.) days or weeks after its accidental loss or deletion. Usually, pulling out a file from a two-month-old backup generates a mix of disbelief, admiration, but above all, a sense of security in the person requesting it. And that is what our work should instill in the people we collaborate with.

The backup server should be hardened. If possible, it should be protected and unreachable from the outside. My best backup servers are those accessible only via VPN, capable of pulling the data on their own. If they are on a LAN, it's even better if they are completely disconnected from the Internet.

For this very reason, backups must always be encrypted. Having a backup means having full access to the data, and the backup server is the prime target for being breached or stolen if the goal is to get your hands on that data. I've seen healthcare facilities' backup servers being targeted (in a rather trivial way, to be honest) by journalists looking for health details of important figures. It is therefore critical that the backup server be as secure as possible.

Based on the type of access, I use two types of encryption:

If the server is local (especially if the ZFS pool is on external disks), I usually install FreeBSD on UFS in read-only mode, as I've described in a previous article, and encrypt the backup disks with GELI. This ensures that in the event of a "dirty" shutdown (more likely in unattended environments), I can reconnect to the host and then reactivate the ZFS pool. This approach makes it nearly impossible to retrieve even the pool's metadata if the disks are stolen, as GELI performs a full-device encryption. For example, an employee of a company I work with stole one of the secondary backup disks (which was located at a different, unmonitored company site) to steal information. He got nothing but a criminal complaint. With this approach, it's also not necessary to further encrypt the datasets, which avoids some issues (which I'll discuss later, in a future post).
If the server is remote, in a datacenter, I usually use ZFS native encryption, encrypting the main backup dataset (and BastilleBSD's, if applicable). Consequently, all child datasets containing backups will also be encrypted. In this case as well, a password will be required after a reboot to unlock those datasets, ensuring that the data cannot be extracted if control of the disks is lost.

Here is an example of how to use GELI to encrypt an entire partition and then create a ZFS pool on it (in the example, the disk is da1 - do not follow these commands blindly, or you will erase all content on the da1 device!):

# WARNING: This destroys the existing partition table on disk da1
gpart destroy -F da1

# Create a new GPT partition table
gpart create -s gpt da1

# Add a freebsd-zfs partition that spans the entire disk
# The -a 1m flag ensures proper alignment
gpart add -t freebsd-zfs -a 1m da1

# Initialize GELI encryption on the new partition (da1p1)
# We use AES-XTS with 256-bit keys and a 4k sector size
# The -b flag means "boot," prompting for the passphrase at boot time
geli init -b -l 256 -s 4096 da1p1
# You will be prompted for a passphrase: choose a strong one and save it!

# Attach the encrypted partition. A new device /dev/da1p1.eli will be created.
# You will be prompted for the passphrase you just set
geli attach da1p1

# (Optional) Check the status of the encrypted device
geli status da1p1

# Create the ZFS pool "bckpool" on the encrypted device
# We enable zstd compression (an excellent compromise) and disable atime
zpool create -O compression=zstd -O atime=off bckpool da1p1.eli

In this setup, the reference pool for everything related to backups will be bckpool - and you'll need to keep this in mind for the next steps. Additionally, after every server reboot, you'll need to "unlock" the disk and import the pool:

# Enter the passphrase when prompted
geli attach da1p1

# Import the ZFS pool, which is now visible
zpool import bckpool

With this method, it's not necessary to encrypt the ZFS datasets, as the underlying disk (or, more precisely, the partition containing the ZFS pool) is already encrypted.

If, instead, you choose to encrypt the ZFS dataset (for example, if you install FreeBSD on the same disks that will hold the data and don't want to use a multi-partition approach), you should create a base encrypted dataset. Inside it, you can create the various backup datasets, VMs, and the BastilleBSD mountpoint. Due to property inheritance, they will all be encrypted as well.

To create an encrypted dataset, a command like this will suffice:

# Creates a new dataset with encryption enabled.
# keylocation=prompt will ask for a passphrase every time it's mounted.
# keyformat=passphrase specifies the key type.
zfs create -o encryption=on -o keylocation=prompt -o keyformat=passphrase zfspool/dataset

In this case, after every reboot, you will need to load the key and mount the dataset:

zfs load-key zfspool/dataset
zfs mount zfspool/dataset

Keep in mind the setup you choose, as many of the subsequent choices and commands will depend on it.

Base System Setup

I'll install BastilleBSD - a useful tool for separating services into jails. It will be helpful for isolating our backup services:

pkg install -y bastille

If you used ZFS for the root filesystem, you can proceed directly with the setup. Otherwise (i.e., ZFS on other disks), you'll need to edit the /usr/local/etc/bastille/bastille.conf file and specify the correct dataset on which to install the jails. Then run:

bastille setup

Once the automatic setup is complete, check the /etc/pf.conf file - it will be automatically configured to only accept SSH connections. Ensure the network interface is set correctly. When you activate pf, you will be kicked out of the server, but you can then reconnect.

service pf start

Let's bootstrap a FreeBSD release for the jails - this will be useful later.

bastille bootstrap 14.3-RELEASE update

Now, we create a local bridge. Jails and VMs can be attached to it, making them fully autonomous. Using VNET jails, for example, will allow the creation of VPNs or tun interfaces inside them, simplifying potential future setups (and increasing security by using a dedicated network stack).

Modify the /etc/rc.conf file and add:

# Add lo1 and bridge0 to the list of cloned interfaces
cloned_interfaces="lo1 bridge0"
# Assign an IP address and netmask to the bridge
ifconfig_bridge0="inet 192.168.0.1 netmask 255.255.255.0"
# Enable gateway functionality for routing
gateway_enable="yes"

Let's also modify /etc/pf.conf to allow the 192.168.0.0/24 subnet to access the Internet via NAT. We will skip packet filtering on bridge0 and enable NAT. This isn't the most secure setup, but it's sufficient to get started:

#...
# Skip PF processing on the internal bridge interface
set skip on bridge0
#...
# NAT traffic from our internal network to the outside world
nat on $ext_if from 192.168.0.0/24 to any -> ($ext_if:0)
#...

To ensure the new settings are correct, it's a good idea to test with a reboot.

Since I often configure vm-bhyve in my setups, I prefer to install it right away, creating the dataset that will contain the VMs and installation templates. Remember that zroot is only valid if you installed the entire system on ZFS; otherwise, you'll need to change it to your own dataset:

# Install required packages
pkg install vm-bhyve grub2-bhyve bhyve-firmware
# Create a dataset to store VMs
zfs create zroot/VMs
# Enable the vm service at boot
sysrc vm_enable="YES"
# Set the directory for VMs, using the ZFS dataset
sysrc vm_dir="zfs:zroot/VMs"
# Initialize vm-bhyve
vm init
# Copy the example templates
cp /usr/local/share/examples/vm-bhyve/* /zroot/VMs/.templates/

At this point, I usually enable the console via tmux. This means that when a VM is launched, it won't open a VNC port by default, but a tmux session connected to the VM's serial port. Let's install and configure tmux:

pkg install -y tmux
vm set console=tmux

Let's also attach the switch we created (bridge0) to vm-bhyve so we can use it:

vm switch create -t manual -b bridge0 public

Now, vm-bhyve is ready.

The basic infrastructure is complete. We now have:

ZFS to ensure data integrity, which will also handle redundancy, etc.
BastilleBSD to manage jails, useful for backing up Linux, NetBSD, OpenBSD, and non-ZFS FreeBSD machines.
vm-bhyve to install specific systems (like Proxmox Backup Server).

Backup Strategies

I use various backup tools, too many to list in this article. So I'll make a broad distinction, describing how to use this server to achieve our goal: securing data.

For FreeBSD servers with ZFS (hosts, VMs, jails, hypervisors, and their respective VMs), I use an extremely useful, efficient, and reliable tool: zfs-autobackup.
For Linux servers (without ZFS), NetBSD, OpenBSD, etc. (any non-ZFS OS), I usually use BorgBackup. There are other fantastic tools like restic, Kopia, etc., but BorgBackup has never let me down and has served me well even on low-power devices and after incredibly complex disasters.
For Proxmox servers (a solution I've used with satisfaction in production since 2013, although I'm recently migrating to FreeBSD/bhyve where possible), I use two possible alternatives (often both at the same time): if the storage is ZFS, I use the zfs-autobackup approach. In either case, the most practical solution is the Proxmox Backup Server. And the Proxmox Backup Server is one of the reasons I proposed installing vm-bhyve: running it in a VM and storing the data on the FreeBSD host gives you the best of both worlds. Some time ago, I tried running it in a FreeBSD jail (via Linuxulator), but it didn't work.

Backups using zfs-autobackup

zfs-autobackup is an extremely useful and effective tool. It allows for "pull" type backups, as well as having an intermediary host that connects to both the source and destination, which is useful if you don't want direct contact between the source and destination. I won't describe the latter setup, but the documentation is clear, and I have several of them in production, ensuring that the production server and its backup server cannot communicate with each other.

I usually create a dataset for each server and instruct zfs-autobackup to keep that server's backups in that dataset. The snapshots taken and transferred will all be from the same instant, so as not to create a time skew (some tools of this kind snapshot a dataset, then transfer it, which can result in minutes of difference between two different datasets from the same server).

I've described in detail how I perform this type of backup in a previous post, so I suggest reading that post for reference.

Let's install zfs-autobackup on the FreeBSD server:

pkg install py311-zfs-autobackup mbuffer

Backups for other servers using BorgBackup

When I don't have ZFS available or need to perform a file-based backup (all or partial), I use a different technique. BorgBackup backups are primarily "push" based, meaning the client will connect to the backup server. This is not optimal or the most secure approach, as the backup server should, in theory, be hardened. Even when protecting everything via VPN, the risk remains that a compromised server could connect to its backup server and alter or delete the backups. I have seen this happen in ransomware cases (especially in the Microsoft world), and so I try to be careful to minimize this type of problem, mainly through snapshots of the backup server (an operation that will be described later).

To ensure the highest possible security, I create a FreeBSD jail on the backup server for each server I need to back up. The advantage of this approach is the complete separation of all servers from each other. By using a regular user inside a jail, a compromised server that connects to its backup server would only be able to reach its own backups, as it would be confined to a user account and, even if it managed to escalate privileges, still be inside a jail.

Let's say, for example, we want to back up a server called "ServerA" (great imagination, I know). We create a dedicated jail on the backup server:

# Create a new VNET jail named "servera" attached to our bridge
bastille create -B servera 14.3-RELEASE 192.168.0.101/24 bridge0

BastilleBSD will automatically set the host's gateway for the jail. In our case, this is incorrect, so we need to modify it and set the jail's gateway to 192.168.0.1 in the /usr/local/bastille/jails/servera/root/etc/rc.conf file:

# ...
defaultrouter="192.168.0.1"
# ...

Restart the jail and connect to it:

bastille restart servera
bastille console servera

Now, inside the jail, we install borgbackup:

pkg install py311-borgbackup

BorgBackup doesn't run a daemon; it's launched by the remote server (which sends its data to the backup server), so it's important that the installed version is compatible with the one on the remote host.

Since we'll be using SSH, let's enable it:

service sshd enable
service sshd start

And create a non-privileged user for this purpose:

# The 'adduser' utility provides an interactive way to create a user.
root@servera:~ # adduser
Username: servera
Full name: Server A
Uid (Leave empty for default): 
Login group [servera]: 
Login group is servera. Invite servera into other groups? []: 
Login class [default]: 
Shell (sh csh tcsh nologin) [sh]: 
Home directory [/home/servera]: 
Home directory permissions (Leave empty for default): 
Use password-based authentication? [yes]: 
Use an empty password? (yes/no) [no]: 
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]: 
Username    : servera
Password    : <random>
Full Name   : Server A
Uid         : 1001
Class       : 
Groups      : servera 
Home        : /home/servera
Home Mode   : 
Shell       : /bin/sh
Locked      : no
OK? (yes/no) [yes]: yes
adduser: INFO: Successfully added (servera) to the user database.
adduser: INFO: Password for (servera) is: JIkdq8Ex

The user is created and can receive SSH connections. After setting everything up, I suggest disabling password-based login in the jail's SSH configuration, using only public key authentication.

As mentioned, the biggest risk of a "push" backup is that a compromised client could access the backup server and delete or encrypt the backup history, rendering it useless.

To drastically mitigate this risk, we can configure SSH to force the client to operate in a special Borg mode called append-only. In this mode, the SSH key used by the client will only have permission to create new archives, not to read or delete old ones. However, this approach could complicate some client-side operations (like mount, prune, etc.), forcing them to be done on the server. For this reason, I won't describe it in this setup, "limiting" myself to taking snapshots of the repositories. It can be a very good practice, so I recommend considering it.

Let's initialize the BorgBackup repository. In this example, for simplicity, I won't set up repository encryption. If the jails are on an encrypted dataset or GELI-encrypted disks, there will still be data encryption on the disks, but there will be no protection against someone who could physically access the server while the disks are mounted. As usual, security is like an onion: every layer helps. Personally, I suggest enabling and using it ALWAYS.

# Switch to the new user
su -l servera
# Initialize a new Borg repo named "servera" with no encryption (for this example)
borg init -e none servera

The jail is ready, but it's unreachable from the outside. There are two ways to make it accessible:

Install a VPN system inside the jail itself. Using tools like Zerotier or Tailscale (which don't need to expose ports) will immediately create the conditions to connect to the jail, which will remain inaccessible from the outside. As the jail is a VNET jail, we're free to choose any of the supported VPN technologies.
Expose a port on the backup server, i.e., on the host, to allow external connections. Many advise against this path as they consider it less secure. It is, but sometimes we don't have the luxury of installing whatever we want on the server we're backing up.

To expose the port, go back to the host and modify the /etc/pf.conf file, creating the rdr and pass rules to let packets in:

# ...
# Redirect incoming traffic on port 1122 to the jail's SSH port (22)
rdr on $ext_if inet proto tcp from any to any port = 1122 -> 192.168.0.101 port 22
# ...
# Allow incoming traffic on port 1122
pass in inet proto tcp from any to any port 1122 flags S/SA keep state

Reload the pf configuration:

service pf reload

The jail will now be reachable on the server's public IP, on port 1122. Obviously, this port number is for illustrative purposes, and I used from any, but for better security, you should replace any with the IP address of the server that will be connecting to perform the backup.

By repeating this process for each server and creating a separate jail for each, you can have isolated jails in separate datasets with their backups, potentially setting space limits using ZFS quotas.

It's important to remember that backing up a live filesystem (i.e., without a snapshot or dumps) has a very high probability of being impossible to restore completely. Databases hate this approach because files will change while being copied and tend to get corrupted. Of course, it depends on the nature of the data (a backup of a static website will have no issues, but a WordPress database probably will), but it's crucial to think about a technique to snapshot the filesystem before proceeding. For example, I have already written about how to create snapshots on FreeBSD with UFS in a previous article: FreeBSD tips and tricks: creating snapshots with UFS.

I will cover other operating systems in a future, dedicated post.

Proxmox Backup Server in a Dedicated VM

Starting with version 4.0 (which is still in beta at the time of this writing), Proxmox Backup Server (PBS) supports storing its data in an S3 bucket. This is excellent news as it decouples the server from the data. There are great open-source S3 implementations, like Minio or SeaweedFS, which allow for clustering, replication, etc. In this "simple" case, we will install Proxmox Backup Server in a small VM, while for the data, we'll install Minio in a native FreeBSD jail. The advantage is undeniable: the VM will only serve as an "intermediary", but the data will rest directly on the FreeBSD host's dataset, natively. It will also be possible to specify other external endpoints, other repositories, etc.

As a philosophy, I tend not to use external providers unless for specific needs, so installing Minio in a jail is a perfect solution to manage this situation.

Let's install PBS by downloading the ISO from their website (https://enterprise.proxmox.com/iso/) - at this moment, the version that supports this setup is 4.0 Beta.

The directory to download to is the vm-bhyve ISOs directory. It's not strictly necessary, but it's useful for not "losing" it somewhere. So, go to the directory and download it:

cd /zroot/VMs/.iso
fetch https://enterprise.proxmox.com/iso/proxmox-backup-server_4.0-BETA-1.iso

Now let's create a VM with vm-bhyve. We can start from the Debian template, but we'll make some modifications to optimize performance. In this example, I'm giving it 30 GB of disk space, 2 GB of RAM, and 2 cores.

If you want to store all backups inside the VM, you'll need to size the virtual disk correctly (or create and attach another one). In this case, I will focus on the "clean" VM that will store its data on a dedicated jail with Minio.

vm create -t debian -s 30G -m 2G -c 2 pbs

Once the empty VM is created, let's modify its options:

vm configure pbs

We will modify the VM to be UEFI and to use the NVME disk driver - bhyve performs significantly better on NVME than virtio, as previously tested:

loader="uefi"
cpu="2"
memory="2G"
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"

Fortunately, the Proxmox team has provided for the installation of the Backup Server on devices without a graphical interface, so the boot menu will allow installation via serial console. Let's launch the installation and connect to the virtual serial console:

cd /zroot/VMs/.iso
vm install pbs proxmox-backup-server_4.0-BETA-1.iso
vm console pbs

Select the installation via Terminal UI (serial console) and proceed normally as if it were a physical host, assigning an IPv4 address from the 192.168.0.x range (in this example, I'll use 192.168.0.3).

This way, the Proxmox Backup Server will run in a VM, with the ability to take snapshots before updates, etc., without any worries.

Once the installation is complete, PBS will reboot and listen on port 8007 of its IP. Again, as with the jails, we have two options: install a VPN system within the VM itself (thus exposing it automatically only on that VPN - generally a more secure operation) or expose port 8007 on the server's public IP.

In the latter case, add the relevant lines to the /etc/pf.conf file on the FreeBSD backup server:

# ...
# Redirect incoming traffic on port 8007 to the PBS VM's web interface
rdr on $ext_if inet proto tcp from any to any port = 8007 -> 192.168.0.3 port 8007
# ...
# Allow that traffic to pass
pass in inet proto tcp from any to any port 8007 flags S/SA keep state

Reload the pf configuration:

service pf reload

The PBS VM configuration is complete. If you chose to use the PBS's internal disk as a repository, no further operations are necessary (other than the normal repository creation, etc., within PBS).

In this case, however, we will use a different approach.

Creating a Minio Jail as a Data Repository for PBS

This approach, in my opinion, has a number of important advantages. The first is that Minio will run in a dedicated jail on the host, at full performance, and will store the data directly on the physical ZFS datapool, thus removing any other layer in between. This jail could potentially be moved to other hosts (by connecting PBS and the jail via VPN or public IP), made redundant thanks to all of Minio's features, etc. Another solution I am successfully testing (in other setups) is SeaweedFS.

Let's create a dedicated jail with Minio and put it on the bridge, so that PBS can access it on the LAN.

bastille create -B minio 14.3-RELEASE 192.168.0.11/24 bridge0

If not configured directly, BastilleBSD will use the host's gateway for the jail, which is incorrect in this case. So let's go modify it and restart the jail. Enter the jail with:

bastille console minio

And modify the /etc/rc.conf file to have the correct gateway (following the example addresses):

# ...
ifconfig_vnet0=" inet 192.168.0.11/24 "
defaultrouter="192.168.0.1"
# ...

Exit the jail and restart it:

bastille restart minio

Enter the jail and install Minio:

bastille console minio
pkg install -y minio

Minio is already able to start, but PBS, even on the LAN, wants an encrypted connection. Fortunately, there's a handy tool that can generate the certificates for us:

# Download the certgen tool
fetch https://github.com/minio/certgen/releases/latest/download/certgen-freebsd-amd64

# Make it executable and run it for our jail's IP
chmod a+rx certgen-freebsd-amd64
./certgen-freebsd-amd64  -host "192.168.0.11"

# Create the necessary directories and set permissions
mkdir -p /usr/local/etc/minio/certs
cp private.key public.crt /usr/local/etc/minio/certs/
chown -R minio:minio /usr/local/etc/minio/certs/

Let's view the certificate's fingerprint. Since it's self-signed, we'll need it for PBS later. For security reasons, PBS will ask for the fingerprint of non-directly verifiable certificates. Run the following command and take note of the result:

openssl x509 -in /usr/local/etc/minio/certs/public.crt -noout -fingerprint -sha256

At this point, enable and configure Minio in /etc/rc.conf. WARNING: The username and password (access key and secret) used in this example are insecure and for testing purposes only. It is strongly recommended to use different values:

# Enable Minio service
minio_enable="YES"
# Set the address for the Minio console
minio_console_address=":8751"
# Set the root user and password as environment variables
minio_env="MINIO_ROOT_USER=testaccess MINIO_ROOT_PASSWORD=testsecret"

Start Minio:

service minio start

If everything went correctly, Minio is now running (with its certificates) and ready to receive connections.

It's now time to create the bucket(s) that PBS will use. There are several ways to do this, but to test that everything is working and to configure PBS, I suggest connecting via an SSH tunnel.

# Create an SSH tunnel from your local machine to the backup server
# Port 8007 is forwarded to the PBS web UI
# Port 8751 is forwarded to the Minio console
ssh user@backupServerIP -L8007:192.168.0.3:8007 -L8751:192.168.0.11:8751

This way, we'll create a tunnel between the FreeBSD backup server and our workstation, mapping 127.0.0.1:8007 to 192.168.0.3:8007 (the PBS web interface) and 127.0.0.1:8751 to 192.168.0.11:8751 (the Minio console port).

Now, connect to https://127.0.0.1:8751, enter the credentials specified in /etc/rc.conf, and create a bucket.

Once the bucket is created, you can configure PBS to use it. Connect to PBS via https://127.0.0.1:8007 and go to S3 Endpoints. Set a name, use 192.168.0.11 as the IP and 9000 as the port, enter the access and secret keys, and the certificate fingerprint we generated earlier. Enable "Path Style" or it will not work.

Then go to Datastores and add it, as you would for any other S3 datastore, by specifying the created bucket and a local directory where the system will keep its cache.

If everything was set up correctly, PBS will create its structure in the bucket, and from that moment on, you can use it. Always keep in mind that this is still a "technology preview", so there may be issues, but from my tests, it is sufficiently reliable.

Taking Local Snapshots of Backups

One of the most common techniques used in ransomware attacks is to also delete or encrypt backups. They often use automated methods, relying on the fact that many (too many!) consider a "backup" to be a simple copy of files to a network share. However, it's not impossible that, in specific cases, they might compromise the machine and connect to the backup server. This is nearly impossible with a "pull" type backup (like the one managed by zfs-autobackup) but is still possible with the "push" approach, which involves using BorgBackup or similar tools.

This happened to one of my clients once - in that case, the problem originated internally, from an employee who wanted to cover up his mistake, inadvertently creating a disaster - but that will be material for another post.

Fortunately, the client had a system that solved the problem: thanks to ZFS, we can have local snapshots on the backup server, which are invisible and inaccessible to the production server. Since we have already installed zfs-autobackup, it's easy to use it for this purpose as well. I've already talked about this in a previous article and won't rewrite the steps here. Just consult that article, keeping in mind that in this case, it's not advisable to snapshot all the datasets on the backup server (the space would grow exponentially), but only those at risk. In the cases analyzed in this post, this applies only to the push part, as PBS will also be accessible only from the Proxmox servers and not from the VMs they contain. If, in this case too, you don't trust those who manage the Proxmox servers, just set up snapshots for the Minio jail as well.

Conclusion

This long post aims to analyze, in a general way, how I believe one can manage reasonably secure backups of their data. Obviously, there are many variables, additional precautions, possible optimizations, hardening, etc., that must be studied on a case-by-case basis. There are old rules, new rules, old and new philosophies. Recently, many people who have embraced the cloud have often stopped thinking about backups, only to realize it when something happens and the data has, indeed, vanished... into the clouds.

In this post, I have generically covered the setup of the backup server, and this demonstrates how FreeBSD, thanks to its features, can be considered an ideal platform for this type of task.

In the next articles in this series, I will examine the client side, i.e., how to structure them for a sufficiently reliable backup, and how to monitor everything - because I've seen this too: people resting easy because the backup was supposedly running every night, but in fact, the backup had been failing every night for more than 4 years.

Stay Tuned and stay...backupped!

Migrating Windows VMs from Proxmox BIOS/KVM to FreeBSD UEFI/bhyve

Stefano Marinelli — Fri, 15 Nov 2024 10:57:00 +0100

A client of mine has several Windows Server VMs, which I had not migrated to FreeBSD/bhyve until a few weeks ago. These VMs were originally installed with the traditional BIOS boot mode, not UEFI, on Proxmox. Fortunately, their virtual disks are on ZFS, which allowed me to test and achieve the final result in just a few steps.

This is because Windows VMs (server or otherwise) often installed on KVM (Proxmox, etc.), especially older ones, are non-UEFI, using the traditional BIOS boot mode. bhyve doesn’t support this setup, but Windows allows changing the boot mode, and I could perform the migration directly on the target FreeBSD server.

Setting Up the Network

Before starting, as usual in my setups, I manually created the bridge and configured a few pf rules for outbound NAT, which these servers need. Let’s begin with the modifications to /etc/rc.conf:

cloned_interfaces="bridge0"
ifconfig_bridge0="inet 192.168.33.1 netmask 255.255.255.0"
gateway_enable="YES"
pf_enable="YES"

Next, for the pf.conf:

ext_if="igb0"

set block-policy return
set skip on bridge0
set skip on lo0

# Allow the VMs to connect to the outside world
nat on $ext_if from {192.168.33.0/24} to any -> ($ext_if)

block in all
antispoof for $ext_if inet
pass in inet proto icmp
pass out quick keep state
pass in inet proto tcp from any to any port ssh flags S/SA keep state

To ensure everything is working as expected, I recommend rebooting the server. It’s not strictly necessary since you could reload network configurations and start pf without a reboot, but I prefer a few extra reboots in these cases. Why? Because system administrators are often afraid to reboot production servers, unsure if they will come back up smoothly. A configuration error could prevent a successful reboot, and this would be a significant issue if the server is already in production. I’ve faced this situation myself—many times! To mitigate this, I make frequent reboots, especially after changing network configurations, ensuring everything starts correctly on reboot.

Installing and Configuring vm-bhyve

Next, I installed vm-bhyve on the target FreeBSD host and configured it:

pkg install vm-bhyve-devel bhyve-firmware
zfs create zroot/VMs
sysrc vm_enable="YES"
sysrc vm_dir="zfs:zroot/VMs"
vm init
cp /usr/local/share/examples/vm-bhyve/* /zroot/VMs/.templates/
vm switch create -t manual -b bridge0 public

Usually, I enable the serial console on tmux, but in this case, I’ll skip that since Windows VMs need a graphical console. If the FreeBSD server is not on your local network, I suggest connecting via SSH with port forwarding for the necessary VNC ports (e.g., -L5900:127.0.0.1:5900) to connect to bhyve via SSH tunnel. Never expose the VNC port directly!

Creating the VM Template

Now, I created an “empty” VM using the “Windows” template with vm-bhyve and adjusted the configuration afterward:

vm create -t windows -s 1G -m 16G -c 4 vm115

I created a virtual disk of 1 GB because it will be replaced by the dataset sent from the current production server, so it’s a fictitious size. At that point, I deleted the disk0 image file:

rm /zroot/VMs/vm115/disk0.img

Modifying the VM Configuration

Next, I modified the VM configuration as follows:

Changed the disk configuration to use a zvol, simplifying the send-receive operation from the original Proxmox host.
In some cases, older Windows installations may not support the nvme driver. If this happens, use ahci-hd instead, as after the conversion Windows may fail to boot and display a BSOD.
Changed the network adapter driver to virtio-net (since it was already installed on the Proxmox VM).
Set the network adapter MAC address to match the original VM.

After running vm configure vm115, the configuration should look something like this:

loader="uefi"
graphics="yes"
xhci_mouse="yes"
cpu="4"
memory="16G"

ahci_device_limit="8"

network0_type="virtio-net"
network0_switch="public"

disk0_type="nvme"
disk0_name="disk0"
disk0_dev="sparse-zvol"

utctime="no"
uuid="myUUID"
network0_mac="theProxmoxVirtualMachineMacAddress"

Copying the Virtual Disk

I took a snapshot of the virtual disks of the VMs on the original server and copied them to the target server:

zfs snapshot vmpool/vm-115-disk-0@toSend01
zfs send -Rcv vmpool/vm-115-disk-0@toSend01 | mbuffer -m 128M | ssh user@FreebsdHostIP "zfs receive -F zroot/VMs/vm115/disk0"

The VM’s disk will be transferred to the target host.

Converting BIOS to UEFI

At this point, bhyve expects a UEFI VM, but the VM we just copied is BIOS-based. In other words, it won’t boot as is.

Luckily, Microsoft provides a tool to convert partitions and change the server from BIOS to UEFI. Of course, I would only use such a tool on a snapshot, and while I could do it directly on the source server, I prefer minimizing downtime and avoiding touching the original server. I want to perform the conversion directly on the FreeBSD host, which also makes disaster recovery easier, allowing me to manage everything on FreeBSD without first restoring the VM on a Proxmox server.

The first step is to download a Windows installation ISO (in my case, Windows Server). I downloaded the ISO from here: Windows Server Evaluation Center.

Once ready, launch the ISO as if you were installing Windows on the VM:

vm install vm115 SERVER_EVAL_x64FRE_en-us.iso

Running the Windows Repair Tool

Now the VM will boot. Connect via VNC (if using VNC tunneling, connect to 127.0.0.1:5900) to proceed. The ISO will prompt you to press a key to boot from the CDROM—go ahead and do that.

Proceed as if you were repairing your installation ("Repair your computer"), then select Troubleshoot -> Command Prompt.

Once at the prompt, run a validation check first (to avoid making changes if the disk has any unusual configurations):

mbr2gpt /validate /disk:0

If the validation is successful, proceed with the actual conversion:

mbr2gpt /convert /disk:0

Windows will convert the partition and indicate success at the end.

Booting the VM

Close the command prompt and shut down the VM. Everything should be ready now—easy peasy.

Boot the VM with:

vm start vm115

Connecting via VNC to the VM will bring up the Windows boot screen. In case of BSOD, change the nvmedriver to ahci-hdand it should be working. If the snapshot was taken while the VM was powered on, it may perform a disk check. If everything went as planned, you should see the Windows login screen.

Additionally, Windows will likely require reactivation of the VM, as it will detect hardware changes.

Final Considerations

At this point, you need to decide whether to continue with this VM or resynchronize it with the original (e.g., if the contents have changed). In the first case, no further action is needed. In the second case, the proper procedure would be to power off the original VM, create a new snapshot, roll back the snapshot on the FreeBSD host, transfer it incrementally—a process I’ve described in a previous post—and then redo the conversion.

To configure the VM to boot automatically when the FreeBSD host boots, add the necessary settings to /etc/rc.conf, as described in a previous article.

From Proxmox to FreeBSD - Story of a Migration

Stefano Marinelli — Mon, 21 Oct 2024 07:33:00 +0200

I have been managing a client's server for several years. I inherited a setup (actually, partly done by me at the time, but following the directions of their internal administrator) based on Proxmox. Originally, the VM disks were qcow2 files, but over time and with Proxmox updates, I managed to create a ZFS pool and move them onto it. For backups, I continued to use Proxmox Backup Server (even though virtualized on bhyve) - a solution we've been using for several years.

The existing VMs/containers were:

An LXC container based on Debian, with Apache as a reverse proxy. The choice of Apache was mainly tied to specific configurations from the company that produces the underlying software. It has always done an excellent job.
A VM running OPNsense — the choice was related to direct use by the client, as it allows easy creation/modification of users for the internal VPN. Some users can access via VPN (OpenVPN) and perform specific operations, such as SSH connections. It has always worked well and is up to date.
Two VMs based on Rocky Linux 9 — both with components of the management software in Java, with the reverse proxy directing requests based on the requested site.
Two VMs based on Rocky Linux 8 — these also have parts of the management software and the databases. These machines are used both by providing specific URLs and by the other two as databases.

The load is not particularly high, and the machines have good performance. Suddenly, however, I received a notification: one of the NVMe drives died abruptly, and the server rebooted. ZFS did its job, and everything remained sufficiently secure, but since it's a leased server and already several years old, I spoke with the client and proposed getting more recent hardware and redoing the setup based on a FreeBSD host.

Unfortunately, I cannot change the setup of the VMs. In my opinion, there are no contraindications since the databases are PostgreSQL and the management applications are Java applications with Tomcat. However, the software house only certifies that type of setup, and considering they are the ones performing updates and fixes, it's appropriate to maintain the setup they suggest. It's a technically sound choice in my opinion, so I can't say anything negative.

Acquiring New Hardware

The first step was, of course, to get the new server. It took a few days because I requested ECC RAM (for a small price variation), and the time extended. As soon as the physical server was ready, I got to work.

I installed FreeBSD 14.1-RELEASE, root on ZFS, creating a mirror setup between the NVMe drives. The I/O is not particularly heavy in this setup, but I don't want to waste potential: even if I can reduce the length of an operation by a few seconds, I see no reason not to do it.

As the first step, I decided to manually create the bridge to which I will connect both the VMs and the reverse proxy, which will be installed inside a FreeBSD jail.

vm-bhyve, the tool I use to manage VMs, allows creating bridges, but I prefer to manage it manually and maintain more complete control over everything. I will also enable pf.

I decided to use zstd as the compression algorithm and disable atime:

zfs set compression=zstd zroot
zfs set atime=off zroot

I then modified the /etc/rc.conf file as follows:

cloned_interfaces="bridge0 lo1"
ifconfig_lo1_name="bastille0"
ifconfig_bridge0="inet 192.168.33.1 netmask 255.255.255.0"
gateway_enable="YES"
pf_enable="YES"
bastille_enable="YES"

I then updated FreeBSD by running the usual command:

freebsd-update fetch install

Configuring the Firewall

I configured the firewall with a basic setup:

ext_if="igb0"

set block-policy return
set skip on bridge0

table <jails> persist
nat on $ext_if from {192.168.33.0/24} to any -> ($ext_if)
nat on $ext_if from <jails> to any -> ($ext_if:0)

# nginx-proxy - to the jail
rdr on $ext_if inet proto tcp from any to PUBLIC_IP port = 80 -> 192.168.33.254 port 80
rdr on $ext_if inet proto tcp from any to PUBLIC_IP port = 443 -> 192.168.33.254 port 443

# opnsense
rdr on $ext_if inet proto tcp from any to PUBLIC_IP port = 1194 -> 192.168.33.253 port 1194
rdr on $ext_if inet proto udp from any to PUBLIC_IP port = 1194 -> 192.168.33.253 port 1194

rdr-anchor "rdr/*"

block in all
antispoof for $ext_if inet
pass in inet proto icmp
pass out quick keep state
pass in inet proto tcp from any to any port {http,https} flags S/SA keep state
pass in inet proto {tcp,udp} from any to any port 1194 flags S/SA keep state
# This will be closed at the end of the setup and will be allowed only via VPN
pass in inet proto tcp from any to any port ssh flags S/SA keep state

Once finished, I rebooted everything to load the new kernel. No problems.

Installing Necessary Tools

I then installed some useful packages:

pkg install tmux py311-zfs-autobackup mbuffer rsync vm-bhyve-devel edk2-bhyve grub2-bhyve bastille

I created the datasets for the jails and VMs:

zfs create zroot/bastille
zfs create zroot/VMs

I then started configuring everything. First, BastilleBSD: I modified /usr/local/etc/bastille/bastille.conf by adding:

## ZFS options
bastille_zfs_enable="YES"
bastille_zfs_zpool="zroot/bastille"

Then I enabled and configured vm-bhyve, enabling the serial console on tmux:

sysrc vm_enable="YES"
sysrc vm_dir="zfs:zroot/VMs"
vm init
cp /usr/local/share/examples/vm-bhyve/* /zroot/VMs/.templates/
vm switch create -t manual -b bridge0 public
vm set console=tmux

Now I bootstrapped FreeBSD 14.1-RELEASE on BastilleBSD:

bastille bootstrap 14.1-RELEASE update

Setting Up the Reverse Proxy Jail

I then started by creating the jail for the reverse proxy:

bastille create -B apache 14.1-RELEASE 192.168.33.254/24 bridge0

Once created, I had to modify the default gateway of the jail because by default it is set to that of the host. It was enough to set 192.168.33.1 in /usr/local/bastille/jails/apache/root/etc/rc.conf.

The configuration of Apache will not be described here as it is closely dependent on the setup, but this jail can reach (in bridge mode) all the VMs that will be placed on the same bridge.

Migrating the VMs

For migrating the VMs, I decided to proceed as follows:

Updating the operating systems of the original VMs - to have the latest version of the kernel and all userland. The VMs are not UEFI, so it will be necessary to have the exact names of the kernel and initrd as they will need to be specified in the configuration file of vm-bhyve.
Creating templates of the VMs with vm-bhyve.
Snapshotting the VM disks and initial transfer to the new server with the VMs still running.
Shutting down all the source VMs.
Creating a new snapshot and performing an incremental transfer.
Adjusting configurations and changing DNS.

I updated everything using dnf, and rebooted when there was also a kernel update. I took note of the versions to use, namely vmlinuz-4.18.0-513.18.1.el8_9.x86_64 and initramfs-4.18.0-513.18.1.el8_9.x86_64.img for Rocky Linux 8.10, and vmlinuz-5.14.0-362.18.1.el9_3.0.1.x86_64 and initramfs-5.14.0-362.18.1.el9_3.0.1.x86_64.img for Rocky Linux 9.4.

Migrating Rocky Linux 8.10 VMs

On the destination server, I created the VMs. I used the linux-zvol template, then modified the configurations:

vm create -t linux-zvol -s 1G -m 8G -c 4 vm100

I created a virtual disk of 1 GB because it will be replaced by the dataset sent from the current production server, so it's a fictitious size. At that point, I deleted the zvol:

zfs destroy zroot/VMs/vm100/disk0

I performed an initial snapshot of the source VMs:

zfs snapshot zfspool/vm-100-disk-0@Move01

And then I copied this first snapshot to the destination machine/VM:

zfs send -R zfspool/vm-100-disk-0@Move01 | mbuffer -s 128k -m 512M | ssh root@destMachine "zfs receive -F zroot/VMs/vm100/disk0"

At the end of the copy, you can do a test and try to boot. For the Rocky Linux 8.10 VMs, I had no problems because the /boot partition is in ext4. I then configured the VM as follows, running the command vm configure vm100:

loader="grub"
cpu="8"
memory="12G"
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="sparse-zvol"
uuid="the-uuid"
network0_mac="the-mac"
grub_run0="linux /vmlinuz-4.18.0-553.22.1.el8_10.x86_64 root=/dev/vda3"
grub_run1="initrd /initramfs-4.18.0-553.22.1.el8_10.x86_64.img"

All I needed to do is: pass to GRUB the name of the kernel and the initramfs. It is now possible to start the machine and do a test and connect to the console:

vm start vm100
vm console vm100

In theory, the machine should boot correctly. It will probably be necessary to change the network interface configurations - in my case, even keeping the virtio and the MAC address, it changed from ens18 to enp0s5 - but everything else should be fine.

At this point, if the source VM was turned off/not reachable or simply there is nothing new to synchronize, the migration is complete. Otherwise, it will be necessary to shut down the source VM, create a new snapshot, and transfer it. Let's start by shutting down the source VM and the destination one (vm stop vm100), roll back to the snapshot transferred to the destination server, create a new snapshot, and perform an incremental transfer:

zfs rollback zroot/VMs/vm100/disk0@Move01

On the source server:

zfs snapshot zfspool/vm-100-disk-0@Move02
zfs send -R -i zfspool/vm-100-disk-0@Move01 zfspool/vm-100-disk-0@Move02 | mbuffer -s 128k -m 512M | ssh root@destMachine "zfs receive -F zroot/VMs/vm100/disk0"

In this way, we have updated the destination VM and transferred only the differences, without altering the configuration of vm-bhyve.

Start the machine - if all goes well, the migration of this VM is finished.

Migrating Rocky Linux 9.4 VMs

For the Rocky Linux 9.4 VMs, the situation was more complex: the partitions were all - even /boot—in XFS. And, for some reason, the GRUB launched by bhyve cannot read from XFS. Therefore, I had to proceed differently.

I copied, using rsync, the /boot of the original VPS (in ZFS) to a directory on the FreeBSD server (specifically, on the VPS I ran the command: rsync -avhHPx --numeric-ids /boot root@FreeBSDHOST:/tmp/). In this way, I kept the files available.

I shut down the machine on the original server and copied its zvol to the new FreeBSD host using the same method as the others. At that point, I recreated the /boot partition of my VPS in ext3 - directly from the FreeBSD host:

pkg install fusefs-ext2
kldload fusefs
mkfs.ext3 /dev/zvol/zroot/VMs/vm104/disk0s1
fuse-ext2 -o force /dev/zvol/zroot/VMs/vm104/disk0s1 /mnt/
cd /mnt
rsync -avhHPx /tmp/boot/. .
umount /mnt

In this way, GRUB will be able to access the /boot partition to load the kernel and initramfs.

Here is the vm-bhyve configuration for this VM:

loader="grub"
cpu="8"
memory="12G"
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="sparse-zvol"
uuid="the-uuid"
network0_mac="the-mac"
grub_run0="linux /vmlinuz-5.14.0-427.37.1.el9_4.x86_64 root=/dev/vda3"
grub_run1="initrd /initramfs-5.14.0-427.37.1.el9_4.x86_64.img"

Launching the machine, however, it will hang during boot and, after a timeout, will ask for administrator credentials to enter a recovery shell. This is because the blkid of the /boot partition has changed, and the fstab of the VM still reports the data of the old XFS partition. In this case, I used the blkid command in the VM and copied the UUID of the new partition. Then modify the /etc/fstab file of the VM and put the new blkid, as well as changing "xfs" to "ext3". After a reboot, the system should start without problems, again with a network card to reconfigure.

This procedure worked correctly for all VMs. In this way, the storage remains on virtio-blk even if it would be optimal for the driver to be changed to nvme. To make this change, you will need to enter the VMs, create a file called /etc/dracut.conf.d/00-custom.conf:

add_drivers+=" nvme "

And regenerate the initramfs - in this way, the nvme driver will be supported at boot:

dracut --regenerate-all --force

It will now be enough to change the configurations of vm-bhyve—virtio-blk becomes nvme, and /dev/vda3 becomes /dev/nvme0n1p3, for example:

loader="grub"
cpu="8"
memory="12G"
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0"
disk0_dev="sparse-zvol"
uuid="the-uuid"
network0_mac="the-mac"
grub_run0="linux /vmlinuz-5.14.0-427.37.1.el9_4.x86_64 root=/dev/nvme0n1p3"
grub_run1="initrd /initramfs-5.14.0-427.37.1.el9_4.x86_64.img"

Migrating the OPNsense VM

For the VM with OPNsense, the procedure was even simpler. It was enough to create a VM of type freebsd-zvol and copy the disk as done for the others. In this case, I replicated the MAC address of the original virtualized network interface (Proxmox server) to ensure that the underlying FreeBSD recognizes it as the same. FreeBSD is less picky about these things.

The final vm-bhyve configuration file will be:

loader="bhyveload"
cpu="2"
memory="1G"
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="sparse-zvol"
uuid="my-uuid"
network0_mac="my-mac"

Configuring Automatic VM Startup

To ensure that the VMs all start at boot, just add them to /etc/rc.conf as follows, giving a 15-second delay between one VM and another:

[...]
vm_enable="YES"
vm_dir="zfs:zroot/VMs"
vm_list="vm100 vm101 [...] opnsense"
vm_delay="15"
[...]

Setting Up Backups

At this point, I configured external backups, every hour, similarly to how I described in a previous article. I also added a local snapshot every 15 minutes, always using zfs-autobackup. To do this, I created a new tag:

zfs set autobackup:localsnap=true zroot

Then I modified the /etc/crontab file, adding this line:

*/15    *       *       *       *       root    /usr/local/bin/zfs-autobackup localsnap --keep-source 15min3h,1h1d > /dev/null 2>&1

That is, it will perform a snapshot every 15 minutes and keep them for 3 hours, then keep one per hour for a day. In this way, in case of quick recovery due to a problem/error, I won't need to transfer the entire dataset/VM from the backup.

Conclusion

The migration is complete: after changing the DNS, the client performed some tests, and everything works properly. The setup has been active for over a week, and there have been no issues. The performance is excellent; I did not perform tests compared to the previous setup since the hardware of the new FreeBSD host is more powerful and modern, so it wouldn't make sense.

Apart from the manager, no user was informed of the change, and in the last week, no reports have been received. The VMs are stable, and the host's load is very low.

The operation actually took less than two hours in total - most of the time has been consumed by the send/receive operations - and gave excellent results. The alternative would have been to install Proxmox on a new host and move the VMs - I probably would have saved a few minutes, but now I can use bhyve, as well as the simplicity and power of the underlying FreeBSD.

From Cloud Chaos to FreeBSD Efficiency

Stefano Marinelli — Thu, 04 Jul 2024 08:41:00 +0200

Introduction

A few months ago, a client asked me to take care of their Kubernetes cluster (hosted on AWS and GCP). In their opinion, the costs were exorbitantly high for relatively simple and lean websites. Sure, they had many visits, but nothing too excessive development-wise.

I kindly declined. Unfortunately, their situation is all too common these days: they hired developers accustomed to working that way, convinced that a system administrator is now unnecessary because "the cloud has infinite potential." They were used to considering optimization as secondary because "we have infinite power" (and this is already a spoiler for the ending).

Being open to dialogue and new experiences, they asked for my opinion on the matter. We talked for a while, and I explained that, in my view, for the type of setup they had (standard, with various replicas and variants, but primarily based on two platforms), it didn't make sense. I saw it as complicating things. An over-engineering of something simple. Like taking a cruise ship to cross a river.

They then asked me to create something simple that would serve as a development server and for backups, to understand what kind of solution I had in mind.

The Solution

So, I started building everything. I began with FreeBSD 13.2-RELEASE, but in the meantime, 14.0-RELEASE came out, so that’s the version I delivered.

I installed the operating system on a physical server, leased from one of the main European providers. Benefiting from one of their auctions (good deals can be found on weekends), they found a sufficiently powerful machine, with 128GB of RAM, 2 NVMe drives of 1TB each, and two spinning disks of 2TB each for less than 100 euros per month. They also took another, less powerful one for additional backups and to back up the first one.

Implementation

I decided to keep the host as clean as possible and concentrated the services in jails (managed by BastilleBSD) and VMs. The machine was divided as follows:

A series of bridges - to be used for different projects. Jails of the same project and/or type use the same bridge and can communicate with each other, sharing some resources (MariaDB, etc.).
A bhyve VM with Alpine Linux - in my opinion, the best distribution for running Docker containers. Do we really need systemd just to launch Docker? They mainly use it as a pre-production test bench, connected via VPN to their company LAN. It is the core of their "online" development, i.e., outside their computers. It has 32GB of RAM, 200GB of disk (obviously bhyve is configured with NVMe drivers), and 4 cores assigned.
A VNET jail with a reverse proxy (nginx) - they know how to modify virtual hosts and generate certificates with certbot, pointing to the underlying jails.
A series of "empty" VNET jails, to be cloned, for each type of setup (they mainly have CMS based on WordPress and Laravel, so with all dependencies inside - nginx, php, redis, etc. except the databases).
A VNET jail with MariaDB installed, to be cloned, to be attached to different projects as needed.
zfs-autobackup performs local snapshots, keeping: one every 15 minutes for 3 hours, one per hour for 24 hours, one per day for 3 days.

Backups are also performed using zfs-autobackup and, in case of disaster recovery in rapid times, a zfs-send (and corresponding zfs-receive) every 10 minutes on another machine (the other, smaller one, also taken at auction), with the same bridges, firewall rules, BastilleBSD, and bhyve installed - ready to start in case of disaster. Being a test server, we didn't consider to implement a proper HA - at the moment, it wouldn't make sense.

They also have another job with zfs-autobackup that performs an additional backup on a server (Debian in their offices). Safe data, in my opinion, are those in storage under your b...ench.

I delivered everything to them and gave a brief course to the more experienced devs on how to manage things. No explanation on the Alpine Linux VM, but I showed them the jails, how to clone, configure, and manage them.

Real-world Testing

I didn't hear from them anymore. After a few weeks, one of the devs contacted me urgently because a junior unfortunately made a mistake and deleted an entire project from one of the jails. I explained that the local snapshots were restorable with a command, and he was thrilled. He restored both the development jail and the one with the database made two minutes before the "mishap" and they restarted immediately.

I realized that this event would change some of their procedures and criteria.

I hadn't heard from anyone for months. This morning, I received a call from their manager, whom I hadn't heard from since the beginning, and he told me how things had been going these months.

Lessons Learned

First, this person has good communication and commercial skills but little technical background. He is open-minded and tends to study carefully what is proposed to him. He doesn't discard any solution a priori, without having touched its pros and cons.

They had leased servers with cPanel and were inserting their content inside them. The devs who arrived a few years ago suggested making a technological transition, eliminating these "obsolete" servers and "outdated" methodologies, pushing everything to the cloud and containerizing everything. When we first talked, he told me how they were "lucky to make that transition because their load had increased enormously and the old servers probably wouldn't have handled the load", instead autoscaling saved them. I had some reservations about autoscaling without particular controls, but clearly, I cannot impose my choices on others.

To cut a long story short: seeing what happened with that junior dev's mistake (and the simplicity with which it was possible to restart immediately), they decided to increase the use of FreeBSD jails and reduce, at least on secondary loads, the use of their Cloud managed with Kubernetes. As they transitioned to jails, however, they noticed some slowdowns. These slowdowns worsened day by day. According to the devs, it would have been appropriate to go back to having, again, autoscaling ("we need moar powaaaaar!!!") but, fortunately, their boss decided to investigate carefully. They realized that these workloads (based on Laravel) were storing sessions on files. Over time, these millions of files (several gigabytes per day) slowed everything down because, for specific operations, Laravel scanned the entire directory. In other words, on the "cloud," they needed much more power than necessary (and much more disk space, but that was cheaper) to carry this load, which was, in fact, unnecessary. After realizing this, they moved the sessions to Redis. Needless to say, everything became extremely faster, even compared to the previous setup on Kubernetes and autoscaling.

At that point, it was clear that one of the problems with their setup is (as often happens) poor optimization. Today, there's a tendency to rush, "throw in" functions, features, libraries, plugins, etc. without considering the interactions and consequences. If it works, it's fine. Even if it increases computational complexity exponentially just to, for example, change the color of an icon (absurd example, but to give an idea).

They then started moving even the main Laravel workloads (thanks to the optimization implemented). At this point, they began moving some of the WordPress sites even though they were extremely concerned. In the cluster, every day, at fairly irregular intervals, the load would rise and everything would slow down until autoscaling started scaling up to the imposed limits. CPU at 100% on all containers, and the devs noticed that the load came from a series of "php" processes. Recreating the containers helped for some minutes, but did not solve the problem.

To their great surprise, all this did not happen on the FreeBSD jails. The load was significantly lower, without any of these spikes. Satisfied, they decided to use this as their final setup. One of the devs, however, wanted to get to the bottom of it and decided to run a test: he moved some of these WordPress sites to the Alpine VM, on Docker. At that point, the spikes resumed, saturating the CPU of the Alpine machine.

Without going into details, they eventually realized that there was a vulnerability in one (or more) of the many plugins installed on the WordPress sites, which was being exploited to inject a process, probably a cryptominer. The name given to the process was "php" - so the devs, not being system experts, did not worry about understanding better whether it was really php or another process pretending to be it. On FreeBSD, all this did not happen because the injected executable could not run - there was no Linux compatibility activated on the server.

Until then, they considered these (expensive) spikes as organic and did not worry too much about them. Paying to have their friendly intruders mine.

Conclusion

They asked me to help, as much as possible, to move other services to FreeBSD. It won't be easy, probably we will need to use bhyve a lot, but they decided that this is the platform they want to focus on in the coming years.

Undoubtedly, this is a success story of FreeBSD and, indirectly, of correct and careful management of one's resources. Too often today, there is the superficial belief that the cloud, with its "infinite" resources, is the solution to all problems. And that Kubernetes is the best solution for everything. I, on the other hand, have always believed that there is the right tool for everything. You can hammer a nail with a screwdriver, but it's not the most suitable and efficient tool.

Today they spend about 1/10 of what they used to spend before, they have more control over their data and the tools they use. Undoubtedly, all this was also caused by poor optimization and control by those who manage the infrastructure, but the question is: how often do people decide that, in the end, it is okay to spend more (especially if it is someone else's money) rather than go crazy for hours behind such a situation? While having defined and limited resources (albeit elevated) poses different problems - but of optimization. And in the age of energy and resource savings, it might be wise to give more importance to optimization.

Abundance led to waste.

Proxmox vs FreeBSD: Which Virtualization Host Performs Better?

Stefano Marinelli — Mon, 10 Jun 2024 05:53:45 +0000

TL;DR: Skip to the Conclusion for a summary.

Preamble

I have always been passionate about virtualization and have consistently used it.

The first solution I installed on my infrastructures (and those of clients) was Xen on NetBSD, with great success. I then used Xen on Linux and, since 2012, OpenNebula, followed by Proxmox in 2013. Proxmox has always given me great satisfaction, and even today I consider it a valuable platform that I install gladly. I have also used other hypervisors like XCP-ng but less frequently, and in recent years, I have started to make extensive use of bhyve.

About two and a half years ago, we began a progressive process of migrating our servers (and those of our clients) from Linux to FreeBSD, using jails (when possible) or VMs on bhyve. In some cases, migrating setups from Proxmox to FreeBSD resulted in performance improvements, even with the same hardware. In some instances, I migrated VMs without notifying clients, and they contacted me a few days later to inquire if we had new hardware because they noticed better performance.

After years, I decided to conduct a test to determine if this was just a perception or if there was a technical basis behind it. Of course, this test has no scientific validity, and the results were obtained on specific hardware and at a specific time, so on different hardware, workload, and situations, the results could be entirely opposite. However, I tried to have as scientific and objective an approach as possible since I am comparing two solutions that I care about and use daily.

Hardware and Test Conditions

I often see comparative tests done on VMs from various providers. In my opinion, this comparison makes no sense because a VM from any provider shares its hardware with many other VMs, so the results will vary depending on the load of the "neighbors" and will never be reliable.

For this test, I decided to take a physical server with the following characteristics:

Intel Core i7-6700
2x SSD M.2 NVMe 512 GB
4x RAM 16384 MB DDR4
NIC 1 Gbit Intel I219-LM

The hardware is not recent, but still very widespread. On more recent hardware, the results might differ, but the test will be based on this configuration.

I installed Proxmox 8.2.2 starting from the Debian template of the provider and manually installed it following the instructions. I created a partition for Proxmox and left one partition free on each of the two NVME drives to create (at different times) the ZFS pool (in mirror) and the LVM on top of the Linux software raid.

After all the tests, I installed FreeBSD 14.1-RELEASE on ZFS on the same host, using bsdinstall from an mfsbsd image since the provider does not directly support installing FreeBSD from its panel or rescue mode.

In both installations, I always trimmed the NVME drives before starting the tests, and in the case of ZFS, I set (both on Proxmox and FreeBSD) compression to zstd and atime to off. No other changes were made compared to the standard installation.

On FreeBSD, the VM was created and managed with vm-bhyve (devel).

On Proxmox, I tested the physical host on ZFS and ext4 and the VM on ZFS and LVM as LVM is the standard and most common setup in Proxmox.

On FreeBSD, I tested the host on ZFS and the VM with both virtio and nvme drivers, on zvol, and as an image file within a ZFS dataset.

I used sysbench installed from the official Debian repository (on Proxmox and VM) and from the FreeBSD package on the respective host.

The VMs, both on Proxmox and FreeBSD, have nearly identical characteristics and default configuration (apart from the nvme drivers set on bhyve, and for that reason, I also tested virtio).

For those who want to reproduce my tests, here are the detailed configurations of the VMs used in bhyve:

FreeBSD bhyve VM Configuration with NVMe Driver:

loader="uefi"
cpu=4
memory=4096M
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"

FreeBSD bhyve VM Configuration with virtio Driver:

loader="uefi"
cpu=4
memory=4096M
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0.img"

Proxmox VM Configuration:

Component	Details
Memory	4.00 GiB [balloon=0]
Processors	4 (1 sockets, 4 cores) [x86-64-v2-AES]
BIOS	Default (SeaBIOS)
Display	Default
Machine	Default (i440fx)
SCSI Controller	VirtIO SCSI single
CD/DVD Drive (ide2)	local:iso/debian-12.5.0-amd64-netinst.iso,media=cdrom,size=629M
Hard Disk (scsi0)	zfspool:vm-100-disk-0,cache=writeback,discard=on,iothread=1,size=50G,ssd=1
Network Device (net0)	virtio=BC:24:11:22:3D:F0,bridge=vmbr0

In all the configurations, I used Debian 12 as the VM operating system, with the file system on ext4.

I chose Debian 12 as it is a stable, widespread, and modern Linux distribution. I did not test a FreeBSD VM because, in my setups, I tend not to virtualize FreeBSD on FreeBSD but to use nested jails.

All tests were performed multiple times, and I took the median results. CPU and RAM were tested only on the first VM (on Proxmox (ZFS) and FreeBSD (ZFS and nvme)) as they are not dependent on the underlying storage. Storage performance, on the other hand, was tested on all configurations.

CPU and RAM Tests on VMs

On both VMs:

sysbench --test=cpu --cpu-max-prime=20000 run
sysbench --test=memory run

Comparative Results

CPU Test

Configuration	Events per Second	Total Time (s)	Latency (avg) (ms)
Proxmox	498.08	10.0010	2.01
FreeBSD	473.65	10.0019	2.11

CPU Percentage Analysis

Difference in Events per Second: ((498.08 - 473.65) / 498.08 approx -4.91\%)
Difference in Total Time: ((10.0019 - 10.0010) / 10.0010 approx +0.009\%)
Difference in Latency (avg): ((2.11 - 2.01) / 2.01 approx +4.98\%)

RAM Test

Configuration	Total Operations	Operations per Second	Total MiB Transferred	MiB/sec	Latency (avg) (ms)
Proxmox	64777227	6476757.59	63259.01	6324.96	0.00
FreeBSD	68621063	6861139.06	67012.76	6700.33	0.00

RAM Percentage Analysis

Difference in Total Operations: ((68621063 - 64777227) / 64777227 approx +5.94\%)
Difference in Operations per Second: ((6861139.06 - 6476757.59) / 6476757.59 approx +5.94\%)
Difference in Total MiB Transferred: ((67012.76 - 63259.01) / 63259.01 approx +5.93\%)
Difference in MiB/sec: ((6700.33 - 6324.96) / 6324.96 approx +5.93\%)

CPU and RAM Comparative Results Table

Test	Metric	Proxmox (KVM)	FreeBSD (bhyve)	Difference (%)
CPU	Events/s	498.08	473.65	-4.91
	Time (s)	10.0010	10.0019	+0.009
	Latency	2.01	2.11	+4.98
RAM	Ops	64777227	68621063	+5.94
	Ops/s	6476757.59	6861139.06	+5.94
	MiB	63259.01	67012.76	+5.93
	MiB/s	6324.96	6700.33	+5.93
	Latency	0.00	0.00	0.00

Interpretation of CPU and RAM Results

CPU Performance:
The VM on FreeBSD has slightly lower CPU performance compared to Proxmox (-4.91% in events per second).
The total execution time is nearly identical, with a negligible difference.
The average latency is slightly higher on FreeBSD (+4.98%).
RAM Performance:
The VM on FreeBSD has better RAM performance compared to Proxmox (+5.94% in operations and MiB/sec).
The average latency is identical in both configurations.

In summary, while Proxmox provides more consistent CPU performance, FreeBSD demonstrates superior memory performance. The choice between Proxmox and FreeBSD may depend on the specific workload requirements and the importance of consistent performance versus higher throughput.

I/O Performance Tests

The test has been conducted using sysbench, with this command line:

sysbench --test=fileio --file-total-size=30G prepare
sysbench --test=fileio --file-total-size=30G --file-test-mode=rndrw  --max-time=300 --max-requests=0 run

I/O Comparative Performance Data with Percentage Differences

Metric	VM on Proxmox (ZFS)	VM on Proxmox (LVM)	VM on FreeBSD (ZFS, NVMe)	VM on FreeBSD (ZFS, Virtio)	VM on FreeBSD (zvol)	Host FreeBSD (ZFS)	Host Proxmox (ZFS)	Host Proxmox (ext4)
File creation speed (MiB/s)	407.82	461.52	1467.83	1398.81	1333.64	1625.67	968.64	633.13
Reads per second	650.09	504.80	11183.44	806.93	11834.53	1234.62	920.95	498.37
Writes per second	433.40	336.54	7455.62	537.95	7889.69	823.08	613.96	332.25
fsyncs per second	1387.08	1076.97	23858.08	1721.79	25247.36	2634.01	1964.96	1063.19
Read throughput (MiB/s)	10.16	7.89	174.74	12.61	184.91	19.29	14.39	7.79
Write throughput (MiB/s)	6.77	5.26	116.49	8.41	123.28	12.86	9.59	5.19
Total events	741163	575588	12749157	919952	13491459	1407592	1049894	568277
Average latency (ms)	0.40	0.52	0.02	0.33	0.02	0.21	0.29	0.53
95th percentile latency (ms)	2.30	3.25	0.06	1.58	0.05	1.32	1.79	2.71
Max latency (ms)	22.65	32.30	35.49	13.60	77.53	9.03	9.47	17.39
Total test time (s)	300.0475	300.1147	300.0020	300.0226	300.0012	300.0416	300.0159	300.1381

Percentage Differences Compared to VM on Proxmox (ZFS)

Metric	VM on Proxmox (LVM)	VM on FreeBSD (ZFS, NVMe)	VM on FreeBSD (ZFS, Virtio)	VM on FreeBSD (zvol)	Host FreeBSD (ZFS)	Host Proxmox (ZFS)	Host Proxmox (ext4)
File creation speed (MiB/s)	+13.18%	+259.77%	+242.99%	+227.02%	+298.62%	+137.52%	+55.25%
Reads per second	-22.34%	+1619.98%	+24.13%	+1720.45%	+89.92%	+41.67%	-23.34%
Writes per second	-22.35%	+1620.26%	+24.12%	+1720.42%	+89.91%	+41.66%	-23.34%
fsyncs per second	-22.36%	+1620.02%	+24.13%	+1720.18%	+89.90%	+41.66%	-23.35%
Read throughput (MiB/s)	-22.34%	+1619.88%	+24.11%	+1719.98%	+89.86%	+41.63%	-23.33%
Write throughput (MiB/s)	-22.30%	+1620.68%	+24.22%	+1720.97%	+89.96%	+41.65%	-23.33%
Total events	-22.34%	+1620.16%	+24.12%	+1720.31%	+89.92%	+41.65%	-23.31%
Average latency (ms)	+30.00%	-95.00%	-17.50%	-95.00%	-47.50%	-27.50%	+32.50%
95th percentile latency (ms)	+41.30%	-97.39%	-31.30%	-97.83%	-42.61%	-22.17%	+17.83%
Max latency (ms)	+42.60%	+56.69%	-39.96%	+242.30%	-60.13%	-58.19%	-23.22%
Total test time (s)	+0.02%	-0.02%	-0.01%	-0.02%	-0.02%	-0.01%	+0.01%

Percentage Differences Compared to VM on Proxmox (LVM) as this is the standard Proxmox setup

Metric	VM on Proxmox (ZFS)	VM on FreeBSD (ZFS, NVMe)	VM on FreeBSD (ZFS, Virtio)	VM on FreeBSD (zvol)	Host FreeBSD (ZFS)	Host Proxmox (ZFS)	Host Proxmox (ext4)
File creation speed (MiB/s)	-11.64%	+218.04%	+203.09%	+188.97%	+252.24%	+109.88%	+37.18%
Reads per second	+28.78%	+2115.42%	+59.85%	+2244.40%	+144.58%	+82.44%	-1.27%
Writes per second	+28.78%	+2115.37%	+59.85%	+2244.35%	+144.57%	+82.43%	-1.27%
fsyncs per second	+28.79%	+2115.30%	+59.87%	+2244.30%	+144.58%	+82.45%	-1.28%
Read throughput (MiB/s)	+28.77%	+2114.70%	+59.82%	+2243.60%	+144.49%	+82.38%	-1.27%
Write throughput (MiB/s)	+28.71%	+2114.64%	+59.89%	+2243.73%	+144.49%	+82.32%	-1.33%
Total events	+28.77%	+2114.98%	+59.83%	+2243.94%	+144.55%	+82.40%	-1.27%
Average latency (ms)	-23.08%	-96.15%	-36.54%	-96.15%	-59.62%	-44.23%	+1.92%
95th percentile latency (ms)	-29.23%	-98.15%	-51.38%	-98.46%	-59.38%	-44.92%	-16.62%
Max latency (ms)	-29.88%	+9.88%	-57.89%	+140.03%	-72.04%	-70.68%	-46.16%
Total test time (s)	-0.02%	-0.04%	-0.03%	-0.04%	-0.02%	-0.03%	+0.01%

Analysis of Performance Data

The performance data collected from various configurations of Proxmox and FreeBSD provides a comprehensive view of the I/O capabilities and highlights some significant differences. Here is an analysis of the key findings:

Comparative Analysis

Hypothesis on NVMe Performance and fsync

An important observation from my tests is that VMs with the bhyve NVMe driver show significantly higher performance compared to the same VMs with the virtio driver or compared to the physical host system. This difference initially led me to hypothesize that the bhyve NVMe driver might not correctly respect fsync operations, returning a positive result before the underlying file system has confirmed the final write. However, this was just a theory based on benchmark results and is not supported by concrete data. Furthermore, some developers have reviewed the code and found no evidence to suggest this behavior, and I personally have never encountered any potential issues that would indicate such a problem.

Specifically, I observed that:

The VM with the virtio driver has performance comparable to Proxmox.
The VM with the NVMe driver, whether on a ZFS dataset or zvol, shows performance superior to the physical FreeBSD host.

Host Physical Systems and Filesystems

File Creation Speed:
- Host FreeBSD (ZFS) shows the highest file creation speed at 1625.67 MiB/s, which is +68.03% compared to Host Proxmox (ZFS) and +156.72% compared to Host Proxmox (ext4).
- Host Proxmox (ext4) has a file creation speed of 633.13 MiB/s, which is -34.62% compared to Host Proxmox (ZFS).
- Read and Write Operations per Second:
- Host FreeBSD (ZFS) demonstrates the highest read and write operations per second with 1234.62 reads/s and 823.08 writes/s.
  - Reads per second: +34.06% compared to Host Proxmox (ZFS) and +147.80% compared to Host Proxmox (ext4).
  - Writes per second: +34.04% compared to Host Proxmox (ZFS) and +147.61% compared to Host Proxmox (ext4).
- Host Proxmox (ext4) shows a lower performance with 498.37 reads/s and 332.25 writes/s.
- Host Proxmox (ZFS) has 920.95 reads/s and 613.96 writes/s.
- fsync Operations per Second:
- Host FreeBSD (ZFS) achieves the highest fsync operations per second at 2634.01 fsyncs/s, which is +34.02% compared to Host Proxmox (ZFS) and +147.73% compared to Host Proxmox (ext4).
- Host Proxmox (ext4) has a lower performance with 1063.19 fsyncs/s.
- Host Proxmox (ZFS) achieves 1964.96 fsyncs/s.
- Throughput:
- Host FreeBSD (ZFS) again leads in throughput with 19.29 MiB/s read and 12.86 MiB/s write.
  - Read throughput: +34.03% compared to Host Proxmox (ZFS) and +147.53% compared to Host Proxmox (ext4).
  - Write throughput: +34.08% compared to Host Proxmox (ZFS) and +147.79% compared to Host Proxmox (ext4).
- Host Proxmox (ext4) has the lowest throughput with 7.79 MiB/s read and 5.19 MiB/s write.
- Host Proxmox (ZFS) has 14.39 MiB/s read and 9.59 MiB/s write.
- Latency:
- Host FreeBSD (ZFS) shows the lowest average latency at 0.21 ms and 95th percentile latency at 1.32 ms.
  - Average latency: -27.59% compared to Host Proxmox (ZFS) and -60.38% compared to Host Proxmox (ext4).
  - 95th percentile latency: -26.27% compared to Host Proxmox (ZFS) and -51.29% compared to Host Proxmox (ext4).
- Host Proxmox (ext4) has the highest average latency at 0.53 ms and 95th percentile latency at 2.71 ms.
- Host Proxmox (ZFS) has an average latency of 0.29 ms and 95th percentile latency of 1.79 ms.

VMs vs Physical Hosts

File Creation Speed:
- VM on FreeBSD (ZFS, NVMe) demonstrates an outstanding file creation speed at 1467.83 MiB/s (+218.04% compared to VM on Proxmox (LVM) and +259.77% compared to VM on Proxmox (ZFS)).
- VM on FreeBSD (zvol) achieves 1333.64 MiB/s, which is also significantly higher than VM on Proxmox (LVM) and VM on Proxmox (ZFS).
- Read and Write Operations per Second:
- VM on FreeBSD (ZFS, NVMe) shows exceptional performance with 11183.44 reads/s and 7455.62 writes/s.
- VM on FreeBSD (zvol) also performs excellently with 11834.53 reads/s and 7889.69 writes/s.
- fsync Operations per Second:
- VM on FreeBSD (ZFS, NVMe) achieves 23858.08 fsyncs/s, and VM on FreeBSD (zvol) achieves 25247.36 fsyncs/s, both significantly higher than any other configuration.
- Throughput:
- VM on FreeBSD (ZFS, NVMe) achieves the highest throughput with 174.74 MiB/s read and 116.49 MiB/s write.
- VM on FreeBSD (zvol) also has high throughput at 184.91 MiB/s read and 123.28 MiB/s write.
- Latency:
- VM on FreeBSD (ZFS, NVMe) shows very low average latency at 0.02 ms and 95th percentile latency at 0.06 ms.
- VM on FreeBSD (zvol) has similarly low latencies, indicating fast response times for I/O operations.

VM Configurations Comparison

File Creation Speed:
- Among VMs, VM on FreeBSD (ZFS, NVMe) leads, followed by VM on FreeBSD (zvol), and then VM on FreeBSD (ZFS, Virtio).
- Read and Write Operations per Second:
- VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) both outperform VM on Proxmox (ZFS) and VM on Proxmox (LVM) configurations significantly.
- VM on Proxmox (ZFS) outperforms VM on Proxmox (LVM) in read and write operations.
- fsync Operations per Second:
- VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) have significantly higher fsync operations compared to VM on Proxmox (ZFS) and VM on Proxmox (LVM).
- Throughput:
- VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) have the highest throughput, followed by VM on Proxmox (ZFS) and then VM on Proxmox (LVM).
- Latency:
- VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) show the lowest latencies among the VMs, indicating faster response times.
- VM on Proxmox (ZFS) shows lower latencies compared to VM on Proxmox (LVM).

Cache Settings and Performance Influence

Cache settings can significantly influence the performance of virtualization systems. In my setup, I did not modify the cache settings for the NVMe and virtio drivers, keeping the default settings. It is possible that the observed performance differences are also due to how different operating systems manage the caches of NVMe devices. I encourage other system administrators to explore the cache settings of their systems to see if changes in this area can influence benchmark results.

Conclusion

Regarding RAM and CPU, the performance of the VMs is comparable. There are slight differences in favor of Proxmox for CPU and FreeBSD for RAM, but in my opinion, these differences are so negligible that they wouldn't sway the decision towards one solution or the other.

The I/O performance data clearly indicates that VM on FreeBSD with NVMe and ZFS outperforms all other configurations by a significant margin. This is evident in the file creation speed, read/write operations per second, fsync operations per second, throughput, and latency metrics.

When comparing physical hosts, Host FreeBSD (ZFS) demonstrates excellent performance, particularly in comparison to Host Proxmox (ZFS) and Host Proxmox (ext4).

When comparing VMs, VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) configurations stand out as the top performers.

The VM using virtio on FreeBSD also shows strong performance, albeit not as high as the NVMe configuration. It significantly outperforms Proxmox configurations in terms of file creation speed, read/write operations per second, and throughput, while maintaining competitive latencies.

The virtio driver provides a stable and reliable option, making it a suitable choice for environments where the NVMe driver cannot be used. This makes FreeBSD with virtio a balanced option for virtualization, offering both high performance and reliability.

By examining these performance metrics, users can make informed decisions about their virtualization and storage configurations to optimize their systems for specific workloads and performance requirements.

In light of these tests and experiments, I can therefore affirm that my sensations (and those of many users) of greater "snappiness" of the VMs on FreeBSD can be confirmed. Certainly, Proxmox is a stable solution, rich in features, battle-tested, and has many other valid points, but FreeBSD, especially with the nvme driver, demonstrates very high performance and a very low overhead in installation and operation.

I will continue to use both solutions with great satisfaction, but I will be even more encouraged to implement virtualization servers based on FreeBSD and bhyve.

Migrating from an Old Linux Server to a New FreeBSD Machine

Stefano Marinelli — Wed, 25 Oct 2023 16:39:37 +0000

Preamble: I believe it's time to bid farewell to this venerable Linux server.

TL;DR: The article chronicles the journey of transitioning from an outdated Linux server, running for 1690 days without updates, to a modern FreeBSD machine. This migration involved using tools like mfsBSD, BastilleBSD, Borg Backup, and bhyve. Despite initial hesitations due to the Linux server's impeccable performance, the transition was smooth, resulting in improved manageability and efficiency. The piece emphasizes the importance of regular system updates and anticipates revisiting the topic in the future with new uptime achievements and updates.

This server loyally served for years as a secondary backup server while also providing a few minor services to users. As it often happens, it remained in operation, neglected and without updates for years. Stable operating systems have the "flaw" of being forgotten, giving the false impression that they don't need maintenance or updates. This machine continued its service without oversight for years. When approached for a service request (not due to malfunctions), I advised the client to upgrade the whole system. A mere update would not suffice, so I suggested starting afresh on new hardware with FreeBSD as the primary OS.

The client was understandably hesitant given the uptime stats:

08:58:43 up 1690 days, 21:32, 4 users, load average: 9.57, 10.15, 8.76

Not a single error, not a single hiccup. From his perspective, a similar setup to what was installed many years ago and still working flawlessly was preferred. Nevertheless, he trusted my expertise and let me proceed.

This server had a plethora of duties, among which:

One of the pivotal tasks was running Proxmox Backup Server via Docker. Proxmox Backup Server requires Debian, but this server was running on Ubuntu 16.04 (previously upgraded from Ubuntu 14.04 – yes, ancient!). Hence, Proxmox Backup Server was still at version 1.x.
Another critical function was storing backups made through BorgBackup on its file system. The /home directory used a mirrored btrfs file system, and each backed-up server had its user on this system. Clients could backup (using a push method) only via VPN and only during specific windows when the server permitted (by adding specific firewall rules via Jenkins. Jenkins also managed connection protocols, snapshots, backups, etc.).
Among the lesser tasks, the server ran a few Docker containers with HandBrake on various presets. The client processed video conversions by uploading the original files via sftp, and after some hours, fetched the converted files from the destination directory. This will not be replicated on the new FreeBSD server since they now handle this operation locally on their high-performance MacBook Pro with Apple Silicon. However, a future restoration isn't off the table.

The first thing I did was install FreeBSD on the new hardware. Given that it's a physical server on Hetzner (an auction pick due to disk space needs over power), and FreeBSD wasn't an option, I used mfsBSD.

After booting the physical server in Linux rescue, I copied mfsBSD onto the disks using dd and restarted. On boot, I SSHed into mfsBSD and executed the installation using bsdinstall—a robust and efficient method.

I set up a raidz1 with all four 6 TB disks, resulting in a final storage space of 21.8T, ample for now without the video files.

To ensure continuity, I kept a setup similar to the old one. The clients would essentially continue with their usual backup procedure without necessitating drastic changes to backup scripts. To avoid storing these backups directly in the physical machine's /home and to leave the door open for future services, I installed BastilleBSD and began setting up several jails. I replaced the old Linux machine's behavior with a VNET FreeBSD jail. In past scenarios, I've created Linux jails (thanks to BastilleBSD) and transferred the old server into the jail using rsync, making minor configuration tweaks. While this usually works, it doesn't address the underlying issue of an obsolete setup. Given the opportunity, I opted for a modern toolset.

Thus, I copied every home directory (along with their historic backups) in its entirety, installed BorgBackup, and re-established the VPN. With a VNET jail, I can craft networking devices and fine-tune configurations. After recreating user accounts, inputting the various SSH authorized_keys, and checking all clients, I set up a snapshot plan on the host. This ensures that if a client is compromised with the potential (however remote) for breach and backup deletion, a ZFS snapshot of the entire jail remains available.

As mentioned, one of the core tools on the old server was Proxmox Backup Server. It's not natively installable on FreeBSD, necessitating a VM. Enter the fantastic bhyve, supported by vm-bhyve.

However, one issue arose: backups would consume vast amounts of space, and I wanted to avoid housing an enormous disk image (or a zvol) with both VMs and backups. So, I opted for a slightly less performant but more flexible solution: installing Debian 12 and Proxmox Backup Server on the VM while placing backups on a separate ZFS dataset on the physical machine, exported via NFS and mounted on the VM.

Given that the physical server has an internal bridge "vm-public" with IP 192.168.124.1 and the VM is at 192.168.124.2, I just created a dataset named zroot/PBS and added the following line to /etc/exports:

/zroot/PBS -alldirs -maproot=root -network 192.168.124.2/32

To enable NFS, insert into /etc/rc.conf:

rpcbind_enable="YES" 
nfs_server_enable="YES" 
mountd_flags="-r" 
rpc_lockd_enable="YES"

Within the VM, create /PBS and include in /etc/fstab:

192.168.124.1:/zroot/PBS /PBS nfs rw,async,soft,intr,noexec 0 0

After Proxmox Backup Server's installation, simply set up the datastores in /PBS/, and they'll directly store on the physical machine's ZFS dataset.

For firewall configurations, I exposed port 8007, redirecting it towards the VM, and everything started functioning smoothly. I then set a Proxmox Backup Server replica from the old to the new server. After completion, I changed the Proxmox Backup Server IP on all Proxmox hosts to point to the new server. Smooth sailing.

The old Ubuntu server also managed other minor services, which have become obsolete and weren't replicated.

The transition was seamless, the client is pleased, and I'm content since each service is now neatly segregated into its jail or VM. The machine's load is minimal, which might pave the way for other tasks, via VPN. Everything now rests on ZFS, and the icing on the cake: I made the client promise not to reach another 1690 days of uptime but to timely update as required.

I'm not entirely convinced the promise will hold—meaning, in a few years, I might yet be discussing this "new" server, highlighting another impressive uptime and another upgrade journey.

Creating a Mikrotik CHR - RouterOS 7 - bhyve VM in FreeBSD

Stefano Marinelli — Tue, 21 Mar 2023 12:27:34 +0000

While I love FreeBSD-based router solutions (OPNsense, PFsense, etc.), I also appreciate and use MikroTik devices and software. They produce and sell reasonably-priced, efficient hardware, and have implemented some interesting proprietary solutions (EoIP, etc.). Additionally, they're a European company. From the smallest Wi-Fi router to the largest enterprise routing platform, they employ the same software with (almost) the same features.

That's why I'm also implementing virtualized MikroTik CHR solutions - they're lightweight, efficient, and can handle a significant amount of traffic with minimal resource overhead.

Our FreeBSD hypervisors run vm-bhyve as a lightweight, efficient, and intelligent management tool for VMs.

Even though MikroTik states that bhyve is not supported (as they believe it's merely a paravirtualization software), CHR based on RouterOS 6 works flawlessly, following the hints provided by the vm-bhyve documentation.

Unfortunately, things change when dealing with RouterOS 7. It doesn't boot at all, and if you upgrade from RoS 6 to RoS 7, it ceases to function entirely.

It seems that the MikroTik CHR image based on RouterOS 7 has an unusual partition table, somewhere between MBR and GPT. For this reason, Proxmox (KVM) doesn't experience any issues, while bhyve seems unable to properly boot the VM. I conducted several tests, but the best solution was suggested by user kriszos on the MikroTik forum (even if kriszos was dealing with Hyper-V).

The script should be used on a machine with the following tools installed: gdisk, wget, unzip, qemu-img, qemu-nbd, and rsync:

#!/bin/bash
wget --no-check-certificate https://download.mikrotik.com/routeros/7.8/chr-7.8.img.zip -O /tmp/chr.img.zip
unzip -p /tmp/chr.img.zip > /tmp/chr.img
rm -rf  chr.qcow2
qemu-img convert -f raw -O qcow2 /tmp/chr.img chr.qcow2
rm -rf /tmp/chr.im*
modprobe nbd
qemu-nbd -c /dev/nbd0 chr.qcow2
rm -rf /tmp/tmp*
mkdir /tmp/tmpmount/
mkdir /tmp/tmpefipart/
mount /dev/nbd0p1 /tmp/tmpmount/
rsync -a /tmp/tmpmount/ /tmp/tmpefipart/
umount /dev/nbd0p1
mkfs -t fat /dev/nbd0p1
mount /dev/nbd0p1 /tmp/tmpmount/
rsync -a /tmp/tmpefipart/ /tmp/tmpmount/
umount /dev/nbd0p1
rm -rf /tmp/tmp*
(
echo 2 # use GPT
echo t # change partition code
echo 1 # select first partition
echo 8300 # change code to Linux filesystem 8300
echo r # Recovery/transformation
echo h # Hybrid MBR
echo 1 2 # partitions added to the hybrid MBR
echo n # Place EFI GPT (0xEE) partition first in MBR (good for GRUB)? (Y/N)
echo   # Enter an MBR hex code (default 83)
echo y # Set the bootable flag? (Y/N)
echo   # Enter an MBR hex code (default 83)
echo n # Set the bootable flag? (Y/N)
echo n # Unused partition space(s) found. Use one to protect more partitions? (Y/N)
echo w # write changes to disk
echo y # confirm
) | gdisk /dev/nbd0
qemu-nbd -d /dev/nbd0
echo "script finished, created file chr.qcow2"

I used the script on a machine with Proxmox installed, and it worked correctly. Once the relevant image was obtained, it was converted to raw format to be fed to bhyve:

qemu-img convert -f qcow2 -O raw chr.qcow2 chr.img

Now that we have the correct image available, here's an example of a VM configuration managed by vm-bhyve:

loader="uefi"
graphics="no"
cpu="2"
memory=“128M"
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name=“chr.img”
uuid=“cafecafe-cafe-cafe-cafe-cafecafecafe”
network0_mac=“ca:fe:ca:fe:ca:fe”

How we are migrating (many of) our servers from Linux to FreeBSD - Part 3 - Proxmox to FreeBSD

Stefano Marinelli — Tue, 14 Mar 2023 13:00:00 +0000

In recent years, we've been migrating many of our servers from Linux to FreeBSD as part of our consolidation and optimization efforts. Specifically, we've been moving services that were previously deployed using Docker onto FreeBSD, and it has proven to be a great choice for handling workloads efficiently.

To this end, we've also been migrating many of our virtual machines (VMs) to FreeBSD, deploying services within FreeBSD jails. In some cases, these jails have even replaced entire VMs and run bare metal. Although we prefer to move to native FreeBSD whenever possible, sometimes it's not the best option for all the services we offer. As a result, one of our most critical physical servers has been left behind for years.

This server was a Proxmox server that we installed many years ago and updated to version 6.4. It hosted some critical services, but upgrading to Proxmox 7.x posed some challenges. In particular, some of the LXC containers required tweaks.

Unfortunately, this server was quite old, with only four physical disks and 64 GB of RAM. It was located in an OVH data center and had been running well until one of the disks started to malfunction once a week, on Sundays. This would trigger a RAID reconstruction that kept the system busy for about two days.

Despite my preference for simple setups, this server had been deployed gradually over many years, and everything was tied together. As a result, unraveling the system to resolve the issues was not a simple task. Sometimes the combination of simple things can make everything complex.

The Proxmox Server

The Proxmox server was configured as the central hub for various services, including primary DNS, web hosting, VOIP, and more. It featured several bridges, each with its own specific purpose, and was connected to a virtual machine running MikroTik CHR. This machine was responsible for consolidating all incoming VPNs from the MikroTik devices we managed, both ours and those belonging to our clients. Additionally, it provided a series of bridges to manage these devices and all server management VPNs and other services. The Proxmox server also housed several virtual machines running Linux, FreeBSD, OpenBSD, and NetBSD, as well as LXC containers.

Over the last two years, we've been migrating most of these virtual machines and containers to FreeBSD-based VMs, which feature their own specific jails. Consequently, most of the VMs we've had to move were BSD-based, while only five Linux VMs remained. The LXC containers hosted a range of services, including servers managed by Virtualmin, a large installation of Zimbra (which was hosted within an LXC container running CentOS 7), as well as some minor Alpine Linux-based machines. We located all these virtual machines and containers in a LAN created and managed by CHR. All public IPs were managed by CHR, which relied on NAT mappings to establish communication between them. CHR had thus become the heart of our system, and if it experienced any issues, it could potentially take down the entire system. Fortunately, it remained stable for years.

Migration - first steps

The first step I took was to install FreeBSD on the new server. Easy peasy. The next step was to find a way for the CHR to migrate to the new server (under bhyve) and continue to manage all the public IPs of the original server. The problem is that OVH, with its failover IPs, ties a specific MAC address to each individual IP address. Therefore, the only way was to create a bridge on the FreeBSD server (on the Proxmox server, I already had the bridge on the physical network card) and create an L2 tunnel between the two servers - I used OpenVPN with tap interfaces, specifically inserted into the bridges. I could have used other methods and techniques, but I wanted to experiment with a setup that could allow, if necessary, to bridge a larger number of physical and virtual servers even if the IPs are all mapped to a single server. OVH does not allow, in fact, the splitting of classes, so a move must be made for the entire class, not for a single IP address.

Initially, MikroTik CHR 7 did not boot on bhyve. In the end, I managed to make it work, but I had other problems, probably related to the MTU of the interfaces. So I thought about taking the opportunity to unbind the LXC containers and VMs from CHR and remove MikroTik from the setup. With RouterOS version 7, in fact, Wireguard-based VPNs are also supported, so within a few days, it was possible to update the few routers still on 6.x and recreate some VPNs using Wireguard. I mapped both the VMs and LXC containers directly to their respective public IPs, greatly simplifying the steps. Everything worked perfectly.

The next step was to test the first migrations, starting from the VMs already on FreeBSD. For simplicity, I created a new FreeBSD VM in bhyve and copied (via zfs-send and zfs-receive) the datasets related to BastilleBSD. All services are installed in jails managed by Bastille, so this was enough to have, in a short time, a new operating server equivalent to the previous one. At that point, I shut down the original server, connected the VM to the bridge linked to the tunnel (after modifying its MAC address), turned on the new FreeBSD VM (on bhyve), and everything started to work correctly - but from the new physical server.

One by one, I moved all the FreeBSD VMs. For Linux, NetBSD, and OpenBSD, I simply copied the images and pointed bhyve to them. Some small specific configuration on vm-bhyve and everything started to work correctly. Where possibile, I replaced the “virtio” with “nvme” as it performs much better on bhyve.

Migration - LXC containers to Virtual Machines

For LXC containers, I initially thought of creating an Alpine Linux virtual machine, installing LXD, and copying each individual container. It worked for some of them, but for others, I started to encounter strange issues, similar to those that would have required manual intervention to upgrade from Proxmox 6.x to 7.x. As is often the case with Linux-based solutions, compatibility is not always preserved between updates, so I would have had to fine-tune all the containers, which I didn't feel like doing. The containers had been created (at the time) to optimize RAM usage on the Proxmox machine, but to date, they have caused more problems than benefits. In some cases, certain processes got "stuck," making it impossible to "reboot" the LXC container, requiring the entire physical node to be rebooted. If they had been virtual machines, I could have given a "kill" command from the virtualizer (to the respective KVM process, in that case) and restarted it.

For greater compatibility and ease of future management, I decided to convert the LXC containers into actual VMs on bhyve. The process was simple:

Creating an empty VM with vm-bhyve and booting the VM with SystemRescueCD.
Creating destination partitions and file systems in the VM, then doing a complete rsync of the original LXC container.
Adjusting the fstab file, installing the kernel on the destination VM, and creating the initrd (some containers were already copies of VMs, so the kernel remained installed and updated, even though it wasn't being used. The initrd, on the other hand, did not include the nvme or virtio drivers, so I had to regenerate it anyway.)
Adjusting the bhyve vm configuration file, doing one last rsync after shutting down the services, shutting down the original LXC container, and starting the bhyve VM.

Everything worked correctly, so one by one, I moved all the containers. The largest one ended up on another physical node (also FreeBSD with bhyve) temporarily because the space on the new server was not sufficient to contain it. It didn't need to be on this server, so no problem.

One by one, the LXC containers started on the new server. Apart from some minor adjustments to the destination VMs (different network interface names, etc.), I didn't encounter any particular problems even after several days. Everything works perfectly.

At the very end, I re-created the MikroTik CHR VM. I’ll keep this setup separate for now, as strictly tied to eoip interfaces. This was the main reason why I haven’t performed the migration before. Things were too tied together and I had to untie everything, step by step.

…and then one of the Linux VMs started to freeze

Several Linux VMs are just the basis on which Docker runs. One of them (not even among the busiest) started, every 12/15 hours, to completely freeze. It stopped responding to ping, and it was impossible to give any type of command from the console. In a word: stuck.

Searching the web, I found some references to this problem and, observing the errors of an ssh session that was left connected (stuck, but still showing the last error), I found it to be a problem similar to the one described in this post, namely:

"watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [khugepaged:67]"

I tried various solutions such as changing the storage driver, the number of cores, the distribution (from Alpine to Debian), etc., but none of these operations solved the issue. I also noticed that the problem occurs with all Linux VMs, but only those with a recent kernel (> 5.10.x) freeze, while the others continue to work. The problem does not occur, however, with the *BSDs.

In the end, I:

Reduced the number of cores to 1 for the VMs that did not have a high load (some remained with multiple cores), hypothesising a problem with allocating cores that were too busy
Gave the command: "/usr/bin/echo 60 > /proc/sys/kernel/watchdog_thresh" to the VM.

The VM became stable, and I have not seen that error/warning on any other machine since. I will investigate further, but I believe it is a problem related to the Linux kernel, which, for some reason, generates a kernel panic if particular situations of CPU concurrency are generated.

The End…and a nice OOM!

After moving everything, I was finally able to migrate the entire class of OVH IPs from one physical server to another. The operation was quite quick, but in order to avoid problems, I notified all users and performed the operation on a Sunday and during off-peak hours. The whole process took about 10 minutes and there were no hitches of any kind.

For safety reasons, I kept the Proxmox machine active for a few more days, but there was no need to use it. However, after a couple of days, I encountered a problem: the largest VM, in some cases, was being "killed" because FreeBSD generated an OOM. I had never seen, from FreeBSD 13.0 onwards, any OOM related to "abuse" of RAM usage by ZFS, but in this case, it actually happened.

In the end, I understood that ZFS, on FreeBSD, is able to release memory, but not quickly enough to manage any "spikes" in individual VMs. In fact, the VMs do not know the situation of the physical host's RAM, so they will tend to occupy all the space allotted to them (even if only for caching). A sudden spike (i.e. if you create and launch a new VM) could cause a sudden increase in RAM usage by the bhyve process, and FreeBSD could be forced to kill it, even if part of the RAM is only ARC cache. While Proxmox supports HA (i.e., control over whether the VM is running), vm-bhyve only launches the VM (bhyve process). I should manage it with tools like monit, but for now, I preferred to simply set limits on ZFS RAM usage using "vfs.zfs.arc_max", and there have been no more problems.

Final considerations

The operation was long but linear. The most complex part was unraveling all the configurations related to MikroTik CHR and the VPNs linked to each individual LXC machine/container. Once everything was implemented on a dedicated VM, the operation was fairly straightforward.

The hardware specifications of the destination physical server are slightly better than the starting one, but the final performance of the setup has greatly improved. The VMs are very responsive (even those that were previously LXC containers running directly on bare metal) and, thanks to ZFS, I can make local snapshots every 5 minutes. In addition, every 10 minutes, I can copy (using the excellent zfs-autobackup) all the VMs and jails to other nodes both as a backup and as an immediate restart in case of disaster. I just need to map the IPs, and everything will start working very quickly. Proxmox also allows you to perform this type of operation with ZFS, but you still need to have Proxmox (in a compatible version) on the target machine. With the current setup, I only need any FreeBSD node that supports bhyve.

Proxmox is an excellent tool, well-developed, open-source, efficient, and stable. We manage many installations, including complex ones (ceph clusters, etc.), and it has never let us down. However, not all tools are ideal for all situations, and for setups like the one described, the new configuration based on FreeBSD has shown significantly interesting performance and greater management and maintenance granularity.

Virtualizing on vm-bhyve is not complex, but it is certainly not comparable, at the current state, to the simplicity of using a clean and complete interface like Proxmox's. A complete HA system is still missing (sure, it's achievable manually, but...), as well as complete management web interface. However, for knowledgeable users, it is undoubtedly a powerful tool that allows you to have excellent FreeBSD as a base. I'm totally satisfied with my migration and the result is far better than I expected.

Creating an Alpine Linux VM on bhyve - with root on ZFS (optionally encrypted)

Stefano Marinelli — Tue, 01 Nov 2022 15:47:35 +0000

Bhyve is great - and we’re using it with a lot of guest operating systems.

One of my favourite Linux distributions is Alpine Linux - it’s great as a docker or lxc/lxd host, is light, stable and easily manageable.

On FreeBSD, vm-bhyve already provides a good template for Alpine Linux, but it’s based on the plain standard image with boot on ext4.

So we need the alpine-extended iso - with zfs module.

Let’s create a new Alpine Linux bhyve VM:

vm create -t alpine -s 50G -m 4G -c 2 alpinevm

Now let’s configure it:

vm configure alpinevm

Let’s change some options: vmlinuz-vanilla and initramfs-vanilla should be changed to vmlinuz-lts and initramfs-lts.

More, we should instruct grub to boot from ZFS. The configuration should be similar to this:

[…]
grub_install0="linux /boot/vmlinuz-lts initrd=/boot/initramfs-lts alpine_dev=cdrom:iso9660 modules=loop,squashfs,sd-mod,usb-storage,sr-mod"
grub_install1="initrd /boot/initramfs-lts"
grub_run0="linux /boot/vmlinuz-lts root=rpool/ROOT/alpine rootfstype=zfs modules=ext4,zfs"
grub_run1="initrd /boot/initramfs-lts"
[…]

It’s now time to start with the installation:

vm install alpinevm alpine-extended.iso

Now, follow this Alpine Linux Wiki guide. Remember that we’re dealing with “vda” devices, not "sda", so change them accordingly. If you don’t want to have an encrypted rootfs dataset, just avoid the encryption line in the zpool create command.

At the end of the installation procedure, reboot and enjoy your new Alpine Linux bhyve VM.

IT Notes - bhyve

FreeBSD vs. SmartOS: Who's Faster for Jails, Zones, and bhyve VMs?

The Lineup: What I Put on the Test Bench

The Benchmark: My sysbench Commands

First Look: CPU and Memory on the Intel N150

Valid CPU Performance Comparison (Single Thread, Intel N150)

Overall Memory Performance Comparison (Intel Processor N150)

Going deeper: Testing on an Intel i7-7500U

CPU Performance Comparison (Intel i7-7500U)

Memory Performance Comparison (Intel i7-7500U)

Breaking Down the Results: What I Learned From Both Tests

Final Thoughts: The Verdict for My Client's New Server

Introducing the illumos Cafe: Another Cozy Corner for OS Diversity

Introducing the illumos Cafe: Another Cozy Corner for OS Diversity

From the BSD Cafe to illumos Cafe

Why Another Cafe?

Why illumos?

The illumos Cafe Project

Community and Philosophy

Technical Setup

Available Services

Registrations and Logo

Looking Ahead

Make Your Own Backup System – Part 2: Forging the FreeBSD Backup Stronghold

Base System Setup

Backup Strategies

Backups using zfs-autobackup

Backups for other servers using BorgBackup

Proxmox Backup Server in a Dedicated VM

Creating a Minio Jail as a Data Repository for PBS

Taking Local Snapshots of Backups

Conclusion

Migrating Windows VMs from Proxmox BIOS/KVM to FreeBSD UEFI/bhyve

Setting Up the Network

Installing and Configuring vm-bhyve

Creating the VM Template

Modifying the VM Configuration

Copying the Virtual Disk

Converting BIOS to UEFI

Running the Windows Repair Tool

Booting the VM

Final Considerations

From Proxmox to FreeBSD - Story of a Migration

Acquiring New Hardware

Configuring the Firewall

Installing Necessary Tools

Setting Up the Reverse Proxy Jail

Migrating the VMs

Migrating Rocky Linux 8.10 VMs

Migrating Rocky Linux 9.4 VMs

Migrating the OPNsense VM

Configuring Automatic VM Startup

Setting Up Backups

Conclusion

From Cloud Chaos to FreeBSD Efficiency

Introduction

The Solution

Implementation

Real-world Testing

Lessons Learned

Conclusion

Proxmox vs FreeBSD: Which Virtualization Host Performs Better?

Preamble

Hardware and Test Conditions

CPU and RAM Tests on VMs

Comparative Results

CPU Test

CPU Percentage Analysis

RAM Test

RAM Percentage Analysis

CPU and RAM Comparative Results Table

Interpretation of CPU and RAM Results

I/O Performance Tests

I/O Comparative Performance Data with Percentage Differences

Percentage Differences Compared to VM on Proxmox (ZFS)

Percentage Differences Compared to VM on Proxmox (LVM) as this is the standard Proxmox setup

Analysis of Performance Data

Comparative Analysis

Hypothesis on NVMe Performance and fsync

Host Physical Systems and Filesystems

The Benchmark: My `sysbench` Commands