Proxmox vs FreeBSD: Which Virtualization Host Performs Better?

TL;DR: Skip to the Conclusion for a summary.

Preamble

I have always been passionate about virtualization and have consistently used it.

The first solution I installed on my infrastructures (and those of clients) was Xen on NetBSD, with great success. I then used Xen on Linux and, since 2012, OpenNebula, followed by Proxmox in 2013. Proxmox has always given me great satisfaction, and even today I consider it a valuable platform that I install gladly. I have also used other hypervisors like XCP-ng but less frequently, and in recent years, I have started to make extensive use of bhyve.

About two and a half years ago, we began a progressive process of migrating our servers (and those of our clients) from Linux to FreeBSD, using jails (when possible) or VMs on bhyve. In some cases, migrating setups from Proxmox to FreeBSD resulted in performance improvements, even with the same hardware. In some instances, I migrated VMs without notifying clients, and they contacted me a few days later to inquire if we had new hardware because they noticed better performance.

After years, I decided to conduct a test to determine if this was just a perception or if there was a technical basis behind it. Of course, this test has no scientific validity, and the results were obtained on specific hardware and at a specific time, so on different hardware, workload, and situations, the results could be entirely opposite. However, I tried to have as scientific and objective an approach as possible since I am comparing two solutions that I care about and use daily.

Hardware and Test Conditions

I often see comparative tests done on VMs from various providers. In my opinion, this comparison makes no sense because a VM from any provider shares its hardware with many other VMs, so the results will vary depending on the load of the “neighbors” and will never be reliable.

For this test, I decided to take a physical server with the following characteristics:

• Intel Core i7-6700
• 2x SSD M.2 NVMe 512 GB
• 4x RAM 16384 MB DDR4
• NIC 1 Gbit Intel I219-LM

The hardware is not recent, but still very widespread. On more recent hardware, the results might differ, but the test will be based on this configuration.

I installed Proxmox 8.2.2 starting from the Debian template of the provider and manually installed it following the instructions. I created a partition for Proxmox and left one partition free on each of the two NVME drives to create (at different times) the ZFS pool (in mirror) and the LVM on top of the Linux software raid.

After all the tests, I installed FreeBSD 14.1-RELEASE on ZFS on the same host, using bsdinstall from an mfsbsd image since the provider does not directly support installing FreeBSD from its panel or rescue mode.

In both installations, I always trimmed the NVME drives before starting the tests, and in the case of ZFS, I set (both on Proxmox and FreeBSD) compression to zstd and atime to off. No other changes were made compared to the standard installation.

On FreeBSD, the VM was created and managed with vm-bhyve (devel).

On Proxmox, I tested the physical host on ZFS and ext4 and the VM on ZFS and LVM as LVM is the standard and most common setup in Proxmox.

On FreeBSD, I tested the host on ZFS and the VM with both virtio and nvme drivers, on zvol, and as an image file within a ZFS dataset.

I used sysbench installed from the official Debian repository (on Proxmox and VM) and from the FreeBSD package on the respective host.

The VMs, both on Proxmox and FreeBSD, have nearly identical characteristics and default configuration (apart from the nvme drivers set on bhyve, and for that reason, I also tested virtio).

For those who want to reproduce my tests, here are the detailed configurations of the VMs used in bhyve:

FreeBSD bhyve VM Configuration with NVMe Driver:

1
2
3
4
5
6
7

loader="uefi"
cpu=4
memory=4096M
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"

FreeBSD bhyve VM Configuration with virtio Driver:

1
2
3
4
5
6
7

loader="uefi"
cpu=4
memory=4096M
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0.img"

Proxmox VM Configuration:

In all the configurations, I used Debian 12 as the VM operating system, with the file system on ext4.

I chose Debian 12 as it is a stable, widespread, and modern Linux distribution. I did not test a FreeBSD VM because, in my setups, I tend not to virtualize FreeBSD on FreeBSD but to use nested jails.

All tests were performed multiple times, and I took the median results. CPU and RAM were tested only on the first VM (on Proxmox (ZFS) and FreeBSD (ZFS and nvme)) as they are not dependent on the underlying storage. Storage performance, on the other hand, was tested on all configurations.

CPU and RAM Tests on VMs

On both VMs:

1
2

sysbench --test=cpu --cpu-max-prime=20000 run
sysbench --test=memory run

Comparative Results

CPU Test

CPU Percentage Analysis

• Difference in Events per Second: ((498.08 - 473.65) / 498.08 approx -4.91%)
• Difference in Total Time: ((10.0019 - 10.0010) / 10.0010 approx +0.009%)
• Difference in Latency (avg): ((2.11 - 2.01) / 2.01 approx +4.98%)

RAM Test

RAM Percentage Analysis

• Difference in Total Operations: ((68621063 - 64777227) / 64777227 approx +5.94%)
• Difference in Operations per Second: ((6861139.06 - 6476757.59) / 6476757.59 approx +5.94%)
• Difference in Total MiB Transferred: ((67012.76 - 63259.01) / 63259.01 approx +5.93%)
• Difference in MiB/sec: ((6700.33 - 6324.96) / 6324.96 approx +5.93%)

CPU and RAM Comparative Results Table

Interpretation of CPU and RAM Results

1. CPU Performance:

   • The VM on FreeBSD has slightly lower CPU performance compared to Proxmox (-4.91% in events per second).
   • The total execution time is nearly identical, with a negligible difference.
   • The average latency is slightly higher on FreeBSD (+4.98%).

2. RAM Performance:

   • The VM on FreeBSD has better RAM performance compared to Proxmox (+5.94% in operations and MiB/sec).
   • The average latency is identical in both configurations.

In summary, while Proxmox provides more consistent CPU performance, FreeBSD demonstrates superior memory performance. The choice between Proxmox and FreeBSD may depend on the specific workload requirements and the importance of consistent performance versus higher throughput.

I/O Performance Tests

The test has been conducted using sysbench, with this command line:

1
2

sysbench --test=fileio --file-total-size=30G prepare
sysbench --test=fileio --file-total-size=30G --file-test-mode=rndrw  --max-time=300 --max-requests=0 run

I/O Comparative Performance Data with Percentage Differences

Percentage Differences Compared to VM on Proxmox (ZFS)

Percentage Differences Compared to VM on Proxmox (LVM) as this is the standard Proxmox setup

Analysis of Performance Data

The performance data collected from various configurations of Proxmox and FreeBSD provides a comprehensive view of the I/O capabilities and highlights some significant differences. Here is an analysis of the key findings:

Comparative Analysis

Hypothesis on NVMe Performance and fsync

An important observation from my tests is that VMs with the bhyve NVMe driver show significantly higher performance compared to the same VMs with the virtio driver or compared to the physical host system. This difference led me to hypothesize that the bhyve NVMe driver might not correctly respect fsync operations, returning a positive result before the underlying file system has confirmed the final write. However, this is just a theory based on benchmark results and is not supported by concrete data.

Specifically, I observed that:

• The VM with the virtio driver has performance comparable to Proxmox.
• The VM with the NVMe driver, whether on a ZFS dataset or zvol, shows performance superior to the physical FreeBSD host.

These observations suggest that the bhyve NVMe driver might “cheat” by returning an ok for fsync operations before they are actually completed. However, further testing and analysis are needed to confirm or refute this hypothesis.

Host Physical Systems and Filesystems

1. File Creation Speed:

   • Host FreeBSD (ZFS) shows the highest file creation speed at 1625.67 MiB/s, which is +68.03% compared to Host Proxmox (ZFS) and +156.72% compared to Host Proxmox (ext4).
   • Host Proxmox (ext4) has a file creation speed of 633.13 MiB/s, which is -34.62% compared to Host Proxmox (ZFS).

2. Read and Write Operations per Second:

   • Host FreeBSD (ZFS) demonstrates the highest read and write operations per second with 1234.62 reads/s and 823.08 writes/s.
     • Reads per second: +34.06% compared to Host Proxmox (ZFS) and +147.80% compared to Host Proxmox (ext4).
     • Writes per second: +34.04% compared to Host Proxmox (ZFS) and +147.61% compared to Host Proxmox (ext4).
   • Host Proxmox (ext4) shows a lower performance with 498.37 reads/s and 332.25 writes/s.
   • Host Proxmox (ZFS) has 920.95 reads/s and 613.96 writes/s.

3. fsync Operations per Second:

   • Host FreeBSD (ZFS) achieves the highest fsync operations per second at 2634.01 fsyncs/s, which is +34.02% compared to Host Proxmox (ZFS) and +147.73% compared to Host Proxmox (ext4).
   • Host Proxmox (ext4) has a lower performance with 1063.19 fsyncs/s.
   • Host Proxmox (ZFS) achieves 1964.96 fsyncs/s.

4. Throughput:

   • Host FreeBSD (ZFS) again leads in throughput with 19.29 MiB/s read and 12.86 MiB/s write.
     • Read throughput: +34.03% compared to Host Proxmox (ZFS) and +147.53% compared to Host Proxmox (ext4).
     • Write throughput: +34.08% compared to Host Proxmox (ZFS) and +147.79% compared to Host Proxmox (ext4).
   • Host Proxmox (ext4) has the lowest throughput with 7.79 MiB/s read and 5.19 MiB/s write.
   • Host Proxmox (ZFS) has 14.39 MiB/s read and 9.59 MiB/s write.

5. Latency:

   • Host FreeBSD (ZFS) shows the lowest average latency at 0.21 ms and 95th percentile latency at 1.32 ms.
     • Average latency: -27.59% compared to Host Proxmox (ZFS) and -60.38% compared to Host Proxmox (ext4).
     • 95th percentile latency: -26.27% compared to Host Proxmox (ZFS) and -51.29% compared to Host Proxmox (ext4).
   • Host Proxmox (ext4) has the highest average latency at 0.53 ms and 95th percentile latency at 2.71 ms.
   • Host Proxmox (ZFS) has an average latency of 0.29 ms and 95th percentile latency of 1.79 ms.

VMs vs Physical Hosts

1. File Creation Speed:

   • VM on FreeBSD (ZFS, NVMe) demonstrates an outstanding file creation speed at 1467.83 MiB/s (+218.04% compared to VM on Proxmox (LVM) and +259.77% compared to VM on Proxmox (ZFS)).
   • VM on FreeBSD (zvol) achieves 1333.64 MiB/s, which is also significantly higher than VM on Proxmox (LVM) and VM on Proxmox (ZFS).

2. Read and Write Operations per Second:

   • VM on FreeBSD (ZFS, NVMe) shows exceptional performance with 11183.44 reads/s and 7455.62 writes/s.
   • VM on FreeBSD (zvol) also performs excellently with 11834.53 reads/s and 7889.69 writes/s.
   • These values suggest that the NVMe driver might not be honoring fsync properly, resulting in inflated performance metrics.

3. fsync Operations per Second:

   • VM on FreeBSD (ZFS, NVMe) achieves 23858.08 fsyncs/s, and VM on FreeBSD (zvol) achieves 25247.36 fsyncs/s, both significantly higher than any other configuration.

4. Throughput:

   • VM on FreeBSD (ZFS, NVMe) achieves the highest throughput with 174.74 MiB/s read and 116.49 MiB/s write.
   • VM on FreeBSD (zvol) also has high throughput at 184.91 MiB/s read and 123.28 MiB/s write.

5. Latency:

   • VM on FreeBSD (ZFS, NVMe) shows very low average latency at 0.02 ms and 95th percentile latency at 0.06 ms.
   • VM on FreeBSD (zvol) has similarly low latencies, indicating fast response times for I/O operations.

VM Configurations Comparison

1. File Creation Speed:

   • Among VMs, VM on FreeBSD (ZFS, NVMe) leads, followed by VM on FreeBSD (zvol), and then VM on FreeBSD (ZFS, Virtio).

2. Read and Write Operations per Second:

   • VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) both outperform VM on Proxmox (ZFS) and VM on Proxmox (LVM) configurations significantly.
   • VM on Proxmox (ZFS) outperforms VM on Proxmox (LVM) in read and write operations.

3. fsync Operations per Second:

   • VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) have significantly higher fsync operations compared to VM on Proxmox (ZFS) and VM on Proxmox (LVM).

4. Throughput:

   • VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) have the highest throughput, followed by VM on Proxmox (ZFS) and then VM on Proxmox (LVM).

5. Latency:

   • VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) show the lowest latencies among the VMs, indicating faster response times.
   • VM on Proxmox (ZFS) shows lower latencies compared to VM on Proxmox (LVM).

Cache Settings and Performance Influence

Cache settings can significantly influence the performance of virtualization systems. In my setup, I did not modify the cache settings for the NVMe and virtio drivers, keeping the default settings. It is possible that the observed performance differences are also due to how different operating systems manage the caches of NVMe devices. I encourage other system administrators to explore the cache settings of their systems to see if changes in this area can influence benchmark results.

Conclusion

Regarding RAM and CPU, the performance of the VMs is comparable. There are slight differences in favor of Proxmox for CPU and FreeBSD for RAM, but in my opinion, these differences are so negligible that they wouldn’t sway the decision towards one solution or the other.

The I/O performance data clearly indicates that VM on FreeBSD with NVMe and ZFS outperforms all other configurations by a significant margin. This is evident in the file creation speed, read/write operations per second, fsync operations per second, throughput, and latency metrics. However, the exceptionally high performance of VM on FreeBSD with NVMe and ZFS suggests that there might be an underlying issue, such as the NVMe driver not honoring fsync properly. This could lead to the VM believing that data has been written when it has not, resulting in artificially inflated performance results.

When comparing physical hosts, Host FreeBSD (ZFS) demonstrates excellent performance, particularly in comparison to Host Proxmox (ZFS) and Host Proxmox (ext4).

When comparing VMs, VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) configurations stand out as the top performers. However, it’s important to consider the potential fsync issue with NVMe storage. VM on Proxmox (ZFS) shows better performance than VM on Proxmox (LVM), but both are outperformed by the FreeBSD configurations.

The VM using virtio on FreeBSD also shows strong performance, albeit not as high as the NVMe configuration. It significantly outperforms Proxmox configurations in terms of file creation speed, read/write operations per second, and throughput, while maintaining competitive latencies.

The virtio driver provides a stable and reliable option, making it a suitable choice for environments where the NVMe driver’s potential fsync issue might be a concern. This makes FreeBSD with virtio a balanced option for virtualization, offering both high performance and reliability.

In conclusion, while the VM on FreeBSD with NVMe and ZFS shows the best performance, it is essential to investigate the potential issue with fsync operations.

By examining these performance metrics, users can make informed decisions about their virtualization and storage configurations to optimize their systems for specific workloads and performance requirements.

In light of these tests and experiments, I can therefore affirm that my sensations (and those of many users) of greater “snappiness” of the VMs on FreeBSD can be confirmed. Certainly, Proxmox is a stable solution, rich in features, battle-tested, and has many other valid points, but FreeBSD, especially with the nvme driver, demonstrates very high performance and a very low overhead in installation and operation.

I will continue to use both solutions with great satisfaction, but I will be even more encouraged to implement virtualization servers based on FreeBSD and bhyve.