Boosting Network Performance in FreeBSD's VNET Jails -

In the world of FreeBSD, jails are a renowned feature that allows for system-level virtualization. As I was setting up the jails for BSDCafe, I stumbled upon an interesting discovery: the network performance of VNET jails was noticeably lower compared to that of VPS or standard jails. Rather than diving into this immediately, I decided to take a mental note and proceed.

As I delved deeper with various tests, a pattern began to emerge. Anytime there was a NAT (Network Address Translation) acting between the internal bridge of the VNET jails - irrespective of whether it was local or bridged via a VPN - the outgoing performance took a nosedive.

From using tcpdump to carrying out MTU (Maximum Transmission Unit) tests, my endeavors seemed fruitless. However, a memory from the past struck me. I recalled setting up a FreeBSD VM on Proxmox (effectively pointing towards an issue with KVM) where I had to make specific tweaks.

To remedy the situation, I made the following modifications:

Added the following to /boot/loader.conf:

hw.vtnet.X.csum_disable=1
hw.vtnet.lro_disable=1

Integrated these lines into /etc/sysctl.conf:

net.link.bridge.pfil_member=0
net.link.bridge.pfil_bridge=0
net.link.bridge.pfil_onlyip=0

And appended to /etc/rc.local (which I already use for initialization):

ifconfig vtnet0 -rxcsum

The end result was exhilarating: not only did the VNET jails now perform at full bandwidth, but even those interconnected via VPN showcased commendable performance.

Interestingly, this seems to be linked to a long-standing bug from 2012, FreeBSD Bug 165059. This issue is even highlighted in the official PFSense documentation.

In the vast landscape of tech, sometimes revisiting the past provides solutions for the present. All’s well that ends well, and I’m pleased to share this resolution with my readers. For those dabbling in FreeBSD, I hope this piece offers some guidance in optimizing your VNET jail setups.

See also