IT Notes - zfs

Time Machine inside a FreeBSD jail

Stefano Marinelli — Wed, 28 Jan 2026 08:52:00 +0000

Many of my clients do not use Microsoft systems on their desktops; they use Linux-based systems or, in some cases, FreeBSD. Many use Apple systems - macOS - and are generally satisfied with them. While I wash my hands of it when it comes to Microsoft systems (telling them they have to manage their desktops autonomously), I am often able to lend a hand with macOS. And one of the main requests they make is to manage the backups of their individual workstations.

macOS, thanks to its Unix base, offers good native tools. Time Machine is transparent and effective, allowing a certain freedom of management. APFS, Apple's current file system, supports snapshots, so the backup will be effectively made on a snapshot. It also supports multiple receiving devices, so you can even have a certain redundancy of the backup itself.

Having many FreeBSD servers, I am often asked to use their resources and storage. To build, in practice, a Time Machine inside one of the servers. And it is a simple and practical operation, quick and "painless". There are many guides, including the excellent one by Benedict Reuschling from which I took inspiration for this one, and I will describe the steps I usually follow to set it all up in just a few minutes.

I usually use BastilleBSD to manage my jails, so the first step is to create a new jail dedicated to the purpose. Here you have to decide on the approach: I suggest using a VNET jail or an "inherit" jail - meaning one that attaches to the host's network stack. On one hand, the inherit approach is less secure but, as often happens, it depends on the complexity of the situation. If, for example, we are using a Raspberry PI dedicated to the purpose, there is no reason to complicate things with bridges, etc., but we can attach directly to the network card with a creation command like:

bastille create tmjail 15.0-RELEASE inherit igb0

Where igb0 is the network interface we want to attach to.

In case we want to attach to the interface but in the form of a bridge, we should use this syntax:

bastille create -V tmjail 15.0-RELEASE 192.168.0.42/24 igb0

Or, if our server already has a bridge (in this case it's bridge0, but yours might be named differently):

bastille create -B tmjail 15.0-RELEASE 192.168.0.42/24 bridge0

At this point, you can choose: do we want to keep the backups inside the jail or in a separate dataset - which can even be on another pool? In some cases, this can be extremely useful: often I have jails running on fast disks (SSD or NVMe) but abundant storage on slower devices. In this example, therefore, I will create an external dataset for the backups (directly from the host) and mount it in the jail. You could also delegate the entire management of the dataset to the jail, which is a different approach.

Let's create a space of 600 GB - already reserved - on the chosen pool. 600 GB is a small space, but it's ok for an example:

zfs create -o quota=600G -o reservation=600G bigpool/tmdata

We can also create separate datasets inside for each user and assign a specific space:

zfs create -o refquota=500g -o refreservation=500g bigpool/tmdata/stefano

We can enter the jail and install what we need, remembering also to create the "mountpoint" for the dataset we just created:

bastille console tmjail 

pkg install -y samba419
mkdir /tmdata

Exit the jail and instruct Bastille to mount the dataset inside the jail every time it is launched:

exit
bastille mount tmjail /bigpool/tmdata /tmdata nullfs rw 0 0

Let's go back into the jail and start with the actual configuration. First, for each Time Machine user, we will create a system user. In my example, I will create the user "stefano", giving him /var/empty as the home directory - this will give an error since we created a Bastille thin jail, but it's not a problem. It happens because in a thin jail some system paths are read-only or not manageable as they are on a full base system, but the user is only needed for ownership and Samba login.

root@tmjail:~ # adduser
Username: stefano
Full name: Stefano
Uid (Leave empty for default):
Login group [stefano]:
Login group is stefano. Invite stefano into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]: nologin
Home directory [/home/stefano]: /var/empty
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]: no
Lock out the account after creation? [no]:
Username    : stefano
Password    : <disabled>
Full Name   : Stefano
Uid         : 1001
Class       :
Groups      : stefano
Home        : /var/empty
Home Mode   :
Shell       : /usr/sbin/nologin
Locked      : no
OK? (yes/no) [yes]: yes
pw: chmod(var/empty): Operation not permitted
pw: chown(var/empty): Operation not permitted
adduser: INFO: Successfully added (stefano) to the user database.
Add another user? (yes/no) [no]: no
Goodbye!

Give the correct permissions to the user:

# If you've not created specific datasets for the users, you'd better create their home directories now
mkdir /tmdata/stefano
chown -R stefano /tmdata/stefano/

Now we configure Samba for Time Machine. The file to create/modify is /usr/local/etc/smb4.conf:

[global]
workgroup = WORKGROUP
security = user
passdb backend = tdbsam
fruit:aapl = yes
fruit:model = MacSamba
fruit:advertise_fullsync = true
fruit:metadata = stream
fruit:veto_appledouble = no
fruit:nfs_aces = no
fruit:wipe_intentionally_left_blank_rfork = yes
fruit:delete_empty_adfiles = yes

[TimeMachine]
path = /tmdata/%U
valid users = %U
browseable = yes
writeable = yes
vfs objects = catia fruit streams_xattr zfsacl
fruit:time machine = yes
create mask = 0600
directory mask = 0700

We have set up Time Machine to support all the necessary features of macOS and to show itself as "Time Machine". Having set path = /tmdata/%U, each user will only see their own path.

At this point, we create the Samba user (meaning the one we will have to type on macOS when we configure the Time Machine):

smbpasswd -a stefano

The Time Machine is seen by macOS because it announces itself via mDNS on the network. This type of service is performed by Avahi, which we are now going to configure. Although not strictly necessary (we can always find the Time Machine by connecting directly to its IP and macOS will remember everything), seeing it announced will help other non-expert users and ourselves when we have to configure another Mac in the future.

Recent Samba releases won't need any specific avahi configuration, so we can skip this step.

We are now ready to enable everything.

service dbus enable
service dbus start
service avahi-daemon enable
service avahi-daemon start
service samba_server enable
service samba_server start

Et voilà. If everything went according to plan, the Time Machine will announce itself on your network (if you have different networks, remember to configure the mDNS proxy on your router) and you will be able to log in (with the smb user you created) and start your first backup.

I suggest encrypting the backups for maximum security and observing, from time to time, your Mac as it silently makes its backups to your trusted FreeBSD server.

Installing Void Linux on ZFS with Hibernation Support

Stefano Marinelli — Mon, 22 Dec 2025 08:43:02 +0000

Introduction

FreeBSD continues to make strides in desktop support, but Linux still holds an advantage in hardware compatibility. After running openSUSE Tumbleweed on my mini PC for several months, I decided it was time to switch to a solution I could control more closely. Not because Tumbleweed doesn't work well - it works great! - but I prefer having direct control over what happens on my machine. And I want native ZFS, because I prefer it over btrfs and it allows me to manage snapshots, backups, and rollbacks just as I do on FreeBSD, using the same tools and procedures.

The choice of Void Linux comes from its BSD-like approach: modular and free of unnecessary complexity. This makes it an excellent solution for this type of setup.

ZFSBootMenu is an extremely powerful tool. It provides an experience similar to FreeBSD's boot loader and natively supports ZFS. I strongly recommend reading the documentation and exploring its features, as some of them - like the built-in SSH daemon - can be genuine lifesavers in recovery scenarios.

Prerequisites and Audience

This guide is not for absolute beginners. If you're new to Linux or Unix-like operating systems, you'd be better served by a ready-to-use distribution like openSUSE Leap (or Tumbleweed for a rolling distribution), Linux Mint, Debian, Ubuntu, or Manjaro. The purpose of this article is to demonstrate a stable, upgradeable, and reasonably secure base setup for users already comfortable with system administration. It uses the glibc variant of Void Linux. The musl version requires different commands, for example for locale generation.

Use at your own risk.

This guide synthesizes instructions from several sources:

Void Linux (UEFI) from ZFSBootMenu - which doesn't address swap. Using a zvol for swap (not the best solution) prevents hibernation and resume. Our approach uses a separate encrypted swap partition that enables proper resume.
Void Linux Full Disk Encryption - excellent for btrfs or ext4, but we want ZFS. We'll borrow the swap configuration approach from here.
Install Void Linux with a desktop environment + Flatpaks - for the desktop portion.

If your setup differs from what's described here (NVMe disk, UEFI boot, Secure Boot disabled), consult the linked guides for explanations and variations.

Installation Script (Optional)

If you want to reproduce this setup quickly, I maintain a script that automates the procedure described in this guide: disk partitioning, ZFS pool and dataset creation, encrypted swap for hibernation resume, dracut configuration, and ZFSBootMenu EFI setup. An optional KDE Plasma desktop installation is also supported.

The script is interactive and will ask for the required parameters (target disk, timezone and keymap, passphrases, desktop options). Requirements, usage instructions, and known limitations are documented in the repository README

That said, I still recommend going through the manual process at least once. Understanding each step is part of the value of this setup, especially when troubleshooting or adapting it to different hardware.

Boot Environment

Since ZFS isn't supported by the base Void Linux image, we'll use hrmpf, an excellent rescue system based on Void Linux that includes ZFS support out of the box.

After booting, you can either proceed directly or SSH into the machine to continue remotely. I generally prefer SSH since it makes copy-paste operations much easier - especially when dealing with UUIDs and long commands. To enable SSH access, set a root password and allow root login:

passwd

Edit /etc/ssh/sshd_config and enable:

PermitRootLogin yes

Restart the SSH daemon:

sv restart sshd

Find the machine's IP address:

ip addr

You can now connect via SSH from another device.

Initial Setup

Set up the environment variables and generate a host ID - we need it for ZFS:

source /etc/os-release
export ID

zgenhostid -f 0x00bab10c

Disk Configuration

Identify your target disk and set up the partition variables. This approach keeps everything consistent and reduces errors:

# Set the base disk - adjust this to match your system
export DISK="/dev/nvme0n1"

# For NVMe disks, partitions are named like nvme0n1p1, nvme0n1p2, etc.
# For SATA/SAS disks (sda, sdb), partitions are named sda1, sda2, etc.
# Set the partition separator accordingly:
export PART_SEP="p"  # Use "p" for NVMe, empty string "" for SATA/SAS

# Define partition numbers
export BOOT_PART="1"
export SWAP_PART="2"
export POOL_PART="3"

# Build full device paths
export BOOT_DEVICE="${DISK}${PART_SEP}${BOOT_PART}"
export SWAP_DEVICE="${DISK}${PART_SEP}${SWAP_PART}"
export POOL_DEVICE="${DISK}${PART_SEP}${POOL_PART}"

Verify your configuration before proceeding:

echo "Boot device: $BOOT_DEVICE"
echo "Swap device: $SWAP_DEVICE"
echo "Pool device: $POOL_DEVICE"

Wipe the Disk

Warning: This operation will irreversibly destroy all data on the selected disk. Double-check that you've selected the correct disk and be sure to have a complete backup of your system!

zpool labelclear -f "$DISK"

wipefs -a "$DISK"
sgdisk --zap-all "$DISK"

Create Partitions

EFI System Partition

If you're not using UEFI boot, adapt this procedure following the appropriate guide linked at the beginning of this post:

sgdisk -n "${BOOT_PART}:1m:+512m" -t "${BOOT_PART}:ef00" "$DISK"

Swap Partition

The swap partition should be slightly larger than your RAM to support hibernation. When you hibernate, the entire contents of RAM are written to swap, so you need enough space to hold it all plus some overhead. In this example, I have 16 GB of RAM, so I'm creating an 18 GB swap partition:

sgdisk -n "${SWAP_PART}:0:+18g" -t "${SWAP_PART}:8200" "$DISK"

ZFS Pool Partition

sgdisk -n "${POOL_PART}:0:-10m" -t "${POOL_PART}:bf00" "$DISK"

Set Up ZFS Encryption

Encrypting the disk is strongly recommended, especially for laptops. Replace SomeKeyphrase with a strong passphrase that's easy to type. Keep in mind that during early boot, the keyboard layout might default to US, so choose a passphrase that's easy to type on a US keyboard layout:

echo 'SomeKeyphrase' > /etc/zfs/zroot.key
chmod 000 /etc/zfs/zroot.key

Create the ZFS Pool

Create the pool with conservative, well-tested options:

zpool create -f -o ashift=12 \
 -O compression=lz4 \
 -O acltype=posixacl \
 -O xattr=sa \
 -O relatime=on \
 -O encryption=aes-256-gcm \
 -O keylocation=file:///etc/zfs/zroot.key \
 -O keyformat=passphrase \
 -o autotrim=on \
 -o compatibility=openzfs-2.2-linux \
 -m none zroot "$POOL_DEVICE"

Create ZFS Datasets

zfs create -o mountpoint=none zroot/ROOT
zfs create -o mountpoint=/ -o canmount=noauto zroot/ROOT/${ID}
zfs create -o mountpoint=/home zroot/home

zpool set bootfs=zroot/ROOT/${ID} zroot

Export and Reimport for Installation

zpool export zroot
zpool import -N -R /mnt zroot
zfs load-key -L prompt zroot

zfs mount zroot/ROOT/${ID}
zfs mount zroot/home

udevadm trigger

Install the Base System

XBPS_ARCH=x86_64 xbps-install \
  -S -R https://mirrors.servercentral.com/voidlinux/current \
  -r /mnt base-system

Copy Host Configuration

Copy the files we generated earlier to the new system:

cp /etc/hostid /mnt/etc
mkdir -p /mnt/etc/zfs
cp /etc/zfs/zroot.key /mnt/etc/zfs

Configure Encrypted Swap

Now we'll set up the encrypted swap partition. This is where the hibernation magic happens - by using a separate LUKS-encrypted partition instead of a ZFS zvol, we can properly resume from hibernation.

Format the swap partition with LUKS:

cryptsetup luksFormat --type luks1 "$SWAP_DEVICE"

Open the encrypted partition, create the swap filesystem, and activate it:

cryptsetup luksOpen "$SWAP_DEVICE" cryptswap
mkswap /dev/mapper/cryptswap
swapon /dev/mapper/cryptswap

Preserve Variables for Chroot

Before entering the chroot, save the disk variables so they remain available inside the new environment:

cat << EOF > /mnt/root/disk-vars.sh
export DISK="$DISK"
export PART_SEP="$PART_SEP"
export BOOT_PART="$BOOT_PART"
export SWAP_PART="$SWAP_PART"
export POOL_PART="$POOL_PART"
export BOOT_DEVICE="$BOOT_DEVICE"
export SWAP_DEVICE="$SWAP_DEVICE"
export POOL_DEVICE="$POOL_DEVICE"
export ID="$ID"
EOF

Enter the Chroot Environment

xchroot /mnt

From this point forward, all commands are executed inside the new system.

First, load the saved variables:

source /root/disk-vars.sh

Configure fstab

Add the swap entry to /etc/fstab:

/dev/mapper/cryptswap   none            swap            defaults        0 0

Set Up Automatic Swap Unlock

To avoid entering the swap password separately after unlocking the ZFS pool, we'll create a keyfile stored on the encrypted ZFS dataset. This is secure because the keyfile only becomes accessible after the ZFS pool is unlocked.

First, install cryptsetup in the new system:

xbps-install -S cryptsetup

Generate a random keyfile and add it to the LUKS partition:

dd bs=1 count=64 if=/dev/urandom of=/boot/volume.key

cryptsetup luksAddKey "$SWAP_DEVICE" /boot/volume.key

chmod 000 /boot/volume.key
chmod -R g-rwx,o-rwx /boot

Add the keyfile to /etc/crypttab:

echo "cryptswap   $SWAP_DEVICE   /boot/volume.key   luks" >> /etc/crypttab

Include the keyfile and crypttab in the initramfs. Create /etc/dracut.conf.d/10-crypt.conf:

install_items+=" /boot/volume.key /etc/crypttab "

Basic System Configuration

Configure keyboard layout and hardware clock. Adjust the keymap and timezone to match your location:

cat << EOF >> /etc/rc.conf
KEYMAP="us"
HARDWARECLOCK="UTC"
EOF

ln -sf /usr/share/zoneinfo/Europe/Rome /etc/localtime

Configure locales:

cat << EOF >> /etc/default/libc-locales
en_US.UTF-8 UTF-8
en_US ISO-8859-1
EOF

echo "LANG=en_US.UTF-8" > /etc/locale.conf

xbps-reconfigure -f glibc-locales

Set the root password:

passwd

Configure ZFS Boot Support

cat << EOF > /etc/dracut.conf.d/zol.conf
nofsck="yes"
add_dracutmodules+=" zfs "
omit_dracutmodules+=" btrfs "
install_items+=" /etc/zfs/zroot.key "
EOF

Install ZFS:

xbps-install -S zfs

Configure ZFSBootMenu

Set the basic boot properties:

zfs set org.zfsbootmenu:commandline="quiet" zroot/ROOT
zfs set org.zfsbootmenu:keysource="zroot/ROOT/${ID}" zroot

The Critical Step: Hibernation Support

Now we need to configure hibernation resume. This is the key insight that makes this setup work: normally, the encrypted ZFS root mounts first, and then it unlocks the swap partition. But when resuming from hibernation, the kernel needs to read the hibernation image from swap before mounting the root filesystem - otherwise, the saved state would be lost.

To solve this, we tell ZFSBootMenu to unlock the swap partition early, before mounting ZFS, by specifying its LUKS UUID.

Get the UUID of your swap partition:

blkid "$SWAP_DEVICE"

You'll see output like:

/dev/...: UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" TYPE="crypto_LUKS" PARTUUID="..."

Store the UUID in a variable for the next step:

SWAP_UUID=$(blkid -s UUID -o value "$SWAP_DEVICE")
echo "Swap UUID: $SWAP_UUID"

Now set the boot parameters using the captured UUID:

zfs set org.zfsbootmenu:commandline="rd.luks.uuid=$SWAP_UUID resume=/dev/mapper/cryptswap" zroot/ROOT/${ID}

Set Up EFI Boot

Create and mount the EFI partition:

mkfs.vfat -F32 "$BOOT_DEVICE"

mkdir -p /boot/efi

Add the EFI partition to /etc/fstab using its UUID:

BOOT_UUID=$(blkid -s UUID -o value "$BOOT_DEVICE")
echo "UUID=$BOOT_UUID    /boot/efi    vfat    defaults    0 0" >> /etc/fstab

Mount it:

mount /boot/efi

Install ZFSBootMenu

xbps-install -S curl

mkdir -p /boot/efi/EFI/ZBM
curl -o /boot/efi/EFI/ZBM/VMLINUZ.EFI -L https://get.zfsbootmenu.org/efi
cp /boot/efi/EFI/ZBM/VMLINUZ.EFI /boot/efi/EFI/ZBM/VMLINUZ-BACKUP.EFI

Configure the EFI boot entries:

xbps-install -S efibootmgr

efibootmgr -c -d "$DISK" -p "$BOOT_PART" \
  -L "ZFSBootMenu (Backup)" \
  -l '\EFI\ZBM\VMLINUZ-BACKUP.EFI'

efibootmgr -c -d "$DISK" -p "$BOOT_PART" \
  -L "ZFSBootMenu" \
  -l '\EFI\ZBM\VMLINUZ.EFI'

Microcode updates

Void Linux is modular, so you may need to install additional packages for your specific hardware. For the Intel microcode, you need the non-free repo: For example:

# For Intel CPUs
xbps-install -S void-repo-nonfree 
xbps-install -S intel-ucode

# For AMD CPUs/GPUs
xbps-install -S linux-firmware-amd

After installing microcode updates, regenerate the boot images and exit:

xbps-reconfigure -fa

Desktop Installation (Optional)

If all you need is a minimal system or a server, you're done and ready to reboot. For a complete desktop environment, continue with the following steps.

Install Core Desktop Packages

xbps-install -S vim nano dbus elogind polkit xorg xorg-fonts xorg-video-drivers xorg-input-drivers dejavu-fonts-ttf terminus-font NetworkManager pipewire alsa-pipewire wireplumber xdg-user-dirs unzip gzip xz 7zip

Install KDE Plasma

xbps-install -S kde-plasma dolphin konsole firefox kdegraphics-thumbnailers ffmpegthumbs vlc ark kwrite discover kf6-purpose

Enable Services

ln -s /etc/sv/NetworkManager /etc/runit/runsvdir/default/
ln -s /etc/sv/dbus /etc/runit/runsvdir/default/
ln -s /etc/sv/udevd /etc/runit/runsvdir/default/
ln -s /etc/sv/polkitd /etc/runit/runsvdir/default/
ln -s /etc/sv/sddm /etc/runit/runsvdir/default/

Configure PipeWire Audio

mkdir -p /etc/xdg/autostart
ln -sf /usr/share/applications/pipewire.desktop /etc/xdg/autostart/

mkdir -p /etc/pipewire/pipewire.conf.d
ln -sf /usr/share/examples/wireplumber/10-wireplumber.conf /etc/pipewire/pipewire.conf.d/
ln -sf /usr/share/examples/pipewire/20-pipewire-pulse.conf /etc/pipewire/pipewire.conf.d/

mkdir -p /etc/alsa/conf.d
ln -sf /usr/share/alsa/alsa.conf.d/50-pipewire.conf /etc/alsa/conf.d
ln -sf /usr/share/alsa/alsa.conf.d/99-pipewire-default.conf /etc/alsa/conf.d

Enable Additional Repositories and Flatpak (Optional)

xbps-install -S void-repo-nonfree void-repo-multilib void-repo-multilib-nonfree

xbps-install -S flatpak
flatpak remote-add --if-not-exists flathub https://dl.flathub.org/repo/flathub.flatpakrepo

Create a Regular User and exit

For desktop use, create a non-root user with appropriate group memberships. Replace username with your desired username.

useradd -m username
passwd username
usermod username -G video,wheel,plugdev,kvm,audio,network
exit

Fix for NetworkManager

xchroot will bind mount /etc/resolv.conf and leave an empty file. Network Manager won't like it. So let's clean it up:

umount -l /mnt/etc/resolv.conf 2>/dev/null || true

rm -f /mnt/etc/resolv.conf
ln -s /run/NetworkManager/resolv.conf /mnt/etc/resolv.conf

Exit and Reboot

umount -n -R /mnt
zpool export zroot
reboot

Post-Installation

If everything went well, after entering your ZFS encryption password, you'll be greeted by the SDDM login screen.

Testing Hibernation

To verify that hibernation works correctly, you can clock the "Hibernate" button or:

loginctl hibernate

The system should power off. When you turn it back on, ZFSBootMenu will prompt for the password, unlock the swap partition, detect the hibernation image, and resume your session exactly where you left off.

If resume fails, check that: 1. The LUKS UUID in the ZFS commandline property matches your swap partition 2. The swap partition is large enough for your RAM 3. The dracut configuration includes the crypttab and keyfile

Conclusion

You now have a fully functional Void Linux system with native ZFS, full disk encryption, and working hibernation. The system is rolling, lightweight, and easy to maintain. Enjoy!

Introducing the illumos Cafe: Another Cozy Corner for OS Diversity

Stefano Marinelli — Mon, 18 Aug 2025 09:04:00 +0200

Introducing the illumos Cafe: Another Cozy Corner for OS Diversity

From the BSD Cafe to illumos Cafe

The idea for this new project was born from the success of the BSD Cafe, an initiative I introduced to the world in July 2023, which received an incredibly positive response. Far more than I ever anticipated. The BSD community already had its well-established hubs: in the Fediverse, places like bsd.network, exquisite.social, and others were already thriving, not to mention all the forums, channels, and Reddit communities.

But in my vision, something was still missing: a hub of services with a positive spirit, built exclusively with open-source tools, where people could come to share, learn, and experience technology with a positive mindset. The BSD Cafe is therefore not just an instance, but a true Cafe - I’ll be speaking more about the BSD Cafe in detail at the next EuroBSDCon.

Why Another Cafe?

In a world increasingly dominated by centralized services under the control (or lack thereof) of the usual big players, it has become essential to create free, independent communities, devoid of the algorithmic and commercial controls that influence our overall experience. From day one, the BSD Cafe has embodied this spirit.

Linux is a good kernel, and there are excellent distributions based on it (some using the GNU userland, others only partially, like Alpine Linux), but it cannot and should not become a monoculture. The alternatives are extremely capable, and for many use cases - in my opinion and experience - they are even more suitable. BSD systems have served me exceptionally well for over 20 years, providing stability and security. At the same time, many other operating systems are renowned for their robustness, reliability, and the quality of their design and implementation.

Why illumos?

illumos is one of them. As the open-source descendant of OpenSolaris, it is an operating system known for its enterprise-grade stability and innovative technologies like ZFS, DTrace, and "zones". It was born from the solid foundations of Solaris and has evolved over time while remaining true to many of its core principles. I have always seen illumos and its distributions as kindred spirits to the BSDs, despite their differences. The philosophy is one of evolution without revolution, of guaranteeing long-term continuity and reliability rather than chasing the latest hype. This is precisely why, for some time now (and thanks in part to the inspiring posts by Joel Carnat, which further sparked my curiosity), I have been running OmniOS and SmartOS alongside my BSD-based setups for certain workloads.

However, there is very little information online about services running on them. So, a few months ago, I began to consider a new project: the illumos Cafe.

The illumos Cafe Project

The illumos Cafe is a project similar to the BSD Cafe (though perhaps less complex, at least initially). It shares the same spirit of positivity and inclusivity and aims to provide services running on illumos-based operating systems to demonstrate that there are no reasons not to use them. Just like with the BSD Cafe, diversifying the operating systems we use - even while using the same platforms - is fundamental to improving the reliability and resilience of the Internet. The Internet was born as a decentralized network, but for most people, it has sadly become just a tool to access the services of big players.

Community and Philosophy

But we want to connect. We want relationships with people, between people. We don't want algorithms. We don't want our data to be monetized by "us and our 65535 partners". We want a network that serves us, an OS that serves us - not an OS that just serves as a vehicle to store our data in "someone else's house". The illumos Cafe, therefore, aims to be a home for anyone interested in developing, using, or who is simply curious about illumos-based operating systems.

Technical Setup

As with the BSD Cafe, the entire setup will be documented. For now, it is very simple: there is a VM (running on FreeBSD and bhyve, on hardware I manage) where I have installed SmartOS. The physical host also runs the reverse proxy (in a jail). Inside the SmartOS VM, there are a series of zones:

Zone 1: nginx (Web Server) - Currently serving the project's homepage.
Zone 2: Mastodon (Social) - Hosting the Mastodon instance and its dependencies at https://mastodon.illumos.cafe.
Zone 3: PostgreSQL (Database) - The Mastodon database, on a dedicated zone.
Zone 4: Redis (Cache) - The Mastodon cache, on a dedicated zone.
Zone 5: snac (LX Zone) - Currently in an LX zone (Alpine) as I ran into some issues getting it to work in a native zone. It will be moved to a native zone as soon as I resolve them. It's serving the snac instance at https://snac.illumos.cafe

Media files are stored on an external physical server (running FreeBSD, the same one as the BSD Cafe, but in a dedicated jail) with SeaweedFS. I was able to compile and run SeaweedFS on illumos without any problems, but at the moment, I don't have a host with enough storage space for the media.

Available Services

More services will arrive over time. For now, two gateways to the Fediverse are already available:

Mastodon - https://mastodon.illumos.cafe
snac - https://snac.illumos.cafe

Both instances share the same rules as the BSD Cafe. Positivity. Supporters, not haters. I want them to be places of enjoyment, not venting. Of friendship, not hate.

Registrations and Logo

Registrations for the Mastodon instance are now open, and the available themes are the default ones plus the colorful TangerineUI - whose orange hue echoes the illumos logo.

The project's logo was not generated by an AI. I made it myself by hastily sticking the illumos SVG onto a coffee cup. Basic, perhaps. But authentic.

Looking Ahead

The BSD Cafe will, of course, remain my primary home. But I want to bring illumos into the Fediverse and provide a home for anyone who wishes to share their interest in this excellent OS.

I will document the entire process, just as I did with Mastodon on FreeBSD, as it is a bit more intricate. Because in my dreams, I see Fediverse statistics showing instances spread fairly evenly across the major open-source operating systems. Because relying on a single OS, even if it's open-source, and ceasing to support the others is also a single point of failure.

Make Your Own Backup System – Part 2: Forging the FreeBSD Backup Stronghold

Stefano Marinelli — Tue, 29 Jul 2025 08:00:00 +0200

With the primary backup strategies and methodologies introduced, we've reached the point where we can get specific: the Backup Server configuration.

When choosing the type of backup server to use, I tend to favor specific setups: either I trust a professional backup service provider (like Colin Percival's Tarsnap), or I want full control over the disks where the backups will be hosted. In both cases, for the past twenty years, my operating system of choice for backup servers has been FreeBSD. With a few rare exceptions for clients with special requests, it covers all my needs. When I require Linux-based solutions, such as the Proxmox Backup Server, I create a VM and manage it within.

I typically use both IPv4 and IPv6. For IPv4, I "play" with NAT and port forwarding. For IPv6, I tend to assign a public IPv6 address to each jail or VM, which is then filtered by the physical server's firewall. Unfortunately, every provider, server, and setup has a different approach to IPv6, making it impossible to cover them all in this article. When a provider allows for routed setups, I use this approach: Make your own VPN: FreeBSD, WireGuard, IPv6, and ad-blocking included - assigning a /72 to the bridge for the jails and VMs.

In my opinion, FreeBSD is a perfect all-rounder for backups, thanks to its ability to completely partition services. You can separate backup services (or specific servers/clients) into different jails or even VMs. Furthermore, using ZFS greatly enhances both flexibility and the range of tools you can use.

The main distinction is usually between local backup servers (physically accessible, though not always attended, and in locations deemed secure) and remote ones, such as leased external servers. I personally use a combination of both. If the services I need to back up are external, in a datacenter, and need to be quickly restorable, I prefer to always have a copy on another server in a different datacenter with good outbound connectivity. This guarantees good bandwidth for restores, which isn't always available from a local connection to the outside world. However, an internal, nearby, and accessible backup server (even a Raspberry Pi or a mini PC) ensures physical access to the data. Whenever possible, I maintain both an external and an internal copy - and they are autonomous, meaning the internal copy is not a replica of the external one, but an additional, independent backup. This ensures that if a problem occurs with the external backup, it won't automatically propagate to the internal one. In any case, the backup must always be in a different datacenter from the one containing the production data. When the fire at the OVH datacenter in Strasbourg caused the entire complex to shut down, many people found themselves in trouble because their backups were in the same, now unreachable, location. I had a copy with another provider, in a different datacenter and country, as well as a local copy.

Despite it being "just" a backup server, I almost always use some form of disk redundancy. If I have two disks, I set up a mirror. With three or more, I use RaidZ1 or RaidZ2. This is because, in my view, backups are nearly as important as production data. The inability to recover data from a backup means it's lost forever. And it happens often, very often, that someone contacts me to recover a file (or a database, etc.) days or weeks after its accidental loss or deletion. Usually, pulling out a file from a two-month-old backup generates a mix of disbelief, admiration, but above all, a sense of security in the person requesting it. And that is what our work should instill in the people we collaborate with.

The backup server should be hardened. If possible, it should be protected and unreachable from the outside. My best backup servers are those accessible only via VPN, capable of pulling the data on their own. If they are on a LAN, it's even better if they are completely disconnected from the Internet.

For this very reason, backups must always be encrypted. Having a backup means having full access to the data, and the backup server is the prime target for being breached or stolen if the goal is to get your hands on that data. I've seen healthcare facilities' backup servers being targeted (in a rather trivial way, to be honest) by journalists looking for health details of important figures. It is therefore critical that the backup server be as secure as possible.

Based on the type of access, I use two types of encryption:

If the server is local (especially if the ZFS pool is on external disks), I usually install FreeBSD on UFS in read-only mode, as I've described in a previous article, and encrypt the backup disks with GELI. This ensures that in the event of a "dirty" shutdown (more likely in unattended environments), I can reconnect to the host and then reactivate the ZFS pool. This approach makes it nearly impossible to retrieve even the pool's metadata if the disks are stolen, as GELI performs a full-device encryption. For example, an employee of a company I work with stole one of the secondary backup disks (which was located at a different, unmonitored company site) to steal information. He got nothing but a criminal complaint. With this approach, it's also not necessary to further encrypt the datasets, which avoids some issues (which I'll discuss later, in a future post).
If the server is remote, in a datacenter, I usually use ZFS native encryption, encrypting the main backup dataset (and BastilleBSD's, if applicable). Consequently, all child datasets containing backups will also be encrypted. In this case as well, a password will be required after a reboot to unlock those datasets, ensuring that the data cannot be extracted if control of the disks is lost.

Here is an example of how to use GELI to encrypt an entire partition and then create a ZFS pool on it (in the example, the disk is da1 - do not follow these commands blindly, or you will erase all content on the da1 device!):

# WARNING: This destroys the existing partition table on disk da1
gpart destroy -F da1

# Create a new GPT partition table
gpart create -s gpt da1

# Add a freebsd-zfs partition that spans the entire disk
# The -a 1m flag ensures proper alignment
gpart add -t freebsd-zfs -a 1m da1

# Initialize GELI encryption on the new partition (da1p1)
# We use AES-XTS with 256-bit keys and a 4k sector size
# The -b flag means "boot," prompting for the passphrase at boot time
geli init -b -l 256 -s 4096 da1p1
# You will be prompted for a passphrase: choose a strong one and save it!

# Attach the encrypted partition. A new device /dev/da1p1.eli will be created.
# You will be prompted for the passphrase you just set
geli attach da1p1

# (Optional) Check the status of the encrypted device
geli status da1p1

# Create the ZFS pool "bckpool" on the encrypted device
# We enable zstd compression (an excellent compromise) and disable atime
zpool create -O compression=zstd -O atime=off bckpool da1p1.eli

In this setup, the reference pool for everything related to backups will be bckpool - and you'll need to keep this in mind for the next steps. Additionally, after every server reboot, you'll need to "unlock" the disk and import the pool:

# Enter the passphrase when prompted
geli attach da1p1

# Import the ZFS pool, which is now visible
zpool import bckpool

With this method, it's not necessary to encrypt the ZFS datasets, as the underlying disk (or, more precisely, the partition containing the ZFS pool) is already encrypted.

If, instead, you choose to encrypt the ZFS dataset (for example, if you install FreeBSD on the same disks that will hold the data and don't want to use a multi-partition approach), you should create a base encrypted dataset. Inside it, you can create the various backup datasets, VMs, and the BastilleBSD mountpoint. Due to property inheritance, they will all be encrypted as well.

To create an encrypted dataset, a command like this will suffice:

# Creates a new dataset with encryption enabled.
# keylocation=prompt will ask for a passphrase every time it's mounted.
# keyformat=passphrase specifies the key type.
zfs create -o encryption=on -o keylocation=prompt -o keyformat=passphrase zfspool/dataset

In this case, after every reboot, you will need to load the key and mount the dataset:

zfs load-key zfspool/dataset
zfs mount zfspool/dataset

Keep in mind the setup you choose, as many of the subsequent choices and commands will depend on it.

Base System Setup

I'll install BastilleBSD - a useful tool for separating services into jails. It will be helpful for isolating our backup services:

pkg install -y bastille

If you used ZFS for the root filesystem, you can proceed directly with the setup. Otherwise (i.e., ZFS on other disks), you'll need to edit the /usr/local/etc/bastille/bastille.conf file and specify the correct dataset on which to install the jails. Then run:

bastille setup

Once the automatic setup is complete, check the /etc/pf.conf file - it will be automatically configured to only accept SSH connections. Ensure the network interface is set correctly. When you activate pf, you will be kicked out of the server, but you can then reconnect.

service pf start

Let's bootstrap a FreeBSD release for the jails - this will be useful later.

bastille bootstrap 14.3-RELEASE update

Now, we create a local bridge. Jails and VMs can be attached to it, making them fully autonomous. Using VNET jails, for example, will allow the creation of VPNs or tun interfaces inside them, simplifying potential future setups (and increasing security by using a dedicated network stack).

Modify the /etc/rc.conf file and add:

# Add lo1 and bridge0 to the list of cloned interfaces
cloned_interfaces="lo1 bridge0"
# Assign an IP address and netmask to the bridge
ifconfig_bridge0="inet 192.168.0.1 netmask 255.255.255.0"
# Enable gateway functionality for routing
gateway_enable="yes"

Let's also modify /etc/pf.conf to allow the 192.168.0.0/24 subnet to access the Internet via NAT. We will skip packet filtering on bridge0 and enable NAT. This isn't the most secure setup, but it's sufficient to get started:

#...
# Skip PF processing on the internal bridge interface
set skip on bridge0
#...
# NAT traffic from our internal network to the outside world
nat on $ext_if from 192.168.0.0/24 to any -> ($ext_if:0)
#...

To ensure the new settings are correct, it's a good idea to test with a reboot.

Since I often configure vm-bhyve in my setups, I prefer to install it right away, creating the dataset that will contain the VMs and installation templates. Remember that zroot is only valid if you installed the entire system on ZFS; otherwise, you'll need to change it to your own dataset:

# Install required packages
pkg install vm-bhyve grub2-bhyve bhyve-firmware
# Create a dataset to store VMs
zfs create zroot/VMs
# Enable the vm service at boot
sysrc vm_enable="YES"
# Set the directory for VMs, using the ZFS dataset
sysrc vm_dir="zfs:zroot/VMs"
# Initialize vm-bhyve
vm init
# Copy the example templates
cp /usr/local/share/examples/vm-bhyve/* /zroot/VMs/.templates/

At this point, I usually enable the console via tmux. This means that when a VM is launched, it won't open a VNC port by default, but a tmux session connected to the VM's serial port. Let's install and configure tmux:

pkg install -y tmux
vm set console=tmux

Let's also attach the switch we created (bridge0) to vm-bhyve so we can use it:

vm switch create -t manual -b bridge0 public

Now, vm-bhyve is ready.

The basic infrastructure is complete. We now have:

ZFS to ensure data integrity, which will also handle redundancy, etc.
BastilleBSD to manage jails, useful for backing up Linux, NetBSD, OpenBSD, and non-ZFS FreeBSD machines.
vm-bhyve to install specific systems (like Proxmox Backup Server).

Backup Strategies

I use various backup tools, too many to list in this article. So I'll make a broad distinction, describing how to use this server to achieve our goal: securing data.

For FreeBSD servers with ZFS (hosts, VMs, jails, hypervisors, and their respective VMs), I use an extremely useful, efficient, and reliable tool: zfs-autobackup.
For Linux servers (without ZFS), NetBSD, OpenBSD, etc. (any non-ZFS OS), I usually use BorgBackup. There are other fantastic tools like restic, Kopia, etc., but BorgBackup has never let me down and has served me well even on low-power devices and after incredibly complex disasters.
For Proxmox servers (a solution I've used with satisfaction in production since 2013, although I'm recently migrating to FreeBSD/bhyve where possible), I use two possible alternatives (often both at the same time): if the storage is ZFS, I use the zfs-autobackup approach. In either case, the most practical solution is the Proxmox Backup Server. And the Proxmox Backup Server is one of the reasons I proposed installing vm-bhyve: running it in a VM and storing the data on the FreeBSD host gives you the best of both worlds. Some time ago, I tried running it in a FreeBSD jail (via Linuxulator), but it didn't work.

Backups using zfs-autobackup

zfs-autobackup is an extremely useful and effective tool. It allows for "pull" type backups, as well as having an intermediary host that connects to both the source and destination, which is useful if you don't want direct contact between the source and destination. I won't describe the latter setup, but the documentation is clear, and I have several of them in production, ensuring that the production server and its backup server cannot communicate with each other.

I usually create a dataset for each server and instruct zfs-autobackup to keep that server's backups in that dataset. The snapshots taken and transferred will all be from the same instant, so as not to create a time skew (some tools of this kind snapshot a dataset, then transfer it, which can result in minutes of difference between two different datasets from the same server).

I've described in detail how I perform this type of backup in a previous post, so I suggest reading that post for reference.

Let's install zfs-autobackup on the FreeBSD server:

pkg install py311-zfs-autobackup mbuffer

Backups for other servers using BorgBackup

When I don't have ZFS available or need to perform a file-based backup (all or partial), I use a different technique. BorgBackup backups are primarily "push" based, meaning the client will connect to the backup server. This is not optimal or the most secure approach, as the backup server should, in theory, be hardened. Even when protecting everything via VPN, the risk remains that a compromised server could connect to its backup server and alter or delete the backups. I have seen this happen in ransomware cases (especially in the Microsoft world), and so I try to be careful to minimize this type of problem, mainly through snapshots of the backup server (an operation that will be described later).

To ensure the highest possible security, I create a FreeBSD jail on the backup server for each server I need to back up. The advantage of this approach is the complete separation of all servers from each other. By using a regular user inside a jail, a compromised server that connects to its backup server would only be able to reach its own backups, as it would be confined to a user account and, even if it managed to escalate privileges, still be inside a jail.

Let's say, for example, we want to back up a server called "ServerA" (great imagination, I know). We create a dedicated jail on the backup server:

# Create a new VNET jail named "servera" attached to our bridge
bastille create -B servera 14.3-RELEASE 192.168.0.101/24 bridge0

BastilleBSD will automatically set the host's gateway for the jail. In our case, this is incorrect, so we need to modify it and set the jail's gateway to 192.168.0.1 in the /usr/local/bastille/jails/servera/root/etc/rc.conf file:

# ...
defaultrouter="192.168.0.1"
# ...

Restart the jail and connect to it:

bastille restart servera
bastille console servera

Now, inside the jail, we install borgbackup:

pkg install py311-borgbackup

BorgBackup doesn't run a daemon; it's launched by the remote server (which sends its data to the backup server), so it's important that the installed version is compatible with the one on the remote host.

Since we'll be using SSH, let's enable it:

service sshd enable
service sshd start

And create a non-privileged user for this purpose:

# The 'adduser' utility provides an interactive way to create a user.
root@servera:~ # adduser
Username: servera
Full name: Server A
Uid (Leave empty for default): 
Login group [servera]: 
Login group is servera. Invite servera into other groups? []: 
Login class [default]: 
Shell (sh csh tcsh nologin) [sh]: 
Home directory [/home/servera]: 
Home directory permissions (Leave empty for default): 
Use password-based authentication? [yes]: 
Use an empty password? (yes/no) [no]: 
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]: 
Username    : servera
Password    : <random>
Full Name   : Server A
Uid         : 1001
Class       : 
Groups      : servera 
Home        : /home/servera
Home Mode   : 
Shell       : /bin/sh
Locked      : no
OK? (yes/no) [yes]: yes
adduser: INFO: Successfully added (servera) to the user database.
adduser: INFO: Password for (servera) is: JIkdq8Ex

The user is created and can receive SSH connections. After setting everything up, I suggest disabling password-based login in the jail's SSH configuration, using only public key authentication.

As mentioned, the biggest risk of a "push" backup is that a compromised client could access the backup server and delete or encrypt the backup history, rendering it useless.

To drastically mitigate this risk, we can configure SSH to force the client to operate in a special Borg mode called append-only. In this mode, the SSH key used by the client will only have permission to create new archives, not to read or delete old ones. However, this approach could complicate some client-side operations (like mount, prune, etc.), forcing them to be done on the server. For this reason, I won't describe it in this setup, "limiting" myself to taking snapshots of the repositories. It can be a very good practice, so I recommend considering it.

Let's initialize the BorgBackup repository. In this example, for simplicity, I won't set up repository encryption. If the jails are on an encrypted dataset or GELI-encrypted disks, there will still be data encryption on the disks, but there will be no protection against someone who could physically access the server while the disks are mounted. As usual, security is like an onion: every layer helps. Personally, I suggest enabling and using it ALWAYS.

# Switch to the new user
su -l servera
# Initialize a new Borg repo named "servera" with no encryption (for this example)
borg init -e none servera

The jail is ready, but it's unreachable from the outside. There are two ways to make it accessible:

Install a VPN system inside the jail itself. Using tools like Zerotier or Tailscale (which don't need to expose ports) will immediately create the conditions to connect to the jail, which will remain inaccessible from the outside. As the jail is a VNET jail, we're free to choose any of the supported VPN technologies.
Expose a port on the backup server, i.e., on the host, to allow external connections. Many advise against this path as they consider it less secure. It is, but sometimes we don't have the luxury of installing whatever we want on the server we're backing up.

To expose the port, go back to the host and modify the /etc/pf.conf file, creating the rdr and pass rules to let packets in:

# ...
# Redirect incoming traffic on port 1122 to the jail's SSH port (22)
rdr on $ext_if inet proto tcp from any to any port = 1122 -> 192.168.0.101 port 22
# ...
# Allow incoming traffic on port 1122
pass in inet proto tcp from any to any port 1122 flags S/SA keep state

Reload the pf configuration:

service pf reload

The jail will now be reachable on the server's public IP, on port 1122. Obviously, this port number is for illustrative purposes, and I used from any, but for better security, you should replace any with the IP address of the server that will be connecting to perform the backup.

By repeating this process for each server and creating a separate jail for each, you can have isolated jails in separate datasets with their backups, potentially setting space limits using ZFS quotas.

It's important to remember that backing up a live filesystem (i.e., without a snapshot or dumps) has a very high probability of being impossible to restore completely. Databases hate this approach because files will change while being copied and tend to get corrupted. Of course, it depends on the nature of the data (a backup of a static website will have no issues, but a WordPress database probably will), but it's crucial to think about a technique to snapshot the filesystem before proceeding. For example, I have already written about how to create snapshots on FreeBSD with UFS in a previous article: FreeBSD tips and tricks: creating snapshots with UFS.

I will cover other operating systems in a future, dedicated post.

Proxmox Backup Server in a Dedicated VM

Starting with version 4.0 (which is still in beta at the time of this writing), Proxmox Backup Server (PBS) supports storing its data in an S3 bucket. This is excellent news as it decouples the server from the data. There are great open-source S3 implementations, like Minio or SeaweedFS, which allow for clustering, replication, etc. In this "simple" case, we will install Proxmox Backup Server in a small VM, while for the data, we'll install Minio in a native FreeBSD jail. The advantage is undeniable: the VM will only serve as an "intermediary", but the data will rest directly on the FreeBSD host's dataset, natively. It will also be possible to specify other external endpoints, other repositories, etc.

As a philosophy, I tend not to use external providers unless for specific needs, so installing Minio in a jail is a perfect solution to manage this situation.

Let's install PBS by downloading the ISO from their website (https://enterprise.proxmox.com/iso/) - at this moment, the version that supports this setup is 4.0 Beta.

The directory to download to is the vm-bhyve ISOs directory. It's not strictly necessary, but it's useful for not "losing" it somewhere. So, go to the directory and download it:

cd /zroot/VMs/.iso
fetch https://enterprise.proxmox.com/iso/proxmox-backup-server_4.0-BETA-1.iso

Now let's create a VM with vm-bhyve. We can start from the Debian template, but we'll make some modifications to optimize performance. In this example, I'm giving it 30 GB of disk space, 2 GB of RAM, and 2 cores.

If you want to store all backups inside the VM, you'll need to size the virtual disk correctly (or create and attach another one). In this case, I will focus on the "clean" VM that will store its data on a dedicated jail with Minio.

vm create -t debian -s 30G -m 2G -c 2 pbs

Once the empty VM is created, let's modify its options:

vm configure pbs

We will modify the VM to be UEFI and to use the NVME disk driver - bhyve performs significantly better on NVME than virtio, as previously tested:

loader="uefi"
cpu="2"
memory="2G"
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"

Fortunately, the Proxmox team has provided for the installation of the Backup Server on devices without a graphical interface, so the boot menu will allow installation via serial console. Let's launch the installation and connect to the virtual serial console:

cd /zroot/VMs/.iso
vm install pbs proxmox-backup-server_4.0-BETA-1.iso
vm console pbs

Select the installation via Terminal UI (serial console) and proceed normally as if it were a physical host, assigning an IPv4 address from the 192.168.0.x range (in this example, I'll use 192.168.0.3).

This way, the Proxmox Backup Server will run in a VM, with the ability to take snapshots before updates, etc., without any worries.

Once the installation is complete, PBS will reboot and listen on port 8007 of its IP. Again, as with the jails, we have two options: install a VPN system within the VM itself (thus exposing it automatically only on that VPN - generally a more secure operation) or expose port 8007 on the server's public IP.

In the latter case, add the relevant lines to the /etc/pf.conf file on the FreeBSD backup server:

# ...
# Redirect incoming traffic on port 8007 to the PBS VM's web interface
rdr on $ext_if inet proto tcp from any to any port = 8007 -> 192.168.0.3 port 8007
# ...
# Allow that traffic to pass
pass in inet proto tcp from any to any port 8007 flags S/SA keep state

Reload the pf configuration:

service pf reload

The PBS VM configuration is complete. If you chose to use the PBS's internal disk as a repository, no further operations are necessary (other than the normal repository creation, etc., within PBS).

In this case, however, we will use a different approach.

Creating a Minio Jail as a Data Repository for PBS

This approach, in my opinion, has a number of important advantages. The first is that Minio will run in a dedicated jail on the host, at full performance, and will store the data directly on the physical ZFS datapool, thus removing any other layer in between. This jail could potentially be moved to other hosts (by connecting PBS and the jail via VPN or public IP), made redundant thanks to all of Minio's features, etc. Another solution I am successfully testing (in other setups) is SeaweedFS.

Let's create a dedicated jail with Minio and put it on the bridge, so that PBS can access it on the LAN.

bastille create -B minio 14.3-RELEASE 192.168.0.11/24 bridge0

If not configured directly, BastilleBSD will use the host's gateway for the jail, which is incorrect in this case. So let's go modify it and restart the jail. Enter the jail with:

bastille console minio

And modify the /etc/rc.conf file to have the correct gateway (following the example addresses):

# ...
ifconfig_vnet0=" inet 192.168.0.11/24 "
defaultrouter="192.168.0.1"
# ...

Exit the jail and restart it:

bastille restart minio

Enter the jail and install Minio:

bastille console minio
pkg install -y minio

Minio is already able to start, but PBS, even on the LAN, wants an encrypted connection. Fortunately, there's a handy tool that can generate the certificates for us:

# Download the certgen tool
fetch https://github.com/minio/certgen/releases/latest/download/certgen-freebsd-amd64

# Make it executable and run it for our jail's IP
chmod a+rx certgen-freebsd-amd64
./certgen-freebsd-amd64  -host "192.168.0.11"

# Create the necessary directories and set permissions
mkdir -p /usr/local/etc/minio/certs
cp private.key public.crt /usr/local/etc/minio/certs/
chown -R minio:minio /usr/local/etc/minio/certs/

Let's view the certificate's fingerprint. Since it's self-signed, we'll need it for PBS later. For security reasons, PBS will ask for the fingerprint of non-directly verifiable certificates. Run the following command and take note of the result:

openssl x509 -in /usr/local/etc/minio/certs/public.crt -noout -fingerprint -sha256

At this point, enable and configure Minio in /etc/rc.conf. WARNING: The username and password (access key and secret) used in this example are insecure and for testing purposes only. It is strongly recommended to use different values:

# Enable Minio service
minio_enable="YES"
# Set the address for the Minio console
minio_console_address=":8751"
# Set the root user and password as environment variables
minio_env="MINIO_ROOT_USER=testaccess MINIO_ROOT_PASSWORD=testsecret"

Start Minio:

service minio start

If everything went correctly, Minio is now running (with its certificates) and ready to receive connections.

It's now time to create the bucket(s) that PBS will use. There are several ways to do this, but to test that everything is working and to configure PBS, I suggest connecting via an SSH tunnel.

# Create an SSH tunnel from your local machine to the backup server
# Port 8007 is forwarded to the PBS web UI
# Port 8751 is forwarded to the Minio console
ssh user@backupServerIP -L8007:192.168.0.3:8007 -L8751:192.168.0.11:8751

This way, we'll create a tunnel between the FreeBSD backup server and our workstation, mapping 127.0.0.1:8007 to 192.168.0.3:8007 (the PBS web interface) and 127.0.0.1:8751 to 192.168.0.11:8751 (the Minio console port).

Now, connect to https://127.0.0.1:8751, enter the credentials specified in /etc/rc.conf, and create a bucket.

Once the bucket is created, you can configure PBS to use it. Connect to PBS via https://127.0.0.1:8007 and go to S3 Endpoints. Set a name, use 192.168.0.11 as the IP and 9000 as the port, enter the access and secret keys, and the certificate fingerprint we generated earlier. Enable "Path Style" or it will not work.

Then go to Datastores and add it, as you would for any other S3 datastore, by specifying the created bucket and a local directory where the system will keep its cache.

If everything was set up correctly, PBS will create its structure in the bucket, and from that moment on, you can use it. Always keep in mind that this is still a "technology preview", so there may be issues, but from my tests, it is sufficiently reliable.

Taking Local Snapshots of Backups

One of the most common techniques used in ransomware attacks is to also delete or encrypt backups. They often use automated methods, relying on the fact that many (too many!) consider a "backup" to be a simple copy of files to a network share. However, it's not impossible that, in specific cases, they might compromise the machine and connect to the backup server. This is nearly impossible with a "pull" type backup (like the one managed by zfs-autobackup) but is still possible with the "push" approach, which involves using BorgBackup or similar tools.

This happened to one of my clients once - in that case, the problem originated internally, from an employee who wanted to cover up his mistake, inadvertently creating a disaster - but that will be material for another post.

Fortunately, the client had a system that solved the problem: thanks to ZFS, we can have local snapshots on the backup server, which are invisible and inaccessible to the production server. Since we have already installed zfs-autobackup, it's easy to use it for this purpose as well. I've already talked about this in a previous article and won't rewrite the steps here. Just consult that article, keeping in mind that in this case, it's not advisable to snapshot all the datasets on the backup server (the space would grow exponentially), but only those at risk. In the cases analyzed in this post, this applies only to the push part, as PBS will also be accessible only from the Proxmox servers and not from the VMs they contain. If, in this case too, you don't trust those who manage the Proxmox servers, just set up snapshots for the Minio jail as well.

Conclusion

This long post aims to analyze, in a general way, how I believe one can manage reasonably secure backups of their data. Obviously, there are many variables, additional precautions, possible optimizations, hardening, etc., that must be studied on a case-by-case basis. There are old rules, new rules, old and new philosophies. Recently, many people who have embraced the cloud have often stopped thinking about backups, only to realize it when something happens and the data has, indeed, vanished... into the clouds.

In this post, I have generically covered the setup of the backup server, and this demonstrates how FreeBSD, thanks to its features, can be considered an ideal platform for this type of task.

In the next articles in this series, I will examine the client side, i.e., how to structure them for a sufficiently reliable backup, and how to monitor everything - because I've seen this too: people resting easy because the backup was supposedly running every night, but in fact, the backup had been failing every night for more than 4 years.

Stay Tuned and stay...backupped!

OSDay 2025 - Why Choose to Use the BSDs in 2025

Stefano Marinelli — Sun, 23 Mar 2025 10:30:00 +0100

This is the text underlying my presentation at OSDay 2025, held on 21 March 2025 in Florence, Italy. There was limited time, so I couldn't go into much detail and had to keep things more general and structured than usual. You can watch the video of my talk on YouTube.

The slides can be downloaded here

Happy reading!

OSDay Florence - 21 March 2025 - Why Choose to Use the BSDs in 2025

"I'm Stefano Marinelli, I solve problems."

I'm the founder and Barista of the BSD Cafe, a community of *BSD enthusiasts.

I work in my company, called Prodottoinrete - a container of ideas and solutions.

I'm passionate about technology and computing, and I've made my passion my profession. Every morning, when I sit in front of the computer, a new world opens up for me to explore.

I've been a Linux user since 1996, before I turned 17. Back then, I used Fidonet and would read about alternative operating systems. I experimented with Linux distributions from CDs, and by 1997, Linux became my everyday system. It was only in 2002 that I began exploring BSD systems, largely thanks to FreeBSD's fantastic handbook.

The relationship we had with Open Source 20-30 years ago was fundamentally different than today. Back then, embracing Open Source meant thinking differently. It meant embracing freedom. We chose Linux and the BSDs when Windows and commercial Unix systems dominated the market. Not because they were simple or free (as in free beer), but for freedom from impositions - both technological and ideological.

I solve problems. And to solve problems effectively, we need to recognize when the landscape has changed.

The reality today is that while we won that war - Open Source is everywhere - we're facing a new challenge. The "mainstream" Open Source world is creating monocultures. The focus has shifted from technologies to specific tools. We're seeing innovation for novelty's sake, not problem-solving.

This shift has profound implications. In a world dominated by cyber threats, where everything is connected and we completely depend on technology, the value of stability has been lost. By stability, I don't just mean that a system doesn't crash. I mean continuity over time, upgradeability, and system visibility.

Instead, the industry seems obsessed with the hype cycle. "New" is prioritized over secure and stable. The mantra has become: - "It will be fixed in the next version" - "We need automatic restarts when it crashes" - "Do we need software that crashes less? We have systemd and Kubernetes to restart crashed workloads!" - "We need moooarrr powaaaaaaar!!!!"

Let me give you a concrete example. A program written in Rust should be memory safe - that's one of the main selling points of the language. But if that program uses unsafe functions and segfaults, what advantage does it offer over a mature C implementation? Stability matters more than the implementation language.

I solve problems. And creating a monoculture does not solve problems - it creates new ones.

Yes, Linux, Docker, and Kubernetes are better than closed source solutions. But when everyone uses the same tools, freedom dies. We use them because "everyone does" rather than because they're the best tool for our specific needs.

If we had only used what everyone else used, we wouldn't have Linux or the BSDs today. There would be no LibreOffice, no Nextcloud. We'd just have Windows variations and expensive Unix systems. We'd be bound by licenses and vendors, stuck with closed solutions.

This is where the BSDs offer a compelling alternative: "Be free and evaluate alternatives. Always."

For those who don't know, the original BSD started in the 1970s (before Linux was conceived). Minix was created as an educational OS because it was believed that BSD, mature and professional, would be the Open Source OS that would dominate the market. A legal case stalled development and scared adopters, but in 1993, NetBSD and FreeBSD emerged. OpenBSD forked from NetBSD later, then DragonflyBSD from FreeBSD.

As Linus Torvalds said in 1993, "If 386BSD had been available when I started on Linux, Linux would probably never had happened."

What makes the BSDs special is their philosophy: - Kernel and userland developed by same teams - Consistency in tools and updates - Excellent documentation - especially OpenBSD, where insufficient docs are considered a bug - Man pages contain virtually everything - Evolution, not Revolution

Let me briefly introduce the main BSD variants that I work with daily:

FreeBSD is a generalist system. It focuses on stability and performance - with HardenedBSD as a security-enhanced fork. It has native ZFS, Boot Environments, and complete separation between OS and packages. It's had container support via jails since 2000 - which predates Linux cgroups by a decade! It offers bhyve virtualization (more efficient than KVM). OPNsense and pfSense are based on FreeBSD, as pf is a powerful firewall. It's used by Netflix for streaming video delivery and forms the foundation for PlayStation consoles. MacOS and iOS also contain some FreeBSD code.

OpenBSD focuses on security and code correctness. Its code is constantly audited and simplified - less is more. The team believes "The more complex the code, the less maintainable." It has security mechanisms like pledge() and unveil(). OpenSSH (and many other nice things) originated and are developed here. Development is driven by team priorities, not user requests. It's ideal for routers, firewalls, and security-critical systems.

NetBSD lives by the motto "Of course it runs NetBSD!" Its focus is on correctness, portability, and proper implementation. It supports 50+ architectures. Development centers on compatibility, which necessitates code quality. It must function on decades-old hardware. It's ideal for systems that require stability without the need for continuous updates, like embedded devices.

I solve problems. And in my experience, the BSDs have consistently proven to be excellent problem-solvers. Here are some real-world benefits I've experienced:

Better stability and security
Simplified administration - upgrades won't destroy your system
Less vulnerability to common attacks - "We don't need this patch, you're running OpenBSD and it's been fixed 20 years ago"
Network interfaces maintain consistent names - ix0 will remain ix0, not renaming from enx3e3300c9e14e to enp10s0f0np0
FreeBSD shows lower system load compared to Linux
FreeBSD handles I/O pressure better - on the same hardware, I've seen 70% time reduction
FreeBSD delivers improved end-user experience/responsiveness
NetBSD provides the comfort of "Don't worry - your platform will be supported for the foreseeable future"

So why choose BSD in 2025? I believe there are several compelling reasons:

Security in an increasingly hostile environment
Stability in a world obsessed with novelty
Performance without unnecessary complexity
Freedom from the mainstream monoculture
Systems designed with coherent philosophy

Don't be afraid to try BSD systems - despite the Beastie mascot, they don't hurt and you'll appreciate them!

See you at BSD Cafe!

Managing ZFS Full Pool Issues with Reserved Space

Stefano Marinelli — Thu, 28 Nov 2024 20:25:00 +0100

Yesterday morning, I received a panicked call from a developer:
"I accidentally filled up the storage, and now I can't perform any operations! My ZFS pool is full!"

I immediately reassured them because I had anticipated this kind of issue. One of the things I almost always do when managing ZFS file systems is to reserve space in a specially created dataset.

This is because ZFS, like all CoW (Copy-on-Write) file systems, can find itself unable to free up space when completely full. By using reserved space, I can always free it up and delete other data, restoring the system to normal operations.

To reserve space, simply create a dataset and assign it a reserved size. Of course, this dataset should not be used for anything else; otherwise, the entire purpose would be defeated.

To create it and reserve space, you only need two simple commands. For example:

zfs create zroot/reserved
zfs set reservation=5G zroot/reserved

This creates the dataset and assigns it 5 GB of reserved space.

Here’s the situation before the operation:

zfs list zroot
NAME    USED  AVAIL  REFER  MOUNTPOINT
zroot  3.02G   109G    96K  /zroot

And here’s the situation after:

zfs list zroot
NAME    USED  AVAIL  REFER  MOUNTPOINT
zroot  8.02G   104G    96K  /zroot

As you can see, the 5 GB are removed from the available space and marked as used, but they are actually empty.

In case of a full file system, you can delete this dataset (or reduce its size) to return to normal file system operation.

Even with this technique, I still recommend not filling ZFS pools beyond 80% of their capacity, as performance degrades significantly past that point.

Migrating Windows VMs from Proxmox BIOS/KVM to FreeBSD UEFI/bhyve

Stefano Marinelli — Fri, 15 Nov 2024 10:57:00 +0100

A client of mine has several Windows Server VMs, which I had not migrated to FreeBSD/bhyve until a few weeks ago. These VMs were originally installed with the traditional BIOS boot mode, not UEFI, on Proxmox. Fortunately, their virtual disks are on ZFS, which allowed me to test and achieve the final result in just a few steps.

This is because Windows VMs (server or otherwise) often installed on KVM (Proxmox, etc.), especially older ones, are non-UEFI, using the traditional BIOS boot mode. bhyve doesn’t support this setup, but Windows allows changing the boot mode, and I could perform the migration directly on the target FreeBSD server.

Setting Up the Network

Before starting, as usual in my setups, I manually created the bridge and configured a few pf rules for outbound NAT, which these servers need. Let’s begin with the modifications to /etc/rc.conf:

cloned_interfaces="bridge0"
ifconfig_bridge0="inet 192.168.33.1 netmask 255.255.255.0"
gateway_enable="YES"
pf_enable="YES"

Next, for the pf.conf:

ext_if="igb0"

set block-policy return
set skip on bridge0
set skip on lo0

# Allow the VMs to connect to the outside world
nat on $ext_if from {192.168.33.0/24} to any -> ($ext_if)

block in all
antispoof for $ext_if inet
pass in inet proto icmp
pass out quick keep state
pass in inet proto tcp from any to any port ssh flags S/SA keep state

To ensure everything is working as expected, I recommend rebooting the server. It’s not strictly necessary since you could reload network configurations and start pf without a reboot, but I prefer a few extra reboots in these cases. Why? Because system administrators are often afraid to reboot production servers, unsure if they will come back up smoothly. A configuration error could prevent a successful reboot, and this would be a significant issue if the server is already in production. I’ve faced this situation myself—many times! To mitigate this, I make frequent reboots, especially after changing network configurations, ensuring everything starts correctly on reboot.

Installing and Configuring vm-bhyve

Next, I installed vm-bhyve on the target FreeBSD host and configured it:

pkg install vm-bhyve-devel bhyve-firmware
zfs create zroot/VMs
sysrc vm_enable="YES"
sysrc vm_dir="zfs:zroot/VMs"
vm init
cp /usr/local/share/examples/vm-bhyve/* /zroot/VMs/.templates/
vm switch create -t manual -b bridge0 public

Usually, I enable the serial console on tmux, but in this case, I’ll skip that since Windows VMs need a graphical console. If the FreeBSD server is not on your local network, I suggest connecting via SSH with port forwarding for the necessary VNC ports (e.g., -L5900:127.0.0.1:5900) to connect to bhyve via SSH tunnel. Never expose the VNC port directly!

Creating the VM Template

Now, I created an “empty” VM using the “Windows” template with vm-bhyve and adjusted the configuration afterward:

vm create -t windows -s 1G -m 16G -c 4 vm115

I created a virtual disk of 1 GB because it will be replaced by the dataset sent from the current production server, so it’s a fictitious size. At that point, I deleted the disk0 image file:

rm /zroot/VMs/vm115/disk0.img

Modifying the VM Configuration

Next, I modified the VM configuration as follows:

Changed the disk configuration to use a zvol, simplifying the send-receive operation from the original Proxmox host.
In some cases, older Windows installations may not support the nvme driver. If this happens, use ahci-hd instead, as after the conversion Windows may fail to boot and display a BSOD.
Changed the network adapter driver to virtio-net (since it was already installed on the Proxmox VM).
Set the network adapter MAC address to match the original VM.

After running vm configure vm115, the configuration should look something like this:

loader="uefi"
graphics="yes"
xhci_mouse="yes"
cpu="4"
memory="16G"

ahci_device_limit="8"

network0_type="virtio-net"
network0_switch="public"

disk0_type="nvme"
disk0_name="disk0"
disk0_dev="sparse-zvol"

utctime="no"
uuid="myUUID"
network0_mac="theProxmoxVirtualMachineMacAddress"

Copying the Virtual Disk

I took a snapshot of the virtual disks of the VMs on the original server and copied them to the target server:

zfs snapshot vmpool/vm-115-disk-0@toSend01
zfs send -Rcv vmpool/vm-115-disk-0@toSend01 | mbuffer -m 128M | ssh user@FreebsdHostIP "zfs receive -F zroot/VMs/vm115/disk0"

The VM’s disk will be transferred to the target host.

Converting BIOS to UEFI

At this point, bhyve expects a UEFI VM, but the VM we just copied is BIOS-based. In other words, it won’t boot as is.

Luckily, Microsoft provides a tool to convert partitions and change the server from BIOS to UEFI. Of course, I would only use such a tool on a snapshot, and while I could do it directly on the source server, I prefer minimizing downtime and avoiding touching the original server. I want to perform the conversion directly on the FreeBSD host, which also makes disaster recovery easier, allowing me to manage everything on FreeBSD without first restoring the VM on a Proxmox server.

The first step is to download a Windows installation ISO (in my case, Windows Server). I downloaded the ISO from here: Windows Server Evaluation Center.

Once ready, launch the ISO as if you were installing Windows on the VM:

vm install vm115 SERVER_EVAL_x64FRE_en-us.iso

Running the Windows Repair Tool

Now the VM will boot. Connect via VNC (if using VNC tunneling, connect to 127.0.0.1:5900) to proceed. The ISO will prompt you to press a key to boot from the CDROM—go ahead and do that.

Proceed as if you were repairing your installation ("Repair your computer"), then select Troubleshoot -> Command Prompt.

Once at the prompt, run a validation check first (to avoid making changes if the disk has any unusual configurations):

mbr2gpt /validate /disk:0

If the validation is successful, proceed with the actual conversion:

mbr2gpt /convert /disk:0

Windows will convert the partition and indicate success at the end.

Booting the VM

Close the command prompt and shut down the VM. Everything should be ready now—easy peasy.

Boot the VM with:

vm start vm115

Connecting via VNC to the VM will bring up the Windows boot screen. In case of BSOD, change the nvmedriver to ahci-hdand it should be working. If the snapshot was taken while the VM was powered on, it may perform a disk check. If everything went as planned, you should see the Windows login screen.

Additionally, Windows will likely require reactivation of the VM, as it will detect hardware changes.

Final Considerations

At this point, you need to decide whether to continue with this VM or resynchronize it with the original (e.g., if the contents have changed). In the first case, no further action is needed. In the second case, the proper procedure would be to power off the original VM, create a new snapshot, roll back the snapshot on the FreeBSD host, transfer it incrementally—a process I’ve described in a previous post—and then redo the conversion.

To configure the VM to boot automatically when the FreeBSD host boots, add the necessary settings to /etc/rc.conf, as described in a previous article.

From Proxmox to FreeBSD - Story of a Migration

Stefano Marinelli — Mon, 21 Oct 2024 07:33:00 +0200

I have been managing a client's server for several years. I inherited a setup (actually, partly done by me at the time, but following the directions of their internal administrator) based on Proxmox. Originally, the VM disks were qcow2 files, but over time and with Proxmox updates, I managed to create a ZFS pool and move them onto it. For backups, I continued to use Proxmox Backup Server (even though virtualized on bhyve) - a solution we've been using for several years.

The existing VMs/containers were:

An LXC container based on Debian, with Apache as a reverse proxy. The choice of Apache was mainly tied to specific configurations from the company that produces the underlying software. It has always done an excellent job.
A VM running OPNsense — the choice was related to direct use by the client, as it allows easy creation/modification of users for the internal VPN. Some users can access via VPN (OpenVPN) and perform specific operations, such as SSH connections. It has always worked well and is up to date.
Two VMs based on Rocky Linux 9 — both with components of the management software in Java, with the reverse proxy directing requests based on the requested site.
Two VMs based on Rocky Linux 8 — these also have parts of the management software and the databases. These machines are used both by providing specific URLs and by the other two as databases.

The load is not particularly high, and the machines have good performance. Suddenly, however, I received a notification: one of the NVMe drives died abruptly, and the server rebooted. ZFS did its job, and everything remained sufficiently secure, but since it's a leased server and already several years old, I spoke with the client and proposed getting more recent hardware and redoing the setup based on a FreeBSD host.

Unfortunately, I cannot change the setup of the VMs. In my opinion, there are no contraindications since the databases are PostgreSQL and the management applications are Java applications with Tomcat. However, the software house only certifies that type of setup, and considering they are the ones performing updates and fixes, it's appropriate to maintain the setup they suggest. It's a technically sound choice in my opinion, so I can't say anything negative.

Acquiring New Hardware

The first step was, of course, to get the new server. It took a few days because I requested ECC RAM (for a small price variation), and the time extended. As soon as the physical server was ready, I got to work.

I installed FreeBSD 14.1-RELEASE, root on ZFS, creating a mirror setup between the NVMe drives. The I/O is not particularly heavy in this setup, but I don't want to waste potential: even if I can reduce the length of an operation by a few seconds, I see no reason not to do it.

As the first step, I decided to manually create the bridge to which I will connect both the VMs and the reverse proxy, which will be installed inside a FreeBSD jail.

vm-bhyve, the tool I use to manage VMs, allows creating bridges, but I prefer to manage it manually and maintain more complete control over everything. I will also enable pf.

I decided to use zstd as the compression algorithm and disable atime:

zfs set compression=zstd zroot
zfs set atime=off zroot

I then modified the /etc/rc.conf file as follows:

cloned_interfaces="bridge0 lo1"
ifconfig_lo1_name="bastille0"
ifconfig_bridge0="inet 192.168.33.1 netmask 255.255.255.0"
gateway_enable="YES"
pf_enable="YES"
bastille_enable="YES"

I then updated FreeBSD by running the usual command:

freebsd-update fetch install

Configuring the Firewall

I configured the firewall with a basic setup:

ext_if="igb0"

set block-policy return
set skip on bridge0

table <jails> persist
nat on $ext_if from {192.168.33.0/24} to any -> ($ext_if)
nat on $ext_if from <jails> to any -> ($ext_if:0)

# nginx-proxy - to the jail
rdr on $ext_if inet proto tcp from any to PUBLIC_IP port = 80 -> 192.168.33.254 port 80
rdr on $ext_if inet proto tcp from any to PUBLIC_IP port = 443 -> 192.168.33.254 port 443

# opnsense
rdr on $ext_if inet proto tcp from any to PUBLIC_IP port = 1194 -> 192.168.33.253 port 1194
rdr on $ext_if inet proto udp from any to PUBLIC_IP port = 1194 -> 192.168.33.253 port 1194

rdr-anchor "rdr/*"

block in all
antispoof for $ext_if inet
pass in inet proto icmp
pass out quick keep state
pass in inet proto tcp from any to any port {http,https} flags S/SA keep state
pass in inet proto {tcp,udp} from any to any port 1194 flags S/SA keep state
# This will be closed at the end of the setup and will be allowed only via VPN
pass in inet proto tcp from any to any port ssh flags S/SA keep state

Once finished, I rebooted everything to load the new kernel. No problems.

Installing Necessary Tools

I then installed some useful packages:

pkg install tmux py311-zfs-autobackup mbuffer rsync vm-bhyve-devel edk2-bhyve grub2-bhyve bastille

I created the datasets for the jails and VMs:

zfs create zroot/bastille
zfs create zroot/VMs

I then started configuring everything. First, BastilleBSD: I modified /usr/local/etc/bastille/bastille.conf by adding:

## ZFS options
bastille_zfs_enable="YES"
bastille_zfs_zpool="zroot/bastille"

Then I enabled and configured vm-bhyve, enabling the serial console on tmux:

sysrc vm_enable="YES"
sysrc vm_dir="zfs:zroot/VMs"
vm init
cp /usr/local/share/examples/vm-bhyve/* /zroot/VMs/.templates/
vm switch create -t manual -b bridge0 public
vm set console=tmux

Now I bootstrapped FreeBSD 14.1-RELEASE on BastilleBSD:

bastille bootstrap 14.1-RELEASE update

Setting Up the Reverse Proxy Jail

I then started by creating the jail for the reverse proxy:

bastille create -B apache 14.1-RELEASE 192.168.33.254/24 bridge0

Once created, I had to modify the default gateway of the jail because by default it is set to that of the host. It was enough to set 192.168.33.1 in /usr/local/bastille/jails/apache/root/etc/rc.conf.

The configuration of Apache will not be described here as it is closely dependent on the setup, but this jail can reach (in bridge mode) all the VMs that will be placed on the same bridge.

Migrating the VMs

For migrating the VMs, I decided to proceed as follows:

Updating the operating systems of the original VMs - to have the latest version of the kernel and all userland. The VMs are not UEFI, so it will be necessary to have the exact names of the kernel and initrd as they will need to be specified in the configuration file of vm-bhyve.
Creating templates of the VMs with vm-bhyve.
Snapshotting the VM disks and initial transfer to the new server with the VMs still running.
Shutting down all the source VMs.
Creating a new snapshot and performing an incremental transfer.
Adjusting configurations and changing DNS.

I updated everything using dnf, and rebooted when there was also a kernel update. I took note of the versions to use, namely vmlinuz-4.18.0-513.18.1.el8_9.x86_64 and initramfs-4.18.0-513.18.1.el8_9.x86_64.img for Rocky Linux 8.10, and vmlinuz-5.14.0-362.18.1.el9_3.0.1.x86_64 and initramfs-5.14.0-362.18.1.el9_3.0.1.x86_64.img for Rocky Linux 9.4.

Migrating Rocky Linux 8.10 VMs

On the destination server, I created the VMs. I used the linux-zvol template, then modified the configurations:

vm create -t linux-zvol -s 1G -m 8G -c 4 vm100

I created a virtual disk of 1 GB because it will be replaced by the dataset sent from the current production server, so it's a fictitious size. At that point, I deleted the zvol:

zfs destroy zroot/VMs/vm100/disk0

I performed an initial snapshot of the source VMs:

zfs snapshot zfspool/vm-100-disk-0@Move01

And then I copied this first snapshot to the destination machine/VM:

zfs send -R zfspool/vm-100-disk-0@Move01 | mbuffer -s 128k -m 512M | ssh root@destMachine "zfs receive -F zroot/VMs/vm100/disk0"

At the end of the copy, you can do a test and try to boot. For the Rocky Linux 8.10 VMs, I had no problems because the /boot partition is in ext4. I then configured the VM as follows, running the command vm configure vm100:

loader="grub"
cpu="8"
memory="12G"
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="sparse-zvol"
uuid="the-uuid"
network0_mac="the-mac"
grub_run0="linux /vmlinuz-4.18.0-553.22.1.el8_10.x86_64 root=/dev/vda3"
grub_run1="initrd /initramfs-4.18.0-553.22.1.el8_10.x86_64.img"

All I needed to do is: pass to GRUB the name of the kernel and the initramfs. It is now possible to start the machine and do a test and connect to the console:

vm start vm100
vm console vm100

In theory, the machine should boot correctly. It will probably be necessary to change the network interface configurations - in my case, even keeping the virtio and the MAC address, it changed from ens18 to enp0s5 - but everything else should be fine.

At this point, if the source VM was turned off/not reachable or simply there is nothing new to synchronize, the migration is complete. Otherwise, it will be necessary to shut down the source VM, create a new snapshot, and transfer it. Let's start by shutting down the source VM and the destination one (vm stop vm100), roll back to the snapshot transferred to the destination server, create a new snapshot, and perform an incremental transfer:

zfs rollback zroot/VMs/vm100/disk0@Move01

On the source server:

zfs snapshot zfspool/vm-100-disk-0@Move02
zfs send -R -i zfspool/vm-100-disk-0@Move01 zfspool/vm-100-disk-0@Move02 | mbuffer -s 128k -m 512M | ssh root@destMachine "zfs receive -F zroot/VMs/vm100/disk0"

In this way, we have updated the destination VM and transferred only the differences, without altering the configuration of vm-bhyve.

Start the machine - if all goes well, the migration of this VM is finished.

Migrating Rocky Linux 9.4 VMs

For the Rocky Linux 9.4 VMs, the situation was more complex: the partitions were all - even /boot—in XFS. And, for some reason, the GRUB launched by bhyve cannot read from XFS. Therefore, I had to proceed differently.

I copied, using rsync, the /boot of the original VPS (in ZFS) to a directory on the FreeBSD server (specifically, on the VPS I ran the command: rsync -avhHPx --numeric-ids /boot root@FreeBSDHOST:/tmp/). In this way, I kept the files available.

I shut down the machine on the original server and copied its zvol to the new FreeBSD host using the same method as the others. At that point, I recreated the /boot partition of my VPS in ext3 - directly from the FreeBSD host:

pkg install fusefs-ext2
kldload fusefs
mkfs.ext3 /dev/zvol/zroot/VMs/vm104/disk0s1
fuse-ext2 -o force /dev/zvol/zroot/VMs/vm104/disk0s1 /mnt/
cd /mnt
rsync -avhHPx /tmp/boot/. .
umount /mnt

In this way, GRUB will be able to access the /boot partition to load the kernel and initramfs.

Here is the vm-bhyve configuration for this VM:

loader="grub"
cpu="8"
memory="12G"
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="sparse-zvol"
uuid="the-uuid"
network0_mac="the-mac"
grub_run0="linux /vmlinuz-5.14.0-427.37.1.el9_4.x86_64 root=/dev/vda3"
grub_run1="initrd /initramfs-5.14.0-427.37.1.el9_4.x86_64.img"

Launching the machine, however, it will hang during boot and, after a timeout, will ask for administrator credentials to enter a recovery shell. This is because the blkid of the /boot partition has changed, and the fstab of the VM still reports the data of the old XFS partition. In this case, I used the blkid command in the VM and copied the UUID of the new partition. Then modify the /etc/fstab file of the VM and put the new blkid, as well as changing "xfs" to "ext3". After a reboot, the system should start without problems, again with a network card to reconfigure.

This procedure worked correctly for all VMs. In this way, the storage remains on virtio-blk even if it would be optimal for the driver to be changed to nvme. To make this change, you will need to enter the VMs, create a file called /etc/dracut.conf.d/00-custom.conf:

add_drivers+=" nvme "

And regenerate the initramfs - in this way, the nvme driver will be supported at boot:

dracut --regenerate-all --force

It will now be enough to change the configurations of vm-bhyve—virtio-blk becomes nvme, and /dev/vda3 becomes /dev/nvme0n1p3, for example:

loader="grub"
cpu="8"
memory="12G"
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0"
disk0_dev="sparse-zvol"
uuid="the-uuid"
network0_mac="the-mac"
grub_run0="linux /vmlinuz-5.14.0-427.37.1.el9_4.x86_64 root=/dev/nvme0n1p3"
grub_run1="initrd /initramfs-5.14.0-427.37.1.el9_4.x86_64.img"

Migrating the OPNsense VM

For the VM with OPNsense, the procedure was even simpler. It was enough to create a VM of type freebsd-zvol and copy the disk as done for the others. In this case, I replicated the MAC address of the original virtualized network interface (Proxmox server) to ensure that the underlying FreeBSD recognizes it as the same. FreeBSD is less picky about these things.

The final vm-bhyve configuration file will be:

loader="bhyveload"
cpu="2"
memory="1G"
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="sparse-zvol"
uuid="my-uuid"
network0_mac="my-mac"

Configuring Automatic VM Startup

To ensure that the VMs all start at boot, just add them to /etc/rc.conf as follows, giving a 15-second delay between one VM and another:

[...]
vm_enable="YES"
vm_dir="zfs:zroot/VMs"
vm_list="vm100 vm101 [...] opnsense"
vm_delay="15"
[...]

Setting Up Backups

At this point, I configured external backups, every hour, similarly to how I described in a previous article. I also added a local snapshot every 15 minutes, always using zfs-autobackup. To do this, I created a new tag:

zfs set autobackup:localsnap=true zroot

Then I modified the /etc/crontab file, adding this line:

*/15    *       *       *       *       root    /usr/local/bin/zfs-autobackup localsnap --keep-source 15min3h,1h1d > /dev/null 2>&1

That is, it will perform a snapshot every 15 minutes and keep them for 3 hours, then keep one per hour for a day. In this way, in case of quick recovery due to a problem/error, I won't need to transfer the entire dataset/VM from the backup.

Conclusion

The migration is complete: after changing the DNS, the client performed some tests, and everything works properly. The setup has been active for over a week, and there have been no issues. The performance is excellent; I did not perform tests compared to the previous setup since the hardware of the new FreeBSD host is more powerful and modern, so it wouldn't make sense.

Apart from the manager, no user was informed of the change, and in the last week, no reports have been received. The VMs are stable, and the host's load is very low.

The operation actually took less than two hours in total - most of the time has been consumed by the send/receive operations - and gave excellent results. The alternative would have been to install Proxmox on a new host and move the VMs - I probably would have saved a few minutes, but now I can use bhyve, as well as the simplicity and power of the underlying FreeBSD.

I Solve Problems

Stefano Marinelli — Thu, 03 Oct 2024 08:53:00 +0200

Addendum: during the event, I was interviewed by Jason Tubnor for the BSD Now Podcast, where I provided further information about the talk and the BSD Cafe project. Here is the link to the episode

This is the text underlying my presentation at EuroBSDCon 2024, on 21 September 2024, in Dublin, Ireland and BSDCan 2025, 13 June 2025, Ottawa, Canada.

The slides can be downloaded here

The BSDCan 2025 video can be viewed here. This is more up-to-date than the EuroBSDCon one.

The EuroBSDCon 2024 video, not yet separated from the live stream, can be viewed here - At first, I was a bit tense, then I relaxed.

Happy reading!

EuroBSDCon Dublin - 21 September 2024 - Why (and how) we're migrating many of our servers from Linux to the BSDs

"I'm Stefano Marinelli, I solve problems."

I’m the founder and Barista of the BSD Cafe, a community of *BSD enthusiasts.

I work in my company, called Prodottoinrete - a container of ideas and solutions.

I’m passionate about technology and computing, and I’ve made my passion my profession. Every morning, when I sit in front of the computer, a new world opens up for me to explore, and I try to share this passion with my clients. Sometimes, I succeed.

I've been a Linux user since 1996, before I turned 17. Back then, I used Fidonet and would read about alternative operating systems. Curiosity got the best of me, and I bought a set of CDs with various Linux distributions at the first opportunity. I tried it, I liked it, but at the time, I didn’t find it advantageous, so I continued using it for secondary tasks while Windows remained my daily driver.

Things changed in late 1997. I decided to go deeper into Linux and, with the purchase of a new, more powerful computer, I realized that aside from gaming, Linux could be my everyday system. This came in handy when I started university in 1998, where the computer science department was very oriented towards Open Source solutions. I was one of the few students who already understood the concepts of Open Source and knew how to use Linux. I was also one of the few who wasn’t baffled when we found Solaris machines in the lab. After all, there were similarities.

Over the years, I became one of the administrators of that lab, which was almost entirely Linux-based.

In 2000, I was fortunate to have Professor Özalp Babaoğlu as my lecturer, which pushed me to explore other operating systems like the BSDs. However, all I had at the time was an old Compaq laptop (486/25 MHz and 4 MB RAM) and no fast internet connection. So, apart from some theoretical studies, I postponed hands-on experiments until I had better resources. By 2002, thanks to a broadband connection and a new computer, I began exploring BSD systems. I started with FreeBSD, largely thanks to its fantastic handbook. I asked my parents to buy a laser printer so I could "print academic material" - but really, I wanted to print all the documentation I could find, starting from the FreeBSD handbook. And it was incredibly helpful.

Before long, FreeBSD became my daily driver - entire nights spent compiling KDE while I slept a meter away from my laptop’s wildly spinning fans - but FreeBSD ran much, much better on that machine than Linux did. Unfortunately, OpenBSD was too slow to be usable with any graphical interface.

In 2003, my final thesis focused on virtualization on Open Source systems, NetBSD/Xen was one of the best and most efficient solutions I tested.

Shortly after, I found a job at a company that was just beginning to offer Linux-based solutions, mainly for web and mail servers, but I was criticized because I completed tasks that didn’t require constant interventions - this hurt the company’s billing. That’s when I set my future guidelines:

I would work for myself, following my own philosophy.
I would not be tied to any particular vendor. I love exploring and learning, so I would always study solutions in depth and recommend the one I thought best suited for the client.
I solve problems - I don’t sell boxes.
I would use and promote Open Source solutions whenever possible.
I would use a BSD whenever feasible and Linux where a BSD was not suitable.

In every technological decision, my priority is solving my clients’ specific problems, not selling a predefined solution. I solve problems.

I was often told that Open Source systems were “toys for universities” and that the real world ran on something else (mostly meaning Windows). I pressed on. In some cases, I offered to be paid only if the results were achieved, showing how much they’d save compared to traditional licensing costs. It worked, but...

They accepted Linux, albeit reluctantly, but rejected BSDs because they didn’t know them. In some cases, I managed to convince them. In others, sadly, I didn’t.

Result: I decided to use the BSDs where clients didn’t have direct access - like email servers or web hosting - while using Linux where they specifically requested it.

NetBSD/Xen as the virtualization base, with excellent longevity and reliability.
OpenBSD as network/firewall entry points.
FreeBSD (especially with the introduction of ZFS) for various services, including backup servers.

It worked. What surprised clients most was the stability and the reduced need for maintenance. They saw more uptime and were happy.

Problem: "If nothing is working, what am I paying you for? If everything’s working, what am I paying you for?" So, I selected clients and situations that understood that if everything works, it’s because of the work behind the scenes. It’s better to pay for everything to work than to pay to fix problems. The problem to solve, in this case, is not stopping the client’s work. And I solve problems.

I managed about 65% *BSD machines and 35% Linux. But as Linux’s popularity grew, so did client demand. It became necessary on multiple occasions to replace or implement Linux instead of a BSD, for specific requests.

At a certain point, Linux virtualization solutions matured, and many of my hosts transitioned from NetBSD/Xen to OpenNebula (often with MooseFS) and then to Proxmox (often with Ceph). This happened because my clients needed to manage their VM lifecycles autonomously, change configurations, migrate hosts, etc.

Proxmox showed excellent reliability and stability. The VMs were often Linux-based (especially if the client needed direct management) or FreeBSD, but in smaller quantities. My own infrastructure, however, remained primarily BSD-based.

Over time, I gradually started to reflect and 'take stock'.

No *BSD has ever caused me to lose data to the point of having to restore from backup. On Linux, I’ve lost data with ext4, XFS, and btrfs. The most catastrophic case was with XFS - just a few files, backed up and restored, but the client was highly demanding and didn’t take it well. The largest failure was with btrfs - after a reboot, a 50 TB filesystem (in mirror, for backups) simply stopped working. No more mounting possible. Data was lost, but I had further backups. The client was informed and understood the situation. Within a few days, the server was rebuilt from scratch on FreeBSD with ZFS - since then, I haven’t lost a single bit.

In 2018, I started introducing Docker and Podman to the developers with specific needs, especially to help them with their development. Many developers began incessantly asking for Docker, unfortunately some of them only wanted to bypass many limitations that traditional setups imposed. Unfortunately, in many cases, those "limitations" were just bad practices - running outdated versions of software and libraries, or not wanting to bother with keeping the stack updated. There are similar problems with other solutions, but at least I can block them. They're great for quick deployment and increased security, but they aren’t always the best choice. Plus, they’re not the only option, despite what many people think these days.

Linux has had major development over the past years, but this has shifted towards specific players’ interests (mainly cloud providers) rather than technical reasons. Maybe not in the kernel, but many Linux distributions and communities now seem focused on the constant push to replace "the old with the new" with solid theoretical reasons but sometimes seemingly without practical benefit.

For me, computing should solve problems and provide opportunities to those who use it. I solve problems. Every change or variation will solve one problem but create new ones. It’s crucial to be mindful not to create worse problems than the ones you’re solving, and today’s enterprise world often seems to overlook that.

It’s common to see software distributed solely via Docker Compose files. Sometimes I use it as an installation tutorial, but I realize they’re just specific pieces precariously glued together (patched files, specific dependency versions, or nothing works).

The trend is to rush, to simplify deployments as much as possible, sweeping structural problems under the rug. The goal is to "innovate", not necessarily improve - just as long as it’s "new" or "how everyone does it, nowadays".

A massive business has grown around Linux: certifications, training, pentesting, certified platforms - the community is losing decision-making power. This is a far cry from the early days.

I solve problems. And these fast-changing technologies risk creating more problems than they solve, at least in certain situations. The workloads I manage often stay up for years, requiring a more stable, upgradable, and consistent approach.

If a client’s problem is to have an e-commerce site, they don’t really care if it runs on Docker, Podman, a FreeBSD jail, or a Raspberry Pi cluster - as long as their problem is solved. In fact, clients are happy when their solution is stable, upgradable, and secure.

There isn’t a single solution to all problems, but many solutions for each problem. My job is to give clients the best solution to solve their specific problem, not the most fashionable one.

That’s why I decided, several years ago, to reverse the proportion and implement and the BSDs for all possible workloads.

Each *BSD has its characteristics and target audience. Sometimes these targets overlap, but it’s generally not difficult to choose the most appropriate solution.

The goal of the migration was to create stable, coherent, upgradable, and secure systems.

By implementing an OpenBSD system, I often don’t need to install any additional packages. When it’s time to upgrade, it’s simple and secure. When a vulnerability emerges, I’m often fortunate to read "OpenBSD is excluded from this issue because it eliminated this risk X years ago..."

By implementing a NetBSD system, I know I’m installing a system that isn’t in a rush to release new versions and will likely run for years with only a few package updates and security patches, when necessary. And it’s quite rare for a patch to be required.

By implementing a FreeBSD system, I know I’ll have ZFS at my disposal, a fast and efficient hypervisor like bhyve, and a native, mature jail system that will ensure service and setup separation coherently and precisely. Not as “add-ons” but built into the system itself.

Moving to the BSDs means, in my experience, reaching systems that are more stable, more easily upgradable, and more consistent in their parts. They don’t chase the hype of the moment, much like the early days of Linux. In the case of FreeBSD, this also means moving to native ZFS and boot environments, giving me greater peace of mind when it comes to upgrades.

The initial strategy was to migrate what would soon need updates anyway, what would be moved (thus requiring a new setup), and what was causing problems and deserved a deeper dive.

First, I decided to migrate to FreeBSD the hypervisors not directly accessed by clients, especially on leased servers. The approach was to create a twin machine (to have an objective performance and stability comparison - it wouldn’t make sense to compare it to a different machine), install FreeBSD, bridge it with the production server, install vm-bhyve, and start copying VMs from Proxmox, reconfiguring the main parameters in the configuration file. In some cases, I already used ZFS on Proxmox, so the first part of the migration was a simple zfs-send/receive. In other cases (e.g., when using Ceph), I made an intermediate move by live-migrating the storage to ZFS and then proceeding as in the first case. The first noticeable effect was a reduction in resources used by the host to handle VM traffic - as expected, only FreeBSD’s basic processes and bhyve were running - but at the same time, there was a significant increase in I/O performance, further enhanced by switching the virtual disk driver from virtio to NVMe. This allowed for in-depth testing and revealed that FreeBSD suffers less under heavy I/O (the VMs on Linux tended to block their I/O, an effect I didn’t notice on FreeBSD) and showed significantly lower loads. In short, it handled the load better.

As an experiment, I decided to migrate two hosts (each with about 10 VMs) of a client - where I had full control—without telling them, over a weekend. By Tuesday, they called me, concerned: they had noticed a massive performance boost and were worried I had upgraded their hardware without approval, thinking that, "given the performance boost", it would cost them a lot. After explaining what I had done, they asked me to run further tests and progressively continue migrating everything. 20 hosts, all based on (sometimes slightly older) versions of Proxmox. And so I did, but taking advantage of their open-mindedness, I went a step further.

Many of their VMs handled simple workloads - PHP websites, or Java-based management systems (running on Tomcat), device monitoring software for industrial machinery, etc. The decision to use VMs had been made by their highly competent internal IT staff to separate environments and dependencies. It was a perfect use case for jails. I decided to try, VM by VM, to replicate the setups inside FreeBSD jails. In some cases, for convenience, I managed to run everything directly in Linux jails (with Linuxulator); in others, it was impossible, so I recreated the setups in individual jails. They immediately noticed an effect: faster operations. It didn’t surprise me. We avoided double buffering (VM and OS), so all the saved RAM could be used by ZFS for its cache and by the host to run other services. Some VMs remained as VMs (e.g., Zimbra). By the end of the operation, I had drastically reduced the number of VMs, replaced by jails, and consequently, the number of hosts. From 20 down to 11 - with a significant monthly cost saving.

The road was now clear, and I progressively continued down this path. One of the most interesting anecdotes: a client told me that they used to start an operation before taking a coffee break, around 15 minutes, to find the task almost done by the time they returned. After the migration, they shared that they launched the process, grabbed their things, and the task was already complete. An estimated reduction from about 18 minutes to 6 minutes on average. I didn’t investigate too much, but I suspect a combination of factors, with the predominant one being bhyve’s NVMe driver.

The main challenge I often face is ideological. Some people are used to thinking that the ideal solution is X - and believe that X is the only solution for their problems. Often, X is the hype of the moment (a few years ago, I fought to convince people that VMware wasn’t necessary and that Proxmox would be a great solution; today, Proxmox is on everyone’s lips - but it’s not the only solution). Often, X is a "cloud" cluster with Kubernetes - running WordPress on it. Even for hosting a law firm’s website, which will be updated every five years.

When I ask, "Okay, but why? Who will manage it? Where will your data really be, and who will safeguard it?", I get blank faces. They hadn’t considered these questions. No one had even mentioned them. "But everyone I spoke to proposed this type of solution...". It’s like at the beginning with Windows, when I proposed *BSD or Linux. Or later with VMware, when I proposed Proxmox. Or now with Kubernetes, when I propose the BSDs.

But the simplest solutions are the easiest to maintain and manage over time. My experience has taught me that setting something up is often the easiest part. The hardest part is returning to it after 1, 5, 10 years. Keeping it running, updating it, stabilizing it. For many, IT isn’t their business, but a tool to achieve their goals. A Kubernetes cluster is fantastic, but it requires maintenance. Or it’s external - so it’s no longer ours. We’ve lost control of the data. For many, it’s unnecessary to complicate things. And with every additional layer, we’re creating more problems.

No BSD system has ever surprised me during an update or a simple reboot. I’ve never encountered, for example, a network interface renaming from enx3e3300c9e14e to enp10s0f0np0* on Linux, effectively locking me out of a server. ix0 will remain ix0.

I’ve never had to recompile ZFS on FreeBSD, only to find that the module wouldn’t load, blocking the filesystem from mounting after reboot.

Many of the developers I work with have embraced the challenge. Most of them are passionate about technology, and learning a new operating method has been very interesting. Almost all of them, after experiments, were positive and, in fact, began explicitly requesting "jails" instead of Docker hosts. They started using BastilleBSD to clone "template" jails and deploy them. They learned to access ZFS automatic snapshots and recover lost files. They learned to manage the resources at their disposal without repeating the usual mantra of "we need mooooar powaaaar!" every time there’s a problem, a slowdown, or a storage overload. They’ve returned to trying to understand what’s happening, rather than just assembling pieces, libraries, and containers without considering the effects.

Others, however, struggled, but remained positive nonetheless. And that’s okay. I solve problems and can’t force my solutions on everyone.

In other cases, the challenge wasn’t technical but "commercial". Often, decision-makers have little to no technical knowledge. Linux sells well. "Cloud" sells even better. A "NetBSD-based solution", unfortunately, has less commercial appeal today. So, they want what they can sell, without focusing too much on the advantages of alternative solutions.

Linux today is subject to many compliance requirements - I’m often asked which version of OpenSSH I’m running - and they complain when the version (the latest from OpenBSD) isn’t considered "secure" because it doesn’t match their procedures (e.g., OpenSSH_9.2p1 Debian-2+deb12u3). They don’t understand when I explain that the "Debian" part refers to the Linux distribution, not a release. Those who prepare these documents are often, sadly, unaware of what they’re asking for. They just have a checklist.

The transition is ongoing, and as I see opportunities, I’m migrating from Linux to the BSDs whenever possible. Today, I can say that 78% of the hypervisors I manage run on FreeBSD, and 66% of the workloads (VPS, jails, hosts, etc.) are running on one of the BSDs - including solutions like OPNsense. None of the clients have experienced major issues. No one has complained about performance or reliability. All feedback has been positive. Many appreciate moving away from IT monocultures, especially when problems arise - following the recent severe SSH vulnerability, many clients contacted me, worried. It was nice to tell some of them that the exposed SSH service was running OpenBSD’s version, and that OpenBSD wasn’t affected as they had developed a secure mechanism back in 2001. They appreciated it. They want more setups based on OpenBSD.

I'm Stefano Marinelli, I solve problems. And I love solving problems using BSD systems.

Moving an entire FreeBSD installation to a new host or VM in a few easy steps

Stefano Marinelli — Mon, 16 Sep 2024 09:41:00 +0200

FreeBSD, especially when installed on ZFS, is incredibly simple to manage, back up, and move to a different system.

I often find myself having to move an entire system from one host to another, from a physical host to a VM, or vice versa, from a VM to a physical host.

By following the approach of keeping the operating system clean and running all services within jails, this operation is usually quite simple: install the operating system on the new host, perform the few necessary configurations (like pf and network interfaces), transfer the jail datasets, restart the jails on the new host, and update the DNS.

However, sometimes I need to move the entire host, including the original operating system. This article will describe this operation when the FreeBSD system uses ZFS as its root file system. This process isn't much more difficult, but there are a few details to keep in mind. Below is a breakdown of the two bootloader scenarios:

UEFI Bootloader
BIOS Bootloader (non-UEFI)

UEFI Bootloader

Start from an ISO or image of mfsbsd. Partition the disk according to the layout of the source system. If the source system is a standard FreeBSD (ZFS) installation, partition the destination disk (assumed to be da0) as follows. WARNING: The first command will destroy any existing partition table on the disk.

gpart destroy -F /dev/da0
gpart create -s gpt da0
gpart add -t efi -s 200M -l efiboot0 da0
newfs_msdos -F 32 -c 1 /dev/da0p1
mount -t msdosfs /dev/da0p1 /mnt
mkdir -p /mnt/EFI/BOOT
cp /boot/loader.efi /mnt/EFI/BOOT/BOOTX64.efi
umount /mnt

Create additional partitions for swap and ZFS:

gpart add -a 1m -t freebsd-swap -s 2g -l swap0 da0
gpart add -a 1m -t freebsd-zfs -l zfs0 da0

Create the ZFS pool:

zpool create -O compression=zstd -O atime=off zroot /dev/gpt/zfs0

On the source system, create a snapshot of all datasets:

zfs snapshot -r zroot@snap01

Important: You will need to connect to the source system using a user with privileges over the entire filesystem. For simplicity, I'll use root, but it's not recommended to leave root accessible via SSH. Limit access to private keys only. To allow root login over SSH, edit /etc/ssh/sshd_config and set PermitRootLogin yes. After moving the server, remember to disable root access.
If mbuffer is installed on the source system, it can help improve the performance of the transfer. If not, remove it from the next pipe command, which should be run from the destination system. The -s will set the block-size, the -m will set the buffer size (128MB, here):

ssh root@sourceip "zfs send -RLv zroot@snap01 | mbuffer -s 128k -m 128M" | zfs receive -F zroot

You have two options now:
If the source server was idle and no further synchronization is required, proceed to the next step.
If the transfer took a long time and services on the source server need to be stopped (e.g., databases), stop what you can, take another snapshot (e.g., zfs snapshot -r zroot@snap02), and synchronize the differences by running another incremental send-receive from the destination server:

ssh root@sourceip "zfs send -RLv -i zroot@snap01 zroot@snap02 | mbuffer -s 128k -m 128M" | zfs receive -F zroot

Set the boot filesystem:

zpool set bootfs=zroot/ROOT/default zroot

BIOS Bootloader

Start from an ISO or image of mfsbsd. Partition the disk according to the layout of the source system. If the source system is a standard FreeBSD (ZFS) installation, partition the destination disk (assumed to be da0) as follows. WARNING: The first command will destroy any existing partition table on the disk.

gpart destroy -F /dev/da0
gpart create -s gpt da0

Create the necessary partitions:

gpart add -a 1m -t freebsd-boot -s 512k -l boot da0
gpart add -a 1m -t freebsd-swap -s 2g -l swap0 da0
gpart add -a 1m -t freebsd-zfs -l zfs0 da0

Create the ZFS pool:

zpool create -O compression=zstd -O atime=off zroot /dev/gpt/zfs0

On the source system, create a snapshot of all datasets:

zfs snapshot -r zroot@snap01

Important: You will need to connect to the source system using a user with privileges over the entire filesystem. For simplicity, I'll use root, but it's not recommended to leave root accessible via SSH. Limit access to private keys only. To allow root login over SSH, edit /etc/ssh/sshd_config and set PermitRootLogin yes.
If mbuffer is installed on the source system, it can help improve the performance of the transfer. If not, remove it from the next pipe command, which should be run from the destination system:

ssh root@sourceip "zfs send -RLv zroot@snap01 | mbuffer -s 128k -m 128M" | zfs receive -F zroot

You have two options now:
If the source server was idle and no further synchronization is required, proceed to the next step.
If the transfer took a long time and services on the source server need to be stopped (e.g., databases), stop what you can, take another snapshot (e.g., zfs snapshot -r zroot@snap02), and synchronize the differences by running another incremental send-receive from the destination server:

ssh root@sourceip "zfs send -RLv -i zroot@snap01 zroot@snap02 | mbuffer -s 128k -m 128M" | zfs receive -F zroot

Install the bootloader on the destination server’s disk:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

Set the boot filesystem:

zpool set bootfs=zroot/ROOT/default zroot

At this point, everything should be ready for boot. Reboot and verify that the various mountpoints (e.g., swap, etc.) and network interface settings are correct. If everything works fine, the server will be an exact copy of the original.

Moving or duplicating an entire FreeBSD system is simple, fast, and reliable. The ability to use ZFS incremental replication is an excellent method for minimizing downtime, especially for large systems. The initial copy will happen "live" while the source system continues to run without issues. Subsequent incrementals will be much faster, making the switch-over process seamless and efficient.

Automating ZFS Snapshots for Peace of Mind

Stefano Marinelli — Wed, 21 Aug 2024 08:41:00 +0200

One feature I couldn't live without anymore is snapshots. As system administrators, we often find ourselves in situations where we've made a mistake, need to revert to a previous state, or need access to a log that has been rotated and disappeared. Since I started using ZFS, all of this has become incredibly simple, and I feel much more at ease when making any modifications.

However, since I don't always remember to create a manual snapshot before starting to work, I use an automatic snapshot system. For this type of snapshot, I use the excellent zfs-autobackup tool - which I also use for backups. The goal is to have a single, flexible, and configurable tool without having to learn different syntaxes.

Why zfs-autobackup?

zfs-autobackup has several advantages that make it perfect (or nearly so) for my purpose:

It operates based on "tags" set on individual datasets. I don't have to specify the dataset; I just assign a specific tag (of my choice) to the datasets, and zfs-autobackup will operate on those datasets, transparently with respect to others. This ensures it will work even on datasets in different zpools.
It's extremely flexible in management. For example, by setting the correct tag to "zroot", it will automatically manage all underlying datasets. However, it's possible to exclude some for more granular snapshot management.
It works well on both FreeBSD and Linux - I use it with satisfaction on both platforms.
Different tags allow for different levels of data retention and operation. For example, the "mylocalsnap" tag will be for local snapshots, while "backup_offsite" will be for backups that will be copied off-site. The two tags (and related snapshots) will be independent, even though they operate on the same datasets.

Installation

On FreeBSD, installation is straightforward, as there's a ready-made package:

pkg install py311-zfs-autobackup

On Linux, it will depend on the specific distribution. Being in Python, it will always be possible to install it using pip.

Configuration

Once installed, you just need to assign the tag to the dataset. For example:

zfs set autobackup:mylocalsnap=true zroot

This will set a tag called "mylocalsnap" on zroot and underlying datasets, i.e., on the entire main file system of FreeBSD.

Usage

Now, you just need to run zfs-autobackup, specifying both the tag and the snapshot retention criteria:

/usr/local/bin/zfs-autobackup mylocalsnap --keep-source 5min1h,1h1d

In this case, it will take a (recursive) snapshot of the datasets that have the "mylocalsnap" tag set to true, keeping one snapshot every 5 minutes for an hour, and one every hour for a day.

On subsequent executions of zfs-autobackup, snapshots that don't meet the previous retention criteria will be deleted.

After running this command, here's the result:

root@fbsnap:~ # zfs list -t snapshot
NAME                                            USED  AVAIL  REFER  MOUNTPOINT
zroot@mylocalsnap-20240820150115                  0B      -    96K  -
zroot/ROOT@mylocalsnap-20240820150115             0B      -    96K  -
zroot/ROOT/default@mylocalsnap-20240820150115     0B      -  1.00G  -
zroot/home@mylocalsnap-20240820150115             0B      -    96K  -
zroot/tmp@mylocalsnap-20240820150115              0B      -   104K  -
zroot/usr@mylocalsnap-20240820150115              0B      -    96K  -
zroot/usr/ports@mylocalsnap-20240820150115        0B      -    96K  -
zroot/usr/src@mylocalsnap-20240820150115          0B      -    96K  -
zroot/var@mylocalsnap-20240820150115              0B      -    96K  -
zroot/var/audit@mylocalsnap-20240820150115        0B      -    96K  -
zroot/var/crash@mylocalsnap-20240820150115        0B      -    96K  -
zroot/var/log@mylocalsnap-20240820150115          0B      -   144K  -
zroot/var/mail@mylocalsnap-20240820150115         0B      -    96K  -
zroot/var/tmp@mylocalsnap-20240820150115          0B      -    96K  -

As you can see, snapshots have been taken of all datasets with the tag set.

Automation

To automate the process, simply modify the /etc/crontab file and add a line like this:

*/5     *       *       *       *       root    /usr/local/bin/zfs-autobackup mylocalsnap --keep-source 5min1h,1h1d

Now, wait a few minutes and check again:

root@fbsnap:~ # zfs list -t snapshot
NAME                                            USED  AVAIL  REFER  MOUNTPOINT
zroot@mylocalsnap-20240820150115                  0B      -    96K  -
zroot/ROOT@mylocalsnap-20240820150115             0B      -    96K  -
zroot/ROOT/default@mylocalsnap-20240820150115   212K      -  1.00G  -
zroot/ROOT/default@mylocalsnap-20240820151000   128K      -  1.00G  -
zroot/home@mylocalsnap-20240820150115             0B      -    96K  -
zroot/tmp@mylocalsnap-20240820150115             72K      -   104K  -
zroot/tmp@mylocalsnap-20240820151000              0B      -   104K  -
zroot/usr@mylocalsnap-20240820150115              0B      -    96K  -
zroot/usr/ports@mylocalsnap-20240820150115        0B      -    96K  -
zroot/usr/src@mylocalsnap-20240820150115          0B      -    96K  -
zroot/var@mylocalsnap-20240820150115              0B      -    96K  -
zroot/var/audit@mylocalsnap-20240820150115        0B      -    96K  -
zroot/var/crash@mylocalsnap-20240820150115        0B      -    96K  -
zroot/var/log@mylocalsnap-20240820150115         64K      -   144K  -
zroot/var/log@mylocalsnap-20240820151000         60K      -   144K  -
zroot/var/mail@mylocalsnap-20240820150115         0B      -    96K  -
zroot/var/tmp@mylocalsnap-20240820150115         64K      -    96K  -
zroot/var/tmp@mylocalsnap-20240820151000          0B      -    96K  -

If everything went as it should, you'll notice that unmodified datasets won't have new snapshots, while those that have been modified since the previous manual execution (e.g., zroot/var/log) will contain both the previous snapshot and the automatic one.

Recovering Files from Snapshots

There are various ways to recover a file from the previous snapshot. One option is to restore the entire snapshot to the current dataset, but this may not be the best option as it will perform a complete restore.

Another alternative is to go to the hidden snapshot directory. For example:

root@fbsnap:~ # cd /var/log/.zfs/snapshot/mylocalsnap-20240820151000/
root@fbsnap:/var/log/.zfs/snapshot/mylocalsnap-20240820151000 # ls -l
total 39
-rw-------  1 root wheel     872 Aug 20 15:06 auth.log
-rw-r--r--  1 root wheel   79079 Aug 20 13:19 bsdinstall_log
-rw-------  1 root wheel    5401 Aug 20 15:10 cron
-rw-r--r--  1 root wheel      63 Aug 20 13:20 daemon.log
-rw-------  1 root wheel      63 Aug 20 13:20 debug.log
-rw-r--r--  1 root wheel      63 Aug 20 13:20 devd.log
-rw-r--r--  1 root wheel      63 Aug 20 13:20 lpd-errs
-rw-r-----  1 root wheel      63 Aug 20 13:20 maillog
-rw-r--r--  1 root wheel   17043 Aug 20 15:00 messages
-rw-r-----  1 root network    63 Aug 20 13:20 ppp.log
-rw-------  1 root wheel      63 Aug 20 13:20 security
-rw-r--r--  1 root wheel     197 Aug 20 15:00 utx.lastlogin
-rw-r--r--  1 root wheel     187 Aug 20 15:00 utx.log
-rw-------  1 root wheel      63 Aug 20 13:20 xferlog

From here, you can read and recover any file present in the individual snapshot. The snapshots are read-only, so you won't be able to write to them.

Creating a Writable Copy of a Snapshot

Should you need a read-write copy of a specific snapshot, you can use the zfs clone command:

root@fbsnap:~ # zfs clone zroot/var/log@mylocalsnap-20240820151000 zroot/recover
root@fbsnap:~ # ls -l /zroot/recover/
total 39
-rw-------  1 root wheel     872 Aug 20 15:06 auth.log
-rw-r--r--  1 root wheel   79079 Aug 20 13:19 bsdinstall_log
-rw-------  1 root wheel    5401 Aug 20 15:10 cron
-rw-r--r--  1 root wheel      63 Aug 20 13:20 daemon.log
-rw-------  1 root wheel      63 Aug 20 13:20 debug.log
-rw-r--r--  1 root wheel      63 Aug 20 13:20 devd.log
-rw-r--r--  1 root wheel      63 Aug 20 13:20 lpd-errs
-rw-r-----  1 root wheel      63 Aug 20 13:20 maillog
-rw-r--r--  1 root wheel   17043 Aug 20 15:00 messages
-rw-r-----  1 root network    63 Aug 20 13:20 ppp.log
-rw-------  1 root wheel      63 Aug 20 13:20 security
-rw-r--r--  1 root wheel     197 Aug 20 15:00 utx.lastlogin
-rw-r--r--  1 root wheel     187 Aug 20 15:00 utx.log
-rw-------  1 root wheel      63 Aug 20 13:20 xferlog

This creates a new dataset zroot/recover that is a writable copy of the snapshot. You can now modify these files as needed, without affecting the original snapshot or the live filesystem.

Cleaning Up

Sometimes you may want to delete all snapshots generated by zfs-autobackup. There are various ways, but the quickest can be to use a simple pipe:

zfs list -t snapshot -o name | grep -i mylocalsnap | xargs -n 1 zfs destroy -vr

By implementing automatic ZFS snapshots, you can work with peace of mind, knowing that you can always revert changes or recover lost files. This setup provides an excellent balance between data protection and system performance.

From Cloud Chaos to FreeBSD Efficiency

Stefano Marinelli — Thu, 04 Jul 2024 08:41:00 +0200

Introduction

A few months ago, a client asked me to take care of their Kubernetes cluster (hosted on AWS and GCP). In their opinion, the costs were exorbitantly high for relatively simple and lean websites. Sure, they had many visits, but nothing too excessive development-wise.

I kindly declined. Unfortunately, their situation is all too common these days: they hired developers accustomed to working that way, convinced that a system administrator is now unnecessary because "the cloud has infinite potential." They were used to considering optimization as secondary because "we have infinite power" (and this is already a spoiler for the ending).

Being open to dialogue and new experiences, they asked for my opinion on the matter. We talked for a while, and I explained that, in my view, for the type of setup they had (standard, with various replicas and variants, but primarily based on two platforms), it didn't make sense. I saw it as complicating things. An over-engineering of something simple. Like taking a cruise ship to cross a river.

They then asked me to create something simple that would serve as a development server and for backups, to understand what kind of solution I had in mind.

The Solution

So, I started building everything. I began with FreeBSD 13.2-RELEASE, but in the meantime, 14.0-RELEASE came out, so that’s the version I delivered.

I installed the operating system on a physical server, leased from one of the main European providers. Benefiting from one of their auctions (good deals can be found on weekends), they found a sufficiently powerful machine, with 128GB of RAM, 2 NVMe drives of 1TB each, and two spinning disks of 2TB each for less than 100 euros per month. They also took another, less powerful one for additional backups and to back up the first one.

Implementation

I decided to keep the host as clean as possible and concentrated the services in jails (managed by BastilleBSD) and VMs. The machine was divided as follows:

A series of bridges - to be used for different projects. Jails of the same project and/or type use the same bridge and can communicate with each other, sharing some resources (MariaDB, etc.).
A bhyve VM with Alpine Linux - in my opinion, the best distribution for running Docker containers. Do we really need systemd just to launch Docker? They mainly use it as a pre-production test bench, connected via VPN to their company LAN. It is the core of their "online" development, i.e., outside their computers. It has 32GB of RAM, 200GB of disk (obviously bhyve is configured with NVMe drivers), and 4 cores assigned.
A VNET jail with a reverse proxy (nginx) - they know how to modify virtual hosts and generate certificates with certbot, pointing to the underlying jails.
A series of "empty" VNET jails, to be cloned, for each type of setup (they mainly have CMS based on WordPress and Laravel, so with all dependencies inside - nginx, php, redis, etc. except the databases).
A VNET jail with MariaDB installed, to be cloned, to be attached to different projects as needed.
zfs-autobackup performs local snapshots, keeping: one every 15 minutes for 3 hours, one per hour for 24 hours, one per day for 3 days.

Backups are also performed using zfs-autobackup and, in case of disaster recovery in rapid times, a zfs-send (and corresponding zfs-receive) every 10 minutes on another machine (the other, smaller one, also taken at auction), with the same bridges, firewall rules, BastilleBSD, and bhyve installed - ready to start in case of disaster. Being a test server, we didn't consider to implement a proper HA - at the moment, it wouldn't make sense.

They also have another job with zfs-autobackup that performs an additional backup on a server (Debian in their offices). Safe data, in my opinion, are those in storage under your b...ench.

I delivered everything to them and gave a brief course to the more experienced devs on how to manage things. No explanation on the Alpine Linux VM, but I showed them the jails, how to clone, configure, and manage them.

Real-world Testing

I didn't hear from them anymore. After a few weeks, one of the devs contacted me urgently because a junior unfortunately made a mistake and deleted an entire project from one of the jails. I explained that the local snapshots were restorable with a command, and he was thrilled. He restored both the development jail and the one with the database made two minutes before the "mishap" and they restarted immediately.

I realized that this event would change some of their procedures and criteria.

I hadn't heard from anyone for months. This morning, I received a call from their manager, whom I hadn't heard from since the beginning, and he told me how things had been going these months.

Lessons Learned

First, this person has good communication and commercial skills but little technical background. He is open-minded and tends to study carefully what is proposed to him. He doesn't discard any solution a priori, without having touched its pros and cons.

They had leased servers with cPanel and were inserting their content inside them. The devs who arrived a few years ago suggested making a technological transition, eliminating these "obsolete" servers and "outdated" methodologies, pushing everything to the cloud and containerizing everything. When we first talked, he told me how they were "lucky to make that transition because their load had increased enormously and the old servers probably wouldn't have handled the load", instead autoscaling saved them. I had some reservations about autoscaling without particular controls, but clearly, I cannot impose my choices on others.

To cut a long story short: seeing what happened with that junior dev's mistake (and the simplicity with which it was possible to restart immediately), they decided to increase the use of FreeBSD jails and reduce, at least on secondary loads, the use of their Cloud managed with Kubernetes. As they transitioned to jails, however, they noticed some slowdowns. These slowdowns worsened day by day. According to the devs, it would have been appropriate to go back to having, again, autoscaling ("we need moar powaaaaar!!!") but, fortunately, their boss decided to investigate carefully. They realized that these workloads (based on Laravel) were storing sessions on files. Over time, these millions of files (several gigabytes per day) slowed everything down because, for specific operations, Laravel scanned the entire directory. In other words, on the "cloud," they needed much more power than necessary (and much more disk space, but that was cheaper) to carry this load, which was, in fact, unnecessary. After realizing this, they moved the sessions to Redis. Needless to say, everything became extremely faster, even compared to the previous setup on Kubernetes and autoscaling.

At that point, it was clear that one of the problems with their setup is (as often happens) poor optimization. Today, there's a tendency to rush, "throw in" functions, features, libraries, plugins, etc. without considering the interactions and consequences. If it works, it's fine. Even if it increases computational complexity exponentially just to, for example, change the color of an icon (absurd example, but to give an idea).

They then started moving even the main Laravel workloads (thanks to the optimization implemented). At this point, they began moving some of the WordPress sites even though they were extremely concerned. In the cluster, every day, at fairly irregular intervals, the load would rise and everything would slow down until autoscaling started scaling up to the imposed limits. CPU at 100% on all containers, and the devs noticed that the load came from a series of "php" processes. Recreating the containers helped for some minutes, but did not solve the problem.

To their great surprise, all this did not happen on the FreeBSD jails. The load was significantly lower, without any of these spikes. Satisfied, they decided to use this as their final setup. One of the devs, however, wanted to get to the bottom of it and decided to run a test: he moved some of these WordPress sites to the Alpine VM, on Docker. At that point, the spikes resumed, saturating the CPU of the Alpine machine.

Without going into details, they eventually realized that there was a vulnerability in one (or more) of the many plugins installed on the WordPress sites, which was being exploited to inject a process, probably a cryptominer. The name given to the process was "php" - so the devs, not being system experts, did not worry about understanding better whether it was really php or another process pretending to be it. On FreeBSD, all this did not happen because the injected executable could not run - there was no Linux compatibility activated on the server.

Until then, they considered these (expensive) spikes as organic and did not worry too much about them. Paying to have their friendly intruders mine.

Conclusion

They asked me to help, as much as possible, to move other services to FreeBSD. It won't be easy, probably we will need to use bhyve a lot, but they decided that this is the platform they want to focus on in the coming years.

Undoubtedly, this is a success story of FreeBSD and, indirectly, of correct and careful management of one's resources. Too often today, there is the superficial belief that the cloud, with its "infinite" resources, is the solution to all problems. And that Kubernetes is the best solution for everything. I, on the other hand, have always believed that there is the right tool for everything. You can hammer a nail with a screwdriver, but it's not the most suitable and efficient tool.

Today they spend about 1/10 of what they used to spend before, they have more control over their data and the tools they use. Undoubtedly, all this was also caused by poor optimization and control by those who manage the infrastructure, but the question is: how often do people decide that, in the end, it is okay to spend more (especially if it is someone else's money) rather than go crazy for hours behind such a situation? While having defined and limited resources (albeit elevated) poses different problems - but of optimization. And in the age of energy and resource savings, it might be wise to give more importance to optimization.

Abundance led to waste.

Enhancing FreeBSD Stability with ZFS Pool Checkpoints

Stefano Marinelli — Mon, 01 Jul 2024 09:41:00 +0200

ZFS offers many interesting features, and one of the most widely used is the ability to create and transfer snapshots of entire datasets, even recursively. This approach is useful for backups or maintaining a specific “point in time” for datasets. For example, on FreeBSD, automatic snapshots of the dataset containing the root file system have been taken with each system upgrade for several releases. This way, thanks to Boot Environments, if there are any problems, it is possible to reboot from a previous clone.

However, sometimes we might need something more. Local snapshots do not protect against the deletion of entire datasets or the activation of new features that could potentially cause problems or incompatibilities.

A very useful tool that I have successfully used for some time is the pool checkpoint feature. This feature, imported from Illumos to FreeBSD in 2018, allows creating a sort of snapshot of the entire pool, including features, metadata, etc.

The checkpoint is different from snapshots of individual datasets. It is not possible to have more than one checkpoint, and some operations like remove, attach, detach, split, and reguid will be impossible when a checkpoint exists. This also has a side effect: if there is a checkpoint, deleting a dataset will not release free space because the data will still be physically present in the storage thanks to the checkpoint.

Additionally, checkpoints are detected by the FreeBSD boot loader. When booting the system, the boot loader will offer the option to perform a "Rewind ZFS checkpoint" and boot from that point, effectively discarding everything that occurred after the checkpoint. This option can be particularly useful in emergencies or when you need to quickly undo recent changes.

Creating a checkpoint is very simple. Just use the command:

zpool checkpoint <pool>

The operation is usually quick. When a checkpoint is present, the command zpool status will show its details. For example:

pool: zroot
state: ONLINE
scan: scrub repaired 0B in 00:00:12 with 0 errors on Fri May 17 13:27:14 2024
checkpoint: created Sun Jun 30 12:30:51 2024, consumes 1.34M
config:

    NAME        STATE     READ WRITE CKSUM
    zroot       ONLINE       0     0     0
      ada1p4    ONLINE       0     0     0

errors: No known data errors

To delete the checkpoint, you can use the command:

zpool checkpoint -d <pool>

To rollback state to checkpoint and remove the checkpoint:

zpool import --rewind-to-checkpoint <pool>

To mount the pool read only (without rolling back the data):

zpool import --read-only=on --rewind-to-checkpoint <pool>

It is therefore possible to generate a checkpoint automatically via cron or manually when necessary, for example, before an operating system upgrade.

For more technical details, I suggest reading this excellent article by Serapheim Dimitropoulos, published in the FreeBSD Journal in January 2019.

Proxmox vs FreeBSD: Which Virtualization Host Performs Better?

Stefano Marinelli — Mon, 10 Jun 2024 05:53:45 +0000

TL;DR: Skip to the Conclusion for a summary.

Preamble

I have always been passionate about virtualization and have consistently used it.

The first solution I installed on my infrastructures (and those of clients) was Xen on NetBSD, with great success. I then used Xen on Linux and, since 2012, OpenNebula, followed by Proxmox in 2013. Proxmox has always given me great satisfaction, and even today I consider it a valuable platform that I install gladly. I have also used other hypervisors like XCP-ng but less frequently, and in recent years, I have started to make extensive use of bhyve.

About two and a half years ago, we began a progressive process of migrating our servers (and those of our clients) from Linux to FreeBSD, using jails (when possible) or VMs on bhyve. In some cases, migrating setups from Proxmox to FreeBSD resulted in performance improvements, even with the same hardware. In some instances, I migrated VMs without notifying clients, and they contacted me a few days later to inquire if we had new hardware because they noticed better performance.

After years, I decided to conduct a test to determine if this was just a perception or if there was a technical basis behind it. Of course, this test has no scientific validity, and the results were obtained on specific hardware and at a specific time, so on different hardware, workload, and situations, the results could be entirely opposite. However, I tried to have as scientific and objective an approach as possible since I am comparing two solutions that I care about and use daily.

Hardware and Test Conditions

I often see comparative tests done on VMs from various providers. In my opinion, this comparison makes no sense because a VM from any provider shares its hardware with many other VMs, so the results will vary depending on the load of the "neighbors" and will never be reliable.

For this test, I decided to take a physical server with the following characteristics:

Intel Core i7-6700
2x SSD M.2 NVMe 512 GB
4x RAM 16384 MB DDR4
NIC 1 Gbit Intel I219-LM

The hardware is not recent, but still very widespread. On more recent hardware, the results might differ, but the test will be based on this configuration.

I installed Proxmox 8.2.2 starting from the Debian template of the provider and manually installed it following the instructions. I created a partition for Proxmox and left one partition free on each of the two NVME drives to create (at different times) the ZFS pool (in mirror) and the LVM on top of the Linux software raid.

After all the tests, I installed FreeBSD 14.1-RELEASE on ZFS on the same host, using bsdinstall from an mfsbsd image since the provider does not directly support installing FreeBSD from its panel or rescue mode.

In both installations, I always trimmed the NVME drives before starting the tests, and in the case of ZFS, I set (both on Proxmox and FreeBSD) compression to zstd and atime to off. No other changes were made compared to the standard installation.

On FreeBSD, the VM was created and managed with vm-bhyve (devel).

On Proxmox, I tested the physical host on ZFS and ext4 and the VM on ZFS and LVM as LVM is the standard and most common setup in Proxmox.

On FreeBSD, I tested the host on ZFS and the VM with both virtio and nvme drivers, on zvol, and as an image file within a ZFS dataset.

I used sysbench installed from the official Debian repository (on Proxmox and VM) and from the FreeBSD package on the respective host.

The VMs, both on Proxmox and FreeBSD, have nearly identical characteristics and default configuration (apart from the nvme drivers set on bhyve, and for that reason, I also tested virtio).

For those who want to reproduce my tests, here are the detailed configurations of the VMs used in bhyve:

FreeBSD bhyve VM Configuration with NVMe Driver:

loader="uefi"
cpu=4
memory=4096M
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"

FreeBSD bhyve VM Configuration with virtio Driver:

loader="uefi"
cpu=4
memory=4096M
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0.img"

Proxmox VM Configuration:

Component	Details
Memory	4.00 GiB [balloon=0]
Processors	4 (1 sockets, 4 cores) [x86-64-v2-AES]
BIOS	Default (SeaBIOS)
Display	Default
Machine	Default (i440fx)
SCSI Controller	VirtIO SCSI single
CD/DVD Drive (ide2)	local:iso/debian-12.5.0-amd64-netinst.iso,media=cdrom,size=629M
Hard Disk (scsi0)	zfspool:vm-100-disk-0,cache=writeback,discard=on,iothread=1,size=50G,ssd=1
Network Device (net0)	virtio=BC:24:11:22:3D:F0,bridge=vmbr0

In all the configurations, I used Debian 12 as the VM operating system, with the file system on ext4.

I chose Debian 12 as it is a stable, widespread, and modern Linux distribution. I did not test a FreeBSD VM because, in my setups, I tend not to virtualize FreeBSD on FreeBSD but to use nested jails.

All tests were performed multiple times, and I took the median results. CPU and RAM were tested only on the first VM (on Proxmox (ZFS) and FreeBSD (ZFS and nvme)) as they are not dependent on the underlying storage. Storage performance, on the other hand, was tested on all configurations.

CPU and RAM Tests on VMs

On both VMs:

sysbench --test=cpu --cpu-max-prime=20000 run
sysbench --test=memory run

Comparative Results

CPU Test

Configuration	Events per Second	Total Time (s)	Latency (avg) (ms)
Proxmox	498.08	10.0010	2.01
FreeBSD	473.65	10.0019	2.11

CPU Percentage Analysis

Difference in Events per Second: ((498.08 - 473.65) / 498.08 approx -4.91\%)
Difference in Total Time: ((10.0019 - 10.0010) / 10.0010 approx +0.009\%)
Difference in Latency (avg): ((2.11 - 2.01) / 2.01 approx +4.98\%)

RAM Test

Configuration	Total Operations	Operations per Second	Total MiB Transferred	MiB/sec	Latency (avg) (ms)
Proxmox	64777227	6476757.59	63259.01	6324.96	0.00
FreeBSD	68621063	6861139.06	67012.76	6700.33	0.00

RAM Percentage Analysis

Difference in Total Operations: ((68621063 - 64777227) / 64777227 approx +5.94\%)
Difference in Operations per Second: ((6861139.06 - 6476757.59) / 6476757.59 approx +5.94\%)
Difference in Total MiB Transferred: ((67012.76 - 63259.01) / 63259.01 approx +5.93\%)
Difference in MiB/sec: ((6700.33 - 6324.96) / 6324.96 approx +5.93\%)

CPU and RAM Comparative Results Table

Test	Metric	Proxmox (KVM)	FreeBSD (bhyve)	Difference (%)
CPU	Events/s	498.08	473.65	-4.91
	Time (s)	10.0010	10.0019	+0.009
	Latency	2.01	2.11	+4.98
RAM	Ops	64777227	68621063	+5.94
	Ops/s	6476757.59	6861139.06	+5.94
	MiB	63259.01	67012.76	+5.93
	MiB/s	6324.96	6700.33	+5.93
	Latency	0.00	0.00	0.00

Interpretation of CPU and RAM Results

CPU Performance:
The VM on FreeBSD has slightly lower CPU performance compared to Proxmox (-4.91% in events per second).
The total execution time is nearly identical, with a negligible difference.
The average latency is slightly higher on FreeBSD (+4.98%).
RAM Performance:
The VM on FreeBSD has better RAM performance compared to Proxmox (+5.94% in operations and MiB/sec).
The average latency is identical in both configurations.

In summary, while Proxmox provides more consistent CPU performance, FreeBSD demonstrates superior memory performance. The choice between Proxmox and FreeBSD may depend on the specific workload requirements and the importance of consistent performance versus higher throughput.

I/O Performance Tests

The test has been conducted using sysbench, with this command line:

sysbench --test=fileio --file-total-size=30G prepare
sysbench --test=fileio --file-total-size=30G --file-test-mode=rndrw  --max-time=300 --max-requests=0 run

I/O Comparative Performance Data with Percentage Differences

Metric	VM on Proxmox (ZFS)	VM on Proxmox (LVM)	VM on FreeBSD (ZFS, NVMe)	VM on FreeBSD (ZFS, Virtio)	VM on FreeBSD (zvol)	Host FreeBSD (ZFS)	Host Proxmox (ZFS)	Host Proxmox (ext4)
File creation speed (MiB/s)	407.82	461.52	1467.83	1398.81	1333.64	1625.67	968.64	633.13
Reads per second	650.09	504.80	11183.44	806.93	11834.53	1234.62	920.95	498.37
Writes per second	433.40	336.54	7455.62	537.95	7889.69	823.08	613.96	332.25
fsyncs per second	1387.08	1076.97	23858.08	1721.79	25247.36	2634.01	1964.96	1063.19
Read throughput (MiB/s)	10.16	7.89	174.74	12.61	184.91	19.29	14.39	7.79
Write throughput (MiB/s)	6.77	5.26	116.49	8.41	123.28	12.86	9.59	5.19
Total events	741163	575588	12749157	919952	13491459	1407592	1049894	568277
Average latency (ms)	0.40	0.52	0.02	0.33	0.02	0.21	0.29	0.53
95th percentile latency (ms)	2.30	3.25	0.06	1.58	0.05	1.32	1.79	2.71
Max latency (ms)	22.65	32.30	35.49	13.60	77.53	9.03	9.47	17.39
Total test time (s)	300.0475	300.1147	300.0020	300.0226	300.0012	300.0416	300.0159	300.1381

Percentage Differences Compared to VM on Proxmox (ZFS)

Metric	VM on Proxmox (LVM)	VM on FreeBSD (ZFS, NVMe)	VM on FreeBSD (ZFS, Virtio)	VM on FreeBSD (zvol)	Host FreeBSD (ZFS)	Host Proxmox (ZFS)	Host Proxmox (ext4)
File creation speed (MiB/s)	+13.18%	+259.77%	+242.99%	+227.02%	+298.62%	+137.52%	+55.25%
Reads per second	-22.34%	+1619.98%	+24.13%	+1720.45%	+89.92%	+41.67%	-23.34%
Writes per second	-22.35%	+1620.26%	+24.12%	+1720.42%	+89.91%	+41.66%	-23.34%
fsyncs per second	-22.36%	+1620.02%	+24.13%	+1720.18%	+89.90%	+41.66%	-23.35%
Read throughput (MiB/s)	-22.34%	+1619.88%	+24.11%	+1719.98%	+89.86%	+41.63%	-23.33%
Write throughput (MiB/s)	-22.30%	+1620.68%	+24.22%	+1720.97%	+89.96%	+41.65%	-23.33%
Total events	-22.34%	+1620.16%	+24.12%	+1720.31%	+89.92%	+41.65%	-23.31%
Average latency (ms)	+30.00%	-95.00%	-17.50%	-95.00%	-47.50%	-27.50%	+32.50%
95th percentile latency (ms)	+41.30%	-97.39%	-31.30%	-97.83%	-42.61%	-22.17%	+17.83%
Max latency (ms)	+42.60%	+56.69%	-39.96%	+242.30%	-60.13%	-58.19%	-23.22%
Total test time (s)	+0.02%	-0.02%	-0.01%	-0.02%	-0.02%	-0.01%	+0.01%

Percentage Differences Compared to VM on Proxmox (LVM) as this is the standard Proxmox setup

Metric	VM on Proxmox (ZFS)	VM on FreeBSD (ZFS, NVMe)	VM on FreeBSD (ZFS, Virtio)	VM on FreeBSD (zvol)	Host FreeBSD (ZFS)	Host Proxmox (ZFS)	Host Proxmox (ext4)
File creation speed (MiB/s)	-11.64%	+218.04%	+203.09%	+188.97%	+252.24%	+109.88%	+37.18%
Reads per second	+28.78%	+2115.42%	+59.85%	+2244.40%	+144.58%	+82.44%	-1.27%
Writes per second	+28.78%	+2115.37%	+59.85%	+2244.35%	+144.57%	+82.43%	-1.27%
fsyncs per second	+28.79%	+2115.30%	+59.87%	+2244.30%	+144.58%	+82.45%	-1.28%
Read throughput (MiB/s)	+28.77%	+2114.70%	+59.82%	+2243.60%	+144.49%	+82.38%	-1.27%
Write throughput (MiB/s)	+28.71%	+2114.64%	+59.89%	+2243.73%	+144.49%	+82.32%	-1.33%
Total events	+28.77%	+2114.98%	+59.83%	+2243.94%	+144.55%	+82.40%	-1.27%
Average latency (ms)	-23.08%	-96.15%	-36.54%	-96.15%	-59.62%	-44.23%	+1.92%
95th percentile latency (ms)	-29.23%	-98.15%	-51.38%	-98.46%	-59.38%	-44.92%	-16.62%
Max latency (ms)	-29.88%	+9.88%	-57.89%	+140.03%	-72.04%	-70.68%	-46.16%
Total test time (s)	-0.02%	-0.04%	-0.03%	-0.04%	-0.02%	-0.03%	+0.01%

Analysis of Performance Data

The performance data collected from various configurations of Proxmox and FreeBSD provides a comprehensive view of the I/O capabilities and highlights some significant differences. Here is an analysis of the key findings:

Comparative Analysis

Hypothesis on NVMe Performance and fsync

An important observation from my tests is that VMs with the bhyve NVMe driver show significantly higher performance compared to the same VMs with the virtio driver or compared to the physical host system. This difference initially led me to hypothesize that the bhyve NVMe driver might not correctly respect fsync operations, returning a positive result before the underlying file system has confirmed the final write. However, this was just a theory based on benchmark results and is not supported by concrete data. Furthermore, some developers have reviewed the code and found no evidence to suggest this behavior, and I personally have never encountered any potential issues that would indicate such a problem.

Specifically, I observed that:

The VM with the virtio driver has performance comparable to Proxmox.
The VM with the NVMe driver, whether on a ZFS dataset or zvol, shows performance superior to the physical FreeBSD host.

Host Physical Systems and Filesystems

File Creation Speed:
- Host FreeBSD (ZFS) shows the highest file creation speed at 1625.67 MiB/s, which is +68.03% compared to Host Proxmox (ZFS) and +156.72% compared to Host Proxmox (ext4).
- Host Proxmox (ext4) has a file creation speed of 633.13 MiB/s, which is -34.62% compared to Host Proxmox (ZFS).
- Read and Write Operations per Second:
- Host FreeBSD (ZFS) demonstrates the highest read and write operations per second with 1234.62 reads/s and 823.08 writes/s.
  - Reads per second: +34.06% compared to Host Proxmox (ZFS) and +147.80% compared to Host Proxmox (ext4).
  - Writes per second: +34.04% compared to Host Proxmox (ZFS) and +147.61% compared to Host Proxmox (ext4).
- Host Proxmox (ext4) shows a lower performance with 498.37 reads/s and 332.25 writes/s.
- Host Proxmox (ZFS) has 920.95 reads/s and 613.96 writes/s.
- fsync Operations per Second:
- Host FreeBSD (ZFS) achieves the highest fsync operations per second at 2634.01 fsyncs/s, which is +34.02% compared to Host Proxmox (ZFS) and +147.73% compared to Host Proxmox (ext4).
- Host Proxmox (ext4) has a lower performance with 1063.19 fsyncs/s.
- Host Proxmox (ZFS) achieves 1964.96 fsyncs/s.
- Throughput:
- Host FreeBSD (ZFS) again leads in throughput with 19.29 MiB/s read and 12.86 MiB/s write.
  - Read throughput: +34.03% compared to Host Proxmox (ZFS) and +147.53% compared to Host Proxmox (ext4).
  - Write throughput: +34.08% compared to Host Proxmox (ZFS) and +147.79% compared to Host Proxmox (ext4).
- Host Proxmox (ext4) has the lowest throughput with 7.79 MiB/s read and 5.19 MiB/s write.
- Host Proxmox (ZFS) has 14.39 MiB/s read and 9.59 MiB/s write.
- Latency:
- Host FreeBSD (ZFS) shows the lowest average latency at 0.21 ms and 95th percentile latency at 1.32 ms.
  - Average latency: -27.59% compared to Host Proxmox (ZFS) and -60.38% compared to Host Proxmox (ext4).
  - 95th percentile latency: -26.27% compared to Host Proxmox (ZFS) and -51.29% compared to Host Proxmox (ext4).
- Host Proxmox (ext4) has the highest average latency at 0.53 ms and 95th percentile latency at 2.71 ms.
- Host Proxmox (ZFS) has an average latency of 0.29 ms and 95th percentile latency of 1.79 ms.

VMs vs Physical Hosts

File Creation Speed:
- VM on FreeBSD (ZFS, NVMe) demonstrates an outstanding file creation speed at 1467.83 MiB/s (+218.04% compared to VM on Proxmox (LVM) and +259.77% compared to VM on Proxmox (ZFS)).
- VM on FreeBSD (zvol) achieves 1333.64 MiB/s, which is also significantly higher than VM on Proxmox (LVM) and VM on Proxmox (ZFS).
- Read and Write Operations per Second:
- VM on FreeBSD (ZFS, NVMe) shows exceptional performance with 11183.44 reads/s and 7455.62 writes/s.
- VM on FreeBSD (zvol) also performs excellently with 11834.53 reads/s and 7889.69 writes/s.
- fsync Operations per Second:
- VM on FreeBSD (ZFS, NVMe) achieves 23858.08 fsyncs/s, and VM on FreeBSD (zvol) achieves 25247.36 fsyncs/s, both significantly higher than any other configuration.
- Throughput:
- VM on FreeBSD (ZFS, NVMe) achieves the highest throughput with 174.74 MiB/s read and 116.49 MiB/s write.
- VM on FreeBSD (zvol) also has high throughput at 184.91 MiB/s read and 123.28 MiB/s write.
- Latency:
- VM on FreeBSD (ZFS, NVMe) shows very low average latency at 0.02 ms and 95th percentile latency at 0.06 ms.
- VM on FreeBSD (zvol) has similarly low latencies, indicating fast response times for I/O operations.

VM Configurations Comparison

File Creation Speed:
- Among VMs, VM on FreeBSD (ZFS, NVMe) leads, followed by VM on FreeBSD (zvol), and then VM on FreeBSD (ZFS, Virtio).
- Read and Write Operations per Second:
- VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) both outperform VM on Proxmox (ZFS) and VM on Proxmox (LVM) configurations significantly.
- VM on Proxmox (ZFS) outperforms VM on Proxmox (LVM) in read and write operations.
- fsync Operations per Second:
- VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) have significantly higher fsync operations compared to VM on Proxmox (ZFS) and VM on Proxmox (LVM).
- Throughput:
- VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) have the highest throughput, followed by VM on Proxmox (ZFS) and then VM on Proxmox (LVM).
- Latency:
- VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) show the lowest latencies among the VMs, indicating faster response times.
- VM on Proxmox (ZFS) shows lower latencies compared to VM on Proxmox (LVM).

Cache Settings and Performance Influence

Cache settings can significantly influence the performance of virtualization systems. In my setup, I did not modify the cache settings for the NVMe and virtio drivers, keeping the default settings. It is possible that the observed performance differences are also due to how different operating systems manage the caches of NVMe devices. I encourage other system administrators to explore the cache settings of their systems to see if changes in this area can influence benchmark results.

Conclusion

Regarding RAM and CPU, the performance of the VMs is comparable. There are slight differences in favor of Proxmox for CPU and FreeBSD for RAM, but in my opinion, these differences are so negligible that they wouldn't sway the decision towards one solution or the other.

The I/O performance data clearly indicates that VM on FreeBSD with NVMe and ZFS outperforms all other configurations by a significant margin. This is evident in the file creation speed, read/write operations per second, fsync operations per second, throughput, and latency metrics.

When comparing physical hosts, Host FreeBSD (ZFS) demonstrates excellent performance, particularly in comparison to Host Proxmox (ZFS) and Host Proxmox (ext4).

When comparing VMs, VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) configurations stand out as the top performers.

The VM using virtio on FreeBSD also shows strong performance, albeit not as high as the NVMe configuration. It significantly outperforms Proxmox configurations in terms of file creation speed, read/write operations per second, and throughput, while maintaining competitive latencies.

The virtio driver provides a stable and reliable option, making it a suitable choice for environments where the NVMe driver cannot be used. This makes FreeBSD with virtio a balanced option for virtualization, offering both high performance and reliability.

By examining these performance metrics, users can make informed decisions about their virtualization and storage configurations to optimize their systems for specific workloads and performance requirements.

In light of these tests and experiments, I can therefore affirm that my sensations (and those of many users) of greater "snappiness" of the VMs on FreeBSD can be confirmed. Certainly, Proxmox is a stable solution, rich in features, battle-tested, and has many other valid points, but FreeBSD, especially with the nvme driver, demonstrates very high performance and a very low overhead in installation and operation.

I will continue to use both solutions with great satisfaction, but I will be even more encouraged to implement virtualization servers based on FreeBSD and bhyve.

How we are migrating (many of) our servers from Linux to FreeBSD - Part 3 - Proxmox to FreeBSD

Stefano Marinelli — Tue, 14 Mar 2023 13:00:00 +0000

In recent years, we've been migrating many of our servers from Linux to FreeBSD as part of our consolidation and optimization efforts. Specifically, we've been moving services that were previously deployed using Docker onto FreeBSD, and it has proven to be a great choice for handling workloads efficiently.

To this end, we've also been migrating many of our virtual machines (VMs) to FreeBSD, deploying services within FreeBSD jails. In some cases, these jails have even replaced entire VMs and run bare metal. Although we prefer to move to native FreeBSD whenever possible, sometimes it's not the best option for all the services we offer. As a result, one of our most critical physical servers has been left behind for years.

This server was a Proxmox server that we installed many years ago and updated to version 6.4. It hosted some critical services, but upgrading to Proxmox 7.x posed some challenges. In particular, some of the LXC containers required tweaks.

Unfortunately, this server was quite old, with only four physical disks and 64 GB of RAM. It was located in an OVH data center and had been running well until one of the disks started to malfunction once a week, on Sundays. This would trigger a RAID reconstruction that kept the system busy for about two days.

Despite my preference for simple setups, this server had been deployed gradually over many years, and everything was tied together. As a result, unraveling the system to resolve the issues was not a simple task. Sometimes the combination of simple things can make everything complex.

The Proxmox Server

The Proxmox server was configured as the central hub for various services, including primary DNS, web hosting, VOIP, and more. It featured several bridges, each with its own specific purpose, and was connected to a virtual machine running MikroTik CHR. This machine was responsible for consolidating all incoming VPNs from the MikroTik devices we managed, both ours and those belonging to our clients. Additionally, it provided a series of bridges to manage these devices and all server management VPNs and other services. The Proxmox server also housed several virtual machines running Linux, FreeBSD, OpenBSD, and NetBSD, as well as LXC containers.

Over the last two years, we've been migrating most of these virtual machines and containers to FreeBSD-based VMs, which feature their own specific jails. Consequently, most of the VMs we've had to move were BSD-based, while only five Linux VMs remained. The LXC containers hosted a range of services, including servers managed by Virtualmin, a large installation of Zimbra (which was hosted within an LXC container running CentOS 7), as well as some minor Alpine Linux-based machines. We located all these virtual machines and containers in a LAN created and managed by CHR. All public IPs were managed by CHR, which relied on NAT mappings to establish communication between them. CHR had thus become the heart of our system, and if it experienced any issues, it could potentially take down the entire system. Fortunately, it remained stable for years.

Migration - first steps

The first step I took was to install FreeBSD on the new server. Easy peasy. The next step was to find a way for the CHR to migrate to the new server (under bhyve) and continue to manage all the public IPs of the original server. The problem is that OVH, with its failover IPs, ties a specific MAC address to each individual IP address. Therefore, the only way was to create a bridge on the FreeBSD server (on the Proxmox server, I already had the bridge on the physical network card) and create an L2 tunnel between the two servers - I used OpenVPN with tap interfaces, specifically inserted into the bridges. I could have used other methods and techniques, but I wanted to experiment with a setup that could allow, if necessary, to bridge a larger number of physical and virtual servers even if the IPs are all mapped to a single server. OVH does not allow, in fact, the splitting of classes, so a move must be made for the entire class, not for a single IP address.

Initially, MikroTik CHR 7 did not boot on bhyve. In the end, I managed to make it work, but I had other problems, probably related to the MTU of the interfaces. So I thought about taking the opportunity to unbind the LXC containers and VMs from CHR and remove MikroTik from the setup. With RouterOS version 7, in fact, Wireguard-based VPNs are also supported, so within a few days, it was possible to update the few routers still on 6.x and recreate some VPNs using Wireguard. I mapped both the VMs and LXC containers directly to their respective public IPs, greatly simplifying the steps. Everything worked perfectly.

The next step was to test the first migrations, starting from the VMs already on FreeBSD. For simplicity, I created a new FreeBSD VM in bhyve and copied (via zfs-send and zfs-receive) the datasets related to BastilleBSD. All services are installed in jails managed by Bastille, so this was enough to have, in a short time, a new operating server equivalent to the previous one. At that point, I shut down the original server, connected the VM to the bridge linked to the tunnel (after modifying its MAC address), turned on the new FreeBSD VM (on bhyve), and everything started to work correctly - but from the new physical server.

One by one, I moved all the FreeBSD VMs. For Linux, NetBSD, and OpenBSD, I simply copied the images and pointed bhyve to them. Some small specific configuration on vm-bhyve and everything started to work correctly. Where possibile, I replaced the “virtio” with “nvme” as it performs much better on bhyve.

Migration - LXC containers to Virtual Machines

For LXC containers, I initially thought of creating an Alpine Linux virtual machine, installing LXD, and copying each individual container. It worked for some of them, but for others, I started to encounter strange issues, similar to those that would have required manual intervention to upgrade from Proxmox 6.x to 7.x. As is often the case with Linux-based solutions, compatibility is not always preserved between updates, so I would have had to fine-tune all the containers, which I didn't feel like doing. The containers had been created (at the time) to optimize RAM usage on the Proxmox machine, but to date, they have caused more problems than benefits. In some cases, certain processes got "stuck," making it impossible to "reboot" the LXC container, requiring the entire physical node to be rebooted. If they had been virtual machines, I could have given a "kill" command from the virtualizer (to the respective KVM process, in that case) and restarted it.

For greater compatibility and ease of future management, I decided to convert the LXC containers into actual VMs on bhyve. The process was simple:

Creating an empty VM with vm-bhyve and booting the VM with SystemRescueCD.
Creating destination partitions and file systems in the VM, then doing a complete rsync of the original LXC container.
Adjusting the fstab file, installing the kernel on the destination VM, and creating the initrd (some containers were already copies of VMs, so the kernel remained installed and updated, even though it wasn't being used. The initrd, on the other hand, did not include the nvme or virtio drivers, so I had to regenerate it anyway.)
Adjusting the bhyve vm configuration file, doing one last rsync after shutting down the services, shutting down the original LXC container, and starting the bhyve VM.

Everything worked correctly, so one by one, I moved all the containers. The largest one ended up on another physical node (also FreeBSD with bhyve) temporarily because the space on the new server was not sufficient to contain it. It didn't need to be on this server, so no problem.

One by one, the LXC containers started on the new server. Apart from some minor adjustments to the destination VMs (different network interface names, etc.), I didn't encounter any particular problems even after several days. Everything works perfectly.

At the very end, I re-created the MikroTik CHR VM. I’ll keep this setup separate for now, as strictly tied to eoip interfaces. This was the main reason why I haven’t performed the migration before. Things were too tied together and I had to untie everything, step by step.

…and then one of the Linux VMs started to freeze

Several Linux VMs are just the basis on which Docker runs. One of them (not even among the busiest) started, every 12/15 hours, to completely freeze. It stopped responding to ping, and it was impossible to give any type of command from the console. In a word: stuck.

Searching the web, I found some references to this problem and, observing the errors of an ssh session that was left connected (stuck, but still showing the last error), I found it to be a problem similar to the one described in this post, namely:

"watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [khugepaged:67]"

I tried various solutions such as changing the storage driver, the number of cores, the distribution (from Alpine to Debian), etc., but none of these operations solved the issue. I also noticed that the problem occurs with all Linux VMs, but only those with a recent kernel (> 5.10.x) freeze, while the others continue to work. The problem does not occur, however, with the *BSDs.

In the end, I:

Reduced the number of cores to 1 for the VMs that did not have a high load (some remained with multiple cores), hypothesising a problem with allocating cores that were too busy
Gave the command: "/usr/bin/echo 60 > /proc/sys/kernel/watchdog_thresh" to the VM.

The VM became stable, and I have not seen that error/warning on any other machine since. I will investigate further, but I believe it is a problem related to the Linux kernel, which, for some reason, generates a kernel panic if particular situations of CPU concurrency are generated.

The End…and a nice OOM!

After moving everything, I was finally able to migrate the entire class of OVH IPs from one physical server to another. The operation was quite quick, but in order to avoid problems, I notified all users and performed the operation on a Sunday and during off-peak hours. The whole process took about 10 minutes and there were no hitches of any kind.

For safety reasons, I kept the Proxmox machine active for a few more days, but there was no need to use it. However, after a couple of days, I encountered a problem: the largest VM, in some cases, was being "killed" because FreeBSD generated an OOM. I had never seen, from FreeBSD 13.0 onwards, any OOM related to "abuse" of RAM usage by ZFS, but in this case, it actually happened.

In the end, I understood that ZFS, on FreeBSD, is able to release memory, but not quickly enough to manage any "spikes" in individual VMs. In fact, the VMs do not know the situation of the physical host's RAM, so they will tend to occupy all the space allotted to them (even if only for caching). A sudden spike (i.e. if you create and launch a new VM) could cause a sudden increase in RAM usage by the bhyve process, and FreeBSD could be forced to kill it, even if part of the RAM is only ARC cache. While Proxmox supports HA (i.e., control over whether the VM is running), vm-bhyve only launches the VM (bhyve process). I should manage it with tools like monit, but for now, I preferred to simply set limits on ZFS RAM usage using "vfs.zfs.arc_max", and there have been no more problems.

Final considerations

The operation was long but linear. The most complex part was unraveling all the configurations related to MikroTik CHR and the VPNs linked to each individual LXC machine/container. Once everything was implemented on a dedicated VM, the operation was fairly straightforward.

The hardware specifications of the destination physical server are slightly better than the starting one, but the final performance of the setup has greatly improved. The VMs are very responsive (even those that were previously LXC containers running directly on bare metal) and, thanks to ZFS, I can make local snapshots every 5 minutes. In addition, every 10 minutes, I can copy (using the excellent zfs-autobackup) all the VMs and jails to other nodes both as a backup and as an immediate restart in case of disaster. I just need to map the IPs, and everything will start working very quickly. Proxmox also allows you to perform this type of operation with ZFS, but you still need to have Proxmox (in a compatible version) on the target machine. With the current setup, I only need any FreeBSD node that supports bhyve.

Proxmox is an excellent tool, well-developed, open-source, efficient, and stable. We manage many installations, including complex ones (ceph clusters, etc.), and it has never let us down. However, not all tools are ideal for all situations, and for setups like the one described, the new configuration based on FreeBSD has shown significantly interesting performance and greater management and maintenance granularity.

Virtualizing on vm-bhyve is not complex, but it is certainly not comparable, at the current state, to the simplicity of using a clean and complete interface like Proxmox's. A complete HA system is still missing (sure, it's achievable manually, but...), as well as complete management web interface. However, for knowledgeable users, it is undoubtedly a powerful tool that allows you to have excellent FreeBSD as a base. I'm totally satisfied with my migration and the result is far better than I expected.

Creating an Alpine Linux VM on bhyve - with root on ZFS (optionally encrypted)

Stefano Marinelli — Tue, 01 Nov 2022 15:47:35 +0000

Bhyve is great - and we’re using it with a lot of guest operating systems.

One of my favourite Linux distributions is Alpine Linux - it’s great as a docker or lxc/lxd host, is light, stable and easily manageable.

On FreeBSD, vm-bhyve already provides a good template for Alpine Linux, but it’s based on the plain standard image with boot on ext4.

So we need the alpine-extended iso - with zfs module.

Let’s create a new Alpine Linux bhyve VM:

vm create -t alpine -s 50G -m 4G -c 2 alpinevm

Now let’s configure it:

vm configure alpinevm

Let’s change some options: vmlinuz-vanilla and initramfs-vanilla should be changed to vmlinuz-lts and initramfs-lts.

More, we should instruct grub to boot from ZFS. The configuration should be similar to this:

[…]
grub_install0="linux /boot/vmlinuz-lts initrd=/boot/initramfs-lts alpine_dev=cdrom:iso9660 modules=loop,squashfs,sd-mod,usb-storage,sr-mod"
grub_install1="initrd /boot/initramfs-lts"
grub_run0="linux /boot/vmlinuz-lts root=rpool/ROOT/alpine rootfstype=zfs modules=ext4,zfs"
grub_run1="initrd /boot/initramfs-lts"
[…]

It’s now time to start with the installation:

vm install alpinevm alpine-extended.iso

Now, follow this Alpine Linux Wiki guide. Remember that we’re dealing with “vda” devices, not "sda", so change them accordingly. If you don’t want to have an encrypted rootfs dataset, just avoid the encryption line in the zpool create command.

At the end of the installation procedure, reboot and enjoy your new Alpine Linux bhyve VM.

How we are migrating (many of) our servers from Linux to FreeBSD - Part 2 - Backups and Disaster Recovery

Stefano Marinelli — Mon, 30 May 2022 03:03:52 +0000

After my post on why we’re migrating (most of) our servers from Linux to FreeBSD, I’ve started to write about how we’re doing it. After covering a basic installation (we’re doing a massive use of jails), I’m going now to describe how we’re performing backups.

Backup is not a tool. Backup is not a software you can buy. Backup is a strategy you need to study and implement to be able to solve your specific problems. You need to understand what you’re doing, otherwise you’ll always have a Schrödinger’s Backup - it may work or not and if you don’t test it well enough (i.e. restore) you’ll find out when it’s too late.

We’re performing backups in many different ways but, for our physical and virtual FreeBSD servers, we have a dual approach. We need both a “ready to use” backup (that will be described here, useful for a fast disaster recovery or prompt restore of specific jails) and a “colder”, more space efficient backup that can be kept for months (or years), more similar to the borg approach on previous posts. Generally speaking, we store our OS (and jails) on ZFS, so I’ll describe this kind of approach here.

Disaster recovery backup - ZFS send/receive

BastilleBSD creates its datasets and mounts them on /usr/local/bastille . There are no databases, all the jails’ configurations are inside that mountpoint so it’s quite easy to backup and restore all the jails or any single jail in one go. More, the “everything-in-a-jail” approach simplifies the restore process as you don’t need to restore the entire host OS, just install an empty FreeBSD server, install BastilleBSD and restore the jails. Or add the jails to already existing FreeBSD systems.

We normally use FreeBSD (or Linux with ZFS) backup servers, well protected and encrypted at rest. For the ZFS send/receive approach, our servers are NOT reachable from the outside. We can ssh into them only using a VPN - they’re too precious to be exposed on the World Wild Web - or, if strictly needed, we expose ssh only using keys, no passwords. We perform the backups using a pull strategy: the backup server connects to the production servers, gets the data, disconnects. The production servers have NO ACCESS to the main backup server. Should they ever be seriously compromised, the backup is safe.

There are many tools that can help to set up this kind of configuration. I’ve tried many of them and found that they all have some good and bad points. The one I decided to use for our servers is zfs-autobackup. It’s easy to use, everything can be set via command line and has a good cron (or Jenkins) output, useful to understand if everything is right.

Let’s consider two servers, one is called “ProdA” and the other is called “Bck” - we obviously want to backup the ProdA into Bck.

Installing ProdA has been covered on a previous post, Bck is quite simple and outside the scope of this post. We just need a protected zfs FreeBSD (or Linux) server. That’s all. Let’s assume that ProdA has a BastilleBSD zfs dataset (and children datasets), with jails and everything needed, as configured in the last post. We now need to install the needed software. On Bck:

pkg install py311-zfs-autobackup mbuffer

On ProdA:

pkg install mbuffer

mbuffer will be used as a ram buffer to avoid read/write spikes (or slowdowns) while sending/receiving the snapshots.

It’s time to prepare the destination dataset. Assuming that Bck has a zroot base dataset, we’ll be creating (as root) a zroot/backups/ProdA

zfs create -p zroot/backups/ProdA

Ok, let’s now go to ProdA

We want to create an unprivileged user that will send the data. We don’t want to allow Bck to connect as root, even if it’s trusted and secure. Let’s create a user called “backupper”. Then, we need to give backupper the right permissions:

zfs allow -u backupper send,snapshot,hold,mount,destroy zroot

Note: if you want Bck to be able to delete the snapshots on ProdA, backupper needs the destroy permission. That means this user can destroy the whole system as can ALSO destroy zroot (or any source dataset you decide). If you’re afraid of this, different approaches must be used (i.e.: local root performing snapshot/cleanups and Bck only transferring them, not hard to achieve with zfs-autobackup). Considering that the Bck is safe, secure and protected, we can tolerate this weakness. Just be sure nobody can break the “backupper” user. Do not use password, use ssh keys and treat this user with the same care you'd use with root.

Now, as root on ProdA:

zfs set autobackup:bck_server=true zroot

We’re setting a custom property, called “autobackup:bck_server”, allowing the zroot (and children) dataset to be backed up by zfs-autobackup. zfs-autobackup will search for all datasets with that property set to “true” (also on different pools) and will backup them. If there’s a specific dataset you don’t want to backup, just set it to “false”. Or if you don’t want to backup the entire zroot but, for example, only “zroot/bastille” (and children), just set autobackup:bck_server=true for that dataset.

ssh config

zfs-autobackup will connect via ssh and zfs-autobackup will try to connect as root. Moreover, even after exchanging the ssh key, Bck will connect many times to ProdA to send its zfs commands (one connection per command). Ssh session initiation is quite long, so there will be some latency. In order to (greatly) speed up this time,

“You can make your ssh connections persistent and greatly speed up zfs-autobackup:

On the server that initiates the backup add this to your ~/.ssh/config:

Host ProdA
User backupper
ControlPath ~/.ssh/control-master-%r@%h:%p
ControlMaster auto
ControlPersist 3600

(Taken from https://github.com/psy0rz/zfs_autobackup/wiki/Performance)

It's now time to go back to Bck and issue a command like this (one line):

/usr/local/bin/zfs-autobackup --ssh-source ProdA bck_server zroot/backups/ProdA --zfs-compressed --no-progress --verbose --buffer 32M --keep-source 0 --no-holds  --set-properties readonly=on --clear-refreservation --keep-target 1d1w,1w1m,1m6m  --destroy-missing 30d --clear-mountpoint

Bck will connect to ProdA, perform the snapshots and start transferring. The most interesting options I used here are:

 --keep-source 0 (only the last snapshot will be kept on ProdA)
 --set-properties readonly=on (be sure the Bck clone is read only, so we will be able to perform an incremental/differential backup next time)
 --keep-target 1d1w,1w1m,1m6m (keep one backup per day for one week, one per week for one month, one per month for six months)
 --destroy-missing 30d (if we've deleted a dataset, keep it for 30 days before removing it from Bck)
 --clear-mountpoint (do not mount the dataset in Bck, as it will cause problems sooner or later)

The first copy will be slow as it'll need to send all the data. The second will be quite fast as only the differences will be transferred.

How to perform a disaster recovery

Ok, your dataset (or datasets) has gone. You need to replace it with the last external backup. You have to retransfer the copy into ProdA (or another FreeBSD host, no difference). Connect to Bck and search for the snapshot you want to restore (zfs list -t snapshot will help). Once identified (one line):

zfs send -R zroot/backups/ProdA/zroot/bastille/bastille/jails/t1@bck_server@20220528005830 | mbuffer -4 -s 128k -m 32M | ssh root@ProdA "zfs receive -F -x canmount -x readonly zroot/bastille/bastille/jails/t1"

Note the -x canmount -x readonly flags. Remember that we altered the canmount and readonly properties of the transferred datasets during the backup, so we must restore them into a normal state.

Once finished, ProdA (or the other, restored host) will show t1 as an available jail and you'll be able to start it.

How we are migrating (many of) our servers from Linux to FreeBSD - Part 1 - System and jails setup

Stefano Marinelli — Sat, 05 Feb 2022 10:10:47 +0000

After my post on why we’re migrating (many of) our servers to FreeBSD, I’ve received a lot of feedback. Many questions, many comments. Many e-mails from Linux users asking how we’re migrating, how jails can replace lxc or (in part) Docker, and how we’re monitoring and performing backups/restores.

I’ll write some posts on how we’re doing it. Of course, it won’t cover all the use cases, but it surely will work for the most common ones.

Let’s start with a general idea of the setup I’m going to describe. One of the things I’ve always tried to do is to leave the host Operating System (VM Hypervisors like XCP-NG or Proxmox Server, FreeBSD with jails and/or Bhyve, Alpine Linux lxc host, Debian with docker etc.) as simple and empty as possible.

That’s why we’re keeping the same setup here, with a FreeBSD host system with as few packages (or ports) as possible. This will ensure easier upgrades, easier backup/restore procedures and a smaller attack surface. Unused services or executables can be problematic, so let’s keep them out of our setup.

I’m going to describe our “basic” FreeBSD host and some jails that will contain the services. There are many jail management systems out there like BastilleBSD, cbsd, iocage, the old ezjail, etc. I love simple (but powerful) solutions, without any database of configured/running jails as it may be a problem in case of backup/recovery. I love the BastilleBSD approach, so that’s the one we’ve been using for our servers. BastilleBSD is a collection of shell scripts, is small, doesn’t need any database, is actively developed, doesn’t need any kind of dependency and works both on ZFS and UFS. iocage, for example, needs ZFS so UFS servers are out and doesn’t seem to be actively developed anymore. Moreover, BastilleBSD doesn’t interfere with other jail systems, so you can mix and match whatever you like. Still, I recommend to choose a jail management system and stick with that. We've done it, and we’re using BastilleBSD.

I won’t describe basic FreeBSD installation. It’s straightforward, easy, fast and it’s full of good documentation out there. The most critical decision is the choice of the file system you’re going to use. We’re using ZFS when possible, as it gives us a lot of good opportunities. For more resource constrained systems (or specific situations where ZFS isn’t recommended nor applicable), we just stick with UFS. It has snapshots, too, so backups and restores are quite easy, too.

Once you’ve installed FreeBSD, you have a fully functional host system. From now on, I’ll assume you’ve installed on ZFS.

First of all, we upgrade our hosts systems to the last security patches. It’s just a matter of:

freebsd-update fetch
freebsd-update install

After this, It’d be better to reboot to be sure that the kernel has been upgraded if needed. As it’s an empty and just installed system, we usually don’t use the Boot Environments for now. Reinstalling is just a matter of minutes and we’re not doing a release upgrade.

Now: ports or packages? We usually use packages: the host system will be simple and empty and we don’t need customised build options. So quarterly packages are more than enough.

First of all, let’s install the “pkg” package management system.

pkg install

BastilleBSD

Now let’s use pkg to install BastilleBSD:

pkg install bastille

The first step is to configure BastilleBSD. All we usually do is: set the ZFS options. bastille_zfs_enable=“YES” and bastille_zfs_pool=“zroot/bastille” and create the dataset:

zfs create zroot/bastille

This way, BastilleBSD will create its base dataset in zroot/bastille/bastille

Why the nested bastille/bastille dataset? Because we sometimes use zroot/bastille to store some information about the underlying jails, so we prefer to keep a nested dataset. BastilleBSD will create its datasets there and will mount the needed datasets in /usr/local/bastille so you will find everything there.

Let’s complete the Bastille installation as described in the official documentation. There are many approaches: loopback, vnet with local interface, vnet with existing bridge. The loopback approach is the easiest and more portable. Generally speaking we tend to use it as it’s easier to deal with when we have to perform an emergency restore to another host. I’ll write more about it in the next posts.

Now let’s bootstrap the FreeBSD 14.1-RELEASE so Bastille will download the needed files and create its base system, then it will apply the security patches.

bastille bootstrap 14.1-RELEASE update

NOTE: BastilleBSD supports other jailed operating systems and, recently, some jailed Linux distributions.

After a while, the system will be ready for its first jail. Now, we generally install a reverse proxy, in order to expose it to web traffic. The reverse proxy will be able to connect to other jails and forward the traffic. It's a good example, so let's do it:

bastille create reverseproxy 14.1-RELEASE 192.168.0.1 bastille0

So we’re creating a jail called reverseproxy, the FreeBSD version is 14.1-RELEASE (of the jail, must be the same or older than the host OS), the jail ip and the loopback interface to use to assign this IP for the jail. The jail will have a dataset (in jails/reverseproxy) with jail configuration, redirect configuration, fstab, etc. and another child dataset (jails/reverseproxy/root) with its root file system. Jails can be thin or thick: BastilleBSD documentation is good, so you can go deeper here.

At this point, a “bastille list -a” should show the jail and it should be running. Now we can enter this jail:

bastille console reverseproxy

and install Nginx (and certbot, if needed):

pkg install py311-certbot-nginx nginx

Configure your nginx (and certbot). As you might guess, I won’t describe it here.

Let’s ensure Nginx will be started at jail launch, so:

service nginx enable

and let’s start it:

service nginx start

Let’s go back to the host and let’s ensure that all the connections to the host ports 80 (http) and 443 (https) will be redirected to the reverseproxy jail:

bastille rdr reverseproxy tcp 80 80
bastille rdr reverseproxy tcp 443 443

No output should be shown, but you can check:

bastille rdr ALL list

Congratulations, you’ve exposed your first jail.

You can create all the jails you want. They will be created and stored in child datasets of zroot/bastille/bastille - this is great for backup purposes, but I’ll describe more about it in another article. They'll also be able to communicate using their private ip addresses. If you've used vnet, you'll need to perform some deeper network configurations (and you can use pf inside the jail!), if using the default loopback device you'll be sharing the host network stack.

Blacklistd

FreeBSD base system has some interesting tools and they get automatically installed. One of those is blacklistd. If you’ve used fail2ban, denyhosts or similar tools, you know what it’s useful for. But it’s integrated and is light. Fail2ban, for example, tends to become heavy and huge as it’s reading from log files. Blacklistd gets notified by the daemon it’s protecting, so load is lower. To enable blacklistd, add to /etc/rc.conf:

blacklistd_enable="YES"
blacklistd_flags="-r" 
pflog_enable="YES"

(I’m assuming you’ve already added pf_enable=“YES” when you’ve installed BastilleBSD. Otherwise, you should add this, too, if you’re using pf.

Now you should add this to pf.conf:

anchor "blacklistd/*" in on $ext_if

Where $ext_if is your external interface.

Last step, let’s get to /etc/ssh/sshd_config and enable blacklistd uncommenting:

UseBlacklist yes

Now reload/start pf, sshd, start pflog and blacklistd and wait. After some time, blacklistctl dump -r will show you some data.

Of course there are many more steps to do, the host should be hardened, network should be configured and firewalled, etc. but it's a basic idea of how we're keeping our host as standard as possible and, then, create the services inside the jails.

Why we're migrating (many of) our servers from Linux to FreeBSD

Stefano Marinelli — Mon, 24 Jan 2022 06:55:00 +0000

More about this in the article I wrote to accompany my talk at EuroBSDCon 2024.

Please note: This article was (unexpectedly) quite popular and many interesting opinions came out from the Hacker News thread that it generated. If you're interested, have a look here: https://news.ycombinator.com/item?id=30057549

There's an Italian version of this article here.

I've been a Linux (or GNU/Linux, for the purists) user since 1996. I've been a FreeBSD user since 2002. I have always successfully used both operating systems, each for specific purposes. I have found, on average, BSD systems to be more stable than their Linux equivalents. By stability, I don't mean uptime (too much uptime means too few kernel security updates, which is wrong). I mean that things work as they should, that they don't "break" from one update to the next, and that you don't have to revise everything because of a missing or modified basic command.

I've always been for development and innovation as long as it doesn't (necessarily, automatically and unreasonably) break everything that is already in place. And the road that the various Linux distributions are taking seems to be that of modifying things that work just for the sake of it or to follow the diktats of the Kernel and those who manage it - but not only.

Some time ago we started a complex, continuous and not always linear operation, that is to migrate, where possible, most of the servers (ours and of our customers) from Linux to FreeBSD.

Why FreeBSD?

There are many alternative operating systems to Linux and the *BSD family is varied and complete. FreeBSD, in my opinion, today is the "all rounder" system par excellence, i.e. well refined and suitable both for use on large servers and small embedded systems. The other BSDs have strengths that, in some fields, make them particularly suitable but FreeBSD, in my humble opinion, is suitable (almost) for every purpose.

So back to the main topic of this article, why am I migrating many of the servers we manage to FreeBSD? The reasons are many, I will list some of them with corresponding explanations.

The system is consistent - kernel and userland are created and managed by the same team

One of the fundamental problems with Linux is that (we shall remember) it is a kernel, everything else is created by different people/companies. On more than one occasion Linus Torvalds as well as other leading Linux kernel developers have remarked that they care about the development of the kernel itself, not how users will use it. In the technical decisions, therefore, they don't take into account what is the real use of the systems but that the kernel will go its own path. This is a good thing, as the development of the Linux kernel is not "held back" by the struggle between distributions and software solutions, but at the same time it is also a disadvantage. In FreeBSD, the kernel and its userland (i.e. all the components of the base operating system) are developed by the same team and there is, therefore, a strong cohesion between the parties. In many Linux distributions it was necessary to "deprecate" ifconfig in favor of ip because new developments in the kernel were no longer supported by ifconfig, without breaking compatibility with other (previous) kernel versions or having functions (on the same network interface) managed by different tools. In FreeBSD, with each release of the operating system, there are both kernel and userland updates, so these changes are consistently incorporated and documented, making the tools compatible with their kernel-side updates.

In other words, in FreeBSD there is no need to " revolutionise" everything every few years and changes are made primarily in the form of additions that can enrich (and not break) each update. If a modification was to change the way it interacts with network devices, ifconfig would be modified to take advantage of that and remain compatible with the "old" syntax. In the long-term, this kind of approach is definitely appreciated by system administrators who find themselves with a linear, consistent, and always well-documented update path.

FreeBSD development is (still) driven by technical interests, not strictly commercial ones.

Linux and related distributions now have contributions from many companies, many of which (e.g. Red Hat) push (justifiably) in the direction of what is convenient for them, their products, and their services. Being big contributors to the project they have a big clout so, indeed, their solutions often become de-facto standards. Consider systemd - was there really a need for such a system? While it brought some advantages, it added some complexity to an otherwise extremely simple and functional system. It remains divisive to this day, with many asking, "but was it really necessary? Did the advantages it brought balance the disadvantages?". 70 binaries just for initialising and logging and a million and a half lines of code just for that? But Red Hat threw the rock...and many followed along. Because sometimes it's nice to follow the trend, the hype of a specific solution.

Even FreeBSD has big companies behind it, collaborating in a more or less direct way. The license is more permissive, so not everyone who uses it commercially contributes to it, but knowing that FreeBSD is at the base of Netflix CDNs, Whatsapp servers (waiting for Meta to replace them, for internal coherence reasons, with Linux servers), Sony Playstations and, in part, macOS, iOS, iPadOS, etc. surely gives confidence on its level. These realities, however, do not have enough clout to drive the development of the core team.

Linux has Docker, Podman, lxc, lxd, etc. but... FreeBSD has jails!

FreeBSD jails are very powerful tools for jailing and separating services. There is controversy about Docker not running on FreeBSD but I believe (like many others) that FreeBSD has a more powerful tool. Jails are older and more mature - and by far - than any containerization solution on Linux. Jails are efficient and are well integrated throughout the operating system. All major commands (ps, kill, top, etc.) are able to display jail information as well. There are many management tools but, in fact, they all do the same thing: they interact with FreeBSD base and create custom configuration files. Personally I'm very comfortable with BastilleBSD but there are a lot of very good tools as well as a sufficiently simple manual management. When I need Docker I launch a Linux machine - often Alpine, which I think is a great minimalist distribution, or Debian. But I'm moving a lot of services from Docker to a dedicated jail on FreeBSD. Docker containers are a great tool for rapid (and consistent) software deployment, but it's not all fun and games. Containers, for example, rely on images that sometimes age and are no longer updated. This is a security issue that should not be overlooked.

Linux has ext4, xfs, btrfs... (and zfs, but with some manual intervention). FreeBSD has UFS2 and ZFS

UFS2 is still a very good and efficient file system and, when configured to use softupdates, capable of performing live snapshots of the file system. This is great for backups. Ext4 and XFS do not support snapshots except through external tools (like DattoBD or snapshots through the volume manager). This works, of course, but it is not native. Btrfs is great in its intentions but still not as stable as it should be after all these years of development. FreeBSD supports ZFS natively in the base system and this brings many advantages: separation of datasets for jails, as well as Boot Environments, to make snapshots before upgrades/changes and to be able to boot (from bootloader) even on a different BE, etc.

The FreeBSD boot procedure is cleaner and simpler.

Linux has always used excellent tools such as grub, lilo (now outdated), etc.. FreeBSD has always used a very linear and consistent boot system, with its own bootloader and dedicated boot partition. Whether on mbr, gpt, etc. things are very similar and consistent. I've never had a problem getting a FreeBSD system to boot after a move or recovery from backup. On Linux, however, grub has sometimes given me problems, even after a simple kernel security update.

FreeBSD's network stack is (still) superior to Linux's - and, often, so is its performance.

Meta has been trying to bring the performance of the Linux network stack up to the level of FreeBSD's for years. Many will ask why, then, not move services to FreeBSD. Large companies with huge datacenters can't change solutions overnight, and their engineers, at any level, are Linux experts. They have invested heavily in btrfs, in Linux, in their specifics. Clearly, upon acquiring Whatsapp, they preferred to migrate the "few" Whatsapp servers to Linux and move them to their datacenters. Regarding the real system performance (i.e. disregarding benchmarks, useful only up to a certain point), FreeBSD shines, especially under high load conditions. Where Linux starts to gasp (ex: waiting for I/O) with 100% CPU, FreeBSD has lower processor load and room for more stuff. In the real world (of my servers and load types), I sometimes experienced severe system slowdowns due to high I/O, even if the data to be processed was not read/write dependent. On FreeBSD this does not happen, and if something is blocking, it blocks THAT operation, not the rest of the system. When performing backups or other important operations this factor becomes extremely important to ensure proper (and stable) system performance.

Straightforward system performance analysis

FreeBSD, in the base system, has all the tools to analyze possible problems and system loads. "vmstat" , in a single line, tells me if the machine is struggling for CPU, for I/O or for Ram. "gstat -a" shows me how much, disk by disk, partition by partition, the storage is active, also in percentage with reference to its performance. "top", then, also has support for figuring out, process by process, how much I/O is being used ("m" option). On Linux, to get the same results, you have to install specific applications, different from distribution to distribution.

Bhyve is more incomplete (but more efficient) than KVM

For my purposes, Bhyve is a great virtualization tool. KVM is definitely more complete but since I don't have any special or specific needs not covered by Bhyve on FreeBSD, I found (on average) better performance with this combination. On FreeBSD, however, KSM is missing which, in some cases, can be very useful.

Will I abandon Linux for FreeBSD? Obviously not, just as I haven't for the last 20 years. Both have their uses, their space, their strengths. But if up to now I have had 80% Linux and 20% FreeBSD, the perspective is to invert the percentages of use and, where possible, directly implement solutions based on FreeBSD.

NOTE: this article has been translated from its Italian original version. Even if it's been reviewed and adapted, there might be some errors.

Efficient backup of lxc containers in Proxmox - ZFS

Stefano Marinelli — Thu, 20 Jan 2022 07:05:00 +0000

I've already written about some of my backup strategies in Proxmox. Proxmox Backup Server is an option, but it's not always the best option, especially if you're using lxc containers.

LVM and Ceph RBD baked containers have already been covered in another post, but one of the (many) great options, if you use Proxmox, is ZFS. I extensively use ZFS both on FreeBSD and Linux (and always wished that BTRFS could reach the same level of reliability).

When I don't need a networked file system (ceph) or want to use LVM, I tend to install Proxmox VMs and lxc containers on ZFS.. Let's now focus on backing up lxc containers.

Proxmox uses ZFS datasets for lxc containers' storage so you'll find all your files on /poolname/subvol-x-disk-y . We can easily backup as we've done in my previous article, we just need a different method for taking snapshots of all those datasets.

ZFS datasets have a hidden .zfs directory that contains all the snapshots that currently exist of that specific dataset. ls won't show it, but you can cd and it will be working.

Of course we can use native zfs send/receive or a tool like zfs-autobackup, which I use daily for local snapshots and remote replication, but we want to save the files, not the zfs dataset, so we can be able to backup to a different file system. Any file system. So we will be using borg.

Let's suppose our ZFS pool is named "proxzfs". Here is a suggested script. Of course, this is my script, it works for me and I'm not responsible if it doesn't work for you/destroys all your data/eats your server/etc.

#!/bin/bash

/usr/sbin/zfs snapshot -r proxzfs@forborg

REPOSITORY=yourpath/server/whatever:borgrepository/
TAG=mytag
borg create -v --stats --compression lz4 --progress    \
   $REPOSITORY::$TAG'-{now:%Y-%m-%dT%H:%M:%S}'          \
   /proxzfs/*/.zfs/snapshot/forborg/  \
   --exclude '*subvolYouMayWantToExclude-disk-0*'

/usr/sbin/zfs destroy -vrR proxzfs@forborg

borg prune -v $REPOSITORY --stats --prefix $TAG'-' \
   --keep-daily=31 --keep-weekly=4 --keep-monthly=12

This small script will create a @forborg snapshot for any dataset it will find under "proxzfs", then will fire up borg and ask it to traverse the forborg snapshots automatically mounted inside the .zfs directory of any dataset.

After that, it will destroy the 'forborg' snapshots and execute a borg prune_._ That will delete the old backups, according to the policy you have established. This step can be avoided here but I prefer to perform it after a backup so my repository is always consistent with my policy.