IT Notes - filesystems

Make Your Own Backup System – Part 2: Forging the FreeBSD Backup Stronghold

Stefano Marinelli — Tue, 29 Jul 2025 08:00:00 +0200

With the primary backup strategies and methodologies introduced, we've reached the point where we can get specific: the Backup Server configuration.

When choosing the type of backup server to use, I tend to favor specific setups: either I trust a professional backup service provider (like Colin Percival's Tarsnap), or I want full control over the disks where the backups will be hosted. In both cases, for the past twenty years, my operating system of choice for backup servers has been FreeBSD. With a few rare exceptions for clients with special requests, it covers all my needs. When I require Linux-based solutions, such as the Proxmox Backup Server, I create a VM and manage it within.

I typically use both IPv4 and IPv6. For IPv4, I "play" with NAT and port forwarding. For IPv6, I tend to assign a public IPv6 address to each jail or VM, which is then filtered by the physical server's firewall. Unfortunately, every provider, server, and setup has a different approach to IPv6, making it impossible to cover them all in this article. When a provider allows for routed setups, I use this approach: Make your own VPN: FreeBSD, WireGuard, IPv6, and ad-blocking included - assigning a /72 to the bridge for the jails and VMs.

In my opinion, FreeBSD is a perfect all-rounder for backups, thanks to its ability to completely partition services. You can separate backup services (or specific servers/clients) into different jails or even VMs. Furthermore, using ZFS greatly enhances both flexibility and the range of tools you can use.

The main distinction is usually between local backup servers (physically accessible, though not always attended, and in locations deemed secure) and remote ones, such as leased external servers. I personally use a combination of both. If the services I need to back up are external, in a datacenter, and need to be quickly restorable, I prefer to always have a copy on another server in a different datacenter with good outbound connectivity. This guarantees good bandwidth for restores, which isn't always available from a local connection to the outside world. However, an internal, nearby, and accessible backup server (even a Raspberry Pi or a mini PC) ensures physical access to the data. Whenever possible, I maintain both an external and an internal copy - and they are autonomous, meaning the internal copy is not a replica of the external one, but an additional, independent backup. This ensures that if a problem occurs with the external backup, it won't automatically propagate to the internal one. In any case, the backup must always be in a different datacenter from the one containing the production data. When the fire at the OVH datacenter in Strasbourg caused the entire complex to shut down, many people found themselves in trouble because their backups were in the same, now unreachable, location. I had a copy with another provider, in a different datacenter and country, as well as a local copy.

Despite it being "just" a backup server, I almost always use some form of disk redundancy. If I have two disks, I set up a mirror. With three or more, I use RaidZ1 or RaidZ2. This is because, in my view, backups are nearly as important as production data. The inability to recover data from a backup means it's lost forever. And it happens often, very often, that someone contacts me to recover a file (or a database, etc.) days or weeks after its accidental loss or deletion. Usually, pulling out a file from a two-month-old backup generates a mix of disbelief, admiration, but above all, a sense of security in the person requesting it. And that is what our work should instill in the people we collaborate with.

The backup server should be hardened. If possible, it should be protected and unreachable from the outside. My best backup servers are those accessible only via VPN, capable of pulling the data on their own. If they are on a LAN, it's even better if they are completely disconnected from the Internet.

For this very reason, backups must always be encrypted. Having a backup means having full access to the data, and the backup server is the prime target for being breached or stolen if the goal is to get your hands on that data. I've seen healthcare facilities' backup servers being targeted (in a rather trivial way, to be honest) by journalists looking for health details of important figures. It is therefore critical that the backup server be as secure as possible.

Based on the type of access, I use two types of encryption:

If the server is local (especially if the ZFS pool is on external disks), I usually install FreeBSD on UFS in read-only mode, as I've described in a previous article, and encrypt the backup disks with GELI. This ensures that in the event of a "dirty" shutdown (more likely in unattended environments), I can reconnect to the host and then reactivate the ZFS pool. This approach makes it nearly impossible to retrieve even the pool's metadata if the disks are stolen, as GELI performs a full-device encryption. For example, an employee of a company I work with stole one of the secondary backup disks (which was located at a different, unmonitored company site) to steal information. He got nothing but a criminal complaint. With this approach, it's also not necessary to further encrypt the datasets, which avoids some issues (which I'll discuss later, in a future post).
If the server is remote, in a datacenter, I usually use ZFS native encryption, encrypting the main backup dataset (and BastilleBSD's, if applicable). Consequently, all child datasets containing backups will also be encrypted. In this case as well, a password will be required after a reboot to unlock those datasets, ensuring that the data cannot be extracted if control of the disks is lost.

Here is an example of how to use GELI to encrypt an entire partition and then create a ZFS pool on it (in the example, the disk is da1 - do not follow these commands blindly, or you will erase all content on the da1 device!):

# WARNING: This destroys the existing partition table on disk da1
gpart destroy -F da1

# Create a new GPT partition table
gpart create -s gpt da1

# Add a freebsd-zfs partition that spans the entire disk
# The -a 1m flag ensures proper alignment
gpart add -t freebsd-zfs -a 1m da1

# Initialize GELI encryption on the new partition (da1p1)
# We use AES-XTS with 256-bit keys and a 4k sector size
# The -b flag means "boot," prompting for the passphrase at boot time
geli init -b -l 256 -s 4096 da1p1
# You will be prompted for a passphrase: choose a strong one and save it!

# Attach the encrypted partition. A new device /dev/da1p1.eli will be created.
# You will be prompted for the passphrase you just set
geli attach da1p1

# (Optional) Check the status of the encrypted device
geli status da1p1

# Create the ZFS pool "bckpool" on the encrypted device
# We enable zstd compression (an excellent compromise) and disable atime
zpool create -O compression=zstd -O atime=off bckpool da1p1.eli

In this setup, the reference pool for everything related to backups will be bckpool - and you'll need to keep this in mind for the next steps. Additionally, after every server reboot, you'll need to "unlock" the disk and import the pool:

# Enter the passphrase when prompted
geli attach da1p1

# Import the ZFS pool, which is now visible
zpool import bckpool

With this method, it's not necessary to encrypt the ZFS datasets, as the underlying disk (or, more precisely, the partition containing the ZFS pool) is already encrypted.

If, instead, you choose to encrypt the ZFS dataset (for example, if you install FreeBSD on the same disks that will hold the data and don't want to use a multi-partition approach), you should create a base encrypted dataset. Inside it, you can create the various backup datasets, VMs, and the BastilleBSD mountpoint. Due to property inheritance, they will all be encrypted as well.

To create an encrypted dataset, a command like this will suffice:

# Creates a new dataset with encryption enabled.
# keylocation=prompt will ask for a passphrase every time it's mounted.
# keyformat=passphrase specifies the key type.
zfs create -o encryption=on -o keylocation=prompt -o keyformat=passphrase zfspool/dataset

In this case, after every reboot, you will need to load the key and mount the dataset:

zfs load-key zfspool/dataset
zfs mount zfspool/dataset

Keep in mind the setup you choose, as many of the subsequent choices and commands will depend on it.

Base System Setup

I'll install BastilleBSD - a useful tool for separating services into jails. It will be helpful for isolating our backup services:

pkg install -y bastille

If you used ZFS for the root filesystem, you can proceed directly with the setup. Otherwise (i.e., ZFS on other disks), you'll need to edit the /usr/local/etc/bastille/bastille.conf file and specify the correct dataset on which to install the jails. Then run:

bastille setup

Once the automatic setup is complete, check the /etc/pf.conf file - it will be automatically configured to only accept SSH connections. Ensure the network interface is set correctly. When you activate pf, you will be kicked out of the server, but you can then reconnect.

service pf start

Let's bootstrap a FreeBSD release for the jails - this will be useful later.

bastille bootstrap 14.3-RELEASE update

Now, we create a local bridge. Jails and VMs can be attached to it, making them fully autonomous. Using VNET jails, for example, will allow the creation of VPNs or tun interfaces inside them, simplifying potential future setups (and increasing security by using a dedicated network stack).

Modify the /etc/rc.conf file and add:

# Add lo1 and bridge0 to the list of cloned interfaces
cloned_interfaces="lo1 bridge0"
# Assign an IP address and netmask to the bridge
ifconfig_bridge0="inet 192.168.0.1 netmask 255.255.255.0"
# Enable gateway functionality for routing
gateway_enable="yes"

Let's also modify /etc/pf.conf to allow the 192.168.0.0/24 subnet to access the Internet via NAT. We will skip packet filtering on bridge0 and enable NAT. This isn't the most secure setup, but it's sufficient to get started:

#...
# Skip PF processing on the internal bridge interface
set skip on bridge0
#...
# NAT traffic from our internal network to the outside world
nat on $ext_if from 192.168.0.0/24 to any -> ($ext_if:0)
#...

To ensure the new settings are correct, it's a good idea to test with a reboot.

Since I often configure vm-bhyve in my setups, I prefer to install it right away, creating the dataset that will contain the VMs and installation templates. Remember that zroot is only valid if you installed the entire system on ZFS; otherwise, you'll need to change it to your own dataset:

# Install required packages
pkg install vm-bhyve grub2-bhyve bhyve-firmware
# Create a dataset to store VMs
zfs create zroot/VMs
# Enable the vm service at boot
sysrc vm_enable="YES"
# Set the directory for VMs, using the ZFS dataset
sysrc vm_dir="zfs:zroot/VMs"
# Initialize vm-bhyve
vm init
# Copy the example templates
cp /usr/local/share/examples/vm-bhyve/* /zroot/VMs/.templates/

At this point, I usually enable the console via tmux. This means that when a VM is launched, it won't open a VNC port by default, but a tmux session connected to the VM's serial port. Let's install and configure tmux:

pkg install -y tmux
vm set console=tmux

Let's also attach the switch we created (bridge0) to vm-bhyve so we can use it:

vm switch create -t manual -b bridge0 public

Now, vm-bhyve is ready.

The basic infrastructure is complete. We now have:

ZFS to ensure data integrity, which will also handle redundancy, etc.
BastilleBSD to manage jails, useful for backing up Linux, NetBSD, OpenBSD, and non-ZFS FreeBSD machines.
vm-bhyve to install specific systems (like Proxmox Backup Server).

Backup Strategies

I use various backup tools, too many to list in this article. So I'll make a broad distinction, describing how to use this server to achieve our goal: securing data.

For FreeBSD servers with ZFS (hosts, VMs, jails, hypervisors, and their respective VMs), I use an extremely useful, efficient, and reliable tool: zfs-autobackup.
For Linux servers (without ZFS), NetBSD, OpenBSD, etc. (any non-ZFS OS), I usually use BorgBackup. There are other fantastic tools like restic, Kopia, etc., but BorgBackup has never let me down and has served me well even on low-power devices and after incredibly complex disasters.
For Proxmox servers (a solution I've used with satisfaction in production since 2013, although I'm recently migrating to FreeBSD/bhyve where possible), I use two possible alternatives (often both at the same time): if the storage is ZFS, I use the zfs-autobackup approach. In either case, the most practical solution is the Proxmox Backup Server. And the Proxmox Backup Server is one of the reasons I proposed installing vm-bhyve: running it in a VM and storing the data on the FreeBSD host gives you the best of both worlds. Some time ago, I tried running it in a FreeBSD jail (via Linuxulator), but it didn't work.

Backups using zfs-autobackup

zfs-autobackup is an extremely useful and effective tool. It allows for "pull" type backups, as well as having an intermediary host that connects to both the source and destination, which is useful if you don't want direct contact between the source and destination. I won't describe the latter setup, but the documentation is clear, and I have several of them in production, ensuring that the production server and its backup server cannot communicate with each other.

I usually create a dataset for each server and instruct zfs-autobackup to keep that server's backups in that dataset. The snapshots taken and transferred will all be from the same instant, so as not to create a time skew (some tools of this kind snapshot a dataset, then transfer it, which can result in minutes of difference between two different datasets from the same server).

I've described in detail how I perform this type of backup in a previous post, so I suggest reading that post for reference.

Let's install zfs-autobackup on the FreeBSD server:

pkg install py311-zfs-autobackup mbuffer

Backups for other servers using BorgBackup

When I don't have ZFS available or need to perform a file-based backup (all or partial), I use a different technique. BorgBackup backups are primarily "push" based, meaning the client will connect to the backup server. This is not optimal or the most secure approach, as the backup server should, in theory, be hardened. Even when protecting everything via VPN, the risk remains that a compromised server could connect to its backup server and alter or delete the backups. I have seen this happen in ransomware cases (especially in the Microsoft world), and so I try to be careful to minimize this type of problem, mainly through snapshots of the backup server (an operation that will be described later).

To ensure the highest possible security, I create a FreeBSD jail on the backup server for each server I need to back up. The advantage of this approach is the complete separation of all servers from each other. By using a regular user inside a jail, a compromised server that connects to its backup server would only be able to reach its own backups, as it would be confined to a user account and, even if it managed to escalate privileges, still be inside a jail.

Let's say, for example, we want to back up a server called "ServerA" (great imagination, I know). We create a dedicated jail on the backup server:

# Create a new VNET jail named "servera" attached to our bridge
bastille create -B servera 14.3-RELEASE 192.168.0.101/24 bridge0

BastilleBSD will automatically set the host's gateway for the jail. In our case, this is incorrect, so we need to modify it and set the jail's gateway to 192.168.0.1 in the /usr/local/bastille/jails/servera/root/etc/rc.conf file:

# ...
defaultrouter="192.168.0.1"
# ...

Restart the jail and connect to it:

bastille restart servera
bastille console servera

Now, inside the jail, we install borgbackup:

pkg install py311-borgbackup

BorgBackup doesn't run a daemon; it's launched by the remote server (which sends its data to the backup server), so it's important that the installed version is compatible with the one on the remote host.

Since we'll be using SSH, let's enable it:

service sshd enable
service sshd start

And create a non-privileged user for this purpose:

# The 'adduser' utility provides an interactive way to create a user.
root@servera:~ # adduser
Username: servera
Full name: Server A
Uid (Leave empty for default): 
Login group [servera]: 
Login group is servera. Invite servera into other groups? []: 
Login class [default]: 
Shell (sh csh tcsh nologin) [sh]: 
Home directory [/home/servera]: 
Home directory permissions (Leave empty for default): 
Use password-based authentication? [yes]: 
Use an empty password? (yes/no) [no]: 
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]: 
Username    : servera
Password    : <random>
Full Name   : Server A
Uid         : 1001
Class       : 
Groups      : servera 
Home        : /home/servera
Home Mode   : 
Shell       : /bin/sh
Locked      : no
OK? (yes/no) [yes]: yes
adduser: INFO: Successfully added (servera) to the user database.
adduser: INFO: Password for (servera) is: JIkdq8Ex

The user is created and can receive SSH connections. After setting everything up, I suggest disabling password-based login in the jail's SSH configuration, using only public key authentication.

As mentioned, the biggest risk of a "push" backup is that a compromised client could access the backup server and delete or encrypt the backup history, rendering it useless.

To drastically mitigate this risk, we can configure SSH to force the client to operate in a special Borg mode called append-only. In this mode, the SSH key used by the client will only have permission to create new archives, not to read or delete old ones. However, this approach could complicate some client-side operations (like mount, prune, etc.), forcing them to be done on the server. For this reason, I won't describe it in this setup, "limiting" myself to taking snapshots of the repositories. It can be a very good practice, so I recommend considering it.

Let's initialize the BorgBackup repository. In this example, for simplicity, I won't set up repository encryption. If the jails are on an encrypted dataset or GELI-encrypted disks, there will still be data encryption on the disks, but there will be no protection against someone who could physically access the server while the disks are mounted. As usual, security is like an onion: every layer helps. Personally, I suggest enabling and using it ALWAYS.

# Switch to the new user
su -l servera
# Initialize a new Borg repo named "servera" with no encryption (for this example)
borg init -e none servera

The jail is ready, but it's unreachable from the outside. There are two ways to make it accessible:

Install a VPN system inside the jail itself. Using tools like Zerotier or Tailscale (which don't need to expose ports) will immediately create the conditions to connect to the jail, which will remain inaccessible from the outside. As the jail is a VNET jail, we're free to choose any of the supported VPN technologies.
Expose a port on the backup server, i.e., on the host, to allow external connections. Many advise against this path as they consider it less secure. It is, but sometimes we don't have the luxury of installing whatever we want on the server we're backing up.

To expose the port, go back to the host and modify the /etc/pf.conf file, creating the rdr and pass rules to let packets in:

# ...
# Redirect incoming traffic on port 1122 to the jail's SSH port (22)
rdr on $ext_if inet proto tcp from any to any port = 1122 -> 192.168.0.101 port 22
# ...
# Allow incoming traffic on port 1122
pass in inet proto tcp from any to any port 1122 flags S/SA keep state

Reload the pf configuration:

service pf reload

The jail will now be reachable on the server's public IP, on port 1122. Obviously, this port number is for illustrative purposes, and I used from any, but for better security, you should replace any with the IP address of the server that will be connecting to perform the backup.

By repeating this process for each server and creating a separate jail for each, you can have isolated jails in separate datasets with their backups, potentially setting space limits using ZFS quotas.

It's important to remember that backing up a live filesystem (i.e., without a snapshot or dumps) has a very high probability of being impossible to restore completely. Databases hate this approach because files will change while being copied and tend to get corrupted. Of course, it depends on the nature of the data (a backup of a static website will have no issues, but a WordPress database probably will), but it's crucial to think about a technique to snapshot the filesystem before proceeding. For example, I have already written about how to create snapshots on FreeBSD with UFS in a previous article: FreeBSD tips and tricks: creating snapshots with UFS.

I will cover other operating systems in a future, dedicated post.

Proxmox Backup Server in a Dedicated VM

Starting with version 4.0 (which is still in beta at the time of this writing), Proxmox Backup Server (PBS) supports storing its data in an S3 bucket. This is excellent news as it decouples the server from the data. There are great open-source S3 implementations, like Minio or SeaweedFS, which allow for clustering, replication, etc. In this "simple" case, we will install Proxmox Backup Server in a small VM, while for the data, we'll install Minio in a native FreeBSD jail. The advantage is undeniable: the VM will only serve as an "intermediary", but the data will rest directly on the FreeBSD host's dataset, natively. It will also be possible to specify other external endpoints, other repositories, etc.

As a philosophy, I tend not to use external providers unless for specific needs, so installing Minio in a jail is a perfect solution to manage this situation.

Let's install PBS by downloading the ISO from their website (https://enterprise.proxmox.com/iso/) - at this moment, the version that supports this setup is 4.0 Beta.

The directory to download to is the vm-bhyve ISOs directory. It's not strictly necessary, but it's useful for not "losing" it somewhere. So, go to the directory and download it:

cd /zroot/VMs/.iso
fetch https://enterprise.proxmox.com/iso/proxmox-backup-server_4.0-BETA-1.iso

Now let's create a VM with vm-bhyve. We can start from the Debian template, but we'll make some modifications to optimize performance. In this example, I'm giving it 30 GB of disk space, 2 GB of RAM, and 2 cores.

If you want to store all backups inside the VM, you'll need to size the virtual disk correctly (or create and attach another one). In this case, I will focus on the "clean" VM that will store its data on a dedicated jail with Minio.

vm create -t debian -s 30G -m 2G -c 2 pbs

Once the empty VM is created, let's modify its options:

vm configure pbs

We will modify the VM to be UEFI and to use the NVME disk driver - bhyve performs significantly better on NVME than virtio, as previously tested:

loader="uefi"
cpu="2"
memory="2G"
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"

Fortunately, the Proxmox team has provided for the installation of the Backup Server on devices without a graphical interface, so the boot menu will allow installation via serial console. Let's launch the installation and connect to the virtual serial console:

cd /zroot/VMs/.iso
vm install pbs proxmox-backup-server_4.0-BETA-1.iso
vm console pbs

Select the installation via Terminal UI (serial console) and proceed normally as if it were a physical host, assigning an IPv4 address from the 192.168.0.x range (in this example, I'll use 192.168.0.3).

This way, the Proxmox Backup Server will run in a VM, with the ability to take snapshots before updates, etc., without any worries.

Once the installation is complete, PBS will reboot and listen on port 8007 of its IP. Again, as with the jails, we have two options: install a VPN system within the VM itself (thus exposing it automatically only on that VPN - generally a more secure operation) or expose port 8007 on the server's public IP.

In the latter case, add the relevant lines to the /etc/pf.conf file on the FreeBSD backup server:

# ...
# Redirect incoming traffic on port 8007 to the PBS VM's web interface
rdr on $ext_if inet proto tcp from any to any port = 8007 -> 192.168.0.3 port 8007
# ...
# Allow that traffic to pass
pass in inet proto tcp from any to any port 8007 flags S/SA keep state

Reload the pf configuration:

service pf reload

The PBS VM configuration is complete. If you chose to use the PBS's internal disk as a repository, no further operations are necessary (other than the normal repository creation, etc., within PBS).

In this case, however, we will use a different approach.

Creating a Minio Jail as a Data Repository for PBS

This approach, in my opinion, has a number of important advantages. The first is that Minio will run in a dedicated jail on the host, at full performance, and will store the data directly on the physical ZFS datapool, thus removing any other layer in between. This jail could potentially be moved to other hosts (by connecting PBS and the jail via VPN or public IP), made redundant thanks to all of Minio's features, etc. Another solution I am successfully testing (in other setups) is SeaweedFS.

Let's create a dedicated jail with Minio and put it on the bridge, so that PBS can access it on the LAN.

bastille create -B minio 14.3-RELEASE 192.168.0.11/24 bridge0

If not configured directly, BastilleBSD will use the host's gateway for the jail, which is incorrect in this case. So let's go modify it and restart the jail. Enter the jail with:

bastille console minio

And modify the /etc/rc.conf file to have the correct gateway (following the example addresses):

# ...
ifconfig_vnet0=" inet 192.168.0.11/24 "
defaultrouter="192.168.0.1"
# ...

Exit the jail and restart it:

bastille restart minio

Enter the jail and install Minio:

bastille console minio
pkg install -y minio

Minio is already able to start, but PBS, even on the LAN, wants an encrypted connection. Fortunately, there's a handy tool that can generate the certificates for us:

# Download the certgen tool
fetch https://github.com/minio/certgen/releases/latest/download/certgen-freebsd-amd64

# Make it executable and run it for our jail's IP
chmod a+rx certgen-freebsd-amd64
./certgen-freebsd-amd64  -host "192.168.0.11"

# Create the necessary directories and set permissions
mkdir -p /usr/local/etc/minio/certs
cp private.key public.crt /usr/local/etc/minio/certs/
chown -R minio:minio /usr/local/etc/minio/certs/

Let's view the certificate's fingerprint. Since it's self-signed, we'll need it for PBS later. For security reasons, PBS will ask for the fingerprint of non-directly verifiable certificates. Run the following command and take note of the result:

openssl x509 -in /usr/local/etc/minio/certs/public.crt -noout -fingerprint -sha256

At this point, enable and configure Minio in /etc/rc.conf. WARNING: The username and password (access key and secret) used in this example are insecure and for testing purposes only. It is strongly recommended to use different values:

# Enable Minio service
minio_enable="YES"
# Set the address for the Minio console
minio_console_address=":8751"
# Set the root user and password as environment variables
minio_env="MINIO_ROOT_USER=testaccess MINIO_ROOT_PASSWORD=testsecret"

Start Minio:

service minio start

If everything went correctly, Minio is now running (with its certificates) and ready to receive connections.

It's now time to create the bucket(s) that PBS will use. There are several ways to do this, but to test that everything is working and to configure PBS, I suggest connecting via an SSH tunnel.

# Create an SSH tunnel from your local machine to the backup server
# Port 8007 is forwarded to the PBS web UI
# Port 8751 is forwarded to the Minio console
ssh user@backupServerIP -L8007:192.168.0.3:8007 -L8751:192.168.0.11:8751

This way, we'll create a tunnel between the FreeBSD backup server and our workstation, mapping 127.0.0.1:8007 to 192.168.0.3:8007 (the PBS web interface) and 127.0.0.1:8751 to 192.168.0.11:8751 (the Minio console port).

Now, connect to https://127.0.0.1:8751, enter the credentials specified in /etc/rc.conf, and create a bucket.

Once the bucket is created, you can configure PBS to use it. Connect to PBS via https://127.0.0.1:8007 and go to S3 Endpoints. Set a name, use 192.168.0.11 as the IP and 9000 as the port, enter the access and secret keys, and the certificate fingerprint we generated earlier. Enable "Path Style" or it will not work.

Then go to Datastores and add it, as you would for any other S3 datastore, by specifying the created bucket and a local directory where the system will keep its cache.

If everything was set up correctly, PBS will create its structure in the bucket, and from that moment on, you can use it. Always keep in mind that this is still a "technology preview", so there may be issues, but from my tests, it is sufficiently reliable.

Taking Local Snapshots of Backups

One of the most common techniques used in ransomware attacks is to also delete or encrypt backups. They often use automated methods, relying on the fact that many (too many!) consider a "backup" to be a simple copy of files to a network share. However, it's not impossible that, in specific cases, they might compromise the machine and connect to the backup server. This is nearly impossible with a "pull" type backup (like the one managed by zfs-autobackup) but is still possible with the "push" approach, which involves using BorgBackup or similar tools.

This happened to one of my clients once - in that case, the problem originated internally, from an employee who wanted to cover up his mistake, inadvertently creating a disaster - but that will be material for another post.

Fortunately, the client had a system that solved the problem: thanks to ZFS, we can have local snapshots on the backup server, which are invisible and inaccessible to the production server. Since we have already installed zfs-autobackup, it's easy to use it for this purpose as well. I've already talked about this in a previous article and won't rewrite the steps here. Just consult that article, keeping in mind that in this case, it's not advisable to snapshot all the datasets on the backup server (the space would grow exponentially), but only those at risk. In the cases analyzed in this post, this applies only to the push part, as PBS will also be accessible only from the Proxmox servers and not from the VMs they contain. If, in this case too, you don't trust those who manage the Proxmox servers, just set up snapshots for the Minio jail as well.

Conclusion

This long post aims to analyze, in a general way, how I believe one can manage reasonably secure backups of their data. Obviously, there are many variables, additional precautions, possible optimizations, hardening, etc., that must be studied on a case-by-case basis. There are old rules, new rules, old and new philosophies. Recently, many people who have embraced the cloud have often stopped thinking about backups, only to realize it when something happens and the data has, indeed, vanished... into the clouds.

In this post, I have generically covered the setup of the backup server, and this demonstrates how FreeBSD, thanks to its features, can be considered an ideal platform for this type of task.

In the next articles in this series, I will examine the client side, i.e., how to structure them for a sufficiently reliable backup, and how to monitor everything - because I've seen this too: people resting easy because the backup was supposedly running every night, but in fact, the backup had been failing every night for more than 4 years.

Stay Tuned and stay...backupped!

Make Your Own Backup System – Part 1: Strategy Before Scripts

Stefano Marinelli — Fri, 18 Jul 2025 09:00:00 +0200

Backup: Beyond the Simple Copy

For as long as I can remember, backup is something that has been underestimated by far too many people. Between flawed techniques, "Schrödinger's backups" (i.e., never tested, thus both valid and invalid at the same time), and conceptual errors about what they are and how they work (RAID is not a backup!), too much data has been lost due to deficiencies in this area.

Nowadays, backup is often an afterthought. Many rely entirely on "the cloud" without ever asking how - or if - their data is actually protected. It's a detail many overlook, but even major cloud providers operate on a shared responsibility model. Their terms often clarify that while they secure the infrastructure, the ultimate responsibility for protecting and backing up your data lies with you. By putting everything "in the cloud", on clusters owned by other companies, or on distributed Kubernetes systems, backup seems unnecessary. When I sometimes ask developers or colleagues how they handle backups for all this, they look at me as if I'm speaking an archaic, unknown, and indecipherable language. The thought has simply never crossed their minds. But data is not ephemeral; it must be preserved in every way possible.

I've always had a philosophy: data must always be restorable (and as quickly as possible), in an open format (meaning you shouldn't have to buy anything to restore or consult it), and consistent. These points may seem obvious, but they are not.

I have encountered various scenarios of data loss:

Datacenters destroyed by fire – I had 142 servers there, but they were all restored in just a few hours.
Server rooms flooded.
Servers destroyed in earthquakes, often due to collapsing walls.
Increasing incidents of various ransomware attacks.
Intentional damage by entities seeking to create problems.
Mistakes made by administrators, which can happen to anyone.

The risk escalates for servers connected to the internet, like e-commerce and email servers. Here, not only is data integrity crucial, but so is the uninterrupted operation of services. This series of posts will revisit some of my old articles to explain my core ideas on the subject and describe, at least in part, my primary techniques.

Many consider a backup to be a simple copy. I often hear people say they have backups because they "copy the data", but this is often wrong and extremely dangerous, providing a false sense of security. Copying the files of a live database is an almost useless operation, as the result will nearly always be impossible to restore. It is essential to at least perform a proper dump and then transfer that file. Yet, many people do this and will only realize their mistake when they face an emergency and need to restore.

The Backup Plan: Asking the Right Questions

Before touching a single file, you must start with a plan, and that plan starts with asking the right questions:

"How much risk am I willing to take? What data do I need to protect? What downtime can I tolerate in case of data loss? What type and amount of storage space do I have available?"

The first question is particularly critical. A common but risky approach is to store a backup on the same machine that requires backing up. While convenient, this method fails in the event of a machine failure. Even relying on a classic USB drive for daily backups is not foolproof, as these devices are as susceptible to failure as any other hardware component. And contrary to popular belief, even high-end uninterruptible power supplies (UPS) are not immune to catastrophic failures.

Thus, the initial step is to establish a management plan, balancing security and cost. The safest backup is often the one stored farthest from the source machine. However, this approach introduces challenges related to space and bandwidth. While local area network (LAN) backups are relatively straightforward, off-network backups involve additional connectivity considerations. This might lead to a compromise on the amount of data backed up to maintain operational speed during both backup and recovery processes.

Safety doesn't always equate to practicality. For instance, with a 200 MBit/sec connection and 2 TB of backup data, the recovery time could be significant. However, if the goal is not rapid restorability but simply ensuring the data is available, the safest backup is often the one closest to us. That is, a backup we can "touch", disconnect, and consult even when offline.

Therefore, it is essential to develop a backup policy tailored to specific needs, keeping in mind that no 'perfect' solution exists.

The Core Decision: Full Disk vs. Individual Files

When planning a backup strategy, one key decision is whether to back up the entire disk or just individual files. Or both of them. Each approach has its pros and cons.

Entire Disk (or Storage) Backup

Advantages:

Complete Recovery: Restoring a full disk backup can quickly revert a system to its exact previous state, boot loader included.
Integration in Virtualization Systems: If your VM is a single file on a filesystem like ZFS or btrfs, you can simply copy that file (after taking a snapshot) to get a complete backup of the VM. Solutions like Proxmox offer easy management of full disk backups, accessible via command line or web interface.
Flexibility in Virtual Environments: Products like the Proxmox Backup Server offer the ability to recover individual files from a full backup, combining the benefits of both approaches.

Disadvantages:

Downtime for Physical Machines: Often, it's necessary to shut down the machine to create a full disk backup, leading to operational interruptions. A hybrid approach, if the physical host is running FreeBSD for example, is to take a snapshot and copy all the host's datasets. The restore process, however, will require some manual operations.
Large Space Requirements: Full disk backups can consume substantial space, including unnecessary data.
Potential Slowdowns and Compatibility Issues: The backup process can be slow and may encounter issues with non-standard file system configurations.

Individual File Backup

While it might seem simpler, backing up individual files can get complicated.

Advantages:

Basic Utility Use: Possible with standard system utilities like tar, cp, rsync, etc.
Granular Backups: Allows for backing up specific files and comparing them to previous versions.
Delta Copying: Only modified parts of the files are backed up, saving space and reducing data transfer.
Portability and Partial Recovery: Files can be moved individually and partially restored as needed.
Compression and Deduplication: These features are often available at the file or block level.
Operational Continuity: Can be done without shutting down the machine.

Disadvantages:

Storage Space Requirements: Simple copy solutions might require significant storage.
Need for File System Snapshot: For efficient and consistent backups, a snapshot (like native ZFS snapshots, btrfs, LVM Volume snapshots, or Microsoft's VSS) is highly recommended before copying.
Hidden Pitfalls: Issues may not become apparent until a backup is needed. And by then, it may be too late.

The Key to Consistency: The Power of Snapshots

Backing up a "live" file system involves a "start" and an "end" moment. During this time, the data can change, leading to fatal inconsistencies. I've encountered such issues in the past: a large MySQL database was compromised, and I was tasked with its recovery. I confidently took the client's last file-based backup and restored various files (not a native dump). Unsurprisingly, the database failed to restart. The large data file had changed too much between the start and end of the copy, rendering it inconsistent. Fortunately, I also had a proper dump, so I managed to recover from that.

The issue is evident: backing up a live file system is risky. An open database, even a basic one like a browser's history, is highly likely to get corrupted, making the backup useless.

The solution is to create a snapshot of the entire file system before beginning the backup. This approach freezes a consistent "point-in-time" view of the data. To date, using snapshots, I have managed to recover everything.

Here are the methods I've explored over the years:

Native File System Snapshot (e.g., BTRFS or ZFS): If your file system inherently supports snapshots, it's wise to use this feature. It's likely to be the most efficient and technically sound option.
LVM Snapshot: For those using LVM, creating a snapshot of the logical volume is a viable approach. This method can lead to some space wastage and, while I still use it, has occasionally caused the file system to freeze during the snapshot's destruction, necessitating a reboot. This has been a rare but recurring issue across different hardware setups, especially under high I/O load.
DattoBD: I've tracked this tool since its inception. I used it extensively in the past, but I sometimes faced stability issues (kernel panics or the inability to delete snapshots, forcing a reboot). For snapshots with Datto, I often use UrBackup scripts, which are convenient and efficient.

The Architecture: Push or Pull?

A longstanding debate among experts is whether backups should be initiated by the client (push) or requested by the server (pull). In my view, it depends.

Generally, I prefer centralized backup systems on dedicated servers, maintained in highly secure environments with minimal services running. Therefore, I lean towards the "pull" method, where the server connects to the client to initiate the backup.

Ideally, the backup server should not be reachable from the outside. It should be protected, hardened, and only be able to reach the setups it needs to back up. The goal is to minimize the possibility that the backup data could be compromised or deleted in case the client machine itself is attacked (which, unfortunately, is not so rare).

This is not always possible, but there are ways to mitigate this problem. One way is to ensure that machines that must be backed up via "push" (i.e., by contacting the backup server themselves) can only access their own space. More importantly, the backup server, for security reasons, should maintain its own filesystem snapshots for a certain period. In this way, even in the worst-case scenario (workload compromised -> connection to backup server -> deletion of backups to demand a ransom), the backup server has its own snapshots. These server-side snapshots should not be accessible from the client host and should be kept long enough to ensure any compromise can be detected in time.

My Guiding Principles for a Good Backup System

Over the years, I've favored having granular control over backups, often finding the need to recover specific files or emails accidentally deleted by clients. A good backup system, in my opinion, should possess these key features:

Instant Recovery and Speed: The system should enable quick recovery and operate at a high processing speed.
External Storage: Backups must be stored externally, not on the same system being backed up. Still, local snapshots are a good idea for immediate rollbacks.
Security: I avoid using mainstream cloud storage services like Dropbox or Google Drive for primary backups. Own your data!
Efficient Space Management: This includes features like compression and deduplication.
Minimal Invasiveness: The system should require minimal additional components to function.

Conclusion: What's Next

The approach to backup is varied, and in this series, I will describe the main scenarios I usually face. I will start with the backup servers and their primary configurations, then move on to the various software and techniques I use.

But that will begin with the next post, where I'm talk about building the backup server which, of course, will be powered by FreeBSD - like all my backup servers for the last 20 years.

Managing ZFS Full Pool Issues with Reserved Space

Stefano Marinelli — Thu, 28 Nov 2024 20:25:00 +0100

Yesterday morning, I received a panicked call from a developer:
"I accidentally filled up the storage, and now I can't perform any operations! My ZFS pool is full!"

I immediately reassured them because I had anticipated this kind of issue. One of the things I almost always do when managing ZFS file systems is to reserve space in a specially created dataset.

This is because ZFS, like all CoW (Copy-on-Write) file systems, can find itself unable to free up space when completely full. By using reserved space, I can always free it up and delete other data, restoring the system to normal operations.

To reserve space, simply create a dataset and assign it a reserved size. Of course, this dataset should not be used for anything else; otherwise, the entire purpose would be defeated.

To create it and reserve space, you only need two simple commands. For example:

zfs create zroot/reserved
zfs set reservation=5G zroot/reserved

This creates the dataset and assigns it 5 GB of reserved space.

Here’s the situation before the operation:

zfs list zroot
NAME    USED  AVAIL  REFER  MOUNTPOINT
zroot  3.02G   109G    96K  /zroot

And here’s the situation after:

zfs list zroot
NAME    USED  AVAIL  REFER  MOUNTPOINT
zroot  8.02G   104G    96K  /zroot

As you can see, the 5 GB are removed from the available space and marked as used, but they are actually empty.

In case of a full file system, you can delete this dataset (or reduce its size) to return to normal file system operation.

Even with this technique, I still recommend not filling ZFS pools beyond 80% of their capacity, as performance degrades significantly past that point.

Migrating Windows VMs from Proxmox BIOS/KVM to FreeBSD UEFI/bhyve

Stefano Marinelli — Fri, 15 Nov 2024 10:57:00 +0100

A client of mine has several Windows Server VMs, which I had not migrated to FreeBSD/bhyve until a few weeks ago. These VMs were originally installed with the traditional BIOS boot mode, not UEFI, on Proxmox. Fortunately, their virtual disks are on ZFS, which allowed me to test and achieve the final result in just a few steps.

This is because Windows VMs (server or otherwise) often installed on KVM (Proxmox, etc.), especially older ones, are non-UEFI, using the traditional BIOS boot mode. bhyve doesn’t support this setup, but Windows allows changing the boot mode, and I could perform the migration directly on the target FreeBSD server.

Setting Up the Network

Before starting, as usual in my setups, I manually created the bridge and configured a few pf rules for outbound NAT, which these servers need. Let’s begin with the modifications to /etc/rc.conf:

cloned_interfaces="bridge0"
ifconfig_bridge0="inet 192.168.33.1 netmask 255.255.255.0"
gateway_enable="YES"
pf_enable="YES"

Next, for the pf.conf:

ext_if="igb0"

set block-policy return
set skip on bridge0
set skip on lo0

# Allow the VMs to connect to the outside world
nat on $ext_if from {192.168.33.0/24} to any -> ($ext_if)

block in all
antispoof for $ext_if inet
pass in inet proto icmp
pass out quick keep state
pass in inet proto tcp from any to any port ssh flags S/SA keep state

To ensure everything is working as expected, I recommend rebooting the server. It’s not strictly necessary since you could reload network configurations and start pf without a reboot, but I prefer a few extra reboots in these cases. Why? Because system administrators are often afraid to reboot production servers, unsure if they will come back up smoothly. A configuration error could prevent a successful reboot, and this would be a significant issue if the server is already in production. I’ve faced this situation myself-many times! To mitigate this, I make frequent reboots, especially after changing network configurations, ensuring everything starts correctly on reboot.

Installing and Configuring vm-bhyve

Next, I installed vm-bhyve on the target FreeBSD host and configured it:

pkg install vm-bhyve-devel bhyve-firmware
zfs create zroot/VMs
sysrc vm_enable="YES"
sysrc vm_dir="zfs:zroot/VMs"
vm init
cp /usr/local/share/examples/vm-bhyve/* /zroot/VMs/.templates/
vm switch create -t manual -b bridge0 public

Usually, I enable the serial console on tmux, but in this case, I’ll skip that since Windows VMs need a graphical console. If the FreeBSD server is not on your local network, I suggest connecting via SSH with port forwarding for the necessary VNC ports (e.g., -L5900:127.0.0.1:5900) to connect to bhyve via SSH tunnel. Never expose the VNC port directly!

Creating the VM Template

Now, I created an “empty” VM using the “Windows” template with vm-bhyve and adjusted the configuration afterward:

vm create -t windows -s 1G -m 16G -c 4 vm115

I created a virtual disk of 1 GB because it will be replaced by the dataset sent from the current production server, so it’s a fictitious size. At that point, I deleted the disk0 image file:

rm /zroot/VMs/vm115/disk0.img

Modifying the VM Configuration

Next, I modified the VM configuration as follows:

Changed the disk configuration to use a zvol, simplifying the send-receive operation from the original Proxmox host.
In some cases, older Windows installations may not support the nvme driver. If this happens, use ahci-hd instead, as after the conversion Windows may fail to boot and display a BSOD.
Changed the network adapter driver to virtio-net (since it was already installed on the Proxmox VM).
Set the network adapter MAC address to match the original VM.

After running vm configure vm115, the configuration should look something like this:

loader="uefi"
graphics="yes"
xhci_mouse="yes"
cpu="4"
memory="16G"

ahci_device_limit="8"

network0_type="virtio-net"
network0_switch="public"

disk0_type="nvme"
disk0_name="disk0"
disk0_dev="sparse-zvol"

utctime="no"
uuid="myUUID"
network0_mac="theProxmoxVirtualMachineMacAddress"

Copying the Virtual Disk

I took a snapshot of the virtual disks of the VMs on the original server and copied them to the target server:

zfs snapshot vmpool/vm-115-disk-0@toSend01
zfs send -Rcv vmpool/vm-115-disk-0@toSend01 | mbuffer -m 128M | ssh user@FreebsdHostIP "zfs receive -F zroot/VMs/vm115/disk0"

The VM’s disk will be transferred to the target host.

Converting BIOS to UEFI

At this point, bhyve expects a UEFI VM, but the VM we just copied is BIOS-based. In other words, it won’t boot as is.

Luckily, Microsoft provides a tool to convert partitions and change the server from BIOS to UEFI. Of course, I would only use such a tool on a snapshot, and while I could do it directly on the source server, I prefer minimizing downtime and avoiding touching the original server. I want to perform the conversion directly on the FreeBSD host, which also makes disaster recovery easier, allowing me to manage everything on FreeBSD without first restoring the VM on a Proxmox server.

The first step is to download a Windows installation ISO (in my case, Windows Server). I downloaded the ISO from here: Windows Server Evaluation Center.

Once ready, launch the ISO as if you were installing Windows on the VM:

vm install vm115 SERVER_EVAL_x64FRE_en-us.iso

Running the Windows Repair Tool

Now the VM will boot. Connect via VNC (if using VNC tunneling, connect to 127.0.0.1:5900) to proceed. The ISO will prompt you to press a key to boot from the CDROM-go ahead and do that.

Proceed as if you were repairing your installation ("Repair your computer"), then select Troubleshoot -> Command Prompt.

Once at the prompt, run a validation check first (to avoid making changes if the disk has any unusual configurations):

mbr2gpt /validate /disk:0

If the validation is successful, proceed with the actual conversion:

mbr2gpt /convert /disk:0

Windows will convert the partition and indicate success at the end.

Booting the VM

Close the command prompt and shut down the VM. Everything should be ready now-easy peasy.

Boot the VM with:

vm start vm115

Connecting via VNC to the VM will bring up the Windows boot screen. In case of BSOD, change the nvmedriver to ahci-hdand it should be working. If the snapshot was taken while the VM was powered on, it may perform a disk check. If everything went as planned, you should see the Windows login screen.

Additionally, Windows will likely require reactivation of the VM, as it will detect hardware changes.

Final Considerations

At this point, you need to decide whether to continue with this VM or resynchronize it with the original (e.g., if the contents have changed). In the first case, no further action is needed. In the second case, the proper procedure would be to power off the original VM, create a new snapshot, roll back the snapshot on the FreeBSD host, transfer it incrementally-a process I’ve described in a previous post-and then redo the conversion.

To configure the VM to boot automatically when the FreeBSD host boots, add the necessary settings to /etc/rc.conf, as described in a previous article.

From Proxmox to FreeBSD - Story of a Migration

Stefano Marinelli — Mon, 21 Oct 2024 07:33:00 +0200

I have been managing a client's server for several years. I inherited a setup (actually, partly done by me at the time, but following the directions of their internal administrator) based on Proxmox. Originally, the VM disks were qcow2 files, but over time and with Proxmox updates, I managed to create a ZFS pool and move them onto it. For backups, I continued to use Proxmox Backup Server (even though virtualized on bhyve) - a solution we've been using for several years.

The existing VMs/containers were:

An LXC container based on Debian, with Apache as a reverse proxy. The choice of Apache was mainly tied to specific configurations from the company that produces the underlying software. It has always done an excellent job.
A VM running OPNsense - the choice was related to direct use by the client, as it allows easy creation/modification of users for the internal VPN. Some users can access via VPN (OpenVPN) and perform specific operations, such as SSH connections. It has always worked well and is up to date.
Two VMs based on Rocky Linux 9 - both with components of the management software in Java, with the reverse proxy directing requests based on the requested site.
Two VMs based on Rocky Linux 8 - these also have parts of the management software and the databases. These machines are used both by providing specific URLs and by the other two as databases.

The load is not particularly high, and the machines have good performance. Suddenly, however, I received a notification: one of the NVMe drives died abruptly, and the server rebooted. ZFS did its job, and everything remained sufficiently secure, but since it's a leased server and already several years old, I spoke with the client and proposed getting more recent hardware and redoing the setup based on a FreeBSD host.

Unfortunately, I cannot change the setup of the VMs. In my opinion, there are no contraindications since the databases are PostgreSQL and the management applications are Java applications with Tomcat. However, the software house only certifies that type of setup, and considering they are the ones performing updates and fixes, it's appropriate to maintain the setup they suggest. It's a technically sound choice in my opinion, so I can't say anything negative.

Acquiring New Hardware

The first step was, of course, to get the new server. It took a few days because I requested ECC RAM (for a small price variation), and the time extended. As soon as the physical server was ready, I got to work.

I installed FreeBSD 14.1-RELEASE, root on ZFS, creating a mirror setup between the NVMe drives. The I/O is not particularly heavy in this setup, but I don't want to waste potential: even if I can reduce the length of an operation by a few seconds, I see no reason not to do it.

As the first step, I decided to manually create the bridge to which I will connect both the VMs and the reverse proxy, which will be installed inside a FreeBSD jail.

vm-bhyve, the tool I use to manage VMs, allows creating bridges, but I prefer to manage it manually and maintain more complete control over everything. I will also enable pf.

I decided to use zstd as the compression algorithm and disable atime:

zfs set compression=zstd zroot
zfs set atime=off zroot

I then modified the /etc/rc.conf file as follows:

cloned_interfaces="bridge0 lo1"
ifconfig_lo1_name="bastille0"
ifconfig_bridge0="inet 192.168.33.1 netmask 255.255.255.0"
gateway_enable="YES"
pf_enable="YES"
bastille_enable="YES"

I then updated FreeBSD by running the usual command:

freebsd-update fetch install

Configuring the Firewall

I configured the firewall with a basic setup:

ext_if="igb0"

set block-policy return
set skip on bridge0

table <jails> persist
nat on $ext_if from {192.168.33.0/24} to any -> ($ext_if)
nat on $ext_if from <jails> to any -> ($ext_if:0)

# nginx-proxy - to the jail
rdr on $ext_if inet proto tcp from any to PUBLIC_IP port = 80 -> 192.168.33.254 port 80
rdr on $ext_if inet proto tcp from any to PUBLIC_IP port = 443 -> 192.168.33.254 port 443

# opnsense
rdr on $ext_if inet proto tcp from any to PUBLIC_IP port = 1194 -> 192.168.33.253 port 1194
rdr on $ext_if inet proto udp from any to PUBLIC_IP port = 1194 -> 192.168.33.253 port 1194

rdr-anchor "rdr/*"

block in all
antispoof for $ext_if inet
pass in inet proto icmp
pass out quick keep state
pass in inet proto tcp from any to any port {http,https} flags S/SA keep state
pass in inet proto {tcp,udp} from any to any port 1194 flags S/SA keep state
# This will be closed at the end of the setup and will be allowed only via VPN
pass in inet proto tcp from any to any port ssh flags S/SA keep state

Once finished, I rebooted everything to load the new kernel. No problems.

Installing Necessary Tools

I then installed some useful packages:

pkg install tmux py311-zfs-autobackup mbuffer rsync vm-bhyve-devel edk2-bhyve grub2-bhyve bastille

I created the datasets for the jails and VMs:

zfs create zroot/bastille
zfs create zroot/VMs

I then started configuring everything. First, BastilleBSD: I modified /usr/local/etc/bastille/bastille.conf by adding:

## ZFS options
bastille_zfs_enable="YES"
bastille_zfs_zpool="zroot/bastille"

Then I enabled and configured vm-bhyve, enabling the serial console on tmux:

sysrc vm_enable="YES"
sysrc vm_dir="zfs:zroot/VMs"
vm init
cp /usr/local/share/examples/vm-bhyve/* /zroot/VMs/.templates/
vm switch create -t manual -b bridge0 public
vm set console=tmux

Now I bootstrapped FreeBSD 14.1-RELEASE on BastilleBSD:

bastille bootstrap 14.1-RELEASE update

Setting Up the Reverse Proxy Jail

I then started by creating the jail for the reverse proxy:

bastille create -B apache 14.1-RELEASE 192.168.33.254/24 bridge0

Once created, I had to modify the default gateway of the jail because by default it is set to that of the host. It was enough to set 192.168.33.1 in /usr/local/bastille/jails/apache/root/etc/rc.conf.

The configuration of Apache will not be described here as it is closely dependent on the setup, but this jail can reach (in bridge mode) all the VMs that will be placed on the same bridge.

Migrating the VMs

For migrating the VMs, I decided to proceed as follows:

Updating the operating systems of the original VMs - to have the latest version of the kernel and all userland. The VMs are not UEFI, so it will be necessary to have the exact names of the kernel and initrd as they will need to be specified in the configuration file of vm-bhyve.
Creating templates of the VMs with vm-bhyve.
Snapshotting the VM disks and initial transfer to the new server with the VMs still running.
Shutting down all the source VMs.
Creating a new snapshot and performing an incremental transfer.
Adjusting configurations and changing DNS.

I updated everything using dnf, and rebooted when there was also a kernel update. I took note of the versions to use, namely vmlinuz-4.18.0-513.18.1.el8_9.x86_64 and initramfs-4.18.0-513.18.1.el8_9.x86_64.img for Rocky Linux 8.10, and vmlinuz-5.14.0-362.18.1.el9_3.0.1.x86_64 and initramfs-5.14.0-362.18.1.el9_3.0.1.x86_64.img for Rocky Linux 9.4.

Migrating Rocky Linux 8.10 VMs

On the destination server, I created the VMs. I used the linux-zvol template, then modified the configurations:

vm create -t linux-zvol -s 1G -m 8G -c 4 vm100

I created a virtual disk of 1 GB because it will be replaced by the dataset sent from the current production server, so it's a fictitious size. At that point, I deleted the zvol:

zfs destroy zroot/VMs/vm100/disk0

I performed an initial snapshot of the source VMs:

zfs snapshot zfspool/vm-100-disk-0@Move01

And then I copied this first snapshot to the destination machine/VM:

zfs send -R zfspool/vm-100-disk-0@Move01 | mbuffer -s 128k -m 512M | ssh root@destMachine "zfs receive -F zroot/VMs/vm100/disk0"

At the end of the copy, you can do a test and try to boot. For the Rocky Linux 8.10 VMs, I had no problems because the /boot partition is in ext4. I then configured the VM as follows, running the command vm configure vm100:

loader="grub"
cpu="8"
memory="12G"
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="sparse-zvol"
uuid="the-uuid"
network0_mac="the-mac"
grub_run0="linux /vmlinuz-4.18.0-553.22.1.el8_10.x86_64 root=/dev/vda3"
grub_run1="initrd /initramfs-4.18.0-553.22.1.el8_10.x86_64.img"

All I needed to do is: pass to GRUB the name of the kernel and the initramfs. It is now possible to start the machine and do a test and connect to the console:

vm start vm100
vm console vm100

In theory, the machine should boot correctly. It will probably be necessary to change the network interface configurations - in my case, even keeping the virtio and the MAC address, it changed from ens18 to enp0s5 - but everything else should be fine.

At this point, if the source VM was turned off/not reachable or simply there is nothing new to synchronize, the migration is complete. Otherwise, it will be necessary to shut down the source VM, create a new snapshot, and transfer it. Let's start by shutting down the source VM and the destination one (vm stop vm100), roll back to the snapshot transferred to the destination server, create a new snapshot, and perform an incremental transfer:

zfs rollback zroot/VMs/vm100/disk0@Move01

On the source server:

zfs snapshot zfspool/vm-100-disk-0@Move02
zfs send -R -i zfspool/vm-100-disk-0@Move01 zfspool/vm-100-disk-0@Move02 | mbuffer -s 128k -m 512M | ssh root@destMachine "zfs receive -F zroot/VMs/vm100/disk0"

In this way, we have updated the destination VM and transferred only the differences, without altering the configuration of vm-bhyve.

Start the machine - if all goes well, the migration of this VM is finished.

Migrating Rocky Linux 9.4 VMs

For the Rocky Linux 9.4 VMs, the situation was more complex: the partitions were all - even /boot-in XFS. And, for some reason, the GRUB launched by bhyve cannot read from XFS. Therefore, I had to proceed differently.

I copied, using rsync, the /boot of the original VPS (in ZFS) to a directory on the FreeBSD server (specifically, on the VPS I ran the command: rsync -avhHPx --numeric-ids /boot root@FreeBSDHOST:/tmp/). In this way, I kept the files available.

I shut down the machine on the original server and copied its zvol to the new FreeBSD host using the same method as the others. At that point, I recreated the /boot partition of my VPS in ext3 - directly from the FreeBSD host:

pkg install fusefs-ext2
kldload fusefs
mkfs.ext3 /dev/zvol/zroot/VMs/vm104/disk0s1
fuse-ext2 -o force /dev/zvol/zroot/VMs/vm104/disk0s1 /mnt/
cd /mnt
rsync -avhHPx /tmp/boot/. .
umount /mnt

In this way, GRUB will be able to access the /boot partition to load the kernel and initramfs.

Here is the vm-bhyve configuration for this VM:

loader="grub"
cpu="8"
memory="12G"
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="sparse-zvol"
uuid="the-uuid"
network0_mac="the-mac"
grub_run0="linux /vmlinuz-5.14.0-427.37.1.el9_4.x86_64 root=/dev/vda3"
grub_run1="initrd /initramfs-5.14.0-427.37.1.el9_4.x86_64.img"

Launching the machine, however, it will hang during boot and, after a timeout, will ask for administrator credentials to enter a recovery shell. This is because the blkid of the /boot partition has changed, and the fstab of the VM still reports the data of the old XFS partition. In this case, I used the blkid command in the VM and copied the UUID of the new partition. Then modify the /etc/fstab file of the VM and put the new blkid, as well as changing "xfs" to "ext3". After a reboot, the system should start without problems, again with a network card to reconfigure.

This procedure worked correctly for all VMs. In this way, the storage remains on virtio-blk even if it would be optimal for the driver to be changed to nvme. To make this change, you will need to enter the VMs, create a file called /etc/dracut.conf.d/00-custom.conf:

add_drivers+=" nvme "

And regenerate the initramfs - in this way, the nvme driver will be supported at boot:

dracut --regenerate-all --force

It will now be enough to change the configurations of vm-bhyve-virtio-blk becomes nvme, and /dev/vda3 becomes /dev/nvme0n1p3, for example:

loader="grub"
cpu="8"
memory="12G"
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0"
disk0_dev="sparse-zvol"
uuid="the-uuid"
network0_mac="the-mac"
grub_run0="linux /vmlinuz-5.14.0-427.37.1.el9_4.x86_64 root=/dev/nvme0n1p3"
grub_run1="initrd /initramfs-5.14.0-427.37.1.el9_4.x86_64.img"

Migrating the OPNsense VM

For the VM with OPNsense, the procedure was even simpler. It was enough to create a VM of type freebsd-zvol and copy the disk as done for the others. In this case, I replicated the MAC address of the original virtualized network interface (Proxmox server) to ensure that the underlying FreeBSD recognizes it as the same. FreeBSD is less picky about these things.

The final vm-bhyve configuration file will be:

loader="bhyveload"
cpu="2"
memory="1G"
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="sparse-zvol"
uuid="my-uuid"
network0_mac="my-mac"

Configuring Automatic VM Startup

To ensure that the VMs all start at boot, just add them to /etc/rc.conf as follows, giving a 15-second delay between one VM and another:

[...]
vm_enable="YES"
vm_dir="zfs:zroot/VMs"
vm_list="vm100 vm101 [...] opnsense"
vm_delay="15"
[...]

Setting Up Backups

At this point, I configured external backups, every hour, similarly to how I described in a previous article. I also added a local snapshot every 15 minutes, always using zfs-autobackup. To do this, I created a new tag:

zfs set autobackup:localsnap=true zroot

Then I modified the /etc/crontab file, adding this line:

*/15    *       *       *       *       root    /usr/local/bin/zfs-autobackup localsnap --keep-source 15min3h,1h1d > /dev/null 2>&1

That is, it will perform a snapshot every 15 minutes and keep them for 3 hours, then keep one per hour for a day. In this way, in case of quick recovery due to a problem/error, I won't need to transfer the entire dataset/VM from the backup.

Conclusion

The migration is complete: after changing the DNS, the client performed some tests, and everything works properly. The setup has been active for over a week, and there have been no issues. The performance is excellent; I did not perform tests compared to the previous setup since the hardware of the new FreeBSD host is more powerful and modern, so it wouldn't make sense.

Apart from the manager, no user was informed of the change, and in the last week, no reports have been received. The VMs are stable, and the host's load is very low.

The operation actually took less than two hours in total - most of the time has been consumed by the send/receive operations - and gave excellent results. The alternative would have been to install Proxmox on a new host and move the VMs - I probably would have saved a few minutes, but now I can use bhyve, as well as the simplicity and power of the underlying FreeBSD.

Automating ZFS Snapshots for Peace of Mind

Stefano Marinelli — Wed, 21 Aug 2024 08:41:00 +0200

One feature I couldn't live without anymore is snapshots. As system administrators, we often find ourselves in situations where we've made a mistake, need to revert to a previous state, or need access to a log that has been rotated and disappeared. Since I started using ZFS, all of this has become incredibly simple, and I feel much more at ease when making any modifications.

However, since I don't always remember to create a manual snapshot before starting to work, I use an automatic snapshot system. For this type of snapshot, I use the excellent zfs-autobackup tool - which I also use for backups. The goal is to have a single, flexible, and configurable tool without having to learn different syntaxes.

Why zfs-autobackup?

zfs-autobackup has several advantages that make it perfect (or nearly so) for my purpose:

It operates based on "tags" set on individual datasets. I don't have to specify the dataset; I just assign a specific tag (of my choice) to the datasets, and zfs-autobackup will operate on those datasets, transparently with respect to others. This ensures it will work even on datasets in different zpools.
It's extremely flexible in management. For example, by setting the correct tag to "zroot", it will automatically manage all underlying datasets. However, it's possible to exclude some for more granular snapshot management.
It works well on both FreeBSD and Linux - I use it with satisfaction on both platforms.
Different tags allow for different levels of data retention and operation. For example, the "mylocalsnap" tag will be for local snapshots, while "backup_offsite" will be for backups that will be copied off-site. The two tags (and related snapshots) will be independent, even though they operate on the same datasets.

Installation

On FreeBSD, installation is straightforward, as there's a ready-made package:

pkg install py311-zfs-autobackup

On Linux, it will depend on the specific distribution. Being in Python, it will always be possible to install it using pip.

Configuration

Once installed, you just need to assign the tag to the dataset. For example:

zfs set autobackup:mylocalsnap=true zroot

This will set a tag called "mylocalsnap" on zroot and underlying datasets, i.e., on the entire main file system of FreeBSD.

Usage

Now, you just need to run zfs-autobackup, specifying both the tag and the snapshot retention criteria:

/usr/local/bin/zfs-autobackup mylocalsnap --keep-source 5min1h,1h1d

In this case, it will take a (recursive) snapshot of the datasets that have the "mylocalsnap" tag set to true, keeping one snapshot every 5 minutes for an hour, and one every hour for a day.

On subsequent executions of zfs-autobackup, snapshots that don't meet the previous retention criteria will be deleted.

After running this command, here's the result:

root@fbsnap:~ # zfs list -t snapshot
NAME                                            USED  AVAIL  REFER  MOUNTPOINT
zroot@mylocalsnap-20240820150115                  0B      -    96K  -
zroot/ROOT@mylocalsnap-20240820150115             0B      -    96K  -
zroot/ROOT/default@mylocalsnap-20240820150115     0B      -  1.00G  -
zroot/home@mylocalsnap-20240820150115             0B      -    96K  -
zroot/tmp@mylocalsnap-20240820150115              0B      -   104K  -
zroot/usr@mylocalsnap-20240820150115              0B      -    96K  -
zroot/usr/ports@mylocalsnap-20240820150115        0B      -    96K  -
zroot/usr/src@mylocalsnap-20240820150115          0B      -    96K  -
zroot/var@mylocalsnap-20240820150115              0B      -    96K  -
zroot/var/audit@mylocalsnap-20240820150115        0B      -    96K  -
zroot/var/crash@mylocalsnap-20240820150115        0B      -    96K  -
zroot/var/log@mylocalsnap-20240820150115          0B      -   144K  -
zroot/var/mail@mylocalsnap-20240820150115         0B      -    96K  -
zroot/var/tmp@mylocalsnap-20240820150115          0B      -    96K  -

As you can see, snapshots have been taken of all datasets with the tag set.

Automation

To automate the process, simply modify the /etc/crontab file and add a line like this:

*/5     *       *       *       *       root    /usr/local/bin/zfs-autobackup mylocalsnap --keep-source 5min1h,1h1d

Now, wait a few minutes and check again:

root@fbsnap:~ # zfs list -t snapshot
NAME                                            USED  AVAIL  REFER  MOUNTPOINT
zroot@mylocalsnap-20240820150115                  0B      -    96K  -
zroot/ROOT@mylocalsnap-20240820150115             0B      -    96K  -
zroot/ROOT/default@mylocalsnap-20240820150115   212K      -  1.00G  -
zroot/ROOT/default@mylocalsnap-20240820151000   128K      -  1.00G  -
zroot/home@mylocalsnap-20240820150115             0B      -    96K  -
zroot/tmp@mylocalsnap-20240820150115             72K      -   104K  -
zroot/tmp@mylocalsnap-20240820151000              0B      -   104K  -
zroot/usr@mylocalsnap-20240820150115              0B      -    96K  -
zroot/usr/ports@mylocalsnap-20240820150115        0B      -    96K  -
zroot/usr/src@mylocalsnap-20240820150115          0B      -    96K  -
zroot/var@mylocalsnap-20240820150115              0B      -    96K  -
zroot/var/audit@mylocalsnap-20240820150115        0B      -    96K  -
zroot/var/crash@mylocalsnap-20240820150115        0B      -    96K  -
zroot/var/log@mylocalsnap-20240820150115         64K      -   144K  -
zroot/var/log@mylocalsnap-20240820151000         60K      -   144K  -
zroot/var/mail@mylocalsnap-20240820150115         0B      -    96K  -
zroot/var/tmp@mylocalsnap-20240820150115         64K      -    96K  -
zroot/var/tmp@mylocalsnap-20240820151000          0B      -    96K  -

If everything went as it should, you'll notice that unmodified datasets won't have new snapshots, while those that have been modified since the previous manual execution (e.g., zroot/var/log) will contain both the previous snapshot and the automatic one.

Recovering Files from Snapshots

There are various ways to recover a file from the previous snapshot. One option is to restore the entire snapshot to the current dataset, but this may not be the best option as it will perform a complete restore.

Another alternative is to go to the hidden snapshot directory. For example:

root@fbsnap:~ # cd /var/log/.zfs/snapshot/mylocalsnap-20240820151000/
root@fbsnap:/var/log/.zfs/snapshot/mylocalsnap-20240820151000 # ls -l
total 39
-rw-------  1 root wheel     872 Aug 20 15:06 auth.log
-rw-r--r--  1 root wheel   79079 Aug 20 13:19 bsdinstall_log
-rw-------  1 root wheel    5401 Aug 20 15:10 cron
-rw-r--r--  1 root wheel      63 Aug 20 13:20 daemon.log
-rw-------  1 root wheel      63 Aug 20 13:20 debug.log
-rw-r--r--  1 root wheel      63 Aug 20 13:20 devd.log
-rw-r--r--  1 root wheel      63 Aug 20 13:20 lpd-errs
-rw-r-----  1 root wheel      63 Aug 20 13:20 maillog
-rw-r--r--  1 root wheel   17043 Aug 20 15:00 messages
-rw-r-----  1 root network    63 Aug 20 13:20 ppp.log
-rw-------  1 root wheel      63 Aug 20 13:20 security
-rw-r--r--  1 root wheel     197 Aug 20 15:00 utx.lastlogin
-rw-r--r--  1 root wheel     187 Aug 20 15:00 utx.log
-rw-------  1 root wheel      63 Aug 20 13:20 xferlog

From here, you can read and recover any file present in the individual snapshot. The snapshots are read-only, so you won't be able to write to them.

Creating a Writable Copy of a Snapshot

Should you need a read-write copy of a specific snapshot, you can use the zfs clone command:

root@fbsnap:~ # zfs clone zroot/var/log@mylocalsnap-20240820151000 zroot/recover
root@fbsnap:~ # ls -l /zroot/recover/
total 39
-rw-------  1 root wheel     872 Aug 20 15:06 auth.log
-rw-r--r--  1 root wheel   79079 Aug 20 13:19 bsdinstall_log
-rw-------  1 root wheel    5401 Aug 20 15:10 cron
-rw-r--r--  1 root wheel      63 Aug 20 13:20 daemon.log
-rw-------  1 root wheel      63 Aug 20 13:20 debug.log
-rw-r--r--  1 root wheel      63 Aug 20 13:20 devd.log
-rw-r--r--  1 root wheel      63 Aug 20 13:20 lpd-errs
-rw-r-----  1 root wheel      63 Aug 20 13:20 maillog
-rw-r--r--  1 root wheel   17043 Aug 20 15:00 messages
-rw-r-----  1 root network    63 Aug 20 13:20 ppp.log
-rw-------  1 root wheel      63 Aug 20 13:20 security
-rw-r--r--  1 root wheel     197 Aug 20 15:00 utx.lastlogin
-rw-r--r--  1 root wheel     187 Aug 20 15:00 utx.log
-rw-------  1 root wheel      63 Aug 20 13:20 xferlog

This creates a new dataset zroot/recover that is a writable copy of the snapshot. You can now modify these files as needed, without affecting the original snapshot or the live filesystem.

Cleaning Up

Sometimes you may want to delete all snapshots generated by zfs-autobackup. There are various ways, but the quickest can be to use a simple pipe:

zfs list -t snapshot -o name | grep -i mylocalsnap | xargs -n 1 zfs destroy -vr

By implementing automatic ZFS snapshots, you can work with peace of mind, knowing that you can always revert changes or recover lost files. This setup provides an excellent balance between data protection and system performance.

From Cloud Chaos to FreeBSD Efficiency

Stefano Marinelli — Thu, 04 Jul 2024 08:41:00 +0200

Introduction

A few months ago, a client asked me to take care of their Kubernetes cluster (hosted on AWS and GCP). In their opinion, the costs were exorbitantly high for relatively simple and lean websites. Sure, they had many visits, but nothing too excessive development-wise.

I kindly declined. Unfortunately, their situation is all too common these days: they hired developers accustomed to working that way, convinced that a system administrator is now unnecessary because "the cloud has infinite potential." They were used to considering optimization as secondary because "we have infinite power" (and this is already a spoiler for the ending).

Being open to dialogue and new experiences, they asked for my opinion on the matter. We talked for a while, and I explained that, in my view, for the type of setup they had (standard, with various replicas and variants, but primarily based on two platforms), it didn't make sense. I saw it as complicating things. An over-engineering of something simple. Like taking a cruise ship to cross a river.

They then asked me to create something simple that would serve as a development server and for backups, to understand what kind of solution I had in mind.

The Solution

So, I started building everything. I began with FreeBSD 13.2-RELEASE, but in the meantime, 14.0-RELEASE came out, so that’s the version I delivered.

I installed the operating system on a physical server, leased from one of the main European providers. Benefiting from one of their auctions (good deals can be found on weekends), they found a sufficiently powerful machine, with 128GB of RAM, 2 NVMe drives of 1TB each, and two spinning disks of 2TB each for less than 100 euros per month. They also took another, less powerful one for additional backups and to back up the first one.

Implementation

I decided to keep the host as clean as possible and concentrated the services in jails (managed by BastilleBSD) and VMs. The machine was divided as follows:

A series of bridges - to be used for different projects. Jails of the same project and/or type use the same bridge and can communicate with each other, sharing some resources (MariaDB, etc.).
A bhyve VM with Alpine Linux - in my opinion, the best distribution for running Docker containers. Do we really need systemd just to launch Docker? They mainly use it as a pre-production test bench, connected via VPN to their company LAN. It is the core of their "online" development, i.e., outside their computers. It has 32GB of RAM, 200GB of disk (obviously bhyve is configured with NVMe drivers), and 4 cores assigned.
A VNET jail with a reverse proxy (nginx) - they know how to modify virtual hosts and generate certificates with certbot, pointing to the underlying jails.
A series of "empty" VNET jails, to be cloned, for each type of setup (they mainly have CMS based on WordPress and Laravel, so with all dependencies inside - nginx, php, redis, etc. except the databases).
A VNET jail with MariaDB installed, to be cloned, to be attached to different projects as needed.
zfs-autobackup performs local snapshots, keeping: one every 15 minutes for 3 hours, one per hour for 24 hours, one per day for 3 days.

Backups are also performed using zfs-autobackup and, in case of disaster recovery in rapid times, a zfs-send (and corresponding zfs-receive) every 10 minutes on another machine (the other, smaller one, also taken at auction), with the same bridges, firewall rules, BastilleBSD, and bhyve installed - ready to start in case of disaster. Being a test server, we didn't consider to implement a proper HA - at the moment, it wouldn't make sense.

They also have another job with zfs-autobackup that performs an additional backup on a server (Debian in their offices). Safe data, in my opinion, are those in storage under your b...ench.

I delivered everything to them and gave a brief course to the more experienced devs on how to manage things. No explanation on the Alpine Linux VM, but I showed them the jails, how to clone, configure, and manage them.

Real-world Testing

I didn't hear from them anymore. After a few weeks, one of the devs contacted me urgently because a junior unfortunately made a mistake and deleted an entire project from one of the jails. I explained that the local snapshots were restorable with a command, and he was thrilled. He restored both the development jail and the one with the database made two minutes before the "mishap" and they restarted immediately.

I realized that this event would change some of their procedures and criteria.

I hadn't heard from anyone for months. This morning, I received a call from their manager, whom I hadn't heard from since the beginning, and he told me how things had been going these months.

Lessons Learned

First, this person has good communication and commercial skills but little technical background. He is open-minded and tends to study carefully what is proposed to him. He doesn't discard any solution a priori, without having touched its pros and cons.

They had leased servers with cPanel and were inserting their content inside them. The devs who arrived a few years ago suggested making a technological transition, eliminating these "obsolete" servers and "outdated" methodologies, pushing everything to the cloud and containerizing everything. When we first talked, he told me how they were "lucky to make that transition because their load had increased enormously and the old servers probably wouldn't have handled the load", instead autoscaling saved them. I had some reservations about autoscaling without particular controls, but clearly, I cannot impose my choices on others.

To cut a long story short: seeing what happened with that junior dev's mistake (and the simplicity with which it was possible to restart immediately), they decided to increase the use of FreeBSD jails and reduce, at least on secondary loads, the use of their Cloud managed with Kubernetes. As they transitioned to jails, however, they noticed some slowdowns. These slowdowns worsened day by day. According to the devs, it would have been appropriate to go back to having, again, autoscaling ("we need moar powaaaaar!!!") but, fortunately, their boss decided to investigate carefully. They realized that these workloads (based on Laravel) were storing sessions on files. Over time, these millions of files (several gigabytes per day) slowed everything down because, for specific operations, Laravel scanned the entire directory. In other words, on the "cloud," they needed much more power than necessary (and much more disk space, but that was cheaper) to carry this load, which was, in fact, unnecessary. After realizing this, they moved the sessions to Redis. Needless to say, everything became extremely faster, even compared to the previous setup on Kubernetes and autoscaling.

At that point, it was clear that one of the problems with their setup is (as often happens) poor optimization. Today, there's a tendency to rush, "throw in" functions, features, libraries, plugins, etc. without considering the interactions and consequences. If it works, it's fine. Even if it increases computational complexity exponentially just to, for example, change the color of an icon (absurd example, but to give an idea).

They then started moving even the main Laravel workloads (thanks to the optimization implemented). At this point, they began moving some of the WordPress sites even though they were extremely concerned. In the cluster, every day, at fairly irregular intervals, the load would rise and everything would slow down until autoscaling started scaling up to the imposed limits. CPU at 100% on all containers, and the devs noticed that the load came from a series of "php" processes. Recreating the containers helped for some minutes, but did not solve the problem.

To their great surprise, all this did not happen on the FreeBSD jails. The load was significantly lower, without any of these spikes. Satisfied, they decided to use this as their final setup. One of the devs, however, wanted to get to the bottom of it and decided to run a test: he moved some of these WordPress sites to the Alpine VM, on Docker. At that point, the spikes resumed, saturating the CPU of the Alpine machine.

Without going into details, they eventually realized that there was a vulnerability in one (or more) of the many plugins installed on the WordPress sites, which was being exploited to inject a process, probably a cryptominer. The name given to the process was "php" - so the devs, not being system experts, did not worry about understanding better whether it was really php or another process pretending to be it. On FreeBSD, all this did not happen because the injected executable could not run - there was no Linux compatibility activated on the server.

Until then, they considered these (expensive) spikes as organic and did not worry too much about them. Paying to have their friendly intruders mine.

Conclusion

They asked me to help, as much as possible, to move other services to FreeBSD. It won't be easy, probably we will need to use bhyve a lot, but they decided that this is the platform they want to focus on in the coming years.

Undoubtedly, this is a success story of FreeBSD and, indirectly, of correct and careful management of one's resources. Too often today, there is the superficial belief that the cloud, with its "infinite" resources, is the solution to all problems. And that Kubernetes is the best solution for everything. I, on the other hand, have always believed that there is the right tool for everything. You can hammer a nail with a screwdriver, but it's not the most suitable and efficient tool.

Today they spend about 1/10 of what they used to spend before, they have more control over their data and the tools they use. Undoubtedly, all this was also caused by poor optimization and control by those who manage the infrastructure, but the question is: how often do people decide that, in the end, it is okay to spend more (especially if it is someone else's money) rather than go crazy for hours behind such a situation? While having defined and limited resources (albeit elevated) poses different problems - but of optimization. And in the age of energy and resource savings, it might be wise to give more importance to optimization.

Abundance led to waste.

Enhancing FreeBSD Stability with ZFS Pool Checkpoints

Stefano Marinelli — Mon, 01 Jul 2024 09:41:00 +0200

ZFS offers many interesting features, and one of the most widely used is the ability to create and transfer snapshots of entire datasets, even recursively. This approach is useful for backups or maintaining a specific “point in time” for datasets. For example, on FreeBSD, automatic snapshots of the dataset containing the root file system have been taken with each system upgrade for several releases. This way, thanks to Boot Environments, if there are any problems, it is possible to reboot from a previous clone.

However, sometimes we might need something more. Local snapshots do not protect against the deletion of entire datasets or the activation of new features that could potentially cause problems or incompatibilities.

A very useful tool that I have successfully used for some time is the pool checkpoint feature. This feature, imported from Illumos to FreeBSD in 2018, allows creating a sort of snapshot of the entire pool, including features, metadata, etc.

The checkpoint is different from snapshots of individual datasets. It is not possible to have more than one checkpoint, and some operations like remove, attach, detach, split, and reguid will be impossible when a checkpoint exists. This also has a side effect: if there is a checkpoint, deleting a dataset will not release free space because the data will still be physically present in the storage thanks to the checkpoint.

Additionally, checkpoints are detected by the FreeBSD boot loader. When booting the system, the boot loader will offer the option to perform a "Rewind ZFS checkpoint" and boot from that point, effectively discarding everything that occurred after the checkpoint. This option can be particularly useful in emergencies or when you need to quickly undo recent changes.

Creating a checkpoint is very simple. Just use the command:

zpool checkpoint <pool>

The operation is usually quick. When a checkpoint is present, the command zpool status will show its details. For example:

pool: zroot
state: ONLINE
scan: scrub repaired 0B in 00:00:12 with 0 errors on Fri May 17 13:27:14 2024
checkpoint: created Sun Jun 30 12:30:51 2024, consumes 1.34M
config:

    NAME        STATE     READ WRITE CKSUM
    zroot       ONLINE       0     0     0
      ada1p4    ONLINE       0     0     0

errors: No known data errors

To delete the checkpoint, you can use the command:

zpool checkpoint -d <pool>

To rollback state to checkpoint and remove the checkpoint:

zpool import --rewind-to-checkpoint <pool>

To mount the pool read only (without rolling back the data):

zpool import --read-only=on --rewind-to-checkpoint <pool>

It is therefore possible to generate a checkpoint automatically via cron or manually when necessary, for example, before an operating system upgrade.

For more technical details, I suggest reading this excellent article by Serapheim Dimitropoulos, published in the FreeBSD Journal in January 2019.

Proxmox vs FreeBSD: Which Virtualization Host Performs Better?

Stefano Marinelli — Mon, 10 Jun 2024 05:53:45 +0000

TL;DR: Skip to the Conclusion for a summary.

Preamble

I have always been passionate about virtualization and have consistently used it.

The first solution I installed on my infrastructures (and those of clients) was Xen on NetBSD, with great success. I then used Xen on Linux and, since 2012, OpenNebula, followed by Proxmox in 2013. Proxmox has always given me great satisfaction, and even today I consider it a valuable platform that I install gladly. I have also used other hypervisors like XCP-ng but less frequently, and in recent years, I have started to make extensive use of bhyve.

About two and a half years ago, we began a progressive process of migrating our servers (and those of our clients) from Linux to FreeBSD, using jails (when possible) or VMs on bhyve. In some cases, migrating setups from Proxmox to FreeBSD resulted in performance improvements, even with the same hardware. In some instances, I migrated VMs without notifying clients, and they contacted me a few days later to inquire if we had new hardware because they noticed better performance.

After years, I decided to conduct a test to determine if this was just a perception or if there was a technical basis behind it. Of course, this test has no scientific validity, and the results were obtained on specific hardware and at a specific time, so on different hardware, workload, and situations, the results could be entirely opposite. However, I tried to have as scientific and objective an approach as possible since I am comparing two solutions that I care about and use daily.

Hardware and Test Conditions

I often see comparative tests done on VMs from various providers. In my opinion, this comparison makes no sense because a VM from any provider shares its hardware with many other VMs, so the results will vary depending on the load of the "neighbors" and will never be reliable.

For this test, I decided to take a physical server with the following characteristics:

Intel Core i7-6700
2x SSD M.2 NVMe 512 GB
4x RAM 16384 MB DDR4
NIC 1 Gbit Intel I219-LM

The hardware is not recent, but still very widespread. On more recent hardware, the results might differ, but the test will be based on this configuration.

I installed Proxmox 8.2.2 starting from the Debian template of the provider and manually installed it following the instructions. I created a partition for Proxmox and left one partition free on each of the two NVME drives to create (at different times) the ZFS pool (in mirror) and the LVM on top of the Linux software raid.

After all the tests, I installed FreeBSD 14.1-RELEASE on ZFS on the same host, using bsdinstall from an mfsbsd image since the provider does not directly support installing FreeBSD from its panel or rescue mode.

In both installations, I always trimmed the NVME drives before starting the tests, and in the case of ZFS, I set (both on Proxmox and FreeBSD) compression to zstd and atime to off. No other changes were made compared to the standard installation.

On FreeBSD, the VM was created and managed with vm-bhyve (devel).

On Proxmox, I tested the physical host on ZFS and ext4 and the VM on ZFS and LVM as LVM is the standard and most common setup in Proxmox.

On FreeBSD, I tested the host on ZFS and the VM with both virtio and nvme drivers, on zvol, and as an image file within a ZFS dataset.

I used sysbench installed from the official Debian repository (on Proxmox and VM) and from the FreeBSD package on the respective host.

The VMs, both on Proxmox and FreeBSD, have nearly identical characteristics and default configuration (apart from the nvme drivers set on bhyve, and for that reason, I also tested virtio).

For those who want to reproduce my tests, here are the detailed configurations of the VMs used in bhyve:

FreeBSD bhyve VM Configuration with NVMe Driver:

loader="uefi"
cpu=4
memory=4096M
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"

FreeBSD bhyve VM Configuration with virtio Driver:

loader="uefi"
cpu=4
memory=4096M
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0.img"

Proxmox VM Configuration:

Component	Details
Memory	4.00 GiB [balloon=0]
Processors	4 (1 sockets, 4 cores) [x86-64-v2-AES]
BIOS	Default (SeaBIOS)
Display	Default
Machine	Default (i440fx)
SCSI Controller	VirtIO SCSI single
CD/DVD Drive (ide2)	local:iso/debian-12.5.0-amd64-netinst.iso,media=cdrom,size=629M
Hard Disk (scsi0)	zfspool:vm-100-disk-0,cache=writeback,discard=on,iothread=1,size=50G,ssd=1
Network Device (net0)	virtio=BC:24:11:22:3D:F0,bridge=vmbr0

In all the configurations, I used Debian 12 as the VM operating system, with the file system on ext4.

I chose Debian 12 as it is a stable, widespread, and modern Linux distribution. I did not test a FreeBSD VM because, in my setups, I tend not to virtualize FreeBSD on FreeBSD but to use nested jails.

All tests were performed multiple times, and I took the median results. CPU and RAM were tested only on the first VM (on Proxmox (ZFS) and FreeBSD (ZFS and nvme)) as they are not dependent on the underlying storage. Storage performance, on the other hand, was tested on all configurations.

CPU and RAM Tests on VMs

On both VMs:

sysbench --test=cpu --cpu-max-prime=20000 run
sysbench --test=memory run

Comparative Results

CPU Test

Configuration	Events per Second	Total Time (s)	Latency (avg) (ms)
Proxmox	498.08	10.0010	2.01
FreeBSD	473.65	10.0019	2.11

CPU Percentage Analysis

Difference in Events per Second: ((498.08 - 473.65) / 498.08 approx -4.91\%)
Difference in Total Time: ((10.0019 - 10.0010) / 10.0010 approx +0.009\%)
Difference in Latency (avg): ((2.11 - 2.01) / 2.01 approx +4.98\%)

RAM Test

Configuration	Total Operations	Operations per Second	Total MiB Transferred	MiB/sec	Latency (avg) (ms)
Proxmox	64777227	6476757.59	63259.01	6324.96	0.00
FreeBSD	68621063	6861139.06	67012.76	6700.33	0.00

RAM Percentage Analysis

Difference in Total Operations: ((68621063 - 64777227) / 64777227 approx +5.94\%)
Difference in Operations per Second: ((6861139.06 - 6476757.59) / 6476757.59 approx +5.94\%)
Difference in Total MiB Transferred: ((67012.76 - 63259.01) / 63259.01 approx +5.93\%)
Difference in MiB/sec: ((6700.33 - 6324.96) / 6324.96 approx +5.93\%)

CPU and RAM Comparative Results Table

Test	Metric	Proxmox (KVM)	FreeBSD (bhyve)	Difference (%)
CPU	Events/s	498.08	473.65	-4.91
	Time (s)	10.0010	10.0019	+0.009
	Latency	2.01	2.11	+4.98
RAM	Ops	64777227	68621063	+5.94
	Ops/s	6476757.59	6861139.06	+5.94
	MiB	63259.01	67012.76	+5.93
	MiB/s	6324.96	6700.33	+5.93
	Latency	0.00	0.00	0.00

Interpretation of CPU and RAM Results

CPU Performance:
The VM on FreeBSD has slightly lower CPU performance compared to Proxmox (-4.91% in events per second).
The total execution time is nearly identical, with a negligible difference.
The average latency is slightly higher on FreeBSD (+4.98%).
RAM Performance:
The VM on FreeBSD has better RAM performance compared to Proxmox (+5.94% in operations and MiB/sec).
The average latency is identical in both configurations.

In summary, while Proxmox provides more consistent CPU performance, FreeBSD demonstrates superior memory performance. The choice between Proxmox and FreeBSD may depend on the specific workload requirements and the importance of consistent performance versus higher throughput.

I/O Performance Tests

The test has been conducted using sysbench, with this command line:

sysbench --test=fileio --file-total-size=30G prepare
sysbench --test=fileio --file-total-size=30G --file-test-mode=rndrw  --max-time=300 --max-requests=0 run

I/O Comparative Performance Data with Percentage Differences

Metric	VM on Proxmox (ZFS)	VM on Proxmox (LVM)	VM on FreeBSD (ZFS, NVMe)	VM on FreeBSD (ZFS, Virtio)	VM on FreeBSD (zvol)	Host FreeBSD (ZFS)	Host Proxmox (ZFS)	Host Proxmox (ext4)
File creation speed (MiB/s)	407.82	461.52	1467.83	1398.81	1333.64	1625.67	968.64	633.13
Reads per second	650.09	504.80	11183.44	806.93	11834.53	1234.62	920.95	498.37
Writes per second	433.40	336.54	7455.62	537.95	7889.69	823.08	613.96	332.25
fsyncs per second	1387.08	1076.97	23858.08	1721.79	25247.36	2634.01	1964.96	1063.19
Read throughput (MiB/s)	10.16	7.89	174.74	12.61	184.91	19.29	14.39	7.79
Write throughput (MiB/s)	6.77	5.26	116.49	8.41	123.28	12.86	9.59	5.19
Total events	741163	575588	12749157	919952	13491459	1407592	1049894	568277
Average latency (ms)	0.40	0.52	0.02	0.33	0.02	0.21	0.29	0.53
95th percentile latency (ms)	2.30	3.25	0.06	1.58	0.05	1.32	1.79	2.71
Max latency (ms)	22.65	32.30	35.49	13.60	77.53	9.03	9.47	17.39
Total test time (s)	300.0475	300.1147	300.0020	300.0226	300.0012	300.0416	300.0159	300.1381

Percentage Differences Compared to VM on Proxmox (ZFS)

Metric	VM on Proxmox (LVM)	VM on FreeBSD (ZFS, NVMe)	VM on FreeBSD (ZFS, Virtio)	VM on FreeBSD (zvol)	Host FreeBSD (ZFS)	Host Proxmox (ZFS)	Host Proxmox (ext4)
File creation speed (MiB/s)	+13.18%	+259.77%	+242.99%	+227.02%	+298.62%	+137.52%	+55.25%
Reads per second	-22.34%	+1619.98%	+24.13%	+1720.45%	+89.92%	+41.67%	-23.34%
Writes per second	-22.35%	+1620.26%	+24.12%	+1720.42%	+89.91%	+41.66%	-23.34%
fsyncs per second	-22.36%	+1620.02%	+24.13%	+1720.18%	+89.90%	+41.66%	-23.35%
Read throughput (MiB/s)	-22.34%	+1619.88%	+24.11%	+1719.98%	+89.86%	+41.63%	-23.33%
Write throughput (MiB/s)	-22.30%	+1620.68%	+24.22%	+1720.97%	+89.96%	+41.65%	-23.33%
Total events	-22.34%	+1620.16%	+24.12%	+1720.31%	+89.92%	+41.65%	-23.31%
Average latency (ms)	+30.00%	-95.00%	-17.50%	-95.00%	-47.50%	-27.50%	+32.50%
95th percentile latency (ms)	+41.30%	-97.39%	-31.30%	-97.83%	-42.61%	-22.17%	+17.83%
Max latency (ms)	+42.60%	+56.69%	-39.96%	+242.30%	-60.13%	-58.19%	-23.22%
Total test time (s)	+0.02%	-0.02%	-0.01%	-0.02%	-0.02%	-0.01%	+0.01%

Percentage Differences Compared to VM on Proxmox (LVM) as this is the standard Proxmox setup

Metric	VM on Proxmox (ZFS)	VM on FreeBSD (ZFS, NVMe)	VM on FreeBSD (ZFS, Virtio)	VM on FreeBSD (zvol)	Host FreeBSD (ZFS)	Host Proxmox (ZFS)	Host Proxmox (ext4)
File creation speed (MiB/s)	-11.64%	+218.04%	+203.09%	+188.97%	+252.24%	+109.88%	+37.18%
Reads per second	+28.78%	+2115.42%	+59.85%	+2244.40%	+144.58%	+82.44%	-1.27%
Writes per second	+28.78%	+2115.37%	+59.85%	+2244.35%	+144.57%	+82.43%	-1.27%
fsyncs per second	+28.79%	+2115.30%	+59.87%	+2244.30%	+144.58%	+82.45%	-1.28%
Read throughput (MiB/s)	+28.77%	+2114.70%	+59.82%	+2243.60%	+144.49%	+82.38%	-1.27%
Write throughput (MiB/s)	+28.71%	+2114.64%	+59.89%	+2243.73%	+144.49%	+82.32%	-1.33%
Total events	+28.77%	+2114.98%	+59.83%	+2243.94%	+144.55%	+82.40%	-1.27%
Average latency (ms)	-23.08%	-96.15%	-36.54%	-96.15%	-59.62%	-44.23%	+1.92%
95th percentile latency (ms)	-29.23%	-98.15%	-51.38%	-98.46%	-59.38%	-44.92%	-16.62%
Max latency (ms)	-29.88%	+9.88%	-57.89%	+140.03%	-72.04%	-70.68%	-46.16%
Total test time (s)	-0.02%	-0.04%	-0.03%	-0.04%	-0.02%	-0.03%	+0.01%

Analysis of Performance Data

The performance data collected from various configurations of Proxmox and FreeBSD provides a comprehensive view of the I/O capabilities and highlights some significant differences. Here is an analysis of the key findings:

Comparative Analysis

Hypothesis on NVMe Performance and fsync

An important observation from my tests is that VMs with the bhyve NVMe driver show significantly higher performance compared to the same VMs with the virtio driver or compared to the physical host system. This difference initially led me to hypothesize that the bhyve NVMe driver might not correctly respect fsync operations, returning a positive result before the underlying file system has confirmed the final write. However, this was just a theory based on benchmark results and is not supported by concrete data. Furthermore, some developers have reviewed the code and found no evidence to suggest this behavior, and I personally have never encountered any potential issues that would indicate such a problem.

Specifically, I observed that:

The VM with the virtio driver has performance comparable to Proxmox.
The VM with the NVMe driver, whether on a ZFS dataset or zvol, shows performance superior to the physical FreeBSD host.

Host Physical Systems and Filesystems

File Creation Speed:
- Host FreeBSD (ZFS) shows the highest file creation speed at 1625.67 MiB/s, which is +68.03% compared to Host Proxmox (ZFS) and +156.72% compared to Host Proxmox (ext4).
- Host Proxmox (ext4) has a file creation speed of 633.13 MiB/s, which is -34.62% compared to Host Proxmox (ZFS).
- Read and Write Operations per Second:
- Host FreeBSD (ZFS) demonstrates the highest read and write operations per second with 1234.62 reads/s and 823.08 writes/s.
  - Reads per second: +34.06% compared to Host Proxmox (ZFS) and +147.80% compared to Host Proxmox (ext4).
  - Writes per second: +34.04% compared to Host Proxmox (ZFS) and +147.61% compared to Host Proxmox (ext4).
- Host Proxmox (ext4) shows a lower performance with 498.37 reads/s and 332.25 writes/s.
- Host Proxmox (ZFS) has 920.95 reads/s and 613.96 writes/s.
- fsync Operations per Second:
- Host FreeBSD (ZFS) achieves the highest fsync operations per second at 2634.01 fsyncs/s, which is +34.02% compared to Host Proxmox (ZFS) and +147.73% compared to Host Proxmox (ext4).
- Host Proxmox (ext4) has a lower performance with 1063.19 fsyncs/s.
- Host Proxmox (ZFS) achieves 1964.96 fsyncs/s.
- Throughput:
- Host FreeBSD (ZFS) again leads in throughput with 19.29 MiB/s read and 12.86 MiB/s write.
  - Read throughput: +34.03% compared to Host Proxmox (ZFS) and +147.53% compared to Host Proxmox (ext4).
  - Write throughput: +34.08% compared to Host Proxmox (ZFS) and +147.79% compared to Host Proxmox (ext4).
- Host Proxmox (ext4) has the lowest throughput with 7.79 MiB/s read and 5.19 MiB/s write.
- Host Proxmox (ZFS) has 14.39 MiB/s read and 9.59 MiB/s write.
- Latency:
- Host FreeBSD (ZFS) shows the lowest average latency at 0.21 ms and 95th percentile latency at 1.32 ms.
  - Average latency: -27.59% compared to Host Proxmox (ZFS) and -60.38% compared to Host Proxmox (ext4).
  - 95th percentile latency: -26.27% compared to Host Proxmox (ZFS) and -51.29% compared to Host Proxmox (ext4).
- Host Proxmox (ext4) has the highest average latency at 0.53 ms and 95th percentile latency at 2.71 ms.
- Host Proxmox (ZFS) has an average latency of 0.29 ms and 95th percentile latency of 1.79 ms.

VMs vs Physical Hosts

File Creation Speed:
- VM on FreeBSD (ZFS, NVMe) demonstrates an outstanding file creation speed at 1467.83 MiB/s (+218.04% compared to VM on Proxmox (LVM) and +259.77% compared to VM on Proxmox (ZFS)).
- VM on FreeBSD (zvol) achieves 1333.64 MiB/s, which is also significantly higher than VM on Proxmox (LVM) and VM on Proxmox (ZFS).
- Read and Write Operations per Second:
- VM on FreeBSD (ZFS, NVMe) shows exceptional performance with 11183.44 reads/s and 7455.62 writes/s.
- VM on FreeBSD (zvol) also performs excellently with 11834.53 reads/s and 7889.69 writes/s.
- fsync Operations per Second:
- VM on FreeBSD (ZFS, NVMe) achieves 23858.08 fsyncs/s, and VM on FreeBSD (zvol) achieves 25247.36 fsyncs/s, both significantly higher than any other configuration.
- Throughput:
- VM on FreeBSD (ZFS, NVMe) achieves the highest throughput with 174.74 MiB/s read and 116.49 MiB/s write.
- VM on FreeBSD (zvol) also has high throughput at 184.91 MiB/s read and 123.28 MiB/s write.
- Latency:
- VM on FreeBSD (ZFS, NVMe) shows very low average latency at 0.02 ms and 95th percentile latency at 0.06 ms.
- VM on FreeBSD (zvol) has similarly low latencies, indicating fast response times for I/O operations.

VM Configurations Comparison

File Creation Speed:
- Among VMs, VM on FreeBSD (ZFS, NVMe) leads, followed by VM on FreeBSD (zvol), and then VM on FreeBSD (ZFS, Virtio).
- Read and Write Operations per Second:
- VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) both outperform VM on Proxmox (ZFS) and VM on Proxmox (LVM) configurations significantly.
- VM on Proxmox (ZFS) outperforms VM on Proxmox (LVM) in read and write operations.
- fsync Operations per Second:
- VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) have significantly higher fsync operations compared to VM on Proxmox (ZFS) and VM on Proxmox (LVM).
- Throughput:
- VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) have the highest throughput, followed by VM on Proxmox (ZFS) and then VM on Proxmox (LVM).
- Latency:
- VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) show the lowest latencies among the VMs, indicating faster response times.
- VM on Proxmox (ZFS) shows lower latencies compared to VM on Proxmox (LVM).

Cache Settings and Performance Influence

Cache settings can significantly influence the performance of virtualization systems. In my setup, I did not modify the cache settings for the NVMe and virtio drivers, keeping the default settings. It is possible that the observed performance differences are also due to how different operating systems manage the caches of NVMe devices. I encourage other system administrators to explore the cache settings of their systems to see if changes in this area can influence benchmark results.

Conclusion

Regarding RAM and CPU, the performance of the VMs is comparable. There are slight differences in favor of Proxmox for CPU and FreeBSD for RAM, but in my opinion, these differences are so negligible that they wouldn't sway the decision towards one solution or the other.

The I/O performance data clearly indicates that VM on FreeBSD with NVMe and ZFS outperforms all other configurations by a significant margin. This is evident in the file creation speed, read/write operations per second, fsync operations per second, throughput, and latency metrics.

When comparing physical hosts, Host FreeBSD (ZFS) demonstrates excellent performance, particularly in comparison to Host Proxmox (ZFS) and Host Proxmox (ext4).

When comparing VMs, VM on FreeBSD (ZFS, NVMe) and VM on FreeBSD (zvol) configurations stand out as the top performers.

The VM using virtio on FreeBSD also shows strong performance, albeit not as high as the NVMe configuration. It significantly outperforms Proxmox configurations in terms of file creation speed, read/write operations per second, and throughput, while maintaining competitive latencies.

The virtio driver provides a stable and reliable option, making it a suitable choice for environments where the NVMe driver cannot be used. This makes FreeBSD with virtio a balanced option for virtualization, offering both high performance and reliability.

By examining these performance metrics, users can make informed decisions about their virtualization and storage configurations to optimize their systems for specific workloads and performance requirements.

In light of these tests and experiments, I can therefore affirm that my sensations (and those of many users) of greater "snappiness" of the VMs on FreeBSD can be confirmed. Certainly, Proxmox is a stable solution, rich in features, battle-tested, and has many other valid points, but FreeBSD, especially with the nvme driver, demonstrates very high performance and a very low overhead in installation and operation.

I will continue to use both solutions with great satisfaction, but I will be even more encouraged to implement virtualization servers based on FreeBSD and bhyve.

Migrating from an Old Linux Server to a New FreeBSD Machine

Stefano Marinelli — Wed, 25 Oct 2023 16:39:37 +0000

Preamble: I believe it's time to bid farewell to this venerable Linux server.

TL;DR: The article chronicles the journey of transitioning from an outdated Linux server, running for 1690 days without updates, to a modern FreeBSD machine. This migration involved using tools like mfsBSD, BastilleBSD, Borg Backup, and bhyve. Despite initial hesitations due to the Linux server's impeccable performance, the transition was smooth, resulting in improved manageability and efficiency. The piece emphasizes the importance of regular system updates and anticipates revisiting the topic in the future with new uptime achievements and updates.

This server loyally served for years as a secondary backup server while also providing a few minor services to users. As it often happens, it remained in operation, neglected and without updates for years. Stable operating systems have the "flaw" of being forgotten, giving the false impression that they don't need maintenance or updates. This machine continued its service without oversight for years. When approached for a service request (not due to malfunctions), I advised the client to upgrade the whole system. A mere update would not suffice, so I suggested starting afresh on new hardware with FreeBSD as the primary OS.

The client was understandably hesitant given the uptime stats:

08:58:43 up 1690 days, 21:32, 4 users, load average: 9.57, 10.15, 8.76

Not a single error, not a single hiccup. From his perspective, a similar setup to what was installed many years ago and still working flawlessly was preferred. Nevertheless, he trusted my expertise and let me proceed.

This server had a plethora of duties, among which:

One of the pivotal tasks was running Proxmox Backup Server via Docker. Proxmox Backup Server requires Debian, but this server was running on Ubuntu 16.04 (previously upgraded from Ubuntu 14.04 – yes, ancient!). Hence, Proxmox Backup Server was still at version 1.x.
Another critical function was storing backups made through BorgBackup on its file system. The /home directory used a mirrored btrfs file system, and each backed-up server had its user on this system. Clients could backup (using a push method) only via VPN and only during specific windows when the server permitted (by adding specific firewall rules via Jenkins. Jenkins also managed connection protocols, snapshots, backups, etc.).
Among the lesser tasks, the server ran a few Docker containers with HandBrake on various presets. The client processed video conversions by uploading the original files via sftp, and after some hours, fetched the converted files from the destination directory. This will not be replicated on the new FreeBSD server since they now handle this operation locally on their high-performance MacBook Pro with Apple Silicon. However, a future restoration isn't off the table.

The first thing I did was install FreeBSD on the new hardware. Given that it's a physical server on Hetzner (an auction pick due to disk space needs over power), and FreeBSD wasn't an option, I used mfsBSD.

After booting the physical server in Linux rescue, I copied mfsBSD onto the disks using dd and restarted. On boot, I SSHed into mfsBSD and executed the installation using bsdinstall-a robust and efficient method.

I set up a raidz1 with all four 6 TB disks, resulting in a final storage space of 21.8T, ample for now without the video files.

To ensure continuity, I kept a setup similar to the old one. The clients would essentially continue with their usual backup procedure without necessitating drastic changes to backup scripts. To avoid storing these backups directly in the physical machine's /home and to leave the door open for future services, I installed BastilleBSD and began setting up several jails. I replaced the old Linux machine's behavior with a VNET FreeBSD jail. In past scenarios, I've created Linux jails (thanks to BastilleBSD) and transferred the old server into the jail using rsync, making minor configuration tweaks. While this usually works, it doesn't address the underlying issue of an obsolete setup. Given the opportunity, I opted for a modern toolset.

Thus, I copied every home directory (along with their historic backups) in its entirety, installed BorgBackup, and re-established the VPN. With a VNET jail, I can craft networking devices and fine-tune configurations. After recreating user accounts, inputting the various SSH authorized_keys, and checking all clients, I set up a snapshot plan on the host. This ensures that if a client is compromised with the potential (however remote) for breach and backup deletion, a ZFS snapshot of the entire jail remains available.

As mentioned, one of the core tools on the old server was Proxmox Backup Server. It's not natively installable on FreeBSD, necessitating a VM. Enter the fantastic bhyve, supported by vm-bhyve.

However, one issue arose: backups would consume vast amounts of space, and I wanted to avoid housing an enormous disk image (or a zvol) with both VMs and backups. So, I opted for a slightly less performant but more flexible solution: installing Debian 12 and Proxmox Backup Server on the VM while placing backups on a separate ZFS dataset on the physical machine, exported via NFS and mounted on the VM.

Given that the physical server has an internal bridge "vm-public" with IP 192.168.124.1 and the VM is at 192.168.124.2, I just created a dataset named zroot/PBS and added the following line to /etc/exports:

/zroot/PBS -alldirs -maproot=root -network 192.168.124.2/32

To enable NFS, insert into /etc/rc.conf:

rpcbind_enable="YES" 
nfs_server_enable="YES" 
mountd_flags="-r" 
rpc_lockd_enable="YES"

Within the VM, create /PBS and include in /etc/fstab:

192.168.124.1:/zroot/PBS /PBS nfs rw,async,soft,intr,noexec 0 0

After Proxmox Backup Server's installation, simply set up the datastores in /PBS/, and they'll directly store on the physical machine's ZFS dataset.

For firewall configurations, I exposed port 8007, redirecting it towards the VM, and everything started functioning smoothly. I then set a Proxmox Backup Server replica from the old to the new server. After completion, I changed the Proxmox Backup Server IP on all Proxmox hosts to point to the new server. Smooth sailing.

The old Ubuntu server also managed other minor services, which have become obsolete and weren't replicated.

The transition was seamless, the client is pleased, and I'm content since each service is now neatly segregated into its jail or VM. The machine's load is minimal, which might pave the way for other tasks, via VPN. Everything now rests on ZFS, and the icing on the cake: I made the client promise not to reach another 1690 days of uptime but to timely update as required.

I'm not entirely convinced the promise will hold-meaning, in a few years, I might yet be discussing this "new" server, highlighting another impressive uptime and another upgrade journey.

How we are migrating (many of) our servers from Linux to FreeBSD - Part 3 - Proxmox to FreeBSD

Stefano Marinelli — Tue, 14 Mar 2023 13:00:00 +0000

In recent years, we've been migrating many of our servers from Linux to FreeBSD as part of our consolidation and optimization efforts. Specifically, we've been moving services that were previously deployed using Docker onto FreeBSD, and it has proven to be a great choice for handling workloads efficiently.

To this end, we've also been migrating many of our virtual machines (VMs) to FreeBSD, deploying services within FreeBSD jails. In some cases, these jails have even replaced entire VMs and run bare metal. Although we prefer to move to native FreeBSD whenever possible, sometimes it's not the best option for all the services we offer. As a result, one of our most critical physical servers has been left behind for years.

This server was a Proxmox server that we installed many years ago and updated to version 6.4. It hosted some critical services, but upgrading to Proxmox 7.x posed some challenges. In particular, some of the LXC containers required tweaks.

Unfortunately, this server was quite old, with only four physical disks and 64 GB of RAM. It was located in an OVH data center and had been running well until one of the disks started to malfunction once a week, on Sundays. This would trigger a RAID reconstruction that kept the system busy for about two days.

Despite my preference for simple setups, this server had been deployed gradually over many years, and everything was tied together. As a result, unraveling the system to resolve the issues was not a simple task. Sometimes the combination of simple things can make everything complex.

The Proxmox Server

The Proxmox server was configured as the central hub for various services, including primary DNS, web hosting, VOIP, and more. It featured several bridges, each with its own specific purpose, and was connected to a virtual machine running MikroTik CHR. This machine was responsible for consolidating all incoming VPNs from the MikroTik devices we managed, both ours and those belonging to our clients. Additionally, it provided a series of bridges to manage these devices and all server management VPNs and other services. The Proxmox server also housed several virtual machines running Linux, FreeBSD, OpenBSD, and NetBSD, as well as LXC containers.

Over the last two years, we've been migrating most of these virtual machines and containers to FreeBSD-based VMs, which feature their own specific jails. Consequently, most of the VMs we've had to move were BSD-based, while only five Linux VMs remained. The LXC containers hosted a range of services, including servers managed by Virtualmin, a large installation of Zimbra (which was hosted within an LXC container running CentOS 7), as well as some minor Alpine Linux-based machines. We located all these virtual machines and containers in a LAN created and managed by CHR. All public IPs were managed by CHR, which relied on NAT mappings to establish communication between them. CHR had thus become the heart of our system, and if it experienced any issues, it could potentially take down the entire system. Fortunately, it remained stable for years.

Migration - first steps

The first step I took was to install FreeBSD on the new server. Easy peasy. The next step was to find a way for the CHR to migrate to the new server (under bhyve) and continue to manage all the public IPs of the original server. The problem is that OVH, with its failover IPs, ties a specific MAC address to each individual IP address. Therefore, the only way was to create a bridge on the FreeBSD server (on the Proxmox server, I already had the bridge on the physical network card) and create an L2 tunnel between the two servers - I used OpenVPN with tap interfaces, specifically inserted into the bridges. I could have used other methods and techniques, but I wanted to experiment with a setup that could allow, if necessary, to bridge a larger number of physical and virtual servers even if the IPs are all mapped to a single server. OVH does not allow, in fact, the splitting of classes, so a move must be made for the entire class, not for a single IP address.

Initially, MikroTik CHR 7 did not boot on bhyve. In the end, I managed to make it work, but I had other problems, probably related to the MTU of the interfaces. So I thought about taking the opportunity to unbind the LXC containers and VMs from CHR and remove MikroTik from the setup. With RouterOS version 7, in fact, Wireguard-based VPNs are also supported, so within a few days, it was possible to update the few routers still on 6.x and recreate some VPNs using Wireguard. I mapped both the VMs and LXC containers directly to their respective public IPs, greatly simplifying the steps. Everything worked perfectly.

The next step was to test the first migrations, starting from the VMs already on FreeBSD. For simplicity, I created a new FreeBSD VM in bhyve and copied (via zfs-send and zfs-receive) the datasets related to BastilleBSD. All services are installed in jails managed by Bastille, so this was enough to have, in a short time, a new operating server equivalent to the previous one. At that point, I shut down the original server, connected the VM to the bridge linked to the tunnel (after modifying its MAC address), turned on the new FreeBSD VM (on bhyve), and everything started to work correctly - but from the new physical server.

One by one, I moved all the FreeBSD VMs. For Linux, NetBSD, and OpenBSD, I simply copied the images and pointed bhyve to them. Some small specific configuration on vm-bhyve and everything started to work correctly. Where possibile, I replaced the “virtio” with “nvme” as it performs much better on bhyve.

Migration - LXC containers to Virtual Machines

For LXC containers, I initially thought of creating an Alpine Linux virtual machine, installing LXD, and copying each individual container. It worked for some of them, but for others, I started to encounter strange issues, similar to those that would have required manual intervention to upgrade from Proxmox 6.x to 7.x. As is often the case with Linux-based solutions, compatibility is not always preserved between updates, so I would have had to fine-tune all the containers, which I didn't feel like doing. The containers had been created (at the time) to optimize RAM usage on the Proxmox machine, but to date, they have caused more problems than benefits. In some cases, certain processes got "stuck," making it impossible to "reboot" the LXC container, requiring the entire physical node to be rebooted. If they had been virtual machines, I could have given a "kill" command from the virtualizer (to the respective KVM process, in that case) and restarted it.

For greater compatibility and ease of future management, I decided to convert the LXC containers into actual VMs on bhyve. The process was simple:

Creating an empty VM with vm-bhyve and booting the VM with SystemRescueCD.
Creating destination partitions and file systems in the VM, then doing a complete rsync of the original LXC container.
Adjusting the fstab file, installing the kernel on the destination VM, and creating the initrd (some containers were already copies of VMs, so the kernel remained installed and updated, even though it wasn't being used. The initrd, on the other hand, did not include the nvme or virtio drivers, so I had to regenerate it anyway.)
Adjusting the bhyve vm configuration file, doing one last rsync after shutting down the services, shutting down the original LXC container, and starting the bhyve VM.

Everything worked correctly, so one by one, I moved all the containers. The largest one ended up on another physical node (also FreeBSD with bhyve) temporarily because the space on the new server was not sufficient to contain it. It didn't need to be on this server, so no problem.

One by one, the LXC containers started on the new server. Apart from some minor adjustments to the destination VMs (different network interface names, etc.), I didn't encounter any particular problems even after several days. Everything works perfectly.

At the very end, I re-created the MikroTik CHR VM. I’ll keep this setup separate for now, as strictly tied to eoip interfaces. This was the main reason why I haven’t performed the migration before. Things were too tied together and I had to untie everything, step by step.

…and then one of the Linux VMs started to freeze

Several Linux VMs are just the basis on which Docker runs. One of them (not even among the busiest) started, every 12/15 hours, to completely freeze. It stopped responding to ping, and it was impossible to give any type of command from the console. In a word: stuck.

Searching the web, I found some references to this problem and, observing the errors of an ssh session that was left connected (stuck, but still showing the last error), I found it to be a problem similar to the one described in this post, namely:

"watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [khugepaged:67]"

I tried various solutions such as changing the storage driver, the number of cores, the distribution (from Alpine to Debian), etc., but none of these operations solved the issue. I also noticed that the problem occurs with all Linux VMs, but only those with a recent kernel (> 5.10.x) freeze, while the others continue to work. The problem does not occur, however, with the *BSDs.

In the end, I:

Reduced the number of cores to 1 for the VMs that did not have a high load (some remained with multiple cores), hypothesising a problem with allocating cores that were too busy
Gave the command: "/usr/bin/echo 60 > /proc/sys/kernel/watchdog_thresh" to the VM.

The VM became stable, and I have not seen that error/warning on any other machine since. I will investigate further, but I believe it is a problem related to the Linux kernel, which, for some reason, generates a kernel panic if particular situations of CPU concurrency are generated.

The End…and a nice OOM!

After moving everything, I was finally able to migrate the entire class of OVH IPs from one physical server to another. The operation was quite quick, but in order to avoid problems, I notified all users and performed the operation on a Sunday and during off-peak hours. The whole process took about 10 minutes and there were no hitches of any kind.

For safety reasons, I kept the Proxmox machine active for a few more days, but there was no need to use it. However, after a couple of days, I encountered a problem: the largest VM, in some cases, was being "killed" because FreeBSD generated an OOM. I had never seen, from FreeBSD 13.0 onwards, any OOM related to "abuse" of RAM usage by ZFS, but in this case, it actually happened.

In the end, I understood that ZFS, on FreeBSD, is able to release memory, but not quickly enough to manage any "spikes" in individual VMs. In fact, the VMs do not know the situation of the physical host's RAM, so they will tend to occupy all the space allotted to them (even if only for caching). A sudden spike (i.e. if you create and launch a new VM) could cause a sudden increase in RAM usage by the bhyve process, and FreeBSD could be forced to kill it, even if part of the RAM is only ARC cache. While Proxmox supports HA (i.e., control over whether the VM is running), vm-bhyve only launches the VM (bhyve process). I should manage it with tools like monit, but for now, I preferred to simply set limits on ZFS RAM usage using "vfs.zfs.arc_max", and there have been no more problems.

Final considerations

The operation was long but linear. The most complex part was unraveling all the configurations related to MikroTik CHR and the VPNs linked to each individual LXC machine/container. Once everything was implemented on a dedicated VM, the operation was fairly straightforward.

The hardware specifications of the destination physical server are slightly better than the starting one, but the final performance of the setup has greatly improved. The VMs are very responsive (even those that were previously LXC containers running directly on bare metal) and, thanks to ZFS, I can make local snapshots every 5 minutes. In addition, every 10 minutes, I can copy (using the excellent zfs-autobackup) all the VMs and jails to other nodes both as a backup and as an immediate restart in case of disaster. I just need to map the IPs, and everything will start working very quickly. Proxmox also allows you to perform this type of operation with ZFS, but you still need to have Proxmox (in a compatible version) on the target machine. With the current setup, I only need any FreeBSD node that supports bhyve.

Proxmox is an excellent tool, well-developed, open-source, efficient, and stable. We manage many installations, including complex ones (ceph clusters, etc.), and it has never let us down. However, not all tools are ideal for all situations, and for setups like the one described, the new configuration based on FreeBSD has shown significantly interesting performance and greater management and maintenance granularity.

Virtualizing on vm-bhyve is not complex, but it is certainly not comparable, at the current state, to the simplicity of using a clean and complete interface like Proxmox's. A complete HA system is still missing (sure, it's achievable manually, but...), as well as complete management web interface. However, for knowledgeable users, it is undoubtedly a powerful tool that allows you to have excellent FreeBSD as a base. I'm totally satisfied with my migration and the result is far better than I expected.

Creating an Alpine Linux VM on bhyve - with root on ZFS (optionally encrypted)

Stefano Marinelli — Tue, 01 Nov 2022 15:47:35 +0000

Bhyve is great - and we’re using it with a lot of guest operating systems.

One of my favourite Linux distributions is Alpine Linux - it’s great as a docker or lxc/lxd host, is light, stable and easily manageable.

On FreeBSD, vm-bhyve already provides a good template for Alpine Linux, but it’s based on the plain standard image with boot on ext4.

So we need the alpine-extended iso - with zfs module.

Let’s create a new Alpine Linux bhyve VM:

vm create -t alpine -s 50G -m 4G -c 2 alpinevm

Now let’s configure it:

vm configure alpinevm

Let’s change some options: vmlinuz-vanilla and initramfs-vanilla should be changed to vmlinuz-lts and initramfs-lts.

More, we should instruct grub to boot from ZFS. The configuration should be similar to this:

[…]
grub_install0="linux /boot/vmlinuz-lts initrd=/boot/initramfs-lts alpine_dev=cdrom:iso9660 modules=loop,squashfs,sd-mod,usb-storage,sr-mod"
grub_install1="initrd /boot/initramfs-lts"
grub_run0="linux /boot/vmlinuz-lts root=rpool/ROOT/alpine rootfstype=zfs modules=ext4,zfs"
grub_run1="initrd /boot/initramfs-lts"
[…]

It’s now time to start with the installation:

vm install alpinevm alpine-extended.iso

Now, follow this Alpine Linux Wiki guide. Remember that we’re dealing with “vda” devices, not "sda", so change them accordingly. If you don’t want to have an encrypted rootfs dataset, just avoid the encryption line in the zpool create command.

At the end of the installation procedure, reboot and enjoy your new Alpine Linux bhyve VM.

How we are migrating (many of) our servers from Linux to FreeBSD - Part 2 - Backups and Disaster Recovery

Stefano Marinelli — Mon, 30 May 2022 03:03:52 +0000

After my post on why we’re migrating (most of) our servers from Linux to FreeBSD, I’ve started to write about how we’re doing it. After covering a basic installation (we’re doing a massive use of jails), I’m going now to describe how we’re performing backups.

Backup is not a tool. Backup is not a software you can buy. Backup is a strategy you need to study and implement to be able to solve your specific problems. You need to understand what you’re doing, otherwise you’ll always have a Schrödinger’s Backup - it may work or not and if you don’t test it well enough (i.e. restore) you’ll find out when it’s too late.

We’re performing backups in many different ways but, for our physical and virtual FreeBSD servers, we have a dual approach. We need both a “ready to use” backup (that will be described here, useful for a fast disaster recovery or prompt restore of specific jails) and a “colder”, more space efficient backup that can be kept for months (or years), more similar to the borg approach on previous posts. Generally speaking, we store our OS (and jails) on ZFS, so I’ll describe this kind of approach here.

Disaster recovery backup - ZFS send/receive

BastilleBSD creates its datasets and mounts them on /usr/local/bastille . There are no databases, all the jails’ configurations are inside that mountpoint so it’s quite easy to backup and restore all the jails or any single jail in one go. More, the “everything-in-a-jail” approach simplifies the restore process as you don’t need to restore the entire host OS, just install an empty FreeBSD server, install BastilleBSD and restore the jails. Or add the jails to already existing FreeBSD systems.

We normally use FreeBSD (or Linux with ZFS) backup servers, well protected and encrypted at rest. For the ZFS send/receive approach, our servers are NOT reachable from the outside. We can ssh into them only using a VPN - they’re too precious to be exposed on the World Wild Web - or, if strictly needed, we expose ssh only using keys, no passwords. We perform the backups using a pull strategy: the backup server connects to the production servers, gets the data, disconnects. The production servers have NO ACCESS to the main backup server. Should they ever be seriously compromised, the backup is safe.

There are many tools that can help to set up this kind of configuration. I’ve tried many of them and found that they all have some good and bad points. The one I decided to use for our servers is zfs-autobackup. It’s easy to use, everything can be set via command line and has a good cron (or Jenkins) output, useful to understand if everything is right.

Let’s consider two servers, one is called “ProdA” and the other is called “Bck” - we obviously want to backup the ProdA into Bck.

Installing ProdA has been covered on a previous post, Bck is quite simple and outside the scope of this post. We just need a protected zfs FreeBSD (or Linux) server. That’s all. Let’s assume that ProdA has a BastilleBSD zfs dataset (and children datasets), with jails and everything needed, as configured in the last post. We now need to install the needed software. On Bck:

pkg install py311-zfs-autobackup mbuffer

On ProdA:

pkg install mbuffer

mbuffer will be used as a ram buffer to avoid read/write spikes (or slowdowns) while sending/receiving the snapshots.

It’s time to prepare the destination dataset. Assuming that Bck has a zroot base dataset, we’ll be creating (as root) a zroot/backups/ProdA

zfs create -p zroot/backups/ProdA

Ok, let’s now go to ProdA

We want to create an unprivileged user that will send the data. We don’t want to allow Bck to connect as root, even if it’s trusted and secure. Let’s create a user called “backupper”. Then, we need to give backupper the right permissions:

zfs allow -u backupper send,snapshot,hold,mount,destroy zroot

Note: if you want Bck to be able to delete the snapshots on ProdA, backupper needs the destroy permission. That means this user can destroy the whole system as can ALSO destroy zroot (or any source dataset you decide). If you’re afraid of this, different approaches must be used (i.e.: local root performing snapshot/cleanups and Bck only transferring them, not hard to achieve with zfs-autobackup). Considering that the Bck is safe, secure and protected, we can tolerate this weakness. Just be sure nobody can break the “backupper” user. Do not use password, use ssh keys and treat this user with the same care you'd use with root.

Now, as root on ProdA:

zfs set autobackup:bck_server=true zroot

We’re setting a custom property, called “autobackup:bck_server”, allowing the zroot (and children) dataset to be backed up by zfs-autobackup. zfs-autobackup will search for all datasets with that property set to “true” (also on different pools) and will backup them. If there’s a specific dataset you don’t want to backup, just set it to “false”. Or if you don’t want to backup the entire zroot but, for example, only “zroot/bastille” (and children), just set autobackup:bck_server=true for that dataset.

ssh config

zfs-autobackup will connect via ssh and zfs-autobackup will try to connect as root. Moreover, even after exchanging the ssh key, Bck will connect many times to ProdA to send its zfs commands (one connection per command). Ssh session initiation is quite long, so there will be some latency. In order to (greatly) speed up this time,

“You can make your ssh connections persistent and greatly speed up zfs-autobackup:

On the server that initiates the backup add this to your ~/.ssh/config:

Host ProdA
User backupper
ControlPath ~/.ssh/control-master-%r@%h:%p
ControlMaster auto
ControlPersist 3600

(Taken from https://github.com/psy0rz/zfs_autobackup/wiki/Performance)

It's now time to go back to Bck and issue a command like this (one line):

/usr/local/bin/zfs-autobackup --ssh-source ProdA bck_server zroot/backups/ProdA --zfs-compressed --no-progress --verbose --buffer 32M --keep-source 0 --no-holds  --set-properties readonly=on --clear-refreservation --keep-target 1d1w,1w1m,1m6m  --destroy-missing 30d --clear-mountpoint

Bck will connect to ProdA, perform the snapshots and start transferring. The most interesting options I used here are:

 --keep-source 0 (only the last snapshot will be kept on ProdA)
 --set-properties readonly=on (be sure the Bck clone is read only, so we will be able to perform an incremental/differential backup next time)
 --keep-target 1d1w,1w1m,1m6m (keep one backup per day for one week, one per week for one month, one per month for six months)
 --destroy-missing 30d (if we've deleted a dataset, keep it for 30 days before removing it from Bck)
 --clear-mountpoint (do not mount the dataset in Bck, as it will cause problems sooner or later)

The first copy will be slow as it'll need to send all the data. The second will be quite fast as only the differences will be transferred.

How to perform a disaster recovery

Ok, your dataset (or datasets) has gone. You need to replace it with the last external backup. You have to retransfer the copy into ProdA (or another FreeBSD host, no difference). Connect to Bck and search for the snapshot you want to restore (zfs list -t snapshot will help). Once identified (one line):

zfs send -R zroot/backups/ProdA/zroot/bastille/bastille/jails/t1@bck_server@20220528005830 | mbuffer -4 -s 128k -m 32M | ssh root@ProdA "zfs receive -F -x canmount -x readonly zroot/bastille/bastille/jails/t1"

Note the -x canmount -x readonly flags. Remember that we altered the canmount and readonly properties of the transferred datasets during the backup, so we must restore them into a normal state.

Once finished, ProdA (or the other, restored host) will show t1 as an available jail and you'll be able to start it.

Why we're migrating (many of) our servers from Linux to FreeBSD

Stefano Marinelli — Mon, 24 Jan 2022 06:55:00 +0000

More about this in the article I wrote to accompany my talk at EuroBSDCon 2024.

Please note: This article was (unexpectedly) quite popular and many interesting opinions came out from the Hacker News thread that it generated. If you're interested, have a look here: https://news.ycombinator.com/item?id=30057549

There's an Italian version of this article here.

I've been a Linux (or GNU/Linux, for the purists) user since 1996. I've been a FreeBSD user since 2002. I have always successfully used both operating systems, each for specific purposes. I have found, on average, BSD systems to be more stable than their Linux equivalents. By stability, I don't mean uptime (too much uptime means too few kernel security updates, which is wrong). I mean that things work as they should, that they don't "break" from one update to the next, and that you don't have to revise everything because of a missing or modified basic command.

I've always been for development and innovation as long as it doesn't (necessarily, automatically and unreasonably) break everything that is already in place. And the road that the various Linux distributions are taking seems to be that of modifying things that work just for the sake of it or to follow the diktats of the Kernel and those who manage it - but not only.

Some time ago we started a complex, continuous and not always linear operation, that is to migrate, where possible, most of the servers (ours and of our customers) from Linux to FreeBSD.

Why FreeBSD?

There are many alternative operating systems to Linux and the *BSD family is varied and complete. FreeBSD, in my opinion, today is the "all rounder" system par excellence, i.e. well refined and suitable both for use on large servers and small embedded systems. The other BSDs have strengths that, in some fields, make them particularly suitable but FreeBSD, in my humble opinion, is suitable (almost) for every purpose.

So back to the main topic of this article, why am I migrating many of the servers we manage to FreeBSD? The reasons are many, I will list some of them with corresponding explanations.

The system is consistent - kernel and userland are created and managed by the same team

One of the fundamental problems with Linux is that (we shall remember) it is a kernel, everything else is created by different people/companies. On more than one occasion Linus Torvalds as well as other leading Linux kernel developers have remarked that they care about the development of the kernel itself, not how users will use it. In the technical decisions, therefore, they don't take into account what is the real use of the systems but that the kernel will go its own path. This is a good thing, as the development of the Linux kernel is not "held back" by the struggle between distributions and software solutions, but at the same time it is also a disadvantage. In FreeBSD, the kernel and its userland (i.e. all the components of the base operating system) are developed by the same team and there is, therefore, a strong cohesion between the parties. In many Linux distributions it was necessary to "deprecate" ifconfig in favor of ip because new developments in the kernel were no longer supported by ifconfig, without breaking compatibility with other (previous) kernel versions or having functions (on the same network interface) managed by different tools. In FreeBSD, with each release of the operating system, there are both kernel and userland updates, so these changes are consistently incorporated and documented, making the tools compatible with their kernel-side updates.

In other words, in FreeBSD there is no need to " revolutionise" everything every few years and changes are made primarily in the form of additions that can enrich (and not break) each update. If a modification was to change the way it interacts with network devices, ifconfig would be modified to take advantage of that and remain compatible with the "old" syntax. In the long-term, this kind of approach is definitely appreciated by system administrators who find themselves with a linear, consistent, and always well-documented update path.

FreeBSD development is (still) driven by technical interests, not strictly commercial ones.

Linux and related distributions now have contributions from many companies, many of which (e.g. Red Hat) push (justifiably) in the direction of what is convenient for them, their products, and their services. Being big contributors to the project they have a big clout so, indeed, their solutions often become de-facto standards. Consider systemd - was there really a need for such a system? While it brought some advantages, it added some complexity to an otherwise extremely simple and functional system. It remains divisive to this day, with many asking, "but was it really necessary? Did the advantages it brought balance the disadvantages?". 70 binaries just for initialising and logging and a million and a half lines of code just for that? But Red Hat threw the rock...and many followed along. Because sometimes it's nice to follow the trend, the hype of a specific solution.

Even FreeBSD has big companies behind it, collaborating in a more or less direct way. The license is more permissive, so not everyone who uses it commercially contributes to it, but knowing that FreeBSD is at the base of Netflix CDNs, Whatsapp servers (waiting for Meta to replace them, for internal coherence reasons, with Linux servers), Sony Playstations and, in part, macOS, iOS, iPadOS, etc. surely gives confidence on its level. These realities, however, do not have enough clout to drive the development of the core team.

Linux has Docker, Podman, lxc, lxd, etc. but... FreeBSD has jails!

FreeBSD jails are very powerful tools for jailing and separating services. There is controversy about Docker not running on FreeBSD but I believe (like many others) that FreeBSD has a more powerful tool. Jails are older and more mature - and by far - than any containerization solution on Linux. Jails are efficient and are well integrated throughout the operating system. All major commands (ps, kill, top, etc.) are able to display jail information as well. There are many management tools but, in fact, they all do the same thing: they interact with FreeBSD base and create custom configuration files. Personally I'm very comfortable with BastilleBSD but there are a lot of very good tools as well as a sufficiently simple manual management. When I need Docker I launch a Linux machine - often Alpine, which I think is a great minimalist distribution, or Debian. But I'm moving a lot of services from Docker to a dedicated jail on FreeBSD. Docker containers are a great tool for rapid (and consistent) software deployment, but it's not all fun and games. Containers, for example, rely on images that sometimes age and are no longer updated. This is a security issue that should not be overlooked.

Linux has ext4, xfs, btrfs... (and zfs, but with some manual intervention). FreeBSD has UFS2 and ZFS

UFS2 is still a very good and efficient file system and, when configured to use softupdates, capable of performing live snapshots of the file system. This is great for backups. Ext4 and XFS do not support snapshots except through external tools (like DattoBD or snapshots through the volume manager). This works, of course, but it is not native. Btrfs is great in its intentions but still not as stable as it should be after all these years of development. FreeBSD supports ZFS natively in the base system and this brings many advantages: separation of datasets for jails, as well as Boot Environments, to make snapshots before upgrades/changes and to be able to boot (from bootloader) even on a different BE, etc.

The FreeBSD boot procedure is cleaner and simpler.

Linux has always used excellent tools such as grub, lilo (now outdated), etc.. FreeBSD has always used a very linear and consistent boot system, with its own bootloader and dedicated boot partition. Whether on mbr, gpt, etc. things are very similar and consistent. I've never had a problem getting a FreeBSD system to boot after a move or recovery from backup. On Linux, however, grub has sometimes given me problems, even after a simple kernel security update.

FreeBSD's network stack is (still) superior to Linux's - and, often, so is its performance.

Meta has been trying to bring the performance of the Linux network stack up to the level of FreeBSD's for years. Many will ask why, then, not move services to FreeBSD. Large companies with huge datacenters can't change solutions overnight, and their engineers, at any level, are Linux experts. They have invested heavily in btrfs, in Linux, in their specifics. Clearly, upon acquiring Whatsapp, they preferred to migrate the "few" Whatsapp servers to Linux and move them to their datacenters. Regarding the real system performance (i.e. disregarding benchmarks, useful only up to a certain point), FreeBSD shines, especially under high load conditions. Where Linux starts to gasp (ex: waiting for I/O) with 100% CPU, FreeBSD has lower processor load and room for more stuff. In the real world (of my servers and load types), I sometimes experienced severe system slowdowns due to high I/O, even if the data to be processed was not read/write dependent. On FreeBSD this does not happen, and if something is blocking, it blocks THAT operation, not the rest of the system. When performing backups or other important operations this factor becomes extremely important to ensure proper (and stable) system performance.

Straightforward system performance analysis

FreeBSD, in the base system, has all the tools to analyze possible problems and system loads. "vmstat" , in a single line, tells me if the machine is struggling for CPU, for I/O or for Ram. "gstat -a" shows me how much, disk by disk, partition by partition, the storage is active, also in percentage with reference to its performance. "top", then, also has support for figuring out, process by process, how much I/O is being used ("m" option). On Linux, to get the same results, you have to install specific applications, different from distribution to distribution.

Bhyve is more incomplete (but more efficient) than KVM

For my purposes, Bhyve is a great virtualization tool. KVM is definitely more complete but since I don't have any special or specific needs not covered by Bhyve on FreeBSD, I found (on average) better performance with this combination. On FreeBSD, however, KSM is missing which, in some cases, can be very useful.

Will I abandon Linux for FreeBSD? Obviously not, just as I haven't for the last 20 years. Both have their uses, their space, their strengths. But if up to now I have had 80% Linux and 20% FreeBSD, the perspective is to invert the percentages of use and, where possible, directly implement solutions based on FreeBSD.

NOTE: this article has been translated from its Italian original version. Even if it's been reviewed and adapted, there might be some errors.

Alpine Linux and LXD perfect setup on BTRFS file system

Stefano Marinelli — Wed, 03 Nov 2021 10:54:10 +0000

Last update: 01 November 2022 - Alpine 3.16 has lxd packages in contrib

Lxc is a great technology. I’ve been using it since 2012 and I’ve always appreciated its lightweight approach and efficiency. Nowadays KVM and hardware virtualization extensions have reached a very good level of performance. While a KVM VM allows to run another kernel and another OS, it ties specific resources to the VM and this isn’t always the best solution. Lxc uses the host’s kernel and shares its ram so the host machine runs lxc container’s processes as local processes - no slowdown, no resource wasting.

Alpine Linux is a great distribution. It’s minimal, reliable, secure and easy to deploy. It’s great for containers but it’s also great for low footprint servers, like hypervisors. It’s good as Xen Dom0, it’s good as KVM host and - no surprise - is great as Lxc host too.

Alpine Linux is great for low energy devices (Raspberry Pis, etc.), too, and has an efficient and reliable out-of-the-box read-only file system support. But I’ll write more about it in another post.

Lxd is a great solution. It uses lxc but simplifies a lot of operations like new container creation, profile management, automatic snapshots, clustering, etc. The “official” best option to use lxd is installing Ubuntu, Snap and so on. I don’t want to use Ubuntu to install Snap. I don’t want to use Snap just to install lxd. That’s why I’m choosing to use lxd with Alpine.

Long story short: this video shows all the steps described below:

Installing Alpine Linux on BTRFS Root File System

Alpine Linux installer supports btrfs as rootfs destination but the setup-alpine command doesn’t allow you to select it: it will partition your disk and create a ext4 standard root file system.

So let’s start with the installation but, then, we’ll interrupt it at disk selection:

Start with setup-alpine and set network, hostname, password, etc. Stop at:

Available disks are:
sda (20.5 GB QEMU     QEMU HARDDISK   )
Which disk(s) would you like to use? (or '?' for help or 'none') [none]

Press ctrl+C and interrupt the script.

Now we can install Alpine Linux on btrfs root system:

test:~# export ROOTFS=btrfs

And, then:

test:~# setup-disk /dev/sda

 The following disk is selected:
 sda    (20.5 GB QEMU     QEMU HARDDISK   )
 How would you like to use it? ('sys', 'data', 'lvm' or '?' for help) [?] sys
 WARNING: The following disk(s) will be erased:
 sda    (20.5 GB QEMU     QEMU HARDDISK   )
 WARNING: Erase the above disk(s) and continue? (y/n) [n] y
 Creating file systems...
 btrfs-progs v5.10
 See http://btrfs.wiki.kernel.org for more information.

 Label:              (null)
 UUID:               6ace888a-2f36-41e8-86b1-3e4214b9e34e
 Node size:          16384
 Sector size:        4096
 Filesystem size:    15.17GiB
 Block group profiles:
 Data:             single            8.00MiB
 Metadata:         DUP             256.00MiB
System:           DUP               8.00MiB
 SSD detected:       no
 Incompat features:  extref, skinny-metadata
 Runtime features:
 Checksum:           crc32c
 Number of devices:  1
 Devices:
 ID        SIZE  PATH
 1    15.17GiB  /dev/sda3

 Installing system on /dev/sda3:
 /mnt/boot is device /dev/sda1

Installation is complete. Now eject the installation media and reboot.

After rebooting, let’s check that root file system is on btrfs:

 test:~# cat /etc/fstab
 UUID=aacbfd28-df47-4945-9c0c-279faef361a0  /   btrfs   rw,relatime,space_cache,subvolid=5,subvol=/ 0 1

Ok, it is. Generally speaking, I prefer to use compression on btrfs. compress=lzo is enough, so you can add it on your fstab.

Now, let’s install lxd.

Edit your /etc/apk/repositories and let it be like this:

http://dl-cdn.alpinelinux.org/alpine/v3.16/main
http://dl-cdn.alpinelinux.org/alpine/v3.16/community

Let’s follow the tutorial I wrote on Alpine Linux wiki and let’s install lxc, lxd, lxcfs and dbus (not needed but recommended to avoid problems with some containers).

After rebooting, you have a fully functional lxd-enabled Alpine Linux installation.

Let’s now configure lxd:

 test:~# lxd init
 Would you like to use LXD clustering? (yes/no) [default=no]:
 Do you want to configure a new storage pool? (yes/no) [default=yes]:
 Name of the new storage pool [default=default]:
 Name of the storage backend to use (btrfs, dir) [default=btrfs]:
 Would you like to create a new btrfs subvolume under /var/lib/lxd? (yes/no) [default=yes]:
 Would you like to connect to a MAAS server? (yes/no) [default=no]:
 Would you like to create a new local network bridge? (yes/no) [default=yes]:
 What should the new bridge be called? [default=lxdbr0]:
 What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
 What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
 Would you like the LXD server to be available over the network? (yes/no) [default=no]:
 Would you like stale cached images to be updated automatically? (yes/no) [default=yes]
 Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:

As you can see, lxd automatically detected that /var/lib/lxd is on a btrfs file system and creates a sub volume.

Now we can launch our first container:

test:~# lxc launch images:alpine/3.16 t1
Creating t1
Starting t1

Ok, done. Let’s check if it’s working:

test:~# lxc info t1
Name: t1
Status: RUNNING
Type: container
Architecture: x86_64
PID: 2648
Created: 2021/11/03 08:55 UTC
Last Used: 2021/11/03 08:55 UTC

Resources:
Processes: 5
CPU usage:
CPU usage (in seconds): 0
Memory usage:
Memory (current): 3.38MiB
Memory (peak): 5.04MiB
Network usage:
eth0:
Type: broadcast
State: UP
Host interface: vethdc207601
MAC address: 00:16:3e:50:d1:ef
MTU: 1500
Bytes received: 3.76kB
Bytes sent: 1.79kB
Packets received: 26
Packets sent: 16
IP addresses:
inet:  10.189.248.21/24 (global)
inet6: fd42:34cb:9408:433c:216:3eff:fe50:d1ef/64 (global)
inet6: fe80::216:3eff:fe50:d1ef/64 (link)
lo:
Type: loopback
State: UP
MTU: 65536
Bytes received: 0B
Bytes sent: 0B
Packets received: 0
Packets sent: 0
IP addresses:
inet:  127.0.0.1/8 (local)
inet6: ::1/128 (local)

It’s running. Everything is ok. For more complex lxc setup (bridges, profiles, etc.) you can follow the excellent lxd documentation.

This kind of setup is easy and lightweight, easy to backup (a new article will follow) and reliable.

Take automatic snapshots of cephfs file systems

Stefano Marinelli — Mon, 29 Jun 2020 05:00:53 +0000

Cephfs is a great tool. It enables users to store and share files while taking advantage of the granularity and performance of Ceph. One of the reasons why I'm using it is the possibility of being used on erasure coded (slow disk) pools and cached on fast ssd disks.

Big amounts of data can be stored and live snapshots can be taken so remote backups can be easily performed. Those snapshots have helped me to restore specific files or directories to specific points in time in a transparent and easy way, multiple times.

Taking a manual snapshot is easy: you'll find a ".snap" hidden directory inside any directory of your (cephfs) file system. You can just "mkdir something" inside the .snap directory and you'll have a full snapshot called "something". To remove a snapshot, simply use the 'rmdir' command and CephFS will delete and trim the space as soon as possible.

To be able to perform it automatically, I've adaped a script called "btrfs-snp", that I generally use to snapshot a BTRFS file system, and created "cephfs-snp".

Using cephfs-snp is quite easy:

# cephfs-snp
Usage: cephfs-snp <dir> (<tag>) (<limit>) (<seconds>)

  dir     │ create snapshot of <dir>
  tag     │ name the snapshot <tag>_<timestamp>
  limit   │ keep <limit> snapshots with this tag. 0 to disable
  seconds │ don't create snapshots before <seconds> have passed from last with this tag. 0 to disable

You can just ask to perform a snapshot or specify more options to have a more granular control of the snapshots created.

It becomes quite interesting when used with cron:

# cat > /etc/cron.hourly/cephfs-snp <<EOF
#!/bin/bash
/usr/local/sbin/$BIN /home hourly  24 3600
/usr/local/sbin/$BIN /home daily    7 86400
/usr/local/sbin/$BIN /home weekly   4 604800
/usr/local/sbin/$BIN /     weekly   4 604800
/usr/local/sbin/$BIN /home monthly 12 2592000
EOF
chmod +x /etc/cron.hourly/cephfs-snp

For more information, you can have a look here: https://github.com/draga79/cephfs-snp

Have fun!

BTRFS: performing automatic snapshots and remote backups

Stefano Marinelli — Sun, 28 Jun 2020 12:00:00 +0000

I'm using BTRFS with satisfaction (I mentioned it in 2014 and, more recently, in the IT-Notes blog). I don't consider it optimal for all types of loads, but it solves many problems in many situations. One of the things I use with more satisfaction is the dynamic snapshot generation function, which guarantees the possibility to have a perfect and immediate copy of a specific volume (or subvolume). When a backup is needed, for example, the use of snapshots is fundamental and having them at file system level is undoubtedly a good help.

When it comes to BTRFS volumes, however, we have additional options. There are native tools that are able to send and receive data to and from BTRFS volumes in an optimized way, taking advantage of the inherent features of the file system itself. Here is a method to automatically take snapshots and transfer them to another BTRFS volume (local or remote).

I've been using two interesting scripts for some time now. In the past I used snapper but I didn't find it optimal for my use. I have therefore continued to search and found some valid alternatives, namely the combination of btrfs-snp and btrfs-sync

btrfs-snp

btrfs-snp allows to be launched specifying a path, a prefix linked to the snapshot taken, the maximum number of snapshots with that prefix to hold and a minimum time between snapshots.

For example:

root@foobar:~# /usr/local/sbin/btrfs-snp / hourly 10 3600

It will take a snapshot of /, with the term hourly as prefix (but it can be any prefix), keep a maximum of 10 snapshots and will not take any more (i.e. the program will exit without action) in less than 3600 seconds from the previous one.

You can of course combine them in this way, to have more snapshots of different types:

/usr/local/sbin/btrfs-snp / hourly 10 3600/usr/local/sbin/btrfs-snp / daily 3 86400/usr/local/sbin/btrfs-snp / weekly 1 604800

It can be launched using cron every hour (or every quarter as it will not take a new snapshot if less than 3600 seconds from the previous one have passed).

btrfs-sync

At this point, we can copy all the snapshots to another file system (local or remote) thanks to btrfs-sync.

The command can be of this type:

root@foobar:~# /usr/local/sbin/btrfs-sync -d -v /.snapshots btrfs@host:/backups/host01/

(it's a one line command)

Obviously the destination file system must be a BTRFS volume.

I usually run the command through a script in the cron.daily, to get a daily copy of the snapshots out.

This kind of setup has already got me out of trouble several times and seems to be quite efficient.

Rotating Docker log files

Stefano Marinelli — Thu, 29 Aug 2019 04:30:00 +0000

Sometimes containers log a lot of things, and the logfile can become huge.

This happens because containers aren't supposed to run for a long time but to be created/disposed regularly. Here's how to fix this.

You have to instruct Docker, at launch, and tell how much you want the log file to grow before rotating and how many old logs you want to keep.

For example,

--log-opt max-size=50m --log-opt max-file=5

will limit log files to 50 MBytes and keep 5 log files, so total occupied space will be 250 MB.

(this article is a copy of the Italian version published on my Italian blog)

BTRFS Best Practices

Stefano Marinelli — Sat, 13 Oct 2018 05:30:00 +0000

I've been using BTRFS for years and have encountered few problems. Some of them are small and can be avoided with some smart improvements. The following advice is based on my experience.

You can find posts online arguing against these options, so your mileage may vary.

Fragmentation, speed and compression

If using rotating disks, employ the space_cache and autodefrag options. The latter can slow down writes but will keep the disk in a usable state for a long time. Otherwise, fragmentation may become a problem.

Generally speaking, I use compression. Modern CPUs are much faster than rotating disks, so compressing data before writing saves space and often improves speed.

Filesystem stats and status

Periodically, check the file system status with the command "btrfs fi df [mountpoint]." Here's an example:

Data, RAID1: total=1.42TiB, used=1.40TiB
System, RAID1: total=32.00MiB, used=256.00KiB
Metadata, RAID1: total=12.00GiB, used=4.68GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

The numbers are quite similar (except for metadata), so no action is needed. Sometimes they're different, and you might encounter a strange situation: you see a lot of free space, but you can't write anymore as the disk appears to be full.

This can happen when the total (allocated) reaches the disk's space, but the used amount is lower. You have space, but you can't use it. A balance will fix this issue.

Perform some manual operations

Ideally, professional file systems should auto-heal and auto-clean. For example, ZFS autoscrubs data periodically. BTRFS still lacks some of these automatic operations, so it's recommended to perform some maintenance. I run them via cron.

My daily job looks like this:

/bin/btrfs fi ba start -dusage=50 -dlimit=2 -musage=50 -mlimit=4 mountpoint

This operation checks and rebalances only a subset of blocks that meet specific conditions. This ensures that the operation only lasts a few minutes. Scheduling it daily will, over time, result in a clean situation.

My weekly job looks like this:

/bin/btrfs scrub start -B -d -c 2 -n 4 mountpoint

This command performs a scrub, checking blocks and detecting (and, if possible, correcting) errors. It's a longer operation, so I don't perform it daily.

I've been using BTRFS since 2012, and thanks to these operations, it still rocks.Its capabilities have helped me in many situations.

I also use BTRFS for my Linux desktop PCs to have snapshots and consistent backups.