IT Notes - series

Make Your Own Backup System – Part 2: Forging the FreeBSD Backup Stronghold

Stefano Marinelli — Tue, 29 Jul 2025 08:00:00 +0200

With the primary backup strategies and methodologies introduced, we've reached the point where we can get specific: the Backup Server configuration.

When choosing the type of backup server to use, I tend to favor specific setups: either I trust a professional backup service provider (like Colin Percival's Tarsnap), or I want full control over the disks where the backups will be hosted. In both cases, for the past twenty years, my operating system of choice for backup servers has been FreeBSD. With a few rare exceptions for clients with special requests, it covers all my needs. When I require Linux-based solutions, such as the Proxmox Backup Server, I create a VM and manage it within.

I typically use both IPv4 and IPv6. For IPv4, I "play" with NAT and port forwarding. For IPv6, I tend to assign a public IPv6 address to each jail or VM, which is then filtered by the physical server's firewall. Unfortunately, every provider, server, and setup has a different approach to IPv6, making it impossible to cover them all in this article. When a provider allows for routed setups, I use this approach: Make your own VPN: FreeBSD, WireGuard, IPv6, and ad-blocking included - assigning a /72 to the bridge for the jails and VMs.

In my opinion, FreeBSD is a perfect all-rounder for backups, thanks to its ability to completely partition services. You can separate backup services (or specific servers/clients) into different jails or even VMs. Furthermore, using ZFS greatly enhances both flexibility and the range of tools you can use.

The main distinction is usually between local backup servers (physically accessible, though not always attended, and in locations deemed secure) and remote ones, such as leased external servers. I personally use a combination of both. If the services I need to back up are external, in a datacenter, and need to be quickly restorable, I prefer to always have a copy on another server in a different datacenter with good outbound connectivity. This guarantees good bandwidth for restores, which isn't always available from a local connection to the outside world. However, an internal, nearby, and accessible backup server (even a Raspberry Pi or a mini PC) ensures physical access to the data. Whenever possible, I maintain both an external and an internal copy - and they are autonomous, meaning the internal copy is not a replica of the external one, but an additional, independent backup. This ensures that if a problem occurs with the external backup, it won't automatically propagate to the internal one. In any case, the backup must always be in a different datacenter from the one containing the production data. When the fire at the OVH datacenter in Strasbourg caused the entire complex to shut down, many people found themselves in trouble because their backups were in the same, now unreachable, location. I had a copy with another provider, in a different datacenter and country, as well as a local copy.

Despite it being "just" a backup server, I almost always use some form of disk redundancy. If I have two disks, I set up a mirror. With three or more, I use RaidZ1 or RaidZ2. This is because, in my view, backups are nearly as important as production data. The inability to recover data from a backup means it's lost forever. And it happens often, very often, that someone contacts me to recover a file (or a database, etc.) days or weeks after its accidental loss or deletion. Usually, pulling out a file from a two-month-old backup generates a mix of disbelief, admiration, but above all, a sense of security in the person requesting it. And that is what our work should instill in the people we collaborate with.

The backup server should be hardened. If possible, it should be protected and unreachable from the outside. My best backup servers are those accessible only via VPN, capable of pulling the data on their own. If they are on a LAN, it's even better if they are completely disconnected from the Internet.

For this very reason, backups must always be encrypted. Having a backup means having full access to the data, and the backup server is the prime target for being breached or stolen if the goal is to get your hands on that data. I've seen healthcare facilities' backup servers being targeted (in a rather trivial way, to be honest) by journalists looking for health details of important figures. It is therefore critical that the backup server be as secure as possible.

Based on the type of access, I use two types of encryption:

If the server is local (especially if the ZFS pool is on external disks), I usually install FreeBSD on UFS in read-only mode, as I've described in a previous article, and encrypt the backup disks with GELI. This ensures that in the event of a "dirty" shutdown (more likely in unattended environments), I can reconnect to the host and then reactivate the ZFS pool. This approach makes it nearly impossible to retrieve even the pool's metadata if the disks are stolen, as GELI performs a full-device encryption. For example, an employee of a company I work with stole one of the secondary backup disks (which was located at a different, unmonitored company site) to steal information. He got nothing but a criminal complaint. With this approach, it's also not necessary to further encrypt the datasets, which avoids some issues (which I'll discuss later, in a future post).
If the server is remote, in a datacenter, I usually use ZFS native encryption, encrypting the main backup dataset (and BastilleBSD's, if applicable). Consequently, all child datasets containing backups will also be encrypted. In this case as well, a password will be required after a reboot to unlock those datasets, ensuring that the data cannot be extracted if control of the disks is lost.

Here is an example of how to use GELI to encrypt an entire partition and then create a ZFS pool on it (in the example, the disk is da1 - do not follow these commands blindly, or you will erase all content on the da1 device!):

# WARNING: This destroys the existing partition table on disk da1
gpart destroy -F da1

# Create a new GPT partition table
gpart create -s gpt da1

# Add a freebsd-zfs partition that spans the entire disk
# The -a 1m flag ensures proper alignment
gpart add -t freebsd-zfs -a 1m da1

# Initialize GELI encryption on the new partition (da1p1)
# We use AES-XTS with 256-bit keys and a 4k sector size
# The -b flag means "boot," prompting for the passphrase at boot time
geli init -b -l 256 -s 4096 da1p1
# You will be prompted for a passphrase: choose a strong one and save it!

# Attach the encrypted partition. A new device /dev/da1p1.eli will be created.
# You will be prompted for the passphrase you just set
geli attach da1p1

# (Optional) Check the status of the encrypted device
geli status da1p1

# Create the ZFS pool "bckpool" on the encrypted device
# We enable zstd compression (an excellent compromise) and disable atime
zpool create -O compression=zstd -O atime=off bckpool da1p1.eli

In this setup, the reference pool for everything related to backups will be bckpool - and you'll need to keep this in mind for the next steps. Additionally, after every server reboot, you'll need to "unlock" the disk and import the pool:

# Enter the passphrase when prompted
geli attach da1p1

# Import the ZFS pool, which is now visible
zpool import bckpool

With this method, it's not necessary to encrypt the ZFS datasets, as the underlying disk (or, more precisely, the partition containing the ZFS pool) is already encrypted.

If, instead, you choose to encrypt the ZFS dataset (for example, if you install FreeBSD on the same disks that will hold the data and don't want to use a multi-partition approach), you should create a base encrypted dataset. Inside it, you can create the various backup datasets, VMs, and the BastilleBSD mountpoint. Due to property inheritance, they will all be encrypted as well.

To create an encrypted dataset, a command like this will suffice:

# Creates a new dataset with encryption enabled.
# keylocation=prompt will ask for a passphrase every time it's mounted.
# keyformat=passphrase specifies the key type.
zfs create -o encryption=on -o keylocation=prompt -o keyformat=passphrase zfspool/dataset

In this case, after every reboot, you will need to load the key and mount the dataset:

zfs load-key zfspool/dataset
zfs mount zfspool/dataset

Keep in mind the setup you choose, as many of the subsequent choices and commands will depend on it.

Base System Setup

I'll install BastilleBSD - a useful tool for separating services into jails. It will be helpful for isolating our backup services:

pkg install -y bastille

If you used ZFS for the root filesystem, you can proceed directly with the setup. Otherwise (i.e., ZFS on other disks), you'll need to edit the /usr/local/etc/bastille/bastille.conf file and specify the correct dataset on which to install the jails. Then run:

bastille setup

Once the automatic setup is complete, check the /etc/pf.conf file - it will be automatically configured to only accept SSH connections. Ensure the network interface is set correctly. When you activate pf, you will be kicked out of the server, but you can then reconnect.

service pf start

Let's bootstrap a FreeBSD release for the jails - this will be useful later.

bastille bootstrap 14.3-RELEASE update

Now, we create a local bridge. Jails and VMs can be attached to it, making them fully autonomous. Using VNET jails, for example, will allow the creation of VPNs or tun interfaces inside them, simplifying potential future setups (and increasing security by using a dedicated network stack).

Modify the /etc/rc.conf file and add:

# Add lo1 and bridge0 to the list of cloned interfaces
cloned_interfaces="lo1 bridge0"
# Assign an IP address and netmask to the bridge
ifconfig_bridge0="inet 192.168.0.1 netmask 255.255.255.0"
# Enable gateway functionality for routing
gateway_enable="yes"

Let's also modify /etc/pf.conf to allow the 192.168.0.0/24 subnet to access the Internet via NAT. We will skip packet filtering on bridge0 and enable NAT. This isn't the most secure setup, but it's sufficient to get started:

#...
# Skip PF processing on the internal bridge interface
set skip on bridge0
#...
# NAT traffic from our internal network to the outside world
nat on $ext_if from 192.168.0.0/24 to any -> ($ext_if:0)
#...

To ensure the new settings are correct, it's a good idea to test with a reboot.

Since I often configure vm-bhyve in my setups, I prefer to install it right away, creating the dataset that will contain the VMs and installation templates. Remember that zroot is only valid if you installed the entire system on ZFS; otherwise, you'll need to change it to your own dataset:

# Install required packages
pkg install vm-bhyve grub2-bhyve bhyve-firmware
# Create a dataset to store VMs
zfs create zroot/VMs
# Enable the vm service at boot
sysrc vm_enable="YES"
# Set the directory for VMs, using the ZFS dataset
sysrc vm_dir="zfs:zroot/VMs"
# Initialize vm-bhyve
vm init
# Copy the example templates
cp /usr/local/share/examples/vm-bhyve/* /zroot/VMs/.templates/

At this point, I usually enable the console via tmux. This means that when a VM is launched, it won't open a VNC port by default, but a tmux session connected to the VM's serial port. Let's install and configure tmux:

pkg install -y tmux
vm set console=tmux

Let's also attach the switch we created (bridge0) to vm-bhyve so we can use it:

vm switch create -t manual -b bridge0 public

Now, vm-bhyve is ready.

The basic infrastructure is complete. We now have:

ZFS to ensure data integrity, which will also handle redundancy, etc.
BastilleBSD to manage jails, useful for backing up Linux, NetBSD, OpenBSD, and non-ZFS FreeBSD machines.
vm-bhyve to install specific systems (like Proxmox Backup Server).

Backup Strategies

I use various backup tools, too many to list in this article. So I'll make a broad distinction, describing how to use this server to achieve our goal: securing data.

For FreeBSD servers with ZFS (hosts, VMs, jails, hypervisors, and their respective VMs), I use an extremely useful, efficient, and reliable tool: zfs-autobackup.
For Linux servers (without ZFS), NetBSD, OpenBSD, etc. (any non-ZFS OS), I usually use BorgBackup. There are other fantastic tools like restic, Kopia, etc., but BorgBackup has never let me down and has served me well even on low-power devices and after incredibly complex disasters.
For Proxmox servers (a solution I've used with satisfaction in production since 2013, although I'm recently migrating to FreeBSD/bhyve where possible), I use two possible alternatives (often both at the same time): if the storage is ZFS, I use the zfs-autobackup approach. In either case, the most practical solution is the Proxmox Backup Server. And the Proxmox Backup Server is one of the reasons I proposed installing vm-bhyve: running it in a VM and storing the data on the FreeBSD host gives you the best of both worlds. Some time ago, I tried running it in a FreeBSD jail (via Linuxulator), but it didn't work.

Backups using zfs-autobackup

zfs-autobackup is an extremely useful and effective tool. It allows for "pull" type backups, as well as having an intermediary host that connects to both the source and destination, which is useful if you don't want direct contact between the source and destination. I won't describe the latter setup, but the documentation is clear, and I have several of them in production, ensuring that the production server and its backup server cannot communicate with each other.

I usually create a dataset for each server and instruct zfs-autobackup to keep that server's backups in that dataset. The snapshots taken and transferred will all be from the same instant, so as not to create a time skew (some tools of this kind snapshot a dataset, then transfer it, which can result in minutes of difference between two different datasets from the same server).

I've described in detail how I perform this type of backup in a previous post, so I suggest reading that post for reference.

Let's install zfs-autobackup on the FreeBSD server:

pkg install py311-zfs-autobackup mbuffer

Backups for other servers using BorgBackup

When I don't have ZFS available or need to perform a file-based backup (all or partial), I use a different technique. BorgBackup backups are primarily "push" based, meaning the client will connect to the backup server. This is not optimal or the most secure approach, as the backup server should, in theory, be hardened. Even when protecting everything via VPN, the risk remains that a compromised server could connect to its backup server and alter or delete the backups. I have seen this happen in ransomware cases (especially in the Microsoft world), and so I try to be careful to minimize this type of problem, mainly through snapshots of the backup server (an operation that will be described later).

To ensure the highest possible security, I create a FreeBSD jail on the backup server for each server I need to back up. The advantage of this approach is the complete separation of all servers from each other. By using a regular user inside a jail, a compromised server that connects to its backup server would only be able to reach its own backups, as it would be confined to a user account and, even if it managed to escalate privileges, still be inside a jail.

Let's say, for example, we want to back up a server called "ServerA" (great imagination, I know). We create a dedicated jail on the backup server:

# Create a new VNET jail named "servera" attached to our bridge
bastille create -B servera 14.3-RELEASE 192.168.0.101/24 bridge0

BastilleBSD will automatically set the host's gateway for the jail. In our case, this is incorrect, so we need to modify it and set the jail's gateway to 192.168.0.1 in the /usr/local/bastille/jails/servera/root/etc/rc.conf file:

# ...
defaultrouter="192.168.0.1"
# ...

Restart the jail and connect to it:

bastille restart servera
bastille console servera

Now, inside the jail, we install borgbackup:

pkg install py311-borgbackup

BorgBackup doesn't run a daemon; it's launched by the remote server (which sends its data to the backup server), so it's important that the installed version is compatible with the one on the remote host.

Since we'll be using SSH, let's enable it:

service sshd enable
service sshd start

And create a non-privileged user for this purpose:

# The 'adduser' utility provides an interactive way to create a user.
root@servera:~ # adduser
Username: servera
Full name: Server A
Uid (Leave empty for default): 
Login group [servera]: 
Login group is servera. Invite servera into other groups? []: 
Login class [default]: 
Shell (sh csh tcsh nologin) [sh]: 
Home directory [/home/servera]: 
Home directory permissions (Leave empty for default): 
Use password-based authentication? [yes]: 
Use an empty password? (yes/no) [no]: 
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]: 
Username    : servera
Password    : <random>
Full Name   : Server A
Uid         : 1001
Class       : 
Groups      : servera 
Home        : /home/servera
Home Mode   : 
Shell       : /bin/sh
Locked      : no
OK? (yes/no) [yes]: yes
adduser: INFO: Successfully added (servera) to the user database.
adduser: INFO: Password for (servera) is: JIkdq8Ex

The user is created and can receive SSH connections. After setting everything up, I suggest disabling password-based login in the jail's SSH configuration, using only public key authentication.

As mentioned, the biggest risk of a "push" backup is that a compromised client could access the backup server and delete or encrypt the backup history, rendering it useless.

To drastically mitigate this risk, we can configure SSH to force the client to operate in a special Borg mode called append-only. In this mode, the SSH key used by the client will only have permission to create new archives, not to read or delete old ones. However, this approach could complicate some client-side operations (like mount, prune, etc.), forcing them to be done on the server. For this reason, I won't describe it in this setup, "limiting" myself to taking snapshots of the repositories. It can be a very good practice, so I recommend considering it.

Let's initialize the BorgBackup repository. In this example, for simplicity, I won't set up repository encryption. If the jails are on an encrypted dataset or GELI-encrypted disks, there will still be data encryption on the disks, but there will be no protection against someone who could physically access the server while the disks are mounted. As usual, security is like an onion: every layer helps. Personally, I suggest enabling and using it ALWAYS.

# Switch to the new user
su -l servera
# Initialize a new Borg repo named "servera" with no encryption (for this example)
borg init -e none servera

The jail is ready, but it's unreachable from the outside. There are two ways to make it accessible:

Install a VPN system inside the jail itself. Using tools like Zerotier or Tailscale (which don't need to expose ports) will immediately create the conditions to connect to the jail, which will remain inaccessible from the outside. As the jail is a VNET jail, we're free to choose any of the supported VPN technologies.
Expose a port on the backup server, i.e., on the host, to allow external connections. Many advise against this path as they consider it less secure. It is, but sometimes we don't have the luxury of installing whatever we want on the server we're backing up.

To expose the port, go back to the host and modify the /etc/pf.conf file, creating the rdr and pass rules to let packets in:

# ...
# Redirect incoming traffic on port 1122 to the jail's SSH port (22)
rdr on $ext_if inet proto tcp from any to any port = 1122 -> 192.168.0.101 port 22
# ...
# Allow incoming traffic on port 1122
pass in inet proto tcp from any to any port 1122 flags S/SA keep state

Reload the pf configuration:

service pf reload

The jail will now be reachable on the server's public IP, on port 1122. Obviously, this port number is for illustrative purposes, and I used from any, but for better security, you should replace any with the IP address of the server that will be connecting to perform the backup.

By repeating this process for each server and creating a separate jail for each, you can have isolated jails in separate datasets with their backups, potentially setting space limits using ZFS quotas.

It's important to remember that backing up a live filesystem (i.e., without a snapshot or dumps) has a very high probability of being impossible to restore completely. Databases hate this approach because files will change while being copied and tend to get corrupted. Of course, it depends on the nature of the data (a backup of a static website will have no issues, but a WordPress database probably will), but it's crucial to think about a technique to snapshot the filesystem before proceeding. For example, I have already written about how to create snapshots on FreeBSD with UFS in a previous article: FreeBSD tips and tricks: creating snapshots with UFS.

I will cover other operating systems in a future, dedicated post.

Proxmox Backup Server in a Dedicated VM

Starting with version 4.0 (which is still in beta at the time of this writing), Proxmox Backup Server (PBS) supports storing its data in an S3 bucket. This is excellent news as it decouples the server from the data. There are great open-source S3 implementations, like Minio or SeaweedFS, which allow for clustering, replication, etc. In this "simple" case, we will install Proxmox Backup Server in a small VM, while for the data, we'll install Minio in a native FreeBSD jail. The advantage is undeniable: the VM will only serve as an "intermediary", but the data will rest directly on the FreeBSD host's dataset, natively. It will also be possible to specify other external endpoints, other repositories, etc.

As a philosophy, I tend not to use external providers unless for specific needs, so installing Minio in a jail is a perfect solution to manage this situation.

Let's install PBS by downloading the ISO from their website (https://enterprise.proxmox.com/iso/) - at this moment, the version that supports this setup is 4.0 Beta.

The directory to download to is the vm-bhyve ISOs directory. It's not strictly necessary, but it's useful for not "losing" it somewhere. So, go to the directory and download it:

cd /zroot/VMs/.iso
fetch https://enterprise.proxmox.com/iso/proxmox-backup-server_4.0-BETA-1.iso

Now let's create a VM with vm-bhyve. We can start from the Debian template, but we'll make some modifications to optimize performance. In this example, I'm giving it 30 GB of disk space, 2 GB of RAM, and 2 cores.

If you want to store all backups inside the VM, you'll need to size the virtual disk correctly (or create and attach another one). In this case, I will focus on the "clean" VM that will store its data on a dedicated jail with Minio.

vm create -t debian -s 30G -m 2G -c 2 pbs

Once the empty VM is created, let's modify its options:

vm configure pbs

We will modify the VM to be UEFI and to use the NVME disk driver - bhyve performs significantly better on NVME than virtio, as previously tested:

loader="uefi"
cpu="2"
memory="2G"
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"

Fortunately, the Proxmox team has provided for the installation of the Backup Server on devices without a graphical interface, so the boot menu will allow installation via serial console. Let's launch the installation and connect to the virtual serial console:

cd /zroot/VMs/.iso
vm install pbs proxmox-backup-server_4.0-BETA-1.iso
vm console pbs

Select the installation via Terminal UI (serial console) and proceed normally as if it were a physical host, assigning an IPv4 address from the 192.168.0.x range (in this example, I'll use 192.168.0.3).

This way, the Proxmox Backup Server will run in a VM, with the ability to take snapshots before updates, etc., without any worries.

Once the installation is complete, PBS will reboot and listen on port 8007 of its IP. Again, as with the jails, we have two options: install a VPN system within the VM itself (thus exposing it automatically only on that VPN - generally a more secure operation) or expose port 8007 on the server's public IP.

In the latter case, add the relevant lines to the /etc/pf.conf file on the FreeBSD backup server:

# ...
# Redirect incoming traffic on port 8007 to the PBS VM's web interface
rdr on $ext_if inet proto tcp from any to any port = 8007 -> 192.168.0.3 port 8007
# ...
# Allow that traffic to pass
pass in inet proto tcp from any to any port 8007 flags S/SA keep state

Reload the pf configuration:

service pf reload

The PBS VM configuration is complete. If you chose to use the PBS's internal disk as a repository, no further operations are necessary (other than the normal repository creation, etc., within PBS).

In this case, however, we will use a different approach.

Creating a Minio Jail as a Data Repository for PBS

This approach, in my opinion, has a number of important advantages. The first is that Minio will run in a dedicated jail on the host, at full performance, and will store the data directly on the physical ZFS datapool, thus removing any other layer in between. This jail could potentially be moved to other hosts (by connecting PBS and the jail via VPN or public IP), made redundant thanks to all of Minio's features, etc. Another solution I am successfully testing (in other setups) is SeaweedFS.

Let's create a dedicated jail with Minio and put it on the bridge, so that PBS can access it on the LAN.

bastille create -B minio 14.3-RELEASE 192.168.0.11/24 bridge0

If not configured directly, BastilleBSD will use the host's gateway for the jail, which is incorrect in this case. So let's go modify it and restart the jail. Enter the jail with:

bastille console minio

And modify the /etc/rc.conf file to have the correct gateway (following the example addresses):

# ...
ifconfig_vnet0=" inet 192.168.0.11/24 "
defaultrouter="192.168.0.1"
# ...

Exit the jail and restart it:

bastille restart minio

Enter the jail and install Minio:

bastille console minio
pkg install -y minio

Minio is already able to start, but PBS, even on the LAN, wants an encrypted connection. Fortunately, there's a handy tool that can generate the certificates for us:

# Download the certgen tool
fetch https://github.com/minio/certgen/releases/latest/download/certgen-freebsd-amd64

# Make it executable and run it for our jail's IP
chmod a+rx certgen-freebsd-amd64
./certgen-freebsd-amd64  -host "192.168.0.11"

# Create the necessary directories and set permissions
mkdir -p /usr/local/etc/minio/certs
cp private.key public.crt /usr/local/etc/minio/certs/
chown -R minio:minio /usr/local/etc/minio/certs/

Let's view the certificate's fingerprint. Since it's self-signed, we'll need it for PBS later. For security reasons, PBS will ask for the fingerprint of non-directly verifiable certificates. Run the following command and take note of the result:

openssl x509 -in /usr/local/etc/minio/certs/public.crt -noout -fingerprint -sha256

At this point, enable and configure Minio in /etc/rc.conf. WARNING: The username and password (access key and secret) used in this example are insecure and for testing purposes only. It is strongly recommended to use different values:

# Enable Minio service
minio_enable="YES"
# Set the address for the Minio console
minio_console_address=":8751"
# Set the root user and password as environment variables
minio_env="MINIO_ROOT_USER=testaccess MINIO_ROOT_PASSWORD=testsecret"

Start Minio:

service minio start

If everything went correctly, Minio is now running (with its certificates) and ready to receive connections.

It's now time to create the bucket(s) that PBS will use. There are several ways to do this, but to test that everything is working and to configure PBS, I suggest connecting via an SSH tunnel.

# Create an SSH tunnel from your local machine to the backup server
# Port 8007 is forwarded to the PBS web UI
# Port 8751 is forwarded to the Minio console
ssh user@backupServerIP -L8007:192.168.0.3:8007 -L8751:192.168.0.11:8751

This way, we'll create a tunnel between the FreeBSD backup server and our workstation, mapping 127.0.0.1:8007 to 192.168.0.3:8007 (the PBS web interface) and 127.0.0.1:8751 to 192.168.0.11:8751 (the Minio console port).

Now, connect to https://127.0.0.1:8751, enter the credentials specified in /etc/rc.conf, and create a bucket.

Once the bucket is created, you can configure PBS to use it. Connect to PBS via https://127.0.0.1:8007 and go to S3 Endpoints. Set a name, use 192.168.0.11 as the IP and 9000 as the port, enter the access and secret keys, and the certificate fingerprint we generated earlier. Enable "Path Style" or it will not work.

Then go to Datastores and add it, as you would for any other S3 datastore, by specifying the created bucket and a local directory where the system will keep its cache.

If everything was set up correctly, PBS will create its structure in the bucket, and from that moment on, you can use it. Always keep in mind that this is still a "technology preview", so there may be issues, but from my tests, it is sufficiently reliable.

Taking Local Snapshots of Backups

One of the most common techniques used in ransomware attacks is to also delete or encrypt backups. They often use automated methods, relying on the fact that many (too many!) consider a "backup" to be a simple copy of files to a network share. However, it's not impossible that, in specific cases, they might compromise the machine and connect to the backup server. This is nearly impossible with a "pull" type backup (like the one managed by zfs-autobackup) but is still possible with the "push" approach, which involves using BorgBackup or similar tools.

This happened to one of my clients once - in that case, the problem originated internally, from an employee who wanted to cover up his mistake, inadvertently creating a disaster - but that will be material for another post.

Fortunately, the client had a system that solved the problem: thanks to ZFS, we can have local snapshots on the backup server, which are invisible and inaccessible to the production server. Since we have already installed zfs-autobackup, it's easy to use it for this purpose as well. I've already talked about this in a previous article and won't rewrite the steps here. Just consult that article, keeping in mind that in this case, it's not advisable to snapshot all the datasets on the backup server (the space would grow exponentially), but only those at risk. In the cases analyzed in this post, this applies only to the push part, as PBS will also be accessible only from the Proxmox servers and not from the VMs they contain. If, in this case too, you don't trust those who manage the Proxmox servers, just set up snapshots for the Minio jail as well.

Conclusion

This long post aims to analyze, in a general way, how I believe one can manage reasonably secure backups of their data. Obviously, there are many variables, additional precautions, possible optimizations, hardening, etc., that must be studied on a case-by-case basis. There are old rules, new rules, old and new philosophies. Recently, many people who have embraced the cloud have often stopped thinking about backups, only to realize it when something happens and the data has, indeed, vanished... into the clouds.

In this post, I have generically covered the setup of the backup server, and this demonstrates how FreeBSD, thanks to its features, can be considered an ideal platform for this type of task.

In the next articles in this series, I will examine the client side, i.e., how to structure them for a sufficiently reliable backup, and how to monitor everything - because I've seen this too: people resting easy because the backup was supposedly running every night, but in fact, the backup had been failing every night for more than 4 years.

Stay Tuned and stay...backupped!

Make Your Own Backup System – Part 1: Strategy Before Scripts

Stefano Marinelli — Fri, 18 Jul 2025 09:00:00 +0200

Backup: Beyond the Simple Copy

For as long as I can remember, backup is something that has been underestimated by far too many people. Between flawed techniques, "Schrödinger's backups" (i.e., never tested, thus both valid and invalid at the same time), and conceptual errors about what they are and how they work (RAID is not a backup!), too much data has been lost due to deficiencies in this area.

Nowadays, backup is often an afterthought. Many rely entirely on "the cloud" without ever asking how - or if - their data is actually protected. It's a detail many overlook, but even major cloud providers operate on a shared responsibility model. Their terms often clarify that while they secure the infrastructure, the ultimate responsibility for protecting and backing up your data lies with you. By putting everything "in the cloud", on clusters owned by other companies, or on distributed Kubernetes systems, backup seems unnecessary. When I sometimes ask developers or colleagues how they handle backups for all this, they look at me as if I'm speaking an archaic, unknown, and indecipherable language. The thought has simply never crossed their minds. But data is not ephemeral; it must be preserved in every way possible.

I've always had a philosophy: data must always be restorable (and as quickly as possible), in an open format (meaning you shouldn't have to buy anything to restore or consult it), and consistent. These points may seem obvious, but they are not.

I have encountered various scenarios of data loss:

Datacenters destroyed by fire – I had 142 servers there, but they were all restored in just a few hours.
Server rooms flooded.
Servers destroyed in earthquakes, often due to collapsing walls.
Increasing incidents of various ransomware attacks.
Intentional damage by entities seeking to create problems.
Mistakes made by administrators, which can happen to anyone.

The risk escalates for servers connected to the internet, like e-commerce and email servers. Here, not only is data integrity crucial, but so is the uninterrupted operation of services. This series of posts will revisit some of my old articles to explain my core ideas on the subject and describe, at least in part, my primary techniques.

Many consider a backup to be a simple copy. I often hear people say they have backups because they "copy the data", but this is often wrong and extremely dangerous, providing a false sense of security. Copying the files of a live database is an almost useless operation, as the result will nearly always be impossible to restore. It is essential to at least perform a proper dump and then transfer that file. Yet, many people do this and will only realize their mistake when they face an emergency and need to restore.

The Backup Plan: Asking the Right Questions

Before touching a single file, you must start with a plan, and that plan starts with asking the right questions:

"How much risk am I willing to take? What data do I need to protect? What downtime can I tolerate in case of data loss? What type and amount of storage space do I have available?"

The first question is particularly critical. A common but risky approach is to store a backup on the same machine that requires backing up. While convenient, this method fails in the event of a machine failure. Even relying on a classic USB drive for daily backups is not foolproof, as these devices are as susceptible to failure as any other hardware component. And contrary to popular belief, even high-end uninterruptible power supplies (UPS) are not immune to catastrophic failures.

Thus, the initial step is to establish a management plan, balancing security and cost. The safest backup is often the one stored farthest from the source machine. However, this approach introduces challenges related to space and bandwidth. While local area network (LAN) backups are relatively straightforward, off-network backups involve additional connectivity considerations. This might lead to a compromise on the amount of data backed up to maintain operational speed during both backup and recovery processes.

Safety doesn't always equate to practicality. For instance, with a 200 MBit/sec connection and 2 TB of backup data, the recovery time could be significant. However, if the goal is not rapid restorability but simply ensuring the data is available, the safest backup is often the one closest to us. That is, a backup we can "touch", disconnect, and consult even when offline.

Therefore, it is essential to develop a backup policy tailored to specific needs, keeping in mind that no 'perfect' solution exists.

The Core Decision: Full Disk vs. Individual Files

When planning a backup strategy, one key decision is whether to back up the entire disk or just individual files. Or both of them. Each approach has its pros and cons.

Entire Disk (or Storage) Backup

Advantages:

Complete Recovery: Restoring a full disk backup can quickly revert a system to its exact previous state, boot loader included.
Integration in Virtualization Systems: If your VM is a single file on a filesystem like ZFS or btrfs, you can simply copy that file (after taking a snapshot) to get a complete backup of the VM. Solutions like Proxmox offer easy management of full disk backups, accessible via command line or web interface.
Flexibility in Virtual Environments: Products like the Proxmox Backup Server offer the ability to recover individual files from a full backup, combining the benefits of both approaches.

Disadvantages:

Downtime for Physical Machines: Often, it's necessary to shut down the machine to create a full disk backup, leading to operational interruptions. A hybrid approach, if the physical host is running FreeBSD for example, is to take a snapshot and copy all the host's datasets. The restore process, however, will require some manual operations.
Large Space Requirements: Full disk backups can consume substantial space, including unnecessary data.
Potential Slowdowns and Compatibility Issues: The backup process can be slow and may encounter issues with non-standard file system configurations.

Individual File Backup

While it might seem simpler, backing up individual files can get complicated.

Advantages:

Basic Utility Use: Possible with standard system utilities like tar, cp, rsync, etc.
Granular Backups: Allows for backing up specific files and comparing them to previous versions.
Delta Copying: Only modified parts of the files are backed up, saving space and reducing data transfer.
Portability and Partial Recovery: Files can be moved individually and partially restored as needed.
Compression and Deduplication: These features are often available at the file or block level.
Operational Continuity: Can be done without shutting down the machine.

Disadvantages:

Storage Space Requirements: Simple copy solutions might require significant storage.
Need for File System Snapshot: For efficient and consistent backups, a snapshot (like native ZFS snapshots, btrfs, LVM Volume snapshots, or Microsoft's VSS) is highly recommended before copying.
Hidden Pitfalls: Issues may not become apparent until a backup is needed. And by then, it may be too late.

The Key to Consistency: The Power of Snapshots

Backing up a "live" file system involves a "start" and an "end" moment. During this time, the data can change, leading to fatal inconsistencies. I've encountered such issues in the past: a large MySQL database was compromised, and I was tasked with its recovery. I confidently took the client's last file-based backup and restored various files (not a native dump). Unsurprisingly, the database failed to restart. The large data file had changed too much between the start and end of the copy, rendering it inconsistent. Fortunately, I also had a proper dump, so I managed to recover from that.

The issue is evident: backing up a live file system is risky. An open database, even a basic one like a browser's history, is highly likely to get corrupted, making the backup useless.

The solution is to create a snapshot of the entire file system before beginning the backup. This approach freezes a consistent "point-in-time" view of the data. To date, using snapshots, I have managed to recover everything.

Here are the methods I've explored over the years:

Native File System Snapshot (e.g., BTRFS or ZFS): If your file system inherently supports snapshots, it's wise to use this feature. It's likely to be the most efficient and technically sound option.
LVM Snapshot: For those using LVM, creating a snapshot of the logical volume is a viable approach. This method can lead to some space wastage and, while I still use it, has occasionally caused the file system to freeze during the snapshot's destruction, necessitating a reboot. This has been a rare but recurring issue across different hardware setups, especially under high I/O load.
DattoBD: I've tracked this tool since its inception. I used it extensively in the past, but I sometimes faced stability issues (kernel panics or the inability to delete snapshots, forcing a reboot). For snapshots with Datto, I often use UrBackup scripts, which are convenient and efficient.

The Architecture: Push or Pull?

A longstanding debate among experts is whether backups should be initiated by the client (push) or requested by the server (pull). In my view, it depends.

Generally, I prefer centralized backup systems on dedicated servers, maintained in highly secure environments with minimal services running. Therefore, I lean towards the "pull" method, where the server connects to the client to initiate the backup.

Ideally, the backup server should not be reachable from the outside. It should be protected, hardened, and only be able to reach the setups it needs to back up. The goal is to minimize the possibility that the backup data could be compromised or deleted in case the client machine itself is attacked (which, unfortunately, is not so rare).

This is not always possible, but there are ways to mitigate this problem. One way is to ensure that machines that must be backed up via "push" (i.e., by contacting the backup server themselves) can only access their own space. More importantly, the backup server, for security reasons, should maintain its own filesystem snapshots for a certain period. In this way, even in the worst-case scenario (workload compromised -> connection to backup server -> deletion of backups to demand a ransom), the backup server has its own snapshots. These server-side snapshots should not be accessible from the client host and should be kept long enough to ensure any compromise can be detected in time.

My Guiding Principles for a Good Backup System

Over the years, I've favored having granular control over backups, often finding the need to recover specific files or emails accidentally deleted by clients. A good backup system, in my opinion, should possess these key features:

Instant Recovery and Speed: The system should enable quick recovery and operate at a high processing speed.
External Storage: Backups must be stored externally, not on the same system being backed up. Still, local snapshots are a good idea for immediate rollbacks.
Security: I avoid using mainstream cloud storage services like Dropbox or Google Drive for primary backups. Own your data!
Efficient Space Management: This includes features like compression and deduplication.
Minimal Invasiveness: The system should require minimal additional components to function.

Conclusion: What's Next

The approach to backup is varied, and in this series, I will describe the main scenarios I usually face. I will start with the backup servers and their primary configurations, then move on to the various software and techniques I use.

But that will begin with the next post, where I'm talk about building the backup server which, of course, will be powered by FreeBSD - like all my backup servers for the last 20 years.

Make Your Own Internet Presence with NetBSD and a 1 euro VPS – Part 1: Your Blog

Stefano Marinelli — Tue, 22 Apr 2025 07:30:36 +0200

Why NetBSD?

For many years, I've been using (and appreciating) NetBSD because it's stable, efficient, and reliable. The codebase has proven its reliability, running without reboots for years without issues. It supports ZFS (though differently than FreeBSD), LVM (useful for those accustomed to it on Linux), the ability to take filesystem snapshots (UFS2, making ZFS less crucial), and it's an excellent virtualization platform. Installation and updates are easy (including via sysupgrade - which I'll cover in a future article). Since it focuses on portable and optimized code (running on ancient architectures requires cleanliness and correctness), it's particularly efficient on low-power devices, like embedded systems or cheap VMs. Therefore, it's one of the best solutions for a small personal setup that can still deliver excellent results and simple management.

Indeed, the market offers very cheap VPS, often with just a single core and little RAM. But a modern single core packs power that a multi-core from just a few years ago could only dream of, and often, the I/O of these machines (a bottleneck for many services) is still decent. I personally use 1 euro per month VPS (VAT included - for those not subject to it, that's less than one euro per month!) with a public IPv4 address and (often) a /64 IPv6 block, ensuring full reachability across the entire network. I'm not providing direct links as I have no affiliations, but netcup's "piko" VPS are among the types I use most often (a 4 euro/month netcup VM handles the entire FediMeteo project), and this type of VM is ideal for our purpose because some providers (like netcup) allow you to upload your own ISO and install your preferred operating system. On VPS like these, I've installed everything - including OmniOS and SmartOS - without problems. And even such a small VPS, with an efficient operating system, can be extremely satisfying.

Why BSSG?

In this article, I'll describe how to create and publish a blog using BSSG as it exemplifies my concept of portability and minimalism. BSSG on NetBSD currently doesn't leverage parallelism provided by tools like GNU Parallel, but for small to medium-sized blogs, this won't be an issue, especially considering these small VMs only have 1 core. Obviously, you can use any Static Site Generator (SSG) (like Hugo, Nikola, 11ty, Pelican, Zola, etc.) - the important thing is to have a static site served by a simple web server.

Let's Start with the Installation

Installing NetBSD is quite straightforward and is clearly covered, complete with explanatory screenshots, in the excellent official NetBSD documentation, which I recommend using as a reference during the process, especially if it's your first time.

In my case, I made sure to use the proposed disk geometry, use the standard automatic partitioning, but enable the "log" and "noatime" options for the filesystem. Both these options will provide a huge advantage in I/O operations, especially with BSSG, as the first enables journaling and the second prevents updating file metadata on every access. BSSG is more I/O bound than CPU bound, so any optimization is beneficial.

Moving forward, I also recommend configuring the network (although installation can be done from packages on the installation ISO). For netcup, you can use DHCPv4 (even though it's a bit slow and sometimes seems to fail, the DHCP client will continue running in the background and eventually work).

For IPv6, I usually configure it manually later, so I'll describe that further down.

I also recommend enabling SSH, adding a regular user (and adding them to the wheel group so they can gain root privileges) - in this case, I'll call the user blog. Also, enable the installation of binary packages, as it will be convenient later to use pkgin to install and update all necessary packages. All these steps are described clearly and in detail in the guide, so I won't detail them here. But they are simple and logical, like all operations on BSD systems.

After installation, reboot. If everything went correctly, you should be able to log in via console or SSH using the "blog" user (or whatever you named it).

First, I suggest configuring the IPv6 address and installing the necessary packages.

For IPv6, in the case of netcup, simply add one of the assigned addresses to the interface. In NetBSD, network interface configurations are stored (similar to OpenBSD) in specific files. For the first virtio interface, the file will be /etc/ifconfig.vioif0.

You need to elevate your privileges to root, open that file with your preferred editor, and add the configuration to the file itself:

nb1euro$ su -l
nb1euro# vi /etc/ifconfig.vioif0

inet6 your-ipv6-addr/64
up

To test everything, perform a reboot and try pinging an IPv6 address (I often use ping6 google.com).

If all goes well, after a few seconds, you should see ping replies, confirming everything is configured correctly.

Regarding packages, the only two strictly necessary ones are bash and a markdown processor (by default, BSSG will use commonmark; otherwise, it can be configured to use pandoc or Markdown.pl). rsync can be useful for deployment. sudo (or doas) can be useful for elevating privileges for certain operations, at least at this stage.

nb1euro$ su -l
nb1euro# pkgin in bash cmark rsync sudo

If you're used to Linux, you can also install the "nano" editor:

nb1euro# pkgin in nano

If sudo was installed, it's now appropriate to grant users in the "wheel" group (like the regular user created during installation) the ability to elevate privileges. Edit the sudoers file (I suggest using the visudo command) and uncomment this line:

## Uncomment to allow members of group wheel to execute any command
%wheel ALL=(ALL:ALL) ALL

At this point, you can switch back to operating as the regular user, downloading and unpacking BSSG:

nb1euro$ ftp https://brew.bsd.cafe/stefano/BSSG/archive/0.15.1.tar.gz
nb1euro$ tar zxfv 0.15.1.tar.gz

Now that BSSG is ready, just initialize a directory with the structure for the new site:

nb1euro$ cd bssg
nb1euro$ ./bssg.sh init /home/blog/myblog

Everything is set to start generating your blog. I recommend reading BSSG's README.md. There are many options, themes, etc., but to get started, you just need to set the site's public URL. For example, if the site will be published as myblog.example.com - just create a file at /home/blog/myblog/config.sh.local (the path defined by the init command) and set the public URL:

SITE_URL="https://myblog.example.com"

This way, all URLs will be absolute URLs, which is necessary to ensure the correct functioning of RSS feeds, sitemaps, etc. This setting assumes HTTPS - if you just want to test the site over HTTP, simply use http and then, optionally, change it to https and regenerate the site later.

You can already create your first test post, directly from the BSSG directory:

nb1euro$ ./bssg.sh post

The system will use nano if it's installed, otherwise it will use vi. Don't worry, in the latter case, BSSG will write the procedure for exiting vi as the post's text 🙂

Once you save the post, BSSG will automatically generate the site. If everything went well, the /home/blog/myblog/output directory will contain the final result. We are therefore ready for the first deployment, which can be done in many different ways. I will cover three:

Using bozohttpd, present by default in NetBSD's base system. It can be used via inetd (launching an httpd process for each connection) or as a daemon. I'll describe the first option, showing in the final benchmarks how, even when used as a daemon, it remains a less performant solution.
Using nginx
Using Caddy

First, it's advisable to obtain a certificate to configure and use HTTPS. If you only want to test using HTTP, this part can be safely bypassed. For solutions 1 and 2, I'll use certbot, which is well-known to many users with Linux experience. Caddy, on the other hand, manages certificates automatically, so there's no need for other solutions and thus no need to install certbot.

nb1euro$ sudo pkgin in py313-certbot

To use bozohttpd, no further installation is necessary. At this point, the options diverge.

Using NetBSD's Integrated httpd

bozohttpd is integrated into NetBSD and, by default, can be launched directly via inetd. This solution, while not extremely efficient or scalable, is simple and requires few resources. It's fine if you expect only a few visits per day, but when used via inetd, the initial latency for each connection is tangible. It can still be useful for some tests or small deployments.

The /etc/inetd.conf file already contains the options to handle this situation:

#http           stream  tcp     nowait:600      _httpd  /usr/libexec/httpd      httpd /var/www
#http           stream  tcp6    nowait:600      _httpd  /usr/libexec/httpd      httpd /var/www

By uncommenting these two lines and restarting inetd (service inetd restart), the server will start responding to HTTP requests on both IPv4 and IPv6.

If you want to add HTTPS support, no problem. Just request a certificate via certbot and specify the webroot.

Run:

nb1euro$ sudo certbot-3.13 certonly

Choose option 2 - the one where you specify the webroot - enter the domain, and when prompted, provide /var/www/ as the webroot.

The certificate will be created. Then, modify the /etc/inetd.conf file to also include support for HTTPS, adding two lines similar to these (obviously, change the certificate paths):

https            stream  tcp     nowait:600      _httpd  /usr/libexec/httpd      httpd -Z /usr/pkg/etc/letsencrypt/live/myblog.example.com/fullchain.pem /usr/pkg/etc/letsencrypt/live/myblog.example.com/privkey.pem /var/www
https            stream  tcp6    nowait:600      _httpd  /usr/libexec/httpd      httpd -Z /usr/pkg/etc/letsencrypt/live/myblog.example.com/fullchain.pem /usr/pkg/etc/letsencrypt/live/myblog.example.com/privkey.pem /var/www

Warning: httpd will run with the permissions of the _httpd user, so make sure all certificates are readable by that user:

nb1euro# chown -R _httpd /usr/pkg/etc/letsencrypt/

Restart inetd, and the server will also respond over HTTPS.

To make your blog public, simply copy the files from the site's output directory to /var/www/ - this time using sudo to bypass permission issues:

nb1euro$ sudo rsync -avhHPx /home/blog/myblog/output/ /var/www/

The site will be immediately visible.

Using nginx

Nginx is fast and efficient, and the performance difference is noticeable (some benchmarks follow below). For an efficient setup ready for a high number of visits, it's advisable to use a web server suited for the purpose, just like nginx.

First, install nginx and the certbot plugin for nginx. This will simplify the installation and renewal of certificates:

nb1euro$ sudo pkgin in py313-certbot-nginx nginx

Copy the startup script to /etc/rc.d - as indicated by the post-installation message. In NetBSD, this operation must be done manually, but it's always pointed out:

nb1euro$ sudo cp /usr/pkg/share/examples/rc.d/nginx /etc/rc.d

Warning: If you previously used httpd from inetd following the previous solution, you must disable it in inetd.conf and restart inetd to free up ports 80 and 443.

Now you can create a virtual host for our new site.

nb1euro$ sudo vi /usr/pkg/etc/nginx/nginx.conf

and add, at the end of the file and before the final closing curly brace:

server {
        listen 80;
        # If you also have configured IPv6 support
        listen [::]:80;

        root /var/www;
        index index.html index.htm;

        server_name myblog.example.com;

        # If you want a long cache for media and css - be careful, this means that if you change to a new theme, it might not be visible immediately as the browser might still use the old cached one
        location ~* \.(jpg|jpeg|png|gif|ico|css|js)$ {
            expires 30d;
            add_header Cache-Control "public, no-transform";
        }

        location / {
                try_files $uri $uri/ =404;
        }
}

Now, it's time to configure the system to enable nginx. Just edit /etc/rc.conf:

nb1euro$ sudo vi /etc/rc.conf

and add:

nginx=YES

Now, you can start nginx:

nb1euro$ sudo service nginx start

Nginx will start listening on port 80. Generating and installing the certificate is very simple:

nb1euro$ sudo certbot-3.13 --nginx -d myblog.example.com

This command will request the certificate and install it, so nginx will already be configured to use it.

As with the previous method, to make your blog public, simply copy the files from the site's output directory to /var/www/ - using sudo to bypass permission issues:

nb1euro$ sudo rsync -avhHPx /home/blog/myblog/output/ /var/www/

The site will be immediately visible.

Using Caddy

Caddy is a convenient and all-in-one solution, efficient and fast. It's packaged for NetBSD and allows you to go online in a flash. I won't delve into the configuration because there are many tutorials (including the official ones), but you just need to install it and run it:

nb1euro$ sudo pkgin in caddy

Once installed, go to the directory you want to serve (e.g., /var/www or directly /home/blog/myblog/output) and run:

nb1euro$ sudo caddy file-server --domain myblog.example.com

Caddy will start, request the certificate, and begin serving your blog over HTTPS as well. To install Caddy as a service (i.e., with a configuration file, etc.), you can proceed similarly to how it's done on Linux. The NetBSD Caddy package doesn't include the rc.d script, but you can copy and paste one (into /etc/rc.d/caddy) from a thread posted on UnitedBSD.

Performance Comparison

I performed some performance tests on these solutions. Here are the results, on a single-core 1 euro/month VPS, from my home connection (which also has its own limitations):

NetBSD httpd via inetd:

Running 10s test @ https://myblog.example.com/
  4 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   213.52ms  173.10ms   1.11s    76.01%
    Req/Sec    12.92      9.19    50.00     75.91%
  371 requests in 10.10s, 1.39MB read
Requests/sec:     36.72
Transfer/sec:    140.65KB

These numbers are quite poor, linked to high latency caused by having to launch bozohttpd for each incoming connection.

NetBSD httpd as a daemon:

Running 10s test @ https://myblog.example.com/
  4 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    35.74ms    6.96ms 108.80ms   81.36%
    Req/Sec    18.29      9.45    50.00     70.88%
  676 requests in 10.10s, 2.53MB read
Requests/sec:     66.92
Transfer/sec:    256.32KB

Here the situation is decidedly better, but not exceptional. httpd isn't designed for high loads or performance.

Nginx as a daemon, 1 worker:

Running 10s test @ https://myblog.example.com/
  4 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    30.69ms    4.87ms  64.14ms   66.01%
    Req/Sec   379.39     65.94   464.00     90.91%
  15026 requests in 10.04s, 56.50MB read
Requests/sec:   1496.65
Transfer/sec:      5.63MB

Here we are on another level, showing truly solid performance. This type of result can handle significantly high loads without particular difficulty. The efficiency of both NetBSD and nginx pays off.

Caddy:

Running 10s test @ https://myblog.example.net/
  4 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    32.10ms    5.75ms  95.04ms   87.44%
    Req/Sec   362.74     64.29   434.00     91.67%
  14374 requests in 10.05s, 54.63MB read
Requests/sec:   1430.82
Transfer/sec:      5.44MB

Caddy shows results comparable to nginx, so the choice between them depends solely on the type of configuration you want to achieve and the experience each person has with the specific platforms.

Conclusion: Efficient Minimalism

We've seen how it's possible to create a personal, professional, and performant online presence with minimal investment. This solution, based on NetBSD and a 1€/month VPS, offers several advantages:

Negligible Cost: For 12€ per year, you can have a website (and more!) completely under your control.
Surprising Performance: As demonstrated by the benchmarks, excellent performance can be achieved even with limited resources (up to 1400-1500 requests/second with nginx or Caddy).
Security and Stability: NetBSD is renowned for its reliability and security, fundamental characteristics for any online service.
Total Control: Unlike free blogging platforms, you have full control over every aspect of your site.
Learning Experience: Managing a BSD system allows you to acquire valuable system administration skills.

This minimalist configuration demonstrates that you don't need to invest in expensive cloud solutions or oversized VPS to have a quality online presence. In an era where the tendency is to think "moooar powaaaar = better results", NetBSD reminds us that efficiency and good design can yield excellent results even with limited resources.

After all, you don't need a thousand-node cloud to write something worth reading.

In the upcoming articles in this series, we will explore how to expand this basic installation with other useful services and how to keep the system updated and secure over time.

Managing ZFS Full Pool Issues with Reserved Space

Stefano Marinelli — Thu, 28 Nov 2024 20:25:00 +0100

Yesterday morning, I received a panicked call from a developer:
"I accidentally filled up the storage, and now I can't perform any operations! My ZFS pool is full!"

I immediately reassured them because I had anticipated this kind of issue. One of the things I almost always do when managing ZFS file systems is to reserve space in a specially created dataset.

This is because ZFS, like all CoW (Copy-on-Write) file systems, can find itself unable to free up space when completely full. By using reserved space, I can always free it up and delete other data, restoring the system to normal operations.

To reserve space, simply create a dataset and assign it a reserved size. Of course, this dataset should not be used for anything else; otherwise, the entire purpose would be defeated.

To create it and reserve space, you only need two simple commands. For example:

zfs create zroot/reserved
zfs set reservation=5G zroot/reserved

This creates the dataset and assigns it 5 GB of reserved space.

Here’s the situation before the operation:

zfs list zroot
NAME    USED  AVAIL  REFER  MOUNTPOINT
zroot  3.02G   109G    96K  /zroot

And here’s the situation after:

zfs list zroot
NAME    USED  AVAIL  REFER  MOUNTPOINT
zroot  8.02G   104G    96K  /zroot

As you can see, the 5 GB are removed from the available space and marked as used, but they are actually empty.

In case of a full file system, you can delete this dataset (or reduce its size) to return to normal file system operation.

Even with this technique, I still recommend not filling ZFS pools beyond 80% of their capacity, as performance degrades significantly past that point.

Moving an entire FreeBSD installation to a new host or VM in a few easy steps

Stefano Marinelli — Mon, 16 Sep 2024 09:41:00 +0200

FreeBSD, especially when installed on ZFS, is incredibly simple to manage, back up, and move to a different system.

I often find myself having to move an entire system from one host to another, from a physical host to a VM, or vice versa, from a VM to a physical host.

By following the approach of keeping the operating system clean and running all services within jails, this operation is usually quite simple: install the operating system on the new host, perform the few necessary configurations (like pf and network interfaces), transfer the jail datasets, restart the jails on the new host, and update the DNS.

However, sometimes I need to move the entire host, including the original operating system. This article will describe this operation when the FreeBSD system uses ZFS as its root file system. This process isn't much more difficult, but there are a few details to keep in mind. Below is a breakdown of the two bootloader scenarios:

UEFI Bootloader
BIOS Bootloader (non-UEFI)

UEFI Bootloader

Start from an ISO or image of mfsbsd. Partition the disk according to the layout of the source system. If the source system is a standard FreeBSD (ZFS) installation, partition the destination disk (assumed to be da0) as follows. WARNING: The first command will destroy any existing partition table on the disk.

gpart destroy -F /dev/da0
gpart create -s gpt da0
gpart add -t efi -s 200M -l efiboot0 da0
newfs_msdos -F 32 -c 1 /dev/da0p1
mount -t msdosfs /dev/da0p1 /mnt
mkdir -p /mnt/EFI/BOOT
cp /boot/loader.efi /mnt/EFI/BOOT/BOOTX64.efi
umount /mnt

Create additional partitions for swap and ZFS:

gpart add -a 1m -t freebsd-swap -s 2g -l swap0 da0
gpart add -a 1m -t freebsd-zfs -l zfs0 da0

Create the ZFS pool:

zpool create -O compression=zstd -O atime=off zroot /dev/gpt/zfs0

On the source system, create a snapshot of all datasets:

zfs snapshot -r zroot@snap01

Important: You will need to connect to the source system using a user with privileges over the entire filesystem. For simplicity, I'll use root, but it's not recommended to leave root accessible via SSH. Limit access to private keys only. To allow root login over SSH, edit /etc/ssh/sshd_config and set PermitRootLogin yes. After moving the server, remember to disable root access.
If mbuffer is installed on the source system, it can help improve the performance of the transfer. If not, remove it from the next pipe command, which should be run from the destination system. The -s will set the block-size, the -m will set the buffer size (128MB, here):

ssh root@sourceip "zfs send -RLv zroot@snap01 | mbuffer -s 128k -m 128M" | zfs receive -F zroot

You have two options now:
If the source server was idle and no further synchronization is required, proceed to the next step.
If the transfer took a long time and services on the source server need to be stopped (e.g., databases), stop what you can, take another snapshot (e.g., zfs snapshot -r zroot@snap02), and synchronize the differences by running another incremental send-receive from the destination server:

ssh root@sourceip "zfs send -RLv -i zroot@snap01 zroot@snap02 | mbuffer -s 128k -m 128M" | zfs receive -F zroot

Set the boot filesystem:

zpool set bootfs=zroot/ROOT/default zroot

BIOS Bootloader

Start from an ISO or image of mfsbsd. Partition the disk according to the layout of the source system. If the source system is a standard FreeBSD (ZFS) installation, partition the destination disk (assumed to be da0) as follows. WARNING: The first command will destroy any existing partition table on the disk.

gpart destroy -F /dev/da0
gpart create -s gpt da0

Create the necessary partitions:

gpart add -a 1m -t freebsd-boot -s 512k -l boot da0
gpart add -a 1m -t freebsd-swap -s 2g -l swap0 da0
gpart add -a 1m -t freebsd-zfs -l zfs0 da0

Create the ZFS pool:

zpool create -O compression=zstd -O atime=off zroot /dev/gpt/zfs0

On the source system, create a snapshot of all datasets:

zfs snapshot -r zroot@snap01

Important: You will need to connect to the source system using a user with privileges over the entire filesystem. For simplicity, I'll use root, but it's not recommended to leave root accessible via SSH. Limit access to private keys only. To allow root login over SSH, edit /etc/ssh/sshd_config and set PermitRootLogin yes.
If mbuffer is installed on the source system, it can help improve the performance of the transfer. If not, remove it from the next pipe command, which should be run from the destination system:

ssh root@sourceip "zfs send -RLv zroot@snap01 | mbuffer -s 128k -m 128M" | zfs receive -F zroot

You have two options now:
If the source server was idle and no further synchronization is required, proceed to the next step.
If the transfer took a long time and services on the source server need to be stopped (e.g., databases), stop what you can, take another snapshot (e.g., zfs snapshot -r zroot@snap02), and synchronize the differences by running another incremental send-receive from the destination server:

ssh root@sourceip "zfs send -RLv -i zroot@snap01 zroot@snap02 | mbuffer -s 128k -m 128M" | zfs receive -F zroot

Install the bootloader on the destination server’s disk:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

Set the boot filesystem:

zpool set bootfs=zroot/ROOT/default zroot

At this point, everything should be ready for boot. Reboot and verify that the various mountpoints (e.g., swap, etc.) and network interface settings are correct. If everything works fine, the server will be an exact copy of the original.

Moving or duplicating an entire FreeBSD system is simple, fast, and reliable. The ability to use ZFS incremental replication is an excellent method for minimizing downtime, especially for large systems. The initial copy will happen "live" while the source system continues to run without issues. Subsequent incrementals will be much faster, making the switch-over process seamless and efficient.

Make Your Own CDN with NetBSD

Stefano Marinelli — Tue, 03 Sep 2024 01:41:00 +0200

Introduction

This article is a spin-off from a previous post on how to create a self-hosted CDN, based on OpenBSD, but this time we'll focus on using NetBSD. The idea is to create reverse proxies with local caching. These proxies would cache the content on the first request and serve it directly afterward. The proxies would be distributed across different regions, and the DNS would route requests to the nearest proxy based on the caller’s location. All this is achieved without relying on external CDNs, using self-managed tools instead.

NetBSD is a lightweight, stable, and secure operating system that supports a wide range of hardware, making it an excellent choice for a caching reverse proxy. Devices that other operating systems may soon abandon, such as early Raspberry Pi models or i386 architecture, are still fully supported by NetBSD and will continue to be so. Additionally, NetBSD is an outstanding platform for virtualization (using Xen or qemu/nvmm) and deserves more attention than it currently receives.

The choice of Varnish is based on several factors, with the main ones being the ability to keep the cache in RAM (which means it can run on read-only systems) and the ability to flush the cache remotely. For example, with each change to my blog, I can choose whether to perform an immediate flush (such as for a new article or an error) or wait for the cache's "natural" expiration (such as for a typo or minor, non-critical changes).

While I won't detail the installation process for NetBSD, as it depends heavily on the hardware you have available, I will guide you through setting up a self-hosted CDN using NetBSD, Varnish, nginx, and the acme.sh or lego tool for SSL certificate management.

Installation

During the installation of NetBSD, ensure that you enable support for binary package management. This will install pkgin, a tool that simplifies package management on NetBSD. If you skip this step during installation, you can still install pkgin later, but it's easier to let the installer handle it.

Once your system is up and running, use pkgin to install the necessary packages: Varnish, nginx, and, depending on your preference, either acme.sh or Go (if you plan to compile lego). Although lego is not available as a precompiled package, you can easily compile it locally using Go, but for simplicity, I recommend using acme.sh.

For this setup, I will present two methods for generating and renewing certificates: using acme.sh or compiling lego - to reach the same final outcome as the OpenBSD article.

Option 1: Using acme.sh (Recommended)

acme.sh is a simple, yet powerful, shell script that handles certificate generation and renewal with ease. To install acme.sh:

pkgin in acmesh varnish nginx

Once acme.sh is installed, you can proceed to configure it for certificate management. It supports DNS authentication and integrates with many DNS providers, making it a flexible choice.

Option 2: Compiling Lego

If you prefer to use lego, you will need to compile it manually, as it is not available as a precompiled package for NetBSD. First, install Go and other necessary packages:

pkgin in go varnish nginx

To compile lego, you’ll need some disk space. Since the /tmp directory in NetBSD is often mounted as tmpfs (using RAM), you may run out of space during compilation if your system has limited memory. You can temporarily disable tmpfs by editing /etc/fstab and commenting out the relevant line:

#tmpfs           /tmp    tmpfs   rw,-m=1777,-s=ram%25

After rebooting, compile lego:

export GO111MODULE=on
go122 install github.com/go-acme/lego/v4/cmd/lego@latest

cp go/bin/lego /usr/pkg/bin/

Once lego is compiled and installed, you can uncomment the /tmp line in /etc/fstab and reboot again.

Configuring Varnish and nginx

First, copy the necessary rc.d scripts for nginx and Varnish:

cp /usr/pkg/share/examples/rc.d/nginx /etc/rc.d/
cp /usr/pkg/share/examples/rc.d/varnishd /etc/rc.d/

Then, add the following to /etc/rc.conf to enable and configure nginx and Varnish:

nginx=YES
varnishd=YES
varnishd_flags="-f /usr/pkg/etc/varnish/default.vcl -T localhost:9999 -a "/var/run/varnish.sock",user=nginx,group=varnish,mode=660 -s default,500m"

In this configuration, Varnish listens on a Unix socket, and nginx connects to it. This approach is more efficient and helps avoid some issues that may arise when exposing Varnish over an IP/port.

Creating the Varnish VCL Configuration

Next, create the VCL configuration file for Varnish at /usr/pkg/etc/varnish/default.vcl:

vcl 4.1;
import std;

# Backend - it-notes.dragas.net
backend it_notes {
    .host = "myBackendIP";
    .port = "80";
}

# ACL - purge - it-notes.dragas.net
acl purge_it_notes {
    "allowedToPurge_IP";
}

sub vcl_recv {
    # it-notes.dragas.net
    if (req.http.Host == "it-notes.dragas.net") {
        set req.backend_hint = it_notes;
        set req.http.Host = "it-notes.dragas.net";

        # PURGE - it-notes.dragas.net
        if (req.method == "PURGE") {
            std.log("Purge request received for " + req.url);

            if (!std.ip(req.http.X-Forwarded-For, "0.0.0.0") ~ purge_it_notes) {
                return (synth(405, "Not allowed."));
            }

            if (req.url == "/" || req.url == "/*") {
                ban("req.http.host == " + req.http.host);
                return(synth(200, "Entire cache has been cleared."));
            }
            return (purge);
        }

    } else {
        # Other domains - 404
        return (synth(404, "Domain not found"));
    }

    if (req.method != "GET" && req.method != "HEAD") {
        return (pipe);
    }

    return (hash);
}

sub vcl_backend_response {
    # TTL - it-notes.dragas.net
    if (bereq.http.host == "it-notes.dragas.net") {
        if (bereq.url ~ "\.(gif|jpg|jpeg|png|webp|ico|css|js)$") {
            set beresp.ttl = 1w;
            set beresp.grace = 1d;
            set beresp.keep = 7d;
            unset beresp.http.Set-Cookie;
            unset beresp.http.Cache-Control;
            set beresp.http.Cache-Control = "public, max-age=604800";
        } else {
            set beresp.ttl = 15m;
            set beresp.grace = 48h;
            set beresp.keep = 7d;
        }
    }

    # Remove some headers
    unset beresp.http.Server;
    unset beresp.http.X-Powered-By;
    unset beresp.http.Via;

    return (deliver);
}

sub vcl_deliver {
    # Add X-Cache header
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
    } else {
        set resp.http.X-Cache = "MISS";
    }

    std.log("Delivering content for " + req.url + " - Cache: " + resp.http.X-Cache);

    # Remove Varnish headers
    unset resp.http.Via;
    unset resp.http.X-Varnish;

    return (deliver);
}

sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }
    return (lookup);
}

sub vcl_hit {
    return (deliver);
}

sub vcl_miss {
    return (fetch);
}

sub vcl_purge {
    std.log("Purge executed for " + req.url);
    return (synth(200, "Purge successful"));
}

sub vcl_synth {
    set resp.http.Content-Type = "text/html; charset=utf-8";
    set resp.http.Retry-After = "5";
    synthetic({"<!DOCTYPE html>
        <html>
            <head>
                <title>"} + resp.status + " " + resp.reason + {"</title>
            </head>
            <body>
                <h1>Status "} + resp.status + " " + resp.reason + {"</h1>
                <p>"} + resp.reason + {"</p>
                <h3>Guru Meditation:</h3>
                <p>XID: "} + req.xid + {"</p>
                <hr>
                <p>Varnish cache server</p>
            </body>
        </html>
    "});
    return (deliver);
}

Configuring nginx

Now, modify the nginx configuration file at /usr/pkg/etc/nginx/nginx.conf. Set the number of worker processes to "auto" to take advantage of all server cores, and configure the reverse proxy for your site(s). Here's an example configuration:

[...]
worker_processes  auto;
[...]

server {
    server_name it-notes.dragas.net;

    location / {
        proxy_method $request_method;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_pass http://unix:/var/run/varnish.sock;
    }

    access_log /var/log/nginx/access.it-notes.dragas.net.log;
    error_log /var/log/nginx/error.it-notes.dragas.net.log;

    listen [::]:443 ssl;
    listen 443 ssl;
    http2 on;
    # If you're using acme.sh, just change the location of the certificates
    ssl_certificate /root/.lego/certificates/it-notes.dragas.net.crt;
    ssl_certificate_key /root/.lego/certificates/it-notes.dragas.net.key;
}

server {
    if ($host = it-notes.dragas.net) {
        return 301 https://$host$request_uri;
    }
    server_name it-notes.dragas.net;
    listen 80;
    listen [::]:80;
    return 404;
}

Starting Varnish and nginx

Finally, start the Varnish and nginx services:

service varnishd start
service nginx start

If everything is configured correctly, both Varnish and nginx will be up and running, ready to handle incoming connections.

Conclusion

Congratulations, you have successfully set up your own CDN on NetBSD. This solution is lightweight, stable, and fully under your control, allowing you to break free from the constraints of major service providers. With NetBSD's broad hardware support and minimal overhead, this setup can run on a wide variety of devices, making it a versatile choice for self-hosted solutions.

I'm using it as a test and as a read-only root filesystem with a RAM-only local cache for my blog on a Raspberry Pi Zero W (first edition), and as soon as I get the new FTTH, I'll probably make it accessible via IPv6 for Italy, putting it physically into production.

If your goal is geo-replication, you can use DNS providers that offer location-based routing or set up your own DNS infrastructure to manage and resolve requests according to the user’s location. With multiple reverse proxies, separate DNS servers, and a well-configured cache, you can achieve a highly resilient system with minimal risk of a single point of failure.

Make Your Own CDN with OpenBSD Base and Just 2 Packages

Stefano Marinelli — Thu, 29 Aug 2024 01:41:00 +0200

Introduction

This article is a "spin-off" from the previous post "Building a Self-Hosted CDN for BSD Cafe Media," which I recommend reading, based on FreeBSD. In that article, I showed how I addressed the issue of geo-replication and geo-distribution of BSD Cafe media content. If you prefer a NetBSD-based setup, there's an article that describes how to do it.

The internet today relies TOO MUCH on just a few big players. When one of them stops working, half the world is impacted because too many services, in my opinion, depend on them. “Too big to fail,” some might say. “Single Point of Failure,” I respond."

The strength of the internet has always been its extreme decentralization, which is now less evident due to this phenomenon.

In this article, I want to show how easy it is to create a self-hosted CDN using OpenBSD and just two external packages: Varnish and Lego.

Actually, only one package is truly needed (Varnish), and SSL certificates could be generated using the built-in acme-client in OpenBSD. However, this might be limiting since acme-client handles certificate generation via traditional methods (using a file in .well-known), but when dealing with a CDN and several reverse proxies listening, you don’t have perfect control over which one will receive the certificate generation validation request.

For convenience and practicality, I’ll use the excellent Lego tool—a Go application that supports many DNS authentication methods, including PowerDNS.

Installation

The steps are quite simple. After installing and updating OpenBSD (using the syspatch command), start by installing the two packages:

obcdn# pkg_add lego varnish
quirks-7.14:updatedb-0p0: ok
quirks-7.14 signed on 2024-08-24T13:35:23Z
quirks-7.14: ok
lego-4.16.1: ok
varnish-7.4.2:bzip2-1.0.8p0: ok
varnish-7.4.2:pcre2-10.37p2: ok
useradd: Warning: home directory `/var/varnish' doesn't exist, and -m was not specified
varnish-7.4.2: ok
The following new rcscripts were installed: /etc/rc.d/varnishd
See rcctl(8) for details.

The next step is to enable Varnish:

rcctl enable varnishd

Varnish has a generic startup script, but it's best to customize the startup options. To do this, modify the /etc/rc.conf.local file:

varnishd_flags="-j unix,user=_varnish,ccgroup=_varnish -f /etc/varnish/default.vcl -T localhost:9999 -a localhost:8080 -s default,500m"

This configuration sets Varnish with a 500 MB cache and listens on localhost, port 8080.

Next, rename the default VCL file to prepare for your own content:

mv /etc/varnish/default.vcl /etc/varnish/default.vcl.distrib

Create a new default.vcl file. Below is an example based on this blog (at the time of writing). You’ll need to adapt it according to your needs, especially if there are cookies or other dynamic content. Note that Varnish will fetch data from a specific backend accessed in http. If privacy is needed, consider creating a VPN between the backend and Varnish, as briefly mentioned in the previous article:

vcl 4.1;
import std;

# Backend - it-notes.dragas.net
backend it_notes {
    .host = "myOriginalServer";
    .port = "80";
}

# ACL - purge - it-notes.dragas.net
acl purge_it_notes {
    "authorizedIPForCachePurge";
}

sub vcl_recv {
    # it-notes.dragas.net
    if (req.http.Host == "it-notes.dragas.net") {
        set req.backend_hint = it_notes;
        set req.http.Host = "it-notes.dragas.net";

        # PURGE - it-notes.dragas.net
        if (req.method == "PURGE") {
            std.log("Purge request received for " + req.url);

            if (!std.ip(req.http.X-Forwarded-For, "0.0.0.0") ~ purge_it_notes) {
                return (synth(405, "Not allowed."));
            }

            if (req.url == "/" || req.url == "/*") {
                ban("req.http.host == " + req.http.host);
                return(synth(200, "Entire cache has been cleared."));
            }
            return (purge);
        }

    } else {
        # Other domains - 404
        return (synth(404, "Domain not found"));
    }

    if (req.method != "GET" && req.method != "HEAD") {
        return (pipe);
    }

    return (hash);
}

sub vcl_backend_response {
    # TTL - it-notes.dragas.net
    if (bereq.http.host == "it-notes.dragas.net") {
        if (bereq.url ~ "\.(gif|jpg|jpeg|png|webp|ico|css|js)$") {
            set beresp.ttl = 1w;
            set beresp.grace = 1d;
            set beresp.keep = 7d;
            unset beresp.http.Set-Cookie;
            unset beresp.http.Cache-Control;
            set beresp.http.Cache-Control = "public, max-age=604800";
        } else {
            set beresp.ttl = 15m;
            set beresp.grace = 48h;
            set beresp.keep = 7d;
        }
    }

    # Remove some headers
    unset beresp.http.Server;
    unset beresp.http.X-Powered-By;
    unset beresp.http.Via;

    return (deliver);
}

sub vcl_deliver {
    # Add X-Cache header
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
    } else {
        set resp.http.X-Cache = "MISS";
    }

    std.log("Delivering content for " + req.url + " - Cache: " + resp.http.X-Cache);

    # Remove Varnish headers
    unset resp.http.Via;
    unset resp.http.X-Varnish;

    return (deliver);
}

sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }
    return (lookup);
}

sub vcl_hit {
    return (deliver);
}

sub vcl_miss {
    return (fetch);
}

sub vcl_purge {
    std.log("Purge executed for " + req.url);
    return (synth(200, "Purge successful"));
}

sub vcl_synth {
    set resp.http.Content-Type = "text/html; charset=utf-8";
    set resp.http.Retry-After = "5";
    synthetic({"<!DOCTYPE html>
        <html>
            <head>
                <title>"} + resp.status + " " + resp.reason + {"</title>
            </head>
            <body>
                <h1>Status "} + resp.status + " " + resp.reason + {"</h1>
                <p>"} + resp.reason + {"</p>
                <h3>Guru Meditation:</h3>
                <p>XID: "} + req.xid + {"</p>
                <hr>
                <p>Varnish cache server</p>
            </body>
        </html>
    "});
    return (deliver);
}

Start Varnish and check if it starts correctly:

rcctl start varnishd

Generating SSL Certificates

Before configuring relayd, you’ll need to generate SSL certificates. Lego supports many DNS providers and provides clear and comprehensive examples, so I suggest reading its README file.

Important: Certificates generated by Lego are not directly compatible with relayd, so you must generate them in the correct format. Add the -k rsa4096 flag to the Lego command to obtain certificates compatible with relayd.

Once generated, the certificates will be in a subdirectory of the directory from which the command was launched. For example, if the command is run as root (which is unnecessary, but just for the example), the certificates will be in /root/.lego/certificates/.

relayd expects certificates in a specific location. Copy them to the appropriate directories. In my example:

obcdn# cp /root/.lego/certificates/it-notes.dragas.net.crt /etc/ssl/
obcdn# cp /root/.lego/certificates/it-notes.dragas.net.key /etc/ssl/private/

Remember to copy the files to the correct directories when renewing. You can also create symbolic links, this will make your life easier but it’s less secure.

Configuring relayd

It’s time to configure relayd. The file is /etc/relayd.conf, and here’s an example configuration:

log state changes
prefork 10

table <itnotes> { 127.0.0.1 }

http protocol "http" {
    match request header append "X-Forwarded-For" value "$REMOTE_ADDR"
    match request header append "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"

    match request path "/*.atom" tag "CACHE"
    match request path "/*.css" tag "CACHE"
    match request path "/*.gif" tag "CACHE"
    match request path "/*.html" tag "CACHE"
    match request path "/*.ico" tag "CACHE"
    match request path "/*.jpg" tag "CACHE"
    match request path "/*.webp" tag "CACHE"
    match request path "/*.js" tag "CACHE"
    match request path "/*.png" tag "CACHE"
    match request path "/*.rss" tag "CACHE"
    match request path "/*.svg" tag "CACHE"
    match request path "/*.xml" tag "CACHE"
    match request path "/*.ttf" tag "CACHE"
    match request path "/*.woff2" tag "CACHE"

    match response tagged "CACHE" header set "Cache-Control" value "public, max-age=604800"

    pass request header "Host" value "it-notes.dragas.net" forward to <itnotes>
}

http protocol "https" {
    match request header append "X-Forwarded-For" value "$REMOTE_ADDR"
    match request header append "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"

    match response header set "Referrer-Policy" value "no-referrer"
    match response header set "X-Content-Type-Options" value "nosniff"
    match response header set "X-Download-Options" value "noopen"
    match response header set "X-Frame-Options" value "SAMEORIGIN"
    match response header set "X-Permitted-Cross-Domain-Policies" value "none"
    match response header set "X-XSS-Protection" value "1; mode=block"
    match response header set "Strict-Transport-Security" value "max-age=15552000; includeSubDomains; preload"

    match request path "/*.atom" tag "CACHE"
    match request path "/*.css" tag "CACHE"
    match request path "/*.gif" tag "CACHE"
    match request path "/*.html" tag "CACHE"
    match request path "/*.ico" tag "CACHE"
    match request path "/*.jpg" tag "CACHE"
    match request path "/*.webp" tag "CACHE"
    match request path "/*.js" tag "CACHE"
    match request path "/*.png" tag "CACHE"
    match request path "/*.rss" tag "CACHE"
    match request path "/*.svg" tag "CACHE"
    match request path "/*.xml" tag "CACHE"
    match request path "/*.ttf" tag "CACHE"
    match request path "/*.woff2" tag "CACHE"

    match response tagged "CACHE" header set "Cache-Control" value "public, max-age=604800"
    tcp { nodelay, sack, socket buffer 65536, backlog 100 }

    tls { keypair "it-notes.dragas.net" }

    pass request header "Host" value "it-notes.dragas.net" forward to <itnotes>
}

relay "http" {
    listen on vio0 port 80
    protocol "http"

    forward to <itnotes> port 8080
}

relay "https" {
    listen on vio0 port 443 tls
    protocol "https"

    forward to <itnotes> port 8080
}

relay "https6" {
    listen on my:ip:v6:address::1 port 443 tls
    protocol "https"

    forward to <itnotes> port 8080
}

The content of the file is quite self-explanatory. Note that:

vio0 is the interface name—it should be modified based on the interface where relayd needs to listen.
I’ve configured relayd to listen on port 80 as well.
IPv4 and IPv6 listeners are separated. If IPv6 is not configured, simply comment out that part. Please, if you can, use IPv6. In a fairer world, everyone would have the right to at least one class of public addresses without having to pay exorbitant fees.
The "keypair" must correspond to the certificate and key names in /etc/ssl and /etc/ssl/private.

Test the configuration, enable, and start relayd:

obcdn# relayd -n
configuration OK
obcdn# rcctl enable relayd
obcdn# rcctl start relayd
relayd(ok)

Final Checks

The stack is ready. A ps command will show the process status:

obcdn# ps aux
USER       PID %CPU %MEM   VSZ   RSS TT  STAT   STARTED       TIME COMMAND
root         1  0.0  0.0   920   276 ??  I       7:10PM    0:00.01 /sbin/init
[...]
_varnish 44999  0.0  0.2  2256  2424 ??  S       8:27PM    0:00.05 varnishd: Varnish-Mgt -i obcdn.my.domain (varnishd)
_varnish 54481  0.0  8.8 34500 88876 ??  S       8:27PM    0:00.60 varnishd: Varnish-Child -i obcdn.my.domain (varnishd)
root     57753  0.0  0.5  4080  4820 ??  IU      8:32PM    0:00.03 /usr/sbin/relayd
_relayd  95940  0.0  0.4  2152  3892 ??  Spc     8:32PM    0:00.01 relayd: pfe (relayd)
_relayd  81032  0.0  0.4  2156  3716 ??  Spc     8:32PM    0:00.01 relayd: hce (relayd)
_relayd  80312  0.0  0.5  2748  5280 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  88919  0.0  0.5  2748  5268 ??  Ipc     8:32PM    0:00.04 relayd: relay (relayd)
_relayd  90186  0.0  0.5  2752  5272 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  32851  0.0  0.5  2736  5284 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  20270  0.0  0.5  2744  5252 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  98826  0.0  0.5  2752  5264 ??  Ipc     8:32PM    0:00.04 relayd: relay (relayd)
_relayd   6454  0.0  0.5  2744  5264 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  90102  0.0  0.5  2740  5276 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  60314  0.0  0.5  2748  5336 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  27246  0.0  0.5  2744  5284 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  87082  0.0  0.4  2100  4508 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  53308  0.0  0.4  2096  4496 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  11697  0.0  0.4  2092  4492 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  83636  0.0  0.4  2092  4500 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  74563  0.0  0.4  2096  4484 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  35910  0.0  0.4  2100  4508 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  24911  0.0  0.4  2096  4504 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  78649  0.0  0.4  2096  4496 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  89586  0.0  0.4  2096  4500 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  11989  0.0  0.4  2100  4500 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
[...]

In this case, both relayd and Varnish are running correctly.

Automating Certificate Renewal

As a final step, remember to create a script to renew the certificates, copy them to /etc/ssl and /etc/ssl/private, and restart relayd.

Conclusion

Congratulations, you now have your own free CDN, with your data, fully under your control. Portable, extendable, controllable, and outside of the big providers' grip.

If your goal is geo-replication, you should use one of the available methods. Some DNS providers allow selection based on the caller's location (like Bunny.net), or, as I prefer, install your own DNS and use tools to manage operations and resolutions, as briefly described in the previous article.

With multiple separate reverse proxies, separate DNS servers (on different providers and possibly different countries or continents) capable of checking if the reverse proxies are operational, you can achieve an extremely low likelihood of encountering a Single Point of Failure, as all components, once the cache is filled, will be nearly autonomous even in the event of a (temporary) backend outage - i.e., the original node.

Building a Self-Hosted CDN for BSD Cafe Media

Stefano Marinelli — Mon, 26 Aug 2024 08:41:00 +0200

Introduction

For just over a year, BSD Cafe's media was hosted on a FreeBSD physical server jail with an outgoing bandwidth of 250 Mbit/sec. To mitigate bandwidth congestion, I initially integrated Cloudflare with a tunnel to serve media (and only media) through Cloudflare. The goal was to georeplicate the media and reduce the load on my server. To do this, the media server had to be on a separate domain managed by Cloudflare since the DNS for the primary bsd.cafe domain was managed by Bunny.net.

The first step (mainly because I discovered that Bunny's DNS does not support IPv6) was to bring the DNS back in-house using two FreeBSD jails (running on different VPS providers), both powered by PowerDNS. PowerDNS supports LUA records, which would come in handy later.

In line with the principles of self-hosting and data ownership, I decided to remove Cloudflare. I created a dedicated subdomain (media.bsd.cafe) and configured the reverse proxy in front of the jail running Minio to respond to that domain. I also reconfigured Mastodon for the new address, and after some fine-tuning, everything worked seamlessly. However, this led to some bandwidth congestion when media was posted, resulting in slower download speeds for users, especially during peak times. This is because, once content is published and federated servers are notified, they all attempt to download the newly published content - media included - almost simultaneously.

Not wanting to abandon my media server (a dedicated jail with spinning disks, offering 4 TB of storage), I opted for a different approach that I’ll describe here, as it might be useful for similar setups.

While this setup was implemented on FreeBSD, the configuration and tools - Nginx, Varnish, WireGuard and PowerDNS - are compatible with many operating systems, including Linux, with only minor adjustments required.

The Approach: Building a Self-Hosted CDN

The idea is to create reverse proxies with local caching. These proxies would cache the content on the first request and serve it directly afterward. The proxies would be distributed across different regions, and the DNS would route requests to the nearest proxy based on the caller’s location. All this is achieved without relying on external CDNs, using self-managed tools instead.

To establish a direct connection between Minio and the reverse proxies, I configured WireGuard inside the jail. The reverse proxies connect via WireGuard, allowing them to access Minio securely as if they were on the same LAN.

No further changes were needed on the media jail itself.

Setting Up the Reverse Proxies

I began configuring the reverse proxies (also running FreeBSD jails, OpenBSD (setup described in another post) and NetBSD (also described in another post), hosted on different providers). The choice of Varnish is based on several factors, with the main ones being the ability to keep the cache in RAM (which means it can run on read-only systems) and the ability to flush the cache remotely. For example, with each change to my blog, I can choose whether to perform an immediate flush (such as for a new article or an error) or wait for the cache's "natural" expiration (such as for a typo or minor, non-critical changes).

First, I connected them via WireGuard to the Minio jail (I won’t detail the steps here; I’ve covered similar setups in other posts). Then, I installed Nginx and Varnish. A more granular setup would have Varnish on a separate jail, but this way, I can move the reverse proxy jails to different hosts with minimal hassle. Currently, these reverse proxies also serve this blog.

Next, I installed and configured Varnish inside the jail:

pkg install varnish7

I created the directory /usr/local/etc/varnish and wrote a custom VCL file to manage this setup, named default.vcl:

vcl 4.1;
import std;

# Backend - it-notes.dragas.net
backend it_notes {
    .host = "itnotesip";
    .port = "itnotesport";
}

# Backend - media.bsd.cafe
backend media_bsd {
    .host = "minioWGip";
    .port = "minioport";
}

# ACL - IPs allowed to purge - it-notes.dragas.net
acl purge_it_notes {
    "a.b.c.d";
}

# ACL - IPs allowed to purge - media.bsd.cafe
acl purge_media_bsd {
    "e.f.g.h";
}

sub vcl_recv {

    # it-notes.dragas.net
    if (req.http.Host == "it-notes.dragas.net") {
        set req.backend_hint = it_notes;
        set req.http.Host = "it-notes.dragas.net";

        # PURGE - it-notes.dragas.net
        if (req.method == "PURGE") {

            std.log("Purge request received for " + req.url);

            if (!std.ip(req.http.X-Real-IP, "0.0.0.0") ~ purge_it_notes) {
                return (synth(405, "Not allowed."));
            }

        if (req.url == "/" || req.url == "/*") {
                ban("req.http.host == " + req.http.host);
                return(synth(200, "Entire cache has been cleared."));
        }
            return (purge);
        }

    # media.bsd.cafe
    } elsif (req.http.Host == "media.bsd.cafe") {
        set req.backend_hint = media_bsd;
        set req.http.Host = "media.bsd.cafe";

        # PURGE - media.bsd.cafe
        if (req.method == "PURGE") {
            if (!std.ip(req.http.X-Real-IP, "0.0.0.0") ~ purge_media_bsd) {
                return (synth(405, "Not allowed."));
            }
            if (req.url == "/" || req.url == "/*") {
                ban("req.http.host == " + req.http.host);
                return(synth(200, "Entire cache has been cleared."));
            }
            return (purge);
        }

    } else {
        # Other domains - 404
        return (synth(404, "Domain not found"));
    }

    if (req.method != "GET" && req.method != "HEAD") {
        return (pipe);
    }

    return (hash);
}

sub vcl_backend_response {
    # TTL - it-notes.dragas.net
    if (bereq.http.host == "it-notes.dragas.net") {
        if (bereq.url ~ "\.(gif|jpg|jpeg|png|ico|css|js)$") {
            set beresp.ttl = 1w;
            set beresp.grace = 1d;
            set beresp.keep = 7d;
            unset beresp.http.Set-Cookie;
            unset beresp.http.Cache-Control;
            set beresp.http.Cache-Control = "public, max-age=604800";
        } else {
            set beresp.ttl = 15m;
            set beresp.grace = 48h;
            set beresp.keep = 7d;
        }

    # TTL - media.bsd.cafe
    } elsif (bereq.http.host == "media.bsd.cafe") {
        if (bereq.url ~ "\.(mp4|mp3|wav|flac|ogg)$") {
            set beresp.ttl = 1d;
            set beresp.grace = 6h;
            set beresp.keep = 3d;
            unset beresp.http.Set-Cookie;
            unset beresp.http.Cache-Control;
            set beresp.http.Cache-Control = "public, max-age=86400";
        } else {
            set beresp.ttl = 30m;
            set beresp.grace = 12h;
            set beresp.keep = 3d;
        }
    }

    # Remove some headers
    unset beresp.http.Server;
    unset beresp.http.X-Powered-By;
    unset beresp.http.Via;

    return (deliver);
}

sub vcl_deliver {
    # ADD header X-Cache
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
    } else {
        set resp.http.X-Cache = "MISS";
    }

  std.log("Delivering content for " + req.url + " - Cache: " + resp.http.X-Cache);


    # Remove Varnish headers
    unset resp.http.Via;
    unset resp.http.X-Varnish;

    return (deliver);
}

sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }
    return (lookup);
}

sub vcl_hit {
    return (deliver);
}

sub vcl_miss {
    return (fetch);
}

sub vcl_purge {
    std.log("Purge executed for " + req.url);
    return (synth(200, "Purge successful"));
}

sub vcl_synth {
    set resp.http.Content-Type = "text/html; charset=utf-8";
    set resp.http.Retry-After = "5";
    synthetic( {"<!DOCTYPE html>
        <html>
            <head>
                <title>"} + resp.status + " " + resp.reason + {"</title>
            </head>
            <body>
                <h1>Status "} + resp.status + " " + resp.reason + {"</h1>
                <p>"} + resp.reason + {"</p>
                <h3>Guru Meditation:</h3>
                <p>XID: "} + req.xid + {"</p>
                <hr>
                <p>Varnish cache server</p>
            </body>
        </html>
    "} );
    return (deliver);
}

This setup allows Varnish to handle both domains with distinct configurations but within the same cache.

To enable Varnish, I updated the /etc/rc.conf file with the following lines, setting a maximum cache size of 2GB:

varnishd_enable="YES"
varnishd_listen="127.0.0.1:8080"
varnishd_config="/usr/local/etc/varnish/default.vcl"
varnishd_storage="default,2000M"

You can now start Varnish:

service varnishd start

The next step is to create two virtual hosts on Nginx (one for it-notes.dragas.net and one for media.bsd.cafe) that will listen on both IPv4 and IPv6 for HTTP and HTTPS. HTTP connections will be redirected to HTTPS, and incoming HTTPS traffic will be passed to Varnish, which will either return cached data or fetch it from the original server (Minio via WireGuard, for media.bsd.cafe). Let's see the media.bsd.cafe part:

server {
   server_name  media.bsd.cafe;

   [...]

   location / {
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_connect_timeout 300;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    chunked_transfer_encoding off;

    expires 12h;
    add_header Cache-Control public;

    add_header X-Cache-Status $upstream_cache_status;
    add_header X-Content-Type-Options nosniff;

    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
    add_header Referrer-Policy "no-referrer-when-downgrade";
    add_header Permissions-Policy "geolocation=(), microphone=(), camera=()";

    proxy_pass http://127.0.0.1:8080;

    [...]

}

[...]

}

server {
    if ($host = media.bsd.cafe) {
        return 301 https://$host$request_uri;
    }
   listen       80;
   listen  [::]:80;
   server_name  media.bsd.cafe;
    return 404;
}

This configuration isn’t complete, but it provides a good idea of how to set up Nginx - each setup will vary. TTL and caching sizes will also differ based on the characteristics of each reverse proxy. For example, one of the proxies has an 8GB cache since I have ample resources there.

Generating certificates is another important aspect. In this case, as the reverse proxies are distributed, they all need to respond to the same addresses. One approach is to generate the certificate on one proxy and distribute it to the others. In my case, I opted to use lego, which, through PowerDNS’s API, adds a DNS record for validation. This way, each reverse proxy can independently generate and renew its certificates when needed.

Configuring DNS for Optimal Routing

Once everything is set up, it’s important to ensure that DNS responds correctly. In my case, I implemented a strategy like this:

Track reverse proxies that respond on port 443 (further refinements are possible and will be done later).
Return the closest reverse proxy based on the client’s IP address.

Unfortunately, PowerDNS on FreeBSD does not include GeoIP support by default, but I have my poudriere ready to compile and install the necessary packages. Alternatively, you could compile it within the jail using the port system.

After that, I installed the geoipupdate package (which requires a free license from MaxMind), updated the IP list, and configured PowerDNS to use the GeoIP database. I added the GeoIP backend alongside the existing SQLite3 backend and specified the database to use:

launch=gsqlite3,geoip
geoip-database-files=mmdb:/usr/local/share/GeoIP/GeoLite2-City.mmdb

Finally, I created a LUA record to return the correct address:

testclosest.bsd.cafe   60      IN      LUA     A "ifportup(443, {'proxy1ip', 'proxy2ip','proxy3ip'}" ", {selector='pickclosest'})"
testclosest.bsd.cafe   60      IN      LUA     AAAA "ifportup(443, {'proxy1ip6', 'proxy2ip6','proxy3ip6'}" ", {selector='pickclosest'})"

And voilà! We now have a small, self-hosted CDN, keeping full control and ownership of our data. Adding a new reverse proxy is straightforward — simply clone an existing proxy, update the WireGuard configuration (adding a peer on the Minio jail and changing the keys on the new proxy), and add it to the DNS.

Happy caching!

FreeBSD Tips and Tricks: Limiting Process Priority in a FreeBSD Jail

Stefano Marinelli — Thu, 11 Jul 2024 08:41:00 +0200

FreeBSD allows for quite comprehensive resource limitation for one or more jails. Beyond the official documentation, there is a good description in MWL's book. There's also an interesting article from Klara Systems that describes some functionalities.

Sometimes, however, we only need the processes running within a specific jail to have a specific priority - higher or lower than others.

There are many methods to achieve this, but the simplest one, in my opinion, is to leverage the properties of the nice(1) command. The main property, in fact, is to transmit the set priority to child processes, so all processes launched by the command that received a different level of "niceness" will inherit its priority.

For example, to give the minimum priority to the services launched within a jail, just modify the .conf file of the jail (in the case of a standard BastilleBSD installation, the file will be /usr/local/bastille/jails/jailname/jail.conf) and change the command

exec.start = '/bin/sh /etc/rc';

to:

exec.start = '/usr/bin/nice -n 20 /bin/sh /etc/rc';

In this way, when rc starts at the jail's boot, it will have a niceness of 20 and will transmit it to all the processes that rc itself launches (thus all the services of the jail).

Note: This will only apply to child processes of rc, not to:

Commands manually launched from the jail console
Services launched by running service servicename start (or restart) from the jail shell. This is because, in this case, the process will not be a child of rc but will derive directly from the console in use.

Using the nice command to set the priority of processes within a FreeBSD jail is a simple and effective method. However, it is important to be aware of the limitations of this approach and ensure that manually executed commands are managed accordingly.

FreeBSD Tips and Tricks: Creating Snapshots with UFS

Stefano Marinelli — Tue, 04 Jun 2024 06:40:45 +0000

One of the main features that every modern file system should support, in my opinion, is the ability to take snapshots. Snapshots can be useful for many reasons, such as making changes or updates with the ability to revert at any time.

On Linux, file systems operating in COW (like ZFS, btrfs, etc.) can handle snapshots efficiently, but traditional file systems like ext4 lack this support. The only way is to add another layer (like LVM) and take a snapshot of the volume, mounting it elsewhere in read-only mode.

FreeBSD provides tools for creating snapshots using both ZFS and UFS. I'll focus on UFS. In the base system, there's a very useful and well-documented command: mksnap_ffs.

Long story short: this video shows all the steps described below:

mksnap_ffs will create a snapshot of the current state of the specified file system and store it in the file you specify. For example, running mksnap_ffs /snap will take a snapshot of the "/" file system and store it in the file "/snap". By creating a memory disk and mapping the /snap file to an md, you can then mount it in read-only mode and retrieve the information. Finally, you can unmount, remove the md (mdconfig -du /dev/mdX), and delete the snapshot with a simple rm (rm /snap). Note that each file system can have up to 20 such snapshots, so it shouldn't be considered on par with ZFS. However, it can be crucial for making a consistent backup of a live system with tools like Borg, Restic, Kopia, etc., or a "simple" rsync.

Steps to Create and Use Snapshots with UFS

Create the Snapshot sh mksnap_ffs /snap This command takes a snapshot of the "/" file system and stores it in the file "/snap".
Create a Memory Disk sh mdconfig /snap This command maps the snapshot file to a memory disk.
Mount the Snapshot sh mount -o ro /dev/md0 /mnt This command mounts the memory disk in read-only mode to /mnt.
Access the Snapshot

You can now access the snapshot via /mnt.
Cleanup sh umount /mnt mdconfig -du 0 rm /snap These commands unmount the snapshot, remove the memory disk, and delete the snapshot file.

By leveraging this built-in feature of FreeBSD, you can ensure your systems are more resilient and easier to manage, particularly for backups and system updates.

FreeBSD Tips and Tricks: Native Read-Only Root File System

Stefano Marinelli — Fri, 31 May 2024 16:53:45 +0000

Many years ago, I needed to create some small servers using PCEngines Alix and had to choose the most suitable operating system. One of the challenges was dealing with dirty shutdowns and subsequent reboot issues, sometimes requiring manual intervention (not always feasible). After testing various Linux distributions and BSD systems, I found the ideal solution: FreeBSD.

One highly useful but often overlooked feature of FreeBSD is its ability to detect that the system is in read-only mode and automatically mount important directories (like /tmp, /var/log, etc.) in tmpfs, allowing the system to function perfectly.

Since then, whenever I need a system that can recover from any type of blackout without manual intervention (such as my Raspberry Pies), I always use this feature. If needed, external storage devices will be mounted automatically (or manually) after boot, and in case of issues, I can always access the device via SSH to perform necessary checks on the external filesystem.

This feature is automatic: simply have the root file system on UFS (not ZFS) and set it to "ro" (from "rw") in /etc/fstab. After a reboot, the system will operate in read-only mode, helping preserve the integrity of the embedded memory.

For updates, just:

Temporarily remount the root file system as rw (mount -uo rw /)
Modify /etc/fstab, changing "ro" back to "rw"
Reboot the device and update it normally

Make your own E-Mail server - Part 2 - Adding Webmail and More with Nextcloud

Stefano Marinelli — Thu, 21 Mar 2024 07:15:00 +0000

In the first part of this series, I detailed how to install and configure a complete mail server within a FreeBSD jail. While accessible via IMAP and SMTP, I intentionally did not cover installing a webmail interface.

Installing a "simple" webmail is not complex, but nowadays, we expect more integrated solutions, including contact books (synchronizable with computers and mobile devices) and calendars. For this reason, in this second part, I will be utilizing Nextcloud.

Nextcloud represents a comprehensive groupware solution today, extendable through "apps" that can be installed directly from the administration menu. This setup offers numerous advantages, including authenticating Nextcloud users through the IMAP server (thus, a mail server user will automatically have a Nextcloud account, and password changes on the mail will reflect on Nextcloud too), the ability to use integrated Contacts (also in webmail) and Calendars, and the option to attach or save attachments directly in Nextcloud.

We will use the reverse proxy configured in part 1, which will "pass" connections to Nextcloud. First, a domain name must be defined, which will be used to connect to the server. In this example, I will use "nextcloud.example.com."

To do this, enter the proxy's jail:

bastille console nginx-proxy

Modify the file /usr/local/etc/nginx/nginx.conf, adjust the worker_processes (so nginx will start a number of processes equal to the number of cores on the machine it's running on), and create the virtual host for Nextcloud:

worker_processes auto;

In the http block, change the maximum file size limit and create the virtual host on http - we will generate the https block and related certificates later:

client_max_body_size 0;

server {
    listen 80;
    server_name nextcloud.example.com;

    location / {
        proxy_buffering off;
        proxy_http_version 1.1;
        fastcgi_hide_header X-Powered-By;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto https;
        proxy_pass https://192.168.123.5;
        proxy_connect_timeout   10m;
        proxy_send_timeout      10m;
        proxy_read_timeout      10m;
    }

    location /.well-known/carddav {
        return 301 $scheme://$host/remote.php/dav;
    }

    location /.well-known/caldav {
        return 301 $scheme://$host/remote.php/dav;
    }
}

The proxy_pass is in https because, later on, Nextcloud will respond in https. Although not necessary, this approach is simpler given the method used to install Nextcloud. Moreover, it allows for the reverse proxy to be on a different server than Nextcloud, without the need for a VPN.

Now, enable and start nginx:

service nginx enable
service nginx start

After setting the listener to port 80, it's now appropriate to generate the certificate.

A simple command will request the certificate and modify the nginx configuration to accept connections over https:

certbot --nginx -d nextcloud.example.com

Upon completion of this command, the virtual host for https will be correctly generated, and an automatic redirect will be set up for all connections arriving on http to be "moved" to https.

Exit the jail, and it's time to create the jail for Nextcloud.

bastille create -B nc 14.2-RELEASE 192.168.123.5/24 bridge0

Enter the jail and add the gateway in rc.conf:

defaultrouter="192.168.123.1"

After restarting the jail, it will now be possible to install Nextcloud.

FreeBSD has a fairly updated package, but when it comes to web applications, I prefer to have more control. Usually, I install everything manually, but for Nextcloud, I found an interesting script made by Gibran Khan that takes care of everything, i.e., installing Apache, PHP, ClamAV, all dependencies. It will also configure Nextcloud and make it (almost) ready to use.

Since Gibran's script was structured for version 27, used PHP 8.2, and had other small things that were not perfect for my use, I decided to modify it slightly, mainly to support Nextcloud 28 and PHP 8.3. Therefore, the description that follows will be related to my fork available here: https://github.com/draga79/NextCloudOnFreeBSD

So, enter the "nc" jail, download the script, and start with the configuration:

bastille console nc
fetch https://github.com/draga79/NextCloudOnFreeBSD/archive/refs/heads/release.zip
unzip release.zip
cd NextCloudOnFreeBSD-release
./pre-install.sh

A configuration file will be generated that we need to modify. The file is install.conf.

You will certainly need to change the HOST_NAME (inserting the one you have chosen, in this case, "nextcloud.example.com") and, unless you are in Italy, I would also suggest changing the COUNTRY_CODE and TIME_ZONE.

Once completed, you can proceed with the actual installation:

./install.sh

At the end of the installation, the login data will appear (which will also be saved in a file named nextcloud.example.com_reference.txt). The installer will also set up a cron job executed by the user "www", which will handle, every 5 minutes, the background operations required for Nextcloud to function properly. In some cases, I've noticed that it's necessary to modify the cronjob, save it, and then it will start working correctly. If that's the case, simply open it with crontab -u www -e, save, and exit.

Now, modify the Nextcloud configuration file, /usr/local/www/nextcloud/config/config.php, and:

Replace overwrite.cli.url with your domain (https://nextcloud.example.com).
Add the proxy to the trusted proxies (if not present, insert the entire block):

 'trusted_proxies' =>
  array (
    0 => '192.168.123.2',
  ),

Modify the file /usr/local/etc/apache24/httpd.conf and change the Listen directive to:

Listen *:443

Restart Apache:

service apache24 restart

To simplify things, it's advisable to make a modification: in /etc/hosts, add a line to ensure the mail server can be reached via LAN through its domain name. Therefore, add the line:

192.168.123.3 mail.example.com

In this manner, when Nextcloud attempts to connect to the mail server, it will do so directly over the LAN. This direct connection is essential for the IMAP authentication plugin, as it verifies that the SSL certificate of the mail server matches the hostname being called.

Now log in to Nextcloud with the admin credentials that the installer provided you earlier. Nextcloud is now installed and initialized. We need to perform two operations - install the Snappymail app (if you want to use it, but generally, I recommend it over the integrated mail client in Nextcloud) and enable IMAP authentication to avoid having duplicate users (i.e., mail user and Nextcloud user).

Click on the menu at the top right (the big A) and go to Apps. Under "Integration," you will now be able to install "Snappymail" and "External User Authentication," which, however, is currently listed as "Untested" because it has been tested for a previous version of Nextcloud (but it works on version 28 too). You will have to click it twice, because the first click will only enable untested apps.

Once the apps are installed, go to "Administration Settings" and "Additional Settings," then to the Snappymail administration panel. Now add your domain, so click Add Domain. The domain name will be that of your emails (example.com). Therefore, in the IMAP server field, enter the name of the mail server (mail.example.com), and set the security to STARTTLS. Do the same for SMTP and SIEVE.

Perform the test (using the username and password of an existing and valid mailbox) and, once everything is validated, save.

You will now be able to log in into your webmail - in the top bar of Nextcloud there will be two icons, one for the integrated webmail client and the other for Snappymail. The integrated webmail client has some features that are not present in Snappymail (such as delayed sending of messages), so it may be useful to keep both active. Currently, Snappymail will require a username and password every time you connect (even if you are already logged into Nextcloud), but it's possible, in user settings, to enable automatic login once authenticated in Nextcloud.

Now is the time to configure IMAP authentication. Return to the Nextcloud jail and modify the Nextcloud configuration file, i.e., /usr/local/www/nextcloud/config/config.php. Add a block like this:

'user_backends' => array(
    array(
        'class' => '\OCA\UserExternal\IMAP',
        'arguments' => array(
            'mail.example.com', 993, 'ssl', 'example.com', false, false
        ),
    ),
),

'example.com' in the fourth field is used to limit the authentication scope to that domain. More information can be found on the official plugin page.

After saving, you will be able to log out as admin and log in with your mail account. Unless there are errors, you will be catapulted into your new user, created and authorized through IMAP authentication.

Congratulations, you are now able to use Nextcloud by logging in with your email credentials and also use it as a basis for webmail, leveraging the contacts module for the email address book as well.

Nextcloud can offer intriguing prospects for everyday use. In addition to its seamless integration with mobile devices, tools like 'Nextcloud Talk' and 'Note' will enable users to avoid the use of external solutions and maintain control over their data.

It will also be possible to synchronize contacts and calendars with mail clients (like Thunderbird) or the address books of various stationary and mobile devices. The procedure varies depending on the device used, but it's just a matter of synchronizing via CalDAV and CardDAV. Alternatively, you can set up Z-Push and use the Microsoft Exchange ActiveSync (EAS) protocol, but this will be described in part 3 of the series.

Addendum: Adding Antivirus Support to Spam Filtering

The automatic installation script, alongside Nextcloud, installed other components including ClamAV to check uploaded files in real time. We can leverage this installation (since ClamAV consumes significant resources, it makes no sense to install two instances) and integrate this instance of ClamAV with rspamd, to also perform antivirus checks on emails.

Modify /usr/local/etc/clamd.conf

Uncomment:

TCPSocket 3310

Restart clamd to listen on port 3310 as well:

service clamav-clamd restart

Return to the mail jail:

bastille console mail

Create the file /usr/local/etc/rspamd/local.d/antivirus.conf with the following content:

clamav {
  # If set, force this action if any virus is found (default unset: no action is forced)
  action = "reject";
  message = '${SCANNER}: virus found: "${VIRUS}"';
  # Scan mime_parts separately - otherwise, the complete mail will be transferred to AV Scanner
  #attachments_only = true; # Before 1.8.1
  #scan_mime_parts = true; # After 1.8.1
  # Scanning Text is suitable for some av scanner databases (e.g., Sanesecurity)
  #scan_text_mime = false; # 1.8.1 +
  #scan_image_mime = false; # 1.8.1 +
  # If `max_size` is set, messages > n bytes in size are not scanned
  #max_size = 20000000;
  # symbol to add (add it to metric if you want non-zero weight)
  symbol = "CLAM_VIRUS";
  # type of scanner: "clamav", "fprot", "sophos", or "savapi"
  type = "clamav";
  # If set true, a log message is emitted for clean messages
  #log_clean = false;
  # Prefix used for caching in Redis: scanner-specific defaults are used. If Redis is enabled and
  # multiple scanners of the same type are present, it's important to set prefix to something unique.
  #prefix = "rs_cl_";
  # For "savapi" you must also specify the following variable
  #product_id = 12345;
  # servers to query (if port is unspecified, scanner-specific default is used)
  # can be specified multiple times to pool servers
  # can be set to a path to a unix socket
  servers = "192.168.123.5:3310";
  # if `patterns` is specified virus name will be matched against provided regexes and the related
  # symbol will be yielded if a match is found. If no match is found, the default symbol is yielded.
  patterns {
    # symbol_name = "pattern";
    JUST_EICAR = '^Eicar-Test-Signature$';
  }
  # In version 1.7.0+ patterns could be extended
  #patterns = {SANE_MAL = 'Sanesecurity\.Malware\.*', CLAM_UNOFFICIAL = 'UNOFFICIAL$'};
  # `whitelist` points to a map of signature names. Hits on these signatures are ignored.
  whitelist = "/usr/local/etc/rspamd/local.d/antivirus.wl";
}

Restart rspamd in the “mail” jail and, from this point on, viruses will be automatically rejected.

Make your own E-Mail server - Part 1 - FreeBSD, OpenSMTPD, Rspamd and Dovecot included

Stefano Marinelli — Fri, 08 Mar 2024 07:15:00 +0000

The main power of the Internet has always been one: decentralization.

Ever since I've been able to, I've always managed my email boxes independently and provided mail hosting services to my clients. Over the years, things have become increasingly complex: on one hand, there's been a disproportionate increase in spam, and on the other, big players (like Google and Microsoft) have tried to gain a lot of ground in managing these services. Private users can obtain many free services, and business users no longer need to worry about the underlying infrastructure. However, in my opinion, the price to pay is very high: the loss of ownership of one's data.

These operators, in fact, use full access to our emails to improve their services or analyze us with the aim of selling us advertisements.

As often happens in these cases, many users appreciate this type of approach and have switched to the services of big players, causing a progressive increase in the level of influence that these companies can have on something free and decentralized like email.

In my opinion, it still makes sense to manage one's own mail servers. Standards evolve, and it is therefore appropriate to follow innovations, adapt to best practices, and secure one's services and users.

Over the last ten years, I have also used and installed various Zimbra OSE servers. Aside from a few minor issues, the setup has proven stable and reliable. Recently, the company that develops this excellent tool has decided to no longer provide packages for the Open Source version. They can be generated with scripts provided by some (great!) users, but they cannot be a solid and stable base in the long term.

I have therefore decided to gradually return to a modular, adaptable, customizable, and updatable mail host setup without worries or headaches. Such a system can include any kind of service (including webmail, integration with caldav/carddav, etc.) and it will be possible to disable what is not strictly necessary. Safer and more secure.

This will be the first in a series of articles, and at the end of this reading, you will have a secure, modern, reliable, and modular mail server. The instructions are designed for FreeBSD and related jails but with very few modifications can be applied to any BSD as well as Linux or other similar operating systems.

Setup Planning

Over the years, I've used many different SMTP servers. Originally, the good old Sendmail. Subsequently, and for many years, I used Exim, also because it was the default system in Debian. I then migrated almost all setups towards Postfix, probably the most widespread smtp server on the net, and I never had any particular problems. In recent years, I have decided to use, when possible, the excellent OpenSMTPD. Being based on OpenBSD and easily installable on other OSes, it shares the primary concepts of security and reliability of OpenBSD as well as the syntax (both in command line and in configuration files) of all other OpenBSD tools. For this type of setup, I will use opensmtpd.

The choice to use FreeBSD (rather than OpenBSD) stems from two main factors:

The possibility of dividing into jails, physically separating the various services and minimizing dependencies.
The possibility of using ZFS, with all the advantages it brings: the ability to take snapshots, the ability to perform backups quickly and efficiently, and the ability to migrate entire jails without particular problems.

The other tools that will be used are:

Rspamd - one of the best integrated systems for checking incoming mail. Customizable, adaptable, and modular, it also provides interesting monitoring systems on the type of incoming and outgoing mail.
Redis - which will be used by rspamd to manage its "memory".
Dovecot and Sieve - for delivering mail to local mailboxes as well as allowing remote email consultation (via imap) and enabling the setting of rules thanks to Pigeonhole Sieve.

The users of the mail server will not be "physical" users of the server itself, but virtual users, totally unrelated to system users.

This article will be long and detailed. The process may seem complex at first glance, but in reality, it's simpler than one might think. In the classic Unix philosophy, each component performs a task and does it well. Configuring the individual components separately will ensure greater scalability, reliability, and resilience to updates as well as simpler debugging compared to all-in-one solutions (like many of those found in convenient, ready-to-use docker-compose.yaml files).

Installation and Configuration

I won't describe the FreeBSD installation and configuration procedure. In my case, I activated a VPS on Hetzner - in this situation, the smallest of the available ARM servers. Yes, it's possible to efficiently manage a mail server for several users on a machine costing less than 4 euros a month. ZFS can be useful, but it's not necessary for this type of setup.

The first step is therefore to choose a location to install everything. The main condition is that the VPS/server/host must have at least one static public IP and, in 2024, I now consider it necessary to have (at least) one IPv6 address. They must also be able to send/receive mail on port 25, which is not guaranteed and not all VPS providers allow. Hetzner, for example, blocks port 25 for new users.

Once you have identified and acquired the server on which to install everything, it's appropriate to check that the IPs (both IPv4 and IPv6) to be used are not on any of the many blocklists/blacklists. My experience indicates that at least 60% of the IPs assigned to me are on one or more of these lists, so the first step is to request delisting, to ensure proper operation when the server is ready.

To Encrypt or Not to Encrypt?

A subsequent assessment is whether to use encryption for the data. Connections to and from the mail server will always be encrypted, but it's also possible to encrypt the data at rest. In the case of FreeBSD, it's possible to enable encryption for the entire installation (the entire virtual disk will be encrypted, and ZFS (or UFS) will be created on top of it) or, in the case of using ZFS, it will be possible to encrypt only the root dataset of the jails, the one that will be used by BastilleBSD. In the first case, it will be necessary to enter the password at every boot of FreeBSD, therefore connect via console at every restart and enter the password. In the second case, FreeBSD will be able to boot but it will be necessary to connect and enter the keys before mounting the BastilleBSD ZFS dataset. None of these solutions will prevent unauthorized access to emails in case of machine compromise. The main advantage will be "only" not to save the emails in clear on disks which, in the case of a VM, will still be shared and managed by third parties. In the case of physical servers, however, things will undoubtedly be different.

Encrypting the entire disk must be done directly during the FreeBSD installation phase, very simply. As for the option to encrypt only the BastilleBSD dataset, you can proceed later, with a command similar to:

zfs create -o encryption=on -o keylocation=prompt -o keyformat=passphrase zroot/bastille

at every reboot, it will be necessary to enter the password and mount the datasets:

zfs load-key -r zroot/bastille
zfs mount -a

Configuring FreeBSD and BastilleBSD

After the installation of the OS, the first step is to configure IPv6 on the VPS. In the case of Hetzner, unfortunately, they only provide a /64, so it will be necessary to segment the assigned network. In this example, it will be divided into /72 subnetworks - to find valid subclasses, it will be possible to use a calculator.

The /etc/rc.conf file should have entries similar to:

ifconfig_vtnet0="DHCP"
ifconfig_vtnet0_ipv6="inet6 2a01:4f8:cafe:cafe::1 prefixlen 72"
ipv6_defaultrouter="fe80::1%vtnet0"

In short, keep the base address assigned by Hetzner, but change the prefix length to 72 - thus giving the possibility of having other networks available.

It is now necessary to enable forwarding for IPv4 and IPv6. Add these lines to the /etc/sysctl.conf file:

net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1

After reboot, test it:

ping6 google.com

If everything has been configured correctly, the ping will be executed and google.com will reply.

It will also be necessary to set up reverse DNS, otherwise many mail servers will reject our emails. This must be done both for the IPv4 address and the designated IPv6 address (within the class - an operation that can also be done after creating the jails, but before starting to send email messages). Also, create the correct A and AAAA records on the DNS for the "mail.example.com" domain.

The FreeBSD installer does not automatically update the operating system, so it is appropriate to do so now:

freebsd-update fetch install

It seems, moreover, that there is still an old bug still present and manifesting when, on a FreeBSD server installed in a VM based on KVM (thus also those of Hetzner), routing is performed (as in our case) between VNET jails and host.

Adding this configuration to /boot/loader.conf will solve the problem:

hw.vtnet.X.tso_disable="1"
hw.vtnet.tso_disable="1"
hw.vtnet.lro_disable="1"
hw.vtnet.X.lro_disable="1"
hw.vtnet.csum_disable="1"
hw.vtnet.X.csum_disable="1"

It's time to install and configure BastilleBSD. This handy tool will make managing jails much simpler and more straightforward, and its configuration is well described on the project page.

Bridge and Firewall

For some time now, I've been favoring the use of VNET jails for these types of setups. I believe that having a complete network stack within the jails gives more freedom in configuration, firewall management (applicable also within individual jails), and technical possibilities (such as creating VPNs within the jails themselves, ensuring greater portability).

I suggest creating a bridge on the FreeBSD server and placing all the jails within this bridge. It will be enough to modify the /etc/rc.conf file and change/add:

cloned_interfaces="lo1 bridge0"
ifconfig_bridge0="inet 192.168.123.1 netmask 255.255.255.0"
ifconfig_bridge0_ipv6="inet6 2a01:4f8:cafe:cafe:100::1 prefixlen 72"
ipv6_gateway_enable="YES"
ipv6_activate_all_interfaces="YES"
gateway_enable="YES"

Now let's configure the firewall. In this example, I will use the IPv4 and IPv6 addresses that I will later assign to the jails. Here's an example of a working pf.conf:

ext_if="vtnet0"

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo
set skip on bridge0

table <jails> persist
# IPv4 private address ranges
table <private> const { 10/8, 172.16/12, 192.168/16 }
nat on $ext_if from 192.168.123.0/24 to ! <private> -> ($ext_if:0)
nat on $ext_if from <jails> to any -> ($ext_if:0)
rdr-anchor "rdr/*"
#rdr via ipv4 to mail
rdr pass on $ext_if proto tcp from any to ($ext_if) port { 25, 465, 587, 143, 993 } -> 192.168.123.3
#rdr via ipv4 to nginx-proxy
rdr pass on $ext_if proto tcp from any to ($ext_if) port { 80, 443 } -> 192.168.123.2

block in all
#PASS ICMP
pass in quick proto icmp from any to any
# Pass ICMP on ipv6
pass quick proto ipv6-icmp
pass out quick keep state
antispoof for $ext_if inet
pass in inet proto tcp from any to any port ssh flags S/SA keep state
#Pass ipv6 to mail jail
pass in quick on $ext_if inet6 proto tcp from any to 2a01:4f8:cafe:cafe:100::25 port { smtp, 465, 587, 143, 993 }
#Pass ipv6 to nginx-proxy jail
pass in quick on $ext_if inet6 proto tcp from any to 2a01:4f8:cafe:cafe:100::80 port { 80, 443 }

At this point, it is advisable to reboot the machine, to be sure that everything comes back up in the right way.

Creating the "nginx-proxy" and "redis" Jails

Let's start with the first jail. It will be an nginx reverse proxy. At the moment it will not be used as such but will be useful as a machine for generating certificates and, in the future, to act as a reverse proxy for various internal web services.

bastille create -B nginx-proxy 14.3-RELEASE 192.168.123.2/24 bridge0

Once the jail is created, it will be necessary to modify some configurations. In the jail itself (bastille console nginx-proxy), modify the /etc/rc.conf file and add the IPv4 gateway and IPv6 configurations, i.e., giving the address specified earlier in the pf.conf of the FreeBSD server and as the gateway the IP address of the related bridge:

defaultrouter="192.168.123.1"
ifconfig_vnet0_ipv6="inet6 2a01:4f8:cafe:cafe:100::80 prefixlen 72"
ipv6_defaultrouter="2a01:4f8:cafe:cafe:100::1"

Restart nginx-proxy with bastille restart nginx-proxy, and you will be able to re-enter the jail. At that point, ping google.com and ping6 google.com should work. The jail will then be able to operate, responding on 80 and 443 both in IPv4 (thanks to the NAT configured previously on the FreeBSD server) and in IPv6, being directly connected in routing.

Now install the necessary software:

pkg install -y nginx py311-certbot py311-certbot-nginx

To automatically renew the certificates, add this line to /etc/periodic.conf:

weekly_certbot_enable="YES"

It's now possible to generate the certificate, which we will also use for all other exposed services:

certbot certonly -d mail.example.com

Once the certificates are generated, we will need to make a small change as opensmtpd (which we will install later in another jail) is (rightly) very restrictive on permissions:

chmod -R 400 /usr/local/etc/letsencrypt/live/mail.example.com/

Now let's create a jail for redis. I usually put it in a dedicated jail also in terms of clustering/multi-server setup:

bastille create -B redis 14.3-RELEASE 192.168.123.4/24 bridge0

Also for the redis jail, it's appropriate to set up the gateway, so in the /etc/rc.conf of the jail:

defaultrouter="192.168.123.1"

I don't configure IPv6 as it's not necessary (this jail will only be reachable from the LAN) but it's possible to do so. Once the jail is restarted (bastille restart redis), it will be able to reach the outside:

pkg install -y redis

Not wanting to disable the protected mode of redis and not wanting to open it without authentication, it will be appropriate to modify the /usr/local/etc/redis.conf configuration file and add the password and some options to optimize its use with rspamd:

requirepass cafecafe
maxmemory 512mb
maxmemory-policy volatile-lru

Of course, set a unique password. Also, change the line concerning the bind:

bind *

Now execute:

service redis enable
service redis start

Redis should be active and ready to receive connections.

The "mail" Jail

It's time to create the jail with the main services:

bastille create -B mail 14.3-RELEASE 192.168.123.3/24 bridge0

and, also in this case, it will be necessary to enter the jail and modify some configurations in /etc/rc.conf:

defaultrouter="192.168.123.1"
ifconfig_vnet0_ipv6="inet6 2a01:4f8:cafe:cafe:100::25 prefixlen 72"
ipv6_defaultrouter="2a01:4f8:cafe:cafe:100::1"

Once the jail is restarted, it will be possible to install the necessary packages:

pkg install -y rspamd opensmtpd opensmtpd-filter-senderscore opensmtpd-filter-rspamd dovecot dovecot-pigeonhole

Please note: the opensmtpd-extras- packages are currently broken so we'll compile the table file support:

pkg install autoconf automake git-tiny
git clone https://github.com/OpenSMTPD/table-passwd.git
cd table-passwd
sh bootstrap
./configure
make
make install
cp table-passwd /usr/local/libexec/opensmtpd/
cp table-passwd.5 /usr/local/share/man/man5/

It is now appropriate to give this jail an FQDN, as it will be the name with which it presents itself when connecting to the outside. Just modify /etc/rc.conf:

hostname="mail.example.com"

The hostname should be equivalent to the reverse DNS set earlier, so the machine will present itself with a name that, doing a reverse lookup, will correspond to the IP of origin.

This jail does not contain the certificates that are present in the nginx-proxy jail. There are various methods to share them, such as using rsync, putting them on an NFS share, etc., but the simplest in this case will be to do a bind mount between the two jails, an operation that BastilleBSD can handle automatically.

Create the correct directory in the "mail" jail:

mkdir /usr/local/etc/letsencrypt

Exit the jail, shut it down, and modify the fstab file of the jail (e.g., /usr/local/bastille/jails/mail/fstab) by adding a line like this:

/usr/local/bastille/jails/nginx-proxy/root/usr/local/etc/letsencrypt /usr/local/bastille/jails/mail/root/usr/local/etc/letsencrypt nullfs ro 0 0

Start the jail, and at that point, the directory created earlier should display the directories and certificates from the nginx-proxy jail.

Due to spammers who, year after year, become increasingly aggressive, many mail servers require perfect configuration of the main sender authentication methodologies. Today, the main (and now necessary, under penalty of message non-delivery) are SPF, DKIM, and DMARC.

The first step is to generate a DKIM key for the domain(s) that will send mail from the server we are configuring. This operation can be performed in a few steps, by placing the keys in the /usr/local/etc/mail/dkim directory:

mkdir /usr/local/etc/mail/dkim
cd /usr/local/etc/mail/dkim
openssl genrsa -out example.com.key 2048
openssl rsa -in example.com.key -pubout -out public.key
chmod 0400 example.com.key
chown rspamd *

Since rspamd will take care of signing outgoing messages, it is appropriate that only the related user be able to read the file with the private key.

Once the pair of keys has been created, it will be necessary to configure DNS to provide the public key to mail servers that will request it. The record should look like this:

mail._domainkey   TXT   "v=DKIM1;k=rsa;p=your-public-key-goes-here"

Once the record has been entered and propagated, it will be possible to test its correctness through one of the many websites that offer this service, remembering that the selector we have configured is "mail" (the first part of the record). Obviously, this is a free text, which can be modified at will (but will then need to be specified in the rspamd configuration, later on).

Being in the DNS configuration phase, it will also be appropriate to create SPF and DMARC records:

SPF record:

example.com. IN TXT "v=spf1 a ip4:your.ip.address ip6:2a01:4f8:cafe:cafe:100::25 mx ~all"

DMARC:

_dmarc.example.com. IN TXT "v=DMARC1;p=none;pct=100;rua=mailto:postmaster@example.com"

These records, obviously modified based on the chosen domain, are configured conservatively but suitable for a test. Once the entire setup is in place, it will be possible to handle things in a more restrictive manner.

The mail server users will be "virtual" users (i.e., not system users of the server), so all their mail will be "handled" by a unique user, which needs to be created:

pw user add vmail -u 2000 -d /var/vmail -s /usr/sbin/nologin
mkdir /var/vmail
chown vmail /var/vmail

Let's now create some files and directories, useful for the subsequent configuration:

touch /usr/local/etc/mail/passwd
touch /usr/local/etc/mail/virtuals
mkdir /usr/local/etc/rspamd/local.d

Remove the current /usr/local/etc/mail/smtpd.conf and replace it with content like this:

table passwd passwd:/usr/local/etc/mail/passwd
table virtuals file:/usr/local/etc/mail/virtuals

pki mail.example.com cert "/usr/local/etc/letsencrypt/live/mail.example.com/fullchain.pem"
pki mail.example.com key "/usr/local/etc/letsencrypt/live/mail.example.com/privkey.pem"

filter check_dyndns phase connect match rdns regex { '.*\.dyn\..*', '.*\.dsl\..*' } \
    disconnect "550 no residential connections - Thou shalt not pass"

filter check_rdns phase connect match !rdns \
    disconnect "550 no rdns - Thou shalt not pass"

filter check_fcrdns phase connect match !fcrdns \
    disconnect "550 no FCrDNS - Thou shalt not pass"

filter senderscore \
    proc-exec "/usr/local/libexec/opensmtpd/opensmtpd-filter-senderscore -blockBelow 10 -junkBelow 70 -slowFactor 5000"

filter rspamd proc-exec "/usr/local/libexec/opensmtpd/opensmtpd-filter-rspamd"

listen on 0.0.0.0 tls pki mail.example.com \
    filter { check_dyndns, check_rdns, check_fcrdns, senderscore, rspamd } auth-optional <passwd>

listen on ::0 tls pki mail.example.com \
    filter { check_dyndns, check_rdns, check_fcrdns, senderscore, rspamd } auth-optional <passwd>

listen on 0.0.0.0 port submission tls-require pki mail.example.com  auth <passwd> filter rspamd mask-src

listen on ::0 port submission tls-require pki mail.example.com  auth <passwd> filter rspamd mask-src

listen on 0.0.0.0 port 465 smtps pki mail.example.com  auth <passwd> filter rspamd mask-src

listen on ::0 port 465 smtps pki mail.example.com  auth <passwd> filter rspamd mask-src

action "local_mail" lmtp "/var/run/dovecot/lmtp" rcpt-to virtual <virtuals>
action "outbound" relay helo mail.example.com

match from any for domain example.com action "local_mail"
match for local action "local_mail"

match from any auth for any action "outbound"
match for any action "outbound"

The beauty of OpenSMTPD lies in its simplicity. Essentially, that's all there is to it, but here's an explanation of what this file does:

table passwd and table virtuals define tables for user credentials and virtual domains/users, crucial for authentication and email forwarding/aliasing.
pki lines specify the SSL/TLS certificate and key for encrypted connections.
filter directives apply various checks and filters to incoming connections, including dynamic DNS and reverse DNS validations, sender reputation scoring, and Rspamd filtering.
listen lines configure the server to listen on all IPv4 and IPv6 addresses for incoming SMTP and submission (port 587) connections, applying TLS and authentication as specified.
action lines define actions for email delivery, using LMTP for local delivery based on virtual mappings and setting up relay actions for outbound emails.
match rules determine how emails are processed, either delivered locally or relayed externally based on source, destination, and authentication.

To test the newly inserted configuration, run smtpd -n.

The next step is to configure Rspamd. Rather than modifying the base configuration files, we'll proceed by creating "overrides", placing specific configurations in the directory created earlier (/usr/local/etc/rspamd/local.d):

For Redis configuration in /usr/local/etc/rspamd/local.d/redis.conf:

servers = "192.168.123.4";
password = "cafecafe";

For SPF settings in /usr/local/etc/rspamd/local.d/spf.conf:

spf_cache_size = 1k;
spf_cache_expire = 1d;
max_dns_nesting = 10;
max_dns_requests = 30;
min_cache_ttl = 5m;

Now, configure DKIM signing, which rspamd will handle for every sent email. The selector must match the one entered in the DNS record previously:

In /usr/local/etc/rspamd/local.d/dkim_signing.conf:

domain {
  example.com {
    path = "/usr/local/etc/mail/dkim/example.com.key";
    selector = "mail";
  }
}

To enable support for some "known" phisher lists, create /usr/local/etc/rspamd/local.d/phishing.conf:

# Configuration options can be added here`
openphish_enabled = true;
phishtank_enabled = true;

For SURBL (Spam URI Real-time Blocklists) configuration in /usr/local/etc/rspamd/local.d/surbl.conf:

# follow redirects when checking URLs in emails for spaminess
redirector_hosts_map = "/usr/local/etc/rspamd/redirectors.inc";

For URL reputation checks in /usr/local/etc/rspamd/local.d/url_reputation.conf:

# check URLs within messages for spaminess
enabled = true;

And for caching some URL tags in Redis, in /usr/local/etc/rspamd/local.d/url_tags.conf:

# cache some URL tags in redis
enabled = true;

Rspamd should be ready for initial testing.

Now it's time to configure Dovecot, which will handle the final phase of delivery and provide mail access, as well as use Sieve for filtering and rule application.

The first step is to start with a default configuration:

cd /usr/local/etc/dovecot/ && cp -R example-config/* .

Next, modify some files. Start with conf.d/15-mailboxes.conf - add auto = create to the main folders, like this:

namespace inbox {
  # These mailboxes are widely used and could perhaps be created automatically:
  separator = .

  mailbox Drafts {
    special_use = \Drafts
    auto = create
  }
  mailbox Junk {
    special_use = \Junk
    auto = create
  }
  mailbox Trash {
    special_use = \Trash
    auto = create
  }

  # For \Sent mailboxes there are two widely used names. We'll mark both of
  # them as \Sent. User typically deletes one of them if duplicates are created.
  mailbox Sent {
    special_use = \Sent
    auto = create
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
}

Modify /usr/local/etc/dovecot/dovecot.conf to include:

protocols = imap lmtp sieve

In conf.d/10-auth.conf, switch to system authentication with passwdfile:

#!include auth-system.conf.ext
!include auth-passwdfile.conf.ext

Then, set up the passdb and userdb configuration in conf.d/auth-passwdfile.conf.ext. Replace the entire file with these contents:

passdb {
  driver = passwd-file
  args = scheme=CRYPT /usr/local/etc/mail/passwd
}

userdb {
  default_fields = uid=vmail gid=vmail home=/var/vmail/%d/%n
  args = /usr/local/etc/mail/passwd
  driver = passwd-file
}

Edit conf.d/10-ssl.conf to update ssl_key and ssl_cert:

ssl_cert = </usr/local/etc/letsencrypt/live/example.com/fullchain.pem
ssl_key = </usr/local/etc/letsencrypt/live/example.com/privkey.pem

In conf.d/20-imap.conf, include:

(...)
mail_plugins = $mail_plugins imap_sieve zlib
mail_max_userip_connections = 100

And in conf.d/20-lmtp.conf:

protocol lmtp {
  mail_fsync = optimized
  mail_plugins = $mail_plugins sieve
}

Set the mail location in conf.d/10-mail.conf:

mail_location = maildir:/var/vmail/%d/%n

maildir_stat_dirs=yes

Now, let's configure Sieve in conf.d/90-plugin.conf:

plugin {
  #setting_name = value
  sieve_plugins = sieve_imapsieve sieve_extprograms
  sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment
  imapsieve_mailbox1_name = Junk
  imapsieve_mailbox1_causes = COPY APPEND
  imapsieve_mailbox1_before = file:/usr/local/lib/dovecot/sieve/report-spam.sieve

  imapsieve_mailbox2_name = *
  imapsieve_mailbox2_from = Junk
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_before = file:/usr/local/lib/dovecot/sieve/report-ham.sieve

  imapsieve_mailbox3_name = Inbox
  imapsieve_mailbox3_causes = APPEND
  imapsieve_mailbox3_before = file:/usr/local/lib/dovecot/sieve/report-ham.sieve
  sieve_pipe_bin_dir = /usr/local/lib/dovecot/sieve

  #Move the spam to Junk folder
  sieve_after = /usr/local/lib/dovecot/sieve/spam-to-folder.sieve
}

And in conf.d/20-managesieve.conf.ext:

service managesieve-login {
  inet_listener sieve {
    port = 4190
  }

  service_count = 1
  vsz_limit = 64M
}

service managesieve {
  process_limit = 1024
}

protocol sieve {
  mail_max_userip_connections = 10

  # Configuration for Sieve script management via ManageSieve protocol
}

plugin {
  sieve_pipe_bin_dir = /usr/local/lib/dovecot/sieve
  sieve = file:/var/vmail/%d/%n/sieve;active=/var/vmail/%d/%n/.dovecot.sieve
  sieve_user_log = file:/var/vmail/%d/%n/sieve/sieve_error.log
}

This setup outlines the Dovecot configuration necessary for handling mail delivery, access, and Sieve-based filtering. With these settings, Dovecot is prepared to manage both incoming and outgoing mails securely and efficiently, including support for managing Sieve scripts via the ManageSieve protocol.

Sieve scripts train Rspamd on spam and ham. Moving email into and out of the junk folder triggers an event to train Rspamd.

These files are located at /usr/local/lib/dovecot/sieve.

Create the report-ham.sieve file:

require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"];

if environment :matches "imap.mailbox" "*" {
  set "mailbox" "${1}";
}

if string "${mailbox}" "Trash" {
  stop;
}

if environment :matches "imap.user" "*" {
  set "username" "${1}";
}

pipe :copy "sa-learn-ham.sh" [ "${username}" ];

Create the report-spam.sieve file:

require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"];

if environment :matches "imap.user" "*" {
  set "username" "${1}";
}

pipe :copy "sa-learn-spam.sh" [ "${username}" ];

Create the spam-to-folder.sieve file:

require ["fileinto","mailbox"];

if header :contains "X-Spam" "Yes" {
 fileinto :create "Junk";
 stop;
}

Compile the files:

sievec report-ham.sieve
sievec report-spam.sieve
sievec spam-to-folder.sieve

Create the following two shell scripts in /usr/local/lib/dovecot/sieve.

Add the following to sa-learn-ham.sh:

#!/bin/sh
exec /usr/local/bin/rspamc -d "${1}" learn_ham

Add the following to sa-learn-spam.sh:

#!/bin/sh
exec /usr/local/bin/rspamc -d "${1}" learn_spam

Make the files executable:

chmod 755 sa-learn-ham.sh
chmod 755 sa-learn-spam.sh

Everything has been configured correctly. However, a fundamental part is missing: the users.

To begin, encrypt the password by executing:

smtpctl encrypt

Input the plain password and hit ENTER. Copy the displayed encrypted password for the next step. Now, insert the entry into /usr/local/etc/mail/passwd:

foo@example.com:your encrypted password goes here::::::

Remember to replace the placeholder with your encrypted password and ensure there are six colons at the end.

Finally, add the virtual user and associate it with the vmail system user in /usr/local/etc/mail/virtuals:

foo@example.com vmail

Note: Additional aliases can be included, such as:

postmaster@example.com foo@example.com

Let's enable and start the services:

service rspamd enable
service rspamd start
service smtpd enable
service smtpd start
service dovecot enable
service dovecot start

At this point, take a look at the logs (especially /var/log/maillog) to get an idea of what's happening. If everything is running correctly, you can now proceed with some tests. Any mail client (such as Thunderbird, etc.) should be able to connect via IMAP and send emails using SMTP, thanks to authentication. Remember that the username corresponds to the one entered in the /usr/local/etc/mail/passwd file and typically matches the full email address.

Once everything is in order, you can set the MX record of the domain "example.com" to "mail.example.com".

One of the key tests to perform is to send an email to a Gmail inbox. If the email is correctly delivered and not marked as Spam, there's a good chance that the setup is correct. There are also several online tools available that can provide useful insights into any configuration errors.

You have just installed and operationalized a complete mail server.

It is now possible to proceed with part 2, which involves the installation of Nextcloud and Snappymail, in order to have a comprehensive groupware system, while keeping one's data on their own servers.

Links:

This article was inspired by several other articles available online that provided me with a lot of useful information. Here is a non-exhaustive list:

Make your own VPN - FreeBSD, Wireguard, ipv6 and ad-blocking included

Stefano Marinelli — Sat, 23 Sep 2023 09:05:51 +0000

Note: This article assumes a setup based on FreeBSD. If you prefer a version based on OpenBSD, it is available here.

VPNs are a fundamental tool for securely connecting to your own servers and devices. Many people use commercial VPNs for various reasons, ranging from not trusting their provider (especially when connecting from a public hotspot) to wanting to "go out" on the Internet with a different IP address, perhaps from another country.

Whatever the reason, solutions are not lacking. I have always set up management VPNs to allow servers and/or clients to communicate with each other using secure channels. Lately, I have been activating IPv6 connectivity on all my devices (both desktop/servers and mobile devices) and I needed to quickly create a node that concentrated some networks and allowed them to go out on the network in IPv6. The tools I used and will describe are:

VPS - in this case, I used a basic Hetzner Cloud VPS (using this link, you will receive 20 euros of cloud credits), but any provider that provides IPv6 connectivity will do - if you want IPv6, of course.
FreeBSD - a versatile, stable, and secure operating system.
Wireguard - lightweight, secure, and at the same time, not very "chatty", so it is also gentle on mobile device batteries. When there is no traffic, it simply does not transmit/receive anything. Well supported by all major desktop and server operating systems as well as Android and iOS devices.
Unbound - can make DNS queries directly to root servers, not through forwarders. It also allows you to insert block-lists and have a result similar to that of Pi-Hole (i.e., ad-blocking).
SpamHaus lists - to immediately stop connections to and from users on blacklists.

The first step is to activate a VPS and install FreeBSD. On the Hetzner Cloud console, there might not be a pre-built FreeBSD image, but only a selection of Linux distributions. Don't worry, just choose any of them and create the VPS. Once done, the FreeBSD ISO image will be available among the "ISO Images". Just insert the virtual CD, restart the VPS, and the FreeBSD installation will appear in the console.

I won't go into detail, the operation is simple and straightforward. The only precaution (in the case of a Hetzner Cloud VPS) is to use "DHCP" for IPv4 but, for now, do not configure IPv6. It will be configured later.

Install all FreeBSD updates (using the freebsd-update fetch install command) and reboot.

It seems, moreover, that there is still an old bug still present and manifesting when, on a FreeBSD server installed in a VM based on KVM (thus also tho se of Hetzner), routing is performed (as in our case) between VNET jails and host.

Adding this configuration to /boot/loader.conf will solve the problem:

hw.vtnet.X.tso_disable="1"
hw.vtnet.tso_disable="1"
hw.vtnet.lro_disable="1"
hw.vtnet.X.lro_disable="1"
hw.vtnet.csum_disable="1"
hw.vtnet.X.csum_disable="1"

Wireguard, on FreeBSD, is now available as a kernel module and the userland can be installed using the pkg install wireguard-tools package manager. This means you can easily keep it updated alongside other software on the system.

The first step is to configure IPv6 on the VPS. In the case of Hetzner, unfortunately, they only provide a /64, so it will be necessary to segment the assigned network. In this example, it will be divided into /72 subnetworks - to find valid subclasses, it will be possible to use a calculator.

The /etc/rc.conf file should have entries similar to:

ifconfig_vtnet0="DHCP"
ifconfig_vtnet0_ipv6="inet6 2001:db8:cafe:cafe::1 prefixlen 72"
ipv6_defaultrouter="fe80::1%vtnet0"

In short, keep the base address assigned by Hetzner, but change the prefix length to 72 - thus giving the possibility of having other networks available.

It is now necessary to enable forwarding for IPv4 and IPv6. Add these lines to the /etc/sysctl.conf file:

net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1

After reboot, test it:

ping6 google.com

If everything has been configured correctly, the ping will be executed and google.com will reply.

To configure Wireguard, a few steps will be necessary. First of all, the private key will need to be created:

wg genkey | tee /dev/stderr | wg pubkey | grep --label PUBLIC -H .

You will get a private key and a public key. Take note of the public key - it will be needed to configure the clients.

Now create a new file called /usr/local/etc/wireguard/wg0.conf:

[Interface]
Address = 172.16.0.1/24,2001:db8:cafe:cafe:100::1/72
ListenPort = 51820
PrivateKey = YUkS6cNTyPbXmtVf/23ppVW3gX2hZIBzlHtXNFRp80w=

A new Wireguard interface called wg0 is being created. Start the Wireguard interface:

service wireguard enable
sysrc wireguard_interfaces="wg0" 
service wireguard start

If everything has been entered correctly, the interface should come up. Check its status:

wg

As for the firewall, FreeBSD comes with no pf configuration. In my setups, I tend to block what is not needed and be permissive with what may be useful. However, I like to keep out the "bad guys," so I use blacklists. pf allows elements to be inserted and removed from tables at runtime, so the firewall can be configured accordingly.

To download and apply the Spamhaus lists, I use a simple but effective script found on the Internet, but for OpenBSD.

For the Spamhaus lists, continue with the FreeBSD script creation.

Create the script in /usr/local/sbin/spamhaus.sh:

#!/bin/sh
#
#this is normally run once per day via cron.
#
echo updating Spamhaus DROP lists:
(
  { fetch -o - https://www.spamhaus.org/drop/drop.txt && \
    fetch -o - https://www.spamhaus.org/drop/dropv6.txt ; \
  } 2>/dev/null | sed "s/;/#/" > /var/db/drop.txt
)
pfctl -t spamhaus -T replace -f /var/db/drop.txt

Make it executable and run it. Pf isn't enabled, so you'll get an error - but this will create the /var/db/drop.txt file:

chmod a+rx /usr/local/sbin/spamhaus.sh
/usr/local/sbin/spamhaus.sh

There are many possibilities to configure pf on FreeBSD. A fairly simple example could be this:

ext_if="vtnet0"
wg0_if="wg0"
wg0_networks="172.16.0.0/24"

set skip on lo

nat on $ext_if from { $wg0_networks } to any -> ($ext_if)

# Spamhaus DROP list:
table <spamhaus> persist file "/var/db/drop.txt"

block drop log quick from <spamhaus>

# Pass ICMP on ipv6
pass quick proto ipv6-icmp
# Block from ipv6 to wg0 network
block in quick on $ext_if inet6 to { 2001:db8:cafe:cafe:100::/72 }
# Pass Wireguard traffic - in and out
pass quick on $wg0_if

# default deny
block in
block out

pass in on $ext_if proto tcp to port ssh
pass in on $ext_if proto udp to port 51820

pass out on $ext_if

This is a very simple configuration: it blocks everything that is present in the list downloaded from Spamhaus, allows NAT from the Wireguard network to the public interface, allows ICMP traffic in IPv6 (necessary for the network to function properly) while blocking incoming traffic to the Wireguard IPv6 LAN (remember that the IPs will be public and directly reachable, so we don't want to expose our devices by default). All traffic on the Wireguard interface will be allowed to pass. Then everything will be blocked and exceptions will be specified, i.e., allowing SSH and Wireguard connections (of course). Authorization will also be granted to allow traffic to exit from the public network interface.

Save this configuration to /etc/pf.conf.

Enable and start pf:

service pf enable 
service pf start

You will probably be kicked out of the system. Don't worry, just reconnect. pf is doing its job.

If everything went correctly, the firewall should have loaded the new rules.

To obtain caching of DNS queries and the related ad-block, it is now time to configure Unbound. Let's install it with:

pkg install unbound

A while ago, I found a script which I slightly adapted. I don't remember where I got it, so I'll paste it here without citing the original creator.

Create a script to update the unbound ad-block, in /usr/local/sbin/unbound-adhosts.sh:

#!/bin/sh
#
# Using blacklist from pi-hole project https://github.com/pi-hole/
# to enable AD blocking in unbound(8)
#
PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin"

# Available blocklists - comment line to disable blocklist
_disconad="https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt"
_discontrack="https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt"
_stevenblack="https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts"

# Global variables
_tmpfile="$(mktemp)" && echo '' > $_tmpfile
_unboundconf="/usr/local/etc/unbound/unbound-adhosts.conf"

# Remove comments from blocklist
simpleParse() {
  fetch -o - $1 | \
  sed -e 's/#.*$//' -e '/^[[:space:]]*$/d' >> $2
}

# Parse DisconTrack
[ -n "${_discontrack}" ] && simpleParse $_discontrack $_tmpfile

# Parse DisconAD
[ -n "${_disconad}" ] && simpleParse $_disconad $_tmpfile

# Parse StevenBlack
[ -n "${_stevenblack}" ] && \
  fetch -o - $_stevenblack | \
  sed -n '/Start/,$p' | \
  sed -e 's/#.*$//' -e '/^[[:space:]]*$/d' | \
  awk '/^0.0.0.0/ { print $2 }' >> $_tmpfile

# Create unbound(8) local zone file
sort -fu $_tmpfile | grep -v "^[[:space:]]*$" | \
awk '{
  print "local-zone: \"" $1 "\" redirect"
  print "local-data: \"" $1 " A 0.0.0.0\""
}' > $_unboundconf && rm -f $_tmpfile

service unbound reload 1>/dev/null

exit 0

After saving the script, make it executable and run it:

chmod a+rx /usr/local/sbin/unbound-adhosts.sh
/usr/local/sbin/unbound-adhosts.sh

Now, the Unbound configuration file in /usr/local/etc/unbound/unbound.conf can be modified as follows:

server:
        verbosity: 1
        log-queries: no
        num-threads: 4
        num-queries-per-thread: 1024
        interface: 127.0.0.1
        interface: 172.16.0.1
        interface: 2001:db8:cafe:cafe:100::1
        interface: ::1
        outgoing-range: 64
        chroot: ""

        access-control: 0.0.0.0/0 refuse
        access-control: 127.0.0.0/8 allow
        access-control: ::0/0 refuse
        access-control: ::1 allow
        access-control: 172.16.0.0/24 allow
        access-control: 2001:db8:cafe:cafe:100::/72 allow

        hide-identity: yes
        hide-version: yes
        auto-trust-anchor-file: "/usr/local/etc/unbound/root.key"
        val-log-level: 2
        aggressive-nsec: yes
        prefetch: yes
        username: "unbound"
        directory: "/usr/local/etc/unbound"
        logfile: "/var/log/unbound.log"
        use-syslog: no
        pidfile: "/var/run/unbound.pid"
        include: /usr/local/etc/unbound/unbound-adhosts.conf

remote-control:
        control-enable: yes
        control-interface: /var/run/unbound.sock

Now, enable and start unbound:

service unbound enable
service unbound start

If everything has been set up correctly, unbound will be able to respond to DNS requests made on 172.16.0.1 and 2001:db8:cafe:cafe:100::1.

Now it is possible to configure the Wireguard client. Create a new configuration by inserting "172.16.0.2/32, 2001:db8:cafe:cafe:100::2/128" (the ones that will later be entered in the peer configuration of the server) in the local IP addresses. Set the DNS server address to "172.16.0.1" and/or its corresponding IPv6 address (in the example, 2001:db8:cafe:cafe:100::1 - yours will be different). In the peer section, insert the server's data, including its public key, IP address:port (in the example, the port is 51820), and allowed addresses (setting "0.0.0.0/0, ::0/0" means "all connections will be sent via Wireguard" - all the traffic will pass through the VPN for both IPv4 and IPv6).Each implementation has its own procedure (Android, iOS, MikroTik, Linux, etc.) but essentially it is sufficient to create the right configuration both on the server and on the client.

Reopen the Wireguard configuration file /usr/local/etc/wireguard/wg0.conf and add:

[Interface]
Address = 172.16.0.1/24,2001:db8:cafe:cafe:100::1/72
ListenPort = 51820
PrivateKey = YUkS6cNTyPbXmtVf/23ppVW3gX2hZIBzlHtXNFRp80w=

[Peer]
PublicKey = *client's public key*
AllowedIPs = 172.16.0.2/32, 2001:db8:cafe:cafe:100::2/128

The client's public key will be shown by the client itself.

Reload the Wireguard configuration:

service wireguard restart

It is also possible to use the VPN only as an ad-blocker, by only routing DNS traffic through it. To achieve this result, configure the client so that the only allowed address is the one of the just-configured unbound (in this example, 172.16.0.1 and/or 2001:db8:cafe:cafe:100::1) - DNS resolution will occur via VPN, but browsing will continue to work through the main provider.

To automatically update the spamhaus and ad-block lists, we will use cron.First, create a script, for example, /usr/local/sbin/update-blocklists.sh:

#!/bin/sh

/usr/local/sbin/unbound-adhosts.sh
/usr/local/sbin/spamhaus.sh

Make it executable:

chmod +x /usr/local/sbin/update-blocklists.sh

Then, add it to the crontab to run daily:

echo "@daily /usr/local/sbin/update-blocklists.sh" >> /etc/crontab

This approach benefits from both update management and security perspectives.

Make your own VPN - OpenBSD, Wireguard, ipv6 and ad-blocking included

Stefano Marinelli — Mon, 03 Apr 2023 04:16:11 +0000

Note: This article assumes a setup based on OpenBSD. If you prefer a version based on FreeBSD, it is available here.

VPS - in this case, I used a basic Hetzner Cloud VPS (using this link, you will receive 20 euros of cloud credits), but any provider that provides IPv6 connectivity will do - if you want IPv6, of course.
OpenBSD - a clean, stable, and secure operating system.
Wireguard - lightweight, secure, and at the same time, not very "chatty", so it is also gentle on mobile device batteries. When there is no traffic, it simply does not transmit/receive anything. Well supported by all major desktop and server operating systems as well as Android and iOS devices.
Unbound - can make DNS queries directly to root servers, not through forwarders. It also allows you to insert block-lists and have a result similar to that of Pi-Hole (i.e., ad-blocking).
SpamHaus lists - to immediately stop connections to and from users on blacklists.

The first step is to activate a VPS and install OpenBSD. On the Hetzner cloud console, there won't be a pre-built OpenBSD image, but only a selection of Linux distributions. Don't worry, just choose any of them and create the VPS. Once done, the OpenBSD ISO image will be available among the "ISO Images". Just insert the virtual CD, restart the VPS, and the OpenBSD installation will appear in the console.

I won't go into detail, the operation is simple and straightforward. The only precaution (in the case of a Hetzner Cloud VPS) is to use "autoconf" for IPv4 but, for now, do not configure IPv6. It will be configured later.

Install all OpenBSD updates (using the syspatch command) and restart, the kernel will be relinked.

Wireguard, on OpenBSD, is fully integrated into the base system and does not require the installation of external packages. This is a big advantage because over time, support for everything related to Wireguard will be managed directly by the main OpenBSD development team.

The /etc/hostname.vio0 file should look something like this:

inet autoconf
inet6 2001:db8:cafe:cafe::1 72 
!route add -net ::/0 fe80::1%vio0

In short, keep the base address assigned by Hetzner, but change the netmask to /72 - thus giving the possibility of having other networks available.

sh /etc/netstart vio0

It will reconfigure the network interface and allow IPv6 to start working. To test it:

ping6 google.com

If everything has been configured correctly, the ping will be executed and google.com will reply.

It is now necessary to enable forwarding for IPv4 and IPv6. Enter these lines in the /etc/sysctl.conf file:

net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1

To apply those changes you can reboot or just type:

sysctl net.inet.ip.forwarding=1
sysctl net.inet6.ip6.forwarding=1

To configure Wireguard, a few steps will be necessary. First of all, the private key will need to be created:

openssl rand -base64 32

Something like this will come out:

YUkS6cNTyPbXmtVf/23ppVW3gX2hZIBzlHtXNFRp80w=

Now create a new file called /etc/hostname.wg0:

172.16.0.1/24 wgport 51820 wgkey YUkS6cNTyPbXmtVf/23ppVW3gX2hZIBzlHtXNFRp80w=
inet6 2001:db8:cafe:cafe:100::1 72

A new Wireguard interface called wg0 is being created. It will have the IPv4 address "172.16.0.1", Wireguard will listen on port 51820, and with the private key created shortly before. It will also have an IPv6 address on one of the subclasses that the provider will have provided.

Save and activate the interface:

sh /etc/netstart wg0

If everything has been entered correctly, it should enable the interface. Now:

ifconfig wg0

And something like this will be returned:

wg0: flags=80c3<UP,BROADCAST,RUNNING,NOARP,MULTICAST> mtu 1420
    index 5 priority 0 llprio 3
    wgport 51820
    wgpubkey xxxxxxxxxxxxxxx=
    groups: wg
    inet 172.16.0.1 netmask 0xffffff00 broadcast 172.16.0.255
    inet6 2001:db8:cafe:cafe:100::1 prefixlen 72

Take note of the wgpubkey - it will be needed to configure the clients.

As for the firewall, OpenBSD comes with a basic pf configuration. In my setups, I tend to block what is not needed and be permissive with what may be useful. However, I like to keep out the "bad guys," so I use blacklists. pf allows elements to be inserted and removed from tables in runtime, so the firewall can be configured accordingly.

To download and apply Spamhaus lists, I use a simple but effective script found on the Internet.

So create the script in /usr/local/sbin/spamhaus.sh:

#!/bin/sh
#
# this is normally run once per day via /etc/daily.local.
#
echo updating Spamhaus DROP lists:
(
  { ftp -o - https://www.spamhaus.org/drop/drop.txt && \
    ftp -o - https://www.spamhaus.org/drop/dropv6.txt ; \
  } 2>/dev/null | sed "s/;/#/" > /var/db/drop.txt
)
pfctl -t spamhaus -T replace -f /var/db/drop.txt

Make it executable and run it:

chmod a+rx /usr/local/sbin/spamhaus.sh
/usr/local/sbin/spamhaus.sh

There are many possibilities to configure pf. A fairly simple example could be this:

ext_if="vio0"
wg0_if = "wg0"
wg0_networks = "172.16.0.0/24"

set skip on lo

# Spamhaus DROP list:
table <spamhaus> persist file "/var/db/drop.txt"

block drop log quick from <spamhaus>

match in all scrub (no-df random-id max-mss 1440)

match out on $ext_if from { $wg0_networks } nat-to ($ext_if)

#Pass ICMP on ipv6
pass quick proto ipv6-icmp
#Block from ipv6 to wg0 network
block in quick on $ext_if inet6 to { 2001:db8:cafe:cafe:100::/72 }
#Pass Wireguard traffic - in and out
pass quick on $wg0_if

# default deny
block in
block out

# By default, do not permit remote connections to X11
block return in on ! lo0 proto tcp to port 6000:6010

# Port build user does not need network
block return out log proto {tcp udp} user _pbuild

pass in on $ext_if proto tcp to port ssh
pass in on $ext_if proto udp to port 51820

pass out on $ext_if

This is a very simple configuration: it blocks everything that is present in the list downloaded from Spamhaus, allows NAT from the Wireguard network to the public interface, allows icmp traffic in IPv6 (necessary for the network to function properly) while blocking incoming traffic to the Wireguard IPv6 LAN (remember that the IPs will be public and directly reachable, so we don't want to expose our devices by default). All traffic on the Wireguard interface will be allowed to pass. Then everything will be blocked and exceptions will be specified, i.e. allowing ssh and Wireguard connections (of course). Authorization will also be granted to allow traffic to exit from the public network interface.

Reload pf configuration:

pfctl -f /etc/pf.conf

If everything went correctly, the firewall should have loaded the new options.

To obtain caching of DNS queries and the related ad-block, it is now time to configure Unbound. A while ago, I found a script which I slightly adapted. I don't remember where I got it, so I'll paste it here without citing the original creator.

Create a script to update the unbound ad-block, in /usr/local/sbin/unbound-adhosts.sh:

#!/bin/ksh
#
# Using blacklist from pi-hole project https://github.com/pi-hole/
# to enable AD blocking in unbound(8)
#
PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin"

# Available blocklists - comment line to disable blocklist
_disconad="https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt"
_discontrack="https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt"
_stevenblack="https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts"

# Global variables
_tmpfile="$(mktemp)" && echo '' > $_tmpfile
_unboundconf="/var/unbound/etc/unbound-adhosts.conf"

# Remove comments from blocklist
function simpleParse {
  ftp -VMo - $1 | \
  sed -e 's/#.*$//' -e '/^[[:space:]]*$/d' >> $2
}

# Parse MalwareDom
#[[ -n ${_malwaredom+x} ]] && simpleParse $_malwaredom $_tmpfile

# Parse ZeusTracker
#[[ -n ${_zeustracker+x} ]] && simpleParse $_zeustracker $_tmpfile

# Parse DisconTrack
[[ -n ${_discontrack+x} ]] && simpleParse $_discontrack $_tmpfile

# Parse DisconAD
[[ -n ${_disconad+x} ]] &&  simpleParse $_disconad $_tmpfile

# Parse StevenBlack
[[ -n ${_stevenblack+x} ]] && \
  ftp -VMo - $_stevenblack | \
  sed -n '/Start/,$p' | \
  sed -e 's/#.*$//' -e '/^[[:space:]]*$/d' | \
  awk '/^0.0.0.0/ { print $2 }' >> $_tmpfile

# Parse hpHosts
[[ -n ${_hostfiles+x} ]] && \
  ftp -VMo - $_hostfiles | \
  sed -n '/START/,$p' | tr -d '^M$' | \
  sed -e 's/#.*$//' -e '/^[[:space:]]*$/d' -e 's/^M$//' | \
  awk '/^127.0.0.1/ { print $2 }' >> $_tmpfile

# Create unbound(8) local zone file
sort -fu $_tmpfile | grep -v "^[[:space:]]*$" | \
awk '{
  print "local-zone: \"" $1 "\" redirect"
  print "local-data: \"" $1 " A 0.0.0.0\""
}' > $_unboundconf && rm -f $_tmpfile

/usr/sbin/rcctl reload unbound 1>/dev/null

exit 0
#EOF

Similarly, make the script executable and run it:

chmod a+rx /usr/local/sbin/unbound-adhosts.sh
/usr/local/sbin/unbound-adhosts.sh

Now, the Unbound configuration file in /var/unbound/etc/unbound.conf can be modified as follows:

# $OpenBSD: unbound.conf,v 1.21 2020/10/28 11:35:58 sthen Exp $

server:
        verbosity: 1
        log-queries: no
        num-threads: 4
        num-queries-per-thread: 1024
        interface: 127.0.0.1
        #interface: 127.0.0.1@5353      # listen on alternative port
        interface: 172.16.0.1
    interface: 2001:db8:cafe:cafe:100::1
        interface: ::1
        outgoing-range: 64
        chroot: ""
        #do-ip6: yes

        # override the default "any" address to send queries; if multiple
        # addresses are available, they are used randomly to counter spoofing
        #outgoing-interface: 192.0.2.1
        #outgoing-interface: 2001:db8::53

        access-control: 0.0.0.0/0 refuse
        access-control: 127.0.0.0/8 allow
        access-control: ::0/0 refuse
        access-control: ::1 allow
        access-control: 172.16.0.0/24 allow
        access-control: 2001:db8:cafe:cafe:100::/72 allow

        hide-identity: yes
        hide-version: yes

        # Perform DNSSEC validation.
        #
        auto-trust-anchor-file: "/var/unbound/db/root.key"
        val-log-level: 2

        # Synthesize NXDOMAINs from DNSSEC NSEC chains.
        # https://tools.ietf.org/html/rfc8198
        #
        aggressive-nsec: yes
        prefetch: yes
        username: "nobody"
        directory: "/var/unbound/etc"
        logfile: "/var/unbound/unbound.log"
        use-syslog: no
        pidfile: "/var/unbound/unbound.pid"
        include: /var/unbound/etc/unbound-adhosts.conf

remote-control:
        control-enable: yes
        control-interface: /var/run/unbound.sock

Before launching unbound, it is necessary to give the appropriate permissions:

chown -R nobody:nobody /var/unbound

Now it is possible to enable and start unbound. Since it needs to load the (long) blocklist, it will take a few seconds:

rcctl enable unbound
rcctl start unbound

If everything has been done correctly, unbound will be able to respond to requests made on 172.16.0.1 and 2001:db8:cafe:cafe:100::1, from their respective LANs.

Now it is possible to configure the Wireguard client. Each implementation has its own procedure (Android, iOS, MikroTik, Linux, etc.) but essentially it is sufficient to create the right configuration both on the server and on the client. For example, the server's public key (visible by typing "ifconfig wg0" on the OpenBSD server) should be inserted into the "peer" that will be created on the client, while the client's public key will be used on the server in this way:

Reopen the file /etc/hostname.wg0 and add:

172.16.0.1/24 wgport 51820 wgkey YUkS6cNTyPbXmtVf/23ppVW3gX2hZIBzlHtXNFRp80w=
inet6 2001:db8:cafe:cafe:100::1 72
wgpeer *client's public key* wgaip 172.16.0.2/32 wgaip 2001:db8:cafe:cafe:100::2/128

Reload the configuration:

sh /etc/netstart wg0

On the client, create a new configuration by inserting "172.16.0.2/32, 2001:db8:cafe:cafe:100::2/128" (the ones that were entered in the peer configuration in the hostname.wg0 file) in the local IP addresses. Set the DNS server address to "172.16.0.1" and/or its corresponding IPv6 address (in the example, 2001:db8:cafe:cafe:100::1 - yours will be different). In the peer, insert the server's data, including its public key, IP address:port (in the example, the port is 51820), and allowed addresses (setting "0.0.0.0/0, ::0/0" means "all connections will be sent via Wireguard" - all the traffic will pass through the VPN for both IPv4 and IPv6).

It is also possible to use the VPN only as an ad-blocker, by only routing DNS traffic through it. To achieve this result, it is sufficient to configure the client so that the only allowed address is the one of the just-configured unbound (in this example, 172.16.0.1 or 2001:db8:cafe:cafe:100::1) - DNS resolution will occur via VPN, but browsing will continue to work through the main provider.

If you want the spamhaus and ad-block lists to be updated automatically, create the /etc/daily.local file and add the following lines:

#!/bin/ksh

/usr/local/sbin/unbound-adhosts.sh
/usr/local/sbin/spamhaus.sh

All of this can be achieved simply with a basic installation of OpenBSD, without the need to install any additional packages. This is an advantage both in terms of update management and security.

How we are migrating (many of) our servers from Linux to FreeBSD - Part 3 - Proxmox to FreeBSD

Stefano Marinelli — Tue, 14 Mar 2023 13:00:00 +0000

In recent years, we've been migrating many of our servers from Linux to FreeBSD as part of our consolidation and optimization efforts. Specifically, we've been moving services that were previously deployed using Docker onto FreeBSD, and it has proven to be a great choice for handling workloads efficiently.

To this end, we've also been migrating many of our virtual machines (VMs) to FreeBSD, deploying services within FreeBSD jails. In some cases, these jails have even replaced entire VMs and run bare metal. Although we prefer to move to native FreeBSD whenever possible, sometimes it's not the best option for all the services we offer. As a result, one of our most critical physical servers has been left behind for years.

This server was a Proxmox server that we installed many years ago and updated to version 6.4. It hosted some critical services, but upgrading to Proxmox 7.x posed some challenges. In particular, some of the LXC containers required tweaks.

Unfortunately, this server was quite old, with only four physical disks and 64 GB of RAM. It was located in an OVH data center and had been running well until one of the disks started to malfunction once a week, on Sundays. This would trigger a RAID reconstruction that kept the system busy for about two days.

Despite my preference for simple setups, this server had been deployed gradually over many years, and everything was tied together. As a result, unraveling the system to resolve the issues was not a simple task. Sometimes the combination of simple things can make everything complex.

The Proxmox Server

The Proxmox server was configured as the central hub for various services, including primary DNS, web hosting, VOIP, and more. It featured several bridges, each with its own specific purpose, and was connected to a virtual machine running MikroTik CHR. This machine was responsible for consolidating all incoming VPNs from the MikroTik devices we managed, both ours and those belonging to our clients. Additionally, it provided a series of bridges to manage these devices and all server management VPNs and other services. The Proxmox server also housed several virtual machines running Linux, FreeBSD, OpenBSD, and NetBSD, as well as LXC containers.

Over the last two years, we've been migrating most of these virtual machines and containers to FreeBSD-based VMs, which feature their own specific jails. Consequently, most of the VMs we've had to move were BSD-based, while only five Linux VMs remained. The LXC containers hosted a range of services, including servers managed by Virtualmin, a large installation of Zimbra (which was hosted within an LXC container running CentOS 7), as well as some minor Alpine Linux-based machines. We located all these virtual machines and containers in a LAN created and managed by CHR. All public IPs were managed by CHR, which relied on NAT mappings to establish communication between them. CHR had thus become the heart of our system, and if it experienced any issues, it could potentially take down the entire system. Fortunately, it remained stable for years.

Migration - first steps

The first step I took was to install FreeBSD on the new server. Easy peasy. The next step was to find a way for the CHR to migrate to the new server (under bhyve) and continue to manage all the public IPs of the original server. The problem is that OVH, with its failover IPs, ties a specific MAC address to each individual IP address. Therefore, the only way was to create a bridge on the FreeBSD server (on the Proxmox server, I already had the bridge on the physical network card) and create an L2 tunnel between the two servers - I used OpenVPN with tap interfaces, specifically inserted into the bridges. I could have used other methods and techniques, but I wanted to experiment with a setup that could allow, if necessary, to bridge a larger number of physical and virtual servers even if the IPs are all mapped to a single server. OVH does not allow, in fact, the splitting of classes, so a move must be made for the entire class, not for a single IP address.

Initially, MikroTik CHR 7 did not boot on bhyve. In the end, I managed to make it work, but I had other problems, probably related to the MTU of the interfaces. So I thought about taking the opportunity to unbind the LXC containers and VMs from CHR and remove MikroTik from the setup. With RouterOS version 7, in fact, Wireguard-based VPNs are also supported, so within a few days, it was possible to update the few routers still on 6.x and recreate some VPNs using Wireguard. I mapped both the VMs and LXC containers directly to their respective public IPs, greatly simplifying the steps. Everything worked perfectly.

The next step was to test the first migrations, starting from the VMs already on FreeBSD. For simplicity, I created a new FreeBSD VM in bhyve and copied (via zfs-send and zfs-receive) the datasets related to BastilleBSD. All services are installed in jails managed by Bastille, so this was enough to have, in a short time, a new operating server equivalent to the previous one. At that point, I shut down the original server, connected the VM to the bridge linked to the tunnel (after modifying its MAC address), turned on the new FreeBSD VM (on bhyve), and everything started to work correctly - but from the new physical server.

One by one, I moved all the FreeBSD VMs. For Linux, NetBSD, and OpenBSD, I simply copied the images and pointed bhyve to them. Some small specific configuration on vm-bhyve and everything started to work correctly. Where possibile, I replaced the “virtio” with “nvme” as it performs much better on bhyve.

Migration - LXC containers to Virtual Machines

For LXC containers, I initially thought of creating an Alpine Linux virtual machine, installing LXD, and copying each individual container. It worked for some of them, but for others, I started to encounter strange issues, similar to those that would have required manual intervention to upgrade from Proxmox 6.x to 7.x. As is often the case with Linux-based solutions, compatibility is not always preserved between updates, so I would have had to fine-tune all the containers, which I didn't feel like doing. The containers had been created (at the time) to optimize RAM usage on the Proxmox machine, but to date, they have caused more problems than benefits. In some cases, certain processes got "stuck," making it impossible to "reboot" the LXC container, requiring the entire physical node to be rebooted. If they had been virtual machines, I could have given a "kill" command from the virtualizer (to the respective KVM process, in that case) and restarted it.

For greater compatibility and ease of future management, I decided to convert the LXC containers into actual VMs on bhyve. The process was simple:

Creating an empty VM with vm-bhyve and booting the VM with SystemRescueCD.
Creating destination partitions and file systems in the VM, then doing a complete rsync of the original LXC container.
Adjusting the fstab file, installing the kernel on the destination VM, and creating the initrd (some containers were already copies of VMs, so the kernel remained installed and updated, even though it wasn't being used. The initrd, on the other hand, did not include the nvme or virtio drivers, so I had to regenerate it anyway.)
Adjusting the bhyve vm configuration file, doing one last rsync after shutting down the services, shutting down the original LXC container, and starting the bhyve VM.

Everything worked correctly, so one by one, I moved all the containers. The largest one ended up on another physical node (also FreeBSD with bhyve) temporarily because the space on the new server was not sufficient to contain it. It didn't need to be on this server, so no problem.

One by one, the LXC containers started on the new server. Apart from some minor adjustments to the destination VMs (different network interface names, etc.), I didn't encounter any particular problems even after several days. Everything works perfectly.

At the very end, I re-created the MikroTik CHR VM. I’ll keep this setup separate for now, as strictly tied to eoip interfaces. This was the main reason why I haven’t performed the migration before. Things were too tied together and I had to untie everything, step by step.

…and then one of the Linux VMs started to freeze

Several Linux VMs are just the basis on which Docker runs. One of them (not even among the busiest) started, every 12/15 hours, to completely freeze. It stopped responding to ping, and it was impossible to give any type of command from the console. In a word: stuck.

Searching the web, I found some references to this problem and, observing the errors of an ssh session that was left connected (stuck, but still showing the last error), I found it to be a problem similar to the one described in this post, namely:

"watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [khugepaged:67]"

I tried various solutions such as changing the storage driver, the number of cores, the distribution (from Alpine to Debian), etc., but none of these operations solved the issue. I also noticed that the problem occurs with all Linux VMs, but only those with a recent kernel (> 5.10.x) freeze, while the others continue to work. The problem does not occur, however, with the *BSDs.

In the end, I:

Reduced the number of cores to 1 for the VMs that did not have a high load (some remained with multiple cores), hypothesising a problem with allocating cores that were too busy
Gave the command: "/usr/bin/echo 60 > /proc/sys/kernel/watchdog_thresh" to the VM.

The VM became stable, and I have not seen that error/warning on any other machine since. I will investigate further, but I believe it is a problem related to the Linux kernel, which, for some reason, generates a kernel panic if particular situations of CPU concurrency are generated.

The End…and a nice OOM!

After moving everything, I was finally able to migrate the entire class of OVH IPs from one physical server to another. The operation was quite quick, but in order to avoid problems, I notified all users and performed the operation on a Sunday and during off-peak hours. The whole process took about 10 minutes and there were no hitches of any kind.

For safety reasons, I kept the Proxmox machine active for a few more days, but there was no need to use it. However, after a couple of days, I encountered a problem: the largest VM, in some cases, was being "killed" because FreeBSD generated an OOM. I had never seen, from FreeBSD 13.0 onwards, any OOM related to "abuse" of RAM usage by ZFS, but in this case, it actually happened.

In the end, I understood that ZFS, on FreeBSD, is able to release memory, but not quickly enough to manage any "spikes" in individual VMs. In fact, the VMs do not know the situation of the physical host's RAM, so they will tend to occupy all the space allotted to them (even if only for caching). A sudden spike (i.e. if you create and launch a new VM) could cause a sudden increase in RAM usage by the bhyve process, and FreeBSD could be forced to kill it, even if part of the RAM is only ARC cache. While Proxmox supports HA (i.e., control over whether the VM is running), vm-bhyve only launches the VM (bhyve process). I should manage it with tools like monit, but for now, I preferred to simply set limits on ZFS RAM usage using "vfs.zfs.arc_max", and there have been no more problems.

Final considerations

The operation was long but linear. The most complex part was unraveling all the configurations related to MikroTik CHR and the VPNs linked to each individual LXC machine/container. Once everything was implemented on a dedicated VM, the operation was fairly straightforward.

The hardware specifications of the destination physical server are slightly better than the starting one, but the final performance of the setup has greatly improved. The VMs are very responsive (even those that were previously LXC containers running directly on bare metal) and, thanks to ZFS, I can make local snapshots every 5 minutes. In addition, every 10 minutes, I can copy (using the excellent zfs-autobackup) all the VMs and jails to other nodes both as a backup and as an immediate restart in case of disaster. I just need to map the IPs, and everything will start working very quickly. Proxmox also allows you to perform this type of operation with ZFS, but you still need to have Proxmox (in a compatible version) on the target machine. With the current setup, I only need any FreeBSD node that supports bhyve.

Proxmox is an excellent tool, well-developed, open-source, efficient, and stable. We manage many installations, including complex ones (ceph clusters, etc.), and it has never let us down. However, not all tools are ideal for all situations, and for setups like the one described, the new configuration based on FreeBSD has shown significantly interesting performance and greater management and maintenance granularity.

Virtualizing on vm-bhyve is not complex, but it is certainly not comparable, at the current state, to the simplicity of using a clean and complete interface like Proxmox's. A complete HA system is still missing (sure, it's achievable manually, but...), as well as complete management web interface. However, for knowledgeable users, it is undoubtedly a powerful tool that allows you to have excellent FreeBSD as a base. I'm totally satisfied with my migration and the result is far better than I expected.

How we are migrating (many of) our servers from Linux to FreeBSD - Part 2 - Backups and Disaster Recovery

Stefano Marinelli — Mon, 30 May 2022 03:03:52 +0000

After my post on why we’re migrating (most of) our servers from Linux to FreeBSD, I’ve started to write about how we’re doing it. After covering a basic installation (we’re doing a massive use of jails), I’m going now to describe how we’re performing backups.

Backup is not a tool. Backup is not a software you can buy. Backup is a strategy you need to study and implement to be able to solve your specific problems. You need to understand what you’re doing, otherwise you’ll always have a Schrödinger’s Backup - it may work or not and if you don’t test it well enough (i.e. restore) you’ll find out when it’s too late.

We’re performing backups in many different ways but, for our physical and virtual FreeBSD servers, we have a dual approach. We need both a “ready to use” backup (that will be described here, useful for a fast disaster recovery or prompt restore of specific jails) and a “colder”, more space efficient backup that can be kept for months (or years), more similar to the borg approach on previous posts. Generally speaking, we store our OS (and jails) on ZFS, so I’ll describe this kind of approach here.

Disaster recovery backup - ZFS send/receive

BastilleBSD creates its datasets and mounts them on /usr/local/bastille . There are no databases, all the jails’ configurations are inside that mountpoint so it’s quite easy to backup and restore all the jails or any single jail in one go. More, the “everything-in-a-jail” approach simplifies the restore process as you don’t need to restore the entire host OS, just install an empty FreeBSD server, install BastilleBSD and restore the jails. Or add the jails to already existing FreeBSD systems.

We normally use FreeBSD (or Linux with ZFS) backup servers, well protected and encrypted at rest. For the ZFS send/receive approach, our servers are NOT reachable from the outside. We can ssh into them only using a VPN - they’re too precious to be exposed on the World Wild Web - or, if strictly needed, we expose ssh only using keys, no passwords. We perform the backups using a pull strategy: the backup server connects to the production servers, gets the data, disconnects. The production servers have NO ACCESS to the main backup server. Should they ever be seriously compromised, the backup is safe.

There are many tools that can help to set up this kind of configuration. I’ve tried many of them and found that they all have some good and bad points. The one I decided to use for our servers is zfs-autobackup. It’s easy to use, everything can be set via command line and has a good cron (or Jenkins) output, useful to understand if everything is right.

Let’s consider two servers, one is called “ProdA” and the other is called “Bck” - we obviously want to backup the ProdA into Bck.

Installing ProdA has been covered on a previous post, Bck is quite simple and outside the scope of this post. We just need a protected zfs FreeBSD (or Linux) server. That’s all. Let’s assume that ProdA has a BastilleBSD zfs dataset (and children datasets), with jails and everything needed, as configured in the last post. We now need to install the needed software. On Bck:

pkg install py311-zfs-autobackup mbuffer

On ProdA:

pkg install mbuffer

mbuffer will be used as a ram buffer to avoid read/write spikes (or slowdowns) while sending/receiving the snapshots.

It’s time to prepare the destination dataset. Assuming that Bck has a zroot base dataset, we’ll be creating (as root) a zroot/backups/ProdA

zfs create -p zroot/backups/ProdA

Ok, let’s now go to ProdA

We want to create an unprivileged user that will send the data. We don’t want to allow Bck to connect as root, even if it’s trusted and secure. Let’s create a user called “backupper”. Then, we need to give backupper the right permissions:

zfs allow -u backupper send,snapshot,hold,mount,destroy zroot

Note: if you want Bck to be able to delete the snapshots on ProdA, backupper needs the destroy permission. That means this user can destroy the whole system as can ALSO destroy zroot (or any source dataset you decide). If you’re afraid of this, different approaches must be used (i.e.: local root performing snapshot/cleanups and Bck only transferring them, not hard to achieve with zfs-autobackup). Considering that the Bck is safe, secure and protected, we can tolerate this weakness. Just be sure nobody can break the “backupper” user. Do not use password, use ssh keys and treat this user with the same care you'd use with root.

Now, as root on ProdA:

zfs set autobackup:bck_server=true zroot

We’re setting a custom property, called “autobackup:bck_server”, allowing the zroot (and children) dataset to be backed up by zfs-autobackup. zfs-autobackup will search for all datasets with that property set to “true” (also on different pools) and will backup them. If there’s a specific dataset you don’t want to backup, just set it to “false”. Or if you don’t want to backup the entire zroot but, for example, only “zroot/bastille” (and children), just set autobackup:bck_server=true for that dataset.

ssh config

zfs-autobackup will connect via ssh and zfs-autobackup will try to connect as root. Moreover, even after exchanging the ssh key, Bck will connect many times to ProdA to send its zfs commands (one connection per command). Ssh session initiation is quite long, so there will be some latency. In order to (greatly) speed up this time,

“You can make your ssh connections persistent and greatly speed up zfs-autobackup:

On the server that initiates the backup add this to your ~/.ssh/config:

Host ProdA
User backupper
ControlPath ~/.ssh/control-master-%r@%h:%p
ControlMaster auto
ControlPersist 3600

(Taken from https://github.com/psy0rz/zfs_autobackup/wiki/Performance)

It's now time to go back to Bck and issue a command like this (one line):

/usr/local/bin/zfs-autobackup --ssh-source ProdA bck_server zroot/backups/ProdA --zfs-compressed --no-progress --verbose --buffer 32M --keep-source 0 --no-holds  --set-properties readonly=on --clear-refreservation --keep-target 1d1w,1w1m,1m6m  --destroy-missing 30d --clear-mountpoint

Bck will connect to ProdA, perform the snapshots and start transferring. The most interesting options I used here are:

 --keep-source 0 (only the last snapshot will be kept on ProdA)
 --set-properties readonly=on (be sure the Bck clone is read only, so we will be able to perform an incremental/differential backup next time)
 --keep-target 1d1w,1w1m,1m6m (keep one backup per day for one week, one per week for one month, one per month for six months)
 --destroy-missing 30d (if we've deleted a dataset, keep it for 30 days before removing it from Bck)
 --clear-mountpoint (do not mount the dataset in Bck, as it will cause problems sooner or later)

The first copy will be slow as it'll need to send all the data. The second will be quite fast as only the differences will be transferred.

How to perform a disaster recovery

Ok, your dataset (or datasets) has gone. You need to replace it with the last external backup. You have to retransfer the copy into ProdA (or another FreeBSD host, no difference). Connect to Bck and search for the snapshot you want to restore (zfs list -t snapshot will help). Once identified (one line):

zfs send -R zroot/backups/ProdA/zroot/bastille/bastille/jails/t1@bck_server@20220528005830 | mbuffer -4 -s 128k -m 32M | ssh root@ProdA "zfs receive -F -x canmount -x readonly zroot/bastille/bastille/jails/t1"

Note the -x canmount -x readonly flags. Remember that we altered the canmount and readonly properties of the transferred datasets during the backup, so we must restore them into a normal state.

Once finished, ProdA (or the other, restored host) will show t1 as an available jail and you'll be able to start it.

How we are migrating (many of) our servers from Linux to FreeBSD - Part 1 - System and jails setup

Stefano Marinelli — Sat, 05 Feb 2022 10:10:47 +0000

After my post on why we’re migrating (many of) our servers to FreeBSD, I’ve received a lot of feedback. Many questions, many comments. Many e-mails from Linux users asking how we’re migrating, how jails can replace lxc or (in part) Docker, and how we’re monitoring and performing backups/restores.

I’ll write some posts on how we’re doing it. Of course, it won’t cover all the use cases, but it surely will work for the most common ones.

Let’s start with a general idea of the setup I’m going to describe. One of the things I’ve always tried to do is to leave the host Operating System (VM Hypervisors like XCP-NG or Proxmox Server, FreeBSD with jails and/or Bhyve, Alpine Linux lxc host, Debian with docker etc.) as simple and empty as possible.

That’s why we’re keeping the same setup here, with a FreeBSD host system with as few packages (or ports) as possible. This will ensure easier upgrades, easier backup/restore procedures and a smaller attack surface. Unused services or executables can be problematic, so let’s keep them out of our setup.

I’m going to describe our “basic” FreeBSD host and some jails that will contain the services. There are many jail management systems out there like BastilleBSD, cbsd, iocage, the old ezjail, etc. I love simple (but powerful) solutions, without any database of configured/running jails as it may be a problem in case of backup/recovery. I love the BastilleBSD approach, so that’s the one we’ve been using for our servers. BastilleBSD is a collection of shell scripts, is small, doesn’t need any database, is actively developed, doesn’t need any kind of dependency and works both on ZFS and UFS. iocage, for example, needs ZFS so UFS servers are out and doesn’t seem to be actively developed anymore. Moreover, BastilleBSD doesn’t interfere with other jail systems, so you can mix and match whatever you like. Still, I recommend to choose a jail management system and stick with that. We've done it, and we’re using BastilleBSD.

I won’t describe basic FreeBSD installation. It’s straightforward, easy, fast and it’s full of good documentation out there. The most critical decision is the choice of the file system you’re going to use. We’re using ZFS when possible, as it gives us a lot of good opportunities. For more resource constrained systems (or specific situations where ZFS isn’t recommended nor applicable), we just stick with UFS. It has snapshots, too, so backups and restores are quite easy, too.

Once you’ve installed FreeBSD, you have a fully functional host system. From now on, I’ll assume you’ve installed on ZFS.

First of all, we upgrade our hosts systems to the last security patches. It’s just a matter of:

freebsd-update fetch
freebsd-update install

After this, It’d be better to reboot to be sure that the kernel has been upgraded if needed. As it’s an empty and just installed system, we usually don’t use the Boot Environments for now. Reinstalling is just a matter of minutes and we’re not doing a release upgrade.

Now: ports or packages? We usually use packages: the host system will be simple and empty and we don’t need customised build options. So quarterly packages are more than enough.

First of all, let’s install the “pkg” package management system.

pkg install

BastilleBSD

Now let’s use pkg to install BastilleBSD:

pkg install bastille

The first step is to configure BastilleBSD. All we usually do is: set the ZFS options. bastille_zfs_enable=“YES” and bastille_zfs_pool=“zroot/bastille” and create the dataset:

zfs create zroot/bastille

This way, BastilleBSD will create its base dataset in zroot/bastille/bastille

Why the nested bastille/bastille dataset? Because we sometimes use zroot/bastille to store some information about the underlying jails, so we prefer to keep a nested dataset. BastilleBSD will create its datasets there and will mount the needed datasets in /usr/local/bastille so you will find everything there.

Let’s complete the Bastille installation as described in the official documentation. There are many approaches: loopback, vnet with local interface, vnet with existing bridge. The loopback approach is the easiest and more portable. Generally speaking we tend to use it as it’s easier to deal with when we have to perform an emergency restore to another host. I’ll write more about it in the next posts.

Now let’s bootstrap the FreeBSD 14.1-RELEASE so Bastille will download the needed files and create its base system, then it will apply the security patches.

bastille bootstrap 14.1-RELEASE update

NOTE: BastilleBSD supports other jailed operating systems and, recently, some jailed Linux distributions.

After a while, the system will be ready for its first jail. Now, we generally install a reverse proxy, in order to expose it to web traffic. The reverse proxy will be able to connect to other jails and forward the traffic. It's a good example, so let's do it:

bastille create reverseproxy 14.1-RELEASE 192.168.0.1 bastille0

So we’re creating a jail called reverseproxy, the FreeBSD version is 14.1-RELEASE (of the jail, must be the same or older than the host OS), the jail ip and the loopback interface to use to assign this IP for the jail. The jail will have a dataset (in jails/reverseproxy) with jail configuration, redirect configuration, fstab, etc. and another child dataset (jails/reverseproxy/root) with its root file system. Jails can be thin or thick: BastilleBSD documentation is good, so you can go deeper here.

At this point, a “bastille list -a” should show the jail and it should be running. Now we can enter this jail:

bastille console reverseproxy

and install Nginx (and certbot, if needed):

pkg install py311-certbot-nginx nginx

Configure your nginx (and certbot). As you might guess, I won’t describe it here.

Let’s ensure Nginx will be started at jail launch, so:

service nginx enable

and let’s start it:

service nginx start

Let’s go back to the host and let’s ensure that all the connections to the host ports 80 (http) and 443 (https) will be redirected to the reverseproxy jail:

bastille rdr reverseproxy tcp 80 80
bastille rdr reverseproxy tcp 443 443

No output should be shown, but you can check:

bastille rdr ALL list

Congratulations, you’ve exposed your first jail.

You can create all the jails you want. They will be created and stored in child datasets of zroot/bastille/bastille - this is great for backup purposes, but I’ll describe more about it in another article. They'll also be able to communicate using their private ip addresses. If you've used vnet, you'll need to perform some deeper network configurations (and you can use pf inside the jail!), if using the default loopback device you'll be sharing the host network stack.

Blacklistd

FreeBSD base system has some interesting tools and they get automatically installed. One of those is blacklistd. If you’ve used fail2ban, denyhosts or similar tools, you know what it’s useful for. But it’s integrated and is light. Fail2ban, for example, tends to become heavy and huge as it’s reading from log files. Blacklistd gets notified by the daemon it’s protecting, so load is lower. To enable blacklistd, add to /etc/rc.conf:

blacklistd_enable="YES"
blacklistd_flags="-r" 
pflog_enable="YES"

(I’m assuming you’ve already added pf_enable=“YES” when you’ve installed BastilleBSD. Otherwise, you should add this, too, if you’re using pf.

Now you should add this to pf.conf:

anchor "blacklistd/*" in on $ext_if

Where $ext_if is your external interface.

Last step, let’s get to /etc/ssh/sshd_config and enable blacklistd uncommenting:

UseBlacklist yes

Now reload/start pf, sshd, start pflog and blacklistd and wait. After some time, blacklistctl dump -r will show you some data.

Of course there are many more steps to do, the host should be hardened, network should be configured and firewalled, etc. but it's a basic idea of how we're keeping our host as standard as possible and, then, create the services inside the jails.

Why we're migrating (many of) our servers from Linux to FreeBSD

Stefano Marinelli — Mon, 24 Jan 2022 06:55:00 +0000

More about this in the article I wrote to accompany my talk at EuroBSDCon 2024.

Please note: This article was (unexpectedly) quite popular and many interesting opinions came out from the Hacker News thread that it generated. If you're interested, have a look here: https://news.ycombinator.com/item?id=30057549

There's an Italian version of this article here.

I've been a Linux (or GNU/Linux, for the purists) user since 1996. I've been a FreeBSD user since 2002. I have always successfully used both operating systems, each for specific purposes. I have found, on average, BSD systems to be more stable than their Linux equivalents. By stability, I don't mean uptime (too much uptime means too few kernel security updates, which is wrong). I mean that things work as they should, that they don't "break" from one update to the next, and that you don't have to revise everything because of a missing or modified basic command.

I've always been for development and innovation as long as it doesn't (necessarily, automatically and unreasonably) break everything that is already in place. And the road that the various Linux distributions are taking seems to be that of modifying things that work just for the sake of it or to follow the diktats of the Kernel and those who manage it - but not only.

Some time ago we started a complex, continuous and not always linear operation, that is to migrate, where possible, most of the servers (ours and of our customers) from Linux to FreeBSD.

Why FreeBSD?

There are many alternative operating systems to Linux and the *BSD family is varied and complete. FreeBSD, in my opinion, today is the "all rounder" system par excellence, i.e. well refined and suitable both for use on large servers and small embedded systems. The other BSDs have strengths that, in some fields, make them particularly suitable but FreeBSD, in my humble opinion, is suitable (almost) for every purpose.

So back to the main topic of this article, why am I migrating many of the servers we manage to FreeBSD? The reasons are many, I will list some of them with corresponding explanations.

The system is consistent - kernel and userland are created and managed by the same team

One of the fundamental problems with Linux is that (we shall remember) it is a kernel, everything else is created by different people/companies. On more than one occasion Linus Torvalds as well as other leading Linux kernel developers have remarked that they care about the development of the kernel itself, not how users will use it. In the technical decisions, therefore, they don't take into account what is the real use of the systems but that the kernel will go its own path. This is a good thing, as the development of the Linux kernel is not "held back" by the struggle between distributions and software solutions, but at the same time it is also a disadvantage. In FreeBSD, the kernel and its userland (i.e. all the components of the base operating system) are developed by the same team and there is, therefore, a strong cohesion between the parties. In many Linux distributions it was necessary to "deprecate" ifconfig in favor of ip because new developments in the kernel were no longer supported by ifconfig, without breaking compatibility with other (previous) kernel versions or having functions (on the same network interface) managed by different tools. In FreeBSD, with each release of the operating system, there are both kernel and userland updates, so these changes are consistently incorporated and documented, making the tools compatible with their kernel-side updates.

In other words, in FreeBSD there is no need to " revolutionise" everything every few years and changes are made primarily in the form of additions that can enrich (and not break) each update. If a modification was to change the way it interacts with network devices, ifconfig would be modified to take advantage of that and remain compatible with the "old" syntax. In the long-term, this kind of approach is definitely appreciated by system administrators who find themselves with a linear, consistent, and always well-documented update path.

FreeBSD development is (still) driven by technical interests, not strictly commercial ones.

Linux and related distributions now have contributions from many companies, many of which (e.g. Red Hat) push (justifiably) in the direction of what is convenient for them, their products, and their services. Being big contributors to the project they have a big clout so, indeed, their solutions often become de-facto standards. Consider systemd - was there really a need for such a system? While it brought some advantages, it added some complexity to an otherwise extremely simple and functional system. It remains divisive to this day, with many asking, "but was it really necessary? Did the advantages it brought balance the disadvantages?". 70 binaries just for initialising and logging and a million and a half lines of code just for that? But Red Hat threw the rock...and many followed along. Because sometimes it's nice to follow the trend, the hype of a specific solution.

Even FreeBSD has big companies behind it, collaborating in a more or less direct way. The license is more permissive, so not everyone who uses it commercially contributes to it, but knowing that FreeBSD is at the base of Netflix CDNs, Whatsapp servers (waiting for Meta to replace them, for internal coherence reasons, with Linux servers), Sony Playstations and, in part, macOS, iOS, iPadOS, etc. surely gives confidence on its level. These realities, however, do not have enough clout to drive the development of the core team.

Linux has Docker, Podman, lxc, lxd, etc. but... FreeBSD has jails!

FreeBSD jails are very powerful tools for jailing and separating services. There is controversy about Docker not running on FreeBSD but I believe (like many others) that FreeBSD has a more powerful tool. Jails are older and more mature - and by far - than any containerization solution on Linux. Jails are efficient and are well integrated throughout the operating system. All major commands (ps, kill, top, etc.) are able to display jail information as well. There are many management tools but, in fact, they all do the same thing: they interact with FreeBSD base and create custom configuration files. Personally I'm very comfortable with BastilleBSD but there are a lot of very good tools as well as a sufficiently simple manual management. When I need Docker I launch a Linux machine - often Alpine, which I think is a great minimalist distribution, or Debian. But I'm moving a lot of services from Docker to a dedicated jail on FreeBSD. Docker containers are a great tool for rapid (and consistent) software deployment, but it's not all fun and games. Containers, for example, rely on images that sometimes age and are no longer updated. This is a security issue that should not be overlooked.

Linux has ext4, xfs, btrfs... (and zfs, but with some manual intervention). FreeBSD has UFS2 and ZFS

UFS2 is still a very good and efficient file system and, when configured to use softupdates, capable of performing live snapshots of the file system. This is great for backups. Ext4 and XFS do not support snapshots except through external tools (like DattoBD or snapshots through the volume manager). This works, of course, but it is not native. Btrfs is great in its intentions but still not as stable as it should be after all these years of development. FreeBSD supports ZFS natively in the base system and this brings many advantages: separation of datasets for jails, as well as Boot Environments, to make snapshots before upgrades/changes and to be able to boot (from bootloader) even on a different BE, etc.

The FreeBSD boot procedure is cleaner and simpler.

Linux has always used excellent tools such as grub, lilo (now outdated), etc.. FreeBSD has always used a very linear and consistent boot system, with its own bootloader and dedicated boot partition. Whether on mbr, gpt, etc. things are very similar and consistent. I've never had a problem getting a FreeBSD system to boot after a move or recovery from backup. On Linux, however, grub has sometimes given me problems, even after a simple kernel security update.

FreeBSD's network stack is (still) superior to Linux's - and, often, so is its performance.

Meta has been trying to bring the performance of the Linux network stack up to the level of FreeBSD's for years. Many will ask why, then, not move services to FreeBSD. Large companies with huge datacenters can't change solutions overnight, and their engineers, at any level, are Linux experts. They have invested heavily in btrfs, in Linux, in their specifics. Clearly, upon acquiring Whatsapp, they preferred to migrate the "few" Whatsapp servers to Linux and move them to their datacenters. Regarding the real system performance (i.e. disregarding benchmarks, useful only up to a certain point), FreeBSD shines, especially under high load conditions. Where Linux starts to gasp (ex: waiting for I/O) with 100% CPU, FreeBSD has lower processor load and room for more stuff. In the real world (of my servers and load types), I sometimes experienced severe system slowdowns due to high I/O, even if the data to be processed was not read/write dependent. On FreeBSD this does not happen, and if something is blocking, it blocks THAT operation, not the rest of the system. When performing backups or other important operations this factor becomes extremely important to ensure proper (and stable) system performance.

Straightforward system performance analysis

FreeBSD, in the base system, has all the tools to analyze possible problems and system loads. "vmstat" , in a single line, tells me if the machine is struggling for CPU, for I/O or for Ram. "gstat -a" shows me how much, disk by disk, partition by partition, the storage is active, also in percentage with reference to its performance. "top", then, also has support for figuring out, process by process, how much I/O is being used ("m" option). On Linux, to get the same results, you have to install specific applications, different from distribution to distribution.

Bhyve is more incomplete (but more efficient) than KVM

For my purposes, Bhyve is a great virtualization tool. KVM is definitely more complete but since I don't have any special or specific needs not covered by Bhyve on FreeBSD, I found (on average) better performance with this combination. On FreeBSD, however, KSM is missing which, in some cases, can be very useful.

Will I abandon Linux for FreeBSD? Obviously not, just as I haven't for the last 20 years. Both have their uses, their space, their strengths. But if up to now I have had 80% Linux and 20% FreeBSD, the perspective is to invert the percentages of use and, where possible, directly implement solutions based on FreeBSD.

NOTE: this article has been translated from its Italian original version. Even if it's been reviewed and adapted, there might be some errors.