IT Notes - tutorial

Monitor your devices with LibreNMS on FreeBSD

Stefano Marinelli — Thu, 07 May 2026 10:45:00 +0000

LibreNMS has been a faithful companion for years now. It quietly handles the monitoring of my servers, devices, and services without demanding much in return - exactly what you want from a tool whose job is to watch over everything else. It's a solid alternative to heavier solutions like Zabbix, and it gives you alerts, data, and graphs on virtually anything reachable over SNMP.

I usually install it on a host that is not reachable from the outside, then let it poll all the devices through a VPN: a single observation point, clean perimeter. The ability to create multiple dashboards - and to filter them by user - has also let me give clients a transparent window onto their own servers. Transparency, in my experience, is always the better long-term bet.

Together with Uptime-Kuma (and the good old Nagios/Munin pair), LibreNMS lives in a FreeBSD jail on my monitoring servers and just does its job.

This post walks through a plain installation of LibreNMS on FreeBSD: package-based, no reverse proxy, no HTTPS, no fancy hardening. The goal is to get to a working setup you can build on top of.

Assumptions

FreeBSD 15.0-RELEASE, in a jail or on a dedicated VM/host
nginx + php-fpm + MySQL 8.4
LibreNMS installed from the official package — not via git clone

One note before we start: in this guide I use plain HTTP just to reach the first-time setup. If your LibreNMS instance won't stay confined to a private network or behind a VPN, configuring HTTPS is mandatory, not optional.

Installation

pkg install librenms mysql84-server python3 nginx

LibreNMS currently depends on PHP 8.4. If you want to speed PHP up, install OPcache too:

pkg install php84-opcache

MySQL

Two settings need to be in place before MySQL starts for the first time. After the first start they cannot be changed without reinitializing the data directory, so it's worth getting them right now.

cd /usr/local/etc/mysql
cp my.cnf.sample my.cnf

In the [mysqld] section, add:

innodb_file_per_table=1
lower_case_table_names=0

Now start MySQL:

service mysql-server enable
service mysql-server start

On a fresh FreeBSD install, the local root user can connect to MySQL without a password from the command line. Connect and create the database and user. I'm using password here as a placeholder - don't.

mysql

CREATE DATABASE librenms CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER 'librenms'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON librenms.* TO 'librenms'@'localhost';
exit

php-fpm

Edit /usr/local/etc/php-fpm.d/www.conf and adjust the listen directives:

listen = /var/run/php-fpm-librenms.sock
listen.owner = www
listen.group = www
listen.mode = 0660

Then create php.ini from the production sample:

cd /usr/local/etc
cp php.ini-production php.ini

And set the timezone in php.ini:

date.timezone = Europe/Rome

nginx

Since this jail (or host) is dedicated to LibreNMS, we can rewrite the server block in /usr/local/etc/nginx/nginx.conf directly:

server {
    listen      80;
    #server_name yourServerName
    root        /usr/local/www/librenms/html;
    index       index.php;

    charset utf-8;
    gzip on;
    gzip_types text/css application/javascript text/javascript application/x-javascript image/svg+xml text/plain text/xsd text/xsl text/xml image/x-icon;

    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    location /api/v0 {
        try_files $uri $uri/ /api_v0.php?$query_string;
    }

    location ~ \.php$ {
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
        set $path_info $fastcgi_path_info;
        try_files $fastcgi_script_name =404;
        include fastcgi_params;
        fastcgi_param SERVER_SOFTWARE "";
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param PATH_INFO $path_info;
        fastcgi_index index.php;
        fastcgi_pass unix:/var/run/php-fpm-librenms.sock;
        fastcgi_buffers 256 4k;
        fastcgi_intercept_errors on;
        fastcgi_read_timeout 14400;
    }

    location ~ /\.(?!well-known).* {
        deny all;
    }
}

Now start nginx and php-fpm:

service nginx enable
service nginx start

service php_fpm enable
service php_fpm start

LibreNMS configuration

Copy the default config:

cp /usr/local/www/librenms/config.php.default /usr/local/www/librenms/config.php

Because we installed from the package, this file already has the right commands and paths for FreeBSD - no need to hunt down mtr, fping, snmpwalk and friends one by one.

Create the directory for RRD graphs and set ownership:

mkdir -p /var/db/librenms/rrd
chown -R www:www /var/db/librenms
chmod 775 /var/db/librenms/rrd

Then the .env file:

cd /usr/local/www/librenms
cp .env.example .env
chown www .env

Edit .env and set at least:

DB_DATABASE - librenms
DB_USERNAME - librenms
DB_PASSWORD - the one you actually used (not password, please)

Then add this line, which tells LibreNMS we still need to run the web installer:

INSTALL=true

A note on permissions. The official LibreNMS documentation suggests chown -R www:www over the entire application tree, but on FreeBSD the package already lays down sane ownership, with storage/ and bootstrap/cache/ writable by www. There's no reason to widen the rest of the codebase. If validate.php complains later about something write-related, the first place to check is:

ls -la /usr/local/www/librenms/storage /usr/local/www/librenms/bootstrap/cache

Now generate the app key as www, since the file is owned by www:

su -m www -c "php artisan key:generate"

And tighten .env:

chmod 600 .env

Refresh the configuration cache:

su -m www -c "lnms config:clear"
su -m www -c "lnms config:cache"

Web installer

Open http://host/install and follow the steps. The validation process may fail. Refreshing the cache picks up the values written to config.php during the install:

su -m www -c "lnms config:clear"
su -m www -c "lnms config:cache"

When the web installer is done, edit .env again and remove the INSTALL=true line if it's still there. Leaving it in place re-exposes the installer to anyone who can reach the URL.

Polling service

LibreNMS needs something to actually run the polls. On FreeBSD, the package ships an rc service that runs the LibreNMS dispatcher, so there's no need to manage cron entries by hand the way most Linux guides assume.

service librenms enable
service librenms start

Validate

cd /usr/local/www/librenms
su -m www -c './validate.php'

You may see a couple of complaints right after starting the service - usually scheduler-related and self-resolving within a few minutes. Re-run validate.php once the dispatcher has had time to settle. Anything still red after that is worth investigating.

Next steps

At this point you can log into the web interface and start adding devices, configuring SNMP, and building dashboards. For that, the official LibreNMS documentation is excellent, and there's no point in me paraphrasing it here.

Time Machine inside a FreeBSD jail

Stefano Marinelli — Wed, 28 Jan 2026 08:52:00 +0000

Many of my clients do not use Microsoft systems on their desktops; they use Linux-based systems or, in some cases, FreeBSD. Many use Apple systems - macOS - and are generally satisfied with them. While I wash my hands of it when it comes to Microsoft systems (telling them they have to manage their desktops autonomously), I am often able to lend a hand with macOS. And one of the main requests they make is to manage the backups of their individual workstations.

macOS, thanks to its Unix base, offers good native tools. Time Machine is transparent and effective, allowing a certain freedom of management. APFS, Apple's current file system, supports snapshots, so the backup will be effectively made on a snapshot. It also supports multiple receiving devices, so you can even have a certain redundancy of the backup itself.

Having many FreeBSD servers, I am often asked to use their resources and storage. To build, in practice, a Time Machine inside one of the servers. And it is a simple and practical operation, quick and "painless". There are many guides, including the excellent one by Benedict Reuschling from which I took inspiration for this one, and I will describe the steps I usually follow to set it all up in just a few minutes.

I usually use BastilleBSD to manage my jails, so the first step is to create a new jail dedicated to the purpose. Here you have to decide on the approach: I suggest using a VNET jail or an "inherit" jail - meaning one that attaches to the host's network stack. On one hand, the inherit approach is less secure but, as often happens, it depends on the complexity of the situation. If, for example, we are using a Raspberry PI dedicated to the purpose, there is no reason to complicate things with bridges, etc., but we can attach directly to the network card with a creation command like:

bastille create tmjail 15.0-RELEASE inherit igb0

Where igb0 is the network interface we want to attach to.

In case we want to attach to the interface but in the form of a bridge, we should use this syntax:

bastille create -V tmjail 15.0-RELEASE 192.168.0.42/24 igb0

Or, if our server already has a bridge (in this case it's bridge0, but yours might be named differently):

bastille create -B tmjail 15.0-RELEASE 192.168.0.42/24 bridge0

At this point, you can choose: do we want to keep the backups inside the jail or in a separate dataset - which can even be on another pool? In some cases, this can be extremely useful: often I have jails running on fast disks (SSD or NVMe) but abundant storage on slower devices. In this example, therefore, I will create an external dataset for the backups (directly from the host) and mount it in the jail. You could also delegate the entire management of the dataset to the jail, which is a different approach.

Let's create a space of 600 GB - already reserved - on the chosen pool. 600 GB is a small space, but it's ok for an example:

zfs create -o quota=600G -o reservation=600G bigpool/tmdata

We can also create separate datasets inside for each user and assign a specific space:

zfs create -o refquota=500g -o refreservation=500g bigpool/tmdata/stefano

We can enter the jail and install what we need, remembering also to create the "mountpoint" for the dataset we just created:

bastille console tmjail 

pkg install -y samba419
mkdir /tmdata

Exit the jail and instruct Bastille to mount the dataset inside the jail every time it is launched:

exit
bastille mount tmjail /bigpool/tmdata /tmdata nullfs rw 0 0

Let's go back into the jail and start with the actual configuration. First, for each Time Machine user, we will create a system user. In my example, I will create the user "stefano", giving him /var/empty as the home directory - this will give an error since we created a Bastille thin jail, but it's not a problem. It happens because in a thin jail some system paths are read-only or not manageable as they are on a full base system, but the user is only needed for ownership and Samba login.

root@tmjail:~ # adduser
Username: stefano
Full name: Stefano
Uid (Leave empty for default):
Login group [stefano]:
Login group is stefano. Invite stefano into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]: nologin
Home directory [/home/stefano]: /var/empty
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]: no
Lock out the account after creation? [no]:
Username    : stefano
Password    : <disabled>
Full Name   : Stefano
Uid         : 1001
Class       :
Groups      : stefano
Home        : /var/empty
Home Mode   :
Shell       : /usr/sbin/nologin
Locked      : no
OK? (yes/no) [yes]: yes
pw: chmod(var/empty): Operation not permitted
pw: chown(var/empty): Operation not permitted
adduser: INFO: Successfully added (stefano) to the user database.
Add another user? (yes/no) [no]: no
Goodbye!

Give the correct permissions to the user:

# If you've not created specific datasets for the users, you'd better create their home directories now
mkdir /tmdata/stefano
chown -R stefano /tmdata/stefano/

Now we configure Samba for Time Machine. The file to create/modify is /usr/local/etc/smb4.conf:

[global]
workgroup = WORKGROUP
security = user
passdb backend = tdbsam
fruit:aapl = yes
fruit:model = MacSamba
fruit:advertise_fullsync = true
fruit:metadata = stream
fruit:veto_appledouble = no
fruit:nfs_aces = no
fruit:wipe_intentionally_left_blank_rfork = yes
fruit:delete_empty_adfiles = yes

[TimeMachine]
path = /tmdata/%U
valid users = %U
browseable = yes
writeable = yes
vfs objects = catia fruit streams_xattr zfsacl
fruit:time machine = yes
create mask = 0600
directory mask = 0700

We have set up Time Machine to support all the necessary features of macOS and to show itself as "Time Machine". Having set path = /tmdata/%U, each user will only see their own path.

At this point, we create the Samba user (meaning the one we will have to type on macOS when we configure the Time Machine):

smbpasswd -a stefano

The Time Machine is seen by macOS because it announces itself via mDNS on the network. This type of service is performed by Avahi, which we are now going to configure. Although not strictly necessary (we can always find the Time Machine by connecting directly to its IP and macOS will remember everything), seeing it announced will help other non-expert users and ourselves when we have to configure another Mac in the future.

Recent Samba releases won't need any specific avahi configuration, so we can skip this step.

We are now ready to enable everything.

service dbus enable
service dbus start
service avahi-daemon enable
service avahi-daemon start
service samba_server enable
service samba_server start

Et voilà. If everything went according to plan, the Time Machine will announce itself on your network (if you have different networks, remember to configure the mDNS proxy on your router) and you will be able to log in (with the smb user you created) and start your first backup.

I suggest encrypting the backups for maximum security and observing, from time to time, your Mac as it silently makes its backups to your trusted FreeBSD server.

Installing Void Linux on ZFS with Hibernation Support

Stefano Marinelli — Mon, 22 Dec 2025 08:43:02 +0000

Introduction

FreeBSD continues to make strides in desktop support, but Linux still holds an advantage in hardware compatibility. After running openSUSE Tumbleweed on my mini PC for several months, I decided it was time to switch to a solution I could control more closely. Not because Tumbleweed doesn't work well - it works great! - but I prefer having direct control over what happens on my machine. And I want native ZFS, because I prefer it over btrfs and it allows me to manage snapshots, backups, and rollbacks just as I do on FreeBSD, using the same tools and procedures.

The choice of Void Linux comes from its BSD-like approach: modular and free of unnecessary complexity. This makes it an excellent solution for this type of setup.

ZFSBootMenu is an extremely powerful tool. It provides an experience similar to FreeBSD's boot loader and natively supports ZFS. I strongly recommend reading the documentation and exploring its features, as some of them - like the built-in SSH daemon - can be genuine lifesavers in recovery scenarios.

Prerequisites and Audience

This guide is not for absolute beginners. If you're new to Linux or Unix-like operating systems, you'd be better served by a ready-to-use distribution like openSUSE Leap (or Tumbleweed for a rolling distribution), Linux Mint, Debian, Ubuntu, or Manjaro. The purpose of this article is to demonstrate a stable, upgradeable, and reasonably secure base setup for users already comfortable with system administration. It uses the glibc variant of Void Linux. The musl version requires different commands, for example for locale generation.

Use at your own risk.

This guide synthesizes instructions from several sources:

Void Linux (UEFI) from ZFSBootMenu - which doesn't address swap. Using a zvol for swap (not the best solution) prevents hibernation and resume. Our approach uses a separate encrypted swap partition that enables proper resume.
Void Linux Full Disk Encryption - excellent for btrfs or ext4, but we want ZFS. We'll borrow the swap configuration approach from here.
Install Void Linux with a desktop environment + Flatpaks - for the desktop portion.

If your setup differs from what's described here (NVMe disk, UEFI boot, Secure Boot disabled), consult the linked guides for explanations and variations.

Installation Script (Optional)

If you want to reproduce this setup quickly, I maintain a script that automates the procedure described in this guide: disk partitioning, ZFS pool and dataset creation, encrypted swap for hibernation resume, dracut configuration, and ZFSBootMenu EFI setup. An optional KDE Plasma desktop installation is also supported.

The script is interactive and will ask for the required parameters (target disk, timezone and keymap, passphrases, desktop options). Requirements, usage instructions, and known limitations are documented in the repository README

That said, I still recommend going through the manual process at least once. Understanding each step is part of the value of this setup, especially when troubleshooting or adapting it to different hardware.

Boot Environment

Since ZFS isn't supported by the base Void Linux image, we'll use hrmpf, an excellent rescue system based on Void Linux that includes ZFS support out of the box.

After booting, you can either proceed directly or SSH into the machine to continue remotely. I generally prefer SSH since it makes copy-paste operations much easier - especially when dealing with UUIDs and long commands. To enable SSH access, set a root password and allow root login:

passwd

Edit /etc/ssh/sshd_config and enable:

PermitRootLogin yes

Restart the SSH daemon:

sv restart sshd

Find the machine's IP address:

ip addr

You can now connect via SSH from another device.

Initial Setup

Set up the environment variables and generate a host ID - we need it for ZFS:

source /etc/os-release
export ID

zgenhostid -f 0x00bab10c

Disk Configuration

Identify your target disk and set up the partition variables. This approach keeps everything consistent and reduces errors:

# Set the base disk - adjust this to match your system
export DISK="/dev/nvme0n1"

# For NVMe disks, partitions are named like nvme0n1p1, nvme0n1p2, etc.
# For SATA/SAS disks (sda, sdb), partitions are named sda1, sda2, etc.
# Set the partition separator accordingly:
export PART_SEP="p"  # Use "p" for NVMe, empty string "" for SATA/SAS

# Define partition numbers
export BOOT_PART="1"
export SWAP_PART="2"
export POOL_PART="3"

# Build full device paths
export BOOT_DEVICE="${DISK}${PART_SEP}${BOOT_PART}"
export SWAP_DEVICE="${DISK}${PART_SEP}${SWAP_PART}"
export POOL_DEVICE="${DISK}${PART_SEP}${POOL_PART}"

Verify your configuration before proceeding:

echo "Boot device: $BOOT_DEVICE"
echo "Swap device: $SWAP_DEVICE"
echo "Pool device: $POOL_DEVICE"

Wipe the Disk

Warning: This operation will irreversibly destroy all data on the selected disk. Double-check that you've selected the correct disk and be sure to have a complete backup of your system!

zpool labelclear -f "$DISK"

wipefs -a "$DISK"
sgdisk --zap-all "$DISK"

Create Partitions

EFI System Partition

If you're not using UEFI boot, adapt this procedure following the appropriate guide linked at the beginning of this post:

sgdisk -n "${BOOT_PART}:1m:+512m" -t "${BOOT_PART}:ef00" "$DISK"

Swap Partition

The swap partition should be slightly larger than your RAM to support hibernation. When you hibernate, the entire contents of RAM are written to swap, so you need enough space to hold it all plus some overhead. In this example, I have 16 GB of RAM, so I'm creating an 18 GB swap partition:

sgdisk -n "${SWAP_PART}:0:+18g" -t "${SWAP_PART}:8200" "$DISK"

ZFS Pool Partition

sgdisk -n "${POOL_PART}:0:-10m" -t "${POOL_PART}:bf00" "$DISK"

Set Up ZFS Encryption

Encrypting the disk is strongly recommended, especially for laptops. Replace SomeKeyphrase with a strong passphrase that's easy to type. Keep in mind that during early boot, the keyboard layout might default to US, so choose a passphrase that's easy to type on a US keyboard layout:

echo 'SomeKeyphrase' > /etc/zfs/zroot.key
chmod 000 /etc/zfs/zroot.key

Create the ZFS Pool

Create the pool with conservative, well-tested options:

zpool create -f -o ashift=12 \
 -O compression=lz4 \
 -O acltype=posixacl \
 -O xattr=sa \
 -O relatime=on \
 -O encryption=aes-256-gcm \
 -O keylocation=file:///etc/zfs/zroot.key \
 -O keyformat=passphrase \
 -o autotrim=on \
 -o compatibility=openzfs-2.2-linux \
 -m none zroot "$POOL_DEVICE"

Create ZFS Datasets

zfs create -o mountpoint=none zroot/ROOT
zfs create -o mountpoint=/ -o canmount=noauto zroot/ROOT/${ID}
zfs create -o mountpoint=/home zroot/home

zpool set bootfs=zroot/ROOT/${ID} zroot

Export and Reimport for Installation

zpool export zroot
zpool import -N -R /mnt zroot
zfs load-key -L prompt zroot

zfs mount zroot/ROOT/${ID}
zfs mount zroot/home

udevadm trigger

Install the Base System

XBPS_ARCH=x86_64 xbps-install \
  -S -R https://mirrors.servercentral.com/voidlinux/current \
  -r /mnt base-system

Copy Host Configuration

Copy the files we generated earlier to the new system:

cp /etc/hostid /mnt/etc
mkdir -p /mnt/etc/zfs
cp /etc/zfs/zroot.key /mnt/etc/zfs

Configure Encrypted Swap

Now we'll set up the encrypted swap partition. This is where the hibernation magic happens - by using a separate LUKS-encrypted partition instead of a ZFS zvol, we can properly resume from hibernation.

Format the swap partition with LUKS:

cryptsetup luksFormat --type luks1 "$SWAP_DEVICE"

Open the encrypted partition, create the swap filesystem, and activate it:

cryptsetup luksOpen "$SWAP_DEVICE" cryptswap
mkswap /dev/mapper/cryptswap
swapon /dev/mapper/cryptswap

Preserve Variables for Chroot

Before entering the chroot, save the disk variables so they remain available inside the new environment:

cat << EOF > /mnt/root/disk-vars.sh
export DISK="$DISK"
export PART_SEP="$PART_SEP"
export BOOT_PART="$BOOT_PART"
export SWAP_PART="$SWAP_PART"
export POOL_PART="$POOL_PART"
export BOOT_DEVICE="$BOOT_DEVICE"
export SWAP_DEVICE="$SWAP_DEVICE"
export POOL_DEVICE="$POOL_DEVICE"
export ID="$ID"
EOF

Enter the Chroot Environment

xchroot /mnt

From this point forward, all commands are executed inside the new system.

First, load the saved variables:

source /root/disk-vars.sh

Configure fstab

Add the swap entry to /etc/fstab:

/dev/mapper/cryptswap   none            swap            defaults        0 0

Set Up Automatic Swap Unlock

To avoid entering the swap password separately after unlocking the ZFS pool, we'll create a keyfile stored on the encrypted ZFS dataset. This is secure because the keyfile only becomes accessible after the ZFS pool is unlocked.

First, install cryptsetup in the new system:

xbps-install -S cryptsetup

Generate a random keyfile and add it to the LUKS partition:

dd bs=1 count=64 if=/dev/urandom of=/boot/volume.key

cryptsetup luksAddKey "$SWAP_DEVICE" /boot/volume.key

chmod 000 /boot/volume.key
chmod -R g-rwx,o-rwx /boot

Add the keyfile to /etc/crypttab:

echo "cryptswap   $SWAP_DEVICE   /boot/volume.key   luks" >> /etc/crypttab

Include the keyfile and crypttab in the initramfs. Create /etc/dracut.conf.d/10-crypt.conf:

install_items+=" /boot/volume.key /etc/crypttab "

Basic System Configuration

Configure keyboard layout and hardware clock. Adjust the keymap and timezone to match your location:

cat << EOF >> /etc/rc.conf
KEYMAP="us"
HARDWARECLOCK="UTC"
EOF

ln -sf /usr/share/zoneinfo/Europe/Rome /etc/localtime

Configure locales:

cat << EOF >> /etc/default/libc-locales
en_US.UTF-8 UTF-8
en_US ISO-8859-1
EOF

echo "LANG=en_US.UTF-8" > /etc/locale.conf

xbps-reconfigure -f glibc-locales

Set the root password:

passwd

Configure ZFS Boot Support

cat << EOF > /etc/dracut.conf.d/zol.conf
nofsck="yes"
add_dracutmodules+=" zfs "
omit_dracutmodules+=" btrfs "
install_items+=" /etc/zfs/zroot.key "
EOF

Install ZFS:

xbps-install -S zfs

Configure ZFSBootMenu

Set the basic boot properties:

zfs set org.zfsbootmenu:commandline="quiet" zroot/ROOT
zfs set org.zfsbootmenu:keysource="zroot/ROOT/${ID}" zroot

The Critical Step: Hibernation Support

Now we need to configure hibernation resume. This is the key insight that makes this setup work: normally, the encrypted ZFS root mounts first, and then it unlocks the swap partition. But when resuming from hibernation, the kernel needs to read the hibernation image from swap before mounting the root filesystem - otherwise, the saved state would be lost.

To solve this, we tell ZFSBootMenu to unlock the swap partition early, before mounting ZFS, by specifying its LUKS UUID.

Get the UUID of your swap partition:

blkid "$SWAP_DEVICE"

You'll see output like:

/dev/...: UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" TYPE="crypto_LUKS" PARTUUID="..."

Store the UUID in a variable for the next step:

SWAP_UUID=$(blkid -s UUID -o value "$SWAP_DEVICE")
echo "Swap UUID: $SWAP_UUID"

Now set the boot parameters using the captured UUID:

zfs set org.zfsbootmenu:commandline="rd.luks.uuid=$SWAP_UUID resume=/dev/mapper/cryptswap" zroot/ROOT/${ID}

Set Up EFI Boot

Create and mount the EFI partition:

mkfs.vfat -F32 "$BOOT_DEVICE"

mkdir -p /boot/efi

Add the EFI partition to /etc/fstab using its UUID:

BOOT_UUID=$(blkid -s UUID -o value "$BOOT_DEVICE")
echo "UUID=$BOOT_UUID    /boot/efi    vfat    defaults    0 0" >> /etc/fstab

Mount it:

mount /boot/efi

Install ZFSBootMenu

xbps-install -S curl

mkdir -p /boot/efi/EFI/ZBM
curl -o /boot/efi/EFI/ZBM/VMLINUZ.EFI -L https://get.zfsbootmenu.org/efi
cp /boot/efi/EFI/ZBM/VMLINUZ.EFI /boot/efi/EFI/ZBM/VMLINUZ-BACKUP.EFI

Configure the EFI boot entries:

xbps-install -S efibootmgr

efibootmgr -c -d "$DISK" -p "$BOOT_PART" \
  -L "ZFSBootMenu (Backup)" \
  -l '\EFI\ZBM\VMLINUZ-BACKUP.EFI'

efibootmgr -c -d "$DISK" -p "$BOOT_PART" \
  -L "ZFSBootMenu" \
  -l '\EFI\ZBM\VMLINUZ.EFI'

Microcode updates

Void Linux is modular, so you may need to install additional packages for your specific hardware. For the Intel microcode, you need the non-free repo: For example:

# For Intel CPUs
xbps-install -S void-repo-nonfree 
xbps-install -S intel-ucode

# For AMD CPUs/GPUs
xbps-install -S linux-firmware-amd

After installing microcode updates, regenerate the boot images and exit:

xbps-reconfigure -fa

Desktop Installation (Optional)

If all you need is a minimal system or a server, you're done and ready to reboot. For a complete desktop environment, continue with the following steps.

Install Core Desktop Packages

xbps-install -S vim nano dbus elogind polkit xorg xorg-fonts xorg-video-drivers xorg-input-drivers dejavu-fonts-ttf terminus-font NetworkManager pipewire alsa-pipewire wireplumber xdg-user-dirs unzip gzip xz 7zip

Install KDE Plasma

xbps-install -S kde-plasma dolphin konsole firefox kdegraphics-thumbnailers ffmpegthumbs vlc ark kwrite discover kf6-purpose

Enable Services

ln -s /etc/sv/NetworkManager /etc/runit/runsvdir/default/
ln -s /etc/sv/dbus /etc/runit/runsvdir/default/
ln -s /etc/sv/udevd /etc/runit/runsvdir/default/
ln -s /etc/sv/polkitd /etc/runit/runsvdir/default/
ln -s /etc/sv/sddm /etc/runit/runsvdir/default/

Configure PipeWire Audio

mkdir -p /etc/xdg/autostart
ln -sf /usr/share/applications/pipewire.desktop /etc/xdg/autostart/

mkdir -p /etc/pipewire/pipewire.conf.d
ln -sf /usr/share/examples/wireplumber/10-wireplumber.conf /etc/pipewire/pipewire.conf.d/
ln -sf /usr/share/examples/pipewire/20-pipewire-pulse.conf /etc/pipewire/pipewire.conf.d/

mkdir -p /etc/alsa/conf.d
ln -sf /usr/share/alsa/alsa.conf.d/50-pipewire.conf /etc/alsa/conf.d
ln -sf /usr/share/alsa/alsa.conf.d/99-pipewire-default.conf /etc/alsa/conf.d

Enable Additional Repositories and Flatpak (Optional)

xbps-install -S void-repo-nonfree void-repo-multilib void-repo-multilib-nonfree

xbps-install -S flatpak
flatpak remote-add --if-not-exists flathub https://dl.flathub.org/repo/flathub.flatpakrepo

Create a Regular User and exit

For desktop use, create a non-root user with appropriate group memberships. Replace username with your desired username.

useradd -m username
passwd username
usermod username -G video,wheel,plugdev,kvm,audio,network
exit

Fix for NetworkManager

xchroot will bind mount /etc/resolv.conf and leave an empty file. Network Manager won't like it. So let's clean it up:

umount -l /mnt/etc/resolv.conf 2>/dev/null || true

rm -f /mnt/etc/resolv.conf
ln -s /run/NetworkManager/resolv.conf /mnt/etc/resolv.conf

Exit and Reboot

umount -n -R /mnt
zpool export zroot
reboot

Post-Installation

If everything went well, after entering your ZFS encryption password, you'll be greeted by the SDDM login screen.

Testing Hibernation

To verify that hibernation works correctly, you can clock the "Hibernate" button or:

loginctl hibernate

The system should power off. When you turn it back on, ZFSBootMenu will prompt for the password, unlock the swap partition, detect the hibernation image, and resume your session exactly where you left off.

If resume fails, check that: 1. The LUKS UUID in the ZFS commandline property matches your swap partition 2. The swap partition is large enough for your RAM 3. The dracut configuration includes the crypttab and keyfile

Conclusion

You now have a fully functional Void Linux system with native ZFS, full disk encryption, and working hibernation. The system is rolling, lightweight, and easy to maintain. Enjoy!

Self-hosting your Mastodon media with SeaweedFS

Stefano Marinelli — Thu, 06 Nov 2025 11:30:02 +0000

Mastodon 4.5.0 is here, and with it come some interesting changes that, in my opinion, might encourage more people to consider it for self-hosting their Fediverse community.

While it may not be as lightweight and simple as other solutions (like snac or GoToSocial or many others), I believe it remains one of the best platforms for managing a medium-sized Fediverse community, thanks in part to the direct feedback that many admins have provided to the developers.

I have previously written about how to install Mastodon in a FreeBSD jail and how to modify its character and poll limits.

One of the most critical initial decisions (which can be changed later, but with extra work) is where to store your media files. Mastodon downloads and re-processes all media it encounters from other instances for three main reasons:

Local Caching: Your users connect to your media server, reducing the load on the original instance.
Security: Re-processing media helps to remove any potential "impurities" before they reach the user's device.
Privacy: It prevents disclosing your users' IP addresses to other instances. A user will only connect to their own instance to fetch all data, including remote content.

At least initially, media files will be the largest part of your instance's storage footprint. It is therefore essential to plan where to store them and to add a regular cleanup script; otherwise, their growth will be exponential.

Mastodon supports uploading media to external S3-compatible solutions, and many admins use the usual commercial providers, paying for data uploads and transfers.

I am a firm believer in "Own Your Data", so I have always used my own self-hosted S3 servers. I initially started with Minio, but over time, I realized that, by design, it doesn't perform well with a multitude of small files (performance degrades). After running some tests, I decided to switch to SeaweedFS.

SeaweedFS "is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek..." - this, combined with the fact that it is a mature and proven piece of software, was enough for me to give it a try. The result? Excellent. The I/O and CPU load on my media server dropped drastically, making SeaweedFS an incredibly suitable solution. Furthermore, some of its features (like the ability to run a filer.sync) allow for efficient and fast replication to other storage, another host, or... anything else.

SeaweedFS works perfectly with Mastodon, and I will explain the steps to get it into production.

I will install SeaweedFS in a dedicated jail and use a dedicated subdomain. This ensures that the media server can be moved to another host at any time without reconfiguring everything or changing domains. SeaweedFS has its own FreeBSD package, installable via pkg, or can be downloaded directly from the project's website.

In either case, I will describe a "test" setup - which can also be used in production without issues. However, I highly recommend diving deeper into the tool, as it is incredibly powerful and flexible and can solve many more problems than one might imagine.

Setting up the SeaweedFS Jail

First, let's create a dedicated jail with BastilleBSD:

bastille create media 14.3-RELEASE 10.0.0.66 bastille0

Now, let's enter the jail and install SeaweedFS (and tmux, which can be useful):

bastille console media
pkg install -y tmux seaweedfs

I suggest launching SeaweedFS in a tmux session so you can monitor its output. Later, you should configure an automatic startup method, such as using the included rc.d file or any other method you prefer.

Create a directory for the data and start SeaweedFS as the "seaweedfs" user:

mkdir -p /seaweedfs/data
chown -R seaweedfs /seaweedfs
su -m seaweedfs
cd /seaweedfs/
/usr/local/bin/weed server -dir /seaweedfs/data -s3

At this point, SeaweedFS will start and create everything it needs to function, including the S3 server.

Configuring Buckets and Users

Now, let's open the weed shell to create the necessary bucket and users:

weed shell
s3.bucket.create -name mastomedia

Still in the weed shell, create a user for Mastodon and grant read permissions for unauthenticated users (which is necessary to serve media to the world):

s3.configure -access_key=mastomedia -secret_key=CHANGEME -buckets=mastomedia -user=mastodon -actions=Read,Write,List,Tagging,Admin -apply
s3.configure -buckets=mastomedia -user=anonymous -actions=Read -apply
s3.configure -buckets=mastomedia -actions=Read -apply

Security Tip: For the -secret_key, avoid using a simple password. You can generate a strong, random key directly from your shell with a command like openssl rand -base64 32.

Done. SeaweedFS is now ready to receive (and serve) media. The next step is to set up a reverse proxy to serve everything over HTTPS. My preferred approach is to configure the system as if it were external, even if the services are in adjacent jails. This might use slightly more resources, but the time and trouble it saves in the future are well worth it.

Nginx Reverse Proxy Configuration

The reverse proxy can be configured something like this:

[...]

server {
   server_name  media.mastodon.example.com;

   ignore_invalid_headers off;
   client_max_body_size 0; # Allow large file uploads without Nginx limits

   location / {
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;

      proxy_connect_timeout 300;
      proxy_http_version 1.1;
      proxy_set_header Connection "";
      chunked_transfer_encoding off;

      expires 1y;
      add_header Cache-Control public;

      add_header X-Cache-Status $upstream_cache_status;
      add_header X-Content-Type-Options nosniff;

      proxy_pass http://10.0.0.66:8333;
   }

# ... other server configurations like SSL ...

}

Mastodon Configuration

Now let's configure Mastodon. If you are running the setup wizard for the first time, here is a summary of the options:

[...]
Do you want to store uploaded files on the cloud? yes
Provider Minio
Minio endpoint URL: https://media.mastodon.example.com
Minio bucket name: mastomedia
Minio access key: mastomedia
Minio secret key: CHANGEME
Do you want to access the uploaded files from your own domain? Yes
Domain for uploaded files: media.mastodon.example.com

If Mastodon is already active, or once the setup is complete, the options in your .env.prod file should be modified to be consistent with what SeaweedFS expects:

S3_ENABLED=true
S3_PROTOCOL=https
S3_REGION=us-east-1
S3_ENDPOINT=https://media.mastodon.example.com
S3_HOSTNAME=media.mastodon.example.com
S3_BUCKET=mastomedia
AWS_ACCESS_KEY_ID=mastomedia
AWS_SECRET_ACCESS_KEY=CHANGEME
S3_FORCE_SINGLE_REQUEST=true
# remove the S3_ALIAS_HOST if it is set

IMPORTANT NOTE: If both services are in jails on the same host (i.e., SeaweedFS is on the same host as Mastodon), you should ensure that the Mastodon jail can reach the SeaweedFS jail through the reverse proxy and not via the external IP. To do this, add the following line to the /etc/hosts file of the Mastodon jail:

10.0.0.1        media.mastodon.example.com

In this example, the reverse proxy is at 10.0.0.1. If you are not using a separate reverse proxy but are exposing Nginx directly from the jail (as described in my Mastodon installation article), use the IP of the Mastodon jail itself instead (e.g., 10.0.0.42).

With this setup, Mastodon will be able to upload media to the SeaweedFS server and generate the correct links for other instances, public visitors, and users of your own instance.

Have fun with SeaweedFS!

Make Your Own Backup System – Part 2: Forging the FreeBSD Backup Stronghold

Stefano Marinelli — Tue, 29 Jul 2025 08:00:00 +0200

With the primary backup strategies and methodologies introduced, we've reached the point where we can get specific: the Backup Server configuration.

When choosing the type of backup server to use, I tend to favor specific setups: either I trust a professional backup service provider (like Colin Percival's Tarsnap), or I want full control over the disks where the backups will be hosted. In both cases, for the past twenty years, my operating system of choice for backup servers has been FreeBSD. With a few rare exceptions for clients with special requests, it covers all my needs. When I require Linux-based solutions, such as the Proxmox Backup Server, I create a VM and manage it within.

I typically use both IPv4 and IPv6. For IPv4, I "play" with NAT and port forwarding. For IPv6, I tend to assign a public IPv6 address to each jail or VM, which is then filtered by the physical server's firewall. Unfortunately, every provider, server, and setup has a different approach to IPv6, making it impossible to cover them all in this article. When a provider allows for routed setups, I use this approach: Make your own VPN: FreeBSD, WireGuard, IPv6, and ad-blocking included - assigning a /72 to the bridge for the jails and VMs.

In my opinion, FreeBSD is a perfect all-rounder for backups, thanks to its ability to completely partition services. You can separate backup services (or specific servers/clients) into different jails or even VMs. Furthermore, using ZFS greatly enhances both flexibility and the range of tools you can use.

The main distinction is usually between local backup servers (physically accessible, though not always attended, and in locations deemed secure) and remote ones, such as leased external servers. I personally use a combination of both. If the services I need to back up are external, in a datacenter, and need to be quickly restorable, I prefer to always have a copy on another server in a different datacenter with good outbound connectivity. This guarantees good bandwidth for restores, which isn't always available from a local connection to the outside world. However, an internal, nearby, and accessible backup server (even a Raspberry Pi or a mini PC) ensures physical access to the data. Whenever possible, I maintain both an external and an internal copy - and they are autonomous, meaning the internal copy is not a replica of the external one, but an additional, independent backup. This ensures that if a problem occurs with the external backup, it won't automatically propagate to the internal one. In any case, the backup must always be in a different datacenter from the one containing the production data. When the fire at the OVH datacenter in Strasbourg caused the entire complex to shut down, many people found themselves in trouble because their backups were in the same, now unreachable, location. I had a copy with another provider, in a different datacenter and country, as well as a local copy.

Despite it being "just" a backup server, I almost always use some form of disk redundancy. If I have two disks, I set up a mirror. With three or more, I use RaidZ1 or RaidZ2. This is because, in my view, backups are nearly as important as production data. The inability to recover data from a backup means it's lost forever. And it happens often, very often, that someone contacts me to recover a file (or a database, etc.) days or weeks after its accidental loss or deletion. Usually, pulling out a file from a two-month-old backup generates a mix of disbelief, admiration, but above all, a sense of security in the person requesting it. And that is what our work should instill in the people we collaborate with.

The backup server should be hardened. If possible, it should be protected and unreachable from the outside. My best backup servers are those accessible only via VPN, capable of pulling the data on their own. If they are on a LAN, it's even better if they are completely disconnected from the Internet.

For this very reason, backups must always be encrypted. Having a backup means having full access to the data, and the backup server is the prime target for being breached or stolen if the goal is to get your hands on that data. I've seen healthcare facilities' backup servers being targeted (in a rather trivial way, to be honest) by journalists looking for health details of important figures. It is therefore critical that the backup server be as secure as possible.

Based on the type of access, I use two types of encryption:

If the server is local (especially if the ZFS pool is on external disks), I usually install FreeBSD on UFS in read-only mode, as I've described in a previous article, and encrypt the backup disks with GELI. This ensures that in the event of a "dirty" shutdown (more likely in unattended environments), I can reconnect to the host and then reactivate the ZFS pool. This approach makes it nearly impossible to retrieve even the pool's metadata if the disks are stolen, as GELI performs a full-device encryption. For example, an employee of a company I work with stole one of the secondary backup disks (which was located at a different, unmonitored company site) to steal information. He got nothing but a criminal complaint. With this approach, it's also not necessary to further encrypt the datasets, which avoids some issues (which I'll discuss later, in a future post).
If the server is remote, in a datacenter, I usually use ZFS native encryption, encrypting the main backup dataset (and BastilleBSD's, if applicable). Consequently, all child datasets containing backups will also be encrypted. In this case as well, a password will be required after a reboot to unlock those datasets, ensuring that the data cannot be extracted if control of the disks is lost.

Here is an example of how to use GELI to encrypt an entire partition and then create a ZFS pool on it (in the example, the disk is da1 - do not follow these commands blindly, or you will erase all content on the da1 device!):

# WARNING: This destroys the existing partition table on disk da1
gpart destroy -F da1

# Create a new GPT partition table
gpart create -s gpt da1

# Add a freebsd-zfs partition that spans the entire disk
# The -a 1m flag ensures proper alignment
gpart add -t freebsd-zfs -a 1m da1

# Initialize GELI encryption on the new partition (da1p1)
# We use AES-XTS with 256-bit keys and a 4k sector size
# The -b flag means "boot," prompting for the passphrase at boot time
geli init -b -l 256 -s 4096 da1p1
# You will be prompted for a passphrase: choose a strong one and save it!

# Attach the encrypted partition. A new device /dev/da1p1.eli will be created.
# You will be prompted for the passphrase you just set
geli attach da1p1

# (Optional) Check the status of the encrypted device
geli status da1p1

# Create the ZFS pool "bckpool" on the encrypted device
# We enable zstd compression (an excellent compromise) and disable atime
zpool create -O compression=zstd -O atime=off bckpool da1p1.eli

In this setup, the reference pool for everything related to backups will be bckpool - and you'll need to keep this in mind for the next steps. Additionally, after every server reboot, you'll need to "unlock" the disk and import the pool:

# Enter the passphrase when prompted
geli attach da1p1

# Import the ZFS pool, which is now visible
zpool import bckpool

With this method, it's not necessary to encrypt the ZFS datasets, as the underlying disk (or, more precisely, the partition containing the ZFS pool) is already encrypted.

If, instead, you choose to encrypt the ZFS dataset (for example, if you install FreeBSD on the same disks that will hold the data and don't want to use a multi-partition approach), you should create a base encrypted dataset. Inside it, you can create the various backup datasets, VMs, and the BastilleBSD mountpoint. Due to property inheritance, they will all be encrypted as well.

To create an encrypted dataset, a command like this will suffice:

# Creates a new dataset with encryption enabled.
# keylocation=prompt will ask for a passphrase every time it's mounted.
# keyformat=passphrase specifies the key type.
zfs create -o encryption=on -o keylocation=prompt -o keyformat=passphrase zfspool/dataset

In this case, after every reboot, you will need to load the key and mount the dataset:

zfs load-key zfspool/dataset
zfs mount zfspool/dataset

Keep in mind the setup you choose, as many of the subsequent choices and commands will depend on it.

Base System Setup

I'll install BastilleBSD - a useful tool for separating services into jails. It will be helpful for isolating our backup services:

pkg install -y bastille

If you used ZFS for the root filesystem, you can proceed directly with the setup. Otherwise (i.e., ZFS on other disks), you'll need to edit the /usr/local/etc/bastille/bastille.conf file and specify the correct dataset on which to install the jails. Then run:

bastille setup

Once the automatic setup is complete, check the /etc/pf.conf file - it will be automatically configured to only accept SSH connections. Ensure the network interface is set correctly. When you activate pf, you will be kicked out of the server, but you can then reconnect.

service pf start

Let's bootstrap a FreeBSD release for the jails - this will be useful later.

bastille bootstrap 14.3-RELEASE update

Now, we create a local bridge. Jails and VMs can be attached to it, making them fully autonomous. Using VNET jails, for example, will allow the creation of VPNs or tun interfaces inside them, simplifying potential future setups (and increasing security by using a dedicated network stack).

Modify the /etc/rc.conf file and add:

# Add lo1 and bridge0 to the list of cloned interfaces
cloned_interfaces="lo1 bridge0"
# Assign an IP address and netmask to the bridge
ifconfig_bridge0="inet 192.168.0.1 netmask 255.255.255.0"
# Enable gateway functionality for routing
gateway_enable="yes"

Let's also modify /etc/pf.conf to allow the 192.168.0.0/24 subnet to access the Internet via NAT. We will skip packet filtering on bridge0 and enable NAT. This isn't the most secure setup, but it's sufficient to get started:

#...
# Skip PF processing on the internal bridge interface
set skip on bridge0
#...
# NAT traffic from our internal network to the outside world
nat on $ext_if from 192.168.0.0/24 to any -> ($ext_if:0)
#...

To ensure the new settings are correct, it's a good idea to test with a reboot.

Since I often configure vm-bhyve in my setups, I prefer to install it right away, creating the dataset that will contain the VMs and installation templates. Remember that zroot is only valid if you installed the entire system on ZFS; otherwise, you'll need to change it to your own dataset:

# Install required packages
pkg install vm-bhyve grub2-bhyve bhyve-firmware
# Create a dataset to store VMs
zfs create zroot/VMs
# Enable the vm service at boot
sysrc vm_enable="YES"
# Set the directory for VMs, using the ZFS dataset
sysrc vm_dir="zfs:zroot/VMs"
# Initialize vm-bhyve
vm init
# Copy the example templates
cp /usr/local/share/examples/vm-bhyve/* /zroot/VMs/.templates/

At this point, I usually enable the console via tmux. This means that when a VM is launched, it won't open a VNC port by default, but a tmux session connected to the VM's serial port. Let's install and configure tmux:

pkg install -y tmux
vm set console=tmux

Let's also attach the switch we created (bridge0) to vm-bhyve so we can use it:

vm switch create -t manual -b bridge0 public

Now, vm-bhyve is ready.

The basic infrastructure is complete. We now have:

ZFS to ensure data integrity, which will also handle redundancy, etc.
BastilleBSD to manage jails, useful for backing up Linux, NetBSD, OpenBSD, and non-ZFS FreeBSD machines.
vm-bhyve to install specific systems (like Proxmox Backup Server).

Backup Strategies

I use various backup tools, too many to list in this article. So I'll make a broad distinction, describing how to use this server to achieve our goal: securing data.

For FreeBSD servers with ZFS (hosts, VMs, jails, hypervisors, and their respective VMs), I use an extremely useful, efficient, and reliable tool: zfs-autobackup.
For Linux servers (without ZFS), NetBSD, OpenBSD, etc. (any non-ZFS OS), I usually use BorgBackup. There are other fantastic tools like restic, Kopia, etc., but BorgBackup has never let me down and has served me well even on low-power devices and after incredibly complex disasters.
For Proxmox servers (a solution I've used with satisfaction in production since 2013, although I'm recently migrating to FreeBSD/bhyve where possible), I use two possible alternatives (often both at the same time): if the storage is ZFS, I use the zfs-autobackup approach. In either case, the most practical solution is the Proxmox Backup Server. And the Proxmox Backup Server is one of the reasons I proposed installing vm-bhyve: running it in a VM and storing the data on the FreeBSD host gives you the best of both worlds. Some time ago, I tried running it in a FreeBSD jail (via Linuxulator), but it didn't work.

Backups using zfs-autobackup

zfs-autobackup is an extremely useful and effective tool. It allows for "pull" type backups, as well as having an intermediary host that connects to both the source and destination, which is useful if you don't want direct contact between the source and destination. I won't describe the latter setup, but the documentation is clear, and I have several of them in production, ensuring that the production server and its backup server cannot communicate with each other.

I usually create a dataset for each server and instruct zfs-autobackup to keep that server's backups in that dataset. The snapshots taken and transferred will all be from the same instant, so as not to create a time skew (some tools of this kind snapshot a dataset, then transfer it, which can result in minutes of difference between two different datasets from the same server).

I've described in detail how I perform this type of backup in a previous post, so I suggest reading that post for reference.

Let's install zfs-autobackup on the FreeBSD server:

pkg install py311-zfs-autobackup mbuffer

Backups for other servers using BorgBackup

When I don't have ZFS available or need to perform a file-based backup (all or partial), I use a different technique. BorgBackup backups are primarily "push" based, meaning the client will connect to the backup server. This is not optimal or the most secure approach, as the backup server should, in theory, be hardened. Even when protecting everything via VPN, the risk remains that a compromised server could connect to its backup server and alter or delete the backups. I have seen this happen in ransomware cases (especially in the Microsoft world), and so I try to be careful to minimize this type of problem, mainly through snapshots of the backup server (an operation that will be described later).

To ensure the highest possible security, I create a FreeBSD jail on the backup server for each server I need to back up. The advantage of this approach is the complete separation of all servers from each other. By using a regular user inside a jail, a compromised server that connects to its backup server would only be able to reach its own backups, as it would be confined to a user account and, even if it managed to escalate privileges, still be inside a jail.

Let's say, for example, we want to back up a server called "ServerA" (great imagination, I know). We create a dedicated jail on the backup server:

# Create a new VNET jail named "servera" attached to our bridge
bastille create -B servera 14.3-RELEASE 192.168.0.101/24 bridge0

BastilleBSD will automatically set the host's gateway for the jail. In our case, this is incorrect, so we need to modify it and set the jail's gateway to 192.168.0.1 in the /usr/local/bastille/jails/servera/root/etc/rc.conf file:

# ...
defaultrouter="192.168.0.1"
# ...

Restart the jail and connect to it:

bastille restart servera
bastille console servera

Now, inside the jail, we install borgbackup:

pkg install py311-borgbackup

BorgBackup doesn't run a daemon; it's launched by the remote server (which sends its data to the backup server), so it's important that the installed version is compatible with the one on the remote host.

Since we'll be using SSH, let's enable it:

service sshd enable
service sshd start

And create a non-privileged user for this purpose:

# The 'adduser' utility provides an interactive way to create a user.
root@servera:~ # adduser
Username: servera
Full name: Server A
Uid (Leave empty for default): 
Login group [servera]: 
Login group is servera. Invite servera into other groups? []: 
Login class [default]: 
Shell (sh csh tcsh nologin) [sh]: 
Home directory [/home/servera]: 
Home directory permissions (Leave empty for default): 
Use password-based authentication? [yes]: 
Use an empty password? (yes/no) [no]: 
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]: 
Username    : servera
Password    : <random>
Full Name   : Server A
Uid         : 1001
Class       : 
Groups      : servera 
Home        : /home/servera
Home Mode   : 
Shell       : /bin/sh
Locked      : no
OK? (yes/no) [yes]: yes
adduser: INFO: Successfully added (servera) to the user database.
adduser: INFO: Password for (servera) is: JIkdq8Ex

The user is created and can receive SSH connections. After setting everything up, I suggest disabling password-based login in the jail's SSH configuration, using only public key authentication.

As mentioned, the biggest risk of a "push" backup is that a compromised client could access the backup server and delete or encrypt the backup history, rendering it useless.

To drastically mitigate this risk, we can configure SSH to force the client to operate in a special Borg mode called append-only. In this mode, the SSH key used by the client will only have permission to create new archives, not to read or delete old ones. However, this approach could complicate some client-side operations (like mount, prune, etc.), forcing them to be done on the server. For this reason, I won't describe it in this setup, "limiting" myself to taking snapshots of the repositories. It can be a very good practice, so I recommend considering it.

Let's initialize the BorgBackup repository. In this example, for simplicity, I won't set up repository encryption. If the jails are on an encrypted dataset or GELI-encrypted disks, there will still be data encryption on the disks, but there will be no protection against someone who could physically access the server while the disks are mounted. As usual, security is like an onion: every layer helps. Personally, I suggest enabling and using it ALWAYS.

# Switch to the new user
su -l servera
# Initialize a new Borg repo named "servera" with no encryption (for this example)
borg init -e none servera

The jail is ready, but it's unreachable from the outside. There are two ways to make it accessible:

Install a VPN system inside the jail itself. Using tools like Zerotier or Tailscale (which don't need to expose ports) will immediately create the conditions to connect to the jail, which will remain inaccessible from the outside. As the jail is a VNET jail, we're free to choose any of the supported VPN technologies.
Expose a port on the backup server, i.e., on the host, to allow external connections. Many advise against this path as they consider it less secure. It is, but sometimes we don't have the luxury of installing whatever we want on the server we're backing up.

To expose the port, go back to the host and modify the /etc/pf.conf file, creating the rdr and pass rules to let packets in:

# ...
# Redirect incoming traffic on port 1122 to the jail's SSH port (22)
rdr on $ext_if inet proto tcp from any to any port = 1122 -> 192.168.0.101 port 22
# ...
# Allow incoming traffic on port 1122
pass in inet proto tcp from any to any port 1122 flags S/SA keep state

Reload the pf configuration:

service pf reload

The jail will now be reachable on the server's public IP, on port 1122. Obviously, this port number is for illustrative purposes, and I used from any, but for better security, you should replace any with the IP address of the server that will be connecting to perform the backup.

By repeating this process for each server and creating a separate jail for each, you can have isolated jails in separate datasets with their backups, potentially setting space limits using ZFS quotas.

It's important to remember that backing up a live filesystem (i.e., without a snapshot or dumps) has a very high probability of being impossible to restore completely. Databases hate this approach because files will change while being copied and tend to get corrupted. Of course, it depends on the nature of the data (a backup of a static website will have no issues, but a WordPress database probably will), but it's crucial to think about a technique to snapshot the filesystem before proceeding. For example, I have already written about how to create snapshots on FreeBSD with UFS in a previous article: FreeBSD tips and tricks: creating snapshots with UFS.

I will cover other operating systems in a future, dedicated post.

Proxmox Backup Server in a Dedicated VM

Starting with version 4.0 (which is still in beta at the time of this writing), Proxmox Backup Server (PBS) supports storing its data in an S3 bucket. This is excellent news as it decouples the server from the data. There are great open-source S3 implementations, like Minio or SeaweedFS, which allow for clustering, replication, etc. In this "simple" case, we will install Proxmox Backup Server in a small VM, while for the data, we'll install Minio in a native FreeBSD jail. The advantage is undeniable: the VM will only serve as an "intermediary", but the data will rest directly on the FreeBSD host's dataset, natively. It will also be possible to specify other external endpoints, other repositories, etc.

As a philosophy, I tend not to use external providers unless for specific needs, so installing Minio in a jail is a perfect solution to manage this situation.

Let's install PBS by downloading the ISO from their website (https://enterprise.proxmox.com/iso/) - at this moment, the version that supports this setup is 4.0 Beta.

The directory to download to is the vm-bhyve ISOs directory. It's not strictly necessary, but it's useful for not "losing" it somewhere. So, go to the directory and download it:

cd /zroot/VMs/.iso
fetch https://enterprise.proxmox.com/iso/proxmox-backup-server_4.0-BETA-1.iso

Now let's create a VM with vm-bhyve. We can start from the Debian template, but we'll make some modifications to optimize performance. In this example, I'm giving it 30 GB of disk space, 2 GB of RAM, and 2 cores.

If you want to store all backups inside the VM, you'll need to size the virtual disk correctly (or create and attach another one). In this case, I will focus on the "clean" VM that will store its data on a dedicated jail with Minio.

vm create -t debian -s 30G -m 2G -c 2 pbs

Once the empty VM is created, let's modify its options:

vm configure pbs

We will modify the VM to be UEFI and to use the NVME disk driver - bhyve performs significantly better on NVME than virtio, as previously tested:

loader="uefi"
cpu="2"
memory="2G"
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"

Fortunately, the Proxmox team has provided for the installation of the Backup Server on devices without a graphical interface, so the boot menu will allow installation via serial console. Let's launch the installation and connect to the virtual serial console:

cd /zroot/VMs/.iso
vm install pbs proxmox-backup-server_4.0-BETA-1.iso
vm console pbs

Select the installation via Terminal UI (serial console) and proceed normally as if it were a physical host, assigning an IPv4 address from the 192.168.0.x range (in this example, I'll use 192.168.0.3).

This way, the Proxmox Backup Server will run in a VM, with the ability to take snapshots before updates, etc., without any worries.

Once the installation is complete, PBS will reboot and listen on port 8007 of its IP. Again, as with the jails, we have two options: install a VPN system within the VM itself (thus exposing it automatically only on that VPN - generally a more secure operation) or expose port 8007 on the server's public IP.

In the latter case, add the relevant lines to the /etc/pf.conf file on the FreeBSD backup server:

# ...
# Redirect incoming traffic on port 8007 to the PBS VM's web interface
rdr on $ext_if inet proto tcp from any to any port = 8007 -> 192.168.0.3 port 8007
# ...
# Allow that traffic to pass
pass in inet proto tcp from any to any port 8007 flags S/SA keep state

Reload the pf configuration:

service pf reload

The PBS VM configuration is complete. If you chose to use the PBS's internal disk as a repository, no further operations are necessary (other than the normal repository creation, etc., within PBS).

In this case, however, we will use a different approach.

Creating a Minio Jail as a Data Repository for PBS

This approach, in my opinion, has a number of important advantages. The first is that Minio will run in a dedicated jail on the host, at full performance, and will store the data directly on the physical ZFS datapool, thus removing any other layer in between. This jail could potentially be moved to other hosts (by connecting PBS and the jail via VPN or public IP), made redundant thanks to all of Minio's features, etc. Another solution I am successfully testing (in other setups) is SeaweedFS.

Let's create a dedicated jail with Minio and put it on the bridge, so that PBS can access it on the LAN.

bastille create -B minio 14.3-RELEASE 192.168.0.11/24 bridge0

If not configured directly, BastilleBSD will use the host's gateway for the jail, which is incorrect in this case. So let's go modify it and restart the jail. Enter the jail with:

bastille console minio

And modify the /etc/rc.conf file to have the correct gateway (following the example addresses):

# ...
ifconfig_vnet0=" inet 192.168.0.11/24 "
defaultrouter="192.168.0.1"
# ...

Exit the jail and restart it:

bastille restart minio

Enter the jail and install Minio:

bastille console minio
pkg install -y minio

Minio is already able to start, but PBS, even on the LAN, wants an encrypted connection. Fortunately, there's a handy tool that can generate the certificates for us:

# Download the certgen tool
fetch https://github.com/minio/certgen/releases/latest/download/certgen-freebsd-amd64

# Make it executable and run it for our jail's IP
chmod a+rx certgen-freebsd-amd64
./certgen-freebsd-amd64  -host "192.168.0.11"

# Create the necessary directories and set permissions
mkdir -p /usr/local/etc/minio/certs
cp private.key public.crt /usr/local/etc/minio/certs/
chown -R minio:minio /usr/local/etc/minio/certs/

Let's view the certificate's fingerprint. Since it's self-signed, we'll need it for PBS later. For security reasons, PBS will ask for the fingerprint of non-directly verifiable certificates. Run the following command and take note of the result:

openssl x509 -in /usr/local/etc/minio/certs/public.crt -noout -fingerprint -sha256

At this point, enable and configure Minio in /etc/rc.conf. WARNING: The username and password (access key and secret) used in this example are insecure and for testing purposes only. It is strongly recommended to use different values:

# Enable Minio service
minio_enable="YES"
# Set the address for the Minio console
minio_console_address=":8751"
# Set the root user and password as environment variables
minio_env="MINIO_ROOT_USER=testaccess MINIO_ROOT_PASSWORD=testsecret"

Start Minio:

service minio start

If everything went correctly, Minio is now running (with its certificates) and ready to receive connections.

It's now time to create the bucket(s) that PBS will use. There are several ways to do this, but to test that everything is working and to configure PBS, I suggest connecting via an SSH tunnel.

# Create an SSH tunnel from your local machine to the backup server
# Port 8007 is forwarded to the PBS web UI
# Port 8751 is forwarded to the Minio console
ssh user@backupServerIP -L8007:192.168.0.3:8007 -L8751:192.168.0.11:8751

This way, we'll create a tunnel between the FreeBSD backup server and our workstation, mapping 127.0.0.1:8007 to 192.168.0.3:8007 (the PBS web interface) and 127.0.0.1:8751 to 192.168.0.11:8751 (the Minio console port).

Now, connect to https://127.0.0.1:8751, enter the credentials specified in /etc/rc.conf, and create a bucket.

Once the bucket is created, you can configure PBS to use it. Connect to PBS via https://127.0.0.1:8007 and go to S3 Endpoints. Set a name, use 192.168.0.11 as the IP and 9000 as the port, enter the access and secret keys, and the certificate fingerprint we generated earlier. Enable "Path Style" or it will not work.

Then go to Datastores and add it, as you would for any other S3 datastore, by specifying the created bucket and a local directory where the system will keep its cache.

If everything was set up correctly, PBS will create its structure in the bucket, and from that moment on, you can use it. Always keep in mind that this is still a "technology preview", so there may be issues, but from my tests, it is sufficiently reliable.

Taking Local Snapshots of Backups

One of the most common techniques used in ransomware attacks is to also delete or encrypt backups. They often use automated methods, relying on the fact that many (too many!) consider a "backup" to be a simple copy of files to a network share. However, it's not impossible that, in specific cases, they might compromise the machine and connect to the backup server. This is nearly impossible with a "pull" type backup (like the one managed by zfs-autobackup) but is still possible with the "push" approach, which involves using BorgBackup or similar tools.

This happened to one of my clients once - in that case, the problem originated internally, from an employee who wanted to cover up his mistake, inadvertently creating a disaster - but that will be material for another post.

Fortunately, the client had a system that solved the problem: thanks to ZFS, we can have local snapshots on the backup server, which are invisible and inaccessible to the production server. Since we have already installed zfs-autobackup, it's easy to use it for this purpose as well. I've already talked about this in a previous article and won't rewrite the steps here. Just consult that article, keeping in mind that in this case, it's not advisable to snapshot all the datasets on the backup server (the space would grow exponentially), but only those at risk. In the cases analyzed in this post, this applies only to the push part, as PBS will also be accessible only from the Proxmox servers and not from the VMs they contain. If, in this case too, you don't trust those who manage the Proxmox servers, just set up snapshots for the Minio jail as well.

Conclusion

This long post aims to analyze, in a general way, how I believe one can manage reasonably secure backups of their data. Obviously, there are many variables, additional precautions, possible optimizations, hardening, etc., that must be studied on a case-by-case basis. There are old rules, new rules, old and new philosophies. Recently, many people who have embraced the cloud have often stopped thinking about backups, only to realize it when something happens and the data has, indeed, vanished... into the clouds.

In this post, I have generically covered the setup of the backup server, and this demonstrates how FreeBSD, thanks to its features, can be considered an ideal platform for this type of task.

In the next articles in this series, I will examine the client side, i.e., how to structure them for a sufficiently reliable backup, and how to monitor everything - because I've seen this too: people resting easy because the backup was supposedly running every night, but in fact, the backup had been failing every night for more than 4 years.

Stay Tuned and stay...backupped!

New Article on BSD Cafe Journal: WordPress on FreeBSD with BastilleBSD

Stefano Marinelli — Mon, 21 Jul 2025 09:30:00 +0200

New Article Published

I'm excited to announce that I have published a new, in-depth article on the BSD Cafe Journal: "WordPress on FreeBSD with BastilleBSD: A Secure Alternative to Linux/Docker".

This piece explores how to create a robust and secure WordPress installation on FreeBSD using BastilleBSD, leveraging the power and isolation of FreeBSD jails as a compelling alternative to the more common Linux and Docker stack.

Future Technical Content

I'm excited to announce that I'm expanding my writing to a new platform! From now on, some of my more technical, long-form articles and tutorials will be published on The BSD Cafe Journal, a fantastic hub for BSD-related content that I'm happy to now contribute to.

This new collaboration complements the work I do here. My personal blog will continue to be my home base, and you won't miss a thing! I'll still be posting my own articles and announcements right here, and I'll always include a direct link to any new content I publish elsewhere. This space will remain as active as ever.

Thank you for reading

Make Your Own Backup System – Part 1: Strategy Before Scripts

Stefano Marinelli — Fri, 18 Jul 2025 09:00:00 +0200

Backup: Beyond the Simple Copy

For as long as I can remember, backup is something that has been underestimated by far too many people. Between flawed techniques, "Schrödinger's backups" (i.e., never tested, thus both valid and invalid at the same time), and conceptual errors about what they are and how they work (RAID is not a backup!), too much data has been lost due to deficiencies in this area.

Nowadays, backup is often an afterthought. Many rely entirely on "the cloud" without ever asking how - or if - their data is actually protected. It's a detail many overlook, but even major cloud providers operate on a shared responsibility model. Their terms often clarify that while they secure the infrastructure, the ultimate responsibility for protecting and backing up your data lies with you. By putting everything "in the cloud", on clusters owned by other companies, or on distributed Kubernetes systems, backup seems unnecessary. When I sometimes ask developers or colleagues how they handle backups for all this, they look at me as if I'm speaking an archaic, unknown, and indecipherable language. The thought has simply never crossed their minds. But data is not ephemeral; it must be preserved in every way possible.

I've always had a philosophy: data must always be restorable (and as quickly as possible), in an open format (meaning you shouldn't have to buy anything to restore or consult it), and consistent. These points may seem obvious, but they are not.

I have encountered various scenarios of data loss:

Datacenters destroyed by fire – I had 142 servers there, but they were all restored in just a few hours.
Server rooms flooded.
Servers destroyed in earthquakes, often due to collapsing walls.
Increasing incidents of various ransomware attacks.
Intentional damage by entities seeking to create problems.
Mistakes made by administrators, which can happen to anyone.

The risk escalates for servers connected to the internet, like e-commerce and email servers. Here, not only is data integrity crucial, but so is the uninterrupted operation of services. This series of posts will revisit some of my old articles to explain my core ideas on the subject and describe, at least in part, my primary techniques.

Many consider a backup to be a simple copy. I often hear people say they have backups because they "copy the data", but this is often wrong and extremely dangerous, providing a false sense of security. Copying the files of a live database is an almost useless operation, as the result will nearly always be impossible to restore. It is essential to at least perform a proper dump and then transfer that file. Yet, many people do this and will only realize their mistake when they face an emergency and need to restore.

The Backup Plan: Asking the Right Questions

Before touching a single file, you must start with a plan, and that plan starts with asking the right questions:

"How much risk am I willing to take? What data do I need to protect? What downtime can I tolerate in case of data loss? What type and amount of storage space do I have available?"

The first question is particularly critical. A common but risky approach is to store a backup on the same machine that requires backing up. While convenient, this method fails in the event of a machine failure. Even relying on a classic USB drive for daily backups is not foolproof, as these devices are as susceptible to failure as any other hardware component. And contrary to popular belief, even high-end uninterruptible power supplies (UPS) are not immune to catastrophic failures.

Thus, the initial step is to establish a management plan, balancing security and cost. The safest backup is often the one stored farthest from the source machine. However, this approach introduces challenges related to space and bandwidth. While local area network (LAN) backups are relatively straightforward, off-network backups involve additional connectivity considerations. This might lead to a compromise on the amount of data backed up to maintain operational speed during both backup and recovery processes.

Safety doesn't always equate to practicality. For instance, with a 200 MBit/sec connection and 2 TB of backup data, the recovery time could be significant. However, if the goal is not rapid restorability but simply ensuring the data is available, the safest backup is often the one closest to us. That is, a backup we can "touch", disconnect, and consult even when offline.

Therefore, it is essential to develop a backup policy tailored to specific needs, keeping in mind that no 'perfect' solution exists.

The Core Decision: Full Disk vs. Individual Files

When planning a backup strategy, one key decision is whether to back up the entire disk or just individual files. Or both of them. Each approach has its pros and cons.

Entire Disk (or Storage) Backup

Advantages:

Complete Recovery: Restoring a full disk backup can quickly revert a system to its exact previous state, boot loader included.
Integration in Virtualization Systems: If your VM is a single file on a filesystem like ZFS or btrfs, you can simply copy that file (after taking a snapshot) to get a complete backup of the VM. Solutions like Proxmox offer easy management of full disk backups, accessible via command line or web interface.
Flexibility in Virtual Environments: Products like the Proxmox Backup Server offer the ability to recover individual files from a full backup, combining the benefits of both approaches.

Disadvantages:

Downtime for Physical Machines: Often, it's necessary to shut down the machine to create a full disk backup, leading to operational interruptions. A hybrid approach, if the physical host is running FreeBSD for example, is to take a snapshot and copy all the host's datasets. The restore process, however, will require some manual operations.
Large Space Requirements: Full disk backups can consume substantial space, including unnecessary data.
Potential Slowdowns and Compatibility Issues: The backup process can be slow and may encounter issues with non-standard file system configurations.

Individual File Backup

While it might seem simpler, backing up individual files can get complicated.

Advantages:

Basic Utility Use: Possible with standard system utilities like tar, cp, rsync, etc.
Granular Backups: Allows for backing up specific files and comparing them to previous versions.
Delta Copying: Only modified parts of the files are backed up, saving space and reducing data transfer.
Portability and Partial Recovery: Files can be moved individually and partially restored as needed.
Compression and Deduplication: These features are often available at the file or block level.
Operational Continuity: Can be done without shutting down the machine.

Disadvantages:

Storage Space Requirements: Simple copy solutions might require significant storage.
Need for File System Snapshot: For efficient and consistent backups, a snapshot (like native ZFS snapshots, btrfs, LVM Volume snapshots, or Microsoft's VSS) is highly recommended before copying.
Hidden Pitfalls: Issues may not become apparent until a backup is needed. And by then, it may be too late.

The Key to Consistency: The Power of Snapshots

Backing up a "live" file system involves a "start" and an "end" moment. During this time, the data can change, leading to fatal inconsistencies. I've encountered such issues in the past: a large MySQL database was compromised, and I was tasked with its recovery. I confidently took the client's last file-based backup and restored various files (not a native dump). Unsurprisingly, the database failed to restart. The large data file had changed too much between the start and end of the copy, rendering it inconsistent. Fortunately, I also had a proper dump, so I managed to recover from that.

The issue is evident: backing up a live file system is risky. An open database, even a basic one like a browser's history, is highly likely to get corrupted, making the backup useless.

The solution is to create a snapshot of the entire file system before beginning the backup. This approach freezes a consistent "point-in-time" view of the data. To date, using snapshots, I have managed to recover everything.

Here are the methods I've explored over the years:

Native File System Snapshot (e.g., BTRFS or ZFS): If your file system inherently supports snapshots, it's wise to use this feature. It's likely to be the most efficient and technically sound option.
LVM Snapshot: For those using LVM, creating a snapshot of the logical volume is a viable approach. This method can lead to some space wastage and, while I still use it, has occasionally caused the file system to freeze during the snapshot's destruction, necessitating a reboot. This has been a rare but recurring issue across different hardware setups, especially under high I/O load.
DattoBD: I've tracked this tool since its inception. I used it extensively in the past, but I sometimes faced stability issues (kernel panics or the inability to delete snapshots, forcing a reboot). For snapshots with Datto, I often use UrBackup scripts, which are convenient and efficient.

The Architecture: Push or Pull?

A longstanding debate among experts is whether backups should be initiated by the client (push) or requested by the server (pull). In my view, it depends.

Generally, I prefer centralized backup systems on dedicated servers, maintained in highly secure environments with minimal services running. Therefore, I lean towards the "pull" method, where the server connects to the client to initiate the backup.

Ideally, the backup server should not be reachable from the outside. It should be protected, hardened, and only be able to reach the setups it needs to back up. The goal is to minimize the possibility that the backup data could be compromised or deleted in case the client machine itself is attacked (which, unfortunately, is not so rare).

This is not always possible, but there are ways to mitigate this problem. One way is to ensure that machines that must be backed up via "push" (i.e., by contacting the backup server themselves) can only access their own space. More importantly, the backup server, for security reasons, should maintain its own filesystem snapshots for a certain period. In this way, even in the worst-case scenario (workload compromised -> connection to backup server -> deletion of backups to demand a ransom), the backup server has its own snapshots. These server-side snapshots should not be accessible from the client host and should be kept long enough to ensure any compromise can be detected in time.

My Guiding Principles for a Good Backup System

Over the years, I've favored having granular control over backups, often finding the need to recover specific files or emails accidentally deleted by clients. A good backup system, in my opinion, should possess these key features:

Instant Recovery and Speed: The system should enable quick recovery and operate at a high processing speed.
External Storage: Backups must be stored externally, not on the same system being backed up. Still, local snapshots are a good idea for immediate rollbacks.
Security: I avoid using mainstream cloud storage services like Dropbox or Google Drive for primary backups. Own your data!
Efficient Space Management: This includes features like compression and deduplication.
Minimal Invasiveness: The system should require minimal additional components to function.

Conclusion: What's Next

The approach to backup is varied, and in this series, I will describe the main scenarios I usually face. I will start with the backup servers and their primary configurations, then move on to the various software and techniques I use.

But that will begin with the next post, where I'm talk about building the backup server which, of course, will be powered by FreeBSD - like all my backup servers for the last 20 years.

How to install FreeBSD on providers that don't support it with mfsBSD

Stefano Marinelli — Wed, 02 Jul 2025 09:30:00 +0200

FreeBSD is an extremely powerful operating system. The ability to isolate services in jails and, thanks to ZFS, the simplicity with which you can create snapshots (both local and remote) make it a perfect system for increasing peace of mind, especially when running many workloads.

Many providers, blinded by the success and large numbers achieved by Linux distributions, have decided to no longer support FreeBSD in their installers. While understandable (they might not have staff experienced with systems other than Linux), this can cause problems for those who want to try using something different. And yes, it makes perfect sense, even just to avoid IT monocultures, which are extremely harmful even in the medium term.

There's an extremely powerful tool that, in my opinion, deserves much more attention than it gets. The tool is called mfsBSD. Using the author's words: "This is a set of scripts that generates a bootable image (and/or ISO file), that creates a working minimal installation of FreeBSD (mfsBSD) or Linux (mfslinux). It is completely loaded into memory."

mfsBSD works intelligently: it can be launched both via UEFI and via "traditional" boot since it has both boot modes enabled. It gets an IP address via DHCP (an operation that works with most providers) and opens an SSH server (with a preset password - so it's advisable to connect immediately and change it, to prevent someone else from doing it for you).

This means that mfsBSD can be used both when you have a console available and, in many cases, without a console, since you'll just need to connect via SSH and start with the traditional installation.

To install FreeBSD using mfsBSD, you just need to follow some very simple steps: all providers, in fact, offer the ability to boot in Linux "rescue mode", generally based on a sufficiently recent version. Set your server (whether physical or VPS, it doesn't matter) in rescue mode and reboot. Once active, connect via SSH (or open a console) to the server in rescue mode.

Now it's sufficient to download the mfsBSD image (for example, from here: https://mfsbsd.vx.sk/files/images/ - in my case, I usually choose the normal image, which at the time of writing this post is https://mfsbsd.vx.sk/files/images/14/amd64/mfsbsd-14.2-RELEASE-amd64.img). At this point, it needs to be written with dd directly to the server's disk (or, if in doubt, disks). For example:

dd if=mfsbsd-14.2-RELEASE-amd64.img of=/dev/sda bs=5M conv=sync

In this case I wrote "sda", but if you had one or more NVMe drives, the correct device would be /dev/nvme0n1 for the first disk, /dev/nvme1n1 for the second, etc.

Reboot. If you have a console, you'll see mfsBSD boot up, enable the SSH server, and position itself at the login. Otherwise, ping the server's IP until it starts responding. At that point, it's sufficient to connect via SSH as the root user.

The root password is mfsroot

As the first thing, change the root password with the passwd command. This is to prevent someone from entering and compromising the machine during the installation time.

Now launch the bsdinstall command and proceed with the normal FreeBSD installation (also setting up mirrors, RAID, etc., if desired), keeping in mind that mfsBSD is running in RAM, so you can overwrite the disk it was installed on without problems.

After the reboot, you'll have your FreeBSD system installed and running.

Make Your Own Internet Presence with NetBSD and a 1 euro VPS – Part 1: Your Blog

Stefano Marinelli — Tue, 22 Apr 2025 07:30:36 +0200

Why NetBSD?

For many years, I've been using (and appreciating) NetBSD because it's stable, efficient, and reliable. The codebase has proven its reliability, running without reboots for years without issues. It supports ZFS (though differently than FreeBSD), LVM (useful for those accustomed to it on Linux), the ability to take filesystem snapshots (UFS2, making ZFS less crucial), and it's an excellent virtualization platform. Installation and updates are easy (including via sysupgrade - which I'll cover in a future article). Since it focuses on portable and optimized code (running on ancient architectures requires cleanliness and correctness), it's particularly efficient on low-power devices, like embedded systems or cheap VMs. Therefore, it's one of the best solutions for a small personal setup that can still deliver excellent results and simple management.

Indeed, the market offers very cheap VPS, often with just a single core and little RAM. But a modern single core packs power that a multi-core from just a few years ago could only dream of, and often, the I/O of these machines (a bottleneck for many services) is still decent. I personally use 1 euro per month VPS (VAT included - for those not subject to it, that's less than one euro per month!) with a public IPv4 address and (often) a /64 IPv6 block, ensuring full reachability across the entire network. I'm not providing direct links as I have no affiliations, but netcup's "piko" VPS are among the types I use most often (a 4 euro/month netcup VM handles the entire FediMeteo project), and this type of VM is ideal for our purpose because some providers (like netcup) allow you to upload your own ISO and install your preferred operating system. On VPS like these, I've installed everything - including OmniOS and SmartOS - without problems. And even such a small VPS, with an efficient operating system, can be extremely satisfying.

Why BSSG?

In this article, I'll describe how to create and publish a blog using BSSG as it exemplifies my concept of portability and minimalism. BSSG on NetBSD currently doesn't leverage parallelism provided by tools like GNU Parallel, but for small to medium-sized blogs, this won't be an issue, especially considering these small VMs only have 1 core. Obviously, you can use any Static Site Generator (SSG) (like Hugo, Nikola, 11ty, Pelican, Zola, etc.) - the important thing is to have a static site served by a simple web server.

Let's Start with the Installation

Installing NetBSD is quite straightforward and is clearly covered, complete with explanatory screenshots, in the excellent official NetBSD documentation, which I recommend using as a reference during the process, especially if it's your first time.

In my case, I made sure to use the proposed disk geometry, use the standard automatic partitioning, but enable the "log" and "noatime" options for the filesystem. Both these options will provide a huge advantage in I/O operations, especially with BSSG, as the first enables journaling and the second prevents updating file metadata on every access. BSSG is more I/O bound than CPU bound, so any optimization is beneficial.

Moving forward, I also recommend configuring the network (although installation can be done from packages on the installation ISO). For netcup, you can use DHCPv4 (even though it's a bit slow and sometimes seems to fail, the DHCP client will continue running in the background and eventually work).

For IPv6, I usually configure it manually later, so I'll describe that further down.

I also recommend enabling SSH, adding a regular user (and adding them to the wheel group so they can gain root privileges) - in this case, I'll call the user blog. Also, enable the installation of binary packages, as it will be convenient later to use pkgin to install and update all necessary packages. All these steps are described clearly and in detail in the guide, so I won't detail them here. But they are simple and logical, like all operations on BSD systems.

After installation, reboot. If everything went correctly, you should be able to log in via console or SSH using the "blog" user (or whatever you named it).

First, I suggest configuring the IPv6 address and installing the necessary packages.

For IPv6, in the case of netcup, simply add one of the assigned addresses to the interface. In NetBSD, network interface configurations are stored (similar to OpenBSD) in specific files. For the first virtio interface, the file will be /etc/ifconfig.vioif0.

You need to elevate your privileges to root, open that file with your preferred editor, and add the configuration to the file itself:

nb1euro$ su -l
nb1euro# vi /etc/ifconfig.vioif0

inet6 your-ipv6-addr/64
up

To test everything, perform a reboot and try pinging an IPv6 address (I often use ping6 google.com).

If all goes well, after a few seconds, you should see ping replies, confirming everything is configured correctly.

Regarding packages, the only two strictly necessary ones are bash and a markdown processor (by default, BSSG will use commonmark; otherwise, it can be configured to use pandoc or Markdown.pl). rsync can be useful for deployment. sudo (or doas) can be useful for elevating privileges for certain operations, at least at this stage.

nb1euro$ su -l
nb1euro# pkgin in bash cmark rsync sudo

If you're used to Linux, you can also install the "nano" editor:

nb1euro# pkgin in nano

If sudo was installed, it's now appropriate to grant users in the "wheel" group (like the regular user created during installation) the ability to elevate privileges. Edit the sudoers file (I suggest using the visudo command) and uncomment this line:

## Uncomment to allow members of group wheel to execute any command
%wheel ALL=(ALL:ALL) ALL

At this point, you can switch back to operating as the regular user, downloading and unpacking BSSG:

nb1euro$ ftp https://brew.bsd.cafe/stefano/BSSG/archive/0.15.1.tar.gz
nb1euro$ tar zxfv 0.15.1.tar.gz

Now that BSSG is ready, just initialize a directory with the structure for the new site:

nb1euro$ cd bssg
nb1euro$ ./bssg.sh init /home/blog/myblog

Everything is set to start generating your blog. I recommend reading BSSG's README.md. There are many options, themes, etc., but to get started, you just need to set the site's public URL. For example, if the site will be published as myblog.example.com - just create a file at /home/blog/myblog/config.sh.local (the path defined by the init command) and set the public URL:

SITE_URL="https://myblog.example.com"

This way, all URLs will be absolute URLs, which is necessary to ensure the correct functioning of RSS feeds, sitemaps, etc. This setting assumes HTTPS - if you just want to test the site over HTTP, simply use http and then, optionally, change it to https and regenerate the site later.

You can already create your first test post, directly from the BSSG directory:

nb1euro$ ./bssg.sh post

The system will use nano if it's installed, otherwise it will use vi. Don't worry, in the latter case, BSSG will write the procedure for exiting vi as the post's text 🙂

Once you save the post, BSSG will automatically generate the site. If everything went well, the /home/blog/myblog/output directory will contain the final result. We are therefore ready for the first deployment, which can be done in many different ways. I will cover three:

Using bozohttpd, present by default in NetBSD's base system. It can be used via inetd (launching an httpd process for each connection) or as a daemon. I'll describe the first option, showing in the final benchmarks how, even when used as a daemon, it remains a less performant solution.
Using nginx
Using Caddy

First, it's advisable to obtain a certificate to configure and use HTTPS. If you only want to test using HTTP, this part can be safely bypassed. For solutions 1 and 2, I'll use certbot, which is well-known to many users with Linux experience. Caddy, on the other hand, manages certificates automatically, so there's no need for other solutions and thus no need to install certbot.

nb1euro$ sudo pkgin in py313-certbot

To use bozohttpd, no further installation is necessary. At this point, the options diverge.

Using NetBSD's Integrated httpd

bozohttpd is integrated into NetBSD and, by default, can be launched directly via inetd. This solution, while not extremely efficient or scalable, is simple and requires few resources. It's fine if you expect only a few visits per day, but when used via inetd, the initial latency for each connection is tangible. It can still be useful for some tests or small deployments.

The /etc/inetd.conf file already contains the options to handle this situation:

#http           stream  tcp     nowait:600      _httpd  /usr/libexec/httpd      httpd /var/www
#http           stream  tcp6    nowait:600      _httpd  /usr/libexec/httpd      httpd /var/www

By uncommenting these two lines and restarting inetd (service inetd restart), the server will start responding to HTTP requests on both IPv4 and IPv6.

If you want to add HTTPS support, no problem. Just request a certificate via certbot and specify the webroot.

Run:

nb1euro$ sudo certbot-3.13 certonly

Choose option 2 - the one where you specify the webroot - enter the domain, and when prompted, provide /var/www/ as the webroot.

The certificate will be created. Then, modify the /etc/inetd.conf file to also include support for HTTPS, adding two lines similar to these (obviously, change the certificate paths):

https            stream  tcp     nowait:600      _httpd  /usr/libexec/httpd      httpd -Z /usr/pkg/etc/letsencrypt/live/myblog.example.com/fullchain.pem /usr/pkg/etc/letsencrypt/live/myblog.example.com/privkey.pem /var/www
https            stream  tcp6    nowait:600      _httpd  /usr/libexec/httpd      httpd -Z /usr/pkg/etc/letsencrypt/live/myblog.example.com/fullchain.pem /usr/pkg/etc/letsencrypt/live/myblog.example.com/privkey.pem /var/www

Warning: httpd will run with the permissions of the _httpd user, so make sure all certificates are readable by that user:

nb1euro# chown -R _httpd /usr/pkg/etc/letsencrypt/

Restart inetd, and the server will also respond over HTTPS.

To make your blog public, simply copy the files from the site's output directory to /var/www/ - this time using sudo to bypass permission issues:

nb1euro$ sudo rsync -avhHPx /home/blog/myblog/output/ /var/www/

The site will be immediately visible.

Using nginx

Nginx is fast and efficient, and the performance difference is noticeable (some benchmarks follow below). For an efficient setup ready for a high number of visits, it's advisable to use a web server suited for the purpose, just like nginx.

First, install nginx and the certbot plugin for nginx. This will simplify the installation and renewal of certificates:

nb1euro$ sudo pkgin in py313-certbot-nginx nginx

Copy the startup script to /etc/rc.d - as indicated by the post-installation message. In NetBSD, this operation must be done manually, but it's always pointed out:

nb1euro$ sudo cp /usr/pkg/share/examples/rc.d/nginx /etc/rc.d

Warning: If you previously used httpd from inetd following the previous solution, you must disable it in inetd.conf and restart inetd to free up ports 80 and 443.

Now you can create a virtual host for our new site.

nb1euro$ sudo vi /usr/pkg/etc/nginx/nginx.conf

and add, at the end of the file and before the final closing curly brace:

server {
        listen 80;
        # If you also have configured IPv6 support
        listen [::]:80;

        root /var/www;
        index index.html index.htm;

        server_name myblog.example.com;

        # If you want a long cache for media and css - be careful, this means that if you change to a new theme, it might not be visible immediately as the browser might still use the old cached one
        location ~* \.(jpg|jpeg|png|gif|ico|css|js)$ {
            expires 30d;
            add_header Cache-Control "public, no-transform";
        }

        location / {
                try_files $uri $uri/ =404;
        }
}

Now, it's time to configure the system to enable nginx. Just edit /etc/rc.conf:

nb1euro$ sudo vi /etc/rc.conf

and add:

nginx=YES

Now, you can start nginx:

nb1euro$ sudo service nginx start

Nginx will start listening on port 80. Generating and installing the certificate is very simple:

nb1euro$ sudo certbot-3.13 --nginx -d myblog.example.com

This command will request the certificate and install it, so nginx will already be configured to use it.

As with the previous method, to make your blog public, simply copy the files from the site's output directory to /var/www/ - using sudo to bypass permission issues:

nb1euro$ sudo rsync -avhHPx /home/blog/myblog/output/ /var/www/

The site will be immediately visible.

Using Caddy

Caddy is a convenient and all-in-one solution, efficient and fast. It's packaged for NetBSD and allows you to go online in a flash. I won't delve into the configuration because there are many tutorials (including the official ones), but you just need to install it and run it:

nb1euro$ sudo pkgin in caddy

Once installed, go to the directory you want to serve (e.g., /var/www or directly /home/blog/myblog/output) and run:

nb1euro$ sudo caddy file-server --domain myblog.example.com

Caddy will start, request the certificate, and begin serving your blog over HTTPS as well. To install Caddy as a service (i.e., with a configuration file, etc.), you can proceed similarly to how it's done on Linux. The NetBSD Caddy package doesn't include the rc.d script, but you can copy and paste one (into /etc/rc.d/caddy) from a thread posted on UnitedBSD.

Performance Comparison

I performed some performance tests on these solutions. Here are the results, on a single-core 1 euro/month VPS, from my home connection (which also has its own limitations):

NetBSD httpd via inetd:

Running 10s test @ https://myblog.example.com/
  4 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   213.52ms  173.10ms   1.11s    76.01%
    Req/Sec    12.92      9.19    50.00     75.91%
  371 requests in 10.10s, 1.39MB read
Requests/sec:     36.72
Transfer/sec:    140.65KB

These numbers are quite poor, linked to high latency caused by having to launch bozohttpd for each incoming connection.

NetBSD httpd as a daemon:

Running 10s test @ https://myblog.example.com/
  4 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    35.74ms    6.96ms 108.80ms   81.36%
    Req/Sec    18.29      9.45    50.00     70.88%
  676 requests in 10.10s, 2.53MB read
Requests/sec:     66.92
Transfer/sec:    256.32KB

Here the situation is decidedly better, but not exceptional. httpd isn't designed for high loads or performance.

Nginx as a daemon, 1 worker:

Running 10s test @ https://myblog.example.com/
  4 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    30.69ms    4.87ms  64.14ms   66.01%
    Req/Sec   379.39     65.94   464.00     90.91%
  15026 requests in 10.04s, 56.50MB read
Requests/sec:   1496.65
Transfer/sec:      5.63MB

Here we are on another level, showing truly solid performance. This type of result can handle significantly high loads without particular difficulty. The efficiency of both NetBSD and nginx pays off.

Caddy:

Running 10s test @ https://myblog.example.net/
  4 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    32.10ms    5.75ms  95.04ms   87.44%
    Req/Sec   362.74     64.29   434.00     91.67%
  14374 requests in 10.05s, 54.63MB read
Requests/sec:   1430.82
Transfer/sec:      5.44MB

Caddy shows results comparable to nginx, so the choice between them depends solely on the type of configuration you want to achieve and the experience each person has with the specific platforms.

Conclusion: Efficient Minimalism

We've seen how it's possible to create a personal, professional, and performant online presence with minimal investment. This solution, based on NetBSD and a 1€/month VPS, offers several advantages:

Negligible Cost: For 12€ per year, you can have a website (and more!) completely under your control.
Surprising Performance: As demonstrated by the benchmarks, excellent performance can be achieved even with limited resources (up to 1400-1500 requests/second with nginx or Caddy).
Security and Stability: NetBSD is renowned for its reliability and security, fundamental characteristics for any online service.
Total Control: Unlike free blogging platforms, you have full control over every aspect of your site.
Learning Experience: Managing a BSD system allows you to acquire valuable system administration skills.

This minimalist configuration demonstrates that you don't need to invest in expensive cloud solutions or oversized VPS to have a quality online presence. In an era where the tendency is to think "moooar powaaaar = better results", NetBSD reminds us that efficiency and good design can yield excellent results even with limited resources.

After all, you don't need a thousand-node cloud to write something worth reading.

In the upcoming articles in this series, we will explore how to expand this basic installation with other useful services and how to keep the system updated and secure over time.

Caching snac Proxied Media with Nginx

Stefano Marinelli — Sat, 08 Feb 2025 16:00:00 +0100

One of the useful yet resource-intensive features of platforms like Mastodon is that they reprocess (and store locally) all multimedia files from other instances.

This behavior is implemented for three valid reasons:

To ensure files do not contain malicious code by reprocessing them locally.
To prevent all users from all instances from overloading the original instance hosting the media by requesting it repeatedly.
To hide the individual IP addresses of users from the original instance.

While this approach has benefits, it also requires significant disk space - often many gigabytes per day.

snac, on the other hand, does not locally process or cache media by default. Instead, media URLs remain unchanged, meaning users will fetch content directly from its original source. This behavior is perfectly fine for many setups (such as mobile devices), but it may not be suitable for everyone.

I suggested a small improvement to snac’s developer, who immediately recognized the benefit for users and implemented an interesting new feature: media proxying via the instance itself.

Enabling Media Proxying in snac

To enable this feature, simply add the following line to your server.json configuration file:

"proxy_media": true

Once enabled, snac will rewrite all media URLs to pass through its own instance. This ensures that original instances will no longer see the IP addresses of individual users, as they will only see the IP of the snac instance itself.

However, this also increases the load and bandwidth consumption of the instance since it must download and forward media every time a user accesses it.

Caching Proxied Media with nginx

To optimize performance, we can configure nginx to cache these proxied files transparently. This way, if multiple users from the snac instance (or the same user at different times) request the same media file, nginx will serve it from the local cache instead of fetching it again. This setup has two key advantages:

Reduces traffic to the original instances.
Lowers the load on the snac instance, as it won’t need to download the same file repeatedly.

To achieve this, create a cache storage area in nginx by adding the following to the http section:

proxy_cache_path /var/cache/nginx/snac_media levels=1:2 keys_zone=snac_media:10m max_size=1g 
                 inactive=1d use_temp_path=off;

This defines a cache directory at /var/cache/nginx/snac_media, with 10 MB allocated for metadata and a maximum cache size of 1 GB. Cached content will be invalidated and removed after one day, ensuring frequently accessed content (like profile avatars) remains cached while rarely used files get replaced over time.

Next, add the following rule to your snac instance's virtual host configuration:

location ~ ^/.+/(x|y)/ {
    proxy_cache snac_media;
    proxy_pass http://snac-ip:8001;
    proxy_set_header Host $host;
    proxy_cache_valid 200 1d;
    proxy_cache_valid 404 1h;
    proxy_ignore_headers "Cache-Control" "Expires";
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    proxy_cache_lock on;
    add_header X-Proxy-Cache $upstream_cache_status;
}

After reloading nginx, the cache will start populating as users request media files. Unrequested content will never enter the cache, ensuring efficient storage usage without unnecessary clutter.

By combining snac’s new media proxying feature with nginx caching, we can achieve a more balanced setup—reducing load on both our instance and external ones, improving privacy, and enhancing performance.

Improving snac Performance with Nginx Proxy Cache

Stefano Marinelli — Wed, 29 Jan 2025 09:00:00 +0100

Some days ago, I migrated my personal Fediverse instance from Akkoma to snac. I appreciate snac a lot and believe it is the best solution available for many use cases.

Akkoma is an excellent tool, but I noticed that even for a small instance like mine, the database grows exponentially, and the database activity remains constant. Despite low load, my disks are continuously "flashing" - which isn't a problem in itself but clearly indicates ongoing activity. In my case, this activity seems unnecessary since it's just a single-user instance.

snac has shown excellent capabilities for managing the FediMeteo project (which I'll write about in detail soon), is lightweight, and has very few dependencies. Moreover, a dedicated snac instance handles sending updates from this blog to the Fediverse.

After successfully transferring my followers from Akkoma to snac without major issues, I started using the new instance. However, as soon as I posted a photo (approximately 4MB), something happened that I somewhat expected but in a different form. My home internet connection (upload speed: 20 Mbit/sec) became saturated, but I also noticed that new connections and smaller entities were resulting in 499 errors - meaning nginx couldn't open new connections to snac. After some investigation, I realized the reason: for every remote instance, Nginx was requesting the multimedia file from snac. Due to saturated connections (snac allows setting the maximum number of active threads), it took several seconds, leading to thread exhaustion in snac. Consequently, subsequent nginx requests resulted in 499 errors as snac could no longer allocate a thread.

To resolve this, I decided to implement direct caching using nginx. My reverse proxy (running in a different FreeBSD jail but this doesn't change the outcome) can cache multimedia files - storing them on first request and serving them directly to everyone who requests them without needing to ask snac every time. This approach is similar to what I use for the media in the BSD Cafe's Mastodon instance, where I employ Varnish to keep everything in RAM.

In snac, images and multimedia files are served from a specific path, such as: https://example.com/user/s/filename.png

The key here is the /s/ segment. By instructing nginx to cache all files containing /s/ in their URL, I can offload some of the work from snac.

I modified my nginx.conf file to include a caching setup:

# Caching configuration for snac
proxy_cache_path /var/cache/nginx/snac_cache levels=1:2 keys_zone=snac:10m max_size=1g inactive=1440m use_temp_path=off;

This creates a section for /var/cache/nginx where I define the caching parameters. It will allocate 10 MB of RAM for metadata, with a maximum cache size of 1 GB (useful if you decide to post some videos). Cached content will be considered invalid and removed after 1440 minutes (24 hours). This ensures that frequently accessed content (like profile avatars and banners) remains in the cache while less frequently accessed content can be replaced with newer content. The goal isn't to have all content cached but rather for nginx to serve files independently during peak times, preventing multiple remote instances from overwhelming snac simultaneously.

In the virtual host configuration for snac, I added this specific override for multimedia content:

# Caching rules for /s/ path
location ~ ^/.+/s/ {
    proxy_cache snac;
    proxy_pass http://snac-jail-ip:8001;
    proxy_set_header Host $host;
    proxy_cache_valid 200 1d;
    proxy_cache_valid 404 1h;
    proxy_ignore_headers "Cache-Control" "Expires";
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    proxy_cache_lock on;
    add_header X-Proxy-Cache $upstream_cache_status;
}

After restarting nginx, the multimedia files will be cached on their first access and served by nginx, leaving snac's threads free to handle everything else. This setup ensures smoother performance and prevents resource exhaustion during periods of high activity or when new content is shared.

Managing ZFS Full Pool Issues with Reserved Space

Stefano Marinelli — Thu, 28 Nov 2024 20:25:00 +0100

Yesterday morning, I received a panicked call from a developer:
"I accidentally filled up the storage, and now I can't perform any operations! My ZFS pool is full!"

I immediately reassured them because I had anticipated this kind of issue. One of the things I almost always do when managing ZFS file systems is to reserve space in a specially created dataset.

This is because ZFS, like all CoW (Copy-on-Write) file systems, can find itself unable to free up space when completely full. By using reserved space, I can always free it up and delete other data, restoring the system to normal operations.

To reserve space, simply create a dataset and assign it a reserved size. Of course, this dataset should not be used for anything else; otherwise, the entire purpose would be defeated.

To create it and reserve space, you only need two simple commands. For example:

zfs create zroot/reserved
zfs set reservation=5G zroot/reserved

This creates the dataset and assigns it 5 GB of reserved space.

Here’s the situation before the operation:

zfs list zroot
NAME    USED  AVAIL  REFER  MOUNTPOINT
zroot  3.02G   109G    96K  /zroot

And here’s the situation after:

zfs list zroot
NAME    USED  AVAIL  REFER  MOUNTPOINT
zroot  8.02G   104G    96K  /zroot

As you can see, the 5 GB are removed from the available space and marked as used, but they are actually empty.

In case of a full file system, you can delete this dataset (or reduce its size) to return to normal file system operation.

Even with this technique, I still recommend not filling ZFS pools beyond 80% of their capacity, as performance degrades significantly past that point.

Moving an entire FreeBSD installation to a new host or VM in a few easy steps

Stefano Marinelli — Mon, 16 Sep 2024 09:41:00 +0200

FreeBSD, especially when installed on ZFS, is incredibly simple to manage, back up, and move to a different system.

I often find myself having to move an entire system from one host to another, from a physical host to a VM, or vice versa, from a VM to a physical host.

By following the approach of keeping the operating system clean and running all services within jails, this operation is usually quite simple: install the operating system on the new host, perform the few necessary configurations (like pf and network interfaces), transfer the jail datasets, restart the jails on the new host, and update the DNS.

However, sometimes I need to move the entire host, including the original operating system. This article will describe this operation when the FreeBSD system uses ZFS as its root file system. This process isn't much more difficult, but there are a few details to keep in mind. Below is a breakdown of the two bootloader scenarios:

UEFI Bootloader
BIOS Bootloader (non-UEFI)

UEFI Bootloader

Start from an ISO or image of mfsbsd. Partition the disk according to the layout of the source system. If the source system is a standard FreeBSD (ZFS) installation, partition the destination disk (assumed to be da0) as follows. WARNING: The first command will destroy any existing partition table on the disk.

gpart destroy -F /dev/da0
gpart create -s gpt da0
gpart add -t efi -s 200M -l efiboot0 da0
newfs_msdos -F 32 -c 1 /dev/da0p1
mount -t msdosfs /dev/da0p1 /mnt
mkdir -p /mnt/EFI/BOOT
cp /boot/loader.efi /mnt/EFI/BOOT/BOOTX64.efi
umount /mnt

Create additional partitions for swap and ZFS:

gpart add -a 1m -t freebsd-swap -s 2g -l swap0 da0
gpart add -a 1m -t freebsd-zfs -l zfs0 da0

Create the ZFS pool:

zpool create -O compression=zstd -O atime=off zroot /dev/gpt/zfs0

On the source system, create a snapshot of all datasets:

zfs snapshot -r zroot@snap01

Important: You will need to connect to the source system using a user with privileges over the entire filesystem. For simplicity, I'll use root, but it's not recommended to leave root accessible via SSH. Limit access to private keys only. To allow root login over SSH, edit /etc/ssh/sshd_config and set PermitRootLogin yes. After moving the server, remember to disable root access.
If mbuffer is installed on the source system, it can help improve the performance of the transfer. If not, remove it from the next pipe command, which should be run from the destination system. The -s will set the block-size, the -m will set the buffer size (128MB, here):

ssh root@sourceip "zfs send -RLv zroot@snap01 | mbuffer -s 128k -m 128M" | zfs receive -F zroot

You have two options now:
If the source server was idle and no further synchronization is required, proceed to the next step.
If the transfer took a long time and services on the source server need to be stopped (e.g., databases), stop what you can, take another snapshot (e.g., zfs snapshot -r zroot@snap02), and synchronize the differences by running another incremental send-receive from the destination server:

ssh root@sourceip "zfs send -RLv -i zroot@snap01 zroot@snap02 | mbuffer -s 128k -m 128M" | zfs receive -F zroot

Set the boot filesystem:

zpool set bootfs=zroot/ROOT/default zroot

BIOS Bootloader

Start from an ISO or image of mfsbsd. Partition the disk according to the layout of the source system. If the source system is a standard FreeBSD (ZFS) installation, partition the destination disk (assumed to be da0) as follows. WARNING: The first command will destroy any existing partition table on the disk.

gpart destroy -F /dev/da0
gpart create -s gpt da0

Create the necessary partitions:

gpart add -a 1m -t freebsd-boot -s 512k -l boot da0
gpart add -a 1m -t freebsd-swap -s 2g -l swap0 da0
gpart add -a 1m -t freebsd-zfs -l zfs0 da0

Create the ZFS pool:

zpool create -O compression=zstd -O atime=off zroot /dev/gpt/zfs0

On the source system, create a snapshot of all datasets:

zfs snapshot -r zroot@snap01

Important: You will need to connect to the source system using a user with privileges over the entire filesystem. For simplicity, I'll use root, but it's not recommended to leave root accessible via SSH. Limit access to private keys only. To allow root login over SSH, edit /etc/ssh/sshd_config and set PermitRootLogin yes.
If mbuffer is installed on the source system, it can help improve the performance of the transfer. If not, remove it from the next pipe command, which should be run from the destination system:

ssh root@sourceip "zfs send -RLv zroot@snap01 | mbuffer -s 128k -m 128M" | zfs receive -F zroot

You have two options now:
If the source server was idle and no further synchronization is required, proceed to the next step.
If the transfer took a long time and services on the source server need to be stopped (e.g., databases), stop what you can, take another snapshot (e.g., zfs snapshot -r zroot@snap02), and synchronize the differences by running another incremental send-receive from the destination server:

ssh root@sourceip "zfs send -RLv -i zroot@snap01 zroot@snap02 | mbuffer -s 128k -m 128M" | zfs receive -F zroot

Install the bootloader on the destination server’s disk:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

Set the boot filesystem:

zpool set bootfs=zroot/ROOT/default zroot

At this point, everything should be ready for boot. Reboot and verify that the various mountpoints (e.g., swap, etc.) and network interface settings are correct. If everything works fine, the server will be an exact copy of the original.

Moving or duplicating an entire FreeBSD system is simple, fast, and reliable. The ability to use ZFS incremental replication is an excellent method for minimizing downtime, especially for large systems. The initial copy will happen "live" while the source system continues to run without issues. Subsequent incrementals will be much faster, making the switch-over process seamless and efficient.

Make your own Read-Only Device with NetBSD

Stefano Marinelli — Tue, 10 Sep 2024 01:41:00 +0200

One detail that is often overlooked when dealing with embedded (or remote) devices is a key point of vulnerability: the file system.

For non-COW file systems (like ext4 on Linux, FFS, etc.), there are situations where a crash or a power outage could cause corruption, requiring manual intervention. This risk is especially present when using root on SD cards or similar storage media in embedded systems. Over time, these media are destined to fail due to numerous writes.

For certain use cases, it's advisable to set up a read-only root file system, which ensures better reliability in case of system issues. Think of scenarios like a router (critical for network access) or a caching reverse-proxy, such as the one described in my series "Make your own CDN".

While FreeBSD natively supports this configuration and some Linux distributions offer custom solutions (e.g., Alpine Linux), NetBSD stands out as an excellent choice for such devices. It supports nearly all embedded devices, is lightweight, and its stability minimizes the need for frequent updates.

The Key Idea

Although NetBSD doesn't provide native read-only support, it's flexible enough to allow for this configuration. Many years ago, I followed a howto to set up a read-only system, and the idea remains simple and effective: NetBSD writes primarily to specific locations, unlike some Linux distributions that attempt to write to various parts of the file system. On NetBSD, the main write targets are the /tmp directory and /var.

With this in mind, we can configure these directories to reside in memory file systems (mfs) and ensure that /var contains everything necessary for the system to function correctly.

1. Prepare the Environment

Start with a basic NetBSD installation (I'll skip the installation steps as there are many tutorials available, and it's straightforward).

As a first step, clean up the /var directory. After each reboot, its contents will be extracted into the mfs and occupy RAM.

Disable `man.db` generation and swap usage:

Edit the following configuration files:

# /etc/rc.conf
makemandb=NO
no_swap=YES

# /etc/daily.conf
run_makemandb=NO

Then, remove the current man.db file:

rm /var/db/man.db

Once this is done, create a compressed archive of the current /var contents to be replicated at every boot:

cd /
tar -cvzf var-image.tar.gz var

2. Create a Custom Startup Script

Next, create a file called /etc/rc.d/mount_mfs_fs with the following content:

#!/bin/sh
#
# mount_mfs_fs: mount memory file system for /var
# by roby, 23 jun 2003 - adapted by Stefano - 01 Sep 2024

# PROVIDE: mount_mfs_fs
# REQUIRE: root

. /etc/rc.subr

name="mount_mfs_fs"
start_cmd="mount_mfs_fs_start"
stop_cmd=":"

mount_mfs_fs_start()
{
    # Check if the /var entry is present and uncommented in /etc/fstab
    if grep -q '^tmpfs[[:space:]]\+/var[[:space:]]\+tmpfs[[:space:]]\+rw,-m1777,-sram%25' /etc/fstab; then
        echo "Mounting memory file system: /var"

        # Mount the file system for /var
        mount /var

        # Extract the contents of the tar file into /var
        tar -xvzpf /var-image.tar.gz -C /

        echo "Mounting memory file systems: Done."
    else
        echo "The tmpfs entry for /var is not present or is commented out in /etc/fstab. Skipping mount and extraction."
    fi
    sleep 5
}

load_rc_config $name
run_rc_command "$1"

Make the script executable:

chmod a+rx /etc/rc.d/mount_mfs_fs

3. Modify Boot Process

Now, modify the /etc/rc.d/mountcritlocal file to require this script to run before its execution:

#!/bin/sh
#
# $NetBSD: mountcritlocal,v 1.17 2022/02/20 14:42:07 alnsn Exp $
#

# PROVIDE: mountcritlocal
# REQUIRE: mount_mfs_fs

$_rc_subr_loaded . /etc/rc.subr

name="mountcritlocal"
start_cmd="mountcritlocal_start"
stop_cmd=":"

mountcritlocal_start()
{
    #       Mount critical file systems that are `local'
    #       (as specified in $critical_filesystems_local)
    #       This usually includes /var.
    #
    mount_critical_filesystems local || return $?
    if checkyesno zfs; then
        mount_critical_filesystems_zfs || return $?
    fi
    return 0
}

load_rc_config $name
load_rc_config_var zfs zfs
run_rc_command "$1"

4. Configure `/etc/fstab`

Edit the /etc/fstab file to mount /tmp and /var in memory:

tmpfs           /tmp        tmpfs   rw,-m1777,-sram%25
tmpfs           /var        tmpfs   rw,-m1777,-sram%25

5. Test the Configuration

You can now reboot the system and check if everything works correctly. Once logged in, the df command should show something like this:

Filesystem      1K-blocks         Used        Avail %Cap Mounted on
/dev/ld0a        18298254       393938     16989404   2% /
tmpfs              524192         4224       519968   0% /var
kernfs                  1            1            0 100% /kern
ptyfs                   1            1            0 100% /dev/pts
procfs                  4            4            0 100% /proc
tmpfs              524192            4       524188   0% /var/shm
tmpfs              524192            4       524188   0% /tmp

6. Set Root to Read-Only

Currently, the system is still running in read-write mode, but /var and /tmp are in memory disks. To switch the root file system to read-only, edit the /etc/fstab file like this:

/dev/ld0a               /       ffs     ro               1 1

On the next reboot, the system will be in pure read-only mode.

7. Perform Updates

To install packages, update the system, or make any changes, you'll need to temporarily switch back to read-write mode. Comment out /var in mfs, change the root file system back to read-write, and reboot:

mount -uw /
vi /etc/fstab
/dev/ld0a               /       ffs     rw               1 1
[...]
#tmpfs           /var    tmpfs   rw,-m1777,-sram%25
[...]

After making the necessary changes, regenerate the /var tarball as done in step 1.

Make Your Own CDN with NetBSD

Stefano Marinelli — Tue, 03 Sep 2024 01:41:00 +0200

Introduction

This article is a spin-off from a previous post on how to create a self-hosted CDN, based on OpenBSD, but this time we'll focus on using NetBSD. The idea is to create reverse proxies with local caching. These proxies would cache the content on the first request and serve it directly afterward. The proxies would be distributed across different regions, and the DNS would route requests to the nearest proxy based on the caller’s location. All this is achieved without relying on external CDNs, using self-managed tools instead.

NetBSD is a lightweight, stable, and secure operating system that supports a wide range of hardware, making it an excellent choice for a caching reverse proxy. Devices that other operating systems may soon abandon, such as early Raspberry Pi models or i386 architecture, are still fully supported by NetBSD and will continue to be so. Additionally, NetBSD is an outstanding platform for virtualization (using Xen or qemu/nvmm) and deserves more attention than it currently receives.

The choice of Varnish is based on several factors, with the main ones being the ability to keep the cache in RAM (which means it can run on read-only systems) and the ability to flush the cache remotely. For example, with each change to my blog, I can choose whether to perform an immediate flush (such as for a new article or an error) or wait for the cache's "natural" expiration (such as for a typo or minor, non-critical changes).

While I won't detail the installation process for NetBSD, as it depends heavily on the hardware you have available, I will guide you through setting up a self-hosted CDN using NetBSD, Varnish, nginx, and the acme.sh or lego tool for SSL certificate management.

Installation

During the installation of NetBSD, ensure that you enable support for binary package management. This will install pkgin, a tool that simplifies package management on NetBSD. If you skip this step during installation, you can still install pkgin later, but it's easier to let the installer handle it.

Once your system is up and running, use pkgin to install the necessary packages: Varnish, nginx, and, depending on your preference, either acme.sh or Go (if you plan to compile lego). Although lego is not available as a precompiled package, you can easily compile it locally using Go, but for simplicity, I recommend using acme.sh.

For this setup, I will present two methods for generating and renewing certificates: using acme.sh or compiling lego - to reach the same final outcome as the OpenBSD article.

Option 1: Using acme.sh (Recommended)

acme.sh is a simple, yet powerful, shell script that handles certificate generation and renewal with ease. To install acme.sh:

pkgin in acmesh varnish nginx

Once acme.sh is installed, you can proceed to configure it for certificate management. It supports DNS authentication and integrates with many DNS providers, making it a flexible choice.

Option 2: Compiling Lego

If you prefer to use lego, you will need to compile it manually, as it is not available as a precompiled package for NetBSD. First, install Go and other necessary packages:

pkgin in go varnish nginx

To compile lego, you’ll need some disk space. Since the /tmp directory in NetBSD is often mounted as tmpfs (using RAM), you may run out of space during compilation if your system has limited memory. You can temporarily disable tmpfs by editing /etc/fstab and commenting out the relevant line:

#tmpfs           /tmp    tmpfs   rw,-m=1777,-s=ram%25

After rebooting, compile lego:

export GO111MODULE=on
go122 install github.com/go-acme/lego/v4/cmd/lego@latest

cp go/bin/lego /usr/pkg/bin/

Once lego is compiled and installed, you can uncomment the /tmp line in /etc/fstab and reboot again.

Configuring Varnish and nginx

First, copy the necessary rc.d scripts for nginx and Varnish:

cp /usr/pkg/share/examples/rc.d/nginx /etc/rc.d/
cp /usr/pkg/share/examples/rc.d/varnishd /etc/rc.d/

Then, add the following to /etc/rc.conf to enable and configure nginx and Varnish:

nginx=YES
varnishd=YES
varnishd_flags="-f /usr/pkg/etc/varnish/default.vcl -T localhost:9999 -a "/var/run/varnish.sock",user=nginx,group=varnish,mode=660 -s default,500m"

In this configuration, Varnish listens on a Unix socket, and nginx connects to it. This approach is more efficient and helps avoid some issues that may arise when exposing Varnish over an IP/port.

Creating the Varnish VCL Configuration

Next, create the VCL configuration file for Varnish at /usr/pkg/etc/varnish/default.vcl:

vcl 4.1;
import std;

# Backend - it-notes.dragas.net
backend it_notes {
    .host = "myBackendIP";
    .port = "80";
}

# ACL - purge - it-notes.dragas.net
acl purge_it_notes {
    "allowedToPurge_IP";
}

sub vcl_recv {
    # it-notes.dragas.net
    if (req.http.Host == "it-notes.dragas.net") {
        set req.backend_hint = it_notes;
        set req.http.Host = "it-notes.dragas.net";

        # PURGE - it-notes.dragas.net
        if (req.method == "PURGE") {
            std.log("Purge request received for " + req.url);

            if (!std.ip(req.http.X-Forwarded-For, "0.0.0.0") ~ purge_it_notes) {
                return (synth(405, "Not allowed."));
            }

            if (req.url == "/" || req.url == "/*") {
                ban("req.http.host == " + req.http.host);
                return(synth(200, "Entire cache has been cleared."));
            }
            return (purge);
        }

    } else {
        # Other domains - 404
        return (synth(404, "Domain not found"));
    }

    if (req.method != "GET" && req.method != "HEAD") {
        return (pipe);
    }

    return (hash);
}

sub vcl_backend_response {
    # TTL - it-notes.dragas.net
    if (bereq.http.host == "it-notes.dragas.net") {
        if (bereq.url ~ "\.(gif|jpg|jpeg|png|webp|ico|css|js)$") {
            set beresp.ttl = 1w;
            set beresp.grace = 1d;
            set beresp.keep = 7d;
            unset beresp.http.Set-Cookie;
            unset beresp.http.Cache-Control;
            set beresp.http.Cache-Control = "public, max-age=604800";
        } else {
            set beresp.ttl = 15m;
            set beresp.grace = 48h;
            set beresp.keep = 7d;
        }
    }

    # Remove some headers
    unset beresp.http.Server;
    unset beresp.http.X-Powered-By;
    unset beresp.http.Via;

    return (deliver);
}

sub vcl_deliver {
    # Add X-Cache header
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
    } else {
        set resp.http.X-Cache = "MISS";
    }

    std.log("Delivering content for " + req.url + " - Cache: " + resp.http.X-Cache);

    # Remove Varnish headers
    unset resp.http.Via;
    unset resp.http.X-Varnish;

    return (deliver);
}

sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }
    return (lookup);
}

sub vcl_hit {
    return (deliver);
}

sub vcl_miss {
    return (fetch);
}

sub vcl_purge {
    std.log("Purge executed for " + req.url);
    return (synth(200, "Purge successful"));
}

sub vcl_synth {
    set resp.http.Content-Type = "text/html; charset=utf-8";
    set resp.http.Retry-After = "5";
    synthetic({"<!DOCTYPE html>
        <html>
            <head>
                <title>"} + resp.status + " " + resp.reason + {"</title>
            </head>
            <body>
                <h1>Status "} + resp.status + " " + resp.reason + {"</h1>
                <p>"} + resp.reason + {"</p>
                <h3>Guru Meditation:</h3>
                <p>XID: "} + req.xid + {"</p>
                <hr>
                <p>Varnish cache server</p>
            </body>
        </html>
    "});
    return (deliver);
}

Configuring nginx

Now, modify the nginx configuration file at /usr/pkg/etc/nginx/nginx.conf. Set the number of worker processes to "auto" to take advantage of all server cores, and configure the reverse proxy for your site(s). Here's an example configuration:

[...]
worker_processes  auto;
[...]

server {
    server_name it-notes.dragas.net;

    location / {
        proxy_method $request_method;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_pass http://unix:/var/run/varnish.sock;
    }

    access_log /var/log/nginx/access.it-notes.dragas.net.log;
    error_log /var/log/nginx/error.it-notes.dragas.net.log;

    listen [::]:443 ssl;
    listen 443 ssl;
    http2 on;
    # If you're using acme.sh, just change the location of the certificates
    ssl_certificate /root/.lego/certificates/it-notes.dragas.net.crt;
    ssl_certificate_key /root/.lego/certificates/it-notes.dragas.net.key;
}

server {
    if ($host = it-notes.dragas.net) {
        return 301 https://$host$request_uri;
    }
    server_name it-notes.dragas.net;
    listen 80;
    listen [::]:80;
    return 404;
}

Starting Varnish and nginx

Finally, start the Varnish and nginx services:

service varnishd start
service nginx start

If everything is configured correctly, both Varnish and nginx will be up and running, ready to handle incoming connections.

Conclusion

Congratulations, you have successfully set up your own CDN on NetBSD. This solution is lightweight, stable, and fully under your control, allowing you to break free from the constraints of major service providers. With NetBSD's broad hardware support and minimal overhead, this setup can run on a wide variety of devices, making it a versatile choice for self-hosted solutions.

I'm using it as a test and as a read-only root filesystem with a RAM-only local cache for my blog on a Raspberry Pi Zero W (first edition), and as soon as I get the new FTTH, I'll probably make it accessible via IPv6 for Italy, putting it physically into production.

If your goal is geo-replication, you can use DNS providers that offer location-based routing or set up your own DNS infrastructure to manage and resolve requests according to the user’s location. With multiple reverse proxies, separate DNS servers, and a well-configured cache, you can achieve a highly resilient system with minimal risk of a single point of failure.

Make Your Own CDN with OpenBSD Base and Just 2 Packages

Stefano Marinelli — Thu, 29 Aug 2024 01:41:00 +0200

Introduction

This article is a "spin-off" from the previous post "Building a Self-Hosted CDN for BSD Cafe Media," which I recommend reading, based on FreeBSD. In that article, I showed how I addressed the issue of geo-replication and geo-distribution of BSD Cafe media content. If you prefer a NetBSD-based setup, there's an article that describes how to do it.

The internet today relies TOO MUCH on just a few big players. When one of them stops working, half the world is impacted because too many services, in my opinion, depend on them. “Too big to fail,” some might say. “Single Point of Failure,” I respond."

The strength of the internet has always been its extreme decentralization, which is now less evident due to this phenomenon.

In this article, I want to show how easy it is to create a self-hosted CDN using OpenBSD and just two external packages: Varnish and Lego.

Actually, only one package is truly needed (Varnish), and SSL certificates could be generated using the built-in acme-client in OpenBSD. However, this might be limiting since acme-client handles certificate generation via traditional methods (using a file in .well-known), but when dealing with a CDN and several reverse proxies listening, you don’t have perfect control over which one will receive the certificate generation validation request.

For convenience and practicality, I’ll use the excellent Lego tool—a Go application that supports many DNS authentication methods, including PowerDNS.

Installation

The steps are quite simple. After installing and updating OpenBSD (using the syspatch command), start by installing the two packages:

obcdn# pkg_add lego varnish
quirks-7.14:updatedb-0p0: ok
quirks-7.14 signed on 2024-08-24T13:35:23Z
quirks-7.14: ok
lego-4.16.1: ok
varnish-7.4.2:bzip2-1.0.8p0: ok
varnish-7.4.2:pcre2-10.37p2: ok
useradd: Warning: home directory `/var/varnish' doesn't exist, and -m was not specified
varnish-7.4.2: ok
The following new rcscripts were installed: /etc/rc.d/varnishd
See rcctl(8) for details.

The next step is to enable Varnish:

rcctl enable varnishd

Varnish has a generic startup script, but it's best to customize the startup options. To do this, modify the /etc/rc.conf.local file:

varnishd_flags="-j unix,user=_varnish,ccgroup=_varnish -f /etc/varnish/default.vcl -T localhost:9999 -a localhost:8080 -s default,500m"

This configuration sets Varnish with a 500 MB cache and listens on localhost, port 8080.

Next, rename the default VCL file to prepare for your own content:

mv /etc/varnish/default.vcl /etc/varnish/default.vcl.distrib

Create a new default.vcl file. Below is an example based on this blog (at the time of writing). You’ll need to adapt it according to your needs, especially if there are cookies or other dynamic content. Note that Varnish will fetch data from a specific backend accessed in http. If privacy is needed, consider creating a VPN between the backend and Varnish, as briefly mentioned in the previous article:

vcl 4.1;
import std;

# Backend - it-notes.dragas.net
backend it_notes {
    .host = "myOriginalServer";
    .port = "80";
}

# ACL - purge - it-notes.dragas.net
acl purge_it_notes {
    "authorizedIPForCachePurge";
}

sub vcl_recv {
    # it-notes.dragas.net
    if (req.http.Host == "it-notes.dragas.net") {
        set req.backend_hint = it_notes;
        set req.http.Host = "it-notes.dragas.net";

        # PURGE - it-notes.dragas.net
        if (req.method == "PURGE") {
            std.log("Purge request received for " + req.url);

            if (!std.ip(req.http.X-Forwarded-For, "0.0.0.0") ~ purge_it_notes) {
                return (synth(405, "Not allowed."));
            }

            if (req.url == "/" || req.url == "/*") {
                ban("req.http.host == " + req.http.host);
                return(synth(200, "Entire cache has been cleared."));
            }
            return (purge);
        }

    } else {
        # Other domains - 404
        return (synth(404, "Domain not found"));
    }

    if (req.method != "GET" && req.method != "HEAD") {
        return (pipe);
    }

    return (hash);
}

sub vcl_backend_response {
    # TTL - it-notes.dragas.net
    if (bereq.http.host == "it-notes.dragas.net") {
        if (bereq.url ~ "\.(gif|jpg|jpeg|png|webp|ico|css|js)$") {
            set beresp.ttl = 1w;
            set beresp.grace = 1d;
            set beresp.keep = 7d;
            unset beresp.http.Set-Cookie;
            unset beresp.http.Cache-Control;
            set beresp.http.Cache-Control = "public, max-age=604800";
        } else {
            set beresp.ttl = 15m;
            set beresp.grace = 48h;
            set beresp.keep = 7d;
        }
    }

    # Remove some headers
    unset beresp.http.Server;
    unset beresp.http.X-Powered-By;
    unset beresp.http.Via;

    return (deliver);
}

sub vcl_deliver {
    # Add X-Cache header
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
    } else {
        set resp.http.X-Cache = "MISS";
    }

    std.log("Delivering content for " + req.url + " - Cache: " + resp.http.X-Cache);

    # Remove Varnish headers
    unset resp.http.Via;
    unset resp.http.X-Varnish;

    return (deliver);
}

sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }
    return (lookup);
}

sub vcl_hit {
    return (deliver);
}

sub vcl_miss {
    return (fetch);
}

sub vcl_purge {
    std.log("Purge executed for " + req.url);
    return (synth(200, "Purge successful"));
}

sub vcl_synth {
    set resp.http.Content-Type = "text/html; charset=utf-8";
    set resp.http.Retry-After = "5";
    synthetic({"<!DOCTYPE html>
        <html>
            <head>
                <title>"} + resp.status + " " + resp.reason + {"</title>
            </head>
            <body>
                <h1>Status "} + resp.status + " " + resp.reason + {"</h1>
                <p>"} + resp.reason + {"</p>
                <h3>Guru Meditation:</h3>
                <p>XID: "} + req.xid + {"</p>
                <hr>
                <p>Varnish cache server</p>
            </body>
        </html>
    "});
    return (deliver);
}

Start Varnish and check if it starts correctly:

rcctl start varnishd

Generating SSL Certificates

Before configuring relayd, you’ll need to generate SSL certificates. Lego supports many DNS providers and provides clear and comprehensive examples, so I suggest reading its README file.

Important: Certificates generated by Lego are not directly compatible with relayd, so you must generate them in the correct format. Add the -k rsa4096 flag to the Lego command to obtain certificates compatible with relayd.

Once generated, the certificates will be in a subdirectory of the directory from which the command was launched. For example, if the command is run as root (which is unnecessary, but just for the example), the certificates will be in /root/.lego/certificates/.

relayd expects certificates in a specific location. Copy them to the appropriate directories. In my example:

obcdn# cp /root/.lego/certificates/it-notes.dragas.net.crt /etc/ssl/
obcdn# cp /root/.lego/certificates/it-notes.dragas.net.key /etc/ssl/private/

Remember to copy the files to the correct directories when renewing. You can also create symbolic links, this will make your life easier but it’s less secure.

Configuring relayd

It’s time to configure relayd. The file is /etc/relayd.conf, and here’s an example configuration:

log state changes
prefork 10

table <itnotes> { 127.0.0.1 }

http protocol "http" {
    match request header append "X-Forwarded-For" value "$REMOTE_ADDR"
    match request header append "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"

    match request path "/*.atom" tag "CACHE"
    match request path "/*.css" tag "CACHE"
    match request path "/*.gif" tag "CACHE"
    match request path "/*.html" tag "CACHE"
    match request path "/*.ico" tag "CACHE"
    match request path "/*.jpg" tag "CACHE"
    match request path "/*.webp" tag "CACHE"
    match request path "/*.js" tag "CACHE"
    match request path "/*.png" tag "CACHE"
    match request path "/*.rss" tag "CACHE"
    match request path "/*.svg" tag "CACHE"
    match request path "/*.xml" tag "CACHE"
    match request path "/*.ttf" tag "CACHE"
    match request path "/*.woff2" tag "CACHE"

    match response tagged "CACHE" header set "Cache-Control" value "public, max-age=604800"

    pass request header "Host" value "it-notes.dragas.net" forward to <itnotes>
}

http protocol "https" {
    match request header append "X-Forwarded-For" value "$REMOTE_ADDR"
    match request header append "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"

    match response header set "Referrer-Policy" value "no-referrer"
    match response header set "X-Content-Type-Options" value "nosniff"
    match response header set "X-Download-Options" value "noopen"
    match response header set "X-Frame-Options" value "SAMEORIGIN"
    match response header set "X-Permitted-Cross-Domain-Policies" value "none"
    match response header set "X-XSS-Protection" value "1; mode=block"
    match response header set "Strict-Transport-Security" value "max-age=15552000; includeSubDomains; preload"

    match request path "/*.atom" tag "CACHE"
    match request path "/*.css" tag "CACHE"
    match request path "/*.gif" tag "CACHE"
    match request path "/*.html" tag "CACHE"
    match request path "/*.ico" tag "CACHE"
    match request path "/*.jpg" tag "CACHE"
    match request path "/*.webp" tag "CACHE"
    match request path "/*.js" tag "CACHE"
    match request path "/*.png" tag "CACHE"
    match request path "/*.rss" tag "CACHE"
    match request path "/*.svg" tag "CACHE"
    match request path "/*.xml" tag "CACHE"
    match request path "/*.ttf" tag "CACHE"
    match request path "/*.woff2" tag "CACHE"

    match response tagged "CACHE" header set "Cache-Control" value "public, max-age=604800"
    tcp { nodelay, sack, socket buffer 65536, backlog 100 }

    tls { keypair "it-notes.dragas.net" }

    pass request header "Host" value "it-notes.dragas.net" forward to <itnotes>
}

relay "http" {
    listen on vio0 port 80
    protocol "http"

    forward to <itnotes> port 8080
}

relay "https" {
    listen on vio0 port 443 tls
    protocol "https"

    forward to <itnotes> port 8080
}

relay "https6" {
    listen on my:ip:v6:address::1 port 443 tls
    protocol "https"

    forward to <itnotes> port 8080
}

The content of the file is quite self-explanatory. Note that:

vio0 is the interface name—it should be modified based on the interface where relayd needs to listen.
I’ve configured relayd to listen on port 80 as well.
IPv4 and IPv6 listeners are separated. If IPv6 is not configured, simply comment out that part. Please, if you can, use IPv6. In a fairer world, everyone would have the right to at least one class of public addresses without having to pay exorbitant fees.
The "keypair" must correspond to the certificate and key names in /etc/ssl and /etc/ssl/private.

Test the configuration, enable, and start relayd:

obcdn# relayd -n
configuration OK
obcdn# rcctl enable relayd
obcdn# rcctl start relayd
relayd(ok)

Final Checks

The stack is ready. A ps command will show the process status:

obcdn# ps aux
USER       PID %CPU %MEM   VSZ   RSS TT  STAT   STARTED       TIME COMMAND
root         1  0.0  0.0   920   276 ??  I       7:10PM    0:00.01 /sbin/init
[...]
_varnish 44999  0.0  0.2  2256  2424 ??  S       8:27PM    0:00.05 varnishd: Varnish-Mgt -i obcdn.my.domain (varnishd)
_varnish 54481  0.0  8.8 34500 88876 ??  S       8:27PM    0:00.60 varnishd: Varnish-Child -i obcdn.my.domain (varnishd)
root     57753  0.0  0.5  4080  4820 ??  IU      8:32PM    0:00.03 /usr/sbin/relayd
_relayd  95940  0.0  0.4  2152  3892 ??  Spc     8:32PM    0:00.01 relayd: pfe (relayd)
_relayd  81032  0.0  0.4  2156  3716 ??  Spc     8:32PM    0:00.01 relayd: hce (relayd)
_relayd  80312  0.0  0.5  2748  5280 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  88919  0.0  0.5  2748  5268 ??  Ipc     8:32PM    0:00.04 relayd: relay (relayd)
_relayd  90186  0.0  0.5  2752  5272 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  32851  0.0  0.5  2736  5284 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  20270  0.0  0.5  2744  5252 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  98826  0.0  0.5  2752  5264 ??  Ipc     8:32PM    0:00.04 relayd: relay (relayd)
_relayd   6454  0.0  0.5  2744  5264 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  90102  0.0  0.5  2740  5276 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  60314  0.0  0.5  2748  5336 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  27246  0.0  0.5  2744  5284 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  87082  0.0  0.4  2100  4508 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  53308  0.0  0.4  2096  4496 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  11697  0.0  0.4  2092  4492 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  83636  0.0  0.4  2092  4500 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  74563  0.0  0.4  2096  4484 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  35910  0.0  0.4  2100  4508 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  24911  0.0  0.4  2096  4504 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  78649  0.0  0.4  2096  4496 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  89586  0.0  0.4  2096  4500 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  11989  0.0  0.4  2100  4500 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
[...]

In this case, both relayd and Varnish are running correctly.

Automating Certificate Renewal

As a final step, remember to create a script to renew the certificates, copy them to /etc/ssl and /etc/ssl/private, and restart relayd.

Conclusion

Congratulations, you now have your own free CDN, with your data, fully under your control. Portable, extendable, controllable, and outside of the big providers' grip.

If your goal is geo-replication, you should use one of the available methods. Some DNS providers allow selection based on the caller's location (like Bunny.net), or, as I prefer, install your own DNS and use tools to manage operations and resolutions, as briefly described in the previous article.

With multiple separate reverse proxies, separate DNS servers (on different providers and possibly different countries or continents) capable of checking if the reverse proxies are operational, you can achieve an extremely low likelihood of encountering a Single Point of Failure, as all components, once the cache is filled, will be nearly autonomous even in the event of a (temporary) backend outage - i.e., the original node.

Building a Self-Hosted CDN for BSD Cafe Media

Stefano Marinelli — Mon, 26 Aug 2024 08:41:00 +0200

Introduction

For just over a year, BSD Cafe's media was hosted on a FreeBSD physical server jail with an outgoing bandwidth of 250 Mbit/sec. To mitigate bandwidth congestion, I initially integrated Cloudflare with a tunnel to serve media (and only media) through Cloudflare. The goal was to georeplicate the media and reduce the load on my server. To do this, the media server had to be on a separate domain managed by Cloudflare since the DNS for the primary bsd.cafe domain was managed by Bunny.net.

The first step (mainly because I discovered that Bunny's DNS does not support IPv6) was to bring the DNS back in-house using two FreeBSD jails (running on different VPS providers), both powered by PowerDNS. PowerDNS supports LUA records, which would come in handy later.

In line with the principles of self-hosting and data ownership, I decided to remove Cloudflare. I created a dedicated subdomain (media.bsd.cafe) and configured the reverse proxy in front of the jail running Minio to respond to that domain. I also reconfigured Mastodon for the new address, and after some fine-tuning, everything worked seamlessly. However, this led to some bandwidth congestion when media was posted, resulting in slower download speeds for users, especially during peak times. This is because, once content is published and federated servers are notified, they all attempt to download the newly published content - media included - almost simultaneously.

Not wanting to abandon my media server (a dedicated jail with spinning disks, offering 4 TB of storage), I opted for a different approach that I’ll describe here, as it might be useful for similar setups.

While this setup was implemented on FreeBSD, the configuration and tools - Nginx, Varnish, WireGuard and PowerDNS - are compatible with many operating systems, including Linux, with only minor adjustments required.

The Approach: Building a Self-Hosted CDN

The idea is to create reverse proxies with local caching. These proxies would cache the content on the first request and serve it directly afterward. The proxies would be distributed across different regions, and the DNS would route requests to the nearest proxy based on the caller’s location. All this is achieved without relying on external CDNs, using self-managed tools instead.

To establish a direct connection between Minio and the reverse proxies, I configured WireGuard inside the jail. The reverse proxies connect via WireGuard, allowing them to access Minio securely as if they were on the same LAN.

No further changes were needed on the media jail itself.

Setting Up the Reverse Proxies

I began configuring the reverse proxies (also running FreeBSD jails, OpenBSD (setup described in another post) and NetBSD (also described in another post), hosted on different providers). The choice of Varnish is based on several factors, with the main ones being the ability to keep the cache in RAM (which means it can run on read-only systems) and the ability to flush the cache remotely. For example, with each change to my blog, I can choose whether to perform an immediate flush (such as for a new article or an error) or wait for the cache's "natural" expiration (such as for a typo or minor, non-critical changes).

First, I connected them via WireGuard to the Minio jail (I won’t detail the steps here; I’ve covered similar setups in other posts). Then, I installed Nginx and Varnish. A more granular setup would have Varnish on a separate jail, but this way, I can move the reverse proxy jails to different hosts with minimal hassle. Currently, these reverse proxies also serve this blog.

Next, I installed and configured Varnish inside the jail:

pkg install varnish7

I created the directory /usr/local/etc/varnish and wrote a custom VCL file to manage this setup, named default.vcl:

vcl 4.1;
import std;

# Backend - it-notes.dragas.net
backend it_notes {
    .host = "itnotesip";
    .port = "itnotesport";
}

# Backend - media.bsd.cafe
backend media_bsd {
    .host = "minioWGip";
    .port = "minioport";
}

# ACL - IPs allowed to purge - it-notes.dragas.net
acl purge_it_notes {
    "a.b.c.d";
}

# ACL - IPs allowed to purge - media.bsd.cafe
acl purge_media_bsd {
    "e.f.g.h";
}

sub vcl_recv {

    # it-notes.dragas.net
    if (req.http.Host == "it-notes.dragas.net") {
        set req.backend_hint = it_notes;
        set req.http.Host = "it-notes.dragas.net";

        # PURGE - it-notes.dragas.net
        if (req.method == "PURGE") {

            std.log("Purge request received for " + req.url);

            if (!std.ip(req.http.X-Real-IP, "0.0.0.0") ~ purge_it_notes) {
                return (synth(405, "Not allowed."));
            }

        if (req.url == "/" || req.url == "/*") {
                ban("req.http.host == " + req.http.host);
                return(synth(200, "Entire cache has been cleared."));
        }
            return (purge);
        }

    # media.bsd.cafe
    } elsif (req.http.Host == "media.bsd.cafe") {
        set req.backend_hint = media_bsd;
        set req.http.Host = "media.bsd.cafe";

        # PURGE - media.bsd.cafe
        if (req.method == "PURGE") {
            if (!std.ip(req.http.X-Real-IP, "0.0.0.0") ~ purge_media_bsd) {
                return (synth(405, "Not allowed."));
            }
            if (req.url == "/" || req.url == "/*") {
                ban("req.http.host == " + req.http.host);
                return(synth(200, "Entire cache has been cleared."));
            }
            return (purge);
        }

    } else {
        # Other domains - 404
        return (synth(404, "Domain not found"));
    }

    if (req.method != "GET" && req.method != "HEAD") {
        return (pipe);
    }

    return (hash);
}

sub vcl_backend_response {
    # TTL - it-notes.dragas.net
    if (bereq.http.host == "it-notes.dragas.net") {
        if (bereq.url ~ "\.(gif|jpg|jpeg|png|ico|css|js)$") {
            set beresp.ttl = 1w;
            set beresp.grace = 1d;
            set beresp.keep = 7d;
            unset beresp.http.Set-Cookie;
            unset beresp.http.Cache-Control;
            set beresp.http.Cache-Control = "public, max-age=604800";
        } else {
            set beresp.ttl = 15m;
            set beresp.grace = 48h;
            set beresp.keep = 7d;
        }

    # TTL - media.bsd.cafe
    } elsif (bereq.http.host == "media.bsd.cafe") {
        if (bereq.url ~ "\.(mp4|mp3|wav|flac|ogg)$") {
            set beresp.ttl = 1d;
            set beresp.grace = 6h;
            set beresp.keep = 3d;
            unset beresp.http.Set-Cookie;
            unset beresp.http.Cache-Control;
            set beresp.http.Cache-Control = "public, max-age=86400";
        } else {
            set beresp.ttl = 30m;
            set beresp.grace = 12h;
            set beresp.keep = 3d;
        }
    }

    # Remove some headers
    unset beresp.http.Server;
    unset beresp.http.X-Powered-By;
    unset beresp.http.Via;

    return (deliver);
}

sub vcl_deliver {
    # ADD header X-Cache
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
    } else {
        set resp.http.X-Cache = "MISS";
    }

  std.log("Delivering content for " + req.url + " - Cache: " + resp.http.X-Cache);


    # Remove Varnish headers
    unset resp.http.Via;
    unset resp.http.X-Varnish;

    return (deliver);
}

sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }
    return (lookup);
}

sub vcl_hit {
    return (deliver);
}

sub vcl_miss {
    return (fetch);
}

sub vcl_purge {
    std.log("Purge executed for " + req.url);
    return (synth(200, "Purge successful"));
}

sub vcl_synth {
    set resp.http.Content-Type = "text/html; charset=utf-8";
    set resp.http.Retry-After = "5";
    synthetic( {"<!DOCTYPE html>
        <html>
            <head>
                <title>"} + resp.status + " " + resp.reason + {"</title>
            </head>
            <body>
                <h1>Status "} + resp.status + " " + resp.reason + {"</h1>
                <p>"} + resp.reason + {"</p>
                <h3>Guru Meditation:</h3>
                <p>XID: "} + req.xid + {"</p>
                <hr>
                <p>Varnish cache server</p>
            </body>
        </html>
    "} );
    return (deliver);
}

This setup allows Varnish to handle both domains with distinct configurations but within the same cache.

To enable Varnish, I updated the /etc/rc.conf file with the following lines, setting a maximum cache size of 2GB:

varnishd_enable="YES"
varnishd_listen="127.0.0.1:8080"
varnishd_config="/usr/local/etc/varnish/default.vcl"
varnishd_storage="default,2000M"

You can now start Varnish:

service varnishd start

The next step is to create two virtual hosts on Nginx (one for it-notes.dragas.net and one for media.bsd.cafe) that will listen on both IPv4 and IPv6 for HTTP and HTTPS. HTTP connections will be redirected to HTTPS, and incoming HTTPS traffic will be passed to Varnish, which will either return cached data or fetch it from the original server (Minio via WireGuard, for media.bsd.cafe). Let's see the media.bsd.cafe part:

server {
   server_name  media.bsd.cafe;

   [...]

   location / {
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_connect_timeout 300;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    chunked_transfer_encoding off;

    expires 12h;
    add_header Cache-Control public;

    add_header X-Cache-Status $upstream_cache_status;
    add_header X-Content-Type-Options nosniff;

    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
    add_header Referrer-Policy "no-referrer-when-downgrade";
    add_header Permissions-Policy "geolocation=(), microphone=(), camera=()";

    proxy_pass http://127.0.0.1:8080;

    [...]

}

[...]

}

server {
    if ($host = media.bsd.cafe) {
        return 301 https://$host$request_uri;
    }
   listen       80;
   listen  [::]:80;
   server_name  media.bsd.cafe;
    return 404;
}

This configuration isn’t complete, but it provides a good idea of how to set up Nginx - each setup will vary. TTL and caching sizes will also differ based on the characteristics of each reverse proxy. For example, one of the proxies has an 8GB cache since I have ample resources there.

Generating certificates is another important aspect. In this case, as the reverse proxies are distributed, they all need to respond to the same addresses. One approach is to generate the certificate on one proxy and distribute it to the others. In my case, I opted to use lego, which, through PowerDNS’s API, adds a DNS record for validation. This way, each reverse proxy can independently generate and renew its certificates when needed.

Configuring DNS for Optimal Routing

Once everything is set up, it’s important to ensure that DNS responds correctly. In my case, I implemented a strategy like this:

Track reverse proxies that respond on port 443 (further refinements are possible and will be done later).
Return the closest reverse proxy based on the client’s IP address.

Unfortunately, PowerDNS on FreeBSD does not include GeoIP support by default, but I have my poudriere ready to compile and install the necessary packages. Alternatively, you could compile it within the jail using the port system.

After that, I installed the geoipupdate package (which requires a free license from MaxMind), updated the IP list, and configured PowerDNS to use the GeoIP database. I added the GeoIP backend alongside the existing SQLite3 backend and specified the database to use:

launch=gsqlite3,geoip
geoip-database-files=mmdb:/usr/local/share/GeoIP/GeoLite2-City.mmdb

Finally, I created a LUA record to return the correct address:

testclosest.bsd.cafe   60      IN      LUA     A "ifportup(443, {'proxy1ip', 'proxy2ip','proxy3ip'}" ", {selector='pickclosest'})"
testclosest.bsd.cafe   60      IN      LUA     AAAA "ifportup(443, {'proxy1ip6', 'proxy2ip6','proxy3ip6'}" ", {selector='pickclosest'})"

And voilà! We now have a small, self-hosted CDN, keeping full control and ownership of our data. Adding a new reverse proxy is straightforward — simply clone an existing proxy, update the WireGuard configuration (adding a peer on the Minio jail and changing the keys on the new proxy), and add it to the DNS.

Happy caching!

Installing Uptime-Kuma in a FreeBSD Jail

Stefano Marinelli — Mon, 22 Jul 2024 07:42:00 +0000

Uptime-Kuma is a very useful tool. Besides being able to monitor whether services, websites, or ports are responding, it can also send notifications using many different services, even combined, and check the expiration of certificates. Additionally, it can create informative dashboards about the status of services.

I have been using it extensively for years, including for BSD Cafe. My initial deployments were on Docker containers, but recently, I have moved everything to FreeBSD jails.

When proceeding with the standard installation described in their documentation, the process will halt because one of the third-party libraries (Playwright) is not officially compatible with FreeBSD. However, since this library is not necessary for the type of checks I want to perform (I am not interested in checking web page components, just their response), it is possible to continue and run everything using a small hack.

It can be installed directly on FreeBSD but, in my case, I prefer using jails. Note that if using traditional jails (i.e., not VNet), "ping" is disabled. Therefore, if using traditional jails and wanting to check hosts via ping, it is advisable to add the line allow.raw_sockets; to the respective jail.conf.

The installation can proceed as follows:

The first step is to install the necessary dependencies:

pkg install node20 npm-node20 git-lite

Once the dependencies are installed, it is advisable to create a user to run everything and use that user for the next steps:

pw add user ukuma -m
echo 'export LC_ALL="en_US.UTF-8"' >> /home/ukuma/.profile
su -l ukuma

At this point, clone the Uptime-Kuma repository and start installing the dependencies:

git clone https://github.com/louislam/uptime-kuma.git
cd uptime-kuma/
npm run setup

The operation will seem to complete correctly. However, running the program will result in a fatal error preventing it from functioning:

ukuma@ukuma:~/uptime-kuma $ node server/server.js
[...]
/home/ukuma/uptime-kuma/node_modules/playwright-core/lib/server/registry/index.js:258
    if (process.platform === 'linux') cacheDirectory = process.env.XDG_CACHE_HOME || _path.default.join(os.homedir(), '.cache'); else if (process.platform === 'darwin') cacheDirectory = _path.default.join(os.homedir(), 'Library', 'Caches'); else if (process.platform === 'win32') cacheDirectory = process.env.LOCALAPPDATA || _path.default.join(os.homedir(), 'AppData', 'Local'); else throw new Error('Unsupported platform: ' + process.platform);
                                                                                                                                                                                                                                                                                                                                                                                             ^

Error: Unsupported platform: freebsd
    at /home/ukuma/uptime-kuma/node_modules/playwright-core/lib/server/registry/index.js:258:388
    at Object.<anonymous> (/home/ukuma/uptime-kuma/node_modules/playwright-core/lib/server/registry/index.js:270:3)
    at Module._compile (node:internal/modules/cjs/loader:1369:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1427:10)
    at Module.load (node:internal/modules/cjs/loader:1206:32)
    at Module._load (node:internal/modules/cjs/loader:1022:12)
    at Module.require (node:internal/modules/cjs/loader:1231:19)
    at require (node:internal/modules/helpers:179:18)
    at Object.<anonymous> (/home/ukuma/uptime-kuma/node_modules/playwright-core/lib/server/index.js:84:17)
    at Module._compile (node:internal/modules/cjs/loader:1369:14)

The issue is clear: Playwright does not support FreeBSD, causing the operation to fail. At this point, there are two options:

Hope that Microsoft decides to support FreeBSD, which is unlikely in the short term.
Trick Playwright into thinking FreeBSD is supported, which may break some related functionality, but it is a small price to pay to have all other features of Uptime-Kuma.

I will use the second method. Open the indicated file (in this case, /home/ukuma/uptime-kuma/node_modules/playwright-core/lib/server/registry/index.js) and modify the indicated line (in this case, 258). Change the line if (process.platform === 'linux') to if (process.platform === 'freebsd') and save it.

Uptime-Kuma will then be able to import the necessary libraries and function, listening on port 3001.

At this point, it will be possible to use it normally, as well as use pm2 to start and manage it automatically, but this goes beyond the scope of this article.

FreeBSD Tips and Tricks: Limiting Process Priority in a FreeBSD Jail

Stefano Marinelli — Thu, 11 Jul 2024 08:41:00 +0200

FreeBSD allows for quite comprehensive resource limitation for one or more jails. Beyond the official documentation, there is a good description in MWL's book. There's also an interesting article from Klara Systems that describes some functionalities.

Sometimes, however, we only need the processes running within a specific jail to have a specific priority - higher or lower than others.

There are many methods to achieve this, but the simplest one, in my opinion, is to leverage the properties of the nice(1) command. The main property, in fact, is to transmit the set priority to child processes, so all processes launched by the command that received a different level of "niceness" will inherit its priority.

For example, to give the minimum priority to the services launched within a jail, just modify the .conf file of the jail (in the case of a standard BastilleBSD installation, the file will be /usr/local/bastille/jails/jailname/jail.conf) and change the command

exec.start = '/bin/sh /etc/rc';

to:

exec.start = '/usr/bin/nice -n 20 /bin/sh /etc/rc';

In this way, when rc starts at the jail's boot, it will have a niceness of 20 and will transmit it to all the processes that rc itself launches (thus all the services of the jail).

Note: This will only apply to child processes of rc, not to:

Commands manually launched from the jail console
Services launched by running service servicename start (or restart) from the jail shell. This is because, in this case, the process will not be a child of rc but will derive directly from the console in use.

Using the nice command to set the priority of processes within a FreeBSD jail is a simple and effective method. However, it is important to be aware of the limitations of this approach and ensure that manually executed commands are managed accordingly.

Blocking Access from or to Specific Countries Using FreeBSD and pf

Stefano Marinelli — Sun, 16 Jun 2024 07:40:45 +0000

In recent times, there has been an exponential increase in malicious (or simply rude) traffic from specific countries. Alternatively, sometimes we simply do not need visitors from other parts of the world accessing our server for various reasons.

On FreeBSD, this operation is very simple, and I have been using a reliable and secure system to manage it automatically for a long time. Of course, as with all geolocation blocks, there is never certainty about the result, as sometimes certain IP blocks officially belong to one country but are actually used by another, or users can resort to VPNs to bypass these types of blocks. However, it remains a valid method to filter out unwanted traffic, especially when it comes to rogue bots that do not respect the robots.txt file and bombard our machines with repeated requests, generating real DDoS attacks.

Installation of ipdbtools

First, install the ipdbtools package (http://www.freshports.org/sysutils/ipdbtools):

pkg install ipdbtools

Next, download the updated global lists:

/usr/local/bin/ipdb-update.sh

Configuring pf

Then, modify the pf configuration. To do this, add the following line at the beginning of the filtering part of your firewall configuration (/etc/pf.conf):

block drop log quick from <blocked_countries>

At this point, simply type:

service pf reload

to reload the pf configuration and start considering the values in the <blocked_countries> table.

If you do not already have a pf.conf because the only requirement was this, simply insert this line in an empty pf.conf and then run:

service pf enable
service pf start

Updating the Blocked Countries List

Next, create a file (in my case, I called it /usr/local/sbin/update_blocked_countries.sh) with content similar to the one below. Replace "CC" with the country codes you want to block. For example, to block France, Germany, and Italy, insert "FR:DE:IT".

UPDATE: pf seems to have difficulty processing very large lists, resulting in errors. For this reason, I modified the following script to insert the lists in 'chunks,' in order to prevent the procedure from failing.

#!/bin/sh

# Original file containing the IP addresses
original_file="/var/db/blocked_countries.txt"

# Generate the blocked countries file
/usr/local/bin/ipup -p -t CC > $original_file

# Number of records per batch
batch_size=10000

# Temporary file for the current batch
temp_file="/tmp/blocked_countries_temp.txt"

# Initialize the line counter
line_count=0

# Function to add a batch of records
add_batch() {
    echo "Adding records from $temp_file to pf table..."
    /sbin/pfctl -t blocked_countries -T add -f "$temp_file"
    if [ $? -ne 0 ]; then
        echo "Error adding records from $temp_file. Exiting."
        exit 1
    fi
    # Empty the temporary file
    > $temp_file
}

# Replace the table with an empty file to avoid conflicts
echo -n > /tmp/empty_blocked_countries.txt
/sbin/pfctl -t blocked_countries -T replace -f /tmp/empty_blocked_countries.txt

# Read the original file line by line
while IFS= read -r line; do
    # Add the line to the temporary file
    echo "$line" >> "$temp_file"
    line_count=$((line_count + 1))

    # If we've reached the batch size, add the records and reset the counter
    if [ $line_count -ge $batch_size ]; then
        add_batch
        line_count=0
    fi
done < "$original_file"

# Add any remaining records
if [ $line_count -gt 0 ]; then
    add_batch
fi

echo "All records added successfully."

Make it executable:

chmod a+rx /usr/local/sbin/update_blocked_countries.sh

Running the command /usr/local/sbin/update_blocked_countries.sh will show the status of the operation, for example:

314159 addresses added.

This means that the list has been generated for the inserted country codes, and pf has updated the <blocked_countries> table.

It is advisable to update the lists occasionally since IP ranges "move" from one country to another.

Automating with Cron

We can use cron to perform this task. Edit the file /etc/crontab by adding a line like:

55 9 * * *    root    /usr/local/bin/ipdb-update.sh > /dev/null 2>&1 && /usr/local/sbin/update_blocked_countries.sh

In this example, at 9:55 every day, the databases will be updated, lists generated, and fed to pf.

This setup will also work correctly on a read-only FreeBSD system on UFS, as described in a previous article. The only precaution in this case is to ensure the crontab runs at every boot since, on every machine restart, the contents of /var will be erased and recreated from scratch. Therefore, also add a line like:

@reboot    root    /usr/local/bin/ipdb-update.sh > /dev/null 2>&1 && /usr/local/sbin/update_blocked_countries.sh

Additionally, in a read-only system, you need to add a mount for tmpfs for the /usr/local/etc/ipdb/IPRanges directory in the /etc/fstab file to make it writable:

tmpfs /usr/local/etc/ipdb/IPRanges/     tmpfs rw 0 0