IT Notes

EnshittifAIcation

Stefano Marinelli — Fri, 20 Mar 2026 11:00:00 +0000

Yesterday morning, first thing after waking up, I checked my emails. One of them was from a client - a sharp person, but not a tech expert - forwarding a message from one of their "digital marketplaces". They claimed that during site crawling, their bot upgrades the connection to HTTP/2, and that this somehow causes issues on their end, so they were asking us to disable HTTP/2 to fix the problem.

I contacted Alex directly - the person (spoiler: not a person) who had sent the email - explaining that if their bot has trouble with HTTP/2 (which, on the contrary, provides significant benefits for the e-commerce experience in question), that's their problem, not ours, and they should fix it. Completely unprompted, I received something unexpected in reply: a guide on how to configure Apache to do what they wanted. The problem? Not only did it completely ignore my stated position, but we don't use Apache - we use nginx. And, I should add, their guide was entirely wrong. I replied pointing all of this out and finally asked to be "escalated to a human, since I was clearly talking to an AI that wasn't understanding any of my responses". The reply was blunt: "That's not possible for this type of issue. Follow our guide or we will suspend your service and your e-commerce visibility." For me, obviously, that's a hard pass. For my client, though, it's a real problem - an intelligent person who understood the situation, but still a problem to solve.

Over the past few months, I've been witnessing a dramatic increase in botnet attacks targeting some of the servers I manage, especially e-commerce ones. These aren't directed at me personally - they also hit servers I manage on behalf of clients. At first I thought they were AI scrapers, but the traffic comes from everywhere, especially from residential connections scattered around the world. I believe these are deliberate disruption campaigns, a side effect of the turbulent geopolitical climate we're living through.

On several of these e-commerce servers, we decided to implement geo-blocking, as I've described previously on this blog. Normally, once you've identified your whitelist countries and the shop isn't a global operation, everything works fine. In other cases, problems arise.

A few days ago, a partner of one of my clients - a company that provides services and needs access to some prepared XML feeds - started complaining they could no longer connect. I asked them for the IP pool they connect from, or at least the country their connections originate from. Their vague reply was: "We can't provide that information because we don't have a fixed IP or set of IPs." They completely ignored the question about the country. I pushed further, but got nowhere - different "people", giving different answers, all wildly off the mark and ignoring what I was actually asking, insisting instead that I whitelist their user agent. I explained, repeatedly, that the block is at the firewall level - meaning I never even see their user agent: if the connection is dropped, there's no handshake, no HTTP headers, nothing. It didn't matter. They kept repeating the same thing without engaging with what I wrote. Eventually they went directly to my client. I'll paste the exact text:

We're having some trouble accessing the site and downloading the XML, as they both currently require a VPN connection. To ensure our Lambda functions can run correctly, could you please:

Remove the location-based restrictions for our access;

Or, allow the User-Agent "REDACTED" in your firewall/server settings?

Please let us know which option works best for you.

Let's break this down:

"Require a VPN connection" - who said anything about a VPN? Pure hallucination.
"Remove the location-based restrictions for our access" - they never once answered: which location?
"Allow the user agent" - I explained, multiple times, that the block is at the firewall level. The connection is dropped before any handshake occurs. There is no user agent to allow.

This morning, another client writes: "The marketing consultancy wants all the server load graphs to get an idea of where we stand." This is the second time in just a few days I've received a request like this. I send both the graphs and the full specs of the dedicated server in use - average load under 5%. The response was staggering: "The internal team, supported by the most advanced AI, believes your current setup is not adequate for the industry, load, and audience you're targeting, and recommends migrating to a cloud VPS with AT LEAST 8 GB of dedicated RAM to ensure sufficient resources, as the current ones are insufficient."

The current ones? 128 GB of RAM. Two modern CPUs. 48 cores total. If we followed their advice, the site would be down within five minutes - and that's just counting legitimate traffic. My client, unaware of the technical differences, asks me if we can implement what they're suggesting.

The shift was abrupt - not unlike when an intern arrives convinced they already know everything, often with the best of intentions: bringing fresh air into an environment that needs "modernising". But with an intern, you can talk. That same confidence often turns into curiosity, hunger to learn, real experience. I've watched eager interns grow into excellent professionals - people who eventually surpassed me in skill and success, and that felt genuinely satisfying, knowing I'd contributed, at least in part, to their growth. With AI, this is impossible. It doesn't grow, doesn't listen, doesn't update its mental model based on what you write back - and above all, it doesn't know what it doesn't know.

That's why I'd like companies to consider that AI systems are stochastic machines, not experts. They can solve some problems, but there's a limit. There will always be a limit, at least with current technology, and we can't afford to ignore it. The damage risks far outweighing the "savings" generated.

The enormous problem with my work these days is the extreme confidence that certain companies project, replacing humans - even senior ones - with AI, with no right of appeal. The result is monstrous confusion, enormous wasted time for everyone, and a widespread erosion of reliability, all papered over by the AI's unshakeable assertiveness - and by those who believe these systems are the Answer to the Ultimate Question of Life, the Universe, and Everything.

Rewarding confidence over actual competence is a bug humanity has always had. It has produced disasters throughout history, it is producing disasters now, and not only in the tech world.

So I find myself wondering: if they're so convinced that AI is better than senior professionals, why don't they replace the bosses with AI? I'm fairly confident the decisions would be considerably better - and humans would end up exactly where they should be.

Why I Love FreeBSD

Stefano Marinelli — Mon, 16 Mar 2026 08:10:00 +0000

When I first laid eyes on the FreeBSD Handbook, back in 2002, I couldn't believe what I was seeing. Six years of Linux, a relationship I've written about elsewhere, across various distributions, had trained me to hunt for documentation in fragments: often incomplete, often outdated, sometimes already stale after barely a year. Here was an operating system that came with a complete, accurate, up-to-date (as much as possible), detailed manual. I was already a convinced believer in Open Source, but I found myself reasoning in very practical terms: if the team behind this OS puts this much care into its documentation, imagine how solid the system itself must be. And so I decided to give it a try. I had a Sony Vaio with no room for a dual boot. I synced everything to a desktop machine with more space, took a breath, and made a decision: I'd install FreeBSD on that laptop and reinstall Linux when the experiment was over.

Spoiler: FreeBSD never left that machine.

At the time I had no idea that this experiment would shape the way I design and run systems for the next twenty years.

I realized almost immediately that GNU/Linux and FreeBSD were so similar they were completely different.

The Unix inspiration was the same, but everything worked differently - and the impression was that FreeBSD was distinctly more mature, less chaotic, more focused. A magnificent cathedral - a form then widely criticized in the circles I moved in - but one that had certain undeniable virtues. Back then I compiled the entire system from source, and I noticed right away that performance was better on that hardware than Linux had ever been. Not only that: Linux would overheat and produce unpredictable results - errors, sudden shutdowns, fans screaming even after compilation finished. My Linux friends continued to insist it was a “hardware problem”, but FreeBSD handled the load far more gracefully. I could read my email in mutt while compiling, something that was practically impossible on Linux, which would slow to a crawl. The fans would settle within seconds of the load ending, and the system felt genuinely more responsive. I never experienced a crash. I was running KDE on all my systems at the time, and the experience on FreeBSD was noticeably superior - more consistent and steady performance, none of the micro-freezes I'd come to accept on Linux, greater overall stability. The one drawback: I compiled everything, including KDE. I was a university student and couldn't leave my laptop in another room - the risk of an "incident" involving one of my flatmates was too real - so I kept it within arm's reach, night after night, fans spinning as KDE and all its applications compiled. At some point I figured out exactly how long the KDE build took, and started using it as a clock: fans running meant it was before four in the morning. Fans silent meant I'd made it past.

The Handbook taught me an enormous amount - more than many of my university courses - including things that had nothing to do with FreeBSD specifically. It taught me the right approach: understand first, act second. The more I read, the more I wanted a printed copy to keep at my desk. So I convinced my parents that I needed a laser printer “for university work”. And the first thing I printed, of course, was the Handbook. That Handbook still contains relevant information today. There have been significant changes over the past twenty-four years, but the foundations are still the same. Many tools still work exactly as they did. Features have been added, but the originals still operate on the same principles. Evolution, not revolution. And when you're building something meant to last, that is - in my view - exactly the right philosophy. Change is good. Innovation is good. On my own machines I've broken and rebuilt things thousands of times. But production environments must be stable and predictable. That, still today, is one of the qualities I value most in every BSD.

Over the years, FreeBSD has served me well. At a certain point it stepped down as my primary desktop - partly because I switched to Mac, partly because of unsupported hardware - but it never stopped being one of my first choices for servers and any serious workload. As I often say: I only have one workstation, and I use it to access hundreds of servers. It's far easier to replace a workstation - I can reconfigure everything in a couple of hours - than to deal with a production server gone sideways, with anxious clients waiting or operations ground to a halt.

FreeBSD has never chased innovation for its own sake. It has never chased hype at the expense of its core purpose. Its motto is "The Power to Serve" - and to do that effectively, efficiently, securely. That is what FreeBSD has been for me.

I love FreeBSD because it has served me for decades without surprises. I love FreeBSD because it innovates while making sure my 2009 servers keep running correctly, requiring only small adjustments at each major update rather than a complete overhaul.

I love FreeBSD because it doesn't rename my network interfaces after a reboot or an upgrade.

And because its jails - around since 2000 - are an effective, efficient, secure, simple, and fully native mechanism: you can manage everything without installing a single external package. I love FreeBSD because ZFS is native, and with it I get native boot environments, which means safe, reversible upgrades. Or, if you're running UFS, you change a single character in fstab and the entire filesystem becomes read-only - cleanly, with no kludges. I love FreeBSD because bhyve is an efficient, lightweight, reliable hypervisor. I love it for its performance, for its features, for everything it has given me.

But I love FreeBSD also - and above all - for its community. Around the BSDs, in general, you find people driven by genuine passion, curiosity, and competence. Over the past twenty years the tech world has attracted many people who appear to be interested in technology. In reality, they are often just looking for something to monetize quickly, even at the cost of destroying it. In the BSD community, that is far less common. At conferences I've had the chance to meet developers in person - to understand their spirit, their skill, and yes, their passion. Not just in the volunteers who contribute for the joy of it, but in those funded by the Foundation as well. And then there are the engineers from companies that rely heavily on FreeBSD - Netflix among them - and they bring the same quality: that engagement, that enthusiasm, that tells you FreeBSD isn't a job for them. It's a pleasure. Which is one of the reasons why every time I attend a BSD conference, I come home even more in love with the project: the vibe of the community, the dedication of the developers, the presence of a Foundation that is strong and effective without being domineering or self-important - which, compared to the foundations of other major Open Source projects, makes it genuinely remarkable. Faces that have been part of this project for over twenty years, and still light up the moment they find their friends and start talking about what they've been working on. That positivity is contagious - and it flows directly into the code, the project, the vision for what comes next. Because that's the heart of it. FreeBSD has always been an operating system written by humans, for humans: built to serve and to be useful, with a consistency, documentation, pragmatism, and craftsmanship that most other projects - particularly mainstream Linux distributions - simply don't have. The Foundation wants to hear from ordinary users. It actively promotes the kind of engagement that brings more people to FreeBSD. Not because big tech companies are pushing to create dependency, but because it believes in the project.

So thank you, FreeBSD, for helping me stay passionate for so many years, for keeping my projects running, for keeping my clients' servers up and my data safe. Thank you, FreeBSD, for never wasting time chasing the trend of the moment, and instead focusing on doing things right. Thank you, FreeBSD, for all the extraordinary people - from across the entire BSD community - you've brought into my life. Friends, not colleagues. Real people. The genuine kind. And when the people running something still believe in it - truly believe in it, after all these years - and the project keeps succeeding, that tells you there is real substance underneath. In the code. In the people. In the community.

FreeBSD doesn't want to be "the best and greatest”. It wants to serve.

The Power to Serve.

Time Machine inside a FreeBSD jail

Stefano Marinelli — Wed, 28 Jan 2026 08:52:00 +0000

Many of my clients do not use Microsoft systems on their desktops; they use Linux-based systems or, in some cases, FreeBSD. Many use Apple systems - macOS - and are generally satisfied with them. While I wash my hands of it when it comes to Microsoft systems (telling them they have to manage their desktops autonomously), I am often able to lend a hand with macOS. And one of the main requests they make is to manage the backups of their individual workstations.

macOS, thanks to its Unix base, offers good native tools. Time Machine is transparent and effective, allowing a certain freedom of management. APFS, Apple's current file system, supports snapshots, so the backup will be effectively made on a snapshot. It also supports multiple receiving devices, so you can even have a certain redundancy of the backup itself.

Having many FreeBSD servers, I am often asked to use their resources and storage. To build, in practice, a Time Machine inside one of the servers. And it is a simple and practical operation, quick and "painless". There are many guides, including the excellent one by Benedict Reuschling from which I took inspiration for this one, and I will describe the steps I usually follow to set it all up in just a few minutes.

I usually use BastilleBSD to manage my jails, so the first step is to create a new jail dedicated to the purpose. Here you have to decide on the approach: I suggest using a VNET jail or an "inherit" jail - meaning one that attaches to the host's network stack. On one hand, the inherit approach is less secure but, as often happens, it depends on the complexity of the situation. If, for example, we are using a Raspberry PI dedicated to the purpose, there is no reason to complicate things with bridges, etc., but we can attach directly to the network card with a creation command like:

bastille create tmjail 15.0-RELEASE inherit igb0

Where igb0 is the network interface we want to attach to.

In case we want to attach to the interface but in the form of a bridge, we should use this syntax:

bastille create -V tmjail 15.0-RELEASE 192.168.0.42/24 igb0

Or, if our server already has a bridge (in this case it's bridge0, but yours might be named differently):

bastille create -B tmjail 15.0-RELEASE 192.168.0.42/24 bridge0

At this point, you can choose: do we want to keep the backups inside the jail or in a separate dataset - which can even be on another pool? In some cases, this can be extremely useful: often I have jails running on fast disks (SSD or NVMe) but abundant storage on slower devices. In this example, therefore, I will create an external dataset for the backups (directly from the host) and mount it in the jail. You could also delegate the entire management of the dataset to the jail, which is a different approach.

Let's create a space of 600 GB - already reserved - on the chosen pool. 600 GB is a small space, but it's ok for an example:

zfs create -o quota=600G -o reservation=600G bigpool/tmdata

We can also create separate datasets inside for each user and assign a specific space:

zfs create -o refquota=500g -o refreservation=500g bigpool/tmdata/stefano

We can enter the jail and install what we need, remembering also to create the "mountpoint" for the dataset we just created:

bastille console tmjail 

pkg install -y samba419
mkdir /tmdata

Exit the jail and instruct Bastille to mount the dataset inside the jail every time it is launched:

exit
bastille mount tmjail /bigpool/tmdata /tmdata nullfs rw 0 0

Let's go back into the jail and start with the actual configuration. First, for each Time Machine user, we will create a system user. In my example, I will create the user "stefano", giving him /var/empty as the home directory - this will give an error since we created a Bastille thin jail, but it's not a problem. It happens because in a thin jail some system paths are read-only or not manageable as they are on a full base system, but the user is only needed for ownership and Samba login.

root@tmjail:~ # adduser
Username: stefano
Full name: Stefano
Uid (Leave empty for default):
Login group [stefano]:
Login group is stefano. Invite stefano into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]: nologin
Home directory [/home/stefano]: /var/empty
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]: no
Lock out the account after creation? [no]:
Username    : stefano
Password    : <disabled>
Full Name   : Stefano
Uid         : 1001
Class       :
Groups      : stefano
Home        : /var/empty
Home Mode   :
Shell       : /usr/sbin/nologin
Locked      : no
OK? (yes/no) [yes]: yes
pw: chmod(var/empty): Operation not permitted
pw: chown(var/empty): Operation not permitted
adduser: INFO: Successfully added (stefano) to the user database.
Add another user? (yes/no) [no]: no
Goodbye!

Give the correct permissions to the user:

# If you've not created specific datasets for the users, you'd better create their home directories now
mkdir /tmdata/stefano
chown -R stefano /tmdata/stefano/

Now we configure Samba for Time Machine. The file to create/modify is /usr/local/etc/smb4.conf:

[global]
workgroup = WORKGROUP
security = user
passdb backend = tdbsam
fruit:aapl = yes
fruit:model = MacSamba
fruit:advertise_fullsync = true
fruit:metadata = stream
fruit:veto_appledouble = no
fruit:nfs_aces = no
fruit:wipe_intentionally_left_blank_rfork = yes
fruit:delete_empty_adfiles = yes

[TimeMachine]
path = /tmdata/%U
valid users = %U
browseable = yes
writeable = yes
vfs objects = catia fruit streams_xattr zfsacl
fruit:time machine = yes
create mask = 0600
directory mask = 0700

We have set up Time Machine to support all the necessary features of macOS and to show itself as "Time Machine". Having set path = /tmdata/%U, each user will only see their own path.

At this point, we create the Samba user (meaning the one we will have to type on macOS when we configure the Time Machine):

smbpasswd -a stefano

The Time Machine is seen by macOS because it announces itself via mDNS on the network. This type of service is performed by Avahi, which we are now going to configure. Although not strictly necessary (we can always find the Time Machine by connecting directly to its IP and macOS will remember everything), seeing it announced will help other non-expert users and ourselves when we have to configure another Mac in the future.

Recent Samba releases won't need any specific avahi configuration, so we can skip this step.

We are now ready to enable everything.

service dbus enable
service dbus start
service avahi-daemon enable
service avahi-daemon start
service samba_server enable
service samba_server start

Et voilà. If everything went according to plan, the Time Machine will announce itself on your network (if you have different networks, remember to configure the mDNS proxy on your router) and you will be able to log in (with the smb user you created) and start your first backup.

I suggest encrypting the backups for maximum security and observing, from time to time, your Mac as it silently makes its backups to your trusted FreeBSD server.

Installing Void Linux on ZFS with Hibernation Support

Stefano Marinelli — Mon, 22 Dec 2025 08:43:02 +0000

Introduction

FreeBSD continues to make strides in desktop support, but Linux still holds an advantage in hardware compatibility. After running openSUSE Tumbleweed on my mini PC for several months, I decided it was time to switch to a solution I could control more closely. Not because Tumbleweed doesn't work well - it works great! - but I prefer having direct control over what happens on my machine. And I want native ZFS, because I prefer it over btrfs and it allows me to manage snapshots, backups, and rollbacks just as I do on FreeBSD, using the same tools and procedures.

The choice of Void Linux comes from its BSD-like approach: modular and free of unnecessary complexity. This makes it an excellent solution for this type of setup.

ZFSBootMenu is an extremely powerful tool. It provides an experience similar to FreeBSD's boot loader and natively supports ZFS. I strongly recommend reading the documentation and exploring its features, as some of them - like the built-in SSH daemon - can be genuine lifesavers in recovery scenarios.

Prerequisites and Audience

This guide is not for absolute beginners. If you're new to Linux or Unix-like operating systems, you'd be better served by a ready-to-use distribution like openSUSE Leap (or Tumbleweed for a rolling distribution), Linux Mint, Debian, Ubuntu, or Manjaro. The purpose of this article is to demonstrate a stable, upgradeable, and reasonably secure base setup for users already comfortable with system administration. It uses the glibc variant of Void Linux. The musl version requires different commands, for example for locale generation.

Use at your own risk.

This guide synthesizes instructions from several sources:

Void Linux (UEFI) from ZFSBootMenu - which doesn't address swap. Using a zvol for swap (not the best solution) prevents hibernation and resume. Our approach uses a separate encrypted swap partition that enables proper resume.
Void Linux Full Disk Encryption - excellent for btrfs or ext4, but we want ZFS. We'll borrow the swap configuration approach from here.
Install Void Linux with a desktop environment + Flatpaks - for the desktop portion.

If your setup differs from what's described here (NVMe disk, UEFI boot, Secure Boot disabled), consult the linked guides for explanations and variations.

Installation Script (Optional)

If you want to reproduce this setup quickly, I maintain a script that automates the procedure described in this guide: disk partitioning, ZFS pool and dataset creation, encrypted swap for hibernation resume, dracut configuration, and ZFSBootMenu EFI setup. An optional KDE Plasma desktop installation is also supported.

The script is interactive and will ask for the required parameters (target disk, timezone and keymap, passphrases, desktop options). Requirements, usage instructions, and known limitations are documented in the repository README

That said, I still recommend going through the manual process at least once. Understanding each step is part of the value of this setup, especially when troubleshooting or adapting it to different hardware.

Boot Environment

Since ZFS isn't supported by the base Void Linux image, we'll use hrmpf, an excellent rescue system based on Void Linux that includes ZFS support out of the box.

After booting, you can either proceed directly or SSH into the machine to continue remotely. I generally prefer SSH since it makes copy-paste operations much easier - especially when dealing with UUIDs and long commands. To enable SSH access, set a root password and allow root login:

passwd

Edit /etc/ssh/sshd_config and enable:

PermitRootLogin yes

Restart the SSH daemon:

sv restart sshd

Find the machine's IP address:

ip addr

You can now connect via SSH from another device.

Initial Setup

Set up the environment variables and generate a host ID - we need it for ZFS:

source /etc/os-release
export ID

zgenhostid -f 0x00bab10c

Disk Configuration

Identify your target disk and set up the partition variables. This approach keeps everything consistent and reduces errors:

# Set the base disk - adjust this to match your system
export DISK="/dev/nvme0n1"

# For NVMe disks, partitions are named like nvme0n1p1, nvme0n1p2, etc.
# For SATA/SAS disks (sda, sdb), partitions are named sda1, sda2, etc.
# Set the partition separator accordingly:
export PART_SEP="p"  # Use "p" for NVMe, empty string "" for SATA/SAS

# Define partition numbers
export BOOT_PART="1"
export SWAP_PART="2"
export POOL_PART="3"

# Build full device paths
export BOOT_DEVICE="${DISK}${PART_SEP}${BOOT_PART}"
export SWAP_DEVICE="${DISK}${PART_SEP}${SWAP_PART}"
export POOL_DEVICE="${DISK}${PART_SEP}${POOL_PART}"

Verify your configuration before proceeding:

echo "Boot device: $BOOT_DEVICE"
echo "Swap device: $SWAP_DEVICE"
echo "Pool device: $POOL_DEVICE"

Wipe the Disk

Warning: This operation will irreversibly destroy all data on the selected disk. Double-check that you've selected the correct disk and be sure to have a complete backup of your system!

zpool labelclear -f "$DISK"

wipefs -a "$DISK"
sgdisk --zap-all "$DISK"

Create Partitions

EFI System Partition

If you're not using UEFI boot, adapt this procedure following the appropriate guide linked at the beginning of this post:

sgdisk -n "${BOOT_PART}:1m:+512m" -t "${BOOT_PART}:ef00" "$DISK"

Swap Partition

The swap partition should be slightly larger than your RAM to support hibernation. When you hibernate, the entire contents of RAM are written to swap, so you need enough space to hold it all plus some overhead. In this example, I have 16 GB of RAM, so I'm creating an 18 GB swap partition:

sgdisk -n "${SWAP_PART}:0:+18g" -t "${SWAP_PART}:8200" "$DISK"

ZFS Pool Partition

sgdisk -n "${POOL_PART}:0:-10m" -t "${POOL_PART}:bf00" "$DISK"

Set Up ZFS Encryption

Encrypting the disk is strongly recommended, especially for laptops. Replace SomeKeyphrase with a strong passphrase that's easy to type. Keep in mind that during early boot, the keyboard layout might default to US, so choose a passphrase that's easy to type on a US keyboard layout:

echo 'SomeKeyphrase' > /etc/zfs/zroot.key
chmod 000 /etc/zfs/zroot.key

Create the ZFS Pool

Create the pool with conservative, well-tested options:

zpool create -f -o ashift=12 \
 -O compression=lz4 \
 -O acltype=posixacl \
 -O xattr=sa \
 -O relatime=on \
 -O encryption=aes-256-gcm \
 -O keylocation=file:///etc/zfs/zroot.key \
 -O keyformat=passphrase \
 -o autotrim=on \
 -o compatibility=openzfs-2.2-linux \
 -m none zroot "$POOL_DEVICE"

Create ZFS Datasets

zfs create -o mountpoint=none zroot/ROOT
zfs create -o mountpoint=/ -o canmount=noauto zroot/ROOT/${ID}
zfs create -o mountpoint=/home zroot/home

zpool set bootfs=zroot/ROOT/${ID} zroot

Export and Reimport for Installation

zpool export zroot
zpool import -N -R /mnt zroot
zfs load-key -L prompt zroot

zfs mount zroot/ROOT/${ID}
zfs mount zroot/home

udevadm trigger

Install the Base System

XBPS_ARCH=x86_64 xbps-install \
  -S -R https://mirrors.servercentral.com/voidlinux/current \
  -r /mnt base-system

Copy Host Configuration

Copy the files we generated earlier to the new system:

cp /etc/hostid /mnt/etc
mkdir -p /mnt/etc/zfs
cp /etc/zfs/zroot.key /mnt/etc/zfs

Configure Encrypted Swap

Now we'll set up the encrypted swap partition. This is where the hibernation magic happens - by using a separate LUKS-encrypted partition instead of a ZFS zvol, we can properly resume from hibernation.

Format the swap partition with LUKS:

cryptsetup luksFormat --type luks1 "$SWAP_DEVICE"

Open the encrypted partition, create the swap filesystem, and activate it:

cryptsetup luksOpen "$SWAP_DEVICE" cryptswap
mkswap /dev/mapper/cryptswap
swapon /dev/mapper/cryptswap

Preserve Variables for Chroot

Before entering the chroot, save the disk variables so they remain available inside the new environment:

cat << EOF > /mnt/root/disk-vars.sh
export DISK="$DISK"
export PART_SEP="$PART_SEP"
export BOOT_PART="$BOOT_PART"
export SWAP_PART="$SWAP_PART"
export POOL_PART="$POOL_PART"
export BOOT_DEVICE="$BOOT_DEVICE"
export SWAP_DEVICE="$SWAP_DEVICE"
export POOL_DEVICE="$POOL_DEVICE"
export ID="$ID"
EOF

Enter the Chroot Environment

xchroot /mnt

From this point forward, all commands are executed inside the new system.

First, load the saved variables:

source /root/disk-vars.sh

Configure fstab

Add the swap entry to /etc/fstab:

/dev/mapper/cryptswap   none            swap            defaults        0 0

Set Up Automatic Swap Unlock

To avoid entering the swap password separately after unlocking the ZFS pool, we'll create a keyfile stored on the encrypted ZFS dataset. This is secure because the keyfile only becomes accessible after the ZFS pool is unlocked.

First, install cryptsetup in the new system:

xbps-install -S cryptsetup

Generate a random keyfile and add it to the LUKS partition:

dd bs=1 count=64 if=/dev/urandom of=/boot/volume.key

cryptsetup luksAddKey "$SWAP_DEVICE" /boot/volume.key

chmod 000 /boot/volume.key
chmod -R g-rwx,o-rwx /boot

Add the keyfile to /etc/crypttab:

echo "cryptswap   $SWAP_DEVICE   /boot/volume.key   luks" >> /etc/crypttab

Include the keyfile and crypttab in the initramfs. Create /etc/dracut.conf.d/10-crypt.conf:

install_items+=" /boot/volume.key /etc/crypttab "

Basic System Configuration

Configure keyboard layout and hardware clock. Adjust the keymap and timezone to match your location:

cat << EOF >> /etc/rc.conf
KEYMAP="us"
HARDWARECLOCK="UTC"
EOF

ln -sf /usr/share/zoneinfo/Europe/Rome /etc/localtime

Configure locales:

cat << EOF >> /etc/default/libc-locales
en_US.UTF-8 UTF-8
en_US ISO-8859-1
EOF

echo "LANG=en_US.UTF-8" > /etc/locale.conf

xbps-reconfigure -f glibc-locales

Set the root password:

passwd

Configure ZFS Boot Support

cat << EOF > /etc/dracut.conf.d/zol.conf
nofsck="yes"
add_dracutmodules+=" zfs "
omit_dracutmodules+=" btrfs "
install_items+=" /etc/zfs/zroot.key "
EOF

Install ZFS:

xbps-install -S zfs

Configure ZFSBootMenu

Set the basic boot properties:

zfs set org.zfsbootmenu:commandline="quiet" zroot/ROOT
zfs set org.zfsbootmenu:keysource="zroot/ROOT/${ID}" zroot

The Critical Step: Hibernation Support

Now we need to configure hibernation resume. This is the key insight that makes this setup work: normally, the encrypted ZFS root mounts first, and then it unlocks the swap partition. But when resuming from hibernation, the kernel needs to read the hibernation image from swap before mounting the root filesystem - otherwise, the saved state would be lost.

To solve this, we tell ZFSBootMenu to unlock the swap partition early, before mounting ZFS, by specifying its LUKS UUID.

Get the UUID of your swap partition:

blkid "$SWAP_DEVICE"

You'll see output like:

/dev/...: UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" TYPE="crypto_LUKS" PARTUUID="..."

Store the UUID in a variable for the next step:

SWAP_UUID=$(blkid -s UUID -o value "$SWAP_DEVICE")
echo "Swap UUID: $SWAP_UUID"

Now set the boot parameters using the captured UUID:

zfs set org.zfsbootmenu:commandline="rd.luks.uuid=$SWAP_UUID resume=/dev/mapper/cryptswap" zroot/ROOT/${ID}

Set Up EFI Boot

Create and mount the EFI partition:

mkfs.vfat -F32 "$BOOT_DEVICE"

mkdir -p /boot/efi

Add the EFI partition to /etc/fstab using its UUID:

BOOT_UUID=$(blkid -s UUID -o value "$BOOT_DEVICE")
echo "UUID=$BOOT_UUID    /boot/efi    vfat    defaults    0 0" >> /etc/fstab

Mount it:

mount /boot/efi

Install ZFSBootMenu

xbps-install -S curl

mkdir -p /boot/efi/EFI/ZBM
curl -o /boot/efi/EFI/ZBM/VMLINUZ.EFI -L https://get.zfsbootmenu.org/efi
cp /boot/efi/EFI/ZBM/VMLINUZ.EFI /boot/efi/EFI/ZBM/VMLINUZ-BACKUP.EFI

Configure the EFI boot entries:

xbps-install -S efibootmgr

efibootmgr -c -d "$DISK" -p "$BOOT_PART" \
  -L "ZFSBootMenu (Backup)" \
  -l '\EFI\ZBM\VMLINUZ-BACKUP.EFI'

efibootmgr -c -d "$DISK" -p "$BOOT_PART" \
  -L "ZFSBootMenu" \
  -l '\EFI\ZBM\VMLINUZ.EFI'

Microcode updates

Void Linux is modular, so you may need to install additional packages for your specific hardware. For the Intel microcode, you need the non-free repo: For example:

# For Intel CPUs
xbps-install -S void-repo-nonfree 
xbps-install -S intel-ucode

# For AMD CPUs/GPUs
xbps-install -S linux-firmware-amd

After installing microcode updates, regenerate the boot images and exit:

xbps-reconfigure -fa

Desktop Installation (Optional)

If all you need is a minimal system or a server, you're done and ready to reboot. For a complete desktop environment, continue with the following steps.

Install Core Desktop Packages

xbps-install -S vim nano dbus elogind polkit xorg xorg-fonts xorg-video-drivers xorg-input-drivers dejavu-fonts-ttf terminus-font NetworkManager pipewire alsa-pipewire wireplumber xdg-user-dirs unzip gzip xz 7zip

Install KDE Plasma

xbps-install -S kde-plasma dolphin konsole firefox kdegraphics-thumbnailers ffmpegthumbs vlc ark kwrite discover kf6-purpose

Enable Services

ln -s /etc/sv/NetworkManager /etc/runit/runsvdir/default/
ln -s /etc/sv/dbus /etc/runit/runsvdir/default/
ln -s /etc/sv/udevd /etc/runit/runsvdir/default/
ln -s /etc/sv/polkitd /etc/runit/runsvdir/default/
ln -s /etc/sv/sddm /etc/runit/runsvdir/default/

Configure PipeWire Audio

mkdir -p /etc/xdg/autostart
ln -sf /usr/share/applications/pipewire.desktop /etc/xdg/autostart/

mkdir -p /etc/pipewire/pipewire.conf.d
ln -sf /usr/share/examples/wireplumber/10-wireplumber.conf /etc/pipewire/pipewire.conf.d/
ln -sf /usr/share/examples/pipewire/20-pipewire-pulse.conf /etc/pipewire/pipewire.conf.d/

mkdir -p /etc/alsa/conf.d
ln -sf /usr/share/alsa/alsa.conf.d/50-pipewire.conf /etc/alsa/conf.d
ln -sf /usr/share/alsa/alsa.conf.d/99-pipewire-default.conf /etc/alsa/conf.d

Enable Additional Repositories and Flatpak (Optional)

xbps-install -S void-repo-nonfree void-repo-multilib void-repo-multilib-nonfree

xbps-install -S flatpak
flatpak remote-add --if-not-exists flathub https://dl.flathub.org/repo/flathub.flatpakrepo

Create a Regular User and exit

For desktop use, create a non-root user with appropriate group memberships. Replace username with your desired username.

useradd -m username
passwd username
usermod username -G video,wheel,plugdev,kvm,audio,network
exit

Fix for NetworkManager

xchroot will bind mount /etc/resolv.conf and leave an empty file. Network Manager won't like it. So let's clean it up:

umount -l /mnt/etc/resolv.conf 2>/dev/null || true

rm -f /mnt/etc/resolv.conf
ln -s /run/NetworkManager/resolv.conf /mnt/etc/resolv.conf

Exit and Reboot

umount -n -R /mnt
zpool export zroot
reboot

Post-Installation

If everything went well, after entering your ZFS encryption password, you'll be greeted by the SDDM login screen.

Testing Hibernation

To verify that hibernation works correctly, you can clock the "Hibernate" button or:

loginctl hibernate

The system should power off. When you turn it back on, ZFSBootMenu will prompt for the password, unlock the swap partition, detect the hibernation image, and resume your session exactly where you left off.

If resume fails, check that: 1. The LUKS UUID in the ZFS commandline property matches your swap partition 2. The swap partition is large enough for your RAM 3. The dracut configuration includes the crypttab and keyfile

Conclusion

You now have a fully functional Void Linux system with native ZFS, full disk encryption, and working hibernation. The system is rolling, lightweight, and easy to maintain. Enjoy!

Why I (still) love Linux

Stefano Marinelli — Mon, 24 Nov 2025 08:52:00 +0100

I know, this title might come as a surprise to many. Or perhaps, for those who truly know me, it won’t. I am not a fanboy. The BSDs and the illumos distributions generally follow an approach to design and development that aligns more closely with the way I think, not to mention the wonderful communities around them, but that does not mean I do not use and appreciate other solutions. I usually publish articles about how much I love the BSDs or illumos distributions, but today I want to talk about Linux (or, better, GNU/Linux) and why, despite everything, it still holds a place in my heart. This will be the first in a series of articles where I’ll discuss other operating systems.

Where It All Began

I started right here, with GNU/Linux, back in 1996. It was my first real prompt after the Commodore 64 and DOS. It was my first step toward Unix systems, and it was love at first shell. I felt a sense of freedom - a freedom that the operating systems I had known up to that point (few, to be honest) had never given me. It was like a “blank sheet” (or rather, a black one) with a prompt on it. I understood immediately that this prompt, thanks to command chaining, pipes, and all the marvels of Unix and Unix-like systems, would allow me to do anything. And that sense of freedom is what makes me love Unix systems to this day.

I was young, but my intuition was correct. And even though I couldn't afford to keep a full Linux installation on that computer long-term due to hardware limitations, I realized that this would be my future. A year later, a new computer arrived, allowing me to use Linux daily, for everything. And successfully, without missing Windows at all (except for a small partition, strictly for gaming).

When I arrived at university, in 1998, I was one of the few who knew it. One of the few who appreciated it. One of the few who hoped to see a flourishing future for it. Everywhere. Widespread. A dream come true. I was a speaker at Linux Days, I actively participated in translation projects, and I wrote articles for Italian magazines. I was a purist regarding the "GNU/Linux" nomenclature because I felt it was wrong to ignore the GNU part - it was fundamental. Because perhaps the "Year of the Linux Desktop" never arrived, but Linux is now everywhere. On my desktop, without a doubt. But also on my smartphone (Android) and on those of hundreds of millions of people. Just as it is in my car. And in countless devices surrounding us - even if we don’t know it. And this is the true success. Let’s not focus too much on the complaint that "it’s not compatible with my device X". It is your device that is not compatible with Linux, not the other way around. Just like when, many years ago, people complained that their WinModems (modems that offloaded all processing to obscure, closed-source Windows drivers) didn't work on Linux. For "early adopters" like me, this concept has always been present, even though, fortunately, things have improved exponentially.

Linux was what companies accepted most willingly (not totally, but still...): the ongoing lawsuits against the BSDs hampered their spread, and Linux seemed like that "breath of fresh air" the world needed.

Linux and its distributions (especially those untethered from corporations, like Debian, Gentoo, Arch, etc.) allowed us to replicate expensive "commercial" setups at a fraction of the cost. Reliability was good, updating was simple, and there was a certain consistency. Not as marked as that of the BSDs, but sufficient.

The world was ready to accept it, albeit reluctantly. Linus Torvalds, despite his sometimes harsh and undiplomatic tone, carried forward the kernel development with continuity and coherence, making difficult decisions but always in line with the project. The "move fast and break things" model was almost necessary because there was still so much to build. I also remember the era when Linux - speaking of the kernel - was designed almost exclusively for x86. The other architectures, to simplify, worked thanks to a series of adaptations that brought most behavior back to what was expected for x86.

And the distributions, especially the more "arduous" ones to install, taught me a lot. The distro-hopping of the early 2000s made me truly understand partitioning, the boot procedure (Lilo first, then Grub, etc.), and for this, I must mainly thank Gentoo and Arch (and the FreeBSD handbook - but this is for another article). I learned the importance of backups the hard way, and I keep this lesson well in mind today. My Linux desktops ran mainly with Debian (initially), then Gentoo, Arch, and openSUSE (which, at the time, was still called "SUSE Linux"), Manjaro, etc. My old 486sx 25Mhz with 4MB (yes, MB) of RAM, powered by Debian, allowed me to download emails (mutt and fetchmail), news (inn + suck), program in C, and create shell scripts - at the end of the 90s.

When Linux Conquered the World

Then the first Ubuntu was launched, and many things changed. I don't know if it was thanks to Ubuntu or simply because the time was ripe, but attention shifted to Linux on the desktop as well (albeit mainly on the computers of us enthusiasts), and many companies began to contribute actively to the system or distributions.

I am not against the participation of large companies in Open Source. Their contributions can be valuable for the development of Open Source itself, and if companies make money from it, good for them. If this ultimately leads to a more complete and valid Open Source product, then I welcome it! It is precisely thanks to mass adoption that Linux cleared the path for the acceptance of Open Source at all levels. I still remember when, just after graduating, I was told that Linux (and Open Source systems like the BSDs) were "toys for universities". I dare anyone to say that today!

But this must be done correctly: without spoiling the original idea of the project and without hijacking (voluntarily or not) development toward a different model. Toward a different evolution. The use of Open Source must not become a vehicle for a business model that tends to close, trap, or cage the user. Or harm anyone. And if it is oriented toward worsening the product solely for one's own gain, I can only be against it.

What Changed Along the Way

And this is where, unfortunately, I believe things have changed in the Linux world (if not in the kernel itself, at least in many distributions). Innovation used to be disruptive out of necessity. Today, in many cases, disruption happens without purpose, and stability is often sacrificed for changes that do not solve real problems. Sometimes, in the name of improved security or stability, a new, immature, and unstable product is created - effectively worsening the status quo.

To give an example, I am not against systemd on principle, but I consider it a tool distant from the original Unix principles - do one thing and do it well - full of features and functions that, frankly, I often do not need. I don't want systemd managing my containerization. For restarting stopped services? There are monit and supervisor - efficient, effective, and optional. And, I might add: services shouldn't crash; they should handle problems in a non-destructive way. My Raspberry Pi A+ doesn't need systemd, which occupies a huge amount of RAM (and precious clock cycles) for features that will never be useful or necessary on that platform.

But "move fast and break things" has arrived everywhere, and software is often written by gluing together unstable libraries or those laden with system vulnerabilities. Not to mention so-called "vibe coding" - which might give acceptable results at certain levels, but should not be used when security and confidentiality become primary necessities or, at least, without an understanding of what has been written.

We are losing much of the Unix philosophy, and many Linux distributions are now taking the path of distancing themselves from a concept of cross-compatibility ("if it works on Linux, I don't care about other operating systems"), of minimalism, of "do one thing and do it well". And, in my opinion, we are therefore losing many of the hallmarks that have distinguished its behavior over the years.

In my view, this depends on two factors: a development model linked to a concept of "disposable" electronics, applied even to software, and the pressure from some companies to push development where they want, not where the project should go. Therefore, in certain cases, the GPL becomes a double-edged sword: on one hand, it protects the software and ensures that contributions remain available. On the other, it risks creating a situation where the most "influential" player can totally direct development because - unable to close their product - they have an interest in the entire project going in the direction they have predisposed. In these cases, perhaps, BSD licenses actually protect the software itself more effectively. Because companies can take and use without an obligation to contribute. If they do, it is because they want to, as in the virtuous case of Netflix with FreeBSD. And this, while it may remove (sometimes precious) contributions to the operating system, guarantees that the steering wheel remains firmly in the hands of those in charge - whether foundations, groups, or individuals.

And Why I Still Care

And so yes, despite all this, I (still) love Linux.

Because it was the first Open Source project I truly believed in (and which truly succeeded), because it works, and because the entire world has developed around it. Because it is a platform on which tons of distributions have been built (and some, like Alpine Linux, still maintain that sense of minimalism that I consider correct for an operating system). Because it has distributions like openSUSE (and many others) that work immediately and without problems on my laptop (suspension and hibernation included) and on my miniPC, a fantastic tool I use daily. Because hardware support has improved immensely, and it is now rare to find incompatible hardware.

Because it has been my life companion for 30 years and has contributed significantly to putting food on the table and letting me sleep soundly. Because it allowed me to study without spending insane amounts on licenses or manuals. Because it taught me, first, to think outside the box. To be free.

So thank you, GNU/Linux.

Even if your btrfs, after almost 18 years, still eats data in spectacular fashion. Even if you rename my network interfaces after a reboot. Even though, at times, I get the feeling that you’re slowly turning into what you once wanted to defeat.

Even if you are not my first choice for many workloads, I foresee spending a lot of time with you for at least the next 30 years.

Static Web Hosting on the Intel N150: FreeBSD, SmartOS, NetBSD, OpenBSD and Linux Compared

Stefano Marinelli — Wed, 19 Nov 2025 09:16:00 +0100

Update: This post has been updated to include Docker benchmarks and a comparison of container overhead versus FreeBSD Jails and illumos Zones.

Note: Some operating systems (FreeBSD and Linux) support kernel TLS (kTLS) and the related SSL_sendfile path in nginx, which can improve HTTPS performance for static files. Since this feature is not available on all the systems included in the comparison (for example NetBSD, OpenBSD and illumos), the benchmarks were run with a common baseline configuration that does not rely on kTLS. The goal is to compare the systems under similar conditions rather than to measure OS specific optimizations.

I often get very specific infrastructure requests from clients. Most of the time it is some form of hosting. My job is usually to suggest and implement the setup that fits their goals, skills and long term plans.

If there are competent technicians on the other side, and they are willing to learn or already comfortable with Unix style systems, my first choices are usually one of the BSDs or an illumos distribution. If they need a control panel, or they already have a lot of experience with a particular stack that will clearly help them, I will happily use Linux and it usually delivers solid, reliable results.

Every now and then someone asks the question I like the least:

“But how does it perform compared to X or Y?”

I have never been a big fan of benchmarks. At best they capture a very specific workload on a very specific setup. They are almost never a perfect reflection of what will happen in the real world.

For example, I discovered that idle bhyve VMs seem to use fewer resources when the host is illumos than when the host is FreeBSD. It looks strange at first sight, but the illumos people are clearly working very hard on this, and the result is a very capable and efficient platform.

Despite my skepticism, from time to time I enjoy running some comparative tests. I already did it with Proxmox KVM versus FreeBSD bhyve, and I also compared Jails, Zones, bhyve and KVM on the same Intel N150 box. That led to the FreeBSD vs SmartOS article where I focused on CPU and memory performance on this small mini PC.

This time I wanted to do something simpler, but also closer to what I see every day: static web hosting.

Instead of synthetic CPU or I/O tests, I wanted to measure how different operating systems behave when they serve a small static site with nginx, both over HTTP and HTTPS.

This is not meant to be a super rigorous benchmark. I used the default nginx packages, almost default configuration, and did not tune any OS specific kernel settings. In my experience, careful tuning of kernel and network parameters can easily move numbers by several tens of percentage points. The problem is that very few people actually spend time chasing such optimizations. Much more often, once a limit is reached, someone yells “we need mooooar powaaaar” while the real fix would be to tune the existing stack a bit.

So the question I want to answer here is more modest and more practical:

With default nginx and a small static site, how much does the choice of host OS really matter on this Intel N150 mini PC?

Spoiler: less than people think, at least for plain HTTP. Things get more interesting once TLS enters the picture.

Disclaimer
These benchmarks are a snapshot of my specific hardware, network and configuration. They are useful to compare relative behavior on this setup. They are not a universal ranking of operating systems. Different CPUs, NICs, crypto extensions, kernel versions or nginx builds can completely change the picture.

Test setup

The hardware is the same Intel N150 mini PC I used in my previous tests: a small, low power box that still has enough cores to be interesting for lab and small production workloads.

On it, I installed several operating systems and environments, always on the bare metal, not nested inside each other. On each OS I installed nginx from the official packages.

Software under test

On the host:

SmartOS, with:
- a Debian 12 LX zone
- an Alpine Linux 3.22 LX zone
- a native SmartOS zone

FreeBSD 14.3-RELEASE:
- nginx running inside a native jail

OpenBSD 7.8:
- nginx on the host

NetBSD 10.1:
- nginx on the host

Debian 13.2:
- nginx on the host

Alpine Linux 3.22:
- nginx on the host
- Docker: Debian 13 container running on the Alpine host (ports mapped)

I also tried to include DragonFlyBSD, but the NIC in this box is not supported. Using a different NIC just for one OS would have made the comparison meaningless, so I excluded it.

nginx configuration

In all environments:

nginx was installed from the system packages
worker_processes was set to auto
the web root contained the same static content

The important part is that I used exactly the same nginx.conf file for all operating systems and all combinations in this article. I copied the same configuration file verbatim to every host, jail and zone. The only changes were the IP address and file paths where needed, for example for the TLS certificate and key.

The static content was a default build of the example site generated by BSSG, my Bash static site generator. The web root was the same logical structure on every OS and container type.

There is no OS specific tuning in the configuration and no kernel level tweaks. This is very close to a “package install plus minimal config” situation.

TLS configuration

For HTTPS I used a very simple configuration, identical on every host.

Self signed certificate created with:

openssl req -x509 -newkey rsa:4096 -nodes -keyout server.key -out server.crt -days 365 -subj "/CN=localhost"

Example nginx server block for HTTPS (simplified):

server {  
listen 443 ssl http2;  
listen [::]:443 ssl http2;  

server_name _;  

ssl_certificate /etc/nginx/ssl/server.crt;  
ssl_certificate_key /etc/nginx/ssl/server.key;  

root /var/www/html;  
index index.html index.htm;  

location / {  
try_files $uri $uri/ =404;  
}  
}

The HTTP virtual host is also the same everywhere, with the root pointing to the BSSG example site.

Load generator

The tests were run from my workstation on the same LAN:

client host: a mini PC machine connected at 2.5 Gbit/s
switch: 2.5 Gbit/s
test tool: wrk

For each target host I ran:

wrk -t4 -c50 -d10s http://IP
wrk -t4 -c10 -d10s http://IP
wrk -t4 -c50 -d10s https://IP
wrk -t4 -c10 -d10s https://IP

Each scenario was executed multiple times to reduce noise; the numbers below are medians (or very close to them) from the runs.

The contenders

To keep things readable, I will refer to each setup as follows:

SmartOS Debian LX → SmartOS host, Debian 12 LX zone
SmartOS Alpine LX → SmartOS host, Alpine 3.22 LX zone
SmartOS Native → SmartOS host, native zone
FreeBSD Jail → FreeBSD 14.3-RELEASE, nginx in a jail
OpenBSD Host → OpenBSD 7.8, nginx on the host
NetBSD Host → NetBSD 10.1, nginx on the host
Debian Host → Debian 13.2, nginx on the host
Alpine Host → Alpine 3.22, nginx on the host
Docker Container → Alpine host, Debian 13 Docker container

Everything uses the same nginx configuration file and the same static site.

Static HTTP results

Let us start with plain HTTP, since this removes TLS from the picture and focuses on the kernel, network stack and nginx itself.

HTTP, 4 threads, 50 concurrent connections

Approximate median wrk results:

Environment	HTTP 50 connections
SmartOS Debian LX	~46.2 k
SmartOS Alpine LX	~49.2 k
SmartOS Native	~63.7 k
FreeBSD Jail	~63.9 k
OpenBSD Host	~64.1 k
NetBSD Host	~64.0 k
Debian Host	~63.8 k
Alpine Host	~63.9 k
Docker Container	~63.7 k

Two things stand out:

All the native or jail/container setups on the hosts that are not LX zones cluster around 63 to 64k requests per second.
The two SmartOS LX zones sit slightly lower, in the 46 to 49k range, which is still very respectable for this hardware.

In other words, as long as you are on the host or in something very close to it (FreeBSD jail, SmartOS native zone, NetBSD, OpenBSD, Linux on bare metal), static HTTP on nginx will happily max out around 64k requests per second with this small Intel N150 CPU.

The Debian and Alpine LX zones on SmartOS are a bit slower, but not dramatically so. They still deliver close to 50k requests per second and, in a real world scenario, you would probably saturate the network or the client long before hitting those numbers.

HTTP, 4 threads, 10 concurrent connections

With fewer concurrent connections, absolute throughput drops, but the relative picture is similar:

SmartOS Native around 44k
NetBSD and Alpine Host around 34 to 35k
FreeBSD, Debian, OpenBSD around 31 to 33k
The Docker Container sits slightly lower at ~30.2k req/s, showing a small overhead from the networking layer
The SmartOS LX zones sit slightly below, around 35 to 37k req/s

The important conclusion is simple:

For plain HTTP static hosting, once nginx is installed and correctly configured, the choice between these operating systems makes very little difference on this hardware. Zones and jails add negligible overhead, LX zones add a small one.

If you are only serving static content over HTTP, your choice of OS should be driven by other factors: ecosystem, tooling, update strategy, your own expertise and preference.

Static HTTPS results

TLS is where things start to diverge more clearly and where CPU utilization becomes interesting.

HTTPS, 4 threads, 50 concurrent connections

Approximate medians:

Environment	HTTPS 50 connections	CPU notes at 50 HTTPS connections
SmartOS Debian LX	~51.4 k	CPU saturated
SmartOS Alpine LX	~40.4 k	CPU saturated
SmartOS Native	~52.8 k	CPU saturated
FreeBSD Jail	~62.9 k	around 60% CPU idle
OpenBSD Host	~39.7 k	CPU saturated
NetBSD Host	~40.4 k	CPU saturated
Debian Host	~62.8 k	about 20% CPU idle
Alpine Host	~62.4 k	small idle headroom, around 7% idle
Docker Container	~62.7 k	CPU saturated

These numbers tell a more nuanced story.

FreeBSD, Debian and Alpine on bare metal form a “fast TLS” group.
All three sit around 62 to 63k requests per second with 50 concurrent HTTPS connections.
FreeBSD does this while using significantly less CPU.
During the HTTPS tests with 50 connections, the FreeBSD host still had around 60% CPU idle. It is the platform that handled TLS load most comfortably in terms of CPU headroom.
Debian and Alpine are close in throughput, but push the CPU harder.
Debian still had some idle time left, Alpine even less. In practice, all three are excellent here, but FreeBSD gives you more room before you hit the wall.
SmartOS, NetBSD and OpenBSD form a “good but heavier” TLS group.
Their HTTPS throughput is in the 40 to 52k req/s range and they reach full CPU usage at 50 concurrent connections. OpenBSD and NetBSD stabilize around 39 to 40k req/s. SmartOS native and the Debian LX zone manage slightly better (around 51 to 53k) but still with the CPU pegged.

HTTPS, 4 threads, 10 concurrent connections

With lower concurrency:

FreeBSD, Debian and Alpine still sit in roughly the 29 to 31k req/s range
SmartOS Native and LX zones are in the mid to high 30k range
The Docker Container drops slightly to ~27.8k req/s
NetBSD and OpenBSD sit around 26 to 27k req/s

The relative pattern is the same: for this TLS workload, FreeBSD and modern Linux distributions on bare metal appear to make better use of the cryptographic capabilities of the CPU, delivering higher throughput or more headroom or both.

What TLS seems to highlight

The HTTPS tests point to something that is not about nginx itself, but about the TLS stack and how well it can exploit the hardware.

On this Intel N150, my feeling is:

FreeBSD, with the userland and crypto stack I am running, is very efficient at TLS here. It delivers the highest throughput while keeping plenty of CPU in reserve.
Debian and Alpine, with their recent kernels and libraries, are also strong performers, close to FreeBSD in throughput, but with less idle CPU.
NetBSD, OpenBSD and SmartOS (native and LX) are still perfectly capable of serving a lot of HTTPS traffic, but they have to work harder to keep up and they hit 100% CPU much earlier.

This matches what I see in day to day operations: TLS performance is often less about “nginx vs something else” and more about the combination of:

the TLS library version and configuration
how well the OS uses the CPU crypto instructions
kernel level details in the network and crypto paths

I suspect the differences here are mostly due to how each system combines its TLS stack (OpenSSL, LibreSSL and friends), its kernel and its hardware acceleration support. It would take a deeper dive into profiling and configuration knobs to attribute the gaps precisely.

In any case, on this specific mini PC, if I had to pick a platform to handle a large amount of HTTPS static traffic, FreeBSD, Debian and Alpine would be my first candidates, in that order.

Zones, jails, containers and Docker: overhead in practice

Another interesting part of the story is the overhead introduced by different isolation technologies.

From these tests and the previous virtualization article on the same N150 machine, the picture is consistent:

FreeBSD jails behave almost like bare metal and are significantly more efficient than Docker.
For both HTTP and HTTPS, running nginx in a jail on FreeBSD 14.3-RELEASE produces numbers practically identical to native hosts.
The contrast with Docker is striking: while the Docker container required 100% CPU to reach peak for the HTTP and HTTPS throughput, the FreeBSD jail delivered the same speed with ~60% of the CPU sitting idle. In terms of performance cost per request, Jails are drastically cheaper.
SmartOS native zones are also very close to the metal.
Static HTTP performance reaches the same 64k req/s region and HTTPS is only slightly behind the "fast TLS" group, although with higher CPU usage.
SmartOS LX zones introduce a noticeable but modest overhead.
Both Debian and Alpine LX zones on SmartOS perform slightly worse than the native zone or FreeBSD jails. For static HTTP they are still very fast. For HTTPS the Debian LX zone remains competitive but costs more CPU, while the Alpine LX zone is slower.
Docker on Linux performs efficiently but eats the margins. I ran an additional test using a Debian 13 Docker container running on the Alpine Linux host. At peak load (50 connections), the throughput was impressive and virtually identical to bare metal: ~63.7k req/s for HTTP and ~62.7k req/s for HTTPS. However, there is a clear cost. First, while the bare metal host maintained a small CPU buffer (~7% idle) during the HTTPS test, Docker saturated the CPU to 100%. Second, at lower concurrency (10 connections), the overhead became visible. The Docker container scored ~30.2k req/s for HTTP and ~27.8k req/s for HTTPS, slightly trailing the ~31-34k and ~29-31k range of the bare metal counterparts. The abstraction layers (NAT, bridging, namespaces) are extremely efficient, but they are not completely free.

This leads to a clear conclusion on efficiency: FreeBSD Jails provide the highest throughput with the lowest CPU cost. LX zones and Docker containers can match the speed (or come close), but they burn significantly more CPU cycles to do so.

What this means for real workloads

It is easy to get lost in tables and percentages, so let us go back to the initial question.

A client wants static hosting.
Does the choice between FreeBSD, SmartOS, NetBSD or Linux matter in terms of performance?

For plain HTTP on this hardware, with nginx and the same configuration:

Not really.
All the native hosts and FreeBSD jails deliver roughly the same maximum throughput, in the 63 to 64k req/s range. SmartOS LX zones are slightly slower but still strong.

For HTTPS:

Yes, it starts to matter a bit more.
FreeBSD stands out for how relaxed the CPU is under high TLS load.
Debian and Alpine are very close in throughput, with more CPU used but still with some headroom.
SmartOS, NetBSD and OpenBSD can still push a lot of HTTPS traffic, but they reach 100% CPU earlier and stabilize at lower request rates.

Does this mean you should always choose FreeBSD or Debian or Alpine for static HTTPS hosting?

Not necessarily.

In real deployments, the bottleneck is rarely the TLS performance of a single node serving a small static site. Network throughput, storage, logging, reverse proxies, CDNs and application layers all play a role.

However, knowing that FreeBSD and current Linux distributions can squeeze more out of a small CPU under TLS is useful when you are:

sizing hardware for small VPS nodes that must serve many HTTPS requests
planning to consolidate multiple services on a low power box
deciding whether you can afford to keep some CPU aside for other tasks (cache, background jobs, monitoring, and so on)

As always, the right answer depends on the complete picture: your skills, your tooling, your backups, your monitoring, the rest of your stack, and your tolerance for troubleshooting when things go sideways.

Final thoughts

From these small tests, my main takeaways are:

Static HTTP is basically solved on all these platforms.
On a modest Intel N150, every system tested can push around 64k static HTTP requests per second with nginx set to almost default settings. For many use cases, that is already more than enough.
TLS performance is where the OS and crypto stack start to matter.
FreeBSD, Debian and Alpine squeeze more HTTPS requests out of the N150, and FreeBSD in particular does it with a surprising amount of idle CPU left. NetBSD, OpenBSD and SmartOS need more CPU to reach similar speeds and stabilize at lower throughput once the CPU is saturated.
Jails and native zones are essentially free, LX zones cost a bit more.
FreeBSD jails and SmartOS native zones show very little overhead for this workload. SmartOS LX zones are still perfectly usable, but if you are chasing every last request per second you will see the cost of the translation layer.
Benchmarks are only part of the story.
If your team knows OpenBSD inside out and has tooling, scripts and workflows built around it, you might happily accept using more CPU on TLS in exchange for security features, simplicity and familiarity. The same goes for NetBSD or SmartOS in environments where their specific strengths shine.

I will not choose an operating system for a client just because a benchmark looks nicer. These numbers are one of the many inputs I consider. What matters most is always the combination of reliability, security, maintainability and the human beings who will have to operate the
system at three in the morning when something goes wrong.

Still, it is nice to know that if you put a tiny Intel N150 in front of a static site and you pick FreeBSD or a modern Linux distribution for HTTPS, you are giving that little CPU a fair chance to shine.

Self-hosting your Mastodon media with SeaweedFS

Stefano Marinelli — Thu, 06 Nov 2025 11:30:02 +0000

Mastodon 4.5.0 is here, and with it come some interesting changes that, in my opinion, might encourage more people to consider it for self-hosting their Fediverse community.

While it may not be as lightweight and simple as other solutions (like snac or GoToSocial or many others), I believe it remains one of the best platforms for managing a medium-sized Fediverse community, thanks in part to the direct feedback that many admins have provided to the developers.

I have previously written about how to install Mastodon in a FreeBSD jail and how to modify its character and poll limits.

One of the most critical initial decisions (which can be changed later, but with extra work) is where to store your media files. Mastodon downloads and re-processes all media it encounters from other instances for three main reasons:

Local Caching: Your users connect to your media server, reducing the load on the original instance.
Security: Re-processing media helps to remove any potential "impurities" before they reach the user's device.
Privacy: It prevents disclosing your users' IP addresses to other instances. A user will only connect to their own instance to fetch all data, including remote content.

At least initially, media files will be the largest part of your instance's storage footprint. It is therefore essential to plan where to store them and to add a regular cleanup script; otherwise, their growth will be exponential.

Mastodon supports uploading media to external S3-compatible solutions, and many admins use the usual commercial providers, paying for data uploads and transfers.

I am a firm believer in "Own Your Data", so I have always used my own self-hosted S3 servers. I initially started with Minio, but over time, I realized that, by design, it doesn't perform well with a multitude of small files (performance degrades). After running some tests, I decided to switch to SeaweedFS.

SeaweedFS "is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek..." - this, combined with the fact that it is a mature and proven piece of software, was enough for me to give it a try. The result? Excellent. The I/O and CPU load on my media server dropped drastically, making SeaweedFS an incredibly suitable solution. Furthermore, some of its features (like the ability to run a filer.sync) allow for efficient and fast replication to other storage, another host, or... anything else.

SeaweedFS works perfectly with Mastodon, and I will explain the steps to get it into production.

I will install SeaweedFS in a dedicated jail and use a dedicated subdomain. This ensures that the media server can be moved to another host at any time without reconfiguring everything or changing domains. SeaweedFS has its own FreeBSD package, installable via pkg, or can be downloaded directly from the project's website.

In either case, I will describe a "test" setup - which can also be used in production without issues. However, I highly recommend diving deeper into the tool, as it is incredibly powerful and flexible and can solve many more problems than one might imagine.

Setting up the SeaweedFS Jail

First, let's create a dedicated jail with BastilleBSD:

bastille create media 14.3-RELEASE 10.0.0.66 bastille0

Now, let's enter the jail and install SeaweedFS (and tmux, which can be useful):

bastille console media
pkg install -y tmux seaweedfs

I suggest launching SeaweedFS in a tmux session so you can monitor its output. Later, you should configure an automatic startup method, such as using the included rc.d file or any other method you prefer.

Create a directory for the data and start SeaweedFS as the "seaweedfs" user:

mkdir -p /seaweedfs/data
chown -R seaweedfs /seaweedfs
su -m seaweedfs
cd /seaweedfs/
/usr/local/bin/weed server -dir /seaweedfs/data -s3

At this point, SeaweedFS will start and create everything it needs to function, including the S3 server.

Configuring Buckets and Users

Now, let's open the weed shell to create the necessary bucket and users:

weed shell
s3.bucket.create -name mastomedia

Still in the weed shell, create a user for Mastodon and grant read permissions for unauthenticated users (which is necessary to serve media to the world):

s3.configure -access_key=mastomedia -secret_key=CHANGEME -buckets=mastomedia -user=mastodon -actions=Read,Write,List,Tagging,Admin -apply
s3.configure -buckets=mastomedia -user=anonymous -actions=Read -apply
s3.configure -buckets=mastomedia -actions=Read -apply

Security Tip: For the -secret_key, avoid using a simple password. You can generate a strong, random key directly from your shell with a command like openssl rand -base64 32.

Done. SeaweedFS is now ready to receive (and serve) media. The next step is to set up a reverse proxy to serve everything over HTTPS. My preferred approach is to configure the system as if it were external, even if the services are in adjacent jails. This might use slightly more resources, but the time and trouble it saves in the future are well worth it.

Nginx Reverse Proxy Configuration

The reverse proxy can be configured something like this:

[...]

server {
   server_name  media.mastodon.example.com;

   ignore_invalid_headers off;
   client_max_body_size 0; # Allow large file uploads without Nginx limits

   location / {
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;

      proxy_connect_timeout 300;
      proxy_http_version 1.1;
      proxy_set_header Connection "";
      chunked_transfer_encoding off;

      expires 1y;
      add_header Cache-Control public;

      add_header X-Cache-Status $upstream_cache_status;
      add_header X-Content-Type-Options nosniff;

      proxy_pass http://10.0.0.66:8333;
   }

# ... other server configurations like SSL ...

}

Mastodon Configuration

Now let's configure Mastodon. If you are running the setup wizard for the first time, here is a summary of the options:

[...]
Do you want to store uploaded files on the cloud? yes
Provider Minio
Minio endpoint URL: https://media.mastodon.example.com
Minio bucket name: mastomedia
Minio access key: mastomedia
Minio secret key: CHANGEME
Do you want to access the uploaded files from your own domain? Yes
Domain for uploaded files: media.mastodon.example.com

If Mastodon is already active, or once the setup is complete, the options in your .env.prod file should be modified to be consistent with what SeaweedFS expects:

S3_ENABLED=true
S3_PROTOCOL=https
S3_REGION=us-east-1
S3_ENDPOINT=https://media.mastodon.example.com
S3_HOSTNAME=media.mastodon.example.com
S3_BUCKET=mastomedia
AWS_ACCESS_KEY_ID=mastomedia
AWS_SECRET_ACCESS_KEY=CHANGEME
S3_FORCE_SINGLE_REQUEST=true
# remove the S3_ALIAS_HOST if it is set

IMPORTANT NOTE: If both services are in jails on the same host (i.e., SeaweedFS is on the same host as Mastodon), you should ensure that the Mastodon jail can reach the SeaweedFS jail through the reverse proxy and not via the external IP. To do this, add the following line to the /etc/hosts file of the Mastodon jail:

10.0.0.1        media.mastodon.example.com

In this example, the reverse proxy is at 10.0.0.1. If you are not using a separate reverse proxy but are exposing Nginx directly from the jail (as described in my Mastodon installation article), use the IP of the Mastodon jail itself instead (e.g., 10.0.0.42).

With this setup, Mastodon will be able to upload media to the SeaweedFS server and generate the correct links for other instances, public visitors, and users of your own instance.

Have fun with SeaweedFS!

The Email They Shouldn't Have Read

Stefano Marinelli — Wed, 08 Oct 2025 14:25:36 +0200

Author's Note: Before we begin, an important clarification. What follows is a horror story based on real events from my career. However, to protect the privacy of the people and companies involved, I have deliberately mixed things up: technologies, contexts, and specific details have been modified or merged with other experiences. I therefore invite you to read this story not as a strict chronicle of a single event, but as an archetype of a widespread problem in the IT world: vendor lock-in and predatory business practices. Any attempt to identify the specific company or software described would lead to an incorrect conclusion. More about it here.

When the phone rang, I was in a meeting - so I didn’t answer. But I recognized the number and sent a quick message:

"I’m with a client right now. If it’s not urgent, please send me an email - otherwise I’ll call you ASAP".

The reply, via SMS, left me speechless:

"That’s exactly the problem. I can’t send you an email. Call me as soon as you can".

From that moment on, my perception of a certain kind of world changed forever.

A few years earlier, a major public institution - let’s call it Agency A - was still running an ancient Exchange mail server. It hadn’t received security updates for ages, the anti-spam was completely ineffective, and the new regulations were clear: embrace Open Source solutions whenever possible.

They had already received a proposal - expensive but, when compared to similar offers made to other organizations, apparently reasonable - for a managed service hosted by an external provider and based on an open source mail stack. The company offered a managed version with its own proprietary additions and enterprise support.

The catch? While such pricing had become almost "normal" in the market, it was still wildly inflated considering what was actually being delivered. Agency A already had solid infrastructure - reputable IP classes, redundant datacenters, everything running smoothly. We had built and maintained that environment for years, and it was still performing perfectly.

The request was simple: “Evaluate this solution, and if it’s suitable, we’ll migrate.”. About 500 active mailboxes, roughly the same number of aliases. Manageable, but far from trivial.

So I started experimenting. I had heard of that stack before but never used it directly. I deployed it in some non-critical environments - ours, and a few test clients who agreed to try it at a discounted rate. Everything worked flawlessly for almost a year. I began to appreciate its design and flexibility. Confident, I told Agency A we could proceed with a pilot migration.

We built a new server, deployed the stack, and assigned a few secondary domains for early adopters. The feedback was great - so good that users started pushing for a full migration.
The IT team planned carefully: created accounts and aliases, migrated selected mailboxes, and kept the old Exchange server online (hidden, for legacy access).

The morning after the MX switch I was tense, waiting for trouble - but it never came. A couple of small questions, nothing serious. The internal team handled everything perfectly. It was a success.

Word spread quickly.

Agency B - smaller, but in some ways more influential - contacted me. They were customers of the same managed-service company that had pitched to Agency A. Once they saw the potential savings (at less than a tenth of the annual cost), the stability, and the freedom of keeping their data on their own servers, they became very interested. Their contract, however, was a five-year deal with automatic renewal - two years left. The legal office said the notice period was six months, so there was time.

They wanted to prepare silently. Their supplier was known for aggressive commercial behavior and often retaliated when customers tried to leave. So we built everything quietly - users, aliases, test setups - and froze the system, waiting for the official termination notice.

That day finally came. The notice was sent, about eight months before expiration. Migration would begin upon confirmation of receipt - or the following month at the latest.

Meanwhile, I learned that Agency C - another institution - was also planning to leave the same provider. They wanted to keep the same software stack for consistency, so I told them about our experience. They asked for a quote, which I prepared (without mentioning Agency B, of course). My margin would have been small, but the project made sense: it was about owning your data, not making money.

Everything seemed to move smoothly - until that SMS.

I called back immediately.

"There’s been a problem with the termination" said the IT manager of Agency B.
"Somehow they found out what we were doing. There are hidden clauses we didn’t know about, and now we can’t leave - at least not for another five years. They know everything. Even your quote.".

I was stunned. How could they possibly know?

Minutes later, my phone rang again - Agency C this time.

"Forget the proposal", they said. "They called us. Threatened us, actually. They even mentioned your name and said they might take legal action against you for unfair competition - claiming they’re the only ‘authorized’ installers of that software. Which is absurd, of course. It’s open source. But our director doesn’t want trouble.".

Sometimes, when public money is involved, people prefer avoiding troubles over doing what’s right.

Something didn’t add up.
Then someone at Agency C noticed a clue: a former interim IT manager still had an email client connected via token authentication - with access to all messages. And that person had signed the original contract with the provider years before. Informally questioned, he admitted contacting them "to warn them" but claimed it was harmless. He never mentioned me - supposedly.

That still didn’t explain how they knew about Agency B’s internal steps. To test a theory, we set a trap: I asked a friend abroad to send Agency B a fake quote, from a company outside the EU.
The following Monday, the provider called Agency B and said, "We advise against working with non-EU companies - compliance can get tricky.".

That strongly suggested it: it looked as if they might have been reading the emails.

The IT manager exploded and confronted them. The response was chilling:

"I’m not saying we do - but we could. It’s in the contract. You should read the fine print, especially the unilateral amendment from two years ago.".

That amendment, quietly accepted, included horrifying clauses:
- notice period extended from 6 to 12 months
- formerly free services could become paid at the provider’s discretion
- and, "for security reasons", they could disable any access other than the webmail - which they promptly did.

All of this happened before the GDPR era, when certain practices could still slip through.

I tried to contact the company’s owner directly - no reply. Calls, emails, nothing. Their support lines were "not authorized to forward requests". I wanted to confront them about the “not accredited” nonsense and the so-called unfair competition. But bullies never like a fair conversation.

I urged Agency B and C to investigate - not only legally, but ethically.
They were horrified, yes - but in the end, nothing changed.
Worse: the provider, invoking that same contract amendment, made previously free features paid ones, increasing their costs by another 30%.
Management wasn’t outraged by the abuse - just by the extra expense, "hard to justify in the budget".

Years later, those directors were gone. The technical staff remained - older, wiser, and determined not to repeat the mistake. They eventually switched providers, though to something "safer", not necessarily better.

I couldn’t solve that problem. The battle had to come from them, and I would have supported them all the way - not for profit, but for principle.
Because when a company that claims to “support open source” behaves like that, we all lose.
We all get labeled the same way.

And that’s the real horror of the story - not the software, but what people do with it.

FreeBSD vs. SmartOS: Who's Faster for Jails, Zones, and bhyve VMs?

Stefano Marinelli — Fri, 19 Sep 2025 10:50:00 +0200

Disclaimer
These benchmarks were performed on my specific hardware and tuned for the workloads I expect to run.
They should not be taken as absolute or universally applicable results.
Different CPUs, storage, networking setups, or workload profiles could produce very different outcomes.
What I’m sharing here is a faithful snapshot of my test environment and use case - a guidepost, not a final verdict.

Years ago, I installed a PCEngines APU at a client's site. It dutifully ran Proxmox with a few small VMs inside. It wasn't a speed demon, but it got the job done. Tasked with running in a closed, uncooled, and unsupervised server closet, it soldiered on for about seven years.

Then, while I was at BSDCan, I got the call. A series of power outages and surges had finally taken their toll, and the APU was dead. It was probably just the power supply, but given its age, we decided it was time for a replacement. I set up a remote bypass to keep them running, but I knew I'd need to install something more powerful soon.

I ordered a modern MiniPC-based on the low-power Intel Processor N150 platform, but with 16GB of RAM and more than enough performance to serve as a decent workstation. I have a similar one in my office running openSUSE Tumbleweed, and it works beautifully.

This time, however, I decided to replace Proxmox with a different virtualization system. This decision wasn't made in a vacuum. In the past, I've put bhyve head-to-head with Proxmox, and my findings were clear: bhyve on FreeBSD is an extremely efficient hypervisor, often outperforming KVM on Proxmox in my tests.

This positive experience is what made FreeBSD with bhyve a top contender. The other path was a KVM-style approach (which would require fewer changes to the VMs), where my options would be NetBSD or an illumos-based OS like SmartOS. Since I had the new hardware on hand, I decided to run some tests to see how these different technologies stacked up against each other, and against the bare metal itself.

The Lineup: What I Put on the Test Bench

My goal was to test every reasonable option on this Intel N150 hardware. The final lineup covered the entire spectrum:

The Baseline:
- FreeBSD 14.3-RELEASE Bare Metal: The ground truth for performance on this hardware.
OS-Level Virtualization (Containers):
- SmartOS Native Zone: The baseline native container on SmartOS.
- SmartOS LX Zone: Running Ubuntu 24.04 and Alpine Linux.
- FreeBSD Native Jail: The baseline native container on FreeBSD.
- FreeBSD Jail with Linux: A jail running a Ubuntu 22.04 userland.
Full Hardware Virtualization (HVM):
- SmartOS bhyve Zone: A FreeBSD guest inside the bhyve hypervisor on a SmartOS host.
- SmartOS KVM Zone: A FreeBSD guest inside the KVM hypervisor on a SmartOS host.
- FreeBSD bhyve VM: A FreeBSD guest inside the bhyve hypervisor on a FreeBSD host.

The Benchmark: My `sysbench` Commands

To keep the comparison fair and simple, I used two core sysbench commands. To ensure consistency, I even compiled sysbench from scratch on the SmartOS native zone to match the versions and compile options on the other systems as closely as possible.

The commands I used in each environment were:

For CPU performance: sysbench --test=cpu --cpu-max-prime=20000 run
For memory performance: sysbench --test=memory run

First Look: CPU and Memory on the Intel N150

My initial tests on the Intel N150 hardware immediately revealed some interesting trends. The sysbench CPU results from any native FreeBSD environment (bare metal or jail) were on a completely different scale from the Linux and SmartOS guests, making a direct comparison meaningless.

However, by excluding the incompatible FreeBSD-native results, we get a very clear picture of the overhead between the various container technologies.

Valid CPU Performance Comparison (Single Thread, Intel N150)

Host OS	Container Tech	Guest OS	CPU Performance (Events/sec)
FreeBSD	Jail (OS-level)	Ubuntu 22.04	1108.18
SmartOS	LX Zone (OS-level)	Ubuntu 24.04	1107.13
SmartOS	Native Zone (OS-level)	SmartOS	1107.04
SmartOS	LX Zone (OS-level)	Alpine Linux	1022.81

The takeaway here was clear: for CPU work, the overhead from these containers is basically a rounding error. For CPU-bound tasks, neither SmartOS Zones nor FreeBSD Jails will be a bottleneck.

The memory results, which were consistent across all platforms, were far more revealing.

Overall Memory Performance Comparison (Intel Processor N150)

Host OS	Virtualization Tech	Guest OS	Memory Performance (Transfer Rate)
SmartOS	LX Zone (OS-level)	Ubuntu 24.04	4970.54 MiB/sec
SmartOS	Native Zone (OS-level)	SmartOS (Native)	4549.97 MiB/sec
FreeBSD	Jail (OS-level)	Ubuntu 22.04	4348.32 MiB/sec
FreeBSD	Bare Metal	FreeBSD (Native)	4005.08 MiB/sec
FreeBSD	Native Jail (OS-level)	FreeBSD (Native)	3990.13 MiB/sec
SmartOS	LX Zone (OS-level)	Alpine Linux	3803.72 MiB/sec
FreeBSD	bhyve VM (Full HVM)	FreeBSD	3636.01 MiB/sec
SmartOS	bhyve Zone (Full HVM)	FreeBSD	3020.15 MiB/sec
SmartOS	KVM Zone (Full HVM)	FreeBSD	205.18 MiB/sec

These initial numbers led to a few conclusions: a virtual layer could be a performance boost, the userland matters, and bhyve clearly outclassed the legacy KVM on SmartOS. However, one result was nagging at me: the performance gap between FreeBSD bare metal (4005.08 MiB/sec) and a native bhyve VM (3636.01 MiB/sec) was about 9%. This was a larger drop than I expected. It prompted a new question: was this overhead inherent to bhyve, or was it a quirk of the new N150 hardware?

Going deeper: Testing on an Intel i7-7500U

To see if more mature, better-supported hardware would tell a different story, I replicated the FreeBSD tests on an older Qotom Mini-PC powered by an Intel i7-7500U. The results were illuminating and dramatically changed the narrative.

CPU Performance Comparison (Intel i7-7500U)

Once again, the CPU tests produced strange results. The native FreeBSD environments all reported incredibly high numbers in the millions of events/sec, while the Ubuntu Linuxulator jail's result was on a completely different, incompatible scale. Frankly, given the massive discrepancy between FreeBSD-native and Linux-based environments, I'm unsure that the sysbench CPU figures can be considered totally reliable in absolute terms.

However, what is useful is comparing the native FreeBSD results against each other. This tells us about relative overhead.

Platform	CPU Performance (Events/sec)	Overhead vs. Bare Metal
FreeBSD Bare Metal	6,377,778	Baseline
FreeBSD Native Jail	6,379,271	~0.0%
FreeBSD bhyve VM	6,346,852	-0.48%

Even if we're skeptical of the absolute numbers, the relative comparison is crystal clear: the CPU overhead of bhyve is less than half a percent. This is the key takeaway.

Memory Performance Comparison (Intel i7-7500U)

The memory benchmarks, in contrast, were consistent and highly informative.

Platform	Memory Performance (Transfer Rate)	Overhead vs. Bare Metal
Ubuntu 22.04 Jail	4856.23 MiB/sec	+7.55%
FreeBSD Native Jail	4517.73 MiB/sec	+0.05%
FreeBSD Bare Metal	4515.24 MiB/sec	Baseline
FreeBSD bhyve VM	4491.60 MiB/sec	-0.52%

This is where the real story is. The memory performance of a bhyve VM was a mere 0.52% slower than bare metal. This is the kind of near-native performance one hopes for from a top-tier hypervisor and stands in stark contrast to the 9% drop seen on the newer N150.

Breaking Down the Results: What I Learned From Both Tests

This comprehensive two-platform analysis paints a much clearer picture.

1. Hardware Really Matters Performance is not an absolute. The difference between the two platforms was stark: on the mature i7-7500U, bhyve’s overhead was less than 1%, while on the newer, budget N150, it was a more significant 9%. This suggests the performance dip is likely due to missing optimizations for that specific CPU architecture, rather than a fundamental flaw in bhyve itself.

2. bhyve's True Potential is Near-Native Speed The i7 tests prove that bhyve is an exceptionally efficient hypervisor on well-supported hardware. The relative CPU overhead was a negligible -0.48%, and more importantly, the reliable memory benchmarks showed a performance drop of just 0.52% compared to bare metal. This is the gold standard for virtualization.

3. FreeBSD Jails are Feather-Light On both platforms, native FreeBSD jails demonstrated almost zero performance overhead. On the i7, both CPU and memory performance were virtually identical to bare metal (a 0.05% difference). The N150 CPU tests further showed that FreeBSD's container implementation is so efficient that running a Linux userland inside a jail delivered the best CPU scores of the entire lineup.

4. SmartOS Zones Are Also Extremely Efficient Just like Jails, SmartOS's native Zones proved to be remarkably lightweight. The N150 CPU tests confirm this, showing that native and LX zones have virtually identical, top-tier performance. On the memory front, the native Zone delivered performance over 13% faster than the FreeBSD bare-metal baseline, pointing to the high efficiency of the illumos kernel.

5. The Linux Userland Excels at Throughput A clear pattern emerged on both testbeds: the Ubuntu userland consistently delivered excellent benchmark results. On the CPU front, Ubuntu on both FreeBSD and SmartOS delivered the highest, and nearly identical, performance scores on the N150. For memory, the story was even more dramatic: the Ubuntu LX Zone on SmartOS was the top performer, beating bare-metal FreeBSD by nearly 25%, while the Ubuntu jail on the i7 also surpassed its host by over 7%.

Final Thoughts: The Verdict for My Client's New Server

So, what's the bottom line for my client's new MiniPC? This benchmarking journey has made the path forward much clearer.

At the beginning of this process, my main question was whether to stick with a KVM-based setup or make the switch to bhyve. The performance data answers that decisively. The legacy KVM on SmartOS showed a crippling performance penalty, making it a non-starter. Given that, the extra effort to migrate the existing VMs to a bhyve-compatible format is absolutely worth it. The performance gain is just too significant to ignore.

The final question, then, is which host OS to use for bhyve: SmartOS or FreeBSD? This is a much tougher call, as both platforms demonstrated incredible strengths.

SmartOS, powered by the illumos kernel, was a true surprise. It delivered astonishing performance on the target N150 hardware. Its key advantage is the raw speed of its containerization for both CPU and memory tasks. The Ubuntu LX Zone not only ran flawlessly but delivered top-tier CPU scores and outperformed the bare-metal FreeBSD baseline in memory by a massive 25% margin. This points to a highly efficient kernel and offers the tantalizing prospect of running ultra-fast Linux containers alongside performant bhyve VMs on the same host.

On the other hand, FreeBSD proved its mastery of bhyve virtualization. The tests on the i7 hardware showed its implementation to be the gold standard, offering virtually zero performance overhead for full hardware virtualization. Its native Jails are equally efficient, and its Linux compatibility layer is so effective that an Ubuntu jail delivered the fastest CPU performance of all containers tested on FreeBSD. For workloads that must live in a full VM, FreeBSD offers the most performant and native bhyve experience, with the reasonable expectation that its support for newer hardware like the N150 will only improve over time.

Ultimately, the choice comes down to the primary workload. It's a decision between the raw container speed and Linux flexibility of SmartOS versus the pure, uncompromising HVM performance of FreeBSD.

But one thing is certain: thanks to this deep dive, the path forward is much clearer, and it's paved by bhyve.

Introducing the illumos Cafe: Another Cozy Corner for OS Diversity

Stefano Marinelli — Mon, 18 Aug 2025 09:04:00 +0200

Introducing the illumos Cafe: Another Cozy Corner for OS Diversity

From the BSD Cafe to illumos Cafe

The idea for this new project was born from the success of the BSD Cafe, an initiative I introduced to the world in July 2023, which received an incredibly positive response. Far more than I ever anticipated. The BSD community already had its well-established hubs: in the Fediverse, places like bsd.network, exquisite.social, and others were already thriving, not to mention all the forums, channels, and Reddit communities.

But in my vision, something was still missing: a hub of services with a positive spirit, built exclusively with open-source tools, where people could come to share, learn, and experience technology with a positive mindset. The BSD Cafe is therefore not just an instance, but a true Cafe - I’ll be speaking more about the BSD Cafe in detail at the next EuroBSDCon.

Why Another Cafe?

In a world increasingly dominated by centralized services under the control (or lack thereof) of the usual big players, it has become essential to create free, independent communities, devoid of the algorithmic and commercial controls that influence our overall experience. From day one, the BSD Cafe has embodied this spirit.

Linux is a good kernel, and there are excellent distributions based on it (some using the GNU userland, others only partially, like Alpine Linux), but it cannot and should not become a monoculture. The alternatives are extremely capable, and for many use cases - in my opinion and experience - they are even more suitable. BSD systems have served me exceptionally well for over 20 years, providing stability and security. At the same time, many other operating systems are renowned for their robustness, reliability, and the quality of their design and implementation.

Why illumos?

illumos is one of them. As the open-source descendant of OpenSolaris, it is an operating system known for its enterprise-grade stability and innovative technologies like ZFS, DTrace, and "zones". It was born from the solid foundations of Solaris and has evolved over time while remaining true to many of its core principles. I have always seen illumos and its distributions as kindred spirits to the BSDs, despite their differences. The philosophy is one of evolution without revolution, of guaranteeing long-term continuity and reliability rather than chasing the latest hype. This is precisely why, for some time now (and thanks in part to the inspiring posts by Joel Carnat, which further sparked my curiosity), I have been running OmniOS and SmartOS alongside my BSD-based setups for certain workloads.

However, there is very little information online about services running on them. So, a few months ago, I began to consider a new project: the illumos Cafe.

The illumos Cafe Project

The illumos Cafe is a project similar to the BSD Cafe (though perhaps less complex, at least initially). It shares the same spirit of positivity and inclusivity and aims to provide services running on illumos-based operating systems to demonstrate that there are no reasons not to use them. Just like with the BSD Cafe, diversifying the operating systems we use - even while using the same platforms - is fundamental to improving the reliability and resilience of the Internet. The Internet was born as a decentralized network, but for most people, it has sadly become just a tool to access the services of big players.

Community and Philosophy

But we want to connect. We want relationships with people, between people. We don't want algorithms. We don't want our data to be monetized by "us and our 65535 partners". We want a network that serves us, an OS that serves us - not an OS that just serves as a vehicle to store our data in "someone else's house". The illumos Cafe, therefore, aims to be a home for anyone interested in developing, using, or who is simply curious about illumos-based operating systems.

Technical Setup

As with the BSD Cafe, the entire setup will be documented. For now, it is very simple: there is a VM (running on FreeBSD and bhyve, on hardware I manage) where I have installed SmartOS. The physical host also runs the reverse proxy (in a jail). Inside the SmartOS VM, there are a series of zones:

Zone 1: nginx (Web Server) - Currently serving the project's homepage.
Zone 2: Mastodon (Social) - Hosting the Mastodon instance and its dependencies at https://mastodon.illumos.cafe.
Zone 3: PostgreSQL (Database) - The Mastodon database, on a dedicated zone.
Zone 4: Redis (Cache) - The Mastodon cache, on a dedicated zone.
Zone 5: snac (LX Zone) - Currently in an LX zone (Alpine) as I ran into some issues getting it to work in a native zone. It will be moved to a native zone as soon as I resolve them. It's serving the snac instance at https://snac.illumos.cafe

Media files are stored on an external physical server (running FreeBSD, the same one as the BSD Cafe, but in a dedicated jail) with SeaweedFS. I was able to compile and run SeaweedFS on illumos without any problems, but at the moment, I don't have a host with enough storage space for the media.

Available Services

More services will arrive over time. For now, two gateways to the Fediverse are already available:

Mastodon - https://mastodon.illumos.cafe
snac - https://snac.illumos.cafe

Both instances share the same rules as the BSD Cafe. Positivity. Supporters, not haters. I want them to be places of enjoyment, not venting. Of friendship, not hate.

Registrations and Logo

Registrations for the Mastodon instance are now open, and the available themes are the default ones plus the colorful TangerineUI - whose orange hue echoes the illumos logo.

The project's logo was not generated by an AI. I made it myself by hastily sticking the illumos SVG onto a coffee cup. Basic, perhaps. But authentic.

Looking Ahead

The BSD Cafe will, of course, remain my primary home. But I want to bring illumos into the Fediverse and provide a home for anyone who wishes to share their interest in this excellent OS.

I will document the entire process, just as I did with Mastodon on FreeBSD, as it is a bit more intricate. Because in my dreams, I see Fediverse statistics showing instances spread fairly evenly across the major open-source operating systems. Because relying on a single OS, even if it's open-source, and ceasing to support the others is also a single point of failure.

Make Your Own Backup System – Part 2: Forging the FreeBSD Backup Stronghold

Stefano Marinelli — Tue, 29 Jul 2025 08:00:00 +0200

With the primary backup strategies and methodologies introduced, we've reached the point where we can get specific: the Backup Server configuration.

When choosing the type of backup server to use, I tend to favor specific setups: either I trust a professional backup service provider (like Colin Percival's Tarsnap), or I want full control over the disks where the backups will be hosted. In both cases, for the past twenty years, my operating system of choice for backup servers has been FreeBSD. With a few rare exceptions for clients with special requests, it covers all my needs. When I require Linux-based solutions, such as the Proxmox Backup Server, I create a VM and manage it within.

I typically use both IPv4 and IPv6. For IPv4, I "play" with NAT and port forwarding. For IPv6, I tend to assign a public IPv6 address to each jail or VM, which is then filtered by the physical server's firewall. Unfortunately, every provider, server, and setup has a different approach to IPv6, making it impossible to cover them all in this article. When a provider allows for routed setups, I use this approach: Make your own VPN: FreeBSD, WireGuard, IPv6, and ad-blocking included - assigning a /72 to the bridge for the jails and VMs.

In my opinion, FreeBSD is a perfect all-rounder for backups, thanks to its ability to completely partition services. You can separate backup services (or specific servers/clients) into different jails or even VMs. Furthermore, using ZFS greatly enhances both flexibility and the range of tools you can use.

The main distinction is usually between local backup servers (physically accessible, though not always attended, and in locations deemed secure) and remote ones, such as leased external servers. I personally use a combination of both. If the services I need to back up are external, in a datacenter, and need to be quickly restorable, I prefer to always have a copy on another server in a different datacenter with good outbound connectivity. This guarantees good bandwidth for restores, which isn't always available from a local connection to the outside world. However, an internal, nearby, and accessible backup server (even a Raspberry Pi or a mini PC) ensures physical access to the data. Whenever possible, I maintain both an external and an internal copy - and they are autonomous, meaning the internal copy is not a replica of the external one, but an additional, independent backup. This ensures that if a problem occurs with the external backup, it won't automatically propagate to the internal one. In any case, the backup must always be in a different datacenter from the one containing the production data. When the fire at the OVH datacenter in Strasbourg caused the entire complex to shut down, many people found themselves in trouble because their backups were in the same, now unreachable, location. I had a copy with another provider, in a different datacenter and country, as well as a local copy.

Despite it being "just" a backup server, I almost always use some form of disk redundancy. If I have two disks, I set up a mirror. With three or more, I use RaidZ1 or RaidZ2. This is because, in my view, backups are nearly as important as production data. The inability to recover data from a backup means it's lost forever. And it happens often, very often, that someone contacts me to recover a file (or a database, etc.) days or weeks after its accidental loss or deletion. Usually, pulling out a file from a two-month-old backup generates a mix of disbelief, admiration, but above all, a sense of security in the person requesting it. And that is what our work should instill in the people we collaborate with.

The backup server should be hardened. If possible, it should be protected and unreachable from the outside. My best backup servers are those accessible only via VPN, capable of pulling the data on their own. If they are on a LAN, it's even better if they are completely disconnected from the Internet.

For this very reason, backups must always be encrypted. Having a backup means having full access to the data, and the backup server is the prime target for being breached or stolen if the goal is to get your hands on that data. I've seen healthcare facilities' backup servers being targeted (in a rather trivial way, to be honest) by journalists looking for health details of important figures. It is therefore critical that the backup server be as secure as possible.

Based on the type of access, I use two types of encryption:

If the server is local (especially if the ZFS pool is on external disks), I usually install FreeBSD on UFS in read-only mode, as I've described in a previous article, and encrypt the backup disks with GELI. This ensures that in the event of a "dirty" shutdown (more likely in unattended environments), I can reconnect to the host and then reactivate the ZFS pool. This approach makes it nearly impossible to retrieve even the pool's metadata if the disks are stolen, as GELI performs a full-device encryption. For example, an employee of a company I work with stole one of the secondary backup disks (which was located at a different, unmonitored company site) to steal information. He got nothing but a criminal complaint. With this approach, it's also not necessary to further encrypt the datasets, which avoids some issues (which I'll discuss later, in a future post).
If the server is remote, in a datacenter, I usually use ZFS native encryption, encrypting the main backup dataset (and BastilleBSD's, if applicable). Consequently, all child datasets containing backups will also be encrypted. In this case as well, a password will be required after a reboot to unlock those datasets, ensuring that the data cannot be extracted if control of the disks is lost.

Here is an example of how to use GELI to encrypt an entire partition and then create a ZFS pool on it (in the example, the disk is da1 - do not follow these commands blindly, or you will erase all content on the da1 device!):

# WARNING: This destroys the existing partition table on disk da1
gpart destroy -F da1

# Create a new GPT partition table
gpart create -s gpt da1

# Add a freebsd-zfs partition that spans the entire disk
# The -a 1m flag ensures proper alignment
gpart add -t freebsd-zfs -a 1m da1

# Initialize GELI encryption on the new partition (da1p1)
# We use AES-XTS with 256-bit keys and a 4k sector size
# The -b flag means "boot," prompting for the passphrase at boot time
geli init -b -l 256 -s 4096 da1p1
# You will be prompted for a passphrase: choose a strong one and save it!

# Attach the encrypted partition. A new device /dev/da1p1.eli will be created.
# You will be prompted for the passphrase you just set
geli attach da1p1

# (Optional) Check the status of the encrypted device
geli status da1p1

# Create the ZFS pool "bckpool" on the encrypted device
# We enable zstd compression (an excellent compromise) and disable atime
zpool create -O compression=zstd -O atime=off bckpool da1p1.eli

In this setup, the reference pool for everything related to backups will be bckpool - and you'll need to keep this in mind for the next steps. Additionally, after every server reboot, you'll need to "unlock" the disk and import the pool:

# Enter the passphrase when prompted
geli attach da1p1

# Import the ZFS pool, which is now visible
zpool import bckpool

With this method, it's not necessary to encrypt the ZFS datasets, as the underlying disk (or, more precisely, the partition containing the ZFS pool) is already encrypted.

If, instead, you choose to encrypt the ZFS dataset (for example, if you install FreeBSD on the same disks that will hold the data and don't want to use a multi-partition approach), you should create a base encrypted dataset. Inside it, you can create the various backup datasets, VMs, and the BastilleBSD mountpoint. Due to property inheritance, they will all be encrypted as well.

To create an encrypted dataset, a command like this will suffice:

# Creates a new dataset with encryption enabled.
# keylocation=prompt will ask for a passphrase every time it's mounted.
# keyformat=passphrase specifies the key type.
zfs create -o encryption=on -o keylocation=prompt -o keyformat=passphrase zfspool/dataset

In this case, after every reboot, you will need to load the key and mount the dataset:

zfs load-key zfspool/dataset
zfs mount zfspool/dataset

Keep in mind the setup you choose, as many of the subsequent choices and commands will depend on it.

Base System Setup

I'll install BastilleBSD - a useful tool for separating services into jails. It will be helpful for isolating our backup services:

pkg install -y bastille

If you used ZFS for the root filesystem, you can proceed directly with the setup. Otherwise (i.e., ZFS on other disks), you'll need to edit the /usr/local/etc/bastille/bastille.conf file and specify the correct dataset on which to install the jails. Then run:

bastille setup

Once the automatic setup is complete, check the /etc/pf.conf file - it will be automatically configured to only accept SSH connections. Ensure the network interface is set correctly. When you activate pf, you will be kicked out of the server, but you can then reconnect.

service pf start

Let's bootstrap a FreeBSD release for the jails - this will be useful later.

bastille bootstrap 14.3-RELEASE update

Now, we create a local bridge. Jails and VMs can be attached to it, making them fully autonomous. Using VNET jails, for example, will allow the creation of VPNs or tun interfaces inside them, simplifying potential future setups (and increasing security by using a dedicated network stack).

Modify the /etc/rc.conf file and add:

# Add lo1 and bridge0 to the list of cloned interfaces
cloned_interfaces="lo1 bridge0"
# Assign an IP address and netmask to the bridge
ifconfig_bridge0="inet 192.168.0.1 netmask 255.255.255.0"
# Enable gateway functionality for routing
gateway_enable="yes"

Let's also modify /etc/pf.conf to allow the 192.168.0.0/24 subnet to access the Internet via NAT. We will skip packet filtering on bridge0 and enable NAT. This isn't the most secure setup, but it's sufficient to get started:

#...
# Skip PF processing on the internal bridge interface
set skip on bridge0
#...
# NAT traffic from our internal network to the outside world
nat on $ext_if from 192.168.0.0/24 to any -> ($ext_if:0)
#...

To ensure the new settings are correct, it's a good idea to test with a reboot.

Since I often configure vm-bhyve in my setups, I prefer to install it right away, creating the dataset that will contain the VMs and installation templates. Remember that zroot is only valid if you installed the entire system on ZFS; otherwise, you'll need to change it to your own dataset:

# Install required packages
pkg install vm-bhyve grub2-bhyve bhyve-firmware
# Create a dataset to store VMs
zfs create zroot/VMs
# Enable the vm service at boot
sysrc vm_enable="YES"
# Set the directory for VMs, using the ZFS dataset
sysrc vm_dir="zfs:zroot/VMs"
# Initialize vm-bhyve
vm init
# Copy the example templates
cp /usr/local/share/examples/vm-bhyve/* /zroot/VMs/.templates/

At this point, I usually enable the console via tmux. This means that when a VM is launched, it won't open a VNC port by default, but a tmux session connected to the VM's serial port. Let's install and configure tmux:

pkg install -y tmux
vm set console=tmux

Let's also attach the switch we created (bridge0) to vm-bhyve so we can use it:

vm switch create -t manual -b bridge0 public

Now, vm-bhyve is ready.

The basic infrastructure is complete. We now have:

ZFS to ensure data integrity, which will also handle redundancy, etc.
BastilleBSD to manage jails, useful for backing up Linux, NetBSD, OpenBSD, and non-ZFS FreeBSD machines.
vm-bhyve to install specific systems (like Proxmox Backup Server).

Backup Strategies

I use various backup tools, too many to list in this article. So I'll make a broad distinction, describing how to use this server to achieve our goal: securing data.

For FreeBSD servers with ZFS (hosts, VMs, jails, hypervisors, and their respective VMs), I use an extremely useful, efficient, and reliable tool: zfs-autobackup.
For Linux servers (without ZFS), NetBSD, OpenBSD, etc. (any non-ZFS OS), I usually use BorgBackup. There are other fantastic tools like restic, Kopia, etc., but BorgBackup has never let me down and has served me well even on low-power devices and after incredibly complex disasters.
For Proxmox servers (a solution I've used with satisfaction in production since 2013, although I'm recently migrating to FreeBSD/bhyve where possible), I use two possible alternatives (often both at the same time): if the storage is ZFS, I use the zfs-autobackup approach. In either case, the most practical solution is the Proxmox Backup Server. And the Proxmox Backup Server is one of the reasons I proposed installing vm-bhyve: running it in a VM and storing the data on the FreeBSD host gives you the best of both worlds. Some time ago, I tried running it in a FreeBSD jail (via Linuxulator), but it didn't work.

Backups using zfs-autobackup

zfs-autobackup is an extremely useful and effective tool. It allows for "pull" type backups, as well as having an intermediary host that connects to both the source and destination, which is useful if you don't want direct contact between the source and destination. I won't describe the latter setup, but the documentation is clear, and I have several of them in production, ensuring that the production server and its backup server cannot communicate with each other.

I usually create a dataset for each server and instruct zfs-autobackup to keep that server's backups in that dataset. The snapshots taken and transferred will all be from the same instant, so as not to create a time skew (some tools of this kind snapshot a dataset, then transfer it, which can result in minutes of difference between two different datasets from the same server).

I've described in detail how I perform this type of backup in a previous post, so I suggest reading that post for reference.

Let's install zfs-autobackup on the FreeBSD server:

pkg install py311-zfs-autobackup mbuffer

Backups for other servers using BorgBackup

When I don't have ZFS available or need to perform a file-based backup (all or partial), I use a different technique. BorgBackup backups are primarily "push" based, meaning the client will connect to the backup server. This is not optimal or the most secure approach, as the backup server should, in theory, be hardened. Even when protecting everything via VPN, the risk remains that a compromised server could connect to its backup server and alter or delete the backups. I have seen this happen in ransomware cases (especially in the Microsoft world), and so I try to be careful to minimize this type of problem, mainly through snapshots of the backup server (an operation that will be described later).

To ensure the highest possible security, I create a FreeBSD jail on the backup server for each server I need to back up. The advantage of this approach is the complete separation of all servers from each other. By using a regular user inside a jail, a compromised server that connects to its backup server would only be able to reach its own backups, as it would be confined to a user account and, even if it managed to escalate privileges, still be inside a jail.

Let's say, for example, we want to back up a server called "ServerA" (great imagination, I know). We create a dedicated jail on the backup server:

# Create a new VNET jail named "servera" attached to our bridge
bastille create -B servera 14.3-RELEASE 192.168.0.101/24 bridge0

BastilleBSD will automatically set the host's gateway for the jail. In our case, this is incorrect, so we need to modify it and set the jail's gateway to 192.168.0.1 in the /usr/local/bastille/jails/servera/root/etc/rc.conf file:

# ...
defaultrouter="192.168.0.1"
# ...

Restart the jail and connect to it:

bastille restart servera
bastille console servera

Now, inside the jail, we install borgbackup:

pkg install py311-borgbackup

BorgBackup doesn't run a daemon; it's launched by the remote server (which sends its data to the backup server), so it's important that the installed version is compatible with the one on the remote host.

Since we'll be using SSH, let's enable it:

service sshd enable
service sshd start

And create a non-privileged user for this purpose:

# The 'adduser' utility provides an interactive way to create a user.
root@servera:~ # adduser
Username: servera
Full name: Server A
Uid (Leave empty for default): 
Login group [servera]: 
Login group is servera. Invite servera into other groups? []: 
Login class [default]: 
Shell (sh csh tcsh nologin) [sh]: 
Home directory [/home/servera]: 
Home directory permissions (Leave empty for default): 
Use password-based authentication? [yes]: 
Use an empty password? (yes/no) [no]: 
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]: 
Username    : servera
Password    : <random>
Full Name   : Server A
Uid         : 1001
Class       : 
Groups      : servera 
Home        : /home/servera
Home Mode   : 
Shell       : /bin/sh
Locked      : no
OK? (yes/no) [yes]: yes
adduser: INFO: Successfully added (servera) to the user database.
adduser: INFO: Password for (servera) is: JIkdq8Ex

The user is created and can receive SSH connections. After setting everything up, I suggest disabling password-based login in the jail's SSH configuration, using only public key authentication.

As mentioned, the biggest risk of a "push" backup is that a compromised client could access the backup server and delete or encrypt the backup history, rendering it useless.

To drastically mitigate this risk, we can configure SSH to force the client to operate in a special Borg mode called append-only. In this mode, the SSH key used by the client will only have permission to create new archives, not to read or delete old ones. However, this approach could complicate some client-side operations (like mount, prune, etc.), forcing them to be done on the server. For this reason, I won't describe it in this setup, "limiting" myself to taking snapshots of the repositories. It can be a very good practice, so I recommend considering it.

Let's initialize the BorgBackup repository. In this example, for simplicity, I won't set up repository encryption. If the jails are on an encrypted dataset or GELI-encrypted disks, there will still be data encryption on the disks, but there will be no protection against someone who could physically access the server while the disks are mounted. As usual, security is like an onion: every layer helps. Personally, I suggest enabling and using it ALWAYS.

# Switch to the new user
su -l servera
# Initialize a new Borg repo named "servera" with no encryption (for this example)
borg init -e none servera

The jail is ready, but it's unreachable from the outside. There are two ways to make it accessible:

Install a VPN system inside the jail itself. Using tools like Zerotier or Tailscale (which don't need to expose ports) will immediately create the conditions to connect to the jail, which will remain inaccessible from the outside. As the jail is a VNET jail, we're free to choose any of the supported VPN technologies.
Expose a port on the backup server, i.e., on the host, to allow external connections. Many advise against this path as they consider it less secure. It is, but sometimes we don't have the luxury of installing whatever we want on the server we're backing up.

To expose the port, go back to the host and modify the /etc/pf.conf file, creating the rdr and pass rules to let packets in:

# ...
# Redirect incoming traffic on port 1122 to the jail's SSH port (22)
rdr on $ext_if inet proto tcp from any to any port = 1122 -> 192.168.0.101 port 22
# ...
# Allow incoming traffic on port 1122
pass in inet proto tcp from any to any port 1122 flags S/SA keep state

Reload the pf configuration:

service pf reload

The jail will now be reachable on the server's public IP, on port 1122. Obviously, this port number is for illustrative purposes, and I used from any, but for better security, you should replace any with the IP address of the server that will be connecting to perform the backup.

By repeating this process for each server and creating a separate jail for each, you can have isolated jails in separate datasets with their backups, potentially setting space limits using ZFS quotas.

It's important to remember that backing up a live filesystem (i.e., without a snapshot or dumps) has a very high probability of being impossible to restore completely. Databases hate this approach because files will change while being copied and tend to get corrupted. Of course, it depends on the nature of the data (a backup of a static website will have no issues, but a WordPress database probably will), but it's crucial to think about a technique to snapshot the filesystem before proceeding. For example, I have already written about how to create snapshots on FreeBSD with UFS in a previous article: FreeBSD tips and tricks: creating snapshots with UFS.

I will cover other operating systems in a future, dedicated post.

Proxmox Backup Server in a Dedicated VM

Starting with version 4.0 (which is still in beta at the time of this writing), Proxmox Backup Server (PBS) supports storing its data in an S3 bucket. This is excellent news as it decouples the server from the data. There are great open-source S3 implementations, like Minio or SeaweedFS, which allow for clustering, replication, etc. In this "simple" case, we will install Proxmox Backup Server in a small VM, while for the data, we'll install Minio in a native FreeBSD jail. The advantage is undeniable: the VM will only serve as an "intermediary", but the data will rest directly on the FreeBSD host's dataset, natively. It will also be possible to specify other external endpoints, other repositories, etc.

As a philosophy, I tend not to use external providers unless for specific needs, so installing Minio in a jail is a perfect solution to manage this situation.

Let's install PBS by downloading the ISO from their website (https://enterprise.proxmox.com/iso/) - at this moment, the version that supports this setup is 4.0 Beta.

The directory to download to is the vm-bhyve ISOs directory. It's not strictly necessary, but it's useful for not "losing" it somewhere. So, go to the directory and download it:

cd /zroot/VMs/.iso
fetch https://enterprise.proxmox.com/iso/proxmox-backup-server_4.0-BETA-1.iso

Now let's create a VM with vm-bhyve. We can start from the Debian template, but we'll make some modifications to optimize performance. In this example, I'm giving it 30 GB of disk space, 2 GB of RAM, and 2 cores.

If you want to store all backups inside the VM, you'll need to size the virtual disk correctly (or create and attach another one). In this case, I will focus on the "clean" VM that will store its data on a dedicated jail with Minio.

vm create -t debian -s 30G -m 2G -c 2 pbs

Once the empty VM is created, let's modify its options:

vm configure pbs

We will modify the VM to be UEFI and to use the NVME disk driver - bhyve performs significantly better on NVME than virtio, as previously tested:

loader="uefi"
cpu="2"
memory="2G"
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"

Fortunately, the Proxmox team has provided for the installation of the Backup Server on devices without a graphical interface, so the boot menu will allow installation via serial console. Let's launch the installation and connect to the virtual serial console:

cd /zroot/VMs/.iso
vm install pbs proxmox-backup-server_4.0-BETA-1.iso
vm console pbs

Select the installation via Terminal UI (serial console) and proceed normally as if it were a physical host, assigning an IPv4 address from the 192.168.0.x range (in this example, I'll use 192.168.0.3).

This way, the Proxmox Backup Server will run in a VM, with the ability to take snapshots before updates, etc., without any worries.

Once the installation is complete, PBS will reboot and listen on port 8007 of its IP. Again, as with the jails, we have two options: install a VPN system within the VM itself (thus exposing it automatically only on that VPN - generally a more secure operation) or expose port 8007 on the server's public IP.

In the latter case, add the relevant lines to the /etc/pf.conf file on the FreeBSD backup server:

# ...
# Redirect incoming traffic on port 8007 to the PBS VM's web interface
rdr on $ext_if inet proto tcp from any to any port = 8007 -> 192.168.0.3 port 8007
# ...
# Allow that traffic to pass
pass in inet proto tcp from any to any port 8007 flags S/SA keep state

Reload the pf configuration:

service pf reload

The PBS VM configuration is complete. If you chose to use the PBS's internal disk as a repository, no further operations are necessary (other than the normal repository creation, etc., within PBS).

In this case, however, we will use a different approach.

Creating a Minio Jail as a Data Repository for PBS

This approach, in my opinion, has a number of important advantages. The first is that Minio will run in a dedicated jail on the host, at full performance, and will store the data directly on the physical ZFS datapool, thus removing any other layer in between. This jail could potentially be moved to other hosts (by connecting PBS and the jail via VPN or public IP), made redundant thanks to all of Minio's features, etc. Another solution I am successfully testing (in other setups) is SeaweedFS.

Let's create a dedicated jail with Minio and put it on the bridge, so that PBS can access it on the LAN.

bastille create -B minio 14.3-RELEASE 192.168.0.11/24 bridge0

If not configured directly, BastilleBSD will use the host's gateway for the jail, which is incorrect in this case. So let's go modify it and restart the jail. Enter the jail with:

bastille console minio

And modify the /etc/rc.conf file to have the correct gateway (following the example addresses):

# ...
ifconfig_vnet0=" inet 192.168.0.11/24 "
defaultrouter="192.168.0.1"
# ...

Exit the jail and restart it:

bastille restart minio

Enter the jail and install Minio:

bastille console minio
pkg install -y minio

Minio is already able to start, but PBS, even on the LAN, wants an encrypted connection. Fortunately, there's a handy tool that can generate the certificates for us:

# Download the certgen tool
fetch https://github.com/minio/certgen/releases/latest/download/certgen-freebsd-amd64

# Make it executable and run it for our jail's IP
chmod a+rx certgen-freebsd-amd64
./certgen-freebsd-amd64  -host "192.168.0.11"

# Create the necessary directories and set permissions
mkdir -p /usr/local/etc/minio/certs
cp private.key public.crt /usr/local/etc/minio/certs/
chown -R minio:minio /usr/local/etc/minio/certs/

Let's view the certificate's fingerprint. Since it's self-signed, we'll need it for PBS later. For security reasons, PBS will ask for the fingerprint of non-directly verifiable certificates. Run the following command and take note of the result:

openssl x509 -in /usr/local/etc/minio/certs/public.crt -noout -fingerprint -sha256

At this point, enable and configure Minio in /etc/rc.conf. WARNING: The username and password (access key and secret) used in this example are insecure and for testing purposes only. It is strongly recommended to use different values:

# Enable Minio service
minio_enable="YES"
# Set the address for the Minio console
minio_console_address=":8751"
# Set the root user and password as environment variables
minio_env="MINIO_ROOT_USER=testaccess MINIO_ROOT_PASSWORD=testsecret"

Start Minio:

service minio start

If everything went correctly, Minio is now running (with its certificates) and ready to receive connections.

It's now time to create the bucket(s) that PBS will use. There are several ways to do this, but to test that everything is working and to configure PBS, I suggest connecting via an SSH tunnel.

# Create an SSH tunnel from your local machine to the backup server
# Port 8007 is forwarded to the PBS web UI
# Port 8751 is forwarded to the Minio console
ssh user@backupServerIP -L8007:192.168.0.3:8007 -L8751:192.168.0.11:8751

This way, we'll create a tunnel between the FreeBSD backup server and our workstation, mapping 127.0.0.1:8007 to 192.168.0.3:8007 (the PBS web interface) and 127.0.0.1:8751 to 192.168.0.11:8751 (the Minio console port).

Now, connect to https://127.0.0.1:8751, enter the credentials specified in /etc/rc.conf, and create a bucket.

Once the bucket is created, you can configure PBS to use it. Connect to PBS via https://127.0.0.1:8007 and go to S3 Endpoints. Set a name, use 192.168.0.11 as the IP and 9000 as the port, enter the access and secret keys, and the certificate fingerprint we generated earlier. Enable "Path Style" or it will not work.

Then go to Datastores and add it, as you would for any other S3 datastore, by specifying the created bucket and a local directory where the system will keep its cache.

If everything was set up correctly, PBS will create its structure in the bucket, and from that moment on, you can use it. Always keep in mind that this is still a "technology preview", so there may be issues, but from my tests, it is sufficiently reliable.

Taking Local Snapshots of Backups

One of the most common techniques used in ransomware attacks is to also delete or encrypt backups. They often use automated methods, relying on the fact that many (too many!) consider a "backup" to be a simple copy of files to a network share. However, it's not impossible that, in specific cases, they might compromise the machine and connect to the backup server. This is nearly impossible with a "pull" type backup (like the one managed by zfs-autobackup) but is still possible with the "push" approach, which involves using BorgBackup or similar tools.

This happened to one of my clients once - in that case, the problem originated internally, from an employee who wanted to cover up his mistake, inadvertently creating a disaster - but that will be material for another post.

Fortunately, the client had a system that solved the problem: thanks to ZFS, we can have local snapshots on the backup server, which are invisible and inaccessible to the production server. Since we have already installed zfs-autobackup, it's easy to use it for this purpose as well. I've already talked about this in a previous article and won't rewrite the steps here. Just consult that article, keeping in mind that in this case, it's not advisable to snapshot all the datasets on the backup server (the space would grow exponentially), but only those at risk. In the cases analyzed in this post, this applies only to the push part, as PBS will also be accessible only from the Proxmox servers and not from the VMs they contain. If, in this case too, you don't trust those who manage the Proxmox servers, just set up snapshots for the Minio jail as well.

Conclusion

This long post aims to analyze, in a general way, how I believe one can manage reasonably secure backups of their data. Obviously, there are many variables, additional precautions, possible optimizations, hardening, etc., that must be studied on a case-by-case basis. There are old rules, new rules, old and new philosophies. Recently, many people who have embraced the cloud have often stopped thinking about backups, only to realize it when something happens and the data has, indeed, vanished... into the clouds.

In this post, I have generically covered the setup of the backup server, and this demonstrates how FreeBSD, thanks to its features, can be considered an ideal platform for this type of task.

In the next articles in this series, I will examine the client side, i.e., how to structure them for a sufficiently reliable backup, and how to monitor everything - because I've seen this too: people resting easy because the backup was supposedly running every night, but in fact, the backup had been failing every night for more than 4 years.

Stay Tuned and stay...backupped!

New Article on BSD Cafe Journal: WordPress on FreeBSD with BastilleBSD

Stefano Marinelli — Mon, 21 Jul 2025 09:30:00 +0200

New Article Published

I'm excited to announce that I have published a new, in-depth article on the BSD Cafe Journal: "WordPress on FreeBSD with BastilleBSD: A Secure Alternative to Linux/Docker".

This piece explores how to create a robust and secure WordPress installation on FreeBSD using BastilleBSD, leveraging the power and isolation of FreeBSD jails as a compelling alternative to the more common Linux and Docker stack.

Future Technical Content

I'm excited to announce that I'm expanding my writing to a new platform! From now on, some of my more technical, long-form articles and tutorials will be published on The BSD Cafe Journal, a fantastic hub for BSD-related content that I'm happy to now contribute to.

This new collaboration complements the work I do here. My personal blog will continue to be my home base, and you won't miss a thing! I'll still be posting my own articles and announcements right here, and I'll always include a direct link to any new content I publish elsewhere. This space will remain as active as ever.

Thank you for reading

Make Your Own Backup System – Part 1: Strategy Before Scripts

Stefano Marinelli — Fri, 18 Jul 2025 09:00:00 +0200

Backup: Beyond the Simple Copy

For as long as I can remember, backup is something that has been underestimated by far too many people. Between flawed techniques, "Schrödinger's backups" (i.e., never tested, thus both valid and invalid at the same time), and conceptual errors about what they are and how they work (RAID is not a backup!), too much data has been lost due to deficiencies in this area.

Nowadays, backup is often an afterthought. Many rely entirely on "the cloud" without ever asking how - or if - their data is actually protected. It's a detail many overlook, but even major cloud providers operate on a shared responsibility model. Their terms often clarify that while they secure the infrastructure, the ultimate responsibility for protecting and backing up your data lies with you. By putting everything "in the cloud", on clusters owned by other companies, or on distributed Kubernetes systems, backup seems unnecessary. When I sometimes ask developers or colleagues how they handle backups for all this, they look at me as if I'm speaking an archaic, unknown, and indecipherable language. The thought has simply never crossed their minds. But data is not ephemeral; it must be preserved in every way possible.

I've always had a philosophy: data must always be restorable (and as quickly as possible), in an open format (meaning you shouldn't have to buy anything to restore or consult it), and consistent. These points may seem obvious, but they are not.

I have encountered various scenarios of data loss:

Datacenters destroyed by fire – I had 142 servers there, but they were all restored in just a few hours.
Server rooms flooded.
Servers destroyed in earthquakes, often due to collapsing walls.
Increasing incidents of various ransomware attacks.
Intentional damage by entities seeking to create problems.
Mistakes made by administrators, which can happen to anyone.

The risk escalates for servers connected to the internet, like e-commerce and email servers. Here, not only is data integrity crucial, but so is the uninterrupted operation of services. This series of posts will revisit some of my old articles to explain my core ideas on the subject and describe, at least in part, my primary techniques.

Many consider a backup to be a simple copy. I often hear people say they have backups because they "copy the data", but this is often wrong and extremely dangerous, providing a false sense of security. Copying the files of a live database is an almost useless operation, as the result will nearly always be impossible to restore. It is essential to at least perform a proper dump and then transfer that file. Yet, many people do this and will only realize their mistake when they face an emergency and need to restore.

The Backup Plan: Asking the Right Questions

Before touching a single file, you must start with a plan, and that plan starts with asking the right questions:

"How much risk am I willing to take? What data do I need to protect? What downtime can I tolerate in case of data loss? What type and amount of storage space do I have available?"

The first question is particularly critical. A common but risky approach is to store a backup on the same machine that requires backing up. While convenient, this method fails in the event of a machine failure. Even relying on a classic USB drive for daily backups is not foolproof, as these devices are as susceptible to failure as any other hardware component. And contrary to popular belief, even high-end uninterruptible power supplies (UPS) are not immune to catastrophic failures.

Thus, the initial step is to establish a management plan, balancing security and cost. The safest backup is often the one stored farthest from the source machine. However, this approach introduces challenges related to space and bandwidth. While local area network (LAN) backups are relatively straightforward, off-network backups involve additional connectivity considerations. This might lead to a compromise on the amount of data backed up to maintain operational speed during both backup and recovery processes.

Safety doesn't always equate to practicality. For instance, with a 200 MBit/sec connection and 2 TB of backup data, the recovery time could be significant. However, if the goal is not rapid restorability but simply ensuring the data is available, the safest backup is often the one closest to us. That is, a backup we can "touch", disconnect, and consult even when offline.

Therefore, it is essential to develop a backup policy tailored to specific needs, keeping in mind that no 'perfect' solution exists.

The Core Decision: Full Disk vs. Individual Files

When planning a backup strategy, one key decision is whether to back up the entire disk or just individual files. Or both of them. Each approach has its pros and cons.

Entire Disk (or Storage) Backup

Advantages:

Complete Recovery: Restoring a full disk backup can quickly revert a system to its exact previous state, boot loader included.
Integration in Virtualization Systems: If your VM is a single file on a filesystem like ZFS or btrfs, you can simply copy that file (after taking a snapshot) to get a complete backup of the VM. Solutions like Proxmox offer easy management of full disk backups, accessible via command line or web interface.
Flexibility in Virtual Environments: Products like the Proxmox Backup Server offer the ability to recover individual files from a full backup, combining the benefits of both approaches.

Disadvantages:

Downtime for Physical Machines: Often, it's necessary to shut down the machine to create a full disk backup, leading to operational interruptions. A hybrid approach, if the physical host is running FreeBSD for example, is to take a snapshot and copy all the host's datasets. The restore process, however, will require some manual operations.
Large Space Requirements: Full disk backups can consume substantial space, including unnecessary data.
Potential Slowdowns and Compatibility Issues: The backup process can be slow and may encounter issues with non-standard file system configurations.

Individual File Backup

While it might seem simpler, backing up individual files can get complicated.

Advantages:

Basic Utility Use: Possible with standard system utilities like tar, cp, rsync, etc.
Granular Backups: Allows for backing up specific files and comparing them to previous versions.
Delta Copying: Only modified parts of the files are backed up, saving space and reducing data transfer.
Portability and Partial Recovery: Files can be moved individually and partially restored as needed.
Compression and Deduplication: These features are often available at the file or block level.
Operational Continuity: Can be done without shutting down the machine.

Disadvantages:

Storage Space Requirements: Simple copy solutions might require significant storage.
Need for File System Snapshot: For efficient and consistent backups, a snapshot (like native ZFS snapshots, btrfs, LVM Volume snapshots, or Microsoft's VSS) is highly recommended before copying.
Hidden Pitfalls: Issues may not become apparent until a backup is needed. And by then, it may be too late.

The Key to Consistency: The Power of Snapshots

Backing up a "live" file system involves a "start" and an "end" moment. During this time, the data can change, leading to fatal inconsistencies. I've encountered such issues in the past: a large MySQL database was compromised, and I was tasked with its recovery. I confidently took the client's last file-based backup and restored various files (not a native dump). Unsurprisingly, the database failed to restart. The large data file had changed too much between the start and end of the copy, rendering it inconsistent. Fortunately, I also had a proper dump, so I managed to recover from that.

The issue is evident: backing up a live file system is risky. An open database, even a basic one like a browser's history, is highly likely to get corrupted, making the backup useless.

The solution is to create a snapshot of the entire file system before beginning the backup. This approach freezes a consistent "point-in-time" view of the data. To date, using snapshots, I have managed to recover everything.

Here are the methods I've explored over the years:

Native File System Snapshot (e.g., BTRFS or ZFS): If your file system inherently supports snapshots, it's wise to use this feature. It's likely to be the most efficient and technically sound option.
LVM Snapshot: For those using LVM, creating a snapshot of the logical volume is a viable approach. This method can lead to some space wastage and, while I still use it, has occasionally caused the file system to freeze during the snapshot's destruction, necessitating a reboot. This has been a rare but recurring issue across different hardware setups, especially under high I/O load.
DattoBD: I've tracked this tool since its inception. I used it extensively in the past, but I sometimes faced stability issues (kernel panics or the inability to delete snapshots, forcing a reboot). For snapshots with Datto, I often use UrBackup scripts, which are convenient and efficient.

The Architecture: Push or Pull?

A longstanding debate among experts is whether backups should be initiated by the client (push) or requested by the server (pull). In my view, it depends.

Generally, I prefer centralized backup systems on dedicated servers, maintained in highly secure environments with minimal services running. Therefore, I lean towards the "pull" method, where the server connects to the client to initiate the backup.

Ideally, the backup server should not be reachable from the outside. It should be protected, hardened, and only be able to reach the setups it needs to back up. The goal is to minimize the possibility that the backup data could be compromised or deleted in case the client machine itself is attacked (which, unfortunately, is not so rare).

This is not always possible, but there are ways to mitigate this problem. One way is to ensure that machines that must be backed up via "push" (i.e., by contacting the backup server themselves) can only access their own space. More importantly, the backup server, for security reasons, should maintain its own filesystem snapshots for a certain period. In this way, even in the worst-case scenario (workload compromised -> connection to backup server -> deletion of backups to demand a ransom), the backup server has its own snapshots. These server-side snapshots should not be accessible from the client host and should be kept long enough to ensure any compromise can be detected in time.

My Guiding Principles for a Good Backup System

Over the years, I've favored having granular control over backups, often finding the need to recover specific files or emails accidentally deleted by clients. A good backup system, in my opinion, should possess these key features:

Instant Recovery and Speed: The system should enable quick recovery and operate at a high processing speed.
External Storage: Backups must be stored externally, not on the same system being backed up. Still, local snapshots are a good idea for immediate rollbacks.
Security: I avoid using mainstream cloud storage services like Dropbox or Google Drive for primary backups. Own your data!
Efficient Space Management: This includes features like compression and deduplication.
Minimal Invasiveness: The system should require minimal additional components to function.

Conclusion: What's Next

The approach to backup is varied, and in this series, I will describe the main scenarios I usually face. I will start with the backup servers and their primary configurations, then move on to the various software and techniques I use.

But that will begin with the next post, where I'm talk about building the backup server which, of course, will be powered by FreeBSD - like all my backup servers for the last 20 years.

How to install FreeBSD on providers that don't support it with mfsBSD

Stefano Marinelli — Wed, 02 Jul 2025 09:30:00 +0200

FreeBSD is an extremely powerful operating system. The ability to isolate services in jails and, thanks to ZFS, the simplicity with which you can create snapshots (both local and remote) make it a perfect system for increasing peace of mind, especially when running many workloads.

Many providers, blinded by the success and large numbers achieved by Linux distributions, have decided to no longer support FreeBSD in their installers. While understandable (they might not have staff experienced with systems other than Linux), this can cause problems for those who want to try using something different. And yes, it makes perfect sense, even just to avoid IT monocultures, which are extremely harmful even in the medium term.

There's an extremely powerful tool that, in my opinion, deserves much more attention than it gets. The tool is called mfsBSD. Using the author's words: "This is a set of scripts that generates a bootable image (and/or ISO file), that creates a working minimal installation of FreeBSD (mfsBSD) or Linux (mfslinux). It is completely loaded into memory."

mfsBSD works intelligently: it can be launched both via UEFI and via "traditional" boot since it has both boot modes enabled. It gets an IP address via DHCP (an operation that works with most providers) and opens an SSH server (with a preset password - so it's advisable to connect immediately and change it, to prevent someone else from doing it for you).

This means that mfsBSD can be used both when you have a console available and, in many cases, without a console, since you'll just need to connect via SSH and start with the traditional installation.

To install FreeBSD using mfsBSD, you just need to follow some very simple steps: all providers, in fact, offer the ability to boot in Linux "rescue mode", generally based on a sufficiently recent version. Set your server (whether physical or VPS, it doesn't matter) in rescue mode and reboot. Once active, connect via SSH (or open a console) to the server in rescue mode.

Now it's sufficient to download the mfsBSD image (for example, from here: https://mfsbsd.vx.sk/files/images/ - in my case, I usually choose the normal image, which at the time of writing this post is https://mfsbsd.vx.sk/files/images/14/amd64/mfsbsd-14.2-RELEASE-amd64.img). At this point, it needs to be written with dd directly to the server's disk (or, if in doubt, disks). For example:

dd if=mfsbsd-14.2-RELEASE-amd64.img of=/dev/sda bs=5M conv=sync

In this case I wrote "sda", but if you had one or more NVMe drives, the correct device would be /dev/nvme0n1 for the first disk, /dev/nvme1n1 for the second, etc.

Reboot. If you have a console, you'll see mfsBSD boot up, enable the SSH server, and position itself at the login. Otherwise, ping the server's IP until it starts responding. At that point, it's sufficient to connect via SSH as the root user.

The root password is mfsroot

As the first thing, change the root password with the passwd command. This is to prevent someone from entering and compromising the machine during the installation time.

Now launch the bsdinstall command and proceed with the normal FreeBSD installation (also setting up mirrors, RAID, etc., if desired), keeping in mind that mfsBSD is running in RAM, so you can overwrite the disk it was installed on without problems.

After the reboot, you'll have your FreeBSD system installed and running.

Vibe Coding Will Rob Us of Our Freedom

Stefano Marinelli — Thu, 05 Jun 2025 08:55:36 +0200

It was one of those Friday afternoons when everyone just wanted to go home, but their PM had that look we all know too well. I manage the infrastructure for this mid-sized e-commerce client, so I get to see their development process from the server side. Their deadline was, as it often is, completely insane.

"Alex", a junior developer on their team - I've been watching his commits for about six months now - got stuck with this backend task. I watched him struggle with it for a day, then suddenly, boom. Using an AI assistant, he churned out what seemed like working code in half the expected time. Their management was thrilled.

Seeing how quickly it came together, I suggested they double-check the code before deployment – I'd rather avoid any potential headaches down the line. But with everything "working" perfectly, they considered it a waste of time. The vibe was right. So, into production it went.

That dismissal didn't sit right with me. So, a few days later, I decided to check the code myself. On the surface, it looked clean, even polished. But then I saw that line. A single, innocuous-looking database query.

The AI had constructed it in a way that was wide open to a classic SQL Injection attack.

Alex's tests, done with "normal" fake user data, passed perfectly. But a malicious actor could have wiped their entire user database with a single, cleverly crafted request. The code worked, but it was a ticking time bomb sitting on my servers. And Alex, who had trusted the tool, had no idea.

Now, before anyone thinks I'm throwing Alex under the bus - absolutely not. From what I can see, he's a smart developer dealing with impossible deadlines. I've seen this pattern with other clients too. He's just a symptom of a much larger, more insidious trend I'm calling "Vibe Coding".

It's this methodology (if we can call it that) where developers, pressured by deadlines, are no longer trained on code structure, but on the "vibe" – that is, on giving the right prompts to AIs and testing only if the output seems to work.

Alex isn't just a case of insecure code. He's proof of how we're becoming dependent on tools we don't control.

We're shifting from being architects to being interior decorators.

An architect understands the foundations, the structural integrity, the load-bearing walls. A decorator can make a room look good, but has no idea if the entire building is about to collapse.

And here's what really scares me. I can already see the future: AIs will require ever-increasing computing power, and developers will no longer be trained on code, but on prompting. It's not hard to imagine where this leads. To the complete loss of programming skills and, in turn, to the complete loss of the ability to write code not controlled by the big players.

Indeed, it wouldn't be difficult for them to force the use of specific languages or to produce specific outcomes. In the end, the program works, and the "vibe coder" will no longer be able to understand what it's really doing.

It works → it can go to production. End of story.

Many developers are terrified of losing their jobs for this very reason: AIs sometimes program better than them. And, in my opinion, they are right to be afraid. But I'm more afraid of a world (and not just in IT) where code will depend exclusively on the companies that sell us AIs.

Today, writing code is something free, potentially doable even on a beat-up laptop. But tomorrow? Will we be completely dependent on AIs (even) for this?

As Serena Sensini rightly argued at OSDay 2025 - and her talk really stuck with me - the point is not to let ourselves be replaced by AIs, but to use them to improve ourselves and our productivity.

We must use these tools to help developers like Alex become true architects, not just decorators who are skilled with a new kind of brush. We must always maintain our skills and our irreplaceability.

Because the day we stop writing code, we will stop being free and independent.

The Day GlusterFS Tried to Kill My Career

Stefano Marinelli — Wed, 21 May 2025 12:55:00 +0200

I was at a client, a healthcare facility, to replace some hard drives. The official line was a familiar one: no budget for significant upgrades. This meant we had to keep the current setup running, a system that was both outdated and, frankly, unreliable by modern standards. We were holding everything together with metaphorical duct tape and sheer willpower, but somehow, it was stable and functional. I had managed to set up a makeshift "cluster" using OpenNebula with GlusterFS for storage (which I later, through hard lessons, replaced with MooseFS and eventually Ceph), squeezing every last drop of performance from the available hardware.

The IT manager, a decent fellow, seemed to genuinely believe the purse strings were just exceptionally tight. However, another company, known to be friends of the general manager, were already providing support on two specific VMs. Their presence felt disproportionate to their limited scope, and there was a subtle, persistent pressure from the GM to involve them more, a context that would only become clear later.

We scheduled an intervention and meticulously notified everyone to disconnect and shut down their machines by 12:30. By 13:00, we had completed the necessary backups, aiming to start the core intervention by 14:00 and have the essential systems back up and running by 15:30. The goal was straightforward: update the underlying systems and check the health of the disks. We shut down all the VMs; everyone disconnected. It was lunchtime.

We updated the servers, rebooted them. And then, one of the disks started throwing errors. GlusterFS, for reasons I never fully pinned down but have strong suspicions about, decided in that critical moment to overwrite both the failing disk and its supposed replica with zeros. I hadn’t changed a single configuration file related to it. From that day forward, GlusterFS, for me, simply ceased to exist as a viable option.

Panic. Pure, cold panic – there were backups, of course, but on a USB 2.0 disk (these were old servers, no USB 3.0 in sight)! The restore time would be agonizingly slow. I immediately stopped everything, feeling like I was about to faint. The IT manager, seeing my face, didn’t immediately grasp what had happened, so I quickly explained the catastrophic data wipe on that specific storage volume. He composed himself and announced to the waiting staff that we would cancel the remainder of the planned intervention, restore from backup, and still aim to have the critical services back up by 15:30 as originally planned, prioritizing the most essential VMs.

Just as we were about to begin the painstaking restore, the "competing" company – the GM’s friends – reached out. They had, against all explicit instructions, powered on one of their VMs and started "doing their own interventions". And now, they were complaining that we had lost some of their data.

This was the moment the general manager and those guys went full "carpe diem". They immediately put me under accusation, loudly proclaiming I had undoubtedly made a critical error that led to this "lost data". It was then the penny truly dropped for the IT manager, and the GM's earlier insistence on involving his 'friends' company took on a sinister new light. Now I can say it: his primary goal hadn't just been about perceived cost-saving; it was to systematically give work to this company, to hand everything over to them, paving the way for them to take over the entire infrastructure, regardless of competence or genuine need.

I wrote a detailed technical report explaining exactly what I had observed, noting various unauthorized SSH logins from the IP addresses associated with those individuals during our clearly demarcated intervention window. Conveniently, the command history on the affected systems had been wiped clean. Of course, not by me.

They continued to harass me for a while. The last, most audacious thing they asked was for me to attend a meeting "to explain in person". They deliberately tried to schedule this meeting for the day before my wedding. And they knew it.

They threatened to demand unspecified (and undoubtedly high) financial compensation for "the lost data". What lost data, you ask? The data they claimed to have entered onto their VMs during our intervention window – the very window they'd been explicitly warned to stay out of and during which they had no business touching the systems.

Final result: no tangible problem for me. I hadn’t done anything wrong. The backups, though slow to restore, were intact and complete. They only calmed down when I proved (with network logs and firewall records in hand) that I wasn’t the only one connected to those machines during the critical period, and my witnesses (two colleagues present with me and the IT manager himself) had seen all my actions and could confirm I hadn’t deviated from standard, careful procedure.

In the end, I realized they would undoubtedly try something like this again. I made the decision to leave the client. Even the IT manager, who had initially tried to believe in the good faith of the leadership and had then witnessed firsthand the technical realities and the subsequent malicious accusations, decided he’d had enough and resigned to change jobs. He understood the financial and political interests at play were far beyond his control.

Unsurprisingly, the general manager managed to install the company he wanted, at astronomical figures. After just two months, true to form, they got their hands fully on the system I had painstakingly kept alive… and broke it. They actually had the gall to ask me for assistance, which I refused. At any price.

After a few years, I found out through the local press that the general manager had ended up in jail for corruption, bribes, and for systematically favoring his friends' companies across many sectors, not just IT. Don't ask me why, but I wasn't surprised at all.

That day, I celebrated. It was a stark reminder that while integrity might not always win the immediate skirmishes in such environments, the rot of corruption often consumes itself in the end. But the cost, for those caught in the interim, can be immense.

The Server That Wasn't Meant to Exist

Stefano Marinelli — Tue, 13 May 2025 16:13:36 +0200

Yesterday I read a piece of news that brought back an important - and painful - episode from my career.
A story about trust, technology... and the kind of problems that can't always be solved.

About 16 years ago, I was contacted by an old friend. He was worried about a situation involving some mutual acquaintances.
To keep it short: an entrepreneur - administrator and owner of several companies - had died suddenly.
He was the kind of man who centralized everything, and his wife and children found themselves struggling to manage things.
One of the sons decided to cash out and leave the family business (focusing on his own career), while the others chose to stay involved in day-to-day operations.
The wife, elderly and retired for years, ended up at the helm, but she was clearly out of her depth.

The main issue was the complete lack of information flow: no digital systems of any kind were in place.
Employees had their own PCs (sometimes even personal laptops), and there was zero control over anything.
All the accounting and administrative data were scattered across individual machines, often taken home at the end of the day.
From the owners’ perspective, all they saw was a huge cash flow coming in - yet the accounts were always in the red.

"If we keep going like this, we’ll be bankrupt in just a few years", I was told.
What I could do was set up a proper IT system, structured to make data management transparent and traceable.

I planned - and got immediate approval for - the purchase of routers, switches, various networking devices and a server with several disks.
The OS of choice, as was my habit at the time, was NetBSD. Thanks to XEN, I set up multiple VMs.
One handled the NAS duties (using Samba, so PCs could connect and store files directly there), another ran Archivista.
I even worked on translating Archivista’s interface into Italian, since it wasn’t yet localized, just to make it easier for users.

As usual in those days, I added a caching proxy (Squid) and a content filter (DansGuardian), to ensure proper usage.
The internet connection was very slow and often collapsed under heavy load - mostly recreational use, as logs revealed.
There was no supervision, and many people were downloading movies and such on company time.

As often happens, not everyone was happy.

One figure in particular - the late owner's former right-hand man - opposed the new system in every possible way.
According to him, none of this was necessary. But the real alarm bell had been his sudden change in lifestyle.
He’d made purchases that didn’t remotely align with his salary.
At the time, the company had no oversight and dealt with a lot of cash. It was all technically legal, especially given the nature of the business. I won’t go into detail - privacy matters.

Once everything was up and running, we trained the employees to use the new system.
Most were thrilled - finally able to work properly, with files in the right place and centralized document management.
OCR, archiving, etc. were all seen as major time-savers and a big boost in efficiency.
Some of the accounting staff remained skeptical, of course.

Since all of this was far from where I lived at the time, I went back to my life once the system was stable and everything in place.
A few quiet days passed - then one morning, my phone rang:

"Good morning, this is XYZ - I handle some technical aspects of a software suite used by the company where you've just installed everything.
We need to install our software, and I understand you set up the server. I’ll need the full server diagram and all the admin passwords".

I explained that it wasn’t a Windows machine, as he assumed (without even having seen it), but NetBSD, running NetBSD and Linux VMs.

A few seconds of silence.

"I see. Then I’ll have to wipe it and install Windows. I need Windows, and I don’t have time to wait for a new server - I’ll proceed tomorrow morning".

I froze. I told him that was not possible - the entire workflow now depended on that machine, and erasing it would be catastrophic.

"I’ll speak with the owners", I said, "and I’m sure they’ll provide you with a separate server within hours".

No use. He started to backpedal.
To my (young) eyes, the goal was now obvious: that server had to disappear, and fast.
He said he would "restore the previous situation", and claimed the server couldn’t remain as-is because he needed it.

I immediately called the owners. Sadly, due to inexperience and inability to handle the situation, they panicked.
They asked me to consider letting him do it, and then redoing the setup later, covering the cost of new hardware and my time.
I refused. I was young, but I already had this mindset: do what’s right, even at the cost of profit.
This was clearly a maneuver to eliminate controls - the server, the centralized filesystem.
The goal was to hide the real accounting data from owners and auditors. Thousands of euros vanished every day through "transactions".
The owners had started to understand, and this new pressure confirmed just how rotten things really were.

I called that man back and told him clearly: the server I’d built wasn’t to be shut down.
If he needed one, I’d deliver a new server by that evening, just for him.

At that point, he finally spoke more openly:

"You don’t get it, do you? I need that server. Not a server.
You’d better go along with this — or you’ll have serious trouble working in this area again".

And other similar, "nice" sentences. I replied calmly:
"Look, I’m just doing a favor for some friends. I don’t have clients in your area — and I don’t want any.
I’d rather do a good job for them than gain new clients".

No way through. He kept pushing - confident in his own sense of "power" - until I said what I had been trying to avoid.

Because I had recognized who he was.
And the disappointment hit me twice as hard - I used to admire him.
He, however, had not recognized me.

"Excuse me, but why are you talking to me like this? You’ve known me since I was a child. Don’t you remember? I’m the nephew of..."

He froze.
He understood immediately.
He connected the dots and knew full well that a single phone call to someone extremely close to me - someone he owed a great deal, both personally and professionally - would have the opposite effect he was aiming for.

That person had helped him greatly over the years. So much so that he folded:

"Oh... I’m so sorry... I didn’t recognize you. I’ll find another solution. Sorry again".

He hung up.
Never heard from him again.

I informed the owners that the issue was resolved (leaving out most of the details), but it didn’t last long.
Within days, a series of "unfortunate events" hit the server: the UPS failed, the server was "accidentally" unplugged and plugged back in incorrectly, and finally... it stopped responding on the network.

It was dead.
And when we opened it, the hard disks were just... gone.

But there was one thing nobody (but the owners) knew.
The server - slowly but surely - had been backing up externally.
All the data up to that point had been copied to a device we’d quietly installed at the owners’ home: a tiny PCEngines Alix, running NetBSD with two USB drives.
It was slow, yes - slow hardware, slow disks - but reliable.
That very device still works today (with FreeBSD) and provides services elsewhere.

I handed all the data to the owners and asked what they intended to do.
They took some time - days, then weeks.
Eventually, they said they’d probably investigate whether there were grounds for a theft report.

I never heard more about it.

But then came a tempting offer:

"Come work for us. Manage our network infrastructure and help us overhaul our internal procedures.
Even if you’ve just bought a house far from here, even if you’d have to leave your other clients -
we’ll pay you enough to forget everything else. Name your price".

They would’ve done it, too.
Our mutual friend urged me:
"They’ve got a huge cash flow, but too many people are taking advantage of them due to lack of control.
Take the job - they’ll treat you like gold, and you’ll really help them".

I didn’t think twice.

I turned it down.

I like my work.
I like doing what I do - and the income is a consequence, not the cause.
I would’ve had to give up my life, my path, to fight battles I don’t enjoy - and that I might not even win.

Because sometimes, dishonest people do win.

I’ve never regretted declining that offer.
I lost touch with all of them years ago, but I later heard things went as I predicted:
the owners gradually backed out.

They made another request later on, which I tried to fulfill — but even that was blocked, just when everything was ready.

At some point, I had to walk away.
Not because I wanted to abandon them in a time of need, but because they weren’t giving me the tools to do what was necessary.
They were so overwhelmed, so unprepared, that they ended up yielding to pressure - often from the very people who were hurting them.

And of course, I’ve left out the worst parts of the story.

(Author's note: Many readers, understandably struck by the severity of the events, have speculated about the involvement of organized crime. I want to clarify that, while the situation was extremely problematic and dishonest, that wasn't the case. The "worst parts" I alluded to referred to other internal dynamics, abuses of trust, and improprieties that I prefer not to detail further for privacy reasons and to avoid weighing down the narrative.)

That’s when I realized:
Some situations are so rotten, they simply can’t be salvaged.

And that’s okay.

I solve problems - it’s what I do best.

But I can’t solve every problem.
Especially not when those involved choose to protect the problem instead of fixing it.

Make Your Own Internet Presence with NetBSD and a 1 euro VPS – Part 1: Your Blog

Stefano Marinelli — Tue, 22 Apr 2025 07:30:36 +0200

Why NetBSD?

For many years, I've been using (and appreciating) NetBSD because it's stable, efficient, and reliable. The codebase has proven its reliability, running without reboots for years without issues. It supports ZFS (though differently than FreeBSD), LVM (useful for those accustomed to it on Linux), the ability to take filesystem snapshots (UFS2, making ZFS less crucial), and it's an excellent virtualization platform. Installation and updates are easy (including via sysupgrade - which I'll cover in a future article). Since it focuses on portable and optimized code (running on ancient architectures requires cleanliness and correctness), it's particularly efficient on low-power devices, like embedded systems or cheap VMs. Therefore, it's one of the best solutions for a small personal setup that can still deliver excellent results and simple management.

Indeed, the market offers very cheap VPS, often with just a single core and little RAM. But a modern single core packs power that a multi-core from just a few years ago could only dream of, and often, the I/O of these machines (a bottleneck for many services) is still decent. I personally use 1 euro per month VPS (VAT included - for those not subject to it, that's less than one euro per month!) with a public IPv4 address and (often) a /64 IPv6 block, ensuring full reachability across the entire network. I'm not providing direct links as I have no affiliations, but netcup's "piko" VPS are among the types I use most often (a 4 euro/month netcup VM handles the entire FediMeteo project), and this type of VM is ideal for our purpose because some providers (like netcup) allow you to upload your own ISO and install your preferred operating system. On VPS like these, I've installed everything - including OmniOS and SmartOS - without problems. And even such a small VPS, with an efficient operating system, can be extremely satisfying.

Why BSSG?

In this article, I'll describe how to create and publish a blog using BSSG as it exemplifies my concept of portability and minimalism. BSSG on NetBSD currently doesn't leverage parallelism provided by tools like GNU Parallel, but for small to medium-sized blogs, this won't be an issue, especially considering these small VMs only have 1 core. Obviously, you can use any Static Site Generator (SSG) (like Hugo, Nikola, 11ty, Pelican, Zola, etc.) - the important thing is to have a static site served by a simple web server.

Let's Start with the Installation

Installing NetBSD is quite straightforward and is clearly covered, complete with explanatory screenshots, in the excellent official NetBSD documentation, which I recommend using as a reference during the process, especially if it's your first time.

In my case, I made sure to use the proposed disk geometry, use the standard automatic partitioning, but enable the "log" and "noatime" options for the filesystem. Both these options will provide a huge advantage in I/O operations, especially with BSSG, as the first enables journaling and the second prevents updating file metadata on every access. BSSG is more I/O bound than CPU bound, so any optimization is beneficial.

Moving forward, I also recommend configuring the network (although installation can be done from packages on the installation ISO). For netcup, you can use DHCPv4 (even though it's a bit slow and sometimes seems to fail, the DHCP client will continue running in the background and eventually work).

For IPv6, I usually configure it manually later, so I'll describe that further down.

I also recommend enabling SSH, adding a regular user (and adding them to the wheel group so they can gain root privileges) - in this case, I'll call the user blog. Also, enable the installation of binary packages, as it will be convenient later to use pkgin to install and update all necessary packages. All these steps are described clearly and in detail in the guide, so I won't detail them here. But they are simple and logical, like all operations on BSD systems.

After installation, reboot. If everything went correctly, you should be able to log in via console or SSH using the "blog" user (or whatever you named it).

First, I suggest configuring the IPv6 address and installing the necessary packages.

For IPv6, in the case of netcup, simply add one of the assigned addresses to the interface. In NetBSD, network interface configurations are stored (similar to OpenBSD) in specific files. For the first virtio interface, the file will be /etc/ifconfig.vioif0.

You need to elevate your privileges to root, open that file with your preferred editor, and add the configuration to the file itself:

nb1euro$ su -l
nb1euro# vi /etc/ifconfig.vioif0

inet6 your-ipv6-addr/64
up

To test everything, perform a reboot and try pinging an IPv6 address (I often use ping6 google.com).

If all goes well, after a few seconds, you should see ping replies, confirming everything is configured correctly.

Regarding packages, the only two strictly necessary ones are bash and a markdown processor (by default, BSSG will use commonmark; otherwise, it can be configured to use pandoc or Markdown.pl). rsync can be useful for deployment. sudo (or doas) can be useful for elevating privileges for certain operations, at least at this stage.

nb1euro$ su -l
nb1euro# pkgin in bash cmark rsync sudo

If you're used to Linux, you can also install the "nano" editor:

nb1euro# pkgin in nano

If sudo was installed, it's now appropriate to grant users in the "wheel" group (like the regular user created during installation) the ability to elevate privileges. Edit the sudoers file (I suggest using the visudo command) and uncomment this line:

## Uncomment to allow members of group wheel to execute any command
%wheel ALL=(ALL:ALL) ALL

At this point, you can switch back to operating as the regular user, downloading and unpacking BSSG:

nb1euro$ ftp https://brew.bsd.cafe/stefano/BSSG/archive/0.15.1.tar.gz
nb1euro$ tar zxfv 0.15.1.tar.gz

Now that BSSG is ready, just initialize a directory with the structure for the new site:

nb1euro$ cd bssg
nb1euro$ ./bssg.sh init /home/blog/myblog

Everything is set to start generating your blog. I recommend reading BSSG's README.md. There are many options, themes, etc., but to get started, you just need to set the site's public URL. For example, if the site will be published as myblog.example.com - just create a file at /home/blog/myblog/config.sh.local (the path defined by the init command) and set the public URL:

SITE_URL="https://myblog.example.com"

This way, all URLs will be absolute URLs, which is necessary to ensure the correct functioning of RSS feeds, sitemaps, etc. This setting assumes HTTPS - if you just want to test the site over HTTP, simply use http and then, optionally, change it to https and regenerate the site later.

You can already create your first test post, directly from the BSSG directory:

nb1euro$ ./bssg.sh post

The system will use nano if it's installed, otherwise it will use vi. Don't worry, in the latter case, BSSG will write the procedure for exiting vi as the post's text 🙂

Once you save the post, BSSG will automatically generate the site. If everything went well, the /home/blog/myblog/output directory will contain the final result. We are therefore ready for the first deployment, which can be done in many different ways. I will cover three:

Using bozohttpd, present by default in NetBSD's base system. It can be used via inetd (launching an httpd process for each connection) or as a daemon. I'll describe the first option, showing in the final benchmarks how, even when used as a daemon, it remains a less performant solution.
Using nginx
Using Caddy

First, it's advisable to obtain a certificate to configure and use HTTPS. If you only want to test using HTTP, this part can be safely bypassed. For solutions 1 and 2, I'll use certbot, which is well-known to many users with Linux experience. Caddy, on the other hand, manages certificates automatically, so there's no need for other solutions and thus no need to install certbot.

nb1euro$ sudo pkgin in py313-certbot

To use bozohttpd, no further installation is necessary. At this point, the options diverge.

Using NetBSD's Integrated httpd

bozohttpd is integrated into NetBSD and, by default, can be launched directly via inetd. This solution, while not extremely efficient or scalable, is simple and requires few resources. It's fine if you expect only a few visits per day, but when used via inetd, the initial latency for each connection is tangible. It can still be useful for some tests or small deployments.

The /etc/inetd.conf file already contains the options to handle this situation:

#http           stream  tcp     nowait:600      _httpd  /usr/libexec/httpd      httpd /var/www
#http           stream  tcp6    nowait:600      _httpd  /usr/libexec/httpd      httpd /var/www

By uncommenting these two lines and restarting inetd (service inetd restart), the server will start responding to HTTP requests on both IPv4 and IPv6.

If you want to add HTTPS support, no problem. Just request a certificate via certbot and specify the webroot.

Run:

nb1euro$ sudo certbot-3.13 certonly

Choose option 2 - the one where you specify the webroot - enter the domain, and when prompted, provide /var/www/ as the webroot.

The certificate will be created. Then, modify the /etc/inetd.conf file to also include support for HTTPS, adding two lines similar to these (obviously, change the certificate paths):

https            stream  tcp     nowait:600      _httpd  /usr/libexec/httpd      httpd -Z /usr/pkg/etc/letsencrypt/live/myblog.example.com/fullchain.pem /usr/pkg/etc/letsencrypt/live/myblog.example.com/privkey.pem /var/www
https            stream  tcp6    nowait:600      _httpd  /usr/libexec/httpd      httpd -Z /usr/pkg/etc/letsencrypt/live/myblog.example.com/fullchain.pem /usr/pkg/etc/letsencrypt/live/myblog.example.com/privkey.pem /var/www

Warning: httpd will run with the permissions of the _httpd user, so make sure all certificates are readable by that user:

nb1euro# chown -R _httpd /usr/pkg/etc/letsencrypt/

Restart inetd, and the server will also respond over HTTPS.

To make your blog public, simply copy the files from the site's output directory to /var/www/ - this time using sudo to bypass permission issues:

nb1euro$ sudo rsync -avhHPx /home/blog/myblog/output/ /var/www/

The site will be immediately visible.

Using nginx

Nginx is fast and efficient, and the performance difference is noticeable (some benchmarks follow below). For an efficient setup ready for a high number of visits, it's advisable to use a web server suited for the purpose, just like nginx.

First, install nginx and the certbot plugin for nginx. This will simplify the installation and renewal of certificates:

nb1euro$ sudo pkgin in py313-certbot-nginx nginx

Copy the startup script to /etc/rc.d - as indicated by the post-installation message. In NetBSD, this operation must be done manually, but it's always pointed out:

nb1euro$ sudo cp /usr/pkg/share/examples/rc.d/nginx /etc/rc.d

Warning: If you previously used httpd from inetd following the previous solution, you must disable it in inetd.conf and restart inetd to free up ports 80 and 443.

Now you can create a virtual host for our new site.

nb1euro$ sudo vi /usr/pkg/etc/nginx/nginx.conf

and add, at the end of the file and before the final closing curly brace:

server {
        listen 80;
        # If you also have configured IPv6 support
        listen [::]:80;

        root /var/www;
        index index.html index.htm;

        server_name myblog.example.com;

        # If you want a long cache for media and css - be careful, this means that if you change to a new theme, it might not be visible immediately as the browser might still use the old cached one
        location ~* \.(jpg|jpeg|png|gif|ico|css|js)$ {
            expires 30d;
            add_header Cache-Control "public, no-transform";
        }

        location / {
                try_files $uri $uri/ =404;
        }
}

Now, it's time to configure the system to enable nginx. Just edit /etc/rc.conf:

nb1euro$ sudo vi /etc/rc.conf

and add:

nginx=YES

Now, you can start nginx:

nb1euro$ sudo service nginx start

Nginx will start listening on port 80. Generating and installing the certificate is very simple:

nb1euro$ sudo certbot-3.13 --nginx -d myblog.example.com

This command will request the certificate and install it, so nginx will already be configured to use it.

As with the previous method, to make your blog public, simply copy the files from the site's output directory to /var/www/ - using sudo to bypass permission issues:

nb1euro$ sudo rsync -avhHPx /home/blog/myblog/output/ /var/www/

The site will be immediately visible.

Using Caddy

Caddy is a convenient and all-in-one solution, efficient and fast. It's packaged for NetBSD and allows you to go online in a flash. I won't delve into the configuration because there are many tutorials (including the official ones), but you just need to install it and run it:

nb1euro$ sudo pkgin in caddy

Once installed, go to the directory you want to serve (e.g., /var/www or directly /home/blog/myblog/output) and run:

nb1euro$ sudo caddy file-server --domain myblog.example.com

Caddy will start, request the certificate, and begin serving your blog over HTTPS as well. To install Caddy as a service (i.e., with a configuration file, etc.), you can proceed similarly to how it's done on Linux. The NetBSD Caddy package doesn't include the rc.d script, but you can copy and paste one (into /etc/rc.d/caddy) from a thread posted on UnitedBSD.

Performance Comparison

I performed some performance tests on these solutions. Here are the results, on a single-core 1 euro/month VPS, from my home connection (which also has its own limitations):

NetBSD httpd via inetd:

Running 10s test @ https://myblog.example.com/
  4 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   213.52ms  173.10ms   1.11s    76.01%
    Req/Sec    12.92      9.19    50.00     75.91%
  371 requests in 10.10s, 1.39MB read
Requests/sec:     36.72
Transfer/sec:    140.65KB

These numbers are quite poor, linked to high latency caused by having to launch bozohttpd for each incoming connection.

NetBSD httpd as a daemon:

Running 10s test @ https://myblog.example.com/
  4 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    35.74ms    6.96ms 108.80ms   81.36%
    Req/Sec    18.29      9.45    50.00     70.88%
  676 requests in 10.10s, 2.53MB read
Requests/sec:     66.92
Transfer/sec:    256.32KB

Here the situation is decidedly better, but not exceptional. httpd isn't designed for high loads or performance.

Nginx as a daemon, 1 worker:

Running 10s test @ https://myblog.example.com/
  4 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    30.69ms    4.87ms  64.14ms   66.01%
    Req/Sec   379.39     65.94   464.00     90.91%
  15026 requests in 10.04s, 56.50MB read
Requests/sec:   1496.65
Transfer/sec:      5.63MB

Here we are on another level, showing truly solid performance. This type of result can handle significantly high loads without particular difficulty. The efficiency of both NetBSD and nginx pays off.

Caddy:

Running 10s test @ https://myblog.example.net/
  4 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    32.10ms    5.75ms  95.04ms   87.44%
    Req/Sec   362.74     64.29   434.00     91.67%
  14374 requests in 10.05s, 54.63MB read
Requests/sec:   1430.82
Transfer/sec:      5.44MB

Caddy shows results comparable to nginx, so the choice between them depends solely on the type of configuration you want to achieve and the experience each person has with the specific platforms.

Conclusion: Efficient Minimalism

We've seen how it's possible to create a personal, professional, and performant online presence with minimal investment. This solution, based on NetBSD and a 1€/month VPS, offers several advantages:

Negligible Cost: For 12€ per year, you can have a website (and more!) completely under your control.
Surprising Performance: As demonstrated by the benchmarks, excellent performance can be achieved even with limited resources (up to 1400-1500 requests/second with nginx or Caddy).
Security and Stability: NetBSD is renowned for its reliability and security, fundamental characteristics for any online service.
Total Control: Unlike free blogging platforms, you have full control over every aspect of your site.
Learning Experience: Managing a BSD system allows you to acquire valuable system administration skills.

This minimalist configuration demonstrates that you don't need to invest in expensive cloud solutions or oversized VPS to have a quality online presence. In an era where the tendency is to think "moooar powaaaar = better results", NetBSD reminds us that efficiency and good design can yield excellent results even with limited resources.

After all, you don't need a thousand-node cloud to write something worth reading.

In the upcoming articles in this series, we will explore how to expand this basic installation with other useful services and how to keep the system updated and secure over time.

Launching BSSG - My Journey from Dynamic CMS to Bash Static Site Generator

Stefano Marinelli — Mon, 07 Apr 2025 08:11:36 +0200

I've had my own website practically forever. Back in the late '90s, I already had a web page on my ISP's server, and since at least 2001, I've had my own homepage on my own server. I've never been a great graphic designer, let alone a skilled webmaster, so I've always tried to keep things minimal and compatible.

Initially, like many others, I wrote HTML pages by hand. Then I used WYSIWYG creation tools, and eventually, I landed on CMS (Content Management Systems).

The Era of Dynamic CMS

I liked CMS because they allowed me to focus on the content and not on the correctness of the generated HTML. Thanks to them, I started writing my first blog shortly afterward.

Over the years, I've used many tools like PHPNuke, FlatNuke (created and developed by my friend Simone Vellei), eventually moving through Joomla and Wordpress. Wordpress always seemed like the most suitable tool for the job, and I used it for many years. Even today, mainly on the sysadmin side, I manage hundreds of Wordpress sites, and they are reasonably reliable, aside from the plugins (because the problem with Wordpress isn't the software itself, but many of the external plugins).

But this is precisely the problem: all dynamic CMS require constant and continuous security updates because, without them, the chances of defacement are extremely high.

Discovering Static Site Generators

And that's precisely why, when I discovered Carlos Fenollosa's bashblog in 2014, it immediately became clear that, indeed, there was no reason to continue down the path of dynamic CMS. I don't write often, I don't update often, there's no reason to regenerate all the content with every visit. Sure, WordPress caching plugins are often quite effective, but they are still add-ons that need to be kept up to date. And I'm not a fan of adding things to streamline. Often, less is more.

So, I started using bashblog for some 'secondary' projects until, in 2015, I migrated my 'old' Italian blog from WordPress to Pelican. Shortly after, I moved from Pelican to Nikola, and that blog is still generated by Nikola, although (that blog's) updates are now extremely rare (so much so that I consider it almost abandoned). I also created the first Docker container for Nikola and, for a long time, it was listed among the deployment methods on their site.

Building My Own: BSSG

But bashblog continued to fascinate me. So in 2015, for fun, I started developing my own Static Site Generator from scratch. I called it (with little imagination), BSSG - Bash Static Site Generator. The plan was for it to be compatible with the main OSes I use, to remain sufficiently simple and straightforward (!!!), and to be tailored to my needs. I intended to use it only and exclusively for small private things, starting with a sort of diary of mine - more professional than personal - and leave the 'official' blogs to more tested and 'professional' tools.

As time went by, I added some small features I liked: theming support, archives, tags (initially absent). Over time, many functions were added, and the script grew large – large enough to make me pause and ask myself some questions about the long-term stability of this solution. So, it remained only for my 'diary', which, however, grew year after year to the point where I needed to devise some kind of optimization. I then developed (more for fun than out of real necessity) a caching system. On rebuild, only what needs to be rebuilt is reconstructed, making the operation sufficiently fast even as the number of posts grows. Obviously, there are limits: using bash and external tools, the efficiency cannot be compared to that of a proper programming language.

Brief Detour: ITNBlog

And it's here that I decided, in preparation for opening a new blog (this one), to create a new tool called ITNBlog. I would develop it in Python and focus a bit more on performance and completeness. But ITNBlog stalled very quickly: time was limited, I'm not a full-time developer, so I realized I would spend too much time on development and too little on content creation.

Therefore, in 2018, I launched this blog but using Ghost, a solution that gave me good results, including performance-wise. I chose Ghost because I thought that, writing content also from my phone while on the go, a real CMS would be useful. Spoiler: no, it didn't turn out that way, so a few years later I decided to migrate this blog to Hugo. Nevertheless, I continued to develop ITNBlog on and off, as a hobby, without any particular ambitions.

At some point, however, I found myself in a particular situation: Hugo deprecated some features, and the theme I had chosen moved forward. But I ended up in an unpleasant situation: using the latest version of Hugo and the current version of the theme would produce unacceptable output; staying with the old version of Hugo while waiting for the theme update meant making a compromise. I actually build the blog from different devices, and they all have different versions of Hugo installed. Change the theme? Feasible, but I would have had to modify almost the entire site.

I considered migrating to manpageblog by gyptazy – I personally love its simplicity and retro look, and it was the main candidate to replace Hugo. I also created a script and migrated all my posts into the correct format.

BSSG to the Rescue (and ITNBlog's Role)

That's when I realized: I would implement the few missing features needed to make ITNBlog sufficiently complete, and this blog would be published using it, ensuring I'd be committed to its development. However, ITNBlog is not mature enough to be released publicly, so for now, it will remain the engine just for my blog. Then I thought again about BSSG – development had stalled some time ago, but it was still in use – and figured that perhaps, with a little tidying up, I could release it.

Because I'm tired of seeing people use dynamic CMS even to implement primarily static blogs or websites – and BSSG, despite its limitations and inefficiencies, works. And there are many themes to choose from. In short, you can install it and generate your blog in seconds.

Why Choose BSSG?

BSSG is the result of a 10-year evolution. The code isn't extremely consistent, some interesting features are missing (which I plan to implement), and it could use refactoring as the build script is monstrously large. But it works, it's portable (and much of the complexity increased precisely because of portability), and it generates sites that achieve very high accessibility and speed scores.

Here are some highlights:

✅ Portability: Uses native OS tools (e.g., md5sum on Linux, md5 on OpenBSD and NetBSD). Portability itself added much of the complexity!
✅ Simple Theming: Themes are just simple CSS files, so the structure remains the same – simplifying theme switching or creating new ones. More than 50 themes are already available!
✅ Essential Features: Supports RSS feed generation, sitemap.xml, OpenGraph tags (to improve social sharing), internationalization (the blog can be in languages other than English – but not multilingual, at least for now), etc.
✅ Built-in Backup and Restore script: It will just copy the configuration file, posts, and pages. Nothing else.
✅ Minimal Dependencies.
✅ Markdown Support: Posts and pages are in Markdown (CommonMark, Pandoc, and markdown.pl are supported).
✅ Feature Images.
✅ Optional GNU Parallel Integration: To speed up build times when there are many posts. This feature significantly impacts the code and has caused me numerous headaches over time. But it's optional (if parallel isn't found, it proceeds traditionally) and only provides benefits when the number of posts increases: with few posts, performance actually degrades.
✅ High Accessibility and Performance Scores: Sites built with BSSG achieve excellent scores.
✅ BSD Licensed: Released under a BSD license.

One of the problems I've always had with all CMS and SSGs has been choosing a theme. In some cases (like Hugo), the theme heavily influences the output, which is both good and bad. Good because it makes each site unique, but bad because it makes switching themes difficult. In the past, I've sometimes found myself having to change themes because they were abandoned and no longer updated. BSSG works differently: theming comes from using a different CSS file, which makes its structure more rigid, but switching from one theme to another is trivial. To help with the choice, I created a script that will build your site using all the themes present in the themes directory, just like on the examples page of the official website. This way, it will be easy to see and test your site with all available themes. If you want to add a touch of originality, you can choose the 'random' theme, and one will be chosen randomly from the list at each site regeneration.

Admin Interface (Experimental)

BSSG is in production use by some clients (for their internal sites), for whom I also created a basic admin interface (using Node Express, partly to chew on a bit of Node), but I don't feel ready to release it immediately as it's not sufficiently tested. It has an integrated Markdown editor and allows post scheduling, generating the files and launching BSSG with the right options at the right time. This could be that connecting link between traditional CMS and SSGs. There are others, but this one is tightly integrated with BSSG.

BSSG is Available Today

Starting today, BSSG is publicly available. It's not perfect, it probably doesn't make sense to do something of this complexity in bash, development will proceed slowly – but it's here, available to anyone who might find it useful.

Happy blogging everyone!

OSDay 2025 - Why Choose to Use the BSDs in 2025

Stefano Marinelli — Sun, 23 Mar 2025 10:30:00 +0100

This is the text underlying my presentation at OSDay 2025, held on 21 March 2025 in Florence, Italy. There was limited time, so I couldn't go into much detail and had to keep things more general and structured than usual. You can watch the video of my talk on YouTube.

The slides can be downloaded here

Happy reading!

OSDay Florence - 21 March 2025 - Why Choose to Use the BSDs in 2025

"I'm Stefano Marinelli, I solve problems."

I'm the founder and Barista of the BSD Cafe, a community of *BSD enthusiasts.

I work in my company, called Prodottoinrete - a container of ideas and solutions.

I'm passionate about technology and computing, and I've made my passion my profession. Every morning, when I sit in front of the computer, a new world opens up for me to explore.

I've been a Linux user since 1996, before I turned 17. Back then, I used Fidonet and would read about alternative operating systems. I experimented with Linux distributions from CDs, and by 1997, Linux became my everyday system. It was only in 2002 that I began exploring BSD systems, largely thanks to FreeBSD's fantastic handbook.

The relationship we had with Open Source 20-30 years ago was fundamentally different than today. Back then, embracing Open Source meant thinking differently. It meant embracing freedom. We chose Linux and the BSDs when Windows and commercial Unix systems dominated the market. Not because they were simple or free (as in free beer), but for freedom from impositions - both technological and ideological.

I solve problems. And to solve problems effectively, we need to recognize when the landscape has changed.

The reality today is that while we won that war - Open Source is everywhere - we're facing a new challenge. The "mainstream" Open Source world is creating monocultures. The focus has shifted from technologies to specific tools. We're seeing innovation for novelty's sake, not problem-solving.

This shift has profound implications. In a world dominated by cyber threats, where everything is connected and we completely depend on technology, the value of stability has been lost. By stability, I don't just mean that a system doesn't crash. I mean continuity over time, upgradeability, and system visibility.

Instead, the industry seems obsessed with the hype cycle. "New" is prioritized over secure and stable. The mantra has become: - "It will be fixed in the next version" - "We need automatic restarts when it crashes" - "Do we need software that crashes less? We have systemd and Kubernetes to restart crashed workloads!" - "We need moooarrr powaaaaaaar!!!!"

Let me give you a concrete example. A program written in Rust should be memory safe - that's one of the main selling points of the language. But if that program uses unsafe functions and segfaults, what advantage does it offer over a mature C implementation? Stability matters more than the implementation language.

I solve problems. And creating a monoculture does not solve problems - it creates new ones.

Yes, Linux, Docker, and Kubernetes are better than closed source solutions. But when everyone uses the same tools, freedom dies. We use them because "everyone does" rather than because they're the best tool for our specific needs.

If we had only used what everyone else used, we wouldn't have Linux or the BSDs today. There would be no LibreOffice, no Nextcloud. We'd just have Windows variations and expensive Unix systems. We'd be bound by licenses and vendors, stuck with closed solutions.

This is where the BSDs offer a compelling alternative: "Be free and evaluate alternatives. Always."

For those who don't know, the original BSD started in the 1970s (before Linux was conceived). Minix was created as an educational OS because it was believed that BSD, mature and professional, would be the Open Source OS that would dominate the market. A legal case stalled development and scared adopters, but in 1993, NetBSD and FreeBSD emerged. OpenBSD forked from NetBSD later, then DragonflyBSD from FreeBSD.

As Linus Torvalds said in 1993, "If 386BSD had been available when I started on Linux, Linux would probably never had happened."

What makes the BSDs special is their philosophy: - Kernel and userland developed by same teams - Consistency in tools and updates - Excellent documentation - especially OpenBSD, where insufficient docs are considered a bug - Man pages contain virtually everything - Evolution, not Revolution

Let me briefly introduce the main BSD variants that I work with daily:

FreeBSD is a generalist system. It focuses on stability and performance - with HardenedBSD as a security-enhanced fork. It has native ZFS, Boot Environments, and complete separation between OS and packages. It's had container support via jails since 2000 - which predates Linux cgroups by a decade! It offers bhyve virtualization (more efficient than KVM). OPNsense and pfSense are based on FreeBSD, as pf is a powerful firewall. It's used by Netflix for streaming video delivery and forms the foundation for PlayStation consoles. MacOS and iOS also contain some FreeBSD code.

OpenBSD focuses on security and code correctness. Its code is constantly audited and simplified - less is more. The team believes "The more complex the code, the less maintainable." It has security mechanisms like pledge() and unveil(). OpenSSH (and many other nice things) originated and are developed here. Development is driven by team priorities, not user requests. It's ideal for routers, firewalls, and security-critical systems.

NetBSD lives by the motto "Of course it runs NetBSD!" Its focus is on correctness, portability, and proper implementation. It supports 50+ architectures. Development centers on compatibility, which necessitates code quality. It must function on decades-old hardware. It's ideal for systems that require stability without the need for continuous updates, like embedded devices.

I solve problems. And in my experience, the BSDs have consistently proven to be excellent problem-solvers. Here are some real-world benefits I've experienced:

Better stability and security
Simplified administration - upgrades won't destroy your system
Less vulnerability to common attacks - "We don't need this patch, you're running OpenBSD and it's been fixed 20 years ago"
Network interfaces maintain consistent names - ix0 will remain ix0, not renaming from enx3e3300c9e14e to enp10s0f0np0
FreeBSD shows lower system load compared to Linux
FreeBSD handles I/O pressure better - on the same hardware, I've seen 70% time reduction
FreeBSD delivers improved end-user experience/responsiveness
NetBSD provides the comfort of "Don't worry - your platform will be supported for the foreseeable future"

So why choose BSD in 2025? I believe there are several compelling reasons:

Security in an increasingly hostile environment
Stability in a world obsessed with novelty
Performance without unnecessary complexity
Freedom from the mainstream monoculture
Systems designed with coherent philosophy

Don't be afraid to try BSD systems - despite the Beastie mascot, they don't hurt and you'll appreciate them!

See you at BSD Cafe!

IT Notes

EnshittifAIcation

Why I Love FreeBSD

Time Machine inside a FreeBSD jail

Installing Void Linux on ZFS with Hibernation Support

Introduction

Prerequisites and Audience

Installation Script (Optional)

Boot Environment

Initial Setup

Disk Configuration

Wipe the Disk

Create Partitions

EFI System Partition

Swap Partition

ZFS Pool Partition

Set Up ZFS Encryption

Create the ZFS Pool

Create ZFS Datasets

Export and Reimport for Installation

Install the Base System

Copy Host Configuration

Configure Encrypted Swap

Preserve Variables for Chroot

Enter the Chroot Environment

Configure fstab

Set Up Automatic Swap Unlock

Basic System Configuration

Configure ZFS Boot Support

Configure ZFSBootMenu

The Critical Step: Hibernation Support

Set Up EFI Boot

Install ZFSBootMenu

Microcode updates

Desktop Installation (Optional)

Install Core Desktop Packages

Install KDE Plasma

Enable Services

Configure PipeWire Audio

Enable Additional Repositories and Flatpak (Optional)

Create a Regular User and exit

Fix for NetworkManager

Exit and Reboot

Post-Installation

Testing Hibernation

Conclusion

Why I (still) love Linux

Where It All Began

When Linux Conquered the World

What Changed Along the Way

And Why I Still Care

Static Web Hosting on the Intel N150: FreeBSD, SmartOS, NetBSD, OpenBSD and Linux Compared

Test setup

Software under test

nginx configuration

TLS configuration

Load generator

The contenders

Static HTTP results

HTTP, 4 threads, 50 concurrent connections

HTTP, 4 threads, 10 concurrent connections

Static HTTPS results

HTTPS, 4 threads, 50 concurrent connections

HTTPS, 4 threads, 10 concurrent connections

What TLS seems to highlight

Zones, jails, containers and Docker: overhead in practice

What this means for real workloads

Final thoughts

Self-hosting your Mastodon media with SeaweedFS

Setting up the SeaweedFS Jail

Configuring Buckets and Users

Nginx Reverse Proxy Configuration

Mastodon Configuration

The Email They Shouldn't Have Read

FreeBSD vs. SmartOS: Who's Faster for Jails, Zones, and bhyve VMs?

The Lineup: What I Put on the Test Bench

The Benchmark: My sysbench Commands

First Look: CPU and Memory on the Intel N150

Valid CPU Performance Comparison (Single Thread, Intel N150)

Overall Memory Performance Comparison (Intel Processor N150)

The Benchmark: My `sysbench` Commands