IT Notes - netbsd

Static Web Hosting on the Intel N150: FreeBSD, SmartOS, NetBSD, OpenBSD and Linux Compared

Stefano Marinelli — Wed, 19 Nov 2025 09:16:00 +0100

Update: This post has been updated to include Docker benchmarks and a comparison of container overhead versus FreeBSD Jails and illumos Zones.

Note: Some operating systems (FreeBSD and Linux) support kernel TLS (kTLS) and the related SSL_sendfile path in nginx, which can improve HTTPS performance for static files. Since this feature is not available on all the systems included in the comparison (for example NetBSD, OpenBSD and illumos), the benchmarks were run with a common baseline configuration that does not rely on kTLS. The goal is to compare the systems under similar conditions rather than to measure OS specific optimizations.

I often get very specific infrastructure requests from clients. Most of the time it is some form of hosting. My job is usually to suggest and implement the setup that fits their goals, skills and long term plans.

If there are competent technicians on the other side, and they are willing to learn or already comfortable with Unix style systems, my first choices are usually one of the BSDs or an illumos distribution. If they need a control panel, or they already have a lot of experience with a particular stack that will clearly help them, I will happily use Linux and it usually delivers solid, reliable results.

Every now and then someone asks the question I like the least:

“But how does it perform compared to X or Y?”

I have never been a big fan of benchmarks. At best they capture a very specific workload on a very specific setup. They are almost never a perfect reflection of what will happen in the real world.

For example, I discovered that idle bhyve VMs seem to use fewer resources when the host is illumos than when the host is FreeBSD. It looks strange at first sight, but the illumos people are clearly working very hard on this, and the result is a very capable and efficient platform.

Despite my skepticism, from time to time I enjoy running some comparative tests. I already did it with Proxmox KVM versus FreeBSD bhyve, and I also compared Jails, Zones, bhyve and KVM on the same Intel N150 box. That led to the FreeBSD vs SmartOS article where I focused on CPU and memory performance on this small mini PC.

This time I wanted to do something simpler, but also closer to what I see every day: static web hosting.

Instead of synthetic CPU or I/O tests, I wanted to measure how different operating systems behave when they serve a small static site with nginx, both over HTTP and HTTPS.

This is not meant to be a super rigorous benchmark. I used the default nginx packages, almost default configuration, and did not tune any OS specific kernel settings. In my experience, careful tuning of kernel and network parameters can easily move numbers by several tens of percentage points. The problem is that very few people actually spend time chasing such optimizations. Much more often, once a limit is reached, someone yells “we need mooooar powaaaar” while the real fix would be to tune the existing stack a bit.

So the question I want to answer here is more modest and more practical:

With default nginx and a small static site, how much does the choice of host OS really matter on this Intel N150 mini PC?

Spoiler: less than people think, at least for plain HTTP. Things get more interesting once TLS enters the picture.

Disclaimer
These benchmarks are a snapshot of my specific hardware, network and configuration. They are useful to compare relative behavior on this setup. They are not a universal ranking of operating systems. Different CPUs, NICs, crypto extensions, kernel versions or nginx builds can completely change the picture.

Test setup

The hardware is the same Intel N150 mini PC I used in my previous tests: a small, low power box that still has enough cores to be interesting for lab and small production workloads.

On it, I installed several operating systems and environments, always on the bare metal, not nested inside each other. On each OS I installed nginx from the official packages.

Software under test

On the host:

SmartOS, with:
- a Debian 12 LX zone
- an Alpine Linux 3.22 LX zone
- a native SmartOS zone

FreeBSD 14.3-RELEASE:
- nginx running inside a native jail

OpenBSD 7.8:
- nginx on the host

NetBSD 10.1:
- nginx on the host

Debian 13.2:
- nginx on the host

Alpine Linux 3.22:
- nginx on the host
- Docker: Debian 13 container running on the Alpine host (ports mapped)

I also tried to include DragonFlyBSD, but the NIC in this box is not supported. Using a different NIC just for one OS would have made the comparison meaningless, so I excluded it.

nginx configuration

In all environments:

nginx was installed from the system packages
worker_processes was set to auto
the web root contained the same static content

The important part is that I used exactly the same nginx.conf file for all operating systems and all combinations in this article. I copied the same configuration file verbatim to every host, jail and zone. The only changes were the IP address and file paths where needed, for example for the TLS certificate and key.

The static content was a default build of the example site generated by BSSG, my Bash static site generator. The web root was the same logical structure on every OS and container type.

There is no OS specific tuning in the configuration and no kernel level tweaks. This is very close to a “package install plus minimal config” situation.

TLS configuration

For HTTPS I used a very simple configuration, identical on every host.

Self signed certificate created with:

openssl req -x509 -newkey rsa:4096 -nodes -keyout server.key -out server.crt -days 365 -subj "/CN=localhost"

Example nginx server block for HTTPS (simplified):

server {  
listen 443 ssl http2;  
listen [::]:443 ssl http2;  

server_name _;  

ssl_certificate /etc/nginx/ssl/server.crt;  
ssl_certificate_key /etc/nginx/ssl/server.key;  

root /var/www/html;  
index index.html index.htm;  

location / {  
try_files $uri $uri/ =404;  
}  
}

The HTTP virtual host is also the same everywhere, with the root pointing to the BSSG example site.

Load generator

The tests were run from my workstation on the same LAN:

client host: a mini PC machine connected at 2.5 Gbit/s
switch: 2.5 Gbit/s
test tool: wrk

For each target host I ran:

wrk -t4 -c50 -d10s http://IP
wrk -t4 -c10 -d10s http://IP
wrk -t4 -c50 -d10s https://IP
wrk -t4 -c10 -d10s https://IP

Each scenario was executed multiple times to reduce noise; the numbers below are medians (or very close to them) from the runs.

The contenders

To keep things readable, I will refer to each setup as follows:

SmartOS Debian LX → SmartOS host, Debian 12 LX zone
SmartOS Alpine LX → SmartOS host, Alpine 3.22 LX zone
SmartOS Native → SmartOS host, native zone
FreeBSD Jail → FreeBSD 14.3-RELEASE, nginx in a jail
OpenBSD Host → OpenBSD 7.8, nginx on the host
NetBSD Host → NetBSD 10.1, nginx on the host
Debian Host → Debian 13.2, nginx on the host
Alpine Host → Alpine 3.22, nginx on the host
Docker Container → Alpine host, Debian 13 Docker container

Everything uses the same nginx configuration file and the same static site.

Static HTTP results

Let us start with plain HTTP, since this removes TLS from the picture and focuses on the kernel, network stack and nginx itself.

HTTP, 4 threads, 50 concurrent connections

Approximate median wrk results:

Environment	HTTP 50 connections
SmartOS Debian LX	~46.2 k
SmartOS Alpine LX	~49.2 k
SmartOS Native	~63.7 k
FreeBSD Jail	~63.9 k
OpenBSD Host	~64.1 k
NetBSD Host	~64.0 k
Debian Host	~63.8 k
Alpine Host	~63.9 k
Docker Container	~63.7 k

Two things stand out:

All the native or jail/container setups on the hosts that are not LX zones cluster around 63 to 64k requests per second.
The two SmartOS LX zones sit slightly lower, in the 46 to 49k range, which is still very respectable for this hardware.

In other words, as long as you are on the host or in something very close to it (FreeBSD jail, SmartOS native zone, NetBSD, OpenBSD, Linux on bare metal), static HTTP on nginx will happily max out around 64k requests per second with this small Intel N150 CPU.

The Debian and Alpine LX zones on SmartOS are a bit slower, but not dramatically so. They still deliver close to 50k requests per second and, in a real world scenario, you would probably saturate the network or the client long before hitting those numbers.

HTTP, 4 threads, 10 concurrent connections

With fewer concurrent connections, absolute throughput drops, but the relative picture is similar:

SmartOS Native around 44k
NetBSD and Alpine Host around 34 to 35k
FreeBSD, Debian, OpenBSD around 31 to 33k
The Docker Container sits slightly lower at ~30.2k req/s, showing a small overhead from the networking layer
The SmartOS LX zones sit slightly below, around 35 to 37k req/s

The important conclusion is simple:

For plain HTTP static hosting, once nginx is installed and correctly configured, the choice between these operating systems makes very little difference on this hardware. Zones and jails add negligible overhead, LX zones add a small one.

If you are only serving static content over HTTP, your choice of OS should be driven by other factors: ecosystem, tooling, update strategy, your own expertise and preference.

Static HTTPS results

TLS is where things start to diverge more clearly and where CPU utilization becomes interesting.

HTTPS, 4 threads, 50 concurrent connections

Approximate medians:

Environment	HTTPS 50 connections	CPU notes at 50 HTTPS connections
SmartOS Debian LX	~51.4 k	CPU saturated
SmartOS Alpine LX	~40.4 k	CPU saturated
SmartOS Native	~52.8 k	CPU saturated
FreeBSD Jail	~62.9 k	around 60% CPU idle
OpenBSD Host	~39.7 k	CPU saturated
NetBSD Host	~40.4 k	CPU saturated
Debian Host	~62.8 k	about 20% CPU idle
Alpine Host	~62.4 k	small idle headroom, around 7% idle
Docker Container	~62.7 k	CPU saturated

These numbers tell a more nuanced story.

FreeBSD, Debian and Alpine on bare metal form a “fast TLS” group.
All three sit around 62 to 63k requests per second with 50 concurrent HTTPS connections.
FreeBSD does this while using significantly less CPU.
During the HTTPS tests with 50 connections, the FreeBSD host still had around 60% CPU idle. It is the platform that handled TLS load most comfortably in terms of CPU headroom.
Debian and Alpine are close in throughput, but push the CPU harder.
Debian still had some idle time left, Alpine even less. In practice, all three are excellent here, but FreeBSD gives you more room before you hit the wall.
SmartOS, NetBSD and OpenBSD form a “good but heavier” TLS group.
Their HTTPS throughput is in the 40 to 52k req/s range and they reach full CPU usage at 50 concurrent connections. OpenBSD and NetBSD stabilize around 39 to 40k req/s. SmartOS native and the Debian LX zone manage slightly better (around 51 to 53k) but still with the CPU pegged.

HTTPS, 4 threads, 10 concurrent connections

With lower concurrency:

FreeBSD, Debian and Alpine still sit in roughly the 29 to 31k req/s range
SmartOS Native and LX zones are in the mid to high 30k range
The Docker Container drops slightly to ~27.8k req/s
NetBSD and OpenBSD sit around 26 to 27k req/s

The relative pattern is the same: for this TLS workload, FreeBSD and modern Linux distributions on bare metal appear to make better use of the cryptographic capabilities of the CPU, delivering higher throughput or more headroom or both.

What TLS seems to highlight

The HTTPS tests point to something that is not about nginx itself, but about the TLS stack and how well it can exploit the hardware.

On this Intel N150, my feeling is:

FreeBSD, with the userland and crypto stack I am running, is very efficient at TLS here. It delivers the highest throughput while keeping plenty of CPU in reserve.
Debian and Alpine, with their recent kernels and libraries, are also strong performers, close to FreeBSD in throughput, but with less idle CPU.
NetBSD, OpenBSD and SmartOS (native and LX) are still perfectly capable of serving a lot of HTTPS traffic, but they have to work harder to keep up and they hit 100% CPU much earlier.

This matches what I see in day to day operations: TLS performance is often less about “nginx vs something else” and more about the combination of:

the TLS library version and configuration
how well the OS uses the CPU crypto instructions
kernel level details in the network and crypto paths

I suspect the differences here are mostly due to how each system combines its TLS stack (OpenSSL, LibreSSL and friends), its kernel and its hardware acceleration support. It would take a deeper dive into profiling and configuration knobs to attribute the gaps precisely.

In any case, on this specific mini PC, if I had to pick a platform to handle a large amount of HTTPS static traffic, FreeBSD, Debian and Alpine would be my first candidates, in that order.

Zones, jails, containers and Docker: overhead in practice

Another interesting part of the story is the overhead introduced by different isolation technologies.

From these tests and the previous virtualization article on the same N150 machine, the picture is consistent:

FreeBSD jails behave almost like bare metal and are significantly more efficient than Docker.
For both HTTP and HTTPS, running nginx in a jail on FreeBSD 14.3-RELEASE produces numbers practically identical to native hosts.
The contrast with Docker is striking: while the Docker container required 100% CPU to reach peak for the HTTP and HTTPS throughput, the FreeBSD jail delivered the same speed with ~60% of the CPU sitting idle. In terms of performance cost per request, Jails are drastically cheaper.
SmartOS native zones are also very close to the metal.
Static HTTP performance reaches the same 64k req/s region and HTTPS is only slightly behind the "fast TLS" group, although with higher CPU usage.
SmartOS LX zones introduce a noticeable but modest overhead.
Both Debian and Alpine LX zones on SmartOS perform slightly worse than the native zone or FreeBSD jails. For static HTTP they are still very fast. For HTTPS the Debian LX zone remains competitive but costs more CPU, while the Alpine LX zone is slower.
Docker on Linux performs efficiently but eats the margins. I ran an additional test using a Debian 13 Docker container running on the Alpine Linux host. At peak load (50 connections), the throughput was impressive and virtually identical to bare metal: ~63.7k req/s for HTTP and ~62.7k req/s for HTTPS. However, there is a clear cost. First, while the bare metal host maintained a small CPU buffer (~7% idle) during the HTTPS test, Docker saturated the CPU to 100%. Second, at lower concurrency (10 connections), the overhead became visible. The Docker container scored ~30.2k req/s for HTTP and ~27.8k req/s for HTTPS, slightly trailing the ~31-34k and ~29-31k range of the bare metal counterparts. The abstraction layers (NAT, bridging, namespaces) are extremely efficient, but they are not completely free.

This leads to a clear conclusion on efficiency: FreeBSD Jails provide the highest throughput with the lowest CPU cost. LX zones and Docker containers can match the speed (or come close), but they burn significantly more CPU cycles to do so.

What this means for real workloads

It is easy to get lost in tables and percentages, so let us go back to the initial question.

A client wants static hosting.
Does the choice between FreeBSD, SmartOS, NetBSD or Linux matter in terms of performance?

For plain HTTP on this hardware, with nginx and the same configuration:

Not really.
All the native hosts and FreeBSD jails deliver roughly the same maximum throughput, in the 63 to 64k req/s range. SmartOS LX zones are slightly slower but still strong.

For HTTPS:

Yes, it starts to matter a bit more.
FreeBSD stands out for how relaxed the CPU is under high TLS load.
Debian and Alpine are very close in throughput, with more CPU used but still with some headroom.
SmartOS, NetBSD and OpenBSD can still push a lot of HTTPS traffic, but they reach 100% CPU earlier and stabilize at lower request rates.

Does this mean you should always choose FreeBSD or Debian or Alpine for static HTTPS hosting?

Not necessarily.

In real deployments, the bottleneck is rarely the TLS performance of a single node serving a small static site. Network throughput, storage, logging, reverse proxies, CDNs and application layers all play a role.

However, knowing that FreeBSD and current Linux distributions can squeeze more out of a small CPU under TLS is useful when you are:

sizing hardware for small VPS nodes that must serve many HTTPS requests
planning to consolidate multiple services on a low power box
deciding whether you can afford to keep some CPU aside for other tasks (cache, background jobs, monitoring, and so on)

As always, the right answer depends on the complete picture: your skills, your tooling, your backups, your monitoring, the rest of your stack, and your tolerance for troubleshooting when things go sideways.

Final thoughts

From these small tests, my main takeaways are:

Static HTTP is basically solved on all these platforms.
On a modest Intel N150, every system tested can push around 64k static HTTP requests per second with nginx set to almost default settings. For many use cases, that is already more than enough.
TLS performance is where the OS and crypto stack start to matter.
FreeBSD, Debian and Alpine squeeze more HTTPS requests out of the N150, and FreeBSD in particular does it with a surprising amount of idle CPU left. NetBSD, OpenBSD and SmartOS need more CPU to reach similar speeds and stabilize at lower throughput once the CPU is saturated.
Jails and native zones are essentially free, LX zones cost a bit more.
FreeBSD jails and SmartOS native zones show very little overhead for this workload. SmartOS LX zones are still perfectly usable, but if you are chasing every last request per second you will see the cost of the translation layer.
Benchmarks are only part of the story.
If your team knows OpenBSD inside out and has tooling, scripts and workflows built around it, you might happily accept using more CPU on TLS in exchange for security features, simplicity and familiarity. The same goes for NetBSD or SmartOS in environments where their specific strengths shine.

I will not choose an operating system for a client just because a benchmark looks nicer. These numbers are one of the many inputs I consider. What matters most is always the combination of reliability, security, maintainability and the human beings who will have to operate the
system at three in the morning when something goes wrong.

Still, it is nice to know that if you put a tiny Intel N150 in front of a static site and you pick FreeBSD or a modern Linux distribution for HTTPS, you are giving that little CPU a fair chance to shine.

The Server That Wasn't Meant to Exist

Stefano Marinelli — Tue, 13 May 2025 16:13:36 +0200

Yesterday I read a piece of news that brought back an important - and painful - episode from my career.
A story about trust, technology... and the kind of problems that can't always be solved.

About 16 years ago, I was contacted by an old friend. He was worried about a situation involving some mutual acquaintances.
To keep it short: an entrepreneur - administrator and owner of several companies - had died suddenly.
He was the kind of man who centralized everything, and his wife and children found themselves struggling to manage things.
One of the sons decided to cash out and leave the family business (focusing on his own career), while the others chose to stay involved in day-to-day operations.
The wife, elderly and retired for years, ended up at the helm, but she was clearly out of her depth.

The main issue was the complete lack of information flow: no digital systems of any kind were in place.
Employees had their own PCs (sometimes even personal laptops), and there was zero control over anything.
All the accounting and administrative data were scattered across individual machines, often taken home at the end of the day.
From the owners’ perspective, all they saw was a huge cash flow coming in - yet the accounts were always in the red.

"If we keep going like this, we’ll be bankrupt in just a few years", I was told.
What I could do was set up a proper IT system, structured to make data management transparent and traceable.

I planned - and got immediate approval for - the purchase of routers, switches, various networking devices and a server with several disks.
The OS of choice, as was my habit at the time, was NetBSD. Thanks to XEN, I set up multiple VMs.
One handled the NAS duties (using Samba, so PCs could connect and store files directly there), another ran Archivista.
I even worked on translating Archivista’s interface into Italian, since it wasn’t yet localized, just to make it easier for users.

As usual in those days, I added a caching proxy (Squid) and a content filter (DansGuardian), to ensure proper usage.
The internet connection was very slow and often collapsed under heavy load - mostly recreational use, as logs revealed.
There was no supervision, and many people were downloading movies and such on company time.

As often happens, not everyone was happy.

One figure in particular - the late owner's former right-hand man - opposed the new system in every possible way.
According to him, none of this was necessary. But the real alarm bell had been his sudden change in lifestyle.
He’d made purchases that didn’t remotely align with his salary.
At the time, the company had no oversight and dealt with a lot of cash. It was all technically legal, especially given the nature of the business. I won’t go into detail - privacy matters.

Once everything was up and running, we trained the employees to use the new system.
Most were thrilled - finally able to work properly, with files in the right place and centralized document management.
OCR, archiving, etc. were all seen as major time-savers and a big boost in efficiency.
Some of the accounting staff remained skeptical, of course.

Since all of this was far from where I lived at the time, I went back to my life once the system was stable and everything in place.
A few quiet days passed - then one morning, my phone rang:

"Good morning, this is XYZ - I handle some technical aspects of a software suite used by the company where you've just installed everything.
We need to install our software, and I understand you set up the server. I’ll need the full server diagram and all the admin passwords".

I explained that it wasn’t a Windows machine, as he assumed (without even having seen it), but NetBSD, running NetBSD and Linux VMs.

A few seconds of silence.

"I see. Then I’ll have to wipe it and install Windows. I need Windows, and I don’t have time to wait for a new server - I’ll proceed tomorrow morning".

I froze. I told him that was not possible - the entire workflow now depended on that machine, and erasing it would be catastrophic.

"I’ll speak with the owners", I said, "and I’m sure they’ll provide you with a separate server within hours".

No use. He started to backpedal.
To my (young) eyes, the goal was now obvious: that server had to disappear, and fast.
He said he would "restore the previous situation", and claimed the server couldn’t remain as-is because he needed it.

I immediately called the owners. Sadly, due to inexperience and inability to handle the situation, they panicked.
They asked me to consider letting him do it, and then redoing the setup later, covering the cost of new hardware and my time.
I refused. I was young, but I already had this mindset: do what’s right, even at the cost of profit.
This was clearly a maneuver to eliminate controls - the server, the centralized filesystem.
The goal was to hide the real accounting data from owners and auditors. Thousands of euros vanished every day through "transactions".
The owners had started to understand, and this new pressure confirmed just how rotten things really were.

I called that man back and told him clearly: the server I’d built wasn’t to be shut down.
If he needed one, I’d deliver a new server by that evening, just for him.

At that point, he finally spoke more openly:

"You don’t get it, do you? I need that server. Not a server.
You’d better go along with this — or you’ll have serious trouble working in this area again".

And other similar, "nice" sentences. I replied calmly:
"Look, I’m just doing a favor for some friends. I don’t have clients in your area — and I don’t want any.
I’d rather do a good job for them than gain new clients".

No way through. He kept pushing - confident in his own sense of "power" - until I said what I had been trying to avoid.

Because I had recognized who he was.
And the disappointment hit me twice as hard - I used to admire him.
He, however, had not recognized me.

"Excuse me, but why are you talking to me like this? You’ve known me since I was a child. Don’t you remember? I’m the nephew of..."

He froze.
He understood immediately.
He connected the dots and knew full well that a single phone call to someone extremely close to me - someone he owed a great deal, both personally and professionally - would have the opposite effect he was aiming for.

That person had helped him greatly over the years. So much so that he folded:

"Oh... I’m so sorry... I didn’t recognize you. I’ll find another solution. Sorry again".

He hung up.
Never heard from him again.

I informed the owners that the issue was resolved (leaving out most of the details), but it didn’t last long.
Within days, a series of "unfortunate events" hit the server: the UPS failed, the server was "accidentally" unplugged and plugged back in incorrectly, and finally... it stopped responding on the network.

It was dead.
And when we opened it, the hard disks were just... gone.

But there was one thing nobody (but the owners) knew.
The server - slowly but surely - had been backing up externally.
All the data up to that point had been copied to a device we’d quietly installed at the owners’ home: a tiny PCEngines Alix, running NetBSD with two USB drives.
It was slow, yes - slow hardware, slow disks - but reliable.
That very device still works today (with FreeBSD) and provides services elsewhere.

I handed all the data to the owners and asked what they intended to do.
They took some time - days, then weeks.
Eventually, they said they’d probably investigate whether there were grounds for a theft report.

I never heard more about it.

But then came a tempting offer:

"Come work for us. Manage our network infrastructure and help us overhaul our internal procedures.
Even if you’ve just bought a house far from here, even if you’d have to leave your other clients -
we’ll pay you enough to forget everything else. Name your price".

They would’ve done it, too.
Our mutual friend urged me:
"They’ve got a huge cash flow, but too many people are taking advantage of them due to lack of control.
Take the job - they’ll treat you like gold, and you’ll really help them".

I didn’t think twice.

I turned it down.

I like my work.
I like doing what I do - and the income is a consequence, not the cause.
I would’ve had to give up my life, my path, to fight battles I don’t enjoy - and that I might not even win.

Because sometimes, dishonest people do win.

I’ve never regretted declining that offer.
I lost touch with all of them years ago, but I later heard things went as I predicted:
the owners gradually backed out.

They made another request later on, which I tried to fulfill — but even that was blocked, just when everything was ready.

At some point, I had to walk away.
Not because I wanted to abandon them in a time of need, but because they weren’t giving me the tools to do what was necessary.
They were so overwhelmed, so unprepared, that they ended up yielding to pressure - often from the very people who were hurting them.

And of course, I’ve left out the worst parts of the story.

(Author's note: Many readers, understandably struck by the severity of the events, have speculated about the involvement of organized crime. I want to clarify that, while the situation was extremely problematic and dishonest, that wasn't the case. The "worst parts" I alluded to referred to other internal dynamics, abuses of trust, and improprieties that I prefer not to detail further for privacy reasons and to avoid weighing down the narrative.)

That’s when I realized:
Some situations are so rotten, they simply can’t be salvaged.

And that’s okay.

I solve problems - it’s what I do best.

But I can’t solve every problem.
Especially not when those involved choose to protect the problem instead of fixing it.

Make Your Own Internet Presence with NetBSD and a 1 euro VPS – Part 1: Your Blog

Stefano Marinelli — Tue, 22 Apr 2025 07:30:36 +0200

Why NetBSD?

For many years, I've been using (and appreciating) NetBSD because it's stable, efficient, and reliable. The codebase has proven its reliability, running without reboots for years without issues. It supports ZFS (though differently than FreeBSD), LVM (useful for those accustomed to it on Linux), the ability to take filesystem snapshots (UFS2, making ZFS less crucial), and it's an excellent virtualization platform. Installation and updates are easy (including via sysupgrade - which I'll cover in a future article). Since it focuses on portable and optimized code (running on ancient architectures requires cleanliness and correctness), it's particularly efficient on low-power devices, like embedded systems or cheap VMs. Therefore, it's one of the best solutions for a small personal setup that can still deliver excellent results and simple management.

Indeed, the market offers very cheap VPS, often with just a single core and little RAM. But a modern single core packs power that a multi-core from just a few years ago could only dream of, and often, the I/O of these machines (a bottleneck for many services) is still decent. I personally use 1 euro per month VPS (VAT included - for those not subject to it, that's less than one euro per month!) with a public IPv4 address and (often) a /64 IPv6 block, ensuring full reachability across the entire network. I'm not providing direct links as I have no affiliations, but netcup's "piko" VPS are among the types I use most often (a 4 euro/month netcup VM handles the entire FediMeteo project), and this type of VM is ideal for our purpose because some providers (like netcup) allow you to upload your own ISO and install your preferred operating system. On VPS like these, I've installed everything - including OmniOS and SmartOS - without problems. And even such a small VPS, with an efficient operating system, can be extremely satisfying.

Why BSSG?

In this article, I'll describe how to create and publish a blog using BSSG as it exemplifies my concept of portability and minimalism. BSSG on NetBSD currently doesn't leverage parallelism provided by tools like GNU Parallel, but for small to medium-sized blogs, this won't be an issue, especially considering these small VMs only have 1 core. Obviously, you can use any Static Site Generator (SSG) (like Hugo, Nikola, 11ty, Pelican, Zola, etc.) - the important thing is to have a static site served by a simple web server.

Let's Start with the Installation

Installing NetBSD is quite straightforward and is clearly covered, complete with explanatory screenshots, in the excellent official NetBSD documentation, which I recommend using as a reference during the process, especially if it's your first time.

In my case, I made sure to use the proposed disk geometry, use the standard automatic partitioning, but enable the "log" and "noatime" options for the filesystem. Both these options will provide a huge advantage in I/O operations, especially with BSSG, as the first enables journaling and the second prevents updating file metadata on every access. BSSG is more I/O bound than CPU bound, so any optimization is beneficial.

Moving forward, I also recommend configuring the network (although installation can be done from packages on the installation ISO). For netcup, you can use DHCPv4 (even though it's a bit slow and sometimes seems to fail, the DHCP client will continue running in the background and eventually work).

For IPv6, I usually configure it manually later, so I'll describe that further down.

I also recommend enabling SSH, adding a regular user (and adding them to the wheel group so they can gain root privileges) - in this case, I'll call the user blog. Also, enable the installation of binary packages, as it will be convenient later to use pkgin to install and update all necessary packages. All these steps are described clearly and in detail in the guide, so I won't detail them here. But they are simple and logical, like all operations on BSD systems.

After installation, reboot. If everything went correctly, you should be able to log in via console or SSH using the "blog" user (or whatever you named it).

First, I suggest configuring the IPv6 address and installing the necessary packages.

For IPv6, in the case of netcup, simply add one of the assigned addresses to the interface. In NetBSD, network interface configurations are stored (similar to OpenBSD) in specific files. For the first virtio interface, the file will be /etc/ifconfig.vioif0.

You need to elevate your privileges to root, open that file with your preferred editor, and add the configuration to the file itself:

nb1euro$ su -l
nb1euro# vi /etc/ifconfig.vioif0

inet6 your-ipv6-addr/64
up

To test everything, perform a reboot and try pinging an IPv6 address (I often use ping6 google.com).

If all goes well, after a few seconds, you should see ping replies, confirming everything is configured correctly.

Regarding packages, the only two strictly necessary ones are bash and a markdown processor (by default, BSSG will use commonmark; otherwise, it can be configured to use pandoc or Markdown.pl). rsync can be useful for deployment. sudo (or doas) can be useful for elevating privileges for certain operations, at least at this stage.

nb1euro$ su -l
nb1euro# pkgin in bash cmark rsync sudo

If you're used to Linux, you can also install the "nano" editor:

nb1euro# pkgin in nano

If sudo was installed, it's now appropriate to grant users in the "wheel" group (like the regular user created during installation) the ability to elevate privileges. Edit the sudoers file (I suggest using the visudo command) and uncomment this line:

## Uncomment to allow members of group wheel to execute any command
%wheel ALL=(ALL:ALL) ALL

At this point, you can switch back to operating as the regular user, downloading and unpacking BSSG:

nb1euro$ ftp https://brew.bsd.cafe/stefano/BSSG/archive/0.15.1.tar.gz
nb1euro$ tar zxfv 0.15.1.tar.gz

Now that BSSG is ready, just initialize a directory with the structure for the new site:

nb1euro$ cd bssg
nb1euro$ ./bssg.sh init /home/blog/myblog

Everything is set to start generating your blog. I recommend reading BSSG's README.md. There are many options, themes, etc., but to get started, you just need to set the site's public URL. For example, if the site will be published as myblog.example.com - just create a file at /home/blog/myblog/config.sh.local (the path defined by the init command) and set the public URL:

SITE_URL="https://myblog.example.com"

This way, all URLs will be absolute URLs, which is necessary to ensure the correct functioning of RSS feeds, sitemaps, etc. This setting assumes HTTPS - if you just want to test the site over HTTP, simply use http and then, optionally, change it to https and regenerate the site later.

You can already create your first test post, directly from the BSSG directory:

nb1euro$ ./bssg.sh post

The system will use nano if it's installed, otherwise it will use vi. Don't worry, in the latter case, BSSG will write the procedure for exiting vi as the post's text 🙂

Once you save the post, BSSG will automatically generate the site. If everything went well, the /home/blog/myblog/output directory will contain the final result. We are therefore ready for the first deployment, which can be done in many different ways. I will cover three:

Using bozohttpd, present by default in NetBSD's base system. It can be used via inetd (launching an httpd process for each connection) or as a daemon. I'll describe the first option, showing in the final benchmarks how, even when used as a daemon, it remains a less performant solution.
Using nginx
Using Caddy

First, it's advisable to obtain a certificate to configure and use HTTPS. If you only want to test using HTTP, this part can be safely bypassed. For solutions 1 and 2, I'll use certbot, which is well-known to many users with Linux experience. Caddy, on the other hand, manages certificates automatically, so there's no need for other solutions and thus no need to install certbot.

nb1euro$ sudo pkgin in py313-certbot

To use bozohttpd, no further installation is necessary. At this point, the options diverge.

Using NetBSD's Integrated httpd

bozohttpd is integrated into NetBSD and, by default, can be launched directly via inetd. This solution, while not extremely efficient or scalable, is simple and requires few resources. It's fine if you expect only a few visits per day, but when used via inetd, the initial latency for each connection is tangible. It can still be useful for some tests or small deployments.

The /etc/inetd.conf file already contains the options to handle this situation:

#http           stream  tcp     nowait:600      _httpd  /usr/libexec/httpd      httpd /var/www
#http           stream  tcp6    nowait:600      _httpd  /usr/libexec/httpd      httpd /var/www

By uncommenting these two lines and restarting inetd (service inetd restart), the server will start responding to HTTP requests on both IPv4 and IPv6.

If you want to add HTTPS support, no problem. Just request a certificate via certbot and specify the webroot.

Run:

nb1euro$ sudo certbot-3.13 certonly

Choose option 2 - the one where you specify the webroot - enter the domain, and when prompted, provide /var/www/ as the webroot.

The certificate will be created. Then, modify the /etc/inetd.conf file to also include support for HTTPS, adding two lines similar to these (obviously, change the certificate paths):

https            stream  tcp     nowait:600      _httpd  /usr/libexec/httpd      httpd -Z /usr/pkg/etc/letsencrypt/live/myblog.example.com/fullchain.pem /usr/pkg/etc/letsencrypt/live/myblog.example.com/privkey.pem /var/www
https            stream  tcp6    nowait:600      _httpd  /usr/libexec/httpd      httpd -Z /usr/pkg/etc/letsencrypt/live/myblog.example.com/fullchain.pem /usr/pkg/etc/letsencrypt/live/myblog.example.com/privkey.pem /var/www

Warning: httpd will run with the permissions of the _httpd user, so make sure all certificates are readable by that user:

nb1euro# chown -R _httpd /usr/pkg/etc/letsencrypt/

Restart inetd, and the server will also respond over HTTPS.

To make your blog public, simply copy the files from the site's output directory to /var/www/ - this time using sudo to bypass permission issues:

nb1euro$ sudo rsync -avhHPx /home/blog/myblog/output/ /var/www/

The site will be immediately visible.

Using nginx

Nginx is fast and efficient, and the performance difference is noticeable (some benchmarks follow below). For an efficient setup ready for a high number of visits, it's advisable to use a web server suited for the purpose, just like nginx.

First, install nginx and the certbot plugin for nginx. This will simplify the installation and renewal of certificates:

nb1euro$ sudo pkgin in py313-certbot-nginx nginx

Copy the startup script to /etc/rc.d - as indicated by the post-installation message. In NetBSD, this operation must be done manually, but it's always pointed out:

nb1euro$ sudo cp /usr/pkg/share/examples/rc.d/nginx /etc/rc.d

Warning: If you previously used httpd from inetd following the previous solution, you must disable it in inetd.conf and restart inetd to free up ports 80 and 443.

Now you can create a virtual host for our new site.

nb1euro$ sudo vi /usr/pkg/etc/nginx/nginx.conf

and add, at the end of the file and before the final closing curly brace:

server {
        listen 80;
        # If you also have configured IPv6 support
        listen [::]:80;

        root /var/www;
        index index.html index.htm;

        server_name myblog.example.com;

        # If you want a long cache for media and css - be careful, this means that if you change to a new theme, it might not be visible immediately as the browser might still use the old cached one
        location ~* \.(jpg|jpeg|png|gif|ico|css|js)$ {
            expires 30d;
            add_header Cache-Control "public, no-transform";
        }

        location / {
                try_files $uri $uri/ =404;
        }
}

Now, it's time to configure the system to enable nginx. Just edit /etc/rc.conf:

nb1euro$ sudo vi /etc/rc.conf

and add:

nginx=YES

Now, you can start nginx:

nb1euro$ sudo service nginx start

Nginx will start listening on port 80. Generating and installing the certificate is very simple:

nb1euro$ sudo certbot-3.13 --nginx -d myblog.example.com

This command will request the certificate and install it, so nginx will already be configured to use it.

As with the previous method, to make your blog public, simply copy the files from the site's output directory to /var/www/ - using sudo to bypass permission issues:

nb1euro$ sudo rsync -avhHPx /home/blog/myblog/output/ /var/www/

The site will be immediately visible.

Using Caddy

Caddy is a convenient and all-in-one solution, efficient and fast. It's packaged for NetBSD and allows you to go online in a flash. I won't delve into the configuration because there are many tutorials (including the official ones), but you just need to install it and run it:

nb1euro$ sudo pkgin in caddy

Once installed, go to the directory you want to serve (e.g., /var/www or directly /home/blog/myblog/output) and run:

nb1euro$ sudo caddy file-server --domain myblog.example.com

Caddy will start, request the certificate, and begin serving your blog over HTTPS as well. To install Caddy as a service (i.e., with a configuration file, etc.), you can proceed similarly to how it's done on Linux. The NetBSD Caddy package doesn't include the rc.d script, but you can copy and paste one (into /etc/rc.d/caddy) from a thread posted on UnitedBSD.

Performance Comparison

I performed some performance tests on these solutions. Here are the results, on a single-core 1 euro/month VPS, from my home connection (which also has its own limitations):

NetBSD httpd via inetd:

Running 10s test @ https://myblog.example.com/
  4 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   213.52ms  173.10ms   1.11s    76.01%
    Req/Sec    12.92      9.19    50.00     75.91%
  371 requests in 10.10s, 1.39MB read
Requests/sec:     36.72
Transfer/sec:    140.65KB

These numbers are quite poor, linked to high latency caused by having to launch bozohttpd for each incoming connection.

NetBSD httpd as a daemon:

Running 10s test @ https://myblog.example.com/
  4 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    35.74ms    6.96ms 108.80ms   81.36%
    Req/Sec    18.29      9.45    50.00     70.88%
  676 requests in 10.10s, 2.53MB read
Requests/sec:     66.92
Transfer/sec:    256.32KB

Here the situation is decidedly better, but not exceptional. httpd isn't designed for high loads or performance.

Nginx as a daemon, 1 worker:

Running 10s test @ https://myblog.example.com/
  4 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    30.69ms    4.87ms  64.14ms   66.01%
    Req/Sec   379.39     65.94   464.00     90.91%
  15026 requests in 10.04s, 56.50MB read
Requests/sec:   1496.65
Transfer/sec:      5.63MB

Here we are on another level, showing truly solid performance. This type of result can handle significantly high loads without particular difficulty. The efficiency of both NetBSD and nginx pays off.

Caddy:

Running 10s test @ https://myblog.example.net/
  4 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    32.10ms    5.75ms  95.04ms   87.44%
    Req/Sec   362.74     64.29   434.00     91.67%
  14374 requests in 10.05s, 54.63MB read
Requests/sec:   1430.82
Transfer/sec:      5.44MB

Caddy shows results comparable to nginx, so the choice between them depends solely on the type of configuration you want to achieve and the experience each person has with the specific platforms.

Conclusion: Efficient Minimalism

We've seen how it's possible to create a personal, professional, and performant online presence with minimal investment. This solution, based on NetBSD and a 1€/month VPS, offers several advantages:

Negligible Cost: For 12€ per year, you can have a website (and more!) completely under your control.
Surprising Performance: As demonstrated by the benchmarks, excellent performance can be achieved even with limited resources (up to 1400-1500 requests/second with nginx or Caddy).
Security and Stability: NetBSD is renowned for its reliability and security, fundamental characteristics for any online service.
Total Control: Unlike free blogging platforms, you have full control over every aspect of your site.
Learning Experience: Managing a BSD system allows you to acquire valuable system administration skills.

This minimalist configuration demonstrates that you don't need to invest in expensive cloud solutions or oversized VPS to have a quality online presence. In an era where the tendency is to think "moooar powaaaar = better results", NetBSD reminds us that efficiency and good design can yield excellent results even with limited resources.

After all, you don't need a thousand-node cloud to write something worth reading.

In the upcoming articles in this series, we will explore how to expand this basic installation with other useful services and how to keep the system updated and secure over time.

Launching BSSG - My Journey from Dynamic CMS to Bash Static Site Generator

Stefano Marinelli — Mon, 07 Apr 2025 08:11:36 +0200

I've had my own website practically forever. Back in the late '90s, I already had a web page on my ISP's server, and since at least 2001, I've had my own homepage on my own server. I've never been a great graphic designer, let alone a skilled webmaster, so I've always tried to keep things minimal and compatible.

Initially, like many others, I wrote HTML pages by hand. Then I used WYSIWYG creation tools, and eventually, I landed on CMS (Content Management Systems).

The Era of Dynamic CMS

I liked CMS because they allowed me to focus on the content and not on the correctness of the generated HTML. Thanks to them, I started writing my first blog shortly afterward.

Over the years, I've used many tools like PHPNuke, FlatNuke (created and developed by my friend Simone Vellei), eventually moving through Joomla and Wordpress. Wordpress always seemed like the most suitable tool for the job, and I used it for many years. Even today, mainly on the sysadmin side, I manage hundreds of Wordpress sites, and they are reasonably reliable, aside from the plugins (because the problem with Wordpress isn't the software itself, but many of the external plugins).

But this is precisely the problem: all dynamic CMS require constant and continuous security updates because, without them, the chances of defacement are extremely high.

Discovering Static Site Generators

And that's precisely why, when I discovered Carlos Fenollosa's bashblog in 2014, it immediately became clear that, indeed, there was no reason to continue down the path of dynamic CMS. I don't write often, I don't update often, there's no reason to regenerate all the content with every visit. Sure, WordPress caching plugins are often quite effective, but they are still add-ons that need to be kept up to date. And I'm not a fan of adding things to streamline. Often, less is more.

So, I started using bashblog for some 'secondary' projects until, in 2015, I migrated my 'old' Italian blog from WordPress to Pelican. Shortly after, I moved from Pelican to Nikola, and that blog is still generated by Nikola, although (that blog's) updates are now extremely rare (so much so that I consider it almost abandoned). I also created the first Docker container for Nikola and, for a long time, it was listed among the deployment methods on their site.

Building My Own: BSSG

But bashblog continued to fascinate me. So in 2015, for fun, I started developing my own Static Site Generator from scratch. I called it (with little imagination), BSSG - Bash Static Site Generator. The plan was for it to be compatible with the main OSes I use, to remain sufficiently simple and straightforward (!!!), and to be tailored to my needs. I intended to use it only and exclusively for small private things, starting with a sort of diary of mine - more professional than personal - and leave the 'official' blogs to more tested and 'professional' tools.

As time went by, I added some small features I liked: theming support, archives, tags (initially absent). Over time, many functions were added, and the script grew large – large enough to make me pause and ask myself some questions about the long-term stability of this solution. So, it remained only for my 'diary', which, however, grew year after year to the point where I needed to devise some kind of optimization. I then developed (more for fun than out of real necessity) a caching system. On rebuild, only what needs to be rebuilt is reconstructed, making the operation sufficiently fast even as the number of posts grows. Obviously, there are limits: using bash and external tools, the efficiency cannot be compared to that of a proper programming language.

Brief Detour: ITNBlog

And it's here that I decided, in preparation for opening a new blog (this one), to create a new tool called ITNBlog. I would develop it in Python and focus a bit more on performance and completeness. But ITNBlog stalled very quickly: time was limited, I'm not a full-time developer, so I realized I would spend too much time on development and too little on content creation.

Therefore, in 2018, I launched this blog but using Ghost, a solution that gave me good results, including performance-wise. I chose Ghost because I thought that, writing content also from my phone while on the go, a real CMS would be useful. Spoiler: no, it didn't turn out that way, so a few years later I decided to migrate this blog to Hugo. Nevertheless, I continued to develop ITNBlog on and off, as a hobby, without any particular ambitions.

At some point, however, I found myself in a particular situation: Hugo deprecated some features, and the theme I had chosen moved forward. But I ended up in an unpleasant situation: using the latest version of Hugo and the current version of the theme would produce unacceptable output; staying with the old version of Hugo while waiting for the theme update meant making a compromise. I actually build the blog from different devices, and they all have different versions of Hugo installed. Change the theme? Feasible, but I would have had to modify almost the entire site.

I considered migrating to manpageblog by gyptazy – I personally love its simplicity and retro look, and it was the main candidate to replace Hugo. I also created a script and migrated all my posts into the correct format.

BSSG to the Rescue (and ITNBlog's Role)

That's when I realized: I would implement the few missing features needed to make ITNBlog sufficiently complete, and this blog would be published using it, ensuring I'd be committed to its development. However, ITNBlog is not mature enough to be released publicly, so for now, it will remain the engine just for my blog. Then I thought again about BSSG – development had stalled some time ago, but it was still in use – and figured that perhaps, with a little tidying up, I could release it.

Because I'm tired of seeing people use dynamic CMS even to implement primarily static blogs or websites – and BSSG, despite its limitations and inefficiencies, works. And there are many themes to choose from. In short, you can install it and generate your blog in seconds.

Why Choose BSSG?

BSSG is the result of a 10-year evolution. The code isn't extremely consistent, some interesting features are missing (which I plan to implement), and it could use refactoring as the build script is monstrously large. But it works, it's portable (and much of the complexity increased precisely because of portability), and it generates sites that achieve very high accessibility and speed scores.

Here are some highlights:

✅ Portability: Uses native OS tools (e.g., md5sum on Linux, md5 on OpenBSD and NetBSD). Portability itself added much of the complexity!
✅ Simple Theming: Themes are just simple CSS files, so the structure remains the same – simplifying theme switching or creating new ones. More than 50 themes are already available!
✅ Essential Features: Supports RSS feed generation, sitemap.xml, OpenGraph tags (to improve social sharing), internationalization (the blog can be in languages other than English – but not multilingual, at least for now), etc.
✅ Built-in Backup and Restore script: It will just copy the configuration file, posts, and pages. Nothing else.
✅ Minimal Dependencies.
✅ Markdown Support: Posts and pages are in Markdown (CommonMark, Pandoc, and markdown.pl are supported).
✅ Feature Images.
✅ Optional GNU Parallel Integration: To speed up build times when there are many posts. This feature significantly impacts the code and has caused me numerous headaches over time. But it's optional (if parallel isn't found, it proceeds traditionally) and only provides benefits when the number of posts increases: with few posts, performance actually degrades.
✅ High Accessibility and Performance Scores: Sites built with BSSG achieve excellent scores.
✅ BSD Licensed: Released under a BSD license.

One of the problems I've always had with all CMS and SSGs has been choosing a theme. In some cases (like Hugo), the theme heavily influences the output, which is both good and bad. Good because it makes each site unique, but bad because it makes switching themes difficult. In the past, I've sometimes found myself having to change themes because they were abandoned and no longer updated. BSSG works differently: theming comes from using a different CSS file, which makes its structure more rigid, but switching from one theme to another is trivial. To help with the choice, I created a script that will build your site using all the themes present in the themes directory, just like on the examples page of the official website. This way, it will be easy to see and test your site with all available themes. If you want to add a touch of originality, you can choose the 'random' theme, and one will be chosen randomly from the list at each site regeneration.

Admin Interface (Experimental)

BSSG is in production use by some clients (for their internal sites), for whom I also created a basic admin interface (using Node Express, partly to chew on a bit of Node), but I don't feel ready to release it immediately as it's not sufficiently tested. It has an integrated Markdown editor and allows post scheduling, generating the files and launching BSSG with the right options at the right time. This could be that connecting link between traditional CMS and SSGs. There are others, but this one is tightly integrated with BSSG.

BSSG is Available Today

Starting today, BSSG is publicly available. It's not perfect, it probably doesn't make sense to do something of this complexity in bash, development will proceed slowly – but it's here, available to anyone who might find it useful.

Happy blogging everyone!

OSDay 2025 - Why Choose to Use the BSDs in 2025

Stefano Marinelli — Sun, 23 Mar 2025 10:30:00 +0100

This is the text underlying my presentation at OSDay 2025, held on 21 March 2025 in Florence, Italy. There was limited time, so I couldn't go into much detail and had to keep things more general and structured than usual. You can watch the video of my talk on YouTube.

The slides can be downloaded here

Happy reading!

OSDay Florence - 21 March 2025 - Why Choose to Use the BSDs in 2025

"I'm Stefano Marinelli, I solve problems."

I'm the founder and Barista of the BSD Cafe, a community of *BSD enthusiasts.

I work in my company, called Prodottoinrete - a container of ideas and solutions.

I'm passionate about technology and computing, and I've made my passion my profession. Every morning, when I sit in front of the computer, a new world opens up for me to explore.

I've been a Linux user since 1996, before I turned 17. Back then, I used Fidonet and would read about alternative operating systems. I experimented with Linux distributions from CDs, and by 1997, Linux became my everyday system. It was only in 2002 that I began exploring BSD systems, largely thanks to FreeBSD's fantastic handbook.

The relationship we had with Open Source 20-30 years ago was fundamentally different than today. Back then, embracing Open Source meant thinking differently. It meant embracing freedom. We chose Linux and the BSDs when Windows and commercial Unix systems dominated the market. Not because they were simple or free (as in free beer), but for freedom from impositions - both technological and ideological.

I solve problems. And to solve problems effectively, we need to recognize when the landscape has changed.

The reality today is that while we won that war - Open Source is everywhere - we're facing a new challenge. The "mainstream" Open Source world is creating monocultures. The focus has shifted from technologies to specific tools. We're seeing innovation for novelty's sake, not problem-solving.

This shift has profound implications. In a world dominated by cyber threats, where everything is connected and we completely depend on technology, the value of stability has been lost. By stability, I don't just mean that a system doesn't crash. I mean continuity over time, upgradeability, and system visibility.

Instead, the industry seems obsessed with the hype cycle. "New" is prioritized over secure and stable. The mantra has become: - "It will be fixed in the next version" - "We need automatic restarts when it crashes" - "Do we need software that crashes less? We have systemd and Kubernetes to restart crashed workloads!" - "We need moooarrr powaaaaaaar!!!!"

Let me give you a concrete example. A program written in Rust should be memory safe - that's one of the main selling points of the language. But if that program uses unsafe functions and segfaults, what advantage does it offer over a mature C implementation? Stability matters more than the implementation language.

I solve problems. And creating a monoculture does not solve problems - it creates new ones.

Yes, Linux, Docker, and Kubernetes are better than closed source solutions. But when everyone uses the same tools, freedom dies. We use them because "everyone does" rather than because they're the best tool for our specific needs.

If we had only used what everyone else used, we wouldn't have Linux or the BSDs today. There would be no LibreOffice, no Nextcloud. We'd just have Windows variations and expensive Unix systems. We'd be bound by licenses and vendors, stuck with closed solutions.

This is where the BSDs offer a compelling alternative: "Be free and evaluate alternatives. Always."

For those who don't know, the original BSD started in the 1970s (before Linux was conceived). Minix was created as an educational OS because it was believed that BSD, mature and professional, would be the Open Source OS that would dominate the market. A legal case stalled development and scared adopters, but in 1993, NetBSD and FreeBSD emerged. OpenBSD forked from NetBSD later, then DragonflyBSD from FreeBSD.

As Linus Torvalds said in 1993, "If 386BSD had been available when I started on Linux, Linux would probably never had happened."

What makes the BSDs special is their philosophy: - Kernel and userland developed by same teams - Consistency in tools and updates - Excellent documentation - especially OpenBSD, where insufficient docs are considered a bug - Man pages contain virtually everything - Evolution, not Revolution

Let me briefly introduce the main BSD variants that I work with daily:

FreeBSD is a generalist system. It focuses on stability and performance - with HardenedBSD as a security-enhanced fork. It has native ZFS, Boot Environments, and complete separation between OS and packages. It's had container support via jails since 2000 - which predates Linux cgroups by a decade! It offers bhyve virtualization (more efficient than KVM). OPNsense and pfSense are based on FreeBSD, as pf is a powerful firewall. It's used by Netflix for streaming video delivery and forms the foundation for PlayStation consoles. MacOS and iOS also contain some FreeBSD code.

OpenBSD focuses on security and code correctness. Its code is constantly audited and simplified - less is more. The team believes "The more complex the code, the less maintainable." It has security mechanisms like pledge() and unveil(). OpenSSH (and many other nice things) originated and are developed here. Development is driven by team priorities, not user requests. It's ideal for routers, firewalls, and security-critical systems.

NetBSD lives by the motto "Of course it runs NetBSD!" Its focus is on correctness, portability, and proper implementation. It supports 50+ architectures. Development centers on compatibility, which necessitates code quality. It must function on decades-old hardware. It's ideal for systems that require stability without the need for continuous updates, like embedded devices.

I solve problems. And in my experience, the BSDs have consistently proven to be excellent problem-solvers. Here are some real-world benefits I've experienced:

Better stability and security
Simplified administration - upgrades won't destroy your system
Less vulnerability to common attacks - "We don't need this patch, you're running OpenBSD and it's been fixed 20 years ago"
Network interfaces maintain consistent names - ix0 will remain ix0, not renaming from enx3e3300c9e14e to enp10s0f0np0
FreeBSD shows lower system load compared to Linux
FreeBSD handles I/O pressure better - on the same hardware, I've seen 70% time reduction
FreeBSD delivers improved end-user experience/responsiveness
NetBSD provides the comfort of "Don't worry - your platform will be supported for the foreseeable future"

So why choose BSD in 2025? I believe there are several compelling reasons:

Security in an increasingly hostile environment
Stability in a world obsessed with novelty
Performance without unnecessary complexity
Freedom from the mainstream monoculture
Systems designed with coherent philosophy

Don't be afraid to try BSD systems - despite the Beastie mascot, they don't hurt and you'll appreciate them!

See you at BSD Cafe!

I Solve Problems

Stefano Marinelli — Thu, 03 Oct 2024 08:53:00 +0200

Addendum: during the event, I was interviewed by Jason Tubnor for the BSD Now Podcast, where I provided further information about the talk and the BSD Cafe project. Here is the link to the episode

This is the text underlying my presentation at EuroBSDCon 2024, on 21 September 2024, in Dublin, Ireland and BSDCan 2025, 13 June 2025, Ottawa, Canada.

The slides can be downloaded here

The BSDCan 2025 video can be viewed here. This is more up-to-date than the EuroBSDCon one.

The EuroBSDCon 2024 video, not yet separated from the live stream, can be viewed here - At first, I was a bit tense, then I relaxed.

Happy reading!

EuroBSDCon Dublin - 21 September 2024 - Why (and how) we're migrating many of our servers from Linux to the BSDs

"I'm Stefano Marinelli, I solve problems."

I’m the founder and Barista of the BSD Cafe, a community of *BSD enthusiasts.

I work in my company, called Prodottoinrete - a container of ideas and solutions.

I’m passionate about technology and computing, and I’ve made my passion my profession. Every morning, when I sit in front of the computer, a new world opens up for me to explore, and I try to share this passion with my clients. Sometimes, I succeed.

I've been a Linux user since 1996, before I turned 17. Back then, I used Fidonet and would read about alternative operating systems. Curiosity got the best of me, and I bought a set of CDs with various Linux distributions at the first opportunity. I tried it, I liked it, but at the time, I didn’t find it advantageous, so I continued using it for secondary tasks while Windows remained my daily driver.

Things changed in late 1997. I decided to go deeper into Linux and, with the purchase of a new, more powerful computer, I realized that aside from gaming, Linux could be my everyday system. This came in handy when I started university in 1998, where the computer science department was very oriented towards Open Source solutions. I was one of the few students who already understood the concepts of Open Source and knew how to use Linux. I was also one of the few who wasn’t baffled when we found Solaris machines in the lab. After all, there were similarities.

Over the years, I became one of the administrators of that lab, which was almost entirely Linux-based.

In 2000, I was fortunate to have Professor Özalp Babaoğlu as my lecturer, which pushed me to explore other operating systems like the BSDs. However, all I had at the time was an old Compaq laptop (486/25 MHz and 4 MB RAM) and no fast internet connection. So, apart from some theoretical studies, I postponed hands-on experiments until I had better resources. By 2002, thanks to a broadband connection and a new computer, I began exploring BSD systems. I started with FreeBSD, largely thanks to its fantastic handbook. I asked my parents to buy a laser printer so I could "print academic material" - but really, I wanted to print all the documentation I could find, starting from the FreeBSD handbook. And it was incredibly helpful.

Before long, FreeBSD became my daily driver - entire nights spent compiling KDE while I slept a meter away from my laptop’s wildly spinning fans - but FreeBSD ran much, much better on that machine than Linux did. Unfortunately, OpenBSD was too slow to be usable with any graphical interface.

In 2003, my final thesis focused on virtualization on Open Source systems, NetBSD/Xen was one of the best and most efficient solutions I tested.

Shortly after, I found a job at a company that was just beginning to offer Linux-based solutions, mainly for web and mail servers, but I was criticized because I completed tasks that didn’t require constant interventions - this hurt the company’s billing. That’s when I set my future guidelines:

I would work for myself, following my own philosophy.
I would not be tied to any particular vendor. I love exploring and learning, so I would always study solutions in depth and recommend the one I thought best suited for the client.
I solve problems - I don’t sell boxes.
I would use and promote Open Source solutions whenever possible.
I would use a BSD whenever feasible and Linux where a BSD was not suitable.

In every technological decision, my priority is solving my clients’ specific problems, not selling a predefined solution. I solve problems.

I was often told that Open Source systems were “toys for universities” and that the real world ran on something else (mostly meaning Windows). I pressed on. In some cases, I offered to be paid only if the results were achieved, showing how much they’d save compared to traditional licensing costs. It worked, but...

They accepted Linux, albeit reluctantly, but rejected BSDs because they didn’t know them. In some cases, I managed to convince them. In others, sadly, I didn’t.

Result: I decided to use the BSDs where clients didn’t have direct access - like email servers or web hosting - while using Linux where they specifically requested it.

NetBSD/Xen as the virtualization base, with excellent longevity and reliability.
OpenBSD as network/firewall entry points.
FreeBSD (especially with the introduction of ZFS) for various services, including backup servers.

It worked. What surprised clients most was the stability and the reduced need for maintenance. They saw more uptime and were happy.

Problem: "If nothing is working, what am I paying you for? If everything’s working, what am I paying you for?" So, I selected clients and situations that understood that if everything works, it’s because of the work behind the scenes. It’s better to pay for everything to work than to pay to fix problems. The problem to solve, in this case, is not stopping the client’s work. And I solve problems.

I managed about 65% *BSD machines and 35% Linux. But as Linux’s popularity grew, so did client demand. It became necessary on multiple occasions to replace or implement Linux instead of a BSD, for specific requests.

At a certain point, Linux virtualization solutions matured, and many of my hosts transitioned from NetBSD/Xen to OpenNebula (often with MooseFS) and then to Proxmox (often with Ceph). This happened because my clients needed to manage their VM lifecycles autonomously, change configurations, migrate hosts, etc.

Proxmox showed excellent reliability and stability. The VMs were often Linux-based (especially if the client needed direct management) or FreeBSD, but in smaller quantities. My own infrastructure, however, remained primarily BSD-based.

Over time, I gradually started to reflect and 'take stock'.

No *BSD has ever caused me to lose data to the point of having to restore from backup. On Linux, I’ve lost data with ext4, XFS, and btrfs. The most catastrophic case was with XFS - just a few files, backed up and restored, but the client was highly demanding and didn’t take it well. The largest failure was with btrfs - after a reboot, a 50 TB filesystem (in mirror, for backups) simply stopped working. No more mounting possible. Data was lost, but I had further backups. The client was informed and understood the situation. Within a few days, the server was rebuilt from scratch on FreeBSD with ZFS - since then, I haven’t lost a single bit.

In 2018, I started introducing Docker and Podman to the developers with specific needs, especially to help them with their development. Many developers began incessantly asking for Docker, unfortunately some of them only wanted to bypass many limitations that traditional setups imposed. Unfortunately, in many cases, those "limitations" were just bad practices - running outdated versions of software and libraries, or not wanting to bother with keeping the stack updated. There are similar problems with other solutions, but at least I can block them. They're great for quick deployment and increased security, but they aren’t always the best choice. Plus, they’re not the only option, despite what many people think these days.

Linux has had major development over the past years, but this has shifted towards specific players’ interests (mainly cloud providers) rather than technical reasons. Maybe not in the kernel, but many Linux distributions and communities now seem focused on the constant push to replace "the old with the new" with solid theoretical reasons but sometimes seemingly without practical benefit.

For me, computing should solve problems and provide opportunities to those who use it. I solve problems. Every change or variation will solve one problem but create new ones. It’s crucial to be mindful not to create worse problems than the ones you’re solving, and today’s enterprise world often seems to overlook that.

It’s common to see software distributed solely via Docker Compose files. Sometimes I use it as an installation tutorial, but I realize they’re just specific pieces precariously glued together (patched files, specific dependency versions, or nothing works).

The trend is to rush, to simplify deployments as much as possible, sweeping structural problems under the rug. The goal is to "innovate", not necessarily improve - just as long as it’s "new" or "how everyone does it, nowadays".

A massive business has grown around Linux: certifications, training, pentesting, certified platforms - the community is losing decision-making power. This is a far cry from the early days.

I solve problems. And these fast-changing technologies risk creating more problems than they solve, at least in certain situations. The workloads I manage often stay up for years, requiring a more stable, upgradable, and consistent approach.

If a client’s problem is to have an e-commerce site, they don’t really care if it runs on Docker, Podman, a FreeBSD jail, or a Raspberry Pi cluster - as long as their problem is solved. In fact, clients are happy when their solution is stable, upgradable, and secure.

There isn’t a single solution to all problems, but many solutions for each problem. My job is to give clients the best solution to solve their specific problem, not the most fashionable one.

That’s why I decided, several years ago, to reverse the proportion and implement and the BSDs for all possible workloads.

Each *BSD has its characteristics and target audience. Sometimes these targets overlap, but it’s generally not difficult to choose the most appropriate solution.

The goal of the migration was to create stable, coherent, upgradable, and secure systems.

By implementing an OpenBSD system, I often don’t need to install any additional packages. When it’s time to upgrade, it’s simple and secure. When a vulnerability emerges, I’m often fortunate to read "OpenBSD is excluded from this issue because it eliminated this risk X years ago..."

By implementing a NetBSD system, I know I’m installing a system that isn’t in a rush to release new versions and will likely run for years with only a few package updates and security patches, when necessary. And it’s quite rare for a patch to be required.

By implementing a FreeBSD system, I know I’ll have ZFS at my disposal, a fast and efficient hypervisor like bhyve, and a native, mature jail system that will ensure service and setup separation coherently and precisely. Not as “add-ons” but built into the system itself.

Moving to the BSDs means, in my experience, reaching systems that are more stable, more easily upgradable, and more consistent in their parts. They don’t chase the hype of the moment, much like the early days of Linux. In the case of FreeBSD, this also means moving to native ZFS and boot environments, giving me greater peace of mind when it comes to upgrades.

The initial strategy was to migrate what would soon need updates anyway, what would be moved (thus requiring a new setup), and what was causing problems and deserved a deeper dive.

First, I decided to migrate to FreeBSD the hypervisors not directly accessed by clients, especially on leased servers. The approach was to create a twin machine (to have an objective performance and stability comparison - it wouldn’t make sense to compare it to a different machine), install FreeBSD, bridge it with the production server, install vm-bhyve, and start copying VMs from Proxmox, reconfiguring the main parameters in the configuration file. In some cases, I already used ZFS on Proxmox, so the first part of the migration was a simple zfs-send/receive. In other cases (e.g., when using Ceph), I made an intermediate move by live-migrating the storage to ZFS and then proceeding as in the first case. The first noticeable effect was a reduction in resources used by the host to handle VM traffic - as expected, only FreeBSD’s basic processes and bhyve were running - but at the same time, there was a significant increase in I/O performance, further enhanced by switching the virtual disk driver from virtio to NVMe. This allowed for in-depth testing and revealed that FreeBSD suffers less under heavy I/O (the VMs on Linux tended to block their I/O, an effect I didn’t notice on FreeBSD) and showed significantly lower loads. In short, it handled the load better.

As an experiment, I decided to migrate two hosts (each with about 10 VMs) of a client - where I had full control—without telling them, over a weekend. By Tuesday, they called me, concerned: they had noticed a massive performance boost and were worried I had upgraded their hardware without approval, thinking that, "given the performance boost", it would cost them a lot. After explaining what I had done, they asked me to run further tests and progressively continue migrating everything. 20 hosts, all based on (sometimes slightly older) versions of Proxmox. And so I did, but taking advantage of their open-mindedness, I went a step further.

Many of their VMs handled simple workloads - PHP websites, or Java-based management systems (running on Tomcat), device monitoring software for industrial machinery, etc. The decision to use VMs had been made by their highly competent internal IT staff to separate environments and dependencies. It was a perfect use case for jails. I decided to try, VM by VM, to replicate the setups inside FreeBSD jails. In some cases, for convenience, I managed to run everything directly in Linux jails (with Linuxulator); in others, it was impossible, so I recreated the setups in individual jails. They immediately noticed an effect: faster operations. It didn’t surprise me. We avoided double buffering (VM and OS), so all the saved RAM could be used by ZFS for its cache and by the host to run other services. Some VMs remained as VMs (e.g., Zimbra). By the end of the operation, I had drastically reduced the number of VMs, replaced by jails, and consequently, the number of hosts. From 20 down to 11 - with a significant monthly cost saving.

The road was now clear, and I progressively continued down this path. One of the most interesting anecdotes: a client told me that they used to start an operation before taking a coffee break, around 15 minutes, to find the task almost done by the time they returned. After the migration, they shared that they launched the process, grabbed their things, and the task was already complete. An estimated reduction from about 18 minutes to 6 minutes on average. I didn’t investigate too much, but I suspect a combination of factors, with the predominant one being bhyve’s NVMe driver.

The main challenge I often face is ideological. Some people are used to thinking that the ideal solution is X - and believe that X is the only solution for their problems. Often, X is the hype of the moment (a few years ago, I fought to convince people that VMware wasn’t necessary and that Proxmox would be a great solution; today, Proxmox is on everyone’s lips - but it’s not the only solution). Often, X is a "cloud" cluster with Kubernetes - running WordPress on it. Even for hosting a law firm’s website, which will be updated every five years.

When I ask, "Okay, but why? Who will manage it? Where will your data really be, and who will safeguard it?", I get blank faces. They hadn’t considered these questions. No one had even mentioned them. "But everyone I spoke to proposed this type of solution...". It’s like at the beginning with Windows, when I proposed *BSD or Linux. Or later with VMware, when I proposed Proxmox. Or now with Kubernetes, when I propose the BSDs.

But the simplest solutions are the easiest to maintain and manage over time. My experience has taught me that setting something up is often the easiest part. The hardest part is returning to it after 1, 5, 10 years. Keeping it running, updating it, stabilizing it. For many, IT isn’t their business, but a tool to achieve their goals. A Kubernetes cluster is fantastic, but it requires maintenance. Or it’s external - so it’s no longer ours. We’ve lost control of the data. For many, it’s unnecessary to complicate things. And with every additional layer, we’re creating more problems.

No BSD system has ever surprised me during an update or a simple reboot. I’ve never encountered, for example, a network interface renaming from enx3e3300c9e14e to enp10s0f0np0* on Linux, effectively locking me out of a server. ix0 will remain ix0.

I’ve never had to recompile ZFS on FreeBSD, only to find that the module wouldn’t load, blocking the filesystem from mounting after reboot.

Many of the developers I work with have embraced the challenge. Most of them are passionate about technology, and learning a new operating method has been very interesting. Almost all of them, after experiments, were positive and, in fact, began explicitly requesting "jails" instead of Docker hosts. They started using BastilleBSD to clone "template" jails and deploy them. They learned to access ZFS automatic snapshots and recover lost files. They learned to manage the resources at their disposal without repeating the usual mantra of "we need mooooar powaaaar!" every time there’s a problem, a slowdown, or a storage overload. They’ve returned to trying to understand what’s happening, rather than just assembling pieces, libraries, and containers without considering the effects.

Others, however, struggled, but remained positive nonetheless. And that’s okay. I solve problems and can’t force my solutions on everyone.

In other cases, the challenge wasn’t technical but "commercial". Often, decision-makers have little to no technical knowledge. Linux sells well. "Cloud" sells even better. A "NetBSD-based solution", unfortunately, has less commercial appeal today. So, they want what they can sell, without focusing too much on the advantages of alternative solutions.

Linux today is subject to many compliance requirements - I’m often asked which version of OpenSSH I’m running - and they complain when the version (the latest from OpenBSD) isn’t considered "secure" because it doesn’t match their procedures (e.g., OpenSSH_9.2p1 Debian-2+deb12u3). They don’t understand when I explain that the "Debian" part refers to the Linux distribution, not a release. Those who prepare these documents are often, sadly, unaware of what they’re asking for. They just have a checklist.

The transition is ongoing, and as I see opportunities, I’m migrating from Linux to the BSDs whenever possible. Today, I can say that 78% of the hypervisors I manage run on FreeBSD, and 66% of the workloads (VPS, jails, hosts, etc.) are running on one of the BSDs - including solutions like OPNsense. None of the clients have experienced major issues. No one has complained about performance or reliability. All feedback has been positive. Many appreciate moving away from IT monocultures, especially when problems arise - following the recent severe SSH vulnerability, many clients contacted me, worried. It was nice to tell some of them that the exposed SSH service was running OpenBSD’s version, and that OpenBSD wasn’t affected as they had developed a secure mechanism back in 2001. They appreciated it. They want more setups based on OpenBSD.

I'm Stefano Marinelli, I solve problems. And I love solving problems using BSD systems.

Make your own Read-Only Device with NetBSD

Stefano Marinelli — Tue, 10 Sep 2024 01:41:00 +0200

One detail that is often overlooked when dealing with embedded (or remote) devices is a key point of vulnerability: the file system.

For non-COW file systems (like ext4 on Linux, FFS, etc.), there are situations where a crash or a power outage could cause corruption, requiring manual intervention. This risk is especially present when using root on SD cards or similar storage media in embedded systems. Over time, these media are destined to fail due to numerous writes.

For certain use cases, it's advisable to set up a read-only root file system, which ensures better reliability in case of system issues. Think of scenarios like a router (critical for network access) or a caching reverse-proxy, such as the one described in my series "Make your own CDN".

While FreeBSD natively supports this configuration and some Linux distributions offer custom solutions (e.g., Alpine Linux), NetBSD stands out as an excellent choice for such devices. It supports nearly all embedded devices, is lightweight, and its stability minimizes the need for frequent updates.

The Key Idea

Although NetBSD doesn't provide native read-only support, it's flexible enough to allow for this configuration. Many years ago, I followed a howto to set up a read-only system, and the idea remains simple and effective: NetBSD writes primarily to specific locations, unlike some Linux distributions that attempt to write to various parts of the file system. On NetBSD, the main write targets are the /tmp directory and /var.

With this in mind, we can configure these directories to reside in memory file systems (mfs) and ensure that /var contains everything necessary for the system to function correctly.

1. Prepare the Environment

Start with a basic NetBSD installation (I'll skip the installation steps as there are many tutorials available, and it's straightforward).

As a first step, clean up the /var directory. After each reboot, its contents will be extracted into the mfs and occupy RAM.

Disable `man.db` generation and swap usage:

Edit the following configuration files:

# /etc/rc.conf
makemandb=NO
no_swap=YES

# /etc/daily.conf
run_makemandb=NO

Then, remove the current man.db file:

rm /var/db/man.db

Once this is done, create a compressed archive of the current /var contents to be replicated at every boot:

cd /
tar -cvzf var-image.tar.gz var

2. Create a Custom Startup Script

Next, create a file called /etc/rc.d/mount_mfs_fs with the following content:

#!/bin/sh
#
# mount_mfs_fs: mount memory file system for /var
# by roby, 23 jun 2003 - adapted by Stefano - 01 Sep 2024

# PROVIDE: mount_mfs_fs
# REQUIRE: root

. /etc/rc.subr

name="mount_mfs_fs"
start_cmd="mount_mfs_fs_start"
stop_cmd=":"

mount_mfs_fs_start()
{
    # Check if the /var entry is present and uncommented in /etc/fstab
    if grep -q '^tmpfs[[:space:]]\+/var[[:space:]]\+tmpfs[[:space:]]\+rw,-m1777,-sram%25' /etc/fstab; then
        echo "Mounting memory file system: /var"

        # Mount the file system for /var
        mount /var

        # Extract the contents of the tar file into /var
        tar -xvzpf /var-image.tar.gz -C /

        echo "Mounting memory file systems: Done."
    else
        echo "The tmpfs entry for /var is not present or is commented out in /etc/fstab. Skipping mount and extraction."
    fi
    sleep 5
}

load_rc_config $name
run_rc_command "$1"

Make the script executable:

chmod a+rx /etc/rc.d/mount_mfs_fs

3. Modify Boot Process

Now, modify the /etc/rc.d/mountcritlocal file to require this script to run before its execution:

#!/bin/sh
#
# $NetBSD: mountcritlocal,v 1.17 2022/02/20 14:42:07 alnsn Exp $
#

# PROVIDE: mountcritlocal
# REQUIRE: mount_mfs_fs

$_rc_subr_loaded . /etc/rc.subr

name="mountcritlocal"
start_cmd="mountcritlocal_start"
stop_cmd=":"

mountcritlocal_start()
{
    #       Mount critical file systems that are `local'
    #       (as specified in $critical_filesystems_local)
    #       This usually includes /var.
    #
    mount_critical_filesystems local || return $?
    if checkyesno zfs; then
        mount_critical_filesystems_zfs || return $?
    fi
    return 0
}

load_rc_config $name
load_rc_config_var zfs zfs
run_rc_command "$1"

4. Configure `/etc/fstab`

Edit the /etc/fstab file to mount /tmp and /var in memory:

tmpfs           /tmp        tmpfs   rw,-m1777,-sram%25
tmpfs           /var        tmpfs   rw,-m1777,-sram%25

5. Test the Configuration

You can now reboot the system and check if everything works correctly. Once logged in, the df command should show something like this:

Filesystem      1K-blocks         Used        Avail %Cap Mounted on
/dev/ld0a        18298254       393938     16989404   2% /
tmpfs              524192         4224       519968   0% /var
kernfs                  1            1            0 100% /kern
ptyfs                   1            1            0 100% /dev/pts
procfs                  4            4            0 100% /proc
tmpfs              524192            4       524188   0% /var/shm
tmpfs              524192            4       524188   0% /tmp

6. Set Root to Read-Only

Currently, the system is still running in read-write mode, but /var and /tmp are in memory disks. To switch the root file system to read-only, edit the /etc/fstab file like this:

/dev/ld0a               /       ffs     ro               1 1

On the next reboot, the system will be in pure read-only mode.

7. Perform Updates

To install packages, update the system, or make any changes, you'll need to temporarily switch back to read-write mode. Comment out /var in mfs, change the root file system back to read-write, and reboot:

mount -uw /
vi /etc/fstab
/dev/ld0a               /       ffs     rw               1 1
[...]
#tmpfs           /var    tmpfs   rw,-m1777,-sram%25
[...]

After making the necessary changes, regenerate the /var tarball as done in step 1.

Make Your Own CDN with NetBSD

Stefano Marinelli — Tue, 03 Sep 2024 01:41:00 +0200

Introduction

This article is a spin-off from a previous post on how to create a self-hosted CDN, based on OpenBSD, but this time we'll focus on using NetBSD. The idea is to create reverse proxies with local caching. These proxies would cache the content on the first request and serve it directly afterward. The proxies would be distributed across different regions, and the DNS would route requests to the nearest proxy based on the caller’s location. All this is achieved without relying on external CDNs, using self-managed tools instead.

NetBSD is a lightweight, stable, and secure operating system that supports a wide range of hardware, making it an excellent choice for a caching reverse proxy. Devices that other operating systems may soon abandon, such as early Raspberry Pi models or i386 architecture, are still fully supported by NetBSD and will continue to be so. Additionally, NetBSD is an outstanding platform for virtualization (using Xen or qemu/nvmm) and deserves more attention than it currently receives.

The choice of Varnish is based on several factors, with the main ones being the ability to keep the cache in RAM (which means it can run on read-only systems) and the ability to flush the cache remotely. For example, with each change to my blog, I can choose whether to perform an immediate flush (such as for a new article or an error) or wait for the cache's "natural" expiration (such as for a typo or minor, non-critical changes).

While I won't detail the installation process for NetBSD, as it depends heavily on the hardware you have available, I will guide you through setting up a self-hosted CDN using NetBSD, Varnish, nginx, and the acme.sh or lego tool for SSL certificate management.

Installation

During the installation of NetBSD, ensure that you enable support for binary package management. This will install pkgin, a tool that simplifies package management on NetBSD. If you skip this step during installation, you can still install pkgin later, but it's easier to let the installer handle it.

Once your system is up and running, use pkgin to install the necessary packages: Varnish, nginx, and, depending on your preference, either acme.sh or Go (if you plan to compile lego). Although lego is not available as a precompiled package, you can easily compile it locally using Go, but for simplicity, I recommend using acme.sh.

For this setup, I will present two methods for generating and renewing certificates: using acme.sh or compiling lego - to reach the same final outcome as the OpenBSD article.

Option 1: Using acme.sh (Recommended)

acme.sh is a simple, yet powerful, shell script that handles certificate generation and renewal with ease. To install acme.sh:

pkgin in acmesh varnish nginx

Once acme.sh is installed, you can proceed to configure it for certificate management. It supports DNS authentication and integrates with many DNS providers, making it a flexible choice.

Option 2: Compiling Lego

If you prefer to use lego, you will need to compile it manually, as it is not available as a precompiled package for NetBSD. First, install Go and other necessary packages:

pkgin in go varnish nginx

To compile lego, you’ll need some disk space. Since the /tmp directory in NetBSD is often mounted as tmpfs (using RAM), you may run out of space during compilation if your system has limited memory. You can temporarily disable tmpfs by editing /etc/fstab and commenting out the relevant line:

#tmpfs           /tmp    tmpfs   rw,-m=1777,-s=ram%25

After rebooting, compile lego:

export GO111MODULE=on
go122 install github.com/go-acme/lego/v4/cmd/lego@latest

cp go/bin/lego /usr/pkg/bin/

Once lego is compiled and installed, you can uncomment the /tmp line in /etc/fstab and reboot again.

Configuring Varnish and nginx

First, copy the necessary rc.d scripts for nginx and Varnish:

cp /usr/pkg/share/examples/rc.d/nginx /etc/rc.d/
cp /usr/pkg/share/examples/rc.d/varnishd /etc/rc.d/

Then, add the following to /etc/rc.conf to enable and configure nginx and Varnish:

nginx=YES
varnishd=YES
varnishd_flags="-f /usr/pkg/etc/varnish/default.vcl -T localhost:9999 -a "/var/run/varnish.sock",user=nginx,group=varnish,mode=660 -s default,500m"

In this configuration, Varnish listens on a Unix socket, and nginx connects to it. This approach is more efficient and helps avoid some issues that may arise when exposing Varnish over an IP/port.

Creating the Varnish VCL Configuration

Next, create the VCL configuration file for Varnish at /usr/pkg/etc/varnish/default.vcl:

vcl 4.1;
import std;

# Backend - it-notes.dragas.net
backend it_notes {
    .host = "myBackendIP";
    .port = "80";
}

# ACL - purge - it-notes.dragas.net
acl purge_it_notes {
    "allowedToPurge_IP";
}

sub vcl_recv {
    # it-notes.dragas.net
    if (req.http.Host == "it-notes.dragas.net") {
        set req.backend_hint = it_notes;
        set req.http.Host = "it-notes.dragas.net";

        # PURGE - it-notes.dragas.net
        if (req.method == "PURGE") {
            std.log("Purge request received for " + req.url);

            if (!std.ip(req.http.X-Forwarded-For, "0.0.0.0") ~ purge_it_notes) {
                return (synth(405, "Not allowed."));
            }

            if (req.url == "/" || req.url == "/*") {
                ban("req.http.host == " + req.http.host);
                return(synth(200, "Entire cache has been cleared."));
            }
            return (purge);
        }

    } else {
        # Other domains - 404
        return (synth(404, "Domain not found"));
    }

    if (req.method != "GET" && req.method != "HEAD") {
        return (pipe);
    }

    return (hash);
}

sub vcl_backend_response {
    # TTL - it-notes.dragas.net
    if (bereq.http.host == "it-notes.dragas.net") {
        if (bereq.url ~ "\.(gif|jpg|jpeg|png|webp|ico|css|js)$") {
            set beresp.ttl = 1w;
            set beresp.grace = 1d;
            set beresp.keep = 7d;
            unset beresp.http.Set-Cookie;
            unset beresp.http.Cache-Control;
            set beresp.http.Cache-Control = "public, max-age=604800";
        } else {
            set beresp.ttl = 15m;
            set beresp.grace = 48h;
            set beresp.keep = 7d;
        }
    }

    # Remove some headers
    unset beresp.http.Server;
    unset beresp.http.X-Powered-By;
    unset beresp.http.Via;

    return (deliver);
}

sub vcl_deliver {
    # Add X-Cache header
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
    } else {
        set resp.http.X-Cache = "MISS";
    }

    std.log("Delivering content for " + req.url + " - Cache: " + resp.http.X-Cache);

    # Remove Varnish headers
    unset resp.http.Via;
    unset resp.http.X-Varnish;

    return (deliver);
}

sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }
    return (lookup);
}

sub vcl_hit {
    return (deliver);
}

sub vcl_miss {
    return (fetch);
}

sub vcl_purge {
    std.log("Purge executed for " + req.url);
    return (synth(200, "Purge successful"));
}

sub vcl_synth {
    set resp.http.Content-Type = "text/html; charset=utf-8";
    set resp.http.Retry-After = "5";
    synthetic({"<!DOCTYPE html>
        <html>
            <head>
                <title>"} + resp.status + " " + resp.reason + {"</title>
            </head>
            <body>
                <h1>Status "} + resp.status + " " + resp.reason + {"</h1>
                <p>"} + resp.reason + {"</p>
                <h3>Guru Meditation:</h3>
                <p>XID: "} + req.xid + {"</p>
                <hr>
                <p>Varnish cache server</p>
            </body>
        </html>
    "});
    return (deliver);
}

Configuring nginx

Now, modify the nginx configuration file at /usr/pkg/etc/nginx/nginx.conf. Set the number of worker processes to "auto" to take advantage of all server cores, and configure the reverse proxy for your site(s). Here's an example configuration:

[...]
worker_processes  auto;
[...]

server {
    server_name it-notes.dragas.net;

    location / {
        proxy_method $request_method;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_pass http://unix:/var/run/varnish.sock;
    }

    access_log /var/log/nginx/access.it-notes.dragas.net.log;
    error_log /var/log/nginx/error.it-notes.dragas.net.log;

    listen [::]:443 ssl;
    listen 443 ssl;
    http2 on;
    # If you're using acme.sh, just change the location of the certificates
    ssl_certificate /root/.lego/certificates/it-notes.dragas.net.crt;
    ssl_certificate_key /root/.lego/certificates/it-notes.dragas.net.key;
}

server {
    if ($host = it-notes.dragas.net) {
        return 301 https://$host$request_uri;
    }
    server_name it-notes.dragas.net;
    listen 80;
    listen [::]:80;
    return 404;
}

Starting Varnish and nginx

Finally, start the Varnish and nginx services:

service varnishd start
service nginx start

If everything is configured correctly, both Varnish and nginx will be up and running, ready to handle incoming connections.

Conclusion

Congratulations, you have successfully set up your own CDN on NetBSD. This solution is lightweight, stable, and fully under your control, allowing you to break free from the constraints of major service providers. With NetBSD's broad hardware support and minimal overhead, this setup can run on a wide variety of devices, making it a versatile choice for self-hosted solutions.

I'm using it as a test and as a read-only root filesystem with a RAM-only local cache for my blog on a Raspberry Pi Zero W (first edition), and as soon as I get the new FTTH, I'll probably make it accessible via IPv6 for Italy, putting it physically into production.

If your goal is geo-replication, you can use DNS providers that offer location-based routing or set up your own DNS infrastructure to manage and resolve requests according to the user’s location. With multiple reverse proxies, separate DNS servers, and a well-configured cache, you can achieve a highly resilient system with minimal risk of a single point of failure.

That Old NetBSD Server, Running Since 2010

Stefano Marinelli — Sun, 27 Aug 2023 05:17:49 +0000

As 2010 was drawing to a close, I found myself on more flights than coffee breaks, constantly testing technical solutions, in search of stability and reliability.

One morning, a client called. They needed to operate various services on their internal network, which essentially meant reconfiguring the entire network behind their firewall. They required a dhcp, an internal DNS, an Apache + PHP server for some internal (and a couple of external) websites, a file server accessible via both NFS and Samba (as Windows PCs needed access), an internal SMTP connecting to an external relay to ensure faster email dispatches for employees given their unstable connectivity, and a few other nuances. My task was to set all this up within a tight two-day window. Given the constraints, I opted for my top choice of the time for such tasks: NetBSD.

I suggested the client invest in Enterprise-grade hardware. However, they insisted on using a server they already had, assembled by a local vendor with “gaming-quality” parts. Though they claimed these were high-quality, it was clearly consumer-grade — lacking dual power supply, remote management capability, and fitted with consumer-grade hard drives.

I always had a feeling that I wasn't their first choice for the job. Perhaps someone had bailed on them, and they had settled for me. They had originally intended to implement everything on Windows 2008 Server, but I persuaded them to try my solution.

With the hardware in hand and less than 48 hours on the clock, I dived in. The OS, as mentioned, was NetBSD 5.1. I compartmentalized the services using several Xen DomU (my favored solution at that time over the then-nascent KVM on Linux). This involved creating multiple partitions on two disks and setting up a unique RAID for each DomU.

After successfully setting everything up, including an external connection to my OpenVPN hub, I handed the server back to the client. Checking in a month later, the feedback was mostly positive, save for some SMB latency issues.

Over the next two to three years, I made occasional adjustments to the machine, but then lost touch with the client. By 2015, my OpenVPN hub was discontinued, taking with it many older or defunct servers, including that NetBSD server.

Fast forward to February 2021, the phone rang. It was that same client, seeking to integrate a new network configuration because of a new firewall. The fact that they were calling meant that the NetBSD server was still alive! Curious, I accepted the task.

To my astonishment, the server was still up & running perfectly. The external services were active but inaccessible, wisely kept hidden from potential threats, while the internal ones were functioning smoothly. NFS was working, as were SMB, the internal DNS, and the SMTP relay. The server was executing about 80% of its original tasks.

What surprised me the most was its uptime. Sadly, I didn't take note, but the last restart had been in 2012 after the Emilia Romagna earthquake. They had a backup generator, so the server always had an efficient uninterrupted power supply.

Nine years of uptime for a server set up in a few hours, on consumer-grade hardware, and left largely unattended for years.

I now understand why I'm not – and will never be – rich. My first (and last) employer complained about my preference for stable and reliable solutions, equating it with lesser profits. According to him, unstable solutions requiring frequent maintenance meant more revenue. To me, a job is well-done when it works consistently, not when it demands constant fixes.

I'm unsure if the server is still operational, but I'll certainly check when I get a chance. Nonetheless, I'm grateful for having trusted NetBSD, a lightweight, stable, secure, and efficient operating system, which, in my view, still doesn't receive the acclaim it truly deserves.