IT Notes - docker

Static Web Hosting on the Intel N150: FreeBSD, SmartOS, NetBSD, OpenBSD and Linux Compared

Stefano Marinelli — Wed, 19 Nov 2025 09:16:00 +0100

Update: This post has been updated to include Docker benchmarks and a comparison of container overhead versus FreeBSD Jails and illumos Zones.

Note: Some operating systems (FreeBSD and Linux) support kernel TLS (kTLS) and the related SSL_sendfile path in nginx, which can improve HTTPS performance for static files. Since this feature is not available on all the systems included in the comparison (for example NetBSD, OpenBSD and illumos), the benchmarks were run with a common baseline configuration that does not rely on kTLS. The goal is to compare the systems under similar conditions rather than to measure OS specific optimizations.

I often get very specific infrastructure requests from clients. Most of the time it is some form of hosting. My job is usually to suggest and implement the setup that fits their goals, skills and long term plans.

If there are competent technicians on the other side, and they are willing to learn or already comfortable with Unix style systems, my first choices are usually one of the BSDs or an illumos distribution. If they need a control panel, or they already have a lot of experience with a particular stack that will clearly help them, I will happily use Linux and it usually delivers solid, reliable results.

Every now and then someone asks the question I like the least:

“But how does it perform compared to X or Y?”

I have never been a big fan of benchmarks. At best they capture a very specific workload on a very specific setup. They are almost never a perfect reflection of what will happen in the real world.

For example, I discovered that idle bhyve VMs seem to use fewer resources when the host is illumos than when the host is FreeBSD. It looks strange at first sight, but the illumos people are clearly working very hard on this, and the result is a very capable and efficient platform.

Despite my skepticism, from time to time I enjoy running some comparative tests. I already did it with Proxmox KVM versus FreeBSD bhyve, and I also compared Jails, Zones, bhyve and KVM on the same Intel N150 box. That led to the FreeBSD vs SmartOS article where I focused on CPU and memory performance on this small mini PC.

This time I wanted to do something simpler, but also closer to what I see every day: static web hosting.

Instead of synthetic CPU or I/O tests, I wanted to measure how different operating systems behave when they serve a small static site with nginx, both over HTTP and HTTPS.

This is not meant to be a super rigorous benchmark. I used the default nginx packages, almost default configuration, and did not tune any OS specific kernel settings. In my experience, careful tuning of kernel and network parameters can easily move numbers by several tens of percentage points. The problem is that very few people actually spend time chasing such optimizations. Much more often, once a limit is reached, someone yells “we need mooooar powaaaar” while the real fix would be to tune the existing stack a bit.

So the question I want to answer here is more modest and more practical:

With default nginx and a small static site, how much does the choice of host OS really matter on this Intel N150 mini PC?

Spoiler: less than people think, at least for plain HTTP. Things get more interesting once TLS enters the picture.

Disclaimer
These benchmarks are a snapshot of my specific hardware, network and configuration. They are useful to compare relative behavior on this setup. They are not a universal ranking of operating systems. Different CPUs, NICs, crypto extensions, kernel versions or nginx builds can completely change the picture.

Test setup

The hardware is the same Intel N150 mini PC I used in my previous tests: a small, low power box that still has enough cores to be interesting for lab and small production workloads.

On it, I installed several operating systems and environments, always on the bare metal, not nested inside each other. On each OS I installed nginx from the official packages.

Software under test

On the host:

SmartOS, with:
- a Debian 12 LX zone
- an Alpine Linux 3.22 LX zone
- a native SmartOS zone

FreeBSD 14.3-RELEASE:
- nginx running inside a native jail

OpenBSD 7.8:
- nginx on the host

NetBSD 10.1:
- nginx on the host

Debian 13.2:
- nginx on the host

Alpine Linux 3.22:
- nginx on the host
- Docker: Debian 13 container running on the Alpine host (ports mapped)

I also tried to include DragonFlyBSD, but the NIC in this box is not supported. Using a different NIC just for one OS would have made the comparison meaningless, so I excluded it.

nginx configuration

In all environments:

nginx was installed from the system packages
worker_processes was set to auto
the web root contained the same static content

The important part is that I used exactly the same nginx.conf file for all operating systems and all combinations in this article. I copied the same configuration file verbatim to every host, jail and zone. The only changes were the IP address and file paths where needed, for example for the TLS certificate and key.

The static content was a default build of the example site generated by BSSG, my Bash static site generator. The web root was the same logical structure on every OS and container type.

There is no OS specific tuning in the configuration and no kernel level tweaks. This is very close to a “package install plus minimal config” situation.

TLS configuration

For HTTPS I used a very simple configuration, identical on every host.

Self signed certificate created with:

openssl req -x509 -newkey rsa:4096 -nodes -keyout server.key -out server.crt -days 365 -subj "/CN=localhost"

Example nginx server block for HTTPS (simplified):

server {  
listen 443 ssl http2;  
listen [::]:443 ssl http2;  

server_name _;  

ssl_certificate /etc/nginx/ssl/server.crt;  
ssl_certificate_key /etc/nginx/ssl/server.key;  

root /var/www/html;  
index index.html index.htm;  

location / {  
try_files $uri $uri/ =404;  
}  
}

The HTTP virtual host is also the same everywhere, with the root pointing to the BSSG example site.

Load generator

The tests were run from my workstation on the same LAN:

client host: a mini PC machine connected at 2.5 Gbit/s
switch: 2.5 Gbit/s
test tool: wrk

For each target host I ran:

wrk -t4 -c50 -d10s http://IP
wrk -t4 -c10 -d10s http://IP
wrk -t4 -c50 -d10s https://IP
wrk -t4 -c10 -d10s https://IP

Each scenario was executed multiple times to reduce noise; the numbers below are medians (or very close to them) from the runs.

The contenders

To keep things readable, I will refer to each setup as follows:

SmartOS Debian LX → SmartOS host, Debian 12 LX zone
SmartOS Alpine LX → SmartOS host, Alpine 3.22 LX zone
SmartOS Native → SmartOS host, native zone
FreeBSD Jail → FreeBSD 14.3-RELEASE, nginx in a jail
OpenBSD Host → OpenBSD 7.8, nginx on the host
NetBSD Host → NetBSD 10.1, nginx on the host
Debian Host → Debian 13.2, nginx on the host
Alpine Host → Alpine 3.22, nginx on the host
Docker Container → Alpine host, Debian 13 Docker container

Everything uses the same nginx configuration file and the same static site.

Static HTTP results

Let us start with plain HTTP, since this removes TLS from the picture and focuses on the kernel, network stack and nginx itself.

HTTP, 4 threads, 50 concurrent connections

Approximate median wrk results:

Environment	HTTP 50 connections
SmartOS Debian LX	~46.2 k
SmartOS Alpine LX	~49.2 k
SmartOS Native	~63.7 k
FreeBSD Jail	~63.9 k
OpenBSD Host	~64.1 k
NetBSD Host	~64.0 k
Debian Host	~63.8 k
Alpine Host	~63.9 k
Docker Container	~63.7 k

Two things stand out:

All the native or jail/container setups on the hosts that are not LX zones cluster around 63 to 64k requests per second.
The two SmartOS LX zones sit slightly lower, in the 46 to 49k range, which is still very respectable for this hardware.

In other words, as long as you are on the host or in something very close to it (FreeBSD jail, SmartOS native zone, NetBSD, OpenBSD, Linux on bare metal), static HTTP on nginx will happily max out around 64k requests per second with this small Intel N150 CPU.

The Debian and Alpine LX zones on SmartOS are a bit slower, but not dramatically so. They still deliver close to 50k requests per second and, in a real world scenario, you would probably saturate the network or the client long before hitting those numbers.

HTTP, 4 threads, 10 concurrent connections

With fewer concurrent connections, absolute throughput drops, but the relative picture is similar:

SmartOS Native around 44k
NetBSD and Alpine Host around 34 to 35k
FreeBSD, Debian, OpenBSD around 31 to 33k
The Docker Container sits slightly lower at ~30.2k req/s, showing a small overhead from the networking layer
The SmartOS LX zones sit slightly below, around 35 to 37k req/s

The important conclusion is simple:

For plain HTTP static hosting, once nginx is installed and correctly configured, the choice between these operating systems makes very little difference on this hardware. Zones and jails add negligible overhead, LX zones add a small one.

If you are only serving static content over HTTP, your choice of OS should be driven by other factors: ecosystem, tooling, update strategy, your own expertise and preference.

Static HTTPS results

TLS is where things start to diverge more clearly and where CPU utilization becomes interesting.

HTTPS, 4 threads, 50 concurrent connections

Approximate medians:

Environment	HTTPS 50 connections	CPU notes at 50 HTTPS connections
SmartOS Debian LX	~51.4 k	CPU saturated
SmartOS Alpine LX	~40.4 k	CPU saturated
SmartOS Native	~52.8 k	CPU saturated
FreeBSD Jail	~62.9 k	around 60% CPU idle
OpenBSD Host	~39.7 k	CPU saturated
NetBSD Host	~40.4 k	CPU saturated
Debian Host	~62.8 k	about 20% CPU idle
Alpine Host	~62.4 k	small idle headroom, around 7% idle
Docker Container	~62.7 k	CPU saturated

These numbers tell a more nuanced story.

FreeBSD, Debian and Alpine on bare metal form a “fast TLS” group.
All three sit around 62 to 63k requests per second with 50 concurrent HTTPS connections.
FreeBSD does this while using significantly less CPU.
During the HTTPS tests with 50 connections, the FreeBSD host still had around 60% CPU idle. It is the platform that handled TLS load most comfortably in terms of CPU headroom.
Debian and Alpine are close in throughput, but push the CPU harder.
Debian still had some idle time left, Alpine even less. In practice, all three are excellent here, but FreeBSD gives you more room before you hit the wall.
SmartOS, NetBSD and OpenBSD form a “good but heavier” TLS group.
Their HTTPS throughput is in the 40 to 52k req/s range and they reach full CPU usage at 50 concurrent connections. OpenBSD and NetBSD stabilize around 39 to 40k req/s. SmartOS native and the Debian LX zone manage slightly better (around 51 to 53k) but still with the CPU pegged.

HTTPS, 4 threads, 10 concurrent connections

With lower concurrency:

FreeBSD, Debian and Alpine still sit in roughly the 29 to 31k req/s range
SmartOS Native and LX zones are in the mid to high 30k range
The Docker Container drops slightly to ~27.8k req/s
NetBSD and OpenBSD sit around 26 to 27k req/s

The relative pattern is the same: for this TLS workload, FreeBSD and modern Linux distributions on bare metal appear to make better use of the cryptographic capabilities of the CPU, delivering higher throughput or more headroom or both.

What TLS seems to highlight

The HTTPS tests point to something that is not about nginx itself, but about the TLS stack and how well it can exploit the hardware.

On this Intel N150, my feeling is:

FreeBSD, with the userland and crypto stack I am running, is very efficient at TLS here. It delivers the highest throughput while keeping plenty of CPU in reserve.
Debian and Alpine, with their recent kernels and libraries, are also strong performers, close to FreeBSD in throughput, but with less idle CPU.
NetBSD, OpenBSD and SmartOS (native and LX) are still perfectly capable of serving a lot of HTTPS traffic, but they have to work harder to keep up and they hit 100% CPU much earlier.

This matches what I see in day to day operations: TLS performance is often less about “nginx vs something else” and more about the combination of:

the TLS library version and configuration
how well the OS uses the CPU crypto instructions
kernel level details in the network and crypto paths

I suspect the differences here are mostly due to how each system combines its TLS stack (OpenSSL, LibreSSL and friends), its kernel and its hardware acceleration support. It would take a deeper dive into profiling and configuration knobs to attribute the gaps precisely.

In any case, on this specific mini PC, if I had to pick a platform to handle a large amount of HTTPS static traffic, FreeBSD, Debian and Alpine would be my first candidates, in that order.

Zones, jails, containers and Docker: overhead in practice

Another interesting part of the story is the overhead introduced by different isolation technologies.

From these tests and the previous virtualization article on the same N150 machine, the picture is consistent:

FreeBSD jails behave almost like bare metal and are significantly more efficient than Docker.
For both HTTP and HTTPS, running nginx in a jail on FreeBSD 14.3-RELEASE produces numbers practically identical to native hosts.
The contrast with Docker is striking: while the Docker container required 100% CPU to reach peak for the HTTP and HTTPS throughput, the FreeBSD jail delivered the same speed with ~60% of the CPU sitting idle. In terms of performance cost per request, Jails are drastically cheaper.
SmartOS native zones are also very close to the metal.
Static HTTP performance reaches the same 64k req/s region and HTTPS is only slightly behind the "fast TLS" group, although with higher CPU usage.
SmartOS LX zones introduce a noticeable but modest overhead.
Both Debian and Alpine LX zones on SmartOS perform slightly worse than the native zone or FreeBSD jails. For static HTTP they are still very fast. For HTTPS the Debian LX zone remains competitive but costs more CPU, while the Alpine LX zone is slower.
Docker on Linux performs efficiently but eats the margins. I ran an additional test using a Debian 13 Docker container running on the Alpine Linux host. At peak load (50 connections), the throughput was impressive and virtually identical to bare metal: ~63.7k req/s for HTTP and ~62.7k req/s for HTTPS. However, there is a clear cost. First, while the bare metal host maintained a small CPU buffer (~7% idle) during the HTTPS test, Docker saturated the CPU to 100%. Second, at lower concurrency (10 connections), the overhead became visible. The Docker container scored ~30.2k req/s for HTTP and ~27.8k req/s for HTTPS, slightly trailing the ~31-34k and ~29-31k range of the bare metal counterparts. The abstraction layers (NAT, bridging, namespaces) are extremely efficient, but they are not completely free.

This leads to a clear conclusion on efficiency: FreeBSD Jails provide the highest throughput with the lowest CPU cost. LX zones and Docker containers can match the speed (or come close), but they burn significantly more CPU cycles to do so.

What this means for real workloads

It is easy to get lost in tables and percentages, so let us go back to the initial question.

A client wants static hosting.
Does the choice between FreeBSD, SmartOS, NetBSD or Linux matter in terms of performance?

For plain HTTP on this hardware, with nginx and the same configuration:

Not really.
All the native hosts and FreeBSD jails deliver roughly the same maximum throughput, in the 63 to 64k req/s range. SmartOS LX zones are slightly slower but still strong.

For HTTPS:

Yes, it starts to matter a bit more.
FreeBSD stands out for how relaxed the CPU is under high TLS load.
Debian and Alpine are very close in throughput, with more CPU used but still with some headroom.
SmartOS, NetBSD and OpenBSD can still push a lot of HTTPS traffic, but they reach 100% CPU earlier and stabilize at lower request rates.

Does this mean you should always choose FreeBSD or Debian or Alpine for static HTTPS hosting?

Not necessarily.

In real deployments, the bottleneck is rarely the TLS performance of a single node serving a small static site. Network throughput, storage, logging, reverse proxies, CDNs and application layers all play a role.

However, knowing that FreeBSD and current Linux distributions can squeeze more out of a small CPU under TLS is useful when you are:

sizing hardware for small VPS nodes that must serve many HTTPS requests
planning to consolidate multiple services on a low power box
deciding whether you can afford to keep some CPU aside for other tasks (cache, background jobs, monitoring, and so on)

As always, the right answer depends on the complete picture: your skills, your tooling, your backups, your monitoring, the rest of your stack, and your tolerance for troubleshooting when things go sideways.

Final thoughts

From these small tests, my main takeaways are:

Static HTTP is basically solved on all these platforms.
On a modest Intel N150, every system tested can push around 64k static HTTP requests per second with nginx set to almost default settings. For many use cases, that is already more than enough.
TLS performance is where the OS and crypto stack start to matter.
FreeBSD, Debian and Alpine squeeze more HTTPS requests out of the N150, and FreeBSD in particular does it with a surprising amount of idle CPU left. NetBSD, OpenBSD and SmartOS need more CPU to reach similar speeds and stabilize at lower throughput once the CPU is saturated.
Jails and native zones are essentially free, LX zones cost a bit more.
FreeBSD jails and SmartOS native zones show very little overhead for this workload. SmartOS LX zones are still perfectly usable, but if you are chasing every last request per second you will see the cost of the translation layer.
Benchmarks are only part of the story.
If your team knows OpenBSD inside out and has tooling, scripts and workflows built around it, you might happily accept using more CPU on TLS in exchange for security features, simplicity and familiarity. The same goes for NetBSD or SmartOS in environments where their specific strengths shine.

I will not choose an operating system for a client just because a benchmark looks nicer. These numbers are one of the many inputs I consider. What matters most is always the combination of reliability, security, maintainability and the human beings who will have to operate the
system at three in the morning when something goes wrong.

Still, it is nice to know that if you put a tiny Intel N150 in front of a static site and you pick FreeBSD or a modern Linux distribution for HTTPS, you are giving that little CPU a fair chance to shine.

From Cloud Chaos to FreeBSD Efficiency

Stefano Marinelli — Thu, 04 Jul 2024 08:41:00 +0200

Introduction

A few months ago, a client asked me to take care of their Kubernetes cluster (hosted on AWS and GCP). In their opinion, the costs were exorbitantly high for relatively simple and lean websites. Sure, they had many visits, but nothing too excessive development-wise.

I kindly declined. Unfortunately, their situation is all too common these days: they hired developers accustomed to working that way, convinced that a system administrator is now unnecessary because "the cloud has infinite potential." They were used to considering optimization as secondary because "we have infinite power" (and this is already a spoiler for the ending).

Being open to dialogue and new experiences, they asked for my opinion on the matter. We talked for a while, and I explained that, in my view, for the type of setup they had (standard, with various replicas and variants, but primarily based on two platforms), it didn't make sense. I saw it as complicating things. An over-engineering of something simple. Like taking a cruise ship to cross a river.

They then asked me to create something simple that would serve as a development server and for backups, to understand what kind of solution I had in mind.

The Solution

So, I started building everything. I began with FreeBSD 13.2-RELEASE, but in the meantime, 14.0-RELEASE came out, so that’s the version I delivered.

I installed the operating system on a physical server, leased from one of the main European providers. Benefiting from one of their auctions (good deals can be found on weekends), they found a sufficiently powerful machine, with 128GB of RAM, 2 NVMe drives of 1TB each, and two spinning disks of 2TB each for less than 100 euros per month. They also took another, less powerful one for additional backups and to back up the first one.

Implementation

I decided to keep the host as clean as possible and concentrated the services in jails (managed by BastilleBSD) and VMs. The machine was divided as follows:

A series of bridges - to be used for different projects. Jails of the same project and/or type use the same bridge and can communicate with each other, sharing some resources (MariaDB, etc.).
A bhyve VM with Alpine Linux - in my opinion, the best distribution for running Docker containers. Do we really need systemd just to launch Docker? They mainly use it as a pre-production test bench, connected via VPN to their company LAN. It is the core of their "online" development, i.e., outside their computers. It has 32GB of RAM, 200GB of disk (obviously bhyve is configured with NVMe drivers), and 4 cores assigned.
A VNET jail with a reverse proxy (nginx) - they know how to modify virtual hosts and generate certificates with certbot, pointing to the underlying jails.
A series of "empty" VNET jails, to be cloned, for each type of setup (they mainly have CMS based on WordPress and Laravel, so with all dependencies inside - nginx, php, redis, etc. except the databases).
A VNET jail with MariaDB installed, to be cloned, to be attached to different projects as needed.
zfs-autobackup performs local snapshots, keeping: one every 15 minutes for 3 hours, one per hour for 24 hours, one per day for 3 days.

Backups are also performed using zfs-autobackup and, in case of disaster recovery in rapid times, a zfs-send (and corresponding zfs-receive) every 10 minutes on another machine (the other, smaller one, also taken at auction), with the same bridges, firewall rules, BastilleBSD, and bhyve installed - ready to start in case of disaster. Being a test server, we didn't consider to implement a proper HA - at the moment, it wouldn't make sense.

They also have another job with zfs-autobackup that performs an additional backup on a server (Debian in their offices). Safe data, in my opinion, are those in storage under your b...ench.

I delivered everything to them and gave a brief course to the more experienced devs on how to manage things. No explanation on the Alpine Linux VM, but I showed them the jails, how to clone, configure, and manage them.

Real-world Testing

I didn't hear from them anymore. After a few weeks, one of the devs contacted me urgently because a junior unfortunately made a mistake and deleted an entire project from one of the jails. I explained that the local snapshots were restorable with a command, and he was thrilled. He restored both the development jail and the one with the database made two minutes before the "mishap" and they restarted immediately.

I realized that this event would change some of their procedures and criteria.

I hadn't heard from anyone for months. This morning, I received a call from their manager, whom I hadn't heard from since the beginning, and he told me how things had been going these months.

Lessons Learned

First, this person has good communication and commercial skills but little technical background. He is open-minded and tends to study carefully what is proposed to him. He doesn't discard any solution a priori, without having touched its pros and cons.

They had leased servers with cPanel and were inserting their content inside them. The devs who arrived a few years ago suggested making a technological transition, eliminating these "obsolete" servers and "outdated" methodologies, pushing everything to the cloud and containerizing everything. When we first talked, he told me how they were "lucky to make that transition because their load had increased enormously and the old servers probably wouldn't have handled the load", instead autoscaling saved them. I had some reservations about autoscaling without particular controls, but clearly, I cannot impose my choices on others.

To cut a long story short: seeing what happened with that junior dev's mistake (and the simplicity with which it was possible to restart immediately), they decided to increase the use of FreeBSD jails and reduce, at least on secondary loads, the use of their Cloud managed with Kubernetes. As they transitioned to jails, however, they noticed some slowdowns. These slowdowns worsened day by day. According to the devs, it would have been appropriate to go back to having, again, autoscaling ("we need moar powaaaaar!!!") but, fortunately, their boss decided to investigate carefully. They realized that these workloads (based on Laravel) were storing sessions on files. Over time, these millions of files (several gigabytes per day) slowed everything down because, for specific operations, Laravel scanned the entire directory. In other words, on the "cloud," they needed much more power than necessary (and much more disk space, but that was cheaper) to carry this load, which was, in fact, unnecessary. After realizing this, they moved the sessions to Redis. Needless to say, everything became extremely faster, even compared to the previous setup on Kubernetes and autoscaling.

At that point, it was clear that one of the problems with their setup is (as often happens) poor optimization. Today, there's a tendency to rush, "throw in" functions, features, libraries, plugins, etc. without considering the interactions and consequences. If it works, it's fine. Even if it increases computational complexity exponentially just to, for example, change the color of an icon (absurd example, but to give an idea).

They then started moving even the main Laravel workloads (thanks to the optimization implemented). At this point, they began moving some of the WordPress sites even though they were extremely concerned. In the cluster, every day, at fairly irregular intervals, the load would rise and everything would slow down until autoscaling started scaling up to the imposed limits. CPU at 100% on all containers, and the devs noticed that the load came from a series of "php" processes. Recreating the containers helped for some minutes, but did not solve the problem.

To their great surprise, all this did not happen on the FreeBSD jails. The load was significantly lower, without any of these spikes. Satisfied, they decided to use this as their final setup. One of the devs, however, wanted to get to the bottom of it and decided to run a test: he moved some of these WordPress sites to the Alpine VM, on Docker. At that point, the spikes resumed, saturating the CPU of the Alpine machine.

Without going into details, they eventually realized that there was a vulnerability in one (or more) of the many plugins installed on the WordPress sites, which was being exploited to inject a process, probably a cryptominer. The name given to the process was "php" - so the devs, not being system experts, did not worry about understanding better whether it was really php or another process pretending to be it. On FreeBSD, all this did not happen because the injected executable could not run - there was no Linux compatibility activated on the server.

Until then, they considered these (expensive) spikes as organic and did not worry too much about them. Paying to have their friendly intruders mine.

Conclusion

They asked me to help, as much as possible, to move other services to FreeBSD. It won't be easy, probably we will need to use bhyve a lot, but they decided that this is the platform they want to focus on in the coming years.

Undoubtedly, this is a success story of FreeBSD and, indirectly, of correct and careful management of one's resources. Too often today, there is the superficial belief that the cloud, with its "infinite" resources, is the solution to all problems. And that Kubernetes is the best solution for everything. I, on the other hand, have always believed that there is the right tool for everything. You can hammer a nail with a screwdriver, but it's not the most suitable and efficient tool.

Today they spend about 1/10 of what they used to spend before, they have more control over their data and the tools they use. Undoubtedly, all this was also caused by poor optimization and control by those who manage the infrastructure, but the question is: how often do people decide that, in the end, it is okay to spend more (especially if it is someone else's money) rather than go crazy for hours behind such a situation? While having defined and limited resources (albeit elevated) poses different problems - but of optimization. And in the age of energy and resource savings, it might be wise to give more importance to optimization.

Abundance led to waste.

The Double-Edged Sword of Docker: Balancing Benefits and Risks

Stefano Marinelli — Mon, 22 Apr 2024 05:30:00 +0000

As a systems administrator, I am deeply concerned about the consequences of the current widespread adoption of technologies like Docker. Having been a proponent and early adopter of containerization for many years, I recognized its potential early on and have been advocating for its use in many of the Linux-based setups I manage.

Initially, this relieved me of some headaches. One recurring issue was dealing with developers requesting "exotic" setups—by exotic, I mean specific (sometimes multiple) versions of PHP on the same VPS, or unique combinations of PHP and MySQL (or MariaDB) that required adding external repositories of all sorts—creating future problems when one of these repositories ceases to exist or be updated, leaving us with an unstable, dangerous, or unupgradable system.

In many cases, I resolved these issues by partitioning components into FreeBSD jails (one jail per service, one for data, with bind mounts as needed—perfect efficiency, excellent upgradability and stability, maximum security). However, this wasn't always feasible. Sometimes, the explicit use of Linux was required, prompting the need for an alternative solution. In the past, I separated components using LXC, similar to FreeBSD jails, but then Docker arrived, and the approach changed.

At that point, the problem seemed solved: I just needed to provide a VPS with Docker, handle backups, data, monitoring, etc., but leave developers the freedom to include the specific versions of components they needed in their setups.

But...

Developers often make poor system administrators. And rightly so, because system administrators are often poor developers. However, this leads to a series of medium-term problems:

Continued use of outdated (or conversely, bleeding-edge and thus unstable) component versions in Dockerfiles, creating stability issues or, worse, security vulnerabilities.
A habitual approach to software crashes as if they were normal. Well-developed software should not crash but autonomously manage issues. When a crash is inevitable, it should indicate a situation so severe that it requires a system administrator's intervention. Instead, the world is full of unstable stacks that crash at the slightest exception, with the mentality, "the container will just restart." This, to me, is unacceptable.
Lack of optimization: I often hear, "I've maxed out the resources on MySQL, we need to scale up." But upon reviewing, I realize that there has been no tuning of its configuration. After some adjustments, the load often decreases by 90%, making it entirely manageable. Yet, we are in an era dominated by major cloud players whose goal is not to optimize our costs (as they claim) but to make us spend more, seemingly simplifying tasks with tools like Kubernetes (and autoscaling) but actually encouraging us to unnecessarily complicate our infrastructure and spend more. Using more resources while contradicting the ecological awareness that marks our times.
Lack of big-picture thinking: As system administrators managing the overall system, we always have a holistic view. Developers, rightly focused on their projects, often lack the depth of understanding to identify the bottleneck in the entire setup. A typical comment I hear is, "the site is slow, we need a more powerful server." In 90% of cases, this is unnecessary and would be completely ineffective. A misimplemented feature launching 50 concurrent long PHP processes wouldn't be solved by increasing from 4 to 8 cores. It would help, sure, but it wouldn't be a solution. Solving it by reducing the processes to two would change everything.
Lack of backup strategy: The average developer focuses on the reproducibility of their setup, maybe keeping a database dump (not always), but seldom addresses the issue of recovery time. I recently had a discussion with a colleague (who calls himself a DevOps) who told me he had "production database dumps, a .tar.gz of individual web app directories, and notes on how he set up that server." When I asked how many systems he managed, he said "over 100, on the same cluster." Asked about disaster recovery, he believed it was "impossible" for such a cluster to be unreachable for long (though the OVH Strasbourg fire should have taught us that nothing is impossible when data is concentrated in one place). Nevertheless, he thought he could restore operations in "about 2 hours per server"—thus, 100 servers would require 200 hours of work. For 99% of setups, these would be totally unacceptable timelines.

Today, we have hardware so powerful that it can handle unimaginable loads from just a few years ago. A bit of planning, optimization, and design can greatly reduce costs, and increase productivity, stability, and system reliability.

Thus, I remain in favor of solutions like Docker, but the turn the entire IT industry is taking towards such solutions worries me because it might improve some aspects but will worsen others. We are simply shifting the problem elsewhere.

In my view, there is no one-size-fits-all solution to any problem; each requires its own study and implementation.

The solution to all the problems we have known was one: 42. And we all know how that turned out.

Migrating from an Old Linux Server to a New FreeBSD Machine

Stefano Marinelli — Wed, 25 Oct 2023 16:39:37 +0000

Preamble: I believe it's time to bid farewell to this venerable Linux server.

TL;DR: The article chronicles the journey of transitioning from an outdated Linux server, running for 1690 days without updates, to a modern FreeBSD machine. This migration involved using tools like mfsBSD, BastilleBSD, Borg Backup, and bhyve. Despite initial hesitations due to the Linux server's impeccable performance, the transition was smooth, resulting in improved manageability and efficiency. The piece emphasizes the importance of regular system updates and anticipates revisiting the topic in the future with new uptime achievements and updates.

This server loyally served for years as a secondary backup server while also providing a few minor services to users. As it often happens, it remained in operation, neglected and without updates for years. Stable operating systems have the "flaw" of being forgotten, giving the false impression that they don't need maintenance or updates. This machine continued its service without oversight for years. When approached for a service request (not due to malfunctions), I advised the client to upgrade the whole system. A mere update would not suffice, so I suggested starting afresh on new hardware with FreeBSD as the primary OS.

The client was understandably hesitant given the uptime stats:

08:58:43 up 1690 days, 21:32, 4 users, load average: 9.57, 10.15, 8.76

Not a single error, not a single hiccup. From his perspective, a similar setup to what was installed many years ago and still working flawlessly was preferred. Nevertheless, he trusted my expertise and let me proceed.

This server had a plethora of duties, among which:

One of the pivotal tasks was running Proxmox Backup Server via Docker. Proxmox Backup Server requires Debian, but this server was running on Ubuntu 16.04 (previously upgraded from Ubuntu 14.04 – yes, ancient!). Hence, Proxmox Backup Server was still at version 1.x.
Another critical function was storing backups made through BorgBackup on its file system. The /home directory used a mirrored btrfs file system, and each backed-up server had its user on this system. Clients could backup (using a push method) only via VPN and only during specific windows when the server permitted (by adding specific firewall rules via Jenkins. Jenkins also managed connection protocols, snapshots, backups, etc.).
Among the lesser tasks, the server ran a few Docker containers with HandBrake on various presets. The client processed video conversions by uploading the original files via sftp, and after some hours, fetched the converted files from the destination directory. This will not be replicated on the new FreeBSD server since they now handle this operation locally on their high-performance MacBook Pro with Apple Silicon. However, a future restoration isn't off the table.

The first thing I did was install FreeBSD on the new hardware. Given that it's a physical server on Hetzner (an auction pick due to disk space needs over power), and FreeBSD wasn't an option, I used mfsBSD.

After booting the physical server in Linux rescue, I copied mfsBSD onto the disks using dd and restarted. On boot, I SSHed into mfsBSD and executed the installation using bsdinstall—a robust and efficient method.

I set up a raidz1 with all four 6 TB disks, resulting in a final storage space of 21.8T, ample for now without the video files.

To ensure continuity, I kept a setup similar to the old one. The clients would essentially continue with their usual backup procedure without necessitating drastic changes to backup scripts. To avoid storing these backups directly in the physical machine's /home and to leave the door open for future services, I installed BastilleBSD and began setting up several jails. I replaced the old Linux machine's behavior with a VNET FreeBSD jail. In past scenarios, I've created Linux jails (thanks to BastilleBSD) and transferred the old server into the jail using rsync, making minor configuration tweaks. While this usually works, it doesn't address the underlying issue of an obsolete setup. Given the opportunity, I opted for a modern toolset.

Thus, I copied every home directory (along with their historic backups) in its entirety, installed BorgBackup, and re-established the VPN. With a VNET jail, I can craft networking devices and fine-tune configurations. After recreating user accounts, inputting the various SSH authorized_keys, and checking all clients, I set up a snapshot plan on the host. This ensures that if a client is compromised with the potential (however remote) for breach and backup deletion, a ZFS snapshot of the entire jail remains available.

As mentioned, one of the core tools on the old server was Proxmox Backup Server. It's not natively installable on FreeBSD, necessitating a VM. Enter the fantastic bhyve, supported by vm-bhyve.

However, one issue arose: backups would consume vast amounts of space, and I wanted to avoid housing an enormous disk image (or a zvol) with both VMs and backups. So, I opted for a slightly less performant but more flexible solution: installing Debian 12 and Proxmox Backup Server on the VM while placing backups on a separate ZFS dataset on the physical machine, exported via NFS and mounted on the VM.

Given that the physical server has an internal bridge "vm-public" with IP 192.168.124.1 and the VM is at 192.168.124.2, I just created a dataset named zroot/PBS and added the following line to /etc/exports:

/zroot/PBS -alldirs -maproot=root -network 192.168.124.2/32

To enable NFS, insert into /etc/rc.conf:

rpcbind_enable="YES" 
nfs_server_enable="YES" 
mountd_flags="-r" 
rpc_lockd_enable="YES"

Within the VM, create /PBS and include in /etc/fstab:

192.168.124.1:/zroot/PBS /PBS nfs rw,async,soft,intr,noexec 0 0

After Proxmox Backup Server's installation, simply set up the datastores in /PBS/, and they'll directly store on the physical machine's ZFS dataset.

For firewall configurations, I exposed port 8007, redirecting it towards the VM, and everything started functioning smoothly. I then set a Proxmox Backup Server replica from the old to the new server. After completion, I changed the Proxmox Backup Server IP on all Proxmox hosts to point to the new server. Smooth sailing.

The old Ubuntu server also managed other minor services, which have become obsolete and weren't replicated.

The transition was seamless, the client is pleased, and I'm content since each service is now neatly segregated into its jail or VM. The machine's load is minimal, which might pave the way for other tasks, via VPN. Everything now rests on ZFS, and the icing on the cake: I made the client promise not to reach another 1690 days of uptime but to timely update as required.

I'm not entirely convinced the promise will hold—meaning, in a few years, I might yet be discussing this "new" server, highlighting another impressive uptime and another upgrade journey.

How we are migrating (many of) our servers from Linux to FreeBSD - Part 3 - Proxmox to FreeBSD

Stefano Marinelli — Tue, 14 Mar 2023 13:00:00 +0000

In recent years, we've been migrating many of our servers from Linux to FreeBSD as part of our consolidation and optimization efforts. Specifically, we've been moving services that were previously deployed using Docker onto FreeBSD, and it has proven to be a great choice for handling workloads efficiently.

To this end, we've also been migrating many of our virtual machines (VMs) to FreeBSD, deploying services within FreeBSD jails. In some cases, these jails have even replaced entire VMs and run bare metal. Although we prefer to move to native FreeBSD whenever possible, sometimes it's not the best option for all the services we offer. As a result, one of our most critical physical servers has been left behind for years.

This server was a Proxmox server that we installed many years ago and updated to version 6.4. It hosted some critical services, but upgrading to Proxmox 7.x posed some challenges. In particular, some of the LXC containers required tweaks.

Unfortunately, this server was quite old, with only four physical disks and 64 GB of RAM. It was located in an OVH data center and had been running well until one of the disks started to malfunction once a week, on Sundays. This would trigger a RAID reconstruction that kept the system busy for about two days.

Despite my preference for simple setups, this server had been deployed gradually over many years, and everything was tied together. As a result, unraveling the system to resolve the issues was not a simple task. Sometimes the combination of simple things can make everything complex.

The Proxmox Server

The Proxmox server was configured as the central hub for various services, including primary DNS, web hosting, VOIP, and more. It featured several bridges, each with its own specific purpose, and was connected to a virtual machine running MikroTik CHR. This machine was responsible for consolidating all incoming VPNs from the MikroTik devices we managed, both ours and those belonging to our clients. Additionally, it provided a series of bridges to manage these devices and all server management VPNs and other services. The Proxmox server also housed several virtual machines running Linux, FreeBSD, OpenBSD, and NetBSD, as well as LXC containers.

Over the last two years, we've been migrating most of these virtual machines and containers to FreeBSD-based VMs, which feature their own specific jails. Consequently, most of the VMs we've had to move were BSD-based, while only five Linux VMs remained. The LXC containers hosted a range of services, including servers managed by Virtualmin, a large installation of Zimbra (which was hosted within an LXC container running CentOS 7), as well as some minor Alpine Linux-based machines. We located all these virtual machines and containers in a LAN created and managed by CHR. All public IPs were managed by CHR, which relied on NAT mappings to establish communication between them. CHR had thus become the heart of our system, and if it experienced any issues, it could potentially take down the entire system. Fortunately, it remained stable for years.

Migration - first steps

The first step I took was to install FreeBSD on the new server. Easy peasy. The next step was to find a way for the CHR to migrate to the new server (under bhyve) and continue to manage all the public IPs of the original server. The problem is that OVH, with its failover IPs, ties a specific MAC address to each individual IP address. Therefore, the only way was to create a bridge on the FreeBSD server (on the Proxmox server, I already had the bridge on the physical network card) and create an L2 tunnel between the two servers - I used OpenVPN with tap interfaces, specifically inserted into the bridges. I could have used other methods and techniques, but I wanted to experiment with a setup that could allow, if necessary, to bridge a larger number of physical and virtual servers even if the IPs are all mapped to a single server. OVH does not allow, in fact, the splitting of classes, so a move must be made for the entire class, not for a single IP address.

Initially, MikroTik CHR 7 did not boot on bhyve. In the end, I managed to make it work, but I had other problems, probably related to the MTU of the interfaces. So I thought about taking the opportunity to unbind the LXC containers and VMs from CHR and remove MikroTik from the setup. With RouterOS version 7, in fact, Wireguard-based VPNs are also supported, so within a few days, it was possible to update the few routers still on 6.x and recreate some VPNs using Wireguard. I mapped both the VMs and LXC containers directly to their respective public IPs, greatly simplifying the steps. Everything worked perfectly.

The next step was to test the first migrations, starting from the VMs already on FreeBSD. For simplicity, I created a new FreeBSD VM in bhyve and copied (via zfs-send and zfs-receive) the datasets related to BastilleBSD. All services are installed in jails managed by Bastille, so this was enough to have, in a short time, a new operating server equivalent to the previous one. At that point, I shut down the original server, connected the VM to the bridge linked to the tunnel (after modifying its MAC address), turned on the new FreeBSD VM (on bhyve), and everything started to work correctly - but from the new physical server.

One by one, I moved all the FreeBSD VMs. For Linux, NetBSD, and OpenBSD, I simply copied the images and pointed bhyve to them. Some small specific configuration on vm-bhyve and everything started to work correctly. Where possibile, I replaced the “virtio” with “nvme” as it performs much better on bhyve.

Migration - LXC containers to Virtual Machines

For LXC containers, I initially thought of creating an Alpine Linux virtual machine, installing LXD, and copying each individual container. It worked for some of them, but for others, I started to encounter strange issues, similar to those that would have required manual intervention to upgrade from Proxmox 6.x to 7.x. As is often the case with Linux-based solutions, compatibility is not always preserved between updates, so I would have had to fine-tune all the containers, which I didn't feel like doing. The containers had been created (at the time) to optimize RAM usage on the Proxmox machine, but to date, they have caused more problems than benefits. In some cases, certain processes got "stuck," making it impossible to "reboot" the LXC container, requiring the entire physical node to be rebooted. If they had been virtual machines, I could have given a "kill" command from the virtualizer (to the respective KVM process, in that case) and restarted it.

For greater compatibility and ease of future management, I decided to convert the LXC containers into actual VMs on bhyve. The process was simple:

Creating an empty VM with vm-bhyve and booting the VM with SystemRescueCD.
Creating destination partitions and file systems in the VM, then doing a complete rsync of the original LXC container.
Adjusting the fstab file, installing the kernel on the destination VM, and creating the initrd (some containers were already copies of VMs, so the kernel remained installed and updated, even though it wasn't being used. The initrd, on the other hand, did not include the nvme or virtio drivers, so I had to regenerate it anyway.)
Adjusting the bhyve vm configuration file, doing one last rsync after shutting down the services, shutting down the original LXC container, and starting the bhyve VM.

Everything worked correctly, so one by one, I moved all the containers. The largest one ended up on another physical node (also FreeBSD with bhyve) temporarily because the space on the new server was not sufficient to contain it. It didn't need to be on this server, so no problem.

One by one, the LXC containers started on the new server. Apart from some minor adjustments to the destination VMs (different network interface names, etc.), I didn't encounter any particular problems even after several days. Everything works perfectly.

At the very end, I re-created the MikroTik CHR VM. I’ll keep this setup separate for now, as strictly tied to eoip interfaces. This was the main reason why I haven’t performed the migration before. Things were too tied together and I had to untie everything, step by step.

…and then one of the Linux VMs started to freeze

Several Linux VMs are just the basis on which Docker runs. One of them (not even among the busiest) started, every 12/15 hours, to completely freeze. It stopped responding to ping, and it was impossible to give any type of command from the console. In a word: stuck.

Searching the web, I found some references to this problem and, observing the errors of an ssh session that was left connected (stuck, but still showing the last error), I found it to be a problem similar to the one described in this post, namely:

"watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [khugepaged:67]"

I tried various solutions such as changing the storage driver, the number of cores, the distribution (from Alpine to Debian), etc., but none of these operations solved the issue. I also noticed that the problem occurs with all Linux VMs, but only those with a recent kernel (> 5.10.x) freeze, while the others continue to work. The problem does not occur, however, with the *BSDs.

In the end, I:

Reduced the number of cores to 1 for the VMs that did not have a high load (some remained with multiple cores), hypothesising a problem with allocating cores that were too busy
Gave the command: "/usr/bin/echo 60 > /proc/sys/kernel/watchdog_thresh" to the VM.

The VM became stable, and I have not seen that error/warning on any other machine since. I will investigate further, but I believe it is a problem related to the Linux kernel, which, for some reason, generates a kernel panic if particular situations of CPU concurrency are generated.

The End…and a nice OOM!

After moving everything, I was finally able to migrate the entire class of OVH IPs from one physical server to another. The operation was quite quick, but in order to avoid problems, I notified all users and performed the operation on a Sunday and during off-peak hours. The whole process took about 10 minutes and there were no hitches of any kind.

For safety reasons, I kept the Proxmox machine active for a few more days, but there was no need to use it. However, after a couple of days, I encountered a problem: the largest VM, in some cases, was being "killed" because FreeBSD generated an OOM. I had never seen, from FreeBSD 13.0 onwards, any OOM related to "abuse" of RAM usage by ZFS, but in this case, it actually happened.

In the end, I understood that ZFS, on FreeBSD, is able to release memory, but not quickly enough to manage any "spikes" in individual VMs. In fact, the VMs do not know the situation of the physical host's RAM, so they will tend to occupy all the space allotted to them (even if only for caching). A sudden spike (i.e. if you create and launch a new VM) could cause a sudden increase in RAM usage by the bhyve process, and FreeBSD could be forced to kill it, even if part of the RAM is only ARC cache. While Proxmox supports HA (i.e., control over whether the VM is running), vm-bhyve only launches the VM (bhyve process). I should manage it with tools like monit, but for now, I preferred to simply set limits on ZFS RAM usage using "vfs.zfs.arc_max", and there have been no more problems.

Final considerations

The operation was long but linear. The most complex part was unraveling all the configurations related to MikroTik CHR and the VPNs linked to each individual LXC machine/container. Once everything was implemented on a dedicated VM, the operation was fairly straightforward.

The hardware specifications of the destination physical server are slightly better than the starting one, but the final performance of the setup has greatly improved. The VMs are very responsive (even those that were previously LXC containers running directly on bare metal) and, thanks to ZFS, I can make local snapshots every 5 minutes. In addition, every 10 minutes, I can copy (using the excellent zfs-autobackup) all the VMs and jails to other nodes both as a backup and as an immediate restart in case of disaster. I just need to map the IPs, and everything will start working very quickly. Proxmox also allows you to perform this type of operation with ZFS, but you still need to have Proxmox (in a compatible version) on the target machine. With the current setup, I only need any FreeBSD node that supports bhyve.

Proxmox is an excellent tool, well-developed, open-source, efficient, and stable. We manage many installations, including complex ones (ceph clusters, etc.), and it has never let us down. However, not all tools are ideal for all situations, and for setups like the one described, the new configuration based on FreeBSD has shown significantly interesting performance and greater management and maintenance granularity.

Virtualizing on vm-bhyve is not complex, but it is certainly not comparable, at the current state, to the simplicity of using a clean and complete interface like Proxmox's. A complete HA system is still missing (sure, it's achievable manually, but...), as well as complete management web interface. However, for knowledgeable users, it is undoubtedly a powerful tool that allows you to have excellent FreeBSD as a base. I'm totally satisfied with my migration and the result is far better than I expected.

Deploying a piece of the Fediverse

Stefano Marinelli — Sun, 15 Jan 2023 09:29:45 +0000

After Elon Musk’s Twitter deal, many users decided to “fly away” from the "traditional" commercial Social Networks. Some for good, some just decided to increase their presence in other, alternative Social Network. That's what I'm doing.

Many of those users decided to join the Fediverse - even if many of them just call it Mastodon, as they don’t understand that Mastodon is just a Software that allows to join the Fediverse.

The Fediverse is composed by thousands of “instances”, some are bigger (like mastodon.social), some are personal (aka: single user instances), many are normal communities with their members. Many of them communicate using the same open protocol, ActivityPub.

Because of this, there’s no “one size fits all” so I’ve started to explore the different solutions. I’ve been mainly focusing on running them on FreeBSD, but I’ve had to fire up Linux for some tests. Here’s what I’ve found out.

Backends / Complete Solutions

Mastodon

I won’t spend too much time on Mastodon as you may find almost everything, everywhere, about it. Tons of articles have been written about Mastodon, so this one would be just another one, surely not the best one. Many consider it to be “the Fediverse” (they just say “Mastodon” to refer to the whole “Fediverse”, and they’re wrong), it’s by far the most installed solution. It’s so popular that there are plenty of clients (both for Android and iOS) that perfectly work with it. Mastodon has its own APIs - and many other Fediverse solutions are using them, just to be able to be compatibile with the Mastodon apps. It also supports backend based “Hide replies” (only show new posts, not all the replies to other posts) in timeline. It’s good as you can filter the replies from any frontend or mobile app as the backend won’t provide them at all, while the client doesn't need to be aware that you're filtering them.

Why I am suggesting it: It’s stable and well done, there’s a lot of documentation and if you install and manage it correctly, you shouldn’t notice anything strange or unexpected. Remember, no software solution can be considered “set and forget” and Mastodon is not an exception. Please, don’t forget that running an instance is not just installing the software. Moderation is a serious issue. More about it later.

Please, consider that: Mastodon is also quite heavy, not easy to scale (even if there’s documentation around) and, by default, is caching everything it sees and knows about. This means that both a single user instance or a thousands of users’ one, will (rapidly) grow because any media will be locally cached.

My first, single user installation grew, in a week, well over 100 GB of occupied storage because of all this caching. It can’t be avoided (you can just tell Mastodon to delete the cache after x days). While it may make sense for a big instance, it can be considered an overkill for a single user one. This is a "known problem", but mainly considered as a feature: all the media will be locally processed, all the contents will be locally stored. So an evil content hidden in a media file will be reprocessed by local ffmpeg or ImageMagick and will be cleared.

Why I’m suggesting another solution: Monopoly is bad and Mastodon is becoming, for many, a synonym of Fediverse . More, it requires much space and it’s resource-hungry. It’s easy to install Mastodon and experience a huge resource drain in just a few days, especially if you’re not a skilled system administrator. While Mastodon is the best solution for a complete microblogging experience, other solutions exist.

To sum up: Installing Mastodon on FreeBSD was easy. Even if I had read about problems, I’ve documented how to do it and it’s stable and reliable. Mastodon is, IMHO, a good piece of software but keep in mind that it could not be the best solution for you and a small, single user instance can become huge in a few weeks. Also, keep in mind that it's the most deployed Fediverse software, so any mobile app, any web app, any hint will work perfectly with Mastodon. Your Fediverse experience will be smooth.

Akkoma (Pleroma fork)

I’ve read about Akkoma in a reddit thread about how difficult was to install Mastodon on FreeBSD. It was described as a Pleroma fork, but actively developed and maintained, faster and with more advanced features. That’s why I decided to try it and - at least for now - stick with it (and not Pleroma, but many of the things I’ll point out here apply to Pleroma, too).

Akkoma (as all the Pleroma forks) is much, much lighter than Mastodon. It’s perfectly able to run a single user instance on a Raspberry PI. Moreover, Christine Lemmer-Webber, coauthor of the ActivityPub protocol, said that Akkoma is a good solution - and I think that her opinion is a qualified one.

Why I am suggesting it: First of all, for its documentation. As for Pleroma, there are installation instructions for a lot of operating systems (yes, also FreeBSD, NetBSD, OpenBSD). It’s light and fast. It doesn’t cache remote media by default, so that’s perfect for a single user instance. You can enable it (both pre-fetching media as soon as the server gets the status, like Mastodon, or just downloading them and caching when the first user meets them), you can also proxy your local media. S3 storage and remote CDNs are supported and everything is customisable. Message size limit is set to 5000 characters by default, but can be adjusted. Everything is configurable and you can choose your favourite frontend. It has quote posts (while Mastodon doesn’t allow them, even if they’re perfectly visible if created by Akkoma).

Please, consider that: Akkoma is not Mastodon. They “talk” using the same language but are different pieces of software. There’s less activity around it (the documentation is good, the support forum is good, but the number of Akkoma installations can’t be compared to Mastodon’s ones). Many Mastodon mobile apps seem to have problems with Akkoma and at the moment there’s a bug (probably it's a Mastodon bug, but users will think it's Akkoma's fault) that, if you’re posting an hashtag, a link will be shown on Mastodon. More, if you want to hide boosts or replies from your timeline, remember that Akkoma won’t perform that at backend’s level, but it should be done by the frontend. Not all frontends and mobile apps support it.

Why I’m suggesting another solution: Well, not exactly. Actually, I’m suggesting to try Akkoma. It’s a good piece of software, developed by friendly people, well accepted by the Fediverse instances’ administrators and has a lot of happy users and instance administrators. I've used Akkoma as my main instance software for more or less one month. The bug I've described and some problems here and there made me move back to Mastodon. I'm keeping my Akkoma instance up, even if not actively used.

To sum up: Akkoma installation is easy and well documented, you have a lot of settings to customise your instance and can fine-tune your installation for your hardware capabilities.

Misskey (and its forks like Firefish, etc.)

Misskey is a very nice piece of software. I had some troubles to run it on FreeBSD (but I didn’t try that much) so I decided to fire up a Linux machine and use Docker.

The interface is nice - it’s Japanese design, so it’s fancy and rich of emojis and effects. I’ve tried it just for a few hours and I appreciated it but decided it wasn’t ok for my needs (at least, for now).

Why I am suggesting it: if you’re building a community, Misskey (or one of its forks) is a very good choice. It’s eye candy, complete and usable. It’s a part of the Fediverse, so no problems to talk to Mastodon, Akkoma, etc. It also has a “drive” feature, useful for many users that want to exchange files.

Why I’m suggesting another solution: the main reason why I had to look at another solution is that it doesn’t support Mastodon APIs so no Mastodon Android or iOS app is working with a Misskey instance. While it may be ok for many users, this could be a problem for others.

To sum up: Misskey is nice, worth trying and a very good solution if it fits your community’s needs. For me, it doesn’t give any advantage over other lighter solutions.

Friendica

Friendica is a project that aims to be similar to Facebook. It’s federated and actively maintained, the interface is nice and familiar to Facebook users. I’ve been able to install it on a FreeBSD jail (as it’s in PHP) and everything worked as expected. I didn’t spend too much time on Friendica but I’m planning to do a deeper test as I’m working on a community of former Facebook users and this could be the right choice.

Why I am suggesting it: If you’re creating a community for (former) Facebook users, they’ll have a familiar feeling in Friendica. The interface is clean and usable, it supports a lot of protocols (ActivityPub, OStatus, diaspora), it supports plugins, can import websites via rss (so automatic post is easy).

Please, consider that: Moderation tools are different from the ones you have on Mastodon or Akkoma (Pleroma, etc.) and you can’t easily report users, especially remote ones as support for Mastodon API is limited. With the current growth of users, it’s easy to find a bad person trying to disturb. Not having effective ways to deal with it may be frustrating.

Why I’m suggesting another solution: I didn’t try Friendica long enough to find some big problems with it.

To sum up: Generally speaking, it is considered a solid and stable solution, actively maintained and, being in php, portable. If you want to create a Facebook-like community, that's the way to go.

snac2

snac2 is a simple, minimalistic ActivityPub instance that supports the Mastodon API. This makes it compatible with platforms like Pleroma, Akkoma, and Mastodon itself. It's written in portable C and it's been created to be light, easy to deploy, and with only two dependencies: openssl and curl. It heavily relies on hard links and doesn't need any database. This is a big, big plus for me. I've performed many tests and found that this is one of the best lightweight solutions to join the Fediverse.

Why I am suggesting it: snac2 is clean and polished. The federation with the other solutions is good, it works beautifully with Tusky (on Android) and tooot (both on Android and iOS) - also consider Enafore as a PWA or web interface - and its integrated web interface is minimal but effective. No javascript, no cookies - clean web. More, the dev is responsive and open to patches and contributions. I've helped with some stress tests and contributed with instructions and patches to make it work on FreeBSD and NetBSD, and they've been merged immediately.

Please, consider that: It's not Mastodon. Some of the Mastodon API features are (currently) not supported so the experience could be different from the other solutions. Some Mastodon apps don't work (the official Mastodon app, for example, can't login). There's no open registration option (users should be manually registered from the cli) and account migration is not supported, at the moment. More, while it's not caching external media, locally published media will stay on the local drives (no S3 upload option), so be prepared to serve those files as well. Testing it from my FTTC connection, I almost DDoSed my internet connection for 15 minutes, publishing a photo.

Why I’m suggesting another solution: Actually, I'm suggesting to try snac2. I think it could be a great solution for a single user instance or for instances managed by tech people, as you can run it just 1 minute after the download. It's light, easy, straightfoward and the dev is a nice person. I'd suggest other solutions if the priority is to offer a full, feature rich Fediverse experience, with registration, big and featureful configuration panels, etc

To sum up: snac2 is a lightweight and effective way to join the Fediverse. Currently, it's my favourite solution for small communities as the "file only" approach and no dependencies are coherent with my ideas. I've migrated my instance from FreeBSD to NetBSD, from external datacenters to my home network and it's just been a matter of a single rsync (-H)

GoToSocial

GoToSocial is a new microblogging platform. Its target it to be a light, customisable, easy to manage and integrated software. It is still at its early alpha stage, it should evolve into a beta at some point in ~~2023~~ 2024. It’s developed in Go and installation in easy and fast, it supports Postgres, Mysql and Sqlite. Basic functionalities have already been integrated and it can federate with (almost) all the other ActivityPub implementations, even if with some small problems. FreeBSD installation was easy and fast, as they provide a amd64 FreeBSD binary. Being written in go, it shouldn't be difficult to self-compile it for other architectures.

Why I am suggesting it: While it doesn’t have an integrated frontend (but Mastodon mobile apps or other frontends can be used, as Enafore Mangane, etc.), it already supports many of the features you’d expect from an ActivityPub implementation. It's fast, suitable for installation on a low end hardware, can be deployed without a reverse proxy as has integrated support for Letsencrypt certificates. But…

Please, consider that: It’s still at alpha stage. Things can still break and it doesn’t support any kind of account migration from Mastodon (while it’s supported by Pleroma and its forks).

Why I’m suggesting another solution: While I think it could be a game changer, I think it’s bit early to deploy it unless you’re a very skilled and experienced administrator. You should understand its “quirks and features”, mainly tied to its alpha status. While I’ve installed and will keep installed a GoToSocial instance, at the moment I’ll just keep it in a test stage server in order to follow its development status.

To sum up: GoToSocial could potentially become one of the most interesting ActivityPub microblogging platforms, at least for small and medium sized communities. It's a bit early to consider it ready for a production deployment, but it's already worth testing.

Frontends

All the Fediverse backend implementations have their own specific features but the users will just interact via a frontend. While Mastodon - but also Pixelfed, Peertube, etc. - are presented as a specific stack (backend + frontend), there are many frontends that can interact via the Mastodon API.

Of course, being the “Mastodon” API, not all the backends are perfectly compatibile/supported by all the frontends. Mobile apps use the Mastodon API, that’s why their compatibility with other implementations like snac2, Pleroma, Akkoma, etc. may not be perfect at all the times.

I won’t describe them all. I’ll just enumerate the ones I’m using, with some notes:

Akkoma’s Pleroma-FE

With the 2022.12 release of Akkoma, Pleroma-FE has been evolved into a proper, nice looking PWA. After an initial configuration of its many options, I found it quite nice and effective. The only problem (common with many PWAs on iOS) is that when you suspend the app, it doesn’t detect it, so if you open it again after two hours, you’ll just see the recent posts, not all the posts of the last two hours. This is because of the way iOS deals with app suspend/resume and many PWAs don’t seem to understand they’ve been suspended.

Mangane

Mangane is a Soapbox fork that aims to improve compatibility with Akkoma. It is nice and clean and the developers are improving it to support Akkoma's features, so it's great. At the moment I've noticed some visual problems, on iOS, with the icons - but Guérin contacted me to ask for information (after reading this article) and opened an issue on to fix it. Guérin has been nice and helpful, making Mangane even more appealing. I’m using it for planned posts on Akkoma, as Pleroma-FE doesn’t support them, yet. I'm also using Mangane as a daily driver, from time to time, as I like it.

Enafore

A light, fast, complete and usable web frontend. I’ve been using it when I needed a simple, clear frontend. It supports “hide replies”, which is great to improve timeline quality. It's among my favourite choices when using a webapp and is a good choice for snac2.

Elk

Elk is a frontend for Mastodon API. It’s nice, clear, eye candy, intelligent. The developers are nice people, open to suggestions. Development is a bit slower compared to the initial pace, but the app is already very complete and stable.

Elk is definitely a very good piece of software, and I recommend to try it.

Moderation and final considerations

One of the things you should be considering is that deploying a piece of the Fediverse isn’t just installing a software and interacting with others. Actually, that’s just the easiest part of the experience, at least if you’re not creating a single user instance.

Many of the people that joined the Fediverse in its early days decided to do it as they felt attacked on other social networks. In the last years, the commercial socials have proven to be the perfect place for negative people, attacking others without being blocked/stopped in an efficient way. Hate causes addiction and the owners of those commercial socials make a lot of money if people interact, showing them ads every time they open the app/website. They make money (also) through people hating each other.

The Fediverse gives the possibility to mute and block users, but also to mute and block entire instances. One of the main tasks of an instance’s administrator is to make sure that everything is ok. While you can define your own instance’s rules, other instances’ admins may block you if they find you’re federating by sending messages agains their rules.

As an administrator, you’re also responsible of keeping your users’ data safe, to avoid sharing/providing illegal contents or offensive stuff. So you’re free to set your rules, but others are free to “defederate” you, if they don’t like the contents your instance is providing. While it’s not an issue for a single user instance, you must be quite careful when opening the registrations as you may find out you've been defederated because of (your) lack of moderation.

There’s some strong criticism against the Fediverse because of this, as many see it as more “censored” than the traditional, commercial social networks. I don’t think it’s true, as you’re free to fire up your instance, decide your rules and act as you want. But you can’t impose others and their instances to follow you, even if you think you’re right. In a free world, everybody should be free to decide if they want to listen to you. But everybody should also be free to create a new space and start sharing their ideas. There’s not a central authority of contents, so only the single admins and users can choose what they want to see or avoid. There's not a commercially driven algorithm that may decide that you should see contents that will cause you anger and hate just because it generates traffic (and money) to the social's owner.

The Fediverse can be a beautiful place to stay.

Rotating Docker log files

Stefano Marinelli — Thu, 29 Aug 2019 04:30:00 +0000

Sometimes containers log a lot of things, and the logfile can become huge.

This happens because containers aren't supposed to run for a long time but to be created/disposed regularly. Here's how to fix this.

You have to instruct Docker, at launch, and tell how much you want the log file to grow before rotating and how many old logs you want to keep.

For example,

--log-opt max-size=50m --log-opt max-file=5

will limit log files to 50 MBytes and keep 5 log files, so total occupied space will be 250 MB.

(this article is a copy of the Italian version published on my Italian blog)