IT Notes - alpine

From Cloud Chaos to FreeBSD Efficiency

Stefano Marinelli — Thu, 04 Jul 2024 08:41:00 +0200

Introduction

A few months ago, a client asked me to take care of their Kubernetes cluster (hosted on AWS and GCP). In their opinion, the costs were exorbitantly high for relatively simple and lean websites. Sure, they had many visits, but nothing too excessive development-wise.

I kindly declined. Unfortunately, their situation is all too common these days: they hired developers accustomed to working that way, convinced that a system administrator is now unnecessary because "the cloud has infinite potential." They were used to considering optimization as secondary because "we have infinite power" (and this is already a spoiler for the ending).

Being open to dialogue and new experiences, they asked for my opinion on the matter. We talked for a while, and I explained that, in my view, for the type of setup they had (standard, with various replicas and variants, but primarily based on two platforms), it didn't make sense. I saw it as complicating things. An over-engineering of something simple. Like taking a cruise ship to cross a river.

They then asked me to create something simple that would serve as a development server and for backups, to understand what kind of solution I had in mind.

The Solution

So, I started building everything. I began with FreeBSD 13.2-RELEASE, but in the meantime, 14.0-RELEASE came out, so that’s the version I delivered.

I installed the operating system on a physical server, leased from one of the main European providers. Benefiting from one of their auctions (good deals can be found on weekends), they found a sufficiently powerful machine, with 128GB of RAM, 2 NVMe drives of 1TB each, and two spinning disks of 2TB each for less than 100 euros per month. They also took another, less powerful one for additional backups and to back up the first one.

Implementation

I decided to keep the host as clean as possible and concentrated the services in jails (managed by BastilleBSD) and VMs. The machine was divided as follows:

A series of bridges - to be used for different projects. Jails of the same project and/or type use the same bridge and can communicate with each other, sharing some resources (MariaDB, etc.).
A bhyve VM with Alpine Linux - in my opinion, the best distribution for running Docker containers. Do we really need systemd just to launch Docker? They mainly use it as a pre-production test bench, connected via VPN to their company LAN. It is the core of their "online" development, i.e., outside their computers. It has 32GB of RAM, 200GB of disk (obviously bhyve is configured with NVMe drivers), and 4 cores assigned.
A VNET jail with a reverse proxy (nginx) - they know how to modify virtual hosts and generate certificates with certbot, pointing to the underlying jails.
A series of "empty" VNET jails, to be cloned, for each type of setup (they mainly have CMS based on WordPress and Laravel, so with all dependencies inside - nginx, php, redis, etc. except the databases).
A VNET jail with MariaDB installed, to be cloned, to be attached to different projects as needed.
zfs-autobackup performs local snapshots, keeping: one every 15 minutes for 3 hours, one per hour for 24 hours, one per day for 3 days.

Backups are also performed using zfs-autobackup and, in case of disaster recovery in rapid times, a zfs-send (and corresponding zfs-receive) every 10 minutes on another machine (the other, smaller one, also taken at auction), with the same bridges, firewall rules, BastilleBSD, and bhyve installed - ready to start in case of disaster. Being a test server, we didn't consider to implement a proper HA - at the moment, it wouldn't make sense.

They also have another job with zfs-autobackup that performs an additional backup on a server (Debian in their offices). Safe data, in my opinion, are those in storage under your b...ench.

I delivered everything to them and gave a brief course to the more experienced devs on how to manage things. No explanation on the Alpine Linux VM, but I showed them the jails, how to clone, configure, and manage them.

Real-world Testing

I didn't hear from them anymore. After a few weeks, one of the devs contacted me urgently because a junior unfortunately made a mistake and deleted an entire project from one of the jails. I explained that the local snapshots were restorable with a command, and he was thrilled. He restored both the development jail and the one with the database made two minutes before the "mishap" and they restarted immediately.

I realized that this event would change some of their procedures and criteria.

I hadn't heard from anyone for months. This morning, I received a call from their manager, whom I hadn't heard from since the beginning, and he told me how things had been going these months.

Lessons Learned

First, this person has good communication and commercial skills but little technical background. He is open-minded and tends to study carefully what is proposed to him. He doesn't discard any solution a priori, without having touched its pros and cons.

They had leased servers with cPanel and were inserting their content inside them. The devs who arrived a few years ago suggested making a technological transition, eliminating these "obsolete" servers and "outdated" methodologies, pushing everything to the cloud and containerizing everything. When we first talked, he told me how they were "lucky to make that transition because their load had increased enormously and the old servers probably wouldn't have handled the load", instead autoscaling saved them. I had some reservations about autoscaling without particular controls, but clearly, I cannot impose my choices on others.

To cut a long story short: seeing what happened with that junior dev's mistake (and the simplicity with which it was possible to restart immediately), they decided to increase the use of FreeBSD jails and reduce, at least on secondary loads, the use of their Cloud managed with Kubernetes. As they transitioned to jails, however, they noticed some slowdowns. These slowdowns worsened day by day. According to the devs, it would have been appropriate to go back to having, again, autoscaling ("we need moar powaaaaar!!!") but, fortunately, their boss decided to investigate carefully. They realized that these workloads (based on Laravel) were storing sessions on files. Over time, these millions of files (several gigabytes per day) slowed everything down because, for specific operations, Laravel scanned the entire directory. In other words, on the "cloud," they needed much more power than necessary (and much more disk space, but that was cheaper) to carry this load, which was, in fact, unnecessary. After realizing this, they moved the sessions to Redis. Needless to say, everything became extremely faster, even compared to the previous setup on Kubernetes and autoscaling.

At that point, it was clear that one of the problems with their setup is (as often happens) poor optimization. Today, there's a tendency to rush, "throw in" functions, features, libraries, plugins, etc. without considering the interactions and consequences. If it works, it's fine. Even if it increases computational complexity exponentially just to, for example, change the color of an icon (absurd example, but to give an idea).

They then started moving even the main Laravel workloads (thanks to the optimization implemented). At this point, they began moving some of the WordPress sites even though they were extremely concerned. In the cluster, every day, at fairly irregular intervals, the load would rise and everything would slow down until autoscaling started scaling up to the imposed limits. CPU at 100% on all containers, and the devs noticed that the load came from a series of "php" processes. Recreating the containers helped for some minutes, but did not solve the problem.

To their great surprise, all this did not happen on the FreeBSD jails. The load was significantly lower, without any of these spikes. Satisfied, they decided to use this as their final setup. One of the devs, however, wanted to get to the bottom of it and decided to run a test: he moved some of these WordPress sites to the Alpine VM, on Docker. At that point, the spikes resumed, saturating the CPU of the Alpine machine.

Without going into details, they eventually realized that there was a vulnerability in one (or more) of the many plugins installed on the WordPress sites, which was being exploited to inject a process, probably a cryptominer. The name given to the process was "php" - so the devs, not being system experts, did not worry about understanding better whether it was really php or another process pretending to be it. On FreeBSD, all this did not happen because the injected executable could not run - there was no Linux compatibility activated on the server.

Until then, they considered these (expensive) spikes as organic and did not worry too much about them. Paying to have their friendly intruders mine.

Conclusion

They asked me to help, as much as possible, to move other services to FreeBSD. It won't be easy, probably we will need to use bhyve a lot, but they decided that this is the platform they want to focus on in the coming years.

Undoubtedly, this is a success story of FreeBSD and, indirectly, of correct and careful management of one's resources. Too often today, there is the superficial belief that the cloud, with its "infinite" resources, is the solution to all problems. And that Kubernetes is the best solution for everything. I, on the other hand, have always believed that there is the right tool for everything. You can hammer a nail with a screwdriver, but it's not the most suitable and efficient tool.

Today they spend about 1/10 of what they used to spend before, they have more control over their data and the tools they use. Undoubtedly, all this was also caused by poor optimization and control by those who manage the infrastructure, but the question is: how often do people decide that, in the end, it is okay to spend more (especially if it is someone else's money) rather than go crazy for hours behind such a situation? While having defined and limited resources (albeit elevated) poses different problems - but of optimization. And in the age of energy and resource savings, it might be wise to give more importance to optimization.

Abundance led to waste.

How to Set Up a Alpine Linux VM Hosting XRDP and XFCE for Secure Remote Desktop Access

Stefano Marinelli — Tue, 14 May 2024 08:05:51 +0000

Introduction

A client recently asked if their approach to remote desktop access was correct. They leave their office PC on and connect to it via remote desktop. Their main requirement is to access internal resources via a browser (they use Brave, so the BSDs cannot be currenly used) and they prefer not to use their home computers for security reasons. I can understand their concern – I wouldn’t be comfortable knowing that a home PC (possibly shared with others) could connect to the company VPN and have unrestricted access.

Setting Up Alpine Linux on a VM

To address this, I downloaded the Alpine Linux Virt ISO from the official site and installed it on a VM in their office datacenter. They use Proxmox, which made the process quite straightforward. I allocated 20GB of disk space, 4GB of RAM, and 2 CPU cores to the VM. For added security, the installation process allows you to encrypt the disk. Note that if you choose this option, you’ll need to access the virtualizer console to re-enter the password every time the VM restarts.

During the Alpine installation, create a non-privileged user who will be using the remote desktop we’re about to set up.

Initial Configuration

Once the installation is complete, you can log in via the console as root or use SSH with the newly created non-privileged user. In the latter case, you’ll first need to switch to the root user:

su -

Enable the community repository by uncommenting it in /etc/apk/repositories:

http://dl-cdn.alpinelinux.org/alpine/v3.20/main
http://dl-cdn.alpinelinux.org/alpine/v3.20/community

Installing Required Packages

Next, install the main packages needed to manage the remote desktop:

apk add xrdp xorgxrdp xorg-server xfce4 xfce4-terminal wireguard-tools ifupdown-ng-wireguard

Edit the /etc/xrdp/xrdp.ini file to ensure xrdp listens only on the VPN’s private IP, avoiding exposure to the LAN (or worse, the WAN):

port=tcp://172.16.16.1:3389

Enable xrdp:

rc-update add xrdp
rc-update add xrdp-sesman

Configuring Wireguard

To set up Wireguard, navigate to /etc/wireguard and create the keys:

wg genkey | tee server.privatekey | wg pubkey > server.publickey

Create a configuration file wg0.conf:

[Interface]
Address = 172.16.16.1/24
ListenPort = 4242
PrivateKey = <server private key value> # the key from the previously generated privatekey file

[Peer]
PublicKey = <client public key value> # client’s public key
AllowedIPs = 172.16.16.2/32

On the client, the configuration should look like this:

[Interface]
PrivateKey = <client private key value>
Address = 172.16.16.2/24

[Peer]
PublicKey = <server public key value>
AllowedIPs = 172.16.16.0/24
Endpoint = <server public ip>:4242

Then, open the /etc/network/interfaces file and add:

auto wg0
iface wg0 inet static
pre-up wg-quick up /etc/wireguard/wg0.conf

Reboot the VM, and everything should be ready. Just be sure to set your router/firewall to forward the 4242 UDP port to the VPS LAN ip for Wireguard access. If the VM has been exposed via public IP, this won't be necessary, but remember that ssh will be exposed, too so take care.

Connecting via Remote Desktop

Use your favorite RDP remote desktop client and point it to 172.16.16.1. You should see a login screen.

Installing Brave Browser

To install Brave Browser on Alpine Linux, the easiest way is to use Flatpak. Open a terminal and, as root, install Flatpak and Brave Browser:

su -
apk add flatpak
flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo
flatpak install flathub com.brave.Browser

After logging out and back into the remote desktop, Brave should appear in the list of applications. Launch it, and you can synchronize it with the Brave installation on your work PC. This setup ensures that everything works seamlessly on the virtual remote desktop.

Conclusion

This approach offers multiple benefits. By exposing the remote desktop via Wireguard, you significantly enhance security without compromising access to internal services. This method ensures that your internal resources remain protected while being easily accessible when needed.

Installing Alpine Linux in a FreeBSD Jail

Stefano Marinelli — Thu, 18 Jan 2024 14:05:51 +0000

Installing Alpine Linux in a FreeBSD Jail

Alpine Linux is one of my favorite Linux distributions, particularly for specialized purposes. I recently faced the task of moving an Alpine Linux-based VPS into a FreeBSD host and considered two approaches:

1. Moving the VPS to bhyve on FreeBSD

The simplest and more conventional method involves transferring the VPS to the FreeBSD host and running it on bhyve. This solution is proven and stable. However, bhyve doesn't support memory ballooning. The assigned and unused memory - used by the Alpine VPS for caching but rendered unnecessary since FreeBSD performs its own caching - led me to explore another method.

2. Using Linuxulator

With the Linuxulator, I thought about creating a jail and copying Alpine Linux into it, attempting to run all necessary services (not many, but complex enough to initially discourage a direct migration to FreeBSD). This approach wasn't pioneering, as it's been used by others (and myself in other situations), but I had never applied it to Alpine Linux before.

The Implementation

I won't describe the entire process of what I did - it turned out to be straightforward: I prepared a Linux jail, configured it, performed an rsync of the entire filesystem of the original VPS, fixed a few things, and started it up. However, I will explain how I create and use Alpine Linux jails in FreeBSD hosts without any jail management software (like BastilleBSD, iocage, ezjail, etc.). Many aren't aware that managing the entire jail lifecycle is already integrated into FreeBSD and that these tools are helpful but not strictly necessary.

Enabling Linuxulator

First, enable and start the Linuxulator on the FreeBSD server:

service linux enable
service linux start

This loads all necessary modules for executing Linux-compiled executables.

Setting Up the Jail

Choose the path for installing the jail. There's no need to use ZFS for this type of setup; UFS is sufficient.

Create the path, for example, /var/jails, and the path for the individual jail, such as /var/jails/alpine01:

mkdir -p /var/jails/alpine01

Download the Alpine Linux base filesystem directly from the Alpine Linux website and decompress it in the newly created directory.

cd /var/jails/alpine01
fetch https://dl-cdn.alpinelinux.org/alpine/v3.20/releases/x86_64/alpine-minirootfs-3.20.2-x86_64.tar.gz
tar zxf alpine-minirootfs-3.20.2-x86_64.tar.gz
rm alpine-minirootfs-3.20.2-x86_64.tar.gz

In theory, the base system is already functional, but some services might complain about missing configurations. It's advisable to create the file /etc/network/interfaces:

echo "auto lo" > /var/jails/alpine01/etc/network/interfaces

And if you want to use openrc:

touch /var/jails/alpine01/run/openrc/softlevel

Starting the Jail

The jail is now ready to be launched. There are two approaches: include its configuration in /etc/jail.conf of the FreeBSD server or create a separate file and launch it manually. The latter allows for interesting possibilities: everything related to the jail lives in the /var/jails directory, which could be on a different device, even NFS or external. This approach is also applicable for FreeBSD jails and is convenient for portable jails or those used from encrypted/separated devices.

Following this second approach, create a file named /var/jail/alpine01.conf containing:

alpine01 {
    host.hostname = "${name}";
    #exec.consolelog = "/var/log/jail_console_${name}.log";
    ip4.addr= 192.168.1.111;
    interface = vtnet0;
    path="/var/jails/${name}";
    allow.raw_sockets=1;
    #exec.start='/sbin/openrc';
    exec.start='/bin/true';
    exec.stop='/bin/true';
    persist;
    mount.fstab="/var/jails/${name}.fstab";
}

I intentionally left the console log lines and the start of openrc commented out (not everyone wants them), and openrc is currently not present in the jail.

Adjust the assigned IP and interface based on your configuration.

Create the jail's fstab, /var/jail/alpine01.fstab:

devfs           /var/jails/alpine01/dev      devfs           rw                      0       0
tmpfs           /var/jails/alpine01/dev/shm  tmpfs           rw,size=1g,mode=1777    0       0
fdescfs         /var/jails/alpine01/dev/fd   fdescfs         rw,linrdlnk             0       0
linprocfs       /var/jails/alpine01/proc     linprocfs       rw                      0       0
linsysfs        /var/jails/alpine01/sys      linsysfs        rw                      0       0

Launching and Testing the Jail

To launch the jail:

jail -crm -f alpine01.conf

For testing:

jexec alpine01 login -f root

Welcome to your new Alpine Linux jail. Once in the jail, don't forget to add the nameserver in /etc/resolv.conf, or the jail won't be able to perform DNS resolutions:

echo "nameserver 1.1.1.1" > /etc/resolv.conf

To simulate a real Alpine Linux boot as closely as possible, I suggest installing openrc:

apk add openrc

Then you can modify the alpine01.conf file. Exit the jail and adjust as follows:

#exec.start='/bin/true';
exec.start='/sbin/openrc';

Now, you can restart the jail and see openrc start services (currently, none) inside the jail:

jail -r alpine01
jail -crm -f alpine01.conf

The jail is now ready.

The Linuxulator can't emulate all Linux syscalls, so some things might not work or only work partially. However, it's more than adequate for many uses. The same approach can also be used for other distributions. A convenient method to download the root file systems of various distributions is to look for those already created for LXC, freely downloadable as regular files.

FreeBSD jail's embrace,
Alpine's freedom finds its place,
Unix worlds in grace.

How we are migrating (many of) our servers from Linux to FreeBSD - Part 3 - Proxmox to FreeBSD

Stefano Marinelli — Tue, 14 Mar 2023 13:00:00 +0000

In recent years, we've been migrating many of our servers from Linux to FreeBSD as part of our consolidation and optimization efforts. Specifically, we've been moving services that were previously deployed using Docker onto FreeBSD, and it has proven to be a great choice for handling workloads efficiently.

To this end, we've also been migrating many of our virtual machines (VMs) to FreeBSD, deploying services within FreeBSD jails. In some cases, these jails have even replaced entire VMs and run bare metal. Although we prefer to move to native FreeBSD whenever possible, sometimes it's not the best option for all the services we offer. As a result, one of our most critical physical servers has been left behind for years.

This server was a Proxmox server that we installed many years ago and updated to version 6.4. It hosted some critical services, but upgrading to Proxmox 7.x posed some challenges. In particular, some of the LXC containers required tweaks.

Unfortunately, this server was quite old, with only four physical disks and 64 GB of RAM. It was located in an OVH data center and had been running well until one of the disks started to malfunction once a week, on Sundays. This would trigger a RAID reconstruction that kept the system busy for about two days.

Despite my preference for simple setups, this server had been deployed gradually over many years, and everything was tied together. As a result, unraveling the system to resolve the issues was not a simple task. Sometimes the combination of simple things can make everything complex.

The Proxmox Server

The Proxmox server was configured as the central hub for various services, including primary DNS, web hosting, VOIP, and more. It featured several bridges, each with its own specific purpose, and was connected to a virtual machine running MikroTik CHR. This machine was responsible for consolidating all incoming VPNs from the MikroTik devices we managed, both ours and those belonging to our clients. Additionally, it provided a series of bridges to manage these devices and all server management VPNs and other services. The Proxmox server also housed several virtual machines running Linux, FreeBSD, OpenBSD, and NetBSD, as well as LXC containers.

Over the last two years, we've been migrating most of these virtual machines and containers to FreeBSD-based VMs, which feature their own specific jails. Consequently, most of the VMs we've had to move were BSD-based, while only five Linux VMs remained. The LXC containers hosted a range of services, including servers managed by Virtualmin, a large installation of Zimbra (which was hosted within an LXC container running CentOS 7), as well as some minor Alpine Linux-based machines. We located all these virtual machines and containers in a LAN created and managed by CHR. All public IPs were managed by CHR, which relied on NAT mappings to establish communication between them. CHR had thus become the heart of our system, and if it experienced any issues, it could potentially take down the entire system. Fortunately, it remained stable for years.

Migration - first steps

The first step I took was to install FreeBSD on the new server. Easy peasy. The next step was to find a way for the CHR to migrate to the new server (under bhyve) and continue to manage all the public IPs of the original server. The problem is that OVH, with its failover IPs, ties a specific MAC address to each individual IP address. Therefore, the only way was to create a bridge on the FreeBSD server (on the Proxmox server, I already had the bridge on the physical network card) and create an L2 tunnel between the two servers - I used OpenVPN with tap interfaces, specifically inserted into the bridges. I could have used other methods and techniques, but I wanted to experiment with a setup that could allow, if necessary, to bridge a larger number of physical and virtual servers even if the IPs are all mapped to a single server. OVH does not allow, in fact, the splitting of classes, so a move must be made for the entire class, not for a single IP address.

Initially, MikroTik CHR 7 did not boot on bhyve. In the end, I managed to make it work, but I had other problems, probably related to the MTU of the interfaces. So I thought about taking the opportunity to unbind the LXC containers and VMs from CHR and remove MikroTik from the setup. With RouterOS version 7, in fact, Wireguard-based VPNs are also supported, so within a few days, it was possible to update the few routers still on 6.x and recreate some VPNs using Wireguard. I mapped both the VMs and LXC containers directly to their respective public IPs, greatly simplifying the steps. Everything worked perfectly.

The next step was to test the first migrations, starting from the VMs already on FreeBSD. For simplicity, I created a new FreeBSD VM in bhyve and copied (via zfs-send and zfs-receive) the datasets related to BastilleBSD. All services are installed in jails managed by Bastille, so this was enough to have, in a short time, a new operating server equivalent to the previous one. At that point, I shut down the original server, connected the VM to the bridge linked to the tunnel (after modifying its MAC address), turned on the new FreeBSD VM (on bhyve), and everything started to work correctly - but from the new physical server.

One by one, I moved all the FreeBSD VMs. For Linux, NetBSD, and OpenBSD, I simply copied the images and pointed bhyve to them. Some small specific configuration on vm-bhyve and everything started to work correctly. Where possibile, I replaced the “virtio” with “nvme” as it performs much better on bhyve.

Migration - LXC containers to Virtual Machines

For LXC containers, I initially thought of creating an Alpine Linux virtual machine, installing LXD, and copying each individual container. It worked for some of them, but for others, I started to encounter strange issues, similar to those that would have required manual intervention to upgrade from Proxmox 6.x to 7.x. As is often the case with Linux-based solutions, compatibility is not always preserved between updates, so I would have had to fine-tune all the containers, which I didn't feel like doing. The containers had been created (at the time) to optimize RAM usage on the Proxmox machine, but to date, they have caused more problems than benefits. In some cases, certain processes got "stuck," making it impossible to "reboot" the LXC container, requiring the entire physical node to be rebooted. If they had been virtual machines, I could have given a "kill" command from the virtualizer (to the respective KVM process, in that case) and restarted it.

For greater compatibility and ease of future management, I decided to convert the LXC containers into actual VMs on bhyve. The process was simple:

Creating an empty VM with vm-bhyve and booting the VM with SystemRescueCD.
Creating destination partitions and file systems in the VM, then doing a complete rsync of the original LXC container.
Adjusting the fstab file, installing the kernel on the destination VM, and creating the initrd (some containers were already copies of VMs, so the kernel remained installed and updated, even though it wasn't being used. The initrd, on the other hand, did not include the nvme or virtio drivers, so I had to regenerate it anyway.)
Adjusting the bhyve vm configuration file, doing one last rsync after shutting down the services, shutting down the original LXC container, and starting the bhyve VM.

Everything worked correctly, so one by one, I moved all the containers. The largest one ended up on another physical node (also FreeBSD with bhyve) temporarily because the space on the new server was not sufficient to contain it. It didn't need to be on this server, so no problem.

One by one, the LXC containers started on the new server. Apart from some minor adjustments to the destination VMs (different network interface names, etc.), I didn't encounter any particular problems even after several days. Everything works perfectly.

At the very end, I re-created the MikroTik CHR VM. I’ll keep this setup separate for now, as strictly tied to eoip interfaces. This was the main reason why I haven’t performed the migration before. Things were too tied together and I had to untie everything, step by step.

…and then one of the Linux VMs started to freeze

Several Linux VMs are just the basis on which Docker runs. One of them (not even among the busiest) started, every 12/15 hours, to completely freeze. It stopped responding to ping, and it was impossible to give any type of command from the console. In a word: stuck.

Searching the web, I found some references to this problem and, observing the errors of an ssh session that was left connected (stuck, but still showing the last error), I found it to be a problem similar to the one described in this post, namely:

"watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [khugepaged:67]"

I tried various solutions such as changing the storage driver, the number of cores, the distribution (from Alpine to Debian), etc., but none of these operations solved the issue. I also noticed that the problem occurs with all Linux VMs, but only those with a recent kernel (> 5.10.x) freeze, while the others continue to work. The problem does not occur, however, with the *BSDs.

In the end, I:

Reduced the number of cores to 1 for the VMs that did not have a high load (some remained with multiple cores), hypothesising a problem with allocating cores that were too busy
Gave the command: "/usr/bin/echo 60 > /proc/sys/kernel/watchdog_thresh" to the VM.

The VM became stable, and I have not seen that error/warning on any other machine since. I will investigate further, but I believe it is a problem related to the Linux kernel, which, for some reason, generates a kernel panic if particular situations of CPU concurrency are generated.

The End…and a nice OOM!

After moving everything, I was finally able to migrate the entire class of OVH IPs from one physical server to another. The operation was quite quick, but in order to avoid problems, I notified all users and performed the operation on a Sunday and during off-peak hours. The whole process took about 10 minutes and there were no hitches of any kind.

For safety reasons, I kept the Proxmox machine active for a few more days, but there was no need to use it. However, after a couple of days, I encountered a problem: the largest VM, in some cases, was being "killed" because FreeBSD generated an OOM. I had never seen, from FreeBSD 13.0 onwards, any OOM related to "abuse" of RAM usage by ZFS, but in this case, it actually happened.

In the end, I understood that ZFS, on FreeBSD, is able to release memory, but not quickly enough to manage any "spikes" in individual VMs. In fact, the VMs do not know the situation of the physical host's RAM, so they will tend to occupy all the space allotted to them (even if only for caching). A sudden spike (i.e. if you create and launch a new VM) could cause a sudden increase in RAM usage by the bhyve process, and FreeBSD could be forced to kill it, even if part of the RAM is only ARC cache. While Proxmox supports HA (i.e., control over whether the VM is running), vm-bhyve only launches the VM (bhyve process). I should manage it with tools like monit, but for now, I preferred to simply set limits on ZFS RAM usage using "vfs.zfs.arc_max", and there have been no more problems.

Final considerations

The operation was long but linear. The most complex part was unraveling all the configurations related to MikroTik CHR and the VPNs linked to each individual LXC machine/container. Once everything was implemented on a dedicated VM, the operation was fairly straightforward.

The hardware specifications of the destination physical server are slightly better than the starting one, but the final performance of the setup has greatly improved. The VMs are very responsive (even those that were previously LXC containers running directly on bare metal) and, thanks to ZFS, I can make local snapshots every 5 minutes. In addition, every 10 minutes, I can copy (using the excellent zfs-autobackup) all the VMs and jails to other nodes both as a backup and as an immediate restart in case of disaster. I just need to map the IPs, and everything will start working very quickly. Proxmox also allows you to perform this type of operation with ZFS, but you still need to have Proxmox (in a compatible version) on the target machine. With the current setup, I only need any FreeBSD node that supports bhyve.

Proxmox is an excellent tool, well-developed, open-source, efficient, and stable. We manage many installations, including complex ones (ceph clusters, etc.), and it has never let us down. However, not all tools are ideal for all situations, and for setups like the one described, the new configuration based on FreeBSD has shown significantly interesting performance and greater management and maintenance granularity.

Virtualizing on vm-bhyve is not complex, but it is certainly not comparable, at the current state, to the simplicity of using a clean and complete interface like Proxmox's. A complete HA system is still missing (sure, it's achievable manually, but...), as well as complete management web interface. However, for knowledgeable users, it is undoubtedly a powerful tool that allows you to have excellent FreeBSD as a base. I'm totally satisfied with my migration and the result is far better than I expected.

Creating an Alpine Linux VM on bhyve - with root on ZFS (optionally encrypted)

Stefano Marinelli — Tue, 01 Nov 2022 15:47:35 +0000

Bhyve is great - and we’re using it with a lot of guest operating systems.

One of my favourite Linux distributions is Alpine Linux - it’s great as a docker or lxc/lxd host, is light, stable and easily manageable.

On FreeBSD, vm-bhyve already provides a good template for Alpine Linux, but it’s based on the plain standard image with boot on ext4.

So we need the alpine-extended iso - with zfs module.

Let’s create a new Alpine Linux bhyve VM:

vm create -t alpine -s 50G -m 4G -c 2 alpinevm

Now let’s configure it:

vm configure alpinevm

Let’s change some options: vmlinuz-vanilla and initramfs-vanilla should be changed to vmlinuz-lts and initramfs-lts.

More, we should instruct grub to boot from ZFS. The configuration should be similar to this:

[…]
grub_install0="linux /boot/vmlinuz-lts initrd=/boot/initramfs-lts alpine_dev=cdrom:iso9660 modules=loop,squashfs,sd-mod,usb-storage,sr-mod"
grub_install1="initrd /boot/initramfs-lts"
grub_run0="linux /boot/vmlinuz-lts root=rpool/ROOT/alpine rootfstype=zfs modules=ext4,zfs"
grub_run1="initrd /boot/initramfs-lts"
[…]

It’s now time to start with the installation:

vm install alpinevm alpine-extended.iso

Now, follow this Alpine Linux Wiki guide. Remember that we’re dealing with “vda” devices, not "sda", so change them accordingly. If you don’t want to have an encrypted rootfs dataset, just avoid the encryption line in the zpool create command.

At the end of the installation procedure, reboot and enjoy your new Alpine Linux bhyve VM.

Why we're migrating (many of) our servers from Linux to FreeBSD

Stefano Marinelli — Mon, 24 Jan 2022 06:55:00 +0000

More about this in the article I wrote to accompany my talk at EuroBSDCon 2024.

Please note: This article was (unexpectedly) quite popular and many interesting opinions came out from the Hacker News thread that it generated. If you're interested, have a look here: https://news.ycombinator.com/item?id=30057549

There's an Italian version of this article here.

I've been a Linux (or GNU/Linux, for the purists) user since 1996. I've been a FreeBSD user since 2002. I have always successfully used both operating systems, each for specific purposes. I have found, on average, BSD systems to be more stable than their Linux equivalents. By stability, I don't mean uptime (too much uptime means too few kernel security updates, which is wrong). I mean that things work as they should, that they don't "break" from one update to the next, and that you don't have to revise everything because of a missing or modified basic command.

I've always been for development and innovation as long as it doesn't (necessarily, automatically and unreasonably) break everything that is already in place. And the road that the various Linux distributions are taking seems to be that of modifying things that work just for the sake of it or to follow the diktats of the Kernel and those who manage it - but not only.

Some time ago we started a complex, continuous and not always linear operation, that is to migrate, where possible, most of the servers (ours and of our customers) from Linux to FreeBSD.

Why FreeBSD?

There are many alternative operating systems to Linux and the *BSD family is varied and complete. FreeBSD, in my opinion, today is the "all rounder" system par excellence, i.e. well refined and suitable both for use on large servers and small embedded systems. The other BSDs have strengths that, in some fields, make them particularly suitable but FreeBSD, in my humble opinion, is suitable (almost) for every purpose.

So back to the main topic of this article, why am I migrating many of the servers we manage to FreeBSD? The reasons are many, I will list some of them with corresponding explanations.

The system is consistent - kernel and userland are created and managed by the same team

One of the fundamental problems with Linux is that (we shall remember) it is a kernel, everything else is created by different people/companies. On more than one occasion Linus Torvalds as well as other leading Linux kernel developers have remarked that they care about the development of the kernel itself, not how users will use it. In the technical decisions, therefore, they don't take into account what is the real use of the systems but that the kernel will go its own path. This is a good thing, as the development of the Linux kernel is not "held back" by the struggle between distributions and software solutions, but at the same time it is also a disadvantage. In FreeBSD, the kernel and its userland (i.e. all the components of the base operating system) are developed by the same team and there is, therefore, a strong cohesion between the parties. In many Linux distributions it was necessary to "deprecate" ifconfig in favor of ip because new developments in the kernel were no longer supported by ifconfig, without breaking compatibility with other (previous) kernel versions or having functions (on the same network interface) managed by different tools. In FreeBSD, with each release of the operating system, there are both kernel and userland updates, so these changes are consistently incorporated and documented, making the tools compatible with their kernel-side updates.

In other words, in FreeBSD there is no need to " revolutionise" everything every few years and changes are made primarily in the form of additions that can enrich (and not break) each update. If a modification was to change the way it interacts with network devices, ifconfig would be modified to take advantage of that and remain compatible with the "old" syntax. In the long-term, this kind of approach is definitely appreciated by system administrators who find themselves with a linear, consistent, and always well-documented update path.

FreeBSD development is (still) driven by technical interests, not strictly commercial ones.

Linux and related distributions now have contributions from many companies, many of which (e.g. Red Hat) push (justifiably) in the direction of what is convenient for them, their products, and their services. Being big contributors to the project they have a big clout so, indeed, their solutions often become de-facto standards. Consider systemd - was there really a need for such a system? While it brought some advantages, it added some complexity to an otherwise extremely simple and functional system. It remains divisive to this day, with many asking, "but was it really necessary? Did the advantages it brought balance the disadvantages?". 70 binaries just for initialising and logging and a million and a half lines of code just for that? But Red Hat threw the rock...and many followed along. Because sometimes it's nice to follow the trend, the hype of a specific solution.

Even FreeBSD has big companies behind it, collaborating in a more or less direct way. The license is more permissive, so not everyone who uses it commercially contributes to it, but knowing that FreeBSD is at the base of Netflix CDNs, Whatsapp servers (waiting for Meta to replace them, for internal coherence reasons, with Linux servers), Sony Playstations and, in part, macOS, iOS, iPadOS, etc. surely gives confidence on its level. These realities, however, do not have enough clout to drive the development of the core team.

Linux has Docker, Podman, lxc, lxd, etc. but... FreeBSD has jails!

FreeBSD jails are very powerful tools for jailing and separating services. There is controversy about Docker not running on FreeBSD but I believe (like many others) that FreeBSD has a more powerful tool. Jails are older and more mature - and by far - than any containerization solution on Linux. Jails are efficient and are well integrated throughout the operating system. All major commands (ps, kill, top, etc.) are able to display jail information as well. There are many management tools but, in fact, they all do the same thing: they interact with FreeBSD base and create custom configuration files. Personally I'm very comfortable with BastilleBSD but there are a lot of very good tools as well as a sufficiently simple manual management. When I need Docker I launch a Linux machine - often Alpine, which I think is a great minimalist distribution, or Debian. But I'm moving a lot of services from Docker to a dedicated jail on FreeBSD. Docker containers are a great tool for rapid (and consistent) software deployment, but it's not all fun and games. Containers, for example, rely on images that sometimes age and are no longer updated. This is a security issue that should not be overlooked.

Linux has ext4, xfs, btrfs... (and zfs, but with some manual intervention). FreeBSD has UFS2 and ZFS

UFS2 is still a very good and efficient file system and, when configured to use softupdates, capable of performing live snapshots of the file system. This is great for backups. Ext4 and XFS do not support snapshots except through external tools (like DattoBD or snapshots through the volume manager). This works, of course, but it is not native. Btrfs is great in its intentions but still not as stable as it should be after all these years of development. FreeBSD supports ZFS natively in the base system and this brings many advantages: separation of datasets for jails, as well as Boot Environments, to make snapshots before upgrades/changes and to be able to boot (from bootloader) even on a different BE, etc.

The FreeBSD boot procedure is cleaner and simpler.

Linux has always used excellent tools such as grub, lilo (now outdated), etc.. FreeBSD has always used a very linear and consistent boot system, with its own bootloader and dedicated boot partition. Whether on mbr, gpt, etc. things are very similar and consistent. I've never had a problem getting a FreeBSD system to boot after a move or recovery from backup. On Linux, however, grub has sometimes given me problems, even after a simple kernel security update.

FreeBSD's network stack is (still) superior to Linux's - and, often, so is its performance.

Meta has been trying to bring the performance of the Linux network stack up to the level of FreeBSD's for years. Many will ask why, then, not move services to FreeBSD. Large companies with huge datacenters can't change solutions overnight, and their engineers, at any level, are Linux experts. They have invested heavily in btrfs, in Linux, in their specifics. Clearly, upon acquiring Whatsapp, they preferred to migrate the "few" Whatsapp servers to Linux and move them to their datacenters. Regarding the real system performance (i.e. disregarding benchmarks, useful only up to a certain point), FreeBSD shines, especially under high load conditions. Where Linux starts to gasp (ex: waiting for I/O) with 100% CPU, FreeBSD has lower processor load and room for more stuff. In the real world (of my servers and load types), I sometimes experienced severe system slowdowns due to high I/O, even if the data to be processed was not read/write dependent. On FreeBSD this does not happen, and if something is blocking, it blocks THAT operation, not the rest of the system. When performing backups or other important operations this factor becomes extremely important to ensure proper (and stable) system performance.

Straightforward system performance analysis

FreeBSD, in the base system, has all the tools to analyze possible problems and system loads. "vmstat" , in a single line, tells me if the machine is struggling for CPU, for I/O or for Ram. "gstat -a" shows me how much, disk by disk, partition by partition, the storage is active, also in percentage with reference to its performance. "top", then, also has support for figuring out, process by process, how much I/O is being used ("m" option). On Linux, to get the same results, you have to install specific applications, different from distribution to distribution.

Bhyve is more incomplete (but more efficient) than KVM

For my purposes, Bhyve is a great virtualization tool. KVM is definitely more complete but since I don't have any special or specific needs not covered by Bhyve on FreeBSD, I found (on average) better performance with this combination. On FreeBSD, however, KSM is missing which, in some cases, can be very useful.

Will I abandon Linux for FreeBSD? Obviously not, just as I haven't for the last 20 years. Both have their uses, their space, their strengths. But if up to now I have had 80% Linux and 20% FreeBSD, the perspective is to invert the percentages of use and, where possible, directly implement solutions based on FreeBSD.

NOTE: this article has been translated from its Italian original version. Even if it's been reviewed and adapted, there might be some errors.

FreeBSD, Raspberry PI - problems booting with USB powered hub connected

Stefano Marinelli — Sat, 04 Dec 2021 10:30:00 +0000

I’ve been using Raspberry Pis since 2012, running both Linux and FreeBSD.

A couple of Raspberry Pi 4 devices are running local services and are performing as backup servers. They’ve been running for one year on Alpine and btrfs but I’ve then decided to convert both of them to FreeBSD and ZFS.

Everything is ok, I can boot from a read only SD card and use the external usb disks as ZFS pools/volumes. The Rpi4 has two USB3 ports, so I’m using a powered USB 3.0 hub. FreeBSD is able to detect it, but there seems to be a problem with u-boot.

If u-boot detects there's a powered hub, it seems to go crazy and continue to try to boot from the USB hub. Even if you have a bootable disk/usb pen, u-boot just goes crazy and isn’t even able to recognize the partitions on the SD card. The result is that the Rpi4 won’t boot and start to loop.

I’ve been able to solve by recompiling u-boot without any USB support. This means I always have to use a SD card (or a network boot), but it’s ok for me. U-Boot won’t even see the USB stack and boot directly from the local SD. Then, when FreeBSD kernel has been loaded, all the USB devices will be detected.

If you want to try my binary file - for Raspberry 4, compiled on 03/12/2021, you can find it here. The file is provided as is, it's working for me but there's no gurantee that it will be okay for you. I don't take responsibility at all.

Alpine Linux and LXD perfect setup on BTRFS file system

Stefano Marinelli — Wed, 03 Nov 2021 10:54:10 +0000

Last update: 01 November 2022 - Alpine 3.16 has lxd packages in contrib

Lxc is a great technology. I’ve been using it since 2012 and I’ve always appreciated its lightweight approach and efficiency. Nowadays KVM and hardware virtualization extensions have reached a very good level of performance. While a KVM VM allows to run another kernel and another OS, it ties specific resources to the VM and this isn’t always the best solution. Lxc uses the host’s kernel and shares its ram so the host machine runs lxc container’s processes as local processes - no slowdown, no resource wasting.

Alpine Linux is a great distribution. It’s minimal, reliable, secure and easy to deploy. It’s great for containers but it’s also great for low footprint servers, like hypervisors. It’s good as Xen Dom0, it’s good as KVM host and - no surprise - is great as Lxc host too.

Alpine Linux is great for low energy devices (Raspberry Pis, etc.), too, and has an efficient and reliable out-of-the-box read-only file system support. But I’ll write more about it in another post.

Lxd is a great solution. It uses lxc but simplifies a lot of operations like new container creation, profile management, automatic snapshots, clustering, etc. The “official” best option to use lxd is installing Ubuntu, Snap and so on. I don’t want to use Ubuntu to install Snap. I don’t want to use Snap just to install lxd. That’s why I’m choosing to use lxd with Alpine.

Long story short: this video shows all the steps described below:

Installing Alpine Linux on BTRFS Root File System

Alpine Linux installer supports btrfs as rootfs destination but the setup-alpine command doesn’t allow you to select it: it will partition your disk and create a ext4 standard root file system.

So let’s start with the installation but, then, we’ll interrupt it at disk selection:

Start with setup-alpine and set network, hostname, password, etc. Stop at:

Available disks are:
sda (20.5 GB QEMU     QEMU HARDDISK   )
Which disk(s) would you like to use? (or '?' for help or 'none') [none]

Press ctrl+C and interrupt the script.

Now we can install Alpine Linux on btrfs root system:

test:~# export ROOTFS=btrfs

And, then:

test:~# setup-disk /dev/sda

 The following disk is selected:
 sda    (20.5 GB QEMU     QEMU HARDDISK   )
 How would you like to use it? ('sys', 'data', 'lvm' or '?' for help) [?] sys
 WARNING: The following disk(s) will be erased:
 sda    (20.5 GB QEMU     QEMU HARDDISK   )
 WARNING: Erase the above disk(s) and continue? (y/n) [n] y
 Creating file systems...
 btrfs-progs v5.10
 See http://btrfs.wiki.kernel.org for more information.

 Label:              (null)
 UUID:               6ace888a-2f36-41e8-86b1-3e4214b9e34e
 Node size:          16384
 Sector size:        4096
 Filesystem size:    15.17GiB
 Block group profiles:
 Data:             single            8.00MiB
 Metadata:         DUP             256.00MiB
System:           DUP               8.00MiB
 SSD detected:       no
 Incompat features:  extref, skinny-metadata
 Runtime features:
 Checksum:           crc32c
 Number of devices:  1
 Devices:
 ID        SIZE  PATH
 1    15.17GiB  /dev/sda3

 Installing system on /dev/sda3:
 /mnt/boot is device /dev/sda1

Installation is complete. Now eject the installation media and reboot.

After rebooting, let’s check that root file system is on btrfs:

 test:~# cat /etc/fstab
 UUID=aacbfd28-df47-4945-9c0c-279faef361a0  /   btrfs   rw,relatime,space_cache,subvolid=5,subvol=/ 0 1

Ok, it is. Generally speaking, I prefer to use compression on btrfs. compress=lzo is enough, so you can add it on your fstab.

Now, let’s install lxd.

Edit your /etc/apk/repositories and let it be like this:

http://dl-cdn.alpinelinux.org/alpine/v3.16/main
http://dl-cdn.alpinelinux.org/alpine/v3.16/community

Let’s follow the tutorial I wrote on Alpine Linux wiki and let’s install lxc, lxd, lxcfs and dbus (not needed but recommended to avoid problems with some containers).

After rebooting, you have a fully functional lxd-enabled Alpine Linux installation.

Let’s now configure lxd:

 test:~# lxd init
 Would you like to use LXD clustering? (yes/no) [default=no]:
 Do you want to configure a new storage pool? (yes/no) [default=yes]:
 Name of the new storage pool [default=default]:
 Name of the storage backend to use (btrfs, dir) [default=btrfs]:
 Would you like to create a new btrfs subvolume under /var/lib/lxd? (yes/no) [default=yes]:
 Would you like to connect to a MAAS server? (yes/no) [default=no]:
 Would you like to create a new local network bridge? (yes/no) [default=yes]:
 What should the new bridge be called? [default=lxdbr0]:
 What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
 What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
 Would you like the LXD server to be available over the network? (yes/no) [default=no]:
 Would you like stale cached images to be updated automatically? (yes/no) [default=yes]
 Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:

As you can see, lxd automatically detected that /var/lib/lxd is on a btrfs file system and creates a sub volume.

Now we can launch our first container:

test:~# lxc launch images:alpine/3.16 t1
Creating t1
Starting t1

Ok, done. Let’s check if it’s working:

test:~# lxc info t1
Name: t1
Status: RUNNING
Type: container
Architecture: x86_64
PID: 2648
Created: 2021/11/03 08:55 UTC
Last Used: 2021/11/03 08:55 UTC

Resources:
Processes: 5
CPU usage:
CPU usage (in seconds): 0
Memory usage:
Memory (current): 3.38MiB
Memory (peak): 5.04MiB
Network usage:
eth0:
Type: broadcast
State: UP
Host interface: vethdc207601
MAC address: 00:16:3e:50:d1:ef
MTU: 1500
Bytes received: 3.76kB
Bytes sent: 1.79kB
Packets received: 26
Packets sent: 16
IP addresses:
inet:  10.189.248.21/24 (global)
inet6: fd42:34cb:9408:433c:216:3eff:fe50:d1ef/64 (global)
inet6: fe80::216:3eff:fe50:d1ef/64 (link)
lo:
Type: loopback
State: UP
MTU: 65536
Bytes received: 0B
Bytes sent: 0B
Packets received: 0
Packets sent: 0
IP addresses:
inet:  127.0.0.1/8 (local)
inet6: ::1/128 (local)

It’s running. Everything is ok. For more complex lxc setup (bridges, profiles, etc.) you can follow the excellent lxd documentation.

This kind of setup is easy and lightweight, easy to backup (a new article will follow) and reliable.