How we are migrating (many of) our servers from Linux to FreeBSD - Part 1

After my post on why we’re migrating (many of) our servers to FreeBSD, I’ve received a lot of feedback. Many questions, many comments. Many e-mails from Linux users asking how we’re migrating, how jails can replace lxc or (in part) Docker, and how we’re monitoring and performing backups/restores.

I’ll write some posts on how we’re doing it. Of course, it won’t cover all the use cases, but it surely will work for the most common ones.

Let’s start with a general idea of the setup I’m going to describe. One of the things I’ve always tried to do is to leave the host Operating System (VM Hypervisors like XCP-NG or Proxmox Server, FreeBSD with jails and/or Bhyve, Alpine Linux lxc host, Debian with docker etc.) as simple and empty as possible.

That’s why we’re keeping the same setup here, with a FreeBSD host system with as few packages (or ports) as possible. This will ensure easier upgrades, easier backup/restore procedures and a smaller attack surface. Unused services or executables can be problematic, so let’s keep them out of our setup.

I’m going to describe our “basic” FreeBSD host and some jails that will contain the services. There are many jail management systems out there like BastilleBSD, cbsd, iocage, the old ezjail, etc. I love simple (but powerful) solutions, without any database of configured/running jails as it may be a problem in case of backup/recovery. I love the BastilleBSD approach, so that’s the one we’ve been using for our servers. BastilleBSD is a collection of shell scripts, is small, doesn’t need any database, is actively developed, doesn’t need any kind of dependency and works both on ZFS and UFS. iocage, for example, needs ZFS so UFS servers are out and doesn’t seem to be actively developed anymore. Moreover, BastilleBSD doesn’t interfere with other jail systems, so you can mix and match whatever you like. Still, I recommend to choose a jail management system and stick with that. We’ve done it, and we’re using BastilleBSD.

I won’t describe basic FreeBSD installation. It’s straightforward, easy, fast and it’s full of good documentation out there. The most critical decision is the choice of the file system you’re going to use. We’re using ZFS when possible, as it gives us a lot of good opportunities. For more resource constrained systems (or specific situations where ZFS isn’t recommended nor applicable), we just stick with UFS. It has snapshots, too, so backups and restores are quite easy, too.

Once you’ve installed FreeBSD, you have a fully functional host system. From now on, I’ll assume you’ve installed on ZFS.

First of all, we upgrade our hosts systems to the last security patches. It’s just a matter of:

freebsd-update fetch
freebsd-update install

After this, It’d be better to reboot to be sure that the kernel has been upgraded if needed. As it’s an empty and just installed system, we usually don’t use the Boot Environments for now. Reinstalling is just a matter of minutes and we’re not doing a release upgrade.

Now: ports or packages? We usually use packages: the host system will be simple and empty and we don’t need customised build options. So quarterly packages are more than enough.

First of all, let’s install the “pkg” package management system.

pkg install

BastilleBSD

Now let’s use pkg to install BastilleBSD:

pkg install bastille

The first step is to configure BastilleBSD. All we usually do is: set the ZFS options. bastille_zfs_enable=“YES” and bastille_zfs_pool=“zroot/bastille” and create the dataset:

zfs create zroot/bastille

This way, BastilleBSD will create its base dataset in zroot/bastille/bastille

Why the nested bastille/bastille dataset? Because we sometimes use zroot/bastille to store some information about the underlying jails, so we prefer to keep a nested dataset. BastilleBSD will create its datasets there and will mount the needed datasets in /usr/local/bastille so you will find everything there.

Let’s complete the Bastille installation as described in the official documentation. There are many approaches: loopback, vnet with local interface, vnet with existing bridge. The loopback approach is the easiest and more portable. Generally speaking we tend to use it as it’s easier to deal with when we have to perform an emergency restore to another host. I’ll write more about it in the next posts.

Now let’s bootstrap the FreeBSD 13.2-RELEASE so Bastille will download the needed files and create its base system, then it will apply the security patches.

bastille bootstrap 14.0-RELEASE update

NOTE: BastilleBSD supports other jailed operating systems and, recently, some jailed Linux distributions.

After a while, the system will be ready for its first jail. Now, we generally install a reverse proxy, in order to expose it to web traffic. The reverse proxy will be able to connect to other jails and forward the traffic. It’s a good example, so let’s do it:

bastille create reverseproxy 14.0-RELEASE 192.168.0.1 bastille0

So we’re creating a jail called reverseproxy, the FreeBSD version is 14.0-RELEASE (of the jail, must be the same or older than the host OS), the jail ip and the loopback interface to use to assign this IP for the jail. The jail will have a dataset (in jails/reverseproxy) with jail configuration, redirect configuration, fstab, etc. and another child dataset (jails/reverseproxy/root) with its root file system. Jails can be thin or thick: BastilleBSD documentation is good, so you can go deeper here.

At this point, a “bastille list -a” should show the jail and it should be running. Now we can enter this jail:

bastille console reverseproxy

and install Nginx (and certbot, if needed):

pkg install py39-certbot-nginx nginx

Configure your nginx (and certbot). As you might guess, I won’t describe it here.

Let’s ensure Nginx will be started at jail launch, so:

service nginx enable

and let’s start it:

service nginx start

Let’s go back to the host and let’s ensure that all the connections to the host ports 80 (http) and 443 (https) will be redirected to the reverseproxy jail:

bastille rdr reverseproxy tcp 80 80
bastille rdr reverseproxy tcp 443 443

No output should be shown, but you can check:

bastille rdr ALL list

Congratulations, you’ve exposed your first jail.

You can create all the jails you want. They will be created and stored in child datasets of zroot/bastille/bastille - this is great for backup purposes, but I’ll describe more about it in another article. They’ll also be able to communicate using their private ip addresses. If you’ve used vnet, you’ll need to perform some deeper network configurations (and you can use pf inside the jail!), if using the default loopback device you’ll be sharing the host network stack.

Blacklistd

FreeBSD base system has some interesting tools and they get automatically installed. One of those is blacklistd. If you’ve used fail2ban, denyhosts or similar tools, you know what it’s useful for. But it’s integrated and is light. Fail2ban, for example, tends to become heavy and huge as it’s reading from log files. Blacklistd gets notified by the daemon it’s protecting, so load is lower. To enable blacklistd, add to /etc/rc.conf:

blacklistd_enable="YES"
blacklistd_flags="-r" 
pflog_enable="YES"

(I’m assuming you’ve already added pf_enable=“YES” when you’ve installed BastilleBSD. Otherwise, you should add this, too, if you’re using pf.

Now you should add this to pf.conf:

anchor "blacklistd/*" in on $ext_if

Where $ext_if is your external interface.

Last step, let’s get to /etc/ssh/sshd_config and enable blacklistd uncommenting:

UseBlacklist yes

Now reload/start pf, sshd, start pflog and blacklistd and wait. After some time, blacklistctl dump -r will show you some data.

Of course there are many more steps to do, the host should be hardened, network should be configured and firewalled, etc. but it’s a basic idea of how we’re keeping our host as standard as possible and, then, create the services inside the jails.

How we are migrating (many of) our servers from Linux to FreeBSD - Part 1 - System and jails setup

BastilleBSD

Blacklistd

See also