Migrating from an Old Linux Server to a New FreeBSD Machine -

Preamble: I believe it’s time to bid farewell to this venerable Linux server.

TL;DR: The article chronicles the journey of transitioning from an outdated Linux server, running for 1690 days without updates, to a modern FreeBSD machine. This migration involved using tools like mfsBSD, BastilleBSD, Borg Backup, and bhyve. Despite initial hesitations due to the Linux server’s impeccable performance, the transition was smooth, resulting in improved manageability and efficiency. The piece emphasizes the importance of regular system updates and anticipates revisiting the topic in the future with new uptime achievements and updates.

This server loyally served for years as a secondary backup server while also providing a few minor services to users. As it often happens, it remained in operation, neglected and without updates for years. Stable operating systems have the “flaw” of being forgotten, giving the false impression that they don’t need maintenance or updates. This machine continued its service without oversight for years. When approached for a service request (not due to malfunctions), I advised the client to upgrade the whole system. A mere update would not suffice, so I suggested starting afresh on new hardware with FreeBSD as the primary OS.

The client was understandably hesitant given the uptime stats:

08:58:43 up 1690 days, 21:32, 4 users, load average: 9.57, 10.15, 8.76

Not a single error, not a single hiccup. From his perspective, a similar setup to what was installed many years ago and still working flawlessly was preferred. Nevertheless, he trusted my expertise and let me proceed.

This server had a plethora of duties, among which:

One of the pivotal tasks was running Proxmox Backup Server via Docker. Proxmox Backup Server requires Debian, but this server was running on Ubuntu 16.04 (previously upgraded from Ubuntu 14.04 – yes, ancient!). Hence, Proxmox Backup Server was still at version 1.x.
Another critical function was storing backups made through BorgBackup on its file system. The /home directory used a mirrored btrfs file system, and each backed-up server had its user on this system. Clients could backup (using a push method) only via VPN and only during specific windows when the server permitted (by adding specific firewall rules via Jenkins. Jenkins also managed connection protocols, snapshots, backups, etc.).
Among the lesser tasks, the server ran a few Docker containers with HandBrake on various presets. The client processed video conversions by uploading the original files via sftp, and after some hours, fetched the converted files from the destination directory. This will not be replicated on the new FreeBSD server since they now handle this operation locally on their high-performance MacBook Pro with Apple Silicon. However, a future restoration isn’t off the table.

The first thing I did was install FreeBSD on the new hardware. Given that it’s a physical server on Hetzner (an auction pick due to disk space needs over power), and FreeBSD wasn’t an option, I used mfsBSD.

After booting the physical server in Linux rescue, I copied mfsBSD onto the disks using dd and restarted. On boot, I SSHed into mfsBSD and executed the installation using bsdinstall—a robust and efficient method.

I set up a raidz1 with all four 6 TB disks, resulting in a final storage space of 21.8T, ample for now without the video files.

To ensure continuity, I kept a setup similar to the old one. The clients would essentially continue with their usual backup procedure without necessitating drastic changes to backup scripts. To avoid storing these backups directly in the physical machine’s /home and to leave the door open for future services, I installed BastilleBSD and began setting up several jails. I replaced the old Linux machine’s behavior with a VNET FreeBSD jail. In past scenarios, I’ve created Linux jails (thanks to BastilleBSD) and transferred the old server into the jail using rsync, making minor configuration tweaks. While this usually works, it doesn’t address the underlying issue of an obsolete setup. Given the opportunity, I opted for a modern toolset.

Thus, I copied every home directory (along with their historic backups) in its entirety, installed BorgBackup, and re-established the VPN. With a VNET jail, I can craft networking devices and fine-tune configurations. After recreating user accounts, inputting the various SSH authorized_keys, and checking all clients, I set up a snapshot plan on the host. This ensures that if a client is compromised with the potential (however remote) for breach and backup deletion, a ZFS snapshot of the entire jail remains available.

As mentioned, one of the core tools on the old server was Proxmox Backup Server. It’s not natively installable on FreeBSD, necessitating a VM. Enter the fantastic bhyve, supported by vm-bhyve.

However, one issue arose: backups would consume vast amounts of space, and I wanted to avoid housing an enormous disk image (or a zvol) with both VMs and backups. So, I opted for a slightly less performant but more flexible solution: installing Debian 12 and Proxmox Backup Server on the VM while placing backups on a separate ZFS dataset on the physical machine, exported via NFS and mounted on the VM.

Given that the physical server has an internal bridge “vm-public” with IP 192.168.124.1 and the VM is at 192.168.124.2, I just created a dataset named zroot/PBS and added the following line to /etc/exports:

/zroot/PBS -alldirs -maproot=root -network 192.168.124.2/32

To enable NFS, insert into /etc/rc.conf:

rpcbind_enable="YES" 
nfs_server_enable="YES" 
mountd_flags="-r" 
rpc_lockd_enable="YES"

Within the VM, create /PBS and include in /etc/fstab:

192.168.124.1:/zroot/PBS /PBS nfs rw,async,soft,intr,noexec 0 0

After Proxmox Backup Server’s installation, simply set up the datastores in /PBS/, and they’ll directly store on the physical machine’s ZFS dataset.

For firewall configurations, I exposed port 8007, redirecting it towards the VM, and everything started functioning smoothly. I then set a Proxmox Backup Server replica from the old to the new server. After completion, I changed the Proxmox Backup Server IP on all Proxmox hosts to point to the new server. Smooth sailing.

The old Ubuntu server also managed other minor services, which have become obsolete and weren’t replicated.

The transition was seamless, the client is pleased, and I’m content since each service is now neatly segregated into its jail or VM. The machine’s load is minimal, which might pave the way for other tasks, via VPN. Everything now rests on ZFS, and the icing on the cake: I made the client promise not to reach another 1690 days of uptime but to timely update as required.

I’m not entirely convinced the promise will hold—meaning, in a few years, I might yet be discussing this “new” server, highlighting another impressive uptime and another upgrade journey.

See also