I have been managing a client's server for several years. I inherited a setup (actually, partly done by me at the time, but following the directions of their internal administrator) based on Proxmox. Originally, the VM disks were qcow2 files, but over time and with Proxmox updates, I managed to create a ZFS pool and move them onto it. For backups, I continued to use Proxmox Backup Server (even though virtualized on bhyve) - a solution we've been using for several years.
The existing VMs/containers were:
- An LXC container based on Debian, with Apache as a reverse proxy. The choice of Apache was mainly tied to specific configurations from the company that produces the underlying software. It has always done an excellent job.
- A VM running OPNsense — the choice was related to direct use by the client, as it allows easy creation/modification of users for the internal VPN. Some users can access via VPN (OpenVPN) and perform specific operations, such as SSH connections. It has always worked well and is up to date.
- Two VMs based on Rocky Linux 9 — both with components of the management software in Java, with the reverse proxy directing requests based on the requested site.
- Two VMs based on Rocky Linux 8 — these also have parts of the management software and the databases. These machines are used both by providing specific URLs and by the other two as databases.
The load is not particularly high, and the machines have good performance. Suddenly, however, I received a notification: one of the NVMe drives died abruptly, and the server rebooted. ZFS did its job, and everything remained sufficiently secure, but since it's a leased server and already several years old, I spoke with the client and proposed getting more recent hardware and redoing the setup based on a FreeBSD host.
Unfortunately, I cannot change the setup of the VMs. In my opinion, there are no contraindications since the databases are PostgreSQL and the management applications are Java applications with Tomcat. However, the software house only certifies that type of setup, and considering they are the ones performing updates and fixes, it's appropriate to maintain the setup they suggest. It's a technically sound choice in my opinion, so I can't say anything negative.
Acquiring New Hardware
The first step was, of course, to get the new server. It took a few days because I requested ECC RAM (for a small price variation), and the time extended. As soon as the physical server was ready, I got to work.
I installed FreeBSD 14.1-RELEASE, root on ZFS, creating a mirror setup between the NVMe drives. The I/O is not particularly heavy in this setup, but I don't want to waste potential: even if I can reduce the length of an operation by a few seconds, I see no reason not to do it.
As the first step, I decided to manually create the bridge to which I will connect both the VMs and the reverse proxy, which will be installed inside a FreeBSD jail.
vm-bhyve, the tool I use to manage VMs, allows creating bridges, but I prefer to manage it manually and maintain more complete control over everything. I will also enable pf.
I decided to use zstd as the compression algorithm and disable atime:
zfs set compression=zstd zroot
zfs set atime=off zroot
I then modified the /etc/rc.conf file as follows:
cloned_interfaces="bridge0 lo1"
ifconfig_lo1_name="bastille0"
ifconfig_bridge0="inet 192.168.33.1 netmask 255.255.255.0"
gateway_enable="YES"
pf_enable="YES"
bastille_enable="YES"
I then updated FreeBSD by running the usual command:
freebsd-update fetch install
Configuring the Firewall
I configured the firewall with a basic setup:
ext_if="igb0"
set block-policy return
set skip on bridge0
table <jails> persist
nat on $ext_if from {192.168.33.0/24} to any -> ($ext_if)
nat on $ext_if from <jails> to any -> ($ext_if:0)
# nginx-proxy - to the jail
rdr on $ext_if inet proto tcp from any to PUBLIC_IP port = 80 -> 192.168.33.254 port 80
rdr on $ext_if inet proto tcp from any to PUBLIC_IP port = 443 -> 192.168.33.254 port 443
# opnsense
rdr on $ext_if inet proto tcp from any to PUBLIC_IP port = 1194 -> 192.168.33.253 port 1194
rdr on $ext_if inet proto udp from any to PUBLIC_IP port = 1194 -> 192.168.33.253 port 1194
rdr-anchor "rdr/*"
block in all
antispoof for $ext_if inet
pass in inet proto icmp
pass out quick keep state
pass in inet proto tcp from any to any port {http,https} flags S/SA keep state
pass in inet proto {tcp,udp} from any to any port 1194 flags S/SA keep state
# This will be closed at the end of the setup and will be allowed only via VPN
pass in inet proto tcp from any to any port ssh flags S/SA keep state
Once finished, I rebooted everything to load the new kernel. No problems.
Installing Necessary Tools
I then installed some useful packages:
pkg install tmux py311-zfs-autobackup mbuffer rsync vm-bhyve-devel edk2-bhyve grub2-bhyve bastille
I created the datasets for the jails and VMs:
zfs create zroot/bastille
zfs create zroot/VMs
I then started configuring everything. First, BastilleBSD: I modified /usr/local/etc/bastille/bastille.conf by adding:
## ZFS options
bastille_zfs_enable="YES"
bastille_zfs_zpool="zroot/bastille"
Then I enabled and configured vm-bhyve, enabling the serial console on tmux:
sysrc vm_enable="YES"
sysrc vm_dir="zfs:zroot/VMs"
vm init
cp /usr/local/share/examples/vm-bhyve/* /zroot/VMs/.templates/
vm switch create -t manual -b bridge0 public
vm set console=tmux
Now I bootstrapped FreeBSD 14.1-RELEASE on BastilleBSD:
bastille bootstrap 14.1-RELEASE update
Setting Up the Reverse Proxy Jail
I then started by creating the jail for the reverse proxy:
bastille create -B apache 14.1-RELEASE 192.168.33.254/24 bridge0
Once created, I had to modify the default gateway of the jail because by default it is set to that of the host. It was enough to set 192.168.33.1 in /usr/local/bastille/jails/apache/root/etc/rc.conf.
The configuration of Apache will not be described here as it is closely dependent on the setup, but this jail can reach (in bridge mode) all the VMs that will be placed on the same bridge.
Migrating the VMs
For migrating the VMs, I decided to proceed as follows:
- Updating the operating systems of the original VMs - to have the latest version of the kernel and all userland. The VMs are not UEFI, so it will be necessary to have the exact names of the kernel and
initrdas they will need to be specified in the configuration file ofvm-bhyve. - Creating templates of the VMs with
vm-bhyve. - Snapshotting the VM disks and initial transfer to the new server with the VMs still running.
- Shutting down all the source VMs.
- Creating a new snapshot and performing an incremental transfer.
- Adjusting configurations and changing DNS.
I updated everything using dnf, and rebooted when there was also a kernel update. I took note of the versions to use, namely vmlinuz-4.18.0-513.18.1.el8_9.x86_64 and initramfs-4.18.0-513.18.1.el8_9.x86_64.img for Rocky Linux 8.10, and vmlinuz-5.14.0-362.18.1.el9_3.0.1.x86_64 and initramfs-5.14.0-362.18.1.el9_3.0.1.x86_64.img for Rocky Linux 9.4.
Migrating Rocky Linux 8.10 VMs
On the destination server, I created the VMs. I used the linux-zvol template, then modified the configurations:
vm create -t linux-zvol -s 1G -m 8G -c 4 vm100
I created a virtual disk of 1 GB because it will be replaced by the dataset sent from the current production server, so it's a fictitious size. At that point, I deleted the zvol:
zfs destroy zroot/VMs/vm100/disk0
I performed an initial snapshot of the source VMs:
zfs snapshot zfspool/vm-100-disk-0@Move01
And then I copied this first snapshot to the destination machine/VM:
zfs send -R zfspool/vm-100-disk-0@Move01 | mbuffer -s 128k -m 512M | ssh root@destMachine "zfs receive -F zroot/VMs/vm100/disk0"
At the end of the copy, you can do a test and try to boot. For the Rocky Linux 8.10 VMs, I had no problems because the /boot partition is in ext4. I then configured the VM as follows, running the command vm configure vm100:
loader="grub"
cpu="8"
memory="12G"
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="sparse-zvol"
uuid="the-uuid"
network0_mac="the-mac"
grub_run0="linux /vmlinuz-4.18.0-553.22.1.el8_10.x86_64 root=/dev/vda3"
grub_run1="initrd /initramfs-4.18.0-553.22.1.el8_10.x86_64.img"
All I needed to do is: pass to GRUB the name of the kernel and the initramfs. It is now possible to start the machine and do a test and connect to the console:
vm start vm100
vm console vm100
In theory, the machine should boot correctly. It will probably be necessary to change the network interface configurations - in my case, even keeping the virtio and the MAC address, it changed from ens18 to enp0s5 - but everything else should be fine.
At this point, if the source VM was turned off/not reachable or simply there is nothing new to synchronize, the migration is complete. Otherwise, it will be necessary to shut down the source VM, create a new snapshot, and transfer it. Let's start by shutting down the source VM and the destination one (vm stop vm100), roll back to the snapshot transferred to the destination server, create a new snapshot, and perform an incremental transfer:
zfs rollback zroot/VMs/vm100/disk0@Move01
On the source server:
zfs snapshot zfspool/vm-100-disk-0@Move02
zfs send -R -i zfspool/vm-100-disk-0@Move01 zfspool/vm-100-disk-0@Move02 | mbuffer -s 128k -m 512M | ssh root@destMachine "zfs receive -F zroot/VMs/vm100/disk0"
In this way, we have updated the destination VM and transferred only the differences, without altering the configuration of vm-bhyve.
Start the machine - if all goes well, the migration of this VM is finished.
Migrating Rocky Linux 9.4 VMs
For the Rocky Linux 9.4 VMs, the situation was more complex: the partitions were all - even /boot—in XFS. And, for some reason, the GRUB launched by bhyve cannot read from XFS. Therefore, I had to proceed differently.
I copied, using rsync, the /boot of the original VPS (in ZFS) to a directory on the FreeBSD server (specifically, on the VPS I ran the command: rsync -avhHPx --numeric-ids /boot root@FreeBSDHOST:/tmp/). In this way, I kept the files available.
I shut down the machine on the original server and copied its zvol to the new FreeBSD host using the same method as the others. At that point, I recreated the /boot partition of my VPS in ext3 - directly from the FreeBSD host:
pkg install fusefs-ext2
kldload fusefs
mkfs.ext3 /dev/zvol/zroot/VMs/vm104/disk0s1
fuse-ext2 -o force /dev/zvol/zroot/VMs/vm104/disk0s1 /mnt/
cd /mnt
rsync -avhHPx /tmp/boot/. .
umount /mnt
In this way, GRUB will be able to access the /boot partition to load the kernel and initramfs.
Here is the vm-bhyve configuration for this VM:
loader="grub"
cpu="8"
memory="12G"
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="sparse-zvol"
uuid="the-uuid"
network0_mac="the-mac"
grub_run0="linux /vmlinuz-5.14.0-427.37.1.el9_4.x86_64 root=/dev/vda3"
grub_run1="initrd /initramfs-5.14.0-427.37.1.el9_4.x86_64.img"
Launching the machine, however, it will hang during boot and, after a timeout, will ask for administrator credentials to enter a recovery shell. This is because the blkid of the /boot partition has changed, and the fstab of the VM still reports the data of the old XFS partition. In this case, I used the blkid command in the VM and copied the UUID of the new partition. Then modify the /etc/fstab file of the VM and put the new blkid, as well as changing "xfs" to "ext3". After a reboot, the system should start without problems, again with a network card to reconfigure.
This procedure worked correctly for all VMs. In this way, the storage remains on virtio-blk even if it would be optimal for the driver to be changed to nvme. To make this change, you will need to enter the VMs, create a file called /etc/dracut.conf.d/00-custom.conf:
add_drivers+=" nvme "
And regenerate the initramfs - in this way, the nvme driver will be supported at boot:
dracut --regenerate-all --force
It will now be enough to change the configurations of vm-bhyve—virtio-blk becomes nvme, and /dev/vda3 becomes /dev/nvme0n1p3, for example:
loader="grub"
cpu="8"
memory="12G"
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0"
disk0_dev="sparse-zvol"
uuid="the-uuid"
network0_mac="the-mac"
grub_run0="linux /vmlinuz-5.14.0-427.37.1.el9_4.x86_64 root=/dev/nvme0n1p3"
grub_run1="initrd /initramfs-5.14.0-427.37.1.el9_4.x86_64.img"
Migrating the OPNsense VM
For the VM with OPNsense, the procedure was even simpler. It was enough to create a VM of type freebsd-zvol and copy the disk as done for the others. In this case, I replicated the MAC address of the original virtualized network interface (Proxmox server) to ensure that the underlying FreeBSD recognizes it as the same. FreeBSD is less picky about these things.
The final vm-bhyve configuration file will be:
loader="bhyveload"
cpu="2"
memory="1G"
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="sparse-zvol"
uuid="my-uuid"
network0_mac="my-mac"
Configuring Automatic VM Startup
To ensure that the VMs all start at boot, just add them to /etc/rc.conf as follows, giving a 15-second delay between one VM and another:
[...]
vm_enable="YES"
vm_dir="zfs:zroot/VMs"
vm_list="vm100 vm101 [...] opnsense"
vm_delay="15"
[...]
Setting Up Backups
At this point, I configured external backups, every hour, similarly to how I described in a previous article. I also added a local snapshot every 15 minutes, always using zfs-autobackup. To do this, I created a new tag:
zfs set autobackup:localsnap=true zroot
Then I modified the /etc/crontab file, adding this line:
*/15 * * * * root /usr/local/bin/zfs-autobackup localsnap --keep-source 15min3h,1h1d > /dev/null 2>&1
That is, it will perform a snapshot every 15 minutes and keep them for 3 hours, then keep one per hour for a day. In this way, in case of quick recovery due to a problem/error, I won't need to transfer the entire dataset/VM from the backup.
Conclusion
The migration is complete: after changing the DNS, the client performed some tests, and everything works properly. The setup has been active for over a week, and there have been no issues. The performance is excellent; I did not perform tests compared to the previous setup since the hardware of the new FreeBSD host is more powerful and modern, so it wouldn't make sense.
Apart from the manager, no user was informed of the change, and in the last week, no reports have been received. The VMs are stable, and the host's load is very low.
The operation actually took less than two hours in total - most of the time has been consumed by the send/receive operations - and gave excellent results. The alternative would have been to install Proxmox on a new host and move the VMs - I probably would have saved a few minutes, but now I can use bhyve, as well as the simplicity and power of the underlying FreeBSD.