Creating a Mikrotik CHR (RouterOS 7) bhyve VM in FreeBSD -

While I love FreeBSD-based router solutions (OPNsense, PFsense, etc.), I also appreciate and use MikroTik devices and software. They produce and sell reasonably-priced, efficient hardware, and have implemented some interesting proprietary solutions (EoIP, etc.). Additionally, they’re a European company. From the smallest Wi-Fi router to the largest enterprise routing platform, they employ the same software with (almost) the same features.

That’s why I’m also implementing virtualized MikroTik CHR solutions - they’re lightweight, efficient, and can handle a significant amount of traffic with minimal resource overhead.

Our FreeBSD hypervisors run vm-bhyve as a lightweight, efficient, and intelligent management tool for VMs.

Even though MikroTik states that bhyve is not supported (as they believe it’s merely a paravirtualization software), CHR based on RouterOS 6 works flawlessly, following the hints provided by the vm-bhyve documentation.

Unfortunately, things change when dealing with RouterOS 7. It doesn’t boot at all, and if you upgrade from RoS 6 to RoS 7, it ceases to function entirely.

It seems that the MikroTik CHR image based on RouterOS 7 has an unusual partition table, somewhere between MBR and GPT. For this reason, Proxmox (KVM) doesn’t experience any issues, while bhyve seems unable to properly boot the VM. I conducted several tests, but the best solution was suggested by user kriszos on the MikroTik forum (even if kriszos was dealing with Hyper-V).

The script should be used on a machine with the following tools installed: gdisk, wget, unzip, qemu-img, qemu-nbd, and rsync:

#!/bin/bash
wget --no-check-certificate https://download.mikrotik.com/routeros/7.8/chr-7.8.img.zip -O /tmp/chr.img.zip
unzip -p /tmp/chr.img.zip > /tmp/chr.img
rm -rf  chr.qcow2
qemu-img convert -f raw -O qcow2 /tmp/chr.img chr.qcow2
rm -rf /tmp/chr.im*
modprobe nbd
qemu-nbd -c /dev/nbd0 chr.qcow2
rm -rf /tmp/tmp*
mkdir /tmp/tmpmount/
mkdir /tmp/tmpefipart/
mount /dev/nbd0p1 /tmp/tmpmount/
rsync -a /tmp/tmpmount/ /tmp/tmpefipart/
umount /dev/nbd0p1
mkfs -t fat /dev/nbd0p1
mount /dev/nbd0p1 /tmp/tmpmount/
rsync -a /tmp/tmpefipart/ /tmp/tmpmount/
umount /dev/nbd0p1
rm -rf /tmp/tmp*
(
echo 2 # use GPT
echo t # change partition code
echo 1 # select first partition
echo 8300 # change code to Linux filesystem 8300
echo r # Recovery/transformation
echo h # Hybrid MBR
echo 1 2 # partitions added to the hybrid MBR
echo n # Place EFI GPT (0xEE) partition first in MBR (good for GRUB)? (Y/N)
echo   # Enter an MBR hex code (default 83)
echo y # Set the bootable flag? (Y/N)
echo   # Enter an MBR hex code (default 83)
echo n # Set the bootable flag? (Y/N)
echo n # Unused partition space(s) found. Use one to protect more partitions? (Y/N)
echo w # write changes to disk
echo y # confirm
) | gdisk /dev/nbd0
qemu-nbd -d /dev/nbd0
echo "script finished, created file chr.qcow2"

I used the script on a machine with Proxmox installed, and it worked correctly. Once the relevant image was obtained, it was converted to raw format to be fed to bhyve:

qemu-img convert -f qcow2 -O raw chr.qcow2 chr.img

Now that we have the correct image available, here’s an example of a VM configuration managed by vm-bhyve:

loader="uefi"
graphics="no"
cpu="2"
memory=“128M"
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name=“chr.img”
uuid=“cafecafe-cafe-cafe-cafe-cafecafecafe”
network0_mac=“ca:fe:ca:fe:ca:fe”

See also