IT Notes - lxc

The Double-Edged Sword of Docker: Balancing Benefits and Risks

Stefano Marinelli — Mon, 22 Apr 2024 05:30:00 +0000

As a systems administrator, I am deeply concerned about the consequences of the current widespread adoption of technologies like Docker. Having been a proponent and early adopter of containerization for many years, I recognized its potential early on and have been advocating for its use in many of the Linux-based setups I manage.

Initially, this relieved me of some headaches. One recurring issue was dealing with developers requesting "exotic" setups-by exotic, I mean specific (sometimes multiple) versions of PHP on the same VPS, or unique combinations of PHP and MySQL (or MariaDB) that required adding external repositories of all sorts-creating future problems when one of these repositories ceases to exist or be updated, leaving us with an unstable, dangerous, or unupgradable system.

In many cases, I resolved these issues by partitioning components into FreeBSD jails (one jail per service, one for data, with bind mounts as needed-perfect efficiency, excellent upgradability and stability, maximum security). However, this wasn't always feasible. Sometimes, the explicit use of Linux was required, prompting the need for an alternative solution. In the past, I separated components using LXC, similar to FreeBSD jails, but then Docker arrived, and the approach changed.

At that point, the problem seemed solved: I just needed to provide a VPS with Docker, handle backups, data, monitoring, etc., but leave developers the freedom to include the specific versions of components they needed in their setups.

But...

Developers often make poor system administrators. And rightly so, because system administrators are often poor developers. However, this leads to a series of medium-term problems:

Continued use of outdated (or conversely, bleeding-edge and thus unstable) component versions in Dockerfiles, creating stability issues or, worse, security vulnerabilities.
A habitual approach to software crashes as if they were normal. Well-developed software should not crash but autonomously manage issues. When a crash is inevitable, it should indicate a situation so severe that it requires a system administrator's intervention. Instead, the world is full of unstable stacks that crash at the slightest exception, with the mentality, "the container will just restart." This, to me, is unacceptable.
Lack of optimization: I often hear, "I've maxed out the resources on MySQL, we need to scale up." But upon reviewing, I realize that there has been no tuning of its configuration. After some adjustments, the load often decreases by 90%, making it entirely manageable. Yet, we are in an era dominated by major cloud players whose goal is not to optimize our costs (as they claim) but to make us spend more, seemingly simplifying tasks with tools like Kubernetes (and autoscaling) but actually encouraging us to unnecessarily complicate our infrastructure and spend more. Using more resources while contradicting the ecological awareness that marks our times.
Lack of big-picture thinking: As system administrators managing the overall system, we always have a holistic view. Developers, rightly focused on their projects, often lack the depth of understanding to identify the bottleneck in the entire setup. A typical comment I hear is, "the site is slow, we need a more powerful server." In 90% of cases, this is unnecessary and would be completely ineffective. A misimplemented feature launching 50 concurrent long PHP processes wouldn't be solved by increasing from 4 to 8 cores. It would help, sure, but it wouldn't be a solution. Solving it by reducing the processes to two would change everything.
Lack of backup strategy: The average developer focuses on the reproducibility of their setup, maybe keeping a database dump (not always), but seldom addresses the issue of recovery time. I recently had a discussion with a colleague (who calls himself a DevOps) who told me he had "production database dumps, a .tar.gz of individual web app directories, and notes on how he set up that server." When I asked how many systems he managed, he said "over 100, on the same cluster." Asked about disaster recovery, he believed it was "impossible" for such a cluster to be unreachable for long (though the OVH Strasbourg fire should have taught us that nothing is impossible when data is concentrated in one place). Nevertheless, he thought he could restore operations in "about 2 hours per server"-thus, 100 servers would require 200 hours of work. For 99% of setups, these would be totally unacceptable timelines.

Today, we have hardware so powerful that it can handle unimaginable loads from just a few years ago. A bit of planning, optimization, and design can greatly reduce costs, and increase productivity, stability, and system reliability.

Thus, I remain in favor of solutions like Docker, but the turn the entire IT industry is taking towards such solutions worries me because it might improve some aspects but will worsen others. We are simply shifting the problem elsewhere.

In my view, there is no one-size-fits-all solution to any problem; each requires its own study and implementation.

The solution to all the problems we have known was one: 42. And we all know how that turned out.

How we are migrating (many of) our servers from Linux to FreeBSD - Part 3 - Proxmox to FreeBSD

Stefano Marinelli — Tue, 14 Mar 2023 13:00:00 +0000

In recent years, we've been migrating many of our servers from Linux to FreeBSD as part of our consolidation and optimization efforts. Specifically, we've been moving services that were previously deployed using Docker onto FreeBSD, and it has proven to be a great choice for handling workloads efficiently.

To this end, we've also been migrating many of our virtual machines (VMs) to FreeBSD, deploying services within FreeBSD jails. In some cases, these jails have even replaced entire VMs and run bare metal. Although we prefer to move to native FreeBSD whenever possible, sometimes it's not the best option for all the services we offer. As a result, one of our most critical physical servers has been left behind for years.

This server was a Proxmox server that we installed many years ago and updated to version 6.4. It hosted some critical services, but upgrading to Proxmox 7.x posed some challenges. In particular, some of the LXC containers required tweaks.

Unfortunately, this server was quite old, with only four physical disks and 64 GB of RAM. It was located in an OVH data center and had been running well until one of the disks started to malfunction once a week, on Sundays. This would trigger a RAID reconstruction that kept the system busy for about two days.

Despite my preference for simple setups, this server had been deployed gradually over many years, and everything was tied together. As a result, unraveling the system to resolve the issues was not a simple task. Sometimes the combination of simple things can make everything complex.

The Proxmox Server

The Proxmox server was configured as the central hub for various services, including primary DNS, web hosting, VOIP, and more. It featured several bridges, each with its own specific purpose, and was connected to a virtual machine running MikroTik CHR. This machine was responsible for consolidating all incoming VPNs from the MikroTik devices we managed, both ours and those belonging to our clients. Additionally, it provided a series of bridges to manage these devices and all server management VPNs and other services. The Proxmox server also housed several virtual machines running Linux, FreeBSD, OpenBSD, and NetBSD, as well as LXC containers.

Over the last two years, we've been migrating most of these virtual machines and containers to FreeBSD-based VMs, which feature their own specific jails. Consequently, most of the VMs we've had to move were BSD-based, while only five Linux VMs remained. The LXC containers hosted a range of services, including servers managed by Virtualmin, a large installation of Zimbra (which was hosted within an LXC container running CentOS 7), as well as some minor Alpine Linux-based machines. We located all these virtual machines and containers in a LAN created and managed by CHR. All public IPs were managed by CHR, which relied on NAT mappings to establish communication between them. CHR had thus become the heart of our system, and if it experienced any issues, it could potentially take down the entire system. Fortunately, it remained stable for years.

Migration - first steps

The first step I took was to install FreeBSD on the new server. Easy peasy. The next step was to find a way for the CHR to migrate to the new server (under bhyve) and continue to manage all the public IPs of the original server. The problem is that OVH, with its failover IPs, ties a specific MAC address to each individual IP address. Therefore, the only way was to create a bridge on the FreeBSD server (on the Proxmox server, I already had the bridge on the physical network card) and create an L2 tunnel between the two servers - I used OpenVPN with tap interfaces, specifically inserted into the bridges. I could have used other methods and techniques, but I wanted to experiment with a setup that could allow, if necessary, to bridge a larger number of physical and virtual servers even if the IPs are all mapped to a single server. OVH does not allow, in fact, the splitting of classes, so a move must be made for the entire class, not for a single IP address.

Initially, MikroTik CHR 7 did not boot on bhyve. In the end, I managed to make it work, but I had other problems, probably related to the MTU of the interfaces. So I thought about taking the opportunity to unbind the LXC containers and VMs from CHR and remove MikroTik from the setup. With RouterOS version 7, in fact, Wireguard-based VPNs are also supported, so within a few days, it was possible to update the few routers still on 6.x and recreate some VPNs using Wireguard. I mapped both the VMs and LXC containers directly to their respective public IPs, greatly simplifying the steps. Everything worked perfectly.

The next step was to test the first migrations, starting from the VMs already on FreeBSD. For simplicity, I created a new FreeBSD VM in bhyve and copied (via zfs-send and zfs-receive) the datasets related to BastilleBSD. All services are installed in jails managed by Bastille, so this was enough to have, in a short time, a new operating server equivalent to the previous one. At that point, I shut down the original server, connected the VM to the bridge linked to the tunnel (after modifying its MAC address), turned on the new FreeBSD VM (on bhyve), and everything started to work correctly - but from the new physical server.

One by one, I moved all the FreeBSD VMs. For Linux, NetBSD, and OpenBSD, I simply copied the images and pointed bhyve to them. Some small specific configuration on vm-bhyve and everything started to work correctly. Where possibile, I replaced the “virtio” with “nvme” as it performs much better on bhyve.

Migration - LXC containers to Virtual Machines

For LXC containers, I initially thought of creating an Alpine Linux virtual machine, installing LXD, and copying each individual container. It worked for some of them, but for others, I started to encounter strange issues, similar to those that would have required manual intervention to upgrade from Proxmox 6.x to 7.x. As is often the case with Linux-based solutions, compatibility is not always preserved between updates, so I would have had to fine-tune all the containers, which I didn't feel like doing. The containers had been created (at the time) to optimize RAM usage on the Proxmox machine, but to date, they have caused more problems than benefits. In some cases, certain processes got "stuck," making it impossible to "reboot" the LXC container, requiring the entire physical node to be rebooted. If they had been virtual machines, I could have given a "kill" command from the virtualizer (to the respective KVM process, in that case) and restarted it.

For greater compatibility and ease of future management, I decided to convert the LXC containers into actual VMs on bhyve. The process was simple:

Creating an empty VM with vm-bhyve and booting the VM with SystemRescueCD.
Creating destination partitions and file systems in the VM, then doing a complete rsync of the original LXC container.
Adjusting the fstab file, installing the kernel on the destination VM, and creating the initrd (some containers were already copies of VMs, so the kernel remained installed and updated, even though it wasn't being used. The initrd, on the other hand, did not include the nvme or virtio drivers, so I had to regenerate it anyway.)
Adjusting the bhyve vm configuration file, doing one last rsync after shutting down the services, shutting down the original LXC container, and starting the bhyve VM.

Everything worked correctly, so one by one, I moved all the containers. The largest one ended up on another physical node (also FreeBSD with bhyve) temporarily because the space on the new server was not sufficient to contain it. It didn't need to be on this server, so no problem.

One by one, the LXC containers started on the new server. Apart from some minor adjustments to the destination VMs (different network interface names, etc.), I didn't encounter any particular problems even after several days. Everything works perfectly.

At the very end, I re-created the MikroTik CHR VM. I’ll keep this setup separate for now, as strictly tied to eoip interfaces. This was the main reason why I haven’t performed the migration before. Things were too tied together and I had to untie everything, step by step.

…and then one of the Linux VMs started to freeze

Several Linux VMs are just the basis on which Docker runs. One of them (not even among the busiest) started, every 12/15 hours, to completely freeze. It stopped responding to ping, and it was impossible to give any type of command from the console. In a word: stuck.

Searching the web, I found some references to this problem and, observing the errors of an ssh session that was left connected (stuck, but still showing the last error), I found it to be a problem similar to the one described in this post, namely:

"watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [khugepaged:67]"

I tried various solutions such as changing the storage driver, the number of cores, the distribution (from Alpine to Debian), etc., but none of these operations solved the issue. I also noticed that the problem occurs with all Linux VMs, but only those with a recent kernel (> 5.10.x) freeze, while the others continue to work. The problem does not occur, however, with the *BSDs.

In the end, I:

Reduced the number of cores to 1 for the VMs that did not have a high load (some remained with multiple cores), hypothesising a problem with allocating cores that were too busy
Gave the command: "/usr/bin/echo 60 > /proc/sys/kernel/watchdog_thresh" to the VM.

The VM became stable, and I have not seen that error/warning on any other machine since. I will investigate further, but I believe it is a problem related to the Linux kernel, which, for some reason, generates a kernel panic if particular situations of CPU concurrency are generated.

The End…and a nice OOM!

After moving everything, I was finally able to migrate the entire class of OVH IPs from one physical server to another. The operation was quite quick, but in order to avoid problems, I notified all users and performed the operation on a Sunday and during off-peak hours. The whole process took about 10 minutes and there were no hitches of any kind.

For safety reasons, I kept the Proxmox machine active for a few more days, but there was no need to use it. However, after a couple of days, I encountered a problem: the largest VM, in some cases, was being "killed" because FreeBSD generated an OOM. I had never seen, from FreeBSD 13.0 onwards, any OOM related to "abuse" of RAM usage by ZFS, but in this case, it actually happened.

In the end, I understood that ZFS, on FreeBSD, is able to release memory, but not quickly enough to manage any "spikes" in individual VMs. In fact, the VMs do not know the situation of the physical host's RAM, so they will tend to occupy all the space allotted to them (even if only for caching). A sudden spike (i.e. if you create and launch a new VM) could cause a sudden increase in RAM usage by the bhyve process, and FreeBSD could be forced to kill it, even if part of the RAM is only ARC cache. While Proxmox supports HA (i.e., control over whether the VM is running), vm-bhyve only launches the VM (bhyve process). I should manage it with tools like monit, but for now, I preferred to simply set limits on ZFS RAM usage using "vfs.zfs.arc_max", and there have been no more problems.

Final considerations

The operation was long but linear. The most complex part was unraveling all the configurations related to MikroTik CHR and the VPNs linked to each individual LXC machine/container. Once everything was implemented on a dedicated VM, the operation was fairly straightforward.

The hardware specifications of the destination physical server are slightly better than the starting one, but the final performance of the setup has greatly improved. The VMs are very responsive (even those that were previously LXC containers running directly on bare metal) and, thanks to ZFS, I can make local snapshots every 5 minutes. In addition, every 10 minutes, I can copy (using the excellent zfs-autobackup) all the VMs and jails to other nodes both as a backup and as an immediate restart in case of disaster. I just need to map the IPs, and everything will start working very quickly. Proxmox also allows you to perform this type of operation with ZFS, but you still need to have Proxmox (in a compatible version) on the target machine. With the current setup, I only need any FreeBSD node that supports bhyve.

Proxmox is an excellent tool, well-developed, open-source, efficient, and stable. We manage many installations, including complex ones (ceph clusters, etc.), and it has never let us down. However, not all tools are ideal for all situations, and for setups like the one described, the new configuration based on FreeBSD has shown significantly interesting performance and greater management and maintenance granularity.

Virtualizing on vm-bhyve is not complex, but it is certainly not comparable, at the current state, to the simplicity of using a clean and complete interface like Proxmox's. A complete HA system is still missing (sure, it's achievable manually, but...), as well as complete management web interface. However, for knowledgeable users, it is undoubtedly a powerful tool that allows you to have excellent FreeBSD as a base. I'm totally satisfied with my migration and the result is far better than I expected.

Why we're migrating (many of) our servers from Linux to FreeBSD

Stefano Marinelli — Mon, 24 Jan 2022 06:55:00 +0000

More about this in the article I wrote to accompany my talk at EuroBSDCon 2024.

Please note: This article was (unexpectedly) quite popular and many interesting opinions came out from the Hacker News thread that it generated. If you're interested, have a look here: https://news.ycombinator.com/item?id=30057549

There's an Italian version of this article here.

I've been a Linux (or GNU/Linux, for the purists) user since 1996. I've been a FreeBSD user since 2002. I have always successfully used both operating systems, each for specific purposes. I have found, on average, BSD systems to be more stable than their Linux equivalents. By stability, I don't mean uptime (too much uptime means too few kernel security updates, which is wrong). I mean that things work as they should, that they don't "break" from one update to the next, and that you don't have to revise everything because of a missing or modified basic command.

I've always been for development and innovation as long as it doesn't (necessarily, automatically and unreasonably) break everything that is already in place. And the road that the various Linux distributions are taking seems to be that of modifying things that work just for the sake of it or to follow the diktats of the Kernel and those who manage it - but not only.

Some time ago we started a complex, continuous and not always linear operation, that is to migrate, where possible, most of the servers (ours and of our customers) from Linux to FreeBSD.

Why FreeBSD?

There are many alternative operating systems to Linux and the *BSD family is varied and complete. FreeBSD, in my opinion, today is the "all rounder" system par excellence, i.e. well refined and suitable both for use on large servers and small embedded systems. The other BSDs have strengths that, in some fields, make them particularly suitable but FreeBSD, in my humble opinion, is suitable (almost) for every purpose.

So back to the main topic of this article, why am I migrating many of the servers we manage to FreeBSD? The reasons are many, I will list some of them with corresponding explanations.

The system is consistent - kernel and userland are created and managed by the same team

One of the fundamental problems with Linux is that (we shall remember) it is a kernel, everything else is created by different people/companies. On more than one occasion Linus Torvalds as well as other leading Linux kernel developers have remarked that they care about the development of the kernel itself, not how users will use it. In the technical decisions, therefore, they don't take into account what is the real use of the systems but that the kernel will go its own path. This is a good thing, as the development of the Linux kernel is not "held back" by the struggle between distributions and software solutions, but at the same time it is also a disadvantage. In FreeBSD, the kernel and its userland (i.e. all the components of the base operating system) are developed by the same team and there is, therefore, a strong cohesion between the parties. In many Linux distributions it was necessary to "deprecate" ifconfig in favor of ip because new developments in the kernel were no longer supported by ifconfig, without breaking compatibility with other (previous) kernel versions or having functions (on the same network interface) managed by different tools. In FreeBSD, with each release of the operating system, there are both kernel and userland updates, so these changes are consistently incorporated and documented, making the tools compatible with their kernel-side updates.

In other words, in FreeBSD there is no need to " revolutionise" everything every few years and changes are made primarily in the form of additions that can enrich (and not break) each update. If a modification was to change the way it interacts with network devices, ifconfig would be modified to take advantage of that and remain compatible with the "old" syntax. In the long-term, this kind of approach is definitely appreciated by system administrators who find themselves with a linear, consistent, and always well-documented update path.

FreeBSD development is (still) driven by technical interests, not strictly commercial ones.

Linux and related distributions now have contributions from many companies, many of which (e.g. Red Hat) push (justifiably) in the direction of what is convenient for them, their products, and their services. Being big contributors to the project they have a big clout so, indeed, their solutions often become de-facto standards. Consider systemd - was there really a need for such a system? While it brought some advantages, it added some complexity to an otherwise extremely simple and functional system. It remains divisive to this day, with many asking, "but was it really necessary? Did the advantages it brought balance the disadvantages?". 70 binaries just for initialising and logging and a million and a half lines of code just for that? But Red Hat threw the rock...and many followed along. Because sometimes it's nice to follow the trend, the hype of a specific solution.

Even FreeBSD has big companies behind it, collaborating in a more or less direct way. The license is more permissive, so not everyone who uses it commercially contributes to it, but knowing that FreeBSD is at the base of Netflix CDNs, Whatsapp servers (waiting for Meta to replace them, for internal coherence reasons, with Linux servers), Sony Playstations and, in part, macOS, iOS, iPadOS, etc. surely gives confidence on its level. These realities, however, do not have enough clout to drive the development of the core team.

Linux has Docker, Podman, lxc, lxd, etc. but... FreeBSD has jails!

FreeBSD jails are very powerful tools for jailing and separating services. There is controversy about Docker not running on FreeBSD but I believe (like many others) that FreeBSD has a more powerful tool. Jails are older and more mature - and by far - than any containerization solution on Linux. Jails are efficient and are well integrated throughout the operating system. All major commands (ps, kill, top, etc.) are able to display jail information as well. There are many management tools but, in fact, they all do the same thing: they interact with FreeBSD base and create custom configuration files. Personally I'm very comfortable with BastilleBSD but there are a lot of very good tools as well as a sufficiently simple manual management. When I need Docker I launch a Linux machine - often Alpine, which I think is a great minimalist distribution, or Debian. But I'm moving a lot of services from Docker to a dedicated jail on FreeBSD. Docker containers are a great tool for rapid (and consistent) software deployment, but it's not all fun and games. Containers, for example, rely on images that sometimes age and are no longer updated. This is a security issue that should not be overlooked.

Linux has ext4, xfs, btrfs... (and zfs, but with some manual intervention). FreeBSD has UFS2 and ZFS

UFS2 is still a very good and efficient file system and, when configured to use softupdates, capable of performing live snapshots of the file system. This is great for backups. Ext4 and XFS do not support snapshots except through external tools (like DattoBD or snapshots through the volume manager). This works, of course, but it is not native. Btrfs is great in its intentions but still not as stable as it should be after all these years of development. FreeBSD supports ZFS natively in the base system and this brings many advantages: separation of datasets for jails, as well as Boot Environments, to make snapshots before upgrades/changes and to be able to boot (from bootloader) even on a different BE, etc.

The FreeBSD boot procedure is cleaner and simpler.

Linux has always used excellent tools such as grub, lilo (now outdated), etc.. FreeBSD has always used a very linear and consistent boot system, with its own bootloader and dedicated boot partition. Whether on mbr, gpt, etc. things are very similar and consistent. I've never had a problem getting a FreeBSD system to boot after a move or recovery from backup. On Linux, however, grub has sometimes given me problems, even after a simple kernel security update.

FreeBSD's network stack is (still) superior to Linux's - and, often, so is its performance.

Meta has been trying to bring the performance of the Linux network stack up to the level of FreeBSD's for years. Many will ask why, then, not move services to FreeBSD. Large companies with huge datacenters can't change solutions overnight, and their engineers, at any level, are Linux experts. They have invested heavily in btrfs, in Linux, in their specifics. Clearly, upon acquiring Whatsapp, they preferred to migrate the "few" Whatsapp servers to Linux and move them to their datacenters. Regarding the real system performance (i.e. disregarding benchmarks, useful only up to a certain point), FreeBSD shines, especially under high load conditions. Where Linux starts to gasp (ex: waiting for I/O) with 100% CPU, FreeBSD has lower processor load and room for more stuff. In the real world (of my servers and load types), I sometimes experienced severe system slowdowns due to high I/O, even if the data to be processed was not read/write dependent. On FreeBSD this does not happen, and if something is blocking, it blocks THAT operation, not the rest of the system. When performing backups or other important operations this factor becomes extremely important to ensure proper (and stable) system performance.

Straightforward system performance analysis

FreeBSD, in the base system, has all the tools to analyze possible problems and system loads. "vmstat" , in a single line, tells me if the machine is struggling for CPU, for I/O or for Ram. "gstat -a" shows me how much, disk by disk, partition by partition, the storage is active, also in percentage with reference to its performance. "top", then, also has support for figuring out, process by process, how much I/O is being used ("m" option). On Linux, to get the same results, you have to install specific applications, different from distribution to distribution.

Bhyve is more incomplete (but more efficient) than KVM

For my purposes, Bhyve is a great virtualization tool. KVM is definitely more complete but since I don't have any special or specific needs not covered by Bhyve on FreeBSD, I found (on average) better performance with this combination. On FreeBSD, however, KSM is missing which, in some cases, can be very useful.

Will I abandon Linux for FreeBSD? Obviously not, just as I haven't for the last 20 years. Both have their uses, their space, their strengths. But if up to now I have had 80% Linux and 20% FreeBSD, the perspective is to invert the percentages of use and, where possible, directly implement solutions based on FreeBSD.

NOTE: this article has been translated from its Italian original version. Even if it's been reviewed and adapted, there might be some errors.

Efficient backup of lxc containers in Proxmox - ZFS

Stefano Marinelli — Thu, 20 Jan 2022 07:05:00 +0000

I've already written about some of my backup strategies in Proxmox. Proxmox Backup Server is an option, but it's not always the best option, especially if you're using lxc containers.

LVM and Ceph RBD baked containers have already been covered in another post, but one of the (many) great options, if you use Proxmox, is ZFS. I extensively use ZFS both on FreeBSD and Linux (and always wished that BTRFS could reach the same level of reliability).

When I don't need a networked file system (ceph) or want to use LVM, I tend to install Proxmox VMs and lxc containers on ZFS.. Let's now focus on backing up lxc containers.

Proxmox uses ZFS datasets for lxc containers' storage so you'll find all your files on /poolname/subvol-x-disk-y . We can easily backup as we've done in my previous article, we just need a different method for taking snapshots of all those datasets.

ZFS datasets have a hidden .zfs directory that contains all the snapshots that currently exist of that specific dataset. ls won't show it, but you can cd and it will be working.

Of course we can use native zfs send/receive or a tool like zfs-autobackup, which I use daily for local snapshots and remote replication, but we want to save the files, not the zfs dataset, so we can be able to backup to a different file system. Any file system. So we will be using borg.

Let's suppose our ZFS pool is named "proxzfs". Here is a suggested script. Of course, this is my script, it works for me and I'm not responsible if it doesn't work for you/destroys all your data/eats your server/etc.

#!/bin/bash

/usr/sbin/zfs snapshot -r proxzfs@forborg

REPOSITORY=yourpath/server/whatever:borgrepository/
TAG=mytag
borg create -v --stats --compression lz4 --progress    \
   $REPOSITORY::$TAG'-{now:%Y-%m-%dT%H:%M:%S}'          \
   /proxzfs/*/.zfs/snapshot/forborg/  \
   --exclude '*subvolYouMayWantToExclude-disk-0*'

/usr/sbin/zfs destroy -vrR proxzfs@forborg

borg prune -v $REPOSITORY --stats --prefix $TAG'-' \
   --keep-daily=31 --keep-weekly=4 --keep-monthly=12

This small script will create a @forborg snapshot for any dataset it will find under "proxzfs", then will fire up borg and ask it to traverse the forborg snapshots automatically mounted inside the .zfs directory of any dataset.

After that, it will destroy the 'forborg' snapshots and execute a borg prune_._ That will delete the old backups, according to the policy you have established. This step can be avoided here but I prefer to perform it after a backup so my repository is always consistent with my policy.

Alpine Linux and LXD perfect setup on BTRFS file system

Stefano Marinelli — Wed, 03 Nov 2021 10:54:10 +0000

Last update: 01 November 2022 - Alpine 3.16 has lxd packages in contrib

Lxc is a great technology. I’ve been using it since 2012 and I’ve always appreciated its lightweight approach and efficiency. Nowadays KVM and hardware virtualization extensions have reached a very good level of performance. While a KVM VM allows to run another kernel and another OS, it ties specific resources to the VM and this isn’t always the best solution. Lxc uses the host’s kernel and shares its ram so the host machine runs lxc container’s processes as local processes - no slowdown, no resource wasting.

Alpine Linux is a great distribution. It’s minimal, reliable, secure and easy to deploy. It’s great for containers but it’s also great for low footprint servers, like hypervisors. It’s good as Xen Dom0, it’s good as KVM host and - no surprise - is great as Lxc host too.

Alpine Linux is great for low energy devices (Raspberry Pis, etc.), too, and has an efficient and reliable out-of-the-box read-only file system support. But I’ll write more about it in another post.

Lxd is a great solution. It uses lxc but simplifies a lot of operations like new container creation, profile management, automatic snapshots, clustering, etc. The “official” best option to use lxd is installing Ubuntu, Snap and so on. I don’t want to use Ubuntu to install Snap. I don’t want to use Snap just to install lxd. That’s why I’m choosing to use lxd with Alpine.

Long story short: this video shows all the steps described below:

Installing Alpine Linux on BTRFS Root File System

Alpine Linux installer supports btrfs as rootfs destination but the setup-alpine command doesn’t allow you to select it: it will partition your disk and create a ext4 standard root file system.

So let’s start with the installation but, then, we’ll interrupt it at disk selection:

Start with setup-alpine and set network, hostname, password, etc. Stop at:

Available disks are:
sda (20.5 GB QEMU     QEMU HARDDISK   )
Which disk(s) would you like to use? (or '?' for help or 'none') [none]

Press ctrl+C and interrupt the script.

Now we can install Alpine Linux on btrfs root system:

test:~# export ROOTFS=btrfs

And, then:

test:~# setup-disk /dev/sda

 The following disk is selected:
 sda    (20.5 GB QEMU     QEMU HARDDISK   )
 How would you like to use it? ('sys', 'data', 'lvm' or '?' for help) [?] sys
 WARNING: The following disk(s) will be erased:
 sda    (20.5 GB QEMU     QEMU HARDDISK   )
 WARNING: Erase the above disk(s) and continue? (y/n) [n] y
 Creating file systems...
 btrfs-progs v5.10
 See http://btrfs.wiki.kernel.org for more information.

 Label:              (null)
 UUID:               6ace888a-2f36-41e8-86b1-3e4214b9e34e
 Node size:          16384
 Sector size:        4096
 Filesystem size:    15.17GiB
 Block group profiles:
 Data:             single            8.00MiB
 Metadata:         DUP             256.00MiB
System:           DUP               8.00MiB
 SSD detected:       no
 Incompat features:  extref, skinny-metadata
 Runtime features:
 Checksum:           crc32c
 Number of devices:  1
 Devices:
 ID        SIZE  PATH
 1    15.17GiB  /dev/sda3

 Installing system on /dev/sda3:
 /mnt/boot is device /dev/sda1

Installation is complete. Now eject the installation media and reboot.

After rebooting, let’s check that root file system is on btrfs:

 test:~# cat /etc/fstab
 UUID=aacbfd28-df47-4945-9c0c-279faef361a0  /   btrfs   rw,relatime,space_cache,subvolid=5,subvol=/ 0 1

Ok, it is. Generally speaking, I prefer to use compression on btrfs. compress=lzo is enough, so you can add it on your fstab.

Now, let’s install lxd.

Edit your /etc/apk/repositories and let it be like this:

http://dl-cdn.alpinelinux.org/alpine/v3.16/main
http://dl-cdn.alpinelinux.org/alpine/v3.16/community

Let’s follow the tutorial I wrote on Alpine Linux wiki and let’s install lxc, lxd, lxcfs and dbus (not needed but recommended to avoid problems with some containers).

After rebooting, you have a fully functional lxd-enabled Alpine Linux installation.

Let’s now configure lxd:

 test:~# lxd init
 Would you like to use LXD clustering? (yes/no) [default=no]:
 Do you want to configure a new storage pool? (yes/no) [default=yes]:
 Name of the new storage pool [default=default]:
 Name of the storage backend to use (btrfs, dir) [default=btrfs]:
 Would you like to create a new btrfs subvolume under /var/lib/lxd? (yes/no) [default=yes]:
 Would you like to connect to a MAAS server? (yes/no) [default=no]:
 Would you like to create a new local network bridge? (yes/no) [default=yes]:
 What should the new bridge be called? [default=lxdbr0]:
 What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
 What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
 Would you like the LXD server to be available over the network? (yes/no) [default=no]:
 Would you like stale cached images to be updated automatically? (yes/no) [default=yes]
 Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:

As you can see, lxd automatically detected that /var/lib/lxd is on a btrfs file system and creates a sub volume.

Now we can launch our first container:

test:~# lxc launch images:alpine/3.16 t1
Creating t1
Starting t1

Ok, done. Let’s check if it’s working:

test:~# lxc info t1
Name: t1
Status: RUNNING
Type: container
Architecture: x86_64
PID: 2648
Created: 2021/11/03 08:55 UTC
Last Used: 2021/11/03 08:55 UTC

Resources:
Processes: 5
CPU usage:
CPU usage (in seconds): 0
Memory usage:
Memory (current): 3.38MiB
Memory (peak): 5.04MiB
Network usage:
eth0:
Type: broadcast
State: UP
Host interface: vethdc207601
MAC address: 00:16:3e:50:d1:ef
MTU: 1500
Bytes received: 3.76kB
Bytes sent: 1.79kB
Packets received: 26
Packets sent: 16
IP addresses:
inet:  10.189.248.21/24 (global)
inet6: fd42:34cb:9408:433c:216:3eff:fe50:d1ef/64 (global)
inet6: fe80::216:3eff:fe50:d1ef/64 (link)
lo:
Type: loopback
State: UP
MTU: 65536
Bytes received: 0B
Bytes sent: 0B
Packets received: 0
Packets sent: 0
IP addresses:
inet:  127.0.0.1/8 (local)
inet6: ::1/128 (local)

It’s running. Everything is ok. For more complex lxc setup (bridges, profiles, etc.) you can follow the excellent lxd documentation.

This kind of setup is easy and lightweight, easy to backup (a new article will follow) and reliable.

Efficient backup of lxc containers in Proxmox

Stefano Marinelli — Tue, 06 Oct 2020 08:22:44 +0000

Sometimes an lxc container can be a better alternative than a KVM virtual machine. It has a smaller overhead, easier and more efficient resource management and lower impact on the physical machine.

You get (potentially) less isolation from the physical machine and you are forced to use the host kernel. Moreover, you cannot run different a different OS.

Proxmox has a great support of lxc containers and they're often a good choice. I've recently converted some VMs to lxc containers and they're faster, lighter and the Ram occupation is quite lower.

There are two major drawbacks: you cannot migrate a running lxc container (it will be shut down, moved and restarted - if using a shared storage, it is often a matter of seconds) and backups to Proxmox Backup Server are slower than a VM's backup.

This is because a VM can monitor, thanks to KVM dirty bitmaps, its disk operations to know in advance which blocks should be checked and copied. When dealing with lxc containers it can't be done, so every backup will need to check every single file - this proved to be efficient in terms of space and deduplication, but quite slow in terms of time. One of my VMs (750 GB stored but with rare disk access) took just a few minutes to be backed up with dirty bitmaps, but more or less 2 hours to backup since it's been transformed to a lxc container.

borg backup proved to be quite efficient, so I tried to find a way to backup those containers performing a snapshot, backing up the snapshot and, then, releasing it.

We cannot use dattobd on the container, so we must snapshot and backup from the host. This has one big advantage: we don't need the container to even know its backup location as every operation will be done by the Proxmox host.

My containers generally are stored in Ceph rbd volumes or LVM thin pools.

I've created a script to deal with both those situations - and it doesn't depend on Proxmox, so it also works if you don't use it.

First of all, create a user and a borg repository on a remote machine and allow the Proxmox host to login via ssh using a key. Then, create a directory (/tobackup in the following scripts, on the host) where the snapshots will be mounted.

In the following example/scripts, no encryption will be used for the backup storage. If you want to encrypt your backup, you will have to perform small modifications.

Here's my rbd_borg_bck.sh :

#!/bin/bash
set -e

export SOURCE_DISK=$1
export USERNAME=$2
export HOST=$3

PATH="/usr/local/jdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11:/usr/pkg/bin:/usr/pkg/sbin"
export PATH

export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes;

echo "Creating and mapping snapshot..."
rbd snap create $SOURCE_DISK@tobackup && DDISK=`rbd map $SOURCE_DISK@tobackup`

echo "Creating backup directory..."
mkdir -p /tobackup/$DDISK

echo "Mounting snapshot..."
mount -o noload $DDISK /tobackup/$DDISK

echo "DOING BACKUP..."

REPOSITORY=$USERNAME@$HOST:$USERNAME/
TAG=daily

borg create -v --stats --progress --compression zlib,9                           \
    $REPOSITORY::$TAG'-{now:%Y-%m-%dT%H:%M:%S}'          \
    /tobackup/$DDISK/                                       \
    --exclude '*/home/*/.cache*'                  \
    --exclude '*/home/*/.local*'                  \
    --exclude '*/home/*/.pki*'                    \
    --exclude '*/home/*/Virtualbox VMs*'          \
    --exclude '*/home/*/.vagrant.d*'              \
    --exclude '*/root/.cache*'                    \
    --exclude '*/var/swap*'

sleep 3s;

echo "Unmounting snapshot..."
umount /tobackup/$DDISK

sleep 3s;

echo "Unmapping snapshot..."
rbd unmap $DDISK

echo "Remove snapshot..."
rbd snap remove $SOURCE_DISK@tobackup

echo "Pruning..."
borg prune -v $REPOSITORY --stats --prefix $TAG'-' \
    --keep-daily=14 --keep-weekly=8 --keep-monthly=12

echo "Done!"

For example, to backup a disk stored in pool "vmrbdpool" named 'vm-101-disk-0' via ssh to server backupserver.mydomain.org, username on the server (and repository name, inside that user's home directory, called server01), you can launch the script this way:

/usr/local/bin/rbd_borg_bck.sh vmrbdpool/vm-101-disk-0 server01 backupserver.mydomain.org

This can also be scheduled and launched via cron (on the Proxmox host) or any other way.

For the LVM thin pools, the script is a bit different. Here's my lvm_thin_borg_bck.sh

#!/bin/bash
set -e

export SOURCE_DISK=$1
export USERNAME=$2
export HOST=$3
export LVM=$4
export DDISK=$4/$2-borgsnap

PATH="/usr/local/jdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11:/usr/pkg/bin:/usr/pkg/sbin"
export PATH

export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes;

echo "Creating and mapping snapshot..."
lvcreate -n $2-borgsnap -s $1
lvchange -ay -Ky $DDISK


echo "Creating backup directory..."
mkdir -p /tobackup/$DDISK

echo "Mounting snapshot..."
mount -o ro /dev/$DDISK /tobackup/$DDISK

echo "DOING BACKUP..."

REPOSITORY=$USERNAME@$HOST:$USERNAME/
TAG=daily

borg create -v --stats --progress --compression zlib,9                           \
    $REPOSITORY::$TAG'-{now:%Y-%m-%dT%H:%M:%S}'          \
    /tobackup/$DDISK/                                       \
    --exclude '*/home/*/.cache*'                  \
    --exclude '*/home/*/.local*'                  \
    --exclude '*/home/*/.pki*'                    \
    --exclude '*/home/*/Virtualbox VMs*'          \
    --exclude '*/home/*/.vagrant.d*'              \
    --exclude '*/root/.cache*'                    

sleep 3s;

echo "Unmounting snapshot..."
umount /tobackup/$DDISK

sleep 3s;

echo "Remove snapshot..."
yes | lvremove $DDISK

echo "Pruning..."
borg prune -v $REPOSITORY --stats --prefix $TAG'-' \
    --keep-daily=14 --keep-weekly=8 --keep-monthly=12

echo "Done!"

In this example, we want to backup a disk "/dev/prox1/vm-120-disk-0", username on the server is server01 (same repository name), server is backupserver.mydomain.org and, as last argument, "prox1" which is the lvm pool.

/usr/local/bin/lvm_thin_borg_bck.sh /dev/prox1/vm-120-disk-0 server01 backupserver.mydomain.org prox1

Borg is efficient and fast and, thanks to this kind of setup, I can create a backup of the 750 GB container in 2 minutes.

Thanks to this kind of setup, backups can be done in a consistent and efficient way.