IT Notes - btrfs

Migrating from an Old Linux Server to a New FreeBSD Machine

Stefano Marinelli — Wed, 25 Oct 2023 16:39:37 +0000

Preamble: I believe it's time to bid farewell to this venerable Linux server.

TL;DR: The article chronicles the journey of transitioning from an outdated Linux server, running for 1690 days without updates, to a modern FreeBSD machine. This migration involved using tools like mfsBSD, BastilleBSD, Borg Backup, and bhyve. Despite initial hesitations due to the Linux server's impeccable performance, the transition was smooth, resulting in improved manageability and efficiency. The piece emphasizes the importance of regular system updates and anticipates revisiting the topic in the future with new uptime achievements and updates.

This server loyally served for years as a secondary backup server while also providing a few minor services to users. As it often happens, it remained in operation, neglected and without updates for years. Stable operating systems have the "flaw" of being forgotten, giving the false impression that they don't need maintenance or updates. This machine continued its service without oversight for years. When approached for a service request (not due to malfunctions), I advised the client to upgrade the whole system. A mere update would not suffice, so I suggested starting afresh on new hardware with FreeBSD as the primary OS.

The client was understandably hesitant given the uptime stats:

08:58:43 up 1690 days, 21:32, 4 users, load average: 9.57, 10.15, 8.76

Not a single error, not a single hiccup. From his perspective, a similar setup to what was installed many years ago and still working flawlessly was preferred. Nevertheless, he trusted my expertise and let me proceed.

This server had a plethora of duties, among which:

One of the pivotal tasks was running Proxmox Backup Server via Docker. Proxmox Backup Server requires Debian, but this server was running on Ubuntu 16.04 (previously upgraded from Ubuntu 14.04 – yes, ancient!). Hence, Proxmox Backup Server was still at version 1.x.
Another critical function was storing backups made through BorgBackup on its file system. The /home directory used a mirrored btrfs file system, and each backed-up server had its user on this system. Clients could backup (using a push method) only via VPN and only during specific windows when the server permitted (by adding specific firewall rules via Jenkins. Jenkins also managed connection protocols, snapshots, backups, etc.).
Among the lesser tasks, the server ran a few Docker containers with HandBrake on various presets. The client processed video conversions by uploading the original files via sftp, and after some hours, fetched the converted files from the destination directory. This will not be replicated on the new FreeBSD server since they now handle this operation locally on their high-performance MacBook Pro with Apple Silicon. However, a future restoration isn't off the table.

The first thing I did was install FreeBSD on the new hardware. Given that it's a physical server on Hetzner (an auction pick due to disk space needs over power), and FreeBSD wasn't an option, I used mfsBSD.

After booting the physical server in Linux rescue, I copied mfsBSD onto the disks using dd and restarted. On boot, I SSHed into mfsBSD and executed the installation using bsdinstall-a robust and efficient method.

I set up a raidz1 with all four 6 TB disks, resulting in a final storage space of 21.8T, ample for now without the video files.

To ensure continuity, I kept a setup similar to the old one. The clients would essentially continue with their usual backup procedure without necessitating drastic changes to backup scripts. To avoid storing these backups directly in the physical machine's /home and to leave the door open for future services, I installed BastilleBSD and began setting up several jails. I replaced the old Linux machine's behavior with a VNET FreeBSD jail. In past scenarios, I've created Linux jails (thanks to BastilleBSD) and transferred the old server into the jail using rsync, making minor configuration tweaks. While this usually works, it doesn't address the underlying issue of an obsolete setup. Given the opportunity, I opted for a modern toolset.

Thus, I copied every home directory (along with their historic backups) in its entirety, installed BorgBackup, and re-established the VPN. With a VNET jail, I can craft networking devices and fine-tune configurations. After recreating user accounts, inputting the various SSH authorized_keys, and checking all clients, I set up a snapshot plan on the host. This ensures that if a client is compromised with the potential (however remote) for breach and backup deletion, a ZFS snapshot of the entire jail remains available.

As mentioned, one of the core tools on the old server was Proxmox Backup Server. It's not natively installable on FreeBSD, necessitating a VM. Enter the fantastic bhyve, supported by vm-bhyve.

However, one issue arose: backups would consume vast amounts of space, and I wanted to avoid housing an enormous disk image (or a zvol) with both VMs and backups. So, I opted for a slightly less performant but more flexible solution: installing Debian 12 and Proxmox Backup Server on the VM while placing backups on a separate ZFS dataset on the physical machine, exported via NFS and mounted on the VM.

Given that the physical server has an internal bridge "vm-public" with IP 192.168.124.1 and the VM is at 192.168.124.2, I just created a dataset named zroot/PBS and added the following line to /etc/exports:

/zroot/PBS -alldirs -maproot=root -network 192.168.124.2/32

To enable NFS, insert into /etc/rc.conf:

rpcbind_enable="YES" 
nfs_server_enable="YES" 
mountd_flags="-r" 
rpc_lockd_enable="YES"

Within the VM, create /PBS and include in /etc/fstab:

192.168.124.1:/zroot/PBS /PBS nfs rw,async,soft,intr,noexec 0 0

After Proxmox Backup Server's installation, simply set up the datastores in /PBS/, and they'll directly store on the physical machine's ZFS dataset.

For firewall configurations, I exposed port 8007, redirecting it towards the VM, and everything started functioning smoothly. I then set a Proxmox Backup Server replica from the old to the new server. After completion, I changed the Proxmox Backup Server IP on all Proxmox hosts to point to the new server. Smooth sailing.

The old Ubuntu server also managed other minor services, which have become obsolete and weren't replicated.

The transition was seamless, the client is pleased, and I'm content since each service is now neatly segregated into its jail or VM. The machine's load is minimal, which might pave the way for other tasks, via VPN. Everything now rests on ZFS, and the icing on the cake: I made the client promise not to reach another 1690 days of uptime but to timely update as required.

I'm not entirely convinced the promise will hold-meaning, in a few years, I might yet be discussing this "new" server, highlighting another impressive uptime and another upgrade journey.

Why we're migrating (many of) our servers from Linux to FreeBSD

Stefano Marinelli — Mon, 24 Jan 2022 06:55:00 +0000

More about this in the article I wrote to accompany my talk at EuroBSDCon 2024.

Please note: This article was (unexpectedly) quite popular and many interesting opinions came out from the Hacker News thread that it generated. If you're interested, have a look here: https://news.ycombinator.com/item?id=30057549

There's an Italian version of this article here.

I've been a Linux (or GNU/Linux, for the purists) user since 1996. I've been a FreeBSD user since 2002. I have always successfully used both operating systems, each for specific purposes. I have found, on average, BSD systems to be more stable than their Linux equivalents. By stability, I don't mean uptime (too much uptime means too few kernel security updates, which is wrong). I mean that things work as they should, that they don't "break" from one update to the next, and that you don't have to revise everything because of a missing or modified basic command.

I've always been for development and innovation as long as it doesn't (necessarily, automatically and unreasonably) break everything that is already in place. And the road that the various Linux distributions are taking seems to be that of modifying things that work just for the sake of it or to follow the diktats of the Kernel and those who manage it - but not only.

Some time ago we started a complex, continuous and not always linear operation, that is to migrate, where possible, most of the servers (ours and of our customers) from Linux to FreeBSD.

Why FreeBSD?

There are many alternative operating systems to Linux and the *BSD family is varied and complete. FreeBSD, in my opinion, today is the "all rounder" system par excellence, i.e. well refined and suitable both for use on large servers and small embedded systems. The other BSDs have strengths that, in some fields, make them particularly suitable but FreeBSD, in my humble opinion, is suitable (almost) for every purpose.

So back to the main topic of this article, why am I migrating many of the servers we manage to FreeBSD? The reasons are many, I will list some of them with corresponding explanations.

The system is consistent - kernel and userland are created and managed by the same team

One of the fundamental problems with Linux is that (we shall remember) it is a kernel, everything else is created by different people/companies. On more than one occasion Linus Torvalds as well as other leading Linux kernel developers have remarked that they care about the development of the kernel itself, not how users will use it. In the technical decisions, therefore, they don't take into account what is the real use of the systems but that the kernel will go its own path. This is a good thing, as the development of the Linux kernel is not "held back" by the struggle between distributions and software solutions, but at the same time it is also a disadvantage. In FreeBSD, the kernel and its userland (i.e. all the components of the base operating system) are developed by the same team and there is, therefore, a strong cohesion between the parties. In many Linux distributions it was necessary to "deprecate" ifconfig in favor of ip because new developments in the kernel were no longer supported by ifconfig, without breaking compatibility with other (previous) kernel versions or having functions (on the same network interface) managed by different tools. In FreeBSD, with each release of the operating system, there are both kernel and userland updates, so these changes are consistently incorporated and documented, making the tools compatible with their kernel-side updates.

In other words, in FreeBSD there is no need to " revolutionise" everything every few years and changes are made primarily in the form of additions that can enrich (and not break) each update. If a modification was to change the way it interacts with network devices, ifconfig would be modified to take advantage of that and remain compatible with the "old" syntax. In the long-term, this kind of approach is definitely appreciated by system administrators who find themselves with a linear, consistent, and always well-documented update path.

FreeBSD development is (still) driven by technical interests, not strictly commercial ones.

Linux and related distributions now have contributions from many companies, many of which (e.g. Red Hat) push (justifiably) in the direction of what is convenient for them, their products, and their services. Being big contributors to the project they have a big clout so, indeed, their solutions often become de-facto standards. Consider systemd - was there really a need for such a system? While it brought some advantages, it added some complexity to an otherwise extremely simple and functional system. It remains divisive to this day, with many asking, "but was it really necessary? Did the advantages it brought balance the disadvantages?". 70 binaries just for initialising and logging and a million and a half lines of code just for that? But Red Hat threw the rock...and many followed along. Because sometimes it's nice to follow the trend, the hype of a specific solution.

Even FreeBSD has big companies behind it, collaborating in a more or less direct way. The license is more permissive, so not everyone who uses it commercially contributes to it, but knowing that FreeBSD is at the base of Netflix CDNs, Whatsapp servers (waiting for Meta to replace them, for internal coherence reasons, with Linux servers), Sony Playstations and, in part, macOS, iOS, iPadOS, etc. surely gives confidence on its level. These realities, however, do not have enough clout to drive the development of the core team.

Linux has Docker, Podman, lxc, lxd, etc. but... FreeBSD has jails!

FreeBSD jails are very powerful tools for jailing and separating services. There is controversy about Docker not running on FreeBSD but I believe (like many others) that FreeBSD has a more powerful tool. Jails are older and more mature - and by far - than any containerization solution on Linux. Jails are efficient and are well integrated throughout the operating system. All major commands (ps, kill, top, etc.) are able to display jail information as well. There are many management tools but, in fact, they all do the same thing: they interact with FreeBSD base and create custom configuration files. Personally I'm very comfortable with BastilleBSD but there are a lot of very good tools as well as a sufficiently simple manual management. When I need Docker I launch a Linux machine - often Alpine, which I think is a great minimalist distribution, or Debian. But I'm moving a lot of services from Docker to a dedicated jail on FreeBSD. Docker containers are a great tool for rapid (and consistent) software deployment, but it's not all fun and games. Containers, for example, rely on images that sometimes age and are no longer updated. This is a security issue that should not be overlooked.

Linux has ext4, xfs, btrfs... (and zfs, but with some manual intervention). FreeBSD has UFS2 and ZFS

UFS2 is still a very good and efficient file system and, when configured to use softupdates, capable of performing live snapshots of the file system. This is great for backups. Ext4 and XFS do not support snapshots except through external tools (like DattoBD or snapshots through the volume manager). This works, of course, but it is not native. Btrfs is great in its intentions but still not as stable as it should be after all these years of development. FreeBSD supports ZFS natively in the base system and this brings many advantages: separation of datasets for jails, as well as Boot Environments, to make snapshots before upgrades/changes and to be able to boot (from bootloader) even on a different BE, etc.

The FreeBSD boot procedure is cleaner and simpler.

Linux has always used excellent tools such as grub, lilo (now outdated), etc.. FreeBSD has always used a very linear and consistent boot system, with its own bootloader and dedicated boot partition. Whether on mbr, gpt, etc. things are very similar and consistent. I've never had a problem getting a FreeBSD system to boot after a move or recovery from backup. On Linux, however, grub has sometimes given me problems, even after a simple kernel security update.

FreeBSD's network stack is (still) superior to Linux's - and, often, so is its performance.

Meta has been trying to bring the performance of the Linux network stack up to the level of FreeBSD's for years. Many will ask why, then, not move services to FreeBSD. Large companies with huge datacenters can't change solutions overnight, and their engineers, at any level, are Linux experts. They have invested heavily in btrfs, in Linux, in their specifics. Clearly, upon acquiring Whatsapp, they preferred to migrate the "few" Whatsapp servers to Linux and move them to their datacenters. Regarding the real system performance (i.e. disregarding benchmarks, useful only up to a certain point), FreeBSD shines, especially under high load conditions. Where Linux starts to gasp (ex: waiting for I/O) with 100% CPU, FreeBSD has lower processor load and room for more stuff. In the real world (of my servers and load types), I sometimes experienced severe system slowdowns due to high I/O, even if the data to be processed was not read/write dependent. On FreeBSD this does not happen, and if something is blocking, it blocks THAT operation, not the rest of the system. When performing backups or other important operations this factor becomes extremely important to ensure proper (and stable) system performance.

Straightforward system performance analysis

FreeBSD, in the base system, has all the tools to analyze possible problems and system loads. "vmstat" , in a single line, tells me if the machine is struggling for CPU, for I/O or for Ram. "gstat -a" shows me how much, disk by disk, partition by partition, the storage is active, also in percentage with reference to its performance. "top", then, also has support for figuring out, process by process, how much I/O is being used ("m" option). On Linux, to get the same results, you have to install specific applications, different from distribution to distribution.

Bhyve is more incomplete (but more efficient) than KVM

For my purposes, Bhyve is a great virtualization tool. KVM is definitely more complete but since I don't have any special or specific needs not covered by Bhyve on FreeBSD, I found (on average) better performance with this combination. On FreeBSD, however, KSM is missing which, in some cases, can be very useful.

Will I abandon Linux for FreeBSD? Obviously not, just as I haven't for the last 20 years. Both have their uses, their space, their strengths. But if up to now I have had 80% Linux and 20% FreeBSD, the perspective is to invert the percentages of use and, where possible, directly implement solutions based on FreeBSD.

NOTE: this article has been translated from its Italian original version. Even if it's been reviewed and adapted, there might be some errors.

Alpine Linux and LXD perfect setup on BTRFS file system

Stefano Marinelli — Wed, 03 Nov 2021 10:54:10 +0000

Last update: 01 November 2022 - Alpine 3.16 has lxd packages in contrib

Lxc is a great technology. I’ve been using it since 2012 and I’ve always appreciated its lightweight approach and efficiency. Nowadays KVM and hardware virtualization extensions have reached a very good level of performance. While a KVM VM allows to run another kernel and another OS, it ties specific resources to the VM and this isn’t always the best solution. Lxc uses the host’s kernel and shares its ram so the host machine runs lxc container’s processes as local processes - no slowdown, no resource wasting.

Alpine Linux is a great distribution. It’s minimal, reliable, secure and easy to deploy. It’s great for containers but it’s also great for low footprint servers, like hypervisors. It’s good as Xen Dom0, it’s good as KVM host and - no surprise - is great as Lxc host too.

Alpine Linux is great for low energy devices (Raspberry Pis, etc.), too, and has an efficient and reliable out-of-the-box read-only file system support. But I’ll write more about it in another post.

Lxd is a great solution. It uses lxc but simplifies a lot of operations like new container creation, profile management, automatic snapshots, clustering, etc. The “official” best option to use lxd is installing Ubuntu, Snap and so on. I don’t want to use Ubuntu to install Snap. I don’t want to use Snap just to install lxd. That’s why I’m choosing to use lxd with Alpine.

Long story short: this video shows all the steps described below:

Installing Alpine Linux on BTRFS Root File System

Alpine Linux installer supports btrfs as rootfs destination but the setup-alpine command doesn’t allow you to select it: it will partition your disk and create a ext4 standard root file system.

So let’s start with the installation but, then, we’ll interrupt it at disk selection:

Start with setup-alpine and set network, hostname, password, etc. Stop at:

Available disks are:
sda (20.5 GB QEMU     QEMU HARDDISK   )
Which disk(s) would you like to use? (or '?' for help or 'none') [none]

Press ctrl+C and interrupt the script.

Now we can install Alpine Linux on btrfs root system:

test:~# export ROOTFS=btrfs

And, then:

test:~# setup-disk /dev/sda

 The following disk is selected:
 sda    (20.5 GB QEMU     QEMU HARDDISK   )
 How would you like to use it? ('sys', 'data', 'lvm' or '?' for help) [?] sys
 WARNING: The following disk(s) will be erased:
 sda    (20.5 GB QEMU     QEMU HARDDISK   )
 WARNING: Erase the above disk(s) and continue? (y/n) [n] y
 Creating file systems...
 btrfs-progs v5.10
 See http://btrfs.wiki.kernel.org for more information.

 Label:              (null)
 UUID:               6ace888a-2f36-41e8-86b1-3e4214b9e34e
 Node size:          16384
 Sector size:        4096
 Filesystem size:    15.17GiB
 Block group profiles:
 Data:             single            8.00MiB
 Metadata:         DUP             256.00MiB
System:           DUP               8.00MiB
 SSD detected:       no
 Incompat features:  extref, skinny-metadata
 Runtime features:
 Checksum:           crc32c
 Number of devices:  1
 Devices:
 ID        SIZE  PATH
 1    15.17GiB  /dev/sda3

 Installing system on /dev/sda3:
 /mnt/boot is device /dev/sda1

Installation is complete. Now eject the installation media and reboot.

After rebooting, let’s check that root file system is on btrfs:

 test:~# cat /etc/fstab
 UUID=aacbfd28-df47-4945-9c0c-279faef361a0  /   btrfs   rw,relatime,space_cache,subvolid=5,subvol=/ 0 1

Ok, it is. Generally speaking, I prefer to use compression on btrfs. compress=lzo is enough, so you can add it on your fstab.

Now, let’s install lxd.

Edit your /etc/apk/repositories and let it be like this:

http://dl-cdn.alpinelinux.org/alpine/v3.16/main
http://dl-cdn.alpinelinux.org/alpine/v3.16/community

Let’s follow the tutorial I wrote on Alpine Linux wiki and let’s install lxc, lxd, lxcfs and dbus (not needed but recommended to avoid problems with some containers).

After rebooting, you have a fully functional lxd-enabled Alpine Linux installation.

Let’s now configure lxd:

 test:~# lxd init
 Would you like to use LXD clustering? (yes/no) [default=no]:
 Do you want to configure a new storage pool? (yes/no) [default=yes]:
 Name of the new storage pool [default=default]:
 Name of the storage backend to use (btrfs, dir) [default=btrfs]:
 Would you like to create a new btrfs subvolume under /var/lib/lxd? (yes/no) [default=yes]:
 Would you like to connect to a MAAS server? (yes/no) [default=no]:
 Would you like to create a new local network bridge? (yes/no) [default=yes]:
 What should the new bridge be called? [default=lxdbr0]:
 What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
 What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
 Would you like the LXD server to be available over the network? (yes/no) [default=no]:
 Would you like stale cached images to be updated automatically? (yes/no) [default=yes]
 Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:

As you can see, lxd automatically detected that /var/lib/lxd is on a btrfs file system and creates a sub volume.

Now we can launch our first container:

test:~# lxc launch images:alpine/3.16 t1
Creating t1
Starting t1

Ok, done. Let’s check if it’s working:

test:~# lxc info t1
Name: t1
Status: RUNNING
Type: container
Architecture: x86_64
PID: 2648
Created: 2021/11/03 08:55 UTC
Last Used: 2021/11/03 08:55 UTC

Resources:
Processes: 5
CPU usage:
CPU usage (in seconds): 0
Memory usage:
Memory (current): 3.38MiB
Memory (peak): 5.04MiB
Network usage:
eth0:
Type: broadcast
State: UP
Host interface: vethdc207601
MAC address: 00:16:3e:50:d1:ef
MTU: 1500
Bytes received: 3.76kB
Bytes sent: 1.79kB
Packets received: 26
Packets sent: 16
IP addresses:
inet:  10.189.248.21/24 (global)
inet6: fd42:34cb:9408:433c:216:3eff:fe50:d1ef/64 (global)
inet6: fe80::216:3eff:fe50:d1ef/64 (link)
lo:
Type: loopback
State: UP
MTU: 65536
Bytes received: 0B
Bytes sent: 0B
Packets received: 0
Packets sent: 0
IP addresses:
inet:  127.0.0.1/8 (local)
inet6: ::1/128 (local)

It’s running. Everything is ok. For more complex lxc setup (bridges, profiles, etc.) you can follow the excellent lxd documentation.

This kind of setup is easy and lightweight, easy to backup (a new article will follow) and reliable.

BTRFS: performing automatic snapshots and remote backups

Stefano Marinelli — Sun, 28 Jun 2020 12:00:00 +0000

I'm using BTRFS with satisfaction (I mentioned it in 2014 and, more recently, in the IT-Notes blog). I don't consider it optimal for all types of loads, but it solves many problems in many situations. One of the things I use with more satisfaction is the dynamic snapshot generation function, which guarantees the possibility to have a perfect and immediate copy of a specific volume (or subvolume). When a backup is needed, for example, the use of snapshots is fundamental and having them at file system level is undoubtedly a good help.

When it comes to BTRFS volumes, however, we have additional options. There are native tools that are able to send and receive data to and from BTRFS volumes in an optimized way, taking advantage of the inherent features of the file system itself. Here is a method to automatically take snapshots and transfer them to another BTRFS volume (local or remote).

I've been using two interesting scripts for some time now. In the past I used snapper but I didn't find it optimal for my use. I have therefore continued to search and found some valid alternatives, namely the combination of btrfs-snp and btrfs-sync

btrfs-snp

btrfs-snp allows to be launched specifying a path, a prefix linked to the snapshot taken, the maximum number of snapshots with that prefix to hold and a minimum time between snapshots.

For example:

root@foobar:~# /usr/local/sbin/btrfs-snp / hourly 10 3600

It will take a snapshot of /, with the term hourly as prefix (but it can be any prefix), keep a maximum of 10 snapshots and will not take any more (i.e. the program will exit without action) in less than 3600 seconds from the previous one.

You can of course combine them in this way, to have more snapshots of different types:

/usr/local/sbin/btrfs-snp / hourly 10 3600/usr/local/sbin/btrfs-snp / daily 3 86400/usr/local/sbin/btrfs-snp / weekly 1 604800

It can be launched using cron every hour (or every quarter as it will not take a new snapshot if less than 3600 seconds from the previous one have passed).

btrfs-sync

At this point, we can copy all the snapshots to another file system (local or remote) thanks to btrfs-sync.

The command can be of this type:

root@foobar:~# /usr/local/sbin/btrfs-sync -d -v /.snapshots btrfs@host:/backups/host01/

(it's a one line command)

Obviously the destination file system must be a BTRFS volume.

I usually run the command through a script in the cron.daily, to get a daily copy of the snapshots out.

This kind of setup has already got me out of trouble several times and seems to be quite efficient.

BTRFS Best Practices

Stefano Marinelli — Sat, 13 Oct 2018 05:30:00 +0000

I've been using BTRFS for years and have encountered few problems. Some of them are small and can be avoided with some smart improvements. The following advice is based on my experience.

You can find posts online arguing against these options, so your mileage may vary.

Fragmentation, speed and compression

If using rotating disks, employ the space_cache and autodefrag options. The latter can slow down writes but will keep the disk in a usable state for a long time. Otherwise, fragmentation may become a problem.

Generally speaking, I use compression. Modern CPUs are much faster than rotating disks, so compressing data before writing saves space and often improves speed.

Filesystem stats and status

Periodically, check the file system status with the command "btrfs fi df [mountpoint]." Here's an example:

Data, RAID1: total=1.42TiB, used=1.40TiB
System, RAID1: total=32.00MiB, used=256.00KiB
Metadata, RAID1: total=12.00GiB, used=4.68GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

The numbers are quite similar (except for metadata), so no action is needed. Sometimes they're different, and you might encounter a strange situation: you see a lot of free space, but you can't write anymore as the disk appears to be full.

This can happen when the total (allocated) reaches the disk's space, but the used amount is lower. You have space, but you can't use it. A balance will fix this issue.

Perform some manual operations

Ideally, professional file systems should auto-heal and auto-clean. For example, ZFS autoscrubs data periodically. BTRFS still lacks some of these automatic operations, so it's recommended to perform some maintenance. I run them via cron.

My daily job looks like this:

/bin/btrfs fi ba start -dusage=50 -dlimit=2 -musage=50 -mlimit=4 mountpoint

This operation checks and rebalances only a subset of blocks that meet specific conditions. This ensures that the operation only lasts a few minutes. Scheduling it daily will, over time, result in a clean situation.

My weekly job looks like this:

/bin/btrfs scrub start -B -d -c 2 -n 4 mountpoint

This command performs a scrub, checking blocks and detecting (and, if possible, correcting) errors. It's a longer operation, so I don't perform it daily.

I've been using BTRFS since 2012, and thanks to these operations, it still rocks.Its capabilities have helped me in many situations.

I also use BTRFS for my Linux desktop PCs to have snapshots and consistent backups.