One feature I couldn't live without anymore is snapshots. As system administrators, we often find ourselves in situations where we've made a mistake, need to revert to a previous state, or need access to a log that has been rotated and disappeared. Since I started using ZFS, all of this has become incredibly simple, and I feel much more at ease when making any modifications.
However, since I don't always remember to create a manual snapshot before starting to work, I use an automatic snapshot system. For this type of snapshot, I use the excellent zfs-autobackup tool - which I also use for backups. The goal is to have a single, flexible, and configurable tool without having to learn different syntaxes.
Why zfs-autobackup?
zfs-autobackup has several advantages that make it perfect (or nearly so) for my purpose:
It operates based on "tags" set on individual datasets. I don't have to specify the dataset; I just assign a specific tag (of my choice) to the datasets, and
zfs-autobackupwill operate on those datasets, transparently with respect to others. This ensures it will work even on datasets in different zpools.It's extremely flexible in management. For example, by setting the correct tag to "zroot", it will automatically manage all underlying datasets. However, it's possible to exclude some for more granular snapshot management.
It works well on both FreeBSD and Linux - I use it with satisfaction on both platforms.
Different tags allow for different levels of data retention and operation. For example, the "mylocalsnap" tag will be for local snapshots, while "backup_offsite" will be for backups that will be copied off-site. The two tags (and related snapshots) will be independent, even though they operate on the same datasets.
Installation
On FreeBSD, installation is straightforward, as there's a ready-made package:
pkg install py311-zfs-autobackup
On Linux, it will depend on the specific distribution. Being in Python, it will always be possible to install it using pip.
Configuration
Once installed, you just need to assign the tag to the dataset. For example:
zfs set autobackup:mylocalsnap=true zroot
This will set a tag called "mylocalsnap" on zroot and underlying datasets, i.e., on the entire main file system of FreeBSD.
Usage
Now, you just need to run zfs-autobackup, specifying both the tag and the snapshot retention criteria:
/usr/local/bin/zfs-autobackup mylocalsnap --keep-source 5min1h,1h1d
In this case, it will take a (recursive) snapshot of the datasets that have the "mylocalsnap" tag set to true, keeping one snapshot every 5 minutes for an hour, and one every hour for a day.
On subsequent executions of zfs-autobackup, snapshots that don't meet the previous retention criteria will be deleted.
After running this command, here's the result:
root@fbsnap:~ # zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
zroot@mylocalsnap-20240820150115 0B - 96K -
zroot/ROOT@mylocalsnap-20240820150115 0B - 96K -
zroot/ROOT/default@mylocalsnap-20240820150115 0B - 1.00G -
zroot/home@mylocalsnap-20240820150115 0B - 96K -
zroot/tmp@mylocalsnap-20240820150115 0B - 104K -
zroot/usr@mylocalsnap-20240820150115 0B - 96K -
zroot/usr/ports@mylocalsnap-20240820150115 0B - 96K -
zroot/usr/src@mylocalsnap-20240820150115 0B - 96K -
zroot/var@mylocalsnap-20240820150115 0B - 96K -
zroot/var/audit@mylocalsnap-20240820150115 0B - 96K -
zroot/var/crash@mylocalsnap-20240820150115 0B - 96K -
zroot/var/log@mylocalsnap-20240820150115 0B - 144K -
zroot/var/mail@mylocalsnap-20240820150115 0B - 96K -
zroot/var/tmp@mylocalsnap-20240820150115 0B - 96K -
As you can see, snapshots have been taken of all datasets with the tag set.
Automation
To automate the process, simply modify the /etc/crontab file and add a line like this:
*/5 * * * * root /usr/local/bin/zfs-autobackup mylocalsnap --keep-source 5min1h,1h1d
Now, wait a few minutes and check again:
root@fbsnap:~ # zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
zroot@mylocalsnap-20240820150115 0B - 96K -
zroot/ROOT@mylocalsnap-20240820150115 0B - 96K -
zroot/ROOT/default@mylocalsnap-20240820150115 212K - 1.00G -
zroot/ROOT/default@mylocalsnap-20240820151000 128K - 1.00G -
zroot/home@mylocalsnap-20240820150115 0B - 96K -
zroot/tmp@mylocalsnap-20240820150115 72K - 104K -
zroot/tmp@mylocalsnap-20240820151000 0B - 104K -
zroot/usr@mylocalsnap-20240820150115 0B - 96K -
zroot/usr/ports@mylocalsnap-20240820150115 0B - 96K -
zroot/usr/src@mylocalsnap-20240820150115 0B - 96K -
zroot/var@mylocalsnap-20240820150115 0B - 96K -
zroot/var/audit@mylocalsnap-20240820150115 0B - 96K -
zroot/var/crash@mylocalsnap-20240820150115 0B - 96K -
zroot/var/log@mylocalsnap-20240820150115 64K - 144K -
zroot/var/log@mylocalsnap-20240820151000 60K - 144K -
zroot/var/mail@mylocalsnap-20240820150115 0B - 96K -
zroot/var/tmp@mylocalsnap-20240820150115 64K - 96K -
zroot/var/tmp@mylocalsnap-20240820151000 0B - 96K -
If everything went as it should, you'll notice that unmodified datasets won't have new snapshots, while those that have been modified since the previous manual execution (e.g., zroot/var/log) will contain both the previous snapshot and the automatic one.
Recovering Files from Snapshots
There are various ways to recover a file from the previous snapshot. One option is to restore the entire snapshot to the current dataset, but this may not be the best option as it will perform a complete restore.
Another alternative is to go to the hidden snapshot directory. For example:
root@fbsnap:~ # cd /var/log/.zfs/snapshot/mylocalsnap-20240820151000/
root@fbsnap:/var/log/.zfs/snapshot/mylocalsnap-20240820151000 # ls -l
total 39
-rw------- 1 root wheel 872 Aug 20 15:06 auth.log
-rw-r--r-- 1 root wheel 79079 Aug 20 13:19 bsdinstall_log
-rw------- 1 root wheel 5401 Aug 20 15:10 cron
-rw-r--r-- 1 root wheel 63 Aug 20 13:20 daemon.log
-rw------- 1 root wheel 63 Aug 20 13:20 debug.log
-rw-r--r-- 1 root wheel 63 Aug 20 13:20 devd.log
-rw-r--r-- 1 root wheel 63 Aug 20 13:20 lpd-errs
-rw-r----- 1 root wheel 63 Aug 20 13:20 maillog
-rw-r--r-- 1 root wheel 17043 Aug 20 15:00 messages
-rw-r----- 1 root network 63 Aug 20 13:20 ppp.log
-rw------- 1 root wheel 63 Aug 20 13:20 security
-rw-r--r-- 1 root wheel 197 Aug 20 15:00 utx.lastlogin
-rw-r--r-- 1 root wheel 187 Aug 20 15:00 utx.log
-rw------- 1 root wheel 63 Aug 20 13:20 xferlog
From here, you can read and recover any file present in the individual snapshot. The snapshots are read-only, so you won't be able to write to them.
Creating a Writable Copy of a Snapshot
Should you need a read-write copy of a specific snapshot, you can use the zfs clone command:
root@fbsnap:~ # zfs clone zroot/var/log@mylocalsnap-20240820151000 zroot/recover
root@fbsnap:~ # ls -l /zroot/recover/
total 39
-rw------- 1 root wheel 872 Aug 20 15:06 auth.log
-rw-r--r-- 1 root wheel 79079 Aug 20 13:19 bsdinstall_log
-rw------- 1 root wheel 5401 Aug 20 15:10 cron
-rw-r--r-- 1 root wheel 63 Aug 20 13:20 daemon.log
-rw------- 1 root wheel 63 Aug 20 13:20 debug.log
-rw-r--r-- 1 root wheel 63 Aug 20 13:20 devd.log
-rw-r--r-- 1 root wheel 63 Aug 20 13:20 lpd-errs
-rw-r----- 1 root wheel 63 Aug 20 13:20 maillog
-rw-r--r-- 1 root wheel 17043 Aug 20 15:00 messages
-rw-r----- 1 root network 63 Aug 20 13:20 ppp.log
-rw------- 1 root wheel 63 Aug 20 13:20 security
-rw-r--r-- 1 root wheel 197 Aug 20 15:00 utx.lastlogin
-rw-r--r-- 1 root wheel 187 Aug 20 15:00 utx.log
-rw------- 1 root wheel 63 Aug 20 13:20 xferlog
This creates a new dataset zroot/recover that is a writable copy of the snapshot. You can now modify these files as needed, without affecting the original snapshot or the live filesystem.
Cleaning Up
Sometimes you may want to delete all snapshots generated by zfs-autobackup. There are various ways, but the quickest can be to use a simple pipe:
zfs list -t snapshot -o name | grep -i mylocalsnap | xargs -n 1 zfs destroy -vr
By implementing automatic ZFS snapshots, you can work with peace of mind, knowing that you can always revert changes or recover lost files. This setup provides an excellent balance between data protection and system performance.