Efficient backup of lxc containers in Proxmox -

Sometimes an lxc container can be a better alternative than a KVM virtual machine. It has a smaller overhead, easier and more efficient resource management and lower impact on the physical machine.

You get (potentially) less isolation from the physical machine and you are forced to use the host kernel. Moreover, you cannot run different a different OS.

Proxmox has a great support of lxc containers and they’re often a good choice. I’ve recently converted some VMs to lxc containers and they’re faster, lighter and the Ram occupation is quite lower.

There are two major drawbacks: you cannot migrate a running lxc container (it will be shut down, moved and restarted - if using a shared storage, it is often a matter of seconds) and backups to Proxmox Backup Server are slower than a VM’s backup.

This is because a VM can monitor, thanks to KVM dirty bitmaps, its disk operations to know in advance which blocks should be checked and copied. When dealing with lxc containers it can’t be done, so every backup will need to check every single file - this proved to be efficient in terms of space and deduplication, but quite slow in terms of time. One of my VMs (750 GB stored but with rare disk access) took just a few minutes to be backed up with dirty bitmaps, but more or less 2 hours to backup since it’s been transformed to a lxc container.

borg backup proved to be quite efficient, so I tried to find a way to backup those containers performing a snapshot, backing up the snapshot and, then, releasing it.

We cannot use dattobd on the container, so we must snapshot and backup from the host. This has one big advantage: we don’t need the container to even know its backup location as every operation will be done by the Proxmox host.

My containers generally are stored in Ceph rbd volumes or LVM thin pools.

I’ve created a script to deal with both those situations - and it doesn’t depend on Proxmox, so it also works if you don’t use it.

First of all, create a user and a borg repository on a remote machine and allow the Proxmox host to login via ssh using a key. Then, create a directory (/tobackup in the following scripts, on the host) where the snapshots will be mounted.

In the following example/scripts, no encryption will be used for the backup storage. If you want to encrypt your backup, you will have to perform small modifications.

Here’s my rbd_borg_bck.sh :

#!/bin/bash
set -e

export SOURCE_DISK=$1
export USERNAME=$2
export HOST=$3

PATH="/usr/local/jdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11:/usr/pkg/bin:/usr/pkg/sbin"
export PATH

export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes;

echo "Creating and mapping snapshot..."
rbd snap create $SOURCE_DISK@tobackup && DDISK=`rbd map $SOURCE_DISK@tobackup`

echo "Creating backup directory..."
mkdir -p /tobackup/$DDISK

echo "Mounting snapshot..."
mount -o noload $DDISK /tobackup/$DDISK

echo "DOING BACKUP..."

REPOSITORY=$USERNAME@$HOST:$USERNAME/
TAG=daily

borg create -v --stats --progress --compression zlib,9                           \
    $REPOSITORY::$TAG'-{now:%Y-%m-%dT%H:%M:%S}'          \
    /tobackup/$DDISK/                                       \
    --exclude '*/home/*/.cache*'                  \
    --exclude '*/home/*/.local*'                  \
    --exclude '*/home/*/.pki*'                    \
    --exclude '*/home/*/Virtualbox VMs*'          \
    --exclude '*/home/*/.vagrant.d*'              \
    --exclude '*/root/.cache*'                    \
    --exclude '*/var/swap*'

sleep 3s;

echo "Unmounting snapshot..."
umount /tobackup/$DDISK

sleep 3s;

echo "Unmapping snapshot..."
rbd unmap $DDISK

echo "Remove snapshot..."
rbd snap remove $SOURCE_DISK@tobackup

echo "Pruning..."
borg prune -v $REPOSITORY --stats --prefix $TAG'-' \
    --keep-daily=14 --keep-weekly=8 --keep-monthly=12

echo "Done!"

For example, to backup a disk stored in pool “vmrbdpool” named ‘vm-101-disk-0’ via ssh to server backupserver.mydomain.org, username on the server (and repository name, inside that user’s home directory, called server01), you can launch the script this way:

/usr/local/bin/rbd_borg_bck.sh vmrbdpool/vm-101-disk-0 server01 backupserver.mydomain.org

This can also be scheduled and launched via cron (on the Proxmox host) or any other way.

For the LVM thin pools, the script is a bit different. Here’s my lvm_thin_borg_bck.sh

#!/bin/bash
set -e

export SOURCE_DISK=$1
export USERNAME=$2
export HOST=$3
export LVM=$4
export DDISK=$4/$2-borgsnap

PATH="/usr/local/jdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11:/usr/pkg/bin:/usr/pkg/sbin"
export PATH

export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes;

echo "Creating and mapping snapshot..."
lvcreate -n $2-borgsnap -s $1
lvchange -ay -Ky $DDISK


echo "Creating backup directory..."
mkdir -p /tobackup/$DDISK

echo "Mounting snapshot..."
mount -o ro /dev/$DDISK /tobackup/$DDISK

echo "DOING BACKUP..."

REPOSITORY=$USERNAME@$HOST:$USERNAME/
TAG=daily

borg create -v --stats --progress --compression zlib,9                           \
    $REPOSITORY::$TAG'-{now:%Y-%m-%dT%H:%M:%S}'          \
    /tobackup/$DDISK/                                       \
    --exclude '*/home/*/.cache*'                  \
    --exclude '*/home/*/.local*'                  \
    --exclude '*/home/*/.pki*'                    \
    --exclude '*/home/*/Virtualbox VMs*'          \
    --exclude '*/home/*/.vagrant.d*'              \
    --exclude '*/root/.cache*'                    

sleep 3s;

echo "Unmounting snapshot..."
umount /tobackup/$DDISK

sleep 3s;

echo "Remove snapshot..."
yes | lvremove $DDISK

echo "Pruning..."
borg prune -v $REPOSITORY --stats --prefix $TAG'-' \
    --keep-daily=14 --keep-weekly=8 --keep-monthly=12

echo "Done!"

In this example, we want to backup a disk “/dev/prox1/vm-120-disk-0”, username on the server is server01 (same repository name), server is backupserver.mydomain.org and, as last argument, “prox1” which is the lvm pool.

/usr/local/bin/lvm_thin_borg_bck.sh /dev/prox1/vm-120-disk-0 server01 backupserver.mydomain.org prox1

Borg is efficient and fast and, thanks to this kind of setup, I can create a backup of the 750 GB container in 2 minutes.

Thanks to this kind of setup, backups can be done in a consistent and efficient way.

See also