IT Notes - tipsandtricks

Caching snac Proxied Media with Nginx

Stefano Marinelli — Sat, 08 Feb 2025 16:00:00 +0100

One of the useful yet resource-intensive features of platforms like Mastodon is that they reprocess (and store locally) all multimedia files from other instances.

This behavior is implemented for three valid reasons:

To ensure files do not contain malicious code by reprocessing them locally.
To prevent all users from all instances from overloading the original instance hosting the media by requesting it repeatedly.
To hide the individual IP addresses of users from the original instance.

While this approach has benefits, it also requires significant disk space - often many gigabytes per day.

snac, on the other hand, does not locally process or cache media by default. Instead, media URLs remain unchanged, meaning users will fetch content directly from its original source. This behavior is perfectly fine for many setups (such as mobile devices), but it may not be suitable for everyone.

I suggested a small improvement to snac’s developer, who immediately recognized the benefit for users and implemented an interesting new feature: media proxying via the instance itself.

Enabling Media Proxying in snac

To enable this feature, simply add the following line to your server.json configuration file:

"proxy_media": true

Once enabled, snac will rewrite all media URLs to pass through its own instance. This ensures that original instances will no longer see the IP addresses of individual users, as they will only see the IP of the snac instance itself.

However, this also increases the load and bandwidth consumption of the instance since it must download and forward media every time a user accesses it.

Caching Proxied Media with nginx

To optimize performance, we can configure nginx to cache these proxied files transparently. This way, if multiple users from the snac instance (or the same user at different times) request the same media file, nginx will serve it from the local cache instead of fetching it again. This setup has two key advantages:

Reduces traffic to the original instances.
Lowers the load on the snac instance, as it won’t need to download the same file repeatedly.

To achieve this, create a cache storage area in nginx by adding the following to the http section:

proxy_cache_path /var/cache/nginx/snac_media levels=1:2 keys_zone=snac_media:10m max_size=1g 
                 inactive=1d use_temp_path=off;

This defines a cache directory at /var/cache/nginx/snac_media, with 10 MB allocated for metadata and a maximum cache size of 1 GB. Cached content will be invalidated and removed after one day, ensuring frequently accessed content (like profile avatars) remains cached while rarely used files get replaced over time.

Next, add the following rule to your snac instance's virtual host configuration:

location ~ ^/.+/(x|y)/ {
    proxy_cache snac_media;
    proxy_pass http://snac-ip:8001;
    proxy_set_header Host $host;
    proxy_cache_valid 200 1d;
    proxy_cache_valid 404 1h;
    proxy_ignore_headers "Cache-Control" "Expires";
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    proxy_cache_lock on;
    add_header X-Proxy-Cache $upstream_cache_status;
}

After reloading nginx, the cache will start populating as users request media files. Unrequested content will never enter the cache, ensuring efficient storage usage without unnecessary clutter.

By combining snac’s new media proxying feature with nginx caching, we can achieve a more balanced setup-reducing load on both our instance and external ones, improving privacy, and enhancing performance.

Improving snac Performance with Nginx Proxy Cache

Stefano Marinelli — Wed, 29 Jan 2025 09:00:00 +0100

Some days ago, I migrated my personal Fediverse instance from Akkoma to snac. I appreciate snac a lot and believe it is the best solution available for many use cases.

Akkoma is an excellent tool, but I noticed that even for a small instance like mine, the database grows exponentially, and the database activity remains constant. Despite low load, my disks are continuously "flashing" - which isn't a problem in itself but clearly indicates ongoing activity. In my case, this activity seems unnecessary since it's just a single-user instance.

snac has shown excellent capabilities for managing the FediMeteo project (which I'll write about in detail soon), is lightweight, and has very few dependencies. Moreover, a dedicated snac instance handles sending updates from this blog to the Fediverse.

After successfully transferring my followers from Akkoma to snac without major issues, I started using the new instance. However, as soon as I posted a photo (approximately 4MB), something happened that I somewhat expected but in a different form. My home internet connection (upload speed: 20 Mbit/sec) became saturated, but I also noticed that new connections and smaller entities were resulting in 499 errors - meaning nginx couldn't open new connections to snac. After some investigation, I realized the reason: for every remote instance, Nginx was requesting the multimedia file from snac. Due to saturated connections (snac allows setting the maximum number of active threads), it took several seconds, leading to thread exhaustion in snac. Consequently, subsequent nginx requests resulted in 499 errors as snac could no longer allocate a thread.

To resolve this, I decided to implement direct caching using nginx. My reverse proxy (running in a different FreeBSD jail but this doesn't change the outcome) can cache multimedia files - storing them on first request and serving them directly to everyone who requests them without needing to ask snac every time. This approach is similar to what I use for the media in the BSD Cafe's Mastodon instance, where I employ Varnish to keep everything in RAM.

In snac, images and multimedia files are served from a specific path, such as: https://example.com/user/s/filename.png

The key here is the /s/ segment. By instructing nginx to cache all files containing /s/ in their URL, I can offload some of the work from snac.

I modified my nginx.conf file to include a caching setup:

# Caching configuration for snac
proxy_cache_path /var/cache/nginx/snac_cache levels=1:2 keys_zone=snac:10m max_size=1g inactive=1440m use_temp_path=off;

This creates a section for /var/cache/nginx where I define the caching parameters. It will allocate 10 MB of RAM for metadata, with a maximum cache size of 1 GB (useful if you decide to post some videos). Cached content will be considered invalid and removed after 1440 minutes (24 hours). This ensures that frequently accessed content (like profile avatars and banners) remains in the cache while less frequently accessed content can be replaced with newer content. The goal isn't to have all content cached but rather for nginx to serve files independently during peak times, preventing multiple remote instances from overwhelming snac simultaneously.

In the virtual host configuration for snac, I added this specific override for multimedia content:

# Caching rules for /s/ path
location ~ ^/.+/s/ {
    proxy_cache snac;
    proxy_pass http://snac-jail-ip:8001;
    proxy_set_header Host $host;
    proxy_cache_valid 200 1d;
    proxy_cache_valid 404 1h;
    proxy_ignore_headers "Cache-Control" "Expires";
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    proxy_cache_lock on;
    add_header X-Proxy-Cache $upstream_cache_status;
}

After restarting nginx, the multimedia files will be cached on their first access and served by nginx, leaving snac's threads free to handle everything else. This setup ensures smoother performance and prevents resource exhaustion during periods of high activity or when new content is shared.

Managing ZFS Full Pool Issues with Reserved Space

Stefano Marinelli — Thu, 28 Nov 2024 20:25:00 +0100

Yesterday morning, I received a panicked call from a developer:
"I accidentally filled up the storage, and now I can't perform any operations! My ZFS pool is full!"

I immediately reassured them because I had anticipated this kind of issue. One of the things I almost always do when managing ZFS file systems is to reserve space in a specially created dataset.

This is because ZFS, like all CoW (Copy-on-Write) file systems, can find itself unable to free up space when completely full. By using reserved space, I can always free it up and delete other data, restoring the system to normal operations.

To reserve space, simply create a dataset and assign it a reserved size. Of course, this dataset should not be used for anything else; otherwise, the entire purpose would be defeated.

To create it and reserve space, you only need two simple commands. For example:

zfs create zroot/reserved
zfs set reservation=5G zroot/reserved

This creates the dataset and assigns it 5 GB of reserved space.

Here’s the situation before the operation:

zfs list zroot
NAME    USED  AVAIL  REFER  MOUNTPOINT
zroot  3.02G   109G    96K  /zroot

And here’s the situation after:

zfs list zroot
NAME    USED  AVAIL  REFER  MOUNTPOINT
zroot  8.02G   104G    96K  /zroot

As you can see, the 5 GB are removed from the available space and marked as used, but they are actually empty.

In case of a full file system, you can delete this dataset (or reduce its size) to return to normal file system operation.

Even with this technique, I still recommend not filling ZFS pools beyond 80% of their capacity, as performance degrades significantly past that point.

Moving an entire FreeBSD installation to a new host or VM in a few easy steps

Stefano Marinelli — Mon, 16 Sep 2024 09:41:00 +0200

FreeBSD, especially when installed on ZFS, is incredibly simple to manage, back up, and move to a different system.

I often find myself having to move an entire system from one host to another, from a physical host to a VM, or vice versa, from a VM to a physical host.

By following the approach of keeping the operating system clean and running all services within jails, this operation is usually quite simple: install the operating system on the new host, perform the few necessary configurations (like pf and network interfaces), transfer the jail datasets, restart the jails on the new host, and update the DNS.

However, sometimes I need to move the entire host, including the original operating system. This article will describe this operation when the FreeBSD system uses ZFS as its root file system. This process isn't much more difficult, but there are a few details to keep in mind. Below is a breakdown of the two bootloader scenarios:

UEFI Bootloader
BIOS Bootloader (non-UEFI)

UEFI Bootloader

Start from an ISO or image of mfsbsd. Partition the disk according to the layout of the source system. If the source system is a standard FreeBSD (ZFS) installation, partition the destination disk (assumed to be da0) as follows. WARNING: The first command will destroy any existing partition table on the disk.

gpart destroy -F /dev/da0
gpart create -s gpt da0
gpart add -t efi -s 200M -l efiboot0 da0
newfs_msdos -F 32 -c 1 /dev/da0p1
mount -t msdosfs /dev/da0p1 /mnt
mkdir -p /mnt/EFI/BOOT
cp /boot/loader.efi /mnt/EFI/BOOT/BOOTX64.efi
umount /mnt

Create additional partitions for swap and ZFS:

gpart add -a 1m -t freebsd-swap -s 2g -l swap0 da0
gpart add -a 1m -t freebsd-zfs -l zfs0 da0

Create the ZFS pool:

zpool create -O compression=zstd -O atime=off zroot /dev/gpt/zfs0

On the source system, create a snapshot of all datasets:

zfs snapshot -r zroot@snap01

Important: You will need to connect to the source system using a user with privileges over the entire filesystem. For simplicity, I'll use root, but it's not recommended to leave root accessible via SSH. Limit access to private keys only. To allow root login over SSH, edit /etc/ssh/sshd_config and set PermitRootLogin yes. After moving the server, remember to disable root access.
If mbuffer is installed on the source system, it can help improve the performance of the transfer. If not, remove it from the next pipe command, which should be run from the destination system. The -s will set the block-size, the -m will set the buffer size (128MB, here):

ssh root@sourceip "zfs send -RLv zroot@snap01 | mbuffer -s 128k -m 128M" | zfs receive -F zroot

You have two options now:
If the source server was idle and no further synchronization is required, proceed to the next step.
If the transfer took a long time and services on the source server need to be stopped (e.g., databases), stop what you can, take another snapshot (e.g., zfs snapshot -r zroot@snap02), and synchronize the differences by running another incremental send-receive from the destination server:

ssh root@sourceip "zfs send -RLv -i zroot@snap01 zroot@snap02 | mbuffer -s 128k -m 128M" | zfs receive -F zroot

Set the boot filesystem:

zpool set bootfs=zroot/ROOT/default zroot

BIOS Bootloader

Start from an ISO or image of mfsbsd. Partition the disk according to the layout of the source system. If the source system is a standard FreeBSD (ZFS) installation, partition the destination disk (assumed to be da0) as follows. WARNING: The first command will destroy any existing partition table on the disk.

gpart destroy -F /dev/da0
gpart create -s gpt da0

Create the necessary partitions:

gpart add -a 1m -t freebsd-boot -s 512k -l boot da0
gpart add -a 1m -t freebsd-swap -s 2g -l swap0 da0
gpart add -a 1m -t freebsd-zfs -l zfs0 da0

Create the ZFS pool:

zpool create -O compression=zstd -O atime=off zroot /dev/gpt/zfs0

On the source system, create a snapshot of all datasets:

zfs snapshot -r zroot@snap01

Important: You will need to connect to the source system using a user with privileges over the entire filesystem. For simplicity, I'll use root, but it's not recommended to leave root accessible via SSH. Limit access to private keys only. To allow root login over SSH, edit /etc/ssh/sshd_config and set PermitRootLogin yes.
If mbuffer is installed on the source system, it can help improve the performance of the transfer. If not, remove it from the next pipe command, which should be run from the destination system:

ssh root@sourceip "zfs send -RLv zroot@snap01 | mbuffer -s 128k -m 128M" | zfs receive -F zroot

You have two options now:
If the source server was idle and no further synchronization is required, proceed to the next step.
If the transfer took a long time and services on the source server need to be stopped (e.g., databases), stop what you can, take another snapshot (e.g., zfs snapshot -r zroot@snap02), and synchronize the differences by running another incremental send-receive from the destination server:

ssh root@sourceip "zfs send -RLv -i zroot@snap01 zroot@snap02 | mbuffer -s 128k -m 128M" | zfs receive -F zroot

Install the bootloader on the destination server’s disk:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

Set the boot filesystem:

zpool set bootfs=zroot/ROOT/default zroot

At this point, everything should be ready for boot. Reboot and verify that the various mountpoints (e.g., swap, etc.) and network interface settings are correct. If everything works fine, the server will be an exact copy of the original.

Moving or duplicating an entire FreeBSD system is simple, fast, and reliable. The ability to use ZFS incremental replication is an excellent method for minimizing downtime, especially for large systems. The initial copy will happen "live" while the source system continues to run without issues. Subsequent incrementals will be much faster, making the switch-over process seamless and efficient.

FreeBSD Tips and Tricks: Limiting Process Priority in a FreeBSD Jail

Stefano Marinelli — Thu, 11 Jul 2024 08:41:00 +0200

FreeBSD allows for quite comprehensive resource limitation for one or more jails. Beyond the official documentation, there is a good description in MWL's book. There's also an interesting article from Klara Systems that describes some functionalities.

Sometimes, however, we only need the processes running within a specific jail to have a specific priority - higher or lower than others.

There are many methods to achieve this, but the simplest one, in my opinion, is to leverage the properties of the nice(1) command. The main property, in fact, is to transmit the set priority to child processes, so all processes launched by the command that received a different level of "niceness" will inherit its priority.

For example, to give the minimum priority to the services launched within a jail, just modify the .conf file of the jail (in the case of a standard BastilleBSD installation, the file will be /usr/local/bastille/jails/jailname/jail.conf) and change the command

exec.start = '/bin/sh /etc/rc';

to:

exec.start = '/usr/bin/nice -n 20 /bin/sh /etc/rc';

In this way, when rc starts at the jail's boot, it will have a niceness of 20 and will transmit it to all the processes that rc itself launches (thus all the services of the jail).

Note: This will only apply to child processes of rc, not to:

Commands manually launched from the jail console
Services launched by running service servicename start (or restart) from the jail shell. This is because, in this case, the process will not be a child of rc but will derive directly from the console in use.

Using the nice command to set the priority of processes within a FreeBSD jail is a simple and effective method. However, it is important to be aware of the limitations of this approach and ensure that manually executed commands are managed accordingly.

FreeBSD Tips and Tricks: Creating Snapshots with UFS

Stefano Marinelli — Tue, 04 Jun 2024 06:40:45 +0000

One of the main features that every modern file system should support, in my opinion, is the ability to take snapshots. Snapshots can be useful for many reasons, such as making changes or updates with the ability to revert at any time.

On Linux, file systems operating in COW (like ZFS, btrfs, etc.) can handle snapshots efficiently, but traditional file systems like ext4 lack this support. The only way is to add another layer (like LVM) and take a snapshot of the volume, mounting it elsewhere in read-only mode.

FreeBSD provides tools for creating snapshots using both ZFS and UFS. I'll focus on UFS. In the base system, there's a very useful and well-documented command: mksnap_ffs.

Long story short: this video shows all the steps described below:

mksnap_ffs will create a snapshot of the current state of the specified file system and store it in the file you specify. For example, running mksnap_ffs /snap will take a snapshot of the "/" file system and store it in the file "/snap". By creating a memory disk and mapping the /snap file to an md, you can then mount it in read-only mode and retrieve the information. Finally, you can unmount, remove the md (mdconfig -du /dev/mdX), and delete the snapshot with a simple rm (rm /snap). Note that each file system can have up to 20 such snapshots, so it shouldn't be considered on par with ZFS. However, it can be crucial for making a consistent backup of a live system with tools like Borg, Restic, Kopia, etc., or a "simple" rsync.

Steps to Create and Use Snapshots with UFS

Create the Snapshot sh mksnap_ffs /snap This command takes a snapshot of the "/" file system and stores it in the file "/snap".
Create a Memory Disk sh mdconfig /snap This command maps the snapshot file to a memory disk.
Mount the Snapshot sh mount -o ro /dev/md0 /mnt This command mounts the memory disk in read-only mode to /mnt.
Access the Snapshot

You can now access the snapshot via /mnt.
Cleanup sh umount /mnt mdconfig -du 0 rm /snap These commands unmount the snapshot, remove the memory disk, and delete the snapshot file.

By leveraging this built-in feature of FreeBSD, you can ensure your systems are more resilient and easier to manage, particularly for backups and system updates.

FreeBSD Tips and Tricks: Native Read-Only Root File System

Stefano Marinelli — Fri, 31 May 2024 16:53:45 +0000

Many years ago, I needed to create some small servers using PCEngines Alix and had to choose the most suitable operating system. One of the challenges was dealing with dirty shutdowns and subsequent reboot issues, sometimes requiring manual intervention (not always feasible). After testing various Linux distributions and BSD systems, I found the ideal solution: FreeBSD.

One highly useful but often overlooked feature of FreeBSD is its ability to detect that the system is in read-only mode and automatically mount important directories (like /tmp, /var/log, etc.) in tmpfs, allowing the system to function perfectly.

Since then, whenever I need a system that can recover from any type of blackout without manual intervention (such as my Raspberry Pies), I always use this feature. If needed, external storage devices will be mounted automatically (or manually) after boot, and in case of issues, I can always access the device via SSH to perform necessary checks on the external filesystem.

This feature is automatic: simply have the root file system on UFS (not ZFS) and set it to "ro" (from "rw") in /etc/fstab. After a reboot, the system will operate in read-only mode, helping preserve the integrity of the embedded memory.

For updates, just:

Temporarily remount the root file system as rw (mount -uo rw /)
Modify /etc/fstab, changing "ro" back to "rw"
Reboot the device and update it normally