IT Notes - security

Monitor your devices with LibreNMS on FreeBSD

Stefano Marinelli — Thu, 07 May 2026 10:45:00 +0000

LibreNMS has been a faithful companion for years now. It quietly handles the monitoring of my servers, devices, and services without demanding much in return - exactly what you want from a tool whose job is to watch over everything else. It's a solid alternative to heavier solutions like Zabbix, and it gives you alerts, data, and graphs on virtually anything reachable over SNMP.

I usually install it on a host that is not reachable from the outside, then let it poll all the devices through a VPN: a single observation point, clean perimeter. The ability to create multiple dashboards - and to filter them by user - has also let me give clients a transparent window onto their own servers. Transparency, in my experience, is always the better long-term bet.

Together with Uptime-Kuma (and the good old Nagios/Munin pair), LibreNMS lives in a FreeBSD jail on my monitoring servers and just does its job.

This post walks through a plain installation of LibreNMS on FreeBSD: package-based, no reverse proxy, no HTTPS, no fancy hardening. The goal is to get to a working setup you can build on top of.

Assumptions

FreeBSD 15.0-RELEASE, in a jail or on a dedicated VM/host
nginx + php-fpm + MySQL 8.4
LibreNMS installed from the official package — not via git clone

One note before we start: in this guide I use plain HTTP just to reach the first-time setup. If your LibreNMS instance won't stay confined to a private network or behind a VPN, configuring HTTPS is mandatory, not optional.

Installation

pkg install librenms mysql84-server python3 nginx

LibreNMS currently depends on PHP 8.4. If you want to speed PHP up, install OPcache too:

pkg install php84-opcache

MySQL

Two settings need to be in place before MySQL starts for the first time. After the first start they cannot be changed without reinitializing the data directory, so it's worth getting them right now.

cd /usr/local/etc/mysql
cp my.cnf.sample my.cnf

In the [mysqld] section, add:

innodb_file_per_table=1
lower_case_table_names=0

Now start MySQL:

service mysql-server enable
service mysql-server start

On a fresh FreeBSD install, the local root user can connect to MySQL without a password from the command line. Connect and create the database and user. I'm using password here as a placeholder - don't.

mysql

CREATE DATABASE librenms CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER 'librenms'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON librenms.* TO 'librenms'@'localhost';
exit

php-fpm

Edit /usr/local/etc/php-fpm.d/www.conf and adjust the listen directives:

listen = /var/run/php-fpm-librenms.sock
listen.owner = www
listen.group = www
listen.mode = 0660

Then create php.ini from the production sample:

cd /usr/local/etc
cp php.ini-production php.ini

And set the timezone in php.ini:

date.timezone = Europe/Rome

nginx

Since this jail (or host) is dedicated to LibreNMS, we can rewrite the server block in /usr/local/etc/nginx/nginx.conf directly:

server {
    listen      80;
    #server_name yourServerName
    root        /usr/local/www/librenms/html;
    index       index.php;

    charset utf-8;
    gzip on;
    gzip_types text/css application/javascript text/javascript application/x-javascript image/svg+xml text/plain text/xsd text/xsl text/xml image/x-icon;

    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    location /api/v0 {
        try_files $uri $uri/ /api_v0.php?$query_string;
    }

    location ~ \.php$ {
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
        set $path_info $fastcgi_path_info;
        try_files $fastcgi_script_name =404;
        include fastcgi_params;
        fastcgi_param SERVER_SOFTWARE "";
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param PATH_INFO $path_info;
        fastcgi_index index.php;
        fastcgi_pass unix:/var/run/php-fpm-librenms.sock;
        fastcgi_buffers 256 4k;
        fastcgi_intercept_errors on;
        fastcgi_read_timeout 14400;
    }

    location ~ /\.(?!well-known).* {
        deny all;
    }
}

Now start nginx and php-fpm:

service nginx enable
service nginx start

service php_fpm enable
service php_fpm start

LibreNMS configuration

Copy the default config:

cp /usr/local/www/librenms/config.php.default /usr/local/www/librenms/config.php

Because we installed from the package, this file already has the right commands and paths for FreeBSD - no need to hunt down mtr, fping, snmpwalk and friends one by one.

Create the directory for RRD graphs and set ownership:

mkdir -p /var/db/librenms/rrd
chown -R www:www /var/db/librenms
chmod 775 /var/db/librenms/rrd

Then the .env file:

cd /usr/local/www/librenms
cp .env.example .env
chown www .env

Edit .env and set at least:

DB_DATABASE - librenms
DB_USERNAME - librenms
DB_PASSWORD - the one you actually used (not password, please)

Then add this line, which tells LibreNMS we still need to run the web installer:

INSTALL=true

A note on permissions. The official LibreNMS documentation suggests chown -R www:www over the entire application tree, but on FreeBSD the package already lays down sane ownership, with storage/ and bootstrap/cache/ writable by www. There's no reason to widen the rest of the codebase. If validate.php complains later about something write-related, the first place to check is:

ls -la /usr/local/www/librenms/storage /usr/local/www/librenms/bootstrap/cache

Now generate the app key as www, since the file is owned by www:

su -m www -c "php artisan key:generate"

And tighten .env:

chmod 600 .env

Refresh the configuration cache:

su -m www -c "lnms config:clear"
su -m www -c "lnms config:cache"

Web installer

Open http://host/install and follow the steps. The validation process may fail. Refreshing the cache picks up the values written to config.php during the install:

su -m www -c "lnms config:clear"
su -m www -c "lnms config:cache"

When the web installer is done, edit .env again and remove the INSTALL=true line if it's still there. Leaving it in place re-exposes the installer to anyone who can reach the URL.

Polling service

LibreNMS needs something to actually run the polls. On FreeBSD, the package ships an rc service that runs the LibreNMS dispatcher, so there's no need to manage cron entries by hand the way most Linux guides assume.

service librenms enable
service librenms start

Validate

cd /usr/local/www/librenms
su -m www -c './validate.php'

You may see a couple of complaints right after starting the service - usually scheduler-related and self-resolving within a few minutes. Re-run validate.php once the dispatcher has had time to settle. Anything still red after that is worth investigating.

Next steps

At this point you can log into the web interface and start adding devices, configuring SNMP, and building dashboards. For that, the official LibreNMS documentation is excellent, and there's no point in me paraphrasing it here.

Evolving the BSD Cafe Network Setup: From Bridging to Routing with FreeBSD

Stefano Marinelli — Thu, 01 Aug 2024 13:05:00 +0200

In the ever-changing landscape of system administration, network configurations often need to grow and evolve to meet new challenges and requirements. This post details my journey from a simple VPS setup to a complex, multi-node network using FreeBSD, jails, VPNs, and advanced routing techniques. Along the way, I'll explore the reasons behind each change and delve into why certain solutions, while functional, may not always be ideal in the long run.

Initial Setup: The Single VPS

My story begins with a single VPS (let's call it VPSSmall) hosted on Hetzner, running FreeBSD. This initial configuration was straightforward and served its purpose well for a time:

Created an internal bridge (bridge0) with IP 192.168.123.1
Used BastilleBSD to create VNET jails with IPs in the 192.168.123.X range
Set up port forwarding to the jails using pf's rdr rules
Utilized a /64 IPv6 block, subdivided into /72 subnets for the bridge and jails

This setup allowed for easy management of multiple services within isolated jails, all sharing the same network namespace.

Growing Pains: Adding a Second VPS

As is often the case in system administration, my needs grew over time. The number of jails increased, and their resource requirements expanded. To address this, I added a second VPS (VPSBig) hosted on a Proxmox server. This VPS isn't directly exposed, and relies on NAT to connect to the outside world. This introduced a new challenge: how to maintain flexibility in moving jails between VPSs without changing their network configurations?

To solve this, I implemented the following setup:

Installed ZeroTier on both VPSs in bridge mode
Created bridge0 on VPSBig (without an IP)
Added ZeroTier interfaces to both bridges

This configuration allowed for seamless movement of jails between nodes by simply transferring the ZFS dataset. The jails could retain their IP addresses regardless of which physical VPS they were running on.

While this setup was functional, it had some drawbacks that I'll discuss in the next section.

The Limitations of Bridging

The bridged setup using ZeroTier, while effective, wasn't without its issues. Here's why I found bridging, in this case, wasn't an ideal long-term solution:

Performance Overhead: Bridging all traffic between VPSs can introduce additional latency and processing overhead, especially when dealing with high-volume traffic.
Scalability Concerns: As the number of VPSs and jails grows, managing a large bridged network becomes increasingly complex. Each new node added to the network increases the potential for broadcast storms and can lead to unnecessary traffic across the entire network.
Security Implications: In a bridged network, all nodes essentially exist on the same network segment. This can potentially allow for lateral movement between jails or VPSs if not carefully managed, increasing the attack surface.
Dependency on ZeroTier: While ZeroTier is a powerful tool, relying on a third-party service for critical infrastructure introduces an external point of failure and potential security considerations.
Limited Control: Bridging provides less granular control over traffic flow compared to routing. This can make it harder to implement complex network policies or optimize traffic paths.
Broadcast Domain Size: Large bridged networks can result in expansive broadcast domains, which can lead to increased network congestion and reduced overall performance.

These limitations prompted me to seek a more robust, scalable, and controllable solution, leading to the next evolution of my network setup.

Refining the Setup: Wireguard and VXLAN

To address the limitations of the bridged setup, I implemented a new configuration involving Wireguard and VXLAN:

Created a Wireguard VPN between VPSSmall and VPSBig
Implemented a VXLAN over the Wireguard tunnel
Replaced ZeroTier with the VXLAN for inter-VPS communication

This setup, described in detail here, offered several advantages:

Improved Security: Wireguard provided a secure, encrypted tunnel between the VPSs.
Better Performance: Direct Wireguard VPN connection often results in lower latency compared to ZeroTier.
Greater Control: By managing all the components of the VPN myself, I have more control over the network configuration and problems.
Reduced Dependency: Eliminating ZeroTier removed a third-party dependency from my critical infrastructure.

While this setup was a significant improvement, it still relied on bridging via VXLAN, which didn't fully address all the scalability and control issues. This realization led to the final evolution of my network.

The Final Evolution: Routing Instead of Bridging

The last step in my network's evolution was to move from a bridged to a routed setup. This change offered even more flexibility and scalability. The new configuration:

VPSSmall: Uses 192.168.122.x/24 and a /72 IPv6 subnet
VPSBig: Uses 192.168.123.x/24 and its original /72 IPv6 subnet
Future nodes can use new private IPv4 ranges and /72 IPv6 subnets as needed

This routed setup provides several key benefits:

Improved Scalability: Each VPS or future node can have its own subnet, making it easier to add new nodes without reconfiguring the entire network.
Better Traffic Control: Routing allows for more granular control over traffic flow, enabling complex network policies and optimizations.
Enhanced Security: With distinct subnets, it's easier to implement security policies and control inter-subnet communication.
Reduced Broadcast Domain: Each subnet forms its own broadcast domain, reducing unnecessary network traffic and improving overall performance.

To ensure jails on VPSBig use the Wireguard tunnel for outgoing traffic while VPSBig itself uses its default gateway, I leveraged FreeBSD's FIB feature. This allows for separate routing tables, providing even more flexibility in managing network traffic.

Implementing Multiple FIBs

Edit /etc/sysctl.conf to enable multiple FIBs:

net.fibs=2

This allows me to use two separate routing tables.

VPSBig Wireguard Configuration

[Interface]
PrivateKey = *VPSBigPrivateKey*
Address = 10.77.0.2/32,oneOfMyIpv6/128

Table = off
PostUp = route -q -n add -inet 0.0.0.0/0 -interface wg0 -fib 1
PostUp = route -q -n add -inet6 ::/1 -interface wg0 -fib 1
PostUp = route -q -n add -inet6 8000::/1 -interface wg0 -fib 1

[Peer]
PublicKey = *VPSSmallPublicKey*
AllowedIPs = 0.0.0.0/0,::0/0
Endpoint = *endpointip:port*
PresharedKey = *presharedkey*
PersistentKeepalive = 30

Let's break down this configuration:

Table = off: This disables Wireguard's automatic routing table management. We're doing this because we want to manually configure the routing.
The PostUp commands are crucial for our manual routing setup:
route -q -n add -inet 0.0.0.0/0 -interface wg0 -fib 1: This adds a default route for IPv4 traffic through the Wireguard interface (wg0) in the alternate routing table (FIB 1).
The next two commands do the same for IPv6 traffic, covering the entire IPv6 address space (::/1 and 8000::/1 together cover all IPv6 addresses).
AllowedIPs = 0.0.0.0/0,::0/0: This tells Wireguard to route all traffic through this peer. However, because we've set Table = off, Wireguard won't actually create these routes - we're doing it manually with our PostUp commands.
PersistentKeepalive = 30: This sends a keepalive packet every 30 seconds, which is necessary because VPSBig is behind NAT and needs to keep the NAT session alive.

VPSSmall Wireguard Configuration

[Interface]
PrivateKey = *VPSSmallPrivateKey*
ListenPort = *port*
Address = 10.77.0.1/24,oneOfMyIpv6/128

[Peer]
PublicKey = *VPSSmallPublicKey*
PresharedKey = *presharedkey*
AllowedIPs = 10.77.0.2/32, 192.168.123.0/24, *theRemote/72ipv6class*/72

In this configuration:

Address = 10.77.0.1/24,oneOfMyIpv6/128: This sets up the Wireguard interface with an IPv4 and IPv6 address.
AllowedIPs = 10.77.0.2/32, 192.168.123.0/24, *theRemote/72ipv6class*/72: This tells Wireguard to route traffic for VPSBig's Wireguard IP (10.77.0.2), VPSBig's local subnet (192.168.123.0/24), and VPSBig's IPv6 subnet through this peer.

The Role of FIB in Routing

The use of FIB (Forwarding Information Base) 1 in the VPSBig configuration is key to our setup. By adding routes to FIB 1, we're creating a separate routing table that can be used by our jails. This allows us to:

Keep the main system (VPSBig itself) routing through its default gateway.
Route all traffic from the jails through the Wireguard tunnel to VPSSmall.

We achieve this by configuring the jails to use FIB 1:

exec.prestart += "ifconfig epairXa fib 1";
exec.prestart += "ifconfig epairXb mtu 1380";

This ensures that each jail uses the alternate routing table (FIB 1) instead of the default routing table, effectively sending all its traffic through the Wireguard tunnel.

If the WireGuard MTU is 1420, setting the jail's MTU to 1380 should be safe enough.

NAT Configuration on VPSSmall

To complete the setup, we need to configure NAT on VPSSmall to allow the jails on VPSBig to access the internet:

nat on vtnet0 from 192.168.123.0/24 to ! <private> -> vtnet0:0

This NAT rule translates the source IP of packets coming from VPSBig's subnet (192.168.123.0/24) to VPSSmall's public IP when they're destined for non-private IP addresses.

Conclusion

My journey from a simple single-VPS setup to this complex, multi-node network illustrates the power and flexibility of FreeBSD. By transitioning from a bridged to a routed setup, I've created a solution that offers improved scalability, security, and control.

Key takeaways from this evolution include:

Adaptability is Crucial: As your needs grow, be prepared to evolve your network architecture.
Understand the Tradeoffs: Each networking approach (bridging, VPNs, routing) has its pros and cons. Choose the one that best fits your current and future needs.
Leverage Advanced Features: FreeBSD's features like jails, FIBs, and pf allow for powerful and flexible network configurations.
Security is Paramount: Always consider the security implications of your network design, especially when dealing with multiple nodes and public-facing services.
Documentation is Key: Keep detailed notes of your network evolution. It helps in troubleshooting and future planning.

Whether you're managing a small personal server or a large-scale infrastructure, the techniques described here can help you build a robust and adaptable network. Remember, network design is an iterative process. Don't be afraid to evolve your setup as your needs change.

Installing Uptime-Kuma in a FreeBSD Jail

Stefano Marinelli — Mon, 22 Jul 2024 07:42:00 +0000

Uptime-Kuma is a very useful tool. Besides being able to monitor whether services, websites, or ports are responding, it can also send notifications using many different services, even combined, and check the expiration of certificates. Additionally, it can create informative dashboards about the status of services.

I have been using it extensively for years, including for BSD Cafe. My initial deployments were on Docker containers, but recently, I have moved everything to FreeBSD jails.

When proceeding with the standard installation described in their documentation, the process will halt because one of the third-party libraries (Playwright) is not officially compatible with FreeBSD. However, since this library is not necessary for the type of checks I want to perform (I am not interested in checking web page components, just their response), it is possible to continue and run everything using a small hack.

It can be installed directly on FreeBSD but, in my case, I prefer using jails. Note that if using traditional jails (i.e., not VNet), "ping" is disabled. Therefore, if using traditional jails and wanting to check hosts via ping, it is advisable to add the line allow.raw_sockets; to the respective jail.conf.

The installation can proceed as follows:

The first step is to install the necessary dependencies:

pkg install node20 npm-node20 git-lite

Once the dependencies are installed, it is advisable to create a user to run everything and use that user for the next steps:

pw add user ukuma -m
echo 'export LC_ALL="en_US.UTF-8"' >> /home/ukuma/.profile
su -l ukuma

At this point, clone the Uptime-Kuma repository and start installing the dependencies:

git clone https://github.com/louislam/uptime-kuma.git
cd uptime-kuma/
npm run setup

The operation will seem to complete correctly. However, running the program will result in a fatal error preventing it from functioning:

ukuma@ukuma:~/uptime-kuma $ node server/server.js
[...]
/home/ukuma/uptime-kuma/node_modules/playwright-core/lib/server/registry/index.js:258
    if (process.platform === 'linux') cacheDirectory = process.env.XDG_CACHE_HOME || _path.default.join(os.homedir(), '.cache'); else if (process.platform === 'darwin') cacheDirectory = _path.default.join(os.homedir(), 'Library', 'Caches'); else if (process.platform === 'win32') cacheDirectory = process.env.LOCALAPPDATA || _path.default.join(os.homedir(), 'AppData', 'Local'); else throw new Error('Unsupported platform: ' + process.platform);
                                                                                                                                                                                                                                                                                                                                                                                             ^

Error: Unsupported platform: freebsd
    at /home/ukuma/uptime-kuma/node_modules/playwright-core/lib/server/registry/index.js:258:388
    at Object.<anonymous> (/home/ukuma/uptime-kuma/node_modules/playwright-core/lib/server/registry/index.js:270:3)
    at Module._compile (node:internal/modules/cjs/loader:1369:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1427:10)
    at Module.load (node:internal/modules/cjs/loader:1206:32)
    at Module._load (node:internal/modules/cjs/loader:1022:12)
    at Module.require (node:internal/modules/cjs/loader:1231:19)
    at require (node:internal/modules/helpers:179:18)
    at Object.<anonymous> (/home/ukuma/uptime-kuma/node_modules/playwright-core/lib/server/index.js:84:17)
    at Module._compile (node:internal/modules/cjs/loader:1369:14)

The issue is clear: Playwright does not support FreeBSD, causing the operation to fail. At this point, there are two options:

Hope that Microsoft decides to support FreeBSD, which is unlikely in the short term.
Trick Playwright into thinking FreeBSD is supported, which may break some related functionality, but it is a small price to pay to have all other features of Uptime-Kuma.

I will use the second method. Open the indicated file (in this case, /home/ukuma/uptime-kuma/node_modules/playwright-core/lib/server/registry/index.js) and modify the indicated line (in this case, 258). Change the line if (process.platform === 'linux') to if (process.platform === 'freebsd') and save it.

Uptime-Kuma will then be able to import the necessary libraries and function, listening on port 3001.

At this point, it will be possible to use it normally, as well as use pm2 to start and manage it automatically, but this goes beyond the scope of this article.

Bridging Networks Across VPS with Wireguard and VXLAN on FreeBSD

Stefano Marinelli — Mon, 15 Jul 2024 08:41:00 +0200

Introduction

In today's interconnected world, system administrators often face the challenge of managing services across multiple Virtual Private Servers (VPS). This article describes an advanced networking setup that allows you to bridge networks between two VPS instances using Wireguard and VXLAN on FreeBSD. This configuration is particularly useful when you need to distribute services across different providers or when you want to leverage the strengths of multiple hosting environments.

Background

At BSD Cafe, we utilize various VPS instances to provide our services. The two main ones are:

A publicly accessible VPS that hosts the reverse proxy and all firewall rules for packet routing.
A larger VPS on a physical host I own, which is not directly exposed to the internet and doesn't have a public IP address.

Most of the jails hosting BSD Cafe services are distributed between these two VPS instances. Occasionally, I need to move services between them for performance reasons or to manage updates efficiently.

To facilitate this flexibility, I've always maintained a bridge on each VPS. Initially, I used Zerotier to establish a connection between these bridges, allowing them to communicate as if they were part of a single, large network.

The New Setup: Wireguard and VXLAN

While the Zerotier setup worked, I decided to switch to a more streamlined solution using Wireguard and VXLAN. Here's why:

Performance: Wireguard offers excellent performance with low overhead.
Simplicity: The configuration is straightforward and easy to maintain.
Security: Wireguard provides strong, modern cryptography.

I had already prepared a Wireguard connection between the two servers from the beginning. Since only one of the servers is publicly accessible, I set up one to only accept connections and the other to connect directly to the public IP of the first, with a 20-second keepalive (which is generally not necessary due to the high traffic between the jails).

To complete the setup, I added two VXLAN interfaces on the VPS instances, added these interfaces to the local bridges, and immediately, packets started flowing between the networks.

Step-by-Step Implementation

Follow these instructions to create a bridge between two different networks using Wireguard and VXLAN on FreeBSD. While I use this setup to connect jails at BSD Cafe, you can use it for various purposes, such as bridging different VM (bhyve) instances across providers.

Prerequisites

Wireguard is now an integral part of FreeBSD, so you no longer need to compile a module or use the Go version. However, we'll use the "wireguard-tools" scripts as they provide the useful "wg-quick" command.

Start by installing the wireguard-tools package on both servers:

pkg install wireguard-tools

Configuration

Server 1 (Public IP)

Generate the Wireguard keys:

wg genkey | tee /dev/stderr | wg pubkey | grep --label PUBLIC -H .

This command will output a private key and a public key. Note down the public key as you'll need it to configure the client.

Let's also add a PSK; it's optional but will increase the security of the entire setup.

wg genpsk

Create a new file /usr/local/etc/wireguard/wg0.conf:

[Interface]
## Default port is 51820 - feel free to change it
PrivateKey = <the private key from the previous command>
ListenPort = 43671
Address = 10.77.0.1/24

PostUp = /sbin/ifconfig vxlan create vxlanid 42 vxlanlocal 10.77.0.1 vxlanremote 10.77.0.2 inet 10.77.1.1/24
PostUp = /sbin/ifconfig bridge0 addm vxlan0 up
PostDown = /sbin/ifconfig vxlan0 destroy

[Peer]
PublicKey = <the other peer's public key>
#If publicly exposed, you can specify the peer ip address/port
#Endpoint = <public_ip>:<port>
AllowedIPs = 10.77.0.2/32
PresharedKey = <the PSK from the previous command>

Modify /etc/rc.conf and add:

wireguard_interfaces="wg0"
wireguard_enable="YES"

Start Wireguard and the VXLAN endpoint:

wg-quick up wg0

Server 2 (Behind NAT)

Generate the Wireguard keys as before.
Create /usr/local/etc/wireguard/wg0.conf:

[Interface]
PrivateKey = <the private key from the previous command>
Address = 10.77.0.2/24

PostUp = /sbin/ifconfig vxlan create vxlanid 42 vxlanlocal 10.77.0.2 vxlanremote 10.77.0.1 inet 10.77.1.2/24
PostUp = /sbin/ifconfig bridge0 addm vxlan0 up
PostDown = /sbin/ifconfig vxlan0 destroy

[Peer]
PublicKey = <the other peer's public key>
Endpoint = <public_ip>:<port>
AllowedIPs = 10.77.0.1/32
PresharedKey = <the PSK from the previous command>
PersistentKeepalive = 20

Modify /etc/rc.conf as before.
Start Wireguard and the VXLAN endpoint:

wg-quick up wg0

Verifying the Connection

To check if the connection is established, run the wg command on either host. This will show you the connection status, the last handshake, and the data transferred.

You can also try pinging the other host's Wireguard and VXLan interface IP address (in this example, 10.77.0.1 or 10.77.0.2 and 10.77.1.1 or 10.77.1.2).

Conclusion

This setup allows the two VXLAN interfaces, inserted into the local bridge, to enable packet transit through Wireguard. This facilitates free passage between the two hosts, effectively creating a single, unified network across your VPS instances.

This configuration is particularly useful for: - Distributing services across different providers - Leveraging both public-facing and private VPS instances - Creating flexible, scalable network architectures

By using Wireguard and VXLAN, you get the benefits of strong encryption, high performance, and the ability to create complex network topologies across physically separate servers.

Remember to always keep your systems updated and regularly review your network configuration to ensure it meets your evolving needs and security requirements.

From Cloud Chaos to FreeBSD Efficiency

Stefano Marinelli — Thu, 04 Jul 2024 08:41:00 +0200

Introduction

A few months ago, a client asked me to take care of their Kubernetes cluster (hosted on AWS and GCP). In their opinion, the costs were exorbitantly high for relatively simple and lean websites. Sure, they had many visits, but nothing too excessive development-wise.

I kindly declined. Unfortunately, their situation is all too common these days: they hired developers accustomed to working that way, convinced that a system administrator is now unnecessary because "the cloud has infinite potential." They were used to considering optimization as secondary because "we have infinite power" (and this is already a spoiler for the ending).

Being open to dialogue and new experiences, they asked for my opinion on the matter. We talked for a while, and I explained that, in my view, for the type of setup they had (standard, with various replicas and variants, but primarily based on two platforms), it didn't make sense. I saw it as complicating things. An over-engineering of something simple. Like taking a cruise ship to cross a river.

They then asked me to create something simple that would serve as a development server and for backups, to understand what kind of solution I had in mind.

The Solution

So, I started building everything. I began with FreeBSD 13.2-RELEASE, but in the meantime, 14.0-RELEASE came out, so that’s the version I delivered.

I installed the operating system on a physical server, leased from one of the main European providers. Benefiting from one of their auctions (good deals can be found on weekends), they found a sufficiently powerful machine, with 128GB of RAM, 2 NVMe drives of 1TB each, and two spinning disks of 2TB each for less than 100 euros per month. They also took another, less powerful one for additional backups and to back up the first one.

Implementation

I decided to keep the host as clean as possible and concentrated the services in jails (managed by BastilleBSD) and VMs. The machine was divided as follows:

A series of bridges - to be used for different projects. Jails of the same project and/or type use the same bridge and can communicate with each other, sharing some resources (MariaDB, etc.).
A bhyve VM with Alpine Linux - in my opinion, the best distribution for running Docker containers. Do we really need systemd just to launch Docker? They mainly use it as a pre-production test bench, connected via VPN to their company LAN. It is the core of their "online" development, i.e., outside their computers. It has 32GB of RAM, 200GB of disk (obviously bhyve is configured with NVMe drivers), and 4 cores assigned.
A VNET jail with a reverse proxy (nginx) - they know how to modify virtual hosts and generate certificates with certbot, pointing to the underlying jails.
A series of "empty" VNET jails, to be cloned, for each type of setup (they mainly have CMS based on WordPress and Laravel, so with all dependencies inside - nginx, php, redis, etc. except the databases).
A VNET jail with MariaDB installed, to be cloned, to be attached to different projects as needed.
zfs-autobackup performs local snapshots, keeping: one every 15 minutes for 3 hours, one per hour for 24 hours, one per day for 3 days.

Backups are also performed using zfs-autobackup and, in case of disaster recovery in rapid times, a zfs-send (and corresponding zfs-receive) every 10 minutes on another machine (the other, smaller one, also taken at auction), with the same bridges, firewall rules, BastilleBSD, and bhyve installed - ready to start in case of disaster. Being a test server, we didn't consider to implement a proper HA - at the moment, it wouldn't make sense.

They also have another job with zfs-autobackup that performs an additional backup on a server (Debian in their offices). Safe data, in my opinion, are those in storage under your b...ench.

I delivered everything to them and gave a brief course to the more experienced devs on how to manage things. No explanation on the Alpine Linux VM, but I showed them the jails, how to clone, configure, and manage them.

Real-world Testing

I didn't hear from them anymore. After a few weeks, one of the devs contacted me urgently because a junior unfortunately made a mistake and deleted an entire project from one of the jails. I explained that the local snapshots were restorable with a command, and he was thrilled. He restored both the development jail and the one with the database made two minutes before the "mishap" and they restarted immediately.

I realized that this event would change some of their procedures and criteria.

I hadn't heard from anyone for months. This morning, I received a call from their manager, whom I hadn't heard from since the beginning, and he told me how things had been going these months.

Lessons Learned

First, this person has good communication and commercial skills but little technical background. He is open-minded and tends to study carefully what is proposed to him. He doesn't discard any solution a priori, without having touched its pros and cons.

They had leased servers with cPanel and were inserting their content inside them. The devs who arrived a few years ago suggested making a technological transition, eliminating these "obsolete" servers and "outdated" methodologies, pushing everything to the cloud and containerizing everything. When we first talked, he told me how they were "lucky to make that transition because their load had increased enormously and the old servers probably wouldn't have handled the load", instead autoscaling saved them. I had some reservations about autoscaling without particular controls, but clearly, I cannot impose my choices on others.

To cut a long story short: seeing what happened with that junior dev's mistake (and the simplicity with which it was possible to restart immediately), they decided to increase the use of FreeBSD jails and reduce, at least on secondary loads, the use of their Cloud managed with Kubernetes. As they transitioned to jails, however, they noticed some slowdowns. These slowdowns worsened day by day. According to the devs, it would have been appropriate to go back to having, again, autoscaling ("we need moar powaaaaar!!!") but, fortunately, their boss decided to investigate carefully. They realized that these workloads (based on Laravel) were storing sessions on files. Over time, these millions of files (several gigabytes per day) slowed everything down because, for specific operations, Laravel scanned the entire directory. In other words, on the "cloud," they needed much more power than necessary (and much more disk space, but that was cheaper) to carry this load, which was, in fact, unnecessary. After realizing this, they moved the sessions to Redis. Needless to say, everything became extremely faster, even compared to the previous setup on Kubernetes and autoscaling.

At that point, it was clear that one of the problems with their setup is (as often happens) poor optimization. Today, there's a tendency to rush, "throw in" functions, features, libraries, plugins, etc. without considering the interactions and consequences. If it works, it's fine. Even if it increases computational complexity exponentially just to, for example, change the color of an icon (absurd example, but to give an idea).

They then started moving even the main Laravel workloads (thanks to the optimization implemented). At this point, they began moving some of the WordPress sites even though they were extremely concerned. In the cluster, every day, at fairly irregular intervals, the load would rise and everything would slow down until autoscaling started scaling up to the imposed limits. CPU at 100% on all containers, and the devs noticed that the load came from a series of "php" processes. Recreating the containers helped for some minutes, but did not solve the problem.

To their great surprise, all this did not happen on the FreeBSD jails. The load was significantly lower, without any of these spikes. Satisfied, they decided to use this as their final setup. One of the devs, however, wanted to get to the bottom of it and decided to run a test: he moved some of these WordPress sites to the Alpine VM, on Docker. At that point, the spikes resumed, saturating the CPU of the Alpine machine.

Without going into details, they eventually realized that there was a vulnerability in one (or more) of the many plugins installed on the WordPress sites, which was being exploited to inject a process, probably a cryptominer. The name given to the process was "php" - so the devs, not being system experts, did not worry about understanding better whether it was really php or another process pretending to be it. On FreeBSD, all this did not happen because the injected executable could not run - there was no Linux compatibility activated on the server.

Until then, they considered these (expensive) spikes as organic and did not worry too much about them. Paying to have their friendly intruders mine.

Conclusion

They asked me to help, as much as possible, to move other services to FreeBSD. It won't be easy, probably we will need to use bhyve a lot, but they decided that this is the platform they want to focus on in the coming years.

Undoubtedly, this is a success story of FreeBSD and, indirectly, of correct and careful management of one's resources. Too often today, there is the superficial belief that the cloud, with its "infinite" resources, is the solution to all problems. And that Kubernetes is the best solution for everything. I, on the other hand, have always believed that there is the right tool for everything. You can hammer a nail with a screwdriver, but it's not the most suitable and efficient tool.

Today they spend about 1/10 of what they used to spend before, they have more control over their data and the tools they use. Undoubtedly, all this was also caused by poor optimization and control by those who manage the infrastructure, but the question is: how often do people decide that, in the end, it is okay to spend more (especially if it is someone else's money) rather than go crazy for hours behind such a situation? While having defined and limited resources (albeit elevated) poses different problems - but of optimization. And in the age of energy and resource savings, it might be wise to give more importance to optimization.

Abundance led to waste.

Blocking Access from or to Specific Countries Using FreeBSD and pf

Stefano Marinelli — Sun, 16 Jun 2024 07:40:45 +0000

In recent times, there has been an exponential increase in malicious (or simply rude) traffic from specific countries. Alternatively, sometimes we simply do not need visitors from other parts of the world accessing our server for various reasons.

On FreeBSD, this operation is very simple, and I have been using a reliable and secure system to manage it automatically for a long time. Of course, as with all geolocation blocks, there is never certainty about the result, as sometimes certain IP blocks officially belong to one country but are actually used by another, or users can resort to VPNs to bypass these types of blocks. However, it remains a valid method to filter out unwanted traffic, especially when it comes to rogue bots that do not respect the robots.txt file and bombard our machines with repeated requests, generating real DDoS attacks.

Installation of ipdbtools

First, install the ipdbtools package (http://www.freshports.org/sysutils/ipdbtools):

pkg install ipdbtools

Next, download the updated global lists:

/usr/local/bin/ipdb-update.sh

Configuring pf

Then, modify the pf configuration. To do this, add the following line at the beginning of the filtering part of your firewall configuration (/etc/pf.conf):

block drop log quick from <blocked_countries>

At this point, simply type:

service pf reload

to reload the pf configuration and start considering the values in the <blocked_countries> table.

If you do not already have a pf.conf because the only requirement was this, simply insert this line in an empty pf.conf and then run:

service pf enable
service pf start

Updating the Blocked Countries List

Next, create a file (in my case, I called it /usr/local/sbin/update_blocked_countries.sh) with content similar to the one below. Replace "CC" with the country codes you want to block. For example, to block France, Germany, and Italy, insert "FR:DE:IT".

UPDATE: pf seems to have difficulty processing very large lists, resulting in errors. For this reason, I modified the following script to insert the lists in 'chunks,' in order to prevent the procedure from failing.

#!/bin/sh

# Original file containing the IP addresses
original_file="/var/db/blocked_countries.txt"

# Generate the blocked countries file
/usr/local/bin/ipup -p -t CC > $original_file

# Number of records per batch
batch_size=10000

# Temporary file for the current batch
temp_file="/tmp/blocked_countries_temp.txt"

# Initialize the line counter
line_count=0

# Function to add a batch of records
add_batch() {
    echo "Adding records from $temp_file to pf table..."
    /sbin/pfctl -t blocked_countries -T add -f "$temp_file"
    if [ $? -ne 0 ]; then
        echo "Error adding records from $temp_file. Exiting."
        exit 1
    fi
    # Empty the temporary file
    > $temp_file
}

# Replace the table with an empty file to avoid conflicts
echo -n > /tmp/empty_blocked_countries.txt
/sbin/pfctl -t blocked_countries -T replace -f /tmp/empty_blocked_countries.txt

# Read the original file line by line
while IFS= read -r line; do
    # Add the line to the temporary file
    echo "$line" >> "$temp_file"
    line_count=$((line_count + 1))

    # If we've reached the batch size, add the records and reset the counter
    if [ $line_count -ge $batch_size ]; then
        add_batch
        line_count=0
    fi
done < "$original_file"

# Add any remaining records
if [ $line_count -gt 0 ]; then
    add_batch
fi

echo "All records added successfully."

Make it executable:

chmod a+rx /usr/local/sbin/update_blocked_countries.sh

Running the command /usr/local/sbin/update_blocked_countries.sh will show the status of the operation, for example:

314159 addresses added.

This means that the list has been generated for the inserted country codes, and pf has updated the <blocked_countries> table.

It is advisable to update the lists occasionally since IP ranges "move" from one country to another.

Automating with Cron

We can use cron to perform this task. Edit the file /etc/crontab by adding a line like:

55 9 * * *    root    /usr/local/bin/ipdb-update.sh > /dev/null 2>&1 && /usr/local/sbin/update_blocked_countries.sh

In this example, at 9:55 every day, the databases will be updated, lists generated, and fed to pf.

This setup will also work correctly on a read-only FreeBSD system on UFS, as described in a previous article. The only precaution in this case is to ensure the crontab runs at every boot since, on every machine restart, the contents of /var will be erased and recreated from scratch. Therefore, also add a line like:

@reboot    root    /usr/local/bin/ipdb-update.sh > /dev/null 2>&1 && /usr/local/sbin/update_blocked_countries.sh

Additionally, in a read-only system, you need to add a mount for tmpfs for the /usr/local/etc/ipdb/IPRanges directory in the /etc/fstab file to make it writable:

tmpfs /usr/local/etc/ipdb/IPRanges/     tmpfs rw 0 0

Make your own E-Mail server - Part 2 - Adding Webmail and More with Nextcloud

Stefano Marinelli — Thu, 21 Mar 2024 07:15:00 +0000

In the first part of this series, I detailed how to install and configure a complete mail server within a FreeBSD jail. While accessible via IMAP and SMTP, I intentionally did not cover installing a webmail interface.

Installing a "simple" webmail is not complex, but nowadays, we expect more integrated solutions, including contact books (synchronizable with computers and mobile devices) and calendars. For this reason, in this second part, I will be utilizing Nextcloud.

Nextcloud represents a comprehensive groupware solution today, extendable through "apps" that can be installed directly from the administration menu. This setup offers numerous advantages, including authenticating Nextcloud users through the IMAP server (thus, a mail server user will automatically have a Nextcloud account, and password changes on the mail will reflect on Nextcloud too), the ability to use integrated Contacts (also in webmail) and Calendars, and the option to attach or save attachments directly in Nextcloud.

We will use the reverse proxy configured in part 1, which will "pass" connections to Nextcloud. First, a domain name must be defined, which will be used to connect to the server. In this example, I will use "nextcloud.example.com."

To do this, enter the proxy's jail:

bastille console nginx-proxy

Modify the file /usr/local/etc/nginx/nginx.conf, adjust the worker_processes (so nginx will start a number of processes equal to the number of cores on the machine it's running on), and create the virtual host for Nextcloud:

worker_processes auto;

In the http block, change the maximum file size limit and create the virtual host on http - we will generate the https block and related certificates later:

client_max_body_size 0;

server {
    listen 80;
    server_name nextcloud.example.com;

    location / {
        proxy_buffering off;
        proxy_http_version 1.1;
        fastcgi_hide_header X-Powered-By;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto https;
        proxy_pass https://192.168.123.5;
        proxy_connect_timeout   10m;
        proxy_send_timeout      10m;
        proxy_read_timeout      10m;
    }

    location /.well-known/carddav {
        return 301 $scheme://$host/remote.php/dav;
    }

    location /.well-known/caldav {
        return 301 $scheme://$host/remote.php/dav;
    }
}

The proxy_pass is in https because, later on, Nextcloud will respond in https. Although not necessary, this approach is simpler given the method used to install Nextcloud. Moreover, it allows for the reverse proxy to be on a different server than Nextcloud, without the need for a VPN.

Now, enable and start nginx:

service nginx enable
service nginx start

After setting the listener to port 80, it's now appropriate to generate the certificate.

A simple command will request the certificate and modify the nginx configuration to accept connections over https:

certbot --nginx -d nextcloud.example.com

Upon completion of this command, the virtual host for https will be correctly generated, and an automatic redirect will be set up for all connections arriving on http to be "moved" to https.

Exit the jail, and it's time to create the jail for Nextcloud.

bastille create -B nc 14.2-RELEASE 192.168.123.5/24 bridge0

Enter the jail and add the gateway in rc.conf:

defaultrouter="192.168.123.1"

After restarting the jail, it will now be possible to install Nextcloud.

FreeBSD has a fairly updated package, but when it comes to web applications, I prefer to have more control. Usually, I install everything manually, but for Nextcloud, I found an interesting script made by Gibran Khan that takes care of everything, i.e., installing Apache, PHP, ClamAV, all dependencies. It will also configure Nextcloud and make it (almost) ready to use.

Since Gibran's script was structured for version 27, used PHP 8.2, and had other small things that were not perfect for my use, I decided to modify it slightly, mainly to support Nextcloud 28 and PHP 8.3. Therefore, the description that follows will be related to my fork available here: https://github.com/draga79/NextCloudOnFreeBSD

So, enter the "nc" jail, download the script, and start with the configuration:

bastille console nc
fetch https://github.com/draga79/NextCloudOnFreeBSD/archive/refs/heads/release.zip
unzip release.zip
cd NextCloudOnFreeBSD-release
./pre-install.sh

A configuration file will be generated that we need to modify. The file is install.conf.

You will certainly need to change the HOST_NAME (inserting the one you have chosen, in this case, "nextcloud.example.com") and, unless you are in Italy, I would also suggest changing the COUNTRY_CODE and TIME_ZONE.

Once completed, you can proceed with the actual installation:

./install.sh

At the end of the installation, the login data will appear (which will also be saved in a file named nextcloud.example.com_reference.txt). The installer will also set up a cron job executed by the user "www", which will handle, every 5 minutes, the background operations required for Nextcloud to function properly. In some cases, I've noticed that it's necessary to modify the cronjob, save it, and then it will start working correctly. If that's the case, simply open it with crontab -u www -e, save, and exit.

Now, modify the Nextcloud configuration file, /usr/local/www/nextcloud/config/config.php, and:

Replace overwrite.cli.url with your domain (https://nextcloud.example.com).
Add the proxy to the trusted proxies (if not present, insert the entire block):

 'trusted_proxies' =>
  array (
    0 => '192.168.123.2',
  ),

Modify the file /usr/local/etc/apache24/httpd.conf and change the Listen directive to:

Listen *:443

Restart Apache:

service apache24 restart

To simplify things, it's advisable to make a modification: in /etc/hosts, add a line to ensure the mail server can be reached via LAN through its domain name. Therefore, add the line:

192.168.123.3 mail.example.com

In this manner, when Nextcloud attempts to connect to the mail server, it will do so directly over the LAN. This direct connection is essential for the IMAP authentication plugin, as it verifies that the SSL certificate of the mail server matches the hostname being called.

Now log in to Nextcloud with the admin credentials that the installer provided you earlier. Nextcloud is now installed and initialized. We need to perform two operations - install the Snappymail app (if you want to use it, but generally, I recommend it over the integrated mail client in Nextcloud) and enable IMAP authentication to avoid having duplicate users (i.e., mail user and Nextcloud user).

Click on the menu at the top right (the big A) and go to Apps. Under "Integration," you will now be able to install "Snappymail" and "External User Authentication," which, however, is currently listed as "Untested" because it has been tested for a previous version of Nextcloud (but it works on version 28 too). You will have to click it twice, because the first click will only enable untested apps.

Once the apps are installed, go to "Administration Settings" and "Additional Settings," then to the Snappymail administration panel. Now add your domain, so click Add Domain. The domain name will be that of your emails (example.com). Therefore, in the IMAP server field, enter the name of the mail server (mail.example.com), and set the security to STARTTLS. Do the same for SMTP and SIEVE.

Perform the test (using the username and password of an existing and valid mailbox) and, once everything is validated, save.

You will now be able to log in into your webmail - in the top bar of Nextcloud there will be two icons, one for the integrated webmail client and the other for Snappymail. The integrated webmail client has some features that are not present in Snappymail (such as delayed sending of messages), so it may be useful to keep both active. Currently, Snappymail will require a username and password every time you connect (even if you are already logged into Nextcloud), but it's possible, in user settings, to enable automatic login once authenticated in Nextcloud.

Now is the time to configure IMAP authentication. Return to the Nextcloud jail and modify the Nextcloud configuration file, i.e., /usr/local/www/nextcloud/config/config.php. Add a block like this:

'user_backends' => array(
    array(
        'class' => '\OCA\UserExternal\IMAP',
        'arguments' => array(
            'mail.example.com', 993, 'ssl', 'example.com', false, false
        ),
    ),
),

'example.com' in the fourth field is used to limit the authentication scope to that domain. More information can be found on the official plugin page.

After saving, you will be able to log out as admin and log in with your mail account. Unless there are errors, you will be catapulted into your new user, created and authorized through IMAP authentication.

Congratulations, you are now able to use Nextcloud by logging in with your email credentials and also use it as a basis for webmail, leveraging the contacts module for the email address book as well.

Nextcloud can offer intriguing prospects for everyday use. In addition to its seamless integration with mobile devices, tools like 'Nextcloud Talk' and 'Note' will enable users to avoid the use of external solutions and maintain control over their data.

It will also be possible to synchronize contacts and calendars with mail clients (like Thunderbird) or the address books of various stationary and mobile devices. The procedure varies depending on the device used, but it's just a matter of synchronizing via CalDAV and CardDAV. Alternatively, you can set up Z-Push and use the Microsoft Exchange ActiveSync (EAS) protocol, but this will be described in part 3 of the series.

Addendum: Adding Antivirus Support to Spam Filtering

The automatic installation script, alongside Nextcloud, installed other components including ClamAV to check uploaded files in real time. We can leverage this installation (since ClamAV consumes significant resources, it makes no sense to install two instances) and integrate this instance of ClamAV with rspamd, to also perform antivirus checks on emails.

Modify /usr/local/etc/clamd.conf

Uncomment:

TCPSocket 3310

Restart clamd to listen on port 3310 as well:

service clamav-clamd restart

Return to the mail jail:

bastille console mail

Create the file /usr/local/etc/rspamd/local.d/antivirus.conf with the following content:

clamav {
  # If set, force this action if any virus is found (default unset: no action is forced)
  action = "reject";
  message = '${SCANNER}: virus found: "${VIRUS}"';
  # Scan mime_parts separately - otherwise, the complete mail will be transferred to AV Scanner
  #attachments_only = true; # Before 1.8.1
  #scan_mime_parts = true; # After 1.8.1
  # Scanning Text is suitable for some av scanner databases (e.g., Sanesecurity)
  #scan_text_mime = false; # 1.8.1 +
  #scan_image_mime = false; # 1.8.1 +
  # If `max_size` is set, messages > n bytes in size are not scanned
  #max_size = 20000000;
  # symbol to add (add it to metric if you want non-zero weight)
  symbol = "CLAM_VIRUS";
  # type of scanner: "clamav", "fprot", "sophos", or "savapi"
  type = "clamav";
  # If set true, a log message is emitted for clean messages
  #log_clean = false;
  # Prefix used for caching in Redis: scanner-specific defaults are used. If Redis is enabled and
  # multiple scanners of the same type are present, it's important to set prefix to something unique.
  #prefix = "rs_cl_";
  # For "savapi" you must also specify the following variable
  #product_id = 12345;
  # servers to query (if port is unspecified, scanner-specific default is used)
  # can be specified multiple times to pool servers
  # can be set to a path to a unix socket
  servers = "192.168.123.5:3310";
  # if `patterns` is specified virus name will be matched against provided regexes and the related
  # symbol will be yielded if a match is found. If no match is found, the default symbol is yielded.
  patterns {
    # symbol_name = "pattern";
    JUST_EICAR = '^Eicar-Test-Signature$';
  }
  # In version 1.7.0+ patterns could be extended
  #patterns = {SANE_MAL = 'Sanesecurity\.Malware\.*', CLAM_UNOFFICIAL = 'UNOFFICIAL$'};
  # `whitelist` points to a map of signature names. Hits on these signatures are ignored.
  whitelist = "/usr/local/etc/rspamd/local.d/antivirus.wl";
}

Restart rspamd in the “mail” jail and, from this point on, viruses will be automatically rejected.

Make your own E-Mail server - Part 1 - FreeBSD, OpenSMTPD, Rspamd and Dovecot included

Stefano Marinelli — Fri, 08 Mar 2024 07:15:00 +0000

The main power of the Internet has always been one: decentralization.

Ever since I've been able to, I've always managed my email boxes independently and provided mail hosting services to my clients. Over the years, things have become increasingly complex: on one hand, there's been a disproportionate increase in spam, and on the other, big players (like Google and Microsoft) have tried to gain a lot of ground in managing these services. Private users can obtain many free services, and business users no longer need to worry about the underlying infrastructure. However, in my opinion, the price to pay is very high: the loss of ownership of one's data.

These operators, in fact, use full access to our emails to improve their services or analyze us with the aim of selling us advertisements.

As often happens in these cases, many users appreciate this type of approach and have switched to the services of big players, causing a progressive increase in the level of influence that these companies can have on something free and decentralized like email.

In my opinion, it still makes sense to manage one's own mail servers. Standards evolve, and it is therefore appropriate to follow innovations, adapt to best practices, and secure one's services and users.

Over the last ten years, I have also used and installed various Zimbra OSE servers. Aside from a few minor issues, the setup has proven stable and reliable. Recently, the company that develops this excellent tool has decided to no longer provide packages for the Open Source version. They can be generated with scripts provided by some (great!) users, but they cannot be a solid and stable base in the long term.

I have therefore decided to gradually return to a modular, adaptable, customizable, and updatable mail host setup without worries or headaches. Such a system can include any kind of service (including webmail, integration with caldav/carddav, etc.) and it will be possible to disable what is not strictly necessary. Safer and more secure.

This will be the first in a series of articles, and at the end of this reading, you will have a secure, modern, reliable, and modular mail server. The instructions are designed for FreeBSD and related jails but with very few modifications can be applied to any BSD as well as Linux or other similar operating systems.

Setup Planning

Over the years, I've used many different SMTP servers. Originally, the good old Sendmail. Subsequently, and for many years, I used Exim, also because it was the default system in Debian. I then migrated almost all setups towards Postfix, probably the most widespread smtp server on the net, and I never had any particular problems. In recent years, I have decided to use, when possible, the excellent OpenSMTPD. Being based on OpenBSD and easily installable on other OSes, it shares the primary concepts of security and reliability of OpenBSD as well as the syntax (both in command line and in configuration files) of all other OpenBSD tools. For this type of setup, I will use opensmtpd.

The choice to use FreeBSD (rather than OpenBSD) stems from two main factors:

The possibility of dividing into jails, physically separating the various services and minimizing dependencies.
The possibility of using ZFS, with all the advantages it brings: the ability to take snapshots, the ability to perform backups quickly and efficiently, and the ability to migrate entire jails without particular problems.

The other tools that will be used are:

Rspamd - one of the best integrated systems for checking incoming mail. Customizable, adaptable, and modular, it also provides interesting monitoring systems on the type of incoming and outgoing mail.
Redis - which will be used by rspamd to manage its "memory".
Dovecot and Sieve - for delivering mail to local mailboxes as well as allowing remote email consultation (via imap) and enabling the setting of rules thanks to Pigeonhole Sieve.

The users of the mail server will not be "physical" users of the server itself, but virtual users, totally unrelated to system users.

This article will be long and detailed. The process may seem complex at first glance, but in reality, it's simpler than one might think. In the classic Unix philosophy, each component performs a task and does it well. Configuring the individual components separately will ensure greater scalability, reliability, and resilience to updates as well as simpler debugging compared to all-in-one solutions (like many of those found in convenient, ready-to-use docker-compose.yaml files).

Installation and Configuration

I won't describe the FreeBSD installation and configuration procedure. In my case, I activated a VPS on Hetzner - in this situation, the smallest of the available ARM servers. Yes, it's possible to efficiently manage a mail server for several users on a machine costing less than 4 euros a month. ZFS can be useful, but it's not necessary for this type of setup.

The first step is therefore to choose a location to install everything. The main condition is that the VPS/server/host must have at least one static public IP and, in 2024, I now consider it necessary to have (at least) one IPv6 address. They must also be able to send/receive mail on port 25, which is not guaranteed and not all VPS providers allow. Hetzner, for example, blocks port 25 for new users.

Once you have identified and acquired the server on which to install everything, it's appropriate to check that the IPs (both IPv4 and IPv6) to be used are not on any of the many blocklists/blacklists. My experience indicates that at least 60% of the IPs assigned to me are on one or more of these lists, so the first step is to request delisting, to ensure proper operation when the server is ready.

To Encrypt or Not to Encrypt?

A subsequent assessment is whether to use encryption for the data. Connections to and from the mail server will always be encrypted, but it's also possible to encrypt the data at rest. In the case of FreeBSD, it's possible to enable encryption for the entire installation (the entire virtual disk will be encrypted, and ZFS (or UFS) will be created on top of it) or, in the case of using ZFS, it will be possible to encrypt only the root dataset of the jails, the one that will be used by BastilleBSD. In the first case, it will be necessary to enter the password at every boot of FreeBSD, therefore connect via console at every restart and enter the password. In the second case, FreeBSD will be able to boot but it will be necessary to connect and enter the keys before mounting the BastilleBSD ZFS dataset. None of these solutions will prevent unauthorized access to emails in case of machine compromise. The main advantage will be "only" not to save the emails in clear on disks which, in the case of a VM, will still be shared and managed by third parties. In the case of physical servers, however, things will undoubtedly be different.

Encrypting the entire disk must be done directly during the FreeBSD installation phase, very simply. As for the option to encrypt only the BastilleBSD dataset, you can proceed later, with a command similar to:

zfs create -o encryption=on -o keylocation=prompt -o keyformat=passphrase zroot/bastille

at every reboot, it will be necessary to enter the password and mount the datasets:

zfs load-key -r zroot/bastille
zfs mount -a

Configuring FreeBSD and BastilleBSD

After the installation of the OS, the first step is to configure IPv6 on the VPS. In the case of Hetzner, unfortunately, they only provide a /64, so it will be necessary to segment the assigned network. In this example, it will be divided into /72 subnetworks - to find valid subclasses, it will be possible to use a calculator.

The /etc/rc.conf file should have entries similar to:

ifconfig_vtnet0="DHCP"
ifconfig_vtnet0_ipv6="inet6 2a01:4f8:cafe:cafe::1 prefixlen 72"
ipv6_defaultrouter="fe80::1%vtnet0"

In short, keep the base address assigned by Hetzner, but change the prefix length to 72 - thus giving the possibility of having other networks available.

It is now necessary to enable forwarding for IPv4 and IPv6. Add these lines to the /etc/sysctl.conf file:

net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1

After reboot, test it:

ping6 google.com

If everything has been configured correctly, the ping will be executed and google.com will reply.

It will also be necessary to set up reverse DNS, otherwise many mail servers will reject our emails. This must be done both for the IPv4 address and the designated IPv6 address (within the class - an operation that can also be done after creating the jails, but before starting to send email messages). Also, create the correct A and AAAA records on the DNS for the "mail.example.com" domain.

The FreeBSD installer does not automatically update the operating system, so it is appropriate to do so now:

freebsd-update fetch install

It seems, moreover, that there is still an old bug still present and manifesting when, on a FreeBSD server installed in a VM based on KVM (thus also those of Hetzner), routing is performed (as in our case) between VNET jails and host.

Adding this configuration to /boot/loader.conf will solve the problem:

hw.vtnet.X.tso_disable="1"
hw.vtnet.tso_disable="1"
hw.vtnet.lro_disable="1"
hw.vtnet.X.lro_disable="1"
hw.vtnet.csum_disable="1"
hw.vtnet.X.csum_disable="1"

It's time to install and configure BastilleBSD. This handy tool will make managing jails much simpler and more straightforward, and its configuration is well described on the project page.

Bridge and Firewall

For some time now, I've been favoring the use of VNET jails for these types of setups. I believe that having a complete network stack within the jails gives more freedom in configuration, firewall management (applicable also within individual jails), and technical possibilities (such as creating VPNs within the jails themselves, ensuring greater portability).

I suggest creating a bridge on the FreeBSD server and placing all the jails within this bridge. It will be enough to modify the /etc/rc.conf file and change/add:

cloned_interfaces="lo1 bridge0"
ifconfig_bridge0="inet 192.168.123.1 netmask 255.255.255.0"
ifconfig_bridge0_ipv6="inet6 2a01:4f8:cafe:cafe:100::1 prefixlen 72"
ipv6_gateway_enable="YES"
ipv6_activate_all_interfaces="YES"
gateway_enable="YES"

Now let's configure the firewall. In this example, I will use the IPv4 and IPv6 addresses that I will later assign to the jails. Here's an example of a working pf.conf:

ext_if="vtnet0"

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo
set skip on bridge0

table <jails> persist
# IPv4 private address ranges
table <private> const { 10/8, 172.16/12, 192.168/16 }
nat on $ext_if from 192.168.123.0/24 to ! <private> -> ($ext_if:0)
nat on $ext_if from <jails> to any -> ($ext_if:0)
rdr-anchor "rdr/*"
#rdr via ipv4 to mail
rdr pass on $ext_if proto tcp from any to ($ext_if) port { 25, 465, 587, 143, 993 } -> 192.168.123.3
#rdr via ipv4 to nginx-proxy
rdr pass on $ext_if proto tcp from any to ($ext_if) port { 80, 443 } -> 192.168.123.2

block in all
#PASS ICMP
pass in quick proto icmp from any to any
# Pass ICMP on ipv6
pass quick proto ipv6-icmp
pass out quick keep state
antispoof for $ext_if inet
pass in inet proto tcp from any to any port ssh flags S/SA keep state
#Pass ipv6 to mail jail
pass in quick on $ext_if inet6 proto tcp from any to 2a01:4f8:cafe:cafe:100::25 port { smtp, 465, 587, 143, 993 }
#Pass ipv6 to nginx-proxy jail
pass in quick on $ext_if inet6 proto tcp from any to 2a01:4f8:cafe:cafe:100::80 port { 80, 443 }

At this point, it is advisable to reboot the machine, to be sure that everything comes back up in the right way.

Creating the "nginx-proxy" and "redis" Jails

Let's start with the first jail. It will be an nginx reverse proxy. At the moment it will not be used as such but will be useful as a machine for generating certificates and, in the future, to act as a reverse proxy for various internal web services.

bastille create -B nginx-proxy 14.3-RELEASE 192.168.123.2/24 bridge0

Once the jail is created, it will be necessary to modify some configurations. In the jail itself (bastille console nginx-proxy), modify the /etc/rc.conf file and add the IPv4 gateway and IPv6 configurations, i.e., giving the address specified earlier in the pf.conf of the FreeBSD server and as the gateway the IP address of the related bridge:

defaultrouter="192.168.123.1"
ifconfig_vnet0_ipv6="inet6 2a01:4f8:cafe:cafe:100::80 prefixlen 72"
ipv6_defaultrouter="2a01:4f8:cafe:cafe:100::1"

Restart nginx-proxy with bastille restart nginx-proxy, and you will be able to re-enter the jail. At that point, ping google.com and ping6 google.com should work. The jail will then be able to operate, responding on 80 and 443 both in IPv4 (thanks to the NAT configured previously on the FreeBSD server) and in IPv6, being directly connected in routing.

Now install the necessary software:

pkg install -y nginx py311-certbot py311-certbot-nginx

To automatically renew the certificates, add this line to /etc/periodic.conf:

weekly_certbot_enable="YES"

It's now possible to generate the certificate, which we will also use for all other exposed services:

certbot certonly -d mail.example.com

Once the certificates are generated, we will need to make a small change as opensmtpd (which we will install later in another jail) is (rightly) very restrictive on permissions:

chmod -R 400 /usr/local/etc/letsencrypt/live/mail.example.com/

Now let's create a jail for redis. I usually put it in a dedicated jail also in terms of clustering/multi-server setup:

bastille create -B redis 14.3-RELEASE 192.168.123.4/24 bridge0

Also for the redis jail, it's appropriate to set up the gateway, so in the /etc/rc.conf of the jail:

defaultrouter="192.168.123.1"

I don't configure IPv6 as it's not necessary (this jail will only be reachable from the LAN) but it's possible to do so. Once the jail is restarted (bastille restart redis), it will be able to reach the outside:

pkg install -y redis

Not wanting to disable the protected mode of redis and not wanting to open it without authentication, it will be appropriate to modify the /usr/local/etc/redis.conf configuration file and add the password and some options to optimize its use with rspamd:

requirepass cafecafe
maxmemory 512mb
maxmemory-policy volatile-lru

Of course, set a unique password. Also, change the line concerning the bind:

bind *

Now execute:

service redis enable
service redis start

Redis should be active and ready to receive connections.

The "mail" Jail

It's time to create the jail with the main services:

bastille create -B mail 14.3-RELEASE 192.168.123.3/24 bridge0

and, also in this case, it will be necessary to enter the jail and modify some configurations in /etc/rc.conf:

defaultrouter="192.168.123.1"
ifconfig_vnet0_ipv6="inet6 2a01:4f8:cafe:cafe:100::25 prefixlen 72"
ipv6_defaultrouter="2a01:4f8:cafe:cafe:100::1"

Once the jail is restarted, it will be possible to install the necessary packages:

pkg install -y rspamd opensmtpd opensmtpd-filter-senderscore opensmtpd-filter-rspamd dovecot dovecot-pigeonhole

Please note: the opensmtpd-extras- packages are currently broken so we'll compile the table file support:

pkg install autoconf automake git-tiny
git clone https://github.com/OpenSMTPD/table-passwd.git
cd table-passwd
sh bootstrap
./configure
make
make install
cp table-passwd /usr/local/libexec/opensmtpd/
cp table-passwd.5 /usr/local/share/man/man5/

It is now appropriate to give this jail an FQDN, as it will be the name with which it presents itself when connecting to the outside. Just modify /etc/rc.conf:

hostname="mail.example.com"

The hostname should be equivalent to the reverse DNS set earlier, so the machine will present itself with a name that, doing a reverse lookup, will correspond to the IP of origin.

This jail does not contain the certificates that are present in the nginx-proxy jail. There are various methods to share them, such as using rsync, putting them on an NFS share, etc., but the simplest in this case will be to do a bind mount between the two jails, an operation that BastilleBSD can handle automatically.

Create the correct directory in the "mail" jail:

mkdir /usr/local/etc/letsencrypt

Exit the jail, shut it down, and modify the fstab file of the jail (e.g., /usr/local/bastille/jails/mail/fstab) by adding a line like this:

/usr/local/bastille/jails/nginx-proxy/root/usr/local/etc/letsencrypt /usr/local/bastille/jails/mail/root/usr/local/etc/letsencrypt nullfs ro 0 0

Start the jail, and at that point, the directory created earlier should display the directories and certificates from the nginx-proxy jail.

Due to spammers who, year after year, become increasingly aggressive, many mail servers require perfect configuration of the main sender authentication methodologies. Today, the main (and now necessary, under penalty of message non-delivery) are SPF, DKIM, and DMARC.

The first step is to generate a DKIM key for the domain(s) that will send mail from the server we are configuring. This operation can be performed in a few steps, by placing the keys in the /usr/local/etc/mail/dkim directory:

mkdir /usr/local/etc/mail/dkim
cd /usr/local/etc/mail/dkim
openssl genrsa -out example.com.key 2048
openssl rsa -in example.com.key -pubout -out public.key
chmod 0400 example.com.key
chown rspamd *

Since rspamd will take care of signing outgoing messages, it is appropriate that only the related user be able to read the file with the private key.

Once the pair of keys has been created, it will be necessary to configure DNS to provide the public key to mail servers that will request it. The record should look like this:

mail._domainkey   TXT   "v=DKIM1;k=rsa;p=your-public-key-goes-here"

Once the record has been entered and propagated, it will be possible to test its correctness through one of the many websites that offer this service, remembering that the selector we have configured is "mail" (the first part of the record). Obviously, this is a free text, which can be modified at will (but will then need to be specified in the rspamd configuration, later on).

Being in the DNS configuration phase, it will also be appropriate to create SPF and DMARC records:

SPF record:

example.com. IN TXT "v=spf1 a ip4:your.ip.address ip6:2a01:4f8:cafe:cafe:100::25 mx ~all"

DMARC:

_dmarc.example.com. IN TXT "v=DMARC1;p=none;pct=100;rua=mailto:postmaster@example.com"

These records, obviously modified based on the chosen domain, are configured conservatively but suitable for a test. Once the entire setup is in place, it will be possible to handle things in a more restrictive manner.

The mail server users will be "virtual" users (i.e., not system users of the server), so all their mail will be "handled" by a unique user, which needs to be created:

pw user add vmail -u 2000 -d /var/vmail -s /usr/sbin/nologin
mkdir /var/vmail
chown vmail /var/vmail

Let's now create some files and directories, useful for the subsequent configuration:

touch /usr/local/etc/mail/passwd
touch /usr/local/etc/mail/virtuals
mkdir /usr/local/etc/rspamd/local.d

Remove the current /usr/local/etc/mail/smtpd.conf and replace it with content like this:

table passwd passwd:/usr/local/etc/mail/passwd
table virtuals file:/usr/local/etc/mail/virtuals

pki mail.example.com cert "/usr/local/etc/letsencrypt/live/mail.example.com/fullchain.pem"
pki mail.example.com key "/usr/local/etc/letsencrypt/live/mail.example.com/privkey.pem"

filter check_dyndns phase connect match rdns regex { '.*\.dyn\..*', '.*\.dsl\..*' } \
    disconnect "550 no residential connections - Thou shalt not pass"

filter check_rdns phase connect match !rdns \
    disconnect "550 no rdns - Thou shalt not pass"

filter check_fcrdns phase connect match !fcrdns \
    disconnect "550 no FCrDNS - Thou shalt not pass"

filter senderscore \
    proc-exec "/usr/local/libexec/opensmtpd/opensmtpd-filter-senderscore -blockBelow 10 -junkBelow 70 -slowFactor 5000"

filter rspamd proc-exec "/usr/local/libexec/opensmtpd/opensmtpd-filter-rspamd"

listen on 0.0.0.0 tls pki mail.example.com \
    filter { check_dyndns, check_rdns, check_fcrdns, senderscore, rspamd } auth-optional <passwd>

listen on ::0 tls pki mail.example.com \
    filter { check_dyndns, check_rdns, check_fcrdns, senderscore, rspamd } auth-optional <passwd>

listen on 0.0.0.0 port submission tls-require pki mail.example.com  auth <passwd> filter rspamd mask-src

listen on ::0 port submission tls-require pki mail.example.com  auth <passwd> filter rspamd mask-src

listen on 0.0.0.0 port 465 smtps pki mail.example.com  auth <passwd> filter rspamd mask-src

listen on ::0 port 465 smtps pki mail.example.com  auth <passwd> filter rspamd mask-src

action "local_mail" lmtp "/var/run/dovecot/lmtp" rcpt-to virtual <virtuals>
action "outbound" relay helo mail.example.com

match from any for domain example.com action "local_mail"
match for local action "local_mail"

match from any auth for any action "outbound"
match for any action "outbound"

The beauty of OpenSMTPD lies in its simplicity. Essentially, that's all there is to it, but here's an explanation of what this file does:

table passwd and table virtuals define tables for user credentials and virtual domains/users, crucial for authentication and email forwarding/aliasing.
pki lines specify the SSL/TLS certificate and key for encrypted connections.
filter directives apply various checks and filters to incoming connections, including dynamic DNS and reverse DNS validations, sender reputation scoring, and Rspamd filtering.
listen lines configure the server to listen on all IPv4 and IPv6 addresses for incoming SMTP and submission (port 587) connections, applying TLS and authentication as specified.
action lines define actions for email delivery, using LMTP for local delivery based on virtual mappings and setting up relay actions for outbound emails.
match rules determine how emails are processed, either delivered locally or relayed externally based on source, destination, and authentication.

To test the newly inserted configuration, run smtpd -n.

The next step is to configure Rspamd. Rather than modifying the base configuration files, we'll proceed by creating "overrides", placing specific configurations in the directory created earlier (/usr/local/etc/rspamd/local.d):

For Redis configuration in /usr/local/etc/rspamd/local.d/redis.conf:

servers = "192.168.123.4";
password = "cafecafe";

For SPF settings in /usr/local/etc/rspamd/local.d/spf.conf:

spf_cache_size = 1k;
spf_cache_expire = 1d;
max_dns_nesting = 10;
max_dns_requests = 30;
min_cache_ttl = 5m;

Now, configure DKIM signing, which rspamd will handle for every sent email. The selector must match the one entered in the DNS record previously:

In /usr/local/etc/rspamd/local.d/dkim_signing.conf:

domain {
  example.com {
    path = "/usr/local/etc/mail/dkim/example.com.key";
    selector = "mail";
  }
}

To enable support for some "known" phisher lists, create /usr/local/etc/rspamd/local.d/phishing.conf:

# Configuration options can be added here`
openphish_enabled = true;
phishtank_enabled = true;

For SURBL (Spam URI Real-time Blocklists) configuration in /usr/local/etc/rspamd/local.d/surbl.conf:

# follow redirects when checking URLs in emails for spaminess
redirector_hosts_map = "/usr/local/etc/rspamd/redirectors.inc";

For URL reputation checks in /usr/local/etc/rspamd/local.d/url_reputation.conf:

# check URLs within messages for spaminess
enabled = true;

And for caching some URL tags in Redis, in /usr/local/etc/rspamd/local.d/url_tags.conf:

# cache some URL tags in redis
enabled = true;

Rspamd should be ready for initial testing.

Now it's time to configure Dovecot, which will handle the final phase of delivery and provide mail access, as well as use Sieve for filtering and rule application.

The first step is to start with a default configuration:

cd /usr/local/etc/dovecot/ && cp -R example-config/* .

Next, modify some files. Start with conf.d/15-mailboxes.conf - add auto = create to the main folders, like this:

namespace inbox {
  # These mailboxes are widely used and could perhaps be created automatically:
  separator = .

  mailbox Drafts {
    special_use = \Drafts
    auto = create
  }
  mailbox Junk {
    special_use = \Junk
    auto = create
  }
  mailbox Trash {
    special_use = \Trash
    auto = create
  }

  # For \Sent mailboxes there are two widely used names. We'll mark both of
  # them as \Sent. User typically deletes one of them if duplicates are created.
  mailbox Sent {
    special_use = \Sent
    auto = create
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
}

Modify /usr/local/etc/dovecot/dovecot.conf to include:

protocols = imap lmtp sieve

In conf.d/10-auth.conf, switch to system authentication with passwdfile:

#!include auth-system.conf.ext
!include auth-passwdfile.conf.ext

Then, set up the passdb and userdb configuration in conf.d/auth-passwdfile.conf.ext. Replace the entire file with these contents:

passdb {
  driver = passwd-file
  args = scheme=CRYPT /usr/local/etc/mail/passwd
}

userdb {
  default_fields = uid=vmail gid=vmail home=/var/vmail/%d/%n
  args = /usr/local/etc/mail/passwd
  driver = passwd-file
}

Edit conf.d/10-ssl.conf to update ssl_key and ssl_cert:

ssl_cert = </usr/local/etc/letsencrypt/live/example.com/fullchain.pem
ssl_key = </usr/local/etc/letsencrypt/live/example.com/privkey.pem

In conf.d/20-imap.conf, include:

(...)
mail_plugins = $mail_plugins imap_sieve zlib
mail_max_userip_connections = 100

And in conf.d/20-lmtp.conf:

protocol lmtp {
  mail_fsync = optimized
  mail_plugins = $mail_plugins sieve
}

Set the mail location in conf.d/10-mail.conf:

mail_location = maildir:/var/vmail/%d/%n

maildir_stat_dirs=yes

Now, let's configure Sieve in conf.d/90-plugin.conf:

plugin {
  #setting_name = value
  sieve_plugins = sieve_imapsieve sieve_extprograms
  sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment
  imapsieve_mailbox1_name = Junk
  imapsieve_mailbox1_causes = COPY APPEND
  imapsieve_mailbox1_before = file:/usr/local/lib/dovecot/sieve/report-spam.sieve

  imapsieve_mailbox2_name = *
  imapsieve_mailbox2_from = Junk
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_before = file:/usr/local/lib/dovecot/sieve/report-ham.sieve

  imapsieve_mailbox3_name = Inbox
  imapsieve_mailbox3_causes = APPEND
  imapsieve_mailbox3_before = file:/usr/local/lib/dovecot/sieve/report-ham.sieve
  sieve_pipe_bin_dir = /usr/local/lib/dovecot/sieve

  #Move the spam to Junk folder
  sieve_after = /usr/local/lib/dovecot/sieve/spam-to-folder.sieve
}

And in conf.d/20-managesieve.conf.ext:

service managesieve-login {
  inet_listener sieve {
    port = 4190
  }

  service_count = 1
  vsz_limit = 64M
}

service managesieve {
  process_limit = 1024
}

protocol sieve {
  mail_max_userip_connections = 10

  # Configuration for Sieve script management via ManageSieve protocol
}

plugin {
  sieve_pipe_bin_dir = /usr/local/lib/dovecot/sieve
  sieve = file:/var/vmail/%d/%n/sieve;active=/var/vmail/%d/%n/.dovecot.sieve
  sieve_user_log = file:/var/vmail/%d/%n/sieve/sieve_error.log
}

This setup outlines the Dovecot configuration necessary for handling mail delivery, access, and Sieve-based filtering. With these settings, Dovecot is prepared to manage both incoming and outgoing mails securely and efficiently, including support for managing Sieve scripts via the ManageSieve protocol.

Sieve scripts train Rspamd on spam and ham. Moving email into and out of the junk folder triggers an event to train Rspamd.

These files are located at /usr/local/lib/dovecot/sieve.

Create the report-ham.sieve file:

require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"];

if environment :matches "imap.mailbox" "*" {
  set "mailbox" "${1}";
}

if string "${mailbox}" "Trash" {
  stop;
}

if environment :matches "imap.user" "*" {
  set "username" "${1}";
}

pipe :copy "sa-learn-ham.sh" [ "${username}" ];

Create the report-spam.sieve file:

require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"];

if environment :matches "imap.user" "*" {
  set "username" "${1}";
}

pipe :copy "sa-learn-spam.sh" [ "${username}" ];

Create the spam-to-folder.sieve file:

require ["fileinto","mailbox"];

if header :contains "X-Spam" "Yes" {
 fileinto :create "Junk";
 stop;
}

Compile the files:

sievec report-ham.sieve
sievec report-spam.sieve
sievec spam-to-folder.sieve

Create the following two shell scripts in /usr/local/lib/dovecot/sieve.

Add the following to sa-learn-ham.sh:

#!/bin/sh
exec /usr/local/bin/rspamc -d "${1}" learn_ham

Add the following to sa-learn-spam.sh:

#!/bin/sh
exec /usr/local/bin/rspamc -d "${1}" learn_spam

Make the files executable:

chmod 755 sa-learn-ham.sh
chmod 755 sa-learn-spam.sh

Everything has been configured correctly. However, a fundamental part is missing: the users.

To begin, encrypt the password by executing:

smtpctl encrypt

Input the plain password and hit ENTER. Copy the displayed encrypted password for the next step. Now, insert the entry into /usr/local/etc/mail/passwd:

foo@example.com:your encrypted password goes here::::::

Remember to replace the placeholder with your encrypted password and ensure there are six colons at the end.

Finally, add the virtual user and associate it with the vmail system user in /usr/local/etc/mail/virtuals:

foo@example.com vmail

Note: Additional aliases can be included, such as:

postmaster@example.com foo@example.com

Let's enable and start the services:

service rspamd enable
service rspamd start
service smtpd enable
service smtpd start
service dovecot enable
service dovecot start

At this point, take a look at the logs (especially /var/log/maillog) to get an idea of what's happening. If everything is running correctly, you can now proceed with some tests. Any mail client (such as Thunderbird, etc.) should be able to connect via IMAP and send emails using SMTP, thanks to authentication. Remember that the username corresponds to the one entered in the /usr/local/etc/mail/passwd file and typically matches the full email address.

Once everything is in order, you can set the MX record of the domain "example.com" to "mail.example.com".

One of the key tests to perform is to send an email to a Gmail inbox. If the email is correctly delivered and not marked as Spam, there's a good chance that the setup is correct. There are also several online tools available that can provide useful insights into any configuration errors.

You have just installed and operationalized a complete mail server.

It is now possible to proceed with part 2, which involves the installation of Nextcloud and Snappymail, in order to have a comprehensive groupware system, while keeping one's data on their own servers.

Links:

This article was inspired by several other articles available online that provided me with a lot of useful information. Here is a non-exhaustive list:

Make your own VPN - FreeBSD, Wireguard, ipv6 and ad-blocking included

Stefano Marinelli — Sat, 23 Sep 2023 09:05:51 +0000

Note: This article assumes a setup based on FreeBSD. If you prefer a version based on OpenBSD, it is available here.

VPNs are a fundamental tool for securely connecting to your own servers and devices. Many people use commercial VPNs for various reasons, ranging from not trusting their provider (especially when connecting from a public hotspot) to wanting to "go out" on the Internet with a different IP address, perhaps from another country.

Whatever the reason, solutions are not lacking. I have always set up management VPNs to allow servers and/or clients to communicate with each other using secure channels. Lately, I have been activating IPv6 connectivity on all my devices (both desktop/servers and mobile devices) and I needed to quickly create a node that concentrated some networks and allowed them to go out on the network in IPv6. The tools I used and will describe are:

VPS - in this case, I used a basic Hetzner Cloud VPS (using this link, you will receive 20 euros of cloud credits), but any provider that provides IPv6 connectivity will do - if you want IPv6, of course.
FreeBSD - a versatile, stable, and secure operating system.
Wireguard - lightweight, secure, and at the same time, not very "chatty", so it is also gentle on mobile device batteries. When there is no traffic, it simply does not transmit/receive anything. Well supported by all major desktop and server operating systems as well as Android and iOS devices.
Unbound - can make DNS queries directly to root servers, not through forwarders. It also allows you to insert block-lists and have a result similar to that of Pi-Hole (i.e., ad-blocking).
SpamHaus lists - to immediately stop connections to and from users on blacklists.

The first step is to activate a VPS and install FreeBSD. On the Hetzner Cloud console, there might not be a pre-built FreeBSD image, but only a selection of Linux distributions. Don't worry, just choose any of them and create the VPS. Once done, the FreeBSD ISO image will be available among the "ISO Images". Just insert the virtual CD, restart the VPS, and the FreeBSD installation will appear in the console.

I won't go into detail, the operation is simple and straightforward. The only precaution (in the case of a Hetzner Cloud VPS) is to use "DHCP" for IPv4 but, for now, do not configure IPv6. It will be configured later.

Install all FreeBSD updates (using the freebsd-update fetch install command) and reboot.

It seems, moreover, that there is still an old bug still present and manifesting when, on a FreeBSD server installed in a VM based on KVM (thus also tho se of Hetzner), routing is performed (as in our case) between VNET jails and host.

Adding this configuration to /boot/loader.conf will solve the problem:

hw.vtnet.X.tso_disable="1"
hw.vtnet.tso_disable="1"
hw.vtnet.lro_disable="1"
hw.vtnet.X.lro_disable="1"
hw.vtnet.csum_disable="1"
hw.vtnet.X.csum_disable="1"

Wireguard, on FreeBSD, is now available as a kernel module and the userland can be installed using the pkg install wireguard-tools package manager. This means you can easily keep it updated alongside other software on the system.

The first step is to configure IPv6 on the VPS. In the case of Hetzner, unfortunately, they only provide a /64, so it will be necessary to segment the assigned network. In this example, it will be divided into /72 subnetworks - to find valid subclasses, it will be possible to use a calculator.

The /etc/rc.conf file should have entries similar to:

ifconfig_vtnet0="DHCP"
ifconfig_vtnet0_ipv6="inet6 2001:db8:cafe:cafe::1 prefixlen 72"
ipv6_defaultrouter="fe80::1%vtnet0"

In short, keep the base address assigned by Hetzner, but change the prefix length to 72 - thus giving the possibility of having other networks available.

It is now necessary to enable forwarding for IPv4 and IPv6. Add these lines to the /etc/sysctl.conf file:

net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1

After reboot, test it:

ping6 google.com

If everything has been configured correctly, the ping will be executed and google.com will reply.

To configure Wireguard, a few steps will be necessary. First of all, the private key will need to be created:

wg genkey | tee /dev/stderr | wg pubkey | grep --label PUBLIC -H .

You will get a private key and a public key. Take note of the public key - it will be needed to configure the clients.

Now create a new file called /usr/local/etc/wireguard/wg0.conf:

[Interface]
Address = 172.16.0.1/24,2001:db8:cafe:cafe:100::1/72
ListenPort = 51820
PrivateKey = YUkS6cNTyPbXmtVf/23ppVW3gX2hZIBzlHtXNFRp80w=

A new Wireguard interface called wg0 is being created. Start the Wireguard interface:

service wireguard enable
sysrc wireguard_interfaces="wg0" 
service wireguard start

If everything has been entered correctly, the interface should come up. Check its status:

wg

As for the firewall, FreeBSD comes with no pf configuration. In my setups, I tend to block what is not needed and be permissive with what may be useful. However, I like to keep out the "bad guys," so I use blacklists. pf allows elements to be inserted and removed from tables at runtime, so the firewall can be configured accordingly.

To download and apply the Spamhaus lists, I use a simple but effective script found on the Internet, but for OpenBSD.

For the Spamhaus lists, continue with the FreeBSD script creation.

Create the script in /usr/local/sbin/spamhaus.sh:

#!/bin/sh
#
#this is normally run once per day via cron.
#
echo updating Spamhaus DROP lists:
(
  { fetch -o - https://www.spamhaus.org/drop/drop.txt && \
    fetch -o - https://www.spamhaus.org/drop/dropv6.txt ; \
  } 2>/dev/null | sed "s/;/#/" > /var/db/drop.txt
)
pfctl -t spamhaus -T replace -f /var/db/drop.txt

Make it executable and run it. Pf isn't enabled, so you'll get an error - but this will create the /var/db/drop.txt file:

chmod a+rx /usr/local/sbin/spamhaus.sh
/usr/local/sbin/spamhaus.sh

There are many possibilities to configure pf on FreeBSD. A fairly simple example could be this:

ext_if="vtnet0"
wg0_if="wg0"
wg0_networks="172.16.0.0/24"

set skip on lo

nat on $ext_if from { $wg0_networks } to any -> ($ext_if)

# Spamhaus DROP list:
table <spamhaus> persist file "/var/db/drop.txt"

block drop log quick from <spamhaus>

# Pass ICMP on ipv6
pass quick proto ipv6-icmp
# Block from ipv6 to wg0 network
block in quick on $ext_if inet6 to { 2001:db8:cafe:cafe:100::/72 }
# Pass Wireguard traffic - in and out
pass quick on $wg0_if

# default deny
block in
block out

pass in on $ext_if proto tcp to port ssh
pass in on $ext_if proto udp to port 51820

pass out on $ext_if

This is a very simple configuration: it blocks everything that is present in the list downloaded from Spamhaus, allows NAT from the Wireguard network to the public interface, allows ICMP traffic in IPv6 (necessary for the network to function properly) while blocking incoming traffic to the Wireguard IPv6 LAN (remember that the IPs will be public and directly reachable, so we don't want to expose our devices by default). All traffic on the Wireguard interface will be allowed to pass. Then everything will be blocked and exceptions will be specified, i.e., allowing SSH and Wireguard connections (of course). Authorization will also be granted to allow traffic to exit from the public network interface.

Save this configuration to /etc/pf.conf.

Enable and start pf:

service pf enable 
service pf start

You will probably be kicked out of the system. Don't worry, just reconnect. pf is doing its job.

If everything went correctly, the firewall should have loaded the new rules.

To obtain caching of DNS queries and the related ad-block, it is now time to configure Unbound. Let's install it with:

pkg install unbound

A while ago, I found a script which I slightly adapted. I don't remember where I got it, so I'll paste it here without citing the original creator.

Create a script to update the unbound ad-block, in /usr/local/sbin/unbound-adhosts.sh:

#!/bin/sh
#
# Using blacklist from pi-hole project https://github.com/pi-hole/
# to enable AD blocking in unbound(8)
#
PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin"

# Available blocklists - comment line to disable blocklist
_disconad="https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt"
_discontrack="https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt"
_stevenblack="https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts"

# Global variables
_tmpfile="$(mktemp)" && echo '' > $_tmpfile
_unboundconf="/usr/local/etc/unbound/unbound-adhosts.conf"

# Remove comments from blocklist
simpleParse() {
  fetch -o - $1 | \
  sed -e 's/#.*$//' -e '/^[[:space:]]*$/d' >> $2
}

# Parse DisconTrack
[ -n "${_discontrack}" ] && simpleParse $_discontrack $_tmpfile

# Parse DisconAD
[ -n "${_disconad}" ] && simpleParse $_disconad $_tmpfile

# Parse StevenBlack
[ -n "${_stevenblack}" ] && \
  fetch -o - $_stevenblack | \
  sed -n '/Start/,$p' | \
  sed -e 's/#.*$//' -e '/^[[:space:]]*$/d' | \
  awk '/^0.0.0.0/ { print $2 }' >> $_tmpfile

# Create unbound(8) local zone file
sort -fu $_tmpfile | grep -v "^[[:space:]]*$" | \
awk '{
  print "local-zone: \"" $1 "\" redirect"
  print "local-data: \"" $1 " A 0.0.0.0\""
}' > $_unboundconf && rm -f $_tmpfile

service unbound reload 1>/dev/null

exit 0

After saving the script, make it executable and run it:

chmod a+rx /usr/local/sbin/unbound-adhosts.sh
/usr/local/sbin/unbound-adhosts.sh

Now, the Unbound configuration file in /usr/local/etc/unbound/unbound.conf can be modified as follows:

server:
        verbosity: 1
        log-queries: no
        num-threads: 4
        num-queries-per-thread: 1024
        interface: 127.0.0.1
        interface: 172.16.0.1
        interface: 2001:db8:cafe:cafe:100::1
        interface: ::1
        outgoing-range: 64
        chroot: ""

        access-control: 0.0.0.0/0 refuse
        access-control: 127.0.0.0/8 allow
        access-control: ::0/0 refuse
        access-control: ::1 allow
        access-control: 172.16.0.0/24 allow
        access-control: 2001:db8:cafe:cafe:100::/72 allow

        hide-identity: yes
        hide-version: yes
        auto-trust-anchor-file: "/usr/local/etc/unbound/root.key"
        val-log-level: 2
        aggressive-nsec: yes
        prefetch: yes
        username: "unbound"
        directory: "/usr/local/etc/unbound"
        logfile: "/var/log/unbound.log"
        use-syslog: no
        pidfile: "/var/run/unbound.pid"
        include: /usr/local/etc/unbound/unbound-adhosts.conf

remote-control:
        control-enable: yes
        control-interface: /var/run/unbound.sock

Now, enable and start unbound:

service unbound enable
service unbound start

If everything has been set up correctly, unbound will be able to respond to DNS requests made on 172.16.0.1 and 2001:db8:cafe:cafe:100::1.

Now it is possible to configure the Wireguard client. Create a new configuration by inserting "172.16.0.2/32, 2001:db8:cafe:cafe:100::2/128" (the ones that will later be entered in the peer configuration of the server) in the local IP addresses. Set the DNS server address to "172.16.0.1" and/or its corresponding IPv6 address (in the example, 2001:db8:cafe:cafe:100::1 - yours will be different). In the peer section, insert the server's data, including its public key, IP address:port (in the example, the port is 51820), and allowed addresses (setting "0.0.0.0/0, ::0/0" means "all connections will be sent via Wireguard" - all the traffic will pass through the VPN for both IPv4 and IPv6).Each implementation has its own procedure (Android, iOS, MikroTik, Linux, etc.) but essentially it is sufficient to create the right configuration both on the server and on the client.

Reopen the Wireguard configuration file /usr/local/etc/wireguard/wg0.conf and add:

[Interface]
Address = 172.16.0.1/24,2001:db8:cafe:cafe:100::1/72
ListenPort = 51820
PrivateKey = YUkS6cNTyPbXmtVf/23ppVW3gX2hZIBzlHtXNFRp80w=

[Peer]
PublicKey = *client's public key*
AllowedIPs = 172.16.0.2/32, 2001:db8:cafe:cafe:100::2/128

The client's public key will be shown by the client itself.

Reload the Wireguard configuration:

service wireguard restart

It is also possible to use the VPN only as an ad-blocker, by only routing DNS traffic through it. To achieve this result, configure the client so that the only allowed address is the one of the just-configured unbound (in this example, 172.16.0.1 and/or 2001:db8:cafe:cafe:100::1) - DNS resolution will occur via VPN, but browsing will continue to work through the main provider.

To automatically update the spamhaus and ad-block lists, we will use cron.First, create a script, for example, /usr/local/sbin/update-blocklists.sh:

#!/bin/sh

/usr/local/sbin/unbound-adhosts.sh
/usr/local/sbin/spamhaus.sh

Make it executable:

chmod +x /usr/local/sbin/update-blocklists.sh

Then, add it to the crontab to run daily:

echo "@daily /usr/local/sbin/update-blocklists.sh" >> /etc/crontab

This approach benefits from both update management and security perspectives.

Make your own VPN - OpenBSD, Wireguard, ipv6 and ad-blocking included

Stefano Marinelli — Mon, 03 Apr 2023 04:16:11 +0000

Note: This article assumes a setup based on OpenBSD. If you prefer a version based on FreeBSD, it is available here.

VPS - in this case, I used a basic Hetzner Cloud VPS (using this link, you will receive 20 euros of cloud credits), but any provider that provides IPv6 connectivity will do - if you want IPv6, of course.
OpenBSD - a clean, stable, and secure operating system.
Wireguard - lightweight, secure, and at the same time, not very "chatty", so it is also gentle on mobile device batteries. When there is no traffic, it simply does not transmit/receive anything. Well supported by all major desktop and server operating systems as well as Android and iOS devices.
Unbound - can make DNS queries directly to root servers, not through forwarders. It also allows you to insert block-lists and have a result similar to that of Pi-Hole (i.e., ad-blocking).
SpamHaus lists - to immediately stop connections to and from users on blacklists.

The first step is to activate a VPS and install OpenBSD. On the Hetzner cloud console, there won't be a pre-built OpenBSD image, but only a selection of Linux distributions. Don't worry, just choose any of them and create the VPS. Once done, the OpenBSD ISO image will be available among the "ISO Images". Just insert the virtual CD, restart the VPS, and the OpenBSD installation will appear in the console.

I won't go into detail, the operation is simple and straightforward. The only precaution (in the case of a Hetzner Cloud VPS) is to use "autoconf" for IPv4 but, for now, do not configure IPv6. It will be configured later.

Install all OpenBSD updates (using the syspatch command) and restart, the kernel will be relinked.

Wireguard, on OpenBSD, is fully integrated into the base system and does not require the installation of external packages. This is a big advantage because over time, support for everything related to Wireguard will be managed directly by the main OpenBSD development team.

The /etc/hostname.vio0 file should look something like this:

inet autoconf
inet6 2001:db8:cafe:cafe::1 72 
!route add -net ::/0 fe80::1%vio0

In short, keep the base address assigned by Hetzner, but change the netmask to /72 - thus giving the possibility of having other networks available.

sh /etc/netstart vio0

It will reconfigure the network interface and allow IPv6 to start working. To test it:

ping6 google.com

If everything has been configured correctly, the ping will be executed and google.com will reply.

It is now necessary to enable forwarding for IPv4 and IPv6. Enter these lines in the /etc/sysctl.conf file:

net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1

To apply those changes you can reboot or just type:

sysctl net.inet.ip.forwarding=1
sysctl net.inet6.ip6.forwarding=1

To configure Wireguard, a few steps will be necessary. First of all, the private key will need to be created:

openssl rand -base64 32

Something like this will come out:

YUkS6cNTyPbXmtVf/23ppVW3gX2hZIBzlHtXNFRp80w=

Now create a new file called /etc/hostname.wg0:

172.16.0.1/24 wgport 51820 wgkey YUkS6cNTyPbXmtVf/23ppVW3gX2hZIBzlHtXNFRp80w=
inet6 2001:db8:cafe:cafe:100::1 72

A new Wireguard interface called wg0 is being created. It will have the IPv4 address "172.16.0.1", Wireguard will listen on port 51820, and with the private key created shortly before. It will also have an IPv6 address on one of the subclasses that the provider will have provided.

Save and activate the interface:

sh /etc/netstart wg0

If everything has been entered correctly, it should enable the interface. Now:

ifconfig wg0

And something like this will be returned:

wg0: flags=80c3<UP,BROADCAST,RUNNING,NOARP,MULTICAST> mtu 1420
    index 5 priority 0 llprio 3
    wgport 51820
    wgpubkey xxxxxxxxxxxxxxx=
    groups: wg
    inet 172.16.0.1 netmask 0xffffff00 broadcast 172.16.0.255
    inet6 2001:db8:cafe:cafe:100::1 prefixlen 72

Take note of the wgpubkey - it will be needed to configure the clients.

As for the firewall, OpenBSD comes with a basic pf configuration. In my setups, I tend to block what is not needed and be permissive with what may be useful. However, I like to keep out the "bad guys," so I use blacklists. pf allows elements to be inserted and removed from tables in runtime, so the firewall can be configured accordingly.

To download and apply Spamhaus lists, I use a simple but effective script found on the Internet.

So create the script in /usr/local/sbin/spamhaus.sh:

#!/bin/sh
#
# this is normally run once per day via /etc/daily.local.
#
echo updating Spamhaus DROP lists:
(
  { ftp -o - https://www.spamhaus.org/drop/drop.txt && \
    ftp -o - https://www.spamhaus.org/drop/dropv6.txt ; \
  } 2>/dev/null | sed "s/;/#/" > /var/db/drop.txt
)
pfctl -t spamhaus -T replace -f /var/db/drop.txt

Make it executable and run it:

chmod a+rx /usr/local/sbin/spamhaus.sh
/usr/local/sbin/spamhaus.sh

There are many possibilities to configure pf. A fairly simple example could be this:

ext_if="vio0"
wg0_if = "wg0"
wg0_networks = "172.16.0.0/24"

set skip on lo

# Spamhaus DROP list:
table <spamhaus> persist file "/var/db/drop.txt"

block drop log quick from <spamhaus>

match in all scrub (no-df random-id max-mss 1440)

match out on $ext_if from { $wg0_networks } nat-to ($ext_if)

#Pass ICMP on ipv6
pass quick proto ipv6-icmp
#Block from ipv6 to wg0 network
block in quick on $ext_if inet6 to { 2001:db8:cafe:cafe:100::/72 }
#Pass Wireguard traffic - in and out
pass quick on $wg0_if

# default deny
block in
block out

# By default, do not permit remote connections to X11
block return in on ! lo0 proto tcp to port 6000:6010

# Port build user does not need network
block return out log proto {tcp udp} user _pbuild

pass in on $ext_if proto tcp to port ssh
pass in on $ext_if proto udp to port 51820

pass out on $ext_if

This is a very simple configuration: it blocks everything that is present in the list downloaded from Spamhaus, allows NAT from the Wireguard network to the public interface, allows icmp traffic in IPv6 (necessary for the network to function properly) while blocking incoming traffic to the Wireguard IPv6 LAN (remember that the IPs will be public and directly reachable, so we don't want to expose our devices by default). All traffic on the Wireguard interface will be allowed to pass. Then everything will be blocked and exceptions will be specified, i.e. allowing ssh and Wireguard connections (of course). Authorization will also be granted to allow traffic to exit from the public network interface.

Reload pf configuration:

pfctl -f /etc/pf.conf

If everything went correctly, the firewall should have loaded the new options.

To obtain caching of DNS queries and the related ad-block, it is now time to configure Unbound. A while ago, I found a script which I slightly adapted. I don't remember where I got it, so I'll paste it here without citing the original creator.

Create a script to update the unbound ad-block, in /usr/local/sbin/unbound-adhosts.sh:

#!/bin/ksh
#
# Using blacklist from pi-hole project https://github.com/pi-hole/
# to enable AD blocking in unbound(8)
#
PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin"

# Available blocklists - comment line to disable blocklist
_disconad="https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt"
_discontrack="https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt"
_stevenblack="https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts"

# Global variables
_tmpfile="$(mktemp)" && echo '' > $_tmpfile
_unboundconf="/var/unbound/etc/unbound-adhosts.conf"

# Remove comments from blocklist
function simpleParse {
  ftp -VMo - $1 | \
  sed -e 's/#.*$//' -e '/^[[:space:]]*$/d' >> $2
}

# Parse MalwareDom
#[[ -n ${_malwaredom+x} ]] && simpleParse $_malwaredom $_tmpfile

# Parse ZeusTracker
#[[ -n ${_zeustracker+x} ]] && simpleParse $_zeustracker $_tmpfile

# Parse DisconTrack
[[ -n ${_discontrack+x} ]] && simpleParse $_discontrack $_tmpfile

# Parse DisconAD
[[ -n ${_disconad+x} ]] &&  simpleParse $_disconad $_tmpfile

# Parse StevenBlack
[[ -n ${_stevenblack+x} ]] && \
  ftp -VMo - $_stevenblack | \
  sed -n '/Start/,$p' | \
  sed -e 's/#.*$//' -e '/^[[:space:]]*$/d' | \
  awk '/^0.0.0.0/ { print $2 }' >> $_tmpfile

# Parse hpHosts
[[ -n ${_hostfiles+x} ]] && \
  ftp -VMo - $_hostfiles | \
  sed -n '/START/,$p' | tr -d '^M$' | \
  sed -e 's/#.*$//' -e '/^[[:space:]]*$/d' -e 's/^M$//' | \
  awk '/^127.0.0.1/ { print $2 }' >> $_tmpfile

# Create unbound(8) local zone file
sort -fu $_tmpfile | grep -v "^[[:space:]]*$" | \
awk '{
  print "local-zone: \"" $1 "\" redirect"
  print "local-data: \"" $1 " A 0.0.0.0\""
}' > $_unboundconf && rm -f $_tmpfile

/usr/sbin/rcctl reload unbound 1>/dev/null

exit 0
#EOF

Similarly, make the script executable and run it:

chmod a+rx /usr/local/sbin/unbound-adhosts.sh
/usr/local/sbin/unbound-adhosts.sh

Now, the Unbound configuration file in /var/unbound/etc/unbound.conf can be modified as follows:

# $OpenBSD: unbound.conf,v 1.21 2020/10/28 11:35:58 sthen Exp $

server:
        verbosity: 1
        log-queries: no
        num-threads: 4
        num-queries-per-thread: 1024
        interface: 127.0.0.1
        #interface: 127.0.0.1@5353      # listen on alternative port
        interface: 172.16.0.1
    interface: 2001:db8:cafe:cafe:100::1
        interface: ::1
        outgoing-range: 64
        chroot: ""
        #do-ip6: yes

        # override the default "any" address to send queries; if multiple
        # addresses are available, they are used randomly to counter spoofing
        #outgoing-interface: 192.0.2.1
        #outgoing-interface: 2001:db8::53

        access-control: 0.0.0.0/0 refuse
        access-control: 127.0.0.0/8 allow
        access-control: ::0/0 refuse
        access-control: ::1 allow
        access-control: 172.16.0.0/24 allow
        access-control: 2001:db8:cafe:cafe:100::/72 allow

        hide-identity: yes
        hide-version: yes

        # Perform DNSSEC validation.
        #
        auto-trust-anchor-file: "/var/unbound/db/root.key"
        val-log-level: 2

        # Synthesize NXDOMAINs from DNSSEC NSEC chains.
        # https://tools.ietf.org/html/rfc8198
        #
        aggressive-nsec: yes
        prefetch: yes
        username: "nobody"
        directory: "/var/unbound/etc"
        logfile: "/var/unbound/unbound.log"
        use-syslog: no
        pidfile: "/var/unbound/unbound.pid"
        include: /var/unbound/etc/unbound-adhosts.conf

remote-control:
        control-enable: yes
        control-interface: /var/run/unbound.sock

Before launching unbound, it is necessary to give the appropriate permissions:

chown -R nobody:nobody /var/unbound

Now it is possible to enable and start unbound. Since it needs to load the (long) blocklist, it will take a few seconds:

rcctl enable unbound
rcctl start unbound

If everything has been done correctly, unbound will be able to respond to requests made on 172.16.0.1 and 2001:db8:cafe:cafe:100::1, from their respective LANs.

Now it is possible to configure the Wireguard client. Each implementation has its own procedure (Android, iOS, MikroTik, Linux, etc.) but essentially it is sufficient to create the right configuration both on the server and on the client. For example, the server's public key (visible by typing "ifconfig wg0" on the OpenBSD server) should be inserted into the "peer" that will be created on the client, while the client's public key will be used on the server in this way:

Reopen the file /etc/hostname.wg0 and add:

172.16.0.1/24 wgport 51820 wgkey YUkS6cNTyPbXmtVf/23ppVW3gX2hZIBzlHtXNFRp80w=
inet6 2001:db8:cafe:cafe:100::1 72
wgpeer *client's public key* wgaip 172.16.0.2/32 wgaip 2001:db8:cafe:cafe:100::2/128

Reload the configuration:

sh /etc/netstart wg0

On the client, create a new configuration by inserting "172.16.0.2/32, 2001:db8:cafe:cafe:100::2/128" (the ones that were entered in the peer configuration in the hostname.wg0 file) in the local IP addresses. Set the DNS server address to "172.16.0.1" and/or its corresponding IPv6 address (in the example, 2001:db8:cafe:cafe:100::1 - yours will be different). In the peer, insert the server's data, including its public key, IP address:port (in the example, the port is 51820), and allowed addresses (setting "0.0.0.0/0, ::0/0" means "all connections will be sent via Wireguard" - all the traffic will pass through the VPN for both IPv4 and IPv6).

It is also possible to use the VPN only as an ad-blocker, by only routing DNS traffic through it. To achieve this result, it is sufficient to configure the client so that the only allowed address is the one of the just-configured unbound (in this example, 172.16.0.1 or 2001:db8:cafe:cafe:100::1) - DNS resolution will occur via VPN, but browsing will continue to work through the main provider.

If you want the spamhaus and ad-block lists to be updated automatically, create the /etc/daily.local file and add the following lines:

#!/bin/ksh

/usr/local/sbin/unbound-adhosts.sh
/usr/local/sbin/spamhaus.sh

All of this can be achieved simply with a basic installation of OpenBSD, without the need to install any additional packages. This is an advantage both in terms of update management and security.

How we are migrating (many of) our servers from Linux to FreeBSD - Part 2 - Backups and Disaster Recovery

Stefano Marinelli — Mon, 30 May 2022 03:03:52 +0000

After my post on why we’re migrating (most of) our servers from Linux to FreeBSD, I’ve started to write about how we’re doing it. After covering a basic installation (we’re doing a massive use of jails), I’m going now to describe how we’re performing backups.

Backup is not a tool. Backup is not a software you can buy. Backup is a strategy you need to study and implement to be able to solve your specific problems. You need to understand what you’re doing, otherwise you’ll always have a Schrödinger’s Backup - it may work or not and if you don’t test it well enough (i.e. restore) you’ll find out when it’s too late.

We’re performing backups in many different ways but, for our physical and virtual FreeBSD servers, we have a dual approach. We need both a “ready to use” backup (that will be described here, useful for a fast disaster recovery or prompt restore of specific jails) and a “colder”, more space efficient backup that can be kept for months (or years), more similar to the borg approach on previous posts. Generally speaking, we store our OS (and jails) on ZFS, so I’ll describe this kind of approach here.

Disaster recovery backup - ZFS send/receive

BastilleBSD creates its datasets and mounts them on /usr/local/bastille . There are no databases, all the jails’ configurations are inside that mountpoint so it’s quite easy to backup and restore all the jails or any single jail in one go. More, the “everything-in-a-jail” approach simplifies the restore process as you don’t need to restore the entire host OS, just install an empty FreeBSD server, install BastilleBSD and restore the jails. Or add the jails to already existing FreeBSD systems.

We normally use FreeBSD (or Linux with ZFS) backup servers, well protected and encrypted at rest. For the ZFS send/receive approach, our servers are NOT reachable from the outside. We can ssh into them only using a VPN - they’re too precious to be exposed on the World Wild Web - or, if strictly needed, we expose ssh only using keys, no passwords. We perform the backups using a pull strategy: the backup server connects to the production servers, gets the data, disconnects. The production servers have NO ACCESS to the main backup server. Should they ever be seriously compromised, the backup is safe.

There are many tools that can help to set up this kind of configuration. I’ve tried many of them and found that they all have some good and bad points. The one I decided to use for our servers is zfs-autobackup. It’s easy to use, everything can be set via command line and has a good cron (or Jenkins) output, useful to understand if everything is right.

Let’s consider two servers, one is called “ProdA” and the other is called “Bck” - we obviously want to backup the ProdA into Bck.

Installing ProdA has been covered on a previous post, Bck is quite simple and outside the scope of this post. We just need a protected zfs FreeBSD (or Linux) server. That’s all. Let’s assume that ProdA has a BastilleBSD zfs dataset (and children datasets), with jails and everything needed, as configured in the last post. We now need to install the needed software. On Bck:

pkg install py311-zfs-autobackup mbuffer

On ProdA:

pkg install mbuffer

mbuffer will be used as a ram buffer to avoid read/write spikes (or slowdowns) while sending/receiving the snapshots.

It’s time to prepare the destination dataset. Assuming that Bck has a zroot base dataset, we’ll be creating (as root) a zroot/backups/ProdA

zfs create -p zroot/backups/ProdA

Ok, let’s now go to ProdA

We want to create an unprivileged user that will send the data. We don’t want to allow Bck to connect as root, even if it’s trusted and secure. Let’s create a user called “backupper”. Then, we need to give backupper the right permissions:

zfs allow -u backupper send,snapshot,hold,mount,destroy zroot

Note: if you want Bck to be able to delete the snapshots on ProdA, backupper needs the destroy permission. That means this user can destroy the whole system as can ALSO destroy zroot (or any source dataset you decide). If you’re afraid of this, different approaches must be used (i.e.: local root performing snapshot/cleanups and Bck only transferring them, not hard to achieve with zfs-autobackup). Considering that the Bck is safe, secure and protected, we can tolerate this weakness. Just be sure nobody can break the “backupper” user. Do not use password, use ssh keys and treat this user with the same care you'd use with root.

Now, as root on ProdA:

zfs set autobackup:bck_server=true zroot

We’re setting a custom property, called “autobackup:bck_server”, allowing the zroot (and children) dataset to be backed up by zfs-autobackup. zfs-autobackup will search for all datasets with that property set to “true” (also on different pools) and will backup them. If there’s a specific dataset you don’t want to backup, just set it to “false”. Or if you don’t want to backup the entire zroot but, for example, only “zroot/bastille” (and children), just set autobackup:bck_server=true for that dataset.

ssh config

zfs-autobackup will connect via ssh and zfs-autobackup will try to connect as root. Moreover, even after exchanging the ssh key, Bck will connect many times to ProdA to send its zfs commands (one connection per command). Ssh session initiation is quite long, so there will be some latency. In order to (greatly) speed up this time,

“You can make your ssh connections persistent and greatly speed up zfs-autobackup:

On the server that initiates the backup add this to your ~/.ssh/config:

Host ProdA
User backupper
ControlPath ~/.ssh/control-master-%r@%h:%p
ControlMaster auto
ControlPersist 3600

(Taken from https://github.com/psy0rz/zfs_autobackup/wiki/Performance)

It's now time to go back to Bck and issue a command like this (one line):

/usr/local/bin/zfs-autobackup --ssh-source ProdA bck_server zroot/backups/ProdA --zfs-compressed --no-progress --verbose --buffer 32M --keep-source 0 --no-holds  --set-properties readonly=on --clear-refreservation --keep-target 1d1w,1w1m,1m6m  --destroy-missing 30d --clear-mountpoint

Bck will connect to ProdA, perform the snapshots and start transferring. The most interesting options I used here are:

 --keep-source 0 (only the last snapshot will be kept on ProdA)
 --set-properties readonly=on (be sure the Bck clone is read only, so we will be able to perform an incremental/differential backup next time)
 --keep-target 1d1w,1w1m,1m6m (keep one backup per day for one week, one per week for one month, one per month for six months)
 --destroy-missing 30d (if we've deleted a dataset, keep it for 30 days before removing it from Bck)
 --clear-mountpoint (do not mount the dataset in Bck, as it will cause problems sooner or later)

The first copy will be slow as it'll need to send all the data. The second will be quite fast as only the differences will be transferred.

How to perform a disaster recovery

Ok, your dataset (or datasets) has gone. You need to replace it with the last external backup. You have to retransfer the copy into ProdA (or another FreeBSD host, no difference). Connect to Bck and search for the snapshot you want to restore (zfs list -t snapshot will help). Once identified (one line):

zfs send -R zroot/backups/ProdA/zroot/bastille/bastille/jails/t1@bck_server@20220528005830 | mbuffer -4 -s 128k -m 32M | ssh root@ProdA "zfs receive -F -x canmount -x readonly zroot/bastille/bastille/jails/t1"

Note the -x canmount -x readonly flags. Remember that we altered the canmount and readonly properties of the transferred datasets during the backup, so we must restore them into a normal state.

Once finished, ProdA (or the other, restored host) will show t1 as an available jail and you'll be able to start it.

How we are migrating (many of) our servers from Linux to FreeBSD - Part 1 - System and jails setup

Stefano Marinelli — Sat, 05 Feb 2022 10:10:47 +0000

After my post on why we’re migrating (many of) our servers to FreeBSD, I’ve received a lot of feedback. Many questions, many comments. Many e-mails from Linux users asking how we’re migrating, how jails can replace lxc or (in part) Docker, and how we’re monitoring and performing backups/restores.

I’ll write some posts on how we’re doing it. Of course, it won’t cover all the use cases, but it surely will work for the most common ones.

Let’s start with a general idea of the setup I’m going to describe. One of the things I’ve always tried to do is to leave the host Operating System (VM Hypervisors like XCP-NG or Proxmox Server, FreeBSD with jails and/or Bhyve, Alpine Linux lxc host, Debian with docker etc.) as simple and empty as possible.

That’s why we’re keeping the same setup here, with a FreeBSD host system with as few packages (or ports) as possible. This will ensure easier upgrades, easier backup/restore procedures and a smaller attack surface. Unused services or executables can be problematic, so let’s keep them out of our setup.

I’m going to describe our “basic” FreeBSD host and some jails that will contain the services. There are many jail management systems out there like BastilleBSD, cbsd, iocage, the old ezjail, etc. I love simple (but powerful) solutions, without any database of configured/running jails as it may be a problem in case of backup/recovery. I love the BastilleBSD approach, so that’s the one we’ve been using for our servers. BastilleBSD is a collection of shell scripts, is small, doesn’t need any database, is actively developed, doesn’t need any kind of dependency and works both on ZFS and UFS. iocage, for example, needs ZFS so UFS servers are out and doesn’t seem to be actively developed anymore. Moreover, BastilleBSD doesn’t interfere with other jail systems, so you can mix and match whatever you like. Still, I recommend to choose a jail management system and stick with that. We've done it, and we’re using BastilleBSD.

I won’t describe basic FreeBSD installation. It’s straightforward, easy, fast and it’s full of good documentation out there. The most critical decision is the choice of the file system you’re going to use. We’re using ZFS when possible, as it gives us a lot of good opportunities. For more resource constrained systems (or specific situations where ZFS isn’t recommended nor applicable), we just stick with UFS. It has snapshots, too, so backups and restores are quite easy, too.

Once you’ve installed FreeBSD, you have a fully functional host system. From now on, I’ll assume you’ve installed on ZFS.

First of all, we upgrade our hosts systems to the last security patches. It’s just a matter of:

freebsd-update fetch
freebsd-update install

After this, It’d be better to reboot to be sure that the kernel has been upgraded if needed. As it’s an empty and just installed system, we usually don’t use the Boot Environments for now. Reinstalling is just a matter of minutes and we’re not doing a release upgrade.

Now: ports or packages? We usually use packages: the host system will be simple and empty and we don’t need customised build options. So quarterly packages are more than enough.

First of all, let’s install the “pkg” package management system.

pkg install

BastilleBSD

Now let’s use pkg to install BastilleBSD:

pkg install bastille

The first step is to configure BastilleBSD. All we usually do is: set the ZFS options. bastille_zfs_enable=“YES” and bastille_zfs_pool=“zroot/bastille” and create the dataset:

zfs create zroot/bastille

This way, BastilleBSD will create its base dataset in zroot/bastille/bastille

Why the nested bastille/bastille dataset? Because we sometimes use zroot/bastille to store some information about the underlying jails, so we prefer to keep a nested dataset. BastilleBSD will create its datasets there and will mount the needed datasets in /usr/local/bastille so you will find everything there.

Let’s complete the Bastille installation as described in the official documentation. There are many approaches: loopback, vnet with local interface, vnet with existing bridge. The loopback approach is the easiest and more portable. Generally speaking we tend to use it as it’s easier to deal with when we have to perform an emergency restore to another host. I’ll write more about it in the next posts.

Now let’s bootstrap the FreeBSD 14.1-RELEASE so Bastille will download the needed files and create its base system, then it will apply the security patches.

bastille bootstrap 14.1-RELEASE update

NOTE: BastilleBSD supports other jailed operating systems and, recently, some jailed Linux distributions.

After a while, the system will be ready for its first jail. Now, we generally install a reverse proxy, in order to expose it to web traffic. The reverse proxy will be able to connect to other jails and forward the traffic. It's a good example, so let's do it:

bastille create reverseproxy 14.1-RELEASE 192.168.0.1 bastille0

So we’re creating a jail called reverseproxy, the FreeBSD version is 14.1-RELEASE (of the jail, must be the same or older than the host OS), the jail ip and the loopback interface to use to assign this IP for the jail. The jail will have a dataset (in jails/reverseproxy) with jail configuration, redirect configuration, fstab, etc. and another child dataset (jails/reverseproxy/root) with its root file system. Jails can be thin or thick: BastilleBSD documentation is good, so you can go deeper here.

At this point, a “bastille list -a” should show the jail and it should be running. Now we can enter this jail:

bastille console reverseproxy

and install Nginx (and certbot, if needed):

pkg install py311-certbot-nginx nginx

Configure your nginx (and certbot). As you might guess, I won’t describe it here.

Let’s ensure Nginx will be started at jail launch, so:

service nginx enable

and let’s start it:

service nginx start

Let’s go back to the host and let’s ensure that all the connections to the host ports 80 (http) and 443 (https) will be redirected to the reverseproxy jail:

bastille rdr reverseproxy tcp 80 80
bastille rdr reverseproxy tcp 443 443

No output should be shown, but you can check:

bastille rdr ALL list

Congratulations, you’ve exposed your first jail.

You can create all the jails you want. They will be created and stored in child datasets of zroot/bastille/bastille - this is great for backup purposes, but I’ll describe more about it in another article. They'll also be able to communicate using their private ip addresses. If you've used vnet, you'll need to perform some deeper network configurations (and you can use pf inside the jail!), if using the default loopback device you'll be sharing the host network stack.

Blacklistd

FreeBSD base system has some interesting tools and they get automatically installed. One of those is blacklistd. If you’ve used fail2ban, denyhosts or similar tools, you know what it’s useful for. But it’s integrated and is light. Fail2ban, for example, tends to become heavy and huge as it’s reading from log files. Blacklistd gets notified by the daemon it’s protecting, so load is lower. To enable blacklistd, add to /etc/rc.conf:

blacklistd_enable="YES"
blacklistd_flags="-r" 
pflog_enable="YES"

(I’m assuming you’ve already added pf_enable=“YES” when you’ve installed BastilleBSD. Otherwise, you should add this, too, if you’re using pf.

Now you should add this to pf.conf:

anchor "blacklistd/*" in on $ext_if

Where $ext_if is your external interface.

Last step, let’s get to /etc/ssh/sshd_config and enable blacklistd uncommenting:

UseBlacklist yes

Now reload/start pf, sshd, start pflog and blacklistd and wait. After some time, blacklistctl dump -r will show you some data.

Of course there are many more steps to do, the host should be hardened, network should be configured and firewalled, etc. but it's a basic idea of how we're keeping our host as standard as possible and, then, create the services inside the jails.

FreeBSD - assign OVH failover IPs to FreeBSD Jails

Stefano Marinelli — Fri, 14 Jan 2022 13:11:54 +0000

OVH (and Soyoustart, of course) network seem to be configured in a "strange" way and setting failover IPs isn’t always as straightforward as you think it should be.

Sometimes you want (or need) to assign a public IP address to a FreeBSD jail without playing with NAT but there’s not much documentation on how to do it inside a jail.

Let’s suppose your FreeBSD host server’s public IP address is 1.2.3.4 and your failover ip is 6.7.8.9.

First of all, go to your Control Panel (OVH/Soyoustart/etc.) and generate a MAC address for the failover public ip address you want to assign to your jail. Let’s assume it’s ca:fe:ca:fe:ca:fe

Now let’s go back to the FreeBSD host and take a note of its gateway (it should be 1.2.3.254, but double check), you’ll need it later.

Now it’s time to create the jail. I love BastilleBSD as it’s light, has no dependencies and is being actively developed. I won’t cover how to install and bootstrap Bastille in this article, for further information have a look at the official documentation.

We need VNET for this purpose, so our jail will have its own complete network stack. If you’ve read that VNET is unstable, you’ve found some old articles. Don’t worry, you can use it now, it's stable.

So, let’s create our jail. Using VNET, a bridge interface will be created and both your physical and your jailed network interfaces will be attached. Let’s suppose our physical host interface is “em0” and let's call our jail "p1":

bastille create -V p1 14.2-RELEASE 6.7.8.9 em0

We're asking Bastille to create a (-V) VNET jail, called p1, it should be a FreeBSD 14.2-RELEASE, its ip will be 6.7.8.9 and the created bridge will be attached to em0. The jail will be created & started, but we’re not ready to use it, yet.

Let's stop the jail:

bastille stop p1

Let’s now modify the jail.conf as we have to set the interface MAC address we’ve generated on the web panel.

You’ll have something like this:

…
vnet;
 vnet.interface = e0b_bastille0;
 exec.prestart += "jib addm bastille0 em0";
 exec.prestart += "ifconfig e0a_bastille0 description \"vnet host interface for Bastille jail p1\"";
 exec.poststop += "jib destroy bastille0"; 
}

Let’s add this line after the exec.prestart += "jib addm bastille0 em0”;

exec.prestart += "ifconfig e0a_bastille0 ether ca:fe:ca:fe:ca:fe”;

Now, let’s configure the network interface inside the jail as Bastille couldn't figure out the "strange" OVH network configuration. Let’s edit the jail’s rc.conf file. If you’ve not messed up with Bastille’s setup, it should be:

/usr/local/bastille/jails/p1/root/etc/rc.conf

Remove the network settings already set by Bastille and replace with something like this:

ifconfig_vnet0="inet 6.7.8.9 netmask 255.255.255.255 broadcast 6.7.8.9"
static_routes="ovh"
route_ovh="-net 1.2.3.254 -iface vnet0"
defaultrouter="1.2.3.254"

The gateway is outside the jail's netmask, so FreeBSD must be instructed to set a static route that will allow connections to flow out reaching the "foreign" gateway (1.2.3.254) via a specific network interface.

Save, exit and start the jail:

bastille start p1

Congratulations, you can ping your jail's public ip and your jail will reach the outside world via its public IP address.

Proxmox Backup Server - hints for a perfect deployment

Stefano Marinelli — Sun, 23 Aug 2020 18:16:57 +0000

Proxmox Backup Server (PBS) has been released. It's still in beta but is already perfectly usable. After many years, it's now possible to perform incremental backups of the VMs and thanks to the qemu dirty bitmaps, backups are also fast.

The documentation is well-done and clear, so I suggest to have a look at it for all its options and features.

In short, it allows to backup any Proxmox VM or Container to a specific PBS server (which can also run Proxmox or not - it's not wise to backup on the same server where the VMs are running). It doesn't require any special file system nor setting as it splits files in chunks (so deduplication is possible and efficient) and can be stored in any supported FS.

I've been using it since the first day and it's quite efficient and reliable, I've backed up hundreds of servers and restored many of them without any problem.

Here's some hints (this post will be updated as needed):

The first backup will be slow - at least as slow as a traditional Proxmox backup. Don't worry, it's perfectly normal.
DO NOT TRIM THE VMs, but use the "discard" option at mount time (if Linux): trim will pass all the dirty blocks and zero them, filling the dirty bitmaps database and slowing down the next backup (even if it will not be larger).
For the same reason, if you want to keep multiple backup servers, remember to backup to ONLY ONE PBS server and then sync it to another one. Backing up the same VMs to multiple servers will confuse the dirty bitmaps and every backup will be slower (not larger), even if it will work.
Sometimes syncing to another server can fail and get stuck. At the moment (0.8.11-1), I found that stopping won't solve the issue and you have to restart the PBS, otherwise the task would seem to be running.

Proxmox Backup System has solved a long time problem: performing efficient, deduplicated, incremental Proxmox VM backups.