IT Notes - vpn

Make Your Own CDN with NetBSD

Stefano Marinelli — Tue, 03 Sep 2024 01:41:00 +0200

Introduction

This article is a spin-off from a previous post on how to create a self-hosted CDN, based on OpenBSD, but this time we'll focus on using NetBSD. The idea is to create reverse proxies with local caching. These proxies would cache the content on the first request and serve it directly afterward. The proxies would be distributed across different regions, and the DNS would route requests to the nearest proxy based on the caller’s location. All this is achieved without relying on external CDNs, using self-managed tools instead.

NetBSD is a lightweight, stable, and secure operating system that supports a wide range of hardware, making it an excellent choice for a caching reverse proxy. Devices that other operating systems may soon abandon, such as early Raspberry Pi models or i386 architecture, are still fully supported by NetBSD and will continue to be so. Additionally, NetBSD is an outstanding platform for virtualization (using Xen or qemu/nvmm) and deserves more attention than it currently receives.

The choice of Varnish is based on several factors, with the main ones being the ability to keep the cache in RAM (which means it can run on read-only systems) and the ability to flush the cache remotely. For example, with each change to my blog, I can choose whether to perform an immediate flush (such as for a new article or an error) or wait for the cache's "natural" expiration (such as for a typo or minor, non-critical changes).

While I won't detail the installation process for NetBSD, as it depends heavily on the hardware you have available, I will guide you through setting up a self-hosted CDN using NetBSD, Varnish, nginx, and the acme.sh or lego tool for SSL certificate management.

Installation

During the installation of NetBSD, ensure that you enable support for binary package management. This will install pkgin, a tool that simplifies package management on NetBSD. If you skip this step during installation, you can still install pkgin later, but it's easier to let the installer handle it.

Once your system is up and running, use pkgin to install the necessary packages: Varnish, nginx, and, depending on your preference, either acme.sh or Go (if you plan to compile lego). Although lego is not available as a precompiled package, you can easily compile it locally using Go, but for simplicity, I recommend using acme.sh.

For this setup, I will present two methods for generating and renewing certificates: using acme.sh or compiling lego - to reach the same final outcome as the OpenBSD article.

Option 1: Using acme.sh (Recommended)

acme.sh is a simple, yet powerful, shell script that handles certificate generation and renewal with ease. To install acme.sh:

pkgin in acmesh varnish nginx

Once acme.sh is installed, you can proceed to configure it for certificate management. It supports DNS authentication and integrates with many DNS providers, making it a flexible choice.

Option 2: Compiling Lego

If you prefer to use lego, you will need to compile it manually, as it is not available as a precompiled package for NetBSD. First, install Go and other necessary packages:

pkgin in go varnish nginx

To compile lego, you’ll need some disk space. Since the /tmp directory in NetBSD is often mounted as tmpfs (using RAM), you may run out of space during compilation if your system has limited memory. You can temporarily disable tmpfs by editing /etc/fstab and commenting out the relevant line:

#tmpfs           /tmp    tmpfs   rw,-m=1777,-s=ram%25

After rebooting, compile lego:

export GO111MODULE=on
go122 install github.com/go-acme/lego/v4/cmd/lego@latest

cp go/bin/lego /usr/pkg/bin/

Once lego is compiled and installed, you can uncomment the /tmp line in /etc/fstab and reboot again.

Configuring Varnish and nginx

First, copy the necessary rc.d scripts for nginx and Varnish:

cp /usr/pkg/share/examples/rc.d/nginx /etc/rc.d/
cp /usr/pkg/share/examples/rc.d/varnishd /etc/rc.d/

Then, add the following to /etc/rc.conf to enable and configure nginx and Varnish:

nginx=YES
varnishd=YES
varnishd_flags="-f /usr/pkg/etc/varnish/default.vcl -T localhost:9999 -a "/var/run/varnish.sock",user=nginx,group=varnish,mode=660 -s default,500m"

In this configuration, Varnish listens on a Unix socket, and nginx connects to it. This approach is more efficient and helps avoid some issues that may arise when exposing Varnish over an IP/port.

Creating the Varnish VCL Configuration

Next, create the VCL configuration file for Varnish at /usr/pkg/etc/varnish/default.vcl:

vcl 4.1;
import std;

# Backend - it-notes.dragas.net
backend it_notes {
    .host = "myBackendIP";
    .port = "80";
}

# ACL - purge - it-notes.dragas.net
acl purge_it_notes {
    "allowedToPurge_IP";
}

sub vcl_recv {
    # it-notes.dragas.net
    if (req.http.Host == "it-notes.dragas.net") {
        set req.backend_hint = it_notes;
        set req.http.Host = "it-notes.dragas.net";

        # PURGE - it-notes.dragas.net
        if (req.method == "PURGE") {
            std.log("Purge request received for " + req.url);

            if (!std.ip(req.http.X-Forwarded-For, "0.0.0.0") ~ purge_it_notes) {
                return (synth(405, "Not allowed."));
            }

            if (req.url == "/" || req.url == "/*") {
                ban("req.http.host == " + req.http.host);
                return(synth(200, "Entire cache has been cleared."));
            }
            return (purge);
        }

    } else {
        # Other domains - 404
        return (synth(404, "Domain not found"));
    }

    if (req.method != "GET" && req.method != "HEAD") {
        return (pipe);
    }

    return (hash);
}

sub vcl_backend_response {
    # TTL - it-notes.dragas.net
    if (bereq.http.host == "it-notes.dragas.net") {
        if (bereq.url ~ "\.(gif|jpg|jpeg|png|webp|ico|css|js)$") {
            set beresp.ttl = 1w;
            set beresp.grace = 1d;
            set beresp.keep = 7d;
            unset beresp.http.Set-Cookie;
            unset beresp.http.Cache-Control;
            set beresp.http.Cache-Control = "public, max-age=604800";
        } else {
            set beresp.ttl = 15m;
            set beresp.grace = 48h;
            set beresp.keep = 7d;
        }
    }

    # Remove some headers
    unset beresp.http.Server;
    unset beresp.http.X-Powered-By;
    unset beresp.http.Via;

    return (deliver);
}

sub vcl_deliver {
    # Add X-Cache header
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
    } else {
        set resp.http.X-Cache = "MISS";
    }

    std.log("Delivering content for " + req.url + " - Cache: " + resp.http.X-Cache);

    # Remove Varnish headers
    unset resp.http.Via;
    unset resp.http.X-Varnish;

    return (deliver);
}

sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }
    return (lookup);
}

sub vcl_hit {
    return (deliver);
}

sub vcl_miss {
    return (fetch);
}

sub vcl_purge {
    std.log("Purge executed for " + req.url);
    return (synth(200, "Purge successful"));
}

sub vcl_synth {
    set resp.http.Content-Type = "text/html; charset=utf-8";
    set resp.http.Retry-After = "5";
    synthetic({"<!DOCTYPE html>
        <html>
            <head>
                <title>"} + resp.status + " " + resp.reason + {"</title>
            </head>
            <body>
                <h1>Status "} + resp.status + " " + resp.reason + {"</h1>
                <p>"} + resp.reason + {"</p>
                <h3>Guru Meditation:</h3>
                <p>XID: "} + req.xid + {"</p>
                <hr>
                <p>Varnish cache server</p>
            </body>
        </html>
    "});
    return (deliver);
}

Configuring nginx

Now, modify the nginx configuration file at /usr/pkg/etc/nginx/nginx.conf. Set the number of worker processes to "auto" to take advantage of all server cores, and configure the reverse proxy for your site(s). Here's an example configuration:

[...]
worker_processes  auto;
[...]

server {
    server_name it-notes.dragas.net;

    location / {
        proxy_method $request_method;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_pass http://unix:/var/run/varnish.sock;
    }

    access_log /var/log/nginx/access.it-notes.dragas.net.log;
    error_log /var/log/nginx/error.it-notes.dragas.net.log;

    listen [::]:443 ssl;
    listen 443 ssl;
    http2 on;
    # If you're using acme.sh, just change the location of the certificates
    ssl_certificate /root/.lego/certificates/it-notes.dragas.net.crt;
    ssl_certificate_key /root/.lego/certificates/it-notes.dragas.net.key;
}

server {
    if ($host = it-notes.dragas.net) {
        return 301 https://$host$request_uri;
    }
    server_name it-notes.dragas.net;
    listen 80;
    listen [::]:80;
    return 404;
}

Starting Varnish and nginx

Finally, start the Varnish and nginx services:

service varnishd start
service nginx start

If everything is configured correctly, both Varnish and nginx will be up and running, ready to handle incoming connections.

Conclusion

Congratulations, you have successfully set up your own CDN on NetBSD. This solution is lightweight, stable, and fully under your control, allowing you to break free from the constraints of major service providers. With NetBSD's broad hardware support and minimal overhead, this setup can run on a wide variety of devices, making it a versatile choice for self-hosted solutions.

I'm using it as a test and as a read-only root filesystem with a RAM-only local cache for my blog on a Raspberry Pi Zero W (first edition), and as soon as I get the new FTTH, I'll probably make it accessible via IPv6 for Italy, putting it physically into production.

If your goal is geo-replication, you can use DNS providers that offer location-based routing or set up your own DNS infrastructure to manage and resolve requests according to the user’s location. With multiple reverse proxies, separate DNS servers, and a well-configured cache, you can achieve a highly resilient system with minimal risk of a single point of failure.

Make Your Own CDN with OpenBSD Base and Just 2 Packages

Stefano Marinelli — Thu, 29 Aug 2024 01:41:00 +0200

Introduction

This article is a "spin-off" from the previous post "Building a Self-Hosted CDN for BSD Cafe Media," which I recommend reading, based on FreeBSD. In that article, I showed how I addressed the issue of geo-replication and geo-distribution of BSD Cafe media content. If you prefer a NetBSD-based setup, there's an article that describes how to do it.

The internet today relies TOO MUCH on just a few big players. When one of them stops working, half the world is impacted because too many services, in my opinion, depend on them. “Too big to fail,” some might say. “Single Point of Failure,” I respond."

The strength of the internet has always been its extreme decentralization, which is now less evident due to this phenomenon.

In this article, I want to show how easy it is to create a self-hosted CDN using OpenBSD and just two external packages: Varnish and Lego.

Actually, only one package is truly needed (Varnish), and SSL certificates could be generated using the built-in acme-client in OpenBSD. However, this might be limiting since acme-client handles certificate generation via traditional methods (using a file in .well-known), but when dealing with a CDN and several reverse proxies listening, you don’t have perfect control over which one will receive the certificate generation validation request.

For convenience and practicality, I’ll use the excellent Lego tool-a Go application that supports many DNS authentication methods, including PowerDNS.

Installation

The steps are quite simple. After installing and updating OpenBSD (using the syspatch command), start by installing the two packages:

obcdn# pkg_add lego varnish
quirks-7.14:updatedb-0p0: ok
quirks-7.14 signed on 2024-08-24T13:35:23Z
quirks-7.14: ok
lego-4.16.1: ok
varnish-7.4.2:bzip2-1.0.8p0: ok
varnish-7.4.2:pcre2-10.37p2: ok
useradd: Warning: home directory `/var/varnish' doesn't exist, and -m was not specified
varnish-7.4.2: ok
The following new rcscripts were installed: /etc/rc.d/varnishd
See rcctl(8) for details.

The next step is to enable Varnish:

rcctl enable varnishd

Varnish has a generic startup script, but it's best to customize the startup options. To do this, modify the /etc/rc.conf.local file:

varnishd_flags="-j unix,user=_varnish,ccgroup=_varnish -f /etc/varnish/default.vcl -T localhost:9999 -a localhost:8080 -s default,500m"

This configuration sets Varnish with a 500 MB cache and listens on localhost, port 8080.

Next, rename the default VCL file to prepare for your own content:

mv /etc/varnish/default.vcl /etc/varnish/default.vcl.distrib

Create a new default.vcl file. Below is an example based on this blog (at the time of writing). You’ll need to adapt it according to your needs, especially if there are cookies or other dynamic content. Note that Varnish will fetch data from a specific backend accessed in http. If privacy is needed, consider creating a VPN between the backend and Varnish, as briefly mentioned in the previous article:

vcl 4.1;
import std;

# Backend - it-notes.dragas.net
backend it_notes {
    .host = "myOriginalServer";
    .port = "80";
}

# ACL - purge - it-notes.dragas.net
acl purge_it_notes {
    "authorizedIPForCachePurge";
}

sub vcl_recv {
    # it-notes.dragas.net
    if (req.http.Host == "it-notes.dragas.net") {
        set req.backend_hint = it_notes;
        set req.http.Host = "it-notes.dragas.net";

        # PURGE - it-notes.dragas.net
        if (req.method == "PURGE") {
            std.log("Purge request received for " + req.url);

            if (!std.ip(req.http.X-Forwarded-For, "0.0.0.0") ~ purge_it_notes) {
                return (synth(405, "Not allowed."));
            }

            if (req.url == "/" || req.url == "/*") {
                ban("req.http.host == " + req.http.host);
                return(synth(200, "Entire cache has been cleared."));
            }
            return (purge);
        }

    } else {
        # Other domains - 404
        return (synth(404, "Domain not found"));
    }

    if (req.method != "GET" && req.method != "HEAD") {
        return (pipe);
    }

    return (hash);
}

sub vcl_backend_response {
    # TTL - it-notes.dragas.net
    if (bereq.http.host == "it-notes.dragas.net") {
        if (bereq.url ~ "\.(gif|jpg|jpeg|png|webp|ico|css|js)$") {
            set beresp.ttl = 1w;
            set beresp.grace = 1d;
            set beresp.keep = 7d;
            unset beresp.http.Set-Cookie;
            unset beresp.http.Cache-Control;
            set beresp.http.Cache-Control = "public, max-age=604800";
        } else {
            set beresp.ttl = 15m;
            set beresp.grace = 48h;
            set beresp.keep = 7d;
        }
    }

    # Remove some headers
    unset beresp.http.Server;
    unset beresp.http.X-Powered-By;
    unset beresp.http.Via;

    return (deliver);
}

sub vcl_deliver {
    # Add X-Cache header
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
    } else {
        set resp.http.X-Cache = "MISS";
    }

    std.log("Delivering content for " + req.url + " - Cache: " + resp.http.X-Cache);

    # Remove Varnish headers
    unset resp.http.Via;
    unset resp.http.X-Varnish;

    return (deliver);
}

sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }
    return (lookup);
}

sub vcl_hit {
    return (deliver);
}

sub vcl_miss {
    return (fetch);
}

sub vcl_purge {
    std.log("Purge executed for " + req.url);
    return (synth(200, "Purge successful"));
}

sub vcl_synth {
    set resp.http.Content-Type = "text/html; charset=utf-8";
    set resp.http.Retry-After = "5";
    synthetic({"<!DOCTYPE html>
        <html>
            <head>
                <title>"} + resp.status + " " + resp.reason + {"</title>
            </head>
            <body>
                <h1>Status "} + resp.status + " " + resp.reason + {"</h1>
                <p>"} + resp.reason + {"</p>
                <h3>Guru Meditation:</h3>
                <p>XID: "} + req.xid + {"</p>
                <hr>
                <p>Varnish cache server</p>
            </body>
        </html>
    "});
    return (deliver);
}

Start Varnish and check if it starts correctly:

rcctl start varnishd

Generating SSL Certificates

Before configuring relayd, you’ll need to generate SSL certificates. Lego supports many DNS providers and provides clear and comprehensive examples, so I suggest reading its README file.

Important: Certificates generated by Lego are not directly compatible with relayd, so you must generate them in the correct format. Add the -k rsa4096 flag to the Lego command to obtain certificates compatible with relayd.

Once generated, the certificates will be in a subdirectory of the directory from which the command was launched. For example, if the command is run as root (which is unnecessary, but just for the example), the certificates will be in /root/.lego/certificates/.

relayd expects certificates in a specific location. Copy them to the appropriate directories. In my example:

obcdn# cp /root/.lego/certificates/it-notes.dragas.net.crt /etc/ssl/
obcdn# cp /root/.lego/certificates/it-notes.dragas.net.key /etc/ssl/private/

Remember to copy the files to the correct directories when renewing. You can also create symbolic links, this will make your life easier but it’s less secure.

Configuring relayd

It’s time to configure relayd. The file is /etc/relayd.conf, and here’s an example configuration:

log state changes
prefork 10

table <itnotes> { 127.0.0.1 }

http protocol "http" {
    match request header append "X-Forwarded-For" value "$REMOTE_ADDR"
    match request header append "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"

    match request path "/*.atom" tag "CACHE"
    match request path "/*.css" tag "CACHE"
    match request path "/*.gif" tag "CACHE"
    match request path "/*.html" tag "CACHE"
    match request path "/*.ico" tag "CACHE"
    match request path "/*.jpg" tag "CACHE"
    match request path "/*.webp" tag "CACHE"
    match request path "/*.js" tag "CACHE"
    match request path "/*.png" tag "CACHE"
    match request path "/*.rss" tag "CACHE"
    match request path "/*.svg" tag "CACHE"
    match request path "/*.xml" tag "CACHE"
    match request path "/*.ttf" tag "CACHE"
    match request path "/*.woff2" tag "CACHE"

    match response tagged "CACHE" header set "Cache-Control" value "public, max-age=604800"

    pass request header "Host" value "it-notes.dragas.net" forward to <itnotes>
}

http protocol "https" {
    match request header append "X-Forwarded-For" value "$REMOTE_ADDR"
    match request header append "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"

    match response header set "Referrer-Policy" value "no-referrer"
    match response header set "X-Content-Type-Options" value "nosniff"
    match response header set "X-Download-Options" value "noopen"
    match response header set "X-Frame-Options" value "SAMEORIGIN"
    match response header set "X-Permitted-Cross-Domain-Policies" value "none"
    match response header set "X-XSS-Protection" value "1; mode=block"
    match response header set "Strict-Transport-Security" value "max-age=15552000; includeSubDomains; preload"

    match request path "/*.atom" tag "CACHE"
    match request path "/*.css" tag "CACHE"
    match request path "/*.gif" tag "CACHE"
    match request path "/*.html" tag "CACHE"
    match request path "/*.ico" tag "CACHE"
    match request path "/*.jpg" tag "CACHE"
    match request path "/*.webp" tag "CACHE"
    match request path "/*.js" tag "CACHE"
    match request path "/*.png" tag "CACHE"
    match request path "/*.rss" tag "CACHE"
    match request path "/*.svg" tag "CACHE"
    match request path "/*.xml" tag "CACHE"
    match request path "/*.ttf" tag "CACHE"
    match request path "/*.woff2" tag "CACHE"

    match response tagged "CACHE" header set "Cache-Control" value "public, max-age=604800"
    tcp { nodelay, sack, socket buffer 65536, backlog 100 }

    tls { keypair "it-notes.dragas.net" }

    pass request header "Host" value "it-notes.dragas.net" forward to <itnotes>
}

relay "http" {
    listen on vio0 port 80
    protocol "http"

    forward to <itnotes> port 8080
}

relay "https" {
    listen on vio0 port 443 tls
    protocol "https"

    forward to <itnotes> port 8080
}

relay "https6" {
    listen on my:ip:v6:address::1 port 443 tls
    protocol "https"

    forward to <itnotes> port 8080
}

The content of the file is quite self-explanatory. Note that:

vio0 is the interface name-it should be modified based on the interface where relayd needs to listen.
I’ve configured relayd to listen on port 80 as well.
IPv4 and IPv6 listeners are separated. If IPv6 is not configured, simply comment out that part. Please, if you can, use IPv6. In a fairer world, everyone would have the right to at least one class of public addresses without having to pay exorbitant fees.
The "keypair" must correspond to the certificate and key names in /etc/ssl and /etc/ssl/private.

Test the configuration, enable, and start relayd:

obcdn# relayd -n
configuration OK
obcdn# rcctl enable relayd
obcdn# rcctl start relayd
relayd(ok)

Final Checks

The stack is ready. A ps command will show the process status:

obcdn# ps aux
USER       PID %CPU %MEM   VSZ   RSS TT  STAT   STARTED       TIME COMMAND
root         1  0.0  0.0   920   276 ??  I       7:10PM    0:00.01 /sbin/init
[...]
_varnish 44999  0.0  0.2  2256  2424 ??  S       8:27PM    0:00.05 varnishd: Varnish-Mgt -i obcdn.my.domain (varnishd)
_varnish 54481  0.0  8.8 34500 88876 ??  S       8:27PM    0:00.60 varnishd: Varnish-Child -i obcdn.my.domain (varnishd)
root     57753  0.0  0.5  4080  4820 ??  IU      8:32PM    0:00.03 /usr/sbin/relayd
_relayd  95940  0.0  0.4  2152  3892 ??  Spc     8:32PM    0:00.01 relayd: pfe (relayd)
_relayd  81032  0.0  0.4  2156  3716 ??  Spc     8:32PM    0:00.01 relayd: hce (relayd)
_relayd  80312  0.0  0.5  2748  5280 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  88919  0.0  0.5  2748  5268 ??  Ipc     8:32PM    0:00.04 relayd: relay (relayd)
_relayd  90186  0.0  0.5  2752  5272 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  32851  0.0  0.5  2736  5284 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  20270  0.0  0.5  2744  5252 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  98826  0.0  0.5  2752  5264 ??  Ipc     8:32PM    0:00.04 relayd: relay (relayd)
_relayd   6454  0.0  0.5  2744  5264 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  90102  0.0  0.5  2740  5276 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  60314  0.0  0.5  2748  5336 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  27246  0.0  0.5  2744  5284 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  87082  0.0  0.4  2100  4508 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  53308  0.0  0.4  2096  4496 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  11697  0.0  0.4  2092  4492 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  83636  0.0  0.4  2092  4500 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  74563  0.0  0.4  2096  4484 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  35910  0.0  0.4  2100  4508 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  24911  0.0  0.4  2096  4504 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  78649  0.0  0.4  2096  4496 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  89586  0.0  0.4  2096  4500 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  11989  0.0  0.4  2100  4500 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
[...]

In this case, both relayd and Varnish are running correctly.

Automating Certificate Renewal

As a final step, remember to create a script to renew the certificates, copy them to /etc/ssl and /etc/ssl/private, and restart relayd.

Conclusion

Congratulations, you now have your own free CDN, with your data, fully under your control. Portable, extendable, controllable, and outside of the big providers' grip.

If your goal is geo-replication, you should use one of the available methods. Some DNS providers allow selection based on the caller's location (like Bunny.net), or, as I prefer, install your own DNS and use tools to manage operations and resolutions, as briefly described in the previous article.

With multiple separate reverse proxies, separate DNS servers (on different providers and possibly different countries or continents) capable of checking if the reverse proxies are operational, you can achieve an extremely low likelihood of encountering a Single Point of Failure, as all components, once the cache is filled, will be nearly autonomous even in the event of a (temporary) backend outage - i.e., the original node.

Building a Self-Hosted CDN for BSD Cafe Media

Stefano Marinelli — Mon, 26 Aug 2024 08:41:00 +0200

Introduction

For just over a year, BSD Cafe's media was hosted on a FreeBSD physical server jail with an outgoing bandwidth of 250 Mbit/sec. To mitigate bandwidth congestion, I initially integrated Cloudflare with a tunnel to serve media (and only media) through Cloudflare. The goal was to georeplicate the media and reduce the load on my server. To do this, the media server had to be on a separate domain managed by Cloudflare since the DNS for the primary bsd.cafe domain was managed by Bunny.net.

The first step (mainly because I discovered that Bunny's DNS does not support IPv6) was to bring the DNS back in-house using two FreeBSD jails (running on different VPS providers), both powered by PowerDNS. PowerDNS supports LUA records, which would come in handy later.

In line with the principles of self-hosting and data ownership, I decided to remove Cloudflare. I created a dedicated subdomain (media.bsd.cafe) and configured the reverse proxy in front of the jail running Minio to respond to that domain. I also reconfigured Mastodon for the new address, and after some fine-tuning, everything worked seamlessly. However, this led to some bandwidth congestion when media was posted, resulting in slower download speeds for users, especially during peak times. This is because, once content is published and federated servers are notified, they all attempt to download the newly published content - media included - almost simultaneously.

Not wanting to abandon my media server (a dedicated jail with spinning disks, offering 4 TB of storage), I opted for a different approach that I’ll describe here, as it might be useful for similar setups.

While this setup was implemented on FreeBSD, the configuration and tools - Nginx, Varnish, WireGuard and PowerDNS - are compatible with many operating systems, including Linux, with only minor adjustments required.

The Approach: Building a Self-Hosted CDN

The idea is to create reverse proxies with local caching. These proxies would cache the content on the first request and serve it directly afterward. The proxies would be distributed across different regions, and the DNS would route requests to the nearest proxy based on the caller’s location. All this is achieved without relying on external CDNs, using self-managed tools instead.

To establish a direct connection between Minio and the reverse proxies, I configured WireGuard inside the jail. The reverse proxies connect via WireGuard, allowing them to access Minio securely as if they were on the same LAN.

No further changes were needed on the media jail itself.

Setting Up the Reverse Proxies

I began configuring the reverse proxies (also running FreeBSD jails, OpenBSD (setup described in another post) and NetBSD (also described in another post), hosted on different providers). The choice of Varnish is based on several factors, with the main ones being the ability to keep the cache in RAM (which means it can run on read-only systems) and the ability to flush the cache remotely. For example, with each change to my blog, I can choose whether to perform an immediate flush (such as for a new article or an error) or wait for the cache's "natural" expiration (such as for a typo or minor, non-critical changes).

First, I connected them via WireGuard to the Minio jail (I won’t detail the steps here; I’ve covered similar setups in other posts). Then, I installed Nginx and Varnish. A more granular setup would have Varnish on a separate jail, but this way, I can move the reverse proxy jails to different hosts with minimal hassle. Currently, these reverse proxies also serve this blog.

Next, I installed and configured Varnish inside the jail:

pkg install varnish7

I created the directory /usr/local/etc/varnish and wrote a custom VCL file to manage this setup, named default.vcl:

vcl 4.1;
import std;

# Backend - it-notes.dragas.net
backend it_notes {
    .host = "itnotesip";
    .port = "itnotesport";
}

# Backend - media.bsd.cafe
backend media_bsd {
    .host = "minioWGip";
    .port = "minioport";
}

# ACL - IPs allowed to purge - it-notes.dragas.net
acl purge_it_notes {
    "a.b.c.d";
}

# ACL - IPs allowed to purge - media.bsd.cafe
acl purge_media_bsd {
    "e.f.g.h";
}

sub vcl_recv {

    # it-notes.dragas.net
    if (req.http.Host == "it-notes.dragas.net") {
        set req.backend_hint = it_notes;
        set req.http.Host = "it-notes.dragas.net";

        # PURGE - it-notes.dragas.net
        if (req.method == "PURGE") {

            std.log("Purge request received for " + req.url);

            if (!std.ip(req.http.X-Real-IP, "0.0.0.0") ~ purge_it_notes) {
                return (synth(405, "Not allowed."));
            }

        if (req.url == "/" || req.url == "/*") {
                ban("req.http.host == " + req.http.host);
                return(synth(200, "Entire cache has been cleared."));
        }
            return (purge);
        }

    # media.bsd.cafe
    } elsif (req.http.Host == "media.bsd.cafe") {
        set req.backend_hint = media_bsd;
        set req.http.Host = "media.bsd.cafe";

        # PURGE - media.bsd.cafe
        if (req.method == "PURGE") {
            if (!std.ip(req.http.X-Real-IP, "0.0.0.0") ~ purge_media_bsd) {
                return (synth(405, "Not allowed."));
            }
            if (req.url == "/" || req.url == "/*") {
                ban("req.http.host == " + req.http.host);
                return(synth(200, "Entire cache has been cleared."));
            }
            return (purge);
        }

    } else {
        # Other domains - 404
        return (synth(404, "Domain not found"));
    }

    if (req.method != "GET" && req.method != "HEAD") {
        return (pipe);
    }

    return (hash);
}

sub vcl_backend_response {
    # TTL - it-notes.dragas.net
    if (bereq.http.host == "it-notes.dragas.net") {
        if (bereq.url ~ "\.(gif|jpg|jpeg|png|ico|css|js)$") {
            set beresp.ttl = 1w;
            set beresp.grace = 1d;
            set beresp.keep = 7d;
            unset beresp.http.Set-Cookie;
            unset beresp.http.Cache-Control;
            set beresp.http.Cache-Control = "public, max-age=604800";
        } else {
            set beresp.ttl = 15m;
            set beresp.grace = 48h;
            set beresp.keep = 7d;
        }

    # TTL - media.bsd.cafe
    } elsif (bereq.http.host == "media.bsd.cafe") {
        if (bereq.url ~ "\.(mp4|mp3|wav|flac|ogg)$") {
            set beresp.ttl = 1d;
            set beresp.grace = 6h;
            set beresp.keep = 3d;
            unset beresp.http.Set-Cookie;
            unset beresp.http.Cache-Control;
            set beresp.http.Cache-Control = "public, max-age=86400";
        } else {
            set beresp.ttl = 30m;
            set beresp.grace = 12h;
            set beresp.keep = 3d;
        }
    }

    # Remove some headers
    unset beresp.http.Server;
    unset beresp.http.X-Powered-By;
    unset beresp.http.Via;

    return (deliver);
}

sub vcl_deliver {
    # ADD header X-Cache
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
    } else {
        set resp.http.X-Cache = "MISS";
    }

  std.log("Delivering content for " + req.url + " - Cache: " + resp.http.X-Cache);


    # Remove Varnish headers
    unset resp.http.Via;
    unset resp.http.X-Varnish;

    return (deliver);
}

sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }
    return (lookup);
}

sub vcl_hit {
    return (deliver);
}

sub vcl_miss {
    return (fetch);
}

sub vcl_purge {
    std.log("Purge executed for " + req.url);
    return (synth(200, "Purge successful"));
}

sub vcl_synth {
    set resp.http.Content-Type = "text/html; charset=utf-8";
    set resp.http.Retry-After = "5";
    synthetic( {"<!DOCTYPE html>
        <html>
            <head>
                <title>"} + resp.status + " " + resp.reason + {"</title>
            </head>
            <body>
                <h1>Status "} + resp.status + " " + resp.reason + {"</h1>
                <p>"} + resp.reason + {"</p>
                <h3>Guru Meditation:</h3>
                <p>XID: "} + req.xid + {"</p>
                <hr>
                <p>Varnish cache server</p>
            </body>
        </html>
    "} );
    return (deliver);
}

This setup allows Varnish to handle both domains with distinct configurations but within the same cache.

To enable Varnish, I updated the /etc/rc.conf file with the following lines, setting a maximum cache size of 2GB:

varnishd_enable="YES"
varnishd_listen="127.0.0.1:8080"
varnishd_config="/usr/local/etc/varnish/default.vcl"
varnishd_storage="default,2000M"

You can now start Varnish:

service varnishd start

The next step is to create two virtual hosts on Nginx (one for it-notes.dragas.net and one for media.bsd.cafe) that will listen on both IPv4 and IPv6 for HTTP and HTTPS. HTTP connections will be redirected to HTTPS, and incoming HTTPS traffic will be passed to Varnish, which will either return cached data or fetch it from the original server (Minio via WireGuard, for media.bsd.cafe). Let's see the media.bsd.cafe part:

server {
   server_name  media.bsd.cafe;

   [...]

   location / {
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_connect_timeout 300;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    chunked_transfer_encoding off;

    expires 12h;
    add_header Cache-Control public;

    add_header X-Cache-Status $upstream_cache_status;
    add_header X-Content-Type-Options nosniff;

    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
    add_header Referrer-Policy "no-referrer-when-downgrade";
    add_header Permissions-Policy "geolocation=(), microphone=(), camera=()";

    proxy_pass http://127.0.0.1:8080;

    [...]

}

[...]

}

server {
    if ($host = media.bsd.cafe) {
        return 301 https://$host$request_uri;
    }
   listen       80;
   listen  [::]:80;
   server_name  media.bsd.cafe;
    return 404;
}

This configuration isn’t complete, but it provides a good idea of how to set up Nginx - each setup will vary. TTL and caching sizes will also differ based on the characteristics of each reverse proxy. For example, one of the proxies has an 8GB cache since I have ample resources there.

Generating certificates is another important aspect. In this case, as the reverse proxies are distributed, they all need to respond to the same addresses. One approach is to generate the certificate on one proxy and distribute it to the others. In my case, I opted to use lego, which, through PowerDNS’s API, adds a DNS record for validation. This way, each reverse proxy can independently generate and renew its certificates when needed.

Configuring DNS for Optimal Routing

Once everything is set up, it’s important to ensure that DNS responds correctly. In my case, I implemented a strategy like this:

Track reverse proxies that respond on port 443 (further refinements are possible and will be done later).
Return the closest reverse proxy based on the client’s IP address.

Unfortunately, PowerDNS on FreeBSD does not include GeoIP support by default, but I have my poudriere ready to compile and install the necessary packages. Alternatively, you could compile it within the jail using the port system.

After that, I installed the geoipupdate package (which requires a free license from MaxMind), updated the IP list, and configured PowerDNS to use the GeoIP database. I added the GeoIP backend alongside the existing SQLite3 backend and specified the database to use:

launch=gsqlite3,geoip
geoip-database-files=mmdb:/usr/local/share/GeoIP/GeoLite2-City.mmdb

Finally, I created a LUA record to return the correct address:

testclosest.bsd.cafe   60      IN      LUA     A "ifportup(443, {'proxy1ip', 'proxy2ip','proxy3ip'}" ", {selector='pickclosest'})"
testclosest.bsd.cafe   60      IN      LUA     AAAA "ifportup(443, {'proxy1ip6', 'proxy2ip6','proxy3ip6'}" ", {selector='pickclosest'})"

And voilà! We now have a small, self-hosted CDN, keeping full control and ownership of our data. Adding a new reverse proxy is straightforward - simply clone an existing proxy, update the WireGuard configuration (adding a peer on the Minio jail and changing the keys on the new proxy), and add it to the DNS.

Happy caching!

Evolving the BSD Cafe Network Setup: From Bridging to Routing with FreeBSD

Stefano Marinelli — Thu, 01 Aug 2024 13:05:00 +0200

In the ever-changing landscape of system administration, network configurations often need to grow and evolve to meet new challenges and requirements. This post details my journey from a simple VPS setup to a complex, multi-node network using FreeBSD, jails, VPNs, and advanced routing techniques. Along the way, I'll explore the reasons behind each change and delve into why certain solutions, while functional, may not always be ideal in the long run.

Initial Setup: The Single VPS

My story begins with a single VPS (let's call it VPSSmall) hosted on Hetzner, running FreeBSD. This initial configuration was straightforward and served its purpose well for a time:

Created an internal bridge (bridge0) with IP 192.168.123.1
Used BastilleBSD to create VNET jails with IPs in the 192.168.123.X range
Set up port forwarding to the jails using pf's rdr rules
Utilized a /64 IPv6 block, subdivided into /72 subnets for the bridge and jails

This setup allowed for easy management of multiple services within isolated jails, all sharing the same network namespace.

Growing Pains: Adding a Second VPS

As is often the case in system administration, my needs grew over time. The number of jails increased, and their resource requirements expanded. To address this, I added a second VPS (VPSBig) hosted on a Proxmox server. This VPS isn't directly exposed, and relies on NAT to connect to the outside world. This introduced a new challenge: how to maintain flexibility in moving jails between VPSs without changing their network configurations?

To solve this, I implemented the following setup:

Installed ZeroTier on both VPSs in bridge mode
Created bridge0 on VPSBig (without an IP)
Added ZeroTier interfaces to both bridges

This configuration allowed for seamless movement of jails between nodes by simply transferring the ZFS dataset. The jails could retain their IP addresses regardless of which physical VPS they were running on.

While this setup was functional, it had some drawbacks that I'll discuss in the next section.

The Limitations of Bridging

The bridged setup using ZeroTier, while effective, wasn't without its issues. Here's why I found bridging, in this case, wasn't an ideal long-term solution:

Performance Overhead: Bridging all traffic between VPSs can introduce additional latency and processing overhead, especially when dealing with high-volume traffic.
Scalability Concerns: As the number of VPSs and jails grows, managing a large bridged network becomes increasingly complex. Each new node added to the network increases the potential for broadcast storms and can lead to unnecessary traffic across the entire network.
Security Implications: In a bridged network, all nodes essentially exist on the same network segment. This can potentially allow for lateral movement between jails or VPSs if not carefully managed, increasing the attack surface.
Dependency on ZeroTier: While ZeroTier is a powerful tool, relying on a third-party service for critical infrastructure introduces an external point of failure and potential security considerations.
Limited Control: Bridging provides less granular control over traffic flow compared to routing. This can make it harder to implement complex network policies or optimize traffic paths.
Broadcast Domain Size: Large bridged networks can result in expansive broadcast domains, which can lead to increased network congestion and reduced overall performance.

These limitations prompted me to seek a more robust, scalable, and controllable solution, leading to the next evolution of my network setup.

Refining the Setup: Wireguard and VXLAN

To address the limitations of the bridged setup, I implemented a new configuration involving Wireguard and VXLAN:

Created a Wireguard VPN between VPSSmall and VPSBig
Implemented a VXLAN over the Wireguard tunnel
Replaced ZeroTier with the VXLAN for inter-VPS communication

This setup, described in detail here, offered several advantages:

Improved Security: Wireguard provided a secure, encrypted tunnel between the VPSs.
Better Performance: Direct Wireguard VPN connection often results in lower latency compared to ZeroTier.
Greater Control: By managing all the components of the VPN myself, I have more control over the network configuration and problems.
Reduced Dependency: Eliminating ZeroTier removed a third-party dependency from my critical infrastructure.

While this setup was a significant improvement, it still relied on bridging via VXLAN, which didn't fully address all the scalability and control issues. This realization led to the final evolution of my network.

The Final Evolution: Routing Instead of Bridging

The last step in my network's evolution was to move from a bridged to a routed setup. This change offered even more flexibility and scalability. The new configuration:

VPSSmall: Uses 192.168.122.x/24 and a /72 IPv6 subnet
VPSBig: Uses 192.168.123.x/24 and its original /72 IPv6 subnet
Future nodes can use new private IPv4 ranges and /72 IPv6 subnets as needed

This routed setup provides several key benefits:

Improved Scalability: Each VPS or future node can have its own subnet, making it easier to add new nodes without reconfiguring the entire network.
Better Traffic Control: Routing allows for more granular control over traffic flow, enabling complex network policies and optimizations.
Enhanced Security: With distinct subnets, it's easier to implement security policies and control inter-subnet communication.
Reduced Broadcast Domain: Each subnet forms its own broadcast domain, reducing unnecessary network traffic and improving overall performance.

To ensure jails on VPSBig use the Wireguard tunnel for outgoing traffic while VPSBig itself uses its default gateway, I leveraged FreeBSD's FIB feature. This allows for separate routing tables, providing even more flexibility in managing network traffic.

Implementing Multiple FIBs

Edit /etc/sysctl.conf to enable multiple FIBs:

net.fibs=2

This allows me to use two separate routing tables.

VPSBig Wireguard Configuration

[Interface]
PrivateKey = *VPSBigPrivateKey*
Address = 10.77.0.2/32,oneOfMyIpv6/128

Table = off
PostUp = route -q -n add -inet 0.0.0.0/0 -interface wg0 -fib 1
PostUp = route -q -n add -inet6 ::/1 -interface wg0 -fib 1
PostUp = route -q -n add -inet6 8000::/1 -interface wg0 -fib 1

[Peer]
PublicKey = *VPSSmallPublicKey*
AllowedIPs = 0.0.0.0/0,::0/0
Endpoint = *endpointip:port*
PresharedKey = *presharedkey*
PersistentKeepalive = 30

Let's break down this configuration:

Table = off: This disables Wireguard's automatic routing table management. We're doing this because we want to manually configure the routing.
The PostUp commands are crucial for our manual routing setup:
route -q -n add -inet 0.0.0.0/0 -interface wg0 -fib 1: This adds a default route for IPv4 traffic through the Wireguard interface (wg0) in the alternate routing table (FIB 1).
The next two commands do the same for IPv6 traffic, covering the entire IPv6 address space (::/1 and 8000::/1 together cover all IPv6 addresses).
AllowedIPs = 0.0.0.0/0,::0/0: This tells Wireguard to route all traffic through this peer. However, because we've set Table = off, Wireguard won't actually create these routes - we're doing it manually with our PostUp commands.
PersistentKeepalive = 30: This sends a keepalive packet every 30 seconds, which is necessary because VPSBig is behind NAT and needs to keep the NAT session alive.

VPSSmall Wireguard Configuration

[Interface]
PrivateKey = *VPSSmallPrivateKey*
ListenPort = *port*
Address = 10.77.0.1/24,oneOfMyIpv6/128

[Peer]
PublicKey = *VPSSmallPublicKey*
PresharedKey = *presharedkey*
AllowedIPs = 10.77.0.2/32, 192.168.123.0/24, *theRemote/72ipv6class*/72

In this configuration:

Address = 10.77.0.1/24,oneOfMyIpv6/128: This sets up the Wireguard interface with an IPv4 and IPv6 address.
AllowedIPs = 10.77.0.2/32, 192.168.123.0/24, *theRemote/72ipv6class*/72: This tells Wireguard to route traffic for VPSBig's Wireguard IP (10.77.0.2), VPSBig's local subnet (192.168.123.0/24), and VPSBig's IPv6 subnet through this peer.

The Role of FIB in Routing

The use of FIB (Forwarding Information Base) 1 in the VPSBig configuration is key to our setup. By adding routes to FIB 1, we're creating a separate routing table that can be used by our jails. This allows us to:

Keep the main system (VPSBig itself) routing through its default gateway.
Route all traffic from the jails through the Wireguard tunnel to VPSSmall.

We achieve this by configuring the jails to use FIB 1:

exec.prestart += "ifconfig epairXa fib 1";
exec.prestart += "ifconfig epairXb mtu 1380";

This ensures that each jail uses the alternate routing table (FIB 1) instead of the default routing table, effectively sending all its traffic through the Wireguard tunnel.

If the WireGuard MTU is 1420, setting the jail's MTU to 1380 should be safe enough.

NAT Configuration on VPSSmall

To complete the setup, we need to configure NAT on VPSSmall to allow the jails on VPSBig to access the internet:

nat on vtnet0 from 192.168.123.0/24 to ! <private> -> vtnet0:0

This NAT rule translates the source IP of packets coming from VPSBig's subnet (192.168.123.0/24) to VPSSmall's public IP when they're destined for non-private IP addresses.

Conclusion

My journey from a simple single-VPS setup to this complex, multi-node network illustrates the power and flexibility of FreeBSD. By transitioning from a bridged to a routed setup, I've created a solution that offers improved scalability, security, and control.

Key takeaways from this evolution include:

Adaptability is Crucial: As your needs grow, be prepared to evolve your network architecture.
Understand the Tradeoffs: Each networking approach (bridging, VPNs, routing) has its pros and cons. Choose the one that best fits your current and future needs.
Leverage Advanced Features: FreeBSD's features like jails, FIBs, and pf allow for powerful and flexible network configurations.
Security is Paramount: Always consider the security implications of your network design, especially when dealing with multiple nodes and public-facing services.
Documentation is Key: Keep detailed notes of your network evolution. It helps in troubleshooting and future planning.

Whether you're managing a small personal server or a large-scale infrastructure, the techniques described here can help you build a robust and adaptable network. Remember, network design is an iterative process. Don't be afraid to evolve your setup as your needs change.

Bridging Networks Across VPS with Wireguard and VXLAN on FreeBSD

Stefano Marinelli — Mon, 15 Jul 2024 08:41:00 +0200

Introduction

In today's interconnected world, system administrators often face the challenge of managing services across multiple Virtual Private Servers (VPS). This article describes an advanced networking setup that allows you to bridge networks between two VPS instances using Wireguard and VXLAN on FreeBSD. This configuration is particularly useful when you need to distribute services across different providers or when you want to leverage the strengths of multiple hosting environments.

Background

At BSD Cafe, we utilize various VPS instances to provide our services. The two main ones are:

A publicly accessible VPS that hosts the reverse proxy and all firewall rules for packet routing.
A larger VPS on a physical host I own, which is not directly exposed to the internet and doesn't have a public IP address.

Most of the jails hosting BSD Cafe services are distributed between these two VPS instances. Occasionally, I need to move services between them for performance reasons or to manage updates efficiently.

To facilitate this flexibility, I've always maintained a bridge on each VPS. Initially, I used Zerotier to establish a connection between these bridges, allowing them to communicate as if they were part of a single, large network.

The New Setup: Wireguard and VXLAN

While the Zerotier setup worked, I decided to switch to a more streamlined solution using Wireguard and VXLAN. Here's why:

Performance: Wireguard offers excellent performance with low overhead.
Simplicity: The configuration is straightforward and easy to maintain.
Security: Wireguard provides strong, modern cryptography.

I had already prepared a Wireguard connection between the two servers from the beginning. Since only one of the servers is publicly accessible, I set up one to only accept connections and the other to connect directly to the public IP of the first, with a 20-second keepalive (which is generally not necessary due to the high traffic between the jails).

To complete the setup, I added two VXLAN interfaces on the VPS instances, added these interfaces to the local bridges, and immediately, packets started flowing between the networks.

Step-by-Step Implementation

Follow these instructions to create a bridge between two different networks using Wireguard and VXLAN on FreeBSD. While I use this setup to connect jails at BSD Cafe, you can use it for various purposes, such as bridging different VM (bhyve) instances across providers.

Prerequisites

Wireguard is now an integral part of FreeBSD, so you no longer need to compile a module or use the Go version. However, we'll use the "wireguard-tools" scripts as they provide the useful "wg-quick" command.

Start by installing the wireguard-tools package on both servers:

pkg install wireguard-tools

Configuration

Server 1 (Public IP)

Generate the Wireguard keys:

wg genkey | tee /dev/stderr | wg pubkey | grep --label PUBLIC -H .

This command will output a private key and a public key. Note down the public key as you'll need it to configure the client.

Let's also add a PSK; it's optional but will increase the security of the entire setup.

wg genpsk

Create a new file /usr/local/etc/wireguard/wg0.conf:

[Interface]
## Default port is 51820 - feel free to change it
PrivateKey = <the private key from the previous command>
ListenPort = 43671
Address = 10.77.0.1/24

PostUp = /sbin/ifconfig vxlan create vxlanid 42 vxlanlocal 10.77.0.1 vxlanremote 10.77.0.2 inet 10.77.1.1/24
PostUp = /sbin/ifconfig bridge0 addm vxlan0 up
PostDown = /sbin/ifconfig vxlan0 destroy

[Peer]
PublicKey = <the other peer's public key>
#If publicly exposed, you can specify the peer ip address/port
#Endpoint = <public_ip>:<port>
AllowedIPs = 10.77.0.2/32
PresharedKey = <the PSK from the previous command>

Modify /etc/rc.conf and add:

wireguard_interfaces="wg0"
wireguard_enable="YES"

Start Wireguard and the VXLAN endpoint:

wg-quick up wg0

Server 2 (Behind NAT)

Generate the Wireguard keys as before.
Create /usr/local/etc/wireguard/wg0.conf:

[Interface]
PrivateKey = <the private key from the previous command>
Address = 10.77.0.2/24

PostUp = /sbin/ifconfig vxlan create vxlanid 42 vxlanlocal 10.77.0.2 vxlanremote 10.77.0.1 inet 10.77.1.2/24
PostUp = /sbin/ifconfig bridge0 addm vxlan0 up
PostDown = /sbin/ifconfig vxlan0 destroy

[Peer]
PublicKey = <the other peer's public key>
Endpoint = <public_ip>:<port>
AllowedIPs = 10.77.0.1/32
PresharedKey = <the PSK from the previous command>
PersistentKeepalive = 20

Modify /etc/rc.conf as before.
Start Wireguard and the VXLAN endpoint:

wg-quick up wg0

Verifying the Connection

To check if the connection is established, run the wg command on either host. This will show you the connection status, the last handshake, and the data transferred.

You can also try pinging the other host's Wireguard and VXLan interface IP address (in this example, 10.77.0.1 or 10.77.0.2 and 10.77.1.1 or 10.77.1.2).

Conclusion

This setup allows the two VXLAN interfaces, inserted into the local bridge, to enable packet transit through Wireguard. This facilitates free passage between the two hosts, effectively creating a single, unified network across your VPS instances.

This configuration is particularly useful for: - Distributing services across different providers - Leveraging both public-facing and private VPS instances - Creating flexible, scalable network architectures

By using Wireguard and VXLAN, you get the benefits of strong encryption, high performance, and the ability to create complex network topologies across physically separate servers.

Remember to always keep your systems updated and regularly review your network configuration to ensure it meets your evolving needs and security requirements.

Migrating from an Old Linux Server to a New FreeBSD Machine

Stefano Marinelli — Wed, 25 Oct 2023 16:39:37 +0000

Preamble: I believe it's time to bid farewell to this venerable Linux server.

TL;DR: The article chronicles the journey of transitioning from an outdated Linux server, running for 1690 days without updates, to a modern FreeBSD machine. This migration involved using tools like mfsBSD, BastilleBSD, Borg Backup, and bhyve. Despite initial hesitations due to the Linux server's impeccable performance, the transition was smooth, resulting in improved manageability and efficiency. The piece emphasizes the importance of regular system updates and anticipates revisiting the topic in the future with new uptime achievements and updates.

This server loyally served for years as a secondary backup server while also providing a few minor services to users. As it often happens, it remained in operation, neglected and without updates for years. Stable operating systems have the "flaw" of being forgotten, giving the false impression that they don't need maintenance or updates. This machine continued its service without oversight for years. When approached for a service request (not due to malfunctions), I advised the client to upgrade the whole system. A mere update would not suffice, so I suggested starting afresh on new hardware with FreeBSD as the primary OS.

The client was understandably hesitant given the uptime stats:

08:58:43 up 1690 days, 21:32, 4 users, load average: 9.57, 10.15, 8.76

Not a single error, not a single hiccup. From his perspective, a similar setup to what was installed many years ago and still working flawlessly was preferred. Nevertheless, he trusted my expertise and let me proceed.

This server had a plethora of duties, among which:

One of the pivotal tasks was running Proxmox Backup Server via Docker. Proxmox Backup Server requires Debian, but this server was running on Ubuntu 16.04 (previously upgraded from Ubuntu 14.04 – yes, ancient!). Hence, Proxmox Backup Server was still at version 1.x.
Another critical function was storing backups made through BorgBackup on its file system. The /home directory used a mirrored btrfs file system, and each backed-up server had its user on this system. Clients could backup (using a push method) only via VPN and only during specific windows when the server permitted (by adding specific firewall rules via Jenkins. Jenkins also managed connection protocols, snapshots, backups, etc.).
Among the lesser tasks, the server ran a few Docker containers with HandBrake on various presets. The client processed video conversions by uploading the original files via sftp, and after some hours, fetched the converted files from the destination directory. This will not be replicated on the new FreeBSD server since they now handle this operation locally on their high-performance MacBook Pro with Apple Silicon. However, a future restoration isn't off the table.

The first thing I did was install FreeBSD on the new hardware. Given that it's a physical server on Hetzner (an auction pick due to disk space needs over power), and FreeBSD wasn't an option, I used mfsBSD.

After booting the physical server in Linux rescue, I copied mfsBSD onto the disks using dd and restarted. On boot, I SSHed into mfsBSD and executed the installation using bsdinstall-a robust and efficient method.

I set up a raidz1 with all four 6 TB disks, resulting in a final storage space of 21.8T, ample for now without the video files.

To ensure continuity, I kept a setup similar to the old one. The clients would essentially continue with their usual backup procedure without necessitating drastic changes to backup scripts. To avoid storing these backups directly in the physical machine's /home and to leave the door open for future services, I installed BastilleBSD and began setting up several jails. I replaced the old Linux machine's behavior with a VNET FreeBSD jail. In past scenarios, I've created Linux jails (thanks to BastilleBSD) and transferred the old server into the jail using rsync, making minor configuration tweaks. While this usually works, it doesn't address the underlying issue of an obsolete setup. Given the opportunity, I opted for a modern toolset.

Thus, I copied every home directory (along with their historic backups) in its entirety, installed BorgBackup, and re-established the VPN. With a VNET jail, I can craft networking devices and fine-tune configurations. After recreating user accounts, inputting the various SSH authorized_keys, and checking all clients, I set up a snapshot plan on the host. This ensures that if a client is compromised with the potential (however remote) for breach and backup deletion, a ZFS snapshot of the entire jail remains available.

As mentioned, one of the core tools on the old server was Proxmox Backup Server. It's not natively installable on FreeBSD, necessitating a VM. Enter the fantastic bhyve, supported by vm-bhyve.

However, one issue arose: backups would consume vast amounts of space, and I wanted to avoid housing an enormous disk image (or a zvol) with both VMs and backups. So, I opted for a slightly less performant but more flexible solution: installing Debian 12 and Proxmox Backup Server on the VM while placing backups on a separate ZFS dataset on the physical machine, exported via NFS and mounted on the VM.

Given that the physical server has an internal bridge "vm-public" with IP 192.168.124.1 and the VM is at 192.168.124.2, I just created a dataset named zroot/PBS and added the following line to /etc/exports:

/zroot/PBS -alldirs -maproot=root -network 192.168.124.2/32

To enable NFS, insert into /etc/rc.conf:

rpcbind_enable="YES" 
nfs_server_enable="YES" 
mountd_flags="-r" 
rpc_lockd_enable="YES"

Within the VM, create /PBS and include in /etc/fstab:

192.168.124.1:/zroot/PBS /PBS nfs rw,async,soft,intr,noexec 0 0

After Proxmox Backup Server's installation, simply set up the datastores in /PBS/, and they'll directly store on the physical machine's ZFS dataset.

For firewall configurations, I exposed port 8007, redirecting it towards the VM, and everything started functioning smoothly. I then set a Proxmox Backup Server replica from the old to the new server. After completion, I changed the Proxmox Backup Server IP on all Proxmox hosts to point to the new server. Smooth sailing.

The old Ubuntu server also managed other minor services, which have become obsolete and weren't replicated.

The transition was seamless, the client is pleased, and I'm content since each service is now neatly segregated into its jail or VM. The machine's load is minimal, which might pave the way for other tasks, via VPN. Everything now rests on ZFS, and the icing on the cake: I made the client promise not to reach another 1690 days of uptime but to timely update as required.

I'm not entirely convinced the promise will hold-meaning, in a few years, I might yet be discussing this "new" server, highlighting another impressive uptime and another upgrade journey.

Make your own VPN - FreeBSD, Wireguard, ipv6 and ad-blocking included

Stefano Marinelli — Sat, 23 Sep 2023 09:05:51 +0000

Note: This article assumes a setup based on FreeBSD. If you prefer a version based on OpenBSD, it is available here.

VPNs are a fundamental tool for securely connecting to your own servers and devices. Many people use commercial VPNs for various reasons, ranging from not trusting their provider (especially when connecting from a public hotspot) to wanting to "go out" on the Internet with a different IP address, perhaps from another country.

Whatever the reason, solutions are not lacking. I have always set up management VPNs to allow servers and/or clients to communicate with each other using secure channels. Lately, I have been activating IPv6 connectivity on all my devices (both desktop/servers and mobile devices) and I needed to quickly create a node that concentrated some networks and allowed them to go out on the network in IPv6. The tools I used and will describe are:

VPS - in this case, I used a basic Hetzner Cloud VPS (using this link, you will receive 20 euros of cloud credits), but any provider that provides IPv6 connectivity will do - if you want IPv6, of course.
FreeBSD - a versatile, stable, and secure operating system.
Wireguard - lightweight, secure, and at the same time, not very "chatty", so it is also gentle on mobile device batteries. When there is no traffic, it simply does not transmit/receive anything. Well supported by all major desktop and server operating systems as well as Android and iOS devices.
Unbound - can make DNS queries directly to root servers, not through forwarders. It also allows you to insert block-lists and have a result similar to that of Pi-Hole (i.e., ad-blocking).
SpamHaus lists - to immediately stop connections to and from users on blacklists.

The first step is to activate a VPS and install FreeBSD. On the Hetzner Cloud console, there might not be a pre-built FreeBSD image, but only a selection of Linux distributions. Don't worry, just choose any of them and create the VPS. Once done, the FreeBSD ISO image will be available among the "ISO Images". Just insert the virtual CD, restart the VPS, and the FreeBSD installation will appear in the console.

I won't go into detail, the operation is simple and straightforward. The only precaution (in the case of a Hetzner Cloud VPS) is to use "DHCP" for IPv4 but, for now, do not configure IPv6. It will be configured later.

Install all FreeBSD updates (using the freebsd-update fetch install command) and reboot.

It seems, moreover, that there is still an old bug still present and manifesting when, on a FreeBSD server installed in a VM based on KVM (thus also tho se of Hetzner), routing is performed (as in our case) between VNET jails and host.

Adding this configuration to /boot/loader.conf will solve the problem:

hw.vtnet.X.tso_disable="1"
hw.vtnet.tso_disable="1"
hw.vtnet.lro_disable="1"
hw.vtnet.X.lro_disable="1"
hw.vtnet.csum_disable="1"
hw.vtnet.X.csum_disable="1"

Wireguard, on FreeBSD, is now available as a kernel module and the userland can be installed using the pkg install wireguard-tools package manager. This means you can easily keep it updated alongside other software on the system.

The first step is to configure IPv6 on the VPS. In the case of Hetzner, unfortunately, they only provide a /64, so it will be necessary to segment the assigned network. In this example, it will be divided into /72 subnetworks - to find valid subclasses, it will be possible to use a calculator.

The /etc/rc.conf file should have entries similar to:

ifconfig_vtnet0="DHCP"
ifconfig_vtnet0_ipv6="inet6 2001:db8:cafe:cafe::1 prefixlen 72"
ipv6_defaultrouter="fe80::1%vtnet0"

In short, keep the base address assigned by Hetzner, but change the prefix length to 72 - thus giving the possibility of having other networks available.

It is now necessary to enable forwarding for IPv4 and IPv6. Add these lines to the /etc/sysctl.conf file:

net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1

After reboot, test it:

ping6 google.com

If everything has been configured correctly, the ping will be executed and google.com will reply.

To configure Wireguard, a few steps will be necessary. First of all, the private key will need to be created:

wg genkey | tee /dev/stderr | wg pubkey | grep --label PUBLIC -H .

You will get a private key and a public key. Take note of the public key - it will be needed to configure the clients.

Now create a new file called /usr/local/etc/wireguard/wg0.conf:

[Interface]
Address = 172.16.0.1/24,2001:db8:cafe:cafe:100::1/72
ListenPort = 51820
PrivateKey = YUkS6cNTyPbXmtVf/23ppVW3gX2hZIBzlHtXNFRp80w=

A new Wireguard interface called wg0 is being created. Start the Wireguard interface:

service wireguard enable
sysrc wireguard_interfaces="wg0" 
service wireguard start

If everything has been entered correctly, the interface should come up. Check its status:

wg

As for the firewall, FreeBSD comes with no pf configuration. In my setups, I tend to block what is not needed and be permissive with what may be useful. However, I like to keep out the "bad guys," so I use blacklists. pf allows elements to be inserted and removed from tables at runtime, so the firewall can be configured accordingly.

To download and apply the Spamhaus lists, I use a simple but effective script found on the Internet, but for OpenBSD.

For the Spamhaus lists, continue with the FreeBSD script creation.

Create the script in /usr/local/sbin/spamhaus.sh:

#!/bin/sh
#
#this is normally run once per day via cron.
#
echo updating Spamhaus DROP lists:
(
  { fetch -o - https://www.spamhaus.org/drop/drop.txt && \
    fetch -o - https://www.spamhaus.org/drop/dropv6.txt ; \
  } 2>/dev/null | sed "s/;/#/" > /var/db/drop.txt
)
pfctl -t spamhaus -T replace -f /var/db/drop.txt

Make it executable and run it. Pf isn't enabled, so you'll get an error - but this will create the /var/db/drop.txt file:

chmod a+rx /usr/local/sbin/spamhaus.sh
/usr/local/sbin/spamhaus.sh

There are many possibilities to configure pf on FreeBSD. A fairly simple example could be this:

ext_if="vtnet0"
wg0_if="wg0"
wg0_networks="172.16.0.0/24"

set skip on lo

nat on $ext_if from { $wg0_networks } to any -> ($ext_if)

# Spamhaus DROP list:
table <spamhaus> persist file "/var/db/drop.txt"

block drop log quick from <spamhaus>

# Pass ICMP on ipv6
pass quick proto ipv6-icmp
# Block from ipv6 to wg0 network
block in quick on $ext_if inet6 to { 2001:db8:cafe:cafe:100::/72 }
# Pass Wireguard traffic - in and out
pass quick on $wg0_if

# default deny
block in
block out

pass in on $ext_if proto tcp to port ssh
pass in on $ext_if proto udp to port 51820

pass out on $ext_if

This is a very simple configuration: it blocks everything that is present in the list downloaded from Spamhaus, allows NAT from the Wireguard network to the public interface, allows ICMP traffic in IPv6 (necessary for the network to function properly) while blocking incoming traffic to the Wireguard IPv6 LAN (remember that the IPs will be public and directly reachable, so we don't want to expose our devices by default). All traffic on the Wireguard interface will be allowed to pass. Then everything will be blocked and exceptions will be specified, i.e., allowing SSH and Wireguard connections (of course). Authorization will also be granted to allow traffic to exit from the public network interface.

Save this configuration to /etc/pf.conf.

Enable and start pf:

service pf enable 
service pf start

You will probably be kicked out of the system. Don't worry, just reconnect. pf is doing its job.

If everything went correctly, the firewall should have loaded the new rules.

To obtain caching of DNS queries and the related ad-block, it is now time to configure Unbound. Let's install it with:

pkg install unbound

A while ago, I found a script which I slightly adapted. I don't remember where I got it, so I'll paste it here without citing the original creator.

Create a script to update the unbound ad-block, in /usr/local/sbin/unbound-adhosts.sh:

#!/bin/sh
#
# Using blacklist from pi-hole project https://github.com/pi-hole/
# to enable AD blocking in unbound(8)
#
PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin"

# Available blocklists - comment line to disable blocklist
_disconad="https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt"
_discontrack="https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt"
_stevenblack="https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts"

# Global variables
_tmpfile="$(mktemp)" && echo '' > $_tmpfile
_unboundconf="/usr/local/etc/unbound/unbound-adhosts.conf"

# Remove comments from blocklist
simpleParse() {
  fetch -o - $1 | \
  sed -e 's/#.*$//' -e '/^[[:space:]]*$/d' >> $2
}

# Parse DisconTrack
[ -n "${_discontrack}" ] && simpleParse $_discontrack $_tmpfile

# Parse DisconAD
[ -n "${_disconad}" ] && simpleParse $_disconad $_tmpfile

# Parse StevenBlack
[ -n "${_stevenblack}" ] && \
  fetch -o - $_stevenblack | \
  sed -n '/Start/,$p' | \
  sed -e 's/#.*$//' -e '/^[[:space:]]*$/d' | \
  awk '/^0.0.0.0/ { print $2 }' >> $_tmpfile

# Create unbound(8) local zone file
sort -fu $_tmpfile | grep -v "^[[:space:]]*$" | \
awk '{
  print "local-zone: \"" $1 "\" redirect"
  print "local-data: \"" $1 " A 0.0.0.0\""
}' > $_unboundconf && rm -f $_tmpfile

service unbound reload 1>/dev/null

exit 0

After saving the script, make it executable and run it:

chmod a+rx /usr/local/sbin/unbound-adhosts.sh
/usr/local/sbin/unbound-adhosts.sh

Now, the Unbound configuration file in /usr/local/etc/unbound/unbound.conf can be modified as follows:

server:
        verbosity: 1
        log-queries: no
        num-threads: 4
        num-queries-per-thread: 1024
        interface: 127.0.0.1
        interface: 172.16.0.1
        interface: 2001:db8:cafe:cafe:100::1
        interface: ::1
        outgoing-range: 64
        chroot: ""

        access-control: 0.0.0.0/0 refuse
        access-control: 127.0.0.0/8 allow
        access-control: ::0/0 refuse
        access-control: ::1 allow
        access-control: 172.16.0.0/24 allow
        access-control: 2001:db8:cafe:cafe:100::/72 allow

        hide-identity: yes
        hide-version: yes
        auto-trust-anchor-file: "/usr/local/etc/unbound/root.key"
        val-log-level: 2
        aggressive-nsec: yes
        prefetch: yes
        username: "unbound"
        directory: "/usr/local/etc/unbound"
        logfile: "/var/log/unbound.log"
        use-syslog: no
        pidfile: "/var/run/unbound.pid"
        include: /usr/local/etc/unbound/unbound-adhosts.conf

remote-control:
        control-enable: yes
        control-interface: /var/run/unbound.sock

Now, enable and start unbound:

service unbound enable
service unbound start

If everything has been set up correctly, unbound will be able to respond to DNS requests made on 172.16.0.1 and 2001:db8:cafe:cafe:100::1.

Now it is possible to configure the Wireguard client. Create a new configuration by inserting "172.16.0.2/32, 2001:db8:cafe:cafe:100::2/128" (the ones that will later be entered in the peer configuration of the server) in the local IP addresses. Set the DNS server address to "172.16.0.1" and/or its corresponding IPv6 address (in the example, 2001:db8:cafe:cafe:100::1 - yours will be different). In the peer section, insert the server's data, including its public key, IP address:port (in the example, the port is 51820), and allowed addresses (setting "0.0.0.0/0, ::0/0" means "all connections will be sent via Wireguard" - all the traffic will pass through the VPN for both IPv4 and IPv6).Each implementation has its own procedure (Android, iOS, MikroTik, Linux, etc.) but essentially it is sufficient to create the right configuration both on the server and on the client.

Reopen the Wireguard configuration file /usr/local/etc/wireguard/wg0.conf and add:

[Interface]
Address = 172.16.0.1/24,2001:db8:cafe:cafe:100::1/72
ListenPort = 51820
PrivateKey = YUkS6cNTyPbXmtVf/23ppVW3gX2hZIBzlHtXNFRp80w=

[Peer]
PublicKey = *client's public key*
AllowedIPs = 172.16.0.2/32, 2001:db8:cafe:cafe:100::2/128

The client's public key will be shown by the client itself.

Reload the Wireguard configuration:

service wireguard restart

It is also possible to use the VPN only as an ad-blocker, by only routing DNS traffic through it. To achieve this result, configure the client so that the only allowed address is the one of the just-configured unbound (in this example, 172.16.0.1 and/or 2001:db8:cafe:cafe:100::1) - DNS resolution will occur via VPN, but browsing will continue to work through the main provider.

To automatically update the spamhaus and ad-block lists, we will use cron.First, create a script, for example, /usr/local/sbin/update-blocklists.sh:

#!/bin/sh

/usr/local/sbin/unbound-adhosts.sh
/usr/local/sbin/spamhaus.sh

Make it executable:

chmod +x /usr/local/sbin/update-blocklists.sh

Then, add it to the crontab to run daily:

echo "@daily /usr/local/sbin/update-blocklists.sh" >> /etc/crontab

This approach benefits from both update management and security perspectives.

Boosting Network Performance in FreeBSD's VNET Jails

Stefano Marinelli — Mon, 14 Aug 2023 15:50:14 +0000

In the world of FreeBSD, jails are a renowned feature that allows for system-level virtualization. As I was setting up the jails for BSDCafe, I stumbled upon an interesting discovery: the network performance of VNET jails was noticeably lower compared to that of VPS or standard jails. Rather than diving into this immediately, I decided to take a mental note and proceed.

As I delved deeper with various tests, a pattern began to emerge. Anytime there was a NAT (Network Address Translation) acting between the internal bridge of the VNET jails - irrespective of whether it was local or bridged via a VPN - the outgoing performance took a nosedive.

From using tcpdump to carrying out MTU (Maximum Transmission Unit) tests, my endeavors seemed fruitless. However, a memory from the past struck me. I recalled setting up a FreeBSD VM on Proxmox (effectively pointing towards an issue with KVM) where I had to make specific tweaks.

To remedy the situation, I made the following modifications:

Added the following to /boot/loader.conf:

hw.vtnet.X.csum_disable=1
hw.vtnet.lro_disable=1

Integrated these lines into /etc/sysctl.conf:

net.link.bridge.pfil_member=0
net.link.bridge.pfil_bridge=0
net.link.bridge.pfil_onlyip=0

And appended to /etc/rc.local (which I already use for initialization):

ifconfig vtnet0 -rxcsum

The end result was exhilarating: not only did the VNET jails now perform at full bandwidth, but even those interconnected via VPN showcased commendable performance.

Interestingly, this seems to be linked to a long-standing bug from 2012, FreeBSD Bug 165059. This issue is even highlighted in the official PFSense documentation.

In the vast landscape of tech, sometimes revisiting the past provides solutions for the present. All's well that ends well, and I'm pleased to share this resolution with my readers. For those dabbling in FreeBSD, I hope this piece offers some guidance in optimizing your VNET jail setups.

Make your own VPN - OpenBSD, Wireguard, ipv6 and ad-blocking included

Stefano Marinelli — Mon, 03 Apr 2023 04:16:11 +0000

Note: This article assumes a setup based on OpenBSD. If you prefer a version based on FreeBSD, it is available here.

VPS - in this case, I used a basic Hetzner Cloud VPS (using this link, you will receive 20 euros of cloud credits), but any provider that provides IPv6 connectivity will do - if you want IPv6, of course.
OpenBSD - a clean, stable, and secure operating system.
Wireguard - lightweight, secure, and at the same time, not very "chatty", so it is also gentle on mobile device batteries. When there is no traffic, it simply does not transmit/receive anything. Well supported by all major desktop and server operating systems as well as Android and iOS devices.
Unbound - can make DNS queries directly to root servers, not through forwarders. It also allows you to insert block-lists and have a result similar to that of Pi-Hole (i.e., ad-blocking).
SpamHaus lists - to immediately stop connections to and from users on blacklists.

The first step is to activate a VPS and install OpenBSD. On the Hetzner cloud console, there won't be a pre-built OpenBSD image, but only a selection of Linux distributions. Don't worry, just choose any of them and create the VPS. Once done, the OpenBSD ISO image will be available among the "ISO Images". Just insert the virtual CD, restart the VPS, and the OpenBSD installation will appear in the console.

I won't go into detail, the operation is simple and straightforward. The only precaution (in the case of a Hetzner Cloud VPS) is to use "autoconf" for IPv4 but, for now, do not configure IPv6. It will be configured later.

Install all OpenBSD updates (using the syspatch command) and restart, the kernel will be relinked.

Wireguard, on OpenBSD, is fully integrated into the base system and does not require the installation of external packages. This is a big advantage because over time, support for everything related to Wireguard will be managed directly by the main OpenBSD development team.

The /etc/hostname.vio0 file should look something like this:

inet autoconf
inet6 2001:db8:cafe:cafe::1 72 
!route add -net ::/0 fe80::1%vio0

In short, keep the base address assigned by Hetzner, but change the netmask to /72 - thus giving the possibility of having other networks available.

sh /etc/netstart vio0

It will reconfigure the network interface and allow IPv6 to start working. To test it:

ping6 google.com

If everything has been configured correctly, the ping will be executed and google.com will reply.

It is now necessary to enable forwarding for IPv4 and IPv6. Enter these lines in the /etc/sysctl.conf file:

net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1

To apply those changes you can reboot or just type:

sysctl net.inet.ip.forwarding=1
sysctl net.inet6.ip6.forwarding=1

To configure Wireguard, a few steps will be necessary. First of all, the private key will need to be created:

openssl rand -base64 32

Something like this will come out:

YUkS6cNTyPbXmtVf/23ppVW3gX2hZIBzlHtXNFRp80w=

Now create a new file called /etc/hostname.wg0:

172.16.0.1/24 wgport 51820 wgkey YUkS6cNTyPbXmtVf/23ppVW3gX2hZIBzlHtXNFRp80w=
inet6 2001:db8:cafe:cafe:100::1 72

A new Wireguard interface called wg0 is being created. It will have the IPv4 address "172.16.0.1", Wireguard will listen on port 51820, and with the private key created shortly before. It will also have an IPv6 address on one of the subclasses that the provider will have provided.

Save and activate the interface:

sh /etc/netstart wg0

If everything has been entered correctly, it should enable the interface. Now:

ifconfig wg0

And something like this will be returned:

wg0: flags=80c3<UP,BROADCAST,RUNNING,NOARP,MULTICAST> mtu 1420
    index 5 priority 0 llprio 3
    wgport 51820
    wgpubkey xxxxxxxxxxxxxxx=
    groups: wg
    inet 172.16.0.1 netmask 0xffffff00 broadcast 172.16.0.255
    inet6 2001:db8:cafe:cafe:100::1 prefixlen 72

Take note of the wgpubkey - it will be needed to configure the clients.

As for the firewall, OpenBSD comes with a basic pf configuration. In my setups, I tend to block what is not needed and be permissive with what may be useful. However, I like to keep out the "bad guys," so I use blacklists. pf allows elements to be inserted and removed from tables in runtime, so the firewall can be configured accordingly.

To download and apply Spamhaus lists, I use a simple but effective script found on the Internet.

So create the script in /usr/local/sbin/spamhaus.sh:

#!/bin/sh
#
# this is normally run once per day via /etc/daily.local.
#
echo updating Spamhaus DROP lists:
(
  { ftp -o - https://www.spamhaus.org/drop/drop.txt && \
    ftp -o - https://www.spamhaus.org/drop/dropv6.txt ; \
  } 2>/dev/null | sed "s/;/#/" > /var/db/drop.txt
)
pfctl -t spamhaus -T replace -f /var/db/drop.txt

Make it executable and run it:

chmod a+rx /usr/local/sbin/spamhaus.sh
/usr/local/sbin/spamhaus.sh

There are many possibilities to configure pf. A fairly simple example could be this:

ext_if="vio0"
wg0_if = "wg0"
wg0_networks = "172.16.0.0/24"

set skip on lo

# Spamhaus DROP list:
table <spamhaus> persist file "/var/db/drop.txt"

block drop log quick from <spamhaus>

match in all scrub (no-df random-id max-mss 1440)

match out on $ext_if from { $wg0_networks } nat-to ($ext_if)

#Pass ICMP on ipv6
pass quick proto ipv6-icmp
#Block from ipv6 to wg0 network
block in quick on $ext_if inet6 to { 2001:db8:cafe:cafe:100::/72 }
#Pass Wireguard traffic - in and out
pass quick on $wg0_if

# default deny
block in
block out

# By default, do not permit remote connections to X11
block return in on ! lo0 proto tcp to port 6000:6010

# Port build user does not need network
block return out log proto {tcp udp} user _pbuild

pass in on $ext_if proto tcp to port ssh
pass in on $ext_if proto udp to port 51820

pass out on $ext_if

This is a very simple configuration: it blocks everything that is present in the list downloaded from Spamhaus, allows NAT from the Wireguard network to the public interface, allows icmp traffic in IPv6 (necessary for the network to function properly) while blocking incoming traffic to the Wireguard IPv6 LAN (remember that the IPs will be public and directly reachable, so we don't want to expose our devices by default). All traffic on the Wireguard interface will be allowed to pass. Then everything will be blocked and exceptions will be specified, i.e. allowing ssh and Wireguard connections (of course). Authorization will also be granted to allow traffic to exit from the public network interface.

Reload pf configuration:

pfctl -f /etc/pf.conf

If everything went correctly, the firewall should have loaded the new options.

To obtain caching of DNS queries and the related ad-block, it is now time to configure Unbound. A while ago, I found a script which I slightly adapted. I don't remember where I got it, so I'll paste it here without citing the original creator.

Create a script to update the unbound ad-block, in /usr/local/sbin/unbound-adhosts.sh:

#!/bin/ksh
#
# Using blacklist from pi-hole project https://github.com/pi-hole/
# to enable AD blocking in unbound(8)
#
PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin"

# Available blocklists - comment line to disable blocklist
_disconad="https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt"
_discontrack="https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt"
_stevenblack="https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts"

# Global variables
_tmpfile="$(mktemp)" && echo '' > $_tmpfile
_unboundconf="/var/unbound/etc/unbound-adhosts.conf"

# Remove comments from blocklist
function simpleParse {
  ftp -VMo - $1 | \
  sed -e 's/#.*$//' -e '/^[[:space:]]*$/d' >> $2
}

# Parse MalwareDom
#[[ -n ${_malwaredom+x} ]] && simpleParse $_malwaredom $_tmpfile

# Parse ZeusTracker
#[[ -n ${_zeustracker+x} ]] && simpleParse $_zeustracker $_tmpfile

# Parse DisconTrack
[[ -n ${_discontrack+x} ]] && simpleParse $_discontrack $_tmpfile

# Parse DisconAD
[[ -n ${_disconad+x} ]] &&  simpleParse $_disconad $_tmpfile

# Parse StevenBlack
[[ -n ${_stevenblack+x} ]] && \
  ftp -VMo - $_stevenblack | \
  sed -n '/Start/,$p' | \
  sed -e 's/#.*$//' -e '/^[[:space:]]*$/d' | \
  awk '/^0.0.0.0/ { print $2 }' >> $_tmpfile

# Parse hpHosts
[[ -n ${_hostfiles+x} ]] && \
  ftp -VMo - $_hostfiles | \
  sed -n '/START/,$p' | tr -d '^M$' | \
  sed -e 's/#.*$//' -e '/^[[:space:]]*$/d' -e 's/^M$//' | \
  awk '/^127.0.0.1/ { print $2 }' >> $_tmpfile

# Create unbound(8) local zone file
sort -fu $_tmpfile | grep -v "^[[:space:]]*$" | \
awk '{
  print "local-zone: \"" $1 "\" redirect"
  print "local-data: \"" $1 " A 0.0.0.0\""
}' > $_unboundconf && rm -f $_tmpfile

/usr/sbin/rcctl reload unbound 1>/dev/null

exit 0
#EOF

Similarly, make the script executable and run it:

chmod a+rx /usr/local/sbin/unbound-adhosts.sh
/usr/local/sbin/unbound-adhosts.sh

Now, the Unbound configuration file in /var/unbound/etc/unbound.conf can be modified as follows:

# $OpenBSD: unbound.conf,v 1.21 2020/10/28 11:35:58 sthen Exp $

server:
        verbosity: 1
        log-queries: no
        num-threads: 4
        num-queries-per-thread: 1024
        interface: 127.0.0.1
        #interface: 127.0.0.1@5353      # listen on alternative port
        interface: 172.16.0.1
    interface: 2001:db8:cafe:cafe:100::1
        interface: ::1
        outgoing-range: 64
        chroot: ""
        #do-ip6: yes

        # override the default "any" address to send queries; if multiple
        # addresses are available, they are used randomly to counter spoofing
        #outgoing-interface: 192.0.2.1
        #outgoing-interface: 2001:db8::53

        access-control: 0.0.0.0/0 refuse
        access-control: 127.0.0.0/8 allow
        access-control: ::0/0 refuse
        access-control: ::1 allow
        access-control: 172.16.0.0/24 allow
        access-control: 2001:db8:cafe:cafe:100::/72 allow

        hide-identity: yes
        hide-version: yes

        # Perform DNSSEC validation.
        #
        auto-trust-anchor-file: "/var/unbound/db/root.key"
        val-log-level: 2

        # Synthesize NXDOMAINs from DNSSEC NSEC chains.
        # https://tools.ietf.org/html/rfc8198
        #
        aggressive-nsec: yes
        prefetch: yes
        username: "nobody"
        directory: "/var/unbound/etc"
        logfile: "/var/unbound/unbound.log"
        use-syslog: no
        pidfile: "/var/unbound/unbound.pid"
        include: /var/unbound/etc/unbound-adhosts.conf

remote-control:
        control-enable: yes
        control-interface: /var/run/unbound.sock

Before launching unbound, it is necessary to give the appropriate permissions:

chown -R nobody:nobody /var/unbound

Now it is possible to enable and start unbound. Since it needs to load the (long) blocklist, it will take a few seconds:

rcctl enable unbound
rcctl start unbound

If everything has been done correctly, unbound will be able to respond to requests made on 172.16.0.1 and 2001:db8:cafe:cafe:100::1, from their respective LANs.

Now it is possible to configure the Wireguard client. Each implementation has its own procedure (Android, iOS, MikroTik, Linux, etc.) but essentially it is sufficient to create the right configuration both on the server and on the client. For example, the server's public key (visible by typing "ifconfig wg0" on the OpenBSD server) should be inserted into the "peer" that will be created on the client, while the client's public key will be used on the server in this way:

Reopen the file /etc/hostname.wg0 and add:

172.16.0.1/24 wgport 51820 wgkey YUkS6cNTyPbXmtVf/23ppVW3gX2hZIBzlHtXNFRp80w=
inet6 2001:db8:cafe:cafe:100::1 72
wgpeer *client's public key* wgaip 172.16.0.2/32 wgaip 2001:db8:cafe:cafe:100::2/128

Reload the configuration:

sh /etc/netstart wg0

On the client, create a new configuration by inserting "172.16.0.2/32, 2001:db8:cafe:cafe:100::2/128" (the ones that were entered in the peer configuration in the hostname.wg0 file) in the local IP addresses. Set the DNS server address to "172.16.0.1" and/or its corresponding IPv6 address (in the example, 2001:db8:cafe:cafe:100::1 - yours will be different). In the peer, insert the server's data, including its public key, IP address:port (in the example, the port is 51820), and allowed addresses (setting "0.0.0.0/0, ::0/0" means "all connections will be sent via Wireguard" - all the traffic will pass through the VPN for both IPv4 and IPv6).

It is also possible to use the VPN only as an ad-blocker, by only routing DNS traffic through it. To achieve this result, it is sufficient to configure the client so that the only allowed address is the one of the just-configured unbound (in this example, 172.16.0.1 or 2001:db8:cafe:cafe:100::1) - DNS resolution will occur via VPN, but browsing will continue to work through the main provider.

If you want the spamhaus and ad-block lists to be updated automatically, create the /etc/daily.local file and add the following lines:

#!/bin/ksh

/usr/local/sbin/unbound-adhosts.sh
/usr/local/sbin/spamhaus.sh

All of this can be achieved simply with a basic installation of OpenBSD, without the need to install any additional packages. This is an advantage both in terms of update management and security.