Introduction
This article is a "spin-off" from the previous post "Building a Self-Hosted CDN for BSD Cafe Media," which I recommend reading, based on FreeBSD. In that article, I showed how I addressed the issue of geo-replication and geo-distribution of BSD Cafe media content. If you prefer a NetBSD-based setup, there's an article that describes how to do it.
The internet today relies TOO MUCH on just a few big players. When one of them stops working, half the world is impacted because too many services, in my opinion, depend on them. “Too big to fail,” some might say. “Single Point of Failure,” I respond."
The strength of the internet has always been its extreme decentralization, which is now less evident due to this phenomenon.
In this article, I want to show how easy it is to create a self-hosted CDN using OpenBSD and just two external packages: Varnish and Lego.
Actually, only one package is truly needed (Varnish), and SSL certificates could be generated using the built-in acme-client in OpenBSD. However, this might be limiting since acme-client handles certificate generation via traditional methods (using a file in .well-known), but when dealing with a CDN and several reverse proxies listening, you don’t have perfect control over which one will receive the certificate generation validation request.
The choice of Varnish is based on several factors, with the main ones being the ability to keep the cache in RAM (which means it can run on read-only systems) and the ability to flush the cache remotely. For example, with each change to my blog, I can choose whether to perform an immediate flush (such as for a new article or an error) or wait for the cache's "natural" expiration (such as for a typo or minor, non-critical changes).
For convenience and practicality, I’ll use the excellent Lego tool—a Go application that supports many DNS authentication methods, including PowerDNS.
Installation
The steps are quite simple. After installing and updating OpenBSD (using the syspatch command), start by installing the two packages:
obcdn# pkg_add lego varnish
quirks-7.14:updatedb-0p0: ok
quirks-7.14 signed on 2024-08-24T13:35:23Z
quirks-7.14: ok
lego-4.16.1: ok
varnish-7.4.2:bzip2-1.0.8p0: ok
varnish-7.4.2:pcre2-10.37p2: ok
useradd: Warning: home directory `/var/varnish' doesn't exist, and -m was not specified
varnish-7.4.2: ok
The following new rcscripts were installed: /etc/rc.d/varnishd
See rcctl(8) for details.
The next step is to enable Varnish:
rcctl enable varnishd
Varnish has a generic startup script, but it's best to customize the startup options. To do this, modify the /etc/rc.conf.local file:
varnishd_flags="-j unix,user=_varnish,ccgroup=_varnish -f /etc/varnish/default.vcl -T localhost:9999 -a localhost:8080 -s default,500m"
This configuration sets Varnish with a 500 MB cache and listens on localhost, port 8080.
Next, rename the default VCL file to prepare for your own content:
mv /etc/varnish/default.vcl /etc/varnish/default.vcl.distrib
Create a new default.vcl file. Below is an example based on this blog (at the time of writing). You’ll need to adapt it according to your needs, especially if there are cookies or other dynamic content. Note that Varnish will fetch data from a specific backend accessed in http. If privacy is needed, consider creating a VPN between the backend and Varnish, as briefly mentioned in the previous article:
vcl 4.1;
import std;
# Backend - it-notes.dragas.net
backend it_notes {
.host = "myOriginalServer";
.port = "80";
}
# ACL - purge - it-notes.dragas.net
acl purge_it_notes {
"authorizedIPForCachePurge";
}
sub vcl_recv {
# it-notes.dragas.net
if (req.http.Host == "it-notes.dragas.net") {
set req.backend_hint = it_notes;
set req.http.Host = "it-notes.dragas.net";
# PURGE - it-notes.dragas.net
if (req.method == "PURGE") {
std.log("Purge request received for " + req.url);
if (!std.ip(req.http.X-Forwarded-For, "0.0.0.0") ~ purge_it_notes) {
return (synth(405, "Not allowed."));
}
if (req.url == "/" || req.url == "/*") {
ban("req.http.host == " + req.http.host);
return(synth(200, "Entire cache has been cleared."));
}
return (purge);
}
} else {
# Other domains - 404
return (synth(404, "Domain not found"));
}
if (req.method != "GET" && req.method != "HEAD") {
return (pipe);
}
return (hash);
}
sub vcl_backend_response {
# TTL - it-notes.dragas.net
if (bereq.http.host == "it-notes.dragas.net") {
if (bereq.url ~ "\.(gif|jpg|jpeg|png|webp|ico|css|js)$") {
set beresp.ttl = 1w;
set beresp.grace = 1d;
set beresp.keep = 7d;
unset beresp.http.Set-Cookie;
unset beresp.http.Cache-Control;
set beresp.http.Cache-Control = "public, max-age=604800";
} else {
set beresp.ttl = 15m;
set beresp.grace = 48h;
set beresp.keep = 7d;
}
}
# Remove some headers
unset beresp.http.Server;
unset beresp.http.X-Powered-By;
unset beresp.http.Via;
return (deliver);
}
sub vcl_deliver {
# Add X-Cache header
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT";
} else {
set resp.http.X-Cache = "MISS";
}
std.log("Delivering content for " + req.url + " - Cache: " + resp.http.X-Cache);
# Remove Varnish headers
unset resp.http.Via;
unset resp.http.X-Varnish;
return (deliver);
}
sub vcl_hash {
hash_data(req.url);
if (req.http.host) {
hash_data(req.http.host);
} else {
hash_data(server.ip);
}
return (lookup);
}
sub vcl_hit {
return (deliver);
}
sub vcl_miss {
return (fetch);
}
sub vcl_purge {
std.log("Purge executed for " + req.url);
return (synth(200, "Purge successful"));
}
sub vcl_synth {
set resp.http.Content-Type = "text/html; charset=utf-8";
set resp.http.Retry-After = "5";
synthetic({"<!DOCTYPE html>
<html>
<head>
<title>"} + resp.status + " " + resp.reason + {"</title>
</head>
<body>
<h1>Status "} + resp.status + " " + resp.reason + {"</h1>
<p>"} + resp.reason + {"</p>
<h3>Guru Meditation:</h3>
<p>XID: "} + req.xid + {"</p>
<hr>
<p>Varnish cache server</p>
</body>
</html>
"});
return (deliver);
}
Start Varnish and check if it starts correctly:
rcctl start varnishd
Generating SSL Certificates
Before configuring relayd, you’ll need to generate SSL certificates. Lego supports many DNS providers and provides clear and comprehensive examples, so I suggest reading its README file.
Important: Certificates generated by Lego are not directly compatible with relayd, so you must generate them in the correct format. Add the -k rsa4096 flag to the Lego command to obtain certificates compatible with relayd.
Once generated, the certificates will be in a subdirectory of the directory from which the command was launched. For example, if the command is run as root (which is unnecessary, but just for the example), the certificates will be in /root/.lego/certificates/.
relayd expects certificates in a specific location. Copy them to the appropriate directories. In my example:
obcdn# cp /root/.lego/certificates/it-notes.dragas.net.crt /etc/ssl/
obcdn# cp /root/.lego/certificates/it-notes.dragas.net.key /etc/ssl/private/
Remember to copy the files to the correct directories when renewing. You can also create symbolic links, this will make your life easier but it’s less secure.
Configuring relayd
It’s time to configure relayd. The file is /etc/relayd.conf, and here’s an example configuration:
log state changes
prefork 10
table <itnotes> { 127.0.0.1 }
http protocol "http" {
match request header append "X-Forwarded-For" value "$REMOTE_ADDR"
match request header append "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"
match request path "/*.atom" tag "CACHE"
match request path "/*.css" tag "CACHE"
match request path "/*.gif" tag "CACHE"
match request path "/*.html" tag "CACHE"
match request path "/*.ico" tag "CACHE"
match request path "/*.jpg" tag "CACHE"
match request path "/*.webp" tag "CACHE"
match request path "/*.js" tag "CACHE"
match request path "/*.png" tag "CACHE"
match request path "/*.rss" tag "CACHE"
match request path "/*.svg" tag "CACHE"
match request path "/*.xml" tag "CACHE"
match request path "/*.ttf" tag "CACHE"
match request path "/*.woff2" tag "CACHE"
match response tagged "CACHE" header set "Cache-Control" value "public, max-age=604800"
pass request header "Host" value "it-notes.dragas.net" forward to <itnotes>
}
http protocol "https" {
match request header append "X-Forwarded-For" value "$REMOTE_ADDR"
match request header append "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"
match response header set "Referrer-Policy" value "no-referrer"
match response header set "X-Content-Type-Options" value "nosniff"
match response header set "X-Download-Options" value "noopen"
match response header set "X-Frame-Options" value "SAMEORIGIN"
match response header set "X-Permitted-Cross-Domain-Policies" value "none"
match response header set "X-XSS-Protection" value "1; mode=block"
match response header set "Strict-Transport-Security" value "max-age=15552000; includeSubDomains; preload"
match request path "/*.atom" tag "CACHE"
match request path "/*.css" tag "CACHE"
match request path "/*.gif" tag "CACHE"
match request path "/*.html" tag "CACHE"
match request path "/*.ico" tag "CACHE"
match request path "/*.jpg" tag "CACHE"
match request path "/*.webp" tag "CACHE"
match request path "/*.js" tag "CACHE"
match request path "/*.png" tag "CACHE"
match request path "/*.rss" tag "CACHE"
match request path "/*.svg" tag "CACHE"
match request path "/*.xml" tag "CACHE"
match request path "/*.ttf" tag "CACHE"
match request path "/*.woff2" tag "CACHE"
match response tagged "CACHE" header set "Cache-Control" value "public, max-age=604800"
tcp { nodelay, sack, socket buffer 65536, backlog 100 }
tls { keypair "it-notes.dragas.net" }
pass request header "Host" value "it-notes.dragas.net" forward to <itnotes>
}
relay "http" {
listen on vio0 port 80
protocol "http"
forward to <itnotes> port 8080
}
relay "https" {
listen on vio0 port 443 tls
protocol "https"
forward to <itnotes> port 8080
}
relay "https6" {
listen on my:ip:v6:address::1 port 443 tls
protocol "https"
forward to <itnotes> port 8080
}
The content of the file is quite self-explanatory. Note that:
vio0is the interface name—it should be modified based on the interface where relayd needs to listen.- I’ve configured relayd to listen on port 80 as well.
- IPv4 and IPv6 listeners are separated. If IPv6 is not configured, simply comment out that part. Please, if you can, use IPv6. In a fairer world, everyone would have the right to at least one class of public addresses without having to pay exorbitant fees.
- The "keypair" must correspond to the certificate and key names in
/etc/ssland/etc/ssl/private.
Test the configuration, enable, and start relayd:
obcdn# relayd -n
configuration OK
obcdn# rcctl enable relayd
obcdn# rcctl start relayd
relayd(ok)
Final Checks
The stack is ready. A ps command will show the process status:
obcdn# ps aux
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
root 1 0.0 0.0 920 276 ?? I 7:10PM 0:00.01 /sbin/init
[...]
_varnish 44999 0.0 0.2 2256 2424 ?? S 8:27PM 0:00.05 varnishd: Varnish-Mgt -i obcdn.my.domain (varnishd)
_varnish 54481 0.0 8.8 34500 88876 ?? S 8:27PM 0:00.60 varnishd: Varnish-Child -i obcdn.my.domain (varnishd)
root 57753 0.0 0.5 4080 4820 ?? IU 8:32PM 0:00.03 /usr/sbin/relayd
_relayd 95940 0.0 0.4 2152 3892 ?? Spc 8:32PM 0:00.01 relayd: pfe (relayd)
_relayd 81032 0.0 0.4 2156 3716 ?? Spc 8:32PM 0:00.01 relayd: hce (relayd)
_relayd 80312 0.0 0.5 2748 5280 ?? Ipc 8:32PM 0:00.03 relayd: relay (relayd)
_relayd 88919 0.0 0.5 2748 5268 ?? Ipc 8:32PM 0:00.04 relayd: relay (relayd)
_relayd 90186 0.0 0.5 2752 5272 ?? Ipc 8:32PM 0:00.03 relayd: relay (relayd)
_relayd 32851 0.0 0.5 2736 5284 ?? Ipc 8:32PM 0:00.03 relayd: relay (relayd)
_relayd 20270 0.0 0.5 2744 5252 ?? Ipc 8:32PM 0:00.03 relayd: relay (relayd)
_relayd 98826 0.0 0.5 2752 5264 ?? Ipc 8:32PM 0:00.04 relayd: relay (relayd)
_relayd 6454 0.0 0.5 2744 5264 ?? Ipc 8:32PM 0:00.03 relayd: relay (relayd)
_relayd 90102 0.0 0.5 2740 5276 ?? Ipc 8:32PM 0:00.03 relayd: relay (relayd)
_relayd 60314 0.0 0.5 2748 5336 ?? Ipc 8:32PM 0:00.03 relayd: relay (relayd)
_relayd 27246 0.0 0.5 2744 5284 ?? Ipc 8:32PM 0:00.03 relayd: relay (relayd)
_relayd 87082 0.0 0.4 2100 4508 ?? Ipc 8:32PM 0:00.02 relayd: ca (relayd)
_relayd 53308 0.0 0.4 2096 4496 ?? Ipc 8:32PM 0:00.02 relayd: ca (relayd)
_relayd 11697 0.0 0.4 2092 4492 ?? Ipc 8:32PM 0:00.02 relayd: ca (relayd)
_relayd 83636 0.0 0.4 2092 4500 ?? Ipc 8:32PM 0:00.02 relayd: ca (relayd)
_relayd 74563 0.0 0.4 2096 4484 ?? Ipc 8:32PM 0:00.02 relayd: ca (relayd)
_relayd 35910 0.0 0.4 2100 4508 ?? Ipc 8:32PM 0:00.02 relayd: ca (relayd)
_relayd 24911 0.0 0.4 2096 4504 ?? Ipc 8:32PM 0:00.02 relayd: ca (relayd)
_relayd 78649 0.0 0.4 2096 4496 ?? Ipc 8:32PM 0:00.02 relayd: ca (relayd)
_relayd 89586 0.0 0.4 2096 4500 ?? Ipc 8:32PM 0:00.02 relayd: ca (relayd)
_relayd 11989 0.0 0.4 2100 4500 ?? Ipc 8:32PM 0:00.02 relayd: ca (relayd)
[...]
In this case, both relayd and Varnish are running correctly.
Automating Certificate Renewal
As a final step, remember to create a script to renew the certificates, copy them to /etc/ssl and /etc/ssl/private, and restart relayd.
Conclusion
Congratulations, you now have your own free CDN, with your data, fully under your control. Portable, extendable, controllable, and outside of the big providers' grip.
If your goal is geo-replication, you should use one of the available methods. Some DNS providers allow selection based on the caller's location (like Bunny.net), or, as I prefer, install your own DNS and use tools to manage operations and resolutions, as briefly described in the previous article.
With multiple separate reverse proxies, separate DNS servers (on different providers and possibly different countries or continents) capable of checking if the reverse proxies are operational, you can achieve an extremely low likelihood of encountering a Single Point of Failure, as all components, once the cache is filled, will be nearly autonomous even in the event of a (temporary) backend outage - i.e., the original node.