IT Notes - web

Aggressive caching for a Mastodon reverse proxy: what to cache, what to never cache, and why content negotiation will eventually betray you

Stefano Marinelli — Fri, 05 Jun 2026 08:44:00 +0000

I have written before about putting a cache in front of snac, and more recently about the HAProxy layer in front of FediMeteo. The general idea is always the same: the reverse proxy should absorb the repetitive, public work that has no business reaching the application server.

This post is the same idea applied to a much louder neighbour: a Mastodon instance. The instance is mastodon.bsd.cafe, the proxy is nginx on FreeBSD, and the configuration below is what I am currently running in production.

Mastodon is heavier than snac in every direction. It has Puma and Sidekiq behind it, more endpoints, more streaming, more federation patterns, and one specific characteristic that complicates everything: it serves multiple representations on the same URLs. The same path returns HTML to a browser, ActivityPub JSON to a remote instance, and sometimes plain JSON to an API client. If the proxy treats the URL as one thing, sooner or later it will return the wrong thing to the wrong client.

Most of the work below comes from that single observation.

If I had to summarize this whole post in a single sentence, it would be this:

Mastodon is not a website. It is a website, an API, and an ActivityPub server, all sharing the same URLs.

Everything else in this configuration - cache keys, variants, bypass rules, the diagnostic headers - is decoration around that one fact.

A popular toot from a friend gets boosted. Twenty federated instances ask for the same ActivityPub object within the same second. Browsers fetch the HTML version of the same URL. If the proxy sees only "a URL", it will eventually betray you: a remote instance will receive HTML, a browser will receive ActivityPub JSON, and you will spend an afternoon wondering why your timeline looks broken on three different servers. I have spent that afternoon. I do not recommend it.

Assumptions before anything else

Before any directive, this configuration assumes a few things about the instance. If any of these does not match your setup, the directives still make sense, but you must read the caveats at the end before adapting them.

The first assumption is that AUTHORIZED_FETCH (secure mode) is disabled. With secure mode off, all ActivityPub GET responses cached at the proxy layer are public and identical regardless of the requesting actor. With secure mode on, Mastodon can legitimately return different bodies depending on which remote actor is asking, and caching them blindly at the proxy becomes at best wasteful, at worst a cache-poisoning surface.

This is not a hypothetical. CVE-2026-25540, fixed in Mastodon 4.3.19, 4.4.13, and 4.5.6, is exactly this kind of mistake, but inside Mastodon's own Rails.cache: the pinned posts and featured hashtags endpoints had actor-dependent ActivityPub responses but were keyed without the actor. The CVE does not directly apply to nginx caches, but the underlying lesson does. Do not cache what depends on the caller unless the caller is part of the cache key. Keep this rule in mind every time you are tempted to cache a federation endpoint "just in case".

The second assumption is that no signed-URL storage backend sits behind /system/ or /media_proxy/. If those paths ever redirect to short-lived presigned S3 or SeaweedFS URLs, my TTLs below are too long: nginx will happily cache a redirect to a URL that has already expired.

The third assumption is that federation traffic uses HTTP Signatures, not the HTTP Authorization header. Mastodon signs federated GETs with the Signature header. The Authorization-based skip-cache rule further down catches API tokens, not signed federation traffic. If you enable AUTHORIZED_FETCH, you must add an explicit skip rule for $http_signature.

I am being deliberate about these assumptions because the configuration that follows is internally consistent only as long as they hold.

The proxy in front of mastodon.bsd.cafe has three jobs:

TLS termination, microcaching of expensive endpoints (especially federation-heavy collections and default public routes), and long-lived caching of immutable assets and user media.

The point is not to replace Mastodon's internal Rails cache. The point is to absorb spiky federation traffic and repetitive asset fetches that would otherwise hit Puma and Rails for every single request.

The strategy is deliberately layered: very long TTL on fingerprinted assets, medium TTL on user-uploaded media, very short microcache on dynamic pages and federation endpoints that get hammered, and explicit bypass rules for anything private, authenticated, actor-dependent, or otherwise unsafe.

Every cacheable layer is keyed correctly for content negotiation. That is the part that matters most.

The cache zone

A single cache zone is shared across all Mastodon locations:

proxy_cache_path /var/cache/nginx/mastodon
                 levels=1:2
                 keys_zone=mastodon_cache:200m
                 max_size=20g
                 inactive=24h
                 use_temp_path=off;

200m of keys zone holds metadata for roughly 1.6 million entries in RAM. The body can grow up to 20g on disk. The two numbers are independent: keys live in shared memory, bodies live on the filesystem, and the cache key is what links them.

inactive=24h evicts anything not requested for a day, even if there is free space. This is intentional. I do not want a long, cold tail of stale entries to squat in the cache forever. I want the working set to remain hot, and I want the rest to fade.

use_temp_path=off is small but important. By default nginx writes a cached response to a temporary file and then renames it into place. If the temp path and cache path are on different filesystems, that cheap rename becomes a real copy. Setting use_temp_path=off puts temporary files directly under the cache directory and avoids that trap. It is the kind of detail nobody mentions until something is suspiciously slow.

Of all the maps in this configuration, only one really earns its place. This one:

map $http_accept $mastodon_cache_variant {
    default                          "default";
    "~*application/activity\+json"   "activitypub";
    "~*application/ld\+json"         "activitypub";
    "~*application/json"             "json";
    "~*text/html"                    "html";
}

Mastodon serves the same URL with different bodies depending on the Accept header. A status URL like /@user/123456789 returns rendered HTML to a browser and an ActivityPub object to another federated instance. If you cache by URL alone, the first request that comes in wins and the next request receives the wrong content type. Instances start federating HTML, browsers start downloading JSON, and the failure is subtle enough to waste hours.

The map normalizes Accept into four buckets - activitypub, json, html, and default - and the result is folded into the cache key in every location that does content negotiation:

proxy_cache_key "$scheme$host$request_uri|accept=$mastodon_cache_variant";

Coalescing equivalent MIME types is intentional. application/activity+json and application/ld+json both map to activitypub, because splitting them across two cache buckets would fragment the cache for no useful operational gain.

A subtle point I want to be explicit about: I do not include $request_method in the cache key. nginx already converts HEADinto GET for caching purposes by default, which is what I want here. A HEAD request on /@user/123 should hit the same cache entry as a GET request on the same URL. Adding the method would only separate them for no benefit.

During rollout I also expose the selected variant as a response header:

add_header X-Cache-Variant $mastodon_cache_variant always;

The header is there to verify the behaviour in production. It can come off once the configuration has proved itself, but I tend to leave it on. A cache that works should be visible. A cache that is invisible can be correct, but it can also be silently wrong, and I would rather know.

This is the first real gotcha, and I want to spend a moment on it because it caught me out the first time I configured a similar setup.

nginx honors the upstream Vary response header in addition to proxy_cache_key. If Mastodon emits Vary: Accept, or worse, Vary: Accept, Cookie, ..., my carefully normalized variant key gets paired with nginx's native Vary handling. The result is that the cache may still fragment on the full, un-normalized Accept header - which defeats the entire point of the variant map.

There is another, very specific failure mode on older or unpatched nginx builds. nginx stores the Vary value in a fixed-size cache metadata field. Historically that field was 42 bytes, which is famously short and almost charmingly suspicious of being a Douglas Adams reference. Modern nginx raised the limit to 128 bytes, which is enough for the common cases but still surprisingly small. If your upstream emits a long Vary header, anything beyond the limit is treated as Vary: *, which means the response is not cached at all. The only signal you get is a critical line in the error log, and unless you are looking for it, you will not see it.

The operational lesson is the same in both cases: if you rely on your own normalized variant key, do not assume upstream Vary is harmless. Check your nginx version, check your error log, and verify cache behaviour via X-Cache-Statusand X-Cache-Variant.

On the locations where the variant map is the cache dimension I care about, I take responsibility explicitly:

proxy_ignore_headers Vary;

This tells nginx to stop using upstream Vary to protect me. That is fine only if my own cache key and request normalization cover every response dimension that matters. In particular, I make sure the backend is not also varying on Accept-Encoding in a way that would create compressed and uncompressed variants behind my back. The cleanest way to avoid that is not to forward Accept-Encoding to the backend at all, and let frontend nginx handle compression itself:

proxy_set_header Accept-Encoding "";

This is the kind of decision I prefer to be explicit about. Ignoring Vary is not magic. It is a responsibility, and it should be paired with the rules that take its place.

Rather than build one giant boolean to decide what bypasses cache, I prefer to decompose the logic into small orthogonal maps. Each map is 1 when caching must be skipped, and the final decision is an OR of all of them.

map $request_method $skip_cache_method {
    default 1;
    GET     0;
    HEAD    0;
}

map $http_authorization $skip_cache_auth {
    default 1;
    ""      0;
}

map $http_cookie $skip_cache_cookie {
    default 1;
    ""      0;
}

map $uri $skip_cache_uri {
    default                  0;
    ~^/auth                  1;
    ~^/oauth                 1;
    ~^/settings              1;
    ~^/admin                 1;
    ~^/api/v1/custom_emojis$ 0;
    ~^/api/v1/instance$      0;
    ~^/api/v2/instance$      0;
    ~^/api/v1/trends/tags$   0;
    ~^/api/oembed$           0;
    ~^/api/                  1;
}

The reasoning is straightforward. Only GET and HEAD are cacheable; everything else, including POST, DELETE, PUT, and ActivityPub deliveries, must pass through. Any request carrying an Authorization header is an API call with a token, and those are never public. Any request with a cookie is potentially logged-in traffic, and caching logged-in pages would leak personal timelines across users. Auth flows, settings, admin, and most of the API bypass the cache by URI, while a small, carefully chosen set of slow-changing public API endpoints is allowed through.

The important caveat I want to underline: the Authorization map does not catch signed federated GETs. Mastodon federation uses HTTP Signatures, which means the relevant request header is Signature. If AUTHORIZED_FETCH is enabled, you must add a parallel map:

map $http_signature $skip_cache_signature {
    default 1;
    ""      0;
}

and then include it in both proxy_cache_bypass and proxy_no_cache. Do this before enabling secure mode, not after.

The maps are used together in each cacheable location:

proxy_cache_bypass $skip_cache_method $skip_cache_auth $skip_cache_cookie $skip_cache_uri;
proxy_no_cache     $skip_cache_method $skip_cache_auth $skip_cache_cookie $skip_cache_uri;

Both directives are necessary. proxy_cache_bypass means "do not read from cache for this request". proxy_no_cache means "do not write this response to cache". Without proxy_no_cache, a logged-in user's response could still poison the anonymous cache. Without proxy_cache_bypass, a request that should have gone straight to the backend might still receive a cached anonymous response. I keep both, every time.

Most locations share a common proxy baseline. There is nothing clever here, but if any of these lines is missing the rest of the configuration quietly does less than expected.

proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Connection "";
proxy_set_header Accept-Encoding "";

proxy_http_version 1.1 and proxy_set_header Connection "" matter for upstream keepalive. Without them, nginx may use HTTP/1.0 semantics upstream and send Connection: close on every request, which makes the keepalive directive on the upstream block far less useful than it looks.

proxy_set_header Accept-Encoding "" keeps backend responses uncompressed so nginx can cache a single representation and handle client-facing compression itself. It also prevents accidental cache fragmentation through Vary: Accept-Encoding, which would otherwise creep in despite the variant map.

These settings are not exciting, and they should not be. The interesting parts of an infrastructure are not always the parts that should be unusual.

The Mastodon server block in my configuration ends up with seven distinct request profiles. Six of them cache; one explicitly does not, because streaming is not a cacheable workload.

I do not group them under one location / with a giant if block. I prefer to keep each profile in its own location, even if some of them look similar. When something goes wrong in production, I want to be able to point at one location and reason about it without holding the rest of the configuration in my head.

location ~ ^/(assets|packs|emoji)/ {
    proxy_cache mastodon_cache;
    proxy_cache_key "$scheme$host$request_uri";
    proxy_ignore_headers Vary;

    proxy_cache_valid 200 301 302 7d;
    proxy_cache_valid 404 10m;

    proxy_cache_lock on;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    proxy_cache_background_update on;

    proxy_cache_bypass $skip_cache_method $skip_cache_auth $skip_cache_cookie $skip_cache_uri;
    proxy_no_cache     $skip_cache_method $skip_cache_auth $skip_cache_cookie $skip_cache_uri;

    proxy_pass http://$custom_upstream;
}

These paths are content-addressed. Webpack fingerprints filenames with hashes, so a new deploy publishes new URLs while the old URLs remain valid. A 7-day TTL is safe because /packs/js/common-abc123.js will never become different content under the same URL. If it does, it has a new URL.

404s get a short 10-minute TTL so a temporarily missing asset can recover quickly.

proxy_cache_lock on is the thundering-herd guard. When a popular asset is not cached and ten clients ask for it at once, nine wait for the first request to populate the cache instead of all ten hammering the backend. I like this directive a lot. It is the kind of small switch that quietly removes a class of problems.

proxy_cache_use_stale together with proxy_cache_background_update is the stale-while-revalidate pattern. If an entry has expired but Mastodon is slow or briefly down, nginx can serve the stale copy and refresh it asynchronously. For static assets this is almost always the right trade-off. The asset has not actually changed under the same URL, and a few extra hours of stale data hurt nobody.

location ~ ^/system/(accounts/avatars|media_attachments/files|custom_emojis/images)/ {
    proxy_cache mastodon_cache;
    proxy_cache_key "$scheme$host$request_uri";
    proxy_ignore_headers Vary;

    proxy_cache_valid 200 302 6h;
    proxy_cache_valid 404 5m;

    proxy_cache_lock on;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    proxy_cache_background_update on;

    proxy_cache_bypass $skip_cache_method $skip_cache_auth $skip_cache_cookie $skip_cache_uri;
    proxy_no_cache     $skip_cache_method $skip_cache_auth $skip_cache_cookie $skip_cache_uri;

    proxy_pass http://$custom_upstream;
}

Avatars, attachment thumbnails, and custom emoji are also effectively content-addressed, because the file path contains an ID. They can still be replaced or deleted, so the TTL is more conservative than for assets: six hours instead of seven days.

The 302 status is also cached, because Mastodon may redirect to another storage location, and the redirect is usually stable enough to cache for hours.

This is also where the caveat about signed URLs really matters. If you ever put a signed-URL backend behind /system/, this TTL must be shorter than the signed URL lifetime, or nginx will eventually serve a redirect to a URL that no longer works. On mastodon.bsd.cafe I do not use signed URLs, so six hours is fine.

location ~ ^/(users|ap/users)/[^/]+/statuses/[0-9]+/replies {
    proxy_cache mastodon_cache;
    proxy_cache_key "$scheme$host$request_uri|accept=$mastodon_cache_variant";
    proxy_ignore_headers Vary;

    proxy_cache_valid 200 30s;
    proxy_cache_valid 404 10s;

    proxy_cache_lock on;
    proxy_cache_lock_timeout 1s;
    proxy_cache_lock_age 5s;

    proxy_cache_bypass $skip_cache_method $skip_cache_auth $skip_cache_cookie $skip_cache_uri;
    proxy_no_cache     $skip_cache_method $skip_cache_auth $skip_cache_cookie $skip_cache_uri;

    proxy_pass http://$custom_upstream;
}

This is the location I tuned most carefully. When a status starts going viral, dozens of federated instances poll /replies to build their thread view, often within the same second. The same URL must serve an HTML thread view to browsers and an ActivityPub OrderedCollection to remote instances, so the variant key is essential here.

A 30-second microcache absorbs the spike without serving meaningfully stale data. A reply that appears 30 seconds late in a federated thread is usually invisible to humans, while the backend relief is very visible.

The lock settings keep backend load and latency bounded. proxy_cache_lock_timeout 1s bounds how long queued requests wait behind the lock. If the timeout expires, they go to the upstream directly, but their responses are not stored in the cache, which prevents a runaway thundering herd from clogging the cache fill path. proxy_cache_lock_age 5s prevents one slow cache-populating request from monopolizing the fill path forever; if the request holding the lock has not completed after 5 seconds, nginx may let another request reach the upstream to retry.

I have currently left proxy_cache_use_stale off on this location while I am still validating the deployment. This is a deliberate debugging stance, not a permanent choice. Stale-while-revalidate is useful in production, but during rollout it can hide upstream issues while I am trying to understand the system. Once the behaviour is stable, the production version will be:

proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
proxy_cache_background_update on;

location ^~ /media_proxy/ {
    proxy_cache mastodon_cache;
    proxy_cache_key "$scheme$host$request_uri";

    proxy_cache_valid 200 10m;
    proxy_cache_valid 301 302 10m;
    proxy_cache_valid 404 1m;

    proxy_ignore_headers Cache-Control Expires Vary;

    proxy_cache_lock on;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    proxy_cache_background_update on;

    proxy_cache_bypass $skip_cache_method $skip_cache_auth $skip_cache_cookie $skip_cache_uri;
    proxy_no_cache     $skip_cache_method $skip_cache_auth $skip_cache_cookie $skip_cache_uri;

    proxy_pass http://$custom_upstream;
}

Mastodon's /media_proxy/ fetches remote media so clients do not leak their IP address to remote servers. The response is the same regardless of Accept, so the cache key intentionally omits the variant. Splitting media proxy responses across html, json, activitypub, and default buckets would only waste storage.

proxy_ignore_headers Cache-Control Expires Vary is deliberate here. Mastodon may emit conservative cache headers, or none at all, and I want the proxy to enforce a short local 10-minute policy regardless of what the backend says.

Set-Cookie is not in the ignore list. nginx's default refusal to cache responses carrying Set-Cookie still applies, and I want it to. It is a safety net I do not want to disable just to win a few cache hits.

The ^~ prefix is a small useful detail. Once this location matches, nginx stops evaluating regex locations. Media proxy traffic can be heavy, and skipping further regex matching is a tiny but free win.

location ~ ^/(users|ap/users)/[^/]+/(followers|following) {
    proxy_pass http://$custom_upstream;
}

This one is a pure proxy, no cache. I want to be explicit that this is a decision, not an omission.

/users/<name>/followers and /users/<name>/following are pagination-heavy, change frequently as people follow and unfollow, and are queried by federation crawlers in ways that would make the cache key proliferate through pages and cursors. The likely hit ratio is poor, the risk of serving stale social graph data is non-trivial, and the cost of caching them - in storage and in mental overhead - is not worth it.

If a remote instance starts hammering these endpoints, the right answer is rate limiting with limit_req_zone, not retrofitting cache as a rate limiter.

Default location: the microcache and streaming without cache

location / {
    proxy_cache mastodon_cache;
    proxy_cache_key "$scheme$host$request_uri|accept=$mastodon_cache_variant";
    proxy_ignore_headers Vary;

    proxy_cache_valid 200 10s;
    proxy_cache_valid 301 302 1m;
    proxy_cache_valid 404 10s;

    proxy_cache_lock on;
    proxy_cache_lock_timeout 5s;

    proxy_cache_bypass $skip_cache_method $skip_cache_auth $skip_cache_cookie $skip_cache_uri;
    proxy_no_cache     $skip_cache_method $skip_cache_auth $skip_cache_cookie $skip_cache_uri;

    proxy_next_upstream error timeout http_502 http_503 http_504;
    proxy_next_upstream_tries 2;

    proxy_pass http://$custom_upstream;
}

Everything not matched by a more specific location falls here: profiles, individual statuses, the about page, the public timeline, and many ActivityPub object fetches.

The TTL is only 10 seconds for 200 responses. That is enough to deduplicate the wave of requests when a popular toot gets boosted or linked from elsewhere, without making the page feel stale to a human visitor.

It is worth being honest that short TTLs still cost CPU. A 10-second microcache on a sustained-traffic URL means the backend regenerates the entry six times per minute. That is vastly better than serving every request from Rails, but it is not free. If your backend cannot comfortably handle that, raise the TTL, or enable stale-while-revalidate on these dynamic paths.

proxy_next_upstream with proxy_next_upstream_tries 2 is the failover trigger. If the primary returns 502, 503, 504, or times out, nginx retries on the backup. The chain is capped at two attempts so a sick upstream cannot hold the request indefinitely.

At the http level:

map $http_upgrade $connection_upgrade {
    default upgrade;
    ""      close;
}

In the server block:

location /api/v1/streaming {
    proxy_buffering off;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_read_timeout 3600s;
    proxy_send_timeout 3600s;
    proxy_pass http://$custom_upstream;
}

Streaming is a WebSocket and SSE-style endpoint. Buffering must be off, otherwise the proxy may hold messages while waiting for buffers to fill. The Upgrade and Connection headers are driven by $connection_upgrade, which is upgradeonly when the client actually sent an Upgrade header. That way a non-WebSocket request to the same path does not get its Connection header mangled.

The hour-long read and send timeouts allow long-lived streams to stay open through quiet periods.

There is no cache here. Streaming is not a cacheable workload, and trying to make it one is one of those ideas that sounds clever for about thirty seconds.

Upstream and failover

upstream mastodonbsdcafe {
    server 192.168.123.33  max_fails=3 fail_timeout=30s;
    server 192.168.122.133 backup;
    keepalive 64;
}

The primary backend is on another VPS; the backup is in a jail next to the reverse proxy. After three consecutive failures, the primary is marked down for 30 seconds. Traffic flips to the backup, then nginx retries the primary after the window.

keepalive 64 holds up to 64 idle TCP connections to the upstream per worker. On a busy instance, this saves real handshake overhead, but only if the proxied connection can actually stay open. That is why the shared proxy settings include proxy_http_version 1.1 and proxy_set_header Connection "". Without those, upstream keepalive does much less than it looks like it should.

I also use an indirection layer:

map $remote_addr $custom_upstream {
    default mastodonbsdcafe;
}

Today everything defaults to the main upstream group. The map exists so that specific client IPs can be pinned to a specific upstream when I am debugging, or so an admin connection can be routed to the backup while the primary is being tested. It costs nothing to have it sitting there, and it has saved me time more than once.

What I log and why

log_format detailed '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent" '
                    'rt=$request_time '
                    'uct=$upstream_connect_time '
                    'uht=$upstream_header_time '
                    'urt=$upstream_response_time '
                    'us=$upstream_status '
                    'ua=$upstream_addr '
                    'cache=$upstream_cache_status '
                    'variant=$mastodon_cache_variant';

access_log /var/log/nginx/access.mastodon.bsd.cafe.log detailed;

add_header X-Cache-Status  $upstream_cache_status always;
add_header X-Cache-Variant $mastodon_cache_variant always;

This log format is purpose-built for the cache layer. For each request it records total request time, upstream connect time, upstream header time, upstream response time, upstream status, which backend served the request, cache status, and which content-negotiation variant was selected.

The cache status is one of the values nginx exposes through $upstream_cache_status: HIT, MISS, BYPASS, EXPIRED, STALE, UPDATING, or REVALIDATED. The response headers expose the same information to the client, which makes it trivial to verify behaviour with curl -I or browser dev tools.

The always qualifier matters. Without it, nginx only adds these headers to a subset of responses, so a 502 from the backend might arrive without the diagnostic headers you need most. I want them on every response, no exceptions.

There is also a small operational detail I find pleasant: a custom 502 page.

error_page 502 /502.html;
location = /502.html {
    root /usr/local/www/mastodon_errors;
    internal;
}

It is not part of the cache strategy, but it makes backend hiccups less ugly. And I block some abusive user agents with 444, which closes the connection without sending any response at all:

if ($http_user_agent ~* "bytespider") {
    return 444;
}

This is not a general bot strategy. It is just a cheap refusal path for traffic I know I do not want.

How I check it actually works

A configuration that I cannot verify is a configuration I do not trust. Here is the short set of commands I keep in a paste buffer for this proxy.

The first verification is variant separation. Three requests to the same URL with different Accept headers should produce three independent cache entries:

for v in 'text/html' \
         'application/activity+json' \
         'application/ld+json; profile="https://www.w3.org/ns/activitystreams"'; do
  printf '%-75s -> ' "$v"
  curl -s -o /dev/null -D - -H "Accept: $v" \
    https://mastodon.bsd.cafe/@someuser/123456789 \
    | awk '/^[Xx]-[Cc]ache/ { printf "%s ", $0 } END { print "" }'
done

On the first pass, every variant should be a MISS. On the second pass, every variant should be a HIT, with X-Cache-Variantshowing the expected bucket.

The second verification is that cookies and Authorization always trigger BYPASS:

curl -I -H 'Cookie: _mastodon_session=test' \
  https://mastodon.bsd.cafe/@someuser

curl -I -H 'Authorization: Bearer fake' \
  https://mastodon.bsd.cafe/api/v1/timelines/home

Both should return X-Cache-Status: BYPASS. If they do not, the skip-cache rules are wrong, and the entire setup is unsafe.

If you intend to enable AUTHORIZED_FETCH, the third verification is for signed GETs. A quick synthetic check that the nginx map fires correctly:

curl -I -H 'Signature: fake' \
       -H 'Accept: application/activity+json' \
       https://mastodon.bsd.cafe/users/someuser

If you added $skip_cache_signature, the result should be X-Cache-Status: BYPASS.

Finally, the logs themselves tell me how the cache is performing in production. Cache status distribution:

awk '{
  for (i = 1; i <= NF; i++)
    if ($i ~ /^cache=/) c[$i]++
}
END {
  for (k in c) print k, c[k]
}' /var/log/nginx/access.mastodon.bsd.cafe.log

A healthy instance shows cache=HIT and cache=BYPASS doing most of the work, with cache=MISS accounting for cold paths and short-TTL refreshes. The same trick works for the variant distribution:

awk '{
  for (i = 1; i <= NF; i++)
    if ($i ~ /^variant=/) v[$i]++
}
END {
  for (k in v) print k, v[k]
}' /var/log/nginx/access.mastodon.bsd.cafe.log

This tells me what my traffic actually looks like. A federation-heavy instance shows a lot of activitypub. An instance with many human visitors shows more html. On mastodon.bsd.cafe the balance shifts depending on what is happening in the wider Fediverse on any given day.

Caveats worth being honest about

I do not like presenting configurations as magic, so I want to be explicit about the conditions under which this one is appropriate.

Short TTLs cost CPU. A 10-second microcache on a sustained-traffic URL means six backend regenerations per minute. That is much better than no cache, but it is not free. If the backend cannot comfortably handle that, raise the TTL or enable stale-while-revalidate on the dynamic paths.

Dynamic stale-while-revalidate is powerful but it hides problems. I currently keep proxy_cache_use_stale off on the dynamic locations because I am still validating behaviour. In steady-state production, stale-while-revalidate is usually the right choice. During rollout, it can quietly hide upstream errors and make debugging harder. Be honest with yourself about which mode you are in.

AUTHORIZED_FETCH changes the threat model. With secure mode disabled, public ActivityPub GET responses are safe to cache as public content, provided your cache key handles content negotiation correctly. With secure mode enabled, ActivityPub responses can become actor-dependent. At that point you must either bypass cache for signed GETs or include the signing actor in the key. The latter usually destroys the hit ratio, so bypassing is the practical answer.

The variant map is a compromise. It covers application/activity+json, application/ld+json, application/json, and text/html. Everything else falls into the default bucket. That is intentional, but the default bucket is still a bucket. If you discover a real client type that matters on your instance, add it explicitly.

Ignoring Vary is a responsibility. proxy_ignore_headers Vary is not magic; it tells nginx to stop protecting you based on upstream Vary. That is fine only if your own cache key and request normalization cover every dimension Vary was protecting. For this configuration that means normalizing Accept into a variant, avoiding backend Accept-Encodingvariation, never caching cookies or authorization, and never caching signed GETs if secure mode is enabled.

Followers and following are uncached on purpose. They are pagination-heavy and change frequently. Caching them would create many low-value entries with questionable freshness. If a remote instance hammers these endpoints, use limit_req_zone. Do not retrofit cache as a rate limiter.

Signed-URL redirects require shorter TTLs. Caching 302s is useful when redirects are stable. It is dangerous when redirects point to short-lived signed URLs. If your media storage returns presigned URLs, your nginx redirect TTL must be shorter than the URL lifetime.

Set-Cookie must remain special. Do not add Set-Cookie to proxy_ignore_headers unless you are absolutely sure the location cannot produce user-specific responses. nginx's default refusal to cache Set-Cookie responses is a safety net. Keep it.

A good configuration is a written form of the assumptions behind a service. When the assumptions change, the configuration must change too.

There is no single brilliant directive in this configuration. The trick is combining long TTLs for immutable assets, medium TTLs for media, tiny TTLs for dynamic public pages, cache locking for thundering-herd protection, strict bypass rules for private or actor-dependent traffic, a normalized content-negotiation key, and enough logging to prove the system is doing what I think it is.

What this layer buys me, in one sentence: fewer requests reach Puma and Rails.

That is the metric I care about. Mastodon is not slow, but it is heavy, and the bigger the instance grows the more it benefits from a layer in front that quietly absorbs the work that does not need to be done by the application. A reverse proxy that caches Mastodon safely has to remember, with every request, that the same URL might mean three different things to three different clients. Once it does, even a very short microcache can remove a surprising amount of load without changing the user-visible behaviour of the instance.

FediMeteo, timezones, and the art of not breaking what already works

Stefano Marinelli — Mon, 25 May 2026 09:14:00 +0000

I have already written about how FediMeteo was born, and about how HAProxy helps reduce the number of requests that reach snac.

Seen from the outside, FediMeteo almost seems still. There is a static homepage, regenerated every hour. There are the city pages, with their forecasts. There are RSS feeds waiting to be fetched, JSON objects waiting to be requested, Fediverse instances refreshing data, subscribing, unsubscribing, retrieving profiles, and reading notes.

That is the visible part.

Behind it, however, FediMeteo is much more than a homepage, a few ActivityPub accounts, and a well-behaved reverse proxy. It is a chain of small pieces, in proper Unix style, each trying to do one thing and do it as well as possible.

That chain, although almost invisible from the outside, was not born already tidy. It changed, was rewritten, adapted to new countries, timezones, ambiguous city names, external service limits, and also to my own mistakes.

Some mistakes were small. Others were much less so.

Because FediMeteo is a human project and, as such, imperfect. Imperfect in the way humans are imperfect, which today almost seems unfashionable. I like that.

The first version of the bot was almost embarrassingly simple, and I was proud of that.

It took a city name as input, asked Nominatim for the coordinates through geopy, called the Open-Meteo API for the current weather and the next several days, and printed a markdown block with current conditions, the forecast for today, the next twelve hours, and the coming days. The text was in Italian. The cities were Italian. The timezone was Europe/Rome. There was nothing to calculate.

Around the script, a small sh wrapper read a list of cities and, for each one, ran the Python program and piped its output into snac note_unlisted. A cron job ran the wrapper every six hours. The output was loose markdown, which snac happily renders, and the integration was: standard output goes into standard input. Nothing fancier than that.

I like this kind of design. It is the part of the Unix philosophy that survives even when fashions change.

When I started adding other European countries, I did not need to change much. I separated the operational logic from the localized strings, moved the strings into one JSON file per country, and spread the cron entries so that not every country posted in the same minute. Each country had its own snac instance, in its own FreeBSD jail, with its own dataset. The bot, internally, was almost the same script as before.

This worked because Europe is, in essence, two or three timezones across most of the countries I cared about.

Then I added Germany, and Germany taught me my first lesson about names.

There are several places called Neustadt in Germany. There is a Frankfurt am Main, and a Frankfurt an der Oder, and they are not the same city. There is a Halle in Saxony-Anhalt and a Halle in North Rhine-Westphalia. Asking Nominatim for "Frankfurt, Germany" produced one of the two, consistently, but not always the one I wanted. Some German users wrote to me, politely, to point out that the forecast for "their" Frankfurt was, in fact, for the other one.

I started thinking about disambiguation, but only enough to fix the immediate cases. The bot still took a single city name. The ambiguous ones I worked around by editing the cities file and hoping for the best.

In hindsight, this was the seed of what would happen later.

The United States broke every assumption the bot had grown up with.

The first problem was the number of cities. I wanted reasonable coverage at state level, which meant identifying the main cities for each of the fifty states. The list ended up at more than 1200 entries. That alone is more cities than every other country in the project combined.

The second problem was timezones. The contiguous United States covers four of them, and Alaska and Hawaii bring the total to six. A "current weather at 12:00" line generated at the same instant for New York and for Los Angeles is technically the same instant, but the two cities are living different parts of the day, and the forecast for "today" is not even quite the same window. A bot that pretended every city was on the same clock would be wrong, sometimes embarrassingly so, every single day.

The third problem was the name thing again, only larger. There are dozens of Springfields. There is a Portland in Oregon and a Portland in Maine. The Germany workaround - editing the cities file by hand and hoping Nominatim picked the right city - was clearly not going to scale to a country where the same name is also a state.

I sat with this for a couple of days before admitting what I already knew.

The bot needed to be rewritten.

What made this hard was not the rewriting itself. It was the requirement to do it without breaking everything else.

By the time I decided to add the United States, the infrastructure around the bot had grown into something I trusted. Jails, snapshots, backup jobs, cron schedules, snac instances on production paths, the HAProxy layer, the homepage cron that aggregated follower counts, and a long list of cities being processed in series every six hours. None of that knew or cared about the bot's internal shape. All of it cared, very much, about the bot's external behavior: a city name and a country code go in, valid markdown comes out, and that markdown ends up in a timeline.

So the contract was clear, even if I had never written it down anywhere. The command-line interface, the output format, the exit codes, the way the wrapper script invoked it, the structure of the JSON country configs - all of it had to keep working. Italian had to keep working. German had to keep working. The cron job that ran every six hours had to keep producing the same shape of output, just with new countries added.

What I changed was almost everything below the surface.

The city argument grew an optional __state suffix, with a double underscore as separator:

python3 main.py springfield__illinois us
python3 main.py springfield__massachusetts us
python3 main.py new_york__new_york us

A city without the suffix continued to work exactly as before, which is what every European country needed. The country config gained a timezone field that could be a fixed string or the literal "auto"; when it was "auto", the bot used timezonefinder against the resolved coordinates to determine the right zone for that specific city. Internally I separated the weather provider behind an interface, so Open-Meteo could remain the primary while MET Norway and wttr.in sat behind as alternatives, with automatic fallback when the primary failed. Units became configurable per country: temperature, wind speed, precipitation. The United States needed Fahrenheit, miles per hour, and inches. Most of Europe wanted Celsius, kilometers per hour, and millimeters. The bot now does either, on a per-country basis, without caring which is which.

I am skipping a lot of small detail here, but the principle was always the same: every new degree of freedom had to be expressible as an optional field in the config or as an optional CLI flag. If a country did not set the new field, the old behavior continued, identical to before.

I tested this by running the new bot against the old country configs and comparing the output line by line. Where it differed, it was a bug in the new bot. Not in the test.

The first cycle after deploying the rewrite was, for every country except the United States, indistinguishable from the cycle before. That was the point.

This is the part of the story I dislike telling, which is precisely why I should tell it.

At some point during the development, while debugging an Open-Meteo response that did not look right, I added a print statement to the error path that dumped the full request URL whenever something went wrong. The full URL of the Open-Meteo customer endpoint includes the apikey query parameter. The print was meant for development. I forgot to remove it.

I deployed.

The next time Open-Meteo had an outage - and small ones happen, sometimes for several minutes at a time - the bot dutifully printed the failing request URL into the post body. For every city. For every cycle that ran during the outage. The wrapper script piped the output into snac note_unlisted without complaint. The posts went out, federated across the Fediverse, with my API key sitting in the text for anyone who cared to read.

Some users were kind enough to write me and tell me. Others were less kind, and made fun of me. Both groups were correct. This should not have happened.

I reported the incident to the Open-Meteo team, who were extremely understanding. They rotated the key immediately and gave me a fresh one. I removed the debug print, and then I did the slightly more useful thing, which was to add redaction at multiple layers - in the bot's output, in the daemon's logging, and in the debug helpers themselves. URL query parameters that look like API keys are masked. Environment variables and config keys named apikey or OPEN_METEO_APIKEY are redacted before any string reaches stdout or a log file. Even JSON-like fields that include open_meteo_apikey are scrubbed if they ever appear in something the program prints.

The lesson is not "be more careful." The lesson is that debug paths leak, sooner or later, so the secrets have to be unreachable from the debug paths in the first place. Now they are.

That afternoon, when I realised what was happening, I closed everything for a minute and looked out of the window. Then I started fixing.

Nominatim is a public service, and it is generous, but it is not infinite. Every city in the project needs coordinates, and at the start of the project every cycle would re-ask Nominatim for every city. Most of the time this worked. Sometimes it did not.

There was one cycle, before I added caching, when Nominatim simply did not respond for one of my queries. The geopy call timed out. The bot raised an exception. The wrapper script gave up on that city and moved on to the next one. A few users noticed that a particular city had not received its forecast that day, and asked what had happened.

I added a coordinate cache, and I am still grateful that I did.

The cache is intentionally boring. The first time the bot resolves a city, it writes the latitude and longitude into a small file under /tmp, named after the city, and the state when present. Every subsequent run reads the file. If the file exists, no Nominatim call is made. If the file is missing, the bot calls Nominatim and writes the file. After the first successful lookup, the cache becomes the source of truth for the coordinates of that city.

This is lighter on Nominatim, faster for every cycle, and much more resilient against transient failures. It is also nice for a reason I did not anticipate.

Nominatim is a geocoder, and like every geocoder it has opinions.

I live in Ferrara, so when I added Italy I made sure Ferrara was in the list, and I checked the first cycle to make sure everything looked right. The forecast came out fine. The temperature was reasonable. The icon matched the sky outside my window. I closed the laptop and forgot about it.

Then, one evening months later, I looked more carefully at the coordinates Nominatim had returned for "Ferrara, Italy", and I realised they did not point to the city. They pointed to a location closer to the centroid of the province, which is a much larger area and mostly countryside. The forecast had been, on average, for a field somewhere outside town, not for the city center.

I am not entirely sure why I had not noticed earlier. Probably because the weather in Ferrara and the weather in the fields outside Ferrara is, on most days, indistinguishable to anyone who is not paying attention. But this is the kind of detail I do not want to leave wrong, especially for my own city.

There are other places where geocoding lands slightly off. Sometimes it is a few kilometers, sometimes a different neighborhood, sometimes genuinely the wrong place.

Because the cache is just a file per city, the fix is also just a file per city. I open the cache file, replace the latitude and longitude with the correct values, save. The next cycle uses the corrected coordinates. No code change, no redeploy, no special tooling. I keep a small list of patched cities in a separate text file, so that if I ever rebuild the cache, I do not lose the manual corrections.

This is the kind of operational simplicity I like. A cache made of plain files costs almost nothing and quietly pays back every time a small problem appears.

For every report it generates, the bot also writes a simplified English text snapshot to /tmp/<city>.txt, or /tmp/<city>__<state>.txt when there is a state.

This is intentional, and it is not a debug artifact. I am not ready to say what I am doing with it yet, but it is part of a future direction for the project. Text is a useful intermediate format, and having a clean, language-neutral representation of every forecast sitting on disk costs almost nothing and might be worth a great deal later.

I prefer to let ideas mature in private before I commit to them in public. So I will leave it at this for the moment.

A full cycle for the United States takes hours.

It is not because the work is heavy. It is because I deliberately inserted a small sleep between cities, to give snac time to dispatch the previous post before the next one is generated. With more than 1200 cities in series, even a short pause adds up. I am not in a hurry. Forecasts that arrive a few minutes apart from each other are not a problem, and the bot was already a polite citizen elsewhere. A polite cycle is fine.

The problem with a slow cycle is not the duration. The problem is what happens to it.

In the original design, the cycle was launched by cron. Every six hours, cron called the wrapper script, the wrapper iterated through the cities file, and for each city it ran the bot and piped the output into snac. There was no scheduler in the project at all. Cron was the scheduler. The wrapper was just a loop.

Restarting snac was harmless. The wrapper would call snac note_unlisted per city, and if snac happened to be unavailable for a moment, that single call might fail, but the loop kept moving and snac was usually back within seconds. Snac itself was not what held the cycle together.

What held the cycle together was the wrapper process. And the wrapper process lived inside the jail.

If the FreeBSD jail was restarted while the wrapper was running, the loop stopped wherever it happened to be. The cron schedule did not care. Six hours later, the next cron tick started a new cycle from the first city, and the cities that had been about to be processed at the moment of the restart were simply skipped for that window. For the United States, this could mean several hundred cities going without an update.

There was a worse case, and it took me longer than it should have to recognise it. If the host was rebooting exactly in the minute when cron should have fired, cron simply did not fire. There was no daemon waiting to pick up the missed tick. The cycle never even started. Six hours of forecasts would be lost, in silence, with nothing in any log to suggest anything had gone wrong.

I lived with this for a long time. Reboots were rare, the impact was limited, and adding state was the kind of thing I always meant to do "next week."

What finally changed it was not a dramatic incident. It was the slow accumulation of small ones. A scheduled VPS reboot. A jail restart after an upgrade. Each one on its own was nothing. Together, they were a steady drip of missed cycles.

So I wrote a daemon.

The crontab entries for the bot went away. There is now a long-running process inside the jail, started at boot, and it does the scheduling itself. The schedule is a list of hours and a minute, read from a JSON config. The daemon wakes up once a minute, checks whether it is time to start a cycle, and either starts one or waits.

The interesting part is the state file.

As the daemon walks through the cities file, it writes its position to a small JSON file: which cities file it is processing, and the index of the next city to handle. The write happens at the boundary between one city and the next, because that is the only place where resuming makes sense. If the daemon is interrupted mid-city, that city is retried on resume; no half-finished post escapes.

When the daemon starts, it reads the state file. If it finds one matching the current cities file, it resumes from the saved index. If the cities file has changed since the state was written, the daemon starts fresh. The check is deliberately conservative: a renamed or modified cities file is treated as a different cycle, because the indices would otherwise be meaningless.

The result is the behavior I should have had from the start. If the host reboots while the United States cycle is running, the daemon comes back up with the jail, reads the state, and continues from where it left off. Every city still gets its update, just with a small gap corresponding to the reboot itself. The cycle finishes. The state file is reset. Life goes on.

And the worst case from the cron days is gone. The daemon does not need anyone to fire it. As long as the jail is running, the daemon is running, and the next scheduled cycle will happen when its hour comes, regardless of what was happening at any specific minute.

Of all the changes I have made to the project, this is the one I like most. It is not exciting work. It is the kind of thing that earns no applause because, when it works, it produces no visible event. But it removes a whole class of small daily annoyances, and it makes a slow process robust against the boring kind of failure: the kind nobody plans for, but that always eventually happens.

The current bot does considerably more than the original Italian script. It handles per-city timezones, three weather providers with automatic fallback, unit conversion for temperature, wind, and precipitation, optional air quality, pressure trend indicators when the provider supplies pressure data, a simplified English text snapshot for future use, a coordinate cache that can be patched by hand, secret redaction at multiple layers, a heartbeat that adapts to whichever HTTP client is installed on the host, and a scheduler-and-resume daemon that survives reboots.

But from the outside, almost nothing has changed.

The European country configs work the same way they always did. The wrapper scripts are unchanged. The snac integration is the same one-line pipe. The HAProxy layer in front does not know or care that the bot was rewritten. The homepage cron that counts followers and regenerates the static page works exactly as before.

The original Italian script does not exist as a file anymore, but it survives as a default. A country config with timezone set to Europe/Rome and no special options behaves, today, exactly as the first version of the bot would have. Everything else is opt-in.

I like this kind of work.

FediMeteo, HAProxy, and the art of not wasting snac threads

Stefano Marinelli — Mon, 18 May 2026 09:44:00 +0000

When I wrote about FediMeteo for the first time, I told the story from the beginning: the idea born almost by chance while checking the weather for a holiday, the memory of my grandfather, who for years had been my personal meteorologist, the decision to build something small and useful, and then the surprise of seeing people actually use it. What began as a personal experiment quickly became a small global service, still running with the same philosophy: FreeBSD, jails, simple scripts, snac, text, emoji, and a lot of small pieces doing their work quietly.

That article was mostly about the birth and growth of the project. This one is about one of the less romantic parts of the same story, although I have to admit that I find a certain beauty in it too: keeping the service light as it grows.

FediMeteo is still intentionally simple from the outside. A homepage, some numbers, a list of countries, and many ActivityPub accounts publishing weather forecasts. The posts are text and emoji. There is no JavaScript requirement to read the pages, no heavy frontend, no unnecessary media attached to every forecast, and no dynamic homepage recalculated at every visit just to show the same numbers. This is not accidental. It is the way I wanted the service to behave from the beginning.

But the more the service is used, the more the small details matter. A request that looks harmless when there are ten followers may become a repeated request when there are thousands of followers, remote instances, crawlers, previews, and other servers fetching the same public objects. In the Fediverse, the same small thing can be asked many times by many different places, each one with a perfectly legitimate reason. The backend doesn't care: it just needs to deal with the requests.

And in FediMeteo, the backend is snac.

I like snac very much precisely because it is small, clear, and efficient. It is not a giant application that tries to be everything. It does a focused job and does it well. But this also means that I want to respect its shape. I do not want to waste its threads on work that the reverse proxy can safely do. A snac thread serving the same public avatar again and again is not a tragedy, but it is still a waste. A snac thread answering the same public ActivityPub object several times in the same minute is doing real work, but often not necessary work.

This is the reason behind the HAProxy tuning I am currently using in front of FediMeteo.

It is not about making the configuration look clever. It is about keeping snac quiet.

A continuation of the same idea

I had already explored the same problem with snac and nginx in two previous posts: Improving snac Performance with Nginx Proxy Cache and Caching snac Proxied Media with Nginx. In both cases, the idea was that the reverse proxy should absorb repeated public requests instead of letting them consume snac resources.

This is especially important because snac uses a limited number of threads. I like that. Limits are healthy. They force us to understand what the service is doing, and they prevent a small program from pretending to be an infinite resource. But limits also make waste visible. If a few threads are busy serving files that could have been served from cache, those threads are not available for something more useful.

With FediMeteo the implementation is different because the reverse proxy is HAProxy, but the reasoning is the same. I have many small snac instances, each one in its own FreeBSD (Bastille) jail, and one public entry point that has to route, terminate TLS, compress, cache, and generally remove as much repetitive work as possible from the backends.

This is, in a way, the natural continuation of the original FediMeteo design. In the first article I wrote that I wanted to manage everything according to the Unix philosophy: small pieces working together. This is another piece of that same puzzle. HAProxy does the edge work. snac does the ActivityPub work. Scripts generate forecasts. cron launches updates. ZFS gives me snapshots. FreeBSD jails keep countries separated. Nothing is particularly heroic by itself, but the whole system becomes pleasant because each part has a clear responsibility.

Why there is almost no media

Before talking about HAProxy, it is worth mentioning one of the most important optimizations, which is not in the proxy configuration at all.

FediMeteo does not use media in its forecasts.

No images attached to the posts, no generated weather cards, no maps for each city, no decorative banners. The forecasts are text and emoji. This was a deliberate decision. Weather information does not become more useful just because it is put inside an image, and every media file used by the service would become something to store, serve, cache, federate, expire, back up, and occasionally debug.

Text and emoji are enough. They are accessible, light, readable in text browsers, friendly to timelines, and understandable even when someone does not know the local language perfectly. This was one of the original design principles of FediMeteo, and it also helps the infrastructure. Less media means less work, fewer cache entries, fewer repeated fetches, fewer surprises.

There is one exception: the avatar.

All FediMeteo accounts use the same avatar, and this is also intentional. I could have used a different avatar for each country, or for each city, or created something visually richer. It would have been nicer in some screenshots, perhaps. It would also have been operationally worse.

With one shared avatar, the reverse proxy has one very useful object to cache. It is public, identical for everyone, small, requested often, and therefore almost always hot in cache. HAProxy can serve it directly instead of asking each snac instance to return the same file. Since avatars are requested by remote instances, browsers, profile previews, and all sorts of federation-related fetches, this single decision removes a surprising amount of pointless backend traffic.

So the avatar is not only a visual identity. It is part of the architecture.

This is the kind of optimization I like most, because it starts before the software. It starts with deciding not to create a problem.

The homepage is static because it can be static

The main homepage follows the same logic.

It is a static HTML page generated from a template. Once per hour, a cron script updates the numbers and statistics. It counts the data I want to show, regenerates the page, and then the page remains static until the next run.

This is not because I cannot make a dynamic page. It is because I do not need one. Boring is good.

The homepage does not need to query all the country instances on every visit. It does not need a database request for each user who opens it. It does not need to ask snac anything in real time. The numbers are useful, but they do not need to be updated every second. Once per hour is enough, and it also fits the spirit of the whole project: do the work when it is needed, then serve the result cheaply.

I have seen too many small services become heavy because the first implementation was convenient rather than appropriate. A cron job and a template are not fashionable, but they are often exactly what a page like this needs.

Many countries, one entry point

FediMeteo is made of many country instances. Each one runs in its own jail and listens on its own internal address and port. From the outside, however, they all live under the same domain structure:

fedimeteo.com
www.fedimeteo.com
it.fedimeteo.com
uk.fedimeteo.com
jp.fedimeteo.com
us.fedimeteo.com
usa.fedimeteo.com
can.fedimeteo.com
canada.fedimeteo.com

And many more.

At the beginning, it is always tempting to write one ACL after another in the HAProxy frontend. It is quick, it is explicit, and for five hostnames it is perfectly fine. But FediMeteo did not remain at five hostnames. As countries and aliases grew, a long chain of ACLs would have turned the frontend into a list of names instead of a description of how the proxy behaves.

So I moved the hostname to backend mapping into a map file:

fedimeteo.com        backend_fedimeteo
www.fedimeteo.com    backend_fedimeteo
it.fedimeteo.com     backend_it
uk.fedimeteo.com     backend_uk
jp.fedimeteo.com     backend_jp
us.fedimeteo.com     backend_us
usa.fedimeteo.com    backend_us
can.fedimeteo.com    backend_ca
canada.fedimeteo.com backend_ca

The frontend then needs only one rule:

use_backend %[req.hdr(host),field(1,:),lower,map(/usr/local/etc/fedimeteo.map,backend_fedimeteo)]

This reads the Host header, removes the port if present, lowercases the result, and looks it up in /usr/local/etc/fedimeteo.map. If nothing matches, it falls back to the main FediMeteo backend.

I like this because it keeps the configuration honest. The frontend contains the policy. The map contains the data. Adding a country means adding an entry to the map and defining a backend. I do not need to make the frontend more complicated every time the service grows.

Backends as small compartments

The country backends are deliberately plain:

backend backend_it
    mode http
    http-reuse safe
    server srv1 10.0.0.2:8001 maxconn 30

backend backend_uk
    mode http
    http-reuse safe
    server srv1 10.0.0.7:8001 maxconn 30

backend backend_jp
    mode http
    http-reuse safe
    server srv1 10.0.0.32:8001 maxconn 30

One backend, one jail, one snac instance. This is exactly the same organizational principle as the rest of the project. If I need to reason about Italy, I look at the Italian jail. If I need to reason about the United Kingdom, I look at the UK jail. If one day I need to move a country elsewhere, the separation is already there.

The maxconn 30 value is not a magic number. It is a ceiling. I want each small backend to have a visible limit in front of it. If something starts hammering a country instance, I prefer the pressure to appear at the HAProxy layer instead of becoming unlimited concurrent work inside snac.

http-reuse safe lets HAProxy reuse backend connections where appropriate. This is another small reduction in unnecessary work. Opening connections repeatedly is not the biggest problem in the world, but avoiding it is still better, especially when many small services sit behind the same proxy.

The front door

The HTTPS frontend listens on IPv4 and IPv6 and offers both HTTP/2 and HTTP/1.1:

frontend https_in
    bind :::443 v4v6 ssl crt /usr/local/etc/certs/ alpn h2,http/1.1
    mode http
    option http-keep-alive

TLS defaults are set globally:

ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

Port 80 only redirects to HTTPS, except for Let's Encrypt challenges:

acl letsencrypt-acl path_beg /.well-known/acme-challenge/
http-request redirect scheme https code 301 unless letsencrypt-acl
use_backend letsencrypt-backend if letsencrypt-acl

In the HTTPS frontend I also set the usual forwarding headers:

http-request set-header X-Real-IP %[src]
http-request set-header X-Forwarded-Proto https

And I add HSTS:

http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"

None of this is unusual, and that is fine. The interesting parts of an infrastructure are not always the parts that should be unusual.

Two caches, because the requests are different

The HAProxy configuration defines two caches:

cache mediacache
  total-max-size 128
  max-object-size 10000000
  max-age 3600
  process-vary on
  max-secondary-entries 12

cache jsoncache
  total-max-size 16
  max-object-size 1000000
  max-age 60
  process-vary on
  max-secondary-entries 12

I keep media and ActivityPub JSON separate because they are not the same kind of traffic.

The media cache is larger and has a longer maximum age. In FediMeteo, this mostly means the shared avatar and a few static-looking objects. Since there is intentionally almost no media, the important cached object is requested very often and remains warm.

The JSON cache is smaller and short-lived. It is there for public ActivityPub GET requests, not to store federation state forever. A 60 second cache is enough to collapse many repeated requests that arrive close together in time, without pretending that ActivityPub responses should be treated like immutable files.

This distinction is important. Caching is not one decision. It is a set of small decisions about what a response means, who can see it, how often it changes, and what happens if it is served again.

Recognizing media

For media, the ACL is based on file extensions:

acl is_media path_end -i .jpg .jpeg .png .gif .webp .svg .ico .mp4 .webm .mp3 .ogg .wav .flac .mov .avi .mkv .m4v

Then I store the result in a transaction variable:

http-request set-var(txn.is_media) bool(true) if is_media

The cache lookup is straightforward:

http-request cache-use mediacache if { var(txn.is_media) -m bool true }

And on the response side:

http-response set-header Cache-Control "max-age=3600, public" if { var(txn.is_media) -m bool true }
http-response del-header Set-Cookie if { var(txn.is_media) -m bool true }
http-response del-header Vary if { var(txn.is_media) -m bool true }
http-response cache-store mediacache if { var(txn.is_media) -m bool true }

The Cache-Control header makes the intent explicit. Set-Cookie is removed because a public media object should not carry session information. Vary is removed because I do not want the same avatar to fragment into many cache entries because of harmless header differences.

This is aggressive only if removed from its context. In this service, with this media policy, it is a reasonable choice. FediMeteo is not serving private media under these paths. It is mostly serving the same public avatar over and over.

For the same reason, I clean the request before it reaches the backend:

http-request del-header Authorization if { var(txn.is_media) -m bool true }
http-request del-header Cookie        if { var(txn.is_media) -m bool true }

I would not do this globally. I do it after deciding that the request is media. Scope is what makes these rules safe.

The result is exactly what I want: the shared avatar becomes an almost perfect cache object. Small, public, repeatedly requested, and served by HAProxy instead of snac.

ActivityPub JSON microcaching

The ActivityPub side starts from the Accept header:

acl is_ap_json   req.hdr(Accept),lower -m sub application/activity+json
acl is_ap_ldjson req.hdr(Accept),lower -m sub application/ld+json
acl is_outbox    path_end /outbox
acl is_get       method GET
acl has_auth     req.hdr(Authorization) -m found
acl has_cookie   req.hdr(Cookie) -m found

This part matters because ActivityPub uses content negotiation. The same path may return HTML to a browser and JSON to a remote instance. If the proxy pretends that a URL is always one thing, it will eventually cache the wrong representation.

So I only mark public ActivityPub GET requests as cacheable:

http-request set-var(txn.is_activitypub) bool(true) if is_get !is_outbox is_ap_json !has_auth !has_cookie
http-request set-var(txn.is_activitypub) bool(true) if is_get !is_outbox is_ap_ldjson !has_auth !has_cookie

There are several decisions here, all important.

It must be a GET, because I am not caching deliveries or anything that changes state. It must not be /outbox, because outbox collections are not the traffic I want to cache here. It must not have Authorization, and it must not have cookies, because authenticated or user-specific requests do not belong in a shared public cache.

Then the cache can be used and populated:

http-request cache-use jsoncache if { var(txn.is_activitypub) -m bool true }

http-response set-header Cache-Control "max-age=60, public" if { var(txn.is_activitypub) -m bool true }
http-response cache-store jsoncache if { var(txn.is_activitypub) -m bool true }

Sixty seconds is short, but useful. Federation often creates small clusters of identical requests. A remote server fetches an actor, another fetches the same actor, something asks for the same object, something retries. I do not need to cache these responses for hours. I only need HAProxy to answer the second and third identical request during the same small burst.

This is microcaching in the most practical sense. It reduces repeated work without changing the nature of the service.

Static media paths

There is also a rule for static paths:

acl is_short_path path_reg ^/[^/]+/s/
http-request cache-use mediacache if is_short_path

This comes from the same observation that led me to cache snac media with nginx. snac uses static media paths, and those paths often represent the kind of public, repeatable traffic that should not consume backend threads if the proxy can serve it. I call them "short", not because they are, but because the first time I saw them, I thought the 's' stood for "short", not "static". The name just stuck.

In FediMeteo this is less central than on a normal social instance, because I deliberately do not use media except for the avatar and basic static objects. Still, the rule fits the general policy: let HAProxy handle repeatable edge work, and let snac spend its threads where they are actually needed.

`Vary`, but not without limits

Both caches have:

process-vary on
max-secondary-entries 12

I want HAProxy to process Vary, because content negotiation is real, especially when ActivityPub is involved. But I also want variation to be bounded. If every slightly different header creates another cache entry, the cache becomes a complicated way to miss.

For media, I remove Vary before storing the response. A shared avatar does not need to vary by Accept. For ActivityPub JSON, I am more careful because the representation matters.

Again, the important thing is not the number itself. It is the decision to make variation explicit and limited.

Seeing whether it works

During rollout, I like to expose a very small diagnostic header:

http-response set-header X-Cache-Status HIT if !{ srv_id -m found }
http-response set-header X-Cache-Status MISS if { srv_id -m found }

This is intentionally simple. If HAProxy selected a backend server, I call it a miss. If no backend server was selected, the response came from cache, so I call it a hit. It is not a complete observability system, but it is enough to answer the first question I usually have after changing a cache rule.

Did this request reach snac?

A test can be as simple as:

curl -I https://it.fedimeteo.com/path/to/avatar.png
curl -I https://it.fedimeteo.com/path/to/avatar.png

The second request should be a hit.

For ActivityPub JSON, the test must use the right Accept header:

curl -I \
  -H 'Accept: application/activity+json' \
  https://it.fedimeteo.com/some/activitypub/object

And I also want to verify that cookies and authorization prevent public caching:

curl -I \
  -H 'Cookie: test=value' \
  -H 'Accept: application/activity+json' \
  https://it.fedimeteo.com/some/activitypub/object

curl -I \
  -H 'Authorization: Bearer fake' \
  -H 'Accept: application/activity+json' \
  https://it.fedimeteo.com/some/activitypub/object

A cache that works should be visible. A cache that is invisible can be correct, but it can also be silently wrong. I prefer to know.

Compression and operational paths

HAProxy also handles gzip compression:

filter compression
compression algo gzip
compression type text/css text/html text/javascript application/javascript text/plain text/xml application/json application/activity+json

This keeps another common responsibility at the edge. The country instances can stay focused on snac and the forecast data, while HAProxy deals with client-facing compression for HTML, JSON, and ActivityPub responses.

There is also a local Prometheus exporter:

frontend prometheus
  bind 127.0.0.1:8405
  mode http
  http-request use-service prometheus-exporter
  no log

And I keep internal operational paths, such as statistics and Grafana, handled before the hostname map. These are small details, but ordering matters. Special paths should be explicit and early. The hostname map is for FediMeteo routing, not for every internal tool I happen to expose behind the same proxy.

What this changes in practice

The nice thing about this configuration is that none of its parts is particularly surprising.

The map keeps hostname routing manageable. The backend definitions keep each country isolated and limited. The static homepage avoids dynamic work for something that changes once per hour. The shared avatar gives HAProxy one very hot media object to serve directly. The media cache keeps public files away from snac. The JSON microcache absorbs short ActivityPub bursts. Header cleanup prevents useless variation. Connection reuse avoids unnecessary backend connection churn.

But all of this is only a longer way of saying one thing:

fewer requests reach snac.

That is the metric I care about here.

Not because snac is slow. If anything, FediMeteo exists in its current form because snac is efficient enough to make this kind of project possible on a very small VPS. But precisely because the whole architecture is small and pleasant, I do not want to waste resources where there is no need.

This is also consistent with the rest of the project. Forecasts are serialized by scripts. Updates happen every six hours. The homepage is regenerated hourly. Countries live in separate jails. Snapshots and backups are handled outside the application. No single component tries to be the entire system.

HAProxy is just another small piece, but it sits in the right place to remove a lot of repeated work.

Caveats

This configuration is not a universal HAProxy recipe for ActivityPub services.

It matches FediMeteo as it is now: almost no media, one shared avatar, static homepage, public forecasts, many small snac instances, and ActivityPub traffic that can benefit from a short public cache when there are no cookies or authorization headers.

If I decide one day to use media in forecasts, the media cache rules will need to be reviewed. If I use different avatars for each city or country, the cache will still work, but I will lose the very nice property of one shared, always-hot avatar. If ActivityPub responses become actor-dependent, public JSON caching must be reconsidered. If one country grows a very different traffic pattern from the others, it may deserve a different limit or policy.

This is why I do not like presenting configurations as magic. A good configuration is a written form of the assumptions behind a service. When the assumptions change, the configuration must change too.

Conclusion

FediMeteo started as a small idea and became larger than I expected, but I still want it to feel small in the right ways. Small does not mean fragile. Small means understandable. It means that each part has a reason to exist, and that unnecessary work is removed before it becomes a problem.

The HAProxy layer follows this idea. It terminates TLS, routes hostnames through a map, reuses backend connections, serves the shared avatar from cache, microcaches public ActivityPub JSON, avoids authenticated and cookie-based traffic, and gives me a small diagnostic header to see what is happening.

There is no single brilliant directive here. There is only the usual work of matching infrastructure to reality.

FediMeteo publishes weather forecasts as text and emoji. The homepage is static HTML updated every hour. The accounts share the same avatar because it is enough, and because it is better for the cache. Each country has its own snac instance in its own FreeBSD jail. HAProxy stands in front of them and tries, quietly, not to bother them unless it has to.

I like this kind of infrastructure.

Not because it is invisible, but because when it works well, it leaves very little to say.

Self-hosting your Mastodon media with SeaweedFS

Stefano Marinelli — Thu, 06 Nov 2025 11:30:02 +0000

Mastodon 4.5.0 is here, and with it come some interesting changes that, in my opinion, might encourage more people to consider it for self-hosting their Fediverse community.

While it may not be as lightweight and simple as other solutions (like snac or GoToSocial or many others), I believe it remains one of the best platforms for managing a medium-sized Fediverse community, thanks in part to the direct feedback that many admins have provided to the developers.

I have previously written about how to install Mastodon in a FreeBSD jail and how to modify its character and poll limits.

One of the most critical initial decisions (which can be changed later, but with extra work) is where to store your media files. Mastodon downloads and re-processes all media it encounters from other instances for three main reasons:

Local Caching: Your users connect to your media server, reducing the load on the original instance.
Security: Re-processing media helps to remove any potential "impurities" before they reach the user's device.
Privacy: It prevents disclosing your users' IP addresses to other instances. A user will only connect to their own instance to fetch all data, including remote content.

At least initially, media files will be the largest part of your instance's storage footprint. It is therefore essential to plan where to store them and to add a regular cleanup script; otherwise, their growth will be exponential.

Mastodon supports uploading media to external S3-compatible solutions, and many admins use the usual commercial providers, paying for data uploads and transfers.

I am a firm believer in "Own Your Data", so I have always used my own self-hosted S3 servers. I initially started with Minio, but over time, I realized that, by design, it doesn't perform well with a multitude of small files (performance degrades). After running some tests, I decided to switch to SeaweedFS.

SeaweedFS "is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek..." - this, combined with the fact that it is a mature and proven piece of software, was enough for me to give it a try. The result? Excellent. The I/O and CPU load on my media server dropped drastically, making SeaweedFS an incredibly suitable solution. Furthermore, some of its features (like the ability to run a filer.sync) allow for efficient and fast replication to other storage, another host, or... anything else.

SeaweedFS works perfectly with Mastodon, and I will explain the steps to get it into production.

I will install SeaweedFS in a dedicated jail and use a dedicated subdomain. This ensures that the media server can be moved to another host at any time without reconfiguring everything or changing domains. SeaweedFS has its own FreeBSD package, installable via pkg, or can be downloaded directly from the project's website.

In either case, I will describe a "test" setup - which can also be used in production without issues. However, I highly recommend diving deeper into the tool, as it is incredibly powerful and flexible and can solve many more problems than one might imagine.

Setting up the SeaweedFS Jail

First, let's create a dedicated jail with BastilleBSD:

bastille create media 14.3-RELEASE 10.0.0.66 bastille0

Now, let's enter the jail and install SeaweedFS (and tmux, which can be useful):

bastille console media
pkg install -y tmux seaweedfs

I suggest launching SeaweedFS in a tmux session so you can monitor its output. Later, you should configure an automatic startup method, such as using the included rc.d file or any other method you prefer.

Create a directory for the data and start SeaweedFS as the "seaweedfs" user:

mkdir -p /seaweedfs/data
chown -R seaweedfs /seaweedfs
su -m seaweedfs
cd /seaweedfs/
/usr/local/bin/weed server -dir /seaweedfs/data -s3

At this point, SeaweedFS will start and create everything it needs to function, including the S3 server.

Configuring Buckets and Users

Now, let's open the weed shell to create the necessary bucket and users:

weed shell
s3.bucket.create -name mastomedia

Still in the weed shell, create a user for Mastodon and grant read permissions for unauthenticated users (which is necessary to serve media to the world):

s3.configure -access_key=mastomedia -secret_key=CHANGEME -buckets=mastomedia -user=mastodon -actions=Read,Write,List,Tagging,Admin -apply
s3.configure -buckets=mastomedia -user=anonymous -actions=Read -apply
s3.configure -buckets=mastomedia -actions=Read -apply

Security Tip: For the -secret_key, avoid using a simple password. You can generate a strong, random key directly from your shell with a command like openssl rand -base64 32.

Done. SeaweedFS is now ready to receive (and serve) media. The next step is to set up a reverse proxy to serve everything over HTTPS. My preferred approach is to configure the system as if it were external, even if the services are in adjacent jails. This might use slightly more resources, but the time and trouble it saves in the future are well worth it.

Nginx Reverse Proxy Configuration

The reverse proxy can be configured something like this:

[...]

server {
   server_name  media.mastodon.example.com;

   ignore_invalid_headers off;
   client_max_body_size 0; # Allow large file uploads without Nginx limits

   location / {
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;

      proxy_connect_timeout 300;
      proxy_http_version 1.1;
      proxy_set_header Connection "";
      chunked_transfer_encoding off;

      expires 1y;
      add_header Cache-Control public;

      add_header X-Cache-Status $upstream_cache_status;
      add_header X-Content-Type-Options nosniff;

      proxy_pass http://10.0.0.66:8333;
   }

# ... other server configurations like SSL ...

}

Mastodon Configuration

Now let's configure Mastodon. If you are running the setup wizard for the first time, here is a summary of the options:

[...]
Do you want to store uploaded files on the cloud? yes
Provider Minio
Minio endpoint URL: https://media.mastodon.example.com
Minio bucket name: mastomedia
Minio access key: mastomedia
Minio secret key: CHANGEME
Do you want to access the uploaded files from your own domain? Yes
Domain for uploaded files: media.mastodon.example.com

If Mastodon is already active, or once the setup is complete, the options in your .env.prod file should be modified to be consistent with what SeaweedFS expects:

S3_ENABLED=true
S3_PROTOCOL=https
S3_REGION=us-east-1
S3_ENDPOINT=https://media.mastodon.example.com
S3_HOSTNAME=media.mastodon.example.com
S3_BUCKET=mastomedia
AWS_ACCESS_KEY_ID=mastomedia
AWS_SECRET_ACCESS_KEY=CHANGEME
S3_FORCE_SINGLE_REQUEST=true
# remove the S3_ALIAS_HOST if it is set

IMPORTANT NOTE: If both services are in jails on the same host (i.e., SeaweedFS is on the same host as Mastodon), you should ensure that the Mastodon jail can reach the SeaweedFS jail through the reverse proxy and not via the external IP. To do this, add the following line to the /etc/hosts file of the Mastodon jail:

10.0.0.1        media.mastodon.example.com

In this example, the reverse proxy is at 10.0.0.1. If you are not using a separate reverse proxy but are exposing Nginx directly from the jail (as described in my Mastodon installation article), use the IP of the Mastodon jail itself instead (e.g., 10.0.0.42).

With this setup, Mastodon will be able to upload media to the SeaweedFS server and generate the correct links for other instances, public visitors, and users of your own instance.

Have fun with SeaweedFS!

Introducing the illumos Cafe: Another Cozy Corner for OS Diversity

Stefano Marinelli — Mon, 18 Aug 2025 09:04:00 +0200

Introducing the illumos Cafe: Another Cozy Corner for OS Diversity

From the BSD Cafe to illumos Cafe

The idea for this new project was born from the success of the BSD Cafe, an initiative I introduced to the world in July 2023, which received an incredibly positive response. Far more than I ever anticipated. The BSD community already had its well-established hubs: in the Fediverse, places like bsd.network, exquisite.social, and others were already thriving, not to mention all the forums, channels, and Reddit communities.

But in my vision, something was still missing: a hub of services with a positive spirit, built exclusively with open-source tools, where people could come to share, learn, and experience technology with a positive mindset. The BSD Cafe is therefore not just an instance, but a true Cafe - I’ll be speaking more about the BSD Cafe in detail at the next EuroBSDCon.

Why Another Cafe?

In a world increasingly dominated by centralized services under the control (or lack thereof) of the usual big players, it has become essential to create free, independent communities, devoid of the algorithmic and commercial controls that influence our overall experience. From day one, the BSD Cafe has embodied this spirit.

Linux is a good kernel, and there are excellent distributions based on it (some using the GNU userland, others only partially, like Alpine Linux), but it cannot and should not become a monoculture. The alternatives are extremely capable, and for many use cases - in my opinion and experience - they are even more suitable. BSD systems have served me exceptionally well for over 20 years, providing stability and security. At the same time, many other operating systems are renowned for their robustness, reliability, and the quality of their design and implementation.

Why illumos?

illumos is one of them. As the open-source descendant of OpenSolaris, it is an operating system known for its enterprise-grade stability and innovative technologies like ZFS, DTrace, and "zones". It was born from the solid foundations of Solaris and has evolved over time while remaining true to many of its core principles. I have always seen illumos and its distributions as kindred spirits to the BSDs, despite their differences. The philosophy is one of evolution without revolution, of guaranteeing long-term continuity and reliability rather than chasing the latest hype. This is precisely why, for some time now (and thanks in part to the inspiring posts by Joel Carnat, which further sparked my curiosity), I have been running OmniOS and SmartOS alongside my BSD-based setups for certain workloads.

However, there is very little information online about services running on them. So, a few months ago, I began to consider a new project: the illumos Cafe.

The illumos Cafe Project

The illumos Cafe is a project similar to the BSD Cafe (though perhaps less complex, at least initially). It shares the same spirit of positivity and inclusivity and aims to provide services running on illumos-based operating systems to demonstrate that there are no reasons not to use them. Just like with the BSD Cafe, diversifying the operating systems we use - even while using the same platforms - is fundamental to improving the reliability and resilience of the Internet. The Internet was born as a decentralized network, but for most people, it has sadly become just a tool to access the services of big players.

Community and Philosophy

But we want to connect. We want relationships with people, between people. We don't want algorithms. We don't want our data to be monetized by "us and our 65535 partners". We want a network that serves us, an OS that serves us - not an OS that just serves as a vehicle to store our data in "someone else's house". The illumos Cafe, therefore, aims to be a home for anyone interested in developing, using, or who is simply curious about illumos-based operating systems.

Technical Setup

As with the BSD Cafe, the entire setup will be documented. For now, it is very simple: there is a VM (running on FreeBSD and bhyve, on hardware I manage) where I have installed SmartOS. The physical host also runs the reverse proxy (in a jail). Inside the SmartOS VM, there are a series of zones:

Zone 1: nginx (Web Server) - Currently serving the project's homepage.
Zone 2: Mastodon (Social) - Hosting the Mastodon instance and its dependencies at https://mastodon.illumos.cafe.
Zone 3: PostgreSQL (Database) - The Mastodon database, on a dedicated zone.
Zone 4: Redis (Cache) - The Mastodon cache, on a dedicated zone.
Zone 5: snac (LX Zone) - Currently in an LX zone (Alpine) as I ran into some issues getting it to work in a native zone. It will be moved to a native zone as soon as I resolve them. It's serving the snac instance at https://snac.illumos.cafe

Media files are stored on an external physical server (running FreeBSD, the same one as the BSD Cafe, but in a dedicated jail) with SeaweedFS. I was able to compile and run SeaweedFS on illumos without any problems, but at the moment, I don't have a host with enough storage space for the media.

Available Services

More services will arrive over time. For now, two gateways to the Fediverse are already available:

Mastodon - https://mastodon.illumos.cafe
snac - https://snac.illumos.cafe

Both instances share the same rules as the BSD Cafe. Positivity. Supporters, not haters. I want them to be places of enjoyment, not venting. Of friendship, not hate.

Registrations and Logo

Registrations for the Mastodon instance are now open, and the available themes are the default ones plus the colorful TangerineUI - whose orange hue echoes the illumos logo.

The project's logo was not generated by an AI. I made it myself by hastily sticking the illumos SVG onto a coffee cup. Basic, perhaps. But authentic.

Looking Ahead

The BSD Cafe will, of course, remain my primary home. But I want to bring illumos into the Fediverse and provide a home for anyone who wishes to share their interest in this excellent OS.

I will document the entire process, just as I did with Mastodon on FreeBSD, as it is a bit more intricate. Because in my dreams, I see Fediverse statistics showing instances spread fairly evenly across the major open-source operating systems. Because relying on a single OS, even if it's open-source, and ceasing to support the others is also a single point of failure.

New Article on BSD Cafe Journal: WordPress on FreeBSD with BastilleBSD

Stefano Marinelli — Mon, 21 Jul 2025 09:30:00 +0200

New Article Published

I'm excited to announce that I have published a new, in-depth article on the BSD Cafe Journal: "WordPress on FreeBSD with BastilleBSD: A Secure Alternative to Linux/Docker".

This piece explores how to create a robust and secure WordPress installation on FreeBSD using BastilleBSD, leveraging the power and isolation of FreeBSD jails as a compelling alternative to the more common Linux and Docker stack.

Future Technical Content

I'm excited to announce that I'm expanding my writing to a new platform! From now on, some of my more technical, long-form articles and tutorials will be published on The BSD Cafe Journal, a fantastic hub for BSD-related content that I'm happy to now contribute to.

This new collaboration complements the work I do here. My personal blog will continue to be my home base, and you won't miss a thing! I'll still be posting my own articles and announcements right here, and I'll always include a direct link to any new content I publish elsewhere. This space will remain as active as ever.

Thank you for reading

Make Your Own Internet Presence with NetBSD and a 1 euro VPS – Part 1: Your Blog

Stefano Marinelli — Tue, 22 Apr 2025 07:30:36 +0200

Why NetBSD?

For many years, I've been using (and appreciating) NetBSD because it's stable, efficient, and reliable. The codebase has proven its reliability, running without reboots for years without issues. It supports ZFS (though differently than FreeBSD), LVM (useful for those accustomed to it on Linux), the ability to take filesystem snapshots (UFS2, making ZFS less crucial), and it's an excellent virtualization platform. Installation and updates are easy (including via sysupgrade - which I'll cover in a future article). Since it focuses on portable and optimized code (running on ancient architectures requires cleanliness and correctness), it's particularly efficient on low-power devices, like embedded systems or cheap VMs. Therefore, it's one of the best solutions for a small personal setup that can still deliver excellent results and simple management.

Indeed, the market offers very cheap VPS, often with just a single core and little RAM. But a modern single core packs power that a multi-core from just a few years ago could only dream of, and often, the I/O of these machines (a bottleneck for many services) is still decent. I personally use 1 euro per month VPS (VAT included - for those not subject to it, that's less than one euro per month!) with a public IPv4 address and (often) a /64 IPv6 block, ensuring full reachability across the entire network. I'm not providing direct links as I have no affiliations, but netcup's "piko" VPS are among the types I use most often (a 4 euro/month netcup VM handles the entire FediMeteo project), and this type of VM is ideal for our purpose because some providers (like netcup) allow you to upload your own ISO and install your preferred operating system. On VPS like these, I've installed everything - including OmniOS and SmartOS - without problems. And even such a small VPS, with an efficient operating system, can be extremely satisfying.

Why BSSG?

In this article, I'll describe how to create and publish a blog using BSSG as it exemplifies my concept of portability and minimalism. BSSG on NetBSD currently doesn't leverage parallelism provided by tools like GNU Parallel, but for small to medium-sized blogs, this won't be an issue, especially considering these small VMs only have 1 core. Obviously, you can use any Static Site Generator (SSG) (like Hugo, Nikola, 11ty, Pelican, Zola, etc.) - the important thing is to have a static site served by a simple web server.

Let's Start with the Installation

Installing NetBSD is quite straightforward and is clearly covered, complete with explanatory screenshots, in the excellent official NetBSD documentation, which I recommend using as a reference during the process, especially if it's your first time.

In my case, I made sure to use the proposed disk geometry, use the standard automatic partitioning, but enable the "log" and "noatime" options for the filesystem. Both these options will provide a huge advantage in I/O operations, especially with BSSG, as the first enables journaling and the second prevents updating file metadata on every access. BSSG is more I/O bound than CPU bound, so any optimization is beneficial.

Moving forward, I also recommend configuring the network (although installation can be done from packages on the installation ISO). For netcup, you can use DHCPv4 (even though it's a bit slow and sometimes seems to fail, the DHCP client will continue running in the background and eventually work).

For IPv6, I usually configure it manually later, so I'll describe that further down.

I also recommend enabling SSH, adding a regular user (and adding them to the wheel group so they can gain root privileges) - in this case, I'll call the user blog. Also, enable the installation of binary packages, as it will be convenient later to use pkgin to install and update all necessary packages. All these steps are described clearly and in detail in the guide, so I won't detail them here. But they are simple and logical, like all operations on BSD systems.

After installation, reboot. If everything went correctly, you should be able to log in via console or SSH using the "blog" user (or whatever you named it).

First, I suggest configuring the IPv6 address and installing the necessary packages.

For IPv6, in the case of netcup, simply add one of the assigned addresses to the interface. In NetBSD, network interface configurations are stored (similar to OpenBSD) in specific files. For the first virtio interface, the file will be /etc/ifconfig.vioif0.

You need to elevate your privileges to root, open that file with your preferred editor, and add the configuration to the file itself:

nb1euro$ su -l
nb1euro# vi /etc/ifconfig.vioif0

inet6 your-ipv6-addr/64
up

To test everything, perform a reboot and try pinging an IPv6 address (I often use ping6 google.com).

If all goes well, after a few seconds, you should see ping replies, confirming everything is configured correctly.

Regarding packages, the only two strictly necessary ones are bash and a markdown processor (by default, BSSG will use commonmark; otherwise, it can be configured to use pandoc or Markdown.pl). rsync can be useful for deployment. sudo (or doas) can be useful for elevating privileges for certain operations, at least at this stage.

nb1euro$ su -l
nb1euro# pkgin in bash cmark rsync sudo

If you're used to Linux, you can also install the "nano" editor:

nb1euro# pkgin in nano

If sudo was installed, it's now appropriate to grant users in the "wheel" group (like the regular user created during installation) the ability to elevate privileges. Edit the sudoers file (I suggest using the visudo command) and uncomment this line:

## Uncomment to allow members of group wheel to execute any command
%wheel ALL=(ALL:ALL) ALL

At this point, you can switch back to operating as the regular user, downloading and unpacking BSSG:

nb1euro$ ftp https://brew.bsd.cafe/stefano/BSSG/archive/0.15.1.tar.gz
nb1euro$ tar zxfv 0.15.1.tar.gz

Now that BSSG is ready, just initialize a directory with the structure for the new site:

nb1euro$ cd bssg
nb1euro$ ./bssg.sh init /home/blog/myblog

Everything is set to start generating your blog. I recommend reading BSSG's README.md. There are many options, themes, etc., but to get started, you just need to set the site's public URL. For example, if the site will be published as myblog.example.com - just create a file at /home/blog/myblog/config.sh.local (the path defined by the init command) and set the public URL:

SITE_URL="https://myblog.example.com"

This way, all URLs will be absolute URLs, which is necessary to ensure the correct functioning of RSS feeds, sitemaps, etc. This setting assumes HTTPS - if you just want to test the site over HTTP, simply use http and then, optionally, change it to https and regenerate the site later.

You can already create your first test post, directly from the BSSG directory:

nb1euro$ ./bssg.sh post

The system will use nano if it's installed, otherwise it will use vi. Don't worry, in the latter case, BSSG will write the procedure for exiting vi as the post's text 🙂

Once you save the post, BSSG will automatically generate the site. If everything went well, the /home/blog/myblog/output directory will contain the final result. We are therefore ready for the first deployment, which can be done in many different ways. I will cover three:

Using bozohttpd, present by default in NetBSD's base system. It can be used via inetd (launching an httpd process for each connection) or as a daemon. I'll describe the first option, showing in the final benchmarks how, even when used as a daemon, it remains a less performant solution.
Using nginx
Using Caddy

First, it's advisable to obtain a certificate to configure and use HTTPS. If you only want to test using HTTP, this part can be safely bypassed. For solutions 1 and 2, I'll use certbot, which is well-known to many users with Linux experience. Caddy, on the other hand, manages certificates automatically, so there's no need for other solutions and thus no need to install certbot.

nb1euro$ sudo pkgin in py313-certbot

To use bozohttpd, no further installation is necessary. At this point, the options diverge.

Using NetBSD's Integrated httpd

bozohttpd is integrated into NetBSD and, by default, can be launched directly via inetd. This solution, while not extremely efficient or scalable, is simple and requires few resources. It's fine if you expect only a few visits per day, but when used via inetd, the initial latency for each connection is tangible. It can still be useful for some tests or small deployments.

The /etc/inetd.conf file already contains the options to handle this situation:

#http           stream  tcp     nowait:600      _httpd  /usr/libexec/httpd      httpd /var/www
#http           stream  tcp6    nowait:600      _httpd  /usr/libexec/httpd      httpd /var/www

By uncommenting these two lines and restarting inetd (service inetd restart), the server will start responding to HTTP requests on both IPv4 and IPv6.

If you want to add HTTPS support, no problem. Just request a certificate via certbot and specify the webroot.

Run:

nb1euro$ sudo certbot-3.13 certonly

Choose option 2 - the one where you specify the webroot - enter the domain, and when prompted, provide /var/www/ as the webroot.

The certificate will be created. Then, modify the /etc/inetd.conf file to also include support for HTTPS, adding two lines similar to these (obviously, change the certificate paths):

https            stream  tcp     nowait:600      _httpd  /usr/libexec/httpd      httpd -Z /usr/pkg/etc/letsencrypt/live/myblog.example.com/fullchain.pem /usr/pkg/etc/letsencrypt/live/myblog.example.com/privkey.pem /var/www
https            stream  tcp6    nowait:600      _httpd  /usr/libexec/httpd      httpd -Z /usr/pkg/etc/letsencrypt/live/myblog.example.com/fullchain.pem /usr/pkg/etc/letsencrypt/live/myblog.example.com/privkey.pem /var/www

Warning: httpd will run with the permissions of the _httpd user, so make sure all certificates are readable by that user:

nb1euro# chown -R _httpd /usr/pkg/etc/letsencrypt/

Restart inetd, and the server will also respond over HTTPS.

To make your blog public, simply copy the files from the site's output directory to /var/www/ - this time using sudo to bypass permission issues:

nb1euro$ sudo rsync -avhHPx /home/blog/myblog/output/ /var/www/

The site will be immediately visible.

Using nginx

Nginx is fast and efficient, and the performance difference is noticeable (some benchmarks follow below). For an efficient setup ready for a high number of visits, it's advisable to use a web server suited for the purpose, just like nginx.

First, install nginx and the certbot plugin for nginx. This will simplify the installation and renewal of certificates:

nb1euro$ sudo pkgin in py313-certbot-nginx nginx

Copy the startup script to /etc/rc.d - as indicated by the post-installation message. In NetBSD, this operation must be done manually, but it's always pointed out:

nb1euro$ sudo cp /usr/pkg/share/examples/rc.d/nginx /etc/rc.d

Warning: If you previously used httpd from inetd following the previous solution, you must disable it in inetd.conf and restart inetd to free up ports 80 and 443.

Now you can create a virtual host for our new site.

nb1euro$ sudo vi /usr/pkg/etc/nginx/nginx.conf

and add, at the end of the file and before the final closing curly brace:

server {
        listen 80;
        # If you also have configured IPv6 support
        listen [::]:80;

        root /var/www;
        index index.html index.htm;

        server_name myblog.example.com;

        # If you want a long cache for media and css - be careful, this means that if you change to a new theme, it might not be visible immediately as the browser might still use the old cached one
        location ~* \.(jpg|jpeg|png|gif|ico|css|js)$ {
            expires 30d;
            add_header Cache-Control "public, no-transform";
        }

        location / {
                try_files $uri $uri/ =404;
        }
}

Now, it's time to configure the system to enable nginx. Just edit /etc/rc.conf:

nb1euro$ sudo vi /etc/rc.conf

and add:

nginx=YES

Now, you can start nginx:

nb1euro$ sudo service nginx start

Nginx will start listening on port 80. Generating and installing the certificate is very simple:

nb1euro$ sudo certbot-3.13 --nginx -d myblog.example.com

This command will request the certificate and install it, so nginx will already be configured to use it.

As with the previous method, to make your blog public, simply copy the files from the site's output directory to /var/www/ - using sudo to bypass permission issues:

nb1euro$ sudo rsync -avhHPx /home/blog/myblog/output/ /var/www/

The site will be immediately visible.

Using Caddy

Caddy is a convenient and all-in-one solution, efficient and fast. It's packaged for NetBSD and allows you to go online in a flash. I won't delve into the configuration because there are many tutorials (including the official ones), but you just need to install it and run it:

nb1euro$ sudo pkgin in caddy

Once installed, go to the directory you want to serve (e.g., /var/www or directly /home/blog/myblog/output) and run:

nb1euro$ sudo caddy file-server --domain myblog.example.com

Caddy will start, request the certificate, and begin serving your blog over HTTPS as well. To install Caddy as a service (i.e., with a configuration file, etc.), you can proceed similarly to how it's done on Linux. The NetBSD Caddy package doesn't include the rc.d script, but you can copy and paste one (into /etc/rc.d/caddy) from a thread posted on UnitedBSD.

Performance Comparison

I performed some performance tests on these solutions. Here are the results, on a single-core 1 euro/month VPS, from my home connection (which also has its own limitations):

NetBSD httpd via inetd:

Running 10s test @ https://myblog.example.com/
  4 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   213.52ms  173.10ms   1.11s    76.01%
    Req/Sec    12.92      9.19    50.00     75.91%
  371 requests in 10.10s, 1.39MB read
Requests/sec:     36.72
Transfer/sec:    140.65KB

These numbers are quite poor, linked to high latency caused by having to launch bozohttpd for each incoming connection.

NetBSD httpd as a daemon:

Running 10s test @ https://myblog.example.com/
  4 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    35.74ms    6.96ms 108.80ms   81.36%
    Req/Sec    18.29      9.45    50.00     70.88%
  676 requests in 10.10s, 2.53MB read
Requests/sec:     66.92
Transfer/sec:    256.32KB

Here the situation is decidedly better, but not exceptional. httpd isn't designed for high loads or performance.

Nginx as a daemon, 1 worker:

Running 10s test @ https://myblog.example.com/
  4 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    30.69ms    4.87ms  64.14ms   66.01%
    Req/Sec   379.39     65.94   464.00     90.91%
  15026 requests in 10.04s, 56.50MB read
Requests/sec:   1496.65
Transfer/sec:      5.63MB

Here we are on another level, showing truly solid performance. This type of result can handle significantly high loads without particular difficulty. The efficiency of both NetBSD and nginx pays off.

Caddy:

Running 10s test @ https://myblog.example.net/
  4 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    32.10ms    5.75ms  95.04ms   87.44%
    Req/Sec   362.74     64.29   434.00     91.67%
  14374 requests in 10.05s, 54.63MB read
Requests/sec:   1430.82
Transfer/sec:      5.44MB

Caddy shows results comparable to nginx, so the choice between them depends solely on the type of configuration you want to achieve and the experience each person has with the specific platforms.

Conclusion: Efficient Minimalism

We've seen how it's possible to create a personal, professional, and performant online presence with minimal investment. This solution, based on NetBSD and a 1€/month VPS, offers several advantages:

Negligible Cost: For 12€ per year, you can have a website (and more!) completely under your control.
Surprising Performance: As demonstrated by the benchmarks, excellent performance can be achieved even with limited resources (up to 1400-1500 requests/second with nginx or Caddy).
Security and Stability: NetBSD is renowned for its reliability and security, fundamental characteristics for any online service.
Total Control: Unlike free blogging platforms, you have full control over every aspect of your site.
Learning Experience: Managing a BSD system allows you to acquire valuable system administration skills.

This minimalist configuration demonstrates that you don't need to invest in expensive cloud solutions or oversized VPS to have a quality online presence. In an era where the tendency is to think "moooar powaaaar = better results", NetBSD reminds us that efficiency and good design can yield excellent results even with limited resources.

After all, you don't need a thousand-node cloud to write something worth reading.

In the upcoming articles in this series, we will explore how to expand this basic installation with other useful services and how to keep the system updated and secure over time.

Launching BSSG - My Journey from Dynamic CMS to Bash Static Site Generator

Stefano Marinelli — Mon, 07 Apr 2025 08:11:36 +0200

I've had my own website practically forever. Back in the late '90s, I already had a web page on my ISP's server, and since at least 2001, I've had my own homepage on my own server. I've never been a great graphic designer, let alone a skilled webmaster, so I've always tried to keep things minimal and compatible.

Initially, like many others, I wrote HTML pages by hand. Then I used WYSIWYG creation tools, and eventually, I landed on CMS (Content Management Systems).

The Era of Dynamic CMS

I liked CMS because they allowed me to focus on the content and not on the correctness of the generated HTML. Thanks to them, I started writing my first blog shortly afterward.

Over the years, I've used many tools like PHPNuke, FlatNuke (created and developed by my friend Simone Vellei), eventually moving through Joomla and Wordpress. Wordpress always seemed like the most suitable tool for the job, and I used it for many years. Even today, mainly on the sysadmin side, I manage hundreds of Wordpress sites, and they are reasonably reliable, aside from the plugins (because the problem with Wordpress isn't the software itself, but many of the external plugins).

But this is precisely the problem: all dynamic CMS require constant and continuous security updates because, without them, the chances of defacement are extremely high.

Discovering Static Site Generators

And that's precisely why, when I discovered Carlos Fenollosa's bashblog in 2014, it immediately became clear that, indeed, there was no reason to continue down the path of dynamic CMS. I don't write often, I don't update often, there's no reason to regenerate all the content with every visit. Sure, WordPress caching plugins are often quite effective, but they are still add-ons that need to be kept up to date. And I'm not a fan of adding things to streamline. Often, less is more.

So, I started using bashblog for some 'secondary' projects until, in 2015, I migrated my 'old' Italian blog from WordPress to Pelican. Shortly after, I moved from Pelican to Nikola, and that blog is still generated by Nikola, although (that blog's) updates are now extremely rare (so much so that I consider it almost abandoned). I also created the first Docker container for Nikola and, for a long time, it was listed among the deployment methods on their site.

Building My Own: BSSG

But bashblog continued to fascinate me. So in 2015, for fun, I started developing my own Static Site Generator from scratch. I called it (with little imagination), BSSG - Bash Static Site Generator. The plan was for it to be compatible with the main OSes I use, to remain sufficiently simple and straightforward (!!!), and to be tailored to my needs. I intended to use it only and exclusively for small private things, starting with a sort of diary of mine - more professional than personal - and leave the 'official' blogs to more tested and 'professional' tools.

As time went by, I added some small features I liked: theming support, archives, tags (initially absent). Over time, many functions were added, and the script grew large – large enough to make me pause and ask myself some questions about the long-term stability of this solution. So, it remained only for my 'diary', which, however, grew year after year to the point where I needed to devise some kind of optimization. I then developed (more for fun than out of real necessity) a caching system. On rebuild, only what needs to be rebuilt is reconstructed, making the operation sufficiently fast even as the number of posts grows. Obviously, there are limits: using bash and external tools, the efficiency cannot be compared to that of a proper programming language.

Brief Detour: ITNBlog

And it's here that I decided, in preparation for opening a new blog (this one), to create a new tool called ITNBlog. I would develop it in Python and focus a bit more on performance and completeness. But ITNBlog stalled very quickly: time was limited, I'm not a full-time developer, so I realized I would spend too much time on development and too little on content creation.

Therefore, in 2018, I launched this blog but using Ghost, a solution that gave me good results, including performance-wise. I chose Ghost because I thought that, writing content also from my phone while on the go, a real CMS would be useful. Spoiler: no, it didn't turn out that way, so a few years later I decided to migrate this blog to Hugo. Nevertheless, I continued to develop ITNBlog on and off, as a hobby, without any particular ambitions.

At some point, however, I found myself in a particular situation: Hugo deprecated some features, and the theme I had chosen moved forward. But I ended up in an unpleasant situation: using the latest version of Hugo and the current version of the theme would produce unacceptable output; staying with the old version of Hugo while waiting for the theme update meant making a compromise. I actually build the blog from different devices, and they all have different versions of Hugo installed. Change the theme? Feasible, but I would have had to modify almost the entire site.

I considered migrating to manpageblog by gyptazy – I personally love its simplicity and retro look, and it was the main candidate to replace Hugo. I also created a script and migrated all my posts into the correct format.

BSSG to the Rescue (and ITNBlog's Role)

That's when I realized: I would implement the few missing features needed to make ITNBlog sufficiently complete, and this blog would be published using it, ensuring I'd be committed to its development. However, ITNBlog is not mature enough to be released publicly, so for now, it will remain the engine just for my blog. Then I thought again about BSSG – development had stalled some time ago, but it was still in use – and figured that perhaps, with a little tidying up, I could release it.

Because I'm tired of seeing people use dynamic CMS even to implement primarily static blogs or websites – and BSSG, despite its limitations and inefficiencies, works. And there are many themes to choose from. In short, you can install it and generate your blog in seconds.

Why Choose BSSG?

BSSG is the result of a 10-year evolution. The code isn't extremely consistent, some interesting features are missing (which I plan to implement), and it could use refactoring as the build script is monstrously large. But it works, it's portable (and much of the complexity increased precisely because of portability), and it generates sites that achieve very high accessibility and speed scores.

Here are some highlights:

✅ Portability: Uses native OS tools (e.g., md5sum on Linux, md5 on OpenBSD and NetBSD). Portability itself added much of the complexity!
✅ Simple Theming: Themes are just simple CSS files, so the structure remains the same – simplifying theme switching or creating new ones. More than 50 themes are already available!
✅ Essential Features: Supports RSS feed generation, sitemap.xml, OpenGraph tags (to improve social sharing), internationalization (the blog can be in languages other than English – but not multilingual, at least for now), etc.
✅ Built-in Backup and Restore script: It will just copy the configuration file, posts, and pages. Nothing else.
✅ Minimal Dependencies.
✅ Markdown Support: Posts and pages are in Markdown (CommonMark, Pandoc, and markdown.pl are supported).
✅ Feature Images.
✅ Optional GNU Parallel Integration: To speed up build times when there are many posts. This feature significantly impacts the code and has caused me numerous headaches over time. But it's optional (if parallel isn't found, it proceeds traditionally) and only provides benefits when the number of posts increases: with few posts, performance actually degrades.
✅ High Accessibility and Performance Scores: Sites built with BSSG achieve excellent scores.
✅ BSD Licensed: Released under a BSD license.

One of the problems I've always had with all CMS and SSGs has been choosing a theme. In some cases (like Hugo), the theme heavily influences the output, which is both good and bad. Good because it makes each site unique, but bad because it makes switching themes difficult. In the past, I've sometimes found myself having to change themes because they were abandoned and no longer updated. BSSG works differently: theming comes from using a different CSS file, which makes its structure more rigid, but switching from one theme to another is trivial. To help with the choice, I created a script that will build your site using all the themes present in the themes directory, just like on the examples page of the official website. This way, it will be easy to see and test your site with all available themes. If you want to add a touch of originality, you can choose the 'random' theme, and one will be chosen randomly from the list at each site regeneration.

Admin Interface (Experimental)

BSSG is in production use by some clients (for their internal sites), for whom I also created a basic admin interface (using Node Express, partly to chew on a bit of Node), but I don't feel ready to release it immediately as it's not sufficiently tested. It has an integrated Markdown editor and allows post scheduling, generating the files and launching BSSG with the right options at the right time. This could be that connecting link between traditional CMS and SSGs. There are others, but this one is tightly integrated with BSSG.

BSSG is Available Today

Starting today, BSSG is publicly available. It's not perfect, it probably doesn't make sense to do something of this complexity in bash, development will proceed slowly – but it's here, available to anyone who might find it useful.

Happy blogging everyone!

FediMeteo: How a Tiny €4 FreeBSD VPS Became a Global Weather Service for Thousands

Stefano Marinelli — Wed, 26 Feb 2025 07:00:00 +0100

Personal Introduction

Weather has always significantly influenced my life. When I was a young athlete, knowing the forecast in advance would have allowed me to better plan my training sessions. As I grew older, I could choose whether to go to school on my motorcycle or, for safety reasons, have my grandfather drive me. And it was him, my grandfather, who was my go-to meteorologist. He followed all weather patterns and forecasts, a remnant of his childhood in the countryside and his life on the move. It's to him that I dedicate FediMeteo.

The idea for FediMeteo started almost by chance while I was checking the holiday weather forecast to plan an outing. Suddenly, I thought how nice it would be to receive regular weather updates for my city directly in my timeline. After reflecting for a few minutes, I registered a domain and started planning.

Design Principles

The choice of operating system was almost automatic. The idea was to separate instances by country, and FreeBSD jails are one of the most useful tools for this purpose.

I initially thought the project would generate little interest. I was wrong. After all, weather affects many of our lives, directly or indirectly. So I decided to structure everything in this way:

I would use a test VPS to see how things would go. The VPS was a small VM on a German provider with 4 shared cores, 4GB of RAM, 120GB of SSD disk space, and a 1Gbit/sec internet connection and now is a 4 euro per month VPS in Milano, Italy - 4 shared cores, 8 GB RAM and 75GB disk space.
I would separate various countries into different instances, for both management and security reasons, as well as to have the possibility of relocating just some of them if needed.
Weather data would come from a reliable and open-source friendly source. I narrowed it down to two options: wttr.in and Open-Meteo, two solutions I know and that have always given me reliable results.
I would pay close attention to accessibility: forecasts would be in local languages, consultable via text browsers, with emojis to give an idea even to those who don't speak local languages, and everything would be accessible without JavaScript or other requirements. One's mother tongue is always more "familiar" than a second language, even if you're fluent.
I would manage everything according to Unix philosophy: small pieces working together. The more years pass, the more I understand how valuable this approach is.
The software chosen to manage the instances is snac. Snac embodies my philosophy of minimal and effective software, perfect for this purpose. It provides clear web pages for those who want to consult via the web, "speaks" the ActivityPub protocol perfectly, produces RSS feeds for each user (i.e., city), has extremely low RAM and CPU consumption, compiles in seconds, and is stable. The developer is an extremely helpful and positive person, and in my opinion, this carries equal weight as everything else.
I would do it for myself. If there was no interest, I would have kept it running anyway, without expanding it. So no anxiety or fear of failure.

Technical Implementation

I started setting up the first "pieces" during the days around Christmas 2024. The scheme was clear: each jail would handle everything internally. A Python script would download data, city by city, and produce markdown. The city coordinates would be calculated via the geopy library and passed to wttr.in and Open-Meteo. No data would be stored locally. This approach gives the ability to process all cities together. Just pass the city and country to the script, and the markdown would be served. At that point, snac comes into play: without the need to use external utilities, the "snac note" command allows posting from stdin by specifying the instance directory and the user to post from. No need to make API calls with external utilities, having to manage API keys, permissions, etc.

Setting Up for Italy

To simplify things, I first structured the jail for Italy. I made a list of the main cities, normalizing them. For example, La Spezia became la_spezia. Forlì, with an accent, became forli - this for maximum compatibility since each city would be a snac user. I then created a script that takes this list and creates snac users via "snac adduser." At that point, after creating all the users, the script would modify the JSON of each user to convert the city name to uppercase, insert the bio (a standard text), activate the "bot" flag, and set the avatar, which was the same for all users at the time. This script is also able to add a new city: just run the script with the (normalized) name of the city, and it will add it - also adding it to the "cities.txt" file, so it will be updated in the next weather update cycle.

Core Application Development

I then created the heart of the service. A Python application (initially only in Italian, then multilingual, separating the operational part from the text) able to receive (via command line) the name of a city and a country code (corresponding to the file with texts in the local language). The script determines the coordinates and then, using API calls, requests the current weather conditions, those for the next 12 hours, and the next 7 days. I conducted experiments with both wttr.in and Open-Meteo, and both gave good results. However, I settled on Open-Meteo because, for my uses, it has always provided very reliable results. This application directly provides an output in Markdown since snac supports it, at least partially.

The cities.txt file is also crucial for updates. I created a script - post.sh, in pure sh, that scrolls through all cities, and for each one, launches the FediMeteo application and publishes its output using snac directly via command line. Once the job is finished, it makes a call to my instance of Uptime-Kuma, which keeps an eye on the situation. In case of failure, the monitoring will alert me that there have been no recent updates, and I can check.

At this point, the system cron takes care of launching post.sh every 6 hours. The requests are serialized, so the cities will update one at a time, and the posts will be sent to followers.

Growth and Unexpected Success

After listing all Italian provincial capitals, I started testing everything. It worked perfectly. Of course, I had to make some adjustments at all levels. For example, one of the problems encountered was that snac did not set the language of the posts, and some users could have missed them. The developer was very quick and, as soon as I exposed the problem, immediately modified the program so that the post could keep the system language, set as an environment variable in the sh script.

After two days, I decided to start adding other countries and announce the project. And the announcement was unexpectedly well received: there were many boosts, and people started asking me to add their cities or countries. I tried to do what I could, within the limits of my physical condition, as in those days, I had the flu that kept me at home with a fever and illness for several days. I started adding many countries in the heart of Europe, translating the main indications into local languages but maintaining emojis so that everything would be understandable even to those who don't speak the local language. There were some small problems reported by some users. One of them: not all weather conditions had been translated, so sometimes they appeared in Italian - as well as errors. In bilingual countries, I tried to include all local languages. Sometimes, unfortunately, making mistakes as I encountered dynamics unknown to me or difficult to interpret. For example, in Ireland, forecasts were published in Irish, but it was pointed out to me that not everyone speaks it, so I modified and published in English.

A Turning Point

The turning point was when FediFollows (@FediFollows@social.growyourown.services - who also manages the site Fedi Directory) started publishing the list of countries and cities, highlighting the project. Many people became aware of FediMeteo and started following the various accounts, the various cities. And from here came requests to add new countries and some new information, such as wind speed. Moreover, I was asked (rightly, to avoid flooding timelines) to publish posts as unlisted - this way, followers would see the posts, but they wouldn't fill local timelines. Snac didn't support this, but again, the snac dev came to my rescue in a few hours.

Scaling Challenges

But with new countries came new challenges. For example, in my original implementation, all units of measurement were in metric/decimal/Celsius - and this doesn't adapt well to realities like the USA. Moreover, focusing on Europe, almost all countries were located in a single timezone, while for larger countries (such as Australia, USA, Canada, etc.), this is totally different. So I started developing a more complete and global version and, in the meantime, added almost all of Europe. The new version would have to be backward compatible, would have to take into account timezone differences for each city, different measurements (e.g., degrees C and F), as well as, initially more difficult part, being able to separate cities with the same name based on states or provinces. I had already seen a similar problem with the implementation of support for Germany, so it had to be addressed properly.

The original goal was to have a VPS for each continent, but I soon realized that thanks to the quality of snac's code and FreeBSD's efficient management, even keeping countries in separate jails, the load didn't increase much. So I decided to challenge myself and the limits of the economical 4 euros per month VPS. That is, to insert as much as possible until seeing what the limits were. Limits that, to date, I have not yet reached. I would also soon exhaust the available API calls for Open-Meteo's free accounts, so I tried to contact the team and explain everything. I was positively surprised to read that they appreciated the project and provided me with a dedicated API key.

Compatible with my free time, I managed to complete the richer and more complete version of my Python program. I'm not a professional dev, I'm more oriented towards systems, so the code is probably quite poor in the eyes of an expert dev. But, in the end, it just needs to take an input and give me an output. It's not a daemon, it's not a service that responds on the network. For that, snac takes care of it.

Expansion to North America

So I decided to start with a very important launch: the USA and Canada. A non-trivial part was identifying the main cities in order to cover, state by state, all the territory. In the end, I identified more than 1200 cities. A number that, by itself, exceeded the sum of all other countries (at that time). And the program, now, is able to take an input with a separator (two underscores: __) between city and state. In this way, it's possible to perfectly understand the differences between city and state: new_york__new_york is an example I like to make, but there are many.

The launch of the USA was interesting: despite having had many previous requests, the reception was initially quite lukewarm, to my extreme surprise. The number of followers in Canada, in a few hours, far exceeded that of the USA. On the contrary, the country with the most followers (in a few days, more than 1000) was Germany. Followed by the UK - which I expected would have been the first.

System Performance

The VPS held up well. Except for the moments when FediFollows launched (after fixing some FreeBSD tuning, the service slowed slightly but didn't crash), the load remained extremely low. So I continued to expand: Japan, Australia, New Zealand, etc.

Current Status

At the time of the last update of this article (21 May 2026), the supported countries are 42: Argentina, Australia, Austria, Belgium, Brazil, Bulgaria, Canada, Cyprus, Croatia, Czechia, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, India, Indonesia, Ireland, Italy, Japan, Latvia, Lithuania, Malta, Mexico, Netherlands, New Zealand, Norway, Poland, Portugal, Romania, Slovakia, Slovenia, South Africa, Spain, Sweden, Switzerland, Taiwan, Turkey, the United Kingdom, and the United States of America (with more regions coming soon!).

Direct followers in the Fediverse are around 8,889 and growing daily, excluding those who follow hashtags or cities via RSS, whose number I can't estimate. However, a quick look at the logs suggests there are many more.

The cities currently covered are 3602 - growing based on new countries and requests.

Challenges Encountered

There have been some problems. The most serious, by my fault, was the API key leak: I had left a debug code active and, the first time Open-Meteo had problems, the error message also included the API call - including the API key. Some users reported it to me (others just mocked) and I fixed the code and immediately reported everything to the Open-Meteo team, who kindly gave me a new API Key and deactivated the old one.

A further problem was related to geopy. It makes a call to Nominatim to determine coordinates. One of the times Nominatim didn't respond, my program wasn't able to determine the position and went into error. I solved this by introducing coordinate caching: now the program, the first time it encounters a city, requests and saves the coordinates. If present, they will be used in the future without making a new request via geopy. This is both lighter on their servers and faster and safer for us.

Infrastructure Details

And the VPS? It has no problems and is surprisingly fast and effective. FreeBSD 15.0-RELEASE, BastilleBSD to manage the jails. Currently, there are 43 jails - one for haproxy, the FediMeteo website, so nginx, and the snac instance for FediMeteo announcements and support - the other 41 for the individual instances. Each of them, therefore, has its autonomous ZFS dataset. Every 15 minutes, there is a local snapshot of all datasets. Every hour, the homepage is regenerated: a small script calculates the number of followers (counting, instance by instance, the followers of individual cities, since I don't publish except in aggregate to avoid possible triangulations and privacy leaks of users). Every hour, moreover, an external backup is made via zfs-autobackup (on encrypted at rest dataset), and once a day, a further backup is made in my datacenter, on disks encrypted with geli. The occupied RAM is 501 MB (yes, exactly: 501 MB), which rises slightly when updates are in progress. Updates normally occur every 6 hours. I have tried, as much as possible, to space them out to avoid overloads in timelines (or on the server itself). Only for the USA, I added a sleep of 5 seconds between one city and another, to give snac the opportunity to better organize the sending of messages. It probably wouldn't be necessary, with the current numbers, but better safe than sorry. In this way, the USA is processed in about 2 and a half hours, but the other jails (thus countries) can work autonomously and send their updates.

The average load of the VPS (taking as reference both the last 24 hours and the last two weeks) is about 25%, as it rises to 70/75% when updates occur for larger instances (such as the USA), or when it is announced by FediFollows. Otherwise, it is on average less than 10%. So, the VPS still has huge margin, and new instances, with new nations, will still be inside it.

Conclusion

This article, although in some parts very conversational, aims to demonstrate how it's possible to build solid, valid, and efficient solutions without the need to use expensive and complex services. Moreover, this is the demonstration of how it's possible to have your online presence without the need to put your data in the hands of third parties or without necessarily having to resort to complex stacks. Sometimes, less is more.

The success of this project demonstrates, once again, that my grandfather was right: weather forecasts interest everyone. He worried about my health and, thanks to his concerns, we spent time together. In the same way, I see many followers and friends talking to me or among themselves about the weather, their experiences, what happens. Again, in my life, weather forecasts have helped sociality and socialization.

Thank you, Grandpa.

Caching snac Proxied Media with Nginx

Stefano Marinelli — Sat, 08 Feb 2025 16:00:00 +0100

One of the useful yet resource-intensive features of platforms like Mastodon is that they reprocess (and store locally) all multimedia files from other instances.

This behavior is implemented for three valid reasons:

To ensure files do not contain malicious code by reprocessing them locally.
To prevent all users from all instances from overloading the original instance hosting the media by requesting it repeatedly.
To hide the individual IP addresses of users from the original instance.

While this approach has benefits, it also requires significant disk space - often many gigabytes per day.

snac, on the other hand, does not locally process or cache media by default. Instead, media URLs remain unchanged, meaning users will fetch content directly from its original source. This behavior is perfectly fine for many setups (such as mobile devices), but it may not be suitable for everyone.

I suggested a small improvement to snac’s developer, who immediately recognized the benefit for users and implemented an interesting new feature: media proxying via the instance itself.

Enabling Media Proxying in snac

To enable this feature, simply add the following line to your server.json configuration file:

"proxy_media": true

Once enabled, snac will rewrite all media URLs to pass through its own instance. This ensures that original instances will no longer see the IP addresses of individual users, as they will only see the IP of the snac instance itself.

However, this also increases the load and bandwidth consumption of the instance since it must download and forward media every time a user accesses it.

Caching Proxied Media with nginx

To optimize performance, we can configure nginx to cache these proxied files transparently. This way, if multiple users from the snac instance (or the same user at different times) request the same media file, nginx will serve it from the local cache instead of fetching it again. This setup has two key advantages:

Reduces traffic to the original instances.
Lowers the load on the snac instance, as it won’t need to download the same file repeatedly.

To achieve this, create a cache storage area in nginx by adding the following to the http section:

proxy_cache_path /var/cache/nginx/snac_media levels=1:2 keys_zone=snac_media:10m max_size=1g 
                 inactive=1d use_temp_path=off;

This defines a cache directory at /var/cache/nginx/snac_media, with 10 MB allocated for metadata and a maximum cache size of 1 GB. Cached content will be invalidated and removed after one day, ensuring frequently accessed content (like profile avatars) remains cached while rarely used files get replaced over time.

Next, add the following rule to your snac instance's virtual host configuration:

location ~ ^/.+/(x|y)/ {
    proxy_cache snac_media;
    proxy_pass http://snac-ip:8001;
    proxy_set_header Host $host;
    proxy_cache_valid 200 1d;
    proxy_cache_valid 404 1h;
    proxy_ignore_headers "Cache-Control" "Expires";
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    proxy_cache_lock on;
    add_header X-Proxy-Cache $upstream_cache_status;
}

After reloading nginx, the cache will start populating as users request media files. Unrequested content will never enter the cache, ensuring efficient storage usage without unnecessary clutter.

By combining snac’s new media proxying feature with nginx caching, we can achieve a more balanced setup—reducing load on both our instance and external ones, improving privacy, and enhancing performance.

Improving snac Performance with Nginx Proxy Cache

Stefano Marinelli — Wed, 29 Jan 2025 09:00:00 +0100

Some days ago, I migrated my personal Fediverse instance from Akkoma to snac. I appreciate snac a lot and believe it is the best solution available for many use cases.

Akkoma is an excellent tool, but I noticed that even for a small instance like mine, the database grows exponentially, and the database activity remains constant. Despite low load, my disks are continuously "flashing" - which isn't a problem in itself but clearly indicates ongoing activity. In my case, this activity seems unnecessary since it's just a single-user instance.

snac has shown excellent capabilities for managing the FediMeteo project (which I'll write about in detail soon), is lightweight, and has very few dependencies. Moreover, a dedicated snac instance handles sending updates from this blog to the Fediverse.

After successfully transferring my followers from Akkoma to snac without major issues, I started using the new instance. However, as soon as I posted a photo (approximately 4MB), something happened that I somewhat expected but in a different form. My home internet connection (upload speed: 20 Mbit/sec) became saturated, but I also noticed that new connections and smaller entities were resulting in 499 errors - meaning nginx couldn't open new connections to snac. After some investigation, I realized the reason: for every remote instance, Nginx was requesting the multimedia file from snac. Due to saturated connections (snac allows setting the maximum number of active threads), it took several seconds, leading to thread exhaustion in snac. Consequently, subsequent nginx requests resulted in 499 errors as snac could no longer allocate a thread.

To resolve this, I decided to implement direct caching using nginx. My reverse proxy (running in a different FreeBSD jail but this doesn't change the outcome) can cache multimedia files - storing them on first request and serving them directly to everyone who requests them without needing to ask snac every time. This approach is similar to what I use for the media in the BSD Cafe's Mastodon instance, where I employ Varnish to keep everything in RAM.

In snac, images and multimedia files are served from a specific path, such as: https://example.com/user/s/filename.png

The key here is the /s/ segment. By instructing nginx to cache all files containing /s/ in their URL, I can offload some of the work from snac.

I modified my nginx.conf file to include a caching setup:

# Caching configuration for snac
proxy_cache_path /var/cache/nginx/snac_cache levels=1:2 keys_zone=snac:10m max_size=1g inactive=1440m use_temp_path=off;

This creates a section for /var/cache/nginx where I define the caching parameters. It will allocate 10 MB of RAM for metadata, with a maximum cache size of 1 GB (useful if you decide to post some videos). Cached content will be considered invalid and removed after 1440 minutes (24 hours). This ensures that frequently accessed content (like profile avatars and banners) remains in the cache while less frequently accessed content can be replaced with newer content. The goal isn't to have all content cached but rather for nginx to serve files independently during peak times, preventing multiple remote instances from overwhelming snac simultaneously.

In the virtual host configuration for snac, I added this specific override for multimedia content:

# Caching rules for /s/ path
location ~ ^/.+/s/ {
    proxy_cache snac;
    proxy_pass http://snac-jail-ip:8001;
    proxy_set_header Host $host;
    proxy_cache_valid 200 1d;
    proxy_cache_valid 404 1h;
    proxy_ignore_headers "Cache-Control" "Expires";
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    proxy_cache_lock on;
    add_header X-Proxy-Cache $upstream_cache_status;
}

After restarting nginx, the multimedia files will be cached on their first access and served by nginx, leaving snac's threads free to handle everything else. This setup ensures smoother performance and prevents resource exhaustion during periods of high activity or when new content is shared.

Increasing or Modifying Character Limits and Poll Options in Mastodon 4.3, 4.4 and 4.5

Stefano Marinelli — Wed, 09 Oct 2024 14:53:00 +0200

Tested with Mastodon 4.5.x

Mastodon 4.3.0 was released just a few hours ago, and many instances have already started updating.

During my tests, I noticed that some things have changed in the core files, and the procedures normally described for changing the character limit of posts no longer work. So, I decided to write this short blog post to document the updated procedure.

Increasing or Modifying the Character Limit

At BSD Cafe, the character limit is set to 5000 (up from the 500 originally allowed by Mastodon). No one has ever used all the available characters, but I prefer that the limit be decided by the poster, not by the platform.

To customize this limit, you need to modify two files:

live/app/javascript/mastodon/features/compose/containers/compose_form_container.js - find the line that contains:

maxChars: state.getIn(['server', 'server', 'configuration', 'statuses', 'max_characters'], 500),

and change 500 to your desired value.

Next, you need to modify another file: live/app/validators/status_length_validator.rb - find the line that contains:

MAX_CHARS = 500

and change it accordingly.

Once done, run the following command:

RAILS_ENV=production bundle exec rails assets:precompile

and restart the Mastodon services to apply the changes.

Increasing or Modifying the Number of Poll Options

Edit the file live/app/validators/poll_options_validator.rb and modify the line:

MAX_OPTIONS = 4

to the value you want. There are also other customizable options in this file.

Again, once done, run the following command:

RAILS_ENV=production bundle exec rails assets:precompile

and restart the Mastodon services to apply the changes.

Changes might not be immediately visible because of caching (browser, web server, etc.), but once these caches expire, the new limits will take effect.

These modifications might be overridden by subsequent Mastodon updates, so be sure to check after every update that they are still valid.

Using a Permanent WebFinger Address for My Fediverse Profile

Stefano Marinelli — Tue, 08 Oct 2024 12:53:00 +0200

Decentralized technologies have always been of great interest to me. In my opinion, the decentralization of the Internet was one of the keys to its success, but year after year, we are gradually losing it. This is happening because we increasingly rely on "mainstream" services for convenience, habit, or trend.

I often hear colleagues say, "I've put my client's email on Y (Y = one of the major players in the email world) to avoid problems." The problems remain, but the colleague can "pass the buck" to the big player of the moment. And, in my experience, clients tend to be much more forgiving when the problem is with a big player.

Once, people used to say: "Nobody ever got fired for buying IBM." Today, I would say, "Nobody ever got fired for putting email on [Google, Microsoft, etc.]."

A few days ago, a colleague mentioned that he struggles to find some of his friends and colleagues on the fediverse because they are spread across various instances, and a simple search doesn't always yield the desired results.

Own Your Data

The fediverse is an extremely effective, resilient, autonomous decentralized communication technology. Unlike email, the many software implementations (such as Mastodon, GoToSocial, Mitra, Akkoma, Pleroma, etc. - soon snac will also support this type of operation) support the ability to "move" one's account. This means that when a user decides to relocate, the server will notify all followers that the user has moved. The followers will automatically stop following the original account and start following the new one. As of today, it is not possible to move past content (which will still be available on the old server).

In my opinion, every government, public entity, association, foundation, etc., that needs to communicate with the public should have its own communication channel, with full control over its data and the messages it delivers. When I read "my Discord server," I feel like responding "there's nothing 'yours' about it - tomorrow morning they could shut everything down, and you would have lost EVERYTHING.". Own your data!

Sometimes, I read that instances are not opened because of "costs not balanced by the number of users." But even public television channels are often economically unprofitable, yet they are considered an essential service for public communication. Open, decentralized technologies that ensure control over one's data should be treated the same way.

The problem, as with email, is tied to the address. A server change necessarily implies an address change, which can sometimes cause problems.

A Permanent Address

For this reason, I decided to implement a kind of "permanent" address that will provide the correct data based on my main server. I decided that this address should be linked to my email address, and since I control the entire domain (including the web server), I can do this quite simply. In this way, if I change servers, users who already follow me will be moved by the "move" operation, while those who do not yet follow me will still be able to find me using my "permanent" address, which will, in turn, provide the new user on the new server.

My setup will be very similar to what I found in an interesting blog post that I based my approach on.

In Practice

My email address is stefano@dragas.it, and I want anyone from any fediverse instance to be able to type it in and find me.
My main account is currently on BSD Cafe: https://mastodon.bsd.cafe/@stefano.

To achieve this, I modified the nginx configuration on the server that serves the "dragas.it" website.

Nginx Configuration

Before the server directive for dragas.it, I added this mapping:

map $query_string $account_name {
    ~resource=acct:stefano@dragas.it$ stefano;
    ~resource=acct%3Astefano%40dragas.it$ stefano;
}

This ensures that requests for those two resources (some clients send the second one, so I wanted to handle that situation) will assign the value "stefano" to the $account_name variable.

WebFinger Exception

Within the server directive of the dragas.it virtual host, I added an exception for webfinger (the protocol used to find user data):

location = /.well-known/webfinger {
    root  /usr/local/www/webfinger;

    if ($account_name) {
      rewrite ^(.*)$ /$account_name.json break;
    }

    try_files $uri = 404;
}

JSON File for WebFinger

At this point, I created the /usr/local/www/webfinger directory and added a file named stefano.json (matching $account_name) with the following content:

{
  "subject": "acct:stefano@dragas.it",
  "aliases": [
    "https://mastodon.bsd.cafe/@stefano",
    "https://mastodon.bsd.cafe/users/stefano"
  ],
  "links": [
    {
      "rel": "http://webfinger.net/rel/profile-page",
      "type": "text/html",
      "href": "https://mastodon.bsd.cafe/@stefano"
    },
    {
      "rel": "self",
      "type": "application/activity+json",
      "href": "https://mastodon.bsd.cafe/users/stefano"
    },
    {
      "rel": "http://ostatus.org/schema/1.0/subscribe",
      "template": "https://mastodon.bsd.cafe/authorize_interaction?uri={uri}"
    }
  ]
}

In practice, this will return a reference to my main Mastodon account.

Testing the Setup

After reloading the nginx configurations, it will be enough to test it:

# curl "https://dragas.it/.well-known/webfinger?resource=acct:stefano@dragas.it"

{
  "subject": "acct:stefano@dragas.it",
  "aliases": [
    "https://mastodon.bsd.cafe/@stefano",
    "https://mastodon.bsd.cafe/users/stefano"
  ],
  "links": [
    {
      "rel": "http://webfinger.net/rel/profile-page",
      "type": "text/html",
      "href": "https://mastodon.bsd.cafe/@stefano"
    },
    {
      "rel": "self",
      "type": "application/activity+json",
      "href": "https://mastodon.bsd.cafe/users/stefano"
    },
    {
      "rel": "http://ostatus.org/schema/1.0/subscribe",
      "template": "https://mastodon.bsd.cafe/authorize_interaction?uri={uri}"
    }
  ]
}

In this case, everything works perfectly. When you search for my email address from any fediverse instance, my BSD Cafe profile will appear.

Conclusion

This approach allows you to define multiple users for each domain. You just need to add the corresponding mapping in the nginx configuration and the .json file with all the data.

A Small Compendium of Fediverse Platforms I Use

Stefano Marinelli — Thu, 12 Sep 2024 18:45:00 +0200

Following my old article about the Fediverse software I've experimented and use, I've decided to convert a Mastodon post into this small blog post.

In the past few days, I revisited several of my old Fediverse instances after some friends asked me to help them set up a new one. While I was at it, I took the opportunity to perform maintenance on some leftover instances I still manage. Here’s a summary of my experience with various platforms:

Akkoma

This is my oldest instance, still running since 2022. It was offline for about 3 or 4 months, but I recently updated it to the latest version and restarted it. After upgrading the software and the database, it didn't show any problem. Akkoma is a very good solution, supports quote posts and emoji reactions.

GoToSocial

I helped a friend update their GoToSocial instance. While the software itself was up-to-date, the underlying system needed an update. I noticed that when the number of followings exceeds 2000, the instance becomes a bit sluggish. PostgreSQL isn’t the issue in this case – it's the GoToSocial process itself that seems to get heavy on the VPS. Despite this, GoToSocial remains very usable, and I see a lot of potential in it. The Mastodon API is well-implemented, and it works seamlessly with major apps.

Mitra

Mitra is another Fediverse platform I’ve been exploring. I helped someone with around 1000 followers and followings migrate from a large Mastodon instance to Mitra. There were no speed issues, though sending messages does make the server slightly "heavier" for a short time. The Mastodon API is partially implemented, but the software is evolving quickly. I find its native interface quite user-friendly, and it’s a platform worth keeping an eye on.

Snac2

I’ve always had a soft spot for Snac2. It doesn’t use a database, and its design choices make it ideal for small instances. One feature I particularly like is how it sends posts to all known instances, which increases visibility and interaction. The interface is basic, with no JavaScript, which is a nice change, but it might feel too minimalistic for users coming from Mastodon. However, the Mastodon API support is steadily improving with each release. Snac2 does struggle with larger numbers, but this is more due to the underlying file system than the software itself. Snac2 now supports moving in/out, so it's easy top test it. I highly recommend it for anyone looking to self-host a small or single-user instance.

Mastodon

My old personal instance of Mastodon was stuck on version 4.1.x and had been offline for a few months. I updated the FreeBSD Jail, upgraded Mastodon to 4.2.12, and then to 4.3.0 without any issues. I also helped a friend migrate their Pleroma-based instance to Mastodon. This user has about 5000 followers and followings, and the instance runs on FreeBSD on an arm64 VPS for around 3 euros per month. Aside from media storage (which isn’t Mastodon’s fault), there were no significant issues. Although Mastodon is sometimes criticized for being resource-intensive, its modular design ensures that even during high load, queues might slow down, but the local timeline and navigation remain reasonably fast. This makes it a strong contender for larger-scale use.

Conclusion

Overall, I feel that these platforms are evolving in the right direction. The developers are doing a fantastic job, and the Fediverse is growing stronger with each new release. Well done to all the devs working hard behind the scenes!

Make Your Own CDN with NetBSD

Stefano Marinelli — Tue, 03 Sep 2024 01:41:00 +0200

Introduction

This article is a spin-off from a previous post on how to create a self-hosted CDN, based on OpenBSD, but this time we'll focus on using NetBSD. The idea is to create reverse proxies with local caching. These proxies would cache the content on the first request and serve it directly afterward. The proxies would be distributed across different regions, and the DNS would route requests to the nearest proxy based on the caller’s location. All this is achieved without relying on external CDNs, using self-managed tools instead.

NetBSD is a lightweight, stable, and secure operating system that supports a wide range of hardware, making it an excellent choice for a caching reverse proxy. Devices that other operating systems may soon abandon, such as early Raspberry Pi models or i386 architecture, are still fully supported by NetBSD and will continue to be so. Additionally, NetBSD is an outstanding platform for virtualization (using Xen or qemu/nvmm) and deserves more attention than it currently receives.

The choice of Varnish is based on several factors, with the main ones being the ability to keep the cache in RAM (which means it can run on read-only systems) and the ability to flush the cache remotely. For example, with each change to my blog, I can choose whether to perform an immediate flush (such as for a new article or an error) or wait for the cache's "natural" expiration (such as for a typo or minor, non-critical changes).

While I won't detail the installation process for NetBSD, as it depends heavily on the hardware you have available, I will guide you through setting up a self-hosted CDN using NetBSD, Varnish, nginx, and the acme.sh or lego tool for SSL certificate management.

Installation

During the installation of NetBSD, ensure that you enable support for binary package management. This will install pkgin, a tool that simplifies package management on NetBSD. If you skip this step during installation, you can still install pkgin later, but it's easier to let the installer handle it.

Once your system is up and running, use pkgin to install the necessary packages: Varnish, nginx, and, depending on your preference, either acme.sh or Go (if you plan to compile lego). Although lego is not available as a precompiled package, you can easily compile it locally using Go, but for simplicity, I recommend using acme.sh.

For this setup, I will present two methods for generating and renewing certificates: using acme.sh or compiling lego - to reach the same final outcome as the OpenBSD article.

Option 1: Using acme.sh (Recommended)

acme.sh is a simple, yet powerful, shell script that handles certificate generation and renewal with ease. To install acme.sh:

pkgin in acmesh varnish nginx

Once acme.sh is installed, you can proceed to configure it for certificate management. It supports DNS authentication and integrates with many DNS providers, making it a flexible choice.

Option 2: Compiling Lego

If you prefer to use lego, you will need to compile it manually, as it is not available as a precompiled package for NetBSD. First, install Go and other necessary packages:

pkgin in go varnish nginx

To compile lego, you’ll need some disk space. Since the /tmp directory in NetBSD is often mounted as tmpfs (using RAM), you may run out of space during compilation if your system has limited memory. You can temporarily disable tmpfs by editing /etc/fstab and commenting out the relevant line:

#tmpfs           /tmp    tmpfs   rw,-m=1777,-s=ram%25

After rebooting, compile lego:

export GO111MODULE=on
go122 install github.com/go-acme/lego/v4/cmd/lego@latest

cp go/bin/lego /usr/pkg/bin/

Once lego is compiled and installed, you can uncomment the /tmp line in /etc/fstab and reboot again.

Configuring Varnish and nginx

First, copy the necessary rc.d scripts for nginx and Varnish:

cp /usr/pkg/share/examples/rc.d/nginx /etc/rc.d/
cp /usr/pkg/share/examples/rc.d/varnishd /etc/rc.d/

Then, add the following to /etc/rc.conf to enable and configure nginx and Varnish:

nginx=YES
varnishd=YES
varnishd_flags="-f /usr/pkg/etc/varnish/default.vcl -T localhost:9999 -a "/var/run/varnish.sock",user=nginx,group=varnish,mode=660 -s default,500m"

In this configuration, Varnish listens on a Unix socket, and nginx connects to it. This approach is more efficient and helps avoid some issues that may arise when exposing Varnish over an IP/port.

Creating the Varnish VCL Configuration

Next, create the VCL configuration file for Varnish at /usr/pkg/etc/varnish/default.vcl:

vcl 4.1;
import std;

# Backend - it-notes.dragas.net
backend it_notes {
    .host = "myBackendIP";
    .port = "80";
}

# ACL - purge - it-notes.dragas.net
acl purge_it_notes {
    "allowedToPurge_IP";
}

sub vcl_recv {
    # it-notes.dragas.net
    if (req.http.Host == "it-notes.dragas.net") {
        set req.backend_hint = it_notes;
        set req.http.Host = "it-notes.dragas.net";

        # PURGE - it-notes.dragas.net
        if (req.method == "PURGE") {
            std.log("Purge request received for " + req.url);

            if (!std.ip(req.http.X-Forwarded-For, "0.0.0.0") ~ purge_it_notes) {
                return (synth(405, "Not allowed."));
            }

            if (req.url == "/" || req.url == "/*") {
                ban("req.http.host == " + req.http.host);
                return(synth(200, "Entire cache has been cleared."));
            }
            return (purge);
        }

    } else {
        # Other domains - 404
        return (synth(404, "Domain not found"));
    }

    if (req.method != "GET" && req.method != "HEAD") {
        return (pipe);
    }

    return (hash);
}

sub vcl_backend_response {
    # TTL - it-notes.dragas.net
    if (bereq.http.host == "it-notes.dragas.net") {
        if (bereq.url ~ "\.(gif|jpg|jpeg|png|webp|ico|css|js)$") {
            set beresp.ttl = 1w;
            set beresp.grace = 1d;
            set beresp.keep = 7d;
            unset beresp.http.Set-Cookie;
            unset beresp.http.Cache-Control;
            set beresp.http.Cache-Control = "public, max-age=604800";
        } else {
            set beresp.ttl = 15m;
            set beresp.grace = 48h;
            set beresp.keep = 7d;
        }
    }

    # Remove some headers
    unset beresp.http.Server;
    unset beresp.http.X-Powered-By;
    unset beresp.http.Via;

    return (deliver);
}

sub vcl_deliver {
    # Add X-Cache header
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
    } else {
        set resp.http.X-Cache = "MISS";
    }

    std.log("Delivering content for " + req.url + " - Cache: " + resp.http.X-Cache);

    # Remove Varnish headers
    unset resp.http.Via;
    unset resp.http.X-Varnish;

    return (deliver);
}

sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }
    return (lookup);
}

sub vcl_hit {
    return (deliver);
}

sub vcl_miss {
    return (fetch);
}

sub vcl_purge {
    std.log("Purge executed for " + req.url);
    return (synth(200, "Purge successful"));
}

sub vcl_synth {
    set resp.http.Content-Type = "text/html; charset=utf-8";
    set resp.http.Retry-After = "5";
    synthetic({"<!DOCTYPE html>
        <html>
            <head>
                <title>"} + resp.status + " " + resp.reason + {"</title>
            </head>
            <body>
                <h1>Status "} + resp.status + " " + resp.reason + {"</h1>
                <p>"} + resp.reason + {"</p>
                <h3>Guru Meditation:</h3>
                <p>XID: "} + req.xid + {"</p>
                <hr>
                <p>Varnish cache server</p>
            </body>
        </html>
    "});
    return (deliver);
}

Configuring nginx

Now, modify the nginx configuration file at /usr/pkg/etc/nginx/nginx.conf. Set the number of worker processes to "auto" to take advantage of all server cores, and configure the reverse proxy for your site(s). Here's an example configuration:

[...]
worker_processes  auto;
[...]

server {
    server_name it-notes.dragas.net;

    location / {
        proxy_method $request_method;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_pass http://unix:/var/run/varnish.sock;
    }

    access_log /var/log/nginx/access.it-notes.dragas.net.log;
    error_log /var/log/nginx/error.it-notes.dragas.net.log;

    listen [::]:443 ssl;
    listen 443 ssl;
    http2 on;
    # If you're using acme.sh, just change the location of the certificates
    ssl_certificate /root/.lego/certificates/it-notes.dragas.net.crt;
    ssl_certificate_key /root/.lego/certificates/it-notes.dragas.net.key;
}

server {
    if ($host = it-notes.dragas.net) {
        return 301 https://$host$request_uri;
    }
    server_name it-notes.dragas.net;
    listen 80;
    listen [::]:80;
    return 404;
}

Starting Varnish and nginx

Finally, start the Varnish and nginx services:

service varnishd start
service nginx start

If everything is configured correctly, both Varnish and nginx will be up and running, ready to handle incoming connections.

Conclusion

Congratulations, you have successfully set up your own CDN on NetBSD. This solution is lightweight, stable, and fully under your control, allowing you to break free from the constraints of major service providers. With NetBSD's broad hardware support and minimal overhead, this setup can run on a wide variety of devices, making it a versatile choice for self-hosted solutions.

I'm using it as a test and as a read-only root filesystem with a RAM-only local cache for my blog on a Raspberry Pi Zero W (first edition), and as soon as I get the new FTTH, I'll probably make it accessible via IPv6 for Italy, putting it physically into production.

If your goal is geo-replication, you can use DNS providers that offer location-based routing or set up your own DNS infrastructure to manage and resolve requests according to the user’s location. With multiple reverse proxies, separate DNS servers, and a well-configured cache, you can achieve a highly resilient system with minimal risk of a single point of failure.

Make Your Own CDN with OpenBSD Base and Just 2 Packages

Stefano Marinelli — Thu, 29 Aug 2024 01:41:00 +0200

Introduction

This article is a "spin-off" from the previous post "Building a Self-Hosted CDN for BSD Cafe Media," which I recommend reading, based on FreeBSD. In that article, I showed how I addressed the issue of geo-replication and geo-distribution of BSD Cafe media content. If you prefer a NetBSD-based setup, there's an article that describes how to do it.

The internet today relies TOO MUCH on just a few big players. When one of them stops working, half the world is impacted because too many services, in my opinion, depend on them. “Too big to fail,” some might say. “Single Point of Failure,” I respond."

The strength of the internet has always been its extreme decentralization, which is now less evident due to this phenomenon.

In this article, I want to show how easy it is to create a self-hosted CDN using OpenBSD and just two external packages: Varnish and Lego.

Actually, only one package is truly needed (Varnish), and SSL certificates could be generated using the built-in acme-client in OpenBSD. However, this might be limiting since acme-client handles certificate generation via traditional methods (using a file in .well-known), but when dealing with a CDN and several reverse proxies listening, you don’t have perfect control over which one will receive the certificate generation validation request.

For convenience and practicality, I’ll use the excellent Lego tool—a Go application that supports many DNS authentication methods, including PowerDNS.

Installation

The steps are quite simple. After installing and updating OpenBSD (using the syspatch command), start by installing the two packages:

obcdn# pkg_add lego varnish
quirks-7.14:updatedb-0p0: ok
quirks-7.14 signed on 2024-08-24T13:35:23Z
quirks-7.14: ok
lego-4.16.1: ok
varnish-7.4.2:bzip2-1.0.8p0: ok
varnish-7.4.2:pcre2-10.37p2: ok
useradd: Warning: home directory `/var/varnish' doesn't exist, and -m was not specified
varnish-7.4.2: ok
The following new rcscripts were installed: /etc/rc.d/varnishd
See rcctl(8) for details.

The next step is to enable Varnish:

rcctl enable varnishd

Varnish has a generic startup script, but it's best to customize the startup options. To do this, modify the /etc/rc.conf.local file:

varnishd_flags="-j unix,user=_varnish,ccgroup=_varnish -f /etc/varnish/default.vcl -T localhost:9999 -a localhost:8080 -s default,500m"

This configuration sets Varnish with a 500 MB cache and listens on localhost, port 8080.

Next, rename the default VCL file to prepare for your own content:

mv /etc/varnish/default.vcl /etc/varnish/default.vcl.distrib

Create a new default.vcl file. Below is an example based on this blog (at the time of writing). You’ll need to adapt it according to your needs, especially if there are cookies or other dynamic content. Note that Varnish will fetch data from a specific backend accessed in http. If privacy is needed, consider creating a VPN between the backend and Varnish, as briefly mentioned in the previous article:

vcl 4.1;
import std;

# Backend - it-notes.dragas.net
backend it_notes {
    .host = "myOriginalServer";
    .port = "80";
}

# ACL - purge - it-notes.dragas.net
acl purge_it_notes {
    "authorizedIPForCachePurge";
}

sub vcl_recv {
    # it-notes.dragas.net
    if (req.http.Host == "it-notes.dragas.net") {
        set req.backend_hint = it_notes;
        set req.http.Host = "it-notes.dragas.net";

        # PURGE - it-notes.dragas.net
        if (req.method == "PURGE") {
            std.log("Purge request received for " + req.url);

            if (!std.ip(req.http.X-Forwarded-For, "0.0.0.0") ~ purge_it_notes) {
                return (synth(405, "Not allowed."));
            }

            if (req.url == "/" || req.url == "/*") {
                ban("req.http.host == " + req.http.host);
                return(synth(200, "Entire cache has been cleared."));
            }
            return (purge);
        }

    } else {
        # Other domains - 404
        return (synth(404, "Domain not found"));
    }

    if (req.method != "GET" && req.method != "HEAD") {
        return (pipe);
    }

    return (hash);
}

sub vcl_backend_response {
    # TTL - it-notes.dragas.net
    if (bereq.http.host == "it-notes.dragas.net") {
        if (bereq.url ~ "\.(gif|jpg|jpeg|png|webp|ico|css|js)$") {
            set beresp.ttl = 1w;
            set beresp.grace = 1d;
            set beresp.keep = 7d;
            unset beresp.http.Set-Cookie;
            unset beresp.http.Cache-Control;
            set beresp.http.Cache-Control = "public, max-age=604800";
        } else {
            set beresp.ttl = 15m;
            set beresp.grace = 48h;
            set beresp.keep = 7d;
        }
    }

    # Remove some headers
    unset beresp.http.Server;
    unset beresp.http.X-Powered-By;
    unset beresp.http.Via;

    return (deliver);
}

sub vcl_deliver {
    # Add X-Cache header
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
    } else {
        set resp.http.X-Cache = "MISS";
    }

    std.log("Delivering content for " + req.url + " - Cache: " + resp.http.X-Cache);

    # Remove Varnish headers
    unset resp.http.Via;
    unset resp.http.X-Varnish;

    return (deliver);
}

sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }
    return (lookup);
}

sub vcl_hit {
    return (deliver);
}

sub vcl_miss {
    return (fetch);
}

sub vcl_purge {
    std.log("Purge executed for " + req.url);
    return (synth(200, "Purge successful"));
}

sub vcl_synth {
    set resp.http.Content-Type = "text/html; charset=utf-8";
    set resp.http.Retry-After = "5";
    synthetic({"<!DOCTYPE html>
        <html>
            <head>
                <title>"} + resp.status + " " + resp.reason + {"</title>
            </head>
            <body>
                <h1>Status "} + resp.status + " " + resp.reason + {"</h1>
                <p>"} + resp.reason + {"</p>
                <h3>Guru Meditation:</h3>
                <p>XID: "} + req.xid + {"</p>
                <hr>
                <p>Varnish cache server</p>
            </body>
        </html>
    "});
    return (deliver);
}

Start Varnish and check if it starts correctly:

rcctl start varnishd

Generating SSL Certificates

Before configuring relayd, you’ll need to generate SSL certificates. Lego supports many DNS providers and provides clear and comprehensive examples, so I suggest reading its README file.

Important: Certificates generated by Lego are not directly compatible with relayd, so you must generate them in the correct format. Add the -k rsa4096 flag to the Lego command to obtain certificates compatible with relayd.

Once generated, the certificates will be in a subdirectory of the directory from which the command was launched. For example, if the command is run as root (which is unnecessary, but just for the example), the certificates will be in /root/.lego/certificates/.

relayd expects certificates in a specific location. Copy them to the appropriate directories. In my example:

obcdn# cp /root/.lego/certificates/it-notes.dragas.net.crt /etc/ssl/
obcdn# cp /root/.lego/certificates/it-notes.dragas.net.key /etc/ssl/private/

Remember to copy the files to the correct directories when renewing. You can also create symbolic links, this will make your life easier but it’s less secure.

Configuring relayd

It’s time to configure relayd. The file is /etc/relayd.conf, and here’s an example configuration:

log state changes
prefork 10

table <itnotes> { 127.0.0.1 }

http protocol "http" {
    match request header append "X-Forwarded-For" value "$REMOTE_ADDR"
    match request header append "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"

    match request path "/*.atom" tag "CACHE"
    match request path "/*.css" tag "CACHE"
    match request path "/*.gif" tag "CACHE"
    match request path "/*.html" tag "CACHE"
    match request path "/*.ico" tag "CACHE"
    match request path "/*.jpg" tag "CACHE"
    match request path "/*.webp" tag "CACHE"
    match request path "/*.js" tag "CACHE"
    match request path "/*.png" tag "CACHE"
    match request path "/*.rss" tag "CACHE"
    match request path "/*.svg" tag "CACHE"
    match request path "/*.xml" tag "CACHE"
    match request path "/*.ttf" tag "CACHE"
    match request path "/*.woff2" tag "CACHE"

    match response tagged "CACHE" header set "Cache-Control" value "public, max-age=604800"

    pass request header "Host" value "it-notes.dragas.net" forward to <itnotes>
}

http protocol "https" {
    match request header append "X-Forwarded-For" value "$REMOTE_ADDR"
    match request header append "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"

    match response header set "Referrer-Policy" value "no-referrer"
    match response header set "X-Content-Type-Options" value "nosniff"
    match response header set "X-Download-Options" value "noopen"
    match response header set "X-Frame-Options" value "SAMEORIGIN"
    match response header set "X-Permitted-Cross-Domain-Policies" value "none"
    match response header set "X-XSS-Protection" value "1; mode=block"
    match response header set "Strict-Transport-Security" value "max-age=15552000; includeSubDomains; preload"

    match request path "/*.atom" tag "CACHE"
    match request path "/*.css" tag "CACHE"
    match request path "/*.gif" tag "CACHE"
    match request path "/*.html" tag "CACHE"
    match request path "/*.ico" tag "CACHE"
    match request path "/*.jpg" tag "CACHE"
    match request path "/*.webp" tag "CACHE"
    match request path "/*.js" tag "CACHE"
    match request path "/*.png" tag "CACHE"
    match request path "/*.rss" tag "CACHE"
    match request path "/*.svg" tag "CACHE"
    match request path "/*.xml" tag "CACHE"
    match request path "/*.ttf" tag "CACHE"
    match request path "/*.woff2" tag "CACHE"

    match response tagged "CACHE" header set "Cache-Control" value "public, max-age=604800"
    tcp { nodelay, sack, socket buffer 65536, backlog 100 }

    tls { keypair "it-notes.dragas.net" }

    pass request header "Host" value "it-notes.dragas.net" forward to <itnotes>
}

relay "http" {
    listen on vio0 port 80
    protocol "http"

    forward to <itnotes> port 8080
}

relay "https" {
    listen on vio0 port 443 tls
    protocol "https"

    forward to <itnotes> port 8080
}

relay "https6" {
    listen on my:ip:v6:address::1 port 443 tls
    protocol "https"

    forward to <itnotes> port 8080
}

The content of the file is quite self-explanatory. Note that:

vio0 is the interface name—it should be modified based on the interface where relayd needs to listen.
I’ve configured relayd to listen on port 80 as well.
IPv4 and IPv6 listeners are separated. If IPv6 is not configured, simply comment out that part. Please, if you can, use IPv6. In a fairer world, everyone would have the right to at least one class of public addresses without having to pay exorbitant fees.
The "keypair" must correspond to the certificate and key names in /etc/ssl and /etc/ssl/private.

Test the configuration, enable, and start relayd:

obcdn# relayd -n
configuration OK
obcdn# rcctl enable relayd
obcdn# rcctl start relayd
relayd(ok)

Final Checks

The stack is ready. A ps command will show the process status:

obcdn# ps aux
USER       PID %CPU %MEM   VSZ   RSS TT  STAT   STARTED       TIME COMMAND
root         1  0.0  0.0   920   276 ??  I       7:10PM    0:00.01 /sbin/init
[...]
_varnish 44999  0.0  0.2  2256  2424 ??  S       8:27PM    0:00.05 varnishd: Varnish-Mgt -i obcdn.my.domain (varnishd)
_varnish 54481  0.0  8.8 34500 88876 ??  S       8:27PM    0:00.60 varnishd: Varnish-Child -i obcdn.my.domain (varnishd)
root     57753  0.0  0.5  4080  4820 ??  IU      8:32PM    0:00.03 /usr/sbin/relayd
_relayd  95940  0.0  0.4  2152  3892 ??  Spc     8:32PM    0:00.01 relayd: pfe (relayd)
_relayd  81032  0.0  0.4  2156  3716 ??  Spc     8:32PM    0:00.01 relayd: hce (relayd)
_relayd  80312  0.0  0.5  2748  5280 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  88919  0.0  0.5  2748  5268 ??  Ipc     8:32PM    0:00.04 relayd: relay (relayd)
_relayd  90186  0.0  0.5  2752  5272 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  32851  0.0  0.5  2736  5284 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  20270  0.0  0.5  2744  5252 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  98826  0.0  0.5  2752  5264 ??  Ipc     8:32PM    0:00.04 relayd: relay (relayd)
_relayd   6454  0.0  0.5  2744  5264 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  90102  0.0  0.5  2740  5276 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  60314  0.0  0.5  2748  5336 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  27246  0.0  0.5  2744  5284 ??  Ipc     8:32PM    0:00.03 relayd: relay (relayd)
_relayd  87082  0.0  0.4  2100  4508 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  53308  0.0  0.4  2096  4496 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  11697  0.0  0.4  2092  4492 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  83636  0.0  0.4  2092  4500 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  74563  0.0  0.4  2096  4484 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  35910  0.0  0.4  2100  4508 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  24911  0.0  0.4  2096  4504 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  78649  0.0  0.4  2096  4496 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  89586  0.0  0.4  2096  4500 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
_relayd  11989  0.0  0.4  2100  4500 ??  Ipc     8:32PM    0:00.02 relayd: ca (relayd)
[...]

In this case, both relayd and Varnish are running correctly.

Automating Certificate Renewal

As a final step, remember to create a script to renew the certificates, copy them to /etc/ssl and /etc/ssl/private, and restart relayd.

Conclusion

Congratulations, you now have your own free CDN, with your data, fully under your control. Portable, extendable, controllable, and outside of the big providers' grip.

If your goal is geo-replication, you should use one of the available methods. Some DNS providers allow selection based on the caller's location (like Bunny.net), or, as I prefer, install your own DNS and use tools to manage operations and resolutions, as briefly described in the previous article.

With multiple separate reverse proxies, separate DNS servers (on different providers and possibly different countries or continents) capable of checking if the reverse proxies are operational, you can achieve an extremely low likelihood of encountering a Single Point of Failure, as all components, once the cache is filled, will be nearly autonomous even in the event of a (temporary) backend outage - i.e., the original node.

Building a Self-Hosted CDN for BSD Cafe Media

Stefano Marinelli — Mon, 26 Aug 2024 08:41:00 +0200

Introduction

For just over a year, BSD Cafe's media was hosted on a FreeBSD physical server jail with an outgoing bandwidth of 250 Mbit/sec. To mitigate bandwidth congestion, I initially integrated Cloudflare with a tunnel to serve media (and only media) through Cloudflare. The goal was to georeplicate the media and reduce the load on my server. To do this, the media server had to be on a separate domain managed by Cloudflare since the DNS for the primary bsd.cafe domain was managed by Bunny.net.

The first step (mainly because I discovered that Bunny's DNS does not support IPv6) was to bring the DNS back in-house using two FreeBSD jails (running on different VPS providers), both powered by PowerDNS. PowerDNS supports LUA records, which would come in handy later.

In line with the principles of self-hosting and data ownership, I decided to remove Cloudflare. I created a dedicated subdomain (media.bsd.cafe) and configured the reverse proxy in front of the jail running Minio to respond to that domain. I also reconfigured Mastodon for the new address, and after some fine-tuning, everything worked seamlessly. However, this led to some bandwidth congestion when media was posted, resulting in slower download speeds for users, especially during peak times. This is because, once content is published and federated servers are notified, they all attempt to download the newly published content - media included - almost simultaneously.

Not wanting to abandon my media server (a dedicated jail with spinning disks, offering 4 TB of storage), I opted for a different approach that I’ll describe here, as it might be useful for similar setups.

While this setup was implemented on FreeBSD, the configuration and tools - Nginx, Varnish, WireGuard and PowerDNS - are compatible with many operating systems, including Linux, with only minor adjustments required.

The Approach: Building a Self-Hosted CDN

The idea is to create reverse proxies with local caching. These proxies would cache the content on the first request and serve it directly afterward. The proxies would be distributed across different regions, and the DNS would route requests to the nearest proxy based on the caller’s location. All this is achieved without relying on external CDNs, using self-managed tools instead.

To establish a direct connection between Minio and the reverse proxies, I configured WireGuard inside the jail. The reverse proxies connect via WireGuard, allowing them to access Minio securely as if they were on the same LAN.

No further changes were needed on the media jail itself.

Setting Up the Reverse Proxies

I began configuring the reverse proxies (also running FreeBSD jails, OpenBSD (setup described in another post) and NetBSD (also described in another post), hosted on different providers). The choice of Varnish is based on several factors, with the main ones being the ability to keep the cache in RAM (which means it can run on read-only systems) and the ability to flush the cache remotely. For example, with each change to my blog, I can choose whether to perform an immediate flush (such as for a new article or an error) or wait for the cache's "natural" expiration (such as for a typo or minor, non-critical changes).

First, I connected them via WireGuard to the Minio jail (I won’t detail the steps here; I’ve covered similar setups in other posts). Then, I installed Nginx and Varnish. A more granular setup would have Varnish on a separate jail, but this way, I can move the reverse proxy jails to different hosts with minimal hassle. Currently, these reverse proxies also serve this blog.

Next, I installed and configured Varnish inside the jail:

pkg install varnish7

I created the directory /usr/local/etc/varnish and wrote a custom VCL file to manage this setup, named default.vcl:

vcl 4.1;
import std;

# Backend - it-notes.dragas.net
backend it_notes {
    .host = "itnotesip";
    .port = "itnotesport";
}

# Backend - media.bsd.cafe
backend media_bsd {
    .host = "minioWGip";
    .port = "minioport";
}

# ACL - IPs allowed to purge - it-notes.dragas.net
acl purge_it_notes {
    "a.b.c.d";
}

# ACL - IPs allowed to purge - media.bsd.cafe
acl purge_media_bsd {
    "e.f.g.h";
}

sub vcl_recv {

    # it-notes.dragas.net
    if (req.http.Host == "it-notes.dragas.net") {
        set req.backend_hint = it_notes;
        set req.http.Host = "it-notes.dragas.net";

        # PURGE - it-notes.dragas.net
        if (req.method == "PURGE") {

            std.log("Purge request received for " + req.url);

            if (!std.ip(req.http.X-Real-IP, "0.0.0.0") ~ purge_it_notes) {
                return (synth(405, "Not allowed."));
            }

        if (req.url == "/" || req.url == "/*") {
                ban("req.http.host == " + req.http.host);
                return(synth(200, "Entire cache has been cleared."));
        }
            return (purge);
        }

    # media.bsd.cafe
    } elsif (req.http.Host == "media.bsd.cafe") {
        set req.backend_hint = media_bsd;
        set req.http.Host = "media.bsd.cafe";

        # PURGE - media.bsd.cafe
        if (req.method == "PURGE") {
            if (!std.ip(req.http.X-Real-IP, "0.0.0.0") ~ purge_media_bsd) {
                return (synth(405, "Not allowed."));
            }
            if (req.url == "/" || req.url == "/*") {
                ban("req.http.host == " + req.http.host);
                return(synth(200, "Entire cache has been cleared."));
            }
            return (purge);
        }

    } else {
        # Other domains - 404
        return (synth(404, "Domain not found"));
    }

    if (req.method != "GET" && req.method != "HEAD") {
        return (pipe);
    }

    return (hash);
}

sub vcl_backend_response {
    # TTL - it-notes.dragas.net
    if (bereq.http.host == "it-notes.dragas.net") {
        if (bereq.url ~ "\.(gif|jpg|jpeg|png|ico|css|js)$") {
            set beresp.ttl = 1w;
            set beresp.grace = 1d;
            set beresp.keep = 7d;
            unset beresp.http.Set-Cookie;
            unset beresp.http.Cache-Control;
            set beresp.http.Cache-Control = "public, max-age=604800";
        } else {
            set beresp.ttl = 15m;
            set beresp.grace = 48h;
            set beresp.keep = 7d;
        }

    # TTL - media.bsd.cafe
    } elsif (bereq.http.host == "media.bsd.cafe") {
        if (bereq.url ~ "\.(mp4|mp3|wav|flac|ogg)$") {
            set beresp.ttl = 1d;
            set beresp.grace = 6h;
            set beresp.keep = 3d;
            unset beresp.http.Set-Cookie;
            unset beresp.http.Cache-Control;
            set beresp.http.Cache-Control = "public, max-age=86400";
        } else {
            set beresp.ttl = 30m;
            set beresp.grace = 12h;
            set beresp.keep = 3d;
        }
    }

    # Remove some headers
    unset beresp.http.Server;
    unset beresp.http.X-Powered-By;
    unset beresp.http.Via;

    return (deliver);
}

sub vcl_deliver {
    # ADD header X-Cache
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
    } else {
        set resp.http.X-Cache = "MISS";
    }

  std.log("Delivering content for " + req.url + " - Cache: " + resp.http.X-Cache);


    # Remove Varnish headers
    unset resp.http.Via;
    unset resp.http.X-Varnish;

    return (deliver);
}

sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }
    return (lookup);
}

sub vcl_hit {
    return (deliver);
}

sub vcl_miss {
    return (fetch);
}

sub vcl_purge {
    std.log("Purge executed for " + req.url);
    return (synth(200, "Purge successful"));
}

sub vcl_synth {
    set resp.http.Content-Type = "text/html; charset=utf-8";
    set resp.http.Retry-After = "5";
    synthetic( {"<!DOCTYPE html>
        <html>
            <head>
                <title>"} + resp.status + " " + resp.reason + {"</title>
            </head>
            <body>
                <h1>Status "} + resp.status + " " + resp.reason + {"</h1>
                <p>"} + resp.reason + {"</p>
                <h3>Guru Meditation:</h3>
                <p>XID: "} + req.xid + {"</p>
                <hr>
                <p>Varnish cache server</p>
            </body>
        </html>
    "} );
    return (deliver);
}

This setup allows Varnish to handle both domains with distinct configurations but within the same cache.

To enable Varnish, I updated the /etc/rc.conf file with the following lines, setting a maximum cache size of 2GB:

varnishd_enable="YES"
varnishd_listen="127.0.0.1:8080"
varnishd_config="/usr/local/etc/varnish/default.vcl"
varnishd_storage="default,2000M"

You can now start Varnish:

service varnishd start

The next step is to create two virtual hosts on Nginx (one for it-notes.dragas.net and one for media.bsd.cafe) that will listen on both IPv4 and IPv6 for HTTP and HTTPS. HTTP connections will be redirected to HTTPS, and incoming HTTPS traffic will be passed to Varnish, which will either return cached data or fetch it from the original server (Minio via WireGuard, for media.bsd.cafe). Let's see the media.bsd.cafe part:

server {
   server_name  media.bsd.cafe;

   [...]

   location / {
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_connect_timeout 300;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    chunked_transfer_encoding off;

    expires 12h;
    add_header Cache-Control public;

    add_header X-Cache-Status $upstream_cache_status;
    add_header X-Content-Type-Options nosniff;

    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
    add_header Referrer-Policy "no-referrer-when-downgrade";
    add_header Permissions-Policy "geolocation=(), microphone=(), camera=()";

    proxy_pass http://127.0.0.1:8080;

    [...]

}

[...]

}

server {
    if ($host = media.bsd.cafe) {
        return 301 https://$host$request_uri;
    }
   listen       80;
   listen  [::]:80;
   server_name  media.bsd.cafe;
    return 404;
}

This configuration isn’t complete, but it provides a good idea of how to set up Nginx - each setup will vary. TTL and caching sizes will also differ based on the characteristics of each reverse proxy. For example, one of the proxies has an 8GB cache since I have ample resources there.

Generating certificates is another important aspect. In this case, as the reverse proxies are distributed, they all need to respond to the same addresses. One approach is to generate the certificate on one proxy and distribute it to the others. In my case, I opted to use lego, which, through PowerDNS’s API, adds a DNS record for validation. This way, each reverse proxy can independently generate and renew its certificates when needed.

Configuring DNS for Optimal Routing

Once everything is set up, it’s important to ensure that DNS responds correctly. In my case, I implemented a strategy like this:

Track reverse proxies that respond on port 443 (further refinements are possible and will be done later).
Return the closest reverse proxy based on the client’s IP address.

Unfortunately, PowerDNS on FreeBSD does not include GeoIP support by default, but I have my poudriere ready to compile and install the necessary packages. Alternatively, you could compile it within the jail using the port system.

After that, I installed the geoipupdate package (which requires a free license from MaxMind), updated the IP list, and configured PowerDNS to use the GeoIP database. I added the GeoIP backend alongside the existing SQLite3 backend and specified the database to use:

launch=gsqlite3,geoip
geoip-database-files=mmdb:/usr/local/share/GeoIP/GeoLite2-City.mmdb

Finally, I created a LUA record to return the correct address:

testclosest.bsd.cafe   60      IN      LUA     A "ifportup(443, {'proxy1ip', 'proxy2ip','proxy3ip'}" ", {selector='pickclosest'})"
testclosest.bsd.cafe   60      IN      LUA     AAAA "ifportup(443, {'proxy1ip6', 'proxy2ip6','proxy3ip6'}" ", {selector='pickclosest'})"

And voilà! We now have a small, self-hosted CDN, keeping full control and ownership of our data. Adding a new reverse proxy is straightforward — simply clone an existing proxy, update the WireGuard configuration (adding a peer on the Minio jail and changing the keys on the new proxy), and add it to the DNS.

Happy caching!

How we are migrating (many of) our servers from Linux to FreeBSD - Part 3 - Proxmox to FreeBSD

Stefano Marinelli — Tue, 14 Mar 2023 13:00:00 +0000

In recent years, we've been migrating many of our servers from Linux to FreeBSD as part of our consolidation and optimization efforts. Specifically, we've been moving services that were previously deployed using Docker onto FreeBSD, and it has proven to be a great choice for handling workloads efficiently.

To this end, we've also been migrating many of our virtual machines (VMs) to FreeBSD, deploying services within FreeBSD jails. In some cases, these jails have even replaced entire VMs and run bare metal. Although we prefer to move to native FreeBSD whenever possible, sometimes it's not the best option for all the services we offer. As a result, one of our most critical physical servers has been left behind for years.

This server was a Proxmox server that we installed many years ago and updated to version 6.4. It hosted some critical services, but upgrading to Proxmox 7.x posed some challenges. In particular, some of the LXC containers required tweaks.

Unfortunately, this server was quite old, with only four physical disks and 64 GB of RAM. It was located in an OVH data center and had been running well until one of the disks started to malfunction once a week, on Sundays. This would trigger a RAID reconstruction that kept the system busy for about two days.

Despite my preference for simple setups, this server had been deployed gradually over many years, and everything was tied together. As a result, unraveling the system to resolve the issues was not a simple task. Sometimes the combination of simple things can make everything complex.

The Proxmox Server

The Proxmox server was configured as the central hub for various services, including primary DNS, web hosting, VOIP, and more. It featured several bridges, each with its own specific purpose, and was connected to a virtual machine running MikroTik CHR. This machine was responsible for consolidating all incoming VPNs from the MikroTik devices we managed, both ours and those belonging to our clients. Additionally, it provided a series of bridges to manage these devices and all server management VPNs and other services. The Proxmox server also housed several virtual machines running Linux, FreeBSD, OpenBSD, and NetBSD, as well as LXC containers.

Over the last two years, we've been migrating most of these virtual machines and containers to FreeBSD-based VMs, which feature their own specific jails. Consequently, most of the VMs we've had to move were BSD-based, while only five Linux VMs remained. The LXC containers hosted a range of services, including servers managed by Virtualmin, a large installation of Zimbra (which was hosted within an LXC container running CentOS 7), as well as some minor Alpine Linux-based machines. We located all these virtual machines and containers in a LAN created and managed by CHR. All public IPs were managed by CHR, which relied on NAT mappings to establish communication between them. CHR had thus become the heart of our system, and if it experienced any issues, it could potentially take down the entire system. Fortunately, it remained stable for years.

Migration - first steps

The first step I took was to install FreeBSD on the new server. Easy peasy. The next step was to find a way for the CHR to migrate to the new server (under bhyve) and continue to manage all the public IPs of the original server. The problem is that OVH, with its failover IPs, ties a specific MAC address to each individual IP address. Therefore, the only way was to create a bridge on the FreeBSD server (on the Proxmox server, I already had the bridge on the physical network card) and create an L2 tunnel between the two servers - I used OpenVPN with tap interfaces, specifically inserted into the bridges. I could have used other methods and techniques, but I wanted to experiment with a setup that could allow, if necessary, to bridge a larger number of physical and virtual servers even if the IPs are all mapped to a single server. OVH does not allow, in fact, the splitting of classes, so a move must be made for the entire class, not for a single IP address.

Initially, MikroTik CHR 7 did not boot on bhyve. In the end, I managed to make it work, but I had other problems, probably related to the MTU of the interfaces. So I thought about taking the opportunity to unbind the LXC containers and VMs from CHR and remove MikroTik from the setup. With RouterOS version 7, in fact, Wireguard-based VPNs are also supported, so within a few days, it was possible to update the few routers still on 6.x and recreate some VPNs using Wireguard. I mapped both the VMs and LXC containers directly to their respective public IPs, greatly simplifying the steps. Everything worked perfectly.

The next step was to test the first migrations, starting from the VMs already on FreeBSD. For simplicity, I created a new FreeBSD VM in bhyve and copied (via zfs-send and zfs-receive) the datasets related to BastilleBSD. All services are installed in jails managed by Bastille, so this was enough to have, in a short time, a new operating server equivalent to the previous one. At that point, I shut down the original server, connected the VM to the bridge linked to the tunnel (after modifying its MAC address), turned on the new FreeBSD VM (on bhyve), and everything started to work correctly - but from the new physical server.

One by one, I moved all the FreeBSD VMs. For Linux, NetBSD, and OpenBSD, I simply copied the images and pointed bhyve to them. Some small specific configuration on vm-bhyve and everything started to work correctly. Where possibile, I replaced the “virtio” with “nvme” as it performs much better on bhyve.

Migration - LXC containers to Virtual Machines

For LXC containers, I initially thought of creating an Alpine Linux virtual machine, installing LXD, and copying each individual container. It worked for some of them, but for others, I started to encounter strange issues, similar to those that would have required manual intervention to upgrade from Proxmox 6.x to 7.x. As is often the case with Linux-based solutions, compatibility is not always preserved between updates, so I would have had to fine-tune all the containers, which I didn't feel like doing. The containers had been created (at the time) to optimize RAM usage on the Proxmox machine, but to date, they have caused more problems than benefits. In some cases, certain processes got "stuck," making it impossible to "reboot" the LXC container, requiring the entire physical node to be rebooted. If they had been virtual machines, I could have given a "kill" command from the virtualizer (to the respective KVM process, in that case) and restarted it.

For greater compatibility and ease of future management, I decided to convert the LXC containers into actual VMs on bhyve. The process was simple:

Creating an empty VM with vm-bhyve and booting the VM with SystemRescueCD.
Creating destination partitions and file systems in the VM, then doing a complete rsync of the original LXC container.
Adjusting the fstab file, installing the kernel on the destination VM, and creating the initrd (some containers were already copies of VMs, so the kernel remained installed and updated, even though it wasn't being used. The initrd, on the other hand, did not include the nvme or virtio drivers, so I had to regenerate it anyway.)
Adjusting the bhyve vm configuration file, doing one last rsync after shutting down the services, shutting down the original LXC container, and starting the bhyve VM.

Everything worked correctly, so one by one, I moved all the containers. The largest one ended up on another physical node (also FreeBSD with bhyve) temporarily because the space on the new server was not sufficient to contain it. It didn't need to be on this server, so no problem.

One by one, the LXC containers started on the new server. Apart from some minor adjustments to the destination VMs (different network interface names, etc.), I didn't encounter any particular problems even after several days. Everything works perfectly.

At the very end, I re-created the MikroTik CHR VM. I’ll keep this setup separate for now, as strictly tied to eoip interfaces. This was the main reason why I haven’t performed the migration before. Things were too tied together and I had to untie everything, step by step.

…and then one of the Linux VMs started to freeze

Several Linux VMs are just the basis on which Docker runs. One of them (not even among the busiest) started, every 12/15 hours, to completely freeze. It stopped responding to ping, and it was impossible to give any type of command from the console. In a word: stuck.

Searching the web, I found some references to this problem and, observing the errors of an ssh session that was left connected (stuck, but still showing the last error), I found it to be a problem similar to the one described in this post, namely:

"watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [khugepaged:67]"

I tried various solutions such as changing the storage driver, the number of cores, the distribution (from Alpine to Debian), etc., but none of these operations solved the issue. I also noticed that the problem occurs with all Linux VMs, but only those with a recent kernel (> 5.10.x) freeze, while the others continue to work. The problem does not occur, however, with the *BSDs.

In the end, I:

Reduced the number of cores to 1 for the VMs that did not have a high load (some remained with multiple cores), hypothesising a problem with allocating cores that were too busy
Gave the command: "/usr/bin/echo 60 > /proc/sys/kernel/watchdog_thresh" to the VM.

The VM became stable, and I have not seen that error/warning on any other machine since. I will investigate further, but I believe it is a problem related to the Linux kernel, which, for some reason, generates a kernel panic if particular situations of CPU concurrency are generated.

The End…and a nice OOM!

After moving everything, I was finally able to migrate the entire class of OVH IPs from one physical server to another. The operation was quite quick, but in order to avoid problems, I notified all users and performed the operation on a Sunday and during off-peak hours. The whole process took about 10 minutes and there were no hitches of any kind.

For safety reasons, I kept the Proxmox machine active for a few more days, but there was no need to use it. However, after a couple of days, I encountered a problem: the largest VM, in some cases, was being "killed" because FreeBSD generated an OOM. I had never seen, from FreeBSD 13.0 onwards, any OOM related to "abuse" of RAM usage by ZFS, but in this case, it actually happened.

In the end, I understood that ZFS, on FreeBSD, is able to release memory, but not quickly enough to manage any "spikes" in individual VMs. In fact, the VMs do not know the situation of the physical host's RAM, so they will tend to occupy all the space allotted to them (even if only for caching). A sudden spike (i.e. if you create and launch a new VM) could cause a sudden increase in RAM usage by the bhyve process, and FreeBSD could be forced to kill it, even if part of the RAM is only ARC cache. While Proxmox supports HA (i.e., control over whether the VM is running), vm-bhyve only launches the VM (bhyve process). I should manage it with tools like monit, but for now, I preferred to simply set limits on ZFS RAM usage using "vfs.zfs.arc_max", and there have been no more problems.

Final considerations

The operation was long but linear. The most complex part was unraveling all the configurations related to MikroTik CHR and the VPNs linked to each individual LXC machine/container. Once everything was implemented on a dedicated VM, the operation was fairly straightforward.

The hardware specifications of the destination physical server are slightly better than the starting one, but the final performance of the setup has greatly improved. The VMs are very responsive (even those that were previously LXC containers running directly on bare metal) and, thanks to ZFS, I can make local snapshots every 5 minutes. In addition, every 10 minutes, I can copy (using the excellent zfs-autobackup) all the VMs and jails to other nodes both as a backup and as an immediate restart in case of disaster. I just need to map the IPs, and everything will start working very quickly. Proxmox also allows you to perform this type of operation with ZFS, but you still need to have Proxmox (in a compatible version) on the target machine. With the current setup, I only need any FreeBSD node that supports bhyve.

Proxmox is an excellent tool, well-developed, open-source, efficient, and stable. We manage many installations, including complex ones (ceph clusters, etc.), and it has never let us down. However, not all tools are ideal for all situations, and for setups like the one described, the new configuration based on FreeBSD has shown significantly interesting performance and greater management and maintenance granularity.

Virtualizing on vm-bhyve is not complex, but it is certainly not comparable, at the current state, to the simplicity of using a clean and complete interface like Proxmox's. A complete HA system is still missing (sure, it's achievable manually, but...), as well as complete management web interface. However, for knowledgeable users, it is undoubtedly a powerful tool that allows you to have excellent FreeBSD as a base. I'm totally satisfied with my migration and the result is far better than I expected.

Deploying a piece of the Fediverse

Stefano Marinelli — Sun, 15 Jan 2023 09:29:45 +0000

After Elon Musk’s Twitter deal, many users decided to “fly away” from the "traditional" commercial Social Networks. Some for good, some just decided to increase their presence in other, alternative Social Network. That's what I'm doing.

Many of those users decided to join the Fediverse - even if many of them just call it Mastodon, as they don’t understand that Mastodon is just a Software that allows to join the Fediverse.

The Fediverse is composed by thousands of “instances”, some are bigger (like mastodon.social), some are personal (aka: single user instances), many are normal communities with their members. Many of them communicate using the same open protocol, ActivityPub.

Because of this, there’s no “one size fits all” so I’ve started to explore the different solutions. I’ve been mainly focusing on running them on FreeBSD, but I’ve had to fire up Linux for some tests. Here’s what I’ve found out.

Backends / Complete Solutions

Mastodon

I won’t spend too much time on Mastodon as you may find almost everything, everywhere, about it. Tons of articles have been written about Mastodon, so this one would be just another one, surely not the best one. Many consider it to be “the Fediverse” (they just say “Mastodon” to refer to the whole “Fediverse”, and they’re wrong), it’s by far the most installed solution. It’s so popular that there are plenty of clients (both for Android and iOS) that perfectly work with it. Mastodon has its own APIs - and many other Fediverse solutions are using them, just to be able to be compatibile with the Mastodon apps. It also supports backend based “Hide replies” (only show new posts, not all the replies to other posts) in timeline. It’s good as you can filter the replies from any frontend or mobile app as the backend won’t provide them at all, while the client doesn't need to be aware that you're filtering them.

Why I am suggesting it: It’s stable and well done, there’s a lot of documentation and if you install and manage it correctly, you shouldn’t notice anything strange or unexpected. Remember, no software solution can be considered “set and forget” and Mastodon is not an exception. Please, don’t forget that running an instance is not just installing the software. Moderation is a serious issue. More about it later.

Please, consider that: Mastodon is also quite heavy, not easy to scale (even if there’s documentation around) and, by default, is caching everything it sees and knows about. This means that both a single user instance or a thousands of users’ one, will (rapidly) grow because any media will be locally cached.

My first, single user installation grew, in a week, well over 100 GB of occupied storage because of all this caching. It can’t be avoided (you can just tell Mastodon to delete the cache after x days). While it may make sense for a big instance, it can be considered an overkill for a single user one. This is a "known problem", but mainly considered as a feature: all the media will be locally processed, all the contents will be locally stored. So an evil content hidden in a media file will be reprocessed by local ffmpeg or ImageMagick and will be cleared.

Why I’m suggesting another solution: Monopoly is bad and Mastodon is becoming, for many, a synonym of Fediverse . More, it requires much space and it’s resource-hungry. It’s easy to install Mastodon and experience a huge resource drain in just a few days, especially if you’re not a skilled system administrator. While Mastodon is the best solution for a complete microblogging experience, other solutions exist.

To sum up: Installing Mastodon on FreeBSD was easy. Even if I had read about problems, I’ve documented how to do it and it’s stable and reliable. Mastodon is, IMHO, a good piece of software but keep in mind that it could not be the best solution for you and a small, single user instance can become huge in a few weeks. Also, keep in mind that it's the most deployed Fediverse software, so any mobile app, any web app, any hint will work perfectly with Mastodon. Your Fediverse experience will be smooth.

Akkoma (Pleroma fork)

I’ve read about Akkoma in a reddit thread about how difficult was to install Mastodon on FreeBSD. It was described as a Pleroma fork, but actively developed and maintained, faster and with more advanced features. That’s why I decided to try it and - at least for now - stick with it (and not Pleroma, but many of the things I’ll point out here apply to Pleroma, too).

Akkoma (as all the Pleroma forks) is much, much lighter than Mastodon. It’s perfectly able to run a single user instance on a Raspberry PI. Moreover, Christine Lemmer-Webber, coauthor of the ActivityPub protocol, said that Akkoma is a good solution - and I think that her opinion is a qualified one.

Why I am suggesting it: First of all, for its documentation. As for Pleroma, there are installation instructions for a lot of operating systems (yes, also FreeBSD, NetBSD, OpenBSD). It’s light and fast. It doesn’t cache remote media by default, so that’s perfect for a single user instance. You can enable it (both pre-fetching media as soon as the server gets the status, like Mastodon, or just downloading them and caching when the first user meets them), you can also proxy your local media. S3 storage and remote CDNs are supported and everything is customisable. Message size limit is set to 5000 characters by default, but can be adjusted. Everything is configurable and you can choose your favourite frontend. It has quote posts (while Mastodon doesn’t allow them, even if they’re perfectly visible if created by Akkoma).

Please, consider that: Akkoma is not Mastodon. They “talk” using the same language but are different pieces of software. There’s less activity around it (the documentation is good, the support forum is good, but the number of Akkoma installations can’t be compared to Mastodon’s ones). Many Mastodon mobile apps seem to have problems with Akkoma and at the moment there’s a bug (probably it's a Mastodon bug, but users will think it's Akkoma's fault) that, if you’re posting an hashtag, a link will be shown on Mastodon. More, if you want to hide boosts or replies from your timeline, remember that Akkoma won’t perform that at backend’s level, but it should be done by the frontend. Not all frontends and mobile apps support it.

Why I’m suggesting another solution: Well, not exactly. Actually, I’m suggesting to try Akkoma. It’s a good piece of software, developed by friendly people, well accepted by the Fediverse instances’ administrators and has a lot of happy users and instance administrators. I've used Akkoma as my main instance software for more or less one month. The bug I've described and some problems here and there made me move back to Mastodon. I'm keeping my Akkoma instance up, even if not actively used.

To sum up: Akkoma installation is easy and well documented, you have a lot of settings to customise your instance and can fine-tune your installation for your hardware capabilities.

Misskey (and its forks like Firefish, etc.)

Misskey is a very nice piece of software. I had some troubles to run it on FreeBSD (but I didn’t try that much) so I decided to fire up a Linux machine and use Docker.

The interface is nice - it’s Japanese design, so it’s fancy and rich of emojis and effects. I’ve tried it just for a few hours and I appreciated it but decided it wasn’t ok for my needs (at least, for now).

Why I am suggesting it: if you’re building a community, Misskey (or one of its forks) is a very good choice. It’s eye candy, complete and usable. It’s a part of the Fediverse, so no problems to talk to Mastodon, Akkoma, etc. It also has a “drive” feature, useful for many users that want to exchange files.

Why I’m suggesting another solution: the main reason why I had to look at another solution is that it doesn’t support Mastodon APIs so no Mastodon Android or iOS app is working with a Misskey instance. While it may be ok for many users, this could be a problem for others.

To sum up: Misskey is nice, worth trying and a very good solution if it fits your community’s needs. For me, it doesn’t give any advantage over other lighter solutions.

Friendica

Friendica is a project that aims to be similar to Facebook. It’s federated and actively maintained, the interface is nice and familiar to Facebook users. I’ve been able to install it on a FreeBSD jail (as it’s in PHP) and everything worked as expected. I didn’t spend too much time on Friendica but I’m planning to do a deeper test as I’m working on a community of former Facebook users and this could be the right choice.

Why I am suggesting it: If you’re creating a community for (former) Facebook users, they’ll have a familiar feeling in Friendica. The interface is clean and usable, it supports a lot of protocols (ActivityPub, OStatus, diaspora), it supports plugins, can import websites via rss (so automatic post is easy).

Please, consider that: Moderation tools are different from the ones you have on Mastodon or Akkoma (Pleroma, etc.) and you can’t easily report users, especially remote ones as support for Mastodon API is limited. With the current growth of users, it’s easy to find a bad person trying to disturb. Not having effective ways to deal with it may be frustrating.

Why I’m suggesting another solution: I didn’t try Friendica long enough to find some big problems with it.

To sum up: Generally speaking, it is considered a solid and stable solution, actively maintained and, being in php, portable. If you want to create a Facebook-like community, that's the way to go.

snac2

snac2 is a simple, minimalistic ActivityPub instance that supports the Mastodon API. This makes it compatible with platforms like Pleroma, Akkoma, and Mastodon itself. It's written in portable C and it's been created to be light, easy to deploy, and with only two dependencies: openssl and curl. It heavily relies on hard links and doesn't need any database. This is a big, big plus for me. I've performed many tests and found that this is one of the best lightweight solutions to join the Fediverse.

Why I am suggesting it: snac2 is clean and polished. The federation with the other solutions is good, it works beautifully with Tusky (on Android) and tooot (both on Android and iOS) - also consider Enafore as a PWA or web interface - and its integrated web interface is minimal but effective. No javascript, no cookies - clean web. More, the dev is responsive and open to patches and contributions. I've helped with some stress tests and contributed with instructions and patches to make it work on FreeBSD and NetBSD, and they've been merged immediately.

Please, consider that: It's not Mastodon. Some of the Mastodon API features are (currently) not supported so the experience could be different from the other solutions. Some Mastodon apps don't work (the official Mastodon app, for example, can't login). There's no open registration option (users should be manually registered from the cli) and account migration is not supported, at the moment. More, while it's not caching external media, locally published media will stay on the local drives (no S3 upload option), so be prepared to serve those files as well. Testing it from my FTTC connection, I almost DDoSed my internet connection for 15 minutes, publishing a photo.

Why I’m suggesting another solution: Actually, I'm suggesting to try snac2. I think it could be a great solution for a single user instance or for instances managed by tech people, as you can run it just 1 minute after the download. It's light, easy, straightfoward and the dev is a nice person. I'd suggest other solutions if the priority is to offer a full, feature rich Fediverse experience, with registration, big and featureful configuration panels, etc

To sum up: snac2 is a lightweight and effective way to join the Fediverse. Currently, it's my favourite solution for small communities as the "file only" approach and no dependencies are coherent with my ideas. I've migrated my instance from FreeBSD to NetBSD, from external datacenters to my home network and it's just been a matter of a single rsync (-H)

GoToSocial

GoToSocial is a new microblogging platform. Its target it to be a light, customisable, easy to manage and integrated software. It is still at its early alpha stage, it should evolve into a beta at some point in ~~2023~~ 2024. It’s developed in Go and installation in easy and fast, it supports Postgres, Mysql and Sqlite. Basic functionalities have already been integrated and it can federate with (almost) all the other ActivityPub implementations, even if with some small problems. FreeBSD installation was easy and fast, as they provide a amd64 FreeBSD binary. Being written in go, it shouldn't be difficult to self-compile it for other architectures.

Why I am suggesting it: While it doesn’t have an integrated frontend (but Mastodon mobile apps or other frontends can be used, as Enafore Mangane, etc.), it already supports many of the features you’d expect from an ActivityPub implementation. It's fast, suitable for installation on a low end hardware, can be deployed without a reverse proxy as has integrated support for Letsencrypt certificates. But…

Please, consider that: It’s still at alpha stage. Things can still break and it doesn’t support any kind of account migration from Mastodon (while it’s supported by Pleroma and its forks).

Why I’m suggesting another solution: While I think it could be a game changer, I think it’s bit early to deploy it unless you’re a very skilled and experienced administrator. You should understand its “quirks and features”, mainly tied to its alpha status. While I’ve installed and will keep installed a GoToSocial instance, at the moment I’ll just keep it in a test stage server in order to follow its development status.

To sum up: GoToSocial could potentially become one of the most interesting ActivityPub microblogging platforms, at least for small and medium sized communities. It's a bit early to consider it ready for a production deployment, but it's already worth testing.

Frontends

All the Fediverse backend implementations have their own specific features but the users will just interact via a frontend. While Mastodon - but also Pixelfed, Peertube, etc. - are presented as a specific stack (backend + frontend), there are many frontends that can interact via the Mastodon API.

Of course, being the “Mastodon” API, not all the backends are perfectly compatibile/supported by all the frontends. Mobile apps use the Mastodon API, that’s why their compatibility with other implementations like snac2, Pleroma, Akkoma, etc. may not be perfect at all the times.

I won’t describe them all. I’ll just enumerate the ones I’m using, with some notes:

Akkoma’s Pleroma-FE

With the 2022.12 release of Akkoma, Pleroma-FE has been evolved into a proper, nice looking PWA. After an initial configuration of its many options, I found it quite nice and effective. The only problem (common with many PWAs on iOS) is that when you suspend the app, it doesn’t detect it, so if you open it again after two hours, you’ll just see the recent posts, not all the posts of the last two hours. This is because of the way iOS deals with app suspend/resume and many PWAs don’t seem to understand they’ve been suspended.

Mangane

Mangane is a Soapbox fork that aims to improve compatibility with Akkoma. It is nice and clean and the developers are improving it to support Akkoma's features, so it's great. At the moment I've noticed some visual problems, on iOS, with the icons - but Guérin contacted me to ask for information (after reading this article) and opened an issue on to fix it. Guérin has been nice and helpful, making Mangane even more appealing. I’m using it for planned posts on Akkoma, as Pleroma-FE doesn’t support them, yet. I'm also using Mangane as a daily driver, from time to time, as I like it.

Enafore

A light, fast, complete and usable web frontend. I’ve been using it when I needed a simple, clear frontend. It supports “hide replies”, which is great to improve timeline quality. It's among my favourite choices when using a webapp and is a good choice for snac2.

Elk

Elk is a frontend for Mastodon API. It’s nice, clear, eye candy, intelligent. The developers are nice people, open to suggestions. Development is a bit slower compared to the initial pace, but the app is already very complete and stable.

Elk is definitely a very good piece of software, and I recommend to try it.

Moderation and final considerations

One of the things you should be considering is that deploying a piece of the Fediverse isn’t just installing a software and interacting with others. Actually, that’s just the easiest part of the experience, at least if you’re not creating a single user instance.

Many of the people that joined the Fediverse in its early days decided to do it as they felt attacked on other social networks. In the last years, the commercial socials have proven to be the perfect place for negative people, attacking others without being blocked/stopped in an efficient way. Hate causes addiction and the owners of those commercial socials make a lot of money if people interact, showing them ads every time they open the app/website. They make money (also) through people hating each other.

The Fediverse gives the possibility to mute and block users, but also to mute and block entire instances. One of the main tasks of an instance’s administrator is to make sure that everything is ok. While you can define your own instance’s rules, other instances’ admins may block you if they find you’re federating by sending messages agains their rules.

As an administrator, you’re also responsible of keeping your users’ data safe, to avoid sharing/providing illegal contents or offensive stuff. So you’re free to set your rules, but others are free to “defederate” you, if they don’t like the contents your instance is providing. While it’s not an issue for a single user instance, you must be quite careful when opening the registrations as you may find out you've been defederated because of (your) lack of moderation.

There’s some strong criticism against the Fediverse because of this, as many see it as more “censored” than the traditional, commercial social networks. I don’t think it’s true, as you’re free to fire up your instance, decide your rules and act as you want. But you can’t impose others and their instances to follow you, even if you think you’re right. In a free world, everybody should be free to decide if they want to listen to you. But everybody should also be free to create a new space and start sharing their ideas. There’s not a central authority of contents, so only the single admins and users can choose what they want to see or avoid. There's not a commercially driven algorithm that may decide that you should see contents that will cause you anger and hate just because it generates traffic (and money) to the social's owner.

The Fediverse can be a beautiful place to stay.

Installing Mastodon inside a FreeBSD jail: A Comprehensive Guide

Stefano Marinelli — Wed, 23 Nov 2022 07:52:02 +0000

Note: Updated for Mastodon 4.5

Introduction

Mastodon and the Fediverse have gained significant popularity, especially during times of uncertainty with traditional social media platforms. As users seek alternative spaces, many are discovering the decentralized nature of Mastodon instances. However, this influx of users has led to challenges for unprepared instances, including performance issues and moderation difficulties.

This guide aims to provide a comprehensive walkthrough for installing Mastodon on a FreeBSD jail, managed by BastilleBSD. While Mastodon documentation tends to be Linux-centric, this tutorial fills the gap for FreeBSD users.

Note: This guide describes a simple, single-jail installation. For production environments, it's recommended to separate services (Valkey, PostgreSQL, etc.) into individual jails. This tutorial assumes a basic understanding of FreeBSD and Unix-like systems.

Prerequisites

A FreeBSD system with BastilleBSD installed
Basic knowledge of FreeBSD jail management
Familiarity with command-line operations

Step 1: Creating the Jail

Let's start by creating a new jail using BastilleBSD:

bastille create mdontest 14.3-RELEASE 10.0.0.42 bastille0

Step 2: Configuring the Jail

As we'll be installing PostgreSQL in the jail, we need to add some configurations to the jail's jail.conf:

sysvmsg=new;
sysvsem=new;
sysvshm=new;

After adding these lines, restart the jail:

bastille restart mdontest
bastille console mdontest

Step 3: Installing Dependencies

Now, let's install the necessary packages:

pkg install -y curl wget gnupg gmake git-lite vips node22 yarn-node22 postgresql18-server postgresql18-contrib ImageMagick7 ffmpeg autoconf nginx valkey py311-certbot py311-certbot-nginx sudo rubygem-bundler rubygem-posix-spawn

Step 4: Enabling and Configuring Services

Enable Valkey, Nginx, and PostgreSQL:

service valkey enable
service nginx enable
service postgresql enable

Valkey Configuration

For simplicity in this jail environment, we'll disable Valkey's protected mode. However, this is not recommended for production environments without proper security measures.

Edit /usr/local/etc/valkey.conf and set:

protected-mode no

Important: In a production environment, ensure proper authentication and network security measures are in place before disabling protected mode.

PostgreSQL Initialization and Configuration

Initialize the PostgreSQL database:

service postgresql initdb

Modify PostgreSQL to accept connections from the jail's services. Edit /var/db/postgres/data18/pg_hba.conf and add:

host    all    all    10.0.0.42/32    trust

Note: In a production environment, consider using more restrictive authentication methods.

Start PostgreSQL and Valkey:

service postgresql start
service valkey start

Step 5: Database Setup

Create the Mastodon database user:

sudo -u postgres psql
CREATE USER mastodon CREATEDB;
\q

Step 6: Creating the Mastodon User

Create a dedicated user for Mastodon:

pw add user mastodon -m
echo 'export LC_ALL="en_US.UTF-8"' >> /home/mastodon/.profile

Step 7: Installing Mastodon

Enable corepack, switch to the Mastodon user and install the software:

corepack enable
su -l mastodon
git clone https://github.com/mastodon/mastodon.git live && cd live
git checkout $(git tag -l | grep '^v[0-9.]*$' | sort -V | tail -n 1)
corepack prepare

Install Ruby and Node dependencies:

export CONFIGURE_ARGS="--with-cflags=\"-Wno-error=incompatible-function-pointer-types\""
export NODE_OPTIONS="--openssl-legacy-provider"
bundle config deployment 'true'
bundle config without 'development test'
bundle install -j$(getconf _NPROCESSORS_ONLN)
yarn install --immutable

Step 8: Mastodon Setup

Run the Mastodon setup command:

RAILS_ENV=production bundle exec rake mastodon:setup

When prompted for the PostgreSQL host, enter 127.0.0.1 (or 10.0.0.42).

Mastodon can now use libvips as a lighter and more modern alternative to ImageMagick. ImageMagick support is being deprecated, so it's suggested to switch to libvips.

To use libvips instead of ImageMagick, set the MASTODON_USE_LIBVIPS environment variable to true into the .env.production:

[...]
MASTODON_USE_LIBVIPS=true

Step 9: Nginx Configuration

In the dist/ directory, you'll find an nginx.conf file. This is not a complete Nginx configuration but a partial one for Mastodon. Integrate this with your existing Nginx setup based on your specific requirements.

Note: Many administrators advise against exposing Mastodon through Cloudflare, as it may interfere with some APIs and disrupt Fediverse interactions.

Step 10: Creating FreeBSD RC Scripts

To manage Mastodon services on FreeBSD, we'll create custom rc scripts. You can find the scripts for mastodon_sidekiq, mastodon_web, and mastodon_streaming at the provided links.

Place these scripts in the /usr/local/etc/rc.d/ directory, make them executable (chmod a+rx /usr/local/etc/rc.d/mastodon_*) and enable them:

service mastodon_sidekiq enable
service mastodon_web enable
service mastodon_streaming enable

Conclusion

Restart the jail or start the services individually. Logs will be appended to /var/log/messages.

You now have a functioning Mastodon instance running in a FreeBSD jail. All services are run by the "daemon" user and are supervised.

Remember to regularly update your Mastodon instance and monitor its performance. For production environments, consider implementing additional security measures and potentially separating services into individual jails.

If you want to change the characters or poll limits, you can refer to this article.

Enjoy your new Mastodon instance!

IT Notes - web

Aggressive caching for a Mastodon reverse proxy: what to cache, what to never cache, and why content negotiation will eventually betray you

Assumptions before anything else

The cache zone

Default location: the microcache and streaming without cache

Upstream and failover

What I log and why

How I check it actually works

Caveats worth being honest about

FediMeteo, timezones, and the art of not breaking what already works

FediMeteo, HAProxy, and the art of not wasting snac threads

A continuation of the same idea

Why there is almost no media

The homepage is static because it can be static

Many countries, one entry point

Backends as small compartments

The front door

Two caches, because the requests are different

Recognizing media

ActivityPub JSON microcaching

Static media paths

Vary, but not without limits

Seeing whether it works

Compression and operational paths

What this changes in practice

Caveats

Conclusion

Self-hosting your Mastodon media with SeaweedFS

Setting up the SeaweedFS Jail

Configuring Buckets and Users

Nginx Reverse Proxy Configuration

Mastodon Configuration

Introducing the illumos Cafe: Another Cozy Corner for OS Diversity

Introducing the illumos Cafe: Another Cozy Corner for OS Diversity

From the BSD Cafe to illumos Cafe

Why Another Cafe?

Why illumos?

The illumos Cafe Project

Community and Philosophy

Technical Setup

Available Services

Registrations and Logo

Looking Ahead

New Article on BSD Cafe Journal: WordPress on FreeBSD with BastilleBSD

New Article Published

Future Technical Content

Make Your Own Internet Presence with NetBSD and a 1 euro VPS – Part 1: Your Blog

Why NetBSD?

Why BSSG?

Let's Start with the Installation

Using NetBSD's Integrated httpd

Using nginx

Using Caddy

Performance Comparison

Conclusion: Efficient Minimalism

Launching BSSG - My Journey from Dynamic CMS to Bash Static Site Generator

The Era of Dynamic CMS

Discovering Static Site Generators

Building My Own: BSSG

Brief Detour: ITNBlog

BSSG to the Rescue (and ITNBlog's Role)

Why Choose BSSG?

Admin Interface (Experimental)

BSSG is Available Today

FediMeteo: How a Tiny €4 FreeBSD VPS Became a Global Weather Service for Thousands

Personal Introduction

Design Principles

Technical Implementation

Setting Up for Italy

Core Application Development

Growth and Unexpected Success

A Turning Point

Scaling Challenges

Expansion to North America

System Performance

Current Status

Challenges Encountered

Infrastructure Details

Conclusion

Caching snac Proxied Media with Nginx

`Vary`, but not without limits