Bridging Networks Across VPS With Wireguard and VXLAN on FreeBSD

Introduction

In today’s interconnected world, system administrators often face the challenge of managing services across multiple Virtual Private Servers (VPS). This article describes an advanced networking setup that allows you to bridge networks between two VPS instances using Wireguard and VXLAN on FreeBSD. This configuration is particularly useful when you need to distribute services across different providers or when you want to leverage the strengths of multiple hosting environments.

Background

At BSD Cafe, we utilize various VPS instances to provide our services. The two main ones are:

1. A publicly accessible VPS that hosts the reverse proxy and all firewall rules for packet routing.
2. A larger VPS on a physical host I own, which is not directly exposed to the internet and doesn’t have a public IP address.

Most of the jails hosting BSD Cafe services are distributed between these two VPS instances. Occasionally, I need to move services between them for performance reasons or to manage updates efficiently.

To facilitate this flexibility, I’ve always maintained a bridge on each VPS. Initially, I used Zerotier to establish a connection between these bridges, allowing them to communicate as if they were part of a single, large network.

The New Setup: Wireguard and VXLAN

While the Zerotier setup worked, I decided to switch to a more streamlined solution using Wireguard and VXLAN. Here’s why:

1. Performance: Wireguard offers excellent performance with low overhead.
2. Simplicity: The configuration is straightforward and easy to maintain.
3. Security: Wireguard provides strong, modern cryptography.

I had already prepared a Wireguard connection between the two servers from the beginning. Since only one of the servers is publicly accessible, I set up one to only accept connections and the other to connect directly to the public IP of the first, with a 20-second keepalive (which is generally not necessary due to the high traffic between the jails).

To complete the setup, I added two VXLAN interfaces on the VPS instances, added these interfaces to the local bridges, and immediately, packets started flowing between the networks.

Overall Network Topology

graph TB
    subgraph "Public Network"
        Internet((Internet))
        Internet2((Internet))
    end

    subgraph "VPS 1 (Public IP)"
        PublicIP[Public IP]
        Firewall[Firewall]
        ReverseProxy[Reverse Proxy]
        Bridge1[Bridge]
        WG1[Wireguard]
        VXLAN1[VXLAN]
        Jail1[Jail 1]
        Jail2[Jail 2]
    end

    subgraph "VPS 2 (Behind NAT)"
        NatDev2[Nat]
        Firewall2[Firewall]
        Bridge2[Bridge]
        WG2[Wireguard]
        VXLAN2[VXLAN]
        Jail3[Jail 3]
        Jail4[Jail 4]
    end

    Internet <--> PublicIP
    PublicIP <--> Firewall
    Internet2 --- NatDev2
    NatDev2 <--> Firewall2
    Firewall2 <--> WG2
    Firewall <--> WG1
    Firewall --> ReverseProxy
    ReverseProxy --> Bridge1
    Bridge1 --- Jail1
    Bridge1 --- Jail2
    Bridge1 --- VXLAN1
    VXLAN1 <---> WG1

    Bridge2 --- Jail3
    Bridge2 --- Jail4
    Bridge2 --- VXLAN2
    VXLAN2 <---> WG2

    Internet <--> Internet2

Step-by-Step Implementation

Follow these instructions to create a bridge between two different networks using Wireguard and VXLAN on FreeBSD. While I use this setup to connect jails at BSD Cafe, you can use it for various purposes, such as bridging different VM (bhyve) instances across providers.

Prerequisites

Wireguard is now an integral part of FreeBSD, so you no longer need to compile a module or use the Go version. However, we’ll use the “wireguard-tools” scripts as they provide the useful “wg-quick” command.

Start by installing the wireguard-tools package on both servers:

1

pkg install wireguard-tools

Configuration

Server 1 (Public IP)

1. Generate the Wireguard keys:

1

wg genkey | tee /dev/stderr | wg pubkey | grep --label PUBLIC -H .

This command will output a private key and a public key. Note down the public key as you’ll need it to configure the client.

Let’s also add a PSK; it’s optional but will increase the security of the entire setup.

1

wg genpsk

2. Create a new file /usr/local/etc/wireguard/wg0.conf:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

[Interface]
## Default port is 51820 - feel free to change it
PrivateKey = <the private key from the previous command>
ListenPort = 43671
Address = 10.77.0.1/24

PostUp = /sbin/ifconfig vxlan create vxlanid 42 vxlanlocal 10.77.0.1 vxlanremote 10.77.0.2 inet 10.77.1.1/24
PostUp = /sbin/ifconfig bridge0 addm vxlan0 up
PostDown = /sbin/ifconfig vxlan0 destroy

[Peer]
PublicKey = <the other peer's public key>
#If publicly exposed, you can specify the peer ip address/port
#Endpoint = <public_ip>:<port>
AllowedIPs = 10.77.0.2/32
PresharedKey = <the PSK from the previous command>

3. Modify /etc/rc.conf and add:

1
2

wireguard_interfaces="wg0"
wireguard_enable="YES"

4. Start Wireguard and the VXLAN endpoint:

1

wg-quick up wg0

Server 2 (Behind NAT)

1. Generate the Wireguard keys as before.

2. Create /usr/local/etc/wireguard/wg0.conf:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

[Interface]
PrivateKey = <the private key from the previous command>
Address = 10.77.0.2/24

PostUp = /sbin/ifconfig vxlan create vxlanid 42 vxlanlocal 10.77.0.2 vxlanremote 10.77.0.1 inet 10.77.1.2/24
PostUp = /sbin/ifconfig bridge0 addm vxlan0 up
PostDown = /sbin/ifconfig vxlan0 destroy

[Peer]
PublicKey = <the other peer's public key>
Endpoint = <public_ip>:<port>
AllowedIPs = 10.77.0.1/32
PresharedKey = <the PSK from the previous command>
PersistentKeepalive = 20

3. Modify /etc/rc.conf as before.

4. Start Wireguard and the VXLAN endpoint:

1

wg-quick up wg0

Verifying the Connection

To check if the connection is established, run the wg command on either host. This will show you the connection status, the last handshake, and the data transferred.

You can also try pinging the other host’s Wireguard and VXLan interface IP address (in this example, 10.77.0.1 or 10.77.0.2 and 10.77.1.1 or 10.77.1.2).

VXLAN Configuration Detail

graph LR
    subgraph "VPS 1 (10.77.0.1)"
        Bridge1[Bridge0]
        VXLAN1[VXLAN0\n10.77.1.1/24]
        WG1[Wireguard\nwg0]
    end

    subgraph "VPS 2 (10.77.0.2)"
        Bridge2[Bridge0]
        VXLAN2[VXLAN0\n10.77.1.2/24]
        WG2[Wireguard\nwg0]
    end

    Bridge1 --- VXLAN1
    Bridge2 --- VXLAN2

    VXLAN1 -.- WG1
    VXLAN2 -.- WG2

    WG1 <--> |Encrypted Tunnel| WG2

    classDef vxlan fill:#f9f,stroke:#333,stroke-width:4px;
    class VXLAN1,VXLAN2 vxlan;

Conclusion

This setup allows the two VXLAN interfaces, inserted into the local bridge, to enable packet transit through Wireguard. This facilitates free passage between the two hosts, effectively creating a single, unified network across your VPS instances.

This configuration is particularly useful for:

• Distributing services across different providers
• Leveraging both public-facing and private VPS instances
• Creating flexible, scalable network architectures

By using Wireguard and VXLAN, you get the benefits of strong encryption, high performance, and the ability to create complex network topologies across physically separate servers.

Remember to always keep your systems updated and regularly review your network configuration to ensure it meets your evolving needs and security requirements.