Forwarding traffic back through WireGuard (Setting dnat for oif wg0, after processing the application)

Question

I use WireGuard as a secure communication channel between two servers in different DCs to hide the existence of the end server (server B). I use nftables as a firewall management tool.

From public server A, the traffic is forwarded keeping the IP address (necessary for the application).

The destination server B receives the packets and the application processes them, but eventually, the server tries to return the packet to the original IP (the sender IP of the original packet) and this becomes a problem.

Masquerading the original IP looks like a simple solution, but it is necessary to keep the original IP already on server B to route these packets back to the WireGuard tunnel.

The tcpdump of the server B:

1:02:36.675958 wg0   In  IP 1.2.3.4.54617 > 10.0.0.2.21: Flags [S], seq 1265491449, win 64240, options [mss 1452,nop,wscale 8,nop,nop,sackOK], length 0
1:02:36.675980 docker0 Out IP 1.2.3.4.54617 > 172.16.0.2.21: Flags [S], seq 1265491449, win 64240, options [mss 1452,nop,wscale 8,nop,nop,sackOK], length 0
1:02:36.676030 docker0 In  IP 172.16.0.2.21 > 1.2.3.4.54617: Flags [S.], seq 1815055360, ack 1265491450, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
1:02:36.676033 enp41s0 Out IP 10.0.0.2.21 > 1.2.3.4.54617: Flags [S.], seq 1815055360, ack 1265491450, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0

• 1.2.3.4 - original IP
• 10.0.0.2 - IP wireguard on B server
• 172.16.0.2 - docker network, let's pretend this is an application (everything works there correctly)

Unfortunately, I have not come up with a solution to this problem, so I am asking for your help. Is it still possible? If yes, by what means?

Update

I decided to use HAProxy, but I don't think that's a very high-performance solution. So I'm still in need of possible solutions to this problem.

Use systemd-networkd to configure the WireGuard tunnel:

# sudo cat /etc/systemd/network/99-wg0.netdev

[NetDev]
Name=wg0
Kind=wireguard
Description=WireGuard tunnel wg0

[WireGuard]
ListenPort=51820
PrivateKey=[key]

[WireGuardPeer]
PublicKey=[key]
PresharedKey=[key]
AllowedIPs=0.0.0.0/0
Endpoint=[server A]:51820

# sudo cat /etc/systemd/network/99-wg0.network

[Match]
Name=wg0

[Network]
Address=10.0.0.2/24
Address=fdc9:281f:04d7:9ee9::2/64

# sudo ip rule:

0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

nftables configurations:

# sudo nft list ruleset         
                                                                                                                                                               [0]
table inet filter {
        chain allow {
                ct state invalid drop comment "early drop of invalid connections"
                ct state { established, related } accept comment "allow tracked connections"

                ip protocol icmp accept comment "allow icmp"
                meta l4proto ipv6-icmp accept comment "allow icmp v6"

                icmp type echo-request limit rate over 10/second burst 4 packets drop comment "No ping floods"
                icmpv6 type echo-request limit rate over 10/second burst 4 packets drop comment "No ping floods"
        }

        chain wireguard {
                tcp dport 21 accept
        }

        chain input {
                type filter hook input priority filter; policy drop;

                iif "lo" accept comment "allow from loopback"

                tcp dport 22 ct state new limit rate 15/minute accept comment "Avoid brute force on SSH"
                tcp dport 22 accept comment "allow sshd"

                ip6 saddr [server A] udp dport 51820 accept comment "Accept wireguard connection from proxy1.vps-da4c9ada.ovh.zolotomc.ru"

                jump allow comment "allowed traffic for input"

                meta pkttype host limit rate 5/second counter packets 1047 bytes 43403 reject with icmpx admin-prohibited
                reject with icmpx host-unreachable
        }

        chain forward {
                type filter hook forward priority filter; policy drop;

                iif "docker0" accept comment "allow outgoing traffic from docker"

                jump allow comment "allowed traffic for forward"

                iif "wg0" jump wireguard comment "Wireguard chain"

                reject with icmpx host-unreachable
        }

        chain output {
                type filter hook output priority filter; policy accept;
        }
}

table inet nat {
        chain prerouting {
                type nat hook prerouting priority dstnat; policy accept;
                iif "wg0" jump wireguard comment "Wireguard chain"
        }

        chain wireguard {
                tcp dport 21 dnat ip to 172.16.0.2
                tcp dport 21 dnat ip6 to fe80::a8d7:f6ff:fe0b:4774
        }

        chain input {
                type nat hook input priority 100; policy accept;
        }

        chain output {
                type nat hook output priority -100; policy accept;
        }

        chain postrouting {
                type nat hook postrouting priority srcnat; policy accept;
                iif "docker0" oif != "docker0" masquerade
        }
}

I tried using Setting packet metainformation, but it reassigns the port to a new random port. And yet, I'm starting to think that it's impossible or too complicated based on just nftables rules.

Answer 1

The required information to reply correctly: "From which interface did the initial packet come from?" is lost when a different packet is seen in reply coming from a network interface. A way to memorize it and reuse it in replies is needed. Here it is: conntrack which memorizes a list of all currently tracked connections. It can also store, per flow a mark which is then called connmark. One can give a meaning to a value.

This allows to apply policy routing to a whole flow rather than just to an individual packet.

The blog To Linux and beyond ! explains it there: Netfilter Connmark.

The idea is to memorize the information when a flow is using the WireGuard interface so that reply packets are associated to the same flow, and routed back through the WireGuard interface instead of the usual route they should have taken. This can be handled by an independant nftables table and associated routing tables/rules to alter the normal fate of the packet. Using arbitrary mark value 0xf00 with the meaning "this flow came from wg0" and arbitrary routing table 1000 to select wg0 for replies.

Prepare the adequate routing table and routing rules to override the normal routes for reply packets:

ip route add default dev wg0 table 1000
ip rule add fwmark 0xf00 lookup 1000

If IPv6 is also used (but OP's configuration lacks IPv6 in AllowedIPs) then also:

ip -6 route add default dev wg0 table 1000
ip -6 rule add fwmark 0xf00 lookup 1000

example using systemd-networkd:

[Match]
Name=wg0

[Network]
Address=10.0.0.2/24
Address=fdc9:281f:04d7:9ee9::2/64

[Route]
Gateway=0.0.0.0
Table=1000

[RoutingPolicyRule]
Table=1000
FirewallMark=0xf00

And add the table below:

replywg0.nft:

table inet replywg0         # for idempotency
delete table inet replywg0  # for idempotency

table inet replywg0 {
    chain prerouting {
        type filter hook prerouting priority -150; policy accept;
        iif wg0 ct mark set 0xf00
        iif != wg0 ct mark 0xf00 meta mark set 0xf00
    }

    chain output {
        type route hook output priority -150; policy accept;
        ct mark 0xf00 meta mark set 0xf00
    }
}

load with:

nft -f replywg0.nft

---

Notes and caveats:

• the packet mark, the only one which affects routing, is not set when coming from wg0, only the connmark is set: else it would select the single route defined in table 1000 and would route back the packet from where it came instead of proceeding to the local system or to Docker containers. If routing table 1000 included all the (unknown to me) relevant additional routes, there would be no need for this special handling.

• the output chain is completely optional if that's only for Docker traffic. It would be useful only for local reply traffic: traffic received from wg0 that didn't go to Docker but to the local system. Note the use of type route instead of type filter here, so a new route lookup can still happen. Anyway some UDP services won't reply properly here: the IP source address would be the wrong one (the one set on enp41s0) and some additional imperfect NAT band-aid would be needed (for example masquerade in table inet nat output).

• Be careful if other needs for marks are put in place later (even WireGuard itself has an option that can set marks): they will likely clash if not handled together properly.

• FTP (which appears in OP's settings) is special: it uses additional connections for data and can require an ALG (usually in passive mode for the server case). As long as it's not encrypted, on Linux it can be handled with the kernel modules nf_conntrack_ftp+nf_nat_ftp and adequate settings, which are described for example there: Secure use of iptables and connection tracking helpers and have (subtly different) equivalent nftables settings. As RELATED traffic (the data traffic) inherits the connmark, the settings described in that blog would even work along the settings in this answer. The toggle nf_conntrack_helper is being completely removed from kernels >= 6.0, so this becomes a required setting.

• I'm assuming that rp_filter has not the value 1 because else probably nothing would have worked through wg0 in the first place. If it was set to 1, additional settings would be required in several places.