Table of Contents
The network situation on 05-hzn-fsn
If you try and connect to boxfox.es from 05-hzn-fsn you notice it, uh, doesn't work, and, in the 05-hzn-fsn /etc/networking/interfaces, you might notice this scuffed weirdness:
auto enp0s31f6
iface enp0s31f6 inet static
address 138.201.22.75/26
gateway 138.201.22.65
post-up sysctl -w net.ipv4.ip_forward=1
post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p tcp -m multiport ! --dport 22,8006 -j DNAT --to 172.30.0.1
post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p udp -j DNAT --to 172.30.0.1
#iface enp0s31f6 inet6 static address 2a01:4f8:171:2d5d::2/64 gateway fe80::1
auto vmbr99
iface vmbr99 inet static
address 172.30.0.0/31
# [...]
post-up iptables -t nat -A POSTROUTING -s '172.30.0.1/31' -o enp0s31f6 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '172.30.0.1/31' -o enp0s31f6 -j MASQUERADE
(In OPNsense: Interface > Neighbours there is a manual entry for the Hetzner gateway 138.201.22.65 -> dc:38:e1:fc:2d:7b - this might be related but i forgot)
What, why?
(Ok, admittedly, I am trying to rember things I did like 3 year ago. This probably isn't entirely complete. There might be something configured elsewhere that's required to make this jank not blow up so lets just hope that doesn't break)
This is required because 05-hzn-fsn is a Proxmox host, and on a Proxmox host, each VM/CT requires it's own IP address, thus, we have a router (running OPNsense) to provide this (along with firewalling, DNS, etc)
The problem arises because it is a dedicated server, with one network interface, that we don't have access to the local terminal of, so if we tried to do the sensible approach of PCI-E passthrough'ing a network adapter, or creating a vmbr device that the host doesn't have an IP on to attach to a VM, we'd get half way through configuring it and then get locked out, because the only way we can manage it is via the network, and the first step of either of those steps is to detatch the host from the networking.
So, what we have to do is have the host have take the external IP address, but, with the provided NAT rules, forward all TCP & UDP traffic to 172.30.0.1 (the OPNsense VM) via the vmbr99 interface, which in there it is configured to forward all traffic coming in there with a source address of 172.30.0.1 to the real ethernet adapter, rewriting the source IP on the way.
This is scuffed, but it works, even if it has the down side of the host not being able to connect to itsself via it's external IP (because if you try to connect to say, port 80 on boxfox.es, it'll get resolved to its own IP, and Linux will see, 'oh you're trying to connect to this IP, which is an IP locally configured on the machine, I can handle that internally', with the connection failing because there's nothing listening on port 80 on the Debian machine itsself.