I went back to fedora and picked a more boring but practical filesystem setup.
The server has a dual port intel i226v NIC and an onboard realtek NIC. The realtek NIC has a fixed IP address and is just for management. the intel NIC has one port going to WAN and the other to LAN.
Originally, I had the WAN and LAN ports thrown into bridges on the linux host. the host doesnt try to join the WAN port by claiming an IP or doing anythign else with it. it’s just here for the OPNsense VM to jump onto and do what it wants. The LAN side was similar but also had a few VLANs I created on the host and passed to the VM as separate interfaces. the VLANs were just passed out from the host and the untagged was a bridge that the host could join.
This did not work. The bridges could not reach out to the world.
I set WAN as direct attachment type in mode bridge to the wan port side of the intel NIC and it created a macvlantap that did reach out of the machine. the LAN and VLAN side still did not work and I cannot do a vtap for that as I need containers on the host to use a few of these.
So I destroyed all the bridges and recreated the LAN. it works! recreate the vlans, WAN dies? reboot the machine, WAN works, VLANs work, but I cannot get the LAN to work as it just keeps dropping the VM’s interface from that bridge.
WHYyYYYYY???ONE!!!/??
I don’t think you want a bridge in fedora. You probably want to pass the Intel card to opnsense and let it handle all the network stuff.
As always, I don’t recommend virtualizing your router, because it’s a huge pain to fix if either it or the host breaks.
I have a failover so it’s not a risk. And I cannot pass the intel card to the VM because the passthrough on this motherboard only works on GPU and one of the NVMe slots. The goals of this build was to put everything in one box and get the fuck off the cloud.
I’d still just give it the interfaces and let it do all the network stuff.
Okay I did that with WAN and that can work there but it’s not possible to do that with LAN because the containers need to be able to talk to those networks too.