blog.rlwinm.de

IPv6 has a the concept of link scope. From IPv6's point of view a bridge interface is a single link (just like multiple hosts connected to a physical Ethernet switch), but if there are IP addresses configured on the member interfaces of a FreeBSD bridge the kernel considers these interfaces as their own links with associated link scope. This will cause IPv6 to break. The only correct configuration is to have no IP addresses configured on the member interfaces. The IP addresses belong exclusively on the bridge interface itself. The member interfaces should be treated as pure Ethernet (OSI layer 2) interfaces instead of both OSI layer 2 (Ethernet like) and OSI layer 3 (IP).

A further complication is that the bridge has to have unmodified access to the Ethernet frames, but most 1Gb/s and faster as well as virtual network interfaces have offloading features like TSO and LRO to rewrite the small (by modern standards) 1500 byte Ethernet frames into “fake” larger frames to reduce the CPU overhead of processing the packet inside each frame. While useful to IP hosts these offloading features have to be disabled to bridge Ethernet or route and filter the IP packets inside.

⚠️ Warning ⚠️ It's easy to lock yourself out of your system by migrating from using its NICs directly to adding them to a bridge. Don't continue unless you have a useable out of band console (keyboard and video console, configured serial port, IPMI SOL/KVM, etc.).

A high-level overview of the configuration.

1. Change the default behaviour of the bridge driver.
2. Prepare first bridge member interface.
3. Create the bridge interface adding its first member interface.
4. Configure the bridge interface.
5. Add or remove other member interfaces as needed.

Change the default behaviour of the bridge driver.

The best way to assign a stable MAC address to the bridge I've found it to add net.link.bridge.inherit_mac=1 to /etc/sysctl.conf and running sysctl -f /etc/sysctl.conf. This causes the bridge interfaces to inherit the MAC address of their first member interface instead of generating a badly randomised one. I find this preferable to manually assigning each bridge a stable MAC address because the bridge MAC address will be as expected by whoever provisioned the member interface (e.g. your hosting provider expecting your system to acquire its IP address via DHCP).

Prepare first bridge member interface.

The first bridge member interface connects the bridge interface itself and its other member interfaces to the network outside your FreeBSD system. Network interfaces have to be “up” to forward traffic. Most types of network interfaces become implicitly “up” by assigning IP addresses to them, but bridge member interfaces must NOT have IP addresses configured on them. The member interfaces must still be brought up. Assuming your first bridge member interface is ix1 add ifconfig_ix1="up" to /etc/rc.conf using sysrc ifconfig_ix1=up.

In the case of my test system I also had to disable the TSO offloading functionality. I carefully inspected the options=...<...> line from the ifconfig ix1 output looking for TSO and LRO offloading features to disable and added the corresponding ifconfig arguments to /etc/rc.conf like this: sysrc ifconfig_ix1+=' -tso -vlanhwtso'.

Make sure there are no other interfering ifconfig_<ifn>_* entries left in /etc/rc.conf at this point e.g. using grep ix1 /etc/rc.conf.

Create the bridge interface adding its first member interface.

The netif rc.d script handles interface configuration and in the case of cloned (pseudo-)interfaces also creates them. Use sysrc cloned_interfaces+=bridge0 to add bridge0 to the list of interfaces to be cloned. The rc.d script can also pass further arguments to ifconfig invocation used to create the interfaces. These are taken from the create_args_<ifn> variables. Bridge interfaces don't default to automatically generate a link-local address required for correct IPv6 operation from its MAC address. Run sysrc create_args_bridge0='inet6 auto_linklocal -ifdisabled addm ix1' to prepare the bridge for IPv6 and add its first member interface as early as possible.

Configure the bridge interface.

Configure IPv4 through the ifconfig_bridge0 entry /etc/rc.conf e.g. by running sysrc ifconfig_bridge0='up DHCP'.

Configure IPv6 through the ifconfig_bridge0_ipv6 entry in /etc/rc.conf e.g. running sysrc ifconfig_bridge0_ipv6='inet6 accept_rtadv'. It's also a good idea to enable rtsold and configure it to use the bridge interface e.g. using sysrc rtsold_enable=YES rtsold_flags='-i -m bridge0'.

Add or remove other member interfaces as needed.

The bridge is now connected to the outside by its first member interface and configured as an IPv4 and IPv6 enabled network interface. You can now add additional members e.g. tap/vmnet interfaces for bhyve guests or one end of an epair for vnet enabled jails.

TL;DR: copy-pasta is my favourite food and damn the consequences

sysrc ifconfig_ix1='up -tso -vlanhwtso'
sysrc create_args_bridge0='inet6 auto_linklocal -ifdisabled addm ix1'
sysrc cloned_interfaces+='bridge0'
sysrc ifconfig_bridge0='up DHCP'
sysrc ifconfig_bridge0_ipv6='inet6 accept_rtadv'
sysrc rtsold_flags='-i -m bridge0'
sysrc rtsold_enable=YES 
shutdown -r now

Mastodon: https://bsd.network/web/@crest