Host Settings

This page contains settings for Linux servers, these instructions are agnostic of the Linux distribution used by the servers. The commands use the ip utility from the iproute2 package. These settings do not persist across reboots. Consult the documentation from the Linux distribution for guides on how to persist the settings used on the server. For example, Netplan or Network Manager. For additional details on options and behavior, consult the kernel bonding driver documentation.

Multi-homing

ESLAG (EVPN Multi-Homing) is the recommended way to provide device redundancy. A server connects to multiple switches (up to 4) without requiring the switches to be directly connected to each other. The host must configure its ports using LACP (IEEE 802.3ad).

Server Settings

The supported server-side setting for bonding is 802.3ad, with hashing policy layer2 or layer2+3. layer2+3 transmit hashing is the recommended setting for the overwhelming majority of scenarios. To configure this in Linux:

sudo ip link add $bond_name type bond miimon 50 mode 802.3ad layer2+3
sudo ip link set $slave1_name master $bond_name
sudo ip link set $slave2_name master $bond_name

The miimon 100 indicates that the bonding driver will use miimon to determine if the link is up, and it will poll every 50 milliseconds.

layer2 or layer2+3 is the hashing mode, which controls the input into the hash function. See the kernel bonding driver documentation for additional details.

After the bond has been created, it can receive a DHCP address or be statically assigned, for example:

sudo ip addr add 10.30.10.10/24 dev bond0

To check the status of this bond:

cat /proc/net/bonding/bond0

VLANs

A Linux host is capable of receiving tagged and untagged traffic. Tagged traffic is traffic that has the VLAN (802.1q) header intact, untagged traffic has the VLAN header removed as it exits the switch.

The Open Network Fabric is capable of emitting tagged and untagged traffic. The setting to change whether traffic is tagged or untagged is on the VPCAttahcment, specifically the nativeVLAN field. The default value of this field is false, meaning the fabric will emit tagged traffic. If a port emits tagged traffic, it means that the Linux host must handle the tag. This is accomplished by creating a link layer interface on the host for the tag:

sudo ip link add link enp2s1 name enp2s1.1001 type vlan id 1001

The name enp2s1.1001 is by convention; it can be customized. The vlan in this example is 1001 and was created when the subnet was created inside the VPC.

When the nativeVLAN field is true, the switch will remove the 802.1q tag from the packet as it exits the switch. When a Linux server is attached to this port, there is no need to create an additional link layer device.

Multiple VLANs on a port

At times, it is desirable to configure a single port that emits more than one VLAN tag, which is called VLAN trunking. To create a trunk port, attach the VPC subnet to the desired port, multiple VPC subnets can be connected to a single connection object:

kubectl fabric vpc attach --vpc-subnet vpc-1/default --connection server-1--leaf-1
kubectl fabric vpc attach --vpc-subnet vpc-2/default --connection server-1--leaf-1

HostBGP container

If using HostBGP subnets, BGP should be running on the host server and an appropriate configuration should be applied. To facilitate these steps, Hedgehog provides a docker container which automatically starts FRR with a valid configuration to join the Fabric.

Warning

Hedgehog does not control host servers; the scripts in the HostBGP container do their best to generate a sane configuration and to address possible pitfalls, but they are not bulletproof, and they cannot fix issues occurring after startup. Users are encouraged to double-check container logs for warnings, to ensure that the generated configuration matches the declared intent, and to monitor the system after the initial container deployment.

As a first step, users should download the docker image from our registry:

docker pull ghcr.io/githedgehog/host-bgp

The command above downloads the latest image for the host-bgp container; alternatively, a specific version can be fetched by adding it as a suffix to the image name, e.g., using version v0.4.0 as a reference:

docker pull ghcr.io/githedgehog/host-bgp:v0.4.0

Pinning a version is recommended, so that a host reboot cannot silently pick up a different image; the examples below all refer to v0.4.0.

Note

Starting with v0.4.0, the image is published as a multi-arch manifest covering linux/amd64 and linux/arm64, so the same image name works on both x86 and ARM hosts and docker pull picks the matching variant automatically. Earlier versions are amd64-only.

The container should then be run with host networking (so that FRR can communicate with the leaves using the host's interfaces) and in privileged mode. Additionally, a few input parameters are required:

an optional ASN to use in BGP. If present, it should be the first parameter; if not specified, the container will use ASN 64999
one or more VPC subnets with their related parameters, in the format <VPC-SUBNET-NAME>:v=<VLAN>:i=<INTERFACE1>[,<ALIAS1>][:i=<INTERFACE2>[,<ALIAS2>]...]:a=<ADDRESS1>[:a=<ADDRESS2>...], where:
- <VPC-SUBNET-NAME> is just a mnemonic ID for the VPC subnet we want to attach to. It can be anything as long as it is a legal name for a route-map or prefix-list in FRR.
- v=<VLAN> is the VLAN ID to be used for the VPC; use 0 for untagged.
- i=<INTERFACE> is an interface to be used to establish a BGP unnumbered session with a Fabric leaf; if a VLAN ID was specified, a corresponding VLAN interface will be created using the provided interface as the master device, named <INTERFACE>.<VLAN> by default.
- <ALIAS> is an optional name for that VLAN interface, given after a comma separator, and used instead of the default <INTERFACE>.<VLAN>. It must be a legal interface name (alphanumeric characters, dashes and underscores, at most 15 characters) and it is only accepted when the VLAN ID is non-zero. An alias is required when the default <INTERFACE>.<VLAN> name would exceed the kernel's 15-character limit on interface names: the container refuses to start rather than truncating the name.
- a=<ADDRESS> is the Virtual IP (or VIP) to be advertised to the leaves; it should have a prefix length of /32 and be part of the subnet the host is attaching to.
an optional list of static routes (which, if present, must go after the VPC subnets), in the format --routes <PREFIX>:i=<INTERFACE>[:d=<DISTANCE>] [<PREFIX2>:i=<INTERFACE2>[:d=<DISTANCE2>]...], where:
- <PREFIX> is the destination IP prefix for the route.
- i=<INTERFACE> is the egress interface for the route.
- d=<DISTANCE> is an optional administrative distance for the route, in the range 1-255 (default 1).
an optional list of per-interface rail IPs, useful to run RoCE from the host (which, if present, must also go after the VPC subnets), in the format --roce <INTERFACE-OR-ALIAS1>:a=<ADDRESS1> [<INTERFACE-OR-ALIAS2>:a=<ADDRESS2>...], where:
- <INTERFACE-OR-ALIAS> refers to an interface declared in the VPC section. In the --roce section, reference it either by its real interface name or by its alias (if one was configured), but not using the comma form (<INTERFACE>,<ALIAS>). The interface must appear in exactly one VPC subnet, since the rail IP rides that subnet's existing BGP session, and it can carry a single rail IP.
- a=<ADDRESS> is a /32 IP address that will be configured on the enslaved VLAN interface (or the "parent" interface itself if not using VLANs) and advertised via the BGP session over that interface only, so that traffic towards it follows a single link. Note that by default Fabric leaves will only accept /32 prefixes belonging to the hostBGP subnet proper; if these per-interface addresses are not part of that, appropriate prefixes including them should be configured as hostBGPExtraPrefixes in the hostBGP subnet.

The --routes and --roce sections can appear in any order relative to each other, as long as both come after the VPC subnets.

As an example, the command might look something like this:

docker run --network=host --privileged --rm --detach --name hostbgp ghcr.io/githedgehog/host-bgp:v0.4.0 64307 vpc-01:v=1001:i=enp2s1:i=enp2s2,to-leaf2:a=10.100.34.5/32 --routes 0.0.0.0/0:i=enp2s0:d=100 --roce enp2s1:a=10.115.1.5/32 to-leaf2:a=10.115.2.7/32

Note

With the above command, any output produced by the container will not be visible from the terminal where it was started. Verify that the container is running correctly with docker ps and inspect its output with docker logs hostbgp. Note that if the container fails at startup, --rm will have already removed it and its logs along with it; in that case, drop both --detach and --rm to see the failure directly.

With the above command:

VLAN interfaces enp2s1.1001 and to-leaf2 would be created, if not already existing
BGP unnumbered sessions would be created on those same interfaces, using ASN 64307
the address 10.100.34.5/32 would be configured on the loopback of the host server and it would be advertised to the leaves
a default route via enp2s0 would be installed with administrative distance 100; this is higher than the default distance for a BGP route (20), meaning that any default route learned via BGP would take priority over this, effectively making it a "backup" route
IP address 10.115.1.5/32 would be configured on enp2s1.1001 and advertised over the corresponding BGP session
IP address 10.115.2.7/32 would be configured on to-leaf2 and advertised over the corresponding BGP session

To further modify the configuration or to troubleshoot the state of the system, an expert user can invoke the FRR CLI using the following command:

docker exec -it hostbgp vtysh

For example, one could use vtysh to verify the configuration generated with the above command:

$ docker exec -t hostbgp vtysh -c "show run"
Building configuration...

Current configuration:
!
frr version 10.5.1_git
frr defaults traditional
hostname server-04
service integrated-vtysh-config
!
ip prefix-list vpc-01 seq 5 permit 10.100.34.5/32
ip prefix-list roce-enp2s1-v1001 seq 5 permit 10.115.1.5/32
ip prefix-list roce-to-leaf2-v1001 seq 5 permit 10.115.2.7/32
!
route-map vpc-01 permit 10
 match ip address prefix-list vpc-01
exit
!
route-map roce-enp2s1-v1001 permit 10
 match ip address prefix-list vpc-01
exit
!
route-map roce-enp2s1-v1001 permit 20
 match ip address prefix-list roce-enp2s1-v1001
exit
!
route-map roce-to-leaf2-v1001 permit 10
 match ip address prefix-list vpc-01
exit
!
route-map roce-to-leaf2-v1001 permit 20
 match ip address prefix-list roce-to-leaf2-v1001
exit
!
ip route 0.0.0.0/0 enp2s0 100
!
interface enp2s1.1001
 ip address 10.115.1.5/32
exit
!
interface lo
 ip address 10.100.34.5/32
exit
!
interface to-leaf2
 ip address 10.115.2.7/32
exit
!
router bgp 64307
 no bgp ebgp-requires-policy
 bgp bestpath as-path multipath-relax
 timers bgp 3 9
 neighbor enp2s1.1001 interface remote-as external
 neighbor to-leaf2 interface remote-as external
 !
 address-family ipv4 unicast
  network 10.100.34.5/32
  network 10.115.1.5/32
  network 10.115.2.7/32
  neighbor enp2s1.1001 route-map roce-enp2s1-v1001 out
  neighbor to-leaf2 route-map roce-to-leaf2-v1001 out
  maximum-paths 4
 exit-address-family
exit
!
end

To stop the container, just run the following command:

docker stop -t 1 hostbgp

Note that stopping the docker container does not currently remove the VIPs from the loopback or any static route configured, nor does it delete the VLAN interfaces. If needed, these should be removed manually; for example, using iproute2 and the reference command above, one could run:

sudo ip route delete 0.0.0.0/0 dev enp2s0
sudo ip address delete dev lo 10.100.34.5/32
sudo ip link delete dev enp2s1.1001
sudo ip link delete dev to-leaf2

Users should consider automating the startup of the hostbgp container at system boot up, to make sure that connectivity is restored in case of a reboot.

Coexistence with systemd-networkd

The hostbgp container runs with --network=host, which means that any route installed by BGP will be visible in the host networking stack. If the server where the container runs is managed by systemd-networkd, it is crucial to ensure that those routes are not removed by systemd. For this reason, users running systemd version >=246 should disable the management of foreign routes, e.g. by creating a drop-in file such as /etc/systemd/networkd.conf.d/10-systemd-frr-coexist.conf with the following content:

[Network]
ManageForeignRoutes=no

Example: multi-VPC multi-homed server

Let's assume that server-03 is attached to both leaf-01 and leaf-02 with unbundled connections server-03--unbundled--leaf-01 and server-03--unbundled--leaf-02, and that we want it to be part of two separate VPCs using host-BGP. We can create the VPCs and attachments e.g. from the control node using the Fabric kubectl plugin:

core@control-1 ~ $ kubectl fabric vpc create --name=vpc-01 --subnet=10.0.1.0/24 --vlan=1001 --host-bgp=true
10:04:09 INF VPC created name=vpc-01
core@control-1 ~ $ kubectl fabric vpc create --name=vpc-02 --subnet=10.0.2.0/24 --vlan=1002 --host-bgp=true
10:04:24 INF VPC created name=vpc-02
core@control-1 ~ $ kubectl fabric vpc attach --name=s3-v1-l1 --conn=server-03--unbundled--leaf-01 --subnet=vpc-01/default
10:05:59 INF VPCAttachment created name=s3-v1-l1
core@control-1 ~ $ kubectl fabric vpc attach --name=s3-v1-l2 --conn=server-03--unbundled--leaf-02 --subnet=vpc-01/default
10:06:08 INF VPCAttachment created name=s3-v1-l2
core@control-1 ~ $ kubectl fabric vpc attach --name=s3-v2-l1 --conn=server-03--unbundled--leaf-01 --subnet=vpc-02/default
10:06:24 INF VPCAttachment created name=s3-v2-l1
core@control-1 ~ $ kubectl fabric vpc attach --name=s3-v2-l2 --conn=server-03--unbundled--leaf-02 --subnet=vpc-02/default
10:06:33 INF VPCAttachment created name=s3-v2-l2

Then we can configure server-03 using the provided container:

docker run --network=host --privileged --rm --detach --name hostbgp ghcr.io/githedgehog/host-bgp:v0.4.0 vpc-01:v=1001:i=enp2s1:i=enp2s2:a=10.0.1.3/32 vpc-02:v=1002:i=enp2s1:i=enp2s2:a=10.0.2.3/32

This will generate the following FRR configuration:

!
ip prefix-list vpc-01 seq 5 permit 10.0.1.3/32
ip prefix-list vpc-02 seq 5 permit 10.0.2.3/32
!
route-map vpc-01 permit 10
 match ip address prefix-list vpc-01
exit
!
route-map vpc-02 permit 10
 match ip address prefix-list vpc-02
exit
!
interface lo
 ip address 10.0.1.3/32
 ip address 10.0.2.3/32
exit
!
router bgp 64999
 no bgp ebgp-requires-policy
 bgp bestpath as-path multipath-relax
 timers bgp 3 9
 neighbor enp2s1.1001 interface remote-as external
 neighbor enp2s1.1002 interface remote-as external
 neighbor enp2s2.1001 interface remote-as external
 neighbor enp2s2.1002 interface remote-as external
 !
 address-family ipv4 unicast
  network 10.0.1.3/32
  network 10.0.2.3/32
  neighbor enp2s1.1001 route-map vpc-01 out
  neighbor enp2s1.1002 route-map vpc-02 out
  neighbor enp2s2.1001 route-map vpc-01 out
  neighbor enp2s2.1002 route-map vpc-02 out
  maximum-paths 4
 exit-address-family
exit
!

And we can verify on either of the leaves attached to server-03 that VIPs are only learned in the VPC they belong to:

leaf-01# show ip route vrf VrfVvpc-01
Codes:  K - kernel route, C - connected, S - static, B - BGP, O - OSPF, A - attached-host
        > - selected route, * - FIB route, q - queued route, r - rejected route, b - backup
       Destination        Gateway                                                                    Dist/Metric   Last Update
--------------------------------------------------------------------------------------------------------------------------------
 B>*   10.0.1.3/32        via fe80::e20:12ff:fefe:401     Ethernet1.1001                             20/0          00:09:43 ago
leaf-01# show ip route vrf VrfVvpc-02
Codes:  K - kernel route, C - connected, S - static, B - BGP, O - OSPF, A - attached-host
        > - selected route, * - FIB route, q - queued route, r - rejected route, b - backup
       Destination        Gateway                                                                    Dist/Metric   Last Update
--------------------------------------------------------------------------------------------------------------------------------
 B>*   10.0.2.3/32        via fe80::e20:12ff:fefe:401     Ethernet1.1002                             20/0          00:09:47 ago
leaf-01#