Calico and MetalLB working together with BGP

When we want to make Calico and MetalLB working together with BGP, we’ll run into the following issue: BGP only allows one session to be established from the same source IP. So, if Calico has a session established with your BGP router, MetalLB cannot establish its own session – it’ll get rejected as a duplicate by BGP’s conflict resolution algorithm. Then, in this post, I will show you a way to workaround this using dummy interfaces

MetalLB is an opensource project of a load-balancer implementation for Kubernetes clusters. Public clouds are offering that service already from their own Kubernetes, like Amazon EKS, using their own cloud multi-purpose load-balancer service ELB. However, if you have your own Kubernetes working on-premises, you can install MetalLB and deploy those services directly from Kubernetes APIs, like in the Cloud.

Calico is a CNI plugin for Kubernetes and creates a flat layer-3 network and assigns a fully routable IP address to every pod (/32 prefixes).

I wanted to keep using the ToR (Top of Rack) routers for the LoadBalancer instead of Spines as it’s normally proposed. Basically, because if you use overlay networks in a Fabric, Spines could be managed as route reflectors for EVPN-iBGP and transport for VxLANs.

Solution: I used a dummy interface in every Kubernetes node as source-address for MetalLB

My Setup is using Calico and MetalLB working together with BGP in Kubernetes v1.20.5. My Cluster was installed with kubeadm with control-planes/master redundancy on Fedora 34 (Kernel version 5.13.4-200). I am using cri-o as my container runtime.

[root@ctl-a1 ~]# kubectl get nodes -o wide
NAME     STATUS   ROLES                  AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                    KERNEL-VERSION           CONTAINER-RUNTIME
ctl-a1   Ready    control-plane,master   12d   v1.20.5   192.168.101.30   <none>        Fedora 34 (Cloud Edition)   5.13.4-200.fc34.x86_64   cri-o://1.20.0
ctl-a2   Ready    control-plane,master   12d   v1.20.5   192.168.101.31   <none>        Fedora 34 (Cloud Edition)   5.13.4-200.fc34.x86_64   cri-o://1.20.0
ctl-a3   Ready    control-plane,master   12d   v1.20.5   192.168.101.32   <none>        Fedora 34 (Cloud Edition)   5.13.4-200.fc34.x86_64   cri-o://1.20.0
wk-a1    Ready    <none>                 12d   v1.20.5   192.168.101.20   <none>        Fedora 34 (Cloud Edition)   5.13.4-200.fc34.x86_64   cri-o://1.20.0
wk-a2    Ready    <none>                 12d   v1.20.5   192.168.101.21   <none>        Fedora 34 (Cloud Edition)   5.13.4-200.fc34.x86_64   cri-o://1.20.0
wk-a3    Ready    <none>                 12d   v1.20.5   192.168.101.22   <none>        Fedora 34 (Cloud Edition)   5.13.4-200.fc34.x86_64   cri-o://1.20.0

This cluster was installed using this procedure at “On-premises Fedora34 K8s HA Cluster with kubeadm and Calico

Servers are connected to two different EVPN domains (or subnets) using VLANs via the interface eth1. I have also created a dummy interface as additional loopback to be used as source-address in metalllb BGP peering.

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:12:15:57 brd ff:ff:ff:ff:ff:ff
    altname enp0s4
    altname ens4
4: VLAN-10@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:12:15:57 brd ff:ff:ff:ff:ff:ff
    inet 192.168.101.30/24 brd 192.168.101.255 scope global noprefixroute VLAN-10
       valid_lft forever preferred_lft forever
    inet6 fe80::f9de:e5e3:ddee:5184/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
5: VLAN-20@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:12:15:57 brd ff:ff:ff:ff:ff:ff
    inet 172.16.101.30/24 brd 172.16.101.255 scope global noprefixroute VLAN-20
       valid_lft forever preferred_lft forever
    inet6 fe80::2ef6:658f:a7e0:3963/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
14: dummy: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 46:cc:d1:82:d4:64 brd ff:ff:ff:ff:ff:ff
    inet 6.4.5.30/32 scope global noprefixroute dummy
       valid_lft forever preferred_lft forever
    inet6 fe80::9d64:7bc8:a43c:4322/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

The dummy interface was created manually as follow:

nmcli connection add type dummy ifname dummy ipv4.method manual ipv4.addresses 6.4.5.30/32

The same dummy interface has been created in all k8s nodes.

# for i in {20,21,22,30,31,32}; do ssh root@172.20.20.$i ifconfig dummy | head -2 ; done
dummy: flags=195<UP,BROADCAST,RUNNING,NOARP>  mtu 1500
        inet 6.4.5.20  netmask 255.255.255.255  broadcast 0.0.0.0
dummy: flags=195<UP,BROADCAST,RUNNING,NOARP>  mtu 1500
        inet 6.4.5.21  netmask 255.255.255.255  broadcast 0.0.0.0
dummy: flags=195<UP,BROADCAST,RUNNING,NOARP>  mtu 1500
        inet 6.4.5.22  netmask 255.255.255.255  broadcast 0.0.0.0
dummy: flags=195<UP,BROADCAST,RUNNING,NOARP>  mtu 1500
        inet 6.4.5.30  netmask 255.255.255.255  broadcast 0.0.0.0
dummy: flags=195<UP,BROADCAST,RUNNING,NOARP>  mtu 1500
        inet 6.4.5.31  netmask 255.255.255.255  broadcast 0.0.0.0
dummy: flags=195<UP,BROADCAST,RUNNING,NOARP>  mtu 1500
        inet 6.4.5.32  netmask 255.255.255.255  broadcast 0.0.0.0

SRL Spine/Leaf Topology

In this case I am using Nokia SRL via Containerlab for Calico and MetalLB working together with BGP. You can see my config for every router in my repo

Next picture shows the eBGP underlay setup with Nokia SRL Spine/Leaf

Next picture shows overlay BGP-EVPN setup with Nokia SRL Spine/Leaf

Next picture shows Calico BGP setup to Nokia SRL Spine/Leaf Overlay VRF

Next picture shows Metallb BGP setup to Nokia SRL Spine/Leaf Overlay VRF

Calico BGP Setup

In order to make Calico and MetalLB working together with BGP, let’s start setting up Calico:

After installing calico using the manifest, then have to setup the BGP configuration as follow

apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  nodeToNodeMeshEnabled: false
  serviceClusterIPs:
  - cidr: 10.96.0.0/12

And now you set up the BGP peers for Calico via manifest

apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: leaf1-tor
spec:
  peerIP: 6.5.3.1
  asNumber: 65310
  nodeSelector: leaf == 'leaf1'
  sourceAddress: "None"
---
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: leaf2-tor
spec:
  peerIP: 6.5.3.2
  asNumber: 65320
  nodeSelector: leaf == 'leaf2'
  sourceAddress: "None"

And wait for the neighbor to establish connection to the router. After that, you should be ok to start using your pods.

Multus and calico

Multus is a CNI meta-plugin. In this case I’ve had also install multus. Check the used manifest for

multus installation for cri-o.

Installation is very simple. Just apply the manifest and you are done. Multus won’t impact any configuration in Calico. It’s just there to add more CNI plugins, like MACVLAN, and use them to add additional ports to pods. However, this post won’t cover anything else related to multus. I will bring something later in other post.

Metallb BGP Setup

After you have installed Calico. You can start installing and setting up Metallb.

For the installation you just need to install the manifests

kubectl apply -f metallb-install-namespace.yaml
kubectl apply -f metallb-install-main-image.yaml

“metallb-install-main-image.yaml” is the same manifest provided in https://metallb.universe.tf/installation/ but with metallb/controller:main and metallb/speaker:main images that have the latest changes. In this case, we need source-address option. More details of this option in Make BGP source address configurable thread.

In this case, we are using the dummy interface as the source-address to avoid conflicts with Calico.

Now, you can setup metallb BGP peers and IP Pool with the following manifest

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    peers:
    - peer-address: 6.5.3.1
      peer-asn: 65310
      my-asn: 65201
      router-id: 6.4.5.20
      source-address: 6.4.5.20
      node-selectors:
      - match-expressions:
        - key: kubernetes.io/hostname
          operator: In
          values: [wk-a1]
    - peer-address: 6.5.3.2
      peer-asn: 65320
      my-asn: 65201
      router-id: 6.4.5.21
      source-address: 6.4.5.21
      node-selectors:
      - match-expressions:
        - key: kubernetes.io/hostname
          operator: In
          values: [wk-a2]
    - peer-address: 6.5.3.1
      peer-asn: 65310
      my-asn: 65201
      router-id: 6.4.5.22
      source-address: 6.4.5.22
      node-selectors:
      - match-expressions:
        - key: kubernetes.io/hostname
          operator: In
          values: [wk-a3]
    - peer-address: 6.5.3.2
      peer-asn: 65320
      my-asn: 65201
      router-id: 6.4.5.30
      source-address: 6.4.5.30
      node-selectors:
      - match-expressions:
        - key: kubernetes.io/hostname
          operator: In
          values: [ctl-a1]
    - peer-address: 6.5.3.1
      peer-asn: 65310
      my-asn: 65201
      router-id: 6.4.5.31
      source-address: 6.4.5.31
      node-selectors:
      - match-expressions:
        - key: kubernetes.io/hostname
          operator: In
          values: [ctl-a2]
    - peer-address: 6.5.3.2
      peer-asn: 65320
      my-asn: 65201
      router-id: 6.4.5.32
      source-address: 6.4.5.32
      node-selectors:
      - match-expressions:
        - key: kubernetes.io/hostname
          operator: In
          values: [ctl-a3]
    address-pools:
    - name: default
      protocol: bgp
      addresses:
      - 10.254.254.240/28

And we are done.

Sharing results

Here you have the output of one of my leaf routers showing how we made Calico and MetalLB working together with BGP

A:leaf1# show network-instance ip-vrf1 protocols bgp neighbor
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BGP neighbor summary for network-instance "ip-vrf1"
Flags: S static, D dynamic, L discovered by LLDP, B BFD enabled, - disabled, * slow
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+------------------------+-----------------------------------+------------------------+--------+-------------+-------------------+-------------------+-----------------+-----------------------------------+
|        Net-Inst        |               Peer                |         Group          | Flags  |   Peer-AS   |       State       |      Uptime       |    AFI/SAFI     |          [Rx/Active/Tx]           |
+========================+===================================+========================+========+=============+===================+===================+=================+===================================+
| ip-vrf1                | 6.4.5.20                          | metallb-bgp            | D      | 65201       | established       | 4d:23h:43m:18s    | ipv4-unicast    | [4/0/33]                          |
| ip-vrf1                | 6.4.5.22                          | metallb-bgp            | D      | 65201       | established       | 4d:23h:43m:18s    | ipv4-unicast    | [3/0/33]                          |
| ip-vrf1                | 6.4.5.31                          | metallb-bgp            | D      | 65201       | established       | 4d:23h:43m:17s    | ipv4-unicast    | [1/0/33]                          |
| ip-vrf1                | 192.168.101.20                    | calico-bgp             | D      | 64512       | established       | 6d:21h:23m:49s    | ipv4-unicast    | [5/1/10]                          |
| ip-vrf1                | 192.168.101.22                    | calico-bgp             | D      | 64512       | established       | 6d:21h:24m:4s     | ipv4-unicast    | [4/1/10]                          |
| ip-vrf1                | 192.168.101.31                    | calico-bgp             | D      | 64512       | established       | 6d:21h:23m:47s    | ipv4-unicast    | [2/1/10]                          |
+------------------------+-----------------------------------+------------------------+--------+-------------+-------------------+-------------------+-----------------+-----------------------------------+
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Summary:
0 configured neighbors, 0 configured sessions are established,0 disabled peers
6 dynamic peersA:leaf1# show network-instance ip-vrf1 protocols bgp neighbor
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BGP neighbor summary for network-instance "ip-vrf1"
Flags: S static, D dynamic, L discovered by LLDP, B BFD enabled, - disabled, * slow
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+------------------------+-----------------------------------+------------------------+--------+-------------+-------------------+-------------------+-----------------+-----------------------------------+
|        Net-Inst        |               Peer                |         Group          | Flags  |   Peer-AS   |       State       |      Uptime       |    AFI/SAFI     |          [Rx/Active/Tx]           |
+========================+===================================+========================+========+=============+===================+===================+=================+===================================+
| ip-vrf1                | 6.4.5.20                          | metallb-bgp            | D      | 65201       | established       | 4d:23h:43m:18s    | ipv4-unicast    | [4/0/33]                          |
| ip-vrf1                | 6.4.5.22                          | metallb-bgp            | D      | 65201       | established       | 4d:23h:43m:18s    | ipv4-unicast    | [3/0/33]                          |
| ip-vrf1                | 6.4.5.31                          | metallb-bgp            | D      | 65201       | established       | 4d:23h:43m:17s    | ipv4-unicast    | [1/0/33]                          |
| ip-vrf1                | 192.168.101.20                    | calico-bgp             | D      | 64512       | established       | 6d:21h:23m:49s    | ipv4-unicast    | [5/1/10]                          |
| ip-vrf1                | 192.168.101.22                    | calico-bgp             | D      | 64512       | established       | 6d:21h:24m:4s     | ipv4-unicast    | [4/1/10]                          |
| ip-vrf1                | 192.168.101.31                    | calico-bgp             | D      | 64512       | established       | 6d:21h:23m:47s    | ipv4-unicast    | [2/1/10]                          |
+------------------------+-----------------------------------+------------------------+--------+-------------+-------------------+-------------------+-----------------+-----------------------------------+
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Summary:
0 configured neighbors, 0 configured sessions are established,0 disabled peers
6 dynamic peers

See you and do not forget to comment.

Leave a Reply

Blog at WordPress.com.

%d bloggers like this: