Configure BGP peering
We have configured Calico to distribute routing information over the Border Gateway Protocol (BGP). This scalable protocol powers routing on the global public Internet.
In many on-premise data centers, each server connects to a top-of-rack (ToR) router operating at the IP layer (layer 3). In that situation, we would need to peer each node with its corresponding ToR router, so that the ToR learns routes to the containers. That configuration is beyond the scope of this guide.
Since we are running in an AWS VPC within a single subnet, the hosts have Ethernet (layer 2) connectivity with one another, meaning there are no routers between them. Thus, they can peer directly with each other.
On one of the nodes in your cluster where you have calicoctl installed, check the status.
sudo calicoctl node status
Result
Calico process is running.
IPv4 BGP status
+---------------+-------------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+---------------+-------------------+-------+----------+-------------+
| 172.31.40.217 | node-to-node mesh | up | 17:38:47 | Established |
| 172.31.40.30 | node-to-node mesh | up | 17:40:09 | Established |
| 172.31.45.29 | node-to-node mesh | up | 17:40:20 | Established |
| 172.31.37.123 | node-to-node mesh | up | 17:40:29 | Established |
+---------------+-------------------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
Alternatively, you can create a CalicoNodeStatus resource to get BGP session status for the node.
Notice there are four BGP sessions, one to each other node in the cluster. In a small cluster, this works well and is highly resilient. However, the total number of BGP sessions scales as the square of the number of nodes, and in a large cluster this creates a lot of overhead.
In this lab we will configure a fixed number of route reflectors. Route reflectors announce their own routes and the routes they receive from other peers. This means nodes only need to peer with the route reflectors to get all the routes in the cluster. This peering arrangement means that the number of BGP sessions scales linearly with the number of nodes.
Choose and label nodes
We will establish three route reflectors, which means we avoid a single point of failure even if we take down a route reflector node for maintenance. In a five node cluster that means that only one BGP session is not needed, since the two non-reflector nodes don't need to peer with one another, but it will save lots of overhead in a large cluster.
Choose three nodes and perform the following for each of them.
Save the node YAML.
calicoctl get node <node name> -o yaml --export > node.yaml
Edit the YAML to add
metadata:
labels:
calico-route-reflector: ''
spec:
bgp:
routeReflectorClusterID: 224.0.0.1
Reapply the YAML
calicoctl apply -f node.yaml
Configure peering
Configure all non-reflector nodes to peer with all route reflectors
calicoctl apply -f - <<EOF
kind: BGPPeer
apiVersion: projectcalico.org/v3
metadata:
name: peer-to-rrs
spec:
nodeSelector: "!has(calico-route-reflector)"
peerSelector: has(calico-route-reflector)
EOF
Configure all route reflectors to peer with each other
calicoctl apply -f - <<EOF
kind: BGPPeer
apiVersion: projectcalico.org/v3
metadata:
name: rrs-to-rrs
spec:
nodeSelector: has(calico-route-reflector)
peerSelector: has(calico-route-reflector)
EOF
Disable the node-to-node mesh
calicoctl create -f - <<EOF
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
name: default
spec:
nodeToNodeMeshEnabled: false
asNumber: 64512
EOF
On a non-reflector node, you should now see only three peerings.
sudo calicoctl node status
Result
Calico process is running.
IPv4 BGP status
+---------------+---------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+---------------+---------------+-------+----------+-------------+
| 172.31.37.123 | node specific | up | 21:52:57 | Established |
| 172.31.40.217 | node specific | up | 21:52:57 | Established |
| 172.31.42.47 | node specific | up | 21:52:57 | Established |
+---------------+---------------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
Alternatively, you can create a CalicoNodeStatus resource to get BGP session status for the node.