Configure egress traffic to multiple external networks

Big Picture

Allows workloads from different namespaces of a Kubernetes cluster to egress onto different external networks that (may) have overlapping IPs with each other.

Value

When traffic from particular applications leaves the cluster to access an external destination, egress gateway enables users to control the source IP of that traffic. However, in this case, egress gateway sees all the external destinations as a flat L3 network. If a user has a network topology which involves multiple external networks and the service endpoints in those networks may have overlapping IPs, it becomes a mandatory requirement that Calico Cloud should be able to direct the egress traffic to the service endpoint in the correct network.

For example, suppose a user has a network setup as below: There are two external networks (Red and Yellow) outside the cluster. A web server on network Red is exposing its service at 105.8.10.1. A database server on network Yellow is exposing its service at 105.8.10.1 too.

Then the cluster could have requirements as follows: Pods in Red namespace should send traffic via egress gateways to 105.8.10.1 on external network Red. Pods in Yellow namespace should send traffic via egress gateways to 105.8.10.1 on external network Yellow.

External network support is introduced into Calico Cloud to meet those requirements. It allows user to associate a egress gateway to external networks so that the egress traffic originated from the client via that egress gateway will be routed to the service endpoints on the associated external networks. In a nutshell, the external network feature adds support for L3 segmentation of outgoing egress traffic.

Prerequisites

• IPs on multiple external networks may overlap with each other, however, those IPs must not overlap with the IPs of pods within the cluster or their hosts.
• IPs of the BGP peers peering with the cluster should not overlap.

Limitations

• This feature should be used in combination with egress gateways. Sending traffic from a client pod to external networks without going through an egress gateway is not supported.
• This feature only supports IPv4.

Before you begin

This feature requires you to understand how to configure a functioning egress gateway deployment in a cluster. For more information on deploying egress gateways, see our other egress gateway guides.

Concepts

Egress node

A node in the cluster which is peering with external networks and populating routes to direct traffic from the egress gateway to the destinations on external networks. Any egress gateway which is associated to an external network should be scheduled to be running on a egress node.

BGP bootstrap routes

BGP bootstrapping routes are routes on the node for traffic from that node to reach other nodes in the cluster, or other destinations outside the cluster, via whatever routers and external networks the user may need. This is considered outside of the scope of Calico Cloud, as both Calico Cloud and external BGP routers on external networks may require bootstrapping routes to be set up before peering with each other.

How to

Enable egress gateway support

In the default FelixConfiguration, set the egressIPSupport field to EnabledPerNamespace or EnabledPerNamespaceOrPerPod, according to the level of granularity that you need in your cluster. For support on a per-namespace basis only:

kubectl patch felixconfiguration default --type='merge' -p \
    '{"spec":{"egressIPSupport":"EnabledPerNamespace"}}'

Or to support both per-namespace and per-pod:

kubectl patch felixconfiguration default --type='merge' -p \
    '{"spec":{"egressIPSupport":"EnabledPerNamespaceOrPerPod"}}'

Enable external network support

In the default FelixConfiguration, set the externalNetworkSupport field to Enabled.

kubectl patch felixconfiguration default --type='merge' -p \
    '{"spec":{"externalNetworkSupport":"Enabled"}}'

Create external network resources

Add an ExternalNetwork resource to the cluster for each external networks. For example, to create an ExternalNetwork for red network.

kubectl create -f - <<EOF
apiVersion: projectcalico.org/v3
kind: ExternalNetwork
metadata:
  name: red
spec:
  routeTableIndex: 500
EOF

danger

Set routeTableIndex to the index of a linux kernel routing table that will be used for the routes associated with the external network. The value should be unique for each external network. The value must not be in the range of RouteTableRanges field in FelixConfiguration, as Calico Cloud uses tables in that range for various purposes. The kernel routing table index must not be used by other processes on the node. A conflict resulting in Calico Cloud using the same routing table index as a configured external network may result in network outages!

Calico Cloud automatically picks up bootstrapping routes into the routing table used by an external network if those routes are pre-configured as a directly connected L2 route by the user in the main routing table. If the main routing table does not have the bootstrapping routes, the user should pre-configure this routing table with bootstrapping routes on each egress node outside of Calico Cloud to include whatever routes are needed for Calico Cloud to reach the BGP peer.

Label egress node

Add a label, e.g. egress: true, to select multiple nodes in the cluster and mark them as egress nodes.

• This label is used to satisfy the nodeSelector of the BGPPeer representing a BGP peer belongs to an external network.
• This label is also used to enforce egress gateway pods associated external networks to be running on egress nodes with the help of setting nodeSelector field in pod spec.

Create BGP peers

For each external router on an external network, create a BGPPeer resource that peers between the egress nodes and the external router; specify the name of the network in the ExternalNetwork field of the BGPPeer resource; set nodeSelector to the label which selects egress nodes.

kubectl create -f - <<EOF
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: some.name
spec:
  nodeSelector: "egress == 'true'"
  peerIP: 172.31.11.1
  asNumber: 63400
  sourceAddress: None
  externalNetwork: red
EOF

Create egress gateways

Use an egress gateway custom resource to deploy a group of egress gateways, and use externalNetworks to associate the egress gateway pods of the deployment with one or more networks as desired. see our other egress gateway guides

kubectl apply -f - <<EOF
apiVersion: operator.tigera.io/v1
kind: EgressGateway
metadata:
  name: egress-gateway
  namespace: default
spec:
  logSeverity: "Info"
  replicas: 1
  ipPools:
  - cidr: "10.10.10.0/30"
  externalNetworks: ["red"]
# Uncomment this block to add ICMP, HTTP probes
# egressGatewayFailureDetection:
#   healthTimeoutDataStoreSeconds: 30
#   icmpProbe:
#     ips:
#     - <IP to probe>
#     - <IP to probe>
#     timeoutSeconds: 15
#     intervalSeconds: 5
#   httpProbe:
#     urls:
#     - <URL to probe>
#     - <URL to probe>
#     timeoutSeconds: 30
#     intervalSeconds: 10
  template:
    metadata:
      labels:
        egress-code: red
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      terminationGracePeriodSeconds: 0
EOF

Where:

• Where externalNetworks associates this egress gateway deployments with external network red. All traffic leaving the egress gateway to an external destination will be routed to the destination on external network red.
• Node selector egress: true forces egress gateway pods of the deployment to be scheduled on egress nodes.

danger

If multiple networks in the externalNetworkNames list (of the egress gateway) advertise the same prefix to Calico Cloud then traffic to those prefixes will flow to a non-deterministic network, which may result in hard-to-debug connectivity issues. It is the user's responsibility to make sure it will never happen.

Use egress gateways

Add annotations to namespaces or pods to configure them to use the relevant egress gateways created by the step above. Egress traffic from the application pods will be routed to destinations within the designated external networks.

Additional resources

• Troubleshooting egress gateways.