Configure multiple Calico Cloud networks on a pod

Big picture

Configure a Kubernetes cluster with multiple Calico Cloud networks on each pod, and enforce security using Calico Cloud tiered network policy.

Value

By default, you can configure only one CNI (network and pod interface) in a cluster. But many deployments require multiple networks (for example, one that is faster or more secure) for sending different types of data. Calico Cloud supports configuring additional Calico Cloud networks and interfaces in your pods using the Multus-CNI plugin. You can then use Calico Cloud tiered policy and other features to enforce security on all of your workload traffic.

Concepts

About the Multus-CNI plugin

Calico Cloud uses the Multus-CNI plugin to create multiple Calico Cloud networks and multiple pod interfaces to access these networks. This extends the default network and pod interface that comes with the Calico CNI.

You install Multus on a cluster, then simply enable Multus in the Calico Cloud Installation resource. Using the Multus NetworkAttachmentDefinition, you define the new networks and reference them as an annotation in the pod resource.

Labels, workload endpoints, and policy

When you set the MultiInterfaceMode field to Multus in the Installation resource, the following network and network interface labels are automatically added to new workload endpoints.

projectcalico.org/network
projectcalico.org/network-namespace
projectcalico.org/network-interface

You can then create Calico Cloud policies using these label selectors to target specific networks or network interfaces.

Limitations

Maximum additional networks per pod

You can define a maximum of nine additional Calico Cloud networks on a pod. If you add a network that exceeds the limit for the pod, networking is not configured and the pod fails to start with an associated error.

Calico Cloud features

Although the following Calico Cloud features are supported for your default Calico Cloud network, they are not supported at this time for additional networks/network interfaces using Multus:

Floating IPs
Specific IPs
Specifying IP pools on a per-namespace or per-pod basis
Egress gateways

Before you begin...

Required

Calico CNI
note
Verify that you are using the Calico Cloud CNI. The CNI plugin used by Kubernetes for AKS, EKS, and GKE may be different, which means this feature will not work.
Install Multus 3.0+ on your Kubernetes cluster
note
Multus is installed on OpenShift 4.0+ clusters.

How to

Configure cluster for multiple networks
Create a new network
Create a pod interface for the new network
Configure the IP pool for the network
Enforce policy on the new network and pod interface
View workload endpoints

Configure cluster for multiple networks

In the Installation custom resource, set the MultiInterfaceMode to Multus.

Create a new network

Create a new network using the Multus NetworkAttachmentDefinition, and set the following required field to "type":"calico".

apiVersion: 'k8s.cni.cncf.io/v1'
kind: NetworkAttachmentDefinition
metadata:
  name: additional-calico-network
spec:
  config: '{
    "cniVersion": "0.3.0",
    "type": "calico",
    "log_level": "info",
    "datastore_type": "kubernetes",
    "mtu": 1410,
    "nodename_file_optional": false,
    "ipam": {
    "type": "calico-ipam",
    "assign_ipv4" : "true",
    "assign_ipv6" : "false"
    },
    "policy": {
    "type": "k8s"
    },
    "kubernetes": {
    "kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
    }
    }'

Create a pod interface for the new network

Create a pod interface that specifies the new network using an annotation.

In the following example, we create a pod with an additional pod interface named, cali1. The pod interface is attached to the network named, additional-calico-network, using the k8s.v1.cni.cncf.io/networks annotation. Note that all networks in k8s.v1.cni.cncf.io/networks are assumed to be Calico Cloud networks.

apiVersion: v1
kind: Pod
metadata:
  name: multus-test-pod-1
  namespace: default
  annotations:
    k8s.v1.cni.cncf.io/networks: additional-calico-network@cali1
spec:
  nodeSelector:
    kubernetes.io/os: linux
  containers:
    - name: multus-test
      command: ['/bin/sh', '-c', 'trap : TERM INT; sleep infinity & wait']
      image: alpine

Configure the IP pool for the network

Although not required, you may want to assign IPs from specific pools to specific network interfaces. If you are using the Calico Cloud IPAM plugin, specify the IP pools in the NetworkAttachmentDefinition custom resource. For example:

 "ipam": {
     "type": "calico-ipam",
     "assign_ipv4" : "true",
     "assign_ipv6" : "false"
     "ipv4_pools": ["10.0.0.0/24", "20.0.0.0/16", "default-ipv4-ippool"],
},

Enforce policy on the new network and pod interface

When MultiInterfaceMode is set to Multus, WorkloadEndpoints are created with these labels:

projectcalico.org/network
projectcalico.org/network-namespace
projectcalico.org/network-interface

You can use these labels to enforce policies on specific interfaces and networks using policy label selectors.

note

Prior to Calico Cloud 3.0, if you were using Kubernetes datastore (kdd mode), the workload endpoint field and name suffix were always eth0. In 3.0, the value for workload labels may not be what you expect. Before creating policies targeting WorkloadEndpoints using the new labels, you should verify label values using the commands in View workload endpoints.

In this policy example, we use the selector field to target all WorkloadEndpoints with the network interface of, cali1.

apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: internal-access.allow-tcp-6379
  namespace: production
spec:
  tier: internal-access
  selector: projectcalico.org/network-interface == cali1
  types:
    - Ingress
    - Egress
  ingress:
    - action: Allow
      metadata:
        annotations:
          from: frontend
          to: database
      protocol: TCP
      source:
        selector: role == 'frontend'
      destination:
        ports:
          - 6379
  egress:
    - action: Allow

View workload endpoints

In the Calico Cloud Manager UI, go to the WorkloadEndpoint page to see all of the WorkloadEndpoints, including the network labels are for targeting WorkloadEndpoints with policy.

Using the CLI...

To view all WorkloadEndpoints for pods (default and new), use the following command.

MULTI_INTERFACE_MODE=multus calicoctl get workloadendpoints -o wide

NAME                                                                 WORKLOAD            NODE                         NETWORKS            INTERFACE         PROFILES                          NATS
test--bo--72vg--kadm--infra--0-k8s-multus--test--pod--1-eth0        multus-test-pod-1   bryan-bo-72vg-kadm-infra-0   192.168.53.129/32   calif887e436e8b   kns.default,ksa.default.default
test--bo--72vg--kadm--infra--0-k8s-multus--test--pod--1-net1        multus-test-pod-1   bryan-bo-72vg-kadm-infra-0   192.168.53.140/32   calim17CD6INXIX   kns.default,ksa.default.default
test--bo--72vg--kadm--infra--0-k8s-multus--test--pod--1-testiface   multus-test-pod-1   bryan-bo-72vg-kadm-infra-0   192.168.53.142/32   calim27CD6INXIX   kns.default,ksa.default.default
test--bo--72vg--kadm--infra--0-k8s-multus--test--pod--1-net3        multus-test-pod-1   bryan-bo-72vg-kadm-infra-0   192.168.52.143/32   calim37CD6INXIX   kns.default,ksa.default.default

To view specific WorkloadEndpoints, use the following command.

MULTI_INTERFACE_MODE=multus calicoctl get workloadendpoint test--bz--72vg--kadm--infra--0-k8s-multus--test--pod--1-net1 -o yaml

apiVersion: projectcalico.org/v3
kind: WorkloadEndpoint
metadata:
  creationTimestamp: '2020-05-04T22:23:05T'
  labels:
    projectcalico.org/namespace: default
    projectcalico.org/network: calico
    projectcalico.org/network-interface: net1
    projectcalico.org/network-namespace: default
    projectcalico.org/orchestrator: k8s
    projectcalico.org/serviceaccount: default
  name: test--bz--72vg--kadm--infra--0-k8s-multus--test--pod--1-net1
  namespace: default
  resourceVersion: '73572'
  uid: b9bb7482-cdb8-48d4-9ae5-58322d48391a
spec:
  endpoint: net1
  interfaceName: calim16CD6INXIX
  ipNetworks:
    - 192.168.52.141/32
  node: bryan-bo-72vg-kadm-infra-0
  orchestrator: k8s
  pod: multus-test-pod-1
  profiles:
    - kns.default
    - ksa.default.default

Configure multiple Calico Cloud networks on a pod

Big picture​

Value​

Concepts​

About the Multus-CNI plugin​

Labels, workload endpoints, and policy​

Limitations​

Before you begin...​

How to​

Configure cluster for multiple networks​

Create a new network​

Create a pod interface for the new network​

Configure the IP pool for the network​

Enforce policy on the new network and pod interface​

View workload endpoints​