Skip to main content

Configure multiple Calico Cloud networks on a pod

Big picture

Configure a Kubernetes cluster with multiple Calico Cloud networks on each pod, and enforce security using Calico Cloud tiered network policy.

Value

By default, you can configure only one CNI (network and pod interface) in a cluster. But many deployments require multiple networks (for example, one that is faster or more secure) for sending different types of data. Calico Cloud supports configuring additional Calico Cloud networks and interfaces in your pods using the Multus-CNI plugin. You can then use Calico Cloud tiered policy and other features to enforce security on all of your workload traffic.

Concepts

About the Multus-CNI plugin

Calico Cloud uses the Multus-CNI plugin to create multiple Calico Cloud networks and multiple pod interfaces to access these networks. This extends the default network and pod interface that comes with the Calico CNI.

You install Multus on a cluster, then simply enable Multus in the Calico Cloud Installation resource. Using the Multus NetworkAttachmentDefinition, you define the new networks and reference them as an annotation in the pod resource.

Labels, workload endpoints, and policy

When you set the MultiInterfaceMode field to Multus in the Installation resource, the following network and network interface labels are automatically added to new workload endpoints.

  • projectcalico.org/network
  • projectcalico.org/network-namespace
  • projectcalico.org/network-interface

You can then create Calico Cloud policies using these label selectors to target specific networks or network interfaces.

Limitations

Maximum additional networks per pod

You can define a maximum of nine additional Calico Cloud networks on a pod. If you add a network that exceeds the limit for the pod, networking is not configured and the pod fails to start with an associated error.

Calico Cloud features

Although the following Calico Cloud features are supported for your default Calico Cloud network, they are not supported at this time for additional networks/network interfaces using Multus:

  • Floating IPs
  • Specific IPs
  • Specifying IP pools on a per-namespace or per-pod basis
  • Egress gateways

Before you begin...

Required

  • Calico CNI

    note

    Verify that you are using the Calico Cloud CNI. The CNI plugin used by Kubernetes for AKS, EKS, and GKE may be different, which means this feature will not work.

  • Install Multus 3.0+ on your Kubernetes cluster

    note

    Multus is installed on OpenShift 4.0+ clusters.

How to

  1. Configure cluster for multiple networks
  2. Create a new network
  3. Create a pod interface for the new network
  4. Configure the IP pool for the network
  5. Enforce policy on the new network and pod interface
  6. View workload endpoints

Configure cluster for multiple networks

In the Installation custom resource, set the MultiInterfaceMode to Multus.

Create a new network

Create a new network using the Multus NetworkAttachmentDefinition, and set the following required field to "type":"calico".

apiVersion: 'k8s.cni.cncf.io/v1'
kind: NetworkAttachmentDefinition
metadata:
name: additional-calico-network
spec:
config: '{
"cniVersion": "0.3.1",
"type": "calico",
"log_level": "info",
"datastore_type": "kubernetes",
"mtu": 1410,
"nodename_file_optional": false,
"ipam": {
"type": "calico-ipam",
"assign_ipv4" : "true",
"assign_ipv6" : "false"
},
"policy": {
"type": "k8s"
},
"kubernetes": {
"kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
}
}'

Create a pod interface for the new network

Create a pod interface that specifies the new network using an annotation.

In the following example, we create a pod with an additional pod interface named, cali1. The pod interface is attached to the network named, additional-calico-network, using the k8s.v1.cni.cncf.io/networks annotation. Note that all networks in k8s.v1.cni.cncf.io/networks are assumed to be Calico Cloud networks.

apiVersion: v1
kind: Pod
metadata:
name: multus-test-pod-1
namespace: default
annotations:
k8s.v1.cni.cncf.io/networks: additional-calico-network@cali1
spec:
nodeSelector:
kubernetes.io/os: linux
containers:
- name: multus-test
command: ['/bin/sh', '-c', 'trap : TERM INT; sleep infinity & wait']
image: alpine

Configure the IP pool for the network

Although not required, you may want to assign IPs from specific pools to specific network interfaces. If you are using the Calico Cloud IPAM plugin, specify the IP pools in the NetworkAttachmentDefinition custom resource. For example:

 "ipam": {
"type": "calico-ipam",
"assign_ipv4" : "true",
"assign_ipv6" : "false"
"ipv4_pools": ["10.0.0.0/24", "20.0.0.0/16", "default-ipv4-ippool"],
},

Enforce policy on the new network and pod interface

When MultiInterfaceMode is set to Multus, WorkloadEndpoints are created with these labels:

  • projectcalico.org/network
  • projectcalico.org/network-namespace
  • projectcalico.org/network-interface

You can use these labels to enforce policies on specific interfaces and networks using policy label selectors.

note

Prior to Calico Cloud 3.0, if you were using Kubernetes datastore (kdd mode), the workload endpoint field and name suffix were always eth0. In 3.0, the value for workload labels may not be what you expect. Before creating policies targeting WorkloadEndpoints using the new labels, you should verify label values using the commands in View workload endpoints.

In this policy example, we use the selector field to target all WorkloadEndpoints with the network interface of, cali1.

apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
name: internal-access.allow-tcp-6379
namespace: production
spec:
tier: internal-access
selector: projectcalico.org/network-interface == cali1
types:
- Ingress
- Egress
ingress:
- action: Allow
metadata:
annotations:
from: frontend
to: database
protocol: TCP
source:
selector: role == 'frontend'
destination:
ports:
- 6379
egress:
- action: Allow

View workload endpoints

In the Calico Cloud Manager UI, go to the WorkloadEndpoint page to see all of the WorkloadEndpoints, including the network labels are for targeting WorkloadEndpoints with policy.

Using the CLI...

To view all WorkloadEndpoints for pods (default and new), use the following command.

MULTI_INTERFACE_MODE=multus calicoctl get workloadendpoints -o wide
NAME                                                                 WORKLOAD            NODE                         NETWORKS            INTERFACE         PROFILES                          NATS
test--bo--72vg--kadm--infra--0-k8s-multus--test--pod--1-eth0 multus-test-pod-1 bryan-bo-72vg-kadm-infra-0 192.168.53.129/32 calif887e436e8b kns.default,ksa.default.default
test--bo--72vg--kadm--infra--0-k8s-multus--test--pod--1-net1 multus-test-pod-1 bryan-bo-72vg-kadm-infra-0 192.168.53.140/32 calim17CD6INXIX kns.default,ksa.default.default
test--bo--72vg--kadm--infra--0-k8s-multus--test--pod--1-testiface multus-test-pod-1 bryan-bo-72vg-kadm-infra-0 192.168.53.142/32 calim27CD6INXIX kns.default,ksa.default.default
test--bo--72vg--kadm--infra--0-k8s-multus--test--pod--1-net3 multus-test-pod-1 bryan-bo-72vg-kadm-infra-0 192.168.52.143/32 calim37CD6INXIX kns.default,ksa.default.default

To view specific WorkloadEndpoints, use the following command.

MULTI_INTERFACE_MODE=multus calicoctl get workloadendpoint test--bz--72vg--kadm--infra--0-k8s-multus--test--pod--1-net1 -o yaml
apiVersion: projectcalico.org/v3
kind: WorkloadEndpoint
metadata:
creationTimestamp: '2020-05-04T22:23:05T'
labels:
projectcalico.org/namespace: default
projectcalico.org/network: calico
projectcalico.org/network-interface: net1
projectcalico.org/network-namespace: default
projectcalico.org/orchestrator: k8s
projectcalico.org/serviceaccount: default
name: test--bz--72vg--kadm--infra--0-k8s-multus--test--pod--1-net1
namespace: default
resourceVersion: '73572'
uid: b9bb7482-cdb8-48d4-9ae5-58322d48391a
spec:
endpoint: net1
interfaceName: calim16CD6INXIX
ipNetworks:
- 192.168.52.141/32
node: bryan-bo-72vg-kadm-infra-0
orchestrator: k8s
pod: multus-test-pod-1
profiles:
- kns.default
- ksa.default.default