Skip to main content
Calico Cloud documentation

Configure multiple Calico Cloud networks on a pod

Big picture​

Configure a Kubernetes cluster with multiple Calico Cloud networks on each pod, and enforce security using Calico Cloud tiered network policy.

Value​

By default, you can configure only one CNI (network and pod interface) in a cluster. But many deployments require multiple networks (for example, one that is faster or more secure) for sending different types of data. Calico Cloud supports configuring additional Calico Cloud networks and interfaces in your pods using the Multus-CNI plugin. You can then use Calico Cloud tiered policy and other features to enforce security on all of your workload traffic.

Concepts​

About the Multus-CNI plugin​

Calico Cloud uses the Multus-CNI plugin to create multiple Calico Cloud networks and multiple pod interfaces to access these networks. This extends the default network and pod interface that comes with the Calico CNI.

You install Multus on a cluster, then simply enable Multus in the Calico Cloud Installation resource. Using the Multus NetworkAttachmentDefinition, you define the new networks and reference them as an annotation in the pod resource.

Labels, workload endpoints, and policy​

When you set the MultiInterfaceMode field to Multus in the Installation resource, the following network and network interface labels are automatically added to new workload endpoints.

  • projectcalico.org/network
  • projectcalico.org/network-namespace
  • projectcalico.org/network-interface

You can then create Calico Cloud policies using these label selectors to target specific networks or network interfaces.

Limitations​

Maximum additional networks per pod

You can define a maximum of nine additional Calico Cloud networks on a pod. If you add a network that exceeds the limit for the pod, networking is not configured and the pod fails to start with an associated error.

Calico Cloud features

Although the following Calico Cloud features are supported for your default Calico Cloud network, they are not supported at this time for additional networks/network interfaces using Multus:

  • Floating IPs
  • Specific IPs
  • Specifying IP pools on a per-namespace or per-pod basis
  • Egress gateways

Before you begin...​

Required

  • Calico CNI

    note

    Verify that you are using the Calico Cloud CNI. The CNI plugin used by Kubernetes for AKS, EKS, and GKE may be different, which means this feature will not work.

  • Install Multus 3.0+ on your Kubernetes cluster

    note

    Multus is installed on OpenShift 4.0+ clusters.

How to​

  1. Configure cluster for multiple networks
  2. Create a new network
  3. Create a pod interface for the new network
  4. Configure the IP pool for the network
  5. Enforce policy on the new network and pod interface
  6. View workload endpoints

Configure cluster for multiple networks​

In the Installation custom resource, set the MultiInterfaceMode to Multus.

Create a new network​

Create a new network using the Multus NetworkAttachmentDefinition, and set the following required field to "type":"calico".

apiVersion: 'k8s.cni.cncf.io/v1'
kind: NetworkAttachmentDefinition
metadata:
  name: additional-calico-network
spec:
  config: '{
    "cniVersion": "0.3.1",
    "type": "calico",
    "log_level": "info",
    "datastore_type": "kubernetes",
    "mtu": 1410,
    "nodename_file_optional": false,
    "ipam": {
    "type": "calico-ipam",
    "assign_ipv4" : "true",
    "assign_ipv6" : "false"
    },
    "policy": {
    "type": "k8s"
    },
    "kubernetes": {
    "kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
    }
    }'

Create a pod interface for the new network​

Create a pod interface that specifies the new network using an annotation.

In the following example, we create a pod with an additional pod interface named, cali1. The pod interface is attached to the network named, additional-calico-network, using the k8s.v1.cni.cncf.io/networks annotation. Note that all networks in k8s.v1.cni.cncf.io/networks are assumed to be Calico Cloud networks.

apiVersion: v1
kind: Pod
metadata:
  name: multus-test-pod-1
  namespace: default
  annotations:
    k8s.v1.cni.cncf.io/networks: additional-calico-network@cali1
spec:
  nodeSelector:
    kubernetes.io/os: linux
  containers:
    - name: multus-test
      command: ['/bin/sh', '-c', 'trap : TERM INT; sleep infinity & wait']
      image: alpine

Configure the IP pool for the network​

Although not required, you may want to assign IPs from specific pools to specific network interfaces. If you are using the Calico Cloud IPAM plugin, specify the IP pools in the NetworkAttachmentDefinition custom resource. For example:

 "ipam": {
     "type": "calico-ipam",
     "assign_ipv4" : "true",
     "assign_ipv6" : "false"
     "ipv4_pools": ["10.0.0.0/24", "20.0.0.0/16", "default-ipv4-ippool"],
},

Enforce policy on the new network and pod interface​

When MultiInterfaceMode is set to Multus, WorkloadEndpoints are created with these labels:

  • projectcalico.org/network
  • projectcalico.org/network-namespace
  • projectcalico.org/network-interface

You can use these labels to enforce policies on specific interfaces and networks using policy label selectors.

note

Prior to Calico Cloud 3.0, if you were using Kubernetes datastore (kdd mode), the workload endpoint field and name suffix were always eth0. In 3.0, the value for workload labels may not be what you expect. Before creating policies targeting WorkloadEndpoints using the new labels, you should verify label values using the commands in View workload endpoints.

In this policy example, we use the selector field to target all WorkloadEndpoints with the network interface of, cali1.

apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: internal-access.allow-tcp-6379
  namespace: production
spec:
  tier: internal-access
  selector: projectcalico.org/network-interface == cali1
  types:
    - Ingress
    - Egress
  ingress:
    - action: Allow
      metadata:
        annotations:
          from: frontend
          to: database
      protocol: TCP
      source:
        selector: role == 'frontend'
      destination:
        ports:
          - 6379
  egress:
    - action: Allow

View workload endpoints​

In the Calico Cloud Manager UI, go to the WorkloadEndpoint page to see all of the WorkloadEndpoints, including the network labels are for targeting WorkloadEndpoints with policy.

Using the CLI...

To view all WorkloadEndpoints for pods (default and new), use the following command.

MULTI_INTERFACE_MODE=multus calicoctl get workloadendpoints -o wide
NAME                                                                 WORKLOAD            NODE                         NETWORKS            INTERFACE         PROFILES                          NATS
test--bo--72vg--kadm--infra--0-k8s-multus--test--pod--1-eth0        multus-test-pod-1   bryan-bo-72vg-kadm-infra-0   192.168.53.129/32   calif887e436e8b   kns.default,ksa.default.default
test--bo--72vg--kadm--infra--0-k8s-multus--test--pod--1-net1        multus-test-pod-1   bryan-bo-72vg-kadm-infra-0   192.168.53.140/32   calim17CD6INXIX   kns.default,ksa.default.default
test--bo--72vg--kadm--infra--0-k8s-multus--test--pod--1-testiface   multus-test-pod-1   bryan-bo-72vg-kadm-infra-0   192.168.53.142/32   calim27CD6INXIX   kns.default,ksa.default.default
test--bo--72vg--kadm--infra--0-k8s-multus--test--pod--1-net3        multus-test-pod-1   bryan-bo-72vg-kadm-infra-0   192.168.52.143/32   calim37CD6INXIX   kns.default,ksa.default.default

To view specific WorkloadEndpoints, use the following command.

MULTI_INTERFACE_MODE=multus calicoctl get workloadendpoint test--bz--72vg--kadm--infra--0-k8s-multus--test--pod--1-net1 -o yaml
apiVersion: projectcalico.org/v3
kind: WorkloadEndpoint
metadata:
  creationTimestamp: '2020-05-04T22:23:05T'
  labels:
    projectcalico.org/namespace: default
    projectcalico.org/network: calico
    projectcalico.org/network-interface: net1
    projectcalico.org/network-namespace: default
    projectcalico.org/orchestrator: k8s
    projectcalico.org/serviceaccount: default
  name: test--bz--72vg--kadm--infra--0-k8s-multus--test--pod--1-net1
  namespace: default
  resourceVersion: '73572'
  uid: b9bb7482-cdb8-48d4-9ae5-58322d48391a
spec:
  endpoint: net1
  interfaceName: calim16CD6INXIX
  ipNetworks:
    - 192.168.52.141/32
  node: bryan-bo-72vg-kadm-infra-0
  orchestrator: k8s
  pod: multus-test-pod-1
  profiles:
    - kns.default
    - ksa.default.default