Skip to main content
Version: 3.18 (latest)

Install in eBPF mode

Big picture​

Install the eBPF dataplane during the initial installation of Calico Enterprise.

Value​

The eBPF dataplane mode has several advantages over standard Linux networking pipeline mode:

  • It scales to higher throughput.

  • It uses less CPU per GBit.

  • It has native support for Kubernetes services (without needing kube-proxy) that:

    • Reduces first packet latency for packets to services.
    • Preserves external client source IP addresses all the way to the pod.
    • Supports DSR (Direct Server Return) for more efficient service routing.
    • Uses less CPU than kube-proxy to keep the dataplane in sync.

To learn more and see performance metrics from our test environment, see the blog, Introducing the Calico eBPF dataplane.

Concepts​

eBPF​

eBPF (or "extended Berkeley Packet Filter"), is a technology that allows safe mini programs to be attached to various low-level hooks in the Linux kernel. eBPF has a wide variety of uses, including networking, security, and tracing. You’ll see a lot of non-networking projects leveraging eBPF, but for Calico Enterprise our focus is on networking, and in particular, pushing the networking capabilities of the latest Linux kernels to the limit.

Before you begin​

Supported architecture and versions

  • x86-64

  • Linux distribution/kernel:

    • Ubuntu 20.04 and 22.04.
    • Red Hat v8.2 with Linux kernel v4.18.0-193 or above (Red Hat have backported the required features to that build).
    • Another supported distribution with Linux kernel v5.3 or above.

  • An underlying network fabric that allows VXLAN traffic between hosts. In eBPF mode, VXLAN is used to forward Kubernetes NodePort traffic.

Unsupported platforms

  • GKE
  • MKE
  • TKG
  • RKE
note

eBPF supports AKS with Calico CNI and Calico Enterprise network policy. However, with AKS with Azure CNI and Calico Enterprise network policy, kube-proxy cannot be disabled so the performance benefits of eBPF are lost. However, there are other reasons to use eBPF other than performance gains, as described in eBPF use cases.

Unsupported features

  • Using NodeLocal DNSCache in your cluster
  • Clusters with some eBPF nodes and some standard dataplane and/or Windows nodes
  • IPv6
  • Host endpoint doNotTrack policy (other policy types are supported)
  • Floating IPs
  • SCTP (either for policy or services)
  • Log action in policy rules
  • Tagged VLAN devices
  • L7 logs
  • Application layer policies
  • Web application firewall (WAF)

Recommendations for performance

For best pod-to-pod performance, we recommend using an underlying network that doesn't require Calico Enterprise to use an overlay. For example:

  • A cluster within a single AWS subnet
  • A cluster using a compatible cloud provider's CNI (such as the AWS VPC CNI plugin)
  • An on-prem cluster with BGP peering configured

If you must use an overlay, we recommend that you use VXLAN, not IPIP. VXLAN has better performance than IPIP in eBPF mode due to various kernel optimizations.

How to​

Installing Calico Enterprise normally consists of the following steps, which are covered by the main installation guide:

  • Create a cluster suitable to run Calico Enterprise
  • Install the Tigera Operator, and the associated Custom Resource Definitions
  • Apply a set of Custom Resources to tell the operator what to install
  • Wait for the operator to provision all the associated resources and report back via its status resource

To install directly in eBPF is very similar; this guide explains the differences:

These steps are explained in more detail below.

Create a suitable cluster​

The basic requirement for eBPF mode is to have a recent-enough kernel (see supported architectures and versions).

Select the appropriate tab below for distribution-specific instructions:

kubeadm supports a number of base OSes; as long as the base OS chosen (such as Ubuntu 20.04) meets the kernel requirements, kubeadm-provisioned clusters are supported.

Since kube-proxy is not required in eBPF mode, you may wish to disable kube-proxy at install time. With kubeadm you can do that by passing the --skip-phases=addon/kube-proxy flag to kubeadm init:

kubeadm init --skip-phases=addon/kube-proxy

Create kubernetes-service-endpoint config map​

In eBPF mode, Calico Enterprise takes over from kube-proxy. This means that, like kube-proxy, it needs to be able to reach the API server directly rather than by using the API server's ClusterIP. To tell Calico Enterprise how to reach the API server we create a ConfigMap with the API server's "real" address. In this guide we do that before installing the Tigera Operator. That means that the operator itself can also use the direct connection and hence it doesn't require kube-proxy to be running.

The tabs below explain how to find the "real" address of the API server for a range of distributions. Note: In all cases it's important that the address used is stable even if your API server is restarted or scaled up/down. If you have multiple API servers, with DNS or other load balancing in front it's important to use the address of the load balancer. This prevents Calico Enterprise from being disconnected if the API servers IP changes.

If you created a cluster manually (for example by using kubeadm) then the right address to use depends on whether you opted for a high-availability cluster with multiple API servers or a simple one-node API server.

  • If you opted to set up a high availability cluster then you should use the address of the load balancer that you used in front of your API servers. As noted in the Kubernetes documentation, a load balancer is required for a HA set-up but the precise type of load balancer is not specified.
  • If you opted for a single control plane node then you can use the address of the control plane node itself. However, it's important that you use a stable address for that node such as a dedicated DNS record, or a static IP address. If you use a dynamic IP address (such as an EC2 private IP) then the address may change when the node is restarted causing Calico Enterprise to lose connectivity to the API server.

Install the Tigera Operator​

Follow the steps in the main install for your platform that installs the Tigera Operator, without applying the custom-resources.yaml (you will update this file in a later step in this doc).

For clusters in AWS, such as kOps and EKS, you must also patch the tigera-operator deployment with DNS config so the operator can resolve the apiserver DNS. AWS DNS server's address is 169.254.169.253.

kubectl patch deployment -n tigera-operator tigera-operator -p '{"spec":{"template":{"spec":{"dnsConfig":{"nameservers":["169.254.169.253"]}}}}}'

Create the Config Map​

Create the following config map in the tigera-operator namespace using the host and port determined above:

kubectl apply -f - <<EOF
kind: ConfigMap
apiVersion: v1
metadata:
  name: kubernetes-services-endpoint
  namespace: tigera-operator
data:
  KUBERNETES_SERVICE_HOST: "<API server host>"
  KUBERNETES_SERVICE_PORT: "<API server port>"
EOF

Tweak and apply installation Custom Resources​

When the main install guide tells you to apply the custom-resources.yaml, typically by running kubectl create with the URL of the file directly, you should instead download the file, so that you can edit it:

  curl -o custom-resources.yaml https://downloads.tigera.io/ee/v3.18.3/manifests/custom-resources.yaml

Edit the file in your editor of choice and find the Installation resource, which should be at the top of the file. To enable eBPF mode, we need to add a new calicoNetwork section inside the spec of the Installation resource, including the linuxDataplane field. For EKS Bottlerocket OS only, you should also add the flexVolumePath setting as shown below.

For example:

# This section includes base Calico Enterprise installation configuration.

kind: Installation
metadata:
  name: default
spec:
   # Added calicoNetwork section with linuxDataplane field
  calicoNetwork:
    linuxDataplane: BPF

   # EKS with Bottlerocket as node image only:
   # flexVolumePath: /var/lib/kubelet/plugins

   # ... remainder of the Installation resource varies by platform ...

   # Install Calico Enterprise
  variant: TigeraSecureEnterprise

Then apply the edited file:

kubectl create -f custom-resources.yaml
tip

If you already created the custom resources, you can switch your cluster over to eBPF mode by updating the installation resource. The operator will automatically roll out the change.

kubectl patch installation.operator.tigera.io default --type merge -p '{"spec":{"calicoNetwork":{"linuxDataplane":"BPF", "hostPorts":null}}}'

Monitor the progress of the installation​

You can monitor progress of the installation with the following command:

watch kubectl get tigerastatus

Disable kube-proxy (or avoid conflicts)​

In eBPF mode, to avoid conflicts with kube-proxy it's necessary to either disable kube-proxy or to configure Calico Enterprise not to clean up kube-proxy's iptables rules. If you didn't disable kube-proxy when starting your cluster then follow the steps below to avoid conflicts:

For a cluster that runs kube-proxy in a DaemonSet (such as a kubeadm-created cluster), you can disable kube-proxy, reversibly, by adding a node selector to kube-proxy's DaemonSet that matches no nodes, for example:

kubectl patch ds -n kube-system kube-proxy -p '{"spec":{"template":{"spec":{"nodeSelector":{"non-calico": "true"}}}}}'

Then, should you want to start kube-proxy again, you can simply remove the node selector.

Next steps​

Recommended

Recommended - Security