Configure MTU to maximize network performance
Big picture
Configure the maximum transmission unit (MTU) for your Calico Cloud environment.
Value
Optimize network performance for workloads by configuring the MTU in Calico Cloud to best suit your underlying network.
Increasing the MTU can improve performance, and decreasing the MTU can resolve packet loss and fragmentation problems when it is too high.
Concepts
MTU and Calico Cloud defaults
The maximum transmission unit (MTU) setting determines the largest packet size that can be transmitted through your network. MTU is configured on the veth attached to each workload, and tunnel devices (if you enable IP in IP, VXLAN, or WireGuard).
In general, maximum performance is achieved by using the highest MTU value that does not cause fragmentation or dropped packets on the path. Maximum bandwidth increases and CPU consumption may drop for a given traffic rate. The improvement is often more significant when pod to pod traffic is being encapsulated (IP in IP, VXLAN, or WireGuard), and splitting and combining such traffic cannot be offloaded to your NICs.
By default, Calico Cloud will auto-detect the correct MTU for your cluster based on node configuration and enabled networking modes. This guide explains how you can override auto-detection of MTU by providing an explicit value if needed.
To ensure auto-detection of MTU works correctly, make sure that the correct encapsulation modes are set in your felix configuration. Disable any unused encapsulations (vxlanEnabled, ipipEnabled, wireguardEnabled and wireguardEnabledV6) in your felix configuration to ensure that auto-detection can pick the optimal MTU for your cluster.
Before you begin...
Required
- Calico CNI
For help on using IP in IP and/or VXLAN overlays, see Configure overlay networking.
For help on using WireGuard encryption, see Configure WireGuard encryption.
How to
Determine MTU size
The following table lists common MTU sizes for Calico Cloud environments. Because MTU is a global property of the network path between endpoints, you should set the MTU to the minimum MTU of any path that packets may take.
Common MTU sizes
| Network MTU | Calico Cloud MTU | Calico Cloud MTU with IP-in-IP (IPv4) | Calico Cloud MTU with VXLAN (IPv4) | Calico Cloud MTU with VXLAN (IPv6) | Calico Cloud MTU with WireGuard (IPv4) | Calico Cloud MTU with WireGuard (IPv6) |
|---|---|---|---|---|---|---|
| 1500 | 1500 | 1480 | 1450 | 1430 | 1440 | 1420 |
| 9000 | 9000 | 8980 | 8950 | 8930 | 8940 | 8920 |
| 1500 (AKS) | 1500 | 1480 | 1450 | 1430 | 1340 | 1320 |
| 1460 (GCE) | 1460 | 1440 | 1410 | 1390 | 1400 | 1380 |
| 9001 (AWS Jumbo) | 9001 | 8981 | 8951 | 8931 | 8941 | 8921 |
| 1450 (OpenStack VXLAN) | 1450 | 1430 | 1400 | 1380 | 1390 | 1370 |
Recommended MTU for overlay networking
The extra overlay header used in IP in IP, VXLAN and WireGuard protocols, reduces the minimum MTU by the size of the header. (IP in IP uses a 20-byte header, IPv4 VXLAN uses a 50-byte header, IPv6 VXLAN uses a 70-byte header, IPv4 WireGuard uses a 60-byte header and IPv6 WireGuard uses an 80-byte header).
When using AKS, the underlying network has an MTU of 1400, even though the network interface will have an MTU of 1500. WireGuard sets the Don't Fragment (DF) bit on its packets, and so the MTU for WireGuard on AKS needs to be set to 60 bytes below (or 80 bytes for IPv6) the 1400 MTU of the underlying network to avoid dropped packets.
If you have a mix of WireGuard and either IP in IP or VXLAN in your cluster, you should configure the MTU to be the smallest of the values of each encap type. The reason for this is that only WireGuard encapsulation will be used between any nodes where both have WireGuard enabled, and IP in IP or VXLAN will then be used between any nodes where both do not have WireGuard enabled. This could be the case if, for example, you are in the process of installing WireGuard on your nodes.
Therefore, we recommend the following:
- If you use IPv4 WireGuard encryption anywhere in your pod network, configure MTU size as “physical network MTU size minus 60”.
- If you use IPv6 WireGuard encryption anywhere in your pod network, configure MTU size as “physical network MTU size minus 80”.
- If you don't use WireGuard, but use IPv4 VXLAN anywhere in your pod network, configure MTU size as “physical network MTU size minus 50”.
- If you don't use WireGuard, but use IPv6 VXLAN anywhere in your pod network, configure MTU size as “physical network MTU size minus 70”.
- If you don't use WireGuard, but use only IP in IP, configure MTU size as “physical network MTU size minus 20”
- Set the workload endpoint MTU and the tunnel MTUs to the same value (so all paths have the same MTU)
eBPF mode
Implementation of NodePorts uses VXLAN tunnel to hand off packets from one node to another, therefore VXLAN MTU setting is used to set the MTUs of workloads (veths) and should be “physical network MTU size minus 50” (see above).
MTU for flannel networking
When using flannel for networking, the MTU for network interfaces should match the MTU of the flannel interface.
- If using flannel with VXLAN, use the “Calico Cloud MTU with VXLAN” column in the table above for common sizes.
Configure MTU
The updated MTU used by Calico Cloud only applies to new workloads.
For Operator installations, edit the Calico Cloud operator Installation resource to set the mtu
field in the calicoNetwork section of the spec. For example:
kubectl patch installation.operator.tigera.io default --type merge -p '{"spec":{"calicoNetwork":{"mtu":1440}}}'
Similarly, for OpenShift:
oc patch installation.operator.tigera.io default --type merge -p '{"spec":{"calicoNetwork":{"mtu":1440}}}'
View current tunnel MTU values
To view the current tunnel size, use the following command:
ip link show
The IP in IP tunnel appears as tunlx (for example, tunl0), along with the MTU size. For example:
