Skip to main content
Calico Cloud documentation

Configure flow log aggregation

Big pictureโ€‹

Configure flow log aggregation level to reduce log volume and costs.

Valueโ€‹

Beyond using filtering to suppress flow logs, Calico Cloud provides controls to aggregate flow logs. Although aggressive aggregation levels reduce flow volume and costs, it can also reduce visibility into specific metadata of allowed and denied traffic. Review this article to see which level of aggregation is suitable for your implementation.

Conceptsโ€‹

Volume and cost versus visibilityโ€‹

Calico Cloud enables flow log aggregation for pod/workload endpoints by default, and uses an aggressive aggregation level to reduce log volume. The level assumes that most users do not need to see pod IP information (due to the ephemeral nature of pod IP address allocation). However, it all depends on your deployment; we recommend reviewing aggregation levels to understand what information gets grouped (and thus suppressed from view).

Aggregation types and levelsโ€‹

For allowed flows, the default aggregation level is 2, AnyConnectionFromSamePodPrefix and for denied flows the default aggregation level is 1, AnyConnectionFromSameSourcePod.

The following table summarizes the aggregation levels by flow log traffic:

LevelNameDescription
0No aggregation
1AnyConnectionFromSameSourcePodIdentity fields below source pod level are masked out. It means that flows, to the same destination, from processes or controllers in the same source pod, are aggregated together.
2AnyConnectionFromSamePodPrefixIn addition to the above, source pod names are aggregated based on their shared prefixes. This means that flows, to the same destination, from pods within the same pod controller (Deployment/ReplicaSet) are aggregated together.
3AnyConnectionBetweenSamePodPrefixesThis level of aggregation builds on the previous two levels and also groups destination pod names based on their shared prefixes.

Understanding aggregation level differencesโ€‹

Here are examples of pod-to-pod flows, highlighting the differences between flow logs at various aggregation levels.

The source port is usually ephemeral and does not convey useful information. By suppressing the source port, AnyConnectionFromSameSourcePod aggregation type minimizes the flow logs generated for traffic coming from different containers within the same pod and going to the same destination endpoint and port. The two flows originating from client-a without aggregation are combined into one.

In Kubernetes, pod controllers (Deployments, DaemonSets, ReplicaSets etc.) can automatically create names for pods. For example, the pods nginx-1 and nginx-2 are created by the ReplicaSet nginx. The controller's name is considered a pod-prefix and is used to aggregate flow log entries (indicated with an asterisk * at the end of the name). Flow logs originating from pods with the same prefix will be aggregated as long as the traffic is on the same protocol, and destined towards the same IP, and destination port. The three flow logs without aggregation originating from client-a and client-b are combined into a single flow log. This aggregation level is called AnyConnectionFromSameSourcePodPrefix.

Finally, with AnyConnectionBetweenSamePodPrefixes we combine source and destination pods that are part of the same pod controller. With level 3, the flow logs are aggregated by the destination port and protocol, as long as they originate from pods with the same pod-prefix and destined for pods of the same pod-prefix. Previously distinct logs are aggregated into a single flow log (see the last row).

Src TrafficDst TrafficPacket counts
Aggr lvlFlowsNameIPPortNameIPPortPkt inPkt out
0 (no aggr)4client-a1.1.1.145556nginx-12.2.2.28012
client-b1.1.2.245666nginx-22.2.3.38022
client-a1.1.1.165533nginx-12.2.2.28013
client-c1.1.1.265534nginx-22.2.3.38034
1 (src port)3client-a1.1.1.1-nginx-12.2.2.28025
client-b1.1.2.2-nginx-12.2.2.28022
client-c1.1.3.3-nginx-22.2.3.38034
2 (src pod-prefix)2client-*--nginx-12.2.2.28047
client-*--nginx-22.2.3.38034
3 (src/dest pod-prefix)1client-*--nginx-*-80711

How toโ€‹

Verify existing aggregation levelโ€‹

Use the following command:

kubectl get felixconfiguration -o yaml

Change default aggregation levelโ€‹

Before changing the default aggregation level, note the following:

  • Although any change in aggregation level affects flow log volume, lowering the aggregation number (especially to 0 for no aggregation) will cause significant impacts to log storage. If you allow more flow logs, ensure that you provision more log storage.
  • Verify that the parameters that you want to see in your aggregation level are not already filtered.

Troubleshoot logs with aggregation levelsโ€‹

When you use flow log aggregation, sometimes you may see two Alerts,

two-alerts

along with two flow log entries. Note that the entries are identical except for the slight timestamp difference.

two-logs

The reason you may see two entries is because of the interaction between the aggregation interval, and the time interval to export logs (flowLogsFlushInterval).

In each aggregation interval, connections/connection attempts can be started or completed. However, flow logs do not start/stop when a connection starts/stops. Letโ€™s assume the default export logs โ€œflushโ€ time of 10 seconds. If a connection is started in one flush interval, but terminates in the next, it is recorded across two entries. To get visibility into flow logs to differentiate the entries, go to Service Graph, flow logs tab, and look at these fields: num_flows, num_flows_started, and num_flows_completed.

The underlying reason for this overlap is a dependency on Linux conntrack, which provides the lifetime of stats that Calico Cloud tracks across different protocols (TCP, ICMP, UDP). For example, for UDP and ICMP, Calico Cloud waits for a conntrack entry to timeout before it considers a โ€œconnectionโ€ closed, and this is usually greater than 10 seconds.

Additional resourcesโ€‹