Filter flow logs
Filter Calico Enterprise flow logs.
Filter Calico Enterprise flow logs to suppress logs of low significance, and troubleshoot threats.
This how-to guide uses the following Calico Enterprise features:
- ConfigMap in the
Container monitoring tools versus flow logs
Container monitoring tools are good for monitoring Kubernetes and orchestrated workloads for CPU usage, network usage, and log aggregation. For example, a data monitoring tool can tell if a pod has turned into a bitcoin miner based on it using more than normal CPU.
Calico Enterprise flow logs provide continuous records of every single packet sent/received by all pods in your Kubernetes cluster. Note that flow logs do not contain all packet data; only the number of packets/bytes that were sent between specific IP/ports, and when. In the previous monitoring tool example, Calico Enterprise flow logs could see the packets running to/from the bitcoin mining network.
Calico Enterprise flow logs tell you when a pod is compromised, specifically:
- Where a pod is sending data to
- If the pod is talking to a known command-and-control server
- Other pods that the compromised pod has been talking to (so you can see if they're compromised too)
Flow log format
A flow log contains these space-delimited fields (unless filtered out).
startTime endTime srcType srcNamespace srcName srcLabels dstType dstNamespace dstName
dstLabels srcIP dstIP proto srcPort dstPort numFlows numFlowsStarted numFlowsCompleted
reporter packetsIn packetsOut bytesIn bytesOut action
1528842551 1528842851 wep dev rails-81531* - wep dev memcached-38456* - - - 6 - 3000 7 3 4 out 154 61 70111 49404 allow
- Fields that are not enabled or are aggregated, are noted by
- Aggregated names (such as “pod prefix”), are noted by
*at the end of the name
dstNamefields contain only a
*, aggregation was performed using other means (such as specific labels), and no unique prefix was present.
Create flow log filters
Create your fluentd filters.
Example: filter out a specific namespace
This example filters out all flow logs whose source or destination namespace is "dev". Additional namespaces could be filtered by adjusting the regular expression "pattern"s, or by adding additional
Example: filter out internet traffic to a specific deployment
This example filters inbound internet traffic to the deployment with pods named,
nginx-internet-*. Note the use of the
and directive to filter out traffic that is both to the deployment, and from the internet (source
Add filters to ConfigMap file
filtersdirectory with a file called
flowwith your desired filters. If you are also adding dns filters, add the
dnsfile to the directory.
fluentd-filtersConfigMap in the
tigera-operatornamespace with the following command.
kubectl create configmap fluentd-filters -n tigera-operator --from-file=filters