Skip to main content

Fluentd metrics

Big picture​

Use the Prometheus monitoring and alerting tool for Fluentd metrics to ensure continuous network visibility.

Value​

Platform engineering teams rely on logs for visibility into their networks. If collecting or storing logs are disrupted, this can impact network visibility. Prometheus can monitor log collection and storage metrics so platform engineering teams are alerted about problems before they occur.

Concepts​

ComponentDescription
PrometheusMonitoring tool that scrapes metrics from instrumented jobs and displays time series data in a visualizer (such as Grafana). For Calico Cloud, the “jobs” that Prometheus can harvest metrics from the Fluentd component.
FluentdSends Calico Cloud logs to Elasticsearch for storage.

How to​

Create Prometheus alerts for Fluentd​

The following example creates a Prometheus rule to monitor some important Fluentd metrics, and alert when they have crossed certain thresholds:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: tigera-prometheus-log-collection-monitoring
namespace: tigera-prometheus
labels:
role: tigera-prometheus-rules
prometheus: calico-node-prometheus
spec:
groups:
- name: tigera-log-collection.rules
rules:
- alert: FluentdPodConsistentlyLowBufferSpace
expr: avg_over_time(fluentd_output_status_buffer_available_space_ratio[5m]) < 75
labels:
severity: Warning
annotations:
summary: "Fluentd pod {{$labels.pod}}'s buffer space is consistently below 75 percent capacity."
description: "Fluentd pod {{$labels.pod}} has very low buffer space. There may be connection issues between Elasticsearch
and Fluentd or there are too many logs to write out, check the logs for the Fluentd pod."

The alerts created in the example are described as follows:​

AlertSeverityRequiresIssue/reason
FluentdPodConsistentlyLowBufferSpaceNon-critical, warningImmediate investigation to ensure logs are being gathered correctly.A Fluentd pod’s available buffer size has averaged less than 75% over the last 5 minutes.

This could mean Fluentd is having trouble communicating with the Elasticsearch cluster, the Elasticsearch cluster is down, or there are simply too many logs to process.