Skip to main content
Version: 3.18 (latest)

Bring your own Prometheus

Big picture​

Scrape Calico Enterprise metrics for Bring Your Own (BYO) Prometheus.

Value​

Calico Enterprise uses the Prometheus monitoring tool to scrape metrics from instrumented jobs, and displays time-series data in a visualizer such as Grafana. You can scrape the following time-series metrics for Calico Enterprise components to your own Prometheus:

  • elasticsearch
  • fluentd
  • calico-node
  • kube-controllers
  • felix
  • typha

To use BYO Prometheus, you must create your own:

  • Service monitors
  • Alerts
note

With BYO Prometheus, Calico Enterprise metrics and alerts are not visible in Manager UI.

Before you begin​

Supported

For the supported version of Prometheus in this release, see the Release Notes (coreos-prometheus).

How to​

Scrape metrics​

Configure TLS certificates

  1. Copy the required secret and configmap to your namespace.

  2. Save the manifest of the required TLS secret and CA configmap.

    kubectl get secret calico-node-prometheus-client-tls -n tigera-prometheus -o yaml >  calico-node-prometheus-client-tls.yaml
    kubectl get configmap -n tigera-prometheus tigera-ca-bundle -o yaml > tigera-ca-bundle.yaml

  3. Edit calico-node-prometheus-client-tls.yaml and tigera-ca-bundle.yaml by changing the namespace to the namespace where your prometheus instance is running.

  4. Apply the manifests to your cluster.

    kubectl apply -f calico-node-prometheus-client-tls.yaml
    kubectl apply -f tigera-ca-bundle.yaml

Create the service monitor

Apply the ServiceMonitor to the namespace where Prometheus is running.

export NAMESPACE=<my-prometheus-namespace>
kubectl apply -f https://downloads.tigera.io/ee/v3.18.3/manifests/prometheus/elasticsearch-metrics-service-monitor.yaml -n $NAMESPACE

The .yamls have no namespace defined so when you apply kubectl, it is applied in the $NAMESPACE.

Verify BYO Prometheus​

Verify metrics in the Prometheus console

  1. Access the Prometheus dashboard using the port-forwarding feature.

    kubectl port-forward pod/byo-prometheus-pod 9090:9090 -n $NAMESPACE

  2. Browse to the Prometheus dashboard: http://localhost:9090.

  3. In the Expression text box, enter your metric name and click the Execute button.

    The Console table is populated with all of your nodes with the number of endpoints.

Verify endpoint authentication

  1. Use the following command to retrieve the tls.key and tls.cert.

    export NAMESPACE=<my-prometheus-namespace>
    kubectl get secret -n $NAMESPACE calico-node-prometheus-client-tls -o yaml

  2. Save the tls.key and tls.cert content into key and cert after base64 decode.

    $:tls_key=<tls.key content>
    $:echo $tls_key|base64 -d >key.pem

    $:tls_cert=<tls.crt content>
    $:echo $cert|base64 -d>cert.pem

  3. Get the ca-bundle certificate using this command:

    kubectl get cm -n $NAMESPACE tigera-ca-bundle -o yaml

  4. Open a new file (bundle.pem) in your favorite editor, and paste the content from "BEGIN CERTIFICATE" to "END CERTIFICATE".

  5. Port-forward the prometheus pods and run this command with the forwarded port.

    curl --cacert bundle.pem --key key.pem  --cert cert.pem https://localhost:8080/metrics

You should be able to see the metrics.

Create policy to secure traffic between pods​

To support zero trust, we recommend that you create Calico Enterprise network policy to allow the traffic between BYO Prometheus pods, and the respective metrics pods. For samples of ingress and egress policies, see Get started with Calico network policy.