Skip to main content

Bring your own Prometheus

Big picture​

Scrape Calico Cloud metrics for Bring Your Own (BYO) Prometheus.

Value​

Calico Cloud uses the Prometheus monitoring tool to scrape metrics from instrumented jobs, and displays time-series data in a visualizer such as Grafana. You can scrape the following time-series metrics for Calico Cloud components to your own Prometheus:

  • elasticsearch
  • fluentd
  • calico-node
  • kube-controllers
  • felix
  • typha (not enabled by default)

Before you begin​

Supported

For the supported version of Prometheus in this release, see the Release Notes (coreos-prometheus).

How to​

Scrape all enabled metrics​

In this section we create a service monitor that scrapes all enabled metrics. To enable metrics that are not enabled by default, please consult the next section.

The following example shows a Prometheus server installed in namespace "external-prometheus" with a serviceMonitorSelector that selects all service monitors with the label k8s-app=tigera-external-prometheus.

  1. Save the following configuration in a file called monitor.yaml.

    apiVersion: operator.tigera.io/v1
    kind: Monitor
    metadata:
      name: tigera-secure
    spec:
      externalPrometheus:
        namespace: external-prometheus
        serviceMonitor:
          labels:
            k8s-app: tigera-external-prometheus

    For a list of all configuration options, see the Installation API reference.

  2. Apply the manifest to your cluster.

    kubectl apply -f monitor.yaml

  3. Verify that the new configuration has been added to your cluster

    export NS=external-prometheus
    kubectl get servicemonitor -n $NS tigera-external-prometheus
    kubectl get serviceaccount -n $NS tigera-external-prometheus
    kubectl get secret -n $NS tigera-external-prometheus
    kubectl get clusterrole tigera-external-prometheus
    kubectl get clusterrolebinding tigera-external-prometheus

    That's it. You should be seeing the new metrics show up in your Prometheus instance within a minute. For more information on verifying metrics, see the section, Verify BYO Prometheus.

Scrape metrics from specific components directly​

We recommend the previous section for scraping all enabled metrics. Read on if you wish to scrape metrics from specific components directly using mTLS, or if you wish to enable metrics that are disabled by default.

Configure TLS certificates

  1. Copy the required secret and configmap to your namespace.

  2. Save the manifest of the required TLS secret and CA configmap.

    kubectl get secret calico-node-prometheus-client-tls -n tigera-prometheus -o yaml >  calico-node-prometheus-client-tls.yaml
    kubectl get configmap -n tigera-prometheus tigera-ca-bundle -o yaml > tigera-ca-bundle.yaml

  3. Edit calico-node-prometheus-client-tls.yaml and tigera-ca-bundle.yaml by changing the namespace to the namespace where your prometheus instance is running.

  4. Apply the manifests to your cluster.

    kubectl apply -f calico-node-prometheus-client-tls.yaml
    kubectl apply -f tigera-ca-bundle.yaml

Create the service monitor

Apply the ServiceMonitor to the namespace where Prometheus is running.

export NAMESPACE=<my-prometheus-namespace>
kubectl apply -f https://downloads.tigera.io/ee/v3.19.0-1.0/manifests/prometheus/elasticsearch-metrics-service-monitor.yaml -n $NAMESPACE

The .yamls have no namespace defined so when you apply kubectl, it is applied in the $NAMESPACE.

Verify BYO Prometheus​

  1. Access the Prometheus dashboard using the port-forwarding feature.

    kubectl port-forward pod/byo-prometheus-pod 9090:9090 -n $NAMESPACE

  2. Browse to the Prometheus dashboard: http://localhost:9090.

  3. In the Expression text box, enter your metric name and click the Execute button.

    The Console table is populated with all of your nodes with the number of endpoints.

Troubleshooting​

This section is applicable only if you experience issues with mTLS after following the Scrape metrics from specific components directly section.

  1. Use the following command to retrieve the tls.key and tls.cert.

    export NAMESPACE=<my-prometheus-namespace>
    kubectl get secret -n $NAMESPACE calico-node-prometheus-client-tls -o yaml

  2. Save the tls.key and tls.cert content into key and cert after base64 decode.

    $:tls_key=<tls.key content>
    $:echo $tls_key|base64 -d >key.pem

    $:tls_cert=<tls.crt content>
    $:echo $cert|base64 -d>cert.pem

  3. Get the ca-bundle certificate using this command:

    kubectl get cm -n $NAMESPACE tigera-ca-bundle -o yaml

  4. Open a new file (bundle.pem) in your favorite editor, and paste the content from "BEGIN CERTIFICATE" to "END CERTIFICATE".

  5. Port-forward the prometheus pods and run this command with the forwarded port.

    curl --cacert bundle.pem --key key.pem  --cert cert.pem https://localhost:8080/metrics

You should be able to see the metrics.

Create policy to secure traffic between pods​

To support zero trust, we recommend that you create Calico Cloud network policy to allow the traffic between BYO Prometheus pods, and the respective metrics pods. For samples of ingress and egress policies, see Get started with Calico network policy.