Bring your own Prometheus
Big picture
Scrape Calico Cloud metrics for Bring Your Own (BYO) Prometheus.
Value
Calico Cloud uses the Prometheus monitoring tool to scrape metrics from instrumented jobs, and displays time-series data in a visualizer such as Grafana. You can scrape the following time-series metrics for Calico Cloud components to your own Prometheus:
- elasticsearch
- fluentd
- calico-node
- kube-controllers
- felix
- typha (not enabled by default)
Before you begin
Supported
For the supported version of Prometheus in this release, see the Release Notes (coreos-prometheus
).
How to
- Scrape all enabled metrics
- Scrape metrics from specific components directly
- Verify BYO Prometheus
- Create policy to secure traffic between pods
- Troubleshooting
Scrape all enabled metrics
In this section we create a service monitor that scrapes all enabled metrics. To enable metrics that are not enabled by default, please consult the next section.
The following example shows a Prometheus server installed in namespace "external-prometheus" with a serviceMonitorSelector
that selects all service monitors with the label k8s-app=tigera-external-prometheus
.
-
Save the following configuration in a file called
monitor.yaml
.apiVersion: operator.tigera.io/v1
kind: Monitor
metadata:
name: tigera-secure
spec:
externalPrometheus:
namespace: external-prometheus
serviceMonitor:
labels:
k8s-app: tigera-external-prometheusFor a list of all configuration options, see the Installation API reference.
-
Apply the manifest to your cluster.
kubectl apply -f monitor.yaml
-
Verify that the new configuration has been added to your cluster
export NS=external-prometheus
kubectl get servicemonitor -n $NS tigera-external-prometheus
kubectl get serviceaccount -n $NS tigera-external-prometheus
kubectl get secret -n $NS tigera-external-prometheus
kubectl get clusterrole tigera-external-prometheus
kubectl get clusterrolebinding tigera-external-prometheusThat's it. You should be seeing the new metrics show up in your Prometheus instance within a minute. For more information on verifying metrics, see the section, Verify BYO Prometheus.
Scrape metrics from specific components directly
We recommend the previous section for scraping all enabled metrics. Read on if you wish to scrape metrics from specific components directly using mTLS, or if you wish to enable metrics that are disabled by default.
- elasticsearch
- fluentd
- calico node
- kube-controllers
- Felix
- Typha
Configure TLS certificates
-
Copy the required secret and configmap to your namespace.
-
Save the manifest of the required TLS secret and CA configmap.
kubectl get secret calico-node-prometheus-client-tls -n tigera-prometheus -o yaml > calico-node-prometheus-client-tls.yaml
kubectl get configmap -n tigera-prometheus tigera-ca-bundle -o yaml > tigera-ca-bundle.yaml
-
Edit
calico-node-prometheus-client-tls.yaml
andtigera-ca-bundle.yaml
by changing the namespace to the namespace where your prometheus instance is running. -
Apply the manifests to your cluster.
kubectl apply -f calico-node-prometheus-client-tls.yaml
kubectl apply -f tigera-ca-bundle.yaml
Create the service monitor
Apply the ServiceMonitor to the namespace where Prometheus is running.
export NAMESPACE=<my-prometheus-namespace>
kubectl apply -f https://downloads.tigera.io/ee/v3.20.0-2.0/manifests/prometheus/elasticsearch-metrics-service-monitor.yaml -n $NAMESPACE
The .yamls have no namespace defined so when you apply kubectl
, it is applied in the $NAMESPACE.
Configure TLS certificates
-
Copy the required secret and configmap to your namespace.
-
Save the manifest of the required TLS secret and CA configmap.
kubectl get secret calico-node-prometheus-client-tls -n tigera-prometheus -o yaml > calico-node-prometheus-client-tls.yaml
kubectl get configmap -n tigera-prometheus tigera-ca-bundle -o yaml > tigera-ca-bundle.yaml
-
Edit
calico-node-prometheus-client-tls.yaml
andtigera-ca-bundle.yaml
and change the namespace to the namespace where your prometheus instance is running. -
Apply the manifests to your cluster.
kubectl apply -f calico-node-prometheus-client-tls.yaml
kubectl apply -f tigera-ca-bundle.yaml
Create the service monitor
Apply the ServiceMonitor to the namespace where Prometheus is running.
export NAMESPACE=<my-prometheus-namespace>
kubectl apply -f https://downloads.tigera.io/ee/v3.20.0-2.0/manifests/prometheus/fluentd-metrics-service-monitor.yaml -n $NAMESPACE
The .yamls have no namespace defined so when you apply kubectl
, it is applied in the $NAMESPACE.
Configure TLS certificates
-
Copy the required secret and configmap to your namespace.
-
Save the manifest of the required TLS secret and CA configmap.
kubectl get secret calico-node-prometheus-client-tls -n tigera-prometheus -o yaml > calico-node-prometheus-client-tls.yaml
kubectl get configmap -n tigera-prometheus tigera-ca-bundle -o yaml > tigera-ca-bundle.yaml
-
Edit
calico-node-prometheus-client-tls.yaml
andtigera-ca-bundle.yaml
by changing the namespace to the namespace where your prometheus instance is running. -
Apply the manifests to your cluster.
kubectl apply -f calico-node-prometheus-client-tls.yaml
kubectl apply -f tigera-ca-bundle.yaml
Create the service monitor
Apply the ServiceMonitor to the namespace where Prometheus is running.
export NAMESPACE=<my-prometheus-namespace>
kubectl apply -f https://downloads.tigera.io/ee/v3.20.0-2.0/manifests/prometheus/calico-node-monitor-service-monitor.yaml -n $NAMESPACE
The .yamls have no namespace defined so when you apply kubectl
, it is applied in $NAMESPACE.
Configure TLS certificates
-
Copy the required secret and configmap to your namespace.
-
Save the manifest of the required TLS secret and CA configmap.
kubectl get secret calico-node-prometheus-client-tls -n tigera-prometheus -o yaml > calico-node-prometheus-client-tls.yaml
kubectl get configmap -n tigera-prometheus tigera-ca-bundle -o yaml > tigera-ca-bundle.yaml
-
Edit
calico-node-prometheus-client-tls.yaml
andtigera-ca-bundle.yaml
by changing the namespace to the namespace where your prometheus instance is running. -
Apply the manifests to your cluster.
kubectl apply -f calico-node-prometheus-client-tls.yaml
kubectl apply -f tigera-ca-bundle.yaml
Create the service monitor
Apply the ServiceMonitor to the namespace where Prometheus is running.
export NAMESPACE=<my-prometheus-namespace>
kubectl apply -f https://downloads.tigera.io/ee/v3.20.0-2.0/manifests/prometheus/kube-controller-metrics-service-monitor.yaml -n $NAMESPACE
The .yamls have no namespace defined so when you apply kubectl
, it is applied in the $NAMESPACE.
Enable metrics
Felix metrics are not enabled by default.
By default, Felix uses port 9091 TCP to publish metrics.
Use the following command to enable Felix metrics.
kubectl patch felixconfiguration default --type merge --patch '{"spec":{"prometheusMetricsEnabled": true}}'
You should see a result similar to:
felixconfiguration.projectcalico.org/default patched
For all Felix configuration values, see Felix configuration.
For all Prometheus Felix configuration values, see Felix Prometheus.
Create a service to expose Felix metrics
kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: felix-metrics-svc
namespace: calico-system
labels:
k8s-app: felix-metrics
spec:
selector:
k8s-app: calico-node
ports:
- port: 9091
targetPort: 9091
EOF
If running Calico Enterprise for Windows, also create a service for Windows nodes:
kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: felix-windows-metrics-svc
namespace: calico-system
labels:
k8s-app: felix-metrics
spec:
clusterIP: None
selector:
k8s-app: calico-node-windows
ports:
- port: 9091
targetPort: 9091
EOF
By default, the Windows firewall blocks listening on ports. For Calico Cloud to manage the Prometheus metrics ports Windows firewall rules, enable the windowsManageFirewallRules
setting in FelixConfiguration:
kubectl patch felixConfiguration default --type merge --patch '{"spec":{"windowsManageFirewallRules": "Enabled"}}'
See the FelixConfiguration reference for more details. You can also add a Windows firewall rule that allows listening on the Prometheus port(s) instead of having Calico Cloud manage it.
Create the service monitor
Apply the ServiceMonitor to the namespace where Prometheus is running.
export NAMESPACE=<my-prometheus-namespace>
kubectl apply -f https://downloads.tigera.io/ee/v3.20.0-2.0/manifests/prometheus/felix-metrics-service-monitor.yaml -n $NAMESPACE
The .yamls have no namespace defined so when you apply kubectl
, it is applied in the $NAMESPACE.
Enable metrics
Typha metrics are not enabled by default.
By default, Typha uses port 9091 TCP to publish metrics. However, if Calico Cloud is installed using the Amazon yaml file, this port will be 9093 because it is set manually using the TYPHA_PROMETHEUSMETRICSPORT environment variable.
Use the following command to enable Typha metrics.
kubectl patch installation default --type=merge -p '{"spec": {"typhaMetricsPort":9093}}'
You should see a result similar to:
installation.operator.tigera.io/default patched
Create the service monitor
Apply the ServiceMonitor to the namespace where Prometheus is running.
export NAMESPACE=<my-prometheus-namespace>
kubectl apply -f https://downloads.tigera.io/ee/v3.20.0-2.0/manifests/prometheus/typha-metrics-service-monitor.yaml -n $NAMESPACE
The .yamls have no namespace defined so when you apply kubectl
, it is applied in the $NAMESPACE.
Verify BYO Prometheus
-
Access the Prometheus dashboard using the port-forwarding feature.
kubectl port-forward pod/byo-prometheus-pod 9090:9090 -n $NAMESPACE
-
Browse to the Prometheus dashboard: http://localhost:9090.
-
In the Expression text box, enter your metric name and click the Execute button.
The Console table is populated with all of your nodes with the number of endpoints.
Troubleshooting
This section is applicable only if you experience issues with mTLS after following the Scrape metrics from specific components directly section.
-
Use the following command to retrieve the tls.key and tls.cert.
export NAMESPACE=<my-prometheus-namespace>
kubectl get secret -n $NAMESPACE calico-node-prometheus-client-tls -o yaml
-
Save the tls.key and tls.cert content into key and cert after base64 decode.
$:tls_key=<tls.key content>
$:echo $tls_key|base64 -d >key.pem
$:tls_cert=<tls.crt content>
$:echo $cert|base64 -d>cert.pem -
Get the ca-bundle certificate using this command:
kubectl get cm -n $NAMESPACE tigera-ca-bundle -o yaml
-
Open a new file (bundle.pem) in your favorite editor, and paste the content from "BEGIN CERTIFICATE" to "END CERTIFICATE".
-
Port-forward the prometheus pods and run this command with the forwarded port.
curl --cacert bundle.pem --key key.pem --cert cert.pem https://localhost:8080/metrics
You should be able to see the metrics.
Create policy to secure traffic between pods
To support zero trust, we recommend that you create Calico Cloud network policy to allow the traffic between BYO Prometheus pods, and the respective metrics pods. For samples of ingress and egress policies, see Get started with Calico network policy.