Workload-based Web Application Firewall (WAF)

note

This feature is tech preview. Tech preview features may be subject to significant changes before they become GA.

Big picture

Protect cloud-native applications from application layer attacks with Calico Enterprise Workload-based Web Application Firewall (WAF).

Value

Our workload-centric Web Application Firewall (WAF) protects your workloads from a variety of application layer attacks originating from within your cluster such as SQL injection. Given that attacks on apps are the leading cause of breaches, you need to secure the HTTP traffic inside your cluster.

Historically, web application firewalls (WAFs) were deployed at the edge of your cluster to filter incoming traffic. Our workload-based WAF solution takes a unique, cloud-native approach to web security by allowing you to implement zero-trust rules for workloads inside your cluster.

Concepts

About Calico Enterprise WAF

WAF is deployed in your cluster along with Envoy DaemonSet. Calico Enterprise proxies selected service traffic through Envoy, checking HTTP requests using the industry-standard ModSecurity with OWASP Core Rule Set v3.3.5 modified for kubernetes workloads.

You simply enable WAF in Manager UI, and determine the services that you want to enable for WAF protection. By default WAF is set to DetectionOnly so no traffic will be denied until you are ready to turn on blocking mode.

Every request that WAF finds an issue with, will result in a Security Event being created for you to review in the UI, regardless of whether the traffic was allowed or denied. This can greatly help in tuning later.

How WAF determines if a request should be allowed or denied

If you configure WAF in blocking mode, WAF will use something called anomaly scoring mode to determine if a request is allowed with 200 OK or denied 403 Forbidden.

This works by matching a single HTTP request against all the configured WAF rules. Each rule has a score and WAF adds all the matched rule scores together, and compares it to the overall anomaly threshold score (100 by default). If the score is under the threshold the request is allowed and if the score is over the threshold the request is denied. Our WAF starts in detection mode only and with a high default scoring threshold so is safe to turn on and then fine-tune the WAF for your specific needs in your cluster.

Before you begin

Not supported

• GKE

Limitations

WAF cannot be used with:

• Host-networked client pods
• TLS traffic
• LoadBalancer services
• Egress gateways
• WireGuard on AKS or EKS (unless you apply a specific kernel variable). Contact Support for help.

note

When selecting and deselecting traffic for WAF, active connections may be disrupted.

caution

Enabling WAF for certain system services may result in an undesired cluster state.

• Do not enable WAF for system service with the following prefixes:

  • tigera-*
  • calico-*
  • kube-system
  • openshift-*

• Do not enable WAF for system services with the following combination of name and namespaces:

  • name: kubernetes, namespace: default
  • name: openshift, namespace: default
  • name: gatekeeper-webhook-service, namespace: gatekeeper-system

The rules are not overridden during upgrade, you will have to manage deploying updates to the OWASP Core Rule Set to the cluster over time.

If you modify the rules, it is recommended to keep your rules in git or similar source control systems.

How to

• Enable WAF on your cluster
• Apply WAF to your services
• View WAF events
• Fine-tuning your WAF
• Disable WAF feature from your cluster

Enable WAF

(Optional) Deploy a sample application

If you don’t have an application to test WAF with or don’t want to use it right away against your own application, we recommend that you install the GoogleCloudPlatform/microservices-demo app:

kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/microservices-demo/release/v0.7.0/release/kubernetes-manifests.yaml

Enable WAF using the CLI

Enable the Policy Sync API in Felix

To enable WAF using the CLI, you must enable the Policy Sync API in Felix. To do this cluster-wide, modify the default FelixConfiguration to set the field policySyncPathPrefix to /var/run/nodeagent:

kubectl patch felixconfiguration default --type='merge' -p '{"spec":{"policySyncPathPrefix":"/var/run/nodeagent"}}'

Enable WAF using kubectl

In the ApplicationLayer custom resource, named tigera-secure, set the webApplicationFirewall field to Enabled.

kubectl apply -f - <<EOF
apiVersion: operator.tigera.io/v1
kind: ApplicationLayer
metadata:
  name: tigera-secure
spec:
  webApplicationFirewall: Enabled
EOF

On the Manager UI, click Threat Defense, Web Application Firewall, Configure Web Application Firewall.

Apply WAF to services

Now that you have deployed WAF in your cluster, you can select the services you want to protect from application layer attacks.

If you have deployed the sample application, you can apply WAF on a service associated with your app, as follows:

kubectl annotate svc frontend -n default --overwrite projectcalico.org/l7-logging=true

Alternatively, you can use the Manager UI to apply WAF to the frontend service.

In this example, we applied WAF to the frontend service. This means that every request that goes through the frontend service is inspected. However, the traffic is not blocked because the WAF rule is set to DetectionOnly by default. You can adjust rules and start blocking traffic by fine-tuning your WAF.

In the previous example, we applied WAF to the frontend service of the sample application. Here, we are applying WAF to a service of your own application.

1. On the Manager UI, click Threat Defense, Web Application Firewall.

2. Select the services you want WAF to inspect, and then click Confirm Selections.

   

3. On the Web Application Firewall page, you can verify that WAF is enabled for a service by locating the service and checking that the Status column says Enabled.

4. To make further changes to a service, click Actions, and then Enable or Disable.

You have now applied WAF rule sets to your own services, and note that the traffic that goes through the selected services will be alerted but not blocked by default.

Trigger a WAF event

If you would like to trigger a WAF event for testing purposes, you can simulate an SQL injection attack inside your cluster by crafting a HTTP request with a query string that WAF will detect as an SQL injection attempt. The query string in this example has some SQL syntax embedded in the text. This is harmless and for demo purposes, but WAF will detect this pattern and create an event for this HTTP request.

Run a simple curl command from any pod inside your cluster targeting a service you have selected for WAF protection e.g. from the demo app above we could attempt to send a simple HTTP request to the cartservice.

  curl http://cartservice/cart?artist=0+div+1+union%23foo*%2F*bar%0D%0Aselect%23foo%0D%0A1%2C2%2Ccurrent_user

Fine-tuning your WAF

Manage your rules

The default rule sets work fine for most deployments. However, you can customize them to suit your requirements. Some common changes are detailed below, for example:

• Enabling blocking mode
• Changing the scoring threshold
• Fine-tuning a rule
• Disabling certain rules

By default, Calico Enterprise ships with Core Rule Set v3.3.5 with the following setup files pre-loaded:

• tigera.conf

There are two ways to edit your rules.

1. Edit the configmap directly using kubectl. The config map combines all the rule files together, so you will need to know how to search and find the exact place in the configmap that you want to update.

2. If you want to modify the rules in any meaningful way, it is recommended that you download the files locally and modify before replacing the configmap. In OWASP, it is recommended you disable rules or change configuration by modifying a set of specific metadata files and exclusion files, and not modify any of the main rule files directly. You can, of course, decide to replace the rule set entirely with your own set of custom rules. This is completely fine once the rules are written in the same SecRule syntax understood by ModSecurity. We recommend you keep a version of your rule files safe in version control like git.

To download and modify the rule files:

1. Clone our operator code which contains the ConfigMap source.

   git clone --depth=1 https://github.com/tigera/operator
   cd operator

2. Change directory into the actual modsec rule set files we use.

   cd pkg/render/applicationlayer/modsec-core-ruleset

3. Make the changes you want in the setup, conf or exclusion files and save.

4. Create a ConfigMap.

   kubectl create cm --dry-run=client --from-file=. -o yaml -n tigera-operator modsecurity-ruleset > $HOME/my-tigera-waf-ruleset.yaml

5. Apply the ConfigMap to your cluster.

   kubectl replace -f $HOME/my-tigera-waf-ruleset.yaml

After you do these steps, the modsecurity-ruleset ConfigMap will be replaced in the tigera-operator namespace, which then triggers a rolling restart of your L7 pods. This means that the HTTP connections going through L7 pods at the time of pod termination will be (RST) reset.

note

It is important to adhere to the Core Rule Set documentation on how to edit the behaviour of your WAF. A good place to begin at is the Installing Core Rule Set.

In many scenarios, the default example CRS configuration will be a good enough starting point. It is recommended to review the example configuration file before you deploy it to make sure it’s right for your environment.

Set WAF rule to block traffic

By default WAF will not block a request even if it has matching rule violations. The rule engine is set to DetectionOnly. You can configure to block traffic instead with an HTTP 403 Forbidden response status code when the combined matched rules scores exceed a certain threshold.

1. Edit the configmap:

   kubectl edit cm -n tigera-operator modsecurity-ruleset
2. Look for SecRuleEngine DetectionOnly and change it to SecRuleEngine On.
3. Save your changes. This triggers a rolling update of the L7 pods.

Action	Description	Disruptive?
DetectionOnly	Traffic is not denied nor dropped. Calico Enterprise will log events.	No
On	Denies HTTP traffic. Calico Enterprise will log the event in Security Events.	Yes
Off	Be cautious about using this option. Traffic is not denied, and there are no events.	No

Change anomaly scoring threshold

The default scoring threshold is set to 100, a high value that is unlikely to result in blocked requests, even if blocking mode is enabled. A critical rule violation scores 5 by default so a single HTTP request would need many rule violations to pass the default threshold of 100. You are encouraged to fine-tune and lower this threshold value iteratively until you are happy with the requests being allowed and denied for your cluster.

1. Edit the crs-setup.conf file and set tx.inbound_anomaly_score_threshold to the value of your choice:

     SecAction \
      "id:900110,\
      phase:1,\
      nolog,\
      pass,\
      t:none,\
      setvar:tx.inbound_anomaly_score_threshold=100,\
      setvar:tx.outbound_anomaly_score_threshold=100"

These are the default scoring points for each severity level, applied to each rule individually. The lower the overall anomaly scoring threshold value is, the more likely it is that traffic will be denied.

Severity	Score	Description
Critical	5	Mostly generated by the application attack rules (93x and 94x files)
Error	4	Generated mostly from outbound leakage rules (95x files)
Warning	3	Generated mostly by malicious client rules (91x files)
Notice	2	Generated mostly by the protocol rules (92x files)

Change other default settings

Core Rule Set has several default values and settings that you can fine-tune. Example: Rule 920420 checks the HTTP request Content-Type header against a list of allowed values. You may want to allow traffic through the WAF that has a certain Content-Type that is not allowed by default, e.g. you may have Content-Types in your micro-services workloads that are not allowed by default.

1. Edit the crs-setup.conf file and add a new Content-Type by modifying id:900220 and appending to the default list named tx.allowed_request_content_type

     SecAction \
      "id:900220,\
       phase:1,\
       nolog,\
       pass,\
       t:none,\
       setvar:'tx.allowed_request_content_type=|application/x-www-form-urlencoded| |multipart/form-data| |multipart/related| |text/xml| |application/xml| |application/soap+xml| |application/json| |application/cloudevents+json| |application/cloudevents-batch+json| |application/grpc|'"

Disable certain rules

You may want to disable a rule entirely. For example, if you find the allowed Content-Type rule mentioned above too restrictive, you can disable it unconditionally by changing RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf.example and renaming that file to remove the .example:

1. Edit RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf.example and add a line such as the following, referencing the rule ID you want to disable.

   SecRuleRemoveById 920420

2. Rename RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf.example to RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf removing the .example in order for ModSecurity to load the file.

View WAF events

Security Events

To view WAF events in a centralized security events dashboard, go to: Threat defense, Security Events. For help, see Security Event Management.

Kibana

To view WAF events In Kibana, select the tigera_secure_ee_waf* index pattern.

Disable WAF for a service

To disable WAF on a service, use the Actions menu on the WAF board, or use the following command:

kubectl annotate svc <service-name> -n <service-namespace> projectcalico.org/l7-logging-

Disable WAF feature from your cluster

To safely disable WAF, determine how to handle ApplicationLayer features and follow the steps:

Disable all ApplicationLayer features, including WAF

kubectl delete applicationlayer tigera-secure

Keep some ApplicationLayer features enabled

To disable WAF but keep some ApplicationLayer features enabled, you must update the ApplicationLayer custom resource.

Note that the ApplicationLayer Specification can specify configuration for Application Logging and Application Layer Policy also.

For the ApplicationLayer custom resource to be valid, at least one of these features have to be enabled, for example:

Valid YAML

WAF enabled, ALP disabled, and Log collection is unspecified (and the default is disabled)

apiVersion: operator.tigera.io/v1
kind: ApplicationLayer
metadata:
  name: tigera-secure
spec:
  webApplicationFirewall: Disabled
  applicationLayerPolicy: Enabled

Invalid YAML

WAF and ALP both set as disabled, log collection is unspecified (and the default is disabled)

apiVersion: operator.tigera.io/v1
kind: ApplicationLayer
metadata:
  name: tigera-secure
spec:
  webApplicationFirewall: Disabled
  applicationLayerPolicy: Disabled