Enforce network policy using Istio tutorial
This tutorial sets up a microservices application, then demonstrates how to use Calico application layer policy to mitigate some common threats.
Prerequisites
- Build a Kubernetes cluster.
- Install Calico on Kubernetes:
- If Calico is not installed on Kubernetes, see Calico on Kubernetes.
- If Calico is already installed on Kubernetes, verify that Calico networking (or a non-Calico CNI) and Calico network policy are installed.
- Install the calicoctl command line tool.
Note: Ensure calicoctl is configured to connect with your datastore. - Enable application layer policy and install Istio.
Note: Ensure that you annotate the default namespace for the Istio sidecar injection (istio-injection=enabled).
kubectl label namespace default istio-injection=enabled
Install the demo application
We will use a simple microservice application to demonstrate Calico application layer policy. The YAO Bank application creates a customer-facing web application, a microservice that serves up account summaries, and an etcd datastore.
Note: You can also view the manifest in your browser.
Verify that the application pods have been created and are ready.
When the demo application has come up, you will see three pods.
View the Kubernetes ServiceAccounts created by the manifest.
You should see a Kubernetes ServiceAccount for each microservice in the application (in addition to the default account).
Examine the Kubernetes Secrets.
You should see output similar to the following.
Notice that Istio CA will have created a secret of type istio.io/key-and-cert for each
service account. These keys and X.509 certificates are used to cryptographically authenticate
traffic in the Istio service mesh, and the corresponding service account identities are used by
Calico in authentication policy.
Determining ingress IP and port
You will use the istio-ingressgateway service to access the YAO Bank application. Determine your
ingress host and port following the Istio instructions. Once you have the
INGRESS_HOST and INGRESS_PORT variables set, you can set the GATEWAY_URL as follows.
Point your browser to http://$GATEWAY_URL/ to confirm the YAO Bank application is functioning
correctly. It may take several minutes for all the services to come up and respond, during which
time you may see 404 or 500 errors.
The need for policy
Although Calico & Istio are running in the cluster, we have not defined any authentication policy. Istio was configured to mutually authenticate traffic between the pods in your application, so only connections with Istio-issued certificates are allowed, and all inter-pod traffic is encrypted with TLS. That’s already a big step in the right direction.
But, let’s consider some deficiencies in this security architecture:
- All incoming connections from workloads in the Istio mesh are equally trusted
- Possession of a key & certificate pair is the only access credential considered.
To understand why these might be a problem, let’s take them one at a time.
Trusting workloads
Trusting connections from any workload in the Istio mesh is a poor security architecture because, like Kubernetes, Istio is designed to host multiple applications. Some of those applications may not be as trusted as others. They may be operated by different users or teams with wildly different security requirements. We don’t want our secure financial application microservices accessible from some hacky prototype another developer is cooking up.
Even within our own application, the best practice is to limit access as much as possible. Only pods that need access to a service should get it. Consider the YAO Bank application. The customer web service does not need, and should not have direct access to the backend database. The customer web service needs to directly interact with clients outside the cluster, some of whom may be malicious. Unfortunately, vulnerabilities in web applications are all too common. For example, an unpatched vulnerability in Apache Struts is what allowed attackers their initial access into the Equifax network where they then launched a devastating attack to steal millions of people’s financial information.
Imagine what would happen if an attacker were to gain control of the customer web pod in our application. Let’s simulate this by executing a remote shell inside that pod.
Notice that from here, we get direct access to the backend database. For example, we can list all the entries in the database like this:
(Piping to python -m json.tool nicely formats the output.)
Single-factor authentication
The possession of a key and certificate pair is a very strong assertion that a connection is authentic because it is based on cryptographic proofs that are believed to be nearly impossible to forge. When we authenticate connections this way we can say with extremely high confidence that the party on the other end is in possession of the corresponding key. However, this is only a proxy for what we actually want to be confident of: that the party on the other end really is the authorized workload we want to communicate with. Keeping the private key a secret is vital to this confidence, and occasionally attackers can find ways to trick applications into giving up secrets they should not. For example, the Heartbleed vulnerability in OpenSSL allowed attackers to trick an affected application into reading out portions of its memory, compromising private keys.
Let’s simulate an attacker who has stolen the private keys of another pod. Since the keys are stored as Kubernetes secrets, we won’t exploit a vulnerability in a service, but instead just mount the secret in a pod that will simulate an attacker.
If you still have your shell open in the customer pod, exit out or open a new terminal tab (we will return to the customer pod later).
Take a look at the 20-attack-pod.yaml manifest in your browser.
It creates a pod and mounts istio.summary secret. This will allow us to masquerade as if we were
the summary service, even though this pod is not run as that service account. Let’s try this out. First, exec into the pod.
Now, we will attack the database. Instead of listing the contents like we did before, let’s try
something more malicious, like changing the account balance with a PUT command.
Unlike when we did this with the customer web pod, we do not have Envoy to handle encryption, so we
have to pass an https URL, the --key and --cert parameters to curl to do the cryptography.
Return to your web browser and refresh to confirm the new balance.
Network policy
We can mitigate both of the above deficiencies with a Calico policy.
Note: You can also view the manifest in your browser.
Let’s examine this policy piece by piece. It consists of three policy objects, one for each microservice.
This policy protects the customer web app. Since this application is customer facing, we do not
restrict what can communicate with it. We do, however, restrict its communications to HTTP GET
requests.
The second policy protects the account summary microservice. We know the only consumer of this service is the customer web app, so we restrict the source of incoming connections to the service account for the customer web app.
The third policy protects the database. Only the summary microservice should have direct access to the database.
Let’s verify our policy is working as intended. First, return to your browser and refresh to ensure policy enforcement has not broken the application.
Next, return to the customer web app. Recall that we simulated an attacker gaining control of that pod by executing a remote shell inside it.
Repeat our attempt to access the database.
We have left out the JSON formatting because we do not expect to get a valid JSON response. This
time we should get a 403 Forbidden response. Only the account summary microservice has database
access according to our policy.
Finally, let’s return to the attack pod that simulated stealing secret keys.
Let’s repeat our attack with stolen keys. We’ll further increase the account balance to highlight whether it succeeds.
You should get no response, and refreshing your browser should not show an increased balance.
You might wonder how Calico was able to detect and prevent this attack—the attacker was able to steal the keys which prove identity in our system. This highlights the value of multi-layer authentication checks. Although our attack pod had the keys to fool the X.509 certificate check, Calico also monitors the Kubernetes API Server for which IP addresses are associated with which service accounts. Since our attack pod has an IP not associated with the account summary service account we disallow the connection.