Kubernetes services
This guide provides educational material that is not specific to Calico.
In this guide you will learn:
- What are Kubernetes services?
- What are the differences between the three main service types and what do you use them for?
- How do services and network policy interact?
- Some options for optimizing how services are handled.
What are Kubernetes services?
Kubernetes Services provide a way of abstracting access to a group of pods as a network service. The group of pods backing each service is usually defined using a label selector.
When a client connects to a Kubernetes service, the connection is load balanced to one of the pods backing the service, as illustrated in this conceptual diagram:
There are three main types of Kubernetes services:
- Cluster IP - which is the usual way of accessing a service from inside the cluster
- Node port - which is the most basic way of accessing a service from outside the cluster
- Load balancer - which uses an external load balancer as a more sophisticated way to access a service from outside the cluster.
Cluster IP services
The default service type is ClusterIP. This allows a service to be accessed within the cluster via a virtual IP
address, known as the service Cluster IP. The Cluster IP for a service is discoverable through Kubernetes DNS. For
example, my-svc.my-namespace.svc.cluster-domain.example. The DNS name and Cluster IP address remain constant for the
life time of the service, even though the pods backing the service may be created or destroyed, and the number of pods
backing the service may change over time.
In a typical Kubernetes deployment, kube-proxy runs on every node and is responsible for intercepting connections to Cluster IP addresses and load balancing across the group of pods backing each service. As part of this process DNAT is used to map the destination IP address from the Cluster IP to the chosen backing pod. Response packets on the connection then have the NAT reverse on their way back to the pod that initiated the connection.
Importantly, network policy is enforced based on the pods, not the service Cluster IP. (i.e. Egress network policy is enforced for the client pod after the DNAT has changed the connection's destination IP to the chosen service backing pod. And because only the destination IP for the connection is changed, ingress network policy for the backing pod sees the original client pod as the source of the connection.)
Node port services
The most basic way to access a service from outside the cluster is to use a service of type NodePort. A Node Port is a
port reserved on each node in the cluster through which the service can be accessed. In a typical Kubernetes deployment,
kube-proxy is responsible for intercepting connections to Node Ports and load balancing them across the pods backing
each service.
As part of this process NAT is used to map the destination IP address and port from the node IP and Node Port, to the chosen backing pod and service port. In addition the source IP address is mapped from the client IP to the node IP, so that response packets on the connection flow back via the original node, where the NAT can be reversed. (It's the node which performed the NAT that has the connection tracking state needed to reverse the NAT.)
Note that because the connection source IP address is SNATed to the node IP address, ingress network policy for the service backing pod does not see the original client IP address. Typically this means that any such policy is limited to restricting the destination protocol and port, and cannot restrict based on the client / source IP. This limitation can be circumvented in some scenarios by using externalTrafficPolicy or by using Calico's eBPF dataplane native service handling (rather than kube-proxy) which preserves source IP address.
Load balancer services
Services of type LoadBalancer expose the service via an external network load balancer (NLB). The exact type of
network load balancer depends on which public cloud provider or, if on-prem, which specific hardware load balancer integration is
integrated with your cluster.
The service can be accessed from outside of the cluster via a specific IP address on the network load balancer, which by default will load balance evenly across the nodes using the service node port.
Most network load balancers preserve the client source IP address, but because the service then goes via a node port, the backing pods themselves do not see the client IP, with the same implications for network policy. As with node ports, this limitation can be circumvented in some scenarios by using externalTrafficPolicy or by using Calico's eBPF dataplane native service handling (rather than kube-proxy) which preserves source IP address.
Advertising service IPs
One alternative to using node ports or network load balancers is to advertise service IP addresses over BGP. This requires the cluster to be running on an underlying network that supports BGP, which typically means an on-prem deployment with standard Top of Rack routers.
Calico supports advertising service Cluster IPs, or External IPs for services configured with one. If you are not using Calico as your network plugin then MetalLB provides similar capabilities that work with a variety of different network plugins.
externalTrafficPolicy:local
By default, whether using service type NodePort or LoadBalancer or advertising service IP addresses over BGP,
accessing a service from outside the cluster load balances evenly across all the pods backing the service, independent
of which node the pods are on. This behavior can be changed by configuring the service with
externalTrafficPolicy:local which specifies that connections should only be load balanced to pods backing the service
on the local node.
When combined with services of type LoadBalancer or with Calico service IP address advertising, traffic is
only directed to nodes that host at least one pod backing the service. This reduces the potential extra network hop
between nodes, and perhaps more importantly, to maintain the source IP address all the way to the pod, so network policy
can restrict specific external clients if desired.
Note that in the case of services of type LoadBalancer, not all Load Balancers support this mode. And in the case of
service IP advertisement, the evenness of the load balancing becomes topology dependent. In this case, pod anti-affinity
rules can be used to ensure even distribution of backing pods across your topology, but this does add some complexity to
deploying the service.
Calico eBPF native service handling
As an alternative to using Kubernetes standard kube-proxy, Calico's eBPF dataplane supports native service handling. This preserves source IP to simplify network policy, offers DSR (Direct Server Return) to reduce the number of network hops for return traffic, and provides even load balancing independent of topology, with reduced CPU and latency compared to kube-proxy.
