Host routes
Neutron allows "host routes" to be configured on a subnet, with each host route comprising
- an IP destination prefix
- a next hop IP for routing to that prefix.
When an instance is launched and gets an IP from that subnet, Neutron arranges, via DHCP, that the instance's routing table gets those routes.
With Calico, a host route's next hop IP should be the local host
networking-calico supports host routes, but it's important to note that a host route is only consistent with Calico when its next hop IP represents the local hypervisor. This is because the local hypervisor, in a Calico setup, always routes all data from an instance and so is always the next hop IP for data to any destination. If the instance's routing table has a route with some other next hop IP, that next hop IP address will effectively be ignored, and the data will likely not pass through the implied router; instead the data will go first to the hypervisor, and then the hypervisor's routing table will determine its next IP hop from there.
Specifically, each host route's next hop IP should be the gateway IP of the subnet that the desired instance NIC is attached to, and from which it got its IP address - where 'desired instance NIC' means the one that you want data for that host route to go through. In networking-calico's usage, subnet gateway IPs represent the local hypervisor, because data sent by an instance is always routed there.
networking-calico avoids unnecessary IP usage by using the subnet gateway IP to represent the local compute host, on every compute host where that subnet is being used. Although that might initially sound odd, it works because no data is ever sent to or from the gateway IP address; the gateway IP is only used as the next hop address for the first IP hop from an instance to its compute host, and then the compute host routes the data again, according to its routing table, to wherever it needs to go. This also means that the gateway IP address really is functioning as each instance's default gateway, in the generally understood sense.
When are host routes useful with Calico?
Host routes are useful with Calico when an instance has multiple NICs and you want to specify which NIC should be used for data to particular prefixes.
When an instance has multiple NICs, it should have a default route through only one of those NICs, and use non-default routes to direct appropriate traffic through the other NICs. Neutron host routes can be used to establish those non-default routes; alternatively they can also be programmed manually in the instance.
For example, suppose an instance has eth0 attached to a subnet with gateway 10.65.0.1, eth1 attached to a subnet with gateway 11.8.0.1, and a default route via eth0. Then a host route like
11.11.0.0/16,11.8.0.1
can be configured for the subnet, to say that data to 11.11.0.0/16 should go out through eth1. The instance's routing table will then be:
default via 10.65.0.1 dev eth0
10.65.0.0/24 dev eth0
11.8.0.0/24 dev eth1
11.11.0.0/16 via 11.8.0.1 dev eth1
When an instance only has a single network attachment, and so a single NIC, host routes cannot make any difference to how data is routed, so it is unhelpful (although also harmless) to configure them. Regardless of what the instance's routing table says, data must exit over the single NIC, and is always layer-2-terminated and rerouted by the host according to the host's routing table. It's required for the host's routing table to cover whatever destinations instances may want to send to, and host routes don't add anything to that.