Skip to main content
Version: 3.18 (latest)

kOps on AWS

Big picture​

Install Calico Enterprise with a self-managed Kubernetes cluster using Kubernetes Operations (kOps) on AWS. kOps is a cluster management tool that provisions cluster VMs and installs Kubernetes. It is a good default choice for most because it gives you access to all Calico Enterprise flexible and powerful networking features. However, other options may work better for your environment.

Before you begin​

CNI support

  • Calico CNI for networking with Calico Enterprise network policy

    The geeky details of what you get:

    PolicyIPAMCNIOverlayRoutingDatastore

  • AWS VPC CNI for networking with Calico Enterprise network policy

    The geeky details of what you get:

    PolicyIPAMCNIOverlayRoutingDatastore

Required

How to​

Select one of the following installation paths:

Install Calico Enterprise networking and network policy​

To use kOps to create a cluster with Calico Enterprise networking and network policy:

  1. Install kOps on your workstation.

  2. Set up your environment for AWS .

  3. Be sure to set up an S3 state store and export its name:

    export KOPS_STATE_STORE=s3://<name-of-your-state-store-bucket>

  4. Configure kOps to use Calico Enterprise for networking. Create a cluster with kOps using the --networking cni flag. For example:

    kops create cluster \
     --zones us-west-2a \
     --networking cni \
     <name-of-your-cluster>
    note

    The name of the cluster must be chosen as a valid DNS name belonging to the root user. It can either be a subdomain of an existing domain name or a subdomain which can be configured on AWS Route 53 service.

    Or, you can add cni to your cluster config. Run kops update cluster --name=<name-of-your-cluster> and set the following networking configuration.

    networking:
      cni: {}
    note

    Setting the --networking cni flag delegates the installation of the CNI to the user for a later stage.

  5. The provisioned kOps cluster will assign its own set of pod network CIDR in the kube-proxy instance different than the one Calico Enterprise expects. To set the cluster cidr for the kube-proxy to match the one expected by Calico Enterprise edit the cluster config kops edit cluster <name-of-your-cluster> and add the kubeProxy config with the clusterCIDR expected by the default Calico Enterprise installation.

    spec:
       ...
       kubeProxy:
          clusterCIDR: 192.168.0.0/16
    note

    For more advanced pod networking CIDR configuration, the requirement is to have ipPools CIDR set by the Calico Enterprise installation to match cluster CIDR set in kube-proxy. Calico's ipPools setting is obtainable in the Installation resource kubectl get installation -o yaml and can be configured by editing the operator manifest found in the install instructions for Calico Enterprise.

  6. The default size of the provisioned instance groups for the cluster might not be sufficient for the full installation of kubernetes and Calico Enterprise. To increase the size of the instance groups run kops edit ig <name-of-instance-group-in-your-cluster> --name <name-of-your-cluster> and edit the following fields accordingly.

       spec:
          ...
          machineType: t3.medium
          maxSize: 1
          minSize: 1

    The name of the instance groups can be obtained from kops get instancegroups --name <name-of-your-cluster>.

  7. Once your cluster has been configured run kops update cluster --name=<name-of-your-cluster> to preview the changes. Then the same command with --yes option (ie. kops update cluster --name=<name-of-your-cluster> --yes) to commit the changes to AWS to create the cluster. It may take 10 to 15 minutes for the cluster to be fully created.

    note

    Once the cluster has been created, the kubectl command should be pointing to the newly created cluster. By default kops>=1.19 does not update kubeconfig to include the cluster certificates, accesses to the cluster through kubectl must be configured.

  8. Validate that nodes are created.

    kubectl get nodes

    The above should return the status of the nodes in the Not Ready state.

  9. KOps does not install any CNI when the flag --networking cni or spec.networking: cni {} is used. In this case the user is expected to install the CNI separately. To Install Calico Enterprise follow the install instructions for Calico Enterprise.

  10. Finally, to delete your cluster once finished, run kops delete cluster <name-of-your-cluster> --yes.

You can further customize the Calico Enterprise install with options listed in the kops documentation.

Install Amazon VPC networking with Calico Enterprise network policy​

You can use Amazon’s VPC CNI plugin for networking, and Calico Enterprise for network policy. The advantage of this approach is that pods are assigned IP addresses associated with Elastic Network Interfaces on worker nodes. The IPs come from the VPC network pool, and therefore do not require NAT to access resources outside the Kubernetes cluster.

Set your kOps cluster configuration to:

networking:
  amazonvpc: {}

After the cluster is up and ready, Install Calico Enterprise.

Next steps​