Skip to main content
Calico Enterprise 3.19 (latest) documentation

Install Calico Enterprise for Windows manually

note

The manual method for installing Calico Enterprise for Windows is deprecated in favor of using the Operator and Windows HostProcess containers (HPC). Support for this method will be dropped in a future Calico Enterprise version.

Big picture​

Install Calico Enterprise for Windows on Kubernetes clusters using the manual installation method. The standard installation for Calico Enterprise for Windows requires more time and expertise to configure. If you need to get started quickly, we recommend the Quickstart guide.

Before you begin​

CNI support

Calico CNI for networking with Calico Enterprise network policy:

The geeky details of what you get by default:

PolicyIPAMCNIOverlayRoutingDatastore

Required

  • Linux and Windows nodes meet requirements

  • The Calico Enterprise for Windows zip archive provided to you by your support representative.

  • If using Calico Enterprise networking:

    • Copy the kubeconfig file (used by kubelet) to each Windows node to the file, c:\k\config.
    • Install and configure calicoctl

  • Download Calico Enterprise for Windows and Kubernetes binaries to each Windows nodes to prepare for install:

  • On each of your Windows nodes, prepare the Windows node for Calico Enterprise for Windows installation:

    • Enable TLS v1.2:
    [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
    • Copy the Calico Enterprise for Windows zip archive to c:\tigera-calico-windows.zip
    • Download the Calico Enterprise for Windows installation script:
    Invoke-WebRequest https://docs.tigera.io/calico-enterprise/3.19/scripts/install-calico-windows.ps1 -OutFile c:\install-calico-windows.ps1
    • Run the Calico Enterprise for Windows installation script:
    c:\install-calico-windows.ps1 -DownloadOnly yes -KubeVersion <your Kubernetes version>

    For example, c:\install-calico-windows.ps1 -DownloadOnly yes -KubeVersion 1.17.9

    After these steps, run ls c:\TigeraCalico and you will see the calico-node.exe binary, install scripts, and other files.

How to​

Manual install​

Because the Kubernetes and Calico Enterprise control components do not run on Windows yet, a hybrid Linux/Windows cluster is required. First you create a Linux cluster for Calico Enterprise components, then you join Windows nodes to the Linux cluster.

note

The manual method for installing Calico Enterprise for Windows is deprecated in favor of using the Operator and Windows HostProcess containers. Support for this method will be dropped in a future Calico Enterprise version.

Kubernetes

  1. Create a Linux cluster
  2. Ensure pods run on the correct nodes
  3. Prepare Windows nodes to join the Linux cluster

Calico Enterprise

  1. Install Calico Enterprise on Linux control and worker nodes
  2. Install Calico Enterprise for Windows and Kubernetes on Windows nodes

Create a Linux cluster​

There are many ways to create a Linux Kubernetes cluster. We regularly test Calico Enterprise for Windows with kubeadm.

Ensure pods run on the correct nodes​

A primary issue of running a hybrid Kubernetes cluster is that many Kubernetes manifests do not specify a node selector to restrict where their pods can run. For example, kubeadm installs kube-proxy (Kubernetes per-host NAT daemon) using a DaemonSet that does not include a node selector. This means that the kube-proxy pod, which only supports Linux, will be scheduled to both Linux and Windows nodes. Services/pods that should run only on Linux nodes (such as the kube-proxy DaemonSet) should be started with a node selector to avoid attempting to schedule them to Windows nodes.

To get around this for kube-proxy:

  1. Use kubectl to retrieve the DaemonSet.

    kubectl get ds kube-proxy -n kube-system -o yaml > kube-proxy.yaml

  2. Modify the kube-proxy.yaml file to include a node selector that selects only Linux nodes:

    spec:
      template:
      ...
        spec:
          nodeSelector:
            kubernetes.io/os: linux
          containers:

  3. Apply the updated manifest.

    kubectl apply -f kube-proxy.yaml

A similar change may be needed for other Kubernetes services (such as kube-dns or core-dns).

Prepare Windows nodes to join the Linux cluster​

On each Windows node, follow the steps below to configure kubelet and kube-proxy service.

Step 1: Configure kubelet

kubelet must be configured to use CNI networking by setting the following command line arguments, depending on the installed container runtime.

For Docker:

  • --network-plugin=cni
  • --cni-bin-dir=<directory for CNI binaries>
  • --cni-conf-dir=<directory for CNI configuration>

For containerd:

  • --container-runtime=remote
  • --container-runtime-endpoint=npipe:////.//pipe//containerd-containerd

The CNI bin and conf dir settings are required by the Calico Enterprise installer to install the CNI binaries and configuration file.

note

Among other parameters, the containerd configuration file includes options to configure the CNI bin and conf dirs.

The following kubelet settings are also important:

  • --hostname-override can be set to $(hostname) to match Calico Enterprise's default. kubelet and Calico Enterprise must agree on the host/nodename; if your network environment results in hostnames that vary over time you should set the hostname override to a static value per host and update Calico Enterprise's nodename accordingly.

  • --node-ip should be used to explicitly set the IP that kubelet reports to the API server for the node. We recommend setting this to the host's main network adapter's IP since we've seen kubelet incorrectly use an IP assigned to a HNS bridge device rather than the host's network adapter.

  • Because of a Windows networking limitation, if using Calico Enterprise IPAM, --max-pods should be set to, at most, the IPAM block size of the IP pool in use minus 4:

    IP pool block sizeMax pods
    /n2^/32-n^ - 4
    /24252
    /25124
    /26 (default)60
    /2728
    /2812
    /294
    /30 or aboveCannot be used

In addition, it's important that kubelet is started after the vSwitch has been created, which happens when Calico Enterprise initializes the dataplane. Otherwise, kubelet can be disconnected for the API server when the vSwitch is created.

AWS users: Add the following argument to kubelet:

--hostname-override=<aws instance private DNS name> (and set the Calico Enterprise nodename variable to match). In addition, you should add KubernetesCluster=<cluster-name> as a tag when creating your Windows instance.

As a quickstart, the Calico Enterprise package includes a sample script at C:\TigeraCalico\kubernetes\kubelet-service.ps1 that:

  • Waits for Calico Enterprise to initialise the vSwitch
  • Starts kubelet with
    • If containerd service is running, the following flags are set:
      • --container-runtime set to remote
      • --container-runtime-endpoint set to npipe:////.//pipe//containerd-containerd
    • Otherwise, the following flags are set for Docker:
      • --network-plugin set to cni
      • --cni-bin-dir set to c:\k\cni
      • --cni-conf-dir set to c:\k\cni\config
      • --pod-infra-container-image set to kubeletwin/pause
    • --kubeconfig set to the path of node kubeconfig file
    • --hostname-override set to match Calico Enterprise's nodename
    • --node-ip set to the IP of the default vEthernet device
    • --cluster-dns set to the IPs of the dns name servers

See the README in the same directory for more details. Feel free to modify the script to adjust other kubelet parameters.

note

The script will pause at the first stage until Calico Enterprise is installed by following the instructions in the next section.

Step 2: Configure kube-proxy on Windows

kube-proxy must be configured as follows:

  • With the correct HNS network name used by the active CNI plugin. kube-proxy reads the HNS network name from an environment variable KUBE_NETWORK
    • With default configuration, Calico Enterprise uses network name "Calico Enterprise"
  • For VXLAN, with the source VIP for the pod subnet allocated to the node. This is the IP that kube-proxy uses when it does SNAT for a NodePort. For Calico Enterprise, the source VIP should be the second IP address in the subnet chosen for the host. For example, if Calico Enterprise chooses an IP block 10.0.0.0/26 then the source VIP should be 10.0.0.2. The script below will automatically wait for the block to be chosen and configure kube-proxy accordingly.
  • For Calico Enterprise policy to function correctly with Kubernetes services, the WinDSR feature gate must be enabled. This requires Windows Server build 17763.1432 or greater and Kubernetes v1.14 or greater. Calico Enterprise will automatically enable the WinDSR feature gate if kubernetes services are managed by Calico Enterprise for Windows.

kube-proxy should be started via a script that waits for the Calico HNS network to be provisioned. The Calico Enterprise package contains a suitable script for use with Calico Enterprise networking at C:\TigeraCalico\kubernetes\kube-proxy-service.ps1. The script:

  • Waits for Calico Enterprise to initialise the vSwitch.
  • Calculates the correct source VIP for the local subnet.
  • Starts kube-proxy with the correct feature gates and hostname to work with Calico Enterprise.

See the README in the same directory for more details. Feel free to modify the script to adjust other kube-proxy parameters.

The script will pause at the first stage until Calico Enterprise is installed by following the instructions in the next section.

Install Calico Enterprise on Linux control and worker nodes​

If using Calico Enterprise BGP networking

  1. Install Calico Enterprise for networking and policy for Kubernetes datastore (kdd).

  2. Disable the default Calico Enterprise IP-in-IP networking (which is not compatible with Windows), by modifying the Calico Enterprise manifest, and setting the CALICO_IPV4POOL_IPIP environment variable to "Never" before applying the manifest.

    If you do apply the manifest with the incorrect value, changing the manifest and re-applying will have no effect. To adjust the already-created IP pool:

    kubectl get ippool -o yaml > ippool.yaml

    Then, modify ippool.yaml by setting the ipipMode to Never and then apply the updated manifest:

    kubectl apply -f ippool.yaml

If using Calico Enterprise VXLAN networking

  1. Modify VXLAN as described in Overlay Networking guide. Note the following:

    • Windows can support only a single type of IP pool so it is important that you use only a single VXLAN IP pool in this mode.
    • Windows supports only VXLAN on port 4789 and VSID ≥ 4096. Calico Enterprise's default (on Linux and Windows) is to use port 4789 and VSID 4096.

  2. Apply the manifest using kubectl, and verify that you have a single pool with VXLANMODE Always.

    kubectl get ippool -o wide

  3. Disable BGP routing by updating the Installation instance:

    kubectl patch ipamconfigurations default --type merge --patch='{"spec": {"strictAffinity": true}}'

For Linux control nodes using Calico Enterprise networking, strict affinity must be set to true. This is required to prevent Linux nodes from borrowing IP addresses from Windows nodes:

calicoctl ipam configure --strictaffinity=true

Install Calico Enterprise for Windows and Kubernetes on Windows nodes​

Follow the steps below on each Windows node to install Kubernetes and Calico Enterprise:

If using Calico Enterprise BGP

Install the RemoteAccess service using the following PowerShell commands:

Install-WindowsFeature RemoteAccess
Install-WindowsFeature RSAT-RemoteAccess-PowerShell
Install-WindowsFeature Routing

Then restart the computer:

Restart-Computer -Force

before running:

Install-RemoteAccess -VpnType RoutingOnly

Sometimes the remote access service fails to start automatically after install. To make sure it is running, execute the following command:

Start-Service RemoteAccess

  1. If using a non-Calico Enterprise network plugin for networking, install and verify it now.

  2. Edit the install configuration file, config.ps1 as follows:

    Set this variable...To...
    $env:KUBE_NETWORKCNI plugin you plan to use. For Calico Enterprise, set the variable to Calico Enterprise.*
    $env:CALICO_NETWORKING_BACKENDwindows-bgp vxlan or none (if using a non-Calico Enterprise CNI plugin).
    $env:CNI_variablesLocation of your Kubernetes installation.
    $env:K8S_SERVICE_CIDRYour Kubernetes service cluster IP CIDR.
    $env:CALICO_DATASTORE_TYPECalico Enterprise datastore you want to use.
    $env:KUBECONFIGLocation of the kubeconfig file Calico Enterprise should use to access the Kubernetes API server. To set up a secure kubeconfig with the correct permissions for Calico Enterprise for Windows, see Create a kubeconfig for Calico Enterprise for Windows.
    $env:NODENAMEHostname used by kubelet. The default uses the node's hostname. Note: If you are using the sample kubelet start-up script from the Calico Enterprise package, kubelet is started with a hostname override that forces it to use this value.
    For AWS to work properly, kubelet should use the node's internal domain name for the AWS integration.
    If using Calico Enterprise BGP networking, the install script will generate a CNI NetConf file from the file cni.conf.template. Certain advanced configuration can be accessed by modifying the template before install. Note: Prior to Kubernetes v1.13, Kubernetes lacked support for setting the correct DNS configuration on each pod. To work around that limitation, the CNI configuration includes DNS settings that are applied to pods whenever the kubelet fails to pass DNS configuration to the CNI plugin. For v1.13 and above, the DNS configuration of the template is ignored in favour of correct per-pod values learned from the kubelet.

  3. Run the installer.

    • Change directory to the location that you unpacked the archive. For example:

      PS C:\... > cd c:\TigeraCalico

    • Run the install script:

      PS C:\... > .\install-calico.ps1
    note

    The installer initializes the Windows vSwitch, which can cause a short connectivity outage as the networking stack is reconfigured. After running that command, you may need to:

    • Reconnect to your remote desktop session.
    • Restart kubelet and kube-proxy if they were already running.
    • If you haven't started kubelet and kube-proxy already, you should do so now. The quickstart scripts provided in the Calico Enterprise package provide an easy way to do this. Calico Enterprise for Windows requires kubelet to be running to complete its per-node configuration (since Kubelet creates the Kubernetes Node resource).
    note

    After you run the installer, do not move the directory because the service registration refers to the path of the directory.

  4. Verify that the Calico Enterprise services are running.

    Get-Service -Name CalicoNode
    Get-Service -Name CalicoFelix

Operator install​

Next steps​