Introduction

The rise of containers has reshaped the way we develop, deploy and maintain the software. Containers allow us to package the different services that constitute an application into separate containers, and to deploy those containers across a set of virtual and physical machines. This gives rise to container orchestration tool to automate the deployment, management, scaling and availability of a container-based application. Kubernetes allows deployment and management of container-based applications at scale. Learn more about backup and disaster recovery for your Kubernetes clusters.

One of the main advantages of Kubernetes is how it brings greater reliability and stability to the container-based distributed application, through the use of dynamic scheduling of containers. But, how do you make sure Kubernetes itself stays up when a component or its master node goes down?
 

Why we need Kubernetes High Availability?

Kubernetes High-Availability is about setting up Kubernetes, along with its supporting components in a way that there is no single point of failure. A single master cluster can easily fail, while a multi-master cluster uses multiple master nodes, each of which has access to same worker nodes. In a single master cluster the important component like API server, controller manager lies only on the single master node and if it fails you cannot create more services, pods etc. However, in case of Kubernetes HA environment, these important components are replicated on multiple masters(usually three masters) and if any of the masters fail, the other masters keep the cluster up and running.

Advantages of multi-master

In the Kubernetes cluster, the master node manages the etcd database, API server, controller manager, and scheduler, along with all the worker nodes. What if we have only a single master node and if that node fails, all the worker nodes will be unscheduled and the cluster will be lost.

In a multi-master setup, by contrast, a multi-master provides high availability for a single cluster by running multiple apiserver, etcd, controller-manager, and schedulers. This does not only provides redundancy but also improves network performance because all the masters are dividing the load among themselves.

A multi-master setup protects against a wide range of failure modes, from a loss of a single worker node to the failure of the master node’s etcd service. By providing redundancy, a multi-master cluster serves as a highly available system for your end-users.

Steps to Achieve Kubernetes HA

Before moving to steps to achieve high-availability, let us understand what we are trying to achieve through a diagram:

(Image Source: Kubernetes Official Documentation)

Master Node: Each master node in a multi-master environment run its’ own copy of Kube API server. This can be used for load balancing among the master nodes. Master node also runs its copy of the etcd database, which stores all the data of cluster. In addition to API server and etcd database, the master node also runs k8s controller manager, which handles replication and scheduler, which schedules pods to nodes.

Worker Node: Like single master in the multi-master cluster also the worker runs their own component mainly orchestrating pods in the Kubernetes cluster. We need 3 machines which satisfy the Kubernetes master requirement and 3 machines which satisfy the Kubernetes worker requirement.

For each master, that has been provisioned, follow the installation guide to install kubeadm and its dependencies. In this blog we will use k8s 1.10.4 to implement HA.

Note: Please note that cgroup driver for docker and kubelet differs in some version of k8s, make sure you change cgroup driver to cgroupfs for docker and kubelet. If cgroup driver for kubelet and docker differs then the master doesn’t come up when rebooted.

Setup etcd cluster

1. Install cfssl and cfssljson

CODE: https://gist.github.com/velotiotech/21736c5db8394cf4a7e912370c8ad65d.js

2 . Generate certificates on master-0

CODE: https://gist.github.com/velotiotech/ec172d8e40025b2b678c659d73c10a04.js

3. Create config.json file in /etc/kubernetes/pki/etcd folder with following content.

CODE: https://gist.github.com/velotiotech/862259eb1976761fdaff11cc13864807.js

4. Create ca-csr.json file in /etc/kubernetes/pki/etcd folder with following content.

CODE: https://gist.github.com/velotiotech/355c584fcc84ed55fea6425f2ca47d90.js

5. Create client.json file in /etc/kubernetes/pki/etcd folder with following content.

CODE: https://gist.github.com/velotiotech/e03ca25303a684393f3a3f8e61ef63d1.js

CODE: https://gist.github.com/velotiotech/773529c62a12802cf017ec5ac96391a2.js

6. Create a directory  /etc/kubernetes/pki/etcd on master-1 and master-2 and copy all the generated certificates into it.

7. On all masters, now generate peer and etcd certs in /etc/kubernetes/pki/etcd. To generate them, we need the previous CA certificates on all masters.

CODE: https://gist.github.com/velotiotech/bfd8c10c124abcede872faa44af2cc63.js

This will replace the default configuration with your machine’s hostname and IP address, so in case if you encounter any problem just check the hostname and IP address are correct and rerun cfssl command.

8. On all masters, Install etcd and set it’s environment file.

CODE: https://gist.github.com/velotiotech/31746658a9a54acc4bb816e2d49636a2.js

9. Now, we will create a 3 node etcd cluster on all 3 master nodes. Starting etcd service on all three nodes as systemd. Create a file /etc/systemd/system/etcd.service on all masters.

CODE: https://gist.github.com/velotiotech/8fac66766a023e25ff46d0f02b3bf6de.js

10. Ensure that you will replace the following placeholder with

• <host_name> : Replace as the master’s hostname</host_name>
• <host_private_ip>: Replace as the current host private IP</host_private_ip>
• <master0_private_ip>: Replace as the master-0 private IP</master0_private_ip>
• <master1_private_ip>: Replace as the master-1 private IP</master1_private_ip>
• <master2_private_ip>: Replace as the master-2 private IP</master2_private_ip>

11. Start the etcd service on all three master nodes and check the etcd cluster health:

CODE: https://gist.github.com/velotiotech/32befa60e1bcadbd5bf9e761116443e5.js

This will show the cluster healthy and connected to all three nodes.

Setup load balancer

There are multiple cloud provider solutions for load balancing like AWS elastic load balancer, GCE load balancing etc. There might not be a physical load balancer available, we can setup a virtual IP load balancer to healthy node master. We are using keepalived for load balancing, install keepalived on all master nodes

CODE: https://gist.github.com/velotiotech/25c7cfabca281dab49327a416c2b9474.js

Create the following configuration file /etc/keepalived/keepalived.conf on all master nodes:

CODE: https://gist.github.com/velotiotech/3b6062287a414fef03653a37bef1ecc7.js

• state is either MASTER (on the first master nodes) or BACKUP (the other master nodes).
• Interface is generally the primary interface, in my case it is eth0
• Priority should be higher for master node e.g 101 and lower for others e.g 100
• Virtual_ip should contain the virtual ip of master nodes

Install the following health check script to /etc/keepalived/check_apiserver.sh on all master nodes:

CODE: https://gist.github.com/velotiotech/d7c6d1629c33727c07e5b806b96981b3.js

CODE: https://gist.github.com/velotiotech/409b993e5374c11d452feadf1a59c100.js

Setup three master node cluster

Run kubeadm init on master0:

Create config.yaml file with following content.

CODE: https://gist.github.com/velotiotech/fcf0ea078879fa7548fe91fa569519d8.js

Please ensure that the following placeholders are replaced:

• <master-private-ip> with the private IPv4 of the master server on which config file resides.</master-private-ip>
• <master0-ip-address>, <master1-ip-address> and <master-2-ip-address> with the IP addresses of your three master nodes</master-2-ip-address></master1-ip-address></master0-ip-address>
• <podcidr> with your Pod CIDR. Please read the </podcidr>CNI network section of the docs for more information. Some CNI providers do not require a value to be set. I am using weave-net as pod network, hence podCIDR will be 10.32.0.0/12
• <load-balancer-ip> with the virtual IP set up in the load balancer in the previous section.</load-balancer-ip>

CODE: https://gist.github.com/velotiotech/0c5ea23043ab70ffd0767b301ac15b8a.js

10. Run kubeadm init on master1 and master2:

First of all copy /etc/kubernetes/pki/ca.crt, /etc/kubernetes/pki/ca.key, /etc/kubernetes/pki/sa.key, /etc/kubernetes/pki/sa.pub to master1’s and master2’s /etc/kubernetes/pki folder.

Note: Copying this files is crucial, otherwise the other two master nodes won’t go into the ready state.

Copy the config file config.yaml from master0 to master1 and master2. We need to change <master-private-ip> to current master host’s private IP.</master-private-ip>

CODE: https://gist.github.com/velotiotech/e4b9e24c2474827da67b9832a38d6272.js

11. Now you can install pod network on all three masters to bring them in the ready state. I am using weave-net pod network, to apply weave-net run:

CODE: https://gist.github.com/velotiotech/dea4d085c2f839cfc3e5d325dbb68ac5.js

12. By default, k8s doesn’t schedule any workload on the master, so if you want to schedule workload on master node as well, taint all the master nodes using the command:

CODE: https://gist.github.com/velotiotech/d348ca1fde7cc64bbe4f00811d90b4fd.js

13. Now that we have functional master nodes, we can join some worker nodes:

Use the join string you got at the end of kubeadm init command

CODE: https://gist.github.com/velotiotech/aa901cc9958e18c9d4fcfd9e69a49f54.js

High Availability in action

The Kubernetes HA cluster will look like:

CODE: https://gist.github.com/velotiotech/cc584efa3c759cdd6aaf9ffc0502eeeb.js

CODE: https://gist.github.com/velotiotech/607ce55ee8a9b6373aa276765edc6795.js

After failing over one master node the Kubernetes cluster is still accessible.

CODE: https://gist.github.com/velotiotech/cc584efa3c759cdd6aaf9ffc0502eeeb.js

CODE: https://gist.github.com/velotiotech/607ce55ee8a9b6373aa276765edc6795.js

Even after one node failed, all the important components are up and running. The cluster is still accessible and you can create more pods, deployment services etc.

CODE: https://gist.github.com/velotiotech/bfc23cf2f6101851ad6a6e05de2b567a.js

Conclusion

High availability is an important part of reliability engineering, focused on making system reliable and avoid any single point of failure of the complete system. At first glance, its implementation might seem quite complex, but high availability brings tremendous advantages to the system that requires increased stability and reliability. Using highly available cluster is one of the most important aspects of building a solid infrastructure.