Guide: Kubernetes Multi-Master HA Cluster with kubeadm

Hello everybody, Tansanrao here! This post is going to guide you into setting up a Multi-Master HA (High-Availability) Kubernetes Cluster on bare-metal or virtual machines.

All our VM images will be based on Ubuntu 20.04.1 Server and for the purpose of this guide, will be Virtual Machines on a VMware ESXi host.

We will require 7 Virtual Machines with a minimum spec of 2 Cores and 4GB RAM per Node for decent performance. Also make sure that you have Static IPs assigned on your DHCP Server.

We are using the following Hostnames & IP Assignments:

• 1 HAProxy Load Balancer Node
  — k8s-haproxy : 192.168.1.112
• 3 Etcd/Kubernetes Master Nodes
  — k8s-master-a : 192.168.1.113
  — k8s-master-b : 192.168.1.114
  — k8s-master-c : 192.168.1.115
• 3 Kubernetes Worker Nodes
  — k8s-node-a : 192.168.1.116
  — k8s-node-b : 192.168.1.117
  — k8s-node-c : 192.168.1.118

We will also require 1 linux client machine, if unavailable, the client tools may be installed on the HAProxy node.

The minimum for production use is 2 physical hosts with at least 1 Master on each with the recommended being 3 hosts with 1 Master and 1 Worker Node Each with an external load balancer. For the sake of this guide, I am running all 7 nodes on the same ESXi host. A single host should be safe enough to use for lab and test environments but do not run anything mission critical on it.

Let’s get started!

Prepare Virtual Machines / Servers

Start by preparing 7 machines with Ubuntu 20.04.1 Server using the correct hostnames and IP addresses. Once done, power on all of them and apply the latest updates using:

sudo apt update && sudo apt upgrade

Setting up Client Tools

Installing cfssl.

CFSSL is an SSL tool by Cloudflare which lets us create our Certs and CAs.

Step 1 - Download the binaries

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

Step 2 - Add the execution permission to the binaries

chmod +x cfssl*

Step 3 - Move the binaries to /usr/local/bin

sudo mv cfssl_linux-amd64 /usr/local/bin/cfssl
sudo mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

Step 4 - Verify the installation

cfssl version

Installing kubectl

Step 1 - Get the binary

make sure it’s the same version as the cluster, in our case we are using v1.19

curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.19.0/bin/linux/amd64/kubectl

Step 2 - Add the execution permission to the binary

chmod +x kubectl

Step 3 - Move the binary to /usr/local/bin

sudo mv kubectl /usr/local/bin

Step 4 - Verify the installation

kubectl version

Installing HAProxy Load Balancer

As we will be deploying three Kubernetes master nodes, we need to deploy an HAProxy Load Balancer in front of them to distribute the traffic.

Step 1 - SSH to the HAProxy VM

ssh [email protected]

Step 2 - Install HAProxy

sudo apt-get install haproxy

Step 3 - Configure HAProxy

sudo nano /etc/haproxy/haproxy.cfg

Enter the following config:

global
...
default
...
frontend kubernetes
bind 192.168.1.112:6443
option tcplog
mode tcp
default_backend kubernetes-master-nodes


backend kubernetes-master-nodes
mode tcp
balance roundrobin
option tcp-check
server k8s-master-a 192.168.1.113:6443 check fall 3 rise 2
server k8s-master-b 192.168.1.114:6443 check fall 3 rise 2
server k8s-master-c 192.168.1.115:6443 check fall 3 rise 2

Step 4 - Restart HAProxy

sudo systemctl restart haproxy

Generating the TLS certificates

These steps can be done on your Linux client if you have one or on the HAProxy machine depending on where you installed the cfssl tool.

Creating a Certificate Authority

Step 1 - Create the certificate authority configuration file

nano ca-config.json

Enter the following config:

{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "kubernetes": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "8760h"
      }
    }
  }
}

Step 2 - Create the certificate authority signing request configuration file

nano ca-csr.json

Enter the following config, Change the names as necessary:

{
  "CN": "Kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
  {
    "C": "IN",
    "L": "Belgaum",
    "O": "Tansanrao",
    "OU": "CA",
    "ST": "Karnataka"
  }
 ]
}

Step 3 - Generate the certificate authority certificate and private key

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

Step 4 - Verify that the ca-key.pem and the ca.pem were generated

ls -la

Creating the certificate for the Etcd cluster

Step 1 - Create the certificate signing request configuration file

nano kubernetes-csr.json

Add the following config:

{
  "CN": "Kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
  {
    "C": "IN",
    "L": "Belgaum",
    "O": "Tansanrao",
    "OU": "CA",
    "ST": "Karnataka"
  }
 ]
}

Step 2 - Generate the certificate and private key

cfssl gencert \
-ca=ca.pem \
-ca-key=ca-key.pem \
-config=ca-config.json \
-hostname=192.168.1.112,192.168.1.113,192.168.1.114,192.168.1.115,127.0.0.1,kubernetes.default \
-profile=kubernetes kubernetes-csr.json | \
cfssljson -bare kubernetes

Step 3 - Verify that the kubernetes-key.pem and the kubernetes.pem file were generated.

ls -la

Step 4 - Copy the certificate to each nodes

scp ca.pem kubernetes.pem kubernetes-key.pem [email protected]:~
scp ca.pem kubernetes.pem kubernetes-key.pem [email protected]:~
scp ca.pem kubernetes.pem kubernetes-key.pem [email protected]:~
scp ca.pem kubernetes.pem kubernetes-key.pem [email protected]:~
scp ca.pem kubernetes.pem kubernetes-key.pem [email protected]:~
scp ca.pem kubernetes.pem kubernetes-key.pem [email protected]:~

Preparing the nodes for kubeadm

Initial Setup for all master and node machines

Copy the commands below and paste them into a setup.sh file and then execute it with . setup.sh.

This script will check for and uninstall older versions of docker and will replace it with the latest version of docker-ce for ubuntu 20.04. It will also add the kubernetes repository and install kubelet, kubeadm, kubectl and will also mark the packages to prevent auto updates.

sudo apt-get remove docker docker-engine docker.io containerd runc

sudo apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

sudo apt-get install -y docker-ce docker-ce-cli containerd.io

sudo usermod -aG docker tansanrao

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

sudo swapoff -a

Now we need to turn swap off for the nodes by editing /etc/fstab on each machine.

sudo nano /etc/fstab

Comment the line that starts with /swap or /swap.img. My /etc/fstab looks like this after making the change.

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/ubuntu-vg/ubuntu-lv during curtin installation
/dev/disk/by-id/dm-uuid-LVM-s96R5iaP77QRtKuZZ0mYLuJcarDuQldMUj3yYFLQDRKWOqz9PHtLTnMMl2cbxpkC / ext4 defaults 0 0
# /boot was on /dev/sda2 during curtin installation
/dev/disk/by-uuid/bcc851c2-bbc4-44c0-bb36-c142eedd63a6 /boot ext4 defaults 0 0
#/swap.img      none    swap    sw      0       0

Installing and configuring Etcd on all 3 Master Nodes

Step 1 - Download and move etcd files and certs to their respective places

sudo mkdir /etc/etcd /var/lib/etcd

sudo mv ~/ca.pem ~/kubernetes.pem ~/kubernetes-key.pem /etc/etcd

wget https://github.com/etcd-io/etcd/releases/download/v3.4.13/etcd-v3.4.13-linux-amd64.tar.gz

tar xvzf etcd-v3.4.13-linux-amd64.tar.gz

sudo mv etcd-v3.4.13-linux-amd64/etcd* /usr/local/bin/

Step 2 - Create an etcd systemd unit file

sudo nano /etc/systemd/system/etcd.service

Enter the following config:

[Unit]
Description=etcd
Documentation=https://github.com/coreos


[Service]
ExecStart=/usr/local/bin/etcd \
  --name 192.168.1.113 \
  --cert-file=/etc/etcd/kubernetes.pem \
  --key-file=/etc/etcd/kubernetes-key.pem \
  --peer-cert-file=/etc/etcd/kubernetes.pem \
  --peer-key-file=/etc/etcd/kubernetes-key.pem \
  --trusted-ca-file=/etc/etcd/ca.pem \
  --peer-trusted-ca-file=/etc/etcd/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth \
  --initial-advertise-peer-urls https://192.168.1.113:2380 \
  --listen-peer-urls https://192.168.1.113:2380 \
  --listen-client-urls https://192.168.1.113:2379,http://127.0.0.1:2379 \
  --advertise-client-urls https://192.168.1.113:2379 \
  --initial-cluster-token etcd-cluster-0 \
  --initial-cluster 192.168.1.113=https://192.168.1.113:2380,192.168.1.114=https://192.168.1.114:2380,192.168.1.115=https://192.168.1.115:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5



[Install]
WantedBy=multi-user.target

Replace the IP address on all fields except the —initial-cluster field to match the machine IP.

Step 3 - Reload the daemon configuration.

sudo systemctl daemon-reload

Step 4 - Enable etcd to start at boot time.

sudo systemctl enable etcd

Step 5 - Start etcd.

sudo systemctl start etcd

Repeat the process for all 3 master nodes and then move to step 6.

Step 6 - Verify that the cluster is up and running.

ETCDCTL_API=3 etcdctl member list

It should give you an output similar to this:

73ea126859b3ba4, started, 192.168.1.114, https://192.168.1.114:2380, https://192.168.1.114:2379, false
a28911111213cc6c, started, 192.168.1.115, https://192.168.1.115:2380, https://192.168.1.115:2379, false
feadb5a763a32caa, started, 192.168.1.113, https://192.168.1.113:2380, https://192.168.1.113:2379, false

Initialising the Master Nodes

Initialising the first Master Node

Step 1 - SSH to the first Master Node

ssh [email protected]

Step 2 - Create the configuration file for kubeadm

nano config.yaml

Enter the following config:

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.19.0
controlPlaneEndpoint: "192.168.1.112:6443"
etcd:
  external:
    endpoints:
      - https://192.168.1.113:2379
      - https://192.168.1.114:2379
      - https://192.168.1.115:2379
    caFile: /etc/etcd/ca.pem
    certFile: /etc/etcd/kubernetes.pem
    keyFile: /etc/etcd/kubernetes-key.pem
networking:
  podSubnet: 10.30.0.0/24
apiServer:
  certSANs:
    - "192.168.1.112"
  extraArgs:
    apiserver-count: "3"

Add any additional domains or IP Addresses that you would want to connect to the cluster under certSANs.

Step 3 - Initialise the machine as a master node

sudo kubeadm init --config=config.yaml

Step 4 - Copy the certificates to the two other masters

sudo scp -r /etc/kubernetes/pki [email protected]:~
sudo scp -r /etc/kubernetes/pki [email protected]:~

Initialising the second Master Node

Step 1 - SSH to the second Master Node

ssh [email protected]

Step 2 - Remove the apiserver.crt and apiserver.key

rm ~/pki/apiserver.*

Step 3 - Move the certificates to the /etc/kubernetes directory.

sudo mv ~/pki /etc/kubernetes/

Step 4 - Create the configuration file for kubeadm

nano config.yaml

Enter the following config:

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.19.0
controlPlaneEndpoint: "192.168.1.112:6443"
etcd:
  external:
    endpoints:
      - https://192.168.1.113:2379
      - https://192.168.1.114:2379
      - https://192.168.1.115:2379
    caFile: /etc/etcd/ca.pem
    certFile: /etc/etcd/kubernetes.pem
    keyFile: /etc/etcd/kubernetes-key.pem
networking:
  podSubnet: 10.30.0.0/24
apiServer:
  certSANs:
    - "192.168.1.112"
  extraArgs:
    apiserver-count: "3"

Step 5 - Initialise the machine as a master node.

sudo kubeadm init --config=config.yaml

Initialising the third master node

Step 1 - SSH to the third Master Node

ssh [email protected]

Step 2 - Remove the apiserver.crt and apiserver.key

rm ~/pki/apiserver.*

Step 3 - Move the certificates to the /etc/kubernetes directory.

sudo mv ~/pki /etc/kubernetes/

Step 4 - Create the configuration file for kubeadm

nano config.yaml

Enter the following config:

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.19.0
controlPlaneEndpoint: "192.168.1.112:6443"
etcd:
  external:
    endpoints:
      - https://192.168.1.113:2379
      - https://192.168.1.114:2379
      - https://192.168.1.115:2379
    caFile: /etc/etcd/ca.pem
    certFile: /etc/etcd/kubernetes.pem
    keyFile: /etc/etcd/kubernetes-key.pem
networking:
  podSubnet: 10.30.0.0/24
apiServer:
  certSANs:
    - "192.168.1.112"
  extraArgs:
    apiserver-count: "3"

Step 5 - Initialise the machine as a master node.

sudo kubeadm init --config=config.yaml

Step 6 - Save the join command printed in the output after the above command

Example Output:

kubeadm join 192.168.1.112:6443 --token c5tkdt.47tjw72synw7qbn9 \
    --discovery-token-ca-cert-hash sha256:069081b1116e821958da62e8d1c185b1df94849bdeb414761e992585f4034ce8 

NOTE: use the output from your terminal and not this post.

Configure kubectl on the client machine

Step 1 - SSH to one of the master nodes

ssh [email protected]

Step 2 - Add permissions to the admin.conf file

sudo chmod +r /etc/kubernetes/admin.conf

Step 3 - From the client machine, copy the configuration file.

scp [email protected]:/etc/kubernetes/admin.conf .

Step 4 - Create and configure the kubectl configuration directory.

mkdir ~/.kube
mv admin.conf ~/.kube/config
chmod 600 ~/.kube/config

Step 5 - Go back to the SSH session and revert the permissions of the config file

sudo chmod 600 /etc/kubernetes/admin.conf

Step 6 - Test to see if you can access the Kubernetes API from the client machine

kubectl get nodes

Expected Output:

NAME           STATUS     ROLES    AGE     VERSION
k8s-master-a   NotReady   master   44m     v1.19.2
k8s-master-b   NotReady   master   11m     v1.19.2
k8s-master-c   NotReady   master   5m50s   v1.19.2

Initialise the worker nodes

SSH into each worker node and execute the kubeadm join command that you copied previously.

sudo kubeadm join 192.168.1.112:6443 --token c5tkdt.47tjw72synw7qbn9 \
    --discovery-token-ca-cert-hash sha256:069081b1116e821958da62e8d1c185b1df94849bdeb414761e992585f4034ce8 

Once all three worker nodes have joined the cluster, test the API to check the available nodes from the client machine.

kubectl get nodes

Expected Output:

NAME           STATUS     ROLES    AGE   VERSION
k8s-master-a   NotReady   master   53m   v1.19.2
k8s-master-b   NotReady   master   20m   v1.19.2
k8s-master-c   NotReady   master   14m   v1.19.2
k8s-node-a     NotReady   <none>   26s   v1.19.2
k8s-node-b     NotReady   <none>   19s   v1.19.2
k8s-node-c     NotReady   <none>   18s   v1.19.2

Deploying the overlay network

We will be using Project Calico as the overlay network but you are free to use any other alternatives such as Flannel or WeaveNet

Apply the manifest to deploy calico overlay

curl https://docs.projectcalico.org/manifests/calico.yaml -O
kubectl apply -f calico.yaml

Check that all the pods deployed correctly

kubectl get pods -n kube-system

Congratulations! Your Bare-metal HA Cluster is ready for use. I recommend setting up Rancher Server for managing it and to setup Traefik as Ingress Controller, Longhorn as a Persistent Volume Provider, Prometheus & Grafana for Metrics and EFK Stack for Logging and Distributed Tracing. Guides for the same are in the works and will be posted in the coming weeks.

Follow me on Twitter & Instagram for behind the scenes and updates.