High Availability Tests¶

Testing High Availability On Single Node¶

High availability ensures that application pods have minimum downtime. At the pod level, high availability can be tested using Kubernetes commands or trying to kill the application from inside the pod. Killing applications from the pod can be done by using kill command.

Test the high availability by getting the name of the pod and then deleting the pod manually using kubectl. This can be done as follows:

$ kubectl get pods
$ kubectl delete pod <pod_name>
$ kubectl get pods

The older pod will be in terminating state, and new pod will come in running state.

High availability within the pod can also be tested by killing the process running inside the pod. One can do this by running the following command:

$ kubectl exec <pod_name> -- /bin/bash -c '/usr/bin/kill -9 $(pidof -s dpdk-testpmd)'

After running this command, the pod will restart and output comes as below:

$ kubectl get pods
NAME                   READY   STATUS   RESTARTS      AGE
dpdk-9d8474bd8-xl7kkr   0/1     Error    0             19m

$ kubectl get pods
NAME                   READY   STATUS    RESTARTS      AGE
dpdk-9d8474bd8-xl7kkr   1/1     Running   1 (44s ago)   2m20s

As shown in the output, same pod terminates with Error status and restarts itself if the process inside the pod is killed. Process or application running inside the pod also gets restarted. This way, high availability on single node cluster can be tested.

Testing High Availability in a multi-worker node cluster¶

In a multi-worker node cluster, when one or more nodes start malfunctioning, Kubernetes will generally attempt to kill the pods on those nodes and create new replacement pods on the other healthy nodes. The way Kubernetes does this is to constantly monitor nodes for exceptional conditions and assign taints to nodes when specific health conditions are met. Pods without tolerances for these taints are killed on those nodes and restarted on nodes that either don’t have any taints or have only those taints that the pods can tolerate.

There are various types of taints, and most are managed by Kubernetes itself. A user can also manually assign taints to nodes. The Kubernetes documentation explains taints & tolerances in more details.

This document outlines 4 different ways to try out HA in a multi-worker node setup.

Add a taint manually

In this method, manually assign the not-ready taint to the node. This will cause all the pods on that node to go into terminating state. Replacement pods will be created on the other healthy nodes.
```
$ kubectl taint nodes <worker-node1-name> node-role.kubernetes.io/not-ready:NoExecute
```
After waiting for sometime (>5 minutes), getting the list of all the running pods on the cluster will show the pods on the tainted node have been killed and replacement pods have been created on the other healthy nodes.
```
$ kubectl get pods -o wide
NAME                            READY   STATUS    RESTARTS   AGE   IP                NODE               NOMINATED NODE   READINESS GATES
dpdk-testpmd-pod-hash1          1/1     Running   0          54m   pod1-ip           worker-node2-name  <none>           <none>
dpdk-testpmd-pod-hash2          1/1     Running   0          45s   pod2-ip           worker-node2-name  <none>           <none>
```
To remove the taint run this command:
```
$ kubectl taint nodes <worker-node1-name> node-role.kubernetes.io/not-ready:NoExecute-
```
Please note the - at the end of that command. That makes Kubernetes remove the previously assigned taint. The replacement pods that were created on the other healthy nodes will continue to run on those nodes. If those pods or the nodes where they are running start malfunctioning, Kubernetes may choose to run those pods on the node that was previously tainted.
Use iptable rules to simulate a network partition

Every worker node needs to send heartbeats to the controller node periodically. The controller node uses these heartbeats to determine if a node is healthy or not. In case of a network partition, the connection between a worker node and a controller node is severed. Therefore, the heartbeats don’t reach the controller node, and the controller node marks the worker node as unreachable.

A network partition can be simulated using iptable rules. To create a iptable rule, the IP address of the controller node is needed. If the IP address of the controller node was used to create the cluster then that can be found in the inventory.ini file. If only the FQDN of the controller node is known, use nslookup <controller-node-fqdn> to get the IP address of the controller node.

To create the iptable rule, run the following command. By running this command on a worker node, all the outgoing communication from this worker node to the controller node is severed.
```
$ sudo iptables -A OUTPUT -d <controller-node-ip> -p tcp -j DROP
```
After the iptable rule is created, the controller node waits for at least 5 minutes before tainting that worker node with the unreachable taint. All the pods on that node will enter terminating state. Replacement pods are created on the other healthy nodes. Note that the pods on the unreachable node will not be completely terminated until that node is able to communicate with the controller node again.

To remove the iptable rule run the following command.
```
$ sudo iptables -D OUTPUT -d <controller-node-ip> -p tcp -j DROP
```
After running this command, any pod that is in terminating state on the unhealthy node will finally be shutdown.
Kill the Kubelet process

The kubelet process on a worker node is responsible for running containers described in the PodSpec and ensuring that they are healthy. Killing the kubelet process on the worker nodes will cause the controller node to think that the worker node has died and it will put all the pods running on that worker node in terminating state and replacement pods will be created on the other healthy nodes to replace them.

To kill the kubelet process use the following command:
```
$ sudo systemctl stop kubelet
```
To bring back the kubelet process use the following command:
```
$ sudo systemctl start kubelet
```
Shut down a worker node

Of all the tests listed in this section, this is the most extreme test. When a node is shutdown, replacement pods are created on healthy worker nodes.

To test this, shut down one of the nodes. Please be aware that any critical process running on that node will be terminated as a consequence of this action. Also, all the users logged into that node will lose their access till the node is online again. There could be other severe consequences of shutting down the node. Please ensure that it is safe to shutdown the node before proceeding.

The following command can be used to shutdown a node:
```
$ sudo systemctl poweroff
```
Then wait for sometime (>5 minutes) for the replacement pods to be created on the other healthy nodes.

The server can be turned on again by pressing the power button on the server or by using the IPMI interface if the server has one.

Once the server is back online, swap needs to be disabled via sudo swapoff. Additionally, any VFs may need to be recreated.