Broker Pod Availability

The advantage of running applications like Kafka on containers is the app level abstraction. This enables Container Orchestration platforms like Kubernetes to scale them as needed and manage the container failures seamlessly. If a Kafka broker pod fails in Kubernetes due to the node failure, Kubernetes with the help of Portworx can respawn the broker pod on a node that has the portworx replica. This is enabled by the Portworx Platform’s storage-aware scheduling called STORK (STorage Orchestrator Runtime for Kubernetes). STORK performs health monitoring of Portworx services, kubelet, and other components of the system, and when a broker failure is detected, it will reach faster than kube-scheduler in rescheduling the broker pod to a healthy node.

When a failed broker is respawned on a new node with access to the original volume, the broker has all the topics and partitions for it to join the list of in-sync replicas. Hence, the broker does not need to replicate huge amounts of data from the ‘leader’ across the network, rather only catch up with the offsets that were added since the broker went offline.

This feature allows the Kafka Broker pod to be available within a short period of time, the time STORK takes to reschedule the pod, as opposed to hours or days in a non-container world, especially when the Kafka broker fails permanently.

The broker pod's availability time after a failure is generally around three minutes or more as STORK scans every two minutes (configurable) for any node failure and once it finds a node failure, it waits for a minute before it attaches the volume to another node. To reduce this time by half, you can configure the parameter --health-monitor-interval from the default of 120 seconds to 30 seconds.

To change the settings, run the following command to edit the stork’s option.

kubectl edit deploy stork -n kube-system

and change the value --health-monitor-interval to 30 in the editor and save the changes. You should see the message deployment.apps/stork edited.

kubectl edit deploy stork -n kube-system
deployment.apps/stork edited

Unless the broker respawn time of three minutes or more is not good enough, you can consider changing the settings.