Kafka, being designed as a distributed and horizontally scalable streaming system, relies on the replication factor of topic-partitions to achieve higher availability of the data if a broker has to fail.
As such for production readiness, you should configure your Kafka topics with a replication factor that's suitable for your application.
Commonly, in production environment where message durability is required, a typical replication factor of 3 is used. Portworx supplements application-level replication through replication at the storage level. Portworx allows you to set the replication between 1 and 3 where 3 offers the highest level of data protection and availability.
In the event of a node outage, replicated volumes allows for the recovery of affected workloads in seconds, since the data is present and can be accessed from other nodes.
Given that both Kafka and Portworx provide replication, what is the ideal replication setting to run Kafka on Kubernetes with Portworx?
Based on our testing, all replication factor combinations across Kafka and Portworx show us that the main decisions when architecting Kafka on Kubernetes are centered around availability, recovery times, and space usage. Our testing revealed that two options were ideal for running Kafka, depending on your business requirements around tolerance to failures.
Option 1: Fast Recoverability
Portworx replication and Kafka replication at 3.
With replication set to two, Portworx replicates the volume at the storage layer which enables quick recovery from pod-, node-, network-, and disk-level failures by quickly respawning the failed broker on another Kubernetes node with the original volume attached. During that short period, when the pod is rescheduled, Kafka’s replication factor of 3 can handle the reassignment of leadership.
Portworx reduces the overall broker pod failure time through STORK and brings back the broker with the original data, which significantly improves the data synchronization by the failed broker with its surviving replica and eliminates the need for Kafka admins to perform any partition rebuild over the network after hard node failures. This option offers quicker recoverability, better availability, resilience, and operational efficiency gains at the cost of some additional storage.
Option 2: Kafka standard configuration
Kafka replication at three and Portworx replication at one.
This is the standard configuration most Kafka users are used to. This is equivalent to running Kafka on a non-Kubernetes environment with Kafka replication at 3, but with some additional benefits.
With this option, during a pod failure or the node failure that does not have the Portworx replica, Portworx can still re-attach its virtual volumes from the surviving nodes to the re-spawned broker pod on another node.
However, if the node that fails owns the Portworx replica, the broker will be unavailable until recovery occurs. During this time, Kafka is still operational as it has two other replicas and when the failed node comes back, the re-spawned broker can synchronize the data from the other Kafka brokers.
It is still operationally better to have Portworx with a single replica than having storage directly on the hosts as in the traditional setup because Portworx can move and reattach its virtual volumes to any other Portworx nodes. This is possible as long as it has an available replica, even if the failed node running the broker is unavailable. This is not something that is possible with the directly attached storage.