Running Apache Kafka on Kubernetes has become the go-to approach for organizations looking to scale their real-time data pipelines in a cloud-native and containerized environment. While deploying Kafka manually on Kubernetes can be complex, Helm — the package manager for Kubernetes — simplifies the process by providing preconfigured, reusable charts.

In this post, you’ll learn how to deploy Kafka on Kubernetes using Helm, including how to manage persistence, expose services, scale brokers, and apply production-ready configurations.

Why Use Helm to Deploy Kafka?

Benefits of using Helm for Kafka deployments:

  • Simplified deployment with parameterized values
  • Consistent infrastructure-as-code setup
  • Easy upgrades and rollbacks
  • Scalable configurations for brokers and Zookeeper
  • Pre-built community Helm charts (e.g., Bitnami, Confluent)

Prerequisites

  • A Kubernetes cluster (local via Minikube or cloud-based like EKS/GKE/AKS)
  • Helm installed on your machine (helm version)
  • kubectl configured to access your cluster
  • StorageClass for persistent volumes

Step 1: Add a Kafka Helm Chart Repository

Use the Bitnami chart (open-source and production-ready):

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

Step 2: Install Kafka with Helm

You can deploy Kafka using default settings:

helm install kafka bitnami/kafka

Or customize using a values.yaml file:

replicaCount: 3

zookeeper:
enabled: true
replicaCount: 3

persistence:
enabled: true
size: 10Gi

externalAccess:
enabled: true
service:
type: LoadBalancer

auth:
enabled: true
clientProtocol: sasl
interBrokerProtocol: sasl
sasl:
mechanisms: plain
clientUsers: [user]
clientPasswords: [password]

Install with custom config:

helm install kafka -f values.yaml bitnami/kafka

Step 3: Verifying Kafka Deployment

Check all pods:

kubectl get pods

Expose Kafka externally (if needed):

kubectl get svc kafka
# or
kubectl port-forward svc/kafka 9092:9092

Step 4: Producing and Consuming Messages

Use Kafka CLI tools via pod exec:

kubectl exec -it kafka-0 -- bash

# Produce
kafka-console-producer.sh --broker-list localhost:9092 --topic test

# Consume
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

Step 5: Monitoring Kafka on Kubernetes

Integrate with Prometheus and Grafana:

metrics:
kafka:
enabled: true
zookeeper:
enabled: true

Then install the kube-prometheus-stack or use Kafka-specific dashboards.

Helm charts also support:

  • Liveness and readiness probes
  • Resource limits
  • Auto-restart on failure

Step 6: Scaling the Kafka Cluster

Update replica count in values.yaml:

replicaCount: 5

Apply changes:

helm upgrade kafka -f values.yaml bitnami/kafka

Ensure your partitions are configured to match the broker count for optimal distribution.

Best Practices for Production Kafka on Kubernetes

  • Use persistent volumes for broker data
  • Enable authentication and TLS encryption
  • Isolate Zookeeper and Kafka nodes across availability zones
  • Implement log retention and topic cleanup policies
  • Monitor replication lag, broker health, and consumer offsets
  • Use network policies to secure communication between services

Conclusion

Deploying Kafka on Kubernetes using Helm enables you to create a scalable, resilient, and automated data streaming infrastructure with minimal manual setup. With Helm charts from Bitnami or Confluent, you gain access to production-grade defaults and modular configurations that simplify cluster management.

Whether you’re testing Kafka locally or deploying multi-broker clusters in production, Helm empowers teams to move faster with repeatable and cloud-native Kafka deployments.