Scaling stateless applications in Kubernetes is relatively straightforward. However, stateful applications—such as databases, distributed systems, and message queues—introduce complexity. They require stable network identities, persistent storage, and ordered deployment and scaling. This is where Kubernetes StatefulSets and Persistent Volumes come into play.

This blog dives deep into how Kubernetes handles stateful workloads and explains how to architect robust, scalable, and resilient deployments using these primitives.

What Are Stateful Applications?

Unlike stateless applications, stateful applications maintain information across sessions or interactions. Examples include:

  • Databases: PostgreSQL, MySQL
  • Message Brokers: Kafka, RabbitMQ
  • Distributed Filesystems: HDFS, GlusterFS
  • Caches: Redis in persistent mode

Scaling these systems requires consistent identity and durable storage to maintain state during restarts, rescheduling, or scaling operations.

Enter StatefulSets

A StatefulSet is a Kubernetes controller designed specifically for managing stateful applications.

Key Features of StatefulSets
  • Stable Pod Identity: Pods are named sequentially (e.g., web-0, web-1, web-2) and retain their identity across reschedules.
  • Stable Network Identity: Each Pod gets a stable DNS entry (e.g., web-0.web.default.svc.cluster.local).
  • Ordered Deployment and Termination: Pods are created, scaled, and deleted in strict order.
  • Persistent Volumes: Each Pod gets its own PersistentVolumeClaim (PVC), ensuring data persists even if the Pod is deleted.

These features make StatefulSets ideal for deploying clustered or replicated services.

Persistent Volumes and Claims

Stateful applications require PersistentVolumes (PVs) and PersistentVolumeClaims (PVCs) to retain data beyond the lifecycle of a Pod.

Storage Workflow
  1. Define a StorageClass (optional, but recommended) for dynamic provisioning.
  2. Each StatefulSet replica automatically generates a PVC.
  3. The volume is mounted into the Pod at a specific mount path.
Example StorageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
Example PVC Template in StatefulSet
volumeClaimTemplates:
- metadata:
    name: data
  spec:
    accessModes: ["ReadWriteOnce"]
    storageClassName: fast-ssd
    resources:
      requests:
        storage: 10Gi

This ensures each Pod has its own dedicated volume named data-web-0, data-web-1, etc.

Example: Scaling a Stateful Redis Cluster

Here’s a simplified StatefulSet for Redis:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
spec:
  serviceName: "redis"
  replicas: 3
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis:7
        ports:
        - containerPort: 6379
        volumeMounts:
        - name: data
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 1Gi

Each Redis Pod gets its own persistent volume and hostname like:

  • redis-0.redis.default.svc.cluster.local
  • redis-1.redis.default.svc.cluster.local

This stable identity is essential for configuring Redis replication or Redis Sentinel.

Managing Scaling and Rolling Updates

Scaling StatefulSets:

kubectl scale statefulset redis --replicas=5

New Pods will be named redis-3, redis-4, etc., and each will get its own PVC. Data from previous Pods remains unaffected.

Rolling Updates:

  • Kubernetes ensures updates proceed one Pod at a time.
  • You can use partition strategy to update only a subset of replicas.
  • Always test storage compatibility before upgrading software versions.
updateStrategy:
  type: RollingUpdate
  rollingUpdate:
    partition: 2

This config ensures only Pods redis-2 and above are updated.

Headless Services for StatefulSets

StatefulSets must be paired with a Headless Service to enable DNS-based discovery for each Pod:

apiVersion: v1
kind: Service
metadata:
  name: redis
spec:
  clusterIP: None
  selector:
    app: redis
  ports:
  - port: 6379

This ensures that DNS entries for each Pod are created, like:

  • redis-0.redis.default.svc.cluster.local
  • redis-1.redis.default.svc.cluster.local

Clients or scripts can then refer to individual replicas reliably.

Best Practices

  • Avoid sharing PVCs among Pods.
  • Choose ReadWriteOnce access mode for most cloud storage backends.
  • Use readiness probes to prevent traffic routing before init is complete.
  • Label StatefulSets and PVCs for easy management and monitoring.
  • Regularly snapshot volumes to prevent data loss during migrations or failures.

Common Pitfalls

  • PVCs Are Not Deleted Automatically: You must manually clean up PVCs if StatefulSet is deleted without removing volumes.
  • Pod Disruption: Use PodDisruptionBudgets to ensure quorum in distributed systems.
  • Storage Latency: Match StorageClass IOPS and latency to application requirements.
  • Zonal Failures: Spread replicas across multiple zones when using cloud providers.

Conclusion

StatefulSets and PersistentVolumes are foundational for scaling stateful applications in Kubernetes. They provide stability, persistence, and reliability—critical for production-grade systems like databases, data lakes, and distributed caches. By mastering these constructs, you can confidently deploy and manage complex applications with consistent state management across your cluster.