In modern cloud-native environments, service discovery plays a vital role in ensuring seamless communication between microservices. Kubernetes, the de facto standard for container orchestration, offers built-in mechanisms to facilitate service discovery using DNS and Kubernetes Services. This blog explores the inner workings of Kubernetes service discovery, with a technical focus tailored for intermediate and advanced users.

What is Service Discovery in Kubernetes?

Service discovery refers to the automatic detection of services within a network. In Kubernetes, this means identifying Pods or Services without hardcoding their IP addresses. Kubernetes abstracts this complexity by assigning Services a stable DNS name and virtual IP (ClusterIP), allowing for scalable and dynamic service resolution.

How Kubernetes Services Work

Kubernetes provides different types of Services to enable access to Pods:

  • ClusterIP (default): Exposes the service on a cluster-internal IP.
  • NodePort: Exposes the service on each Node’s IP at a static port.
  • LoadBalancer: Integrates with external load balancers in supported cloud providers.
  • ExternalName: Maps a service to a DNS name outside the cluster.

Each of these types plays a role in service discovery depending on the use case.

DNS-Based Service Discovery

Kubernetes uses CoreDNS (or kube-dns in older versions) to resolve service names within the cluster.

CoreDNS Basics

CoreDNS runs as a cluster DNS and handles queries like:

my-service.my-namespace.svc.cluster.local

When a Pod wants to communicate with my-service, Kubernetes automatically resolves this DNS name to the corresponding ClusterIP. This abstraction allows services to scale and change IPs without requiring reconfiguration.

Search Path and Resolution Order

The search path is defined in /etc/resolv.conf inside each Pod:

search my-namespace.svc.cluster.local svc.cluster.local cluster.local

This configuration enables flexible resolution using short names like my-service.

Headless Services and Stateful Applications

For advanced scenarios like StatefulSets or direct Pod-to-Pod communication, Headless Services (clusterIP: None) are used.

How It Works

Headless services allow clients to discover individual Pod IPs instead of routing through a ClusterIP. For example, a StatefulSet named web with 3 replicas will result in DNS records like:

web-0.web.default.svc.cluster.local
web-1.web.default.svc.cluster.local
web-2.web.default.svc.cluster.local

This pattern is crucial for databases like Cassandra, Kafka, or HDFS, where stable network identities are needed.

Endpoints and EndpointSlices

When a Service is created, Kubernetes generates associated Endpoints or EndpointSlices to track the backing Pods.

  • Endpoints store IP addresses and ports.
  • EndpointSlices (introduced in v1.17) scale better and support more metadata.

These resources are automatically updated as Pods come and go, ensuring the DNS name always points to live Pods.

Best Practices for DNS and Service Discovery

  • Avoid hardcoding IP addresses; always use DNS.
  • Use readiness probes to prevent routing traffic to unready Pods.
  • Prefer ClusterIP services for internal communication.
  • Use Headless Services for StatefulSets and when client-side load balancing is needed.
  • Monitor DNS performance using CoreDNS metrics and logs.

Common Pitfalls and Troubleshooting

1. DNS Resolution Failure

  • Check if coredns Pods are running:

    kubectl get pods -n kube-system -l k8s-app=kube-dns

  • Verify that /etc/resolv.conf inside the Pod has the correct search path.

2. Misconfigured Services

  • Ensure that selectors in your Service match the Pod labels.
  • Use kubectl describe service to validate service endpoints.

3. Network Policies Blocking DNS

  • Verify that network policies allow traffic to the CoreDNS Pods.

Real-World Example

Suppose you have a frontend service that needs to communicate with a backend:

Deployment YAML (Backend)

apiVersion: v1
kind: Service
metadata:
  name: backend
spec:
  selector:
    app: backend
  ports:
    - port: 80
      targetPort: 8080

Your frontend code can simply make a request to:

http://backend

Thanks to DNS and Kubernetes Services, the backend is discoverable without knowing its IP address.

Conclusion

Kubernetes makes service discovery seamless using DNS and Services, abstracting away the complexity of dynamic IPs and scaling. Understanding these mechanisms is essential for designing resilient and scalable microservices. Whether you’re deploying a large-scale analytics engine like Spark or setting up a high-availability HDFS cluster, mastering Kubernetes service discovery will significantly streamline your development and operations.