In Hadoop, the Namenode is the single source of truth for HDFS metadata — including file system structure, file-to-block mappings, and permissions. If the Namenode fails, the entire file system becomes inaccessible, even though the actual data blocks remain intact on the DataNodes.

To overcome this single point of failure, HDFS supports Namenode High Availability (HA) — allowing multiple Namenodes to ensure uninterrupted access to metadata in production environments.

In this blog, we’ll explore how metadata is managed in HDFS, how Namenode HA works, and how to implement a resilient architecture with automatic failover and shared metadata consistency.

Importance of Metadata in HDFS

The Namenode manages:

  • File system hierarchy
  • Block locations and replication info
  • File permissions and ownership
  • File sizes, timestamps, and quotas

All metadata is held in-memory on the Namenode and persisted to:

  • FsImage: The checkpointed metadata snapshot
  • EditLog: A write-ahead log of all changes since the last checkpoint

If the Namenode fails without HA, clients lose access to HDFS — even if DataNodes are functional.

Namenode High Availability Architecture

Namenode HA introduces:

  • Active Namenode: Handles all client requests
  • Standby Namenode: In sync but passive
  • Shared Journal (QJM or NFS): Ensures consistent edit logs between nodes
  • ZooKeeper Failover Controller (ZKFC): Manages automatic failover
+------------------+
|    ZooKeeper     |
+--------+---------+
|
+-----------+-----------+
|                       |
+------------+        +-------------+
|  Active NN |        |  Standby NN |
+------------+        +-------------+
|                         |
|        Shared Journal   |
+-----------+-------------+
|
+------+------+
| JournalNodes |
+-------------+

Configuring Namenode HA

  1. Enable HA in core-site.xml:
<property>
<name>fs.defaultFS</name>
<value>hdfs://cluster1</value>
</property>
<property>
<name>dfs.nameservices</name>
<value>cluster1</value>
</property>
<property>
<name>dfs.ha.namenodes.cluster1</name>
<value>nn1,nn2</value>
</property>
  1. Define Namenode RPC addresses:
<property>
<name>dfs.namenode.rpc-address.cluster1.nn1</name>
<value>nn1-host:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.cluster1.nn2</name>
<value>nn2-host:8020</value>
</property>
  1. Set shared edit directory using JournalNodes:
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://jn1:8485;jn2:8485;jn3:8485/cluster1</value>
</property>
  1. Enable automatic failover with ZKFC:
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>

Start ZKFC on both Namenodes to monitor and trigger failover.

Metadata Synchronization and Consistency

To keep the Standby Namenode in sync:

  • It continuously reads the edit logs from JournalNodes
  • It maintains its own in-memory copy of the metadata
  • Upon failover, it becomes Active and serves client requests immediately

Checkpointing occurs periodically:

  • The FsImage is updated from edit logs
  • Both Active and Standby maintain consistent views

Failover and Recovery

Failover occurs automatically when:

  • Active NN becomes unresponsive
  • ZKFC detects failure via ZooKeeper heartbeat
  • Standby NN is promoted to Active

Manual failover (for maintenance):

hdfs haadmin -failover nn1 nn2

Monitor failovers and sync health using:

hdfs haadmin -getServiceState nn1
hdfs haadmin -checkHealth nn2

Best Practices

  • Use at least 3 JournalNodes for quorum
  • Deploy ZKFC on both Namenodes
  • Store edit logs and FsImage on separate disks for durability
  • Monitor metadata metrics using Prometheus or Cloudera Manager
  • Backup metadata regularly via hdfs dfsadmin -fetchImage

Monitoring and Troubleshooting

  • Use hdfs fsck / to verify metadata integrity
  • Check ZKFC and JournalNode logs for failover events
  • Use jps to confirm daemons (NN, DN, ZKFC, JournalNode) are running
  • Audit metadata changes with Hadoop audit logs

Conclusion

High Availability for the Namenode is essential for building a fault-tolerant Hadoop environment. By configuring HA with JournalNodes and ZKFC, and ensuring metadata synchronization between Active and Standby nodes, you can deliver continuous access to HDFS — even in the face of hardware or network failures.

With proper setup and monitoring, Namenode HA ensures that metadata — the heart of HDFS — is always available, consistent, and protected.