Hadoop Distributed File System (HDFS) is designed for reliability and fault tolerance. To ensure that file system operations are not lost during system failures, HDFS uses a critical mechanism known as the Write-Ahead Log (WAL) — commonly referred to in Hadoop as the edit log.

In this post, we’ll dive deep into HDFS write-ahead logging, how it supports data reliability, the role of the NameNode, and best practices for managing edit logs in production environments.

What Is a Write-Ahead Log in HDFS?

A write-ahead log (WAL) is a file that records every metadata operation (like file creation, deletion, or block replication) before it’s applied to the HDFS namespace. In HDFS, this is implemented via the edit log, which resides on the NameNode.

The edit log ensures:

  • Durability: No file system changes are lost on failure
  • Atomicity: Metadata operations are applied in order
  • Recovery: The file system can be reconstructed after a crash

Components of HDFS Metadata Storage

HDFS metadata is maintained in two files:

  1. FsImage: Snapshot of the entire namespace (written periodically)
  2. Edit Log: A sequential log of recent changes made to the file system

Together, these files allow the NameNode to recover the exact state of the file system during a restart.

How Edit Logs Work

When a client performs an operation like creating a file:

  1. The operation is first written to the edit log on the NameNode
  2. Once recorded, the in-memory namespace is updated
  3. Later, the FsImage is updated via a checkpoint (merging the FsImage and edit log)

This ensures metadata changes are durable even if the system crashes after the edit log is written but before the FsImage is updated.

Checkpointing and Secondary NameNode

Over time, the edit log grows large. To avoid long recovery times, a process called checkpointing is used:

  • The Secondary NameNode (or Checkpoint Node) periodically:
    • Downloads the latest FsImage and edit log
    • Merges them to produce a new FsImage
    • Uploads the updated FsImage back to the NameNode
hdfs dfsadmin -rollEdits
hdfs dfsadmin -saveNamespace

These commands help manually roll the edit logs and force a checkpoint if needed.

JournalNodes in HDFS High Availability (HA)

In HA configurations, HDFS uses a set of JournalNodes to store the write-ahead logs (edit logs) in a shared and replicated fashion.

Workflow:

  • Both Active and Standby NameNodes write to a shared Quorum Journal Manager (QJM)
  • Edit logs are written to majority of JournalNodes before being committed
  • Standby node reads edit logs to stay in sync

This guarantees fault tolerance and seamless failover in case the Active NameNode crashes.

HDFS Edit Log Directory Configuration

The edit log storage paths are defined in hdfs-site.xml:

<property>
<name>dfs.namenode.name.dir</name>
<value>file:///dfs/nn/name</value>
</property>

<property>
  <name>dfs.namenode.edits.dir</name>
  <value>file:///dfs/nn/edits</value>
</property>

Ensure that:

  • Edit logs are stored on reliable disks
  • There are multiple directories for redundancy
  • Disk health is monitored continuously

Recovery Using Edit Logs

If the NameNode crashes:

  1. On startup, it loads the FsImage
  2. Applies all changes from the edit log
  3. Builds the current namespace state
  4. Begins serving client requests

The consistency of HDFS metadata relies heavily on the integrity of the edit logs.

Monitoring and Managing Edit Logs

Best practices include:

  • Enable automatic log rolling: 
    <property>
<name>dfs.namenode.edits.roll.interval</name>
<value>600</value> <!-- in seconds -->
</property>
  • Periodically monitor log sizes and checkpoint intervals
  • Use High Availability (HA) with JournalNodes for production reliability
  • Regularly back up FsImage and edit logs to off-cluster storage
  • Use log rotation and retention policies to prevent disk fill-up

Conclusion

Write-ahead logging via the HDFS edit log is a foundational component of Hadoop’s reliability. It ensures that every change to the file system metadata is durable and recoverable, even in the event of a NameNode failure.

By understanding how edit logs, FsImage, and JournalNodes work together — and by implementing best practices — you can build a resilient, fault-tolerant data lake infrastructure capable of supporting critical enterprise workloads.