Apache Hudi is a powerful data lake framework that enables streaming data ingestion, incremental processing, and ACID-compliant updates. While Hudi works out of the box with sensible defaults, tuning the right configuration parameters can significantly improve performance, reliability, and resource usage for production workloads.

In this blog, we’ll explore advanced configuration tuning strategies for Hudi clusters — including write performance, compaction tuning, memory management, and query optimization. Whether you’re running Hudi with Spark on YARN, Kubernetes, or EMR, these tips can help you squeeze out maximum performance.

Key Configuration Categories in Hudi

  1. Write Path Optimization
  2. Compaction and Cleaning Tuning
  3. Memory and Parallelism
  4. Table and File Format Settings
  5. Query Layer Enhancements (Presto, Hive, Spark)

1. Optimizing Hudi Write Path

Hudi write performance is largely governed by how efficiently it handles inserts, upserts, and deletes.

Key configs:

hoodie.upsert.shuffle.parallelism=200
hoodie.insert.shuffle.parallelism=200
hoodie.bulkinsert.shuffle.parallelism=300
hoodie.datasource.write.operation=upsert
hoodie.datasource.write.precombine.field=ts
hoodie.datasource.write.recordkey.field=id
  • Record key uniquely identifies records.
  • Precombine field selects the most recent record based on timestamp.
  • Shuffle parallelism should match or exceed executor cores × number of nodes.

💡 Use bulk_insert for large initial loads and upsert for streaming.

2. Tuning Compaction and Cleaning

If you use MERGE_ON_READ tables, compaction is key to balancing performance and freshness.

Recommended settings:

hoodie.compact.inline=true
hoodie.compact.inline.max.delta.commits=5
hoodie.compaction.async.enabled=true
hoodie.clean.automatic=true
hoodie.cleaner.policy=KEEP_LATEST_COMMITS
hoodie.keep.min.commits=20
hoodie.keep.max.commits=30
  • Inline compaction minimizes latency but consumes resources.
  • Async compaction improves ingestion throughput.
  • Cleaner config ensures data versioning doesn’t blow up storage.

💡 Compaction frequency depends on read latency requirements.

3. Managing Memory and Parallelism

Hudi jobs run on Apache Spark, so proper memory and executor tuning is crucial:

--executor-memory 8G \
--driver-memory 4G \
--executor-cores 4 \
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer

Hudi-specific memory tuning:

hoodie.memory.merge.fraction=0.6
hoodie.memory.compaction.fraction=0.6
hoodie.copyonwrite.insert.split.size=500000
hoodie.copyonwrite.record.size.estimate=1024
  • Increase merge/compaction fraction if OOMs occur.
  • Tune record size estimate based on payload type (JSON, Avro, Parquet).

💡 Monitor Spark UI for memory spills and adjust accordingly.

4. File and Table Format Optimization

To balance file count, performance, and query speed:

hoodie.parquet.small.file.limit=134217728   # 128MB
hoodie.copyonwrite.small.file.limit=268435456   # 256MB
hoodie.parquet.max.file.size=536870912      # 512MB
hoodie.copyonwrite.record.size.estimate=1024
  • Avoid too many small files (NameNode strain).
  • Use hoodie.metadata.enable=true for faster listing.
  • Prefer Snappy compression for balanced performance:
hoodie.parquet.compression.codec=snappy

💡 Enable file sizing config to control compaction output granularity.

5. Query Performance Tuning

For downstream analytics engines (Spark, Hive, Presto, Trino):

Enable metadata table:

hoodie.metadata.enable=true
hoodie.metadata.index.column.stats.enable=true
hoodie.metadata.index.bloom.filter.enable=true

Hive sync:

hoodie.datasource.hive_sync.enable=true
hoodie.datasource.hive_sync.mode=hms
hoodie.datasource.hive_sync.database=analytics
hoodie.datasource.hive_sync.table=hudi_events

💡 Using Hive Metastore sync allows easy integration with BI tools.

Advanced Features to Explore

  • Clustering: Organizes data files for better range queries
  • Bootstrap: Ingest historical data without rewriting
  • Multitenancy: Use partitioning and ACLs to isolate tenants
hoodie.clustering.inline=true
hoodie.clustering.inline.max.commits=4

Monitoring and Metrics

Monitor job health with:

  • Spark UI for shuffle, GC, memory
  • Hudi metrics via Prometheus, Datadog, or Grafana
  • Table health with hoodie-cli:
hoodie-cli
> connect --path hdfs:///datalake/hudi/events
> show fsview all
> show archived commits

Best Practices

  • Start with COPY_ON_WRITE and migrate to MERGE_ON_READ for streaming
  • Use async compaction for better write throughput
  • Monitor commit timelines for ingestion lag
  • Periodically run repair and file sizing operations
  • Profile memory and shuffle stages in Spark UI regularly

Conclusion

Advanced configuration tuning is key to scaling Apache Hudi for production. By optimizing writes, compactions, memory, and metadata access, you ensure low-latency ingestion, efficient storage, and fast analytical queries.

Use these configuration patterns as a blueprint to build high-performance Hudi pipelines that power modern data lakes and lakehouses at scale.