In large-scale data lakes, how you organize your data significantly affects performance, cost, and scalability. Apache Hudi offers flexible partitioning strategies that allow you to structure data for efficient ingestion, querying, and management.

This post dives deep into Hudi’s partitioning strategies, how they work, when to use them, and best practices to achieve optimal performance for real-time analytics and batch ETL workloads.

What is Partitioning in Hudi?

Partitioning in Hudi divides a dataset into logical groups based on one or more column values (e.g., event_date, region). Each partition is physically stored as a separate directory in the underlying filesystem (S3, HDFS, etc.).

Partitioning improves:

  • Query performance via pruning
  • Write efficiency by reducing the number of updated files
  • Scalability by avoiding large directory listings

Types of Partitioning in Hudi

Apache Hudi supports two main types of partitioning:

  1. Static Partitioning
    Partition path is explicitly specified during the write.

  2. Dynamic Partitioning
    Partition values are derived from the input dataset during ingestion.

Configuring Partitioning in Hudi

Specify partitioning via:

hoodie.datasource.write.partitionpath.field=event_date
hoodie.datasource.write.hive_style_partitioning=true

Example partitioned write:

df.write.format("hudi") \
.option("hoodie.datasource.write.partitionpath.field", "event_date") \
.option("hoodie.datasource.write.precombine.field", "event_ts") \
.option("hoodie.datasource.write.operation", "upsert") \
.option("hoodie.table.name", "orders_table") \
.mode("append") \
.save("s3://datalake/orders_table")

This will create directories like:

/orders_table/event_date=2024-04-01/
event_date=2024-04-02/

Multi-Level Partitioning

You can partition by multiple fields:

hoodie.datasource.write.partitionpath.field=region,event_date

Which creates:

/region=US/event_date=2024-04-01/
/region=IN/event_date=2024-04-02/

This helps distribute data more evenly and avoid high file counts in a single directory.

Choosing the Right Partition Column

A good partition column:

  • Has low to moderate cardinality
  • Is frequently used in query filters
  • Does not change often for the same record key

✅ Good:

  • event_date
  • country
  • customer_segment

❌ Bad:

  • user_id
  • transaction_id (too many partitions)

Partition Pruning for Query Performance

When querying Hudi via Spark, Hive, or Athena, partition pruning ensures only relevant partitions are scanned.

Spark example:

SELECT * FROM orders_table
WHERE event_date = '2024-04-01';

To enable partition pruning:

  • Use Hive-style partitioning
  • Avoid using functions on partition columns in WHERE clauses
  • Ensure the partition column is part of the DataFrame filter

Clustering to Optimize Partition Files

Over time, partitions can become skewed or filled with small files. Hudi supports clustering to reorganize data within partitions.

Enable inline clustering:

hoodie.clustering.inline=true
hoodie.clustering.inline.max.commits=4
hoodie.clustering.plan.strategy.sort.columns=event_ts

Clustering rewrites partition files into optimized sizes, improving:

  • Read performance
  • Metadata management
  • Query scan time

Partitioning and Compaction in MOR Tables

For Merge-on-Read (MOR) tables:

  • Each partition may have log files + base files
  • Compaction rewrites partitions into optimized Parquet files

Tune compaction frequency to avoid latency:

hoodie.compact.inline=true
hoodie.compact.inline.max.delta.commits=3

Use partition filters when triggering compaction to limit scope.

Best Practices

  • Limit partitions per table to avoid metadata bloat
  • Use date-based partitioning for time-series data
  • Monitor partition sizes — aim for 100MB to 1GB per file
  • Use clustering and compaction to keep partitions clean
  • Enable Hudi’s metadata table for fast partition listings
  • Avoid deeply nested partition paths unless necessary

Conclusion

Effective partitioning is one of the most important performance levers in Hudi. By choosing the right partition fields, enabling Hive-style partitioning, and leveraging clustering, you can dramatically improve the scalability and manageability of your lakehouse architecture.

As your data volume grows, well-designed partitioning will ensure Hudi remains fast, flexible, and cost-effective — even at petabyte scale.