Apache Pulsar is designed to support high-throughput, low-latency messaging at scale — making it ideal for applications like IoT telemetry, financial transaction processing, and real-time analytics.

To take full advantage of Pulsar’s capabilities in high-volume scenarios, it’s essential to optimize the consumer side. This involves tuning the client configuration, choosing the right subscription modes, leveraging batching and async processing, and scaling horizontally.

This post explores best practices and configuration options to maximize consumer performance in Pulsar environments that demand millions of messages per second.

Understand Pulsar Consumer Internals

A Pulsar consumer:

  • Connects to a topic partition via a broker
  • Pulls messages from the broker’s managed ledger (backed by BookKeeper)
  • Acknowledges messages (either individually or cumulatively)
  • Supports multiple subscription types (Exclusive, Shared, Failover, Key_Shared)

Each of these aspects can be optimized for throughput, latency, or resource utilization.

Key Tuning Strategies

1. Choose the Right Subscription Type
  • Exclusive: Single consumer per topic/partition
  • Shared: Multiple consumers per topic (messages distributed in round-robin)
  • Failover: One active consumer, others standby
  • Key_Shared: Parallel processing with ordering per key

For high-throughput needs:

  • Use Shared for load-balanced parallel consumption
  • Use Key_Shared if key-based ordering is important
2. Enable Batching for Higher Throughput

Batching reduces network overhead by grouping messages:

Consumer<byte[]> consumer = client.newConsumer()
.topic("high-volume-topic")
.subscriptionName("analytics-sub")
.subscriptionType(SubscriptionType.Shared)
.batchReceivePolicy(BatchReceivePolicy.builder()
.maxNumMessages(100)
.maxNumBytes(1024 * 1024)
.timeout(100, TimeUnit.MILLISECONDS)
.build())
.subscribe();
  • Tune maxNumMessages, maxNumBytes, and timeout based on your processing latency budget
  • Enable producer-side batching too for complementary gains
3. Use Asynchronous Message Processing

Async consumers allow concurrent processing without blocking the receive loop:

consumer.receiveAsync().thenAccept(msg -> {
process(msg.getValue());
consumer.acknowledgeAsync(msg);
});
  • Reduces thread contention
  • Improves CPU utilization under load
4. Adjust Receiver Queue Size

The receiver queue controls how many messages are buffered on the client:

receiverQueueSize=1000
  • A larger queue improves throughput by reducing back-and-forth with the broker
  • For latency-sensitive apps, smaller values may be better
5. Tune Acknowledgment Mode
  • Individual Acknowledgment for per-message control
  • Cumulative Acknowledgment for sequential processing with fewer acks

Use cumulative acks when processing in order:

consumer.acknowledgeCumulative(msg);

It reduces broker metadata and improves throughput.

6. Use Multiple Consumers per Partition
  • For high-volume topics, create multiple consumer instances within a Shared subscription
  • Each consumer processes messages from one or more partitions
  • Scale horizontally with consumer group replicas
7. Leverage Key_Shared for Ordered Parallelism

When processing requires key-based affinity, use Key_Shared:

subscriptionType = SubscriptionType.Key_Shared

This enables multiple consumers to process in parallel while ensuring all messages for a key go to the same consumer.

8. Monitor and Tune Network and Disk

Pulsar performance also depends on:

  • Broker → Consumer network throughput
  • Disk I/O on BookKeeper and broker machines
  • Garbage collection and heap tuning in the JVM

Use Prometheus and Grafana to track:

  • pulsar_consumer_receive_rate
  • pulsar_consumer_ack_rate
  • consumer_backlog
  • broker_dispatch_rate

Example: High-Throughput Consumer Configuration

Consumer<byte[]> consumer = client.newConsumer()
.topic("sensor-data")
.subscriptionName("iot-processing")
.subscriptionType(SubscriptionType.Shared)
.receiverQueueSize(5000)
.ackTimeout(60, TimeUnit.SECONDS)
.batchReceivePolicy(BatchReceivePolicy.builder()
.maxNumMessages(200)
.timeout(50, TimeUnit.MILLISECONDS)
.build())
.subscribe();

This setup handles high ingestion rates with efficient batching and queue management.

Best Practices

  • Start with Shared subscription for max parallelism
  • Use batching and asynchronous receive
  • Scale horizontally with multiple consumer instances
  • Tune receiver queue size and ack strategies
  • Monitor with broker and client-side metrics
  • Profile processing time per message to avoid slow consumers

Conclusion

Apache Pulsar is built to handle high message volumes, but achieving optimal consumer performance requires thoughtful tuning and architecture design. By leveraging batching, async APIs, parallelism, and proper subscription strategies, you can build consumers that scale to millions of messages per second — while maintaining low latency and high reliability.

Optimize now to prepare your applications for the next wave of streaming data at scale.