With billions of devices continuously generating data, Internet of Things (IoT) systems require robust, real-time data platforms to handle the scale, velocity, and variability of incoming messages. Apache Pulsar, with its native support for high-throughput ingestion, geo-replication, and fault tolerance, is ideally suited for powering real-time IoT data pipelines.

In this blog, we’ll explore how Pulsar enables scalable and fault-tolerant IoT data processing, from edge device ingestion to cloud-based analytics and visualization.

Why Apache Pulsar for IoT?

Apache Pulsar offers features essential for IoT:

  • Multi-tenant architecture for device and service isolation
  • Low-latency, high-throughput ingestion
  • Built-in geo-replication for cross-region failover
  • Durable storage via Apache BookKeeper
  • Topic compaction and message expiry for resource efficiency
  • Pulsar Functions for edge analytics and stream transformations

IoT Pipeline Architecture with Pulsar

[IoT Devices / Edge Gateways]
↓
[Pulsar Producers (MQTT, HTTP, SDKs)]
↓
[Pulsar Brokers & Bookies]
↓
[Stream Processing (Pulsar Functions, Flink, Spark)]
↓
[Data Lake / Monitoring / Real-Time Dashboards]

Each layer supports:

  • Scalability: horizontal partitioning, topic sharding
  • Resilience: persistent storage and message acknowledgments
  • Low latency: millisecond-level publish/subscribe

Ingesting IoT Data into Pulsar

Use protocol gateways (e.g., MQTT, HTTP) or native clients to ingest sensor readings, telemetry, logs, etc.

MQTT Bridge Example:

  • Devices publish to MQTT topics
  • Gateway translates MQTT → Pulsar
MQTT Topic: iot/devices/temperature
→ Pulsar Topic: persistent://iot/telemetry/temperature

Alternatively, build direct producers using the Pulsar Client SDKs (Java, Python, C++, Go).

Partitioning and Scalability

Use partitioned topics to scale per device or region:

pulsar-admin topics create-partitioned-topic \
persistent://iot/sensors/temperature --partitions 50

This allows:

  • Parallel writes from thousands of devices
  • Scalable consumers
  • Load balancing across brokers

Use key-based routing for device-level ordering.

Real-Time Processing with Pulsar Functions

Use Pulsar Functions for in-line processing such as:

  • Unit conversions (e.g., °F to °C)
  • Filtering noise or anomalies
  • Enrichment with metadata (device type, location)
  • Aggregating values per time window

Example Python function:

def process(input, context):
data = json.loads(input)
if data["temperature"] > 80:
return json.dumps({"alert": "High temp", "value": data["temperature"]})
return None

Integration with Downstream Systems

Sink processed data to:

  • Time-series DBs (InfluxDB, Timescale)
  • Monitoring tools (Grafana, Prometheus)
  • Cloud warehouses (BigQuery, Snowflake via Pulsar IO connectors)
  • Object storage (S3, GCS for batch analytics)

Pulsar IO provides ready-to-use connectors for popular destinations.

Ensuring Fault Tolerance

Apache Pulsar ensures resilience via:

  • Persistent storage using BookKeeper journals
  • Message acknowledgments and retries
  • Replication across bookies
  • Geo-replication between data centers or cloud regions
  • Dead Letter Topics (DLQ) for failed messages

Use message deduplication to avoid reprocessing during retries:

producerName=my-device
enableBatching=true
producerAccessMode=Shared
sendTimeoutMs=30000
enableDeduplication=true

Monitoring and Metrics

Track system health and throughput with:

  • Prometheus + Grafana dashboards
  • Broker metrics (publish rate, backlog, latency)
  • Topic-level stats (per partition)
  • Function logs and error tracking

Use pulsar-admin topics stats and pulsar-admin functions stats for CLI monitoring.

Best Practices

✅ Use partitioned topics to parallelize high-volume data
✅ Apply message TTL and compaction for ephemeral telemetry
✅ Route device data using device ID as key
✅ Isolate tenants per use case or customer
✅ Enable replication and persistence for mission-critical data
✅ Validate schema and message format at edge
✅ Use DLQs for non-compliant messages

Conclusion

Apache Pulsar delivers the scalability, flexibility, and reliability needed for real-time IoT data processing. From edge to cloud, Pulsar supports high-velocity ingestion, low-latency streaming, and durable message delivery across globally distributed architectures.

Whether you’re building smart city platforms, industrial telemetry systems, or connected device networks — Pulsar enables your IoT stack to operate in real time, at scale, and with confidence.