Kafka Streaming

Build event-streaming systems with Apache Kafka. Writes producers and consumers, designs topics and partitioning, handles consumer groups and offsets, and implements idempotent, exactly-once processing.

This skill builds reliable Kafka streaming pipelines. It writes producers and consumers in Python and other clients, designs topics, partitioning, and replication, manages consumer groups and offset handling, and implements patterns like dead-letter queues, idempotent consumers, and exactly-once delivery for event-driven architectures.

When to use

Use when building Kafka producers and consumers, designing topics and partitioning, handling consumer groups and offsets, or implementing reliable event-streaming patterns.

Examples

Write a producer

Publish events to a topic

Write a Python confluent-kafka producer that publishes order events to an orders topic with keyed partitioning and delivery callbacks

Reliable consumer

Process with offset control

Build a Kafka consumer with a consumer group that commits offsets after successful processing and routes failures to a dead-letter topic

Topic design

Plan partitioning

Help me choose partition count and key strategy for a high-throughput clickstream topic that needs ordering per user