Apache Kafka Streaming

Build event-driven architectures with Apache Kafka. Generates producers, consumers, stream processors, schema definitions with Avro/Protobuf, and Kafka Connect configurations for real-time data pipelines.

This skill covers the Kafka ecosystem for event-driven architectures. It generates KafkaJS producers and consumers in TypeScript, Kafka Streams topology definitions, Schema Registry with Avro/Protobuf, Connect source and sink connectors, topic configuration with partitioning strategies, consumer group management, and exactly-once semantics. Also covers Confluent Cloud, Redpanda, and WarpStream alternatives.

When to use

Use when building event-driven microservices, creating real-time data pipelines, implementing CQRS/event sourcing patterns, or setting up Kafka Connect integrations.

Examples

Event sourcing

Implement event sourcing with Kafka and TypeScript

Build an event sourcing system with KafkaJS: define domain events with Avro schemas, create a producer for order events, and a consumer that builds read models in PostgreSQL

Connect pipeline

Set up CDC from PostgreSQL to Elasticsearch

Configure Kafka Connect with Debezium PostgreSQL source connector and Elasticsearch sink connector for real-time search index updates