Bytewax Python Streaming

Build Python-native stream processing pipelines with Bytewax. Generates dataflows, stateful operators, windowing logic, Kafka/Redis connectors, and Kubernetes deployments — no JVM required.

Bytewax is a stream processing framework built on the Timely Dataflow runtime, with a pure-Python API. This skill writes Bytewax dataflows using map, filter, reduce, and stateful operators, configures input/output connectors for Kafka, Redis, S3, and HTTP, builds tumbling/sliding/session windows, sets up recovery state for fault tolerance, and deploys to Kubernetes with the bytewax-platform operator.

When to use

Use when teams want stream processing without learning Flink/Spark Streaming, building ML feature pipelines, or processing Kafka events with native Python libraries (pandas, numpy, sklearn).

Examples

Real-time anomaly detection

Sliding window stats on Kafka events

Build a Bytewax dataflow that consumes metrics from Kafka, maintains a 5-minute sliding window of mean/stddev per host, and emits to a Slack alert sink when z-score > 3

Online ML feature pipeline

Streaming feature engineering

Write a Bytewax pipeline that consumes click events, joins with user state from Redis, computes rolling 1h/24h features, and writes them to a feature store