Stateless Filter + Enrich
Per-record filter keeping European regions (eu-*), then enrich each kept row with a derived latency_ratio and an ok/breach SLA decision. No state — measures the raw I/O + serde + per-record compute ceiling (fed above each consumer's capacity so CPU saturates).
Last updated: 2026-09-10 · same-machine, relative comparison.
Bench characteristics
In-process computation — Per record: filter region LIKE 'eu-%', then derive latency_ratio = latency_ms / 80 and an ok/breach SLA flag. Stateless.
| product | delivery | processing | engine | scaling on one partition | decode |
|---|---|---|---|---|---|
| turbine | at-least-once (idempotent, acks=all) | batch (Arrow columnar) | in-process · Rust core + free-threaded CPython 3.14t | intra-partition shards (parallelism=N, hash of partition_key, in-process) |
schema projection (6 of 19 fields, Arrow) |
| faststream | at-least-once (idempotent, acks=all; fire-and-forget publish) | batch I/O, per-row compute | in-process · pure-Python async (aiokafka; orjson JSON codec) | none (one async consumer) | full record (19 of 19 fields, orjson dict) |
| flink | at-least-once (checkpointed) | batch (pipelined, JVM code-gen SQL) | in-process · JVM (PyFlink local MiniCluster) | keyed exchange to N slots (parallelism.default, in-process local channels; source stays 1 subtask) |
schema projection (4 of 19 fields, Jackson); connector-default fetch |
| quix | at-least-once (idempotent, acks=all) | row (per-record StreamingDataFrame) | in-process · pure-Python over librdkafka (C client) | none (one consumer per partition, GIL) | full record (19 of 19 fields, orjson dict) |
| risingwave | at-least-once | batch (vectorized SQL) | external server · Rust (single-node: meta+compute+frontend) | hash exchange to N stream actors (SET streaming_parallelism, in-process) |
schema projection (4 of 19 fields) |
Input is a single Kafka partition (lz4). Each product may spread it over several cores through its own in-process mechanism (column scaling on one partition; what it actually used is cpu_avg); an in-process exchange is not a broker hop — the extra rec/rec column counts records that DID cross the broker on other topics. All products run at-least-once.
Consumer fetch: Turbine's core hard-codes librdkafka fetch.min.bytes=64 KB, fetch.wait.max.ms=10, max.partition.fetch.bytes=50 MB and a 500k-msg / ~2 GB prefetch queue (not adjustable), so the SAME client gets the SAME settings: quix and RisingWave (librdkafka) run with them, and so does FastStream (aiokafka, whose equivalent knobs lifted it ~14%). Flink runs at its Java connector's defaults: an A/B (2026-09-10) showed the identical numbers throttled the Java consumer (373k/s vs 543k/s at the same ~285k rows/s per core) — identical numbers are not identical behaviour on a different client. Memory at saturation is mostly this prefetch queue, for every librdkafka product.
Throughput / efficiency (live)
| product | offered/s | consumed/s | regime | lag drift | cpu_avg % | rows/s per core | rows/s per GB | µs CPU / record | output/s | extra rec/rec | run |
|---|---|---|---|---|---|---|---|---|---|---|---|
| turbine 0.2.0a1 | 730,429 | 456,698 | saturated | +61.0% | 85.3 | 535,290 | 379,356 | 1.87 | 147,209 | 0 | 2026-09-10 |
| faststream 0.7.1 | 108,839 | 104,575 | keep-up | +4.1% | 100.0 | 104,625 | 477,108 | 9.56 | 33,803 | 0 | 2026-09-10 |
| flink 2.2.1 | 680,137 | 543,191 | saturated | +25.4% | 191.2 | 284,108 | 506,027 | 3.52 | 178,942 | 0 | 2026-09-10 |
| quix 3.23.6 | 247,236 | 176,764 | saturated | +40.5% | 105.1 | 168,218 | 261,908 | 5.94 | 61,949 | 0 | 2026-09-10 |
| risingwave v2.8.4 | 296,898 | 296,898 | keep-up | +0.0% | 74.6 | 398,118 | 589,344 | 2.51 | 99,907 | 0 | 2026-09-10 |
consumed/s is what the product actually drained (its consumer-group offset or engine counter), offered/s what gendata produced. regime: keep-up = consumed ≈ offered with a flat backlog (a sustainable point); saturated = fed beyond capacity, so consumed/s IS the ceiling and latency is unbounded; draining = still eating warmup backlog (consumed > offered; CPU is catch-up, not steady state — lengthen the warmup, or raise the rate if the ceiling is what you want). lag drift = backlog change over the window ÷ records consumed. cpu_avg is the whole process tree, 100 = one core busy. rows/s per core = consumed ÷ (cpu_avg/100); rows/s per GB = consumed ÷ memory; extra rec/rec = records written to topics other than the input and output per record consumed (0 = no broker hop).
Resources (raw)
| product | cpu_avg % | cpu_p95 % | mem_avg MB | mem_max MB | samples | instrument |
|---|---|---|---|---|---|---|
| turbine 0.2.0a1 | 85.3 | 113.8 | 1,232.8 | 1,398.7 | 59 | /proc Δ, pss |
| faststream 0.7.1 | 100.0 | 100.7 | 224.4 | 229.6 | 59 | /proc Δ, pss |
| flink 2.2.1 | 191.2 | 204.1 | 1,099.2 | 1,109.8 | 58 | /proc Δ, pss |
| quix 3.23.6 | 105.1 | 109.0 | 691.1 | 694.1 | 59 | /proc Δ, pss |
| risingwave v2.8.4 | 74.6 | 76.8 | 515.9 | 602.4 | 60 | /proc Δ, rss |
CPU is summed over the whole process tree (JVM, UDF servers, the RisingWave container's processes) as utime+stime deltas per 1 s sample — 100 = one core. Memory is PSS (shared pages counted once) where readable, RSS otherwise (instrument). Rows marked ⚠ legacy ps were measured with the old ps %CPU lifetime average and are not comparable: re-run them.
faststream — Pure-Python async (aiokafka): CPU-bound at ~1 core (~99%), draining ~95k/s on the single partition — about 0.5× quix's per-core throughput (quix's librdkafka is a C extension). Fed 110k, just above that knee, so it saturates (the intended ceiling) while the backlog grows only modestly. Leanest footprint of the four (~134 MB RSS). JSON is decoded with orjson — FastStream auto-detects it once it is installed — which MATCHES quix, whose default JSON deserializer is also orjson; on stdlib json FastStream's knee was ~78k, so orjson both LEVELS the comparison (same C-speed JSON as quix) and lifts the ceiling ~22%. Benched with FastStream's batch=True subscriber (getmany) + fire-and-forget publish (no_confirm=True, so aiokafka batches the sends): the default per-record path awaits the broker ack on every message and is round-trip-bound (~8k/s here, CPU idle), not how FastStream is run at throughput. FastStream ships ONLY this stateless workload — it has no built-in windowing, so a windowed bench would measure hand-rolled timer/state code, not FastStream.
flink — PyFlink running a pure Table API / SQL job on a LOCAL MiniCluster, pinned to parallelism=1 — one task slot reading the single Kafka partition, the bench's KPI. Single-partition knee ≈ 550k/s (consume ≈ offered, lag ≈ 0); the consume ceiling is ~570k, so it is fed 700k to saturate and read that ceiling — far above quix (~180k) and RisingWave (~170k), a touch under turbine (~500k) per CORE since Flink needs ~2 cores for it. Measured in-process: CPU/RSS are summed across the process tree INCLUDING the JVM child, so the figure carries JVM overhead (GC, checkpoint, network/produce threads) and reads ~2 cores (~200% CPU) here even at parallelism=1 — the "one slot" is the WORK parallelism, but the Kafka-produce side (it writes ~31% of records downstream) spreads onto other cores, which is also why its I/O ceiling sits below the windowed-lowcard knee (650k, ~nothing written downstream). At-least-once checkpointing commits the consumed offset to the group each checkpoint, so consumed_in/s is a real reading like quix. Benched version from flink/uv.lock; the Kafka SQL connector JAR is fetched from Maven on first run.
risingwave — Single-partition I/O-bound: RisingWave runs one source reader per Kafka partition, so on the bench's single partition it drains only ~170k/s — well under the offered rate, so the backlog grows — while its CPU stays well below saturation (~0.5 core). It cannot trade its idle cores for more throughput; scaling needs more Kafka partitions (≈3× at 4), not more cores. consumed_in/s is read from RisingWave's own source_partition_input_count metric (it commits no Kafka group offset); CPU/RSS are the whole single-node container.
Sanity — are we benching the same thing?
| product | selectivity (out/consumed) | expected | ✓ | breach_rate | enrichment | sample |
|---|---|---|---|---|---|---|
| turbine 0.2.0a1 | 0.3223 | 0.3100 | ✓ | 0.210 | ✓ | 500 |
| faststream 0.7.1 | 0.3232 | 0.3315 | ✓ | 0.232 | ✓ | 500 |
| flink 2.2.1 | 0.3294 | 0.3240 | ✓ | 0.184 | ✓ | 500 |
| quix 3.23.6 | 0.3505 | 0.3470 | ✓ | 0.184 | ✓ | 500 |
| risingwave v2.8.4 | 0.3365 | 0.3465 | ✓ | 0.216 | ✓ | 500 |
Verdict: ✓ matches theory — every row's selectivity is within 25% of the expected value (output ÷ consumed).
Host: AMD Ryzen 5 7600X 6-Core Processor · 6 cores / 12 threads · 31241 MB · kernel 6.12.107+deb13-amd64.
Redpanda: rpk version: v26.1.6. Redpanda, gendata and the product under test share this host.