Windowed Low Cardinality
Tumbling 30s event-time window grouped by region (16 fixed keys), with the per-region SLO rollup an operations dashboard computes — latency distribution, SLA breaches, errors, traffic volume, host saturation. 12 aggregates over 9 of the record's columns: the tiny key set keeps state trivial, so decoding and per-record compute, not the state store, bound the capacity. Capacity is read in drain mode, on a backlog carrying synthetic event time (every product meets the same windows); the live runs, at each product's max sustainable single-partition rate (its knee), give the close latency.
Last updated: 2026-11-10 · same-machine, relative comparison.
flowchart LR
src[("bench-input<br/>1 partition")]
s0{{"<b>key by</b><br/>16 keys (region)"}}
s1[["<b>window</b><br/>tumbling 30 s, event time<br/>12 aggregates over 9 columns"]]
sink[("bench-out")]
state[("keyed state<br/>one open window per key")]
probe>"latency probe<br/>arrival − window_end_ms"]
src --> s0
s0 --> s1
s1 -->|"at-least-once"| sink
s1 -.- state
sink -.-> probeBench characteristics
In-process computation — Tumbling 30s event-time window on the Kafka message timestamp, 1 s grace, GROUP BY region (16 fixed keys) → count, avg / min / max / stddev_pop(latency_ms), count(latency_ms > 80) (the stateless SLA), sum(error_count), sum(request_size + response_size), avg(cpu_usage), max(memory_mb), max(queue_depth), sum(throughput); one pane per key per window close.
| product | delivery | processing | engine | scaling on one partition | decode | state |
|---|---|---|---|---|---|---|
| turbine | at-least-once (idempotent, acks=all) | batch (Arrow columnar; 100k rows / 500 ms) | in-process · Rust core + free-threaded CPython 3.14t | intra-partition shards (parallelism=N, hash of partition_key, in-process) |
schema projection (13 of 19 fields, Arrow) | own RocksDB store (merge operator for windows) |
| turbine-resident | at-least-once (idempotent, acks=all) | batch (Arrow columnar; 100k rows / 500 ms) | in-process · Rust core + free-threaded CPython 3.14t | none — one unkeyed shard per partition (resident storage refuses parallelism) |
schema projection (13 of 19 fields, Arrow) | resident typed columns in memory, checkpointed whole (storage="resident") |
| arroyo | at-least-once (checkpointed) | batch (vectorized SQL, Arrow / DataFusion; source batches of 100k rows / 500 ms) | in-process · Rust (arroyo run: controller + one worker process) |
hash shuffle to N subtasks (arroyo run --parallelism, in-process queues) |
schema projection (9 of 19 fields) | in-memory tables, checkpointed to the state dir |
| flink | at-least-once (checkpointed) | batch (pipelined, JVM code-gen SQL) | in-process · JVM (local MiniCluster; Python only submits the job) | keyed exchange to N slots (parallelism.default, in-process local channels; source stays 1 subtask) |
schema projection (10 of 19 fields, Jackson); connector-default fetch | RocksDB (embedded, on-disk LSM) |
| quix | at-least-once (idempotent, acks=all) | row (per-record StreamingDataFrame) | in-process · pure-Python over librdkafka (C client) | none (one consumer per partition, GIL) | full record (19 of 19 fields, orjson dict) | RocksDB (embedded); changelog topic OFF for the bench |
| risingwave | at-least-once | batch (vectorized SQL) | external server · Rust (single-node: meta+compute+frontend) | hash exchange to N stream actors (SET streaming_parallelism, in-process) |
schema projection (9 of 19 fields) | server-managed (Hummock LSM store) |
Input is a single Kafka partition (lz4). Each product may spread it over several cores through its own in-process mechanism (column scaling on one partition; what it actually used is cpu_avg); an in-process exchange is not a broker hop — the extra rec/rec column counts records that DID cross the broker on other topics. All products run at-least-once, windows are tumbling 30 s event-time on the Kafka message timestamp.
Consumer fetch: Turbine's core hard-codes librdkafka fetch.min.bytes=64 KB, fetch.wait.max.ms=10, max.partition.fetch.bytes=50 MB and a 500k-msg / ~2 GB prefetch queue (not adjustable), so the SAME client gets the SAME settings: quix and RisingWave (librdkafka) run with them, and so does FastStream (aiokafka, whose equivalent knobs lifted it ~14%). Flink runs at its Java connector's defaults: an A/B (2026-09-10) showed the identical numbers throttled the Java consumer (373k/s vs 543k/s at the same ~285k rows/s per core) — identical numbers are not identical behaviour on a different client. Memory at saturation is mostly this prefetch queue, for every librdkafka product.
Drain ceiling (--prefill: static backlog, no producer running)
| product | backlog | event time/s | warmup + window | drained/s | cpu_avg % | rows/s per core | rows/s per GB | µs CPU / record | panes/s | run |
|---|---|---|---|---|---|---|---|---|---|---|
| turbine 0.3.5a1 | 200M | 430k | 40 + 120 s | 929,207 | 253.5 | 366,490 | 452,991 | 2.73 | 1.19 | 2026-11-10 |
| turbine-resident 0.3.5a1 | 200M | 430k | 40 + 120 s | 899,912 | 252.2 | 356,887 | 534,303 | 2.80 | 1.19 | 2026-11-10 |
| arroyo 0.15.0 | 200M | 430k | 40 + 240 s | 564,290 | 127.4 | 442,755 | 487,627 | 2.26 | 0.66 | 2026-10-27 |
| flink 2.2.1 | 200M | 430k | 55 + 240 s | 507,149 | 162.2 | 312,654 | 356,996 | 3.20 | 0.66 | 2026-10-27 |
| quix 3.23.6 | 160M | 430k | 268 + 516 s | 50,817 | 101.4 | 50,095 | 69,157 | 19.96 | 0.06 | 2026-10-20 |
| risingwave v3.1.0 | 160M | 430k | 47 + 120 s | 369,169 | 149.4 | 247,175 | 408,002 | 4.05 | 0.40 | 2026-10-20 |
How to read this table
gendata wrote the backlog and exited BEFORE the consumer started, so the window measures a drain with no producer competing for the box — the product's consume ceiling free of gendata's CPU and 10 Hz burstiness. Not comparable with a live table (different regime); ⚠ exhausted = the backlog ran out mid-window, rate understated.
The backlog's records carry synthetic event time, event time/s records per second of it (gendata --event-time-rate): it stands for a live stream at that rate, so every product meets the SAME windows — the same records and keys per window, which live runs at each product's own knee do not — and drained/s is its capacity on that problem, closes included. Event time runs at drained ÷ event-time rate × the wall clock, panes/s with it. A product draining far below the event-time rate gets a longer warmup and window (≥ its first close, then two more, at its live knee's pace: warmup + window). No latency here: a pane's age is the backlog's. 0 ⚠ panes/s: no pane was published during the window — the drain did not run the whole pipeline, its drained/s is not comparable.
Throughput / efficiency (live, at each product's knee) — capped by the machine: gendata produces ~430k/s at most, so read capacity in the drain table
| product | offered/s | consumed/s | regime (lag drift) | cpu_avg % | rows/s per core | rows/s per GB | µs CPU / record | panes/s | extra rec/rec | run |
|---|---|---|---|---|---|---|---|---|---|---|
| turbine 0.3.5a1 | 425,085 | 420,495 | keep-up (+0.9%) | 120.9 | 347,792 | 357,434 | 2.88 | 0.53 | 0 | 2026-11-10 |
| turbine-resident 0.3.5a1 | 425,239 | 421,524 | keep-up (+0.9%) | 122.1 | 345,259 | 372,403 | 2.90 | 0.53 | 0 | 2026-11-10 |
| arroyo 0.15.0 | 425,321 | 423,335 | keep-up (+0.5%) | 101.3 | 418,074 | 592,394 | 2.39 | 0.53 | 0 | 2026-10-27 |
| flink 2.2.1 | 424,282 | 422,877 | keep-up (+0.4%) | 141.3 | 299,348 | 280,406 | 3.34 | 0.53 | 0 | 2026-10-27 |
| quix 3.23.6 | 49,456 | 49,269 | keep-up (+0.4%) | 100.1 | 49,223 | 482,595 | 20.32 | 0.53 | 0 | 2026-10-20 |
| risingwave v3.1.0 | 346,174 | 345,964 | keep-up (+0.1%) | 154.5 | 223,856 | 628,292 | 4.47 | 0.53 | 0 | 2026-10-20 |
How to read this table
consumed/s is what the product actually drained (its consumer-group offset or engine counter), offered/s what gendata produced; a dotted underline marks the median of a repeated batch — hover it for the run count and the min – max. regime (lag drift): keep-up = consumed ≈ offered with a flat backlog (a sustainable point); saturated = fed beyond capacity, so consumed/s IS the ceiling and latency is unbounded; draining = still eating warmup backlog (consumed > offered; CPU is catch-up, not steady state — lengthen the warmup, or raise the rate if the ceiling is what you want). The lag drift in parentheses = backlog change over the window ÷ records consumed — the evidence for the regime (≈ 0 at keep-up, positive when saturated, negative while draining). cpu_avg is the whole process tree, 100 = one core busy. rows/s per core = consumed ÷ (cpu_avg/100); rows/s per GB = consumed ÷ memory; extra rec/rec = records written to topics other than the input and output per record consumed (0 = no broker hop).
panes/s is the window-emission cadence (≈ keys ÷ window size), a liveness signal, not a throughput figure — it only moves when windows close, which needs the consumer to keep up.
Latency (close latency — pane published minus window end)
| product | regime | p50 ms | p90 ms | p99 ms | p99.9 ms | max ms | records | setup floor p50 / p99 ms | added p50 / p99 ms |
|---|---|---|---|---|---|---|---|---|---|
| turbine 0.3.5a1 | keep-up | 1,335.6 | 1,352.5 | 1,352.5 | 1,352.5 | 1,352.5 | 192 | 106.2 / 204.5 | 1,229.4 / 1,148.0 |
| turbine-resident 0.3.5a1 | keep-up | 1,322.5 | 1,366.5 | 1,366.5 | 1,366.5 | 1,366.5 | 192 | 106.0 / 204.4 | 1,216.5 / 1,162.1 |
| arroyo 0.15.0 | keep-up | 1,971.1 | 2,555.5 | 2,555.5 | 2,555.5 | 2,555.5 | 384 | 106.4 / 204.5 | 1,864.7 / 2,351.0 |
| flink 2.2.1 | keep-up | 1,311.5 | 1,423.1 | 1,423.1 | 1,423.1 | 1,423.1 | 384 | 106.5 / 204.8 | 1,205.0 / 1,218.4 |
| quix 3.23.6 | keep-up | 1,112.2 | 1,124.0 | 1,124.0 | 1,124.0 | 1,124.0 | 192 | 105.7 / 204.5 | 1,006.5 / 919.5 |
| risingwave v3.1.0 | keep-up | 5,156.8 | 5,189.0 | 5,189.0 | 5,189.0 | 5,189.0 | 192 | 106.5 / 204.6 | 5,050.3 / 4,984.4 |
How to read this table
Measured by a probe consumer tailing the output topic (C client, 10 ms max fetch wait — the probe adds ≲10 ms, the same for every product). Only shown for keep-up runs: under saturation the figure is the backlog's age and grows without bound. setup floor is a harness producer writing one small record every 10 ms (acks=all, 5 ms linger) to its own topic during the same run, measured by the same probe: the broker's produce→visible latency at the products' output shape (on Redpanda, small produces surface on its flush cadence), which every product inherits. added = product − floor, the part the product is responsible for (fetch, batching, commit/epoch cadence, produce). The floor is a low-volume topic's (100 small records/s, surfacing on Redpanda's 100 ms flush.ms): an output topic carrying several MB/s — every record scored and republished — surfaces sooner: a percentile under the floor's reads below, added does not apply. Colours: each percentile against the best (lowest) of its column — green under 2×, yellow from 2×, orange from 5×, red from 10× (the ratio on hover).
Resources (raw)
| product | cpu_avg % | cpu_p95 % | mem_avg MB | mem_max MB | samples | instrument |
|---|---|---|---|---|---|---|
| turbine 0.3.5a1 | 253.5 | 264.0 | 2,100.5 | 2,408.4 | 117 | /proc Δ, pss |
| turbine 0.3.5a1 | 120.9 | 133.2 | 1,204.7 | 1,252.6 | 118 | /proc Δ, pss |
| turbine-resident 0.3.5a1 | 252.2 | 263.3 | 1,724.7 | 2,024.9 | 117 | /proc Δ, pss |
| turbine-resident 0.3.5a1 | 122.1 | 137.0 | 1,159.1 | 1,233.0 | 118 | /proc Δ, pss |
| arroyo 0.15.0 | 127.4 | 144.8 | 1,185.0 | 1,332.9 | 235 | /proc Δ, pss |
| arroyo 0.15.0 | 101.3 | 109.7 | 731.8 | 786.5 | 236 | /proc Δ, pss |
| flink 2.2.1 | 162.2 | 174.6 | 1,454.7 | 1,520.6 | 235 | /proc Δ, pss |
| flink 2.2.1 | 141.3 | 154.0 | 1,544.3 | 1,549.6 | 234 | /proc Δ, pss |
| quix 3.23.6 | 100.1 | 101.4 | 104.5 | 105.1 | 118 | /proc Δ, pss |
| quix 3.23.6 | 101.4 | 108.0 | 752.4 | 799.1 | 507 | /proc Δ, pss |
| risingwave v3.1.0 | 149.4 | 159.9 | 926.5 | 984.5 | 118 | /proc Δ, rss |
| risingwave v3.1.0 | 154.5 | 160.5 | 563.9 | 592.1 | 118 | /proc Δ, rss |
How to read this table
CPU is summed over the whole process tree (JVM, UDF servers, the RisingWave container's processes) as utime+stime deltas per 1 s sample — 100 = one core. Memory is PSS (shared pages counted once) where readable, RSS otherwise (instrument). Rows marked ⚠ legacy ps were measured with the old ps %CPU lifetime average and are not comparable: re-run them.
Product notes
turbine — The rollup is folded in Rust, like the highcard window: since 0.3.1a1 StdDev has a native form (turbine-fast/Turbine.Fast#131), and a Multi is folded natively only when every aggregate is. Up to 0.2.2a2 its 12 aggregates went through pyarrow kernels called from Python on every batch — the older releases' rows measure that.
arroyo — arroyo run executes the SQL pipeline locally (the release binary; Python only writes the SQL): its controller and one worker process, both in the sampled process tree, the worker running parallelism = the run's cores (3). Event time is the Kafka message timestamp, its watermark 1 s behind it (Arroyo's default, the bench's grace); the GROUP BY hash-shuffles to the subtasks over in-process queues. Window state lives in memory, checkpointed every 10 s to the local state dir.
flink — Local MiniCluster (PyFlink only submits the job), parallelism = the run's cores (the source one subtask on the single partition, a keyed exchange to the slots; 3 since 2026-10-20, 1 before). RocksDB keyed-state backend (EmbeddedRocksDBStateBackend, on-disk LSM) — apples-to-apples with quix/Turbine/RisingWave, not Flink's default heap (RAM) — but with only 16 region keys the state never spills past the memtable, so it performs like heap. At-least-once checkpointing to the local filesystem. CPU/RSS are summed across the process tree INCLUDING the JVM child (GC / checkpoint / network threads). expected.output_rate is the fixed 16-region pane cadence (0.53). Its knee is above gendata's live cap: its row is read at the cap.
Sanity — ✓ matches theory
Are we benching the same thing?
| product | panes/s | expected | sample |
|---|---|---|---|
| turbine 0.3.5a1 · drain | 1.19 | 1.14 | 192 |
| turbine 0.3.5a1 | 0.53 | 0.53 | 80 |
| turbine-resident 0.3.5a1 · drain | 1.19 | 1.08 | 176 |
| turbine-resident 0.3.5a1 | 0.53 | 0.53 | 80 |
| arroyo 0.15.0 · drain | 0.66 | 0.73 | 192 |
| arroyo 0.15.0 | 0.53 | 0.53 | 144 |
| flink 2.2.1 · drain | 0.66 | 0.68 | 192 |
| flink 2.2.1 | 0.53 | 0.53 | 160 |
| quix 3.23.6 | 0.53 | 0.53 | 80 |
| quix 3.23.6 · drain | 0.06 | 0.06 | 48 |
| risingwave v3.1.0 · drain | 0.40 | 0.46 | 80 |
| risingwave v3.1.0 | 0.53 | 0.53 | 80 |
Aggregated output: checked against the expected pane cadence (expected.output_rate in bench.yaml; × drained ÷ event-time rate for a windowed drain), not per-record selectivity.
Verdict: ✓ matches theory — panes/s within 2× of the expected cadence for every row.
Turbine releases
| line | version | bench | regime | offered/s | consumed/s | rows/s per core | µs CPU / record | rows/s per GB | p99 ms | Δ efficiency | Δ capacity | run |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| turbine | 0.3.5a1 | 2 | keep-up | 425,085 | 420,495 | 347,792 | 2.88 | 357,434 | 1,352.5 | +0.3% | - | 2026-11-10 |
| turbine | 0.3.4a1 | 2 | keep-up | 421,622 | 418,674 | 346,790 | 2.88 | 332,527 | 1,257.7 | +2.4% | - | 2026-11-10 |
| turbine | 0.3.3a1 | 2 | keep-up | 423,332 | 418,537 | 338,693 | 2.95 | 341,951 | 1,380.0 | +1.8% | - | 2026-11-06 |
| turbine | 0.3.2a1 | 2 | keep-up | 423,281 | 420,323 | 332,717 | 3.01 | 301,548 | 1,429.6 | -7.7% ⚠ | - | 2026-11-04 |
| turbine | 0.3.1a2 | 2 | keep-up | 424,029 | 420,548 | 360,370 | 2.77 | 285,081 | 1,413.5 | -0.4% | - | 2026-11-01 |
| turbine | 0.3.1a1 | 2 | keep-up | 424,341 | 419,846 | 361,751 | 2.76 | 288,761 | 1,376.2 | ≠ bench | ≠ bench | 2026-10-28 |
| turbine | 0.2.2a2 | 1 ⚠ deprecated | keep-up | 419,010 | 414,432 | 281,743 | 3.55 | 607,932 | 1,831.2 | -6.4% ⚠ | - | 2026-10-24 |
| turbine | 0.2.2a1 | 1 ⚠ deprecated | keep-up | 423,899 | 421,164 | 300,985 | 3.32 | 565,027 | 2,159.3 | -12.2% ⚠ | - | 2026-10-18 |
| turbine | 0.2.1a4 | 1 ⚠ deprecated | keep-up | 425,396 | 422,423 | 342,869 | 2.92 | 487,407 | 1,248.7 | +6.8% | - | 2026-10-18 |
| turbine | 0.2.1a3 | 1 ⚠ deprecated | keep-up | 414,720 | 411,868 | 321,144 | 3.11 | 462,962 | 1,982.3 | +40.8% | - | 2026-10-18 |
| turbine | 0.2.1a2 | 1 ⚠ deprecated | saturated | 425,164 | 355,675 | 228,053 | 4.38 | 375,811 | - | -23.5% ⚠ | -10.0% ⚠ | 2026-10-18 |
| turbine | 0.2.0a2 | 1 ⚠ deprecated | saturated | 425,060 | 395,414 | 298,121 | 3.35 | 420,979 | - | - | - | 2026-10-18 |
| turbine · drain | 0.3.5a1 | 2 | drain | — | 929,207 | 366,490 | 2.73 | 452,991 | - | +1.5% | +0.2% | 2026-11-10 |
| turbine · drain | 0.3.4a1 | 2 | drain | — | 927,631 | 361,145 | 2.77 | 429,452 | - | +0.8% | +0.3% | 2026-11-10 |
| turbine · drain | 0.3.3a1 | 2 | drain | — | 924,868 | 358,280 | 2.79 | 427,924 | - | -3.5% | -1.5% | 2026-11-06 |
| turbine · drain | 0.3.2a1 | 2 | drain | — | 938,676 | 371,410 | 2.69 | 512,031 | - | +2.8% | +1.6% | 2026-11-03 |
| turbine · drain | 0.3.1a2 | 2 | drain | — | 924,280 | 361,469 | 2.77 | 488,872 | - | +1.2% | +2.9% | 2026-11-01 |
| turbine · drain | 0.3.1a1 | 2 | drain | — | 898,409 | 357,281 | 2.80 | 452,035 | - | -3.4% | -6.3% ⚠ | 2026-10-28 |
| turbine · drain | 0.2.2a2 | 2 | drain | — | 958,374 | 369,755 | 2.70 | 492,674 | - | - | - | 2026-10-27 |
| turbine-resident | 0.3.5a1 | 2 | keep-up | 425,239 | 421,524 | 345,259 | 2.90 | 372,403 | 1,366.5 | -3.2% | - | 2026-11-10 |
| turbine-resident | 0.3.4a1 | 2 | keep-up | 425,075 | 420,994 | 356,593 | 2.80 | 374,873 | 1,359.0 | - | - | 2026-11-10 |
| turbine-resident · drain | 0.3.5a1 | 2 | drain | — | 899,912 | 356,887 | 2.80 | 534,303 | - | -5.8% ⚠ | -6.2% ⚠ | 2026-11-10 |
| turbine-resident · drain | 0.3.4a1 | 2 | drain | — | 958,899 | 378,784 | 2.64 | 565,429 | - | - | - | 2026-11-10 |
How to read this table
The Turbine lines measured on several releases, newest first: a regression check, not part of the cross-product comparison above (which shows the pinned release only). Δ compares each release with the one below it: efficiency = rows/s per core (the inverse of µs CPU per record), capacity = consumed/s, only when both runs were fed beyond capacity (saturated or drain — at a keep-up point consumed/s is just the offered rate). ⚠ = worse by more than 5%, about the single-run noise: confirm with --repeat 3. Two releases measured in two sessions also differ by the host-to-host spread — same flavor, another physical host: −0.7 % consumed / −3.5 % rows/s per core in the c3 control of 2026-10-07 —, which leaves little headroom under the 5 %: re-measure both releases in ONE session (suite.py run … --version X) before acting on a ⚠. No delta is computed across a change of bench version (bench, from bench.yaml), host or run config; ⚠ deprecated rows were measured on an older bench definition — re-run that release with suite.py run turbine WORKLOAD --version X.
Host: ovh:c3-32 (ovh:c3-32@DE1, ovh:c3-32@SBG5) · AMD EPYC-Milan Processor (2400 MHz) · 16 cores / 16 threads · 31345 MB · kernel 6.12.100+deb13-cloud-amd64.
Redpanda: rpk version: v26.2.2. Redpanda, gendata and the product under test share this host.