Windowed High Cardinality — exactly-once
The low-cardinality bench's tumbling 30s window, aggregating count + avg(latency_ms) grouped by user_id from a key pool PINNED across products (500k users — see tenants below) — one open window per active user. Stresses the keyed-state store rather than framework overhead. Capacity is read in drain mode, on a backlog carrying synthetic event time (every product meets the same windows and keys); the live runs, at each product's max sustainable single-partition rate (its knee), give the close latency.
Last updated: 2026-11-10 · same-machine, relative comparison.
flowchart LR
src[("bench-input<br/>1 partition")]
s0{{"<b>key by</b><br/>500k keys (user_id)"}}
s1[["<b>window</b><br/>tumbling 30 s, event time<br/>count, avg(latency_ms)"]]
sink[("bench-out")]
state[("keyed state<br/>one open window per key")]
probe>"latency probe<br/>arrival − window_end_ms"]
src --> s0
s0 --> s1
s1 -->|"exactly-once"| sink
s1 -.- state
sink -.-> probeBench characteristics
In-process computation — Tumbling 30s event-time window on the Kafka message timestamp, 1 s grace, GROUP BY user_id (key pool pinned to 500k users — identical for every product regardless of its rate) → count + avg(latency_ms); one pane per key per window close.
| product | delivery | processing | engine | scaling on one partition | decode | state |
|---|---|---|---|---|---|---|
| turbine-eos | exactly-once (Kafka transaction per batch; crash recovery: replay) | batch (Arrow columnar; 100k rows / 500 ms) | in-process · Rust core + free-threaded CPython 3.14t | intra-partition shards (parallelism=N, hash of partition_key, in-process) |
schema projection (6 of 19 fields, Arrow) | own RocksDB store (merge operator for windows) |
| arroyo-eos | exactly-once (checkpoints every 10 s; Kafka transaction per checkpoint, 2PC sink) | batch (vectorized SQL, Arrow / DataFusion; source batches of 100k rows / 500 ms) | in-process · Rust (arroyo run: controller + one worker process) |
hash shuffle to N subtasks (arroyo run --parallelism, in-process queues) |
schema projection (2 of 19 fields) | in-memory tables, checkpointed to the state dir |
| flink-eos | exactly-once (EXACTLY_ONCE checkpoints every 10 s; Kafka transaction per checkpoint, 2PC sink) | batch (pipelined, JVM code-gen SQL) | in-process · JVM (local MiniCluster; Python only submits the job) | keyed exchange to N slots (parallelism.default, in-process local channels; source stays 1 subtask) |
schema projection (3 of 19 fields, Jackson); connector-default fetch | RocksDB (embedded, on-disk LSM) |
| quix-eos | exactly-once (Kafka transaction per commit, every 5 s) | row (per-record StreamingDataFrame) | in-process · pure-Python over librdkafka (C client) | none (one consumer per partition, GIL) | full record (19 of 19 fields, orjson dict) | RocksDB (embedded); changelog topic OFF for the bench |
Input is a single Kafka partition (lz4). Each product may spread it over several cores through its own in-process mechanism (column scaling on one partition; what it actually used is cpu_avg); an in-process exchange is not a broker hop — the extra rec/rec column counts records that DID cross the broker on other topics. Every line runs exactly-once: its output becomes visible to a read_committed consumer at its transaction's commit (column delivery), windows are tumbling 30 s event-time on the Kafka message timestamp.
Consumer fetch: Turbine's core hard-codes librdkafka fetch.min.bytes=64 KB, fetch.wait.max.ms=10, max.partition.fetch.bytes=50 MB and a 500k-msg / ~2 GB prefetch queue (not adjustable), so the SAME client gets the SAME settings: quix and RisingWave (librdkafka) run with them, and so does FastStream (aiokafka, whose equivalent knobs lifted it ~14%). Flink runs at its Java connector's defaults: an A/B (2026-09-10) showed the identical numbers throttled the Java consumer (373k/s vs 543k/s at the same ~285k rows/s per core) — identical numbers are not identical behaviour on a different client. Memory at saturation is mostly this prefetch queue, for every librdkafka product.
Drain ceiling (--prefill: static backlog, no producer running)
| product | backlog | event time/s | warmup + window | drained/s | cpu_avg % | rows/s per core | rows/s per GB | µs CPU / record | panes/s | run |
|---|---|---|---|---|---|---|---|---|---|---|
| turbine-eos 0.3.5a1 | 200M | 300k | 40 + 120 s | 421,171 | 274.8 | 153,277 | 138,365 | 6.52 | 23,656.47 | 2026-11-10 |
| arroyo-eos 0.15.0 | 200M | 300k | 40 + 240 s | 567,229 | 119.9 | 473,212 | 408,848 | 2.11 | 31,080.30 | 2026-11-06 |
| flink-eos 2.2.1 | 200M | 300k | 268 + 515 s | 54,395 | 179.9 | 30,244 | 30,527 | 33.06 | 2,897.56 | 2026-11-06 |
| quix-eos 3.23.6 | 140M | 300k | 385 + 750 s | 28,045 | 106.0 | 26,445 | 16,826 | 37.81 | 1,330.84 | 2026-11-07 |
How to read this table
gendata wrote the backlog and exited BEFORE the consumer started, so the window measures a drain with no producer competing for the box — the product's consume ceiling free of gendata's CPU and 10 Hz burstiness. Not comparable with a live table (different regime); ⚠ exhausted = the backlog ran out mid-window, rate understated.
The backlog's records carry synthetic event time, event time/s records per second of it (gendata --event-time-rate): it stands for a live stream at that rate, so every product meets the SAME windows — the same records and keys per window, which live runs at each product's own knee do not — and drained/s is its capacity on that problem, closes included. Event time runs at drained ÷ event-time rate × the wall clock, panes/s with it. A product draining far below the event-time rate gets a longer warmup and window (≥ its first close, then two more, at its live knee's pace: warmup + window). No latency here: a pane's age is the backlog's. 0 ⚠ panes/s: no pane was published during the window — the drain did not run the whole pipeline, its drained/s is not comparable.
Throughput / efficiency (live, at each product's knee) — capped by the machine: gendata produces ~430k/s at most, so read capacity in the drain table
| product | offered/s | consumed/s | regime (lag drift) | cpu_avg % | rows/s per core | rows/s per GB | µs CPU / record | panes/s | extra rec/rec | run |
|---|---|---|---|---|---|---|---|---|---|---|
| turbine-eos 0.3.5a1 | 424,025 | 423,991 | keep-up (-0.0%) | 237.8 | 178,263 | 131,410 | 5.61 | 16,479.47 | 0 | 2026-11-10 |
| arroyo-eos 0.15.0 | 395,486 | 383,756 | keep-up (+3.1%) | 91.7 | 418,622 | 378,749 | 2.39 | 16,581.04 | 0 | 2026-11-06 |
| flink-eos 2.2.1 | 34,626 | 34,451 | keep-up (+0.5%) | 127.9 | 26,933 | 21,983 | 37.13 | 14,260.41 | 0 | 2026-11-06 |
| quix-eos 3.23.6 | 23,743 | 22,238 | saturated (+6.8%) | 104.8 | 21,225 | 19,993 | 47.11 | 12,522.16 | 2.16 | 2026-11-06 |
How to read this table
consumed/s is what the product actually drained (its consumer-group offset or engine counter), offered/s what gendata produced; a dotted underline marks the median of a repeated batch — hover it for the run count and the min – max. regime (lag drift): keep-up = consumed ≈ offered with a flat backlog (a sustainable point); saturated = fed beyond capacity, so consumed/s IS the ceiling and latency is unbounded; draining = still eating warmup backlog (consumed > offered; CPU is catch-up, not steady state — lengthen the warmup, or raise the rate if the ceiling is what you want). The lag drift in parentheses = backlog change over the window ÷ records consumed — the evidence for the regime (≈ 0 at keep-up, positive when saturated, negative while draining). cpu_avg is the whole process tree, 100 = one core busy. rows/s per core = consumed ÷ (cpu_avg/100); rows/s per GB = consumed ÷ memory; extra rec/rec = records written to topics other than the input and output per record consumed (0 = no broker hop).
panes/s is the window-emission cadence (≈ keys ÷ window size), a liveness signal, not a throughput figure — it only moves when windows close, which needs the consumer to keep up.
Cost of exactly-once
| line | vs | regime | rows/s per core | µs CPU / record | consumed/s | close p50 ms | measured |
|---|---|---|---|---|---|---|---|
| turbine-eos · drain | turbine | drain → drain | -1.6% | 6.42 → 6.52 | -8.1% | - | interleaved |
| turbine-eos | turbine | keep-up → keep-up | -1.3% | 5.54 → 5.61 | - | 4,366 → 4,464 | interleaved |
| arroyo-eos · drain | arroyo | drain → drain | -1.8% | 2.08 → 2.11 | -4.9% | - | separately |
| arroyo-eos | arroyo | keep-up → keep-up | -8.2% | 2.19 → 2.39 | - | 3,739 → 9,733 | separately |
| flink-eos · drain | flink | drain → drain | -5.4% | 31.29 → 33.06 | -4.7% | - | separately |
| flink-eos | flink | keep-up → keep-up | -6.8% | 34.59 → 37.13 | - | 2,995 → 13,591 | separately |
| quix-eos · drain | quix | drain → drain | -17.7% | 31.11 → 37.81 | -15.5% | - | separately |
| quix-eos | quix | keep-up → saturated | -16.7% | 39.25 → 47.11 | - | - | separately |
How to read this table
Each exactly-once line against its at-least-once line — the same product, workload, mode and cores, the newest batch of each. rows/s per core: the change in efficiency (the inverse of µs CPU per record, shown at-least-once → exactly-once); consumed/s: the change in capacity, only when both runs were fed beyond it (drain or saturated — at a keep-up point it is the offered rate); the latency p50 when both kept up — an exactly-once line's output becomes visible to the probe, a read_committed consumer, at its commit (a batch's transaction for Turbine, a checkpoint's for Flink and Arroyo, a commit's for Quix: the interval is in the characteristics' delivery and processing columns — on the latency bench every exactly-once line commits at Turbine's batch timeout, a transaction per batch, the others through BENCH_COMMIT_INTERVAL_MS (Arroyo every 1 s, the shortest checkpoint interval it accepts), which their at-least-once lines, publishing without waiting for a commit, do not set: ≠ config there). measured: interleaved = both lines in one interleaved batch (one host, run by run); separately = two sessions, the difference then carrying the host-to-host spread too (a few %; a Flink VM once read 30 % slower, before the session gate). · ≠ config / ≠ host / ≠ bench: the two runs differ there too — read the figures with it. Output records/s read from offsets include one transaction marker per transaction that wrote to the output — noise on the per-record workloads (≲ 1 per 10k records), up to ~1 in 16 panes on windowed-lowcard's 0.53 panes/s.
Latency (close latency — pane published minus window end)
| product | regime | p50 ms | p90 ms | p99 ms | p99.9 ms | max ms | records | setup floor p50 / p99 ms | added p50 / p99 ms |
|---|---|---|---|---|---|---|---|---|---|
| turbine-eos 0.3.5a1 | keep-up | 4,463.8 | 6,355.9 | 6,960.8 | 7,052.5 | 7,257.4 | 5982784 | 106.6 / 204.8 | 4,357.1 / 6,756.0 |
| arroyo-eos 0.15.0 | keep-up | 9,733.0 | 13,719.1 | 15,684.6 | 15,772.9 | 15,979.3 | 10966840 | 106.5 / 205.2 | 9,626.5 / 15,479.4 |
| flink-eos 2.2.1 | keep-up | 13,591.0 | 23,018.2 | 25,949.0 | 26,024.3 | 26,233.4 | 10038515 | 106.5 / 204.7 | 13,484.5 / 25,744.3 |
| quix-eos 3.23.6 | saturated | — | — | — | — | — | 4537344 | — | — |
How to read this table
Measured by a probe consumer tailing the output topic (C client, 10 ms max fetch wait — the probe adds ≲10 ms, the same for every product). Only shown for keep-up runs: under saturation the figure is the backlog's age and grows without bound. setup floor is a harness producer writing one small record every 10 ms (acks=all, 5 ms linger) to its own topic during the same run, measured by the same probe: the broker's produce→visible latency at the products' output shape (on Redpanda, small produces surface on its flush cadence), which every product inherits. added = product − floor, the part the product is responsible for (fetch, batching, commit/epoch cadence, produce). The floor is a low-volume topic's (100 small records/s, surfacing on Redpanda's 100 ms flush.ms): an output topic carrying several MB/s — every record scored and republished — surfaces sooner: a percentile under the floor's reads below, added does not apply. Colours: each percentile against the best (lowest) of its column — green under 2×, yellow from 2×, orange from 5×, red from 10× (the ratio on hover).
Resources (raw)
| product | cpu_avg % | cpu_p95 % | mem_avg MB | mem_max MB | samples | instrument |
|---|---|---|---|---|---|---|
| turbine-eos 0.3.5a1 | 274.8 | 389.6 | 3,117.0 | 3,555.7 | 117 | /proc Δ, pss |
| turbine-eos 0.3.5a1 | 237.8 | 342.7 | 3,303.9 | 3,588.6 | 116 | /proc Δ, pss |
| arroyo-eos 0.15.0 | 119.9 | 173.3 | 1,420.7 | 1,712.3 | 234 | /proc Δ, pss |
| arroyo-eos 0.15.0 | 91.7 | 124.5 | 1,037.5 | 1,282.0 | 234 | /proc Δ, pss |
| flink-eos 2.2.1 | 179.9 | 343.8 | 1,824.6 | 1,846.4 | 503 | /proc Δ, pss |
| flink-eos 2.2.1 | 127.9 | 360.1 | 1,604.8 | 1,611.1 | 233 | /proc Δ, pss |
| quix-eos 3.23.6 | 106.0 | 124.1 | 1,706.8 | 1,926.4 | 727 | /proc Δ, pss |
| quix-eos 3.23.6 | 104.8 | 123.6 | 1,139.0 | 1,355.0 | 117 | /proc Δ, pss |
How to read this table
CPU is summed over the whole process tree (JVM, UDF servers, the RisingWave container's processes) as utime+stime deltas per 1 s sample — 100 = one core. Memory is PSS (shared pages counted once) where readable, RSS otherwise (instrument). Rows marked ⚠ legacy ps were measured with the old ps %CPU lifetime average and are not comparable: re-run them.
Sanity — ✓ matches theory
Are we benching the same thing?
| product | panes/s | expected | sample |
|---|---|---|---|
| turbine-eos 0.3.5a1 · drain | 23,656.47 | 24,315.80 | 499 |
| turbine-eos 0.3.5a1 | 16,479.47 | 16,666.67 | 499 |
| arroyo-eos 0.15.0 · drain | 31,080.30 | 31,627.92 | 497 |
| arroyo-eos 0.15.0 | 16,581.04 | 16,666.67 | 497 |
| flink-eos 2.2.1 · drain | 2,897.56 | 3,021.93 | 497 |
| flink-eos 2.2.1 | 14,260.41 | 14,625.73 | 497 |
| quix-eos 3.23.6 · drain | 1,330.84 | 1,558.06 | 499 |
| quix-eos 3.23.6 | 12,522.16 | 12,717.87 | 499 |
Aggregated output: checked against the expected pane cadence (expected.output_rate in bench.yaml; × drained ÷ event-time rate for a windowed drain), not per-record selectivity.
Verdict: ✓ matches theory — panes/s within 2× of the expected cadence for every row.
Turbine releases
| line | version | bench | regime | offered/s | consumed/s | rows/s per core | µs CPU / record | rows/s per GB | p99 ms | Δ efficiency | Δ capacity | run |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| turbine-eos | 0.3.5a1 | 3 | keep-up | 424,025 | 423,991 | 178,263 | 5.61 | 131,410 | 6,960.8 | +1.6% | - | 2026-11-10 |
| turbine-eos | 0.3.4a1 | 3 | keep-up | 424,770 | 424,923 | 175,442 | 5.70 | 121,051 | 7,287.0 | - | - | 2026-11-10 |
| turbine-eos · drain | 0.3.5a1 | 3 | drain | — | 421,171 | 153,277 | 6.52 | 138,365 | - | +0.4% | +4.5% | 2026-11-10 |
| turbine-eos · drain | 0.3.4a1 | 3 | drain | — | 402,905 | 152,729 | 6.55 | 113,728 | - | - | - | 2026-11-10 |
How to read this table
The Turbine lines measured on several releases, newest first: a regression check, not part of the cross-product comparison above (which shows the pinned release only). Δ compares each release with the one below it: efficiency = rows/s per core (the inverse of µs CPU per record), capacity = consumed/s, only when both runs were fed beyond capacity (saturated or drain — at a keep-up point consumed/s is just the offered rate). ⚠ = worse by more than 5%, about the single-run noise: confirm with --repeat 3. Two releases measured in two sessions also differ by the host-to-host spread — same flavor, another physical host: −0.7 % consumed / −3.5 % rows/s per core in the c3 control of 2026-10-07 —, which leaves little headroom under the 5 %: re-measure both releases in ONE session (suite.py run … --version X) before acting on a ⚠. No delta is computed across a change of bench version (bench, from bench.yaml), host or run config; ⚠ deprecated rows were measured on an older bench definition — re-run that release with suite.py run turbine WORKLOAD --version X.
Host: ovh:c3-32 (ovh:c3-32@DE1, ovh:c3-32@SBG5, ovh:c3-32@UK1) · AMD EPYC-Milan Processor (2400 MHz) · 16 cores / 16 threads · 31345 MB · kernel 6.12.100+deb13-cloud-amd64.
Redpanda: rpk version: v26.2.2. Redpanda, gendata and the product under test share this host.