Model Inference
Score every record with the SAME frozen 4-16-1 PyTorch MLP (models/, trained once by models/make_model.py). No filter, so selectivity is 1.0 — this prices what each framework costs around a forward pass: decode, feature extraction, the call itself, encode, produce. The model is deliberately tiny so the frameworks, not the matmul, are what is measured. Benched at each product's max sustainable single-partition rate (its knee).
Last updated: 2026-11-10 · same-machine, relative comparison.
flowchart LR
src[("bench-input<br/>1 partition")]
s0["<b>map</b><br/>4 features scaled to [0, 1]"]
s1(["<b>model</b><br/>MLP 4-16-1 → risk"])
sink[("bench-out")]
src --> s0
s0 --> s1
s1 -->|"at-least-once"| sinkBench characteristics
In-process computation — Per record: four gendata metrics (latency_ms, queue_depth, cpu_usage, error_count) divided by their documented maxima into [0, 1], fed to a frozen 4-16-1 MLP (Linear/ReLU/Linear/Sigmoid) → risk. Stateless; every record is scored and republished.
| product | delivery | processing | engine | scaling on one partition | decode |
|---|---|---|---|---|---|
| turbine | at-least-once (idempotent, acks=all) | batch (Arrow columnar; 100k rows / 500 ms) | in-process · Rust core + free-threaded CPython 3.14t | intra-partition shards (parallelism=N, hash of partition_key, in-process) |
schema projection (9 of 19 fields, Arrow) |
| turbine-torch | at-least-once (idempotent, acks=all) | batch (Arrow columnar → DLPack → torch eager; 100k rows / 500 ms) | in-process · Rust core + free-threaded CPython 3.14t | intra-partition shards (parallelism=N) |
schema projection (9 of 19 fields, Arrow) |
| arroyo | at-least-once (checkpointed) | vectorized SQL around a Rust UDF on ONNX Runtime (one ORT run per record) | in-process · Rust (arroyo cluster: API + compiler + controller + one worker process) |
none for a per-record job (chained into the subtask reading the partition) | schema projection (7 of 19 fields) |
| faststream | at-least-once (idempotent, acks=all; fire-and-forget publish) | batch I/O (getmany 100k rows / 500 ms), per-row compute | in-process · pure-Python async (aiokafka; orjson JSON codec) | none (one async consumer) | full record (19 of 19 fields, orjson dict) |
| flink | at-least-once (checkpointed) | batch (pipelined, JVM code-gen SQL) | in-process · JVM (local MiniCluster; Python only submits the job) | rebalance to N slots (parallelism.default, in-process local channels; source stays 1 subtask) |
schema projection (7 of 19 fields, Jackson); connector-default fetch |
| pyflink | at-least-once (checkpointed) | Arrow batches (100k rows) over gRPC to a Python worker (pandas UDF, PROCESS mode) → torch eager | in-process · JVM (local MiniCluster) + Python UDF worker | rebalance to N slots (parallelism.default, in-process local channels; source stays 1 subtask) |
schema projection (7 of 19 fields, Jackson); connector-default fetch |
| quix | at-least-once (idempotent, acks=all) | row (per-record StreamingDataFrame) | in-process · pure-Python over librdkafka (C client) | none (one consumer per partition, GIL) | full record (19 of 19 fields, orjson dict) |
| risingwave | at-least-once | batch (vectorized SQL) | external server · Rust (single-node: meta+compute+frontend) | hash exchange to N stream actors (SET streaming_parallelism, in-process) |
schema projection (7 of 19 fields) |
Input is a single Kafka partition (lz4). Each product may spread it over several cores through its own in-process mechanism (column scaling on one partition; what it actually used is cpu_avg); an in-process exchange is not a broker hop — the extra rec/rec column counts records that DID cross the broker on other topics. All products run at-least-once.
Consumer fetch: Turbine's core hard-codes librdkafka fetch.min.bytes=64 KB, fetch.wait.max.ms=10, max.partition.fetch.bytes=50 MB and a 500k-msg / ~2 GB prefetch queue (not adjustable), so the SAME client gets the SAME settings: quix and RisingWave (librdkafka) run with them, and so does FastStream (aiokafka, whose equivalent knobs lifted it ~14%). Flink runs at its Java connector's defaults: an A/B (2026-09-10) showed the identical numbers throttled the Java consumer (373k/s vs 543k/s at the same ~285k rows/s per core) — identical numbers are not identical behaviour on a different client. Memory at saturation is mostly this prefetch queue, for every librdkafka product.
Drain ceiling (--prefill: static backlog, no producer running)
| product | backlog | drained/s | cpu_avg % | rows/s per core | rows/s per GB | µs CPU / inference | output/s | run |
|---|---|---|---|---|---|---|---|---|
| turbine 0.3.5a1 | 120M | 825,187 | 361.8 | 228,062 | 463,947 | 4.38 | 847,168 | 2026-11-10 |
| turbine-torch 0.3.5a1 | 120M | 952,463 | 351.5 | 270,985 | 496,895 | 3.69 | 914,475 | 2026-11-10 |
| arroyo 0.15.0 | 120M | 357,441 | 217.8 | 164,132 | 180,470 | 6.09 | 367,170 | 2026-10-28 |
| faststream 0.7.1 | 120M | 42,278 | 99.8 | 42,374 | 58,415 | 23.60 | 40,744 | 2026-10-28 |
| flink 2.2.1 | 60M | 357,104 | 482.1 | 74,075 | 221,443 | 13.50 | 363,281 | 2026-10-20 |
| pyflink 2.2.1 | 120M | 320,794 | 275.5 | 116,445 | 148,513 | 8.59 | 325,008 | 2026-10-28 |
| quix 3.23.6 | 60M | 29,768 | 102.7 | 28,992 | 32,709 | 34.49 | 30,372 | 2026-10-20 |
| risingwave v3.1.0 | 60M | 148,990 | 173.2 | 86,016 | 123,715 | 11.63 | 150,470 | 2026-10-20 |
How to read this table
gendata wrote the backlog and exited BEFORE the consumer started, so the window measures a drain with no producer competing for the box — the product's consume ceiling free of gendata's CPU and 10 Hz burstiness. Not comparable with a live table (different regime); ⚠ exhausted = the backlog ran out mid-window, rate understated.
Resources (raw)
| product | cpu_avg % | cpu_p95 % | mem_avg MB | mem_max MB | samples | instrument |
|---|---|---|---|---|---|---|
| turbine 0.3.5a1 | 361.8 | 386.2 | 1,821.3 | 1,929.1 | 58 | /proc Δ, pss |
| turbine-torch 0.3.5a1 | 351.5 | 371.6 | 1,962.8 | 2,064.1 | 58 | /proc Δ, pss |
| arroyo 0.15.0 | 217.8 | 251.1 | 2,028.1 | 2,345.2 | 59 | /proc Δ, pss |
| faststream 0.7.1 | 99.8 | 101.0 | 741.1 | 851.7 | 59 | /proc Δ, pss |
| flink 2.2.1 | 482.1 | 498.5 | 1,651.3 | 1,663.7 | 59 | /proc Δ, pss |
| pyflink 2.2.1 | 275.5 | 290.7 | 2,211.9 | 2,230.4 | 58 | /proc Δ, pss |
| quix 3.23.6 | 102.7 | 109.7 | 931.9 | 941.2 | 59 | /proc Δ, pss |
| risingwave v3.1.0 | 173.2 | 207.5 | 1,233.2 | 1,245.3 | 58 | /proc Δ, mixed |
How to read this table
CPU is summed over the whole process tree (JVM, UDF servers, the RisingWave container's processes) as utime+stime deltas per 1 s sample — 100 = one core. Memory is PSS (shared pages counted once) where readable, RSS otherwise (instrument). Rows marked ⚠ legacy ps were measured with the old ps %CPU lifetime average and are not comparable: re-run them.
Product notes
turbine — NATIVE operator — infer=ml.OnnxModel(...) on the subscription: the engine runs ONNX Runtime on every emitted batch in the worker's own thread (one session per partition), no Python in the loop; the handler only produces the four normalised feature columns (Arrow kernels). Same frozen network as everyone else (the ONNX export of the .pt, asserted against the card's probe rows at startup). Known surcharge: infer= appends the score to what the handler emits, so the four float32 feature columns are published too — four columns the other products do not write. Compare with turbine-torch, the same app with eager PyTorch in the handler.
turbine-torch — CONTROL LINE — Turbine with the model as eager PyTorch inside the handler (Arrow columns → DLPack, zero copy per column → one forward per batch), the iso-runtime row against the other products' eager-PyTorch lines. The gap to turbine is what Python-in-the-loop costs Turbine — here less than the native line's four published feature columns.
arroyo — The model is a Rust UDF on ONNX Runtime (arroyo/risk_udf.rs), compiled by Arroyo's own compiler service from its source — its Python UDFs run once per row in an embedded sub-interpreter, where numpy / torch do not load. arroyo run takes no UDF, so this workload runs arroyo cluster (API, controller, compiler and one worker process, all sampled) and submits the pipeline over its REST API. One ORT run per record, like Flink's Java UDF, on the same ONNX export and ONNX Runtime release as Turbine's operator. The per-record job is chained into the subtask reading the partition: no exchange spreads the model over the other cores.
faststream — CPU-bound on one core; the leanest footprint of the inference lines. The forward IS batched — the batch=True subscriber hands the handler a list — so what is being paid is the pure-Python loop around it and one publish per record.
flink — NATIVE — a Java ScalarFunction (flink/java/RiskUdf.java) on ONNX Runtime Java 1.29.0, the release Turbine's operator loads, on the same ONNX artefact, inside the JVM operator: the job is pure JVM like every other Flink workload (Python only submits it). Flink's scalar UDFs are called once per row — it has no vectorised Java UDF — so this is one ORT run PER RECORD, the framework's shape (Turbine runs one per batch). The features are normalised by the SQL around the call. Runs at cores 3 (the UDF rebalanced over 3 slots, ~10 % more CPU per record than cores 1 on the dev box); pyflink, the same job with the model as a pandas UDF, runs at cores 3 too (since 2026-10-20).
pyflink — CONTROL LINE — the flink job with the model as a Pandas UDF in PROCESS mode: every batch is serialised to Arrow, crosses gRPC to a separate Python worker (eager PyTorch, one thread, one forward per batch), and comes back — what "run a model in PyFlink" costs. The gap to flink is what that hop costs Flink. Rows measured before 2026-10-13 (as flink: ~10 cores at 81k/s) ran torch with its default thread pool in the worker — a bench bug (the driver's model check shipped the model pre-built, so the worker never pinned one thread), not PyFlink's cost: do not quote them. THREAD mode would remove the hop but forbids pandas UDFs and is GIL-bound.
quix — One PyTorch call PER RECORD — quixstreams' StreamingDataFrame has no batching primitive, so this is not a handicap, it is the framework: every other product here batches the forward because its API hands over a batch.
risingwave — Measured as the CONTAINER + the host UDF server (see benches.py also_process_tree): RisingWave cannot run a model in-engine — embedded Python UDFs are disabled by default and stdlib-only — so the model lives in an Arrow Flight server and every batch crosses that boundary. One thing the batching does NOT buy: arrow-udf boxes every incoming cell to a Python object before the function sees it, whatever the mode, so the columnar wire format does not survive to the model.
Sanity — ✓ matches theory
Are we benching the same thing?
| product | selectivity (out/consumed) | expected | ✓ |
|---|---|---|---|
| turbine 0.3.5a1 | 1.0252 | 1.0000 | ✓ |
| turbine-torch 0.3.5a1 | 0.9391 | 1.0000 | ✓ |
| arroyo 0.15.0 | 1.0082 | 1.0000 | ✓ |
| faststream 0.7.1 | 0.9669 | 1.0000 | ✓ |
| flink 2.2.1 | 1.0174 | 1.0000 | ✓ |
| pyflink 2.2.1 | 1.0111 | 1.0000 | ✓ |
| quix 3.23.6 | 1.0188 | 1.0000 | ✓ |
| risingwave v3.1.0 | 1.0024 | 1.0000 | ✓ |
Verdict: ✓ matches theory — every row's selectivity is within 25% of the expected value (output ÷ consumed).
Turbine releases
| line | version | bench | regime | offered/s | consumed/s | rows/s per core | µs CPU / inference | rows/s per GB | p99 ms | Δ efficiency | Δ capacity | run |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| turbine · drain | 0.3.5a1 | 1 | drain | — | 825,187 | 228,062 | 4.38 | 463,947 | - | -6.6% ⚠ | -5.3% ⚠ | 2026-11-10 |
| turbine · drain | 0.3.4a1 | 1 | drain | — | 871,252 | 244,053 | 4.10 | 484,188 | - | +0.4% | +0.8% | 2026-11-10 |
| turbine · drain | 0.3.3a1 | 1 | drain | — | 864,264 | 242,985 | 4.12 | 464,259 | - | -5.2% ⚠ | +0.5% | 2026-11-06 |
| turbine · drain | 0.3.2a1 | 1 | drain | — | 860,212 | 256,343 | 3.90 | 419,811 | - | +2.4% | -3.4% | 2026-11-03 |
| turbine · drain | 0.3.1a2 | 1 | drain | — | 890,112 | 250,296 | 4.00 | 436,754 | - | +1.5% | +3.7% | 2026-11-01 |
| turbine · drain | 0.3.1a1 | 1 | drain | — | 858,742 | 246,596 | 4.06 | 427,018 | - | -12.2% ⚠ | +41.2% | 2026-10-28 |
| turbine · drain | 0.2.2a2 | 1 | drain | — | 608,349 | 280,703 | 3.56 | 293,823 | - | ≠ config | ≠ config | 2026-10-28 |
| turbine · drain | 0.2.2a1 | 1 | drain | — | 594,639 | 270,287 | 3.70 | 284,953 | - | -0.9% | +14.4% | 2026-10-18 |
| turbine · drain | 0.2.1a4 | 1 | drain | — | 519,819 | 272,629 | 3.67 | 294,150 | - | +1.6% | -0.3% | 2026-10-18 |
| turbine · drain | 0.2.1a3 | 1 | drain | — | 521,358 | 268,464 | 3.72 | 287,747 | - | -1.5% | -1.0% | 2026-10-18 |
| turbine · drain | 0.2.1a2 | 1 | drain | — | 526,715 | 272,527 | 3.67 | 287,069 | - | -5.5% ⚠ | +82.2% | 2026-10-18 |
| turbine · drain | 0.2.0a2 | 1 | drain | — | 289,138 | 288,449 | 3.47 | 206,260 | - | - | - | 2026-10-18 |
| turbine-torch · drain | 0.3.5a1 | 1 | drain | — | 952,463 | 270,985 | 3.69 | 496,895 | - | +2.2% | +3.4% | 2026-11-10 |
| turbine-torch · drain | 0.3.4a1 | 1 | drain | — | 920,902 | 265,159 | 3.77 | 469,146 | - | -5.2% ⚠ | -5.9% ⚠ | 2026-11-10 |
| turbine-torch · drain | 0.3.3a1 | 1 | drain | — | 978,793 | 279,620 | 3.58 | 504,797 | - | -1.5% | +2.5% | 2026-11-06 |
| turbine-torch · drain | 0.3.2a1 | 1 | drain | — | 954,768 | 283,978 | 3.52 | 476,586 | - | +4.1% | +0.4% | 2026-11-03 |
| turbine-torch · drain | 0.3.1a2 | 1 | drain | — | 950,791 | 272,696 | 3.67 | 500,475 | - | -2.8% | -1.8% | 2026-11-01 |
| turbine-torch · drain | 0.3.1a1 | 1 | drain | — | 968,298 | 280,466 | 3.57 | 478,907 | - | -6.7% ⚠ | +21.2% | 2026-10-28 |
| turbine-torch · drain | 0.2.2a2 | 1 | drain | — | 798,837 | 300,467 | 3.33 | 396,947 | - | ≠ config | ≠ config | 2026-10-28 |
| turbine-torch · drain | 0.2.2a1 | 1 | drain | — | 764,775 | 291,800 | 3.43 | 389,755 | - | -1.2% | +30.5% | 2026-10-18 |
| turbine-torch · drain | 0.2.1a4 | 1 | drain | — | 585,924 | 295,344 | 3.39 | 331,219 | - | -1.1% | -1.1% | 2026-10-18 |
| turbine-torch · drain | 0.2.1a3 | 1 | drain | — | 592,630 | 298,744 | 3.35 | 339,976 | - | -4.5% | -1.0% | 2026-10-18 |
| turbine-torch · drain | 0.2.1a2 | 1 | drain | — | 598,918 | 312,693 | 3.20 | 324,542 | - | -3.5% | +69.3% | 2026-10-18 |
| turbine-torch · drain | 0.2.0a2 | 1 | drain | — | 353,777 | 323,949 | 3.09 | 240,478 | - | - | - | 2026-10-18 |
How to read this table
The Turbine lines measured on several releases, newest first: a regression check, not part of the cross-product comparison above (which shows the pinned release only). Δ compares each release with the one below it: efficiency = rows/s per core (the inverse of µs CPU per record), capacity = consumed/s, only when both runs were fed beyond capacity (saturated or drain — at a keep-up point consumed/s is just the offered rate). ⚠ = worse by more than 5%, about the single-run noise: confirm with --repeat 3. Two releases measured in two sessions also differ by the host-to-host spread — same flavor, another physical host: −0.7 % consumed / −3.5 % rows/s per core in the c3 control of 2026-10-07 —, which leaves little headroom under the 5 %: re-measure both releases in ONE session (suite.py run … --version X) before acting on a ⚠. No delta is computed across a change of bench version (bench, from bench.yaml), host or run config; ⚠ deprecated rows were measured on an older bench definition — re-run that release with suite.py run turbine WORKLOAD --version X.
Host: ovh:c3-32 (ovh:c3-32@DE1, ovh:c3-32@SBG5, ovh:c3-32@UK1) · AMD EPYC-Milan Processor (2400 MHz) · 16 cores / 16 threads · 31345 MB · kernel 6.12.100+deb13-cloud-amd64.
Redpanda: rpk version: v26.2.2. Redpanda, gendata and the product under test share this host.