Model Inference
Score every record with the SAME frozen 4-16-1 PyTorch MLP (models/, trained once by models/make_model.py). No filter, so selectivity is 1.0 — this prices what each framework costs around a forward pass: decode, feature extraction, the call itself, encode, produce. The model is deliberately tiny so the frameworks, not the matmul, are what is measured. Benched at each product's max sustainable single-partition rate (its knee).
Last updated: 2026-09-10 · same-machine, relative comparison.
Bench characteristics
In-process computation — Per record: four gendata metrics (latency_ms, queue_depth, cpu_usage, error_count) divided by their documented maxima into [0, 1], fed to a frozen 4-16-1 MLP (Linear/ReLU/Linear/Sigmoid) → risk. Stateless; every record is scored and republished.
| product | delivery | processing | engine | scaling on one partition | decode |
|---|---|---|---|---|---|
| turbine | at-least-once (idempotent, acks=all) | batch (Arrow columnar) | in-process · Rust core + free-threaded CPython 3.14t | intra-partition shards (parallelism=N, hash of partition_key, in-process) |
schema projection (6 of 19 fields, Arrow) |
| turbine-torch | at-least-once (idempotent, acks=all) | batch (Arrow columnar → DLPack → torch eager) | in-process · Rust core + free-threaded CPython 3.14t | intra-partition shards (parallelism=N) |
schema projection (9 of 19 fields, Arrow) |
| faststream | at-least-once (idempotent, acks=all; fire-and-forget publish) | batch I/O, per-row compute | in-process · pure-Python async (aiokafka; orjson JSON codec) | none (one async consumer) | full record (19 of 19 fields, orjson dict) |
| flink | at-least-once (checkpointed) | batch (pipelined, JVM code-gen SQL) | in-process · JVM (PyFlink local MiniCluster) | keyed exchange to N slots (parallelism.default, in-process local channels; source stays 1 subtask) |
schema projection (4 of 19 fields, Jackson); connector-default fetch |
| quix | at-least-once (idempotent, acks=all) | row (per-record StreamingDataFrame) | in-process · pure-Python over librdkafka (C client) | none (one consumer per partition, GIL) | full record (19 of 19 fields, orjson dict) |
| risingwave | at-least-once | batch (vectorized SQL) | external server · Rust (single-node: meta+compute+frontend) | hash exchange to N stream actors (SET streaming_parallelism, in-process) |
schema projection (4 of 19 fields) |
Input is a single Kafka partition (lz4). Each product may spread it over several cores through its own in-process mechanism (column scaling on one partition; what it actually used is cpu_avg); an in-process exchange is not a broker hop — the extra rec/rec column counts records that DID cross the broker on other topics. All products run at-least-once.
Consumer fetch: Turbine's core hard-codes librdkafka fetch.min.bytes=64 KB, fetch.wait.max.ms=10, max.partition.fetch.bytes=50 MB and a 500k-msg / ~2 GB prefetch queue (not adjustable), so the SAME client gets the SAME settings: quix and RisingWave (librdkafka) run with them, and so does FastStream (aiokafka, whose equivalent knobs lifted it ~14%). Flink runs at its Java connector's defaults: an A/B (2026-09-10) showed the identical numbers throttled the Java consumer (373k/s vs 543k/s at the same ~285k rows/s per core) — identical numbers are not identical behaviour on a different client. Memory at saturation is mostly this prefetch queue, for every librdkafka product.
Throughput / efficiency (live)
| product | offered/s | consumed/s | regime | lag drift | cpu_avg % | rows/s per core | rows/s per GB | µs CPU / inference | output/s | extra rec/rec | run |
|---|---|---|---|---|---|---|---|---|---|---|---|
| turbine 0.2.0a1 | 395,733 | 359,194 | saturated | +10.4% | 90.4 | 397,307 | 277,981 | 2.52 | 367,026 | 0 | 2026-09-10 |
| turbine-torch 0.2.0a1 | 482,078 | 436,025 | saturated | +10.8% | 94.0 | 463,763 | 328,924 | 2.16 | 405,915 | 0 | 2026-09-10 |
| faststream 0.7.1 | 74,217 | 57,441 | saturated | +29.4% | 99.9 | 57,525 | 143,267 | 17.38 | 57,420 | 0 | 2026-09-10 |
| flink 2.2.1 | 147,958 | 116,114 | saturated | +27.7% | 808.5 | 14,362 | 100,223 | 69.63 | 117,083 | 0 | 2026-09-10 |
| quix 3.23.6 | 39,592 | 39,307 | keep-up | +0.7% | 86.4 | 45,496 | 141,496 | 21.98 | 39,577 | 0 | 2026-09-10 |
| risingwave v2.8.4 | 148,416 | 148,492 | keep-up | -0.1% | 105.7 | 140,527 | 175,964 | 7.12 | 148,374 | 0 | 2026-09-10 |
consumed/s is what the product actually drained (its consumer-group offset or engine counter), offered/s what gendata produced. regime: keep-up = consumed ≈ offered with a flat backlog (a sustainable point); saturated = fed beyond capacity, so consumed/s IS the ceiling and latency is unbounded; draining = still eating warmup backlog (consumed > offered; CPU is catch-up, not steady state — lengthen the warmup, or raise the rate if the ceiling is what you want). lag drift = backlog change over the window ÷ records consumed. cpu_avg is the whole process tree, 100 = one core busy. rows/s per core = consumed ÷ (cpu_avg/100); rows/s per GB = consumed ÷ memory; extra rec/rec = records written to topics other than the input and output per record consumed (0 = no broker hop).
Resources (raw)
| product | cpu_avg % | cpu_p95 % | mem_avg MB | mem_max MB | samples | instrument |
|---|---|---|---|---|---|---|
| turbine 0.2.0a1 | 90.4 | 99.1 | 1,323.2 | 1,490.3 | 59 | /proc Δ, pss |
| turbine-torch 0.2.0a1 | 94.0 | 105.6 | 1,357.4 | 1,515.8 | 59 | /proc Δ, pss |
| faststream 0.7.1 | 99.9 | 101.1 | 410.6 | 415.9 | 59 | /proc Δ, pss |
| flink 2.2.1 | 808.5 | 839.0 | 1,186.4 | 1,214.8 | 58 | /proc Δ, pss |
| quix 3.23.6 | 86.4 | 88.4 | 284.5 | 284.8 | 59 | /proc Δ, pss |
| risingwave v2.8.4 | 105.7 | 166.3 | 864.1 | 886.8 | 58 | /proc Δ, mixed |
CPU is summed over the whole process tree (JVM, UDF servers, the RisingWave container's processes) as utime+stime deltas per 1 s sample — 100 = one core. Memory is PSS (shared pages counted once) where readable, RSS otherwise (instrument). Rows marked ⚠ legacy ps were measured with the old ps %CPU lifetime average and are not comparable: re-run them.
turbine — NATIVE operator — infer=ml.OnnxModel(...) on the subscription: the engine runs ONNX Runtime on every emitted batch in the worker's own thread (one session per partition), no Python in the loop; the handler only produces the four normalised feature columns (Arrow kernels). Same frozen network as everyone else (the ONNX export of the .pt, asserted against the card's probe rows at startup). Known surcharge: infer= appends the score to what the handler emits, so the four float32 feature columns are published too — four columns the other products do not write. Compare with turbine-torch, the same app with eager PyTorch in the handler. PROVISIONAL RATE — not swept.
turbine-torch — CONTROL LINE — Turbine with the model as eager PyTorch inside the handler (Arrow columns → DLPack, zero copy per column → one forward per batch), the iso-runtime row against the four other products. The gap to turbine is what Python-in-the-loop costs Turbine. PROVISIONAL RATE — not swept; the pre-2026-09-10 run drained 228k at 200k offered (working a backlog off, ~0.56 cores, ~770 MB) on the legacy instrument.
faststream — Knee ~57k/s at one core (75k offered drains the same 57k): 60k offered, 55.4k drained at ~0.99 cores (pegged) and a growing backlog, so the knee is around 55k. Leanest footprint of the five (~335 MB). The forward IS batched — the batch=True subscriber hands the handler a list — so what is being paid is the pure-Python loop around it and one publish per record.
flink — PROVISIONAL RATE — not swept. Pandas UDF in PROCESS mode: every batch is serialised to Arrow, crosses gRPC to a separate Python worker, and comes back — that hop is what "run a model in PyFlink" costs, and it is where the CPU goes: with a fully columnar UDF body (numpy stack → one forward → numpy back) the job still reads ~6.8 cores at 100k/s on the /proc instrument, the same as the old per-row transpose did. The model is not the cost; the Arrow/gRPC round trip and the JVM around it are. THREAD mode would remove the hop but forbids pandas UDFs and is GIL-bound.
quix — PROVISIONAL RATE — not swept. At 20k offered it drained 19.3k (slightly behind) at ~0.43 cores, so it is near its ceiling without being CPU-saturated: the cost is one PyTorch call PER RECORD, since quixstreams' StreamingDataFrame has no batching primitive. That is the framework's shape, not a bench handicap — every other product here batches the forward because its API hands over a batch.
risingwave — Swept to 150k (keep-up, ~1.06 cores; knee above). At 50k offered it kept up exactly (lag 0) at ~0.57 cores and ~865 MB, measured as CONTAINER + the host UDF server (see benches.py also_process_tree): RisingWave cannot run a model in-engine — embedded Python UDFs are disabled by default and stdlib-only — so the model lives in an Arrow Flight server and every batch crosses that boundary. One thing the batching does NOT buy: arrow-udf boxes every incoming cell to a Python object before the function sees it, whatever the mode, so the columnar wire format does not survive to the model.
Sanity — are we benching the same thing?
| product | selectivity (out/consumed) | expected | ✓ |
|---|---|---|---|
| turbine 0.2.0a1 | 1.0218 | 1.0000 | ✓ |
| turbine-torch 0.2.0a1 | 0.9309 | 1.0000 | ✓ |
| faststream 0.7.1 | 0.9996 | 1.0000 | ✓ |
| flink 2.2.1 | 1.0083 | 1.0000 | ✓ |
| quix 3.23.6 | 1.0069 | 1.0000 | ✓ |
| risingwave v2.8.4 | 0.9992 | 1.0000 | ✓ |
Verdict: ✓ matches theory — every row's selectivity is within 25% of the expected value (output ÷ consumed).
Host: AMD Ryzen 5 7600X 6-Core Processor · 6 cores / 12 threads · 31241 MB · kernel 6.12.107+deb13-amd64.
Redpanda: rpk version: v26.1.6. Redpanda, gendata and the product under test share this host.