Crash recovery
Does a product deliver what its strongest guarantee promises when its process dies mid-stream? That is exactly-once for every line in the main tables but RisingWave's, at-least-once — its Kafka sink has no transactions, so it is audited for what an exactly-once claim would owe, its duplicates shown. Each run prefills bench-input with a backlog stamped with synthetic event time and closed by a train of sentinel records a minute later, then a product line drains it from a wiped state dir while its whole process tree is SIGKILLed several times and restarted on the same state dir (a server, RisingWave: its container's process, restarted on the same disk) — a host crash with the local disk intact. Once the backlog is drained, an oracle recomputes every closed window pane from the input topic itself (no reference run to trust) and the output topic is classified against it pane by pane: as a transactional consumer sees it (read_committed), and with the records of aborted transactions (read_uncommitted).
Protocol — every line drains the same backlog, 12M records (5,000 per second of event time: a 30 s window closes every 150k records), through the same 10 kills, at the cores its performance runs use, next to a crash-free control on the same backlog, in two kill scenarios (the tables' kill at column):
- random — kill k is released at a seeded random phase of the commit cycle once its position is reached: a crash at a random instant, which lands inside a commit window a few times in a hundred — the scenario that reaches Turbine's state gap (a kill between a batch's transaction commit and its RocksDB write), which kills while publishing miss.
- publishing — kill k is released, once its position is reached, by records in the output topic that no commit covers yet — an open transaction holding records (LSO < high watermark) for a transactional line, the high watermark moving for the others: the crash lands while the product publishes a window close. An exactly-once line must abort what it had published (aborted records); an at-least-once one publishes it again (duplicates).
Last updated: 2026-11-10.
flowchart LR
gen["gendata prefill<br/>synthetic event time<br/>+ sentinels closing the last windows"] --> src[("bench-input<br/>1 partition")]
src --> app[["product line<br/>state dir on local disk"]]
kill{{"SIGKILL × k<br/>whole process tree,<br/>restart on the same state"}} -.-> app
app -->|"its delivery guarantee"| sink[("bench-out")]
src -.-> oracle["oracle<br/>every closed pane<br/>recomputed from the input"]
sink -->|"read_committed<br/>read_uncommitted"| audit{"audit"}
oracle --> audit
audit --> verdict["exact · duplicated · wrong · missing · extra<br/>+ records of aborted transactions"]windowed-highcard
Each line runs its windowed-highcard app — 500k keys (user_id), one partition.
Verdict
| product | kill at | kills | exact | duplicated | wrong · missing · extra | commit window hit | verdict |
|---|---|---|---|---|---|---|---|
| turbine-eos 0.3.5a1 | random | 10 / 10 (seeds A + B) | 10,365,076 | 0 | 0 · 0 · 0 | 0–20,000 aborted · 0–1 × state gap replayed | ✓ exactly-once |
| turbine-eos 0.3.5a1 | publishing | 10 / 10 (seeds A + B) | 10,364,144 | 0 | 0 · 0 · 0 | 46,210–53,452 aborted | ✓ exactly-once |
| arroyo-eos 0.15.0 | random | 3 / 10 (seeds A + B) | - | - | - | 3 × commit not restored | ✗ never recovered |
| arroyo-eos 0.15.0 | publishing | 10 / 10 (seeds A + B) | 10,370,013 | 0 | 0 · 0 · 0 | 6,913,298–6,996,902 aborted | ✓ exactly-once |
| flink-eos 2.2.1 | random | 9 / 10 (seeds A + B) | 10,368,746 | 0 | 0 · 0 · 0 | 4,652,066–5,642,575 aborted | ✓ exactly-once |
| flink-eos 2.2.1 | publishing | 9 / 10 (seeds A + B) | 10,367,390 | 0 | 0 · 0 · 0 | 75,075–261,892 aborted | ✓ exactly-once |
| quix-eos 3.23.6 | random | 10 / 10 (seeds A + B) | 10,370,137 | 0 | 0 · 0 · 0 | 165,348–208,811 aborted | ✓ exactly-once |
| quix-eos 3.23.6 | publishing | 10 / 10 (seeds A + B) | 10,356,718 | 0 | 0 · 0 · 0 | 2,306–3,181 aborted | ✓ exactly-once |
| risingwave v3.1.0 | random | 10 / 10 (seed A; B inconclusive) | 10,358,692 | 12,800 | 0 · 0 · 0 | - | ✗ not exactly-once |
| risingwave v3.1.0 | publishing | 10 / 10 (seeds A + B) | 10,026,210 | 338,030 | 0 · 0 · 0 | - | ✗ not exactly-once |
How to read this table
- One row per line and kill scenario (kill at): the runs of its kill seeds (A, B — two trials, each on its own VM: Run configuration) merged, each figure the worse seed's — the fewest exact panes, the most duplicated / wrong / missing / extra, the worse verdict. A seed whose run is inconclusive is set aside while the other concludes (kills says so). Every seed's verdict is in the verdict's tooltip.
- kills — done / planned, the fewer of the seeds'. Kill k waits for the committed offset to pass k/(planned+1) of the backlog and to have moved since the last restart (never kill a restore), then a seeded jitter (a random phase of the commit cycle) or, for kills while publishing, the output topic's next move; a line that drains the backlog first gets fewer kills.
- exact … extra — every pane the oracle expects, read back under
read_committed(what a transactional consumer downstream sees): exact = once with the right count and average; duplicated = more than once, every copy right; wrong = a copy with a wrong value; missing = never emitted; extra = a pane of no closed window of the input. - commit window hit — the proof that kills landed inside the commit path, without which a clean run proves nothing (
ok but inconclusive), from the lowest seed's to the highest (neither better nor worse; each seed's in the tooltip). aborted = recordsread_uncommittedsees andread_committeddoes not, the output of the transactions a kill aborted; state gap = a crash the product itself reported at restart, read from its log (TurbineEOS state gap: a kill between a batch's transaction commit and its RocksDB write, the batch replayed at the restart; Arroyo commit not restored: a kill between a completed checkpoint and its sink's commit, logged once per sink subtask). - verdict — hover it for the full line, with the input events lost by a consumer that keeps each pane's last copy (an upsert table) when panes are wrong or missing. never recovered: after a restart the product made no committed progress again (nothing new reached a
read_committedconsumer), the backlog left undrained — no output to audit, the stall is the verdict. - quix-eos commits every 5 s in one Kafka transaction: the panes published since the last commit, its state's changelog writes and the input offsets. A kill at a random instant aborts everything published since that commit; one while publishing fires as a close starts, so it aborts fewer records. Restarted on the same state dir, it restores its RocksDB from the changelog's committed tail.
- risingwave has no exactly-once Kafka sink — at-least-once by its documentation, still no transaction in the release benched; only its Iceberg sink has an exactly-once mode — so the one delivery it has stands with the exactly-once claims: what its users get through a crash. Its restarted server resumes from its last checkpoint (1 s) and publishes again what it had published since. It publishes a window close right at a checkpoint, so a crash at a random instant lands in that span only now and then (one kill run of two on the reference machine), one while it publishes every time.
- arroyo-eos commits a checkpoint's Kafka transactions once the checkpoint has completed. Killed in between, its restored job asks the restarted sink to commit a transaction whose producer died with the process — "Restoring from commit phase not yet implemented", its log says (commit not restored) — and waits for that commit forever: no checkpoint follows, a restart included, so nothing it publishes again reaches a
read_committedconsumer (never recovered), and the pending transaction is aborted by the broker at its timeout.
Recovery
| product | kill at | kills | records / life | recovery p50 | recovery max | drain | crash-free | cost / kill |
|---|---|---|---|---|---|---|---|---|
| turbine-eos 0.3.5a1 | random | 10 (seeds A + B) | 1.08M–1.1M | 1.5 s | 5.0 s | 165.2 s | 137.6–143.0 s | 2.2 s |
| turbine-eos 0.3.5a1 | publishing | 10 (seeds A + B) | 1.09M | 3.5 s | 5.7 s | 188.5 s | 137.6–143.0 s | 4.6 s |
| arroyo-eos 0.15.0 | random | 3 (seeds A + B) | 1.29M–1.34M | 3.1 s | 3.9 s | never | 25.7–27.4 s | - |
| arroyo-eos 0.15.0 | publishing | 10 (seeds A + B) | 1.1M–1.22M | 3.1 s | 4.0 s | 48.5 s | 25.7–27.4 s | 2.1 s |
| flink-eos 2.2.1 | random | 9 (seeds A + B) | 902k–1.28M | 13.9 s | 22.9 s | 216.6 s | 101.3–105.2 s | 11.5 s |
| flink-eos 2.2.1 | publishing | 9 (seeds A + B) | 1.11M–1.4M | 14.0 s | 18.9 s | 169.5 s | 104.9–122.7 s | 6.0 s |
| quix-eos 3.23.6 | random | 10 (seeds A + B) | 1.06M–1.08M | 46.9 s | 53.2 s | 1141.9 s | 561.4–565.1 s | 58.0 s |
| quix-eos 3.23.6 | publishing | 10 (seeds A + B) | 1.08M–1.12M | 47.0 s | 52.8 s | 1162.0 s | 561.4–565.1 s | 59.7 s |
| risingwave v3.1.0 | random | 10 (seeds A + B) | 1.11M–1.12M | 2.4 s | 6.1 s | 139.0 s | 115.6–124.4 s | 1.5 s |
| risingwave v3.1.0 | publishing | 10 (seeds A + B) | 1M–1.08M | 2.9 s | 7.8 s | 138.9 s | 115.6–124.4 s | 1.9 s |
How to read this table
- One row per line and kill scenario, its seeds merged as in the verdict table — every seed's kill run counts here, whatever its verdict: each time is the worse seed's, records / life and crash-free the span over the seeds (neither better nor worse). Each kill run is timed next to the crash-free control of its VM (Crash-free controls). Colours: each time against the best (lowest) of its column — green under 2×, yellow from 2×, orange from 5×, red from 10× (the ratio on hover); never in purple (off the scale),
≈ 0in green; crash-free, a span, has none. - records / life — median committed progress between a restart and the next kill.
- recovery — per restart: process start → the committed offset back past its value at the kill (start, state restore, replay, then one commit cycle). It carries the line's commit cadence whatever its restore speed: a Kafka transaction a second for Turbine exactly-once, 5 s for Turbine at-least-once, a 10 s checkpoint for Flink, a 1 s one for RisingWave (whose restart is a whole server's) and for Arroyo (its default is 10 s: at that cadence it drains the backlog before a second kill can follow a restart). Quix's is its consumer group's: it subscribes, so a restarted consumer gets its partition only once the killed one's session has expired (librdkafka's default
session.timeout.ms, 45 s; its state restored in about a second after that), then a 5 s commit. - drain — first start → the committed offset at the end of the backlog, kills and restarts included; crash-free — the same line's
--kills 0run on the same backlog, cores and VM; cost / kill — (drain − crash-free) ÷ kills: the time each crash costs end to end (restart, restore, the work redone since the last commit), the worse seed's own — its drain and crash-free may be other than the row's worse drain and the span's ends.≈ 0when the two drains differ by less than 5%, about a single run's noise: the kills cost less than two crash-free runs differ (the figure is in its tooltip).
Reference lines: at-least-once witnesses
The lines a product runs below its strongest one, through the same protocol: the reference for what exactly-once costs. The at-least-once witnesses write no transaction: after a kill they redo and re-emit the work since their last commit (duplicates), and their recovery carries their commit cadence (5 s for Turbine, a 10 s checkpoint for Flink; Quix's group session, 45 s, like its exactly-once line). The columns read as above.
| product | kill at | kills | exact | duplicated | wrong · missing · extra | commit window hit | verdict |
|---|---|---|---|---|---|---|---|
| turbine 0.3.5a1 | random | 10 / 10 (seeds A + B) | 9,273,192 | 1,096,957 | 0 · 0 · 0 | - | ✗ not exactly-once |
| turbine 0.3.5a1 | publishing | 10 / 10 (seeds A + B) | 9,446,611 | 916,866 | 0 · 0 · 0 | - | ✗ not exactly-once |
| arroyo 0.15.0 | random | 10 / 10 (seeds A + B) | 6,160,245 | 4,216,325 | 0 · 0 · 0 | - | ✗ not exactly-once |
| arroyo 0.15.0 | publishing | 10 / 10 (seeds A + B) | 4,850,713 | 5,515,511 | 0 · 0 · 0 | - | ✗ not exactly-once |
| flink 2.2.1 | random | 9 / 10 (seeds A + B) | 6,154,984 | 4,200,950 | 20,577 · 0 · 0 | - | ✗ not exactly-once |
| flink 2.2.1 | publishing | 9 / 10 (seeds A + B) | 10,298,159 | 49,852 | 18,153 · 0 · 0 | - | ✗ not exactly-once |
| quix 3.23.6 | random | 10 / 10 (seed B; A inconclusive) | 10,346,565 | 23,183 | 0 · 0 · 0 | - | ✗ not exactly-once |
| quix 3.23.6 | publishing | 10 / 10 (seeds A + B) | 10,356,472 | 7,582 | 0 · 0 · 0 | - | ✗ not exactly-once |
| product | kill at | kills | records / life | recovery p50 | recovery max | drain | crash-free | cost / kill |
|---|---|---|---|---|---|---|---|---|
| turbine 0.3.5a1 | random | 10 (seeds A + B) | 1.09M–1.13M | 5.3 s | 5.4 s | 149.5 s | 130.3–135.3 s | 1.9 s |
| turbine 0.3.5a1 | publishing | 10 (seeds A + B) | 1.09M–1.12M | 5.3 s | 5.3 s | 149.8 s | 130.3–135.3 s | 1.9 s |
| arroyo 0.15.0 | random | 10 (seeds A + B) | 1.12M–1.18M | 1.9 s | 2.8 s | 42.5 s | 21.1–22.3 s | 2.0 s |
| arroyo 0.15.0 | publishing | 10 (seeds A + B) | 1.12M–1.16M | 2.4 s | 3.0 s | 39.4 s | 21.1–22.3 s | 1.7 s |
| flink 2.2.1 | random | 9 (seeds A + B) | 1.1M–1.17M | 15.0 s | 22.0 s | 211.0 s | 99.2–105.1 s | 11.2 s |
| flink 2.2.1 | publishing | 9 (seeds A + B) | 1.13M–1.39M | 12.6 s | 20.1 s | 171.2 s | 106.4–117.2 s | 6.5 s |
| quix 3.23.6 | random | 10 (seeds A + B) | 1.06M–1.11M | 45.6 s | 47.8 s | 1052.7 s | 476.6–484.8 s | 56.8 s |
| quix 3.23.6 | publishing | 10 (seeds A + B) | 1.07M–1.11M | 45.5 s | 47.6 s | 1073.0 s | 476.6–484.8 s | 59.6 s |
Crash-free controls
Each kill run's crash-free control: the same line on the same backlog, cores and VM, without kills (--kills 0), whose drain is the recovery table's crash-free; one row per line and the scenarios it controls, its seeds merged like the kill runs'. Every one must audit exact, or the oracle or the app is wrong and the kill runs of its VM (control of) prove nothing.
| product | control of | exact | duplicated | wrong · missing · extra | verdict |
|---|---|---|---|---|---|
| turbine 0.3.5a1 | random + publishing (seeds A + B) | 10,372,835 | 0 | 0 · 0 · 0 | ✓ control |
| turbine-eos 0.3.5a1 | random + publishing (seeds A + B) | 10,370,640 | 0 | 0 · 0 · 0 | ✓ control |
| arroyo 0.15.0 | random + publishing (seeds A + B) | 10,368,604 | 0 | 0 · 0 · 0 | ✓ control |
| arroyo-eos 0.15.0 | random + publishing (seeds A + B) | 10,367,787 | 0 | 0 · 0 · 0 | ✓ control |
| flink 2.2.1 | random (seeds A + B) | 10,358,330 | 0 | 0 · 0 · 0 | ✓ control |
| flink 2.2.1 | publishing (seeds A + B) | 10,363,741 | 0 | 0 · 0 · 0 | ✓ control |
| flink-eos 2.2.1 | random (seeds A + B) | 10,369,504 | 0 | 0 · 0 · 0 | ✓ control |
| flink-eos 2.2.1 | publishing (seeds A + B) | 10,371,495 | 0 | 0 · 0 · 0 | ✓ control |
| quix 3.23.6 | random + publishing (seeds A + B) | 10,368,386 | 0 | 0 · 0 · 0 | ✓ control |
| quix-eos 3.23.6 | random + publishing (seeds A + B) | 10,363,201 | 0 | 0 · 0 · 0 | ✓ control |
| risingwave v3.1.0 | random + publishing (seeds A + B) | 10,370,245 | 0 | 0 · 0 · 0 | ✓ control |
Kill timelines
One row per life — kill k is the life after the k-th kill: its recovery (red: restart → the committed offset back past its value at the kill), then its processing up to the next kill (blue), with the records it committed and what the product reported at that restart. Time in min:s since the first start.
turbine-eos — random, 12M records, 10 kills, 3 cores, seed A (2026-11-10 11:07)
---
displayMode: compact
config:
gantt:
leftPadding: 237
useWidth: 1000
themeCSS: ".taskText, .taskTextOutsideRight, .taskTextOutsideLeft { display: none; } .task { stroke: none; } .sectionTitle, .grid .tick text { font-family: var(--md-mermaid-font-family); fill: var(--md-mermaid-label-fg-color) !important; } .sectionTitle { font-size: 14px; } .grid .tick text { font-size: 13px; } .grid .tick line { stroke: var(--md-default-fg-color--lighter); stroke-opacity: 0.5; }"
---
gantt
dateFormat x
axisFormat %M:%S
todayMarker off
tickInterval 15second
section start · 1st commit 0.4 s · +1.16M
first commit :0, 411
processing :active, 411, 13443
section kill 1 · 1.2 s · +1.03M
recovery :crit, 13443, 14662
processing :active, 14662, 26424
section kill 2 · 4.0 s · +1.11M
recovery :crit, 26425, 30462
processing :active, 30462, 43861
section kill 3 · 1.4 s · +1.12M
recovery :crit, 43861, 45277
processing :active, 45277, 58312
section kill 4 · 1.4 s · +1.1M
recovery :crit, 58313, 59735
processing :active, 59735, 73091
section kill 5 · 1.3 s · +1.08M
recovery :crit, 73091, 74403
processing :active, 74403, 87010
section kill 6 · 1.6 s · +1.13M
recovery :crit, 87011, 88630
processing :active, 88630, 101517
section kill 7 · 1.4 s · +1.01M
recovery :crit, 101517, 102928
processing :active, 102928, 115011
section kill 8 · 3.8 s · +1.1M
recovery :crit, 115011, 118841
processing :active, 118841, 132184
section kill 9 · 5.0 s · +1.08M
recovery :crit, 132185, 137221
processing :active, 137221, 150165
section kill 10 · 1.5 s · to the end
recovery :crit, 150165, 151682
processing :active, 151682, 165169turbine-eos — random, 12M records, 10 kills, 3 cores, seed B (2026-11-10 10:14)
---
displayMode: compact
config:
gantt:
leftPadding: 333
useWidth: 1000
themeCSS: ".taskText, .taskTextOutsideRight, .taskTextOutsideLeft { display: none; } .task { stroke: none; } .sectionTitle, .grid .tick text { font-family: var(--md-mermaid-font-family); fill: var(--md-mermaid-label-fg-color) !important; } .sectionTitle { font-size: 14px; } .grid .tick text { font-size: 13px; } .grid .tick line { stroke: var(--md-default-fg-color--lighter); stroke-opacity: 0.5; }"
---
gantt
dateFormat x
axisFormat %M:%S
todayMarker off
tickInterval 15second
section start · 1st commit 0.3 s · +1.12M
first commit :0, 302
processing :active, 302, 12671
section kill 1 · 4.2 s · +1.13M
recovery :crit, 12671, 16911
processing :active, 16911, 29711
section kill 2 · 1.2 s · +1.04M
recovery :crit, 29712, 30925
processing :active, 30925, 42578
section kill 3 · 4.1 s · +1.09M
recovery :crit, 42578, 46714
processing :active, 46714, 59422
section kill 4 · 1.0 s · +1.16M
recovery :crit, 59422, 60448
processing :active, 60448, 73233
section kill 5 · 1.9 s · +1.07M
recovery :crit, 73234, 75157
processing :active, 75157, 87254
section kill 6 · 1.0 s · +1.07M
recovery :crit, 87255, 88270
processing :active, 88270, 100264
section kill 7 · 4.8 s · +1.07M
recovery :crit, 100265, 105107
processing :active, 105107, 118333
section kill 8 · 1.6 s · +1.08M · state gap replayed 10k
recovery :crit, 118334, 119955
processing :active, 119955, 131967
section kill 9 · 1.2 s · +1.13M
recovery :crit, 131968, 133181
processing :active, 133181, 146148
section kill 10 · 1.4 s · to the end
recovery :crit, 146148, 147574
processing :active, 147574, 159754turbine-eos — publishing, 12M records, 10 kills, 3 cores, seed A (2026-11-10 11:17)
---
displayMode: compact
config:
gantt:
leftPadding: 230
useWidth: 1000
themeCSS: ".taskText, .taskTextOutsideRight, .taskTextOutsideLeft { display: none; } .task { stroke: none; } .sectionTitle, .grid .tick text { font-family: var(--md-mermaid-font-family); fill: var(--md-mermaid-label-fg-color) !important; } .sectionTitle { font-size: 14px; } .grid .tick text { font-size: 13px; } .grid .tick line { stroke: var(--md-default-fg-color--lighter); stroke-opacity: 0.5; }"
---
gantt
dateFormat x
axisFormat %M:%S
todayMarker off
tickInterval 20second
section start · 1st commit 0.4 s · +1.1M
first commit :0, 410
processing :active, 410, 15528
section kill 1 · 5.6 s · +1.15M
recovery :crit, 15529, 21082
processing :active, 21082, 36531
section kill 2 · 1.8 s · +1.05M
recovery :crit, 36532, 38364
processing :active, 38364, 51875
section kill 3 · 5.2 s · +1.07M
recovery :crit, 51876, 57031
processing :active, 57031, 71503
section kill 4 · 5.2 s · +1.09M
recovery :crit, 71504, 76747
processing :active, 76747, 91229
section kill 5 · 1.7 s · +1.14M
recovery :crit, 91230, 92957
processing :active, 92957, 108587
section kill 6 · 1.8 s · +1.05M
recovery :crit, 108588, 110399
processing :active, 110399, 122730
section kill 7 · 5.4 s · +1.09M
recovery :crit, 122730, 128167
processing :active, 128167, 140556
section kill 8 · 1.6 s · +1.09M
recovery :crit, 140557, 142174
processing :active, 142174, 154959
section kill 9 · 5.7 s · +1.12M
recovery :crit, 154959, 160701
processing :active, 160701, 173845
section kill 10 · 1.8 s · to the end
recovery :crit, 173846, 175660
processing :active, 175660, 188531turbine-eos — publishing, 12M records, 10 kills, 3 cores, seed B (2026-11-10 10:24)
---
displayMode: compact
config:
gantt:
leftPadding: 230
useWidth: 1000
themeCSS: ".taskText, .taskTextOutsideRight, .taskTextOutsideLeft { display: none; } .task { stroke: none; } .sectionTitle, .grid .tick text { font-family: var(--md-mermaid-font-family); fill: var(--md-mermaid-label-fg-color) !important; } .sectionTitle { font-size: 14px; } .grid .tick text { font-size: 13px; } .grid .tick line { stroke: var(--md-default-fg-color--lighter); stroke-opacity: 0.5; }"
---
gantt
dateFormat x
axisFormat %M:%S
todayMarker off
tickInterval 15second
section start · 1st commit 0.3 s · +1.1M
first commit :0, 309
processing :active, 309, 12207
section kill 1 · 4.8 s · +1.09M
recovery :crit, 12208, 17047
processing :active, 17047, 29698
section kill 2 · 1.5 s · +1.09M
recovery :crit, 29699, 31238
processing :active, 31238, 43758
section kill 3 · 1.5 s · +1.11M
recovery :crit, 43759, 45275
processing :active, 45275, 57982
section kill 4 · 1.8 s · +1.07M
recovery :crit, 57983, 59792
processing :active, 59792, 72043
section kill 5 · 1.9 s · +1.09M
recovery :crit, 72044, 73963
processing :active, 73963, 86622
section kill 6 · 1.7 s · +1.14M
recovery :crit, 86622, 88341
processing :active, 88341, 101290
section kill 7 · 2.2 s · +1.05M
recovery :crit, 101290, 103521
processing :active, 103521, 115455
section kill 8 · 5.1 s · +1.08M
recovery :crit, 115456, 120590
processing :active, 120590, 133300
section kill 9 · 1.7 s · +1.1M
recovery :crit, 133300, 135018
processing :active, 135018, 148151
section kill 10 · 1.7 s · to the end
recovery :crit, 148152, 149870
processing :active, 149870, 162867arroyo-eos — random, 12M records, 3 kills, 3 cores, seed A (2026-10-23 17:02)
---
displayMode: compact
config:
gantt:
leftPadding: 237
useWidth: 1000
themeCSS: ".taskText, .taskTextOutsideRight, .taskTextOutsideLeft { display: none; } .task { stroke: none; } .sectionTitle, .grid .tick text { font-family: var(--md-mermaid-font-family); fill: var(--md-mermaid-label-fg-color) !important; } .sectionTitle { font-size: 14px; } .grid .tick text { font-size: 13px; } .grid .tick line { stroke: var(--md-default-fg-color--lighter); stroke-opacity: 0.5; }"
---
gantt
dateFormat x
axisFormat %M:%S
todayMarker off
tickInterval 15second
section start · 1st commit 2.5 s · +1.49M
first commit :0, 2533
processing :active, 2533, 4714
section kill 1 · 2.3 s · +1.15M
recovery :crit, 4714, 6978
processing :active, 6978, 7130
section kill 2 · 3.9 s · +1.29M
recovery :crit, 7131, 11001
processing :active, 11001, 12926
section kill 3 · never recovered
no commit :crit, 12927, 132944arroyo-eos — random, 12M records, 4 kills, 3 cores, seed B (2026-10-23 17:05)
---
displayMode: compact
config:
gantt:
leftPadding: 237
useWidth: 1000
themeCSS: ".taskText, .taskTextOutsideRight, .taskTextOutsideLeft { display: none; } .task { stroke: none; } .sectionTitle, .grid .tick text { font-family: var(--md-mermaid-font-family); fill: var(--md-mermaid-label-fg-color) !important; } .sectionTitle { font-size: 14px; } .grid .tick text { font-size: 13px; } .grid .tick line { stroke: var(--md-default-fg-color--lighter); stroke-opacity: 0.5; }"
---
gantt
dateFormat x
axisFormat %M:%S
todayMarker off
tickInterval 15second
section start · 1st commit 2.3 s · +1.42M
first commit :0, 2318
processing :active, 2318, 4006
section kill 1 · 2.0 s · +1.28M
recovery :crit, 4007, 6034
processing :active, 6034, 8972
section kill 2 · 1.8 s · +792k
recovery :crit, 8972, 10794
processing :active, 10794, 11065
section kill 3 · 1.9 s · +1.4M
recovery :crit, 11065, 12980
processing :active, 12980, 16609
section kill 4 · never recovered
no commit :crit, 16609, 136690arroyo-eos — publishing, 12M records, 10 kills, 3 cores, seed A (2026-10-23 17:10)
---
displayMode: compact
config:
gantt:
leftPadding: 237
useWidth: 1000
themeCSS: ".taskText, .taskTextOutsideRight, .taskTextOutsideLeft { display: none; } .task { stroke: none; } .sectionTitle, .grid .tick text { font-family: var(--md-mermaid-font-family); fill: var(--md-mermaid-label-fg-color) !important; } .sectionTitle { font-size: 14px; } .grid .tick text { font-size: 13px; } .grid .tick line { stroke: var(--md-default-fg-color--lighter); stroke-opacity: 0.5; }"
---
gantt
dateFormat x
axisFormat %M:%S
todayMarker off
tickInterval 5second
section start · 1st commit 2.5 s · +1.47M
first commit :0, 2523
processing :active, 2523, 4083
section kill 1 · 3.6 s · +980k
recovery :crit, 4083, 7717
processing :active, 7717, 7852
section kill 2 · 3.6 s · +1.31M
recovery :crit, 7853, 11484
processing :active, 11484, 13135
section kill 3 · 1.8 s · +724k
recovery :crit, 13136, 14962
processing :active, 14962, 16610
section kill 4 · 3.4 s · +1.48M
recovery :crit, 16611, 20056
processing :active, 20056, 21613
section kill 5 · 1.8 s · +736k
recovery :crit, 21613, 23438
processing :active, 23438, 25092
section kill 6 · 1.8 s · +1.12M
recovery :crit, 25093, 26924
processing :active, 26924, 29089
section kill 7 · 3.8 s · +1.55M
recovery :crit, 29090, 32848
processing :active, 32848, 34605
section kill 8 · 2.0 s · +811k
recovery :crit, 34606, 36643
processing :active, 36643, 38297
section kill 9 · 2.2 s · +1.38M
recovery :crit, 38298, 40526
processing :active, 40526, 42684
section kill 10 · 2.2 s · to the end
recovery :crit, 42685, 44906
processing :active, 44906, 46919arroyo-eos — publishing, 12M records, 10 kills, 3 cores, seed B (2026-10-23 17:13)
---
displayMode: compact
config:
gantt:
leftPadding: 237
useWidth: 1000
themeCSS: ".taskText, .taskTextOutsideRight, .taskTextOutsideLeft { display: none; } .task { stroke: none; } .sectionTitle, .grid .tick text { font-family: var(--md-mermaid-font-family); fill: var(--md-mermaid-label-fg-color) !important; } .sectionTitle { font-size: 14px; } .grid .tick text { font-size: 13px; } .grid .tick line { stroke: var(--md-default-fg-color--lighter); stroke-opacity: 0.5; }"
---
gantt
dateFormat x
axisFormat %M:%S
todayMarker off
tickInterval 5second
section start · 1st commit 2.1 s · +1.27M
first commit :0, 2120
processing :active, 2120, 3773
section kill 1 · 1.8 s · +1.02M
recovery :crit, 3773, 5594
processing :active, 5594, 7751
section kill 2 · 4.0 s · +1.56M
recovery :crit, 7752, 11797
processing :active, 11797, 13546
section kill 3 · 2.2 s · +1.03M
recovery :crit, 13547, 15786
processing :active, 15786, 17432
section kill 4 · 3.6 s · +979k
recovery :crit, 17433, 21060
processing :active, 21060, 21197
section kill 5 · 3.6 s · +1.29M
recovery :crit, 21198, 24830
processing :active, 24830, 26483
section kill 6 · 3.1 s · +549k
recovery :crit, 26484, 29615
processing :active, 29615, 29758
section kill 7 · 2.0 s · +1.13M
recovery :crit, 29759, 31796
processing :active, 31796, 33951
section kill 8 · 2.0 s · +1.07M
recovery :crit, 33952, 35981
processing :active, 35981, 38135
section kill 9 · 3.5 s · +1.31M
recovery :crit, 38136, 41663
processing :active, 41663, 43317
section kill 10 · 3.1 s · to the end
recovery :crit, 43317, 46439
processing :active, 46439, 48452flink-eos — random, 12M records, 9 kills, 3 cores, seed A (2026-10-20 09:58)
---
displayMode: compact
config:
gantt:
leftPadding: 243
useWidth: 1000
themeCSS: ".taskText, .taskTextOutsideRight, .taskTextOutsideLeft { display: none; } .task { stroke: none; } .sectionTitle, .grid .tick text { font-family: var(--md-mermaid-font-family); fill: var(--md-mermaid-label-fg-color) !important; } .sectionTitle { font-size: 14px; } .grid .tick text { font-size: 13px; } .grid .tick line { stroke: var(--md-default-fg-color--lighter); stroke-opacity: 0.5; }"
---
gantt
dateFormat x
axisFormat %M:%S
todayMarker off
tickInterval 20second
section start · 1st commit 15.5 s · +1.42M
first commit :0, 15526
processing :active, 15526, 22048
section kill 1 · 17.1 s · +1.82M
recovery :crit, 22049, 39146
processing :active, 39146, 39525
section kill 2 · 13.9 s · +1.28M
recovery :crit, 39525, 53458
processing :active, 53458, 56338
section kill 3 · 12.2 s · +654k
recovery :crit, 56339, 68546
processing :active, 68546, 70907
section kill 4 · 12.3 s · +968k
recovery :crit, 70907, 83212
processing :active, 83212, 90705
section kill 5 · 11.8 s · +818k
recovery :crit, 90706, 102540
processing :active, 102540, 109435
section kill 6 · 14.8 s · +2.5M
recovery :crit, 109436, 124238
processing :active, 124238, 144753
section kill 7 · 15.7 s · +1.32M
recovery :crit, 144753, 160467
processing :active, 160467, 161465
section kill 8 · 7.9 s · +222k
recovery :crit, 161466, 169361
processing :active, 169361, 173711
section kill 9 · 22.9 s · to the end
recovery :crit, 173711, 196594
processing :active, 196594, 196594flink-eos — random, 12M records, 10 kills, 3 cores, seed B (2026-10-20 09:59)
---
displayMode: compact
config:
gantt:
leftPadding: 243
useWidth: 1000
themeCSS: ".taskText, .taskTextOutsideRight, .taskTextOutsideLeft { display: none; } .task { stroke: none; } .sectionTitle, .grid .tick text { font-family: var(--md-mermaid-font-family); fill: var(--md-mermaid-label-fg-color) !important; } .sectionTitle { font-size: 14px; } .grid .tick text { font-size: 13px; } .grid .tick line { stroke: var(--md-default-fg-color--lighter); stroke-opacity: 0.5; }"
---
gantt
dateFormat x
axisFormat %M:%S
todayMarker off
tickInterval 20second
section start · 1st commit 10.4 s · +1.76M
first commit :0, 10408
processing :active, 10408, 20786
section kill 1 · 11.2 s · +822k
recovery :crit, 20786, 32013
processing :active, 32013, 39103
section kill 2 · 12.1 s · +1.46M
recovery :crit, 39103, 51224
processing :active, 51224, 59846
section kill 3 · 8.7 s · +1.6M
recovery :crit, 59847, 68515
processing :active, 68515, 82725
section kill 4 · 10.4 s · +316k
recovery :crit, 82725, 93111
processing :active, 93111, 99955
section kill 5 · 11.9 s · +775k
recovery :crit, 99956, 111877
processing :active, 111877, 119936
section kill 6 · 17.1 s · +1.66M
recovery :crit, 119936, 137004
processing :active, 137004, 141663
section kill 7 · 10.9 s · +764k
recovery :crit, 141664, 152569
processing :active, 152569, 157683
section kill 8 · 12.2 s · +871k
recovery :crit, 157684, 169920
processing :active, 169920, 170241
section kill 9 · 14.4 s · +933k
recovery :crit, 170241, 184686
processing :active, 184686, 189139
section kill 10 · 12.6 s · to the end
recovery :crit, 189139, 201743
processing :active, 201743, 216634flink-eos — publishing, 12M records, 9 kills, 3 cores, seed A (2026-10-20 22:43)
---
displayMode: compact
config:
gantt:
leftPadding: 243
useWidth: 1000
themeCSS: ".taskText, .taskTextOutsideRight, .taskTextOutsideLeft { display: none; } .task { stroke: none; } .sectionTitle, .grid .tick text { font-family: var(--md-mermaid-font-family); fill: var(--md-mermaid-label-fg-color) !important; } .sectionTitle { font-size: 14px; } .grid .tick text { font-size: 13px; } .grid .tick line { stroke: var(--md-default-fg-color--lighter); stroke-opacity: 0.5; }"
---
gantt
dateFormat x
axisFormat %M:%S
todayMarker off
tickInterval 15second
section start · 1st commit 20.1 s · +1.89M
first commit :0, 20069
processing :active, 20069, 20594
section kill 1 · 8.5 s · +316k
recovery :crit, 20595, 29113
processing :active, 29113, 29476
section kill 2 · 13.7 s · +2.32M
recovery :crit, 29477, 43193
processing :active, 43193, 51685
section kill 3 · 12.6 s · +528k
recovery :crit, 51686, 64313
processing :active, 64313, 64878
section kill 4 · 8.8 s · +2.13M
recovery :crit, 64878, 73712
processing :active, 73712, 84122
section kill 5 · 16.9 s · +1.59M
recovery :crit, 84122, 101063
processing :active, 101063, 101542
section kill 6 · 15.9 s · +1.4M
recovery :crit, 101543, 117479
processing :active, 117479, 117886
section kill 7 · 13.0 s · +707k
recovery :crit, 117886, 130926
processing :active, 130926, 131062
section kill 8 · 9.1 s · +506k
recovery :crit, 131063, 140148
processing :active, 140148, 140282
section kill 9 · 18.9 s · to the end
recovery :crit, 140283, 159154
processing :active, 159154, 159155flink-eos — publishing, 12M records, 10 kills, 3 cores, seed B (2026-10-20 22:39)
---
displayMode: compact
config:
gantt:
leftPadding: 237
useWidth: 1000
themeCSS: ".taskText, .taskTextOutsideRight, .taskTextOutsideLeft { display: none; } .task { stroke: none; } .sectionTitle, .grid .tick text { font-family: var(--md-mermaid-font-family); fill: var(--md-mermaid-label-fg-color) !important; } .sectionTitle { font-size: 14px; } .grid .tick text { font-size: 13px; } .grid .tick line { stroke: var(--md-default-fg-color--lighter); stroke-opacity: 0.5; }"
---
gantt
dateFormat x
axisFormat %M:%S
todayMarker off
tickInterval 15second
section start · 1st commit 9.8 s · +2.34M
first commit :0, 9832
processing :active, 9832, 20050
section kill 1 · 15.7 s · +1.23M
recovery :crit, 20050, 35754
processing :active, 35754, 35891
section kill 2 · 8.1 s · +254k
recovery :crit, 35891, 44025
processing :active, 44025, 44160
section kill 3 · 9.9 s · +1.97M
recovery :crit, 44161, 54063
processing :active, 54063, 64929
section kill 4 · 10.3 s · +462k
recovery :crit, 64930, 75263
processing :active, 75263, 75715
section kill 5 · 16.6 s · +998k
recovery :crit, 75716, 92295
processing :active, 92295, 92432
section kill 6 · 15.0 s · +1.09M
recovery :crit, 92433, 107478
processing :active, 107478, 108007
section kill 7 · 18.7 s · +1.62M
recovery :crit, 108007, 126688
processing :active, 126688, 127095
section kill 8 · 13.0 s · +1.13M
recovery :crit, 127096, 140146
processing :active, 140146, 140282
section kill 9 · 10.3 s · +371k
recovery :crit, 140282, 150588
processing :active, 150588, 151325
section kill 10 · 18.2 s · to the end
recovery :crit, 151325, 169498
processing :active, 169498, 169498quix-eos — random, 12M records, 10 kills, 1 core, seed A (2026-11-07 12:37)
---
displayMode: compact
config:
gantt:
leftPadding: 243
useWidth: 1000
themeCSS: ".taskText, .taskTextOutsideRight, .taskTextOutsideLeft { display: none; } .task { stroke: none; } .sectionTitle, .grid .tick text { font-family: var(--md-mermaid-font-family); fill: var(--md-mermaid-label-fg-color) !important; } .sectionTitle { font-size: 14px; } .grid .tick text { font-size: 13px; } .grid .tick line { stroke: var(--md-default-fg-color--lighter); stroke-opacity: 0.5; }"
---
gantt
dateFormat x
axisFormat %M:%S
todayMarker off
tickInterval 120second
section start · 1st commit 10.2 s · +1.24M
first commit :0, 10159
processing :active, 10159, 58937
section kill 1 · 45.6 s · +1.04M
recovery :crit, 58938, 104512
processing :active, 104512, 160815
section kill 2 · 45.9 s · +1.06M
recovery :crit, 160816, 206730
processing :active, 206730, 263496
section kill 3 · 46.2 s · +1.05M
recovery :crit, 263497, 309715
processing :active, 309715, 375638
section kill 4 · 47.5 s · +1.2M
recovery :crit, 375639, 423154
processing :active, 423154, 483428
section kill 5 · 51.9 s · +1.05M
recovery :crit, 483428, 535285
processing :active, 535285, 588816
section kill 6 · 51.3 s · +1.05M
recovery :crit, 588817, 640082
processing :active, 640082, 702186
section kill 7 · 46.9 s · +1.05M
recovery :crit, 702186, 749073
processing :active, 749073, 804529
section kill 8 · 52.6 s · +1.17M
recovery :crit, 804529, 857118
processing :active, 857118, 928782
section kill 9 · 47.0 s · +1.08M
recovery :crit, 928783, 975765
processing :active, 975765, 1031900
section kill 10 · 46.0 s · to the end
recovery :crit, 1031901, 1077928
processing :active, 1077928, 1141696quix-eos — random, 12M records, 10 kills, 1 core, seed B (2026-11-07 12:37)
---
displayMode: compact
config:
gantt:
leftPadding: 237
useWidth: 1000
themeCSS: ".taskText, .taskTextOutsideRight, .taskTextOutsideLeft { display: none; } .task { stroke: none; } .sectionTitle, .grid .tick text { font-family: var(--md-mermaid-font-family); fill: var(--md-mermaid-label-fg-color) !important; } .sectionTitle { font-size: 14px; } .grid .tick text { font-size: 13px; } .grid .tick line { stroke: var(--md-default-fg-color--lighter); stroke-opacity: 0.5; }"
---
gantt
dateFormat x
axisFormat %M:%S
todayMarker off
tickInterval 120second
section start · 1st commit 8.4 s · +1.12M
first commit :0, 8360
processing :active, 8360, 53485
section kill 1 · 45.9 s · +1.18M
recovery :crit, 53485, 99371
processing :active, 99371, 172941
section kill 2 · 48.2 s · +1.07M
recovery :crit, 172942, 221093
processing :active, 221093, 274555
section kill 3 · 46.5 s · +1.03M
recovery :crit, 274556, 321061
processing :active, 321061, 385752
section kill 4 · 45.9 s · +1.07M
recovery :crit, 385752, 431659
processing :active, 431659, 490383
section kill 5 · 45.6 s · +1.09M
recovery :crit, 490384, 535981
processing :active, 535981, 610074
section kill 6 · 46.7 s · +1.16M
recovery :crit, 610074, 656777
processing :active, 656777, 714224
section kill 7 · 46.2 s · +1.05M
recovery :crit, 714225, 760423
processing :active, 760423, 825965
section kill 8 · 46.7 s · +1.05M
recovery :crit, 825966, 872690
processing :active, 872690, 920617
section kill 9 · 46.4 s · +1.2M
recovery :crit, 920617, 967054
processing :active, 967054, 1030852
section kill 10 · 53.2 s · to the end
recovery :crit, 1030853, 1084027
processing :active, 1084027, 1141909quix-eos — publishing, 12M records, 10 kills, 1 core, seed A (2026-11-07 13:03)
---
displayMode: compact
config:
gantt:
leftPadding: 237
useWidth: 1000
themeCSS: ".taskText, .taskTextOutsideRight, .taskTextOutsideLeft { display: none; } .task { stroke: none; } .sectionTitle, .grid .tick text { font-family: var(--md-mermaid-font-family); fill: var(--md-mermaid-label-fg-color) !important; } .sectionTitle { font-size: 14px; } .grid .tick text { font-size: 13px; } .grid .tick line { stroke: var(--md-default-fg-color--lighter); stroke-opacity: 0.5; }"
---
gantt
dateFormat x
axisFormat %M:%S
todayMarker off
tickInterval 120second
section start · 1st commit 8.9 s · +1.19M
first commit :0, 8858
processing :active, 8858, 58236
section kill 1 · 47.8 s · +1.05M
recovery :crit, 58236, 106060
processing :active, 106060, 164051
section kill 2 · 46.8 s · +1.18M
recovery :crit, 164051, 210888
processing :active, 210888, 282265
section kill 3 · 47.4 s · +1.21M
recovery :crit, 282266, 329647
processing :active, 329647, 393028
section kill 4 · 46.9 s · +908k
recovery :crit, 393028, 439956
processing :active, 439956, 496902
section kill 5 · 46.1 s · +1.05M
recovery :crit, 496902, 543005
processing :active, 543005, 603091
section kill 6 · 47.0 s · +1.09M
recovery :crit, 603091, 650084
processing :active, 650084, 722477
section kill 7 · 46.0 s · +1.16M
recovery :crit, 722477, 768486
processing :active, 768486, 829510
section kill 8 · 46.1 s · +1.03M
recovery :crit, 829510, 875567
processing :active, 875567, 939574
section kill 9 · 47.0 s · +1.22M
recovery :crit, 939575, 986599
processing :active, 986599, 1048222
section kill 10 · 47.2 s · to the end
recovery :crit, 1048222, 1095452
processing :active, 1095452, 1158481quix-eos — publishing, 12M records, 10 kills, 1 core, seed B (2026-11-07 13:03)
---
displayMode: compact
config:
gantt:
leftPadding: 237
useWidth: 1000
themeCSS: ".taskText, .taskTextOutsideRight, .taskTextOutsideLeft { display: none; } .task { stroke: none; } .sectionTitle, .grid .tick text { font-family: var(--md-mermaid-font-family); fill: var(--md-mermaid-label-fg-color) !important; } .sectionTitle { font-size: 14px; } .grid .tick text { font-size: 13px; } .grid .tick line { stroke: var(--md-default-fg-color--lighter); stroke-opacity: 0.5; }"
---
gantt
dateFormat x
axisFormat %M:%S
todayMarker off
tickInterval 120second
section start · 1st commit 8.5 s · +1.11M
first commit :0, 8457
processing :active, 8457, 57517
section kill 1 · 47.8 s · +1.1M
recovery :crit, 57518, 105354
processing :active, 105354, 173382
section kill 2 · 46.1 s · +1.29M
recovery :crit, 173382, 219520
processing :active, 219520, 279429
section kill 3 · 46.7 s · +874k
recovery :crit, 279429, 326160
processing :active, 326160, 382863
section kill 4 · 47.3 s · +1.23M
recovery :crit, 382864, 430208
processing :active, 430208, 492726
section kill 5 · 46.5 s · +1.05M
recovery :crit, 492727, 539244
processing :active, 539244, 613993
section kill 6 · 46.1 s · +1.05M
recovery :crit, 613994, 660095
processing :active, 660095, 718125
section kill 7 · 52.8 s · +1.03M
recovery :crit, 718125, 770933
processing :active, 770933, 828663
section kill 8 · 47.9 s · +1.21M
recovery :crit, 828663, 876582
processing :active, 876582, 940581
section kill 9 · 45.9 s · +1.06M
recovery :crit, 940582, 986510
processing :active, 986510, 1059842
section kill 10 · 51.0 s · to the end
recovery :crit, 1059843, 1110811
processing :active, 1110811, 1161961risingwave — random, 12M records, 10 kills, 3 cores, seed A (2026-10-20 23:28)
---
displayMode: compact
config:
gantt:
leftPadding: 237
useWidth: 1000
themeCSS: ".taskText, .taskTextOutsideRight, .taskTextOutsideLeft { display: none; } .task { stroke: none; } .sectionTitle, .grid .tick text { font-family: var(--md-mermaid-font-family); fill: var(--md-mermaid-label-fg-color) !important; } .sectionTitle { font-size: 14px; } .grid .tick text { font-size: 13px; } .grid .tick line { stroke: var(--md-default-fg-color--lighter); stroke-opacity: 0.5; }"
---
gantt
dateFormat x
axisFormat %M:%S
todayMarker off
tickInterval 15second
section start · 1st commit 3.9 s · +1.16M
first commit :0, 3899
processing :active, 3899, 7287
section kill 1 · 3.4 s · +1.11M
recovery :crit, 7287, 10654
processing :active, 10654, 19837
section kill 2 · 2.1 s · +1.02M
recovery :crit, 19837, 21908
processing :active, 21908, 30740
section kill 3 · 2.2 s · +1.22M
recovery :crit, 30740, 32973
processing :active, 32973, 44605
section kill 4 · 2.5 s · +1.02M
recovery :crit, 44605, 47149
processing :active, 47149, 57554
section kill 5 · 2.1 s · +1.19M
recovery :crit, 57554, 59689
processing :active, 59689, 70347
section kill 6 · 5.8 s · +968k
recovery :crit, 70347, 76131
processing :active, 76131, 85759
section kill 7 · 6.1 s · +1.22M
recovery :crit, 85759, 91869
processing :active, 91869, 104511
section kill 8 · 3.3 s · +915k
recovery :crit, 104511, 107790
processing :active, 107790, 116431
section kill 9 · 2.3 s · +1.12M
recovery :crit, 116431, 118753
processing :active, 118753, 128721
section kill 10 · 1.8 s · to the end
recovery :crit, 128721, 130558
processing :active, 130558, 138999risingwave — random, 12M records, 10 kills, 3 cores, seed B (2026-10-20 23:24)
---
displayMode: compact
config:
gantt:
leftPadding: 237
useWidth: 1000
themeCSS: ".taskText, .taskTextOutsideRight, .taskTextOutsideLeft { display: none; } .task { stroke: none; } .sectionTitle, .grid .tick text { font-family: var(--md-mermaid-font-family); fill: var(--md-mermaid-label-fg-color) !important; } .sectionTitle { font-size: 14px; } .grid .tick text { font-size: 13px; } .grid .tick line { stroke: var(--md-default-fg-color--lighter); stroke-opacity: 0.5; }"
---
gantt
dateFormat x
axisFormat %M:%S
todayMarker off
tickInterval 15second
section start · 1st commit 3.9 s · +1.13M
first commit :0, 3898
processing :active, 3898, 6681
section kill 1 · 4.1 s · +1.2M
recovery :crit, 6681, 10767
processing :active, 10767, 20081
section kill 2 · 2.0 s · +1.03M
recovery :crit, 20081, 22086
processing :active, 22086, 30827
section kill 3 · 2.3 s · +1.02M
recovery :crit, 30827, 33124
processing :active, 33124, 41887
section kill 4 · 2.2 s · +1.11M
recovery :crit, 41887, 44078
processing :active, 44078, 55134
section kill 5 · 2.6 s · +1.08M
recovery :crit, 55134, 57686
processing :active, 57686, 65452
section kill 6 · 5.7 s · +1.11M
recovery :crit, 65452, 71148
processing :active, 71148, 79392
section kill 7 · 3.1 s · +1.14M
recovery :crit, 79392, 82448
processing :active, 82448, 93409
section kill 8 · 2.2 s · +1.11M
recovery :crit, 93409, 95613
processing :active, 95613, 104059
section kill 9 · 2.9 s · +1.09M
recovery :crit, 104059, 106996
processing :active, 106996, 116878
section kill 10 · 1.8 s · to the end
recovery :crit, 116878, 118693
processing :active, 118693, 127416risingwave — publishing, 12M records, 10 kills, 3 cores, seed A (2026-10-20 23:19)
---
displayMode: compact
config:
gantt:
leftPadding: 237
useWidth: 1000
themeCSS: ".taskText, .taskTextOutsideRight, .taskTextOutsideLeft { display: none; } .task { stroke: none; } .sectionTitle, .grid .tick text { font-family: var(--md-mermaid-font-family); fill: var(--md-mermaid-label-fg-color) !important; } .sectionTitle { font-size: 14px; } .grid .tick text { font-size: 13px; } .grid .tick line { stroke: var(--md-default-fg-color--lighter); stroke-opacity: 0.5; }"
---
gantt
dateFormat x
axisFormat %M:%S
todayMarker off
tickInterval 15second
section start · 1st commit 3.9 s · +1.29M
first commit :0, 3901
processing :active, 3901, 9536
section kill 1 · 3.7 s · +989k
recovery :crit, 9536, 13198
processing :active, 13198, 21016
section kill 2 · 2.9 s · +1.46M
recovery :crit, 21016, 23932
processing :active, 23932, 37454
section kill 3 · 3.1 s · +830k
recovery :crit, 37454, 40572
processing :active, 40572, 48875
section kill 4 · 2.4 s · +1.52M
recovery :crit, 48875, 51281
processing :active, 51281, 65297
section kill 5 · 6.4 s · +959k
recovery :crit, 65297, 71672
processing :active, 71672, 79835
section kill 6 · 2.7 s · +833k
recovery :crit, 79835, 82528
processing :active, 82528, 91226
section kill 7 · 2.0 s · +992k
recovery :crit, 91226, 93247
processing :active, 93247, 102577
section kill 8 · 2.8 s · +1.13M
recovery :crit, 102577, 105379
processing :active, 105379, 113977
section kill 9 · 2.7 s · +1.01M
recovery :crit, 113977, 116701
processing :active, 116701, 125341
section kill 10 · 4.3 s · to the end
recovery :crit, 125341, 129614
processing :active, 129614, 138880risingwave — publishing, 12M records, 10 kills, 3 cores, seed B (2026-10-20 23:15)
---
displayMode: compact
config:
gantt:
leftPadding: 237
useWidth: 1000
themeCSS: ".taskText, .taskTextOutsideRight, .taskTextOutsideLeft { display: none; } .task { stroke: none; } .sectionTitle, .grid .tick text { font-family: var(--md-mermaid-font-family); fill: var(--md-mermaid-label-fg-color) !important; } .sectionTitle { font-size: 14px; } .grid .tick text { font-size: 13px; } .grid .tick line { stroke: var(--md-default-fg-color--lighter); stroke-opacity: 0.5; }"
---
gantt
dateFormat x
axisFormat %M:%S
todayMarker off
tickInterval 15second
section start · 1st commit 3.9 s · +1.35M
first commit :0, 3904
processing :active, 3904, 9567
section kill 1 · 7.8 s · +1.1M
recovery :crit, 9567, 17331
processing :active, 17331, 25003
section kill 2 · 2.8 s · +886k
recovery :crit, 25003, 27812
processing :active, 27812, 36387
section kill 3 · 2.0 s · +1.06M
recovery :crit, 36387, 38404
processing :active, 38404, 47827
section kill 4 · 2.7 s · +1.38M
recovery :crit, 47827, 50525
processing :active, 50525, 64232
section kill 5 · 2.3 s · +1.22M
recovery :crit, 64232, 66532
processing :active, 66532, 75583
section kill 6 · 2.8 s · +1.05M
recovery :crit, 75583, 78426
processing :active, 78426, 86943
section kill 7 · 2.8 s · +915k
recovery :crit, 86943, 89787
processing :active, 89787, 98319
section kill 8 · 3.1 s · +1.01M
recovery :crit, 98319, 101383
processing :active, 101383, 109713
section kill 9 · 2.8 s · +1.53M
recovery :crit, 109713, 112512
processing :active, 112512, 126059
section kill 10 · 2.8 s · to the end
recovery :crit, 126059, 128825
processing :active, 128825, 134360Run configuration
| run | product | seed | backlog | prefill rate | event time | kill at | jitter | cores | env | kills | drained |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 2026-11-10 11:26 | turbine | A (42) | 12M | 400k/s | 5k/s | - | 0–5 s | 3 | BENCH_BATCH_SIZE=10000 TURBINE_STATE_CACHE_SIZE=2000000 |
0 / 0 | 130.3 s |
| 2026-11-10 11:35 | turbine | A (42) | 12M | 400k/s | 5k/s | random | 0–5 s | 3 | BENCH_BATCH_SIZE=10000 TURBINE_STATE_CACHE_SIZE=2000000 |
10 / 10 | 149.5 s |
| 2026-11-10 11:45 | turbine | A (42) | 12M | 400k/s | 5k/s | publishing | - | 3 | BENCH_BATCH_SIZE=10000 TURBINE_STATE_CACHE_SIZE=2000000 |
10 / 10 | 149.8 s |
| 2026-11-10 10:33 | turbine | B (43) | 12M | 400k/s | 5k/s | - | 0–5 s | 3 | BENCH_BATCH_SIZE=10000 TURBINE_STATE_CACHE_SIZE=2000000 |
0 / 0 | 135.3 s |
| 2026-11-10 10:42 | turbine | B (43) | 12M | 400k/s | 5k/s | random | 0–5 s | 3 | BENCH_BATCH_SIZE=10000 TURBINE_STATE_CACHE_SIZE=2000000 |
10 / 10 | 145.3 s |
| 2026-11-10 10:51 | turbine | B (43) | 12M | 400k/s | 5k/s | publishing | - | 3 | BENCH_BATCH_SIZE=10000 TURBINE_STATE_CACHE_SIZE=2000000 |
10 / 10 | 139.9 s |
| 2026-11-10 10:58 | turbine-eos | A (42) | 12M | 400k/s | 5k/s | - | 0–1 s | 3 | BENCH_BATCH_SIZE=10000 TURBINE_STATE_CACHE_SIZE=2000000 |
0 / 0 | 143.0 s |
| 2026-11-10 11:07 | turbine-eos | A (42) | 12M | 400k/s | 5k/s | random | 0–1 s | 3 | BENCH_BATCH_SIZE=10000 TURBINE_STATE_CACHE_SIZE=2000000 |
10 / 10 | 165.2 s |
| 2026-11-10 11:17 | turbine-eos | A (42) | 12M | 400k/s | 5k/s | publishing | - | 3 | BENCH_BATCH_SIZE=10000 TURBINE_STATE_CACHE_SIZE=2000000 |
10 / 10 | 188.5 s |
| 2026-11-10 10:05 | turbine-eos | B (43) | 12M | 400k/s | 5k/s | - | 0–1 s | 3 | BENCH_BATCH_SIZE=10000 TURBINE_STATE_CACHE_SIZE=2000000 |
0 / 0 | 137.6 s |
| 2026-11-10 10:14 | turbine-eos | B (43) | 12M | 400k/s | 5k/s | random | 0–1 s | 3 | BENCH_BATCH_SIZE=10000 TURBINE_STATE_CACHE_SIZE=2000000 |
10 / 10 | 159.8 s |
| 2026-11-10 10:24 | turbine-eos | B (43) | 12M | 400k/s | 5k/s | publishing | - | 3 | BENCH_BATCH_SIZE=10000 TURBINE_STATE_CACHE_SIZE=2000000 |
10 / 10 | 162.9 s |
| 2026-10-23 17:17 | arroyo | A (42) | 12M | 400k/s | 5k/s | - | 0–1 s | 3 | ARROYO__DEFAULT_CHECKPOINT_INTERVAL=1s |
0 / 0 | 21.1 s |
| 2026-10-23 17:25 | arroyo | A (42) | 12M | 400k/s | 5k/s | random | 0–1 s | 3 | ARROYO__DEFAULT_CHECKPOINT_INTERVAL=1s |
10 / 10 | 34.9 s |
| 2026-10-23 17:34 | arroyo | A (42) | 12M | 400k/s | 5k/s | publishing | - | 3 | ARROYO__DEFAULT_CHECKPOINT_INTERVAL=1s |
10 / 10 | 38.0 s |
| 2026-10-23 17:20 | arroyo | B (43) | 12M | 400k/s | 5k/s | - | 0–1 s | 3 | ARROYO__DEFAULT_CHECKPOINT_INTERVAL=1s |
0 / 0 | 22.3 s |
| 2026-10-23 17:29 | arroyo | B (43) | 12M | 400k/s | 5k/s | random | 0–1 s | 3 | ARROYO__DEFAULT_CHECKPOINT_INTERVAL=1s |
10 / 10 | 42.5 s |
| 2026-10-23 17:38 | arroyo | B (43) | 12M | 400k/s | 5k/s | publishing | - | 3 | ARROYO__DEFAULT_CHECKPOINT_INTERVAL=1s |
10 / 10 | 39.4 s |
| 2026-10-23 16:59 | arroyo-eos | A (42) | 12M | 400k/s | 5k/s | - | 0–1 s | 3 | ARROYO__DEFAULT_CHECKPOINT_INTERVAL=1s |
0 / 0 | 25.7 s |
| 2026-10-23 17:02 | arroyo-eos | A (42) | 12M | 400k/s | 5k/s | random | 0–1 s | 3 | ARROYO__DEFAULT_CHECKPOINT_INTERVAL=1s |
3 / 10 | no |
| 2026-10-23 17:10 | arroyo-eos | A (42) | 12M | 400k/s | 5k/s | publishing | - | 3 | ARROYO__DEFAULT_CHECKPOINT_INTERVAL=1s |
10 / 10 | 46.9 s |
| 2026-10-23 17:02 | arroyo-eos | B (43) | 12M | 400k/s | 5k/s | - | 0–1 s | 3 | ARROYO__DEFAULT_CHECKPOINT_INTERVAL=1s |
0 / 0 | 27.4 s |
| 2026-10-23 17:05 | arroyo-eos | B (43) | 12M | 400k/s | 5k/s | random | 0–1 s | 3 | ARROYO__DEFAULT_CHECKPOINT_INTERVAL=1s |
4 / 10 | no |
| 2026-10-23 17:13 | arroyo-eos | B (43) | 12M | 400k/s | 5k/s | publishing | - | 3 | ARROYO__DEFAULT_CHECKPOINT_INTERVAL=1s |
10 / 10 | 48.5 s |
| 2026-10-20 10:07 | flink | A (42) | 12M | 400k/s | 5k/s | - | 0–10 s | 3 | - | 0 / 0 | 99.2 s |
| 2026-10-20 10:18 | flink | A (42) | 12M | 400k/s | 5k/s | random | 0–10 s | 3 | - | 10 / 10 | 211.0 s |
| 2026-10-20 10:08 | flink | B (43) | 12M | 400k/s | 5k/s | - | 0–10 s | 3 | - | 0 / 0 | 105.1 s |
| 2026-10-20 10:19 | flink | B (43) | 12M | 400k/s | 5k/s | random | 0–10 s | 3 | - | 9 / 10 | 194.5 s |
| 2026-10-20 22:52 | flink | A (42) | 12M | 400k/s | 5k/s | - | 0–10 s | 3 | - | 0 / 0 | 117.2 s |
| 2026-10-20 23:01 | flink | A (42) | 12M | 400k/s | 5k/s | publishing | - | 3 | - | 9 / 10 | 159.6 s |
| 2026-10-20 22:48 | flink | B (43) | 12M | 400k/s | 5k/s | - | 0–10 s | 3 | - | 0 / 0 | 106.4 s |
| 2026-10-20 22:57 | flink | B (43) | 12M | 400k/s | 5k/s | publishing | - | 3 | - | 10 / 10 | 171.2 s |
| 2026-10-20 09:48 | flink-eos | A (42) | 12M | 400k/s | 5k/s | - | 0–10 s | 3 | - | 0 / 0 | 105.2 s |
| 2026-10-20 09:58 | flink-eos | A (42) | 12M | 400k/s | 5k/s | random | 0–10 s | 3 | - | 9 / 10 | 196.6 s |
| 2026-10-20 09:48 | flink-eos | B (43) | 12M | 400k/s | 5k/s | - | 0–10 s | 3 | - | 0 / 0 | 101.3 s |
| 2026-10-20 09:59 | flink-eos | B (43) | 12M | 400k/s | 5k/s | random | 0–10 s | 3 | - | 10 / 10 | 216.6 s |
| 2026-10-20 22:33 | flink-eos | A (42) | 12M | 400k/s | 5k/s | - | 0–10 s | 3 | - | 0 / 0 | 104.9 s |
| 2026-10-20 22:43 | flink-eos | A (42) | 12M | 400k/s | 5k/s | publishing | - | 3 | - | 9 / 10 | 159.2 s |
| 2026-10-20 22:29 | flink-eos | B (43) | 12M | 400k/s | 5k/s | - | 0–10 s | 3 | - | 0 / 0 | 122.7 s |
| 2026-10-20 22:39 | flink-eos | B (43) | 12M | 400k/s | 5k/s | publishing | - | 3 | - | 10 / 10 | 169.5 s |
| 2026-11-07 13:18 | quix | A (42) | 12M | 400k/s | 5k/s | - | 0–5 s | 1 | - | 0 / 0 | 476.6 s |
| 2026-11-07 13:42 | quix | A (42) | 12M | 400k/s | 5k/s | random | 0–5 s | 1 | - | 10 / 10 | 1039.5 s |
| 2026-11-07 14:06 | quix | A (42) | 12M | 400k/s | 5k/s | publishing | - | 1 | - | 10 / 10 | 1073.0 s |
| 2026-11-07 13:17 | quix | B (43) | 12M | 400k/s | 5k/s | - | 0–5 s | 1 | - | 0 / 0 | 484.8 s |
| 2026-11-07 13:41 | quix | B (43) | 12M | 400k/s | 5k/s | random | 0–5 s | 1 | - | 10 / 10 | 1052.7 s |
| 2026-11-07 14:06 | quix | B (43) | 12M | 400k/s | 5k/s | publishing | - | 1 | - | 10 / 10 | 1065.5 s |
| 2026-11-07 12:12 | quix-eos | A (42) | 12M | 400k/s | 5k/s | - | 0–5 s | 1 | - | 0 / 0 | 561.4 s |
| 2026-11-07 12:37 | quix-eos | A (42) | 12M | 400k/s | 5k/s | random | 0–5 s | 1 | - | 10 / 10 | 1141.7 s |
| 2026-11-07 13:03 | quix-eos | A (42) | 12M | 400k/s | 5k/s | publishing | - | 1 | - | 10 / 10 | 1158.5 s |
| 2026-11-07 12:12 | quix-eos | B (43) | 12M | 400k/s | 5k/s | - | 0–5 s | 1 | - | 0 / 0 | 565.1 s |
| 2026-11-07 12:37 | quix-eos | B (43) | 12M | 400k/s | 5k/s | random | 0–5 s | 1 | - | 10 / 10 | 1141.9 s |
| 2026-11-07 13:03 | quix-eos | B (43) | 12M | 400k/s | 5k/s | publishing | - | 1 | - | 10 / 10 | 1162.0 s |
| 2026-10-20 23:10 | risingwave | A (42) | 12M | 400k/s | 5k/s | - | 0–1 s | 3 | BENCH_DDL=background |
0 / 0 | 124.4 s |
| 2026-10-20 23:19 | risingwave | A (42) | 12M | 400k/s | 5k/s | publishing | - | 3 | BENCH_DDL=background |
10 / 10 | 138.9 s |
| 2026-10-20 23:28 | risingwave | A (42) | 12M | 400k/s | 5k/s | random | 0–1 s | 3 | BENCH_DDL=background |
10 / 10 | 139.0 s |
| 2026-10-20 23:06 | risingwave | B (43) | 12M | 400k/s | 5k/s | - | 0–1 s | 3 | BENCH_DDL=background |
0 / 0 | 115.6 s |
| 2026-10-20 23:15 | risingwave | B (43) | 12M | 400k/s | 5k/s | publishing | - | 3 | BENCH_DDL=background |
10 / 10 | 134.4 s |
| 2026-10-20 23:24 | risingwave | B (43) | 12M | 400k/s | 5k/s | random | 0–1 s | 3 | BENCH_DDL=background |
10 / 10 | 127.4 s |
What a clean run does not prove
- Only the transactions that close windows carry output, so aborted under-counts the kills that hit an open transaction; a clean run without any is inconclusive, never a pass.
- Turbine commits a batch's transaction before its RocksDB write. A kill in between leaves the local state a batch behind the broker: its
replaypolicy re-consumes that batch with the outputs suppressed, logged and counted (state gap). Random kills land there a few times in a hundred restarts, so a run of 10 kills may go through none. - Flink has no such window to choose a policy for: its checkpoint stores the state and the source offsets together, and the sink commits the checkpoint's transaction only after. A kill between the two is recovered by the restored job re-committing the pending transaction — a ms-wide window, logged at DEBUG only, not counted. Its lossy case is a restart slower than the transaction timeout (15 min here): the broker aborts the pending transaction, and the connector logs that this "most likely signals data loss" — not exercised.
- A life must commit before it can be killed: a line committing on a 10 s checkpoint processes far more records per life than one committing every second (records / life); the backlog is sized so that Flink fits its kills.
- RisingWave is a server: its job is submitted once, in the background (a job still being created in the foreground — a drain's whole backfill — is dropped when the cluster restarts), the kill SIGKILLs the container's process and the restart is a
docker starton the same writable layer (state and meta store), from which the server recovers the job itself. Its committed position is its source's state, persisted on every 1 s checkpoint and read from the server over SQL: unreadable while it is down, the position at a kill is the restored one, read at the restart (at worst one checkpoint ahead). - Not covered yet: a lost local disk (restore from the remote snapshot / changelog).
Not shown: 80 run(s) off the protocol, without an audit (interrupted), or re-run since on the same configuration and seed.
Host: ovh:c3-32 (ovh:c3-32@DE1, ovh:c3-32@SBG5, ovh:c3-32@UK1) · AMD EPYC-Milan Processor (2400 MHz) · 16 cores / 16 threads · 31345 MB · kernel 6.12.100+deb13-cloud-amd64.