Aggregate benchmark comparison

Multi-seed, cross-mechanism comparison across all five fault classes, replacing single-run inspection with averaged, confidence-bounded metrics.

Overview

The individual-run pages linked from the sidebar each show one trajectory graph for one seed. This page instead aggregates 8 independent seed realizations per (fault × target × duration × fault mode × recovery mechanism) combination, with the two trained mechanisms (supervisor and fault_tolerant_marl) evaluated on each of 3 training seeds - 4,160 individual evaluation runs in total - into means with 95% confidence intervals, and further breaks each result down by fault target (sort agent, press agent, or both) and fault duration (transient or permanent). Six recovery arms are compared throughout: none (no recovery, the control condition), rule_based (oracle reconfiguration, told the ground-truth fault state), rule_based_detect (the same corrective heuristic, but required to detect the fault itself from observable signals), supervisor (a trained hierarchical supervisor policy), fault_tolerant_marl (agents trained under domain-randomized faults), and llm_replanning (an LLM agent that reasons over system state and issues a recovery plan).

Note on rule_based vs. the "Rule-based reconfiguration" pages elsewhere on this site: the "Rule-based reconfiguration" trajectory pages show both variants: the oracle (rule_based) and rule_based_detect. rule_based_detect is the same corrective action set (reconfigure the affected channel using a fixed heuristic) gated behind self-detection instead of ground truth, and is included here as a distinct arm because detection quality turns out to matter a great deal - see the metric glossary and per-fault sections below.

Why multi-seed

A single evaluation run answers "what happened in this one random draw," not "how does this mechanism perform in general." Two failure modes make a single run actively misleading here: a fault instance can be too mild to produce any measurable effect (a null result - the run looks identical to a fault-free control episode within normal noise), and a full-override oracle mechanism can produce byte-identical outcomes across unrelated fault types purely because it discards the fault's effect entirely once it engages. Averaging 8 independently-seeded realizations per combination - each with its own episode dynamics and its own fault-injection timing - turns these single-draw artifacts into an honest signal, the same reason a hardware benchmark reports an average over many passes rather than one frame's timing.

How to read these charts

Every chart is a grid of bar plots: rows are the fault's target (sort agent / press agent / both), columns are duration (transient / permanent). Bars within a cell are the six recovery arms, in the same left-to-right order every time (none, rule_based, rule_based_detect, supervisor, fault_tolerant_marl, llm_replanning), and the black whisker on each bar is a 95% confidence interval computed across the 8 seed realizations (24 runs for supervisor and fault_tolerant_marl, which are evaluated on each of 3 training seeds). A missing bar means no data for that arm in that cell (for example, none and fault_tolerant_marl have no fault detector, so they never appear on the detection-quality charts).

Metric glossary

A time-to-recovery metric was evaluated and dropped from this comparison. It is only meaningful over events that both recovered and were not a null result, and that qualifying subset differs enormously in size and composition between mechanisms - a mechanism that fully neutralizes a fault has that event excluded as a null result altogether, leaving only its hardest, unresolved cases to average over, while a mechanism that never touches the fault keeps a larger, easier-composed pool. This produced misleading "faster recovery" numbers for mechanisms doing objectively less. Fraction-of-window-healthy above is reported instead, since it is computed identically for every event regardless of outcome.

Sensor noise

Additive noise on observation vectors - the mildest fault class in the suite. Null-result rates run 33-44% across arms, meaning a large share of individual instances at the default noise level (σ=0.1) are not reliably distinguishable from ordinary episode noise. Read the confidence intervals carefully here in particular.

Corrected degradation-area (lower is better)

Weighted safety-violation excess (lower is better)

Fraction of window spent healthy (higher is better)

Relapse count (lower is better)

False-recovery rate (lower is better)

Intervention recall (higher is better)

Findings. For a fault this mild, target matters more than recovery mechanism: every arm looks close to perfect on target=sort, and the real separation only appears on target=press/both, where none and llm_replanning are the weakest performers on both degradation-area and safety-excess, fault_tolerant_marl has the lowest pooled damage after the oracle, and supervisor's best-in-class recall does not translate into lower damage. rule_based_detect and llm_replanning trade a lower false-positive rate for lower recall than supervisor.

Effect size vs. no recovery (Glass's Δ)

Glass's Δ = (mechanism mean - none mean) / none's own standard deviation, pooled across every target × duration combo for this fault (n=48 runs per arm; 144 for supervisor and fault_tolerant_marl, which pool 3 training seeds). Negative is an improvement for degradation-area, safety-excess, and relapse-count; positive is an improvement for fraction-healthy. Conventional magnitude bands (Cohen, 1988): |Δ| ≈ 0.2 small, 0.5 medium, 0.8 large - not a significance test, read alongside n.

Actuator degradation

Motor/actuator attenuation in one of three modes (stuck, restrict, slip), pooled across modes in the charts below. Null-result rates are lower here (17-33%) than sensor noise - a more consistently measurable fault.

Corrected degradation-area (lower is better)

Weighted safety-violation excess (lower is better)

Fraction of window spent healthy (higher is better)

Relapse count (lower is better)

False-recovery rate (lower is better)

Intervention recall (higher is better)

Findings. Every self-detecting or oracle mechanism beats both none and fault_tolerant_marl by a wide margin on safety; fault_tolerant_marl tracks none almost exactly on both headline metrics, meaning the fault-tolerant training did not measurably help against this fault class in aggregate (see cross-fault conclusions for why).

Effect size vs. no recovery (Glass's Δ)

Glass's Δ = (mechanism mean - none mean) / none's own standard deviation, pooled across every target × duration combo for this fault (n=144 runs per arm; 432 for supervisor and fault_tolerant_marl, which pool 3 training seeds). Negative is an improvement for degradation-area, safety-excess, and relapse-count; positive is an improvement for fraction-healthy. Conventional magnitude bands (Cohen, 1988): |Δ| ≈ 0.2 small, 0.5 medium, 0.8 large - not a significance test, read alongside n.

Agent dropout

An agent's participation is completely zeroed out - one of the two most severe fault classes tested, alongside Byzantine.

Corrected degradation-area (lower is better)

Weighted safety-violation excess (lower is better)

Fraction of window spent healthy (higher is better)

Relapse count (lower is better)

False-recovery rate (lower is better)

Intervention recall (higher is better)

Findings. This is the strongest evidence in the suite for active correction: none and fault_tolerant_marl are statistically indistinguishable from each other and both roughly 17x worse than rule_based on safety-violation excess. Every real corrective mechanism, however crude, is worth deploying against a fully-dropped agent.

Effect size vs. no recovery (Glass's Δ)

Glass's Δ = (mechanism mean - none mean) / none's own standard deviation, pooled across every target × duration combo for this fault (n=48 runs per arm; 144 for supervisor and fault_tolerant_marl, which pool 3 training seeds). Negative is an improvement for degradation-area, safety-excess, and relapse-count; positive is an improvement for fraction-healthy. Conventional magnitude bands (Cohen, 1988): |Δ| ≈ 0.2 small, 0.5 medium, 0.8 large - not a significance test, read alongside n.

Communication loss

Inter-agent messaging is dropped or staled; only the press channel is affected by construction, so every chart below has a single target row. The one fault class where fault_tolerant_marl outright wins rather than tying none.

Corrected degradation-area (lower is better)

Weighted safety-violation excess (lower is better)

Fraction of window spent healthy (higher is better)

Relapse count (lower is better)

False-recovery rate (lower is better)

Intervention recall (higher is better)

Findings. This fault only degrades the press channel's forward-looking signal about the sort stage - the press agent's own local sensing stays accurate - making it the mildest, most partial-information fault in the suite. That is consistent with fault_tolerant_marl's domain-randomized training transferring well here while it does not for the near-total failures (agent dropout, Byzantine): the degradation is small enough that a policy trained for general robustness has an easier job than a policy trying to substitute for a fully broken or adversarial teammate. fault_tolerant_marl also turns most communication-loss instances into null results (81% of its runs, against 53% for none).

Effect size vs. no recovery (Glass's Δ)

Glass's Δ = (mechanism mean - none mean) / none's own standard deviation, pooled across every target × duration combo for this fault (n=32 runs per arm; 96 for supervisor and fault_tolerant_marl, which pool 3 training seeds). Negative is an improvement for degradation-area, safety-excess, and relapse-count; positive is an improvement for fraction-healthy. Conventional magnitude bands (Cohen, 1988): |Δ| ≈ 0.2 small, 0.5 medium, 0.8 large - not a significance test, read alongside n.

Byzantine / adversarial agents

A cooperative agent's policy is replaced with an adversarial one, in one of three modes (fixed-malicious, worst-action, random-malicious), pooled across modes below.

Corrected degradation-area (lower is better)

Weighted safety-violation excess (lower is better)

Fraction of window spent healthy (higher is better)

Relapse count (lower is better)

False-recovery rate (lower is better)

Intervention recall (higher is better)

Findings. Same shape as agent dropout: none and fault_tolerant_marl are tied at the bottom, every corrective mechanism is a large, clear improvement on safety, and rule_based_detect (30.2) outperforms supervisor (37.3) and, more clearly, llm_replanning (42.8) - as on most fault types, supervisor's higher recall does not translate into lower damage.

Effect size vs. no recovery (Glass's Δ)

Glass's Δ = (mechanism mean - none mean) / none's own standard deviation, pooled across every target × duration combo for this fault (n=144 runs per arm; 432 for supervisor and fault_tolerant_marl, which pool 3 training seeds). Negative is an improvement for degradation-area, safety-excess, and relapse-count; positive is an improvement for fraction-healthy. Conventional magnitude bands (Cohen, 1988): |Δ| ≈ 0.2 small, 0.5 medium, 0.8 large - not a significance test, read alongside n.

Cross-fault conclusions

Doing nothing is safe only against the mildest faults

none and fault_tolerant_marl are statistically indistinguishable from each other on both degradation-area and safety-violation excess for actuator degradation, agent dropout, and Byzantine - and both sit roughly an order of magnitude worse on safety than any actively-correcting mechanism for agent dropout (~17x) and Byzantine (~13x). For faults that fully compromise an agent's decision-making, leaving it uncorrected is measurably dangerous, not just measurably suboptimal - this is the strongest, most defensible result in the suite. The one exception is communication loss, the mildest and most partial fault tested, where fault_tolerant_marl outright wins.

The oracle's outcome does not depend on the fault type

rule_based reports near-identical degradation-area, safety-excess, and healthy-fraction across sensor noise, actuator degradation, agent dropout, and Byzantine (comms_loss differs because it is structurally a press-only fault). This is not an artifact of the evaluation - it is a direct consequence of the mechanism's design. Once an oracle recovery mechanism engages, it fully overrides the affected channel's decision with a state-following heuristic for the remainder of the fault window; the specific mechanism that corrupted the original decision becomes irrelevant, because that original decision is discarded entirely. What matters to an oracle is only when and where it engages, never what caused the engagement. This is a real, structural property worth stating plainly rather than a coincidence to explain away.

Detection quality is a genuine precision/recall trade-off between mechanisms

The three self-detecting arms sit at different points on the trade-off. supervisor has the highest recall of any real detector on sensor noise (0.68-0.74, vs. 0.16-0.54 for the other two) but also the highest false-recovery rate on every fault type (~0.57-0.58 pooled, vs. ~0.41-0.47). rule_based_detect and llm_replanning - which share the same underlying detector - sit closer together, trading lower recall for fewer false alarms. In this sweep the extra recall does not buy lower damage: rule_based_detect matches or beats supervisor on pooled degradation-area for every fault type except actuator degradation (15.7 vs. 15.0).

Pooled numbers hide fault-target structure large enough to change conclusions

Averaging across a fault's target (sort agent / press agent / both) can average together results that differ by 50x or more, as with supervisor under sensor noise: near-zero corrected degradation-area on target=sort, roughly 70x that on target=press/both. A pooled headline number for that cell would read as an unremarkable middling result; the faceted view shows it is actually "irrelevant on one channel, a large loss on the other." Any comparison across mechanisms should be read at the target/duration level, not the pooled level, before it is trusted.

What this does and does not establish

Null-result rates ranged from 0% to 81% across (fault, recovery) combinations - a meaningful fraction of individual fault instances, especially for sensor noise, are not statistically distinguishable from ordinary episode noise at the configured injection strength. This is expected and is not filtered out of the reported means; with 8 seeds per cell it washes out rather than dominating the average, but it does mean the confidence intervals shown should be read as indicative rather than as a formal significance test - no paired significance testing across mechanisms has been run yet. The safety-violation-excess and degradation-area results for agent dropout, Byzantine, and actuator degradation are large enough relative to their confidence intervals to be read with confidence; results for the milder sensor-noise and communication-loss classes, and any single target/duration cell with a wide error bar, should be treated as directional rather than conclusive.