Research documentation ยท Experimental record
Research

Evaluation metrics

The common scoreboard used across fault and recovery combinations.

MetricDefinition
Degradation-areaReward lost below the pre-fault baseline, summed from fault onset to the end of the episode, minus the same quantity for a matched fault-free control episode (floored at zero).
Safety-violation excessWeighted count of container overflow states (5 × catastrophic + 2 × severe + 1 × near-miss) from fault onset to the end of the episode, minus the same count for the matched fault-free control episode (floored at zero).
Fraction of time healthyShare of steps from fault onset to the end of the episode on which reward is within tolerance of the pre-fault baseline.
Relapse countNumber of times performance falls back below the healthy threshold after a confirmed recovery (15 consecutive healthy steps).
False-recovery rateShare of a mechanism's interventions made on steps where no fault was affecting either channel. Only defined for mechanisms with a detector.
Intervention recallShare of steps with a fault actually affecting a channel on which the mechanism intervened. Only defined for mechanisms with a detector.

Experimental reporting

Every combination is run over 8 random seeds (and over 3 training seeds for the two trained mechanisms). Results are reported as means with 95% confidence intervals, plus Glass's Δ effect sizes against the no-recovery control; see the Results page.