Gate decisions
Every decision the gate has made, from its append-only ledger. A candidate passes a suite only when the lower bound of the 95% interval for its difference from the baseline stays above minus the suite's margin. Records are content-addressed: the same comparison under the same spec is the same id.
e3fdc06fa056e927 blocked 2026-09-20T00:40:46Z
2026-09/openweights-control → 2026-09-run2/openweights-control, spec drift-blocks (1b6d78c7491b)
| Suite | Verdict | Items | Baseline | Candidate | Difference (95% CI) | Margin | Reason |
|---|---|---|---|---|---|---|---|
| closed_form_reasoning | block | 120 | 96.7% | 95.0% | -1.7 (-5.0 to +1.7) | 3% | cannot rule out a 3% drop: -1.7% (-5.0% to +1.7%) over 120 items, 3 worse and 1 better (p 0.222, Holm 0.222) |
| multiple_choice | warn | 100 | 82.0% | 83.0% | +1.0 (+0.0 to +3.0) | 3% | 100 paired items, fewer than the 118 needed to see a 3% drop at 80% power: under-powered, warns only |
| instruction_following | warn | 60 | 95.0% | 96.7% | +1.7 (+0.0 to +5.0) | 3% | 60 paired items, fewer than the 118 needed to see a 3% drop at 80% power: under-powered, warns only |
| structured_extraction | warn | 20 | 95.0% | 95.0% | +0.0 (+0.0 to +0.0) | 3% | 20 paired items, fewer than the 118 needed to see a 3% drop at 80% power: under-powered, warns only |
| structured_extraction_heldout | warn | 20 | 90.0% | 90.0% | +0.0 (+0.0 to +0.0) | 3% | 20 paired items, fewer than the 118 needed to see a 3% drop at 80% power: under-powered, warns only |
| refusal_calibration | warn | 40 | 92.5% | 95.0% | +2.5 (+0.0 to +7.5) | 3% | 40 paired items, fewer than the 118 needed to see a 3% drop at 80% power: under-powered, warns only |
| long_context_recall | warn | 20 | 100.0% | 100.0% | +0.0 (+0.0 to +0.0) | 3% | 20 paired items, fewer than the 118 needed to see a 3% drop at 80% power: under-powered, warns only |
| paraphrase_robustness | warn | 40 | 95.0% | 97.5% | +2.5 (+0.0 to +7.5) | 3% | 40 paired items, fewer than the 118 needed to see a 3% drop at 80% power: under-powered, warns only |
16d849d843b034f3 passed 2026-09-20T00:40:41Z
2026-09/anthropic-alias → 2026-09-run2/anthropic-alias, spec drift-blocks (1b6d78c7491b)
| Suite | Verdict | Items | Baseline | Candidate | Difference (95% CI) | Margin | Reason |
|---|---|---|---|---|---|---|---|
| closed_form_reasoning | pass | 120 | 100.0% | 100.0% | +0.0 (+0.0 to +0.0) | 3% | non-inferior: +0.0% (+0.0% to +0.0%) over 120 items stays above -3% (p 0.000, Holm 0.000) |
| multiple_choice | warn | 100 | 93.0% | 92.0% | -1.0 (-5.0 to +2.0) | 3% | 100 paired items, fewer than the 118 needed to see a 3% drop at 80% power: under-powered, warns only |
| instruction_following | warn | 60 | 95.0% | 95.0% | +0.0 (+0.0 to +0.0) | 3% | 60 paired items, fewer than the 118 needed to see a 3% drop at 80% power: under-powered, warns only |
| structured_extraction | warn | 20 | 90.0% | 90.0% | +0.0 (+0.0 to +0.0) | 3% | 20 paired items, fewer than the 118 needed to see a 3% drop at 80% power: under-powered, warns only |
| structured_extraction_heldout | warn | 20 | 85.0% | 85.0% | +0.0 (+0.0 to +0.0) | 3% | 20 paired items, fewer than the 118 needed to see a 3% drop at 80% power: under-powered, warns only |
| refusal_calibration | warn | 40 | 85.0% | 85.0% | +0.0 (+0.0 to +0.0) | 3% | 40 paired items, fewer than the 118 needed to see a 3% drop at 80% power: under-powered, warns only |
| long_context_recall | warn | 20 | 100.0% | 100.0% | +0.0 (+0.0 to +0.0) | 3% | 20 paired items, fewer than the 118 needed to see a 3% drop at 80% power: under-powered, warns only |
| paraphrase_robustness | warn | 40 | 100.0% | 100.0% | +0.0 (+0.0 to +0.0) | 3% | 40 paired items, fewer than the 118 needed to see a 3% drop at 80% power: under-powered, warns only |