Backtesting and live-fire exercises answer different questions
A defensive agent can look strong in a backtest and still be brittle when traces change. The useful governance move is not to pick one score, but to keep the questions—and the evidence—separate.
In one reported evaluation, detections that passed backtesting were tested against eight unseen attacks. Five of 11 detections from an optimized open pipeline fired at least once, compared with 10 of 35 from a frontier system. Firing was only the first gate: the evaluation also checked whether detections stayed quiet on available test traffic and whether independent reviewers found behavioral grounding, multiple signals, and no environment-specific strings. After those checks, three open-pipeline detections qualified as “gold”; none from the frontier system did.
That result is interesting, but it is not a universal ranking. The source calls the evaluation a directional system-level case study: it covers one scenario family, small detection sets, and limited benign traffic. Three live-fire runs also had harness failures, although their telemetry was retained. Those boundaries matter when translating a result into a safety claim.
A practical evaluation record can have two lanes:
- Known-trace evidence: what passed on the backtest corpus, with the corpus and selection rules named.
- Novel-trace evidence: which previously unseen variations were exercised, what fired, what remained quiet, and which independent review gates were passed.
Design the second lane before the test, not after a surprising result. Preserve the attack-family boundary, benign-traffic coverage, harness incidents, and reviewer criteria alongside the outcome. If a detection fires on a novel trace but fails grounding or noise checks, record that as a useful failure—not as generalized capability.
The decision boundary is modest: live-fire evidence can justify confidence about tested variations and review gates, but it cannot establish production false-positive behavior or cross-scenario generalization by itself. Expanding that claim requires new scenario families, stronger benign-traffic coverage, and repeatable oversight.