When a Contamination Audit Says “Clean”: False-Clean Rate as a Robustness Metric for Benchmark Gaming
Abstract
A benchmark-contamination audit is a measurement instrument, and like any instrument it has an error rate. The error that matters under strategic optimization is the false-clean rate: how often the audit certifies a genuinely contaminated model as clean. In the field this rate is unrecoverable, because a model flagged "clean" has no ground truth to check it against. We recover it by injecting contamination under control across 11 LoRA and full fine-tuning regimes on MMLU (5 base-model families), giving every model a known label. The false-clean rate then depends on what signal the audit reads. A membership detector (Min-K% Prob), which asks whether the model finds the exact benchmark tokens unusually likely, false-cleans 0 of 11 contaminated models. Every signature detector, which instead looks for a behavioral fingerprint, has a mechanism-specific blind spot: three independently constructed surface/paraphrase auditors each false-clean 6 of 11 (55%), and an output-collapse detector misses 5 of 11 (45%). The surface audit fails in both directions — naively thresholded, it also false-alarms on clean models whose pretraining corpus already prefers the canonical wording. Independent measurement-science checks confirm the missed regimes carry genuine contamination, not detector noise. The blind spot is corpus-bound, persisting across seven model families and three scales. The operative axis is thus membership vs. signature: the robust certificate uses a membership detector, while a surface or paraphrase audit annotates memorization type without certifying coverage.