What Black-Box Benchmarks Miss: Open-Ended Generation Under-Measures Memorization in Foundation Models

Public benchmarks are commonly audited for pretraining contamination with a black-box test: prompt the model, read its answer, and compare it to ground truth. We show that this audit can issue false-clean verdicts, certifying a benchmark as uncontaminated even when the model has stored the answers, because it reads the less sensitive of two valid instruments. The question “does the model know a value?” has two operational answers that the literature often conflates: a generation readout, which asks whether decoding emits the value, and a candidate-conditioned logprob readout, which asks whether the value is top-ranked among magnitude-matched alternatives. On a real, non-fine-tuned 3B open model, the logprob readout ranks stored mountain elevations above their nearest real neighbor 98% of the time, while generation emits them only 50% of the time, recovering 20/42 values that generation gets wrong with no reverse cases. A known-exposure canary shows the gap opens at partial exposure and closes at saturation. We further show that missed contamination can propagate into downstream signals that never query the benchmark: a date-only “market sentiment” score is observationally identical to the model’s own recalled values, and residualizing on recall collapses its truth-correlation from 0.74 to 0.02. We define this empirical readout gap, show why naive logprob ranking is itself confounded by numeric priors, and propose a corrected, logprob-aware audit protocol using magnitude-matched distractors. Our results suggest that black-box leakage tests should be treated as lower bounds, not certificates of benchmark cleanliness.