Your Benchmark Score Is Not a Measurement: Capability Does Not Determine Cue-Robustness
Abstract
A benchmark score is reported as a stable property of a model. We show it is instead one point on a cue-response curve, and we quantify the curve's width as a measurement-validity statistic. A single leading cue, a one-sentence assertion of an answer that supplies no evidence and is of the kind that leaks into prompts, retrieved context, and multi-agent pipelines, moves the same model on the same MMLU-Pro items across a manipulable range of 66 points (pooled), in a graded dose-response. We measure this range per model with a paired item-bootstrap confidence interval and find that accuracy predicts cue-robustness but does not determine it: a higher-scoring model can be decisively less robust, for example DeepSeek-V3.2 is more manipulable than the lower-scoring Sonnet 4.6 (paired bootstrap P=0.999). Read through generalizability theory, the cue is a method facet and the manipulable range summarizes the score variation attributable to it, the very thing a single reliability coefficient hides. The effect holds for the cues that dominate deployment, with a RAG-style retrieved snippet moving the score even more than a bare assertion (pooled range 0.80 vs 0.66), and it spans every subject domain. We recast cue-sensitivity as a construct-validity property of the score and propose reporting a cue-robustness interval, the manipulable range with its bootstrap CI, via a cheap black-box protocol. Where cues are unavoidable, this interval reports the reliability that a single accuracy number conceals.