Don’t Blink: Evidence Collapse during Multimodal Reasoning
Abstract
Reasoning VLMs expose a practical interpretability question: can visual-grounding signals measured across an entire generated reasoning trace be turned into actionable reliability diagnostics? We characterize this as evidence collapse: across the three reasoning VLMs studied here, attention to visual tokens, including manually annotated evidence regions, declines over the reasoning trajectory. This creates a gap between interpretability and actionability: the model's text can appear confident while its internal use of visual evidence has already decayed. Full-response entropy is the strongest transferable monitor we test, beating answer-span entropy and vision-based trajectory features on standard discrimination and calibration metrics, but it is text-only and cannot tell whether confidence is visually grounded. Adding vision naively does not close this gap because the relationship between visual disengagement and error is task-conditional. Low-entropy, visually disengaged predictions are hazardous on tasks requiring sustained visual reference, yet benign or inverted on symbolic tasks where early visual extraction suffices. A single global fusion rule cannot represent this sign flip, explaining why fusion degrades transfer. A targeted vision veto makes the interpretability signal actionable: it reduces selective risk where the interaction predicts danger and raises it where the same signal predicts safety. Overall, visual grounding is an actionable but task-conditional interpretability signal: whether disengagement should trigger intervention depends on the structure of the task, not on the model alone. Code will be provided after acceptance.