Localizing and Repairing Context-Failures in Audio-Language Models

As language models become increasingly multimodal, they offer the promise of using textual context to improve the decoding of visual and audio signals. In impaired speech recognition, however, clinical context intended to help recognize atypical speech can instead make frozen audio-language models perform worse. We study this failure in Gemma-4-E2B on the Speech Accessibility Project (SAP) rated subset. Adding full clinical context raises Word Error Rate (WER) with the largest damage on utterances the audio-only prompt already transcribes cleanly. Length-matched controls show that much of this degradation is triggered by non-instructional prompt content rather than by clinical semantics. We localize a repairable part of the failure with Matched Residual Patching (MRP). At a middle decoder layer, replacing the model's internal state for the final input token with the corresponding audio-only state closes 57-60% of the WER gap under bidirectional patching. Building on this localization, we introduce self-contrastive residual alignment (SCRA), an inference-time edit that predicts and removes the prompt-induced residual shift. A two-pass variant, SCRA2p, constructs the contrast from the same input with the non-instructional block stripped, requiring no cached paired dataset. The full SCRA method learns a low-rank linear corrector which, at rank 128, matches the MRP ceiling, closes 60% of the gap between audio-only and full clinical-context prompting, and slightly but significantly improves over the audio-only baseline on difficult samples. These results identify a repairable residual-stream failure mode in frozen audio-language models and provide a one-pass residual-edit primitive for mitigation.