Gaze Attention: Selective Visual Attention for Multimodal Large Language Models

Multimodal large language models (MLLMs) attend to all visual tokens at each generation step, so the cost of reasoning over images and videos grows with the full visual context. Humans are more efficient: rather than processing an entire scene, they selectively fixate on the regions relevant to their intended description. In this work, we introduce Gaze Attention, a novel mechanism that enables MLLMs to selectively attend to task-relevant visual regions during generation. Specifically, we spatially group visual embeddings—stored as key-value caches—into compact gaze regions, each represented by a lightweight descriptor. At each decoding step, the model dynamically selects the most relevant regions and restricts attention to them, reducing redundant computation while enhancing focus. To mitigate the loss of global context caused by localized attention, we further propose learnable context tokens appended to each image or frame, allowing the model to maintain holistic visual awareness. Extensive experiments on image and video understanding benchmarks demonstrate that Gaze Attention matches or surpasses dense-attention baselines while attending to up to 90\% fewer visual KV entries, reducing attention FLOPs by ${\sim}81\%$ and KV-cache memory by ${\sim}79\%$.