Mechanistic Analysis of Safety Disparities Across Vision and Text Modalities in Vision-Language Models

Modern large language models (LLMs) are fine-tuned for safety and helpfulness, enabling them to refuse harmful or sensitive requests while complying with benign ones. However, capability gaps between text and image modalities leave vision-language models (VLMs) more susceptible to a diverse array of jailbreaks. In this work, we use mechanistic interpretability (MI) techniques to determine why the vision modality introduces such vulnerabilities. We find that although images and text share a geometrically similar refusal direction, harmful images have weaker projection along this shared direction. Using sparse auto-encoders (SAEs), we show that harmful images fire safety-relevant features less frequently than their text counterparts. Most notably, we find that a refusal vector learned from text inputs ablates image refusal almost entirely, whereas activation steering in the reverse direction has substantially weaker effect, indicating that the text refusal signal is both stronger and more resistant to override. Taken together, these findings highlight a safety-specific brittleness in the vision modalities of VLMs. Broadly, this work underscores the need for vision-specific safety fine-tuning rather than inheriting safety mechanisms directly from text pipelines.