MMIB: A Mechanistic Interpretability Benchmark for Vision-Language Models

Large vision-language models (VLMs) have achieved strong multimodal reasoning performance, yet their internal decision-making processes remain unclear. Mechanistic Interpretability (MI) aims to uncover the circuits and representations responsible for model behavior. While benchmarks such as mechanistic interpretability benchmark (MIB) enable rigorous evaluation of MI methods in large language models, no standardized framework exists for multimodal architectures, so interpretability studies in VLMs remain difficult to compare and validate. In this work, we introduce Multimodal Mechanistic Interpretability Benchmark (MMIB), a benchmark for evaluating MI methods in VLMs. MMIB extends the dual-track framework of MIB to multimodal settings on two different models. The first track evaluates circuit localization using the Circuit Performance Ratio (CPR) and Circuit-Model Distance (CMD) metrics under modality-specific counterfactual intervention. The second track evaluates causal variable localization using interchange intervention accuracy (IIA). We also study a metric that computes the cross-modal information flow. We introduce image- and text-based counterfactual inputs, enabling structured causal validation across both modalities. Our results show that across the settings we evaluated, the faithfulness metrics show promising consistency and interpretability in multimodal models. These findings demonstrate that mechanistic benchmarking can be extended beyond language-only models to heterogeneous multimodal systems.