Measuring Representation Reorganization from Language Models to Vision-Language Models

When a language model is adapted into a vision-language model, a single representation score can lead to the wrong analysis decision. We compare three paired model families on three captioned-image datasets and ask three separate questions: does image/text mismatch change sparse latent activations, do the language model's concept neighbors remain similar in the vision-language model, and are input conditions separable in dense hidden-state space? The answers diverge. In this grid, SmolVLM2 has the lowest sparse movement but the strongest neighborhood rewriting; Qwen2.5-VL preserves the most language-model neighbors; PaliGemma2 gives the clearest dense condition separation. Low sparse movement therefore does not imply preserved language-model concept neighborhoods. Our contribution is to show that these probes can disagree and are therefore not interchangeable: sparse movement tracks mismatch sensitivity, neighbor survival tracks concept-neighborhood preservation, and dense separation tracks condition geometry.