Who Is Left of Whom? Tracing Spatial Evidence and Role Binding in Relative-Position Reasoning

Reasoning about relative positions between objects is a fundamental component of spatial reasoning, but current evaluations often rely on individual-instance accuracy, which does not necessarily imply robust understanding of relations. For example, a model may correctly answer "Is the cat to the left of the dog?" while failing on "Is the dog to the left of the cat?", or struggling to remain consistent when the positions of two objects are swapped. In this paper, we study object-level relative-position reasoning in VLMs and LLMs across three matched settings: VLMs with images, VLMs with textual scene descriptions, and their LLM backbones with the same descriptions. Through activation patching, we find that spatial reasoning proceeds in stages. We then extract object-centered spatial/order IDs and use source-to-query patching and steering to confirm the link between visual or textual evidence to query-side object representations and predictions. Additionally, we identify a stable target/reference role direction in hidden states, which causally modulates role-sensitive judgments. Our analyses reveal two interacting components: object-centered spatial/order IDs, which encode the source-derived positions of queried objects, and target/reference role binding, which determines the direction of relational comparison. These components are not only diagnostic but also actionable: steering query-object spatial IDs changes the model's spatial belief, while synthetic role-direction steering improves paired consistency on real images without retraining.