When Are Routing Weights Actionable? Causal Probes for Interpreting Block Attention Residuals

Architectures that expose internal routing weights promise a more actionable form of mechanistic interpretability: rather than inferring information flow indirectly, one might inspect a routing tensor and decide which pathways matter. We test this promise in Block Attention Residuals (Block AttnRes), introduced by Chen et al. (2026), which replace fixed additive residuals (He et al., 2015) with a learned softmax over earlier depth-source representations. This surfaces cross-layer routing as an inspectable tensor in the forward pass, making Block AttnRes a tempting interpretability target: information flow normally inferred indirectly is now directly observable. We ask whether such exposure suffices for mechanistic interpretation. We probe two same-scale (0.6B) Block AttnRes checkpoints under identical routing-ablation interventions: a vanilla Qwen3 inference-wrapped through a deterministic recency-bias schedule that the codebase admits as a routing-equivalent loading path, and a Block AttnRes Qwen3 trained from scratch with routing as part of optimization. The wrapped baseline’s routing weights are content-independent and reproduce the schedule’s analytic prediction. The trained AttnRes checkpoint instead exhibits three localised routing motifs: an embedding-source pathway through early-layer MLP, a current-state pathway through early-layer attention and MLP, and an older-history pathway through late-layer attention. Beyond this stratification, we find a sharp dissociation between average routing mass and causal importance: in both sublayers, the largest mass slice is not the largest causal contribution, and one source family carries appreciable mass with no detectable causal role under intervention. Our results identify a failure mode for actionable interpretability: architectural visibility can produce plausible but non-actionable explanations. Routing weights should therefore be treated as intervention targets and hypothesis generators, not as explanations until causally validated.