The Fragility of Reasoning-Importance Attribution for Pruning Language Models

Attribution-guided pruning, which scores components by their causal contribution and protects the important ones, is increasingly proposed for compressing reasoning language models. We ask a prior question: is the reasoning-importance signal these methods consume reliable to measure? Across four small reasoning models, it is fragile in three ways that change pruning decisions. First, scoring final-answer tokens, saturates in every chain-of-thought model we test (answer-token NLL 10−4–10−3) because the model states its answer inside its own reasoning; guiding head pruning with this target costs 27.6 and 32.2 accuracy points relative to scoring reasoning tokens on the two models we power with five seeds, while perplexity barely moves and even reverses sign at 7B. Second, causal importance is not localizable to single units: single-neuron effects sit at the measurement floor, their cross-split ranking is noise, and they cancel: a group’s ablation effect disagrees in sign with the summed effects of its members in 90% of groups. Reproducible importance emerges only at groups of roughly 100 neurons. Third, contrastively isolated “reasoning-specific” importance is not separable from generic importance, and protecting it is worse than protecting raw importance. We additionally pre-registered the hypothesis that distillation- induced concentration of importance predicts the pruning cost; it is not supported, and since our grid confounds concentration with scale we report it as unresolved. The result is a short set of measurement rules: score reasoning tokens, attribute at coarse granularity, and evaluate with accuracy rather than perplexity.