Benchmark Items as Measurement Instruments: Differential Item Functioning Across Response-Derived LLM Regimes

Large language model benchmarks usually compare systems with aggregate accuracy, although the underlying observations are model-by-item responses. We study whether benchmark items function comparably across model populations after conditioning on overall performance. Using 4,409 models and 5,760 BIG-Bench Hard (BBH) items, we construct separate model and item holdouts: response-derived groups are learned from 4,033 group-discovery items and 3,179 discovery models; Mantel--Haenszel differential item functioning (MH-DIF) is estimated on 1,727 disjoint items; and item-pool consequences are evaluated on 1,230 held-out models. We introduce score-residualized response clustering (SRRC), which clusters normalized residual domain profiles after removing a cubic function of overall score, then assigns held-out models with frozen centroids. The selected two-regime solution is balanced, partially aligned with architecture metadata, stable across initializations (mean adjusted Rand index \(=0.998\)), and weakly associated with overall score (\(R^2=0.0027\)). Under a descriptive ETS-style DIF categorization, 746 of 1,727 items (43.2\%) fall into the largest-effect category (Category C) while all 20 score-matching strata remain usable. The held-out Regime 1 - Regime 2 gap is only $+1.33$ percentage points over all test items, but becomes $+13.63$ or $-11.38$ points on item pools favoring the respective regimes. Discovery and evaluation item effects correlate strongly ($r=0.941$), and domain- and difficulty-matched random pools do not reproduce these shifts. These results show that item composition can materially alter regime-level comparisons even when aggregate performance is similar. They support response-derived DIF as a diagnostic for benchmark comparability within this BBH extraction, while not establishing benchmark-wide invalidity or causal model types.