The dstate Measurement Problem: A Construct-Validity and Non-Stationarity Audit of Recurrent-State Capacity in 20 Linear Architectures
Zacharie Bugaud
Abstract
Linear recurrent architectures (Mamba/Mamba-2, RWKV, GLA, xLSTM, MesaNet, and their hybrids) advertise a nominal state-size hyperparameter $\dstate$ and an ordinal accuracy ranking on recall benchmarks. We argue --- and demonstrate --- that both behave as measurement instruments with serious, diagnosable validity failures, and we use AIMS's frame (interactive measurement, strategic optimisation, non-stationarity) to name them. \textbf{Pathology 1 --- Confounded instrument.} In a $1{,}064$-run controlled audit across $20$ architecture families and seven nominal state sizes, four widely-cited reference implementations (RWKV-6, RetNet, HGRN-2, GSA) are \emph{flat in $\dstate$}: the nominal knob is silently coupled to other backbone hyperparameters (head count, expansion ratio, slot count) and does not independently control capacity. The $d_{\text{state}}$ label has no consistent referent across the zoo. \textbf{Pathology 2 --- Construct invalidity.} The AR-capacity ranking is orthogonal to byte-level next-token loss (GLA, at chance on AR, is the strongest of six rows on LM loss). Treating AR as a proxy for downstream quality --- as the field implicitly does --- is a textbook construct mismatch. \textbf{Pathology 3 --- Measurement non-stationarity.} The small-scale ranking does not survive scale-up of the same recipe: at $\sim\!1\,$B parameters under controlled compute starvation, only xLSTM and RWKV-7 retain Associative Recall while MesaNet and Gated DeltaNet collapse. The construct ``recall capacity at recipe $R_1$'' is not the same construct as ``recall capacity at recipe $R_2$''. Capacity exponents have wide bootstrap CIs (the Mamba-2 95\% CI for the $\dstate$ scaling exponent contains zero) and should not be reported as laws. We package these diagnostics as \statebench, a measurement protocol with three explicit design commitments imported from psychometrics and causal inference: (1) \emph{masked targets} (loss only at construct-relevant positions, in the spirit of IRT item control); (2) \emph{capacity curves not scalars} ($\Kstar$ is a summary of a full curve, both released); (3) a \emph{disclosed, single, fixed recipe} with explicitly labelled scale-tier adaptations. We release all $1{,}064$ summary JSONs, task generators, configs, a state-memory normalisation table, and a leaderboard scaffold whose protocol is designed to resist recipe-shopping, vocabulary inflation, and target leakage --- the strategic-optimisation pathologies AIMS targets.
Chat is not available.
Successful Page Load