Attention as a Dynamic MLP: A Lens on the Expressivity-Efficiency Gap

Subquadratic and linear-attention sequence mixers cut the per-token cost of attention, but they often saturate below full quadratic attention and leave a persistent quality gap. This paper asks a simple question behind that gap: when a mixer trades cost for a cheaper score computation, what function-class capability does it give up? We answer it with one analytical tool. An autoregressive attention head is a two-layer MLP whose weights are instantiated from the context, and whose scores are a width-$t$ hidden layer. In this view the nonlinearity on the scores does routing: it selects an input-conditioned subset of a context-built memory pool. We give two self-contained results. First, replacing the gating nonlinearity with the ungated normalization used by linear attention collapses this routing: the readout is forced to use the entire pool with no input-conditioned selection, which is exactly the capability subquadratic mixers tend to lose. Second, the same gating admits a clean complexity account: activation is a total-variation contraction on the memory pool, so the pool size gives an explicit bound on the expressive complexity of the head. We are careful about scope: this complexity is function-class expressivity, not compute time or space. A small matched-budget study illustrates the predicted gap.