Fully-Recurrent Test-Time Training
Zefan Cai ⋅ Hao Tan ⋅ Junjie Hu
Abstract
KV Binding test-time training turns a Transformer cache into an online associative memory, but the usual form binds each layer's keys to that same layer's values, leaving shallow binding states trained on shallow targets even when useful long-context information sits near the top of the network. We introduce \method{}, a recurrent residual variant that keeps layer-specific key addresses while using the final layer's values as a shared write target, so for chunk $t$ and layer $\ell$ the binding state is trained on $(K_t^\ell,V_t^L)$. This makes the binding fully recurrent: the learned map is a test-time state carried across chunks rather than an auxiliary signal used only within one. During generation, layer $\ell$ reads only the previous state $B_{t-1}^\ell$, injects the retrieved final-layer value residual into attention, and writes $V_t^L$ only after the final layer has computed it, preserving causality while letting earlier layers retrieve deep value information from past chunks. The current evidence is a matched-run RULER analysis: the residual-TTT implementation (\texttt{vgap95}) has positive deltas in 11 of 18 RLACT pairs, including all six complete 124M diffusion pairs, while outcomes are mixed at 760M and in other recurrent families. This supports a scoped design principle rather than a blanket win: depth-local keys can address a shared deep value state, but the benefit depends on recurrent update and projection configuration. We also specify matched protocols for testing the same interface in novel view synthesis and autoregressive video generation.
Chat is not available.
Successful Page Load