Dual-Stream Decoding for Accelerated Large Language Models

Autoregressive language models are trained to generate text one token at a time, causing inference latency and cost to scale linearly with output length. However, modern large language models often exhibit semi-autoregressive predictive capabilities, frequently aided by speculative decoding or other multi-token prediction methods. In contrast, discrete diffusion models promise parallel text generation but fundamentally struggle to model sequential correlations due to their reliance on mean-field approximations, effectively ignoring the causality inherent in natural language. We introduce $\textbf{Cross-Regression}$, an approach aimed at achieving true hybridization of autoregressive and discrete diffusion sequence modeling. Cross-Regression is a decoding algorithm that exploits latent information about future tokens embedded in pretrained transformer hidden states via a dual-stream construction: a $\textbf{control stream}$ that preserves exact autoregressive probabilities, and a $\textbf{predictive stream}$ that proposes parallel candidates. Cross-Regression uses residual $\textbf{energy guidance}$ to refine predictions without discarding intermediate computation, while enabling early acceptance analogous to speculative decoding. The method can operate in a $\underline{lossless}$ accelerated mode that preserves the original model distribution, or in a faster $\underline{lossy}$ mode with controllable deviation. Experiments demonstrate substantial speedups in text generation, reducing the number of sequential forward passes by up to $\textbf{5+}$ times on models with up to $\textbf{70B}$ parameters, while remaining compatible with KV caching and without requiring prohibitively expensive training.