Feature Evolution in Diffusion Language Models

Diffusion language models generate text through iterative denoising rather than left-to-right prediction, introducing a diffusion-time axis along which linguistic representations may be organized. We investigate whether this temporal structure gives rise to stage-wise feature dynamics in a pretrained diffusion language model. Using sparse autoencoders trained on Dream-v0-Base-7B, we analyze feature activations across layers, token positions, and denoising timesteps. Contrary to a simple stage-wise account, we find that SAE features rarely separate cleanly into early- or late-stage behaviors, and that token classes such as punctuation, mathematical symbols, function words, and content words are not consistently revealed at distinct diffusion stages. Instead, we identify recurring activation geometries: features that track masked token positions, features that activate near token commitment boundaries, and features that selectively respond to particular unmasked tokens. These results suggest that linguistic computation in diffusion language models is temporally distributed, with feature activity reflecting continual refinement across denoising time rather than sharply separated phases of syntax, structure, or semantics.