Latent Linguistic Relations During Diffusion Language Model Denoising

Recent advances in diffusion-based language models motivate understanding how these models build linguistic structure during denoising. Diffusion language models generate text by iteratively denoising a partially masked sequence, exposing intermediate states that autoregressive models lack. We use these states to ask when, during denoising, a small diffusion model (DiffuGPT-S) forms interpretable linguistic structure, evaluating on held-out natural sentences from a gold Universal Dependencies (UD) treebank. We report three findings. (1) DiffuGPT-S contains attention heads that track grammatical relations: the strongest object→verb head (L4 H0) predicts the correct receiver with 75.5% held-out accuracy, and subject→verb, adjective→noun, and determiner→noun heads also transfer. A timestep-resolved analysis shows this structure begins to form before full token resolution—L4 H0 predicts the object→verb receiver with 41.6% accuracy while both relation endpoints are still masked, rising to 63.9% once one is revealed. (2) A depth wise division of labor: linear probes recover part-of-speech (token-class) information most accurately from shallow and middle layers, while a logit lens recovers exact token identity far better from deep layers (40.2% at layer 10 vs. 1.2% at layer 0); direct logit attribution and matched ablation show selected relation heads contribute relation-specific support for the target token. (3) Attention structure evolves over diffusion time, with deep-layer heads becoming reliably more selective while shallow layers are heterogeneous. Together these give a quantitative, time-resolved account of how linguistic structure forms during diffusion-LM denoising.