Interpreting Diffusion Language Models Over Denoising Time

Mechanistic interpretability has uncovered important internal structure in autoregressive language models, but comparable methods for diffusion-based large language models (dLLMs) remain underdeveloped. Unlike autoregressive models, dLLMs generate by iteratively denoising an entire sequence, making denoising time a central axis of computation. In this work, we study dLLMs with sparse autoencoders (SAEs), cross-layer transcoders (CLTs), timestep-aware tracing, and feature-level interventions. We train timestep-balanced feature models on Dream and LLaDA activations and use them to analyze how sparse features evolve over denoising trajectories. Our experiments show that SAEs recover interpretable features whose activations exhibit stable, task- and decoder-dependent temporal structure. In Dream case studies on sorting and poetry rhyme-cloze, these traces reveal features distributed across layers and denoising steps rather than concentrated only at the end of generation. Timing-controlled interventions show selective causal leverage: some middle-window sorting bundles strongly affect held-out exact match under particular decoders, while rhyme-cloze steering and broader transfer checks are much weaker. CLT population probes further show that cross-layer fanout edits can perturb denoising trajectories, but current CLT effects are not yet specific to top-ranked features. Together, these results establish a practical foundation for trajectory-level interpretability in dLLMs and support a calibrated claim: denoising time is an interpretable organizational axis, and some task- and decoder-conditioned feature bundles can be causally probed with matched timing controls. We will open-source our training code, SAE and CLT checkpoints, and temporal tracing visualization tools to facilitate future dLLM interpretability research.