Measuring the Reliability of Natural Language Autoencoders

**Accepted paper — no in-person presentation.** This paper remains accepted. No in-person poster presentation is scheduled. Natural language autoencoders (NLAs) have recently been proposed as an interpretability primitive for large language models, mapping residual-stream activations to natural-language descriptions and back. Because the activation verbalizer (AV) samples stochastically, repeated verbalizations of the same activation yield different descriptions and different reconstructed vectors, creating a measurement reliability problem that has not been characterized. We evaluate NLA reliability across four conditions --- PRISM, Bias in Bios, MMLU-Choice, and MMLU-NoChoice --- using Gemma-3-12B layer-32 activations and $K{=}12$ stochastic AV samples per activation. Applying Generalizability Theory (G-theory), we decompose reconstruction-consistency variance into signal attributable to the activation and noise attributable to AV sampling. Across most conditions, fidelity G-coefficients exceed 0.90 at $n'{=}1$ sample and consistency G-coefficients reach 0.90 at $n'{=}3$, yielding concrete sampling recommendations for practitioners. A cautionary exception arises in MMLU-Choice, where high within-item consistency coexists with near-zero item-discriminability and a 20-point drop in linear probe accuracy.