Evaluation Awareness in Language Models: Mapping Behavior to Mechanism

Recent work has shown that large language models (LLMs) can internally distinguish between evaluation and deployment contexts, a phenomenon known as evaluation awareness, raising concerns about the reliability of safety benchmarks. We mapped prefix-conditioned SAE activation differences in LLaMA 3.1 8B and Gemma 3-1B. We constructed matching eval-like and deployment-like prompt pairs using five different data sets spanning honesty, rule-following, and risk detection. Then, we used pre-trained sparse autoencoders (SAEs) on intermediate model representations to identify interpretable features that co-activate with the eval-awareness signal. We found that when LLaMA 3.1 8B detects an evaluation prompt, it suppresses features relating to human consequences and simultaneously amplifies formal, structured knowledge, while Gemma 3-1B amplifies features relating to an LLMs inner mechanics. This research can help us understand which features are being altered within a model due to evaluation awareness.