Interpreting Prefill Awareness

Frontier models can recognize when prior assistant messages have been tampered with, a capability termed prefill awareness. This is a concern for AI safety methods that rely on prefill attacks, such as alignment evaluations and control interventions, as models might learn to game them. To this end, we explore how Opus 4.5 gets this signal by ablating high-level properties of the prefill attacks and try to reduce the rate of verbalized awareness using this information. We find that the main signals the models use often come from superficial stylistic differences, and this awareness drops significantly when the attacker model is from the same family. Accounting for stylistic differences reduces Opus' verbalized awareness, on BigCodeBench with prefills from Gemini 2.5 Pro, from a balanced accuracy of 100% to 54.2%. We also ask what open-source models are capable of and observe that, while they don't exhibit verbalized awareness like Opus 4.5, we can train a linear probe to classify prefilled responses and find these probes generalize to held-out models and datasets.