Auditing Prompt Injection Defenses Before They Reach Users: What Strong Adversaries Reveal

Large language models are increasingly deployed in user-facing products that handle sensitive personal data while interacting with untrusted external content. This makes them a prime target for prompt injection attacks, in which an adversary injects malicious instructions into that content to hijack the model. Fine-tuning-based defenses mitigate these attacks by training models to follow benign user instructions and ignore injected ones, and are already shipping in production systems. Lacking formal guarantees, they are evaluated empirically against the most effective realistic attacks available at the time, often reporting near-zero attack success rates. We argue this is fragile: a defense that looks robust today may fall to a more sophisticated attack tomorrow -- after being deployed to handle users' emails or credentials. We therefore advocate auditing defenses under a deliberatively strong adversary before release, a principle long established in cryptography and differential privacy. Our key insight is that the fine-tuning process itself yields useful artifacts: intermediate checkpoints that serve as stepping stones for optimization-based attacks. Although unavailable to real attackers, the vulnerabilities they expose are genuine and exploitable by realistic adversaries. Instantiating our attack-agnostic framework with GCG and BEAST, we raise attack success rates against state-of-the-art defenses from near zero to as high as 96\%, revealing weaknesses missed by prior evaluations. Our results show that current evaluation practices can substantially overestimate robustness, and that auditing defenses under strong adversaries before deployment shortens the feedback loop and helps prevent attacks from causing real-world harm to users.