Visual Memory Attacks Can Persist Through Context Masking

Vision-language models (VLMs) are increasingly deployed as autonomous agents that ingest untrusted content, including images. Recent work shows that an adversarial image can plant a hidden backdoor, where the model behaves normally until a chosen trigger makes it emit an attacker-chosen response (Schlarmann & Hein, 2026). While such attacks are concerning, a natural defence is to remove adversarial images from the context before a model acts, and the cheapest version of this masks the image's attention while reusing the prefix cache it helped build. We show that this is insufficient against an adaptive attacker. We develop Persistent Visual Memory Injection (P-VMI), which optimizes the image to route its signal into the key/value (KV) cache written by the surrounding turns, so masking the image at the trigger removes the direct path but not its cached effect. On Qwen3-VL-8B-Instruct, P-VMI reaches up to $\sim 90 \%$ target success in its strongest setting, and persists even under a stricter defence that shows the image once and masks it through every later turn. A cache-swap ablation localizes the effect to the KV cache, indicating that defences must discard or recompute the poisoned cache rather than merely mask attention.