Typed Slot Policies for Prompt-Injection Defense in Computer-Use Agents, and Why the Attribution Signal Is Faithful
Akshatha Arunkumar ⋅ Shivank Garg
Abstract
Computer-use agents propose structured actions over screens, DOM trees, and documents; untrusted content in those observations can hijack a sensitive action slot, such as an email recipient or a shell command. Page- or action-level defenses are too coarse: the same untrusted region is often legitimate evidence for one slot and an attack on another. We present ASP-Guard, a training-free runtime monitor that authorizes each action slot by its semantic type over a shared region-to-slot mask-diff attribution signal, escalating unknown slots to a human rather than failing open. On a visual prompt-injection benchmark the typed predicate catches or escalates every one of the landed attacks with zero silent allows, and Pareto-dominates an untyped sink-blocking rule that collapses from 1.00 to 0.33 as attack families diversify. We then ask why the attribution signal works: across 88 landing cases from three open models spanning two architecture families, activation patching shows the region mask-diff flags is the region whose residual activations causally mediate the injected payload (pooled Spearman $\rho=0.38$, 95% CI [0.28, 0.47], N=88), so the attribution signal is mechanistically faithful in a slot-conditioned form. The deployed single-action signal is itself degenerate single-step, which we show is a latent susceptibility that surfaces only in the action frame. We are explicit about the adaptive and text-only attacks that still slip past a per-action gate, and release our code and evaluation suite.
Chat is not available.
Successful Page Load