SurveyAttentionBench: A Counterfactual Test of Survey Attention-Check Robustness in Web Agents

We propose SurveyAttentionBench, a controlled benchmark for evaluating whether autonomous web agents can complete online surveys and pass attention checks under varying observation exposure. By varying survey design, attention-check type, and interface metadata while holding tasks fixed, the benchmark distinguishes genuine visual/reasoning ability from success driven by exposed cues. The benchmark evaluates when agents succeed, why they fail, and whether their success reflects robust reasoning or reliance on interface-level shortcuts.