Look Before You Write: Counterfactual Reward Shaping for Visual Grounding in Reinforcement Learning for Vision–Language Models

Reinforcement learning with verifiable rewards (RLVR) is useful for vision–language model post-training, but final-answer verifiers reward correctness without checking whether visual information was used. This allows GRPO-trained VLMs to exploit language priors rather than visual evidence. To this end, we propose Counterfactual Visual GRPO, a minimal reward-shaping modification that compares each rollout on the real image with a paired rollout on a corrupted counterfactual image. Both are scored by the same verifier, and the shaped reward is $r_i = r_i^{\mathrm{real}} - \lambda r_i^{\mathrm{corr}}$. The corrupt rollout receives no gradient; it only serves as a per-prompt baseline that discounts answers reachable without reliable pixels. We implement the method using Qwen3-VL-30B with LoRA in a Tinker-based GRPO pipeline and evaluate on Geometry3K. Across three seeds, Counterfactual Visual GRPO is comparable to but does not outperform vanilla GRPO: vanilla reaches 43.0 ± 2.1% held-out accuracy while ours reaches 42.8 ± 0.5%, and a paired bootstrap over prompts pooled across seeds finds no statistically significant difference (gap −0.2 points, $p = 0.90$). A flip-rate diagnostic nonetheless shows that the grounding signal is active: blurring the image changes the verifier outcome on 37% of held-out prompts and drops accuracy from 42.3% to 9.0%. Future work includes scaling the method to more data and compute, as well as evaluating the method on more domains.