Structured Formats Collapse Refusal Representations: Evidence for Format-Dependent Safety in Instruction-Tuned LLMs
Abstract
Agentic structured formats (tool calls, function execution, system prompts) induce format-dependent safety representations: models preserve relative harmfulness ordering (AUC = 0.997 across 10 formats) but shift activation distributions so calibrated safety boundaries fail. Using representation probing, sparse feature analysis, and activation interventions across five models (1.1–2.7B parameters, four architecture families), we show native agentic formats suppress refusal-direction discrimination to 1.3–6.5% of baseline at the last-token readout site, while content tokens retain 65–76%. Format-token ablation restores discrimination monotonically (2.1% → 48.6%), establishing that learned format tokens drive the suppression. The precise signature: a direct-trained classifier achieves perfect AUC on tool_call format yet drops to 50% accuracy (random chance); per-format recalibration with five samples recovers >95%. Frontier validation: GPT-4o refusal drops 86% → 30% (p < 10−4 ); Gemini 2.5 Flash 95% → 45% (p = 0.006); Claude remains format-invariant (p = 1.0), showing the vulnerability is defensible.