JARVISC: Jailbreak Analysis via Rubrics & Variable Input Seed Complexity

Existing jailbreak benchmarks evaluate whether attacks elicit unsafe model behavior, but often treat harmful seed prompts as interchangeable units of intent or harm category. This ignores a key prompt-side factor as prompts vary in how precisely they define successful task completion. As a result, vague requests and detailed, constrained prompts are evaluated equivalently, allowing generic non-refusing responses to be counted as successful attacks. We introduce structural complexity $C(p)$, a prompt-level metric that quantifies how explicitly a prompt specifies the requirements for success. Analyzing existing benchmarks shows that they concentrate in low-complexity regions and underrepresent prompts where success requires satisfying concrete, structured requirements. To expand coverage, we develop an attack-guided prompt evolution pipeline that generates 200 evolved prompts using attack resistance as the selection signal. We further show that standard evaluation protocols break down in this regime, where responses can appear harmful and non-refusing while failing to meet the prompt's actual requirements, and propose a prompt-specific rubric evaluator that measures jailbreak success through requirement-level satisfaction rather than coarse compliance signals. We evaluate representative jailbreak methods against multiple target models under binary LLM-as-a-Judge, StrongREJECT, and prompt-specific rubric evaluation across existing datasets and our expanded prompts. We find that existing jailbreak methods perform poorly on our evolved benchmark under rubric evaluation with the strongest attack reaching only 24\% success, and most methods falling below 20\%. We also find that standard evaluators substantially overestimate success, with binary evaluation exceeding rubric-verified success by 65.8 points and StrongREJECT by 31.7 points on our evolved benchmark. These results show that jailbreak evaluation must account for prompt structure, as attacks that cross the refusal boundary often fail to satisfy the concrete requirements needed for task completion.