Your Judge Is a Confound: Evaluator and Attack Choice Distort Jailbreak-Safety Measurement for Fine-Tuned LLMs

Red-teaming reports for fine-tuned LLMs headline a single attack-success rate (ASR), implicitly treating it as a property of the model. We show it is governed as much by how it is measured. Across 25 configurations (five open-weight families × Base FP16, 4-bit, LoRA, QLoRA, and full fine-tuning), attacked with black-box, white-box, and one replayed multi-turn jailbreak over HarmBench and JailbreakBench and scored by three automated evaluators calibrated against 750 blind annotations (412-sample clean consensus), two measurement choices change the reported conclusion. First, evaluator choice inflates measured black-box Delta ASR roughly threefold and reorders which methods look most harmful: against human consensus, GPT-4o-mini runs at a 45.7% false-positive rate (still 39.2% under a StrongREJECT rubric) versus 8.6% for a fine-tuned classifier. Second, the black-box suite and a gradient-guided white-box attack, AutoDAN, give directionally opposite verdicts for the same method (black-box-suite vs. AutoDAN interaction, p=1.6 x 10^-7), and AutoDAN’s apparent LoRA “protection” is seed-noise (-30.0 ± 24.1 pp), not a deployable defense. Only once these are controlled does a model-level signal emerge: re-training a three-family subset at a fixed seed to remove a seed/recipe confound, a TOST procedure establishes that LoRA and full fine-tuning degrade black-box safety equivalently (±5 pp) and seed-stably, while QLoRA’s higher variance prevents certifying it equivalent at the reduced sample size. This residual signal is scoped, not universal. It is a behavioral equivalence in a benign one-epoch regime, not a verdict on adaptation algorithms in general. We argue safety claims about fine-tuned LLMs must jointly report adaptation method, attack family, seed/recipe, and a human-calibrated evaluator, and release a protocol for doing so.