Inference-Time Consensus for Mitigating Hidden Behaviors from LLM Fine-Tuning

Language models are increasingly fine-tuned on data aggregated from many external sources, and only a small amount of poisoned data is enough to install hidden, source-specific misbehavior. In this setting, useful behavior may be shared across sources, whereas poisoned behavior is source-specific. We first show that standard training-time defenses cannot exploit this structure. Distilling a single model against the sources under KL regularization provably only dilutes source-specific behavior rather than removing it. Instead, we fine-tune a separate model on each source's dataset and compose these models directly at decoding time, using a token-wise minimum that keeps only the next-token behavior the sources agree on, so the aggregate stays safe unless every source is poisoned. We further relax exact agreement to tolerate partial support across sources and different surface expressions of the same intention. Across controlled poisoning tasks, subliminal learning, and emergent misalignment, consensus decoding suppresses source-specific behavior while preserving shared behavior, including cases where union training and weight averaging retain the unwanted behavior.