Characterizing Learning-Resistant Preferences

What can prevent a preference from being learned during RLHF? We characterize these learning-resistant preferences with respect to three factors. First, we find that RLHF methods are highly sensitive to the initial data likelihood under the language model: on controlled tasks, the probability margin between the likelihood of the chosen response and rejected response strongly predicts learning. Second, when adding the complexity of real-world, noisy preference data, the probability of the chosen response often increases only negligibly relative to the rejected response--far less than the improvement possible with cleaner datasets or supervised fine-tuning. Finally, we find that variance in human preferences causes labels from single annotators to exhibit poor generalization. Together, these three obstacles---sensitivity to initial likelihood, PPO's relative difficulty learning from noisy data, and poor generalization when there is inter-rater disagreement---hamper RLHF methods, sometimes making them learn to prefer the rejected response over the chosen response on over half of training examples. These results suggest that the effectiveness of RLHF methods depends strongly on the noisiness of the data and its likelihood under the reference model.