History-Conditioned Preference Optimization for Multi-Turn Language Interaction

Multi-turn interaction can generate supervision that is unavailable from isolated prompt-response pairs because clarification, repair, commitment tracking, and evolving user goals become observable only over the course of interaction. Standard preference optimization usually attaches supervision to prompt-response pairs, which leaves this interaction-generated signal unmodeled. We study that gap through Temporal-DPO (T-DPO), a history-conditioned preference framework in which response utility depends on the current turn and a latent session state summarizing the preceding interaction trajectory. We evaluate this idea with an offline preference-structure analysis on the PRISM Alignment Dataset using a frozen SmolLM2 instruct/base model pair across three settings that vary context length and sample size. Across all three settings, history-conditioned scoring improves turn-3-and-beyond Spearman correlation relative to prompt-local scoring, indicating that later-turn preference judgments depend on accumulated interaction history. Aggregate pairwise accuracy remains comparable across scoring methods, and temporal-versus-shuffled-history gaps are small. These results suggest that interaction history carries preference information that prompt-local formulations do not recover, particularly in later turns. We introduce a history-conditioned preference objective together with an offline framework for measuring session-dependent preference structure in multi-turn language interaction.