Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction
Abstract
Predicting item difficulty is a central challenge in educational assessment, where accurate estimates are critical for fairness and effective test construction. Existing methods are either expensive to scale or limited in predictive power, even with recent Large Language Model(LLM) related methods. Meanwhile, Large Reasoning Models (LRMs) expose detailed reasoning traces that resemble human problem solving, creating new opportunities for difficulty modeling. We propose a new perspective that treats item difficulty as an observable consequence of problem-solving burden. Building on this formulation, we introduce a structured framework that converts LRM reasoning traces into sentence-level episode sequences using a cognitively grounded taxonomy and then derives compact process features for downstream prediction, which is named Epi2Diff (Episode to Difficulty). This produces a structured process-based representation of each assessment item. Across four real-world human difficulty datasets, our method consistently outperforms various methods, including fine-tuning small encoder-only language models, LLM In-Context Learning, and LLM Supervised Finetuning with full parameters or LoRA. Epi2Diff delivers an 8.0% relative gain in classification and a 24.4% reduction in RMSE for regression against strong baselines. We also find that harder items are associated with more effortful and iterative reasoning dynamics. Moreover, our method not only predicts the difficulty value, but it also provides a new perspective on the explanation of the predicted difficulty based on the episode transition of LRM responses. These results suggest that structured reasoning dynamics are both predictive and interpretable, and provide a new lens for understanding the utility of LRMs in educational measurement.