Do LLMs Encode Event Co-Evolution Beyond Textual Similarity?
Chongrui Ye ⋅ Yuxiang Liu ⋅ Haofei Yu ⋅ Jiayu Liu ⋅ Shanyong Wang ⋅ Jiaxuan You
Abstract
Whether large language models encode how real-world events co-evolve, beyond the textual similarity of their descriptions, is an open question. Existing event-relation benchmarks derive labels from human annotations of event-mentioning text and are therefore tied to textual similarity, while event-forecasting benchmarks measure individual outcomes rather than inter-event structure. We introduce \textsc{OpinionBench}, an event-relation benchmark over prediction-market events with ordinal edge labels from change-point synchrony of probability time series. Text enters only as a TF-IDF candidate filter, and at scoring time a partial-rank correlation removes the effect of lexical and held-out semantic similarity, so a model earns credit only for signal beyond text. Because markets continuously produce timestamped events, the benchmark refreshes with post-cutoff data, mitigating training-set contamination. After controlling for textual similarity, TF-IDF collapses to near-zero partial-$\rho$ and the strongest dense-embedding baseline reaches only $0.11$, while frontier LLMs reach $0.21$. The advantage scales within model families and shows no detectable memorization on post-cutoff events. We report it as a robust but modest effect, conditional on a TF-IDF-selected candidate pool.
Chat is not available.
Successful Page Load