Understanding Stochastic Reasoning Dynamics in LLM Judges through Markov Models

Large language models (LLMs) are increasingly used as automated judges, yet conventional evaluation focuses primarily on final verdicts rather than on how reasoning evolves across repeated executions. We introduce a framework for analyzing stochastic reasoning dynamics in LLM judges by representing generated reasoning trajectories as empirical Markov processes over discrete semantic states. Our methodology separates deterministic verdict generation from stochastic rationale generation, enabling variability in generated reasoning to be analyzed while keeping the underlying evaluation decision fixed. Controlled semantic perturbations produce evaluation instances with known fault labels, and the resulting trajectories are characterized using transition entropy, absorption time, state revisit rate, split-half distance, and complementary distributional comparisons. Experiments on two software-engineering benchmarks reveal consistent structural differences across in-sample, out-of-sample, and sensitivity regimes. Out-of-sample reasoning follows less repetitive yet less reproducible trajectories despite nearly identical trajectory lengths, whereas increasing the number of sampled trajectories improves structural coverage and estimation stability. These results demonstrate that stochastic reasoning dynamics provide complementary information beyond final evaluation outcomes, offering a principled perspective for analyzing robustness and reproducibility in LLM-based evaluation.