When Model and Judge Are Both AI: Decomposing the Reliability of Open-Ended Math Grading

When one AI model writes a mathematical proof and another grades it for partial credit, the now-routine way to evaluate open-ended reasoning at scale, a leaderboard of the scores invites a doubt: is it ranking which model reasons better, or which one the judge favors? We turn this into a measurement question, does the grading pipeline (which judge, which rubric) decide which model \emph{wins} or only how \emph{high} it scores, and answer it with a \emph{fully crossed} generalizability-theory (G-theory) decomposition of $15$ models $\times\,15$ USAMO proof problems (2024--2026) $\times\,3$ rubrics $\times\,3$ independent-vendor judges $\times\,3$ replications ($6{,}048$ gradings), placing between-model signal, grading pipeline, and content on one common variance scale. The answer is a clean decoupling: grading choices move a model's absolute \emph{score} but barely touch its \emph{rank}; what reorders models is the content it is tested on. Its warning half is the less appreciated one. Grading choices move the absolute \emph{level}, a holistic rubric shifts scores by $+0.76/7$ and the judges run ${\sim}0.5/7$ lenient against a human standard, so absolute scores are choice-sensitive and not human-anchored: a caution for score thresholds, reward signals, and progress claims. The \emph{order}, by contrast, is robust: swapping the judge (three vendors, one held out of screening) or the rubric preserves the ranking (Spearman $\rho \geq 0.98$); the reordering force is content, the model$\times$item interaction ($v_{pi}{=}0.023$), ${\approx}29$--$46\times$ the grading-pipeline interactions ($20\%$ of total variance against under $1\%$). Reproducible is not choice-invariant: the ranking is dependable across problem samples ($G{=}0.95$, $\Phi{=}0.90$) even as its level stays choice-sensitive, because \Phicoef\ reproduces the judges' own scale rather than certifying it. Only \emph{items} raise ranking reliability, so a dependable order costs ${\sim}15$ grading calls per model against ${\sim}27\times$ for the reflexive $3\times3\times3$ design: spend the grading budget on problems, not judges, rubrics, or re-samples.