Delphi-Psychometric Evaluation of 3D Generative Models: Combining Structured LLM Elicitation with Item Response Theory

Benchmarks for 3D generative models aggregate automated metrics without principled analysis of reliability, item difficulty, or judge consistency. We introduce a framework combining three previously disconnected methodologies: Delphi-structured multi-round LLM judging (for scalable, calibrated elicitation), Item Response Theory (IRT) and Many-Facet Rasch Modelling (MFRM) (for psychometric analysis of the resulting data), applied to Hi3DBench, a public benchmark spanning six image-to-3D generation models and 510 prompts across five quality criteria. We report five empirical contributions: (E1) the five object-level criteria load onto a single latent quality factor (1-factor CFA: CFI=0.95, RMSEA=0.09), validating within-level aggregation; a cross-level analysis adding Hi3DEval’s four material-scoring dimensions confirms they form a psychometrically distinct factor (2F CFI=0.883 vs. 1F CFI=0.641, ∆BIC=−1,763), so object-level and material-level scores should not be combined; (E2) evaluation criterion accounts for 76% of score variance—dwarfing model (1.4%) and prompt (1.0%) contributions, revealing that automated metric scale artefacts dominate the benchmark; Delphi LLM judging on a uniform 0–9 scale cross-validates this finding by making model quality the dominant source (37%), and achieves strong reliability (ICC= 0.81–0.88; inter-judge SD reduced 68% across Delphi rounds); (E3) in a flipped 2PL IRT orientation (prompts as persons, models as items, N = 470 eligible prompts), model stringency spans a > 8-logit range (HunyuanDiT-3D b=−5.29 to CRM b = 2.83); prompt generatability ˆ θp ∈[−1.07,1.14] with 75 near-ceiling prompts (p≥0.8); (E4) descriptive sensitivity analysis reveals 44% of items show |gap|≥0.40 in pass rates between TripoSR and the five Objaverse-trained models; because Group B contains exactly one model, Objaverse exposure and TripoSR-specific factors are perfectly confounded and cannot be separated; (E5) an IRT-θ-stratified 30-item tinyEval3D subset recovers the full-benchmark model ranking exactly (Spearman ρ= 1.0, LOO-CV ρ= 0.954), reducing evaluation cost by 94%, with model rankings confirmed by Delphi LLM judging (ρ= 0.90 agreement with automated metric rankings). Our framework exposes structural inefficiencies in current 3D benchmarks and provides a reusable methodology for constructing psychometrically grounded evaluations.