Matching the Distribution, Not the Mean: Learning Human Rating Distributions with VLMs

Repeated human ratings encode both central tendency and disagreement, but model-alignment pipelines often collapse them to a single target. We test whether vision-language models (VLMs) can instead learn empirical response distributions. The main experiment uses 360 rock images, six perceptual features, and approximately twenty ratings per image-feature pair. In each fold, models train on two of the three rock families and are evaluated on the remaining family, giving 120 test images per fold. Across five open-weight VLMs, training on the full human histogram lowers mean human-model KL divergence from 1.20 for rounded-mean targets and 0.62 for mode targets to 0.24. It also raises mean Pearson correlation with human mean ratings from 0.908 and 0.874 to 0.938. Secondary checks ask whether these gains transfer to human similarity judgments on the same rock images, whether feature ratings remain linearly decodable from model embeddings, whether fine-tuning degrades general image-understanding performance, and whether the pattern appears on a small natural-photograph quality task. Preserving repeated judgments therefore improves distributional fidelity on test images while retaining expected-rating alignment.