Why Scoring AI's Human Simulations Like a Math Test Gets It Wrong

A new paper out of Renmin University argues that the standard way of grading AI models that simulate human survey responses is broken, and it starts from an uncomfortable premise: ask the same person the same question twice, and they may give two different answers. "Rethinking the Evaluation and Optimization of LLM-Based Social Simulation," posted to arXiv on August 20 by Pei Wang, Xu Chen, and Ji-Rong Wen, contends that treating a single recorded human answer as the one correct target — the way benchmarks currently score coding or math problems — is the wrong yardstick for tasks that are inherently subjective.

The authors' fix starts with what they call the subjectivity coefficient, an entropy-based measure that separates objective tasks, where one answer is verifiably right, from subjective ones like social simulation, where a recorded response is just one draw from a wider distribution of plausible human reactions. From there they build SALT, short for Subjectivity-Adaptive soft-Label Training, which constructs soft training labels from semantically similar contexts rather than forcing a model to reproduce a single hard-coded answer, scaling the softness of the label to how subjective the question actually is.

To test the approach, the team built SUBJSIM, a benchmark of 19,300 contexts drawn from 193 annotators answering 100 subjective questions, explicitly designed to score models against the full spread of human responses rather than a lone ground truth. Renmin's Gaoling School of Artificial Intelligence, where Chen and Wen are based, has built a research program around simulating populations of people with LLM agents, including prior work like CiteAgent for citation-network behavior and a TOIS paper on user-behavior simulation, so this paper reads as another installment in that line rather than a standalone experiment.

The critique lands inside a live argument in the field. A May 2026 paper, "Stop Drawing Scientific Claims from LLM Social Simulations Without Robustness Audits," and an ICLR 2025 blog post asking whether LLM simulations can truly reflect humanity have both made the case that language models tend to nail the modal, most-common human response while flattening the disagreement and uncertainty around it — which is precisely the gap SALT's distribution-matching approach is aimed at closing. A separate paper, "Restoring Heterogeneity in LLM-based Social Simulation," proposes audience segmentation as an alternative route to the same complaint: that simulated populations collapse toward homogeneous, majority-vote answers.

The soft-label instinct itself is not new to this paper; it echoes a critique that reward-model researchers have raised for years about RLHF, where majority-vote scoring can erase minority preferences during training. Papers like "Distributions In, Distributions Out" and "Majority Vote Silences Minority Values," the latter examining annotator disagreement in the HateXplain hate-speech dataset, make the same argument in adjacent corners of NLP. What this paper does is import that argument directly into the specific, and growing, subfield of using LLMs to stand in for human survey and experiment participants — a subfield now substantial enough to support its own venues, including a Social Agents in the Wild workshop at ACL 2026 and a dedicated 2025 ACM workshop on LLM agents for social simulation.

One practical wrinkle the paper doesn't address in its abstract: distribution-matching scores are harder to explain to a policy researcher or market analyst than a single accuracy number, which could slow adoption even if the underlying method is more faithful to how people actually behave. As of this fetch, no code or data repository was linked from the paper's arXiv listing, so replicating SUBJSIM or testing SALT against it will, for now, mean waiting on the authors to publish one.