Individually Sensible, Collectively Harmful: Simulating Commons Failure in LLM Societies

Large language models are increasingly deployed as agents in simulations of societies, markets, and organizations, raising a fundamental question: can individually sensible AI agents nevertheless produce collectively harmful outcomes? We study this question in a repeated commons dilemma with externalities. We introduce Creeping Trap, a controlled social simulation in which LLM agents repeatedly choose private extraction levels for immediate profit while jointly accumulating catastrophe risk and imposing harm on silent stakeholders. The environment includes analytically derived welfare and behavioral reference strategies, allowing individually sensible behavior and collective welfare to be evaluated simultaneously. Across nine contemporary LLMs and 990 episodes, including a 400-episode confirmatory study, simulated societies consistently over-extract; in the confirmatory study, 396 of 400 episodes produce negative aggregate welfare. This failure is not arbitrary irrationality: observed behavior lies near a broad low-regret region of local best response that even simple heuristics attain, indicating an institutional failure generated by individually sensible decisions. We further show that institutional design changes collective outcomes: surfacing stakeholder harm consistently reduces extraction, whereas shortening accountability horizons increases extraction and induces end-of-term defection. Finally, prompt wording substantially changes measured behavior, making prompt formulation a first-order experimental variable. Creeping Trap provides a controlled testbed for studying institutional interventions that improve collective outcomes in AI societies.