Paying to Look: How Language Models Respond to Changes in Search Costs

Recent developments in the capabilities of large language models (LLMs) have allowed for increasingly effective deployment in which users delegate decisions. In many economically relevant settings, LLMs must make dynamic decisions under incomplete information with costly information acquisition, requiring them to have an understanding of their decision-making behavior at future dates. In this study, we evaluate LLMs as agents in a sequential search game with costly exploration to capture the core tradeoff between the value of continued search and its cost across various common economic contexts. Our main finding is that meaningful sensitivity of exploration to changes in search costs, which is an essential feature of payoff-optimal decisions for forward-looking Bayesian decision-makers, is an emergent capability for the most recent wave of LLMs with reasoning. By comparison, older or non-reasoning LLMs' decision-making is highly insensitive to exploration costs. This discrepancy in model behavior holds across different contextual framings of the game. Our results imply that newer reasoning LLMs may be better at internalizing search costs when making choices, and that this feature serves as an important dimension for benchmarking the economic behavior of LLMs and implementing cost-effective applications in dynamic contexts.