Beyond expert users: agents should help users construct preferences, not just elicit them

Agentic assistants typically assume an expert user: one who can answer all questions about what they want and who will accept a correct solution when they see it. We argue both assumptions are unrealistic. First, users are boundedly rational --- they enter interactions with incomplete, underspecified requirements, sometimes unaware that a relevant consideration even exists. Second, agents must communicate solutions in a way that users can recognize as correct, since a user who cannot verify an output cannot act on it. To formalize these principles, we draw on social science frameworks to introduce CoPref, a model of how non-expert users develop task preferences depending on agent dialog actions. We study these ideas concretely in the setting of agentic recommender systems, where we propose CoShop, an interactive benchmark. In CoShop, agents must converse with CoPref users to elicit and develop their preferences, and then write a report from which users must identify the correct item. Evaluating eight frontier models, we find that no agent exceeds 50% accuracy on CoShop despite ten turns of interaction. Failures concentrate not in task execution, but in how agents converse with and present results to users.