WebTrace: Evaluating LLM Re-identification of Anonymized Text with Web Search

As Large Language Models (LLMs) are increasingly augmented with tool use capabilities such as web search, they can actively retrieve and cross-reference public information, creating privacy risks that go beyond memorization. One concrete manifestation is re-identification: we show that an LLM agent with web search access can link anonymized interview transcripts from the Anthropic Interviewer dataset to specific, named individuals. Yet without ground-truth identities, the accuracy and coverage of such attacks cannot be rigorously measured. To bridge this gap, we introduce WebTrace, a benchmark of synthetic interview transcripts grounded in 822 real individuals with verified identities. We use this benchmark to evaluate seven open-weight and proprietary LLMs on re-identification accuracy, confidence calibration, token consumption, and latency. Our findings reveal an alarming shift: web search substantially increases re-identification success, making cheap, precise, and fully automated end-to-end attacks practically feasible. Open-weight models achieve performance on par with proprietary models while operating at roughly half the cost. These results reveal a pressing dual-use privacy challenge for LLMs and establish WebTrace as a foundation for systematic measurement of adversarial privacy risks.