LegalScope: Measuring Exam-to-Case Transfer in LLM Legal Reasoning

Accepted paper — no in-person presentation.

This paper remains accepted. No in-person poster presentation is scheduled.

Public legal exams are widely used to evaluate legal reasoning in large language models, yet it remains unclear whether exam performance reflects the ability to reason over real-case facts. We introduce LegalScope, a dual-track benchmark that pairs 868open-ended legal-exam questions with 76 lawyer-reviewed issue-stance prompts derived from de-identified Chinese civil judgments. LegalScope evaluates models under both reference-answer scoring and case-based rubric scoring, enabling direct analysis of exam-to-case transfer. Across 20 model groups, public-exam scores correlate with real-case scores but do not fully predict them, and real-case prompts expose a recurring weakness in extracting operative legal and factual constraints. We further show that automated evaluation aligns strongly with human review on exam answers but less reliably on real-case legal analysis, highlighting the need for expert-grounded case evaluation.