When Do Stored-Solution Skills Help Agents? A Controlled Study of Routing, Accumulation, and Blind Reuse
Abstract
LLM agents increasingly discover skills—on a hard task they author and store a reusable procedure, growing ever-larger libraries (Voyager, TROVE, CRAFT)—on the implicit premise that new tasks need new skills. We ask a narrower question: for skills that are stored worked examples retrieved by similarity, when does discovery help beyond composing the tools and memories already available? We compare composition, naive accumulation (store every solved task), a simple novelty gate, and a refactoring-style compression baseline using released skill libraries and executed code-repair studies scored by real tests (pytest and SWE-bench Lite). The contribution is primarily behavioral and methodological, not a new agent architecture. First, a proxy analysis shows large released libraries are redundant under a heuristic capability label, but the proxy is domain-fragile and diagnostic rather than load-bearing. Second, controlled blind-reuse probes show that stored examples matter when a task stream contains recurring non-inferable conventions, and in that regime a small semantically-routed library matches or slightly exceeds much larger accumulated libraries; on SWE-bench, forbidding skill invention shows no detectable solve-rate loss, but the study is explicitly underpowered for small effects. Third, the simple novelty gate is a measurement device and practical baseline: it exposes a size–reuse frontier on which online gating is leaner, while refactoring-style compression is competitive. We distill three evaluation pitfalls—scaffold quality, sparse-versus-blind feedback, and tasks already inside a model's blind competence—that can manufacture spurious compose-versus-discover effects. Within this scoped construct, discovery is best understood as routing reusable information from solved examples, invoked on novelty, rather than a default response to every task. Code and data: https://anonymous.4open.science/r/skill-gate-anonymous-release-E34F/