The System Must Evolve: Benchmarking Long-Horizon Automation Over Dependent Tasks

Large language model (LLM) agents are increasingly deployed as long-horizon, lifelong systems, yet most benchmarks measure isolated task completion or heterogeneous task streams rather than whether an agent can accumulate, retain, and reuse capabilities that later tasks structurally depend on. We introduce Throughline, a continual-agent benchmark inspired by the production-chain game Factorio. Throughline abstracts away spatial layout and real-time control while preserving hierarchical recipes, throughput constraints, and dependency-structured task progression, and scores every rollout with a deterministic simulator rather than an LLM judge. Over a shared 20-sub-task cumulative campaign, in which the factory and recipe library persist so that each sub-task extends earlier ones, we evaluate frontier API models and local continual-learning (CL) baselines. We surface four effects: frontier models span a wide capability range and collapse at different depths of the dependency chain while sharing a wall at the final composite target; the best memory representation is decisive yet model-specific; local CL is rehearsal-sensitive, with naive fine-tuning catastrophically forgetting while rich rehearsal recovers to the offline supervised ceiling; and the residual failure is mis-extension of an inherited factory rather than loss of earlier targets. Throughline is a diagnostic testbed for memory, CL, and reinforcement-learning methods that must build and reuse evolving capability stacks.