TraceML: An Empirical Analysis of Human-Agent Planning in Machine Learning Development

Large language models write correct code for isolated problems but are far weaker at autonomous machine-learning development, where an agent improves data pipelines, models, and validation over hours of feedback, and on most competitions still finishes below strong human competitors. Outcome-based benchmarks record this gap but not its cause: they grade the final submission and discard the process behind it, hiding where an agent's workflow parts ways with an expert's. We introduce TraceML, which pairs human and agent runs on the same competitions under one version-level schema: 4,465 human Kaggle trajectories over 134 competitions, and a matched subset of 7 competitions with 430 human and 207 agent trajectories. Each trajectory is an ordered set of code versions with scores, timestamps, and step-level labels for action, intent, edit size, and score effect. We find humans and agents differ in how they explore, when they build validation and ensembling structure, and how they allocate effort, with each agent scaffold settling into a narrow loop rather than the multi-phase process experts use. A short planning prompt built from human practice leads an agent to adopt the practices we name while its effort profile stays agent-like, so instruction closes part of the gap and leaves part open. We release TraceML, the extraction and labeling code, and the labeler checkpoints at {https://huggingface.co/datasets/TraceML-HF/TraceML}.