Width Recovers, Depth Continues: Architectural Tradeoffs in Small Transformers for Symbolic Regression

Small models can expose architectural phenomena that are difficult to find when scaled up. In this work, we study the trade-off of depth and width allocation, in decoder-only Transformers trained for symbolic skeleton regression. Given a few points of a function, the models generate a prefix form expression skeleton, whose constants are later fitted by a verifier. Across 400k to 50M parameter models trained under equivalent training token budgets, we compare wide, balanced, and narrow architectures. Models are trained only on expressions of depth 1-3, and evaluated both in distribution and on deeper out-of-distribution expressions of depths 4-6. We find that architecture shape induces a clear capability split. Wide models achieve the strongest free-generation performance, with the best observed models reaching 11.47\% success@1 and 18.70\% success@100 on depth 4--6 expressions under a strict evaluation-grid MSE threshold. In contrast, narrow models achieve much lower error when asked to continue correct prefixes of deep expressions, even though they perform worse at generating full expressions independently, suggesting that depth improves conditional continuation once the model is on the correct trajectory. OOD functional success does not require exact expression recovery: the models usually find a shallower expression than the target, and exact recovery of deeper OOD skeletons is almost nonexistent (0.1\% at $K=10$ and 1.0\% at $K=100$ on any individual split). Overall, width helps models discover functionally correct solutions, while depth allows them to continue deeper expressions once given the correct beginning.