Dimension Is Free, Horizon Is Not: When Evolution Strategies Rival Policy Gradients for LLM Fine-Tuning

Weight-perturbation Evolution Strategies (ES) are increasingly proposed as a scalable alternative to action-space policy gradients (PG; PPO, GRPO) for LLM fine-tuning (Qiu et al., 2025), even as a parallel literature reports that "all RL algorithms perform similarly" (Yue et al., 2025). We give a single account that explains both. ES and PG are one score-function estimator of the same gradient $g^\star = \nabla_\theta \mathbb{E}[R]$; they differ only in the variance of the estimate, and the two variances scale with orthogonal problem properties. The naive reading — $\mathrm{Var}(\hat{g}{\mathrm{ES}}) \propto D$, the parameter count — suggests ES cannot scale to large models. We show this is wrong: that variance is isotropic noise in reward-null directions, so ambient dimension is free (a blessing of dimensionality), and ES's training cost scales only with the intrinsic dimension $D{\mathrm{eff}}$. PG instead pays the effective horizon $T_{\mathrm{eff}} = T \cdot A$, whose per-step action term $A$ is bounded for categorical tokens ($1 - \sum_v p_v^2 \leq 1$) but unbounded for continuous control. In a trained who-wins phase diagram these orthogonal costs draw three regimes — a tie where both solve the task, PG winning at high $D_{\mathrm{eff}}$ / short horizon, and ES winning at long horizon — so ES is never strictly better when credit is clean: its entire advantage is immunity to long, credit-poor reward, with $D_{\mathrm{eff}}$ setting only the margin. This turns method choice into placement on a $(D_{\mathrm{eff}}, T_{\mathrm{eff}})$ map with two levers — a strong prior lowers $D_{\mathrm{eff}}$, credit assignment lowers $T_{\mathrm{eff}}$ — on which most LLM fine-tuning sits near the tie/boundary (low $D_{\mathrm{eff}}$, moderate horizon), with ES's decisive edge predicted only for the long, credit-poor tail (multi-turn, terminal-reward agents).