Predicting Neural Scaling Laws without Training: A Data Manifold Oracle

Fitting neural scaling laws is expensive because it requires training families of models at multiple scales for every new corpus. We study the nature of neural scaling laws, asking what properties of the data determine a corpus’s scaling exponent and irreducible loss, and how much can be known about either before any model is trained. We develop and rigorously analyze a training-free alternative, a data manifold oracle that reads scaling behavior directly from the data by combining a compression measurement, the LZ77 entropy-rate estimate hμ, which can be computed in milliseconds, with a single geometric side input, an expansion scale λ. For a class of conformal-expanding sources, we give the pipeline a theorem-level foundation based on four theorem pillars unified by a dimension ladder connecting geometric, algorithmic, recurrence-practical, and smooth-integer notions of dimension. Specifically, the intrinsic dimension of the data manifold is recovered to leading order as d = h_μ/λ; the best-in-class neural cross-entropy scaling exponent, an intrinsic property of the corpus’s algorithmic-geometric structure rather than the exponent of any particular trained run, which we study empirically, is provably at least 2/d; a single typical sample determines d almost surely; and an obstruction theorem shows that no symbolic measurement alone can determine d, meaning that the geometric side input is necessary rather than incidental. The analysis sharply separates what compression can certify, including entropy floors and lower bounds on exponents, from what it cannot. We prove that the finite-sample compression residual carries no dimension power law and reclassify the popular residual-exponent interpretation of compression curves as an empirically supported regularity whose explanation is captured by a precise open conjecture concerning an effective expansion scale for rough sources. Empirically, the oracle’s two readouts predict measured loss floors with R² = 0.92 and rank web, code, and mathematics corpora by data-scaling exponent with ρ = 0.84 across 41 corpora. New filter-dynamics experiments place real language in the long-memory, sub-exponential, multi-timescale regime required by the conjecture: the optimal filter synchronizes sub-exponentially with a probe-invariant but corpus-dependent synchronization exponent that remains stable across probes ranging from 124 million to 32 billion parameters and across state-space, recurrent neural network, and attention-based filters, while varying systematically across corpora. A post-training corollary converts the same readouts into a target-conditioned data-selection rule. Together, these results recast scaling-law estimation from a training problem into a data-measurement problem, providing practitioners with a certified, gradient-free tool for corpus comparison and data curation.