Cheap Scores First: Cascading Ablated Auxiliary Models for Targeted Data Selection

Pretraining data selection and curation are critical steps for training high-quality LLMs. Model-aware data selection methods such as CoLoR-Filter (Brandfonbrener et al., 2024) use a pair of small auxiliary LLMs to score and select pretraining data. Under compute constraints, we investigate the trade-offs in transformer-block deletion ablations of 150M-class auxiliary models. Ablating the middle two blocks leaves CoLoR scores largely intact, with a Spearman rank correlation of 0.89 against the full-model scores on a sampled C4 pool, whereas top- and bottom-two deletions degrade the correlation to 0.53 and 0.49. However, around selection thresholds where data decisions are actually made, we find that the scored ablations collapse to near chance. To mitigate this, we propose a cascading approach: score samples cheaply with an ablated model, followed by final filtering with the full model. Our proposed cascading approach improves AUC from 0.51 to 0.68 around the hardest boundary threshold. As an end-to-end evaluation, we train 410M-class models on cascade-selected data and find Books validation loss to be competitive with the baseline (5.004 vs. 5.038 cross-entropy loss, averaged over three training seeds) despite selecting substantially different data (0.49 Jaccard overlap). These results suggest auxiliary-model depth is a practical lever for cheaper targeted data selection.