Coverage before Difficulty: Hard-Example Selection Collapses under Small-Budget SFT

Difficulty-based data selection, which keeps the examples a model finds hardest, is a popular way to stretch limited supervision. We ask what it does under a small budget: supervised fine-tuning (SFT) of Qwen2.5-0.5B on 256 examples, reproducible on a free-tier Colab. Scoring difficulty by per-example response negative log-likelihood (NLL) over a four-family Tulu-3 pool (code, general, instruction-following, math), we find that difficulty is strongly confounded with task family: the globally hardest examples are almost all instruction-following, while intrinsically low-entropy families such as math are never selected. Unconstrained hard selection therefore collapses the training mixture onto a single family, posting the best macro-NLL improvement even as math regresses and downstream GSM8K accuracy drops below the base model. A simple coverage-before-difficulty constraint, an equal per-family quota with difficulty applied within each family, recovers most of the headline gain while turning every family positive and holding GSM8K at or above baseline.