Metadata Community Detection as Pretraining Data Mixture: Evidence from the Pushshift Reddit Dataset

Pretraining data curation is essential for training efficient language models. Existing mixture methods rely on domains either inherited from data source or imposed via a fixed taxonomy, neither of which adapts to the structure of the corpus itself. We argue that the metadata of commonly used pretraining corpora already carries this structure, along whose network edges textual similarity mixes assortatively. We propose CommunityMix, a mixture method that partitions metadata networks via community detection. We present a case study on Reddit, showing that domains induced from the user-subreddit affiliation network outperform both uniform sampling and classifier-based construction on DLCM CORE.