Methods and Opportunities at Small Scale (MOSS)

As the current LLM research community pushes towards massive scale, this workshop takes a complementary approach by focusing on the under-explored small-scale regime, where scale encompasses compute, data, and model size. We ask: to what extent is scale necessary, and how far can we push toward smaller settings while maintaining competitive performance and enabling scientifically meaningful discoveries?

Beyond transferring insights from small- to large-scale settings, we emphasize the intrinsic value of understanding small-scale regimes. Practically, studying small-scale limits can yield more efficient algorithms and system designs. Small-scale settings also enable controlled, systematic experimentation with rapid iterations, thereby facilitating scientific progress.

This workshop aims to highlight the methods, opportunities, and insights enabled by small-scale experimentation, and to foster discussion on how such approaches can broaden participation and accelerate progress in LLM research.