Understanding Without Knowing: Format and Context Impact on 100M-Parameter Domain QA

We study domain question answering in language models below 100M parameters, a regime where broad world knowledge cannot be stored and specialization to a narrow domain is the only route to competence. In this regime a single design choice—whether a question is posed as balanced yes/no or as multiple choice—turns out to determine whether the model can use its domain knowledge at all, even though a 7B model is indifferent to it. We show this with WikiDoc-BMCQA, a medical benchmark posing identical facts in both formats, with an optional text passage to test closed-book and open-book behavior. Posed as binary yes/no (B), closed-book accuracy of finetuned MobileLLM (125M) and Mamba (130M) falls below a no-knowledge baseline that predicts from surface phrasing alone; posed as multiple choice (MC), the same facts become genuinely accessible above that baseline. Supplying the source passage (open-book) lifts both binary and multiple choice accuracies to almost that of a larger Qwen 7B instruct model, showing that reading abilites are instead saturated even at the 100M scale. The same format and context effects replicate on three disjoint public benchmarks (Mintaka, SciQ). These results suggest a graded, architecture-invariant property of domain knowledge at 100M-scale. While the scale-robust route to using such tiny models remains supplying context, knowledge at this scale is present, but only discriminatively accessible.