Demonstrations, CoT, and Prompting: A Theoretical Analysis of In-Context Learning

In-Context Learning (ICL) enables pretrained LLMs to adapt to downstream tasks by conditioning on a small set of input-output demonstrations, without any parameter updates. Although there have been many theoretical efforts to explain how ICL works, most either rely on strong architectural or data assumptions, or fail to capture the impact of key practical factors such as demonstration selection, Chain-of-Thought (CoT) prompting, the number of demonstrations, and prompt templates. We address this gap by establishing a theoretical analysis of ICL under mild assumptions that links these design choices to generalization behavior. We derive an upper bound on the ICL test loss, showing that performance is governed by (i) the quality of selected demonstrations, quantified by Lipschitz constants of the ICL loss along paths connecting test prompts to pretraining samples, (ii) an intrinsic ICL capability of the pretrained model, and (iii) the degree of distribution shift. We extend this framework to CoT prompting, and further characterize how ICL performance sensitivity to prompt templates varies with the number of demonstrations. Together, our theory proves that pretraining equips the model to generalize beyond observed tasks, and characterizes how this capability is shaped by demonstration quality and prompt design. All theoretical insights are corroborated by experiments.