Understanding Why LoRA Placement Matters: An Empirical and Mechanistic Study in LLMs

Low-Rank Adaptation (LoRA) is widely used for parameter-efficient fine-tuning, yet the choice of which architectural components to adapt remains less understood. In this work, we examine LoRA placement as an architecture-level adaptation choice, asking whether placement-dependent behavior can be explained by trainable parameter count alone, or whether the location where adaptation enters the model computation also plays a central role. We study this question in small-scale language models in the 0.6B-3B class, covering different placement strategies. Our results show that broader adaptation does not necessarily improve performance and can even degrade it. Attention-based placement provides the most robust efficiency-performance trade-off across models, while MLP-based and combined strategies can achieve competitive accuracy at higher computational cost. We further analyze LoRA behavior from both weight-space and representation-space perspectives. Effective placements produce structured parameter updates and moderate representation changes, whereas weaker placements either perturb representations excessively or modify internal computation insufficiently. Overall, our study suggests that LoRA placement should be understood not only as a parameter-budget choice, but also as an architecture-level adaptation decision that shapes internal model behavior.