Shifting Mechanisms: How Positional Encoding Choice Shapes Long-Context Retrieval

Despite supporting context windows of millions of tokens, language models still retrieve unreliably as context length grows. Yet it remains unclear how the choice of positional encoding (PE) impacts long-context retrieval. We take a mechanistic approach to answer this question, tracing how PE choice shapes the internal mechanisms a model uses for retrieval, and how those mechanisms in turn determine long-context performance. We find that PE choice shapes mechanism allocation: RoPE architectures learn a positional-dominant allocation, while hybrid architectures shift toward a semantic-dominant allocation. This analysis reveals two distinct paths for long-context scaling: Path 1 reduces positional encoding degradation to preserve the positional mechanism, while Path 2 avoids the positional mechanism by shifting toward semantic mechanisms that do not degrade with context length. We argue that Path 2 is the more principled direction, as it sidesteps theoretical limitations associated with RoPE at long context.