TRONTOR: To RoPE or Not to RoPE for Long-Context Modeling

While Rotary Position Embedding (RoPE) is widely used in large language models (LLMs), its inductive biases can impair long-context reasoning and lead to attention sinks. In this work, we propose TRONTOR, a method that effectively combines RoPE and No Positional Encoding (NoPE) through a simple fine-tuning phase, bypassing the need for expensive pretraining modifications. Our preliminary experiments demonstrate that mixing RoPE and NoPE at the attention-head level can enhance long-context capabilities, while static selection strategies fail to capture the varying complexity of inputs. Based on the observation, TRONTOR adaptively switches between RoPE and NoPE at the token level via context-aware thresholding. Furthermore, a simple non-weighted averaging of RoPE bases is sufficient within active RoPE heads, avoiding the complex intra-head routing mechanisms required by prior studies. Evaluated on the RULER benchmark using Llama-3.2 (1B and 3B) and Qwen3 (1.7B and 4B), TRONTOR consistently demonstrates superior performance by mitigating attention sinks and dynamically reallocating attention to relevant mid-context tokens.