Beyond Merging and Dropping: Adaptive Online KV Cache Compaction

As Large Language Models (LLMs) increasingly rely on long Chain-of-Thought reasoning, linear Key-Value (KV) cache growth creates severe memory and latency bottlenecks. Existing learnable cache compressions are typically offline and enforce rigid, uniform compression ratios. To address this, we propose an adaptive online compaction method that combines token merging with learnable token dropping. By extending the STILL compactor with Gumbel-Sigmoid gating, our approach dynamically prunes unimportant latent token representations from a dense, merged initialization. This hybrid mechanism enables optimal, heterogeneous cache allocation across layers and heads while retaining reasoning capabilities. Trained sequentially on Open-R1 \citep{openr1}, our compactor reaches stable 8x compression while keeping teacher--student divergence bounded across repeated sequential compaction. On Qwen3-1.7B, our method achieves 27.50\% Avg@4 on AIME 26 and 19.69\% on HMMT-Feb 26, outperforming vanilla STILL by 2.50 and 0.76 percentage points and DMS by 1.67 and 1.51 points, respectively. It reduces peak KV payload by approximately 83\% relative to uncompressed inference and improves throughput by more than 22\% over vanilla STILL.