Align as Act: Innovations-Based Reward Decomposition for LLM Agents

In this paper, we propose a new step reward design for large language model (LLM) agents based on innovations: an information-theoretically optimal reward decomposition. Our method defines an intrinsic reward as the log energy of the innovation signal, measuring the amount of net new information provided by each step relative to the agent’s past interaction history. This intrinsic reward can be computed directly from embedding vectors in the semantic vector space via Gram-Schmidt orthogonalization, without training any additional reward or world models. The proposed design enables the agent to suppress redundant steps by definition and concentrate on progress-aligned transitions. Our method can be seamlessly integrated into standard on-policy objectives (e.g., PPO, GRPO) by augmenting rewards during training. Experiments on agentic benchmarks of ALFWorld and WebShop demonstrate that our method consistently improves performance and sample efficiency over strong RL baselines, suggesting that innovations-driven intrinsic rewards provide a practical principle for credit assignment in LLM agent RL post-training.