Shutdownable Agents through Length-Neutral Policy Optimization

AI agents are increasingly used to autonomously solve long-horizon tasks. Research suggests that models trained in this way might resist shutdown, increasing loss of control risk. Neutrality+ is a decision rule designed to mitigate this problem. Agents satisfying Neutrality+ maximize the sum of expected utilities conditional on each trajectory length. This is theorized to keep agents shutdownable and allow them to be useful. We introduce Length-Neutral Policy Optimization (LNPO), the first known method for training models to satisfy Neutrality+. We also introduce gridworld environments that act as analogs of scenarios where agents are instrumentally incentivized to resist shutdown via scheming, sandbagging, avoiding monitoring, or acting differently under monitoring. We compare agents trained with Proximal Policy Optimization (PPO) to agents trained with LNPO and observe that LNPO agents resist shutdown at substantially lower rates, with reductions ranging from 40--71\% depending on the specific shutdown resistance behavior. We test to see if LNPO degrades performance in scenarios where shutdown is irrelevant and find no observed performance degradation.