Understanding Machine Unlearning Through the Lens of Mode Connectivity

Machine Unlearning aims to remove undesired information from trained models without requiring full retraining from scratch. Despite recent advancements, their underlying loss landscapes and optimization dynamics received less attention. In this paper, we investigate and analyze machine unlearning through the lens of mode connectivity--the phenomenon where independently trained models can be connected by smooth low-loss paths in the parameter space. We define and study mode connectivity in unlearning (MCU) across a range of overlooked conditions, including models trained curriculum learning, second-order optimization, and cross-method connectivity. Our findings show distinct patterns of loss landscapes across various datasets, training paradigms, and unlearning algorithms. With MCU, we analyze the mechanistic (dis)similarity between retraining from scratch and unlearning, as well as between unlearning algorithms. We also demonstrate MCU can be used to improve generalization of unlearning and defending against relearning attacks. To the best of our knowledge, this is the first study of loss landscape analysis of machine unlearning with mode connectivity.