Towards Identifying Redundancy in Chain-of-Thought Reasoning

Improving our understanding of how efficiently language models can reason requires us to also understand reasoning's antithesis: what doesn't qualify as efficient language model reasoning? In other words, when considering language models' reasoning, do we have a clear idea of what constitutes redundancy? In this work, we focus on strategies to identify redundancy in the math word problem domain. We first argue that under specific circumstances designed to circumvent its shortcomings, a redundancy-detecting approach based on step deletion can successfully pinpoint redundant steps. Using those specific cases as a ground truth, we then test two methods for detecting redundancy that are based on screening for repetitiveness as compared to earlier reasoning in that problem, which are potentially more generalizable. However, neither of these methods manages to satisfy our two verification tests, demonstrating that detecting redundancy in chains of thought remains a difficult problem.