Can Language Models Learn to Skip Steps?

Thank you for your valuable and constructive feedback. We provide specific responses and clarifications as follows.

[W1]

Thank you for the suggestion. We acknowledge the importance of testing our method across different models. Following your advice, we performed additional experiments on Phi-3-mini for the Analog of Algebra task. These experiments confirm that our method is generalizable to other model series. We will include the results across all tasks in the revision.

Table: Phi-3-mini on Analog of Algebra.

[W2]

Thank you for your valuable suggestion! We acknowledge the importance of evaluating the generalizability of our method. We are actively working on additional experiments and will provide an update on the results once the experiments are completed.

[W3]

For the three datasets used in our work, we use heuristic rules to automatically generate the reasoning processes. For example, in the Analog of Algebra dataset, we first create full-step reasoning data using standard algebra rules on operators. Then, we replace the variables with analog symbols. We also plan to release the code for creating the dataset in the future.

Thank you for your valuable and constructive feedback. We provide specific responses and clarifications as follows.

[W1]

Thank you for your valuable suggestion! We acknowledge the importance of evaluating the generalizability of our method. We are actively working on additional experiments and will provide an update on the results once the experiments are completed.

[W2]

In the iteration phase, we query the model to generate shorter answers on the training set and filter based on the correctness of the final answers. The data used are the labels required for training any model, so our filtering process does not rely on any additional information beyond the training data.

When generating answers with fewer steps, the model determines which steps to skip or omit. Thus, the skipping behaviors are entirely generated by the model itself, and the filtering serves as guidance on how to strengthen such behaviors.

Correct answers can indeed be subsets of the full-length solution. The models may actively omit certain steps from the full-length answer to generate the shorter output.

[Q1]

Thank you for this insightful question. Our accuracy metric does not measure the correctness of intermediate steps. Automatically evaluating the correctness of the internal reasoning process is a common issue and can be challenging [1]. Previous work [2] has shown that training with incorrect reasoning traces can still lead to improvements. From our manual analysis of Analog of Algebra on In-domain test set, we observe 50 skipped answers and find that 96% of them are correct in its reasoning process. We'll provide the analysis across all tasks in the revision.

[1]Solving math word problems with process- and outcome-based feedback.
[2]MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models.

[Q2]

The primary motivation of this metric is to measure whether the model has learned the ability to skip steps. Even when the accuracy is not high, the model still uses fewer steps overall while maintaining its overall accuracy. This consistent performance, both when accuracy is low and high, indicates that the model is developing this step-skipping ability and demonstrating it on both in-domain and out-of-domain data.

[Q3]

Thank you for your insightful question. We will include the analysis across all tasks in the revision. Here we present the results for the Analog of Algebra task. The Skipping Ratio indicates the proportion of answers where steps were skipped in the entire test set. The Acc measures the accuracy of these skipping answers. The results show that as iterations progress, the model tends to skip steps in more problems, and the accuracy increases accordingly. In the OOD-hard setting, due to the lengthy and difficult nature of the problems, the model initially tends to use shorter answers with lower accuracy. However, the overall accuracy increases with each iteration.

Table: Skipping ratio across the iterations and the accuracy of problems with skipping-step answers.

[Q4]

These two tasks involve more steps and actions in the answers. From our manual observation, we noticed that the model tends to overfit on the multi-digit addition task. As the training data only contains numbers with up to three digits, the model overfits to three-digit additions. Here is an incorrect answer from the model's prediction:

Q: 4423 + 684
Prediction:

1. Add the 10^0's place digits: 2 + 4 (plus carry 0) = 6 (write down 6, no carry over).
2. Add the 10^1's place digits: 4 + 8 (plus carry 1) = 12 (write down 2, carry over 1).
3. Add the 10^2's place digits: 4 + 6 (plus carry 1) = 11 (write down 1, carry over 1).
4. Carry the final carry over 1 to the next place.
   Thus, the answer is 11266.

In this example, the model actually performs the addition for 442 + 684, which indicates a biased overfitting from the initialization phase. During the iteration phase, the skipping data helps to mitigate this overfitting, thereby improving accuracy.

[Q5]

On the task of Analog of Algebra, the original training set contains 5770 data. As the iteration proceeds, the number of mixed data points (Total) increases significantly, nearly tripling by the fifth iteration. Here, $i$ indicates the number of valid answers that skip $i$ step in the reasoning process.

Table: The change in data volume across iterations.

[Q6]

In the warm start phase, we only use i=1 in the manual skipping data.

[Q7]

Thank you for pointing it out. We will revise the figure reference accordingly.

Thank you for your constructive feedback and valuable suggestion! We acknowledge the importance of using established practical benchmarks. We are currently conducting additional experiments with these datasets and will provide an update on the results once the experiments are completed.

Thank you for your insightful and encouraging feedback. We are pleased that you found our approach and results promising and will continue to refine and expand upon these ideas in future revisions.

[W1]

We only require the step number as input when generating the skipped data for the training set during the iteration phase. At inference time, we train an additional standard model (described as Eq. 2), which only takes the question as input, without the need for a number of reasoning steps. The instruction used is:

"Transform the expression to isolate ❤ on the left side of the ↔.
Question: [[QUESTION]]
Answer:"

This approach ensures the model is robust and practical for real-world applications, as it does not rely on the ground-truth number of reasoning steps during inference.

[W2]

We apologize for the confusion regarding the results in Sections 5.1 and 5.2. We will provide more detailed captions to clarify this. In Section 5.1, we analyze the initialized model $M_0$, which is trained with full step data only and requires the step number as input. When asked to generate answers in fewer steps, the model struggles to maintain accuracy because it has not been trained on the skipping-step data. The primary purpose of $M_0$ is to generate skipped step data during the iteration phase, and it is not expected to achieve high accuracy initially—only sufficient accuracy to generate usable data.

In Section 5.2, we evaluate the resulting standard model $M^{standard}$, which is trained on both the full step and the generated skipped data, and does not require the step number as input. This training method allows $M^{standard}$ to maintain accuracy while using fewer steps, demonstrating the effectiveness of our approach.

[W3]

Thank you for your valuable suggestion! We acknowledge the importance of using established benchmarks. We are actively working on additional experiments using these datasets and will provide an update on the results once the experiments are completed.

[Q1]

Mixing the data allows us to gradually process the iterations. At the beginning of the iterations, the valid skipping data may not be enough and they cannot help the model fully understand the task. For example, on Analog of Algebra, Iter 1 only has less than 150 valid skipping data to use. We are also working on additional ablation analysis to provide empirical support following your advice.

[Q2]

Thank you for this insightful suggestion! It would indeed be valuable to investigate where the improvements come from. We will consider incorporating this experiment in future revisions to better understand the underlying reasons.

[Q3]

Thank you for pointing this out. We will include this in the related work.

[Q4]

Thank you for providing the references. We are excited to see related work in the community that shares similar ideas and interests! We will incorporate these references in the revised version.

[Q5, Q6]

Thank you for pointing out these typos. We will revise them accordingly.

[Limitation]

We additionally conduct experiments with Phi-3-mini on the Analog of Algebra task. The results confirm that our method is generalizable to other model series. We will include the results across all tasks in the revision.

Table: Phi-3-mini on Analog of Algebra.