Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs

• The main experiments in Table 1 and Table 2 should include the baseline of naive DPO experiments.

• The author uses different source data for SFT and Step-DPO separately. If we use the whole dataset adopted in SFT and Step-DPO in the SFT phase [2], how would the performance of SFT and DPO compare? This could establish a strong baseline.

• If we construct a paired dataset, the author can use the naive DPO loss function to optimize the LLMs. Therefore, this paper could be regarded as a data construction pipeline for step-level paired data.

• Considering the process supervision proposed in [1] and related works [4], the novelty and data construction methods are limited.

• Accuracy gains are small, many < 1%.

1. In terms of results, this work ultimately achieved improvements of 1.4% and 1.6% relative to DPO on MATH and GSM8K respectively using Qwen2-72B. My research area is primarily RL, and I'm new to LLMs. Therefore, I won't make judgments on the significance of the performance improvements in the paper. In this regard, I will refer to other reviewers' opinions later. Additionally, I have some questions about the technical design.
2. The construction of the dataset is the core foundation of the proposed method, but the entire annotation and generation process is based on GPT-4 and the initial model $\pi_{ref}$. How is the quality of the data and annotations controlled? I understand that the authors mention relying on models rather than human effort can make this process user-friendly, but it also makes the process a black box and uncontrollable. In this process, it seems we can only rely on comparing performance results after the fine-tuning process to determine the effectiveness of the method.

The current data construction relies on two basic assumptions: (1) There exists a powerful model like GPT-4 that can accurately point out erroneous steps in the reasoning chain. (2) It relies on $\pi_{ref}$ to generate effective step-wise rectification.
These two assumptions are problematic and are not sufficiently discussed in the paper. The experimental results demonstrate that the proposed data pipeline can indeed help the learning of step-wise DPO. However, what are the underlying principles, and what new insights can it provide to researchers in the field? Discussion of these questions can help further enhance the impact of this work.

3. If relying on the existence of a stronger model, or even one that is close to a ground-truth model, then this process is not simply about improving the model's reasoning ability but is closer to knowledge distillation.
4. What kind of improvement can the step-wise rectification based on $\pi_{ref}$ provide? In lines 276-283, the authors mention sampling the referred model multiple times and retaining results where the final answer matches the ground truth. How large is the scope of this data, and does it represent capabilities that the model already possesses but have not been exploited? Moreover, if only mining this part of the capability, the improvement the model can achieve seems to be limited.

1. The significance of the new method's dominance over existing models/results is not clear in several cases. The lack of meaningful error bars in figures and tables means that some improvements appear possibly marginal in Tables 1, 3, and 4.
2. In some cases, such as for Qwen-72B-Instruct, the bulk of improvements seems to come from the updated prompting strategy used by the authors, not the Step-DPO method. For example, the abstract concludes with the statement that the Step-DPO-trained Qwen-72B-Instruct surpasses GPT-4-1106, Claude-3-Opus, and Gemini-1.5-Pro on GSM8K and MATH. In fact, it seems it does not surpass Claude-3-Opus on GSM8K. Additionally, Qwen-72B-Instruct already surpassed all of these three models on MATH (and GPT-4-1106 and Gemini-1.5-Pro on GSM8K) after the improved prompting strategy, meaning that the conclusion that Step-DPO allowed for these results seems possibly unfounded.
3. In terms of contribution, it is unclear to me that the framing of Step-DPO as a DPO variant is founded. It seems to me that the formulation in Equation 2 is precisely that of Equation 1, with the exception of changing what is the context and completion. I am not sure why we need a full equation statement of Step-DPO. The paper does mention that "Rejecting an entire undesirable answer in DPO may also discard preceding correct reasoning steps, introducing significant noise and negatively impacting training." (201-203). I agree that there is a major issue with the formulation of DPO for tasks with only some portions that are wrong; it could diminish the likelihood of initial correct reasoning steps. However, as long as the portion of the response before the deviation point is the same, it is effectively part of the context and therefore no gradients will be derived from it. This means that it seems to me that the approach presented in the paper is effectively exactly the same as ordinary DPO, but with the initial portion of both completions identical and the portion of the completion after the deviation clipped after one reasoning step. Mostly, the contribution of this work seems to be pair-preference dataset creation for a targeted purpose.

• Key information is not clear: in the so-called step localization stage, about how to label the step-wise reasoning, the authors described "This process can be done manually or using GPT-4". However, it is not clear how this is done in detail. Since this is very important in the entire processing, it would be necessary to know more. Specifically,

  1. how many data is labelled by human, and how many is from GPT-4
  2. For human annotation, how many annotators are hired for each question; what is the question to annotate, e.g., is it a binary classification? Also, since many math questions are not easy, what efforts do the authors take to ensure annotation quality.
  3. For GPT-4 annotation, what is the prompt? Since model could make mistakes , how to reduce this error risk?
  4. One alternative (and straightforward) way is to using a small amount of labelled data to fine-tune a llm to be a PRM, and use it to give step-wise supervision siginals.

• In table3, compared to normal DPO, the step-wise DPO's gain is very limited. It's not very convincing to claim the advantage of step-wise DPO. Also, why only using 5K data for this ablation instead of the entire dataset?

• Similarly in table4, the marginal improvement of in distribution data is hard to conclude it is better.