Iterative Reasoning Preference Optimization

We thank the reviewer for the comments.

Weakness 1: Correct; we trained using IRPO by leveraging GSM8K/MATH/ARC training set, and we tested on the respective test sets, but not other datasets. This paper doesn’t focus on generalization to other datasets, but the reviewer raises a valid point. We believe that generalizing to other datasets might need a larger number of prompts (e.g., by leveraging a dozen different datasets for IRPO training) and it would be an effort to find good-quality test data as well.

Weakness 2: We were not aware of this paper at the time of writing, and we will cite this paper. But as explained below, in fact, this algorithm is already considered in our baseline.

This paper is similar to STaR (Zelikman et al.) from 2022. Our submission includes a variant of STaR baseline that uses temperature sampling without rationalization (i.e., without providing the correct answer first before generating the problem). In Rest-EM, in the binary reward case, when the solution has the wrong final answer, r=0; in this case we do not take any gradient. Therefore, the Rest-EM’s approach is essentially our variant of STaR. We’ll clarify this piece of related literature and our baselines in our revision, and thank you for suggesting!

We thank the reviewer for the feedback and insights.

Re: novelty.

• Our method is not self-rewarding because the reward signals come from the ground-truth labels instead of the model itself. More specifically, self-rewarding LM requires generating prompts, and using LMs to evaluate the sampled generations. But LMs are horrible at evaluating reasoning-related generations (e.g., according to RewardBench – although there has been some progress in the past few weeks). So we believe that self-rewarding LMs cannot be successfully directly applied using llama-2-70b-chat, but using IRPO in an unsupervised fashion is a really promising research avenue!
• We are proposing a loss that integrates NLL while the self-rewarding LM uses standard DPO training. Through our experiments (in this IRPO submission), we demonstrate the importance of the NLL term on positive examples.
• The focus of self-rewarding was general instruction tuning and our paper focuses on reasoning, math, and science reasoning in particular. In fact, the self-rewarding did not bring improvement in the math category.

Q1:

• IMPORTANT CLARIFICATION on “more data”: The “twice as much data” baseline means that we actually generated twice as much data – there is no duplication. We apologize for the confusion and will emphasize this point in the paper.
• On the second bullet point: “Using the positive & negative samples for DPO” is actually our DPO baselines in the tables. “Using positive samples for SFT” is actually the STaR baseline in the tables.
• On the third bullet point: (c, x, y). The DPO baseline is essentially RPO but without the NLL term.

Q2: Yes. We experimented with different beta’s (0.05, 0.1, 0.5, 1.0) and the probability trends are similar within each dataset. We are investigating why sometimes naive DPO leads to decreasing chosen AND rejected probabilities like in Figure 2. Our current hypothesis is that it’s related to the fact that both current and rejected generations are sampled from the most recent model – both generations have high probabilities under the most recent model distribution.

Thank you for the detailed review and suggestions!

Weakness 1 (longer training): Thank you for pointing out this issue. In each iteration, we do train longer (5000 steps) but end up selecting an earlier checkpoint – the selected checkpoints (by validation accuracy) are usually trained for 2000 or 3000 steps (e.g., for GSM8K, M1 is trained for 2000 steps and M2 for 2000 steps). This is because training longer makes the model overfit to the training and hurts validation performance. We use the same checkpoint selection process for both RPO and DPO. We’ll include more detailed discussion in the revision.

Importantly, in Table 1 of the submission, we also included results where the model is trained on twice as much data instead of two iterations (using STaR and using iterative RPO, also for a max of 5000 steps). As shown in Table 1, this strategy (the last row) doesn’t match the performance of doing two separate iterations (2nd row). This highlights the advantage of iterations over longer training.

Another piece of evidence (that iterative training is helpful) is that recent work (Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data; https://arxiv.org/abs/2404.14367) has demonstrated the importance for on-policy data (i.e., iterations). We’ll make sure to discuss this issue more thoroughly.

Weakness 2 (hyperparameters for baselines): We conducted a grid search for SFT, iteration 1 of DPO, and iteration 1 of RPO. For iteration>1, we use the same hyperparameter as iteration=1 (except for num_training_step which is selected individually for each iteration). All experiments share a similar config file; therefore, for all baselines, learning rates are all tuned from the range of 5e-7 to 5e-6. DPO and RPO are tuned in the same sets of hyperparameters (if the hyperparameter exists in both methods). We will clarify this issue, and thanks for pointing this out.

Weakness 3: With DPO training, the probability of rejected samples decreases regardless of NLL (see Fig3 a). Addition of NLL affects the chosen samples and makes them go up in probability. In contrast, their probability doesn’t go up much in DPO training without NLL. The chosen samples are correct solutions, so their higher probability means the model is likely to generate that correct solution. The direct way to measure if the model is generating correct solutions or not is to look at the test set accuracy, which we report in the paper.

In other words, our hypothesis is that without NLL, both chosen and rejected probabilities decrease, and given that probabilities sum to one over sequences, the probabilities might be going to unforeseen low-quality sequences.

Q1: Yes. Please see weakness 2 above. Thanks again for raising this point.

Q2: We have eyeballed ~30 generations for each dataset; to the best of our ability (given that MATH is difficult), if the answer matches, then our generated CoT is almost always correct for these particular datasets, especially GSM8K and MATH.

Q3: Generation is relatively cheap these days given the low-precision toolkits (e.g., vLLM; https://github.com/vllm-project/vllm) – using 8 V100 (32GB) GPUs, we can sample a few thousand responses (for our tasks) from 70B models in just a few minutes. All training in this paper is done using eight nodes, each containing eight A100 GPUs (briefly mentioned in Section 3); each 10 steps take around 2 minutes; training 5000 steps (plus validation and checkpoint saving) takes less than a day.

We thank the reviewer for the review and the support!

We presented DPO results in the paper. We showed that DPO in iteration 1 does significantly worse than our RPO’s iteration 1; hence we did not try further iterations of standard DPO (aka Iterative DPO). The reviewer raises a good point that we can check what future iteration results are like. SPIN doesn’t include the NLL term in IRPO, so it’s closer to iterative DPO than IRPO; moreover, SPIN assumes always using the reference CoT but we do not; they also report only modest gains in reasoning tasks.

The reviewer raises a good point about self-rewarding LM. The original algorithm requires generating prompts and evaluating the sampled responses using the LM itself. IRPO assumes knowing what the correct answer is (e.g., the answer may be 7.5 for a math question), but we wouldn’t know the correct answers for augmented prompts. LMs are quite bad at generating evaluations of a response (e.g., according to RewardBench – although there has been some progress in the past few weeks). So we believe that self-rewarding LM cannot be successfully directly applied using llama-2-70b-chat, but using IRPO in an unsupervised fashion is a really promising research avenue to keep exploring!