The paper does not address the scalability of the proposed methods to even larger models or different types of LLMs beyond the ones tested.

Response: We thank the reviewer for the suggestion. Here we believe provide a theoretical analysis would be enough since we know exactly how much memory and time each of the proposed algorithm would need comparing to SFT. For Algorithm 1 in the paper, we need to maintain a reward model and a policy model (which is the LLM), and this is doubling the standard LLM fine-tuning. In short, the memory consumption and computation time of Algorithm 1 is similar to the standard RLHF process, where we also have the reward learning and policy optimization processes; For Algorithm 2, we simply need to maintain the policy (LLM) model, and the memory consumption would be exactly the same as the standard SFT, whereas the computation time would involving generating for the entire training sample, which would be of similar level as the standard RLHF process. We summarize the memory consumption and the computational time of the proposed methods in the table below, assuming that the reward and policy models are of same size. Here Forward means the memory required for storing a model in inference mode, and Backward is the memory required for storing a model in training mode, including weights, activations and gradients; also SFT means the computational time as standard SFT, and Generation means the time to generate continuations for each of the input training prompts. 2SFT+Generation is roughly the same time as standard RLHF.

The evaluation results only present the empirical performance but lack experiments on computation costs introduced by this method.

Response: We thank the reviewer for the question. We refer to the answer to the previous question, where we do include the computational time analysis of the proposed methods. In short, Algorithm 1 is similar to RLHF, and Algortihm 2 is similar to a generation plus a DPO process.

It seems that RFT and IRFT perform equally. What's the takeaway from these two methods? Moreover, how to choose them wisely.

Response: The difference between RFT and IRFT are similar to that of RLHF and DPO. RFT produces an explicit reward model for further generalization and IRFT is more memory and time efficient. We advocate using IRFT if you are not looking for an explicit reward model from the demonstration dataset (see our general response for the detail of the generalization ability of the reward model).

Typo in L226, there's an error in referring to the Appendix.

Response: We thank the reviewer for the reminder. We modfied it correspondingly.

In L204-206 "In 204 practice however we take a relatively small T and large K, because frequent on-line sampling is time 205 consuming" How does this parameter choice affect the final performance, especially compared to large T and Large K?

Response: We did our test on a large T and small K, i.e. we generate samples for every training batch and do the update. The performance is largely similar to the SFT performance (less than 1% lift from the baseline), as shown below (see our general rebuttal to all reviewers for the implementation details):

where T=#batch is the case when $T$ is large and $K$ is small.

In short, we believe that generating too frequently would not only be time-consuming, but also detrimental to the model performance (since we take to much variance into account at each stochastic gradient step for generation). We would recommend a reasonable frequency in generation which leads to the optimal performance, such as what we observed when we generate 5 to 10 times for the entire training dataset.

The experiments are not very convincing. For the Open LLM leaderboard experiments, the gain is really small (around 1 or 2%). It seems likely that this gain is due to variance in the training process, especially since IRFT has ~1% variance in different hyperparameter settings.

Response: To further show the strength of our algorithms, we tested two extra experiments and inclide the results in the general rebuttal to all reviewers. First, we test our algorithm for 7b models with parameter-efficient fine-tuning (PEFT) settings and show that the proposed method could yield a 2.3% improvement comparing to the baselines. Additionally, we also conduct a reward accuracy analysis in the general rebuttal where we show that the implicit reward learned by our method is more accurate than that of SFT and SPIN (please also see our answer to your third question). We hope this could give more strong evidence on the superiority of the proposed method.

There isn’t much discussion of the computational cost of the algorithms.

Response: We thank the reviewer for bringing this issue up. We refer to the general rebuttal to all reviewers where we include a detailed computational cost analysis.

How accurate is the learned reward model?

Response: We conduct a very simple experiment to show how accurate the reward learned by our model is. Since we train our 7b model with a high-quality dataset (ultrachat) and the corresponding implicit reward $r=\log(\frac{\pi_{\mathbf{\theta}}}{\pi_{\text{ref}}})$ should already be pretty accurate in terms of distinguish the good over the rejected continuations. Therefore we construct the implicit reward by equation $r=\log(\frac{\pi_{\mathbf{\theta}}}{\pi_{\text{ref}}})$ where we compare different $\pi_{\mathbf{\theta}}$ (pretrained, SFT, SPIN and IRFT) on the ultrafeedback dataset (note that we did not do training on this dataset) which is a preference dataset. We believe that the accurate reward model $r=\log(\frac{\pi_{\mathbf{\theta}}}{\pi_{\text{ref}}})$ should be able to distinguish the preferred and rejected continuations, and we compute the ratio of $r(\text{preferred})>r(\text{rejected})$ and obtain the follow table:

where we can see that IRFT significantly improves the implicit reward's distinguishability of chosen over rejected. We hope this addresses the reviewer's concern.

The experimental setting is not very clear. Is Algorithm 1 used on the top 10K data selected by the reward model? If so this is really problematic: shouldn’t you fine tune on the whole dataset (or a randomly downsampled subset)? Can you assume that you will have access to 10K samples with high reward scores? And wouldn't just be better to use the reward model used for data selection for RL?

Response: We thank the reviewer for the question. For the experiment on Algorithm 1, we want to demonstrate that with high quality demonstration data, our proposed Algorithm 1 is more capable of learning an accurate reward model that is able to distinguish the good over the rejected continuations, by assigning higher scores to the good continuations. Note that we need to construct a "high quality dataset" at the first place. The experiment setting is that we use a well-established reward model (beaver-7b-v3.0-reward) to pick up good demonstration data (which are the top 10k data in terms of reward score), and fairly train the model with SFT and our method on these 10k data, then compute the rewards of the continuations generated by SFT trained model and our trained model.

We are not assuming we always have access to top 10k samples, instead, top 10k sample is used as a relatively simple way to obtain high quality data to verify our presumption that our method is more capable of getting better reward if we have good demonstration data.

In practice, the implication of this experiment is that, if you have good demonstration data (under certain score criteria) at hand and would like your language model to align with it, then our proposed method should be more capable of extract this underlying score criteria and better aligning with it. We do not assume that we always have a well-established reward model on hand in practice.

How robust is IRFT to hyperparameter setting compared to SFT?

Response: We thank the reviewer for the question and we believe IRFT is robust to hyperparameters in general. We test all our experiment of IRFT, SPIN and SFT with the same learning rate, epochs and batchsize. We do have new hyperparameters $T$ and $K$. However when you fix $T$, total epochs and the training dataset, $K$ is automatically determined. Therefore the only parameter one needs to determine in extra is $T$. We found that in practice taking $T=5$ (or multiples of 5, meaning training for more epochs) would result in the best performance for both 1b and 7b models. In short, the only extra parameter we have is $T$ and we believe $T=5$ is readily a good candidate for most of the alignment tasks.

Is 1-2% increase on the OpenLLM benchmark meaningful?

Response: Yes, we do believe that it is meaningful. The openLLM leaderboard benchmark is one of the most heavily-used (SPIN also primarily used this) influential benchmark in the community where it does not rely on large models such as GPT4 but only on multiple downstream tasks to test the performance of different LLMs. For models like Llama3-70B, they in general also only achieve only 1 to 2% increase over the previous state-of-the-art (if you check their hugginface webpage, they also tested on tasks such as MMLU, GSM8K and Winogrande). Therefore we believe we are making meaningful progress using the same tasks as SPIN on 7b models.

The experimental results seem to support the efficacy of the proposed algorithms, however the gains across various tasks with IRFT seem fairly marginal (it would help if Table 2 and 3 contained error bars, perhaps with policies finetuned with different seeds). As it stands, it seems like different benchmarks results are sensitive to different hyperparameters (choice of T), and the baselines outperform IRFT on 3/6 of the benchmarks in Table 3, suggesting that the additional complexity of the method does not necessarily lead to improved performance across all tasks. For example, the best average performance of IRFT is 61.03, whereas the best baseline achieves 61.02.

Response: To further show the strength of our algorithms, we tested our algorithm for 7b models with parameter-efficient fine-tuning (PEFT) settings and show that the proposed method could yield a 2.3% improvement comparing to the baselines. The result tables are in our general rebuttal to all reviewers. We hope this could give more strong evidence on the superiority of the proposed method.

Furthermore, the results seem more mixed in Table 3 compared to Table 2, suggesting that it's not clear the method is still effective as model sizes increase.

Response: Again please see our new results included in the general rebuttal to all reviewers. First, we show that for the LoRA setting, our proposed method significantly outperforms SFT. Additionally, we also conduct a reward accuracy analysis in the general rebuttal where we show that the implicit reward learned by our method is more accurate than that of SFT and SPIN. We believe the extra evidence should further support that our proposed method yields significant improvements over SFT.

It would help to present the SFT results in Table 3. Even though the authors note that further SFT could degrade performance, it would help to see how effectively using the demonstration data with IRFT or SPIN compares against naive SFT.

Response: We thank the reviewer for this suggestion. Our claim "SFT could degrade performance" is largely from [1] page 8 "SFT on further epochs 2 and 3 fails to yield more than 1% improvement". We also verified this during the rebuttal period. Due to the limit time during the rebuttal period, we were only able to conduct the experiment on LoRA and the result can be see in the general rebuttal to all reviewers, where we indeed see less than 1% lift of SFT from the baseline, verifying the claim made by [1]. We will rephrase "SFT could degrade performance" to "SFT on further epochs 2 and 3 fails to yield more than 1% improvement" in the revised paper.

Minor note: line 157 seems incomplete, "even when the demonstration policy is"...(extreme)?

Response: We thank the reviewer for the reminder. We meant to say "even when the demonstration policy is extreme" and will modify it accordingly.

Minor note: the notation is a bit confusing, with theta referring to both the model parameters and the reward model parameters. For clarity, it would help if separate variables were used.

Response: We thank the reviewer for the reminder. For RFT (Algorithm 1), the $\theta$ is the parameter for reward since the policy is determined by tge reward; For IRFT (Algorithm 2) $\theta$ is the parameter for the policy. We will modify the $\theta$ in Algorithm 1 into $\phi$ in our revised paper.

Minor note: missing reference to appendix in line 226.

Response: We thank the reviewer for the reminder. We modfied it correspondingly.

Minor note: grammar error in line 320 -- "SPIN and IRFT are both capable of further improv(ing) the performance.."

Response: We thank the reviewer for the reminder. We modfied it correspondingly.

The paper is missing more thorough discussion on the method’s limitations and weaknesses (e.g. sensitivity to T, more complicated training process compared to SFT, etc.).

Response: We thank the reviewer for raising this issue up. We believe IRFT is robust to hyperparameters in general. We test all our experiment of IRFT, SPIN and SFT with the same learning rate, epochs and batchsize. We do have new hyperparameters $T$ and $K$. However when you fix $T$, total epochs and the training dataset, $K$ is automatically determined. Therefore the only parameter one needs to determine in extra is $T$. We found that in practice taking $T=5$ (or multiples of 5, meaning training for more epochs) would result in the best performance for both 1b and 7b models. In short, the only extra parameter we have is $T$ and we believe $T=5$ is readily a good candidate for most of the alignment tasks.

Please refer to our general rebuttal to all authors for the computational cost analysis.

References:

[1] Chen, Zixiang, et al. "Self-play fine-tuning converts weak language models to strong language models." International Conference on Machine Learning (2024).

1. The training procedure is complicated, meaning there are many issues in tunning. Is there any computation cost analysis for better understanding the limitations of this method?

Response: We thank the reviewer for bringing this issue up. For Algorithm 1 in the paper, we need to maintain a reward model and a policy model (which is the LLM), and this is doubling the standard LLM fine-tuning. In short, the memory consumption and computation time of Algorithm 1 is similar to the standard RLHF process, where we also have the reward learning and policy optimization processes; For Algorithm 2, we simply need to maintain the policy (LLM) model, and the memory consumption would be exactly the same as the standard SFT, whereas the computation time would involving generating for the entire training sample, which would be of similar level as the standard RLHF process. We summarize the memory consumption and the computational time of the proposed methods in the table below, assuming that the reward and policy models are of same size. Here Forward means the memory required for storing a model in inference mode, and Backward is the memory required for storing a model in training mode, including weights, activations and gradients; also SFT means the computational time as standard SFT, and Generation means the time to generate continuations for each of the input training prompts. 2SFT+Generation is roughly the same time as standard RLHF.

2. Though the author discusses SPIN, the truth is that equation 4.7 in SPIN is very similar to equation 6 in this paper.

Response: As we stated from Line 251 page 7, our Algorithm 2 include the algorithm in SPIN as a special case and our inverse reinforcement learning framework naturally introduces a difference of log probabilities formulation. We believe that the claim "the truth is that ... is very similar" is not very accurate, but instead we would say that the algorithms coincide under certain conditions. There is still a key difference between (4.7) in SPIN and our equation 6: in our equation 6 we take the negative sample $\tilde{y}$ generated by our current model $p_{\mathbf{\theta}}$, whereas in SPIN the negative sample $\tilde{y}$ is sampled from the model from the based model of current training epoch $p_{\mathbf{\theta}_t}$.

3. The experimental results are not strong. It did not surpass the SPIN by a large margin. Are there any other baselines that can be incorporated such as DPO?

Response: We don't think it necessary to compare with DPO. As we made it clear in the abstract, our proposed framwork is a method for the supervised fine-tune stage of alignment, which means that we do not have access to a preference dataset but only a demonstration dataset. Direct preference optimization (DPO) is not designed for aligning demonstration dataset. In fact, our proposed method can be regarded as a DPO-type algorithm for the supervised fine-tune stage. So we cannot compare our method fairly with DPO, at least not when DPO utilizes preference data. However, in order to further show the strength of our algorithms, we tested our algorithm for 7b models with additional parameter-efficient fine-tuning (PEFT) settings and show that the proposed method could yield a 2.3% improvement comparing to the baselines. The result tables are in our general rebuttal to all reviewers. We hope this could give more strong evidence on the superiority of the proposed methods.

4. How can the reward model obtain the generalization power? Can the author say more about it, in my understanding, the main issue that affects the generalization is the scale of the demonstration dataset.

Response: The power of this proposed method is that we can learn a reward model even without the preference dataset, and we completely agree with the reviewer on this point that the generalization power relies on the scale and quality of the demonstration dataset. For example, we train our 7b model with a high-quality dataset (ultrachat) and we believe that the corresponding implicit reward $r=\log(\frac{\pi_{\mathbf{\theta}}}{\pi_{\text{ref}}})$ should already obtain certain generalization ability. Therefore we did a simple test below to show the generalization power of the reward model learned from the proposed algorithm: we construct the implicit reward by equation $r=\log(\frac{\pi_{\mathbf{\theta}}}{\pi_{\text{ref}}})$ where we compare different $\pi_{\mathbf{\theta}}$ (SFT, SPIN and IRFT) on the ultrafeedback dataset (note that we did not do training on this dataset) which is a preference dataset. We believe that the reward model $r=\log(\frac{\pi_{\mathbf{\theta}}}{\pi_{\text{ref}}})$ with a better generalization ability should be able to distinguish the preferred and rejected continuations, and we compute the ratio of $r(\text{preferred})>r(\text{rejected})$ and obtain the follow table:

where we can see that IRFT significantly inproves the implicit reward's distinguishability of chosen over rejected, indicating a superior generalization ability.