Dear Reviewer V4Mi,

Thanks for your careful and insightful comments, we provide our responses to your questions in the following.

W1: lack of clarity on RLR

A1: The RLR uses the filtered instructions in the data selection stage, but the responses are the corresponding pairwise responses in the preference dataset; we use the samples in the preference dataset to refine the reward model ensembles.

W2: Limited evaluation dataset

A2: To answer this question, we conduct addtional experiments based on the ultra feedback dataset [1] to train the policy, the results are presented in the following.

Reward Model Evaluation Results

• Helpful results

• Harmless results

GPT Comparision Results

From the above results, we can see that UGDA still outperforms than all the baselines, which further verify the effectiveness of our architecture. Additionally, we do not apply different random seeds for the new dataset experiments due to the time limitation of the rebuttal. If needed, we will add the results with the error bars.

References

[1] Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W., ... & Sun, M. (2024, June). ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback. In Forty-first International Conference on Machine Learning.

Dear Reviewer Jzfj,

Thanks for your careful and insightful comments, we provide our responses to your questions in the following.

W1: Comparison with iterative PPO/DPO method and reward ensembles.

A1: We have conducted the reward ensembles method UWO [1] in the current version of our paper as the baseline. Additionally, follow your advice, we conduct the experiment based on the DPO [2] and iterative DPO [3], the results are based on the Gemma 7B policy.

Benchmark Evaluation Results

Reward Model Evaluation Results

GPT Comparision Results

From the results, we can see that iterative DPO outperforms DPO, but is less effective than UGDA. This may be because of the superior generalization ability of PPO compared with DPO.

References

[1] Coste, T., Anwar, U., Kirk, R., & Krueger, D. (2023). Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:2310.02743.

[2] Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., & Finn, C. (2024). Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.

[3] Xu, J., Lee, A., Sukhbaatar, S., & Weston, J. (2023). Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.16682.

W2: Relabeled reward data validation.

A2: We provide the comparision of the relabeled reward data with human or GPT annotator in Table 1, which shows the similarity of the human labelers and GPT-4, which is suffecient for supporting us to use GPT annotator in the following experiments. If needed, we are willing to conduct more experiments to verify the results with your advice.

Q1: Validation dataset construction.

A1: We build the validation dataset by randomly selecting 10% samples of the perference dataset, specially each samople is constructed by the instruction with chosen response. We will add the related description in the experimental setup in our final version.

Q2: The computational complexity.

A2: The complexity of reward model training with ensembles scales with the preference data size $\mathcal{D}_p$ and the number of ensembles $K$, which is $\mathcal{O}(|\mathcal{D}_p|\cdot K)$.

For the gradient based data selection, we divide this into two stage to analysis the complexity. The gradient feature computation is the most expensive step, and the cost scales linearly with candidate dataset size $|\mathcal{D}\_{\text{train}}|$, the number of checkpoints $N$ and the projection dimension $d$, which is $\mathcal{O}(|\mathcal{D}\_{\text{train}}|\cdot N \cdot d)$. From the experimental results in our work, we can see that a small projection dimension is sufficient.

The data selection process requires minimal computation, and scales with the size of $|\mathcal{D}\_{\text {train}}|$ and $|\mathcal{D}\_{\text{val}}|$, which is $\mathcal{O}\left(|\mathcal{D}\_{\text {train}}| \cdot\left|\mathcal{D}\_{\text {val }}\right|\right)$.

Furthermore, we will add the above analysis to our final version.

Dear Reviewer wH1u,

Thanks for your careful and insightful comments, we provide our responses to your questions in the following.

W1: Motivation and algorithm design

A1: In the training procedure of RLHF, for the gradient-based data selection, our motivation is to identify the most relevant data from these extensive datasets to effectively develop specific capabilities, which can be used for the further reward model refining. Specifically, we study the problem of identifying the influence of training samples on the prediction of the validation samples. The data influence can be learned by gradient information [1,2].

Additionally, our experiments based on the complex QA tasks, thus we can not directly define the samples that are out of distribution by the data visualization. Thus, we use a heuristic way to define the out of distribution samples by the length of the instruction, specially, to achieve the fair comparision, we sort the samples by the length of the instructions, we use 25% of them to be relabeled by the GPT annotator and further refine the reward models. We show the results of the policy optimization in the following, and we use OOD (length) to represent the above method.

From the results, we can observe that, if we use the length of instructions as the criterion to select the samples, the results are not much more superior than the random selection for the policy optimizaiton. This may be because the samples that influence the gradient may not only be the OOD samples, which means, in the policy optimization process, the learned policy can generalize to some OOD scenarios. Thus, these OOD samples may not be the most important factor affecting the gradient.

References

[1] Pruthi, G., Liu, F., Kale, S., & Sundararajan, M. (2020). Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33, 19920-19930.

[2] Han, X., Simig, D., Mihaylov, T., Tsvetkov, Y., Celikyilmaz, A., & Wang, T. (2023). Understanding in-context learning via supportive pretraining data. arXiv preprint arXiv:2306.15091.

W2: Evaluation

A2: As shown in the scaling law of the reward model [1], the bigger reward model has strong ability, thus we use it to evaluate the result. To answer this question, we additionally use the Gemma-2 27B to evaluate the results.

• Helpful results

• Harmless results

From the results, we can see that our UGDA still outperform than other baselines, the results show that our method can acheive better performance with different bigger reward model evaluation, which is independent of the specific model structure.

References

[1] Gao, L., Schulman, J., & Hilton, J. (2023, July). Scaling laws for reward model overoptimization. In International Conference on Machine Learning (pp. 10835-10866). PMLR.

W3: Presentation.

A3: We will follow your advice and revise all the misleading presentations. And we will also check the full paper again to polish the presentation in our final version.

Thanks for providing additional experiment results and clarifications. The authors have addressed my concerns. The proposed method has more solid empirical justifications and I have raised the score.

The paper can still benefit further from a clearer presentation in the revision.

Dear Reviewer DX3G,

Thanks for your careful and insightful comments, we provide our responses to your questions in the following.

W1: Lacks comparisons to active learning and data augmentation baselines.

A1: We conduct experiments with the ablation study which only uses uncertainty. This can be seen as an active learning-based method.

To achieve the fair comparision, we have conducted the experiments based on the random data augmentation as the baseline of our method in the experiments, which can be seen as the data augmentation baseline. The results are shown in Section 5.3.

W2: Off-policy methods.

A2: We follow your advice to conduct experimental analysis based on the DPO [1] and iterative DPO [2]. Specially, the for the iterative DPO, to achieve the fair comparison, we only apply one round iteration with 25% samples for data augmentation, we use GPT annotator as the reward model for iterative DPO. Moreover, the experimental results are based on the Gemma 7B policy.

Benchmark Evaluation Results

Reward Model Evaluation Results

GPT Comparision Results

From the results, we can see that iterative DPO outperforms DPO, but is less effective than UGDA. This may be because of the superior generalization ability of PPO compared with DPO.

References

[1] Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., & Finn, C. (2024). Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.

[2] Xu, J., Lee, A., Sukhbaatar, S., & Weston, J. (2023). Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.16682.

Q1: Could you apply the method to the full fine-tuning reward ensembles?

A1: Our method can be directly applied to the full fine-tuning without additional modification, but due to the computational cost and the time limitation, we can not provide the fine-tuning results during the rebuttal, we are sorry for this. From the results of the LoRA fine-tuning, the effectiveness of our UGDA can be verified.