Q-Adapter: Customizing Pre-trained LLMs to New Preferences with Forgetting Mitigation

W1: In Table 1, under the Category: Business, the PR (PPO) method shows a significantly lower performance compared to the other baselines (this is not as evident in other categories).

We used the PPO algorithm implemented by TRL [1] and adopted the same LoRA training config as other methods. The phenomenon is that PPO is extremely unstable in the business category of the dataset. We tried a few runs and the results shown in Table 1 are the best outcome we observed in the experiments. We speculate that the instability of online PPO [2, 3] may be exacerbated in our training settings. Improving the stability of PPO can be an important direction to enhance LLM customization.

[1] Leandro von Werra, etc. TRL: Transformer Reinforcement Learning. https://github.com/huggingface/trl

[2] Huang, S., Dossa, R. F. J., Raffin, A., Kanervisto, A., & Wang, W. (2022). The 37 implementation details of proximal policy optimization. The ICLR Blog Track 2023.

[3] Zheng, R., Dou, S., Gao, S., Hua, Y., Shen, W., Wang, B., ... & Huang, X. (2023). Delve into PPO: Implementation matters for stable rlhf. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following.

W2: The study could include full parameter fine-tuning and additional baselines for comparison, such as other forgetting mitigation techniques or model fine-tuning strategies (e.g., NEFTune), to provide a more comprehensive evaluation of Q-Adapter's performance.

Thanks for your suggestion. We add the comparisons to NEFTune, along with KTO and IPO proposed by other reviewers. Please refer to global_Q1 in our global comment above. Since our proposed Q-Adapter only learns parameters from the LoRA modules, we keep the same parameter size for all baselines in training and do not tune full parameters to ensure fair comparisons.

W1: Also they only use GPT-4o as a judge (Judgments from LLMs part). Although it is common to use LLM-as-a-judge for similar tasks, the authors may consider using more models, especially those with more explainability or reasoning ability for their ranking and evaluation.

Thanks for your suggestion. We introduce two more powerful LLMs, Claude-3.5-Sonnet and Deepseek-chat, to evaluate generation qualities. We present the results in global_Q2 from our general comment above, showing that results among these models are relatively consistent. Note that we only adopt LLM judgments when objective metric is inaccessible, particularly in evaluating generation qualities from downstream test data.

Q1: In the experiment, the authors do not compare their method with some SOTA methods other than PPO and DPO (See Questions); Authors may consider referring to and comparing with: https://arxiv.org/abs/2404.14723 https://arxiv.org/abs/2402.01306 https://arxiv.org/abs/2405.00675 https://proceedings.mlr.press/v238/gheshlaghi-azar24a.html

We have added three baselines, KTO [1], IPO [2], and NEFTune [3], where KTO and IPO are policy regularization methods mentioned above and NEFTune are an enhanced SFT method mentioned by Reviewer 16NH. Please refer to global_Q1 for the new results, from which we find that our method, Q-Adapter can outperforms these methods.

[1] Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., & Kiela, D. (2024). KTO: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306.

[2] Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., & Calandriello, D. (2024). A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics (pp. 4447-4455).

[3] Jain, N., Chiang, P. Y., Wen, Y., Kirchenbauer, J., Chu, H. M., Somepalli, G., ... & Goldstein, T. (2023). NEFTune: Noisy embeddings improve instruction finetuning. arXiv preprint arXiv:2310.05914.

W1: According to Section 4, the effectiveness of this framework highly depends on the quality and relevance of the training preference data. The author may further discuss the impact of noise and the limitation of the model's ability to align with user preferences accurately within more datasets.

We do not assume the quality and particular relevance of the training preference data in Section 4. These factors on data and noise may affect the performance of both our method and our baselines. In fact, the dataset we used is not perfect. For instance, the HH-RLHF dataset is shown to contain noise [1], but the experimental results indicate that our method outperforms the baselines. Further investigating how noise or model capability affects LLM customization is an interesting topic though not falling to our main contributions. We have revised our paper to add the discussion in Section 7.

[1] Yeh, M. H., Tao, L., Wang, J., Du, X., & Li, Y. (2024). How Reliable Is Human Feedback For Aligning Large Language Models?. arXiv preprint arXiv:2410.01957.

W2: In Section 5, the authors only compare the Q-Adapter with two Policy Regularization baselines, DPO and PPO. The authors should include more up-to-date baselines in their experiments to better validate the effectiveness of their method.

We originally choose DPO and PPO as the representative methods which adopt policy regularization to mitigate forgetting. To include more up-to-date approaches, we have added three baselines including KTO, IPO and NEFTune. Please refer to global_Q1 and global_Q2 in the global comment above for the results, where Q-Adapter still outperforms these baselines.

Q1: According to Section 5.1, the evaluation metrics may need further clarification. It seems that in all tasks, including MMLU, MMLU Pro, GSM8K, BBH, and IFEval, the Base Model shows the highest score compared to other methods.

Our evaluation metrics in general benchmarks, including MMLU, MMLU Pro, GSM8K, BBH, and IFEval, is to measure the forgetting issue when we fine-tune models in downstream data. The performance drop is expected since our customization process uses a preference dataset with different traits from the math, reasoning, or instruction following capabilities indicated by the benchmarks above. In general, our method provides a better trade-off between general capabilities and downstream performance. We present a further clarification in global_Q3.

Q2: In Section 5.2, the authors evaluate the model's capabilities in learning new preferences only by demonstrating the evaluation of OpenAI GPT-4o. This evaluation is insufficient; the author may further discuss the evaluation results from other LLMs and human labelers.

Thanks for your suggestion. Although the evaluator of GPT-4o can contain bias when make judgments, the auto-evaluation process of AlpacaEval along with the length-controlled technique can highly correlate to humans. In global_Q2, we present the evaluation results using two powerful LLMs, Claude-3.5-Sonnet and Deepseek-chat. We find these models provide relatively consistent results and Q-Adapter still maintain good performance.

Thank you for the clarification; I will keep my score.

W1: In Section Judgments from LLMs, the evaluation with GPT-4o is not convincing. The LLM-as-a-judge is intended to provide an objective and robust assessment of model outputs across different dimensions. However, simply using GPT-4o for evaluation actually introduces potential biases and lacks the transparency needed for reliable performance validation.

Thanks for your suggestion. As stated in our experiment setup, we ONLY use LLM judgments when there is NO objective metrics to quantify the generation qualities in downstream test data. When it is applicable to use objective metrics (i.e., measuring anti-forgetting capabilities), we have adopted them in general benchmarks in Table 1, including widely used assessments in MMLU, BBH, GSM8k and so on. Moreover, to ablate potential bias of GPT-4o, we add two more evaluators, Claude-3.5-Sonnet and Deepseek-chat, to evaluate generation texts from different methods. We present the results in global_Q2 from the global comment above, where these evaluators show relatively consistent results.

W2: For the experimental result, the authors may add more analysis as the performance of this Q-Adapter does not significantly outperform baseline methods. For example, there is minimal improvement in the BBH dataset (Table 1), and without error analysis, it is challenging to demonstrate the method's effectiveness.

We use Table 1 to measure the anti-forgetting capability of different methods during customization. In general, our method achieves the highest scores in 19 out of 20 comparisons from five benchmarks in four different datasets. To further describe the degree of forgetting, we find that the average score decrease of Q-Adapter is 30.0% less than the best baseline (Replay) and 43.2% less than the average performance of all baselines in Table 1, indicating the advantage of our method.

W3: Also the paper lacks a thorough discussion on the computational overhead or practical considerations associated with deploying Q-Adapter.

We state the training cost of Q-Adapter in Appendix B.2 in our original paper. We can train Q-Adapter for 3 epochs in a single machine with 4x NVIDIA GeForce RTX 4090 GPUs. In inference time, the 8B version of Q-Adapter can be deployed in a single NVIDIA GeForce RTX 4090 GPU. Although Q-Adapter requires both the outputs from the base and the adapter model, we propose a caching workaround, which can save 25% the inference time compared to simply forwarding twice. We have updated Appendix B.2 to include the discussion. Thanks for your comment.

Q1: The result in Table 1 is confusing, as the base model seems to have the best performance. Could any training harm the model's performance?

It is potential to harm the model’s general benchmark scores when we train the model in a different domain-specific data distribution. As stated in W2, the results are expected since all the methods suffer from the forgetting issue when we tune a model in a specific downstream dataset, which is our customization process intending to address. Please refer to global_Q3 for further clarifications.

Q2: The paper employs the DSP and HH-RLHF datasets but does not explain the criteria for selecting these datasets over other domain-specific or preference-aligned datasets (Section 5.1, Experimental Setup). What are the key attributes of these datasets that make them particularly suitable for evaluating Q-Adapter’s capabilities in preserving and adapting model preferences?

In our original submission, we present details on why we choose DSP and HH-RLHF at the beginning of Sections 5.2 and 5.3, respectively. In Section 5.1 of the revised PDF, we have added a new paragraph on the reasons for choosing them to conduct experiments. In short, we do not deliberately select datasets that will give our method an advantage over the baselines. The reasons for choosing DSP and HH-RLHF are simply that 1) they are suitable for our considered customization scenarios and 2) they allow us to easily define $r_1$ and $r_2$, thereby facilitating subsequent evaluations.