Thank you for your helpful comments! We appreciate your comments that our method may have a great impact on the LLM community. We address your questions below.

Comment 1.1: "The major contribution of this paper is to propose the problem of RLHF in a continual manner, however, the motivation and necessity of this problem is lack of detailed explanation and support, hindering readers’ understanding of its value.

• Response: The motivation of Continual RLHF: In real-world applications, learning continuously changing human preferences is more practical than learning invariable human preferences. For example, the progression from the onset of the COVID-19 virus in human society to widespread infections and finally to achieving herd immunity has seen corresponding changes in government policies and human perspective. An AI agent that keeps pace with the times should exhibit behavior that aligns with current government policies and human understanding preferences at different stages, rather than remaining static.

However, traditional alignment methods, namely Reinforcement Learning from Human Feedback (RLHF)[1,2], lack flexibility for continual learning (CL) of human preferences. Existing work[3] often necessitates retraining models to adapt to dynamic preferences, which needs huge computation resources. Hence, there is a pressing need for research into continual alignment methods to address this limitation, enabling LLMs to better adhere to evolving human preferences and values while generating helpful responses. We note that reviewers za1s and Loz4 mentioned the motivation of CPPO under Strengths as well.

Comment 1.2: "The authors should introduce more related work that retrain or finetune the LM when new data arrive. "

• Response: According to your suggestion, we have added more related work in Section 5.2 of the new PDF.

Comment 1.3: " Furthermore, the storage and computational cost of different approaches should be analyzed, for example, the retraining methods, replay-based methods and regularization-based methods."

• Response: The memory and computational complexity of the continual learning methods depend on the actual experiment setting. In general, the storage complexity of Replay-based methods is positively correlated with the number of replayed samples and linearly increases with the number of tasks. Regularization-based methods, such as EWC, typically require storing the weights of historical models and Fisher matrix information, leading to storage costs that linearly increase with the number of tasks. On the other hand, Re-training methods necessitate retraining the entire historical dataset, resulting in cumulative computational costs being quadratic with respect to the number of tasks.

The methods in Section 5.2 have been applied in NLP scenarios for continual pre-training, task incremental learning, class incremental learning, and domain incremental learning[4]. To the best of our knowledge, these methods have not been experimentally validated in the continual learning of the RLHF scenario.

References

[1] Nisan Stiennon, et al. Learning to summarize from human feedback, arxiv 2020.

[2] Long Ouyang, et al. Training language models to follow instructions with human feedback, arxiv 2022.

[3] Yuntao Bai, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arxiv 2022.

[4] Magdalena Biesialska, et.al. International Committee on Computational Linguistics. 2020.

Comment 2.1: "The experiment part is weak. In the main experiment to evaluate the continual learning ability of CPPO in NLP tasks, only one benchmark Reddit TL;DR summarization is considered. What’s more, the length of the continual task stream is only 2, which is not enough for a solid evaluation of continual learning ability compared to related work... "

• Response: To further validate the proposed method, we add a new benchmark for Domain-Incremental Learning (DIL) on 18 domains, comprising 3 tasks. We build this benchmark based on the Stanford Human Preferences Dataset (SHP)[1] dataset. Our task design methodology can also be extended to sequences of 6, 9, or even 18 tasks. Here, we only present the final experimental results. The details of the SHP dataset, task design, and evaluation metrics can be referenced in the new PDF Appendix Section H.

Comment 2.2: "The proposed CPPO method can also be applied to other continual reinforcement learning tasks other than RLHF. Simpler conventional control tasks like DoorGym should be included to further validate the soundness of CPPO."

• Response: We believe that the evaluation of DoorGym is out of the scope of our paper. Our research focuses on the alignment of LM, following previous works [2-6] which are evaluated only on NLP tasks. In this paper, we have evaluated CPPO on 5 representative NLP datasets, including Reddit TL;DR for summarization, CNN Daily Mail for summarization, IMDB for sentiment text generation, Random Walks for generating the shortest path, and SHP for open-domain QA.

References

[1] Kawin Ethayarajh, et al. Understanding dataset difficulty with V-usable information. PMLR, 2022.

[2] Yuntao Bai, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arxiv, 2022.

[3] Yuntao Bai, et al. Constitutional ai: Harmlessness from ai feedback. arxiv, 2022.

[4] Rafael Rafailov, et al. Your language model is secretly a reward model, NeurIPS, 2023

[5] Nisan Stiennon, et al. Learning to summarize from human feedback. arxiv, 2020.

[6] Zheng Yuan, et al. Rrhf: Rank responses to align language models with human feedback without tear, NeurIPS, 2023

Thank you for your helpful comments! We appreciate your comments about our main idea and the motivation behind it make sense. We address your questions below.

Comment: "I wonder about the performance of 1) $M_{\pi_2}$ on the Task-1 test set and 2) $M_{\pi_1}$ on the Task-2 test set. The result of the first experiment can show how much knowledge the model retains for the first task. In addition, the second experiment is supposed to show a mediocre result to prove the two tasks have a clear difference and that the experiment setting is indeed a continual learning setting. The performance gap of $M_{\pi_1}$ and $M_{\pi_2}$ on Task-2 can reveal to what extent the two data distributions mismatch."

Response: The reference PM score of $M_{\pi_2}$ on Task 1 can be computed by $rPM(M_{2}, \mathbb{D}_1^{test} )=rPMS_1 - SFR$.

Here, we supplement the performance of $M_{\pi_1}$ on Task 2, comparing CPPO with PPO.

Comparing the evaluations in columns 2 and 4 for $M_{\pi_1}$ and $M_{\pi_2}$ on Task-2, we observe that model $M_{\pi_1}$ significantly underperforms $M_{\pi_2}$. This indicates a substantial difference in the data distributions between the two tasks.

Comparing the evaluations in columns 1 and 3 for $M_{\pi_1}$ and $M_{\pi_2}$ on Task-1, we observe that the policy model trained with PPO exhibits forgetting, indicating a risk of catastrophic forgetting during continually learning of Task-1 and Task-2. The model trained with CPPO demonstrates Backward Transfer (BWT) capability, specifically a negative SFR, suggesting that CPPO overcomes forgetting in this task and that learning new human preferences leads to a positive impact on retaining old preferences.

Thank you for your helpful comments! We appreciate your comments that our idea is novel and our approach is clearly explained. We address your questions below.

Comment 1: "Please provide confidence intervals and t-tests for table 8. It would be good to show the proposed method is significantly better than PPO."

• Response: We conduct t-tests and compute confidence intervals and p-values for Table 8 in the original paper. The detailed results are shown in the table below. In particular, the proposed method CPPO improves the PPO baseline by 8.23% with a p-value of 0.037, indicating that CPPO is significantly better than PPO.

We are grateful for your time and efforts. We have been eagerly waiting for your feedback on our response. We will be here waiting and hope to see it before the discussion period ends. We understand that you are very busy, but would highly appreciate it if you could take into account our response when updating the rating and having discussions with AC and other reviewers. Please let us know if you have further comments.

Considering that you may not be familiar with the RLHF field, we provide some RLHF references[1-3] that are highly relevant to our study. We hope that these works will assist you in understanding and evaluating the research content of our paper.

References

[1] Stiennon, Nisan, et al. "Learning to summarize with human feedback." Advances in Neural Information Processing Systems 33 (2020): 3008-3021.

[2] Ouyang, Long, et al. "Training language models to follow instructions with human feedback." Advances in Neural Information Processing Systems 35 (2022): 27730-27744.

[3] Bai, Yuntao, et al. "Training a helpful and harmless assistant with reinforcement learning from human feedback." arXiv preprint arXiv:2204.05862 (2022).

Comment 2: "What problem formulation is equation 4 a solution to?"

• Response: In the continual RLHF scenario, the human preference dataset evolves over time, often consisting of a sequence of tasks or domains. Each task or domain may have its own set of data, and these tasks are presented to the model sequentially. The model needs to adapt to new tasks without forgetting previously learned ones.

• Task Formulation: In this paper, we propose the task of continually learning human preferences under an offline continual learning setting[1]. Formally. we consider a task sequence of $T = {T_1 \rightarrow T_2 \rightarrow ... }$ to continually learn a policy model on the corresponding human preference datasets $HF =\{HF_1 \rightarrow HF_2 \rightarrow ...\}$ and prompt datasets $S =\{S_1 \rightarrow S_2 \rightarrow ...\}$. For each task $T_t (t=1,2,...)$, the policy $\pi_t$ is initialized by $\pi_{t-1}$ and then is trained against the reward model $r_t$ , where the reward model $r_t$ is learned on $HF_t$. The initial policy $\pi_0$ is the SFT model, namely, $\pi_0 = \pi_{SFT}$. The final objective is to learn a policy model $\pi_\theta$ that maximizes the overall reward on all of the learned human preferences: $max_{\theta} \Sigma_{t=1}^{T} E_{s\sim{S_t}, a\sim \pi_{\theta}(\cdot \mid s)} \bigl[r_t(s,a)\bigr]$. We also add the task formulation to the new PDF.

Comment 3: "The key difference with generic RLHF seems to be that the optimization of the two terms is only over a subset of the data in equation 4. Why does it help us to essentially throw away data? Or is this motivated as part of a computational constraint from some implicitly considered problem formulation that is not spelled out? "

• Response:

Eq.4 derives from the learning objective of the Policy Consolidation[2] method, as described in Eq.3 (in the new PDF): $\max_{\theta} E_{s\sim {S_t}, a\sim \pi_{\theta}(\cdot \mid s)} \bigl[r_t(s,a)\bigr] - E_{s \in {S_{t-1}}} D_{KL}({P_{\pi_{\theta}}} (a|s)||{P_{\pi_{t-1}}}(a|s))$.

The relationship of $D_1$ in Eq.4 and $S_t$ in Eq.3 is $D_1 \subset S_t$, analogously $D_2 \subset S_{t-1}$. Eq.4 is based on the intuitive objective: to make the language model have a high probability of generating a response with a high reward. Next, we clarify the reason that the objective Eq.4 is optimized on the subset of rollouts, not all of them.

Due to the LM usually using the top-P or top-K sampling method to generate the response, namely responses with high generation probability are more potentially generated. Therefore, the impact of low-probability samples on the final evaluation is weaker than the high-probability samples. Hence, we expect only to maximize the average reward on the rollout set with high generation probability in $S_t$, namely $D_1$.

To prevent forgetting, we focus more on the high-reward samples, because we expect to keep the ability of the old policy to generate high-reward rather than low-reward samples. In our learning strategy, those responses with low rewards can be forgotten when learning new preferences. Hence, we minimize the KL divergence on the responses with high rewards in $S_{t-1}$, namely $D_2$, which is essentially a form of function regularization[1].

The above intuition seeks a "compromise" between plasticity (learning new preference) and stability (retaining old preference). It is similar to the EWC algorithm, where parameters with high importance are subjected to stronger regularization to keep the old knowledge, while parameters with low importance receive weaker regularization and have more capacity to learn new knowledge.

References:

[1] Magdalena Biesialska, et al. Continual lifelong learning in natural language processing: A survey. ICCL, 2020.

[2] Christos Kaplanis, et al. Policy consolidation for continual reinforcement learning. PMLR, 2019.

Comment 4: "The hard cutoff based on hyperparameter k seems a bit weird to me. Towards what metric would the hyperparameter k be optimized for if it can't be equation 4 itself? This gets even weirder for me when the hard cutoff is then relaxed in equation 6. I don't get why it was ever introduced to begin with."

• Response: The introduction of $k$ is derived from the mean squared error (MSE) method for anomaly detection, aiming to identify different types of rollouts. In our scenario, if the generation probability or reward value deviates from its mean value $\mu$ exceeds $k$ times the standard deviation $\sigma$, namely $|p-\mu(p)|>k\cdot \sigma(p)$ or $|r-\mu(r)|>k\cdot \sigma(r)$, it is labeled as an anomaly, corresponding to a non-normal rollout type. Hence, hyperparameter $k$ controls the threshold of different rollout types. If we increase $k$, the size of the normal region in Figure 1 will increase, while the size of other regions will decrease. This means that the majority of rollout samples will learn like vanilla PPO.

Comment 5: "Why does equation 6 make equation 5 easier to optimize? I buy that equation 6 is more general than equation 5, but this is the key motivation for this sense and it is really not clear to me why this would be the case. Especially considering that focusing on only a subset of the data would require less computation, which I presume is a major constraint of the implicit problem formulation."

• Response: Since most rollout samples belong to the normal type (refer to Figure 1), optimizing using Eq.5 implies discarding a large portion of rollout samples, significantly reducing training efficiency (as the discarded samples also require time to generate). Therefore, for normal-type samples, we adopt a learning strategy similar to PPO, setting the weight $\alpha(x)$ to (or close to) 1. In this way, CPPO fully utilizes each rollout sample and adaptively adjusts the learning weights based on the reward and generation probability.

Thank you for your helpful comments! We appreciate your positive feedback on our research motivation and experimental results. We address your questions below.

Comment 1: "What is the statistical significance of the reported results across random seeds?"

• Response: Due to time and computational constraints, we currently do not have the statistical significance results for all the baselines. In subsequent versions of the paper, we will include the corresponding experimental results. In the table below, we present the mean and variance of the evaluation results for some methods across three random seeds (due to the randomness, there may be slight variations compared to the original PDF).

We've taken your initial feedback into careful consideration and incorporated them into our manuscript as indicated in our response. Could you kindly confirm whether our responses have appropriately addressed your concerns? If you find that we have properly addressed your concerns, we kindly request that you consider adjusting your initial score accordingly. We understand that you are very busy, but would highly appreciate it if you could take into account our response when having discussions with AC and other reviewers. Please let us know if you have further comments.

Thank you again for your time and effort in reviewing our work.

We noticed your concerns regarding Eq.5 in the Weaknesses part, and we guess that you may have questions about the derivation in Appendix B. Here, we will list out the derivation step by step, and if you have any specific doubts about a particular step, please feel free to point them out. And we are eagerly waiting for your feedback on our response.

from Eq.4 to Eq.5 (same with the new PDF Appendix B)

By introducing the actor-critic version, the clipped ratio, the objective Eq. 4. can be written as the objective Eq. 5. Next, we provide the derivation step by step:

1. We utilize notations $I_{D_1}(x)$ and $I_{D_2}(x)$ to rewrite the Eq.4 as $\max_{\theta} \mathbb E_{x \sim \pi_{\theta}} I_{D_1}(x) \cdot R(x) - \mathbb E_{x \sim \pi_{t-1}} I_{D_2}(x) \cdot D_{KL}({P_{\pi_{\theta}}} (x)\parallel P_{\pi_{t-1}}(x))$.

2. Then we introduce the importance sampling like PPO, the above objective can be written as $\max_{\theta} \mathbb E_{x \sim \pi_{t-1}} I_{D_1}(x) \cdot \frac{{P_{\pi_{\theta}}} (x)}{{P_{\pi_{t-1}}} (x)} {R}(x) - \mathbb E_{x \sim \pi_{t-1}} I_{D_2}(x) \cdot D_{KL}({P_{\pi_{\theta}}} (x)\parallel{P_{\pi_{t-1}}}(x))$.

3. In the PPO method, the objective is to maximize the expectation of the advantage function instead of the reward value. By introducing the advantage function instead of the reward, the above objective can be written as $\max_{\theta} \mathbb E_{x \sim \pi_{t-1}} I_{D_1}(x) \cdot \frac{{P_{\pi_{\theta}}} (x)}{{P_{\pi_{t-1}}} (x)} {A}(x) - \mathbb E_{x \sim \pi_{t-1}} I_{D_2}(x) \cdot D_{KL}({P_{\pi_{\theta}}} (x)\parallel{P_{\pi_{t-1}}}(x))$.

4. By introducing the knowledge retention penalty instead of KL-divergence, the above objective is written as: $\max_{\theta} \mathbb E_{x \sim \pi_{t-1}} I_{D_1}(x) \cdot \frac{{P_{\pi_{\theta}}} (x)}{{P_{\pi_{t-1}}} (x)} {A}(x) - \mathbb E_{x \sim \pi_{t-1}} I_{D_2}(x) \cdot L^{KR}(x)$.

5. In the CL task, the new policy $\pi_{t}$ is generally initialized by the old policy $\pi_{t-1}$. In CPPO, we treat the $\pi_{t-1}$ and $\pi_{t}$ as the reference model and policy model respectively. Then, we consider the actor-critic version, the clipped ratio, and the entropy bonus used in PPO, the above objective can be written as ${J}({\theta})^{'} = L^{ {I_{D_1}} \cdot CLIP+ {I_{D_2}} \cdot KR + VF + S}(\theta)$, namely the Eq. 5.

We thank all reviewers again for their insightful comments and questions.

We have attempted to carefully address all of your questions in the individual responses and in the edited paper draft. If you have any additional questions during the last day of the discussion period, we would be delighted to provide further clarification.

Thank you for your attention to our work. The code is complete. The new task continues training based on the checkpoint saved from the previous task, so we have only provided the code for the first task's training.

Thanks for your comments. Because we are directly modifying the trlx code when implementing CPPO, it is consistent with trlx's dependency environment. This is the repo of CPPO: https://git.openi.org.cn/Hanlard/CPPO It includes the requirements.txt file of dependencies. We hope this is helpful to you.

CPPO has two versions, namely, CPPO (Heuristic) and CPPO (Learn). The CPPO (Learn) uses the learnable balance weights, hence it does not need the hyperparameters like ub, and lb.