Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn

On other possible baseline methods, e.g., L2 regularization, LayerNorm (Lyle et al., 2024) continual backprop (Dohare et al., 2024) and ReDo (Sokar et al. 2023)

In addition to existing empirical evidence for LayerNorm, L2 Regularization and ReDo, we additionally implemented and ran LayerNorm, ReDo and AdamRel in our continual ProcGen experiments.

[Existing Evidence] The comparison between LayerNorm and TRAC (i.e., our major baseline and codebase) in continual Gym control tasks can be found in Appendix D, Figure 15 of TRAC paper (Muppidi et al., NeurIPS 2024). Their results show TRAC outperforms LayerNorm (i.e., LayerNorm Adam) in their Figure 15. For L2 regularization, Lyle et al. (2024) mentioned “We found that batch normalization and L2 regularization interfere with learning” in their Section 4.2, RL evaluation. In our work, we did L2 regularization and we found it performed very similarly (slightly worse) to L2 Init (i.e., Regenerative Regularization).

In another paper (Juliani and Ash, NeurIPS 2024), their Figure 5, ReDo, LayerNorm and other related methods are compared with L2 Init (i.e., Regen Reg in the figure) and L2 norm in ProcGen CoinRun. The results show that L2 Init performs comparably with ReDo and LayerNorm and slightly outperforms L2 norm (which is consistent with our findings as mentioned above).

For continual backpropagation, it is similar to ReDo which uses a different recycling metric, and in Appendix C.4, Figure 25 of the ReDo paper, they achieved similar results. Therefore, we only consider ReDo here.

[Additional Experiments] Please refer to the response to Reviewer cU5H and Reviewer mKWp.

Questions on the buffer

[On buffer flush and awareness of task switch]

In our method, i.e., C-CHAIN PPO, both $B_{ref}$ and $B_{train}$ are sampled from the online interaction data collected by the policy in the current iteration (e.g., every 2048 interactions). Thus, it does not matter for C-CHAIN PPO whether the buffer is flushed after each task switch and the agent does not need to be aware of task switches.

[The impact of the size of $B_{ref}$]

We did the experiments for different batch sizes for $B_{ref}$ on continual Gym control tasks. Our findings were that using a 2x, 4x or 8x batch size for $B_{ref}$ sometimes improved the learning performance, but not consistently. To some degree, increasing the batch size of $B_{ref}$ acted similarly to increasing the regularization coefficient, as both of them reduce more churn. Thus, we did not search for the best batch size for $B_{ref}$ to alleviate the hyperparameter choice burden.

Questions on concrete experimental results in continual RL

[Explanation for the slow decreasing of the approximate rank for C-CHAIN in Fig. 4]

For C-CHAIN, the approximate rank decreases very slowly, from around 85 to 80 over 10M steps (in contrast, the approximate rank decreases is 75 to 30 for vanilla PPO). We think it is reasonable as the agent continually consumes the plasticity to learn new tasks and our method is not perfect.

[The performance in MountainCar]

We think MountainCar is a bit more difficult than the other three Gym control tasks, as it needs more exploration and TRAC almost totally failed in it (note that MountainCar is not included in TRAC paper). One possible reason for C-CHAIN’s low early-stage performance might also be due to the exploration feature of MountainCar. Reducing churn could also be viewed to prevent the generalization of exploration behavior learned by the policy in some states to similar states. Therefore, C-CHAIN learns slowly but improves steadily, while vanilla PPO learns quicker and collapses.

[The performance on Fruitbot, Jumper, and Plunder]

By checking the task details in the official document of ProcGen, tasks like Leaper, Jumper are stuff-collection tasks with sparse rewards, and especially Leaper is an episodic-reward environment.

To gain more understanding, we provided the average scores of the max/mean/min curves for Leaper, Fruitbot, Plunder, Jumper below. We can observe that C-CHAIN improves the average Max scores over PPO Vanilla and the average Mean scores (by improving Max or by reducing failures when Max is similar); while it does not fully address the (near-)zero performance due to the limited exploration ability of the PPO base agent.

Other remaining questions

Due to the space limitation, we will provide them during the discussion stage.

We sincerely appreciate the reviewer’s valuable comments and recognition. Our response aims to add more discussion and clarification on the points mentioned in the inspiring comments.

In addition, we provide additional experimental results for AdamRel [1], ReDo and LayerNorm, along with Reliable Metrics (Agarwal et al., NeurIPS 2021). Besides, we provide two additional continual RL experimental setups in DMC.

On the data distribution regarding the expectation in Eq. 4 and $S_x$

We appreciate the reviewer for pointing this out. Yes, $S_x$ denotes the stochastic variable of sampling $B_{train}$, which should be defined along with a sampling distribution. For illustrative purposes, using a uniform distribution is sufficient. And we expect to take the concrete form of the sampling distribution (intuitively, it should depend on factors like on-policy learning/off-policy learning, exploration policy, experience replay strategy, etc.) into the analysis in the future. We will clarify this point as suggested.

On the mentioned related [1], [2], [3]

[Additional Discussions]

For some discussions on [2] and [3], the Proximal Feature Optimization (PFO) regularization proposed in [2] is closely related to DR3 (Kumar et al., ICLR 2022) and CHAIN (Tang and Berseth, NeurIPS 2024). This is because that the feature difference can be viewed as the gradient difference when the network is viewed as a linear approximation i.e., $\pi(s) = \phi(s) W$ as in DR3; in another view, regularizing feature difference should share some overlapped effects with regularizing network-output difference as in CHAIN.

And for [3], we found the experimental results for ProcGen CoinRun by the Figure 5 in their paper provide a useful reference of the learning performance of other related methods not included in our submission version. For example, ReDo and LayerNorm perform comparably with Regen reg (i.e., the L2 Init baseline method adopted in our paper).

[Additional Experimental Results]

We added AdamRel [1], ReDo and LayerNorm to our experimental comparison in 16 ProcGen tasks. The aggregate comparison is shown below. We can find ReDo and LayerNorm perform comparably with L2 Init, which is basically consistent with the evidence in existing papers.

Moreover, we provide the aggregation evaluation by using Reliable Metrics (Agarwal et al., NeurIPS 2021). The evaluation uses Mean, Median, Inter-Quantile Mean (IQM) and Optimality Gap, with 95% confidence intervals and 50000 bootstrap replications (recommended by Reliable Metrics official implementation). It aggregates over 9 methods, 6 seeds for each of 16 ProcGen tasks, i.e., 864 runs with 10M steps for each.

The results are shown in the table below (the corresponding plot will be added to the revision). We can observe that C-CHAIN performs the best regarding all four metrics and outperforms the second-best with no overlap of confidence intervals for Median, IQM, and a minor one for Mean.

On other advice

[The position of Figure 1] We are glad to know that Figure 1 helps to understand the NTK equations. We will move Figure 1 close to Equation 2 and 5 with additional explanations to improve the smoothness of the introduction of the NTK equations.

[The broader connection to optimizing neural networks] We appreciate the reviewer’s inspiring comments. We will discuss these broader connections in our revision.

We sincerely appreciate the reviewer’s valuable comments and the recognition of our method and experiments. Our response aims to address these aspects in detail.

On the experimental setups, task difference, and additional CRL setups

[Clarification on environment choice and task difference]

We followed the experimental setting in TRAC paper (Muppidi et al., NeurIPS 2024) and took it as a major baseline. For a more comprehensive experimental comparison, we added MountainCar (where TRAC fails almost totally) to the continual Gym control suite, and we extended the 4 ProcGen tasks used in TRAC to all 16 tasks provided by ProcGen suite.

These tasks are representative CRL scenarios with semantically related tasks, and they are challenging to the vanilla PPO agent in the sense that apparent degradation occurs as the learning goes on (for most of them).

[Additional experimental setups in DMC]

As suggested by the reviewer, instead of chaining totally different Atari games (Abbas et al., 2023), we additionally established two continual RL settings in DeepMind Control (DMC) suite:

• Continual Walker: chain Walker-stand, Walker-walk, Walker-run
• Continual Quadruped: chain Quadruped-walk, Quadruped-run, Quadruped-walk (repeat because only two Quadruped tasks are available in DMC).

Similarly, we compare PPO Oracle, PPO Vanilla, and C-CHAIN in the three settings. We run 1M steps for each task, i.e., 3M in total for each setting. The results are averaged over 12 random seeds for each configuration, as shown below (the corresponding learning curves will be added to our revision).

We can observe that C-CHAIN improves PPO in continual DMC. Actually, from the learning curves (will be added in the revision), we found PPO learns more slowly with a decreased slope in the second and the third task, and C-CHAIN learns faster and achieves higher scores.

On the choice of baseline methods

[Existing evidence in prior work]

For the choice of other baseline methods, as we mentioned in Line 315-325, TRAC outperforms Concatenated ReLU (Abbas et al., 2023), EWC (Schwarz et al., 2018), Modulating Masks (Nath et al., 2023) in their paper, as well as LayerNorm (Lyle et al., 2024) in Figure 15 of TRAC paper (i.e., LayerNorm Adam). Besides, more related methods (e.g., ReDo) are compared with L2 Init (i.e., Regen Reg) in ProcGen CoinRun in Figure 5 of (Juliani and Ash, NeurIPS 2024). The results show that L2 Init performs comparably with ReDo and LayerNorm.

[Additional results for three more baselines and reliable aggregate evaluation] Please refer to the response to Reviewer cU5H, due to the space limitation.

On a more precise mathematical formulation of the connection between churn, NTK rank, and plasticity loss

In this paper, we are more focused on using formal expressions to describe, analyze and dig out intuitive insights. To give a thorough and rigorous theory, we need (at least) definitions and assumptions from three aspects:

• [RL-oriented definition of plasticity] To our knowledge, the existing formal definition of plasticity (i.e., Target-fitting capacity, Definition 1 in Clare et al., 2022) is based on a general family of objective functions, which is too vague and broad for RL. This is why a direct plasticity loss metric was not used in our paper, and instead we used the rank loss of NTK since it is the best empirical practice in the literature (Clare Lyle et al., 2024). One possible RL-oriented definition could be developed upon the Value Improvement Path (Definition 1, Dabney et al., 2021).
• [Proper assumptions on deep RL] A Rigorous theoretical analysis on deep RL learning process is challenging as we need concrete assumptions on network structure, optimization, objective function. Theoretical analysis methods and results under practical assumptions are still lacking in deep RL.
• [Proper assumptions on continual learning setting] In the literature of continual RL, task sequence and switch scheme are usually manually designed for experimentation, little formal definition and assumption of task distribution and task switch have been established so far.

Due to the lack of theoretical foundation above, providing a thorough and rigorous theory is out of the scope of this paper. But as suggested, we will update the discussion in Section 4 to provide more formal support and additional discussions for the connections between churn, NTK and plasticity.

On the results for the tasks like Leaper

Please refer to the response to Reviewer 65iv.

Other remaining questions

Due to the space limitation, we will provide them during the discussion stage.

We sincerely appreciate the reviewer’s valuable comments and the recognition of the importance of the topic studied in our paper.

On the reviewer’s comments including “the empirical evaluation doesn't show conclusive results”, “the improvement is not consistent”, and “little insight is provided to explain it”

[Additional reliable aggregate evaluation]

To obtain a conclusive evaluation, we provide the aggregation evaluation by using Reliable Metrics (Agarwal et al., NeurIPS 2021). The evaluation uses Mean, Median, Inter-Quantile Mean (IQM) and Optimality Gap, with 95% confidence intervals and 50000 bootstrap replications (recommended by Reliable Metrics official github implementation). It aggregates over 9 methods, 6 seeds for each of 16 ProcGen tasks, i.e., 864 runs with 10M steps for each.

The results are shown in the table below (the corresponding plot will be added to the revision). We can observe that C-CHAIN performs the best regarding all four metrics and outperforms the second-best with no overlap of Confidence Intervals for Median, IQM, and a minor one for Mean.

Due to the space limitation, please refer to the concrete markdown table we will upload in the common response or the discussion response after the rebuttal deadline.

[On the consistency of improvement]

As summarized by Table 1, our method C-CHAIN (PPO) consistently outperforms vanilla PPO in all 20 envs. And C-CHAIN performs the best in 15 out of 16 tasks (and performs the second-best in the remaining one) on continual ProcGen benchmark. In this sense, our method achieved overall consistent improvement.

[On the insights and explanations]

For continual CartPole and continual LunarLander, where L2 Init outperforms C-CHAIN, we provided the possible explanation in Line 328-365. Our insight is, for simple tasks like CartPole and LunarLander where the agent can find a good policy near parameter initialization, L2 Init performs the best; in contrast, it limits the policy learning as in continual ProcGen where L2 init is outperformed by C-CHAIN in most tasks.

On the experimental design

We followed the experimental settings used in TRAC (as also mentioned by the reviewer’s comment “The authors selected standard benchmarking tasks accepted by the community to study loss of plasticity”), and we further extended the ProcGen tasks from the four ones used in TRAC to all 16 tasks, as well as continual MountainCar where TRAC collapses.

[On the task diversity and the varying difficulties]

Since the tasks are different in game logic, reward density, visual complexity, and etc., they provide a range of environments of diversity and varying difficulties, which is also a commonly adopted principle of designing a benchmark. Therefore, we think it is natural for Oracle PPO or vanilla PPO to have different levels of performance across different tasks.

On the math derivation

[How did both $\nabla_{\theta} f_{\theta}(\bar x)$ and $\nabla_{\theta} f_{\theta}(x)$ become $G_{\theta}$?]

We refer the reviewer to the definition of the NTK matrix (i.e., the first line for entry definition and the second line for matrix definition in vector form) in Equation 2.

As in the second line of Equation 4, $\nabla_{\theta} f_{\theta}(\bar x)^{\top} \nabla_{\theta} f_{\theta}(x)$ becomes $N_{\theta}(\bar x, x)$ by the definition in Equation 2. Then, Equation 5 is the vector form of Equation 4, correspondingly, the entry definition $N_{\theta}(\bar x, x)$ becomes $N_{\theta} = G_{\theta}^{\top}G_{\theta}$ as in Equation 2.

[The transition from Equation 4 and Equation 5, about the diagonal $S$ matrix]

As mentioned above, Equation 5 is the vector form of Equation 4. Therefore, the sampling $x \sim B_{train}$ is replaced by the $S$ matrix as below in Equation 5. More specifically, $S$ has the same size as $N_{\theta}$, i.e., $|X|$ by $|X|$. As mentioned in Line 155-157, $S$ is a diagonal matrix, and its diagonal is ${0,1}$-binary with 1 for the sampled data and 0 for the non-sampled.

We appreciate the reviewer for pointing this out. We will make this clearer as suggested by the reviewer.

On the evaluation of C-CHAIN in continual supervised learning

We provided two explanations in Section 5.3. A significant difference to note is that RL suffers from the chain of churn, formally characterized in (Tang and Berseth, NeurIPS 2024), which stems from the iterative policy improvement and policy evaluation nature (i.e., the Generalized Policy Iteration paradigm), while SL does not as it learns from static labels.

This means that the exacerbation of churn during the process of NTK rank loss on both the policy network side and the value network side can further (negatively) affect each other due to the chain effect. Since C-CHAIN is proposed from the perspective of churn, this explains to some extent why C-CHAIN is not that effective for CSL.