Neuron-level Balance between Stability and Plasticity in Deep Reinforcement Learning

We sincerely thank the reviewer for their detailed feedback and constructive suggestions. We understand the concerns raised and will address them to strengthen the quality of our paper. Below, we provide a point-by-point response to the reviewer's comments.

Limited Experimental Scenarios and Benchmark Selection We appreciate the suggestion to extend our experiments to more complex and diverse benchmarks beyond Meta-World and Atari. While we initially chose Meta-World and Atari due to their established difficulty and representativeness in reinforcement learning research, we acknowledge that the limited number of tasks tested may not fully demonstrate the generalizability of our method. In future work, we will include more challenging benchmarks such as DeepMind Control Suite, Gym-Mujoco, BabyAI [1], and the lifelong learning manipulator benchmark LIBERO [2]. These additional experiments will further validate the effectiveness of our approach in diverse and complex settings.

Sequential Task Learning and Mainstream Experimental Design

Selection of Baselines for Comparison As mentioned in our general response, we recognize the need to expand our experimental design to multi-task sequential setups and to compare against a broader range of state-of-the-art baselines, particularly those addressing plasticity loss and continual RL learning. These enhancements will provide a more comprehensive evaluation of our method and better align with mainstream practices in continual reinforcement learning research.

Evaluation Metrics for Plasticity We acknowledge the need for more rigorous evaluation metrics to demonstrate our method’s impact on plasticity. In our current evaluation, we focused on stability-plasticity trade-offs but did not include specific metrics like the covariance metric [6] or the FAU metric [7]. In the revised paper, we will extend our evaluation to include these plasticity metrics, providing a more quantitative assessment of our method's effectiveness in balancing stability and plasticity.

We are grateful for the additional references provided and will incorporate the relevant literature into our related work section. Here are some specific changes we plan to implement based on your feedback:

• Expanded Experiments: We will include additional tasks from the suggested benchmarks to provide a broader evaluation of our method's applicability.
• Longer Task Sequences: We will test our approach on longer sequences of tasks, aligning with the designs used in recent continual learning research.
• Comprehensive Baseline Comparisons: We will include state-of-the-art baselines that address plasticity loss and lifelong learning challenges, enhancing the rigor of our comparative analysis.
• Incorporating Plasticity Metrics: We will utilize the covariance and FAU metrics to provide a deeper understanding of our method’s impact on plasticity.

We greatly appreciate the detailed and constructive feedback, which will guide us in enhancing the quality of our paper. Your insights have been invaluable in highlighting areas for improvement, and we are committed to making the necessary revisions to strengthen our contribution to the field. Thank you once again for your thoughtful review.

We greatly appreciate the reviewer's insightful feedback and constructive comments, which have provided valuable guidance for improving our work. Below, we address each of the reviewer's concerns and outline the changes we plan to implement in our revised paper.

Limited Task Overlap and Generality of the Method We acknowledge the limitation of our current experiments, which only focus on sequential tasks involving opening/closing the same object. To demonstrate the generality of our method, we will extend our experiments on the Meta-World environment by exploring new pairs of tasks, such as "Push", "Reach", "Pick Place", "Basketball", and "Sweep Into". Additionally, as suggested, we will move some of the Atari experiments from the supplementary material to the main paper to provide a more comprehensive evaluation. We will also include additional results beyond the best-performing ones to offer a more complete picture of our method’s capabilities.

Performance on Multiple Sequential Tasks and Mask Ratio Analysis We agree that testing our approach on only two sequential tasks raises questions about its scalability to longer task sequences. In the revised version, we will extend our evaluation to include learning multiple sequential tasks (e.g., 10 tasks) to assess the method's performance under more challenging settings. We will also investigate the impact of the mask ratio on stability and plasticity by varying the number of RL skill neurons, which is currently treated as a hyperparameter. This analysis will be moved from the appendix to the main paper, with a detailed study of the values "300" and "400" and their influence on learning dynamics.

Ablation Study on Gradient Masking and Experience Replay We appreciate the suggestion to conduct a more thorough ablation study. In the revised paper, we will add experiments that isolate the effects of gradient masking and experience replay. Specifically, we will include:

• An ablation experiment using only gradient masking without experience replay to demonstrate the individual contribution of each component.
• An analysis of whether other baselines can benefit from the same experience replay mechanism. This will help clarify whether our method’s improvements are due to the gradient masking strategy or the replay buffers.

Additional Baselines Addressing Catastrophic Forgetting We acknowledge that our current baselines may not sufficiently cover the literature on catastrophic forgetting in reinforcement learning. We will expand our experimental setup to include baselines such as Progressive Neural Networks [2] and other methods known for mitigating catastrophic forgetting [1]. We will also clarify why certain continual learning approaches are not directly applicable to DRL and highlight the unique challenges our method addresses in this context.

Paper Length and Figure Adjustments Thank you for the suggestions on optimizing the paper length. We will relocate Figure 3 to the appendix and adjust the layout of existing figures to save space, for instance, by arranging plots in 3 figures per row or converting bar plots into compact tables. This will allow us to integrate more experimental results and discussions into the main paper without exceeding the page limit.

Responses to Specific Comments:

• Evaluation Criterion Function (line 216) We appreciate the need for clarity regarding the evaluation criterion. The function varies based on the environment: in Meta-World, it is based on task success rates, while in Atari, it relies on reward scores. The degree to which the agent approaches the goal is measured using this criterion. We will clarify this in the revised text with examples for each domain.
• Clarification on Stability and Plasticity (line 60) The sentence will be revised for clarity: "Stability and plasticity depend on the learning capabilities of neurons. High learning capacity enhances plasticity, allowing the model to adapt to new tasks, while low capacity helps retain prior knowledge, promoting stability."
• Redundancy in Section 3.2 We agree that Paragraph 2 of Section 3.2 overlaps with the related work section. We will streamline this content to avoid redundancy.
• Visualization of Average Return in Figure 7 Thank you for this suggestion. We will adjust Figure 7 to include average returns on the x-axis, enabling better comparisons with other plots.
• Bibliography Duplication We will correct the duplication of the reference to "Fengshuo Bai et al." in the bibliography.

We are grateful for your thoughtful recommendations. These revisions will significantly enhance the clarity, comprehensiveness, and impact of our paper. Thank you once again for your detailed review and suggestions.

We sincerely appreciate the reviewer's detailed feedback and valuable suggestions, which have greatly helped us identify areas for improvement in our work. Below, we address each of your concerns and outline the modifications we plan to make in our revised paper.

1. Comprehensive Review of Existing Approaches in Continual RL We acknowledge that our related work section primarily focused on general continual learning and did not sufficiently address continual RL methods, especially in the context of stability-plasticity trade-offs. In the revised version, we will include a comprehensive survey of continual RL, particularly focusing on structure-based methods $1$

$2$

$3$ and rehearsal-based approaches $4$

$5$ , as suggested. This will provide better context and positioning for our work. Additionally, we will expand the discussion to include neuron-level methods to highlight the relevance of our approach. 2. Clear Definition of the Continual Learning Problem We agree that the definition of the continual learning setting in our paper was not sufficiently clear. We focused on a two-task setup, which led to ambiguity regarding whether the approach is task-incremental or class-incremental. In the revised paper, we will explicitly define the continual RL problem, detailing whether the method applies to both the policy and critic networks, and clarifying the extent of access to previous task data. Furthermore, we will include a precise definition of the "RL skill neuron" to enhance clarity. 3. Novelty of Combining Task-Specific Skill Neurons with Gradient Masking We understand the reviewer's concern regarding the novelty of combining task-specific skill neurons with gradient masking, as similar ideas were explored in PackNet [3] and CoTASP [2]. However, our method identifies RL skill neurons driven by reinforcement learning objectives, specifically focusing on neurons that contribute to the policy's performance. This distinction allows us to retain RL-specific knowledge more effectively. In the revision, we will include a thorough comparison with these methods in both the related work and experimental sections to highlight our contributions more clearly. 4. Limitations in the Score Function We acknowledge the reviewer's concerns regarding our score function:

• (4.1) Raw Activation Values: The implementation indeed uses absolute activation values, but this was not clearly explained. We will clarify this in the paper.
• (4.2) Activation Baseline: We compare each neuron's activation with its own average over time, rather than across neurons, to capture its relative importance. The choice to compare each neuron's activation with its own average over time is inspired by practices in neuroscience, where neural activity is often evaluated relative to its baseline firing rate to highlight deviations that indicate task-specific importance. This neuron-specific baseline allows us to focus on each neuron's unique contribution, avoiding the noise and dilution that can arise when averaging across neurons with varying roles. By capturing these individual deviations, our method effectively identifies neurons that are particularly relevant for skill retention, ensuring a more precise assessment of their functional importance.
• (4.3) Binary Counts in Eq (3): We appreciate the suggestion to consider finer distinctions in activation levels. In future work, we will explore more granular measures to enhance precision.
• (4.4) Reward Signal in q(t): Our intention was to use q(t) as a global measure of goal achievement. We will improve the explanation to avoid confusion.
• (4.5) Intuition Behind Eq (4): The scoring mechanism aimed to consider both high and low activation scenarios. We will refine our explanation and include a statistical analysis of the score distribution in the experiments.

5. Similarity to Dormant Neurons The method for identifying RL skill neurons was designed to consider both activation levels and task performance, unlike dormant neurons that rely solely on activation thresholds. Continual RL tasks are inherently goal-driven, with each task demanding specific behaviors and policies to optimize performance. Simply relying on activation thresholds might fail to capture neurons that are critical for task-specific goals.By integrating both activation levels and task performance into our scoring function, we ensure that the selection process aligns with the task's objectives. This dual consideration allows us to identify neurons that not only exhibit consistent activity but also contribute meaningfully to achieving task-specific goals. To demonstrate the strengthen of our method, we will include a comparison with dormant neuron-based methods in the experiments to demonstrate the benefits of our approach.

6. Gradient Masking Strategy We appreciate the reviewer's suggestion to explore alternative gradient masking strategies. Currently, we mask gradients connected to identified skill neurons based on their output dimensions. We plan to experiment with additional strategies, such as blocking gradients between specific neurons across layers, in future work.
7. Gradient Masking with Experience Replay We agree that relying on experience replay may limit the applicability of our approach in certain settings. However, we observed that gradient masking alone led to some knowledge forgetting, which was mitigated by incorporating experience replay from the training buffer. In the revised paper, we will discuss potential methods to reduce dependency on replay data and explore alternative rehearsal strategies.
8. Implementation for the Skill Neurons
9. Simple Fine-tuning Baseline
10. Limitations of Experimental Domain Regarding the implementation of skill neurons, the selection of baselines, and the experimental domain, we refer to our general response for a detailed discussion of our planned improvements. Briefly, we acknowledge the need to provide more detailed descriptions of our methodology, adopt advanced continual RL baselines, and expand our experiments to more challenging multi-task settings such as Meta-World's CW10 and CW20. These enhancements aim to address the limitations identified and further validate the robustness and applicability of our approach.
11. Ablation Study Suggestions We appreciate the suggestions for a more detailed ablation study. In addition to our current analysis, we will include comparisons with: (1) alternative activation functions; (2) the score function using Acc(N) only; (3) dormant neuron-based scoring; (4) randomly selected skill neurons; and (5) gradient masking without replay.

Responses to Specific Questions:

• Evaluation Criteria (qθ(t)): We chose different evaluation criteria based on task-specific objectives. For Meta-World, we used success rates, aligning with standard evaluations for these tasks.
• Time Steps (T) in Eq (1) and Eq (2): This is a hyperparameter set to 1e6 for all experiments. We agree that this choice should be more adaptive and will explore fine-tuning it for different environments.
• Reward-based Evaluation and Skill Neurons: If neuron activation and reward performance are positively correlated, we classify it as an RL skill neuron. We will add graphical representations to highlight these correlations.
• Use of Raw vs. Absolute Activation: Thanks for the valuable suggestions, we do use absolute activation values and we revise the expression to absolute activation values in paper for better clarity.
• Impact of Activation Functions: The current approach uses tanh in SAC networks. We will explore other activation functions in future studies to verify the generality of our method.
• Experience Replay Frequency: We retain the replay buffer of size 1e6 for the previous task. During training, we only use this stored data.
• Learning Curves Clarification: The upper part of each figure represents the first task, while the lower part corresponds to the second task. We will adjust the figures for better clarity.
• Sampling and q(t) Computation: The evaluation criteria q(t) are directly obtained from interactions with the environment, not from replay buffers.
• Computational Complexity: The identification of RL skill neurons involves additional evaluation steps after training the first task, with complexity similar to the RL algorithm itself.
• Algorithm 1 Enhancement: We will revise Algorithm 1 in the appendix to include the steps for computing RL skill neurons.

We are grateful for your constructive feedback and believe these revisions will significantly improve the clarity and impact of our work. Thank you for your insightful suggestions.

Thank you for your constructive feedback and for highlighting the key areas where our work can be further strengthened. We appreciate the opportunity to clarify and improve our contributions in response to your insightful comments.

1. Limited Experimental Scope and Scalability Concerns:

   We acknowledge your concern regarding the scalability of our method, especially in the context of long-term continual learning settings. The current experiments were indeed limited to two task sequences, which may not fully capture the scalability and robustness of our approach. In response, we have planned additional experiments with a sequence of approximately 10 tasks to evaluate the scalability and effectiveness of our method in more complex and extended scenarios. These results will be included in the revised version to provide a deeper understanding of the method’s performance across a broader range of tasks.

2. Restricted Baseline Comparisons:

   We appreciate your suggestion to include comparisons with more diverse and established baselines from the reinforcement learning literature. In the current version, we focused on addressing the stability-plasticity dilemma by comparing our approach with three key baselines. However, we understand the importance of benchmarking against additional methods commonly used in continual reinforcement learning. We are in the process of incorporating comparisons with the methods you suggested ([1], [2], and [3]), as well as other state-of-the-art baselines. This will allow us to present a more comprehensive evaluation of our approach's performance relative to existing techniques.

We are confident that these enhancements will address the concerns raised and further demonstrate the strengths and scalability of our proposed method.

Thank you once again for your valuable comments.