Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning

We sincerely thank the reviewer for their insightful comments and valuable suggestions. Below we address the concerns raised:

Response to 1 and 2

We used the state-vector version of DMC rather than image-based inputs, as CleanRL's PPO benchmark results¹ show that PPO can achieve moderate performance with vector states. We specifically chose environments where PPO demonstrates learning capability while leaving room for improvement. This experimental design ensures the tasks are learnable (as evidenced by baseline performance) while providing sufficient headroom to demonstrate the benefits of enhanced plasticity. We will clarify this experimental setup in the revision. We apologize for any confusion caused by the heavily smoothed visualization of the baseline performance. We have updated Fig. 9 to show the raw baseline data, which more accurately reflects the actual baseline results with returns consistently reaching 4-5 and occasional peaks above 10. This revision provides a clearer representation of the baseline algorithm's true performance.

Response to 3

We acknowledge that Fig. 5 inadvertently combined results from different parameter settings. We have regenerated the figure using consistent settings across all methods, with data from 5 random seeds for each experiment. The revised figure will be included in the updated manuscript along with explicit documentation of the seed count.

Response to 4

Following the reviewer's suggestions, we have expanded our evaluation in two key aspects. First, we incorporated two additional plasticity metrics: neuronal dormant ratio and gradient covariance. Second, we conducted experiments replacing tanh with ReLU in middle layers. The new results, detailed in the Appendix A.4.1, demonstrate our method's effectiveness across different activation functions while providing further validation of its plasticity properties.

Response to 5

We apologize for the presentation issues in our manuscript. Similar to the previously acknowledged error in Fig. 5, Fig. 14 was incorrectly labeled and contained results from mismatched experimental configurations. Additionally, we failed to clarify that the figure shows activation norm data, where we used absolute values of tanh outputs as neuron scores. We have now replotted all metrics with consistent experimental settings and conducted additional validation with ReLU activation functions in the Appendix A.4.1, and changed Fig. 14 to Fig. 15. The new results demonstrate our method's consistency with standard plasticity metrics when using ReLU.

We sincerely thank the reviewer for their insightful comments and valuable suggestions. Below we address the concerns raised:

Response to 1

While the plasticity-stability trade-off provides broad concept, our proposed concept of "neuron regeneration" serves two essential purposes that justify its introduction:

1. Precision and Specificity: Unlike the broader plasticity-stability trade-off concept, "neuron regeneration" focuses precisely on neuron-level phenomena and mechanisms. By explicitly describing operations at the individual neuron level, this term provides a more concrete and specific way to discuss neural network plasticity. This neuron-level specificity helps researchers better understand and analyze the actual mechanisms involved in maintaining network plasticity.

2. Conceptual Clarity and Biological Inspiration: The term "neuron regeneration" deliberately draws parallels with biological processes, making it more intuitive for researchers to understand the purpose and mechanisms of our approach. This biological analogy helps convey how neural networks can maintain stability while enabling continuous learning through targeted neuron-level interventions.

We refined the definition to highlight neuron-level operations. This more precise terminology contributes to the field by focusing specifically on neuronal mechanisms involved in maintaining network plasticity and stability, rather than relying on broad concepts.

Response to 2

The title "Stay Hungry, Keep Learning" intentionally captures our method's core philosophy - maintaining continuous learning capability and avoiding saturation, which directly aligns with our cycle reset and inner distillation mechanisms. Just as biological systems maintain learning capacity through constant renewal, our method sustains the network's ability to acquire new knowledge.

Regarding the framework name, we agree that "Sustainable Back Propagation" might not fully reflect our method's essence. We have revised it to "Sustainable Backup Propagation (SBP)", which more accurately describes our approach of maintaining sustainable learning through knowledge backup and support mechanisms. This new terminology better aligns with our method's core components: the cycle reset that provides sustainable learning capability and the knowledge distillation that serves as a backup and support system for preserving critical information.

We also acknowledge that there was a typographical error in the abstract where "Sustainable Back Propagation (SBP)" was incorrectly written as "Sustainable Brain Plasticity". This was an oversight during manuscript preparation. Throughout the revised paper and in all our technical discussions, we consistently use and define SBP as "Sustainable Backup Propagation", which accurately reflects the nature of our proposed method. We have corrected these terminology issues in the revised manuscript to maintain precision and consistency.

Response to 3

We conducted experiments comparing our cyclic reset strategy with targeted approaches. Through analysis of weight norms (Fig. 5) and gradient norms (Fig. 6), we demonstrate that cyclic reset is the primary factor in reducing weight magnitudes and maintaining gradient flow, while distillation serves as a knowledge preservation mechanism that maintains learning efficiency and ensures performance improvement. Additionally, we validate the necessity of resetting all neurons through comparative experiments that examine different reset ratios. The results consistently show that higher reset ratios correlate with both smaller weight magnitudes and larger gradient values, supporting our hypothesis about the relationship between reset coverage and network plasticity. This relationship demonstrates why full neural reset is more effective than partial reset strategies. Details have been added to the Appendix A.4.4.

Response to 4

Beyond the indirect metrics of weight and gradient norms, we have now incorporated direct plasticity measurements - the neuronal dormant ratio and activation norm. These additional metrics further validate our framework's effectiveness in maintaining network plasticity. Details are provided in the Appendix A.4.1.

Response to 5

Our focus on PPO was strategic: on-policy algorithms face unique plasticity challenges due to their small online buffer and sensitivity to data distribution changes. This makes them an ideal testbed for our framework - success in this challenging setting would suggest broader applicability. Following the reviewer's suggestion, we validated this hypothesis with SAC experiments, which demonstrated performance improvements even with limited distillation buffer sizes. These results across both paradigms confirm our framework's effectiveness, while highlighting its particular value for on-policy methods where plasticity challenges are more acute. Details are provided in the Appendix A.5.

We sincerely thank the reviewer for their insightful comments and valuable suggestions. Below we address the concerns raised:

Response to alpha

Initially, we experimented with $\alpha=0.5$ (symmetric KL divergence), a theoretically well-established choice. However, our empirical results showed suboptimal performance in our setting. Through extensive ablation studies, we explored different $\alpha$ values to better balance the forward and reverse KL divergence terms. We have included comprehensive results comparing $\alpha=0.5$ with other $\alpha$ values in Fig. 10.

Response to plasticity and weight magnitude

Neural plasticity measurement remains an open challenge that can be approached through multiple metrics. Weight-based metrics suggest higher plasticity correlates with smaller weight magnitudes [1]. We focus on gradient norms as they directly indicate learning capacity and help identify plasticity loss through gradient vanishing. Additional metrics include activation patterns, where lower dormant ratio [2] suggests greater plasticity, and more recent approaches using covariance matrices [3]. While no single metric fully captures plasticity, examining these complementary measures together provides a more comprehensive understanding of network adaptability.

Response to typo

We thank the reviewer for this technical correction. We will revise Section 2.1 to more accurately describe PPO's clipping mechanism. The mechanism sets gradients to zero when $r_t(\theta)$ exceeds $[1-\epsilon, 1+\epsilon]$, thereby preventing policy updates rather than merely limiting their magnitude. We will also correct the duplicate citation of Schulman et al. 2017 in the References.

References

[1] Dohare et al. "Loss of plasticity in deep continual learning." Nature (2024)

[2] Sokar et al. "The dormant neuron phenomenon in deep reinforcement learning." ICML (2023)

[3] Obando et al. "In value-based deep reinforcement learning, a pruned network is a good network." Architecture (2024)

We sincerely thank the reviewer for their insightful comments and valuable suggestions. Below we address the concerns raised:

Response to Weaknesses

Response to 1.1

Our Inner Distillation module represents a novel framework that extends significantly beyond traditional knowledge distillation.

1. First, while conventional KD transfers knowledge between separate teacher and student networks, our approach innovates by enabling fine-grained knowledge transfer between different components within the same network at the neuronal level.

2. Second, we specifically design our distillation mechanism to address the unique plasticity-stability challenge in neural resets - preserving crucial knowledge while maintaining adaptability. The introduction of bi-directional KL divergence loss, tailored for this intra-network knowledge transfer, enables precise control over knowledge preservation and adaptation at the neuron level. This specialized application and adaptation of distillation principles creates a fundamentally new approach to maintaining neural plasticity.

Response to 1.2 and Questions1

While previous works demonstrate quick recovery in some scenarios, our research reveals that this is highly algorithm-dependent. Specifically, algorithms with limited replay buffers like PPO face significant recovery challenges compared to SAC's quick recovery with large buffers. Moreover, our approach addresses a broader goal beyond simple recovery: maintaining continuous learning efficiency throughout training.

Regarding computational costs, our method introduces modest overhead through additional distillation epochs (3%~35% as shown in Table 2). We have added a detailed computational cost comparison between our distillation approach and direct recovery training in Appendix 4.5. While our results indicate that the distillation process consumes more computational time due to our conservative setting of the distillation loss coefficient (0.01), this parameter can be optimized to reduce the computational overhead.

More importantly, Fig. 7 demonstrates that this computational investment yields substantial performance improvements. In contrast, simply adding extra training steps after reset only achieves marginally better results than cycle reset, and generally underperforms compared to basic PPO. This suggests that within the PPO framework, merely increasing training steps is insufficient to fully recover the lost knowledge, even when achieving similar performance metrics.

Response to 3.1

While [2] correctly identifies that critics experience more severe plasticity loss, we argue that addressing plasticity in PPO remains both crucial and insightful. Unlike value-based methods with large replay buffers, PPO's on-policy nature and limited buffer size create distinct plasticity challenges due to continuous distribution shifts in training data. Therefore, this makes PPO an important test case for plasticity enhancement.

The observation that PPO+Cycle alone brings no improvement actually strengthens our argument: it demonstrates that plasticity loss in PPO requires a more sophisticated solution than simple weight resetting. This finding motivated our development of the distillation mechanism, which effectively addresses both the plasticity maintenance and knowledge preservation challenges. Our results suggest that PPO performance limitations may be related to plasticity constraints than fundamental algorithmic limitations.

Response to 3.2

We have added experiments with SAC in Appendix A.5 to enable direct comparison. While these new results demonstrate our method's advantages in optimal conditions, we maintain that our original PPO results, even if sub-optimal, provide valuable insights. Our method's ability to achieve superior performance when baselines converge to sub-optimal solutions specifically highlights its effectiveness in overcoming learning barriers. The combination of optimal SAC results and challenging PPO scenarios strengthens our argument - our approach not only excels under favorable conditions but also demonstrates significant improvements when traditional methods struggle.

Response to Questions 2

Our research primarily addresses the plasticity challenges in on-policy RL methods, where only a small replay buffer (e.g., 8,192 samples in our experiments) can be utilized to recover useful knowledge that might be temporarily lost during plasticity-enhancing resets. In contrast, off-policy methods can leverage much larger replay buffers (typically 1M samples) to mitigate such knowledge loss. Although we focus on on-policy methods due to their inherent limitations in knowledge recovery, we have also included comprehensive experimental results with SAC in Appendix A.5 to provide a more thorough validation of our approach. These supplementary results demonstrate that our method's benefits extend beyond its intended scope, showing improvements even in off-policy settings.