Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages

2. For Visual RL Community:

🌟 We uncover that the primary obstacle to achieving sample-efficient VRL is, in fact, the catastrophic plasticity loss of the critic module, rather than the previously widely attributed poor visual representation capability of the encoder. Our detailed deconstruction of plasticity loss in VRL offers a clear and novel pathway for enhancing its sample efficiency.

Given that the primary distinction between VRL and state-based RL lies in the concurrent optimization of task-specific policies and learning of compact state representations from high-dimensional observations, previous studies have naturally attributed the sample inefficiency in VRL primarily to inadequate visual representation capabilities.Recent advancements in VRL, focusing on enhancing sample efficiency, have primarily centered around the development of advanced representation learning techniques to forge more effective encoders. However, recent empirical studies [5, 6] suggest that these methods do not consistently improve training efficiency, indicating that insufficient representation may not be the primary bottleneck hindering the sample efficiency of current algorithms.

Although recent studies increasingly acknowledge that merely enhancing visual representation no longer yields significant improvements in sample efficiency, the primary impediment to further advancements in VRL sample efficiency remains elusive. This paper, through a deconstruction of plasticity loss in VRL, identifies that the catastrophic plasticity loss in the critic module is indeed this key bottleneck. As reviewer Kom1 notes, 'the paper can provide a promising potential for a new methodology to deal with plasticity loss,' our research at least suggests two viable pathways for further enhancing VRL sample efficiency: 1. Designing more effective interventions to better maintain the critic's plasticity, with data augmentation (DA) being an example of such interventions; 2. Utilizing the critic's plasticity level as a metric to dynamically adjust the algorithmic design of RL training, as exemplified by Adaptive Reward Reshaping (ARR).

[5] Li X, Shang J, Das S, et al. Does self-supervised learning really improve reinforcement learning from pixels?. NeurIPS 2022.

[6] Hansen N, Yuan Z, Ze Y, et al. On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baseline. ICML 2023.

Thank you for your feedback! Here are our responses:

We acknowledge your perspective that data augmentation (DA) is a well-established factor in the field of Visual Reinforcement Learning (VRL). However, the relationship between DA and plasticity loss remains under-explored, which constitutes a primary contribution of our study. This aspect has also been favorably recognized by all the other three reviewers. Thus, we respectfully submit that our study provides more nuanced insights than a mere reaffirmation of DA's effectiveness. Specifically, our investigations yield substantial insights into at least the following three key areas:

---

1. For Data Augmentation:

🌟 Effectively mitigating catastrophic plasticity loss presents the most plausible explanation for the indispensability of data augmentation in achieving sample-efficient VRL, rather than merely offering a new perspective on the effectiveness of DA.

Contrary to its incremental role in enhancing test performance in supervised learning tasks such as image classification, DA plays an indispensable and decisive role in numerous visual RL tasks. As illustrated in [1], even simple random shift transformations applied to input observations can lead to significant performance improvements in previously unsuccessful algorithm. By analogy, if DA in supervised learning boosts algorithm performance from 80 to 90 points, in VRL it's more like transforming algorithms from a baseline of 20 points to an impressive 90.

Despite the DA operations in VRL being similar to those in supervised CV tasks, the explanations for DA's underlying mechanisms in CV do not readily apply to VRL scenarios. For example, DA can be considered as an a priori mechanism in tasks with distribution shifts between training and testing phases, adeptly mitigating these discrepancies. However, [2] note that all the environments from DMC employ a camera that is fixed relative to the agent’s position. Hence, robustness to shifts does not appear to introduce any useful inductive bias about the underlying tasks. Additionally, previous studies have widely held that in supervised vision tasks, DA can enhance representation learning by inducing invariance or equivariance constraints [3]. However, evidence from [4] demonstrates that adding explicit alignment regularization to DA does not improve its performance, but rather, can have a detrimental effect.

In summary, to the best of our knowledge, existing research fails to adequately explain how DA fundamentally impacts VRL training, turning previously ineffective algorithms into highly sample-efficient ones. Our study introduces a novel connection between DA and plasticity within VRL training, demonstrating that DA can significantly mitigate plasticity loss in the critic module. In the absence of DA, the detrimental impact of catastrophic plasticity loss hinders the effective training of VRL agents, thus elucidating the pronounced performance disparity in VRL scenarios with or without DA and providing a newperspective on the underlying working mechanism of DA.

[1] Ma G, Zhang L, Wang H, et al. Learning Better with Less: Effective Augmentation for Sample-Efficient Visual Reinforcement Learning. NeurIPS 2023.

[2] Cetin E, Ball P J, Roberts S, et al. Stabilizing off-policy deep reinforcement learning from pixels. ICML 2022.

[3] Wang H, Huang Z, Wu X, et al. Toward learning robust and invariant representations with alignment regularization and data augmentation. KDD 2022

[4] Klee D, Walters R, Platt R. Understanding the Mechanism behind Data Augmentation’s Success on Image-based RL. RLDM 2022.

1. Effectiveness:

The emergence of the high RR dilemma stems from the fact that increasing the replay ratio can enhance sample efficiency by boosting data reuse frequency, yet this is often counteracted by the more severe plasticity loss that accompanies it. Adaptive RR effectively capitalizes on the sample efficiency benefits of a high RR, while skillfully circumventing the dire plasticity loss consequences.

Prior to introducing Adaptive RR, the most effective method to address the high RR dilemma involved periodically resetting parts of the network while employing a high static RR. In the table below, we present a comparison between RR=2 (with Reset) and Adaptive RR across three representative tasks. Compared to high RR with Reset, Adaptive RR demonstrates superior advantages in at least three aspects:

(1) Higher sample efficiency. While applying Resets to a high replay ratio of 2 does improve its sample efficiency, it does not match the performance levels attained by Adaptive RR.

(2) Greater computing efficiency. Unlike high RR with Reset, which consistently maintains a high update frequency, Adaptive RR employs a lower RR in the initial stages of training, thereby enhancing computing efficiency.

(3) Enhanced safety during interaction. As emphasized in [1], the performance collapse caused by Resets can lead to a higher likelihood of violating safety constraints during agent-environment interactions. Adaptive RR, on the other hand, effectively avoids this issue.

[1] Kim W, Shin Y, Park J, et al. Sample-Efficient and Safe Deep Reinforcement Learning via Reset Deep Ensemble Agents. NeurIPS 2023.

Applicability:

Firstly, while our original manuscript primarily reported the experimental results from 6 DMC tasks, these tasks encompass a diverse range of challenging characteristics in continuous control tasks. For instance, Finger Turn Hard presents a typical sparse reward setting, while Quadruped Run features a large state-action space. The consistent performance improvement of our proposed Adaptive RR across these tasks demonstrates its broad applicability in diverse continuous control environments.

Furthermore, we acknowledge the reviewer's concerns about Adaptive RR's universality. To validate its effectiveness across varied tasks, we extensively tested it in the discrete Atari-100k tasks. The ensuing data, as shown in the table below, demonstrate that Adaptive RR surpassed both lower and higher static RR settings in 15 of the 17 diverse tasks. Conducted with 5 seeds per task, these findings will be elaborated in our forthcoming manuscript revision, highlighting Adaptive RR's broad applicability.

We recognize the reviewer's concerns in Weakness 2, mainly regarding the effectiveness and applicability of Adaptive RR, and its relevance to the concepts revisited in earlier sections. In our subsequent response, we'll further validate Adaptive RR's efficacy and versatility. For now, our focus is on clarifying the deep-seated link between our methods and the extensive exploration we conducted on data, modules, and training stages.

🌟 Our discovery that the critic’s plasticity is a pivotal factor in sample efficiency suggests that the FAU of the critic module could be adaptively used to discern the current training stage. Without a thorough investigation into how the plasticity of different modules differently impacts training, it would be challenging to precisely select the critic’s FAU as a metric for dynamically guiding the adjustments in RR. In networks with an actor-critic architecture, it might seem intuitive to use the average FAU of both actor and critic as the metric for guiding RR adjustments. However, as illustrated in Figure 17 in our paper's appendix, the divergent trends of FAU in actor and critic can lead to confusion, thereby obscuring the observable trends in the agent's plasticity.

🌟 Our investigation into the distinct properties of plasticity across different training stages directly motivates the implementation of Adaptive RR. Initially, a low RR is employed to prevent catastrophic plasticity loss. As training progresses and plasticity dynamics stabilize, increasing the RR can enhance reuse frequency. While the method of gradually increasing RR is simple, its effectiveness stems from our nuanced understanding of the complex issue of plasticity loss in DRL. Furthermore, our investigation provides robust empirical support for all the implementation details of the Adaptive RR approach.

🌟 The significant impact of DA on plasticity offers us a comparative experimental approach to study modules and stages. Section 3 of our paper not only reveals DA as an effective intervention to mitigate plasticity loss, but also the pronounced plasticity differences caused by the presence or absence of DA provide a compelling basis for comparative analysis. This allows for a deeper exploration of how plasticity varies and evolves across different modules and training stages.

I acknowledge the efforts made by the authors. I think my major concern "limited insights revealed" is partially alleviated, regarding the new discussion. However, I slightly disagree with the new emphasis from authors:

We uncover that the primary obstacle to achieving sample-efficient VRL is, in fact, the catastrophic plasticity loss of the critic module, rather than the previously widely attributed poor visual representation capability of the encoder. Our detailed deconstruction of plasticity loss in VRL offers a clear and novel pathway for enhancing its sample efficiency.

I think the visual representation ability is still the primary obstacle. As cited by the authors, the paper [6] actually shows that data augmentation is the key factor not only for visual RL but also for other settings like imitation learning. I think the claim made by authors "the primary obstacle to achieving sample-efficient VRL is, in fact, the catastrophic plasticity loss of the critic module, rather than the previously widely attributed poor visual representation capability of the encoder" lacks enough support, since the critic part also shares the visual representation.

Thank the authors for their detailed discussions and additional experiments and I would raise my score accordingly. Further considering the understanding from previous papers, I would tend to not give a positive score.

[6] Hansen N, Yuan Z, Ze Y, et al. On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baseline. ICML 2023.

Thank you!

It would be good to see these discussions in the updated manuscript in the future and more in-depth experiments.

I have raised my score.

It is interesting to see that another recent work DrM explores a similar thing and shows the potential of mitigating plasticity loss. I would be glad to see a direct discussion and comparison with DrM in the paper, such as the main difference, deeper connection, and further potential. I would consider raising my score accordingly.

Also, I hope all the related papers that authors mention during the discussion period should be included in their main paper.

I appreciate that the authors have prioritized responding to low-scoring reviewers first, but I would like to emphasize that I think this paper, with a few changes, could merit a >6 score, as I think Figure 1 alone is especially striking and worth being highlighted in a conference paper. Specifically:

1. The paper really needs to evaluate against ReDO. This will also help a lot to clarify the somewhat murky causality around dormant neurons and learning dynamics.

2. Including even a single non-Mujoco domain (ideally a more discrete, visually rich setting like an environment from Atari-100K or ProcGen) would go a long way towards validating the generality of the observations in this paper. It is worth noting that Schwarzer et al. noticed significant benefits from incorporating resets, though they evaluate on an expensive domain. Perhaps MinAtar could be a cheaper alternative?

3. A more mechanistic characterization of the effect of data augmentation (in lieu of additional evaluation environments) would give some indication of how environment-specific this strategy is. For example, to what extent is data augmentation mitigating overfitting/generalization issues vs network optimization pathologies?

Here we provide clarifications and explanations in response to the questions raised by the reviewer.

---

Q1: The definition of FAU is a bit unclear: is it measuring average activity of neurons over a large batch, i.e. the sparsity of the network's representation, or is the expression inside the indicator function nonzero if the unit is active for any input, i.e. FAU = 1 - (fraction of dormant neurons)?

A: In our paper, the Fraction of Active Units (FAU) is defined as a metric for measuring the average activity of neurons over a large batch, in line with the approaches previously outlined in [1] and [2]. Thank you for your attention to this detail. We will further clarify this concept in the subsequent versions of our manuscript.

[1] Lyle C, Zheng Z, Nikishin E, et al. Understanding plasticity in neural networks. ICML 2023.

[2] Lee H, Cho H, Kim H, et al. PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning. NeurIPS 2023.

---

Q2: The replay ratio adaptation method is not described in sufficient detail. How specifically is the "recovery of plasticity" measured? How robust is the method to different variations on this measurement?

A: After initiating training, we monitor the FAU of the critic module at every check interval of $I$ steps. When the FAU difference between consecutive checkpoints falls below the minimal threshold $\tau$, it signifies the 'recovery of plasticity'. In both DMC and Atari environments, the minimal threshold $\tau$ is set to $0.001$; for DMC, the interval $I$ is set to $5 \times 10^4$ steps, and for Atari, it is set to $2 \times 10^3$ steps. This experimental design stems from the observation that the critic's FAU rapidly decreases during the early stages of training and subsequently recovers. Once its value stabilizes at a relatively high level, the trend in FAU change becomes more gradual. Consequently, a minimal FAU difference between successive checkpoints suggests that the agent has moved beyond the early stages, which are more susceptible to catastrophic plasticity loss. At this point, we can safely increase the replay ratio to pursue higher sample efficiency.

Furthermore, compared to judging the critic's FAU against a specific absolute value, our method of observing FAU trends to determine its training stage offers greater robustness. As illustrated in Figure 17 of our Appendix, setting a specific FAU value as the criterion for 'recovery' often requires adjusting the threshold for each specific task. In contrast, our approach demonstrates considerable robustness in hyperparameters. The minimal threshold $\tau$, set at $0.001$ for both DMC and Atari environments, effectively works across various tasks. The only adjustment needed is selecting an appropriate check interval based on the total allowed steps.

W2: ... I have intentions to raise my score if the authors are able to show experimental results that their findings regarding the relationship between DA & reset also holds for other DMC environments.

A: Thank you for your interest in exploring whether incorporating Resets into DA can further enhance sample efficiency. Indeed, beyond the three tasks discussed in the main text, we have presented comparative results for eight other distinct tasks in Figures 11 and 12 of the Appendix. These tasks represent a diverse range of challenges and types within the DMC environments. Among these, only in 'Reacher Hard' did applying Resets noticeably improve the algorithm's sample efficiency and convergence performance.

We also acknowledge the differences in our experimental results compared to the original study on Resets in [1]. We believe the primary reason for these discrepancies lies in our use of DrQ-v2[2] as the baseline algorithm, as opposed to the original DrQ[3] used in [1]. 🌟 On one hand, DrQ-v2, with its sophisticated design enhancements, significantly improves sample efficiency and can effectively train in tasks where the original DrQ struggles. For instance, Figure 18 in [1] indicates that DrQ fails to train effectively in tasks like Quadruped Walk, Quadruped Run, and Reacher Hard, where the introduction of Resets greatly enhances performance. In contrast, DrQ-v2 with DA already achieves high sample efficiency in these tasks, leaving little room for further improvement through Resets. 🌟 On the other hand, DrQ sets the critic's replay ratio (RR) at 1, unlike the basic configuration of DrQ-v2, which is RR=0.5. As discussed in [1], higher RR leads to more severe plasticity loss, making the improvements from Resets more pronounced. In the basic configuration of DrQ-v2 (RR=0.5), our experiments indeed show that DA alone is sufficient to maintain plasticity.

However, as RR increases, the utility of Resets becomes apparent. We conducted comparative experiments in three tasks with and without Resets at RR=2. The results in the table below consistently show that Resets can further enhance sample efficiency by recovering plasticity in high RR scenarios, aligning with the conclusions in [1].

[1] The Primacy Bias in Deep Reinforcement Learning, Nikshin et. al., ICML 2022

[2] Yarats D, Fergus R, Lazaric A, et al. Mastering visual continuous control: Improved data-augmented reinforcement learning. ICLR 2022.

[3] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. ICLR 2021.

We appreciate your thorough review and insightful feedback. We will address each of your comments and concerns below and also in our revised manuscript.

---

W1: The manuscript would benefit if it were to make their main message clearer by aligning their findings and their proposed method.

A: We are grateful to the reviewer for their attention to the structure of our paper. Indeed, the development of Adaptive RR is intricately linked to our in-depth analysis of plasticity loss in VRL.

The challenge of the high RR dilemma in DRL is well-recognized, stemming from a trade-off between improved sample efficiency due to greater data reuse and the accompanying catastrophic plasticity loss. Our thorough examination of plasticity from various perspectives provided a clear path to address this issue. Every part of our paper connects together to make a complete story. Without all these parts, we wouldn’t have been able to suggest Adaptive RR as a simple but effective way to deal with the high RR problem.

🌟 Our discovery that the critic’s plasticity is a pivotal factor in sample efficiency suggests that the FAU of the critic module could be adaptively used to discern the current training stage. Without a thorough investigation into how the plasticity of different modules differently impacts training, it would be challenging to precisely select the critic’s FAU as a metric for dynamically guiding the adjustments in RR. In networks with an actor-critic architecture, it might seem intuitive to use the average FAU of both actor and critic as the metric for guiding RR adjustments. However, as illustrated in Figure 17 in our paper's appendix, the divergent trends of FAU in actor and critic can lead to confusion, thereby obscuring the observable trends in the agent's plasticity.

🌟 Our investigation into the distinct properties of plasticity across different training stages directly motivates the implementation of Adaptive RR. Initially, a low RR is employed to prevent catastrophic plasticity loss. As training progresses and plasticity dynamics stabilize, increasing the RR can enhance reuse frequency. While the method of gradually increasing RR is simple, its effectiveness stems from our nuanced understanding of the complex issue of plasticity loss in DRL. Furthermore, our investigation provides robust empirical support for all the implementation details of the Adaptive RR approach.

🌟 The significant impact of DA on plasticity offers us a comparative experimental approach to study modules and stages. Section 3 of our paper not only reveals DA as an effective intervention to mitigate plasticity loss, but also the pronounced plasticity differences caused by the presence or absence of DA provide a compelling basis for comparative analysis. This allows for a deeper exploration of how plasticity varies and evolves across different modules and training stages.

W3: ... it is essential for the authors to provide the following results in order to verify whether the proposed adaptive RR is indeed superior compared to static RR.

A: To demonstrate the scalability of Adaptive RR, we evaluated its effectiveness under configurations of RR=4 and RR=8. As shown in the table below, 🌟 the performance of Adaptive RR (from 0.5 to 8) significantly exceed that of a static RR set at 8. 🌟 Furthermore, Adaptive RR (from 0.5 to 4) not only surpass static high or low RR but also outperform the original ARR (from 0.5 to 2) in 2 out of 3 tasks presented in the main text. This strongly validates the universality of Adaptive RR and its potential to achieve even greater efficacy.

---

W4: ...their experimental environment is confined to DMC...

A: Firstly, while our original manuscript focused on 6 DMC tasks, these encompass a wide spectrum of challenging features within continuous control tasks. For example, Finger Turn Hard is characterized by a typical sparse reward environment, whereas Quadruped Run has an extensive state-action space. The consistent enhancement in performance with our proposed Adaptive RR across these tasks underscores its wide-ranging suitability for various continuous control scenarios. Additionally, as we have demonstrated in our response to W3, Adaptive RR has the potential to scale to even higher RR values.

Furthermore, in response to the reviewer's concerns regarding the universality of Adaptive RR, we conducted comprehensive evaluations on Atari-100k tasks. The results, as detailed in the forthcoming table, indicate that Adaptive RR consistently outperforms both lower and higher static RR in 15 of 17 discrete control tasks. Each task underwent five random runs, and these robust findings, which will be expanded upon in our revised manuscript, strongly validate Adaptive RR's broad applicability and effectiveness.

W2: DrQ seems to be using an update frequency of 1 while DrQ-v2 is using an update frequency of 2, which means that from the default setting DrQ-v2's replay ratio is higher, and naturally DrQ-v2 should enjoy Reset better than DrQ does.

A: We agree with the reviewer that the default RR in DrQ = 1.

DrQ: We perform one training update every time we receive a new observation.

However, we emphatically assert that the default RR for DrQ-v2 is indeed set at 0.5.

In light of the reviewer's comments, we revisited the 'default set of hyper-parameters' in the DrQ-v2 original paper's appendix and identified a potential source of the misunderstanding: Agent update frequency = 2. The naming of this hyperparameter could easily lead to confusion with the Replay Ratio (RR). However, in this context, frequency=2 actually denotes that the agent is updated every two steps. Conversely, RR is defined as the number of gradient updates per environment step. Hence, the default RR in DrQ-v2 should indeed be calculated as 1 / Agent update frequency, equating to 0.5.

We further checked the DrQ-v2 code and found that Agent update frequency = 2 is named as update_every_steps: 2 in the code, as seen in config.yaml#L41. The code utilizing update_every_steps to adjust the update frequency appears in drqv2.py#L230-L234. Specifically, it is implemented as follows:

def update(self, replay_iter, step):
    metrics = dict()

    if step % self.update_every_steps != 0:
        return metrics

This section of the code refers to updating the agent when step is an integer multiple of update_every_steps. The default value for update_every_steps is set to 2, thereby resulting in a default replay ratio of 0.5.

It's also worth noting that the adjustment of the update frequency in the original implementation does not directly allow for a RR greater than 1. For example, setting update_every_steps to 0.5 still results in updating the agent every step, which effectively means an RR of 1.

To accommodate higher RR settings in our experiments, we introduced an additional hyperparameter replay_ratio. We modified the code in train.py#L180-L182 to implement these higher RR settings. Our code modifications, including the introduction of the replay_ratio hyperparameter and adjustments to accommodate higher RR, are detailed in the Supplementary Material.

# try to update the agent (original)
if not seed_until_step(self.global_step):
    metrics = self.agent.update(self.replay_iter, self.global_step)
    self.logger.log_metrics(metrics, self.global_frame, ty='train')

Our Revision:

When RR>1: set update_every_steps=1, self.agent.replay_ratio=RR;

When RR<=1: set update_every_steps=1/RR, self.agent.replay_ratio=1;

# try to update the agent (ours)
if not seed_until_step(self.global_step):
    for _iter in range(self.agent.replay_ratio):
        metrics = self.agent.update(self.replay_iter, self.global_step)
    self.logger.log_metrics(metrics, self.global_frame, ty='train')

In conclusion, the default RR for DrQ-v2 is indeed set at 0.5. Furthermore, as demonstrated in Figure 7 of our paper, the default RR of 0.5 achieves a high level of sample efficiency compared to settings of 1, 2, and 4. This optimal efficiency is attained with the lowest computational cost. Thus, using the default setting as a high-level baseline, it is entirely reasonable for us to compare the effectiveness of DA and Reset in mitigating plasticity loss under these conditions.

Despite this, we greatly appreciate the reviewer's suggestion to compare DA and Reset under high RR conditions, which is crucial for enhancing our manuscript. Our experiments under high RR conditions are in complete agreement with the findings in [1]. We will add these comparisons to our forthcoming revision for a more thorough and meticulous examination of DA and Reset.

[1] The Primacy Bias in Deep Reinforcement Learning, Nikshin et. al., ICML 2022.

---

Thanks again for the valuable suggestions. We look forward to further discussions with the reviewer.

First of all, I thank the authors for their extensive and insightful discussion and would like to say that most of my concerns have been resolved. However, still I have some concerns regarding the author's comments and would like to leave a few comments below.

Weakness 1 : Although I agree that the thorough investigation of whether FAU of the critic module or actor module is more important and investigating how adaptive RR operates is insightful, this still doesn't seem to be related to DA. How about removing the DA section from the manuscript? Although I understand that DA is an effective approach in mitigating plasticity loss, this still doesn't seem to be related with the major findings of this manuscript.

Weakness 2 : Although it is interesting that Reset works differently between DrQ and DrQ-v2, your argument that DrQ's replay ratio is higher than of DrQ-v2 seems to be wrong. Check Appendix B.3 of [1] and Appendix B of [2]. From what I understand, DrQ seems to be using an update frequency of 1 while DrQ-v2 is using an update frequency of 2, which means that from the default setting DrQ-v2's replay ratio is higher, and naturally DrQ-v2 should enjoy Reset better than DrQ does (as shown by the extra experiments for Cheetah, Walker and Quadruped Run for RR=2). So, this raises doubt regarding the rationale of the phenomenon observed in the author's manuscript.

[1] Image Augmentation is All You Need: Regularizing Deep Reinforcement Learning from Pixels, Kostrikov et. al., ICLR 2021

[2] Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning, Yarats et. al., arXiv 2021

Despite my remaining concerns, I personally found this manuscript insightful and enjoyed reviewing this paper. Therefore, I wish to increase my score to 6. Thank you for your hard work.

W3: Applicability Concerns: ... how the findings translate to non-actor-critic methods or across diverse benchmarks. For instance, assessing the proposed adaptive replay ratio method on a well-established sample-efficiency benchmark such as Atari-100k, with Rainbow algorithm, would improve the practical significance of the adaptive RR method.

A: We thank the reviewer for the suggestion to extend our evaluations to non-actor-critic methods and discrete control tasks. This is an excellent complement to our previous experiments in continuous DMC tasks. We have completed additional experiments in the Atari-100k environment, utilizing DrQ($\epsilon$) as our baseline, which is a Rainbow DQN-based approach. Due to time and computational resource constraints, we have directly adopted the default replay ratio (RR=1) results from the BBF for all tasks, except for SpaceInvaders and Enduro. For the other experiments we conducted, each was performed with 5 random runs. In addition to experiments with three different static RRs, we also reran the ReDO experiments to serve as our baseline. The results are presented in the following table.

🌟 Adaptive RR outperforms other configurations, including ReDO, in 11 out of the 17 tasks. This not only demonstrates that Adaptive RR can strike a superior trade-off between reuse frequency and plasticity loss, but also shows that dynamically adjusting the RR based on the critic's overall plasticity level (specifically referring to the Q network in Atari) enables competitive performance through effective neuron-level network parameter resetting.

---

We look forward to further discussions with the reviewer.

We are grateful for your comprehensive review and valuable feedback. We apologize for the delayed response, as considerable time was required to complete the necessary experiments. We will address each of your comments and concerns in the following responses, as well as in our revised manuscript.

---

W1: Metrics-Related Concerns: ... While no single metric may definitively quantify plasticity loss, a consistency of findings across multiple measures would strengthen the empirical foundation of their arguments.

A: We fully concur with the reviewer's point that introducing additional metrics is essential for a more comprehensive characterization of an agent's plasticity. Beyond FAU, we have employed Weight Norm and Feature Rank as indicators to quantify plasticity. Specifically, we utilized these more comprehensive metrics to more accurately compare the characteristics of different modules and to contrast the effects of data augmentation with other interventions. The results of these experiments have been updated in Figures 19 and 20 of the paper's appendix. This revisitation of VRL's plasticity using a broader range of metrics has provided us with deeper insights into the distinct characteristics of the actor and critic.

🌟 DA significantly enhances the feature rank of both the actor and critic networks, effectively preventing implicit under-parameterization [1]. Combined with previous observations that DA results in a low FAU for the actor network and a high FAU for the critic network, we can infer that a sparse actor network still retains the necessary expressivity. Conversely, the critic is required to maintain ample plasticity and active states to continuously learn from the non-stationary targets presented by newly collected experiences.

[1] Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning. ICLR 2021.

---

W2: Analytical Concerns: ... what happens when these are combined ... Providing a theoretical framework or empirical evidence for DA's efficacy in reducing plasticity loss would be a valuable addition to the field, potentially guiding future development of more robust VRL systems.

A: The reviewer's suggestion for a more in-depth analysis and comparison of DA with other interventions is indeed necessary. It not only aids in understanding the powerful ability of DA to mitigate plasticity loss but also assists in designing more efficient VRL systems. As demonstrated in the PLASTIC study [2], combining various effective interventions can consistently maintain plasticity across a range of complex scenarios. Due to time and computational resource constraints, we plan to conduct experiments in future manuscript versions comparing DA with other interventions when combined, as [2] has already adequately demonstrated the potential of such combinations. During this rebuttal period, we will focus our discussion on why DA, as opposed to interventions like layer normalization, possesses a stronger capability in maintaining plasticity. To gain a deeper understanding, we employed a variety of metrics to measure the differences in training dynamics when using various interventions. The experimental results are displayed in Figures 20 of the paper's appendix and will be incorporated into the main text in subsequent versions of the manuscript. Our principal findings and understandings can be summarized as follows:

🌟 Although CReLU, by altering the activation function, forces the FAU of the critic to always be 0.5, it does not significantly enhance the critic's feature rank. This implies that while this method ensures each neuron is activated, these neurons do not possess sufficient expressivity.

🌟 Layer Normalization (LN) fails to effectively recover the FAU of the critic in the early stages of training. This inability prevents it from continuously learning from new experiences in later stages, aligning with the trend observed in its episode return training curve. After initial effective learning, algorithms using only LN quickly reach a training plateau, maintaining suboptimal performance without further improvement.

🌟 Employing DA effectively elevates the critic's FAU to a high level in the initial training phase, thereby avoiding catastrophic plasticity loss. This allows the agent to continuously optimize the accuracy of Q estimation through interactions with the environment. Additionally, with DA, we also achieve an actor with high expressivity. These two aspects together contribute to the effectiveness of DA in maintaining plasticity, making it an essential method for achieving sample-efficient VRL.

[2] PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning. NeurIPS 2023.