A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning

Thank you for taking the time to review our paper and provide helpful feedback.

Our intention with this work was to focus on the on-policy setting, given that the majority of the work in the area of plasticity loss has focused exclusively on the off-policy setting. Our decision not to include off-policy experiments was based on the fact that we believed there was ample evidence characterizing the problem and various solutions already existing in the literature. We agree that a direct comparison utilizing a single codebase between the two may have presented additional value, but we chose to instead allocate resources towards extensive e experiments specifically in the on-policy setting.

We agree that the information in Figures 4 and 5 could be presented in a more interpretable way. Our goal is to allow readers to see the entire trends for each method, but given the number of methods and task conditions, this presents a challenge to represent succinctly. Thank you for pointing out the rliable library. We have used it to generate a potential replacement for Figure 4, which you can find here. If it seems like an improvement, we will also replace Figure 5 using the same layout, and move the current figures to the appendix.

In our experiments which involved resetting the “final layer”, we reset both the critic and policy heads. Preliminary experiments (not reported in the paper) suggested that resetting both was superior to resetting either individually. We believe that this differs from the off-policy setting due to the critical role of the policy in the collection of samples. Whereas in off-policy learning there is a stable buffer of data to draw on, in the on-policy setting the data distribution used for training is much more sensitive to the policy. As such, resetting the policy head increases the overall policy entropy, thus ensuring that the data distribution used for training can remain diverse enough to prevent policy collapse. That said, the majority of methods considered here act on the entire network, rather than just the final layers of the network. As such, we believe that our results are largely robust to this choice. Relatedly, because we were considering discrete action policies, we utilized a shared encoder network, as this is the standard set-up in the PPO algorithm.

The total number of epochs was chosen such that convergence would always be reached within a single round of the experiment. As such, running experiments longer did not result in measurably different behavior than what has been discussed in this paper, and we did not find policy collapse or degradation within a single round. We believe that this suggests that the effects we observe in our experiments are indeed the result of plasticity loss. We will state this clearly in the camera ready.

Thank you for your valuable feedback on our paper.

Both you and another reviewer have brought up the readability of our graphs. We are working to improve this for the camera-ready version. Reviewer NjbE suggested utilizing the “rliable” library to generate plots. We took their suggestion and generated a potential new version of Figure 4 using the library, which you can see here. Please let us know if it improves readability, and if so we will replace both Figure 4 and 5 with this new layout and move the current figures to the appendix.

Thank you for pointing out the typographical errors. We have addressed them in our current draft of the manuscript.

The issue with the shrink-perturb method is that it is performed only occasionally, rather than at every step of SGD. This is fine in the original context under which it was introduced, where the problem setting enjoys a clear indicator of when it should be applied. In non-contrived RL settings, however, the environment might not provide this information—an implementation of shrink-perturb would either require us to either set the intervention cadance as a hyperparameter or to use additional machinery to detect appropriate points for application. We refer to methods that are applied at every step of SGD (like soft shrink perturb) as continuous, and we argue that this property is desirable in realistic RL settings.

It is worth clarifying our results concerning predictive measures for plasticity loss. We find that weight magnitude and dead unit count, when considered independently, are the two significantly predictive metrics of plasticity loss. Although we find that by itself gradient magnitude is not predictive, it becomes predictive as part of a Generalized Linear Model (GLM) which uses all the measures as predictors. This suggests that gradient magnitude is able to account for additional variance in plasticity loss which is not accounted for by weight magnitude, suggesting that it may contribute to some aspect of plasticity loss. In contrast, dead unit count is no longer predictive in the GLM due to its high correlation with weight magnitude, which is the more fundamental variable (because dead unit count is a function of weight magnitude).

We believe our results deviate from previous findings largely because of the breadth of settings we considered. Rather than studying only a single task, type of distribution shift, and model architecture, which might provide only narrow insight into the plasticity loss phenomenon, we explore several. Of course, our study also deviates from prior work in that we also consider only on-policy agents, and the underlying dynamics of plasticity loss in the two settings may differ.

We considered Montezuma's Revenge to be a more "natural" RL task, presented to show a use for what we have learned from environments studied earlier in the paper, which were chosen specifically to probe the plasticity loss pathology. As such, we unfortunately did not log the same diagnostic information for Montazuma's Revenge as we did for earlier tasks. Fitting these agents takes more time than is afforded in the NeurIPS discussion period, but we will collect this information for the camera ready copy.

The reward on the y-axis of Figure 6 is the agent's score in the game, reflecting how many objects it has collected, new rooms it has entered, and enemies it has vanquished. Our code for this is only a small modification to a popular PyTorch RND implementation.

We appreciate you taking the time to read and review our paper.

We used five replicates per experiment unless indicated otherwise, and the shaded regions of the graphs correspond to standard error. We will revise all figure captions to make both of these points explicit.

We used a learning rate of 5e-4 for all experiments, as displayed in Appendix C. We further use the default PyTorch values for Adam's $\beta_1$ and $\beta_2$, and we will update the section to reflect this detail.

We agree that it is possible that using the “Non-stationary Adam” method from Dohare et al. would improve performance on the tasks considered here. That said, we do not find evidence of policy degradation within-rounds (see for example Figure 2 upper-left in contrast to Figure 1 of Dohare et al.). Note that we chose the number of training epochs to reflect the asymptotic behavior of the policy—longer-running experiments, which were used to select this parameter, did not show evidence of policy collapse when the task/environment distribution is held fixed. Regardless, we will include a discussion “Non-stationary Adam” in our related work.