Q1. … leads to feature collapse, but … it is uncommon to see PPO run for 6 or 8 epochs (Figure 1, 2, and 3).

A1. The collapse phenomenon we show is certainly not exclusive to running with more epochs. The phenomenon does happen with the “standard” hyperparameters of the environments we study. This is the case on MuJoCo, and on ALE in Figures 1, 2, and 3, mentioned by the reviewer, we see the start of the collapse at 100M steps, which motivates us to increase the number of epochs to accelerate the collapse, and allows us to show that the phenomenon is due to non-stationarity and the inability of the policy to adapt to new targets.
In this sense, increasing the number of epochs is a tool in our analysis and is not a condition for collapse.
To observe collapse with the “standard” hyperparameters in Figures 1, 2, and 3 it suffices to train for longer. We show this on ALE/Phoenix and ALE/NameThisGame with 200M steps in Figures 34 and 35 in the PDF of the rebuttal.

Q2. While they attempt to explain feature collapse in the critic, the explanation is insufficient

A2. We do not attempt to explain the collapse of the critic network (see lines 147-149) which has been studied in previous works, e.g., Kumar et al., 2022 through implicit regularization and Lyle et al., 2022 who argue that sparser targets are more detrimental.
We make a few observations about the representation dynamics of the critic network in relation to the sparsity of the environment (lines 178-184), which then allow us to draw conclusions about sharing an actor-critic trunk in section 4 (lines 339-355).

If the reviewer meant the actor network, then we do describe its features and dynamics and connect them to non-stationarity. By showing that this collapse also holds in on-policy policy optimization in addition to offline, off-policy, and value-based RL as shown by previous work, we strengthen the hypothesis of non-stationarity being a major cause of the phenomenon.

Ultimately, discovering deeper causes beyond non-stationarity remains an open question as discussed in our conclusion (389-391).

Q3. While they emphasize that the focus of the paper is not on improving the performance of PPO, they do not provide a clear solution to mitigate feature collapse in PPO (Figure 6).

A3. In this work, we study interventions that have been shown in other settings to help and present their effects on the representation dynamics of PPO. This serves two purposes. 1) showing which ones can help and under which conditions (which environment features), and 2) when they do help and improve the trust region at the same time, strengthen the relation we have drawn between representation collapse and trust-region collapse. We have shown that sharing the actor-critic network when the environment has a rich reward signal does mitigate collapse (Figure 6, top and middle), and applying PFO also mitigates collapse when sharing the actor-critic trunk does not (Figure 6, 19, 20).

However, like any other empirical study, and as mentioned in our limitations, there is no guarantee on how well our results would generalize to other settings. We hope that our paper encourages further theoretical studies, and only then will we have provably or “clearly” mitigated the problem.

Q4. … no clear correlation between the number of epochs and the plasticity loss of the critic network … it contrasts with findings from value-based plasticity studies.

A4. Runs with higher epochs tend to have higher plasticity loss which can be explained by a stronger overfitting to previous targets and is consistent with the findings from value-based methods.

There are some exceptions, which have probably raised the reviewer’s attention, which can be explained by the behavior of the policy. When the policy collapses earlier in a run with a higher number of epochs, the critic's plasticity loss is prematurely stopped (there are no more changing objectives) and ends up lower than in a run with a lower number of epochs which has continued training until the end.

This is the case of Figure 7 and is explained in its caption.

Q5. How about replacing the clipping operation with a KL penalty? …

A5. We have also performed early experiments with PPO-KL. However, the adaptive coefficient of the KL trust region makes the algorithm extremely high variance and often collapses due to the coefficient exploding or collapsing independently of feature dynamics.

Nevertheless, for the runs where the coefficient was stable, we did indeed observe the same behavior as PPO-Clip, specifically that the features became more aliased until they collapsed, followed by a trust region collapse characterized by a blow-up of the KL divergence.

Q6. ...how about applying regularization in the weight space using EWMA update?

A6. PFO doesn’t stem from the sole motivation of regularizing the representations, it also seeks to do so by extending the trust region set by the clipping in the output space to an additional regularizer in the feature space.

The idea is to not impose an explicit constraint or a regularization on the weights directly to allow them to adapt to the moving targets but to regularize their dynamics through the features and output trust regions.

Q7. It would be beneficial to investigate Phasic Policy Gradient (PPG) [2]

A7. This is a great suggestion. We would put PPG in the same box of algorithms that add an auxiliary representation loss to the policy network while not interfering too much with its objective, unlike sharing the actor-critic trunk, which we show can be detrimental when its rank is low to start with.

In the scope of this work, we did not investigate such algorithms, as a thorough analysis would require assessing a comprehensive sample of these algorithms such as PPG, InFer (Lyle et al., 2022), and CURL (Laskin et al., 2020). We reserve this study for future work.

Thank you for your responses to my questions. My answers to some of your responses are shown below.

Q1. … leads to feature collapse, but … it is uncommon to see PPO run for 6 or 8 epochs (Figure 1, 2, and 3).

A1. Thank you for the detailed explanation! Increasing the number of epochs makes sense to me now.

Q2. While they attempt to explain feature collapse in the critic, the explanation is insufficient.

A2. I would like you to elaborate on Lines 182-184. However, I understand that the main focus of this work is not on the plasticity of the critic network. Thanks!

Q4. … no clear correlation between the number of epochs and the plasticity loss of the critic network … it contrasts with findings from value-based plasticity studies.

A4. I think I missed the caption. Thank you for letting me know.

Q5. How about replacing the clipping operation with a KL penalty? …

A5. Thank you for clarifying my question! Including these results will strengthen your paper.

All of my concerns have been well addressed, and therefore I am increasing the score from 5 to 6.

Updated: After reviewing feedback from other reviewers, I found that similar studies had been conducted prior to this work. Therefore, I have decided to maintain the original score.

Q1. The toy setting was good demonstration of the problems with trust region constraints. It may help to introduce this sooner … The paper would benefit from outlining the trust region insight sooner as it is a core idea that only gets explained in detail on page 7 of the paper.

A1. We thank the reviewer for the recommendation. This is indeed a key point of interest to the readers. We will update the introduction section to describe the insights sooner and point to the theoretical derivation for interested readers.

Q2. The results on the regularization term are a bit confusing and mixed. The narrative would benefit from clarifying the aspects that remain unresolved or ambiguous that merit further attention. The Gravitar analysis of sparse reward environments was helpful. But I was not really sure what to make of the effects on episode returns compared to representation collapse. Some way of teasing these effects out more cleanly would help.

A2. Figure 6 presents summaries of runs at the end of training aggregating runs with different numbers of epochs. It, therefore, shows the distribution of returns and representation metrics across different hyperparameters.
With this, when a method like PFO presents an improvement of the lower tail of episodic returns and, at the same time, an improvement of the representation metrics (no extremely poor ranks or fully dead networks), we can say that the method improves robustness to representation collapse and performance collapse, and strengthens the link we’ve drawn between the two in the earlier sections. In this sense, the effects on episode returns and representation collapse should be appreciated in tandem.
It should also be noted that the strong relation between them mainly holds around the collapsing regime, as described in Figure 4 and explained in our global rebuttal answer GA2.

Q3. line 143: "do not include normalization layers". Does this help plasticity preservation? (even if only to cite prior work on the effects)

A3. Lyle et al. (2023) conducted a study with non-stationary supervised learning and value-based methods and found that normalization layers do provide improvements to plasticity. In the scope of this work, we did not look at interventions that change the “degree” of non-stationarity in the input or output of the network, such as layer normalization, batch normalization, and running observation or reward normalization, as these would require a different analysis. As stated in our discussion and limitation plan to address these in future work.

Q4. lines 151-153: "We vary the number of epochs as a way to control the effect of non-stationarity, which gives the agent a more significant number of optimization steps per rollout while not changing the optimal target it can reach due to clipping, as opposed to changing the value of ε in the trust region for example." It might be interesting to contrast the effects of varying epsilon as well.

A4. As mentioned, varying epsilon would change the optimum of the surrogate objective so it’s not exactly clear if we can use this as a tool to investigate the relation between non-stationarity and feature collapse, like we did the number of epochs.
Nevertheless, per the reviewer’s request, we rerun Figure 1 with ALE/Phoenix with different values of epsilon and the baseline number of epochs of 4. As expected, and as shown in Figure 36 of the additional rebuttal PDF, we observe no apparent correlation with the time of collapse as both doubling (epsilon =0.2) and halving (epsilon =0.05) the baseline epsilon (epsilon =0.1) can yield training curves with a delayed representation and performance collapse.

Q5. line 280 (minor): a_i should be a_1 in this case

A5. Indeed. We thank the reviewer for the detailed feedback!

Q6. Figure 6 (minor). The caption only mentions "Top" and "Bottom", omitting the middle row of "NameThisGame".

A6. Top and middle share the same properties. We will add this to the caption.

Q1. The authors claim to open data and code; however, I could not locate them. Therefore, I apologize if I overlooked their presence.

A1. Yes, these are available on an anonymous GitHub repo mentioned in Appendix line 578. The GitHub repo contains further links to the Weights&Biases project with all the interactive training curves and links to download the raw logs.

Q2. Please provide a more detailed explanation on Eq.2. I don't understand the meaning of phi, and should the formula be s_t instead of S_t?

A2. $\phi(S_t)$ refers to the vector of the preactivations of the penultimate layer of the network on the state $S_t$. As PFO can also take all the pre-activations in the network, $\phi(S_t)$ could also be the concatenation of the preactivations of all the layers.
We will add these details before the equation in the camera-ready version. (This is analogous to $\Phi$ defined in the background (lines 100-104) as the feature matrix, consisting of the activations of the last hidden layer.) $S_t$ refers to the state that we capitalize as it is a random variable in the expectation over trajectories, like all other random variables in the background (lines 58 - 66) This is the notation from Sutton and Barto (2018).

Therefore, the equation is essentially an L2 norm between the features of the same state at the old policy and the learned one: $\|\| \phi_\theta(S_t) - \phi_\text{old}(S_t)\|\|_2^2$.

Q3. In section 4, the authors introduce an intervention that allows the actor and critic to share all the layers. The experiment results illustrate that this can bring some improvements but not significantly. I am more curious about whether the reason for this is the sparsity of reward or the environment itself. I think the latter is more likely, and we can verify this by setting up a control experiment with different reward functions in the same environment.

A3. This is a good observation! The representation of the critic plays a key role in the reasoning. When the critic, if trained separately, is subject to a degraded representation (or collapse) with a very low rank and many dead units, we observed that sharing its representation with the policy makes the policy collapse faster. When the environments are similar – we consider two ALE environments, e.g., Phoenix and Gravitar, to be similar because of the observation space, etc – the main difference in making a critic collapse or not is sparsity of the reward as observed by Lyle et al. (2022). Therefore in this case, it is fair to say that the reason for the collapse is mainly due to reward sparsity.

Nevertheless, one needs to be careful when comparing environments when more than one degree of freedom changes; for example, this would make comparisons of experimental results obtained on Mujoco and ALE environments harder to attribute to sparsity only.

Per the reviewer's request we ran an experiment on ALE/Phoenix (a dense reward environment), with a reward mask randomly masking a reward with 90% chance, and compared the effects of sharing the actor-critic trunk. The results are in the additional PDF of the general rebuttal. As expected, while with dense rewards, sharing the trunk was beneficial in ALE/Phoenix (Figure 21 Appendix), with the sparse reward, the opposite is true: sharing the trunk is detrimental (Figure 37 Rebuttal). This confirms our conclusion. We thank the reviewer for suggesting the experiment, we will add it to the Appendix of the paper. We hope this strengthens the reviewer's support for the paper.

Q1.a Figures can be plotted with better quality, don't overlap the labels (in Figure 3),

A1.a Indeed. We will correct this in the camera-ready version.

Q1.b and maybe set the titles to the quantity of interest.

A1b. Can the reviewer please elaborate on this or give an example? What is meant by title and quantity of interest?

Q2.a isn't it expected that the policy learns to lose rank, by compressing state information to only correlate with actions?

A2.a Indeed, however, we distinguish between a low rank that’s beneficial and an extremely low rank that is detrimental. We have clarified this in answer GA1 in the global rebuttal.

Q2.b I understand that several works are measuring the feature rank at the last policy/critic layer. But why that is sufficient evidence that the policy is losing rank? For instance, if my action space is binary, then the policy's last layer could learn to be rank-2. More information might be stored in previous layers.

A2.b The penultimate layer serves as a bottleneck for decoding the actions or the value, so when this layer reaches an extremely low rank (as described in the previous answer, to distinguish for a beneficial low rank), this becomes irrecoverable. This also strongly correlates with plasticity loss, as this layer serves as a bottleneck for reconstructing information from the input state.

Q3.a The proposed Proximal Feature Optimization objective is a bit confusing. First it should be a l2 norm? What does it mean by (phi(s) - phi_old(s))^2.

A3.a Yes, we thank for pointing this out. $\phi(s)$ is a vector containing the pre-activations. The square of the difference should be more clearly written as the squared norm of the difference between the vectors: $\|\| \phi_\theta(S_t) - \phi_\text{old}(S_t)\|\|_2^2$.

Q3.b Secondly, from a high-level, isn't this just a regularization loss as in continual learning that prevents forgetting? But based on my knowledge it will exacerbate the the loss of plasticity. If you want to preserve rank, should not you add some reconstruction loss?

A3.b PFO regularizes the change in features at every batch, unlike methods that prevent forgetting, which regularize the parameters towards a fixed optimum of a previous task.
In this sense, PFO is sought to be an extension of the clipping trust region applied in the output space to a trust region in the feature space. It comes from our observation that the clipping is not enough to satisfy the trust region.
It also allows us to address the undesired symptoms we observed (high feature norm and aliased features).
Other auxiliary losses, such as a reconstruction loss, can also be beneficial in this case but do not necessarily constrain or regularize the step size.
Finally, to support our claim that regularizing representations would prevent trust region and performance collapse, we found that using a regularization that directly targets one of the undesired symptoms we observed would make the connection more straightforward.

Q4. From Figure 6, does it mean that the proposed method does not consistently improve the loss of plasticity?

A4. It does consistently improve the plasticity loss. This can not show in the the figure when the non-regularized model collapses too early. This is described in the caption of Figure 18: "the tails of the plasticity loss on Phoenix with interventions can be higher than without interventions on the runs where the models collapse too early without interventions, leading to the plasticity loss of the non-collapsed models with interventions eventually becoming higher."

Q5. If the main goal is to prevent the preactivation norm from going larger, why not just regularize it to be smaller?

A5. The increasing preactivation norm phenomenon can be seen as an effect of a bigger issue: the failure of the clipping trust region imposed on the output to ensure a trust region on the rest of the network. Therefore, as mentioned in A3.b, PFO is sought to extend this trust region. Analogously to constraining ratios between the previous and current policy, it regularizes the difference in features between the previous and current policy.
This allows PFO to address both the feature norm and the trust region issue. This is described in lines (319-322), which we will rephrase to highlight this point more.

Q5. Using the term "plasticity loss" as a metric is confusing because it also refers to the overall phenomena …

A5. Yes, we use this metric to compute the loss from fitting a random initialization's trajectories. It is defined in the background (114-122) and to fit the actor we use a KL divergence. In the paper, we distinguish between the metric called plasticity loss and the phenomenon called loss of plasticity. This is the value of a loss, that’s why we call it plasticity loss. However, the phenomenon has also been termed plasticity loss by Lyle et al. (2024), so we agree that reusing it for the loss can be confusing. We propose to state our distinction between the two forms in our background, making it less confusing. Otherwise, we can change the loss term to to random fit error.

Q6. Fig.1 Why does the preactivation norm plateau at 10^4?

A6. Typically, the preactivation norm increases until all the neurons of the feature layer are dead (in the case of ReLU). After that, all the gradients for the weight matrices are 0, so the norm flattens out.

Q7.a Which data is used to produce the box plots? …

A7.a Each run is summarized with an average over the last 5% of its progress. The 15 runs consist of 3 epoch values (4, 6, 8) x 5 seeds. This is presented in lines 307-308 and further detailed in the appendix lines 641 - 644, 658-659.

Q7.b In the box plots, how are outliers determined? …

A7.b The whiskers are determined by the highest observed datapoint below Q3 + 1.5 IQR (similarly for the lower one) (default of matplotlib). The outliers are points outside of the whiskers.

We thank the reviewer for the question and will add this information to Appendix lines 659+.

Q9. I do not think it would be fair to see PPO has more nonstationarity than other policy gradient methods…

A9. We agree that, from this perspective, REINFORCE could be considered as more nonstationary than PPO. We will remove this "more nonstationary" observation; it is not critical to the work. The critical point of in work is that PPO performs multiple epochs on the same data and, in this sense, can be more prone to "overfitting" to previous experiences, which can become worse with more epochs.

Q10. Line 257. "The underlying assumption...approximately orthogonal..." Could you give some more support to this claim? …

A10. The presumed orthogonality here is between the states where the policy should act differently, like with two different groups of aggregated states. Otherwise, for the states where the policy should be similar (same group of aggregated states), some alignment in the features and a reduction of dimensionality are indeed desired for generalization. However, with clipping, this alignment should be associated with the right relative feature norm; otherwise, the clipping will be violated, as observed in Figure 5, where a larger alpha would make the relative deviation of the action probabilities larger.

Q11. Line 8-9: "...policy optimization methods which are often thought capable of training indefinitely"...

A11. This is indeed arguable. We believe that trust region methods aim to overcome this high variance and instability. However, this is not a critical claim, and we can remove it. We thank the reviewer for the suggestion.

Q15. Interestingly, the value network is not the problem. …

A15. Yes, this is one key point of our work. Policy networks also suffer from representation collapse independently of value networks, as noted in the caption of Figure 1. Our intuition and the main motivation of this work is that, similarly to value networks, policy networks are also subject to the non-stationarity of their inputs and outputs (background, lines 88+), and optimizing deep neural networks under non-stationarity is known to cause issues.

Q16. A suggestion would be to try the Atari environemnts identified in [1] to benefit from plasticity injection…

A16. Phoenix is one of the environments we have in common and where we see the most impactful results.

For this work, we did not want to select environments with prior knowledge of existing collapse in value-based methods to demonstrate the collapse in policy optimization; we rather used the unbiased sample recommended in Atari-5 (Aitchison et al. 2023) because we did no want to unconsciously cherry-pick environments, where certain approaches may perform better. Moreover, we would like to highlight that, on Atari, we could not find any large-scale study of on-policy RL algorithms and PPO demonstrating the collapse of the actor’s representation from which we could pick our environments at the time of submission.

Q17. A minor suggestion is to use stable rank [2] as a measure of the rank …

A17. We have tracked 5 different measures of the rank, spanning continuous or discrete and relative or absolute metrics. We conducted a thorough comparison of these in Appendix E.

We address the key points in the rebuttal and the remaining ones in a comment.

Q1. line 193 feels a bit out of place … investigation is not mentioned earlier … suggest removing or summarizing the paragraph …

A1. This information is relevant to practitioners in diagnosing the kind of collapse they face. One common collapse or stagnation is due to entropy collapse and lack of exploration. In this work, we highlight that when collapse is associated with a high entropy, it may likely be due to collapsed representations. We agree with the suggestion of the reviewer and will summarize this paragraph in a sentence.

Q2. … research questions Q1. and Q2. do not mention the trust-region although it is a large part of the paper.

A2. Q1 and Q2 serve as motivation for section 3.1, where they are answered. These are not the only questions addressed in the work. The trust-region exposition in section 3.2 is only motivated after answering Q1 and Q2 and observing collapse.
We will move Q1 and Q2 inside section 3.1 to clarify their scope and reformulate section 3.2 with two questions, Q3 and Q4, to mirror 3.1 and ensure the question format covers all the main points of our work.

Q3. I would be careful about mentioning that low feature rank necessarily means that the representation is poor … I am also concerned about the link between feature rank and trust-region violation …

A3. Reviewer XWGd has also raised this concern. We have clarified this in GA1 and GA2 of our global rebuttal.

Q4.a how the average probability ratio is computed? …

A4.a Yes, this is correct, and more details can be found in the Appendix lines 635-640.

Q4.b If so, this seems potentially misleading …

A4.b The mean ratio without conditioning does not give information about learning or the trust region. The PPO-Clip objective increases the ratios of actions with positive advantages and decreases those with negative advantages until they reach the clip limit. Therefore, to have a signal for learning, we have to at least condition on the sign of the advantage. Still, a takeaway from the toy example is that taking the mean ratio of actions with, say, a positive advantage under bad representations would mix the ratios that suffer from interference (Figure 5 right) and would give a misleading average.

Therefore, we found that the right way to quantify the trust-region violation is to condition on the violation itself (e.g., above $1+\epsilon$). We took the mean rather than the count because fully optimizing the PPO objective should push the actions until the clip limit, so it's not clear whether a high clip count would be worse than a lower one.

Q4.c they may overemphasize small probabilities in the denominator.

A4.c This is true but has not been a critical issue for our analysis. In Figure 4 with ALE the smallest probability ratio is around 0.4, and in Figure 6, the largest excess ratios are between 1.5 and 2.5.

Q4.d I would suggest also reporting how often the clipping criteria are violated …

A4.d Yes, we track this. As discussed above (A4.b) it’s not clear what the count of the clip fraction should be and it is not an interesting quantity for our study.

Q4.e consider ratios less than 1 or greater than 1 seperately

A4.e We do separate them in Figure 4, where we look at the ratios below $1-\epsilon$ and isolate the causal relation around poor representations. For Figure 6, aggregating them into a ratio provides a better summary with larger plots.

Q8. How is the PFO regularizer related to adding a regularizer in parameter space? …

A8. We can construct a spectrum ranging from regularizing the parameters to the outputs. The lower end regularizes the parameters like an L2 weight difference. The higher end regularizes the network's output like PPO clipping and almost like a target network in value-based methods. PFO sits in the middle of this spectrum, where the regularization allows both the network's weights and final outputs to change without explicit constraints while maintaining a regularization of the feature space. Also, PFO should mostly be seen combined with PPO, to extend the trust region to the feature space.

Q12. Is there a reason why the paper measures feature norms instead?

A12. Other work we cite, like Lyle et al. (2024), has also looked at feature norms in addition to weight norms. We noticed that the model weights were consistently and steadily increasing regardless of collapse; however, the feature norm showed a sudden jump around collapse. We found this to be a more apparent symptom that researchers and practitioners would want to investigate further.

Q13. degradation from training more epochs could be attributed to difficulties of offline RL

A13. Off-policiness could be a good characterization here as the algorithm has access to the action probabilities of the collected data and the policy that collected it more generally. In PPO the more epochs performed, the more the on-policy approximation becomes off-policy. However, the PPO-Clip objective is supposed to be robust to slight changes in the number of epochs, as it only depends on epsilon. Therefore, we view observing a drastic collapse by moving from four to six epochs in Figure 1 not as a failure of standard RL algorithms in the off-policy setting but as a failure of the learning dynamics of PPO-Clip.

Q14. I find the proposed solution PFO to be slightly unsatisfying, …

A14. The first purpose of PFO in this work, as the reviewer mentions, is to show that explicitly regularizing the feature dynamics builds robustness to trust-region and performance collapse. Now as with most empirical studies and regularizations using a coefficient, there is no guarantee on how long PFO will stand the test of time, but we hope that our paper encourages further theoretical studies and only then will we have provably mitigated the problem.

We are happy that our clarifications addressed all your previous concerns.

I have to agree that certain points of Reviewer qwmj are still concerning, including the ones around the performance of the baseline.

We understand the concern; however, this is a matter of clarification. We take the thoroughness of our implementation with utmost importance and have clarified all the necessary bits in our reply to the reviewer, which we summarize below:

1. Recall that our results hold on Atari and we have already replicated them with the CleanRL implementation the reviewer is using.
2. Recall that our implementation for MuJoCo is based on recent implementations for continuous action spaces influenced by seminal work studying implementation details in PPO and that we find this setting more relevant to our audience.
3. Replicate our implementation in CleanRL, which is still exhibiting collapse, so the reviewer can more easily inspect the code.
4. Adapt the default CleanRL implementation the reviewer is using with a fully state-dependent action distribution and observe collapse in the setting they are familiar with and interested in.

I am also still a bit underwhelmed by the fact that trust-region failures seem to only occur when the representation has already collapsed and the performance is poor.

Unfortunately, this is a fact. We extensively investigated the link in the early stages of the project to find strong evidence throughout training, which to us was also the more exciting claim to prove; however, countless examples made us realize that the link did not necessarily hold throughout training but only around the collapsing regime. Using only positive examples or a narrow view of the correlations, to claim that such a link is present throughout training would be extremely misleading and has been an issue in offline RL. The work by Gulcehre et al. (2022) that the reviewer references is an excellent example of this. They conclude that previous exciting associations between performance and rank in offline RL are misleading and do not hold when considering a large experimental scope.

Our conclusion is that trust-region violations become more evident when representations are about to collapse. This does not mean that one should only be concerned about the link when performance is starting to deteriorate. The representations don’t collapse all of a sudden; they deteriorate throughout training until they reach collapse. So, mitigating representation degradation should happen throughout training and not only when around the collapsing regime.

We hope these clarifications address your concerns. Thank you for your input, and we look forward to your feedback.

Q1. … PFO is closely related to DR3 and RD …. More discussions are necessary.

A1. We thank the reviewer for pointing these out. Although these regularizations emerge from value-based offline-RL challenges, a discussion of their similarities with PFO can be valuable to our audience. We can add the following to our camera-ready version.

Other interventions regularizing feature representations have been studied in value-based offline RL. Kumar et al. (2022) propose DR3, which counteracts an implicit regularization in TD learningby minimizing the dot product between the features of the estimated and target states.
Ma et al. (2024) propose Representation Distinction (RD) which tries to avoid unwanted generalization by minimizing the dot product between the features of state-action pairs sampled from the learned policy and those sampled from the dataset or an OOD policy.

Both are related to PFO as the methods directly tackle an undesired feature learning dynamic, but there is no motivation for DR3 or RD in online RL, and PFO is conceptually different. The implicit regularization that DR3 counteracts is not present in on-policy RL as shown by Kumar et al. (2022) in the SARSA experiment, and PFO differs from DR3 as it extends a trust region rather than counteracts an implicit bias. Whence the different implementations: PFO regularizes the state features between two consecutive policies, as opposed to consecutive ones of the same policy in DR3.
Similarly, the overestimation studied by Ma et al. (2024) in the vicious backup-generalization cycle is broken by on-policy data. PFO’s motivation to bring the trust region to the feature space resembles RD’s motivation to bring the overestimation constraint to the feature space. However, they are again different in their implementation as RD regularizes state features between the learned policy and the dataset policy or an OOD policy.
Finally, PFO is applied to the actor, while both DR3 and RD are applied to the critic.

Q2. … do not provide sufficient support for the point that "mitigating the learning issues in feature rank, plasticity loss of PPO can improve the learning performance in terms of episode return". PFO … does not bring clear improvement in terms of learning performance like episode return ... This cripples the significance of PFO and also the potential causal connection …

A2. First, we claim that the causal connection holds around the collapse regime, not necessarily throughout training, and that discovering and describing such a relation only around collapse is nontrivial. We have clarified these arguments in GA1 and GA2 in the global rebuttal.

Second, we distinguish between consistently improving the best run performance (i.e. claiming to be “better” than PPO) and improving the aggregate performance across multiple runs and hyperparameters, e.g., robustness to collapse in our case. The limitation the reviewer raises is about the former, however this work argues about the latter.

In this sense, the significance of PFO is realized by a) successfully mitigating collapse as observed in Figure 6 with a significantly higher median performance indicated by the black lines, and b) improving the representation metrics and obtaining a better trust region, therefore strengthening the relation we’ve drawn between representation, trust region, and performance around collapse.

Ultimately, this improvement can result in best performance improvement as well but less consistently as it requires running for long enough to observe a collapse with the standard (tuned) hyperprameters. This is the case for NameThisGame in Figures 18 and 22, Humanoid in Figures 19 and 25, and Hopper in Figures 20 and 26, where PFO performs better than the baseline PPO. This is not a central claim of the paper, as it requires a larger computational budget to show for all environments.
To strengthen these points, we have added Figure 34 to the rebuttal PDF where we show that on ALE/Phoenix with the tuned standard hyperparameters, the agent collapses when training for longer and this is mitigated with PFO, which attains the best performance in that case.

Q3. The epsiode return curves … for MuJoCo Hopper look strange. Is there any explanation to the collapse? A regular implementation of PPO should work in Hopper when the epoch num is 10.

A3. We use the same hyperparameters as the original PPO implementation, which are also the default ones in popular codebases (as noted in lines 140-143).
Our setting differs from a “regular” implementation in the following main points, which can help explain why a collapse is not typically observed in previous work.

1. We train for longer and collapse always happens after the default 1M steps. Dohare et al. (2023) also observe collapse when training for longer.
2. We do not decay the learning rate (line 154) we are interested in agents that can ideally train indefinitely.
3. we parameterize the action space by a TanhNormal distribution with both state-dependent mean and variance following the implementation of Haarnoja et al. (2018). (lines 614-616).

Q4. … 4 MuJoCo tasks and 5 Atari games in the Experimental Setup ... However … only 3 Atari games … and 2 MuJoCo tasks … used in Section 4. Are there more results on the remaining tasks/games?

A4. Indeed, as noted in lines 305-306, while we used 4 MuJoCo tasks and 5 Atari games to demonstrate the collapse phenomenon with enough evidence, we did not use all of the environments for the evaluation. We selected the environments where the collapse was the most consistent to test multiple interventions while maintaining a reasonable compute budget (this is common practice like in Gulcehre et al. 2023 and Kumar et al. 2021). We believe this to be a right tradeoff between claiming that interventions are necessary to prevent collapse and providing enough insights about several types of interventions.

We sincerely thank the AC for their commitment to a thorough review process and for including us in the discussion of important concerns that were raised during the private discussions between the reviewers that we would not have otherwise had an opportunity to respond to. This clarifies the recent comments we have received and allows us to discuss them constructively. We hope to address them during the time left for the discussion phase.

several papers that already refute (1): it has been overlooked in on-policy policy optimization methods which are often thought capable of training indefinitely… and establish (2) revealing that PPO agents are also affected by feature rank deterioration and loss of plasticity.

We respectfully disagree. These works do not refute (1) nor establish (2). This is a misleading claim.

First, the “it” in (1) refers to “networks … exhibit a decrease in the rank of their representations, a decrease in their ability to regress to arbitrary targets, called plasticity” (line 5 abstract, and lines 25-56 introduction.) and “raises the question of how much PPO agents are impacted by the same representation degradation attributed to non-stationarity.” (lines 34-36 introduction).

We agree that the “thought capable of training indefinitely” bit is controversial and that it is common to believe the opposite. We discussed this with reviewer YzGo and mentioned that it was not a critical finding of the paper and that we would remove it.

However, none of the works mentioned discusses the representation degradation of PPO, i.e., the “overlooked” bit of (1) and the main point of (2).

Dohare et al. (2021) have a single RL experiment with an environment with non-stationary transition dynamics and only track online performance. There is no curve about representation degradation (representation rank, dead units, feature norm, etc.), and the “plasticity loss” refers to the degradation of the online performance of this environment with non-stationary dynamics. This confuses assessing the ability to learn a sequence of optimal policies with the evolution of the learning network to fit the same arbitrary function, which we measure with the random label fitting capability.

Dohare et al. (2023a) track dead units and effective rank but only in the supervised learning experiments, confirming our point that representation degradation is overlooked in online RL. They also perform the same experiment as Dohare et al. (2021), using a non-stationary environment to measure only online performance.

Dohare et al. (2023b) have experimented in stationary online RL with PPO as we do, but again, they only track online performance in the complex tasks (MuJoCo), with no figures of representation degradation and no figures of plasticity loss. With all due respect, all the claims made in the paper about plasticity appear to be speculations. Representation is plotted once for a toy example, but it is one-dimensional, so already collapsed and does not show an evolution from rich to poor.

These works are important in developing a better understanding of training deep neural networks under non-stationarity; however, they do not specialize in the specific topic of representation dynamics we discuss in our work. Importantly, it is specifically tracking representations that allows us to draw a connection between representation and trust region degradation.

Mujoco tasks in the latter paper, which are highly relevant ... natural to build on these results. However, these existing results are not acknowledged. It is important because the acknowledgment would have allowed the reviewers to compare your results.

To the best of our knowledge, Dohare et al. 2023b do not provide code with their paper, and their setting is not fully described. Several implementation details are missing, e.g., whether they decay the learning rate, the value of the Adam $\beta$ hyperparameters used for the plots (unlike for L2, which is well described), and the action distribution used. Furthermore, some of the design choices in their setting are unusual, e.g., a 4x larger minibatch size effectively dividing the number of gradient updates per batch by 4 (less overfitting) and no gradient norm clipping. For these reasons and because we had a more modern setting in mind with an action space fully dependent on the representations, basing our implementation on a replication of theirs did not seem like a natural choice to us.

It is also not clear if highlighting their specific results on MuJoCo or basing our implementation on theirs would have provided a stronger reference to compare our results. To the best of our knowledge, the paper has not been replicated, the code is not available, and the runs have not been thoroughly reviewed. In fact, reviews of the paper in EWRL16 state “reviewer points out some issues with the empirical results”.