The Ladder in Chaos: Improving Policy Learning by Harnessing the Parameter Evolving Path in A Low-dimensional Space

Q1: The learning curves of TD3 and TD3-PPTB

We provide the missing learning curves in Figure 9 (for RAD and RAD-PPTB) and Figure 10 (for TD3 and TD3-PPTB) of the one-page pdf uploaded.

We hypothesize that the higher convergence performance in Ant and Hopper can be explained by PPTB’s effect in preventing the plasticity loss in the minor temporal SVD directions to some degree.

In fact, the policy path trimming (PPT) method proposed in our paper can be viewed as a special way to do network resetting to alleviate the primacy bias as in [Nikishin et al., 2022].

Concretely, our method differs from the vanilla network resetting method in [Nikishin et al., 2022] on two points: (1) we do reset in a transformed subspace (with temporal SVD) rather than the original space; (2) we only reset the parameter update in the minor directions and maintain the learned information in the major directions.

From the perspective of plasticity, our method periodically rolls back the plasticity of the network in the directions that are orthogonal to the ones that represent the effective knowledge learned so far.

Q2: “The author only provided results for 100k and 500k steps in the RAD results shown in Table 2. What about the results for 1000k or 2000k steps?”

We follow the evaluation setting as that in the original paper of RAD (see Table 1 in the original paper).

Q3: Some of the reported improvements in AUC appear to be incorrect. For instance, (189 ± 2 (103.33%)) and (148 ± 15 (81.48%)) seem to contain errors.

As described in Line 277-279, for each task, we report the comparative improvement by normalizing with a random-agent baseline as 0 and the DRL base algorithm (i.e., TD3 or RAD) as 1.

This resembles the convention used for Atari tasks, where the random agent is taken as 0 and the human performance is taken as 1.

Q4: “in the experiment details, a 2000 or 1e5-dimensional neural network is too large for Mujoco training. Most papers indicate that commonly used model dimensions are 64 or 128”

For the dimension of each hidden layer, we follow the convention and use 256 for TD3 and RAD. Thus 1e5 is the scale of the total number of network parameters (and 2000 is the window size of the historical policies maintained), not the dimension of each hidden layer.

Q5: Can you talk about how to determine the P_b in your experiment? If is not set properly, will it cause excessive oscillation during the training process?

In our experiments, we found that choosing different $P_b$ may have a difference in the improvement achieved upon the baseline algorithm, but it is quite safe to choose among a relatively large range without incurring a clear performance drop. Choosing a large $P_b$ should be safer as it makes less impact on the parameter update in principle according to the temporal SVD.

This is also supported by our empirical results for the SVD parameter reconstruction shown in Table 3 to Table 8 in the appendix.

Dear Reviewer,

We hope that you've had a chance to read our responses and clarification. As the end of the discussion period is approaching, we would greatly appreciate it if you could confirm that our updates have addressed your concerns.

Q1: Why TD3 and RAD are selected?

The motivation for this work starts from the investigation of the dynamics of policy network parameters. Thus, we choose TD3 and RAD for the following reasons:

• They are popular deep AC methods (note RAD is SAC-based), where an explicit policy network is trained. This ruled out the value-based RL methods.
• TD3 and SAC are off-policy methods. They are usually more sample-efficient while less stable than on-policy algorithms like PPO.
• RAD takes visual observations as input. Thus, our choices cover both proprioceptive inputs and image-based inputs.

Q2: Wouldn't other algorithms have similar phenomena when updating strategies?

As suggested by Reviewer zpVk, in Figure 11 of the one-page pdf, we can observe similar phenomena for Behavior Cloning with Adam or SGD optimizers, but with different concrete patterns.

Besides, we observed similar phenomena for DDPG (omitted due to space limitation).

Q3: Other suggestions on the experiment

We appreciate the reviewer’s valuable suggestions on the experiment.

We will present the results for this if we are able to finish this before the discussion stage ends.

Dear Reviewer,

We hope that you've had a chance to read our responses and clarification. As the end of the discussion period is approaching, we would greatly appreciate it if you could confirm that our updates have addressed your concerns.

Thank you for your response. I've read the other reviews and the rebuttal. I’m keeping my initial score.

Q1: The additional investigation for Behavior Cloning policy

We appreciate the reviewer for pointing out this insightful and inspiring point.

We provide additional results in Figure 11 of the one-page pdf to present additional investigation for Behavior Cloning in D4RL halfcheetah-medium-replay with Adam and SGD optimizers.

As noted by the reviewer, we found that BC (Adam), BC (SGD) and the RL cases reported in our paper show different patterns in terms of temporal SVD evolvement. The key observation is that BC policies do not turn out to show the harmonics in the major temporal SVD directions (the leftmost plot in Figure 11b and 11d). In contrast, we can observe that the amplitude of the oscillations seems to decrease throughout training.

We deem that this observation is very interesting which could lead to a deeper understanding of the learning dynamics of BC and RL agents.

For the parameter update analysis, compared with the RL cases reported in our paper, the parameter update amount is more Gaussian distribution like for BC (Adam). This indicates the difference in policy parameter update dynamics between BC (i.e., supervised learning) and RL.

Somewhat surprisingly, the asymmetry of parameter update amount turns out to be more severe in BC (SGD). This is a bit counterintuitive because Adam exhibits less asymmetry than SGD.

We are more than willing to discuss this point with the reviewer in the discussion stage.

Q2: The analysis for DDPG

We appreciate the reviewer for pointing out this.

Actually, we also did the parameter update analysis and the temporal SVD analysis for DDPG. We found the empirical results are quite similar and the patterns in parameter update and temporal SVD directions are almost the same as those of TD3. Therefore, the plots are omitted and not included in the one-page pdf.

This indicates that the phenomena revealed in this work are not closely related to the overestimation bias things (which is the focus of clipped double Q learning in TD3, SAC, RAD).

Q3: Lack of training curves

We provide the missing learning curves in Figure 9 (for RAD and RAD-PPTB) and Figure 10 (for TD3 and TD3-PPTB) of the one-page pdf uploaded.

Q4: “Line 434: you say that you choose the boosting coefficient $𝑝_𝑏$ from a set of values. Do you mean that you search from this set for a good hyperpameter value, or do you randomly sample from this set at each training iteration?”

The complete process is that we first pre-designate a set of candidates of $p_b$ according to our empirical results for the temporal SVD parameter reconstruction (as shown by Table 3-8 in the appendix). Then we narrow the set and search from this set for a good hyperparameter.

Moreover, In our experiments, we found that choosing different $P_b$ may have a difference in the improvement achieved upon the baseline algorithm, but it is quite safe to choose among a relatively large range without incurring a clear performance drop. Choosing a large $P_b$ should be safer as it makes less impact on the parameter update in principle according to the temporal SVD. This is also supported by our empirical results for the SVD parameter reconstruction.

Q5: The related work in deep learning literature

We appreciate the reviewer for pointing out the reference paper. We have added it to our draft and will add more in our related work section to strengthen the discussion.

Q6: Other suggestions on the experiment

We appreciate the reviewer’s valuable suggestions on the experiment.

We will present the results for this if we are able to finish this before the discussion stage ends.

Thank you for the updated results and rebuttal!

We provide the missing learning curves in Figure 9 (for RAD and RAD-PPTB) and Figure 10 (for TD3 and TD3-PPTB) of the one-page pdf uploaded.

The PPTB versions certainly look better. I encourage you to add them to the appendix in a future version of the paper.

As noted by the reviewer, we found that BC (Adam), BC (SGD) and the RL cases reported in our paper show different patterns in terms of temporal SVD evolvement. The key observation is that BC policies do not turn out to show the harmonics in the major temporal SVD directions (the leftmost plot in Figure 11b and 11d). In contrast, we can observe that the amplitude of the oscillations seems to decrease throughout training.

(^ emphasis mine)

Upon looking at the newly provided plots (Fig 11b + 11d in the 1-page PDF), I don't see how one can infer that. Here's what I see:

1. The BC-training dynamics do exhibit oscillations just like the RL-training dynamics in the main paper (e.g. Fig 7).
2. The oscillation amplitudes for the major components do decrease over BC training, just as they also decrease during RL training.

Unless I'm very mistaken in my interpretation (happy to be proven wrong!) your experiments reveal that the oscillatory patterns in the top SVD directions are not specific to RL, but occur in vanilla supervised deep learning too (irrespective of Adam or plain SGD). And these experiments should go into the paper. This also suggests that maybe PPTB need not be an RL-specific heuristic but could also help supervised deep learning (but that will require more experiments, so the paper raises more questions than it answers...)

I think that these findings necessitate substantial edits to the paper. E.g.

• Emphasising early on that the empirical observations are a broader deep learning phenomenon, not RL-specific
• Adjusting/removing claims that make it sound like the oscillations stem from RL pathologies (e.g. noisy policy gradients)
• Linking the paper more strongly with related work on low-dim subspace training

For this key reason, I'm maintaining my score at the current level.

I still believe your approach is interesting and original, and I'd encourage you to try and publish a future version with an updated story!

A side comment:

If you look closely at Fig 7 column 2 in the paper (the major-component u[k] curves for all 4 MuJoCo envs) you'll find that the oscillatory curves are nearly identical for Hopper-v4, Walker2d-v4, and Ant-v4 but not HalfCheetah-v4. I don't have any intuition for this, but it seems surprising. As this is an empirical investigation, the authors should think about what that means, how/why that might happen, and discuss that in the paper.

We greatly appreciate the time and effort devoted by the reviewer to reviewing our work and participating in the discussion.

The comments and discussions provided by the reviewer are really valuable and inspiring, and we believe these comments are leading our work to a higher quality and level.

We will carefully take the suggestions and improve our organization and presentation to include the additional results done in the rebuttal and discussion stages, i.e., the parameter and temporal SVD analysis for DDPG, BC (Adam and SGD), as well as the additional evaluation of PPTB for DoubleDQN in MinAtar (as suggested by Reviewer dZpR).

Concretely, we plan to start by presenting the learning process of policy networks from a relatively general DL angle to cover aspects like optimizers (e.g., Adam, SGD) and learning paradigms (i.e., RL, SL). Then we plan to discuss these aspects separately by presenting the empirical observations regarding the network parameter analysis and the temporal SVD analysis, with a focus on online RL policy learning as presented in our draft. Afterward, we discuss the differences in the empirical observations in different settings, by establishing the connection between the distinct features of different learning settings and the difference in the patterns of the distribution of network parameter update amount or the oscillations of temporal SVD directions.

Here are some results and conclusions we’ve already observed:

1. by comparing the results of TD3 and DDPG (as mentioned in Q2 of our rebuttal), we can find that the overestimation bias (i.e., the major difference between TD3 and DDPG) does not seem to be a factor that causes the oscillations.
2. by comparing TD3 and BC (Adam), we can find that RL and SL lead different distributions of network parameter update amount (as Figure 4a v.s., Figure 11a) and the “harmonic v.s., wavelet” oscillations (as Figure 7a v.s., Figure 11b).
3. by comparing BC (Adam) and BC (SGD) as shown in Figure 11 a,b v.s., Figure c,d, we can find that the momentum mechanism used in Adam leads to a relatively more evenly distributed network parameter update amount, while SGD has a large number of minor updates (also reflected by the flat $u[1]$ curve in Figure 11d, the first column) and a long tail in distribution.

These observations are the empirical support for our previous response “the pattern of the oscillations is different in different learning problems”. And with the second observation outlined above, we might not be able to fully agree with “However, the RL and SL training dynamics, empirically, did not show distinguishably different oscillatory patterns” in the comment of the reviewer. We are doing more BC results to investigate this point further.

Q1: the evaluation of the algorithms. For example, reporting the max return in the evaluation of algorithm or using the standard deviation across runs, whereas the standard error or boostrapped confidence intervals would be more appropriate

Technically, we are not reporting the “max return” in our evaluation. We report the “maximum of the mean return”. To be concrete, this is obtained by first (1) computing the curve of mean return evaluation across multiple runs, and then (2) taking the maximum of the mean evaluation curve.

We follow the evaluation scheme presented in TD3 paper (see “Max Average Return over 10 trials” in the caption of Table 1 in TD3 paper).

The rationality of this evaluation scheme is that, for online learning, it is possible to store checkpoints throughout learning. For the opposite case, in offline RL, the convention is to report the mean final score.

In Line 279, we wrote, “We report the means and standard deviation errors across six independent trials”. We meant to say “standard errors” rather than “standard deviation errors”. This is a mistake in writing. We have amended it in our revision.

We appreciate the reviewer for the suggestion of using other more reliable evaluation metrics. We will take the suggestion in our revision.

Q2: “I think expanding a bit more on the analysis could be interesting”

As suggested by Reviewer zpVk, we provide additional results in Figure 11 of the one-page pdf to present additional investigation for Behavior Cloning in D4RL halfcheetah-medium-replay with Adam and SGD optimizers.

Compared with the RL cases reported in our paper, the parameter update amount is more Gaussian-disribution like for BC (Adam). This indicates the difference in policy parameter update dynamics between BC (i.e., supervised learning) and RL. Somewhat surprisingly, the asymmetry of parameter update amount turns out to be more severe in BC (SGD). This is a bit counterintuitive because Adam exhibits less asymmetry than SGD.

Moreover, we found that BC (Adam), BC (SGD) and the RL cases reported in our paper show different patterns in terms of temporal SVD evolvement.

We believe that these results are interesting to potential audiences of this paper and worth a more in-depth discussion.

Q3: “By constraining the parameter evolution path to focus on the previous main directions, would the effect of primacy bias or related phenomena be even stronger? Could we be losing out on performance due to prematurely committing to certain update directions?”

In fact, the policy path trimming (PPT) method proposed in our paper can be viewed as a special way to do network resetting to alleviate the primacy bias as in [Nikishin et al., 2022].

Concretely, our method differs from the vanilla network resetting method in [Nikishin et al., 2022] on two points: (1) we do reset in a transformed subspace (with temporal SVD) rather than the original space; (2) we only reset the parameter update in the minor directions and maintain the learned information in the major directions.

From the perspective of plasticity, our method periodically rolls back the plasticity of the network in the directions that are orthogonal to the ones that represent the effective knowledge learned so far.

We appreciate the reviewer’s insightful comments and we believe that more study can be done in the future by taking into consideration both policy update subspace and the plasticity loss problem.

Q4: "Only upper 80% values according to $\Delta_{j}^{\text{apc}}$ are taken to plot...for meaningful analysis" Could you elaborate why this decision was made? Does it have to do with many of the values being close to zero?

This is because many parameters have a very minor value (i.e., very close to zero) of accumulated parameter change (as revealed by the parameter update asymmetry phenomenon).

We have clarified this in our draft to eliminate the confusion.

Q5: The clarification questions

• [Line 157: What is $\alpha_k$? Is it Singular Value Information Amount?] $\alpha$ is the Singular Value Information Amount for a dimensionality number $k$, which is defined in Line 155.

• [Fig 2 b) 4th and 5th figure from left. Interesting findings. So basically, the detour ratio is smaller for the Again, I wonder if that has more to do with noise or curvature. Perhaps there is less noise in these directions] Yes, the detour ratio is smaller for the major directions. We consider that it is closely related to the curvature of the landscape of policy objective function.

• [In the 3rd fig from the left, it's hard to see much since the black curve masks everything else. Consider using different colors or some transparency. It would be nice to see how the paths get increasingly noisy.] We appreciate the reviewer’s suggestion. We will replace the plot with the frequency information by Fourier transformation for a better presentation.

• [More description for Policy Path Trimming and Policy Path Boosting] The conversion from the original parameter space to the subspace is done by performing temporal SVD. The corresponding left singular vectors are taken as the new coordinates in the subspace. This is described between Line 145-152. We have taken the writing suggestions provided by the reviewer and added more detailed descriptions in Section 4.1 and 4.2 to eliminate the confusion.

Q6: Other suggestions

We sincerely appreciate the reviewer for providing insightful comments along with very useful reference papers.

Due to the time limit of the rebuttal stage, we may not be able to respond to each point. However, we are more than willing to discuss more in the discussion stage.

Dear Reviewer,

We hope that you've had a chance to read our responses and clarification. As the end of the discussion period is approaching, we would greatly appreciate it if you could confirm that our updates have addressed your concerns.

Q1: “Computing SVD is both time intensive and requires storing a wide range of previous parameters. It therefore requires a lot more compute, which is why such methods are typically not used”

We apologize for causing the reviewers to misunderstand the memory consumption and computational overhead of our method.

We would like to further clarify this point in the following:

[Computation cost] We would like to clarify that (1) each SVD operation in our experiments takes roughly 1 second by using torch.linalg.svd, and (2) the SVD operation is not performed at every time step but rather at sparse intervals (e.g., $t_p \ge 1000$). The practical computation cost in wall-clock time is less than 5% of the total training time, which is worth it compared to the benefits SVD brings.

Moreover, the temporal SVD operations remain scalable as matrix size increases, thanks to the use of efficient linear algebra libraries. This scalability has been validated by numerous successful cases, such as the widely adopted large model fine-tuning technique, LoRA (Low-Rank Adaptation) [1]. The effective application of LoRA clearly demonstrates that SVD is not a significant issue in terms of memory overhead and computational cost.

[Memory cost] As described in Appendix B, PPBT does not require storing all historical policies but instead stores policies at sparse intervals. In our implementation, the size of the stored policy parameter matrix is $[2000, \approx 1e6]$, making the memory overhead entirely acceptable.

We have clarified this point in our draft.

Q2: Can you comment more on the relationship between your work and other, less compute intensive methods, that aim to allow more uniform convergence among the different data eigenvalues such as momentum? Is there an explicit connection here? Have you thought about that?

In Q1, We provide the response to the misunderstanding on the “compute-intensive” point.

We are more than willing to discuss the relationship between our work and any concrete related work provided by the reviewer in the discussion stage.

Reference:

[1] LoRA: Low-Rank Adaptation of Large Language Models. 2021

Dear Reviewer,

We hope that you've had a chance to read our responses and clarification. As the end of the discussion period is approaching, we would greatly appreciate it if you could confirm that our updates have addressed your concerns.

Q1: “Do you think these results apply beyond just RL? I see no reason why this phenomenon is RL-specific. Doesn't the fact that these methods use momentum (e.g. Adam) make this phenomenon obviously true? Does this phenomenon persist when using plain SGD? What about when you observe the gradient updates as opposed to the parameter updates?”

We appreciate the reviewer for pointing out this. We agree that it will be interesting and valuable to extend our experimental investigation to more general deep learning settings.

To this end, we provide additional results in Figure 11 of the one-page pdf to present additional investigation for Behavior Cloning in D4RL halfcheetah-medium-replay with Adam and SGD optimizers.

Compared with the RL cases reported in our paper, the parameter update amount is more Gaussian-disribution like for BC (Adam). This indicates the difference in policy parameter update dynamics between BC (i.e., supervised learning) and RL. Somewhat surprisingly, the asymmetry of parameter update amount turns out to be more severe in BC (SGD). This is a bit counterintuitive because Adam exhibits less asymmetry than SGD.

Moreover, we found that BC (Adam), BC (SGD) and the RL cases reported in our paper show different patterns in terms of temporal SVD evolvement.

In this paper, we start from and focus more on online RL and we defer the study across different learning paradigms in the future.

Q2: “why do you believe this method does not exacerbate issues of primacy bias mentioned in the paper?”

In fact, the policy path trimming (PPT) method proposed in our paper can be viewed as a special way to do network resetting to alleviate the primacy bias as in [Nikishin et al., 2022].

Concretely, our method differs from the vanilla network resetting method in [Nikishin et al., 2022] on two points: (1) we do reset in a transformed subspace (with temporal SVD) rather than the original space; (2) we only reset the parameter update in the minor directions and maintain the learned information in the major directions.

From the perspective of plasticity, our method periodically rolls back the plasticity of the network in the directions that are orthogonal to the ones that represent the effective knowledge learned so far.

We appreciate the reviewer’s insightful comments and we believe that more study can be done in the future by taking into consideration both policy update subspace and the plasticity loss problem.

Q3: “Do you think this would work equally well in sparse(r) reward settings? My intuition is that it would not, since the early parameter updates likely do not contain significant information about the reward. The continuous control tasks evaluated are particularly dense. Also, doesn't pruning the noisier directions harm exploration?”

We appreciate the reviewer for pointing out this inspiring point. First, we consider that the asymmetry in terms of parameter update amount and the concentration of temporal SVD information should be more severe in sparse-reward settings, as the self-distillation mechanism can dominate when the reward signal contains little information [1].

Second, our method does not aim to address the exploration problem and thus it is less likely to improve the learning performance especially when the baseline algorithm fails with very sparse rewards.

However, we do not think that our method necessarily harms exploration, because the trimming happens at a very sparse interval and moreover the effect of the parameter update of the minor directions in exploration is unclear yet. One possible understanding is that the periodic trimming rolls back the plasticity (as discussed in Q3) and could encourage learning new behaviors with the parameters in these minor directions.

We are also running additional experiments in several sparse-reward DMC tasks. We will present the results for this if we are able to finish this before the discussion stage ends.

Reference:

[1] DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization. 2021

Dear Reviewer,

We hope that you've had a chance to read our responses and clarification. As the end of the discussion period is approaching, we would greatly appreciate it if you could confirm that our updates have addressed your concerns.

We appreciate the reviewer's further feedback!

"It is hard to tell which results are statistically significant (I would recommend using standard error!) and also plotting standard RL training curves with the appropriate error regions"

Sorry for missing the response to this point in our rebuttal. Actually, we responded to the point of "standard error" in the Q1 of our rebuttal to the Reviewer iaGj, (but we missed it here).

In Line 279, we wrote, “We report the means and standard deviation errors across six independent trials”. We meant to say “standard errors” rather than “standard deviation errors”. This is a mistake in writing. We have amended it in our revision.

In our one-page pdf material, we provided the learning curves with means and standard errors (Figure 9 and Figure 10), which should help to show a more complete comparison than the scores we reported in the tables in our draft.

Please let us know if there are any remaining questions or concerns that we can address to improve your assessment. We are willing to discuss more with the reviewer to improve our work further.

Thanks for the response! I had a quick question on your thoughts on my comments on statistical significance (the second point on improvements in the weaknesses). I don't see it in the response, but may have missed it. In general, the difference in performance seems extremely marginal, though I do understand that most RL benchmarks have long since saturated.

I do believe most of my concerns have been addressed, though would hope to get a quick answer to the above.