Parseval Regularization for Continual Reinforcement Learning

We would like to thank the reviewer for their comprehensive assessment, also including positive aspects.

For the concern about computational complexity, we point the reviewer to the shared rebuttal statement, where we discuss this in detail. We address the other points individually below:

• “How are the parameters for Parseval regularization chosen and tuned? Is there a standard procedure or does it require extensive experimentation?”

To tune the regularization coefficient, we do a coarse sweep of 4 values: $10^{-i}$ for $i \in (2,3,4,5)$, (as outlined in Appendix C.5). We find that Parseval regularization is fairly robust to the strength of the regularization coefficient and a setting of $10^{-3}$ or $10^{-4}$ works best for the environments we used.

• “While the empirical results are strong, the theoretical analysis might be lacking in depth. A more rigorous exploration of the underlying mechanisms and theoretical guarantees could strengthen the paper’s contributions.”

We agree theoretical results would be of great interest. Unfortunately, given the current state of deep learning theory it is difficult to give rigorous proofs of such optimization benefits for practical neural networks. One potential direction would be to consider the simplified setting of deep linear networks, where orthogonal initialization has been shown to scale better with respect to depth in a supervised learning setting on a single task [1]. Orthogonality of weights is also preserved by following the gradient flow in linear neural networks. Since we are considering the continual RL setting with nonlinear networks, this would add substantial complications to the problem setting and would likely require many new insights to develop the theory, making it more suitable as a future work.

[1] “Exact solutions to the nonlinear dynamics of learning in deep linear neural networks” Saxe et al.

• “...It is unclear how well the method scales to more complex, high-dimensional tasks or real-world applications.”

We agree it would be interesting for future work to consider more challenging tasks and real-world applications. In this paper, we have demonstrated the utility of Parseval regularization in a variety of environments using sequences of standard benchmark tasks. These results provide evidence for its effectiveness in a broader range of settings and the simplicity of the approach makes it easy to incorporate into RL agents. Finally, since computational costs for running continual RL experiments are already fairly high due to having to train the agent on multiple tasks in a sequence, it is overly costly to run experiments where learning a single task may take a long time.

• “Although the paper compares Parseval regularization with a few alternative algorithms, it might benefit from a broader range of baseline comparisons. Including more state-of-the-art methods could provide a more comprehensive evaluation of its effectiveness."

We compare to many baselines from the plasticity loss literature covering all the major kinds of algorithms including Shrink-and-Perturb (injecting randomness and weight decay), layer norm (normalization to improve curvature of loss surface), concatenated ReLU (dealing with dead units) and regenerative regularization (regularization towards the initialization). We are open to considering other algorithms we may have missed.

• ”How do the different components of Parseval regularization (regularizing row norms vs. angles between weight vectors) interact with each other, and is there an optimal balance between these components?”

Parseval regularization encourages parameter matrices to be orthogonal, which improves the gradient propagation properties (avoiding vanishing and exploding gradients). As such, from a theoretical standpoint, it would make sense to keep the two parts (regularizing row norms and angles) as a single objective. From our ablation studies (see Fig. 6), we saw that both components contribute to the success of Parseval regularization.

• ”How does Parseval regularization perform across different network architectures and activation functions? Are there specific types of networks where it is particularly beneficial or less effective?”

In Fig. 5, we can see that Parseval regularization works well with a variety of activation functions although it seems to make the largest difference for the tanh function (which also performs best overall).

• ”How does Parseval regularization specifically influence policy entropy and optimization dynamics? Are there scenarios where it might lead to suboptimal performance due to these influences?”

In Appendix A.3, we reported results on the entropy of the policy during training for Parseval regularization and other methods. We saw that the entropy tended to be higher for Parseval regularization. To investigate this further, we experimented with adding different levels of entropy regularization but did not find a systematic link between the entropy regularization strength and the performance. Based on this, we hypothesize that higher entropy may be a byproduct of better performance but entropy does not have a causal link to performance.

We would like to thank the reviewer for the specific and concise points.

For the first point concerning the computational complexity, we have discussed this in detail in the shared rebuttal statement and would direct the reviewer’s attention there.

Concerning the memory requirement, Parseval regularization only requires at most $d^2$ additional memory coming from the computation of the Frobenius norm of the product of parameter matrices, where $d$ is the width of a dense layer. Note that we compute this norm sequentially per-layer so we only require a single matrix of memory overall. In our experiments, since $d=64$, this amounts to $64^2=4096$ additional float32 numbers, equivalent to approximately 16 kb of memory, a negligible amount.

Concerning the references:

• “Superposition of many models into one” by Cheung et al.

This superposition technique is interesting although it addresses a different aspect of continual learning: catastrophic forgetting. In contrast, our paper focuses on the issues of plasticity and improving learning on new tasks. Additionally, the superposition technique requires knowledge of the task identity to change the "context" of the superposition whereas we are working in the task-agnostic setting, where the agent does not receive a signal when tasks change or a label for the current task. It must learn new tasks as they come using only the base observations and rewards. As such, we could not directly apply the superposition algorithm as it is unclear how to extend the method to the task-agnostic setting, although this could be a fruitful research direction. As a side note, it is interesting to see that the superposition technique also makes use of orthogonal matrices albeit in a completely different manner than Parseval regularization.

• “Memory-efficient Reinforcement Learning with Value-based Knowledge Consolidation” by Lan et al.

This paper also focuses on the problem of catastrophic forgetting rather than plasticity in continual learning. Inspecting the proposed algorithm (MeDQN) closer, it is tailored to DQN or could be adapted to other algorithms which utilize a target network and off-policy updates. It's main goal is to eliminate the need for maintaining a large replay buffer through a sampling technique and an auxiliary loss to matching the target network. Since we used PPO in our experiments, which does not have a target network and is on-policy, it would not be possible to apply their algorithm. Exploring the interaction between Parseval regularization and other algorithms such as MeDQN would be an interesting avenue for future work.

Thank you for pointing us to the references, we will mention these works in the paper.

We would like to thank the reviewer for the clear feedback and for recognizing the qualities of the paper.

As the main concern was centered around the computational complexity of Parseval regularization, we would like to point the reviewer to the shared rebuttal statement where we have discussed this in-depth. We hope this response has sufficiently addressed this issue and we are happy to answer any additional questions.

To clarify, the task in Fig.5 consists of the MetaWorld suite of environments described in the paragraph starting on line 181. We will fix the caption.

We will also modify the title as suggested and agree it is more suitable.

Thank you for providing the computational complexity and runtime of parseval regularization. I did not consider the effect of mini-batches on computational complexity. I suggest that you add a discussion of computational complexity to the main paper and provide the table of runtime of different algorithms in the appendix. Because the computational complexity and runtime are not too large, my main concern about the paper has been addressed. I'm raising my score to reflect that.

The other reviews raise concerns about comparison with other methods and point out some other limitations of this work. A comparison with the methods suggested by other reviewers is not critical as those methods address catastrophic forgetting, not plasticity loss. Additionally, although the other reviewer provides exciting directions for future work, they are not necessary for the main claims made in this paper. I believe that this paper, in its current form, is a good contribution to the community, and it should be accepted.