We thank you so much for your insightful review, feedback, and opportunity to improve our work!

PACE vs. Mechanic

Firstly, we address your feedback by conducting a more rigorous evaluation of Mechanic across our benchmarks. Implementing your suggestion has substantiated our claim that PACE performs slightly better empirically.

Results

We conducted additional experiments, running 10 seeds for Mechanic on Procgen and Atari environments and 25 seeds on Control environments. We show the average normalized reward over the entire lifelong run for each environment for PPO when using both PACE and Mechanic in Supplementary PDF Figure 2.

Following your suggestion, all results below are average normalized rewards (normalized scores between each environment, and for atari, normalized for each game; see more details on this in the Author Rebuttal).

Procgen Results for Mechanic vs PACE over 10 seeds

Control Results for Mechanic vs PACE over 25 seeds

Atari Results for Mechanic vs PACE over 10 seeds

Conclusion

While Mechanic effectively combats plasticity loss and performs similarly to PACE, it consistently underperforms PACE by a slight margin across all settings. This result may align with the theoretical expectation that our scaling tuner provides better OCO performance guarantees.

Insight

While Mechanic is also a parameter-free optimizer, it has only been evaluated in the context of stationary supervised DL. By demonstrating that Mechanic performs well in a variety of non-stationary RL settings, we strengthen our claim that parameter-free optimization deserves more attention in the context of lifelong RL. We believe these results further validate the insights derived from PACE:

• parameter-free OCO is all about better regularization and
• better regularization mitigates loss of plasticity.

We are happy to include these additional Mechanic results in our main experiments.

Weight Decay Tuning

We appreciate your question regarding the tuning of weight decay with Adam, as explored by Lyle et al. 2023 for mitigating plasticity loss.

• In our experiments, weight decay was set to 0.
• Following your suggestion, we conducted a HP sweep using PyTorch’s AdamW optimizer across three control environments with 15 seeds for each of the following weight decay values: 0.0001, 0.001, 0.01, 0.1, 1.0, 10.0, 15.0, 5.0, and 50.0.
• Our findings show varying weight decay values do offer benefits (Supplementary PDF Figures 4a-c), but do not outperform PACE with no weight decay.
• Like L2 regularization strengths, the best weight decay values are environment-specific and sensitive to distribution shifts (see Supplementary PDF Figure 4d), complicating the presetting of these values without a prior hyperparameter sweep or knowledge of expected distribution shifts.

Plasticity Loss Mitigation Baselines

In appreciation of your suggestion, we have included plasticity injection (following the simple methodology laid out in Nikishin et al.) as a baseline in our control environments.

• We inject plasticity at the start of every distribution shift.
• While plasticity injection does help mitigate plasticity loss, it is outperformed by PACE across the 3 control settings (Supplementary PDF Figure 3b).

Additionally, we included another plasticity loss mitigation technique, layer normalization, as suggested by Reviewer eCfr.

• While layer normalization improves the performance of baseline Adam, it is outperformed by PACE across the 3 control settings.
• However, we also find that layer normalization with PACE improves over PACE without layer normalization (Supplementary PDF Figure 3b).

Minor: Reward Normalizizaton

Thank you for pointing out that our results can be made more clear by normalizing the scores in Table 1 (we will follow the normalization procedure in the Author Rebuttal).

Please let us know if you have any other concerns/comments, and we'll be glad to address those!

Dear Reviewer,

Following up on our previous comment regarding the statistical significance of our results, we have conducted additional experiments on the Gym Control suite, which are more feasible, quicker, and efficient to run on CPUs than the other experiments. Specifically, we increased the sample size to 300 seeds for CartPole, Acrobot, and LunarLander. While our rebuttal initially reported the mean and standard deviations of the seed runs, we now provide a more stringent analysis, including the mean, standard error, and p-values, to make our claim as strong as possible. P-values were calculated using two-sample t-tests. For each control task. we tested the hypothesis that the means between Mechanic and PACE are the same ($\mu_{\text{PACE}} = \mu_{\text{Mechanic}}$)) (Null Hypothesis, H0) against the alternative that they are different (Alternative Hypothesis, H1). P-values here are helpful to quantify the probability of observing the data under the assumption that the null hypothesis is true.

Results:

These results support our argument that PACE outperforms Mechanic in the Gym Control tasks. The p-values here, significantly below the commonly accepted threshold of 0.05, strongly reject the null hypothesis in favor of the alternative. This rejection supports our claim about the efficacy of PACE over Mechanic. Additionally, the relatively low standard errors associated with the mean scores for each method further indicate a high level of precision in our results, reinforcing the robustness of our findings.

Even with these results, we reiterate that our primary contribution remains the demonstration that parameter-free OCO helps mitigate the loss of plasticity in lifelong RL.

Please let us know if this has changed your mind or if you have any other concerns/questions that were not addressed here.

Thank you for your very thorough review! Especially, we appreciate your valuable comments on the writing and organization of this paper. We'll incorporate those in the camera-ready revision.

Regarding your questions: we realized that we went a bit too fast when introducing the algorithm. So first of all we would like to clarify its main idea, as well as the role of each component.

Idea of PACE

The overall structure of PACE results from a decade of research on parameter-free OCO. The latter as a branch of optimization theory concerns proving regret bounds under convex losses, which has a similar spirit as our problem but very much idealized (e.g., there isn't any notion of plasticity etc).

• The central component of PACE is the direction-magnitude decomposition from (Cutkosky & Orabona, 2018), whose main idea is to apply a carefully designed 1D algorithm (parameter-free tuner) on top of a base optimizer to adjust how far the iterates drift away from the initialization. With the right tuner (Alg 2), this essentially acts as a data-dependent regularizer. It helps mitigating plasticity loss due to the high plasticity associated with the initialization (around the origin; see the insights of Abbas et al, 2023).
• The tuner we use is based on the erfi function, due to (Zhang et al, 2024b). The design of this tuner is a long story (solving a PDE called the Backward Heat Equation), but intuitively, the idea is that the shape of the erfi function trades off the distance of the iterates from the origin and from the empirical optimum. Such a tradeoff is mathematically optimal in a quite strong sense. The tuner takes the inner product $\langle g_t,\theta_t-\theta_{\mathrm{ref}} \rangle$ as input, which is essentially the projection of the gradient along the update direction (since tuner controls the magnitude rather than direction, it only cares about the gradient component on the direction of $\theta_t-\theta_{\mathrm{ref}}$).
• As in much of practical model training, the tuner requires discounting. To automatically select the discount factor, we use the additive aggregation technique from (Cutkosky, 2019). The argument is very simple: if we sum up two parameter-free OCO algorithms, then the resulting algorithm performs almost as well as the best one among the two. That is, (at least in the convex setting) we can select the optimal discount factor for free.

We note that these three components crucially work together to guarantee good regret bounds in the convex setting. Since this is often the minimum requirement for any reasonable optimizer, we kept all three components in the nonconvex setting as well (otherwise the algorithm wouldn't make sense). This (ruling out design altenatives) is in our opinion a big benefit of theory-driven research.

Minor issues

• On the convexity. We essentially mean that although parameter-free OCO builds on the structure of convexity, it's intriguing to see that it works well on lifelong RL, a nonconvex and nonstationary task. We'll make this clearer.
• Stability. Here we mean the iterates shouldn't drift too far. This is the same reason for the importance of weight decay in DL, which is somewhat widely accepted by the community.

Questions

• A round refers to an iteration.
• We agree with you that the concept of discounted regret isn't so relevant to our exposition, so we'll remove it.
• Loss rescaling means on Line 5 of Alg 2 (the tuner), we multiply $\beta$ during the update of $v_t$ and $\sigma_t$. The theory behind this is justified in (Zhang et al, 2024a).
• Distribution shift in Figure 7. In the control environments, a distribution shift refers to altering the observation's dimensions by adding or subtracting random noise for a duration of 200 steps. This noise is changed every 200 steps. On the x-axis of Figure 7, each distribution shift number corresponds to a different 200-step interval. For example, Distribution Shift 1 covers steps 0-200, Shift 2 covers 200-400, and so forth.
• Cumulative reward sum. The discounted regret is a suitable metric for the theoretical analysis in OCO, but in reality (experiments and real world), we only care about cumulative reward sum (as in almost all existing works). Somewhat inevitably, this is one of the common aspects where theory and practice differ.
• Uncertainty set $\mathcal{U}$. Intuitively (in the context of lifelong RL) this could be thought as the loss landscape: since we only have gradient feedback, the loss landscape remains uncertain for our algorithm throughout its operation.

Since parameter-free OCO is itself a quite sophisticated field, our exposition focused mostly on the high level main idea. Please let us know if you have any other concerns / comments, and we'll be glad to address those!

We thank you for your continued engagement with our work!

Ablation study

Regarding your concern about the ablation study, as previously noted in our rebuttal, the components of our parameter-free algorithm are designed to function integrally, which is foundational to maintaining its parameter-free nature.

However, we appreciate your suggestion that exploring the individual impact of each component could highlight their respective contributions.

We conducted an ablation study to quantify the impact of removing specific components, expressed as average normalized rewards over 25 seeds:

We ablate the erfi tuner by replacing it with a baseline 1D online gradient descent tuner, where the parameter is $s_t$. Additionally, we ablate the additive aggregation component by employing a single tuner with $\beta$=1.

The ablation study demonstrates that the erfi tuner plays the most critical role in performance. Moreover, removing discounting diminishes the benefits of additive aggregation (which automatically selects the best discounting), though some advantages are retained in performance.

---

A simple revision for the clarity of writing concern

We are pleased that our detailed explanation in the rebuttal has addressed your inquiries about how PACE works. We recognize that your concern is regarding the clarity of the writing in the method section. To address this concern, we revise a part of Section 3. Here is a modified excerpt we will include in the camera-ready text following your suggestions:

PACE for Lifelong RL: In lifelong RL, a key issue is the excessive drifting of weights $\theta_t$, which can detrimentally affect adapting to new tasks. To address this, PACE enforces proximity to a well-chosen reference point $\theta_{\mathrm{ref}}$, providing a principled solution derived from a decade of research in parameter-free Online Convex Optimization (OCO). Unlike traditional methods such as $L_2$ regularization or resetting, PACE avoids hyperparameter tuning, utilizing the unique properties of OCO to maintain weight stability and manage drift effectively.

The core of PACE, similar to other parameter-free optimizers, incorporates three techniques:

1. Direction-Magnitude Decomposition: Inspired by Cutkosky & Orabona (2018), this technique employs a carefully designed one-dimensional algorithm, the "parameter-free tuner," atop a base optimizer. This setup acts as a data-dependent regularizer, controlling the extent to which the iterates deviate from their initialization, thereby minimizing loss of plasticity, which is crucial given the high plasticity at the origin (Abbas et al., 2023).
2. Erfi Potential Function: Building on the previous concept, the tuner utilizes the Erfi potential function, as developed by Zhang et al. (2024b). This function is crafted to effectively balance the distance of the iterates from both the origin and the empirical optimum. It manages the update magnitude by focusing on the gradient projection along the direction $\theta_t - \theta_{\mathrm{ref}}$.
3. Additive Aggregation: The tuner above necessitates discounting. Thus, we employ Additive Aggregation by Cutkosky (2019). This approach enables the combination of multiple parameter-free OCO algorithms, each with different discount factors, to approximate the performance of the best-performing algorithm. Importantly, it facilitates the automatic selection of the optimal discount factor during training.

These three components crucially work together to guarantee good regret bounds in the convex setting and are the minimum requirement for any reasonable parameter-free optimizer.

These revisions follow your suggestions on introducing the concepts utilized in the algorithm and explaining the interactions among the 3 components.

We are happy to include the ablation study and writing revisions in the camera-ready version. Do you have any other concerns or suggestions that were not addressed here?

Thank you for your insightful review and constructive feedback!

Lifelong RL baselines

Firstly, we acknowledge your comment on the lack of baselines. We recognize the important contributions of Ben-Iwhiwhu et al., 2023, Rolnick et al., 2019, Schwarz et al., 2018, and others in the field of lifelong RL. However, we didn't include these baselines since we consider our setting and scope to be a bit different (i.e., no task recycling, shared curricula, or privileged task boundaries; more on this soon). However, following your suggestion, we incorporated these baselines and found that the results (detailed below) support our hypothesis.

Results

Using open-source code from Ben-Iwhiwhu et al., 2023, we evaluate Adam and PACE across Impala, Online EWC, CLEAR, and Modulating Masks in our Procgen games. We modify the continual RL setup from Ben-Iwhiwhu et al. (2023) and Powers et al. (2022) to our lifelong RL setting (modifications are detailed in the Author Rebuttal).

We present a comparison of average normalized rewards over 5 seeds for each method and game in Supplementary PDF Figures 1b-e and plot the average reward curves, excluding STD fill for clarity, in Supplementary PDF Figure 1a for Starpilot.

In the Procgen environments, PACE consistently outperforms Adam across various learning algorithms:

• IMPALA by an average of 21.83%
• Online EWC by an average of 15.86%
• CLEAR by an average of 14.41%
• Modulating Masks by an average of 10.14%.

In Supplementary PDF Figure 1a, we also observe when using Adam, these methods are still vulnerable to loss of plasticity in later levels (especially Online EWC).

We are happy to include these results in our work and revise the introduction to acknowledge the history of these methods in the context of lifelong RL.

Clarifying our Lifelong Setting

• Our setting does not revisit old tasks nor assure any task similarity.
• We operate in a strictly online setup, prioritizing the evaluation of rapid adaptation to new tasks without task boundaries (which mask-based and EWC-based continual RL approaches require).
• In our setting, we demonstrate that plasticity loss occurs when adapting to new tasks never seen before.

In contrast,

• Ben-Iwhiwhu et al. (2023) focus on task recycling to balance learning new tasks with performance on previously seen ones. We consider this as a continual RL setting; indeed, Ben-Iwhiwhu et al., 2023 use "CORA: A Continual Reinforcement Learning Benchmark" (Powers et al., 2022) for Procgen experiments, where previously seen tasks are evaluated (backward transfer) during training.
• In this same setting, Abbas et al. (2023) observe loss of plasticity when tasks are revisited.

Clarifying our scope and hypothesis

We would like to clarify our work specifically addresses the loss of plasticity problem in lifelong RL through the lens of optimizers, such as Adam.

Hypothesis: Parameter-free optimization mitigates loss of plasticity more effectively than Adam, regardless of the learning algorithm or architecture employed (e.g., PPO, PPO with CReLU).

Conclusion: We introduce a new parameter-free optimizer, PACE, and show it better preserves plasticity compared to Adam across 4 Procgen, 2 Atari, and 3 Gym Control lifelong settings.

We would like to note that PACE is orthogonal to learning or policy algorithms and architectures (eg. PPO, CLEAR, Modulating Masks). As seen in the results above, PACE can be integrated with other RL algorithms simply by replacing the optimizer (like Adam or RMSPROP) that they use to train with.

Evaluation Choice: We opted to assess many environments with different input dimensions and distribution shifts, rather than many algorithms, to evaluate different modes of plasticity loss and isolate the effects of Adam and PACE. However, we have addressed your feedback on incorporating more baselines in Supplementary PDF Figure 1 and above.

Comparison to parameter-free OCO

We agree that our explanation of parameter-free OCO is somewhat abstract, given the space constraints. We will enhance more clarity in the camera-ready version!

On the (un-)necessity of an ablation study: Parameter-free OCO has evolved over the last decade into a standard framework, primarily based on extending 1D algorithms to higher dimensions (Cutkosky & Orabona (2018)) and a data-dependent method for selecting the discount factor (Cutkosky (2019)). These contributions are critical for ensuring optimal performance in convex settings—considered the benchmark for optimizers—and are equally vital in nonconvex applications. Omitting these would undermine the algorithm’s effectiveness and theoretical motivations.

Our novelty. When moving beyond convexity, existing works on parameter-free OCO consider training DL models. Intuitively, this doesn't fully utilize the capability of such algorithms: by design, OCO algorithms are able to handle even adversarially varying nature, whereas DL training is "stationary." Our work goes beyond this limitation by considering lifelong RL as an intrinsically "nonstationary" problem. This is a nontrivial "out-of-the-box" move for OCO, as the latter cannot capture the loss of plasticity we aim to address. In our opinion, the insight is pleasingly neat:

• parameter-free OCO is all about better regularization and
• better regularization mitigates loss of plasticity.

Furthermore, PACE employs the 1D algorithm from Zhang et al. (2024), which is theoretically optimal yet untested in real-world deep learning training.

Minor: Adam

We use default PyTorch Adam, which we note is adaptive but still relies on an initial LR as a hyperparameter (in contrast, PACE is insensitive to LR).

Please let us know if you have any other concerns/comments!

The authors made a significant effort to improve the paper, in particular with the addition of baselines. Some concerns still remain, but the modifications grant an increase of my initial assessment.

PS: I'm not able to access the new version of the paper. I'm not sure whether the authors have uploaded it.

Thank you for your thoughtful review and for highlighting the strengths of our work!

Layer normalization helps:

Thank you for suggesting the use of layer normalization, as highlighted by Lyle et al. for addressing plasticity loss, in our experiments. We implement layer normalization in our control experiments for both PPO with Adam and PPO with PACE (Figure 3 in Supplementary PDF). Our results show that while layer normalization improves the performance of baseline Adam, it does not outperform PACE. Moreover, we find your hypothesis to be true: PACE with layer normalization performs better than PACE without layer normalization (see Figure 3b in Supplementary PDF).

Comments

• Thank you for your close reading of our method and the appendix. We unfortunately did not have the opportunity to examine the relationship between the parameter norm and our scaling approach, but we are excited to explore this analysis in future work!

• Regarding your question about updating $\theta_{t}^{Base}$ by evaluating the gradients at $\theta_{t}$, intuitively, there is more stability around the reference point. Therefore, updating the gradients from a base update near this safe region helps prevent issues with gradients that might otherwise explode or vanish if updated at $\theta_t$.

• We agree with your observation that the success of parameter-free optimization may not indicate a “hidden” convexity, but rather that these methods may have broader applicability. This is an important point, and future research should indeed investigate why these methods are effective in such settings.

• Regarding Figure 2, we appreciate your feedback and will update the caption to better explain our visual abstraction of the setting.

• You correctly pointed out that in Figure 6, while PACE combats plasticity loss, it may still be affected by it, as the reward does not reach the same level as during the first distribution shift. Although good, we believe PACE's scaling isn’t always optimal, especially in a non-convex setting. There may be a more "optimal" set of scaling values such that the performance is as good as the first task, and this is something that may evolve as parameter-free methods continue to improve.

Please let us know if you have any other concerns or comments, and we'll be glad to address them!