We thank the reviewer for the insightful feedback and positive assessment of our work, especially on solid theoretical grounding, advancing the formal statement of CRL, intuitive mathematical formulation, and principled learning decomposition. We aim at addressing the concerns you raised in your review.

Q1: [Weakness (1)] The performance of the Q-value-based update may degrade when Q-values have inconsistent scales across tasks.

A1: This potential performance degradation in the presence of inconsistent scales exactly motivates the policy-based definition of catastrophic forgetting (Lines 168 to 171), which we show is more versatile than the Q-value-based one and serves as a preferable alternative. In addition, policies may inherently enjoy lower variance than value functions, contributing to improved performance and stability.

To address this issue, we propose Softmax transformation (starting from Line 177) to transform the Q-value-based incremental update to a policy-based incremental update under KL divergence (in Proposition 2). This softmax transformation strategy fundamentally solves the inconsistent scale issue across tasks, as aligning Q-functions directly across tasks can be more brittle. In contrast, aligning policies is cleaner in mathematical formulation and more robust against diverse settings.

Q2: [Weakness (2)] No ablation or sensitivity study is presented for the hypothesis test-based warm-up mechanism, which may introduce fragility under noisy returns or limited samples.

A2: Increasing the policy evaluation steps $n$ for the hypothesis test is helpful in general under noisy returns at the cost of reducing the policy optimization updates. In our experiments, we consistently let $n=600$ across all sequences of games on MinAtar as we find it is sufficient to maintain favorable CRL-relevant metrics. In Table 6 of Appendix E.3, we have provided the ablation study by varying the policy evaluation steps $n$, suggesting that increasing the policy evaluation step does not significantly impact the Continual RL metrics.

Q3: [Weakness (3)] The exposition could benefit from clearer visual diagrams summarizing the update mechanisms or data flow between the fast and meta learners.

A3: Thanks for this great suggestion, and we promise to improve Figure 1 to better exhibit the update mechanisms and interplay between the fast and meta learners.

Q4:[Weakness (4)] No results are shown on longer sequences of tasks (e.g., 10+tasks), which would better test scalability and forgetting

A4: Most existing continual RL benchmarks normally use 10 tasks in the sequence. For example, we follow the MinAtar setting in [5] with 7 games in the sequence and the Meta-world environment in [2,3] with 10 tasks. We will evaluate a longer sequence of tasks in the revised paper and put the results in the appendix for reference.

Q5:[Weakness (5)] The neuroscience motivation, while engaging, does not clearly influence the architecture beyond analogy.

A5: We believe this is a fundamental challenge to fully connect the growing of neurons in brains with the continual RL algorithm design. Although this research direction is beyond the current scope of our paper, we fully acknowledge this important future direction in continual RL.

Q6:[Question 1] The adaptive warm-up mechanism relies on a hypothesis test between fast, meta, and reset strategies using early task returns. However, early rollouts can be noisy or sparse, especially in high-variance environments. How sensitive is the adaptive warm-up mechanism to noise in early evaluations?

A6: We employ 600 policy evaluation steps, and it turns out that in our experiments the choice of best warm-up is not sensitive to the potentially noisy returns in the early rollouts. In particular, when the adaptive warm-up mechanism chooses the meta warm-up, we often find the average rewards evaluated by the meta policy are much higher than the other two in our experiments. We agree that there may be some very noisy environments when the trustworthy policy evaluation and comparison become more important, such as the safety-critical scenarios we mentioned in Lines 218-219. This could motivate follow-up work to specifically investigate the policy evaluation effect in noisy environments in the context of RL, but for our experiments, we have not found the sensitivity of the policy's values in the early rollout.

Q7:[Question 2] The current framework assumes known task boundaries, which simplifies the design but limits applicability innon-stationary environments without such information. How well does the method generalize to task-agnostic continual RL?

A7: Most continual RL methods assume the task boundary to be known in the literature, as no access to the task boundary will pose a fundamental challenge for continual RL. We think the hypothesis test in the adaptive meta warm-up strategy can be very useful to address this issue. In particular, we can develop an online hypothesis test to discriminate the current data distribution within a sliding time window and the new distribution of the new arriving samples potentially from a new task. This will be a very interesting research direction that relies on the advanced hypothesis test techniques, and we leave it as future work.

Q8:[Question 3] How does the method handle tasks with inconsistent Q-value or reward scales?

A8: Please refer to A1.

Q9:[Question 4] The framework includes a meta buffer, separate learners, and evaluation steps. While said to be scalable,resource costs are not quantified. Can the authors clarify the memory and computational cost of FAME?

A9: In terms of space complexity, the dual learner system of FAME requires an additional memory copy as the fast learner (the normal learner in baselines such as Reset and Finetune), and an additional meta replay buffer with the same size as the buffer of the fast size, which is scalable as the total sample memory is fixed and independent of the number of future tasks.

In terms of the computational cost, we employ the same number of agent's interaction steps with the environment for a fair comparison. In other words, the policy evaluation occurs at the cost of reducing the policy optimization update steps in total. The main additional computation cost is the updating of the meta-learner, which is conducted in a supervised learning way. We found it only takes around a couple of minutes across 200 epochs to update the meta-learners in MinAtar after one task finishes. This additional computation overhead is negligible to the overall computation cost in the online training of RL algorithms.

Overall, FAME has a negligible computation cost increase and double space memory to achieve competitive continual RL performance, which is much more efficient and scalable.

Q10:Limitations and Paper Formatting

A10: We acknowledge that in Section 5 we discuss the limitations of FAME approach and point out some interesting future directions. We thank the reviewer for mentioning the potential societal impact, and we promise to add this discussion in future revisions.

Regarding the paper formatting, we are grateful to the reviewer for highlighting these issues. We commit to improving the presentation of Figure 1 for better clarity and cleaning up potential confusion and typos in notations.

Thank you again for pointing out these potential areas of improvement. We appreciate your suggestions. Please let us know if you have any further comments or feedback.

Reference

[1] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. Conference on robot learning. PMLR, 2020.
[2] Malagon, Mikel, Josu Ceberio, and Jose A. Lozano. Self-composing policies for scalable continual reinforcement learning. ICML 2024.
[3] Wesley Chung, Lynn Cherif, David Meger, Doina Precup. Parseval regularization for continual reinforcement learning. NeurIPS 2024.
[4] Hongjoon Ahn, Jinu Hyeon, Youngmin Oh, Bosun Hwang, Taesup Moon. Prevalence of Negative Transfer in Continual Reinforcement Learning: Analyses and a Simple Baseline. ICLR 2025.
[5] Anand, Nishanth, and Doina Precup. Prediction and control in continual reinforcement learning. NeurIPS 2023
[6] Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Hadsell,R. Progressive neural networks. arxiv 2016
[7] Mallya, A. and Lazebnik, S. Packnet: Adding multiple tasks to a single network by iterative pruning. CVPR 2018

We would like to sincerely express our gratitude to the reviewer for their time and effort in reviewing our paper. We appreciate the reviewer's positive recognition of our work on a solid mathematical foundation, a new perspective from biology, and broader application in both value- and policy-based algorithms. We address each concern you raised and are happy to answer any further questions.

Q1: [Weakness 1] The authors perform the one-vs-all hypothesis to initialize a fast learner. Considering that there are threecandidate learners including a randomly initialized one, the time and space complexity to perform the hypothesis could be a concern to apply this mechanism in real-world application. It would be nice to have an ablation study how much computational overhead it takes and how much improvements it gains compared to using one of them only.

A1: In adaptive meta warm-up with the one-vs-all hypothesis test, it essentially implies that we replace the policy optimization in the early training phase with the policy evaluation given that we maintain the number of agent interactions with the environment for a fair comparison with other baseline methods. The computation within the policy evaluation steps of our adaptive meta warm-up is even lower than the normal training with the policy optimization throughout, as we do not need to update the policy parameters in the policy evaluation. After the policy evaluation, we conduct the hypothesis test among three scalars with negligible overhead. Overall, the computational overhead of our approach with the adaptive meta-warm-up strategy is also the same as the normal training, e.g., Reset and Finetune method.

Regarding the ablation study among the fast, meta, and random learners, if there is only a random learner, our fast learning only selects the random learner as an initialization, degenerating to a Reset method. If there is only a fast learner accessible, our fast learning degrades to the Finetune method. In conclusion, two baseline methods (Finetune and Reset) are the special case in our adaptive meta warm-up strategy, and the performance of them naturally serves as the ablation study.

Q2: [Weakness 2] The baselines in this paper are too weak to evaluate. There are lots of works in continual RL [1]. I suggest theauthors compare with SOTA continual RL algorithms to make the results more convincing.

A2: We add PackNet [6] as the baseline method in Meta-world, which serves as a powerful baseline in multi-task and continual RL settings. The results are provided in Tables 1 below. Notably, Packnet requires the task identifiers and the number of all tasks to be known, which is less practical in real applications. Although keeping track of model parameters of the past tasks in Packnet contributes to zero forgetting, but it is less beneficial in the other two metrics compared with our FAME approaches. We also add more baseline methods in Atari games in Table 2 in A3.

Table 1: Average Results in Meta-world across three Sequences over 10 seeds (3 seeds on PackNet). The best and second-best methods in each metric are in bold and underlined, respectively.

Q3: [Weakness 3] Additionally, running experiments on three different tasks repeatedly in shuffled order is not enough to show the knowledge transfer and resistance to catastrophic forgetting of this work. Referring to [1], a typical benchmarkruns 6 different Atari games to evaluate a continual RL algorithm. I suggest the authors to run more tasks to make the evaluation more solid.

A3: We follow the MinAtar benchmark, which is established based on the shuffled order in [5]. Although shuffling order becomes less challenging, it often happens in real life and is particularly useful to demonstrate when our meta warm-up is most suitable when the old environment occurs again. Following your suggestions, we conduct experiments on Atari games by following [2], and add more baseline methods, including PackNet [6] and ProgressiveNet [7]. The result is provided in Table 2.

Table 2: Results on Atari Games, including ALE/Freeway-v5 environment with 7 different playing modes and ALE/SpaceInvaders-v5 environment with 10 different playing modes. The implementation is adapted from [2]. The best and second-best methods in each metric are in bold and underlined, respectively.

Q4: [Questions 1] Does the replay buffer size increase linearly with the task? ....

A4: We allow the number of collected data in the meta buffer to linearly increase with the task under the constraint of the replay buffer size. However, we only store a small proportion ($1\\%$ in our experiments) of state and action pairs in the last training phase of each task into the meta buffer (Line 232 in Section 3.1.3), which is very efficient in space complexity. That contributes to a meta buffer with the same size as the original buffer in baseline methods, e.g., Reset and Finetune. Our performance is achieved under this constraint for a fair comparison. We can expect the performance improvement if we allow a larger size of replay buffer to take advantage of more historical data.

It may be possible to investigate the generative replay buffer, but training the generative replay buffer may still be vulnerable to the (catastrophic) forgetting issue, i.e., losing the generation ability on the old samples when new samples are continually used to update. Therefore, the efficiency of the samples collected in the replay buffer seems more pressing, and our FAME approach enjoys a high sample efficiency in learning the meta-learner.

Q5:[Questions 2] The authors choose KL divergence and Wasserstein Distance without an explanation. The main results in theexperiments section show that FAME-WD is always worse than FAME-KL, so what is the motivation to bring WD inthis work?

A5: The KL and Wasserstein distances are perhaps the most commonly used distances in RL when comparing the difference of policy distribution. The KL divergence is much easier for computation, while it is known that Wasserstein distance is able to capture more data geometry and can be more suitable to compare more complicated policy distributions at the cost of heavy computation. When Gaussian policy is used (which is common in policy-based RL, e,g., PPO, DDPG), the Wasserstein distance can be simplified, allowing us to derive an incremental meta-learner update rule in Proposition 3.

In Table 2 of our paper, FAME-KL outperforms FAME-WD, but in Figure 3 (right), FAME-WD has higher average performance in the early tasks of the sequence. In addition, in Table 7 in Appendix F.2 on other sequences of tasks, FAME-WD still performs better in many cases across some metrics. In conclusion, introducing FAME-MD is more theoretically grounded, and we do observe the superiority of FAME-MD in some cases. We will add a detailed explanation in the revised paper to help readers to understand this theoretical motivation and technical contribution.

Thanks again for the reviewer's consistent dedication to reviewing our work, and we are happy to answer any further questions or concerns. Should this rebuttal address your concerns, we would be grateful for an increased score.

Reference

[1] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. Conference on robot learning. PMLR, 2020.
[2] Malagon, Mikel, Josu Ceberio, and Jose A. Lozano. Self-composing policies for scalable continual reinforcement learning. ICML 2024.
[3] Wesley Chung, Lynn Cherif, David Meger, Doina Precup. Parseval regularization for continual reinforcement learning. NeurIPS 2024.
[4] Hongjoon Ahn, Jinu Hyeon, Youngmin Oh, Bosun Hwang, Taesup Moon. Prevalence of Negative Transfer in Continual Reinforcement Learning: Analyses and a Simple Baseline. ICLR 2025.
[5] Anand, Nishanth, and Doina Precup. Prediction and control in continual reinforcement learning. NeurIPS 2023
[6] Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Hadsell,R. Progressive neural networks. arxiv 2016
[7] Mallya, A. and Lazebnik, S. Packnet: Adding multiple tasks to a single network by iterative pruning. CVPR 2018

Great thanks to the authors for addressing my concerns! The illustrations of computational and space overhead, as well as experiments are comprehensive. I updated my score.

1. However, I remain concerned about FAME-WD. Although the theoretical advantages of the Wasserstein distance are compelling, they do not consistently translate into empirical improvements over FAME-KL. Is it significantly more difficult to train a FAME-WD model that can outperform FAME-KL? If so, its greater computational cost and mixed results could limit its practical applicability. In the final version, I would appreciate guidance on scenarios in which the Wasserstein distance offers clear benefits, or at least some recommendations for selecting hyperparameters that optimize FAME-WD’s performance.
2. As noted in my comments:

In Appendix A, Related Work section, the authors cite papers numbered above 45, but the reference list has only 44 papers and there is no extra reference list in appendix.

It appears that some citations are missing. I would appreciate these being added in the final version.

We thank the reviewer for the positive feedback of our work, especially for the well-established mathematical foundations, the novelty of our approach, and the significant improvement in experiments. We aim at addressing the concerns you raised in your review.

Q1:[Weakness 1] The experiments are done with only 3 random seeds. I would recommend performing with at least 5 random seeds to reduce the variability in the experimental results.

A1: Thank you for this advice. For Meta-world, we now provide an updated summary of results over 10 seeds in Table 1 below, consistently substantiating the superiority of FAME. For the MinAtar environment, we clarify that our experiments are averaged over 10 sequences, each sequence with 3 seeds (30 seeds in total).

Table 1: Average Results in Meta-world across three sequences over 10 seeds (3 seeds on PackNet). The best and second-best methods in each metric are in bold and underlined, respectively.

Q2:[Weakness 2] $\tilde{Q}_k^M$ is not defined in Eq 3 and 4.

A2: We thank the reviewer for the helpful comment. $\tilde{Q}_k^M$ denotes a candidate Q function used to approximate $Q_k^M$ and we will state this in the revised paper to avoid ambiguity.

Q3:[Weakness 3] The proposed approach requires the knowledge of task information and task boundary. How will the approachfare where task boundary is unclear? While to the credits of the authors, they did indicate this as a limitation in the limitation section.

A3: Most continual RL methods assume the task boundary to be known in the literature, as no access to the task boundary will pose a fundamental challenge for continual RL. We think the hypothesis test in the adaptive meta warm-up strategy has great potential to address this issue within the FAME approach. In particular, we can develop an online hypothesis test to discriminate between the current data distribution within a sliding time window and the new distribution of the new arriving samples. This will be a very interesting research direction by leveraging the advanced hypothesis test techniques, and we leave it as future work.

Q4:[Weakness 4] In line 43-44, the authors mentioned that many existing methods have been proposed that trade off stability and plasticity, without quantifying explicit objectives to optimise. It will be nice if the authors can provide some citations and elaborate what could be some of these explicit objectives to consider.

A4: For example, [5] decomposes the value function into two with different time scales and optimizes them, but it is not clear whether the permanent value function holds the general knowledge and is able to explicitly reduce catastrophic forgetting. After a careful literature review, we strongly believe the foundations of continual RL, such as a principled measure of MDP distance or similarity (Definition 1), and catastrophic forgetting (Definition 2), play a pivotal role in developing continual RL algorithms in the community. We promise to add more citations to support this claim and add more details of our explicit objective, i.e., definitions 1 and 2, in the revised version of our paper.

Q5:[Weakness 5] The Minatar benchmark is somewhat simplistic in terms of representation and game dynamics complexity. How would the model fare in more complex games like Atari?

A5: Thanks for this suggestion. The choice of Minatar follows [5]. Albeit with a relatively lighter computational cost, MinAtar allows us to sweep a range of hyperparameters and to analyze the mechanism of our proposed method in detail, such as the selection ratio in Figure 2 (right). To strengthen our empirical evidence, we have also added experiments in Atari games by following [2] and adding more baseline methods. The results are provided in Table 2 below.

Table 2: Results on Atari Games, including ALE/Freeway-v5 environment with 7 different playing modes and ALE/SpaceInvaders-v5 environment with 10 different playing modes. The implementation is adapted from [2]. The best and second-best methods in each metric are in bold and underlined, respectively.

Q6:[Question 1] How is the ideas proposed in definition 1 and definition 2 differs or relate to bisimulation metrics[1]?

A6: Thanks for sharing this reference. Albeit being complimentary, bisimulation metrics [1] provide additional evidence to support the optimal value functions that can be naturally used to measure the state similarity, which is consistent with Definition 1. From this perspective, Definition 1 targets the overall state similarity by averaging all states (and actions) to define MDP distance and also extends the MDP difference from the difference of optimal value functions to policies. Based on Definition 1, Definition 2 is considered as a weighted MDP difference with the weights determined by the old tasks and new policy or value functions, serving as the foundation of quantifying catastrophic forgetting.

Q7:[Question 1] What about interference? Are there signs of interference for the tasks studied in the experiments? If so, how doesthe proposed model cope with limiting the effects of catastrophic interference?

A7: If we understand correctly, catastrophic interference in continual RL is the loss of performance on previous tasks when learning new ones, which also refers to catastrophic forgetting. The purpose of a meta-learner is exactly to minimize this catastrophic forgetting, defined based on Definitions 1 and 2. Feel free to clarify the questions if you find we have a misunderstanding here.

Thank you again for pointing out these potential areas of improvement. We appreciate your suggestions. Please let us know if you have any further comments or feedback.

Reference

[1] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. Conference on robot learning. PMLR, 2020.
[2] Malagon, Mikel, Josu Ceberio, and Jose A. Lozano. Self-composing policies for scalable continual reinforcement learning. ICML 2024.
[3] Wesley Chung, Lynn Cherif, David Meger, Doina Precup. Parseval regularization for continual reinforcement learning. NeurIPS 2024.
[4] Hongjoon Ahn, Jinu Hyeon, Youngmin Oh, Bosun Hwang, Taesup Moon. Prevalence of Negative Transfer in Continual Reinforcement Learning: Analyses and a Simple Baseline. ICLR 2025.
[5] Anand, Nishanth, and Doina Precup. Prediction and control in continual reinforcement learning. NeurIPS 2023
[6] Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Hadsell,R. Progressive neural networks. arxiv 2016
[7] Mallya, A. and Lazebnik, S. Packnet: Adding multiple tasks to a single network by iterative pruning. CVPR 2018

I thank the authors for the clarification.

Having read the comments from the authors on A7, I am concerned that the authors have misunderstood the difference between catastrophic interference and catastrophic forgetting. Catastrophic interference refers to the inability of the agent to learn new knowledge after prior learning of other task(s) while catastrophic forgetting refers to the inability of the agent to recall previously learned knowledge.

Can the authors discuss how their approach will fare in the catastrophic interference scenario?

We thank the reviewer for the appreciation of our work, especially for the novel use of the meta policy, a principled measure of forgetting, the Q-function to policy alignment, and memory efficiency. We aim at addressing the concerns you raised in your review.

Q1: The evaluation is weak. It is based on seven simplified Atari-style environments and a single 10-task Meta-World sequence, with all results averaged over just three seeds.

A1: For the MinAtar environment, we clarify that our experiments are averaged over 10 sequences, each sequence with 3 seeds (30 seeds in total). For Meta-world, in the submitted paper, we also provide similar results of two extra sequences of tasks in Appendix F.2 across 3 seeds, and sequences of tasks are also given. To reduce the randomness, we now provide an updated summary of results for each sequence of tasks over 10 seeds (3 seeds on PackNet [7]) in Tables 1.1-1.3, consistently substantiating the superiority of FAME-KL and FAME-MD. More explanation of Packnet [7] is given in A3.

Table 1.1: Results of Sequence 1 in Meta-world averaged over 10 seeds. The best and second-best methods in each metric are in bold and underlined, respectively.

Table 1.2: Results of Sequence 2 in Meta-world averaged over 10 seeds. The best and second-best methods in each metric are in bold and underlined, respectively.

Table 1.3: Results of Sequence 3 in Meta-world averaged over 10 seeds. The best and second-best methods in each metric are in bold and underlined, respectively.

Q2: The authors do not evaluate FAME on established continual RL benchmarks such as Continual World [1], COOM [2], or CoRA [3].

A2: The Continual World Benchmark is built on top of Meta-World as a testbed, an established benchmark in meta-learning and multi-task learning and continual RL communities. Continual world defines a specific sequence of tasks (e.g., CW10 or CW20) using tasks from Meta-world, while Meta-world offers more flexible evaluation on various sequences of tasks. For example, many recent continual RL work, such as [2,3,4], is evaluating their methods on Meta-world directly. Section 4.2 of our paper is based on Meta-world, with the implementation adapted from [3] (Wesley Chung et al. NeurIPS 2024). Section 4.1 in MinAtar follows [5] (Anand Nishanth et al. NeurIPS 2023) as well as the implementation to better elucidate the mechanism of our FAME approach, especially the adaptive meta warm-up strategy via Figure 2. Our evaluation is highly established on common continual RL benchmarks in existing literature.

Q3: In terms of baselines, the paper compares only against simple adaptations of standard RL algorithms, while omitting well-established continual learning methods like EWC, AGEM, MAS, and PackNet.

A3: We add the Packnet in our empirical comparison, as Packnet is considered as a more powerful and widely used baseline as opposed to others. The results are provided in Tables 1.1-1.3, where our FAME approach also outperforms Packnet across average performance and forward transfer. Notably, Packnet achieves zero forgetting by storing (masks of) model parameters of previous tasks by additionally knowing the task identifiers and the number of tasks in advance, which is less practical in real scenarios. The average result across three sequences in Meta-world and additional comparisons in Atari games with more baseline methods (ProgressiveNet [6]) are provided in Tables 1 and 2 below.

Table 1: Average Results in Meta-world across three Sequences over 10 Seeds (3 seeds on PackNet). The best and second-best methods in each metric are in bold and underlined, respectively.

Table 2: Results on Atari Games, including ALE/Freeway-v5 environment with 7 different playing modes and ALE/SpaceInvaders-v5 environment with 10 different playing modes. The implementation is adapted from [2]. The best and second-best method in each metric are in bold and underlined, respectively.

Q4: FAME’s adaptive warm-up mechanism appears to provide the most benefit when tasks are revisited, rather than when encountering entirely new environments. ... Revisiting tasks is a less challenging setting than tackling a continual stream of novel tasks in non-repeating, task-incremental scenarios.

A4: The shuffled order of tasks is particularly useful to illuminate the mechanism of our adaptive meta warm-up strategy. In real applications, the new arriving task is agnostic, which can be either totally unknown or re-encountered. Although a shuffled order of sequence is slightly less challenging to totally non-repeating scenarios, evaluating mixed types of new tasks is also meaningful for general continual RL scenarios (Section 4.1).

The adaptive meta warm-up offers more flexibility to choose the best initialization or warm-up for an agnostic task. For instance, it is very intuitive to choose a meta warm-up more often when an old task is revisited, while it chooses a reset (random initialization) when a new environment occurs. In addition to the scenarios with mixed types of new arriving environments, in Section 4.2, we focus on the non-repeating and task-incremental scenarios, suggesting consistent performance. The continual RL scenarios with shuffled order of tasks and incremental tasks in Sections 4.1 and 4.2 of our experiments provide a more comprehensive evaluation of the continual RL performance.

Q5: How sensitive is FAME to the choice of temperature in the softmax transformation and the size of the meta buffer?

A5: Our FAME method is robust to the choice of hyperparameters for the considered ranges in our ablation study analysis in Appendix E.3. For instance, we directly leverage categorical transformation, i.e., Softmax transformation with the temperature $\tau=1$, and keep the maximum size of the meta buffer as the same as the standard on used in DQN-Reset and DQN-Finetune. Our FAME approach under these simple choices of hyper-parameters already suggests desirable performance. The sensitivity analysis of all other hyperparameters with a line search is provided in Appendix E.3, including the regularization hyperparameter $\lambda$, the warm-up step $L$, the policy evaluation step $n$, and the weight estimation step $N$, demonstrating that our FAME approach is not sensitive to these hyperparameters for the considered ranges.

Thank you again for pointing out these potential areas of improvement. We appreciate your suggestions. Please let us know if you have any further comments or feedback.

Reference

[1] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. Conference on robot learning. PMLR, 2020.
[2] Malagon, Mikel, Josu Ceberio, and Jose A. Lozano. Self-composing policies for scalable continual reinforcement learning. ICML 2024.
[3] Wesley Chung, Lynn Cherif, David Meger, Doina Precup. Parseval regularization for continual reinforcement learning. NeurIPS 2024.
[4] Hongjoon Ahn, Jinu Hyeon, Youngmin Oh, Bosun Hwang, Taesup Moon. Prevalence of Negative Transfer in Continual Reinforcement Learning: Analyses and a Simple Baseline. ICLR 2025.
[5] Anand, Nishanth, and Doina Precup. Prediction and control in continual reinforcement learning. NeurIPS 2023
[6] Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Hadsell,R. Progressive neural networks. arxiv 2016
[7] Mallya, A. and Lazebnik, S. Packnet: Adding multiple tasks to a single network by iterative pruning. CVPR 2018

Hi Reviewer cESJ,

I hope you don't mind me jumping in here. I would like to seek your clarification on the dismissal of the last point. Are you against the use of Minatar for studying continual RL or the use of repeating tasks for studying continual RL?

It is common in the continual RL literature to use repeating tasks, especially in Atari [8] and Minatar [5]. The latter which the authors have cited. I listed just two papers here but there is much more in the literature.

[5] Anand, Nishanth, and Doina Precup. Prediction and control in continual reinforcement learning. NeurIPS 2023

[8] Abbas, Zaheer, et al. "Loss of plasticity in continual deep reinforcement learning." Conference on lifelong learning agents. PMLR, 2023.