Thanks for the valuable review.

Q1 Is data relabeling also used for all the baselines with fast policies? If not, please do so.

A1 Thank you for the valuable suggestion. We will include baselines with data relabeling in the final version. Due to time constraints, we currently provide a comparison between our algorithm without relabeling and the baselines:

Q2, W3, W4, W6 Report PEARL alone (the "slow" learner from prior work). Also report the "fast" learner used in Algorithm 2 step 10 with experience relabeling. The absence of these two baselines are of major concern to me.

A2 LILAC, which we use as one of our baselines, is just the non-stationary version of PEARL. PEARL is a meta-RL algorithm that requires the task id, which is not available in the non-stationary problem setting. LILAC transfers PEARL into the non-stationary MDPs scenario.

Another baseline, ITER, can be viewed as the ‘fast’ learner only, as ITER also adopts the mechanism of periodically resetting the model.

Based on the comparisons with these baselines, our method demonstrates strong performance, addressing concerns regarding the adequacy of baseline coverage. Nonetheless, we recognize the importance of clearer presentation and will improve the baseline annotations and related discussions in the next revision.

Q3, W13 Increase seeds to at least 10 and include confidence intervals. Otherwise the claimed gains may not be significant.

A3 Thank you for the helpful advice. We will increase the number of seeds in the next version. For now, we include BFSF results with three additional seeds:

Q4, W1 Please define all symbols in Theorem 2.1 and provide a proof sketch. Under what realistic conditions can the bound be < V_max? If the bound is always loose, consider reframing the theory section.

A4 The symbol definitions appear on Line 142; we reproduce and briefly explain them below:

• $M,M’$: the original and relabeled MDPs.
• $H$: the horizon.
• $r_{max},V_{max}$: the maximum reward and value.
• $r_{diff}$: the maximum reward gap between $M,M’$.
• $U_{r,r'}(\pi)=E_{(s,a)\sim\rho^\pi_{M}}[r’(s,a)-r(s,a)]$: the average reward difference between two MDPs under the same occupancy measure $\rho_M^{\pi}$.
• $D_{\ell_{1}}$: the L1 distance.

Proof Sketch The bound decomposes the suboptimality into two parts that bridge the true MDP $M$ and the relabeled MDP $M'$:

1. $|\eta_M(\pi^\ast_M)-\eta_{M’}(\pi^\ast_{M’})|$: the difference between the optimal returns in $M$ and $M’$, corresponding to the first RHS term.
2. $|\eta_{M’}(\pi^\ast_{M’})-\eta_{M}(\pi^\ast_{M’})|$: the return difference when applying the same policy $\pi_{M’}^\ast$ under two different MDPs, $M$ and $M'$. This term can be further decomposed into two parts, corresponding to the second and third terms in Theorem 2.1, because the MDPs differ in both reward and dynamics.

Tightness: The bound is not as large as $V_{max}$, as $V_{max}$ can be viewed as a constant multiplied by an L1 distance (e.g. $V_{max}D_{\ell_1}(p_{M’}(s,a),p_M(s,a))$). The relabeled MDP $M’$ trained to closely approximate the original MDP $M$, so the L1 distance tends to be small after training.

Q5, W12 Explain algorithmic timing. Describe precisely how training, dual-reset, and environment interaction interleave.

A5 In each iteration, we first select either the ‘fast’ or ‘slow’ policy. We then collect 5 episodes, with each episode consisting of 200 steps, through environment interaction. Afterward, we perform 2000 steps of RL training. The model reset occurs every 50 episodes.

Q6, W11 In Fig. 2, what causes the periodic drops in BFSF between switches, and why are the drops absent at 6 k / 10 k steps on Meta-World Reach?

A6 The periodic drops are caused by abrupt task changes with potentially conflicting objectives — for example, in Cheetah-Dir, the tasks involve moving left versus right. In contrast, in Meta-World Reach, although goals are randomly sampled across the space, neighboring tasks may occasionally result in nearby goals, leading to smoother transitions and less pronounced performance drops.

Q7 Line 497 says "As we cannot include images in this rebuttal" in the appendix of the submission. Please include said images.

A7 We apologize for the missing image. Since images cannot be included in the rebuttal itself, we will ensure that the final version of the paper includes the missing figure with the performance curves.

W2 There is no discounting in the MDP problem definition. Is there no discounting in the experiments? In general, critical hyper-parameters like discount factor, window size w, reset frequency ν, RL algorithm for line 10 in Alg. 2) and number of seeds (only 3) are either buried in the appendix or absent, undermining statistical reliability.

Answer: Thank you for pointing this out. We will make sure to explicitly present these key hyperparameters alongside the main definitions in future versions for better clarity. Below are the values used in our experiments:

• Discount factor: $0.99$. The MDP definition omits discounting since our environments are finite-horizon tasks where discounting is not strictly necessary. However, we use standard discounted return during training.

• Window size $w$: $100$.

• Reset frequency $\nu$: $50$.

W5 Unlike the authors claim on line 471, Table 3 shows (I assume average returns) that the approach is highly sensitive to the window size w and reset frequency v with a very clear trend (higher w and v lead to poorer performance).

Answer: We agree that the performance varies across different choices of $w$ and $\nu$, as shown in Table 3. However, even under suboptimal hyperparameters, our approach consistently outperforms the baseline (LILAC average return: 757.8), demonstrating the robustness and effectiveness of BFSF.

W6 There's also too much noise in the figure. E.g. It is hard to justify the "Dual-Reset Ablation" interpretation given by the authors given the noise. An easier domain like Cheetah-vel or Ant-Goal (from looking at the results in Fig 2) would have been better for this ablation.

Answer: Thank you for the suggestion. We agree that domains like Cheetah-vel or Ant-goal may yield cleaner curves and help highlight component effects. We chose Cheetah-dir because it presents a more challenging setting where interference is more severe, making the benefits of our design more evident. That said, we appreciate the reviewer’s feedback and will consider ways to improve the clarity and readability of the curves in future revisions.

W14 The main novelty in this work seems to be from the specific Bayesian inference and dual-reset ideas. Unfortunately, Figure 4 does not support the claim that these are useful significant components of the approach.

Answer: Thank you for the comment. While Bayesian inference and the dual-reset mechanism are indeed important components of our method, we would like to clarify that the core novelty lies in the dynamic switching between the fast and slow policies, which directly addresses the interference problem in non-stationary environments. The dual-reset and data relabeling mechanisms help these modules function effectively, while Bayesian inference provides a theoretically grounded and adaptive way to select between them.

More specifically, compared to the ablations without dual-reset or Bayesian inference, BFSF demonstrates a clear advantage at the beginning of the third phase: these ablated versions struggle to leverage experience from the first phase due to interference, while BFSF maintains better policy continuity. This suggests that the full BFSF framework, including Bayesian switching and dual resets, contributes meaningfully to mitigating interference.

W8-12 Weakness in clarity.

Answer: Thank you for the careful reading and constructive feedback. We will correct all typographical errors and improve figure clarity in the final version.

Thanks for the valuable review.

Q1, W1 Could the Bayesian selection mechanism be enhanced with exploration to avoid early commitment bias? The current policy selection is fully deterministic based on posterior mean estimates. This may lead to under-utilization of one policy (e.g., the ‘slow’ policy) if the other dominates early on. Have you considered incorporating stochastic policy selection mechanisms to encourage balanced exploration of both ‘fast’ and ‘slow’ policies, especially in the early phases?

A1 Exploration is inherently incorporated in our approach. Specifically, we only utilize data within a recent window, which keeps the effective sample size small, and we set the prior value $\mu_0$ to a dynamically updated upper bound to encourage exploration. As a result, if a policy has not been sufficiently selected, the prior strongly influences the Bayesian estimate, leading to a large posterior value that naturally encourages exploration of that policy.

Moreover, the ‘slow’ and ‘fast’ policies do not compete exclusively as in standard bandit settings. The experience gathered by one policy can be efficiently leveraged by the other. Evidence for this can be seen in existing offline meta-RL research [1], where even fully offline datasets are sufficient to support effective context-based policies.

Lastly, our algorithm selects between the ‘fast’ and ‘slow’ policies by sampling from their respective posterior distributions and choosing based on the probability that one sample exceeds the other, which inherently provides exploration capability.

Q2, W3 How robust is the ‘fast’ policy’s performance to inaccuracies in the context-based dynamics model used for relabeling? Since relabeling heavily depends on the quality of the learned model, it would be helpful to understand how model errors affect ‘fast’ policy training. Have you evaluated the impact of model approximation errors on policy performance?

A2 Thank you for the question. The context-based dynamics model is used to relabel past experiences to make them more relevant to the current task, thereby accelerating the fast policy’s adaptation. When the model is inaccurate, the relabeled data may become less helpful, but the fast policy can still rely on newly collected (on-policy) data. As a result, the impact of model errors is typically limited to short-term adaptation performance, and does not severely affect long-term learning.

To evaluate the sensitivity to model quality, we conducted an ablation study on the relabeling percentage (Appendix B.3), summarized below:

As the results show, our algorithm maintains relatively stable performance across a wide range of relabeling percentages, demonstrating that our method is robust and does not heavily depend on precise tuning.

Q3, W2 What happens when fast policy dominates data collection for extended periods? Although slow policy is updated in each epoch, it depends on data collected through interaction through the past policy. It would be helpful to clarify how such imbalance is mitigated in practice, and whether you considered mechanisms for guaranteeing minimum data exposure for each policy.

A3 This concern relates closely to the explanation in A1: the ‘slow’ and ‘fast’ policies do not compete exclusively, and the experience collected by one policy can be effectively leveraged by the other.

To better answer your question, we can analyze how the ‘fast’ and ‘slow’ policy functions in a real experiment in Figure 5. In the beginning, ‘fast’ policy dominates as there are not enough multi-task data for ‘slow’ policy to generalize, while in the tasks afterward, the ‘slow’ context-based policy serves as a good initial point and the ‘fast’ policy further improves efficiently. To summarize, the ‘slow’ policy provides reasonable performance when encountering a new task, while the ‘fast’ policy dominates the improvement inside a task, which can also be utilized by the ’slow’ policy.

Q4, W7 In Figure 4(b), the ablation results suggest that the fast policy and relabeling have the most significant impact on overall performance, whereas the removal of Bayesian inference seems to cause only a minor degradation. Could you clarify the relative importance of the Bayesian selection mechanism in the overall framework? Specifically, how critical is Bayesian inference to the stability and adaptability of BFSF? If the system performs almost as well without Bayesian inference and slow policy, what unique benefits does it provide in more challenging or uncertain settings?

A4 We appreciate the reviewer’s insightful comment. As shown in Figure 4(b), the fast policy and relabeling indeed have the most direct and substantial impact on performance, with the removal of Bayesian inference resulting in only a minor drop in performance under the current benchmarks. However, it’s important to note that the ablation without Bayesian inference can be viewed as a deterministic approximation of the Bayesian posterior, using the mean. The full Bayesian approach still provides a principled and theoretically grounded framework for decision-making.

While the fast policy and relabeling can improve performance independently in simpler tasks or when prior knowledge is strong, the true strength of Bayesian inference shines in more complex and uncertain environments. In such settings, Bayesian reasoning helps mitigate early commitment biases, improving the agent’s ability to adapt over time. By weighing policies based on their long-term effectiveness, Bayesian inference ensures greater stability and robustness, especially in non-stationary or noisy conditions.

W4 The framework introduces several new hyperparameters (e.g., reset frequency, window size for Bayesian updates), but sensitivity to these is not extensively studied or theoretically justified.

Answer: Appendix B.3 provides a sensitivity study on the window of recent data $w$ and the reset frequency $\nu$. Additionally, we present the following study for reference:

Sensitivity Study on the window of recent data $w$: (For reference, the average return of the baseline (LILAC) is $757.8$.)

Sensitivity Study on the reset frequency $\nu$:

In the Bayesian update, the prior value $\mu_0$ is set to a dynamically updated upper bound to encourage exploration. The value of $\phi_0$ is set to 1 without further tuning.

[1] Kate et. al., Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables. ICML 2019.

Thanks for the valuable review.

Q1 How does the computational overhead of maintaining dual policies, meta-RL training, and Bayesian inference scale with the number of tasks and complexity of environments?

A1 Thank you for the question. The computational overhead of our method remains modest and does not scale with the number of tasks, since we always maintain only two policies (fast and slow), regardless of how many tasks are encountered. We do not allocate a separate policy per task; instead, generalization is achieved through a context-based conditioning mechanism.

The Bayesian inference module only performs selection between the two policies based on recent returns, and thus its overhead is negligible and does not scale with task number or environment complexity.

Compared to context-based baselines (e.g. LILAC), the primary additional cost comes from training and maintaining extra 'fast' dual policies. In practice, this results in only a moderate increase in training time.

Q2 How sensitive is the framework to key hyperparameters like the window size (w=100), reset frequency (ν=50), and relabeling percentage (50%)?

A2 Appendix B.3 provides sensitive studies on the window of recent data $w$, the reset frequency $\nu$ and the relabel percentage. Additionally, we present the following study for reference:

Sensitivity study on the window of recent data $w$: (For reference, the average return of the baseline (LILAC) is $757.8$.)

Sensitivity study on the reset frequency $\nu$:

Sensitivity Study on the relabel percentage:

Q3 Since the method assumes unknown task boundaries, how would it perform in scenarios with gradual task transitions or when task changes are very subtle?

A3 The experiments in gradual task transitions are provided in Appendix B.3. Additionally, we present the result for reference:

Q4 Would BFSF be effective in other domains like discrete action spaces, vision-based tasks, or environments with fundamentally different dynamics than the tested locomotion tasks?

A4 BFSF is designed to address the interference problem in non-stationary MDPs, a challenge that may also arise in domains like discrete action spaces and vision-based tasks. We appreciate the valuable suggestion and agree that exploring these domains will be crucial for further validation.