We sincerely appreciate your detailed and constructive feedback. Below are our responses to your concerns.

1. Comparison with BMPC (Weakness 1)

Limited novelty: The key approach of imitating planner behavior to address actor divergence (Eq.2), as presented in this paper, appears to mirror similar methodologies already employed in BMPC (Eq.6 in BMPC paper). From my perspective, the novelty of the proposed method in this regard is not immediately apparent. The authors should delineate the core contributions of this work that distinguish it from the existing BMPC.

We appreciate the opportunity to clarify the differences between our work and BMPC. While both methods leverage planner-generated actions for imitation in MBRL, our approach is conceptually distinct and practically more efficient in several key aspects:

1. Time efficiency

• BMPC relies on frequent re-planning during training to obtain up-to-date planner actions for behavior cloning, which introduces significant computational overhead.
• In contrast, our method employs a bootstrap alignment mechanism that aligns the actor with the stored planner actions, eliminating the need for explicit re-planning. This design shows a clear advantage in time efficiency. (see following Table 6: Training time with re-planning).

2. Simplicity and generality

• BMPC incorporates several auxiliary designs beyond its core planning framework. For instance, it periodically alters max_policy_std to control exploration levels. While such heuristics can be effective in specific tasks, they require manual tuning and are sensitive to domain choices, limiting generality and robustness.

• In contrast, our approach focuses on a minimal yet effective alignment mechanism that improves policy quality without introducing domain-specific tricks or altering core components. This simplicity not only makes our method easier to implement and integrate, but also more stable and broadly applicable across tasks.

3. Performance
Empirically, our method consistently outperforms BMPC across various tasks, particularly in high-dimensional settings like Dog-run and Humanoid-run. While BMPC is prone to early convergence and unstable updates, our method maintains steady progress and achieves higher final returns with greater learning efficiency.

Summary

• BMPC reflects a transitional stage in the development of planning-driven MBRL—a design that gestures toward the right direction but remains incomplete. It relies on several loosely motivated heuristics, such as periodically adjusting max_policy_std and fresh planner actions in replay buffer, to promote exploration and maintain training stability. These handcrafted tweaks suggest that BMPC is still evolving and lacks the robustness needed for broader applicability.

• In contrast, our BOOM offers a fully-formed, lightweight solution that seamlessly integrates into existing MBRL frameworks. Without introducing new components or tuning-sensitive mechanisms, we achieve consistent gains in stability and performance. We believe this kind of alignment-based strategy not only improves practical usability but also opens up a promising direction for more systematic advancement in planner-policy integration.

2. Improved statistical validation (Weakness 2)

Limited Statistical Validation: Experimental results (Fig. 2, Table 1) based on only three random seeds are insufficient for robust validation.

We have increased the number of random seeds from 3 to 5 for our method on all benchmark tasks. Due to time constraints, we were only able to extend the seed count for our method rather than all baselines. The updated results are shown in the following table, and our method still consistently outperforms baselines with slightly reduced variance.

Table 1: Benchmark with increased number of seeds

Notably, the only task where performance dropped was the hurdle, which requires the humanoid to continuously jump over multiple hurdles. We observed that some seeds resulted in the agent missing 1–2 jumps and falling prematurely, leading to relatively low scores around 350–400, while successful seeds achieved 450–500. The overall average across 5 seeds is 417, which, despite some instability, still significantly outperforms prior baselines—BMPC achieves only 197.1 ± 12.1, and Dreamer (10M) only 135.7 ± 6.1.

3. Add ablation study (Weakness 2)

Ablation experiments are confined to a single task, limiting generalizability.

We conduct additional ablation experiments on the challenging H1hand-slide task from Humanoid-Bench, which complements the original ablation on Dog-run.

Table 2: Ablation on the alignment metric

Table 3: Ablation on the Q-weight mechanism

Table 4: Ablation on the alignment coefficient

As shown, the H1hand-slide involves higher-dimensional action space and thus requires relatively larger alignment coefficient. Overall, the trend still aligns well with our original claim about the effectiveness of the alignment mechanism. We believe these additional ablation study strengthen the empirical robustness of our claims.

4. Clarify the action in Eq.(2) and Eq.(3) (Q1)

In Eq.(2), action a is extracted from a "planning-augmented behavior policy," yet in Eq.(3) and Algorithm 1, it appears to be sampled from the replay buffer. These methods are distinct, especially as MPPI planning actions evolve throughout training. The author needs to clarify the difference between these two methods.

To clarify: Eq.(2) reflects our theoretical design — in an ideal setting, every training sample from the replay buffer would be paired with a freshly re-planned action generated using the current world model and policy. This corresponds to what we refer to as the "planning-augmented behavior policy."

However, in practice, continuously re-generating planner actions for all replay samples during every policy update introduces significant computational overhead. To address this, we adopt a practical compromise: when collecting experience, we store the MPPI-generated planner action into the replay buffer alongside the transition. As shown in Eq.(3), during training, we directly imitate these stored actions instead of re-planning on the fly.

While this may lead to some degradation due to action staleness, we mitigate this by applying a soft Q-weight mechanism. This allows to focus more on high-quality historical behaviors, reducing the impact of suboptimal actions.

We apologize for the confusion and will add a dedicated “Practical Implementation Details” section in the revision to make this distinction explicit.

5. More experiments and discussions on the re-planning mechanism (Q2)

Following Q1, due to the model bias in the initial world model training, the interaction data stored in the replay buffer is poor. Therefore, I have concerns about the effectiveness of using actions from the replay buffer for alignment. Why not adopt the re-planning approach used in BMPC?

We fully agree that actions stored in the replay buffer may be suboptimal, particularly due to model bias in the early stages. To evaluate the benefit of using up-to-date planner actions, we incorporated a re-planning mechanism from BMPC into our BOOM framework. Specifically, during every $k=10$ policy updates, we select $b=20$ samples from each training batch (batch size 256), and regenerate their corresponding planner actions using the current policy and world model. The re-planned action will replace the old action in the buffer.

Table 5: TAR with re-planning

Table 6: Training time with re-planning

This introduction of re-planning led to a consistent improvement in performance—from 819.7 to 860.1 TAR on the dog-run task—suggesting that aligning with fresh actions from the current model-planner pair can indeed enhance training. However, we also observed a notable trade-off: the overall training time increased by approximately 50% percent due to additional rollouts and planning.

We appreciate your suggestion, as it helped us identify a meaningful performance gain. However, we believe the increased time cost may limit the practicality of frequent re-planning in larger-scale or real-time applications. Given the relatively modest improvement and the added computational cost, we believe re-planning is not strictly necessary. That said, if performance is the top priority, it remains a reliable trick worth considering.

6. Conclusion

After extensive effort, we have added the requested more experiments with more seeds, clarified the difference from BMPC, and add experiments employing the re-planning mechanism from BMPC. We hope our responses address your concerns and kindly request your consideration for a higher score.

Thank you for your detailed and thoughtful comments. We’re sorry that some parts of our previous response may have led to misunderstandings, and we genuinely appreciate the opportunity to clarify them below.

1. Efficiency

We appreciate your insight regarding torch.compile. We acknowledge that we did not apply this optimization, as our work was built upon the TD-MPC2 codebase. However, as your results indicate, even with torch.compile, replanning introduces a 45.6% training time overhead ((9.9 – 6.8)/6.8), which is closely consistent with our reported 51.3% ((56.3 – 37.2)/37.2). This suggests that replanning still constitutes a non-negligible computational cost and impacts training efficiency though torch.compile is employed.

2. Simplicity and generality

What we actually mean is the additional parameters introduced by BMPC: expl_log_std_min and expl_log_std_max (Line 26 and 27 in bmpc/bmpc/config.yaml). Periodically, the policy’s max_policy_std and min_policy_std are replaced by expl_log_std_max and expl_log_std_min, which appeared to be a heuristic trick rather than a principled design. We’ll revise our wording to reflect this more precisely.

See Line 145-151 in bmpc/bmpc/common/world_model.py for the heuristic trick

# Gaussian policy prior
mean, log_std = self._pi(z).chunk(2, dim=-1)
if expl:
    log_std = math.log_std(log_std, self.expl_log_std_min, self.expl_log_std_dif)
else:
    log_std = math.log_std(log_std, self.log_std_min, self.log_std_dif)
eps = torch.randn_like(mean)

Given the log_std and random eps, the action is calculated by: Line 167-173 in bmpc/bmpc/common/world_model.py

# Reparameterization trick
if self.cfg.bmpc:
    mean = torch.tanh(mean)
    action = (mean + eps * log_std.exp()).clamp(-1,1)
else:
    action = mean + eps * log_std.exp()
    mean, action, log_prob = math.squash(mean, action, log_prob)

3. Performance

We believe the performance discrepancy stems from two key factors:

(1) In the BMPC config (Line 30 in bmpc/bmpc/config.yaml), if steps was set to 2_000_000, this may have led to 4M environment steps due to action_repeat=2. The correct setting should be 1_000_000 to ensure 2M environmental steps. This may explain why your reproduction results are inconsistent with the results reported in the bmpc paper.

Please see Line 53-59 in bmpc/bmpc/envs/dmcontrol.py for action repeat

def step(self, action):
    reward = 0
    action = action.astype(self.action_spec_dtype)
    for _ in range(2):
        step = self.env.step(action)
        reward += step.reward
    return self._obs_to_array(step.observation), reward, False, defaultdict(float)

(2) BMPC sets log_std_min: -3 and log_std_max: 1 (Line 59 and 60 in bmpc/bmpc/config.yaml), instead of TD-MPC2's default setting -10 and 2. Since this setting controls the randomness range of the policy and may contribute to performance differences, We use the TD-MPC2 setting throughout for a fair comparison. This may also have some impact. Notably, even when strictly following the BMPC setting, their performance as reported is still lower or comparable to ours.

4. Summary

We appreciate your feedback and agree that our wording may have been overly critical. We recognize BMPC’s soild contribution in mitigating actor divergence via pure imitation with replanning, and we will revise our text to reflect this.

At the same time, our method achieves stronger time-efficiency and performance.

• Time-efficiency: Both your reproduction results and our experiments indicate that the replanning mechanism used by BMPC incurs 40~50% increase in computational time compared to the variant without replanning.

• Performance: After clarifying the environment step issue, we find that even under strict adherence to BMPC’s parameter settings, the reported performance remains below 700 on Dog-run and below 600 on Humanoid-run, which is either lower than or comparable to ours.

5. Clarify the novelty compared to BMPC

Regarding theoretical grounding, our method is also theoretically well-grounded. In fact, Our framework also favors re-planning, as reflected in the Eq.(3) and supported by our experiments with re-planning—just as you suggested—which improve the performance on the dog-run task from 819.7 to 860.1. By the way, We also highlight that our method maintains strong performance both with and without re-planning, which supports its robustness.

Compared to BMPC, our major novelty lies in explicitly retaining max-Q for policy improvement. And the additional bootstrap alignment we propose is mainly designed to enhance the accuracy of Q-value learning. At the same time, the proposed bootstrap alignment alleviated the reliance on re-planning.

Our central claims is that retaining max-Q enables higher performance than pure imitation as in BMPC. This is because Q-values provide a direct signal of action quality, and even the latest MPPI-generated actions are not strictly optimal (MPPI is a sample-based optimization method and cannot obtain the optimal solution of the objective function, i.e., cumulative return). This suggests that our potential performance ceiling is inherently higher.

6. Conclusion

We hope this clarifies the key points and helps resolve your concerns. We sincerely appreciate your review and would be grateful if you would consider re-evaluating our work. Please feel free to reach out with any further questions or suggestions.

We sincerely appreciate your detailed and constructive feedback. Below are our responses to your concerns.

1. How world model learning is improved in BOOM (Weakness 1)

World model is an important component proposed in this paper as it can be told from the title. However, it didn't receive much attention in the main body of this paper, and the way to use it seems like a direct plug-in to replace the standard dynamics model in model-based planning.

We appreciate your comment and would like to clarify the motivation, structure, and the improvements to the world model in our framework.

Motivation: Our work investigates the challenges and solutions in integrating off-policy RL with planning-driven world model frameworks. One key challenge is actor divergence: the mismatch between the policy and planner, leading the $(s,a)$ (from planner) and $(s',a')$ (from policy since $a'=\pi(s')$) in the TD error follows different distributions, which easily leads to value over-estimation and impairs policy performance. (note that value is a core component of our world model). To mitigate this issue, our proposed BOOM framework leverages a policy-planner alignment mechanism, which not only stabilizes policy learning but also improves the learning of the world model itself.

Structure: Concretely, our world model consists of four components:

• A latent encoder that encodes observations into a compact representation space

• A dynamics model that predicts latent transitions

• A reward predictor that provides rewards used for planning

• A Q-value estimator that enables policy improvement and provides terminal value for planning.

Improvements: The reasons why we improve the learning of these components are:

(1) Q-value learning is improved due to policy-planner consistency. By aligning the policy with the planner, we reduce the discrepancy between the collected data and the actual policy behavior. This improves the distributional matching during training, enabling the value estimator to learn from more consistent and policy-relevant trajectories. As a result, the predicted values used in planning become more accurate and reliable.

(2) Encoder, reward, and dynamics learning are all improved due to a unified TD loss. We adopt a TD-style learning objective that jointly optimizes the value function along with the other components of the world model.

TD-style world model loss = latent dynamics loss + reward loss + Q-value loss

As the value predictions become more accurate due to better distributional match, the gradients flowing into the encoder, dynamics model, and reward predictor become more informative, leading to improved overall model quality.

In summary, although our world model is based on standard components, our framework improves its learning through designed bootstrap alignment mechanism. These enhancements contribute directly to the improved performance and stability observed in BOOM.

2. Comparison with offline RL (Weakness 2)

The entire actor divergence problem is similar to the problem of distribution shift in offline RL. The proposed design is basically a policy regularization by following a very similar way as in offline RL where one tries to regularize the policy update to be closer to the behavior policy for the offline dataset. In particular, the likelihood-free alignment metric in eq.3 has been widely used in offline RL to directly use the offline data without modeling the offline behavior policy

We thank you for raising this important point regarding the similarity between the actor divergence problem in planning-driven MBRL and the distribution shift challenge in offline RL. While there are some surface-level resemblances—such as the use of policy regularization to keep the updated policy close to a behavior policy—the motivations and practical implications differ fundamentally between the two settings.

(1) Focus of algorithm design

• Offline RL centers on a fixed offline dataset, relying heavily on behavior cloning (BC) to restrict the policy within the dataset’s support. To improve policy performance, offline RL cautiously relaxes BC for poor actions, but always aims to keep the policy close to known data to avoid extrapolation error.

• BOOM centers on the policy and seeks to maximize Q-values and align with planner-generated actions, both aimed at improving policy performance. Ultimately, the policy and planner co-adapt and converge to the optimal solution through online interaction.

(2) Relationship between two policy objectives (max Q and min KL)

• In offline RL, maximizing Q-values and BC often conflict: Q-values outside the dataset distribution cannot be accurately estimated, so maximizing Q risks pushing the policy toward unsupported actions; BC pulls the policy back toward known actions. This tension forces a conservative balance.

• In BOOM, the two objectives are largely complementary: the planner generally produces higher-quality actions than the policy. Policy alignment with the planner improves performance and strengthens policy-planner consistency, which in turn leads to more accurate Q estimates. More accurate Q-values then enable better maximization by the policy and allow the planner to generate even higher-quality actions. This positive feedback loop bootstraps policy and planner consistently toward faster convergence to the optimal solution.

This distinction highlights that BOOM is not an offline method regularized around given dataset, but an active, planner-supervised online RL algorithm that enables consistent and progressive improvement. This is also supported by our ablation experiments on the alignment coefficient. In a large range, the performance is good and stable.

Concise summary for quick understanding

## Offline RL
1. (Fixed Dataset)
  Static data collected from a behavior policy.

2. (Policy Regularization) 
  Constraint the policy close to the dataset, while also leaving little room for improvement by maximizing Q-values.

## BOOM (Online RL)
1. (Policy)
Align with planner's higher-quality actions for better behavior than max-Q alone.

2. (World Model)
Better distribution matching improves accuracy.

3. (Planner)
Improved model results in higher-quality actions for sampling.

4. (Policy)
High-quality data enhances policy training and completes the bootstrap loop.

3. Comparison with advantage weighted policy regularization in offline RL (Weakness 3)

The soft Q-weighted mechanism also shares a similar idea advantage weighted policy regularization in offline RL.

For the Advantage-weighting mechanism and offline RL algorithm AWAC, we have supplemented the comparative experiments, ablation analysis and discussion in the response to (the 2nd reviewer HgMu ). Results indicate that advantage-weighting is slightly less effective mainly due to over-selectivity. AWAC performs significantly worse, likely because it relies solely on weighted imitation loss, whereas our BOOM benefits from combining Q-value loss with alignment loss, which is better suited for online RL.

4. Clarify contributions (Q1)

While the performance looks promising, it is hard to think of some new technical contributions from this paper to the research community. I would consider to change my score if the novelty can be well justified.

Thank you for your candid and constructive feedback. While our method builds upon existing techniques in MBRL and offline RL, we believe the primary contribution of our work lies in clearly identifying and effectively addressing a critical yet under-explored issue in planning-driven MBRL — the divergence between the actor and the planner. This problem, though fundamental, has not received adequate attention in prior research.

Our contributions are justified as follows:

(1) Bottleneck identification of planning-based MBRL. We highlight the actor divergence as a under-explored challenge in planning-based MBRL. While similar to distributional shift issues in offline RL, this problem arises even in fully online and model-based contexts, and has distinct implications for model and policy learning.

(2) Simple yet effective solution. We propose a bootstrap mechanism that is easy to implement and integrates seamlessly with existing components, effectively aligning the policy with planner actions to improve stability and performance.

(3) Shift in focus. Rather than improving the structure and training loss of world models or adopting stronger planner, we focus on closing the gap between policy and planner while still maximizing Q-values. This complements existing directions in MBRL and offers a new angle for further performance improvement.

(4) Strong empirical evidence. We validate BOOM on multiple challenging benchmarks, where it consistently delivers stable performance gains. These results support our hypothesis that alignment is key to reliable planning-based learning.

Summary: A minimalist yet powerful approach for planning-driven MBRL. Our method aims to improve planning-driven MBRL through a simple bootstrap alignment mechanism that requires minimal changes to existing components. This straightforward and easy-to-implement approach consistently delivers strong and stable performance gains without introducing complex modifications or additional computational overhead.

We believe this minimalistic yet effective strategy represents a promising direction for further research. We genuinely hope this work can serve as a valuable contribution to the community, inspiring others to explore not only similarly simple yet impactful ideas, but also more sophisticated and powerful solutions in planning-driven MBRL.

5. Conclusion

We have clarified how model learning benefits from our method, elaborate the difference from offline RL and highlight the contributions. We hope our responses address your concerns and kindly request your consideration for a higher score.

Thank you for your detailed and thoughtful comments. We’re sorry that some parts of our previous response may have led to misunderstandings, and we genuinely appreciate the opportunity to clarify them below.

1. Technical novelty

Thank you for your thoughtful feedback and for recognizing our identification of the actor divergence issue. We fully agree with your points regarding the differing motivations and implications of policy regularization in our work compared to that in offline RL. Indeed, although reverse KL-based regularization and Q-weighted coefficients are not new individually, our contribution lies in identifying their specific utility in mitigating actor divergence and enhancing the accuracy of Q-value learning in planning-driven MBRL, which, to our knowledge, has not been systematically addressed in prior work.

2. Comparison with AWAC

We also appreciate your concern regarding the inclusion of AWAC in our experiments. We agree that AWAC is originally designed for offline RL, and that performing bad in an online planning context is expected. While AWAC is not a fully online method, we use it to illustrate how alternative regularization strategies influence performance, and to clarify the distinction between our method and AWAC.

3. Ablation on the weighting mechanism

More importantly, as you rightly pointed out, a more appropriate way to evaluate the effectiveness of our proposed alignment loss is to perform an ablation on the weighting: modifying BOOM by replacing our soft Q-weighted policy regularization with the advantage-weighted policy regularization used in AWAC.

In fact, we have already conducted this exact experiment (in the response to Reviewer HgMu: 3. Ablation on the Advantage-weighting mechanism (Q1)). Due to space limitations in the initial rebuttal, we did not elaborate further. We apologize for the oversight and any confusion it may have caused.

For this suggested ablation, below we present the complete results and insights:

we have explored an alternative weighting scheme based on the advantage function $A(s,a) = Q(s,a) - V(s)$, which is commonly used in actor-critic methods to provide more stable gradients.

Specifically, we implemented an advantage-weighted variant of our method, where $V(s)$ is estimated by averaging multiple samples from the current policy following AWAC. (The python code is included for clarity):

# Approximation of advantage
# Number of policy samples per state to estimate V more accurately
num_pi_samples = 10

with torch.no_grad():
    qs = self.model.Q(zs, actions_beta)

    vs_samples = []
    for _ in range(num_pi_samples):
        sampled_actions_pi = self.model.pi(zs)
        vs_samples.append(self.model.Q(zs, sampled_actions_pi))
    vs = torch.stack(vs_samples, dim=0).mean(dim=0)

    adv = (qs - vs).detach()

While advantage-weighted imitation can offer more stable gradients, we observed a slight ~5% performance drop. We hypothesize that this is because advantage emphasizes the relative quality of planner actions compared to current policy actions, leading to a more selective imitation signal. However, our objective is not only to imitate actions that are strictly better, but to promote consistent alignment between the policy and the planner over time.

As online RL progresses, both the policy and planner co-evolve toward optimality. Overly selective weighting can hinder this mutual alignment, especially in early phases where the planner is not perfect. This observation is consistent with our earlier ablation on the temperature ($\tau$) of soft Q-weight (See the response to the Weakness 1 of Reviewer HgMu), where a smaller $\tau$ also led to degraded performance due to hard-max style weighting.

To conclude, while advantage-weighted imitation offers a more selective filter, our results suggest that such over-selectivity can be counterproductive. It is also worth noting that additional policy sampling is required for advantage estimation. In contrast, the soft Q-weight mechanism provides a better balance between stability and performance. Given the observed performance degradation and the additional computational burden, we find advantage-based weighting to be unsuitable for our setting.

This ablation supports our core claim that the proposed alignment loss—not just the choice of likelihood-free KL, but its soft-Q weighting formulation—is critical for stabilizing policy and improving performance in BOOM.

4. Conclusion

Thank you again for the valuable suggestions. We hope that our clarifications and additional experiments have addressed your concerns, and we would greatly appreciate your reconsideration for a higher score.

We sincerely appreciate your detailed and constructive feedback. Below are our responses to your concerns.

1. Ablation on the soft Q-weight temperature $\tau$ (Weakness 1)

The soft Q-weighting mechanism introduces a temperature hyperparameter $\tau$. In similar softmax-based weighting schemes, this parameter can be highly influential, controlling the "hardness" of the weighting. The paper lacks an ablation study on $\tau$, leaving its sensitivity unevaluated.

We thank you for highlighting the importance of the temperature parameter $\tau$, which controls the sharpness of the softmax distribution over Q-values. A smaller $\tau$ results in a distribution closer to hard-max, while a larger $\tau$ leads to more uniform weighting.

We conducted an ablation study on the Dog-run task by varying $\tau$ from 0.1 to 10.0:

The results show that performance remains stable across a broad range of $\tau$ values. Notably, a slightly higher $\tau$ (e.g., 2.0) can lead to marginal performance improvements. In contrast, setting $\tau$ too low results in performance degradation due to overly selective.

2. Ablation on the gradient clipping norm (Q2)

In equation 7, the bound of the expected Q-value depends on $L_Q$. Have the authors considered explicitly or implicitly controlling this value, such as through gradient clipping or a regularization term?

We agree that gradient clipping is a practical and effective strategy to mitigate instability during Q-function updates. As suggested, we explicitly investigated its role in our setting. Following TD-MPC2, we initially adopted a clipping norm of 20, as listed in Table 2 of Appendix B.2.

To evaluate the sensitivity of our method to this choice, we conducted an ablation study on the Dog-run task by varying gradient clipping norm from 2 to 200:

These results reveal the following:

• A moderate clipping threshold of 20 yields the best performance and aligns with the default setting in TD-MPC2.
• A smaller norm (2) still maintains reasonable stability, suggesting some robustness to tighter control.
• A large norm (200) weakens the effect of clipping, allowing larger gradients that potentially destabilize training, leading to performance degradation.

Overall, this confirms that clipping is an effective and necessary component, and the default choice inherited from TD-MPC2 is already close to optimal.

3. Ablation on the Advantage-weighting mechanism (Q1)

In Figure 4(b), the authors investigated the importance of soft Q-weighting mechanism. Have the authors considered making ablation studies on alternative weighting schemes, such as using the advantage A(s, a), which might provide more stable learning signals?

Yes, we have explored an alternative weighting scheme based on the advantage function $A(s,a) = Q(s,a) - V(s)$, which is commonly used in actor-critic methods to provide more stable gradients.

Specifically, we implemented an advantage-weighted variant of our method, where $V(s)$ is estimated by averaging multiple samples from the current policy (python code included for clarity):

# Approximation of advantage
# Number of policy samples per state to estimate V more accurately
num_pi_samples = 10

with torch.no_grad():
    qs = self.model.Q(zs, actions_beta)

    vs_samples = []
    for _ in range(num_pi_samples):
        sampled_actions_pi = self.model.pi(zs)
        vs_samples.append(self.model.Q(zs, sampled_actions_pi))
    vs = torch.stack(vs_samples, dim=0).mean(dim=0)

    adv = (qs - vs).detach()

While advantage-weighted imitation can offer more stable gradients, we observed a slight ~5% performance drop and a ~2% increase in training time due to additional policy sampling. We hypothesize that this is because advantage emphasizes the relative quality of planner actions compared to current policy actions, leading to a more selective imitation signal. However, our objective is not only to imitate actions that are strictly better, but to promote consistent alignment between the policy and the planner over time.

As online RL progresses, both the policy and planner co-evolve toward optimality. Overly selective weighting can hinder this mutual alignment, especially in early phases where the planner and value are not perfect. This observation is consistent with our earlier ablation on the temperature ($\tau$) of soft Q-weight, where a smaller $\tau$ also led to degraded performance due to more hard-max style weighting.

In summary, while advantage-weighted imitation offers a more selective filter, our results suggest that such over-selectivity can be counterproductive, leading to degraded performance. In contrast, the soft Q-weight mechanism provides a better balance between stability and performance. Given its effectiveness and lower overhead, it serves as a practical and sufficient choice, making advantage-based filtering unnecessary in our setting.

4. Comparison with Offline RL algorithm AWAC (Q3)

Equation (13) in AWAC (https://arxiv.org/abs/2006.09359) proposes a weighted behavior cloning objective that appears similar to Equation (4) in this paper. Could the authors clarify whether these objectives are equivalent or highlight the key differences between them?

(1) Conceptual differences:
AWAC is mainly designed for offline RL, aiming to balance behavioral cloning with performance optimization under the constraint of a fixed dataset. By applying Lagrangian relaxation analysis, AWAC derives a theoretical result stating that the optimal policy satisfies: $\pi^*(a|s) \propto \pi_\beta(a|s)\exp(\frac{1}{\lambda}A(s,a))$. Consequently, at the algorithmic level, AWAC directly relaxes the imitation objective using advantage-based weighting. This approach constrains the learned policy within the dataset coverage while allowing relaxation of imitation on poor actions, enabling slightly better performance than pure behavioral cloning.

$L_\text{AWAC} = \Sigma w_A \text{KL}(\beta || \pi)$

In contrast, BOOM operates in an online RL setting where fresh planner rollouts are continuously available. Both the policy and planner are jointly optimized toward the optimal solution over time, enabling BOOM to focus more on maintaining alignment, rather than solely on selectively imitating only the highest-advantage actions.

$L_\text{BOOM} = Q(s,a_\pi) + \lambda_\text{align} \Sigma w_Q \text{KL}(\beta || \pi)$

(2) Empirical comparison:
Since AWAC fundamentally employs an imitation-style loss similar to BMPC, and BMPC incorporates a re-planning technique that refreshes planner actions in the replay buffer, we extended the comparison by adding re-planning variants for both AWAC and our BOOM (denoted as AWAC + replan and BOOM + replan). This ensures a fair comparison regarding the use of up-to-date planner data for policy alignment.

The Python code is provided for clarity, with the awac_lambda value set to 1.0, following the original paper’s setting.

# AWAC-style weighted imitation loss
weights = torch.exp(adv / self.cfg.awac_lambda)
log_pis_action = self.model.log_pi_action(zs, actions_beta)
pi_loss = (- weights * log_pis_action).mean()

(3) Results analysis: As shown in the results, AWAC performs significantly worse than BOOM (The performance gap is so large that we did not add more ablation experiments to the hyper-parameter awac_lambda). Although both methods benefit from re-planning—with AWAC exhibiting a larger relative improvement—AWAC still lags far behind BOOM in absolute performance. It is important to note that re-planning incurs approximately 50% additional training time due to the overhead of repeated planning. Remarkably, BOOM achieves strong performance even without re-planning. This advantage likely stems from BOOM’s combined use of Q-function loss and imitation loss, which better supports learning in the online RL setting compared to AWAC’s sole reliance on weighted imitation loss.

In summary, although AWAC and our BOOM share conceptual similarities, differences in their practical settings and algorithmic designs make AWAC less suitable for our online planning-driven MBRL framework. Notably, BOOM retains the original Q-function loss alongside its particularly designed bootstrap alignment, which enhances learning effectiveness. Therefore, BOOM offers a more effective and computationally efficient approach for maintaining policy-planner consistency and achieving superior performance.

We appreciate your insightful question and will clarify these distinctions in the revised manuscript.

5. Conclusion

We sincerely appreciate your detailed and constructive feedback. We have added the requested experiments on soft Q-weight temperature, grad clipping norm, Advantage-based weights and comparison with AWAC. We hope our responses address your concerns and kindly request your consideration for a higher score.

We sincerely appreciate your detailed and constructive feedback. Below are our responses to your concerns.

1. Clarify confusing names

This is just the off-policy leaning problem [Sutton& Barton, 1999]. Although not proposed by this paper, why do you need a new name for it? Actor divergence is a confusing name. This issue leads to two problems: (1) Distribution shift in value learning: again this is the classical problem of off-policy learning, see [GTD, TDC, TDRC]. (2) Inevitable Actor DivergenceWhen Off-policy RL Meets Online Planning: this is a confusing subsection title.

Regarding "Actor divergence," we understand that for experienced RL researchers, this term may appear redundant with known off-policy challenges. However, in our planning-driven MBRL setting, we face a unique form of divergence: between the policy and the planner—both of which serve as “actors” but generate data from different distributions. Our goal was to make this divergence intuitive to readers less familiar with classical theory. We will explicitly link this terminology to traditional off-policy learning problem to mitigate misleading.

For the "Distribution shift in value learning," we fully agree that this is a classical off-policy learning problem, extensively studied in foundational works such as GTD, TDC, and TDRC. We will cite the recommended literature to clarify the connection.

Finally, regarding the subsection title "Inevitable Actor Divergence When Off-policy RL Meets Online Planning," our goal was to highlight the intrinsic tension that arises when off-policy RL is combined with model-based planning. We also agree that the title could be clearer—something like “Distribution Shift in Planning-Driven Model-Based RL” may better reflect the core challenge being addressed.

2. Discussion of GTD methods

It’s surprising an off-policy learning missed gradient TD methods, which discusses the sample distribution mismatch, and importance sampling in early RL times.

We acknowledge the importance of GTD, TDC, and TDRC for stable off-policy learning with linear function approximation. While these are foundational, our method uses deep RL with nonlinear networks, where accurate TD learning typically relies on empirical tricks such as target networks. We will include citations and briefly discuss these classical approaches for completeness and historical context.

3. Survey on world models

Section 3.1: this latent dynamics model started from earlier research, e.g., ACE. There is no literature survey (not even discussions) on world models. This is one shortcoming of the paper.

We appreciate this point and will expand related works. Below, we categorize world models into three major types based on their objectives and roles in policy learning:

1. Information-preserving latent models. These methods aim to learn a fully informative latent representation of the environment by combining with auxiliary reconstruction losses. This ensures that the latent space retains all relevant features from high-dimensional observations, supporting accurate imagination. Notably, PlaNet [1] and Dreamer [2] use stochastic recurrent latent dynamics models (e.g., RSSMs), which are trained with an additional decoder to reconstruct the original observations. Although this decoder is discarded at test time, the reconstruction loss regularizes the latent space, improving model stability, generalization. These models are typically used in imagination-driven MBRL, where the policy and value functions are trained entirely on imagined rollouts.

2. Task-aligned latent models trained via one unified TD loss. The latent space in these methods is not required to be fully reconstructive; instead, it only needs to encode information task-aligned critical for decision-making, enabling greater efficiency and stronger performance in high-dimensional control. Early examples include ACE [3], VPN [4] and TreeQN [5], which perform latent tree search over value predictions without decoding observations. More recent approaches like TD-MPC [6] adopts a unified TD loss, where the encoder, dynamics, reward, and value functions are jointly optimized. This family of methods is well-suited for planning-driven MBRL, where latent rollouts are used for online planning.

3. Trajectory-level sequence models for offline RL. A third category treats RL as a sequence modeling problem, without explicitly learning a policy. Instead, models such as Decision Transformer [7] and Trajectory Transformer [8] are trained offline to autoregressively predict future tokens (states, actions, rewards) conditioned on past trajectories and return-to-go. These methods rely on powerful sequence models (e.g., Transformers) to capture behavioral patterns and temporal dependencies. They do not require a separately defined policy network—the Transformer itself acts as a conditional policy, directly outputting the next action given the desired return and current observations. Extending this trend to the pixel level, video generation models like Sora [9] predict future frames directly from visual inputs using large-scale spatiotemporal Transformers. While not trained as agents, they serve as high-capacity world simulators and open new avenues for embodied planning in visually complex domains.

Our approach follows the 2nd category: we adopt a latent model from TD-MPC [6] and improve its learning by introducing a bootstrap alignment mechanism that explicitly encourages consistency between the policy and the planner. This consistency suppresses distributional shift and leads to more accurate model learning.

4. Add tree-search baseline

Another weakness of the paper is that it only compares with DreamerV3, TD-MPC and one of its variants.

Continuous control with Tree Search. This is a good baseline to compare against too.

Following ACE [3], we implement a multi-actor tree-search variant as a new baseline. Specifically, we rollout multiple candidate trajectories by alternately using multiple actors in a tree-unfolding manner, and select the best one before refinement with MPPI. Results are shown below:

Note we now use 5 seeds, so the result of BOOM is updated

As shown, tree search does not bring significant improvements, likely for two main reasons: (1) modern deep RL often uses stochastic policies (e.g., diagonal Gaussian), so multiple actors produce behaviors with limited diversity; (2) even if minor differences exist among actors, the MPPI planner further refines and smooths the actions, resulting in similar outputs. Therefore, tree search yields minimal performance gain while adding additional computational overhead. For these reasons, We choose to adopt a single shared policy for simplicity and efficiency.

5. Clarify small advantages on some domains

Some domains are very strong, including Humanoid-run, Dog-run, H1hand-walk, etc. However, some domains the advantages are small or tied, such as dog-walk, dog-trot, Dog-stand.

Indeed, simpler tasks such as Dog-stand, Dog-walk, and Dog-trot have lower difficulty (specifically, lower speed requirements) and smaller performance gaps across methods. Baselines already achieve near-optimal returns (>900), leaving little headroom for improvement.

Nonetheless, our method maintains performance advantages in these tasks while providing more stable training. Importantly, on the most challenging high-speed run tasks, our approach yields significant gains, highlighting its robustness under harder control demands.

6. Survey on other model-based RL methods

Have you surveyed other model-based DRL algorithms?

Beyond imagination-based and planning-based methods, another emerging paradigm involves synthesizing training data. These methods generate additional trajectoried or transitions to support policy learning. For example, Diffuser [10] uses a diffusion model to iteratively denoise trajectories for planning. Synthetic Experience Replay (SER) [11] generates additional replay data via diffusion-based upsampling. Prioritized Generative Replay (PGR) [12] improves data efficiency by generating high-value transitions independent of the current policy. However, these approaches often rely on training large generative models and have mostly been applied to low-dimensional tasks (e.g., Cheetah, Reacher), without showing consistent gains over standard planning-based methods. For these reasons, we did not include them in our comparison, though we consider them complementary and promising directions for future work.

[1] D. Hafner et al. Learning latent dynamics for planning from pixels. ICML, 2019.

[2] D. Hafner et al. Mastering diverse control tasks through world models. Nature, 2025.

[3] S. Zhang et al. ACE: An Actor Ensemble Algorithm for Continuous Control with Tree Search. AAAI, 2019.

[4] J. Oh et al. Value prediction network. NeurIPS, 2017.

[5] G. Farquhar et al. TreeQN and ATreeC: Differentiable Tree-Structured Models for Deep RL. ICLR, 2018.

[6] N. Hansen et al. TD-MPC2: Scalable, Robust World Models for Continuous Control. ICLR, 2024.

[7] L. Chen et al. Decision Transformer: Reinforcement Learning via Sequence Modeling. Arxiv, 2021.

[8] M. Janner et al. Offline Reinforcement Learning as One Big Sequence Modeling Problem. NeurIPS, 2021.

[9] Brooks T et al. Video generation models as world simulators. OpenAI Blog, 2024.

[10] M. Janner et al. Planning with diffusion for flexible behavior synthesis. ICML, 2022.

[11] C. Lu et al. Synthetic experience replay. NeurIPS, 2023.

[12] R. Wang et al. Prioritized generative replay. ICLR, 2025.

7. Conclusion

We have clarified the confusing names, added the requested tree-search baselines, and more literature reviews on the world model development and other model-based methods. We hope our responses address your concerns and kindly request your consideration for a higher score.