SafeDreamer: Safe Reinforcement Learning with World Models

(Weaknesses #3) Discussion of results. The discusison of results is limited, it is clear all methods achieve the similar performance in terms of costs. However for rewards the three algorithms can perform quite differently across the environments. It would be nice if the authors could adress this, and perhaps explain or at least provide some insight into why one algorithm may be working better than another (w.r.t reward) on a specific task.

Re: Thank you very much for your valuable suggestions. Here is our analysis of each task according the experimental results (see Figure 4 and Appendix C):

1. In PointGoal1, we observed that the cost for OSRP was twice as high as that for OSRP-Lag, which might be attributed to errors in the cost model. OSRP-Lag resulted in a more conservative policy, suggesting that introducing a cost critic through the Lagrangian method can mitigate the errors brought by estimating cost return solely based on the cost model.
2. In RacecarGoal1, due to the larger size of the Racecar, it is more prone to touching obstacles. Therefore, OSRP-Lag tends to be more conservative in this environment compared to the other two algorithms.
3. In PointButton1, where the agent needs to avoid dynamically moving obstacles, we found that OSRP had higher rewards than BSRP and was close to Unsafe DreamerV3. This is because OSRP can utilize the world model for online planning, allowing real-time prediction of dynamic changes in the environment. BSRP, relying solely on the actor network for action output without online planning, struggles to predict real-time dynamic changes in obstacles.
4. In PointPush1, the optimal policy involves the agent learning to wedge its head in the middle gap of the box to push it quickly. We noticed that BSRP could converge to this optimal policy, whereas OSRP and OSRP-Lag had difficulty discovering it during online planning. This may be due to the limited length and number of planning trajectories, making it hard to search for dexterous strategies within a finite time. BSRP's significant advantage in this environment suggests it might be better suited for static environments that require more precise operations.
5. In PointGoal2, with more obstacles in the environment, OSRP-Lag, which introduces a cost critic to estimate cost return, tends to be more conservative than OSRP. This leads to a slower increase in reward for OSRP-Lag. Although OSRP's reward increases as quickly as BSRP's, its cost is higher compared to the other two algorithms, indicating that relying solely on the cost model for planning can result in errors.

(Weaknesses #4) Some references are wrong in section 5.2 these need fixing

Re: Thank you very much for your careful reading. We have updated the correct citations for MBPPO-Lag and DreamerV3 in Section 5.2 and have highlighted them in orange.

Thank you very much for recognizing our experiments. The comprehensive and rich experimental section is a major highlight of our paper. Our response to your comments is as follows, which includes some explanations, additional experiments, and adjustments to the manuscript. If our responses have addressed your concerns, we sincerely hope you would consider adjusting the score accordingly.

(Weaknesses #1) A key problem is there is little dicussion or elaboration about the choices made in this paper and what the alternatives could be. Unfortunately the related work section is relegated to the appendix, while this does not pose an immediate issue I don't find this section to be particularly comprehensive. Yes, there are references to older approaches from the literature, both model-free and model-based. However, there is no discussion on where this research sits in the broader context of safe RL. Not all safe RL research uses the CMDP framework with simple cumulative constraints. It would be nice for the authors to provide a broader overview of safe RL and the problems associated with deploying RL in the realworld. And perhaps discuss and motivate why they only consider simple cumulative constraints and even critique this setup and discuss its limitations.

Re: Thank you very much for your valuable suggestions. In response to your comments, we have made the following adjustments:

1. We have moved the related work to the main paper.
2. We added some other relevant papers in related work, including algorithms outside the CMDP framework, such as those addressing constrained problems from the perspective of energy functions, and so on.
3. We have included additional discussions about SafeRL and its deployment in real-world environments. Similar to your view, we believe that SafeRL can facilitate the deployment of RL in the real world, and we have further incorporated this discussion into Section 6: Conclusion.

(Weaknesses #2) Originality. All three algorithms are essentially combinations of existing methods, DreamerV3, PID Lagrangian, CCEM and the Augmented Lagrangian. This is not an issue per se, since this is common for RL research. However, very little discussion is provided for why these methods have been chosen and what the possible alternatives are from the literature and if these may obtain better results or not.

Re: In the context of model-based RL, there are generally two ways of utilizing a world model.

1. One is to use the world model for online planning.
2. The other is to use trajectories rolled out by the world model for offline policy optimization, which in this work is referred to as 'background planning'.

In this work, our main argument is that the world model can facilitate the achievement of zero cost in safe RL, whether through online planning or background planning, as demonstrated by our algorithms OSRP and BSRP-Lag. BSRP-Lag employs an augmented Lagrangian for policy optimization to optimize the action distribution generated by the actor. Conversely, OSRP uses CCEM in conjunction with the world model for online optimization of the action distribution. However, due to the constraints of computational resources and errors in the cost model, online planning is limited in its ability to balance long-term safety and rewards. To address this, we introduced Lagrangian and critics into online planning, resulting in OSRP-Lag.

Ultimately, the three world model-based algorithms we propose achieve nearly zero cost with both vector and visual inputs, a feat not accomplished by previous work.

We argue that the choice of Lagrangian method is not the key to achieving zero cost. The crucial factor is how to accurately model safety-relevant states using the world model. As the environment becomes more complex, the error in each component of the world model also increases. Therefore, it's necessary to adaptively modify to reconcile variances in signal magnitudes and ubiquitous instabilities within all model elements. Additionally, we have incorporated an ablation study in Appendix C.5 to examine the impact of various components of the SafeDreamer on its performance. We believe our work will serve as a reference for the SafeRL community in achieving zero cost.

I really do appreciate the time and effort you have put into this rebuttal period. Several things are a lot clearer to me now and this work seems particularly promising. I still need to check whether the revised paper reflects the changes I wanted to see and I will update my score in due course if I am happy with the revised paper. In mean meantime I have one more slight confusion that needs clearing up. See above (or below).

(Question #3) Would you consider incorporating the Bayesian method from (As et al. 2022)? And how would you expect this to affect performance?

Re: We regard LAMBDA as a highly significant paper in the realm of safe model-based RL, and it was indeed the inspiration behind our development of SafeDreamer. We believe that using a Bayesian method to sample multiple world model parameters for policy optimization can enhance policy performance in dexterous tasks, such as the Point Push task. This is because it effectively allows training across different world models, somewhat akin to the domain randomization concept frequently used in sim2real. Different world models might represent varying physical parameters, such as a vehicle's maximum speed or friction. Consequently, this approach can lead to more robust policy improvements. We think that when an environment is sufficiently complex and the world model has been exposed to a diverse range of data distributions, the sampled world models could represent these different distributions, thereby training more robust policies. However, given that current environments like Safety Gym are relatively simple, we believe this method will play a crucial role when faced with more complex future environments.

(Question #4) In your implementation, is there a reason for the cost predictor head being a real valued predictor rather than a binary classifier? And what is the effect of scaling the costs by 10.0 (line 208 agent.py), is this necessary to learn a good cost predictor?

Re: We considered that in some environments, the cost is not merely binary but can be real-valued. Anticipating future research expansions into more varied environments, we opted to use real-valued costs. During the training of the cost head, we employed discrete regression on twohot encoded targets, effectively transforming the regression problem into a classification problem. In environments where the cost is binary, a real-valued predictor with discrete regression training is, to some extent, equivalent to a binary classifier. Moreover, in the Safety-Gym environment, the cost is sparsely distributed. A common approach to predict sparse and imbalanced data is to use weighted loss. Applying a scaling factor to sparse values is essentially equivalent to increasing their frequency of occurrence in the dataset. In our experiments, we found that this approach aids in the convergence of the cost head. It's worth mentioning that LAMBDA[1] has used such techniques, which also inspired us: https://github.com/yardenas/la-mbda/blob/master/la_mbda/models.py#L124

[1] As, Y., Usmanova, I., Curi, S., & Krause, A. (2022). Constrained policy optimization via bayesian world models.

(Concern #1) Thanks. I agree that taking adopting the Bayesian paradigm will be important future work. Just to clarify, scaling the costs by 10.0 was the best configuration? Correct me if I'm wrong but in LAMBDA they do something similar to balance the dataset but find that 50.0 was the best configuration? Again this may be me misunderstanding, but this isn't reflected in the hyperparameters, $\beta_c = 1.0$, but shouldn't it then be $\beta=10$? This seems like a fairly crucial component of the model, since learning an accurate cost function (predictor head) is paramount? It might be nice to see an ablation study for this value.

Re: Thank you very much for your valuable suggestion. We feel your deep understanding of the Model-based RL and Safe RL domains, and find your advice to be very insightful and informative. Indeed, in LAMBDA, setting the value to 50 is to balance the dataset. However, in SafeDreamer, setting it to 50 is not the optimal choice. We conducted detailed ablation studies on this parameter, the results of which can be seen at

1. https://sites.google.com/view/safedreamer#:~:text=Ablation%20studies%20on-,the,-weight%20of%20the%20cost%20model%20loss%20in%20SafetyPointGoal1
2. https://sites.google.com/view/safedreamer#:~:text=Ablation%20studies%20on%20the-,weight,-of%20the%20cost%20model%20loss%20in%20SafetyPointGoal2

The related content has also been updated in the revised version of our paper, see Appendix C.5 (Figure 26 and Figure 27).

(Concern #2) Thank you these insights are actually very helpful for understanding the tradeoffs between the different algorithms. It would be nice to something like this in the paper.

Re: Thank you very much for your valuable suggestion. Following your advice, we have further updated the latest revision of our paper to include detailed ablation experiments on the hyperparameter $\beta$. These results have been updated in Appendix C.5. Additionally, in response to “understanding the tradeoffs between different algorithms”, we have added an extra paragraph in Section 6.3:Results, titled "Performance Analysis under Different Tasks". This section describes insights gained from extensive experiments, including the performance of algorithms in different settings and the impact of various algorithmic components on performance.

Thank you very much for your thorough review and for your patient communication with us. We are eagerly looking forward to your approval!

We sincerely appreciate your recognition of our work and the increase in our score. It is an honor for us to address your concerns, and your invaluable insights will be integrated into the final revised version.

Thank you very much for your thorough review and valuable suggestions. In response to each of your points, we have provided detailed replies and made corresponding adjustments to our paper. These include adding new experimental environments, revising the paragraph, and modifying the algorithm's description. We sincerely hope that our responses can address your concerns, and you can adjust your score accordingly.

(Weaknesses #1) While the paper compares SafeDreamer with several existing SafeRL methods, it lacks a comprehensive comparison with a broader range of related work in the field. A more extensive comparison could provide a clearer context for SafeDreamer's contributions and limitations. For example, the authors noticed the failure of achieving zero cost of Lagrangian methods and CPO and concluded the reason behind it is the lack of long-horizon planning. Actually there are several literatures [1, 2] that have discussed this issue from the perspective of cost function design. In general, more informative cost functions could solve this issue of not achieving zero costs of Lagrangian methods and CPO.

Re: Thank you very much for your valuable suggestions. Following your advice, we have expanded the Related Work section, and moved the Related Work from Appendix A to the main paper. We have carefully readed the two works [1][2] and found that their focus on Cost Function design aligns with our view that existing methods such as CPO and Lagrangian approaches cannot achieve zero cost. We have now included these works in the Related Work section and have discussed them there.

[1] Ma, H., Liu, C., Li, S. E., Zheng, S., & Chen, J. (2022, May). Joint synthesis of safety certificate and safe control policy using constrained reinforcement learning. In Learning for Dynamics and Control Conference (pp. 97-109). PMLR.

[2] He, T., Zhao, W., & Liu, C. (2023). Autocost: Evolving intrinsic cost for zero-violation reinforcement learning. arXiv preprint arXiv:2301.10339.

(Question #1) I might have missed some details. But why the blues in Figure 5 is not complete? The robustness of converged performance is worth reporting. Also, I am curious to see whether OSRP-Lag is able to converge to zero costs.

Re: I'm very sorry for causing the misunderstanding. The point we aim to convey through this figure is that even when model-free algorithms are given more iterations, they fail to achieve nearly zero-cost. Given that model-based planning is more time-consuming compared to model-free algorithms, we did not run as many steps as the model-free algorithms. Additionally, OSRP-Lag does not converge to zero cost; we only claim it achieves nearly zero cost. In environments with inherent randomness, utilizing a world model indeed ensures that the final converging policy has a lower cost. However, due to model errors and limited data availability, the world model can't account for every possible situation, making it difficult to converge to zero cost. A potential solution lies in incorporating pessimistic estimates during policy learning, allowing for conservative strategies even in unseen scenarios. Finally, we have showcased the algorithm's training curve at 8M steps in appendix C.5. Due to time and computational resource constraints, we only ran this for the PointGoal1 experiment from scratch. If you could consider increasing your score, we will update the training curves for all environments at 8M steps in ready versions. More details can be referred to Figure 21 in appendix C.5.

(Question #2) Why choose different cost thresholds for different environments in Figure 6? I think the threshold of value 0 makes the most sense to me. Also, I noticed that OSRP and OSRP-Lag seem to be too conservative when the cost threshold is around 14 in SafetyPointButton1, which might reveal the imbalance between performance and safety satisfaction.

Re: Firstly, we apologize for the misunderstanding and thank you for your careful reading. We did not choose different cost thresholds for different environments. In Figure 6, the cost thresholds for all environments are set to 2.0. Regarding your observation that 'OSRP and OSRP-Lag seem to be too conservative when the cost threshold is around 14 in SafetyPointButton1', we believe you might have mistaken the dashed line of the baseline algorithm MPC:sim for the cost threshold.

The reason we did not set the cost threshold to zero is due to potential errors in the trained cost model and cost value model. Therefore, even in a completely safe state, the estimated cost return by these models might not be zero but a very small positive value. This implies that any trajectory in the planning process may not meet the constraints based on these estimations, making online planning overly conservative and unable to converge.

(MAJOR-Baselines #2) It is not clear how exactly the baselines from Ray 2019 were modified for the vision-based partial observations (Ray 2019 uses state-vector observations). According to the caption of Fig 4, it seems like the exact experimental procedure of Ray 2019 was used. This possibly implies that no CNN (to handle the image observations) and RNN (to handle the partial observations) were used for the model-free baselines. Meanwhile, SafeDreamer uses recurrence in its model and possibly also uses a CNN (whether it uses a CNN or not was also not stated in the paper, so I can only guess). This is a severe concern that could actually explain why almost all of the model-free baselines have extremely poor performance inconsistent with prior works.

We did not add a vision module (such as CNN or RNN) to the Model Free algorithms. A major advantage of SafeDreamer is its ability to support both vector and visual inputs. In our comparisons with Model Free algorithms, we only used vector inputs. We tried adding specific vision modules to Model Free algorithms, but the performance was not satisfactory. Extending existing Model Free Safe RL algorithms to support visual input scenarios presents significant challenges. Additionally, we have added detailed descriptions of the hyperparameters and network structures for both Model Free and Model Based algorithms. For more details, please refer to Appendix B (see Tables 2, 3, 4, 5, 6). Regarding the statement, 'This is a severe concern that could actually explain why almost all of the model-free baselines have extremely poor performance inconsistent with prior works,' prior works [1][2][3][4] used a setting of cost limit = 25.0. Under this more relaxed safety constraint, they were able to achieve more satisfactory performance. However, our paper sets the cost limit = 2.0, which is very close to zero cost, posing a significant challenge to existing algorithms. In situations with a low cost limit, this leads to the Lagrangian multiplier continuously penalizing the policy, resulting in overly conservative strategies – that is, situations with low costs and also low rewards, which is the reason for the Poor Performance noted in our paper.

Once again, we reiterate: The experiments in our paper are based on factual experimentation and are fair and comparable. We strictly respect the scientific norms of research paper submission!

[1] Ray, A., Achiam, J., & Amodei, D. (2019). Benchmarking safe exploration in deep reinforcement learning.

[2] Zhang, Y., Vuong, Q., & Ross, K. (2020). First order constrained optimization in policy space. Advances in Neural Information Processing Systems, 33, 15338-15349.

[3] Yang, L., Ji, J., Dai, J., Zhang, L., Zhou, B., Li, P., ... & Pan, G. (2022). Constrained update projection approach to safe policy optimization. Advances in Neural Information Processing Systems, 35, 9111-9124.

[4] As, Y., Usmanova, I., Curi, S., & Krause, A. (2022). Constrained policy optimization via bayesian world models. Eleventh International Conference on Learning Representations.

(MAJOR-Baselines #3) The model-free baselines are extremely old relative to the model-based baselines. There are a number of relevant recent model-free baselines like Saute RL and ROSARL that also aim to achieve near-zero cost.

Re: Your suggestion is valuable, and within the limited time available for our rebuttal, we have added the SauteRL you recommended, including PPO-Saute and TRPO-Saute, with their cost limits set to 2.0. Detailed hyperparameters are provided in Appendix B. However, due to time constraints and limited computing resources, we were unable to add ROSARL as a baseline, which would require certain modifications to ROSARL, including:

1. Changing the Safety Gym setting to reset the episode upon collision with an obstacle, along with modifications to the reward settings. The original text states: We modify the environment so that any collision with a hazard results in episode termination with a reward of −1, thereby making the problem much harder.
2. Fixing the position of obstacles so that their location does not update with each episode reset. The original text states: The goal’s location is randomly reset when the agent reaches it, while the locations of the obstacles remain unchanged.

We hope you can see our commitment and take every one of your suggestions as valuable input. If you could consider increasing our score, we promise to fully incorporate the two baselines you suggested in the ready version.

We are very grateful for your recognition of the experimental part of our work and also for your thorough review and valuable suggestions. We have provided detailed responses to each of your comments and have made corresponding revisions to the paper. This includes adding new experimental environments, correcting fuzzy expressions, and adjusting the layout of the main paper and appendix.

We sincerely hope that our responses can address your concerns, and that you could consider adjusting your score. We would be immensely grateful for this.

(MAJOR-Contributions #1) SafeDreamer (OSRP, OSRP-Lag, and BSRP-Lag) seems like a straightforward combination of prior works (Dreamerv3, CCEM, Augmented Lagrangian, and PID Lagrangian). For example, if I understand correctly, OSRP (Sec 3.2) is exactly CCEM but using the TD(\lambda) objectives from DreamerV3 (for the cost and rewards estimates). It is not clear what are nuances that make this combination not as straightforward as one would expect. No theory nor empirical analysis is given to gain better insights into the behavior of SafeDreamer when all these components are combined.

Re: Recently, many works have been integrating traditional MPC (Model Predictive Control)-Based methods with TD (Temporal Difference) learning. Examples like TDMPC[1] and LOOP[2] represent a combination of MPC with TD learning, known as MPC with a terminal value function. However, directly transferring these methods to the SafeRL domain presents challenges.

1. Due to the need to meet safety constraints, it's crucial for the world model to accurately predict changes in safety-related states, such as the relative position of obstacles to the agent. The prediction of obstacle positions can be particularly problematic; a single incorrect prediction can lead to a cascade of errors in subsequent position predictions. Therefore, using methods like CCEM to calculate the total cost of a trajectory might only yield accurate cost estimates for the initial steps. In approaches like TDMPC or LOOP, which use a value model to estimate future costs at the terminal step of planning, the accumulated error in the final step's state can be significant, leading to large inaccuracies in cost value estimation. Instead, we use TD-lambda, where each step's cost estimation depends not only on the cost model but also on the predicted cost value.
2. Traditional CCEM requires a ground truth simulator to perform safe planning on ground truth states. However, in visual tasks, accurately predicting every pixel's change is challenging and resource-intensive. Our method involves planning on the latent hidden state, which avoids the need for precise pixel predictions. Experimental evidence shows that this approach can achieve nearly zero-cost performance, indicating that the latent hidden state implicitly captures almost all safety-related visual information. We believe that conducting safe planning on the latent hidden state could offer new insight for the SafeRL community.
3. We have added an ablation study of SafeDreamer's design and configuration choices in Appendix C.5.

[1] Hansen, N., Wang, X., & Su, H. (2022). Temporal difference learning for model predictive control.

[2] Sikchi, H., Zhou, W., & Held, D. (2022, January). Learning off-policy with online planning. In Conference on Robot Learning (pp. 1622-1633). PMLR.

(MAJOR-Contributions #2) While the shown empirical evaluations of SafeDreamer reporting great reward and cost performances in SafetyGym tasks are decent contributions by themselves, I think they are not sufficiently significant for publication. Some theoretical/empirical analysis of SafeDreamer and evaluations in other domains (other than SafetyGym) would improve the contributions.

Re: Thank you for your valuable suggestions. To further validate our method, we conducted additional experiments beyond Safety-Gym, incorporating tests in the Safe Drive scenario, focused on safe autonomous driving tasks. Specifically, we conducted supplementary tests of the SafeDreamer in three tasks on MetaDrive, Car-Racing, and FormulaOne. The results demonstrate that SafeDreamer significantly outperforms the baseline algorithms. We have updated the descriptions of these environments and the corresponding experimental data in the revised version of our manuscript. For more details, please refer to Appendix C.1 - C.3. Some video demos can be seen in https://sites.google.com/view/safedreamer#:~:text=SafeDreamer-,MetaDrive,-SafeDreamer

(Additional Concerns #4) 3- I am very familiar with the baselines used in Ray 2019, and I know that they all still maximise rewards (at different rates) even with a near-zero cost threshold (but with very poor cost performance). Admittedly, there may be aspects of the specific setting here that make them fail so badly and weirdly. The given possible explanation using the Lagrangian doesn't seem correct to me either. Can the authors report results using a cost threshold of 25 for sanity check? Since that is the same cost threshold used in Ray 2019, that should give similar results to Ray 2019 for the model-free baselines, justifying the authors' hypothesis to an acceptable degree.

Re: From your comments, we can sense your deep familiarity with SafeRL and Safety Gym[2]. We want to clarify that we did not intentionally make unfair adjustments or subjectively bias the Model-free baseline. We opted not to use the baseline provided by Safety Gym[2], Safety-Start-Agent[1], but instead used OmniSafe[3], for the following reasons:

1. Safety-Start-Agent is currently based on TensorFlow 1.X, which posed some compatibility issues on our 4090 servers, although it is still runnable.
2. Safety-Start-Agent currently includes only three SafeRL algorithms (CPO, PPO-Lag, TRPO-Lag) and PPO and TRPO, whereas OmniSafe encompasses a wider range of algorithms and is based on Pytorch, making it more user-friendly and easier to replicate.

Regarding your viewpoint, we strongly agree. Yesterday, we reran the three SafeRL algorithms from Safety-Start-Agent, and the results were as you described: “they all still maximize rewards (at different rates) even with a near-zero cost threshold (but with very poor cost performance).” We believe this phenomenon can be explained. Considering that the goal of SafeRL is to maximize rewards while meeting constraints, both high rewards without meeting constraints and low rewards with constraints are actually signs of the algorithm’s failure. We attribute this to the control of the algorithm's hyperparameters.

[1] Safety-Starter-Agent: A companion repo to the paper "Benchmarking Safe Exploration in Deep Reinforcement Learning" URL: https://github.com/openai/safety-starter-agents

[2] Safety Gym: https://github.com/openai/safety-gym

[3] Ji, J., Zhou, J., Zhang, B., Dai, J., Pan, X., Sun, R., ... & Yang, Y. (2023). OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Research. arXiv preprint arXiv:2305.09304. OmniSafe Repo: https://github.com/PKU-Alignment/omnisafe

The open-source repository of OmniSafe has already garnered over 700 stars (URL: https://github.com/PKU-Alignment/omnisafe), and we have observed that its Issue and Discussion sections are quite active. Hence, it has been recognized to a certain extent by the SafeRL community.

About:

Since that is the same cost threshold used in Ray 2019, that should give similar results to Ray 2019 for the model-free baselines, justifying the authors’ hypothesis to an acceptable degree.

Re: We acknowledge and agree with your suggestion, and believe that including this in our paper will make the research more solid. Based on the setting of a cost threshold of 25.0, we have added experiments in two environments within the limited time available.

Notes: We commit to completing experiments in other environments and will report these in detail in our paper.

Partial experimental results are available at: https://sites.google.com/view/safedreamer#:~:text=limit%3D25%20in-,PoinGoal1,-(Low%2Ddimensional).

Due to time constraints, our testing was limited to the PointGoal1 environment. We observed that the baseline's trend aligns closely with the Safety-Gym paper: the baseline's cost began to converge around 25, while maintaining a high level of reward, consistent with Figure 7 in the original Safety Gym paper. On the other hand, SafeDreamer was still able to converge rapidly to zero cost, even with a cost limit set at 25. We attribute this to the following reasons:

1. Given the relative simplicity of the PointGoal1 environment, SafeDreamer can quickly identify the optimal strategy that meets constraints by relying on accurate modeling of the World Model and planning within it (sorting candidate trajectories by cost).
2. A cost near zero is also a solution for cost ≤ 25, and its convergence to zero also indirectly reflects the superiority of the World Model.

Notes: Dear reviewer 96af, we did not intend to deliberately lower the performance of the Baseline to highlight the superiority of SafeDreamer. We firmly believe that such actions would not be in line with scientific norms and research ethics. We greatly hope to gain your professional endorsement. If you are satisfied with our revised version and our responses, we hope you could consider adjusting the score accordingly.

Thank you very much for your thorough review and valuable suggestions. In response to each of your points, we have provided detailed replies and made corresponding adjustments to our paper. These include adding new experimental environments, revising the paragraph, and modifying the algorithm's description. We sincerely hope that our responses can address your concerns, and you can adjust your score accordingly.

(Weakness #1) The paper proposes OSPR, OSRP-Lag and BSRP-Lag but the differences between them is not well motivated and presented.

Re: In the realm of model-based RL, there are generally two ways to utilize a world model

1. one involves using the world model for online planning. This approach allows the use of the world model for inference during testing, thereby offering enhanced adaptability to dynamically changing environments.
2. the other involves using the world model’s rollouts for offline policy optimization, referred to as background planning in our work. This method does not require the use of a world model for online planning during testing, thus reducing computational overhead.

Online planning denotes the real-time optimization of action distributions by the agent through the world model during interaction with the environment. In contrast, background planning refers to training an actor offline using the world model, which is then employed to output action distributions when interacting with the real environment. In this work, we aim to articulate the viewpoint that the world model can facilitate the implementation of SafeRL at nearly zero cost, whether through online planning or background planning. This is substantiated by the performance of our algorithms: OSRP and BSRP-Lag.

1. BSRP-Lag employs augmented lagrangian methods for policy optimization to enhance the action distributions produced by the actor.
2. OSRP uses the constrained cross-entropy method (CCEM) in conjunction with the world model for online optimization of action distributions.

Given the computational resource constraints that limit the online planning horizon, it's challenging to consider long-term trade-offs between rewards and costs. Therefore, we introduce the Lagrangian method in online planning to balance these long-term costs and rewards, as demonstrated in OSRP-Lag.

In our experiments, we observed that OSRP performs better in complex, dynamically changing environments, while BSRP excels in static environments requiring dexterous action. For instance, in PointButton1, where the agent must avoid dynamically moving obstacles, OSRP achieved higher rewards compared to BSRP, nearing the DreamerV3 (only considering the reward). This is attributable to OSRP's ability to use the world model for online planning, enabling real-time prediction of environmental dynamics. In contrast, BSRP, relying solely on action outputs from the actor network without online planning, struggles to predict the real-time dynamics of obstacles. Furthermore, in PointPush1, the optimal policy involves the agent learning to quickly push the box by wedging its head in the middle gap of the box. BSRP converged to this optimal policy, whereas OSRP and OSRP-Lag found it challenging to discover this approach through online planning, likely due to the limited length and number of planning trajectories, hindering the discovery of dexterous strategies within a finite time. The significant advantage of BSRP in this scenario suggests that it may be better suited for static environments requiring dexterous operations.

Ultimately, the three world model-based algorithms achieve nearly zero-cost with both vector and visual inputs, a feat not previously accomplished in prior works.

(Weakness #2) In Section 3.2, it seems the sampled trajectories contain two parts, one set of the trajs are deduced from a Normal action distribution and another set is from current policy. This part is confusing and not well presented. The notation is not aligned between SafeDreamer paragraph and the Algorithm 1. I would recommend rephrase the method part and mark the Algorithm line number in the text.

Re: Thank you very much for the valuable suggestion. Indeed, the action trajectory is generated by both the normal action distribution and the actor network. We have employed techniques similar to LOOP [1] and TDMPC [2]. By integrating a small number of actions generated by the actor network into the action trajectory produced by the Gaussian distribution, we can guide the planning toward a more rapid convergence. We have rephrased the method section in the main paper and marked the algorithm line numbers for clarity.

[1] Hansen, N., Wang, X., & Su, H. (2022). Temporal difference learning for model predictive control.

[2] Sikchi, H., Zhou, W., & Held, D. (2022, January). Learning off-policy with online planning. In Conference on Robot Learning (pp. 1622-1633). PMLR.