FOSP: Fine-tuning Offline Safe Policy through World Models

We are very grateful to the reviewer for pointing out some of the weaknesses it considers. We arewilling to respond to each of the weaknesses and also answer the reviewer's questions one by one.

Weaknesses:

1. In Figure 4, the results for DreamerV3 could be omitted.

We appreciate the reviewer’s suggestion. We have included the omitted images in Appendix Figure 14.

2. There are too few obstacles in the real-world environment, making it difficult to assess the agent's obstacle avoidance behavior. The website does not provide any differences between the proposed algorithm and the baseline in the video demos.

We admit that real-world experiments are relatively simple. However, it is hard for us to add obstacles to the experiment because the robotic workspace is limited. Since its end effector is big enough, we need to leave enough space in the workspace for it to perform tasks. Besides, because of the limited height of the workspace, the robotic arm needs to plan a complex trajectory to avoid collision obstacles. Therefore, to some extent, it reflects the method’s ability to avoid obstacles. The website has showcased videos comparing the proposed algorithm with the baseline. The differences are the same as Figure 12 in the paper. The baseline encounters issues with constraint violations and planning difficulties, while our approach effectively completes the tasks.

3. Some questions in Figure 4.

Figure 4 (Figure 3 in the new version) has been smoothed so it seems like FOSP does not show improvement in PointGoal2 and PointGoal1.

Here are some evaluation results about FOSP fine-tuning on PointGoal2 and PointGoal1:

The results indicate that in these two environments, there is indeed a slight increase, though not substantial. The cost also has been decreased. So, it still highlights the importance of fine-tuning in FOSP. As for the reason why the improvement isn't very significant, it is true that the policy trained on offline data is not optimal. But PointGoal1 and PointGoal2 are relatively simple compared to other tasks. Thus, it is possible that the policy can achieve near-optimal performance in relatively simple environments by relying solely on offline data.

As for the experiments for SafeDreamer in PointButton1, although there was no significant improvement in performance, the fine-tuning phase still helped reduce the cost:

Questions:

1. About the videos of the SafetyFadingEasy1 and Hard1

The videos are presented at https://anonymous.4open.science/r/PointFading_video-F34F In a way, these two environments indeed tested the agent's memory capabilities. The model (Dreamerv3, which our method is based on) does possess some memory capability. The deterministic part of the dynamic model allows it to remember information over many time steps [1]. However, relying solely on the model's memory capacity to generalize across these two environments is challenging (SafeDreamer, which includes a similar design, does not perform as well as our method, as shown in Figure 17). We should focus on the increasing of task difficulty. During the fine-tuning process, with the disappearance of the goals, the agent needs to quickly find a path to reach the goal. And our method can quickly capture this ability. We have added it to Appendix H.3.

2. Why do we use $Q(s,a)$ instead of $V(s)$?

It is common in reinforcement learning that use state value function $V(s)$ and action-state value function $Q(s,a)$ to optimize the problem. The differences between them are $V(s)$ can evaluate which states are more desirable but not directly decide the action and $Q(s,a)$ can directly choose which action to take in any state by selecting the action with the highest value. These two approaches are equivalent since $V^*(s) = \max_{a} Q^*(s,a)$ (* represents optimal). While Dreamerv3 uses $V(s)$, it only considers state value to adjust its policy. In contrast, we leverage $Q(s,a)$ as the critic to focus on the state-action pairs’ value and directly optimize the policy.

3. About the image-based setting adaptation for Recovery RL

Please refer to the part on Recovery RL in Appendix F. We use the same method as [3] by using a VAE-based model to capture the high-dimensional information.

4. About the modification of SafeDreamer for offline setting

We do not modify it to the offline setting because it will become a new algorithm if we design it as an offline algorithm. Our experiments provide a straightforward validation that our method outperforms SafeDreamer. Additionally, the reviewer may ask how we can be certain whether the improvement in FOSP's performance is solely due to the offline design (since no offline modifications were made to SafeDreamer), and how the other design components contribute. We have answered this question and conducted ablation studies in Section 5.3 to demonstrate that each module contributes to FOSP's performance. Given that FOSP is built upon SafeDreamer, we can confidently assert that each module in our algorithm is necessary for its effectiveness.

References:

[1] Hafner et al., "Learning Latent Dynamics for Planning from Pixels", 2018.

[2] Kostrikov et al. "Offline reinforcement learning with implicit q-learning." arXiv preprint arXiv:2110.06169 (2021).

[3] Brijen et al. “Recovery rl: Safe reinforcement learning with learned recovery zones.” IEEE Robotics and Automation Letters, 6 (3):4915–4922, 2021.

The Figure 14 still overshadow the cost performance of the main baselines(eg. safedreamer, recovery RL).

We are very grateful to the reviewer for pointing out some of the weaknesses it considers. We are willing to respond to each of the weaknesses and also answer the reviewer's questions one by one.

Weaknesses:

1. Confusing presentation

We are sorry about the confusion. This paper intends to train a vision-based safe policy that can leverage offline data and quickly adapt to novel environments while maintaining safety properties. It’s a non-trivial problem because real-world scenarios often differ significantly from the exact conditions represented in the dataset. Prior works either struggle with limitations in offline learning or face challenges in safe policy fine-tuning. Our main contribution is proposing a new framework to solve this problem. The most promising previous work is SafeDreamer, which is a strong vision-only model-based safe RL algorithm. However, SafeDreamer has limitations in offline learning. First, it suffers from value overestimation that will result in an inaccurate value estimation. Second, its policy learning approach struggles to handle offline constraints (i.e., the policy needs to stay close to the behavior policy) and safety constraints simultaneously, leading to suboptimal task performance and safety trade-offs. Third, SafeDreamer is unable to safely generalize its offline trained policy during online fine-tuning. To address these problems, we propose several solutions building upon SafeDreamer. We first leverage IQL [1] to address the value overestimation and make sure it can learn an accurate Q-value. To ensure safety while not degrading performance, we introduce the reachability estimation function from RESPO [2] and derive a new safe reinforcement learning algorithm that can be adapted to our framework. Inspired by PEX [3], we propose a safe adaptation of its methodology, enabling robust fine-tuning. These components are seamlessly integrated into a unified framework, ensuring compatibility and synergy among the methods. The experiments demonstrate that the proposed method can solve safe offline-to-online and safe generalization tasks. Our framework is novel and we made some technical contributions to the algorithm (e.g., deriving safe RL based on REF, a safe version of PEX) to ensure they can solve our problem.

2. Why would FOSP outperform SafeDreamer

SafeDreamer is an online safe model-based RL algorithm. It is used to improve vision-based tasks in safe RL. Our method is better than SafeDreamer according to the analysis in Weakness 1. Furthermore, we can analyze the outperforming through experiments from two perspectives:

1. The offline-to-online tasks: we use the same method to train FOSP and SafeDreamer, i.e., offline to online (Table 1). SafeDreamer suffers from the distribution shift in the offline setting and it cannot well handle the constraints during offline training. So, it cannot perform well at the initial stage of fine-tuning, which would hinder its subsequent fine-tuning. Meanwhile, SafeDreamer lacks an offline-to-online design, making it difficult to fine-tune the RL policy.

2. The online tasks: we conduct online training only for SafeDreamer, while FOSP undergoes offline-to-online training (Table 2). For a fair comparison, FOSP's online training steps are the same as SafeDreamer's online training steps (just like in the real world, offline training does not require any cost). When SafeDreamer trains only online, it lacks the safety guidance from the offline dataset. It also struggles with step limits as it always requires a large number of training steps to achieve optimal performance [4]. In contrast, FOSP can take advantage of both the offline dataset and the fine-tuning phase.

It is why FOSP outperforms SafeDreamer.

3. Are all the design choices necessary?

Yes, the experimental results have been shown in Section 5.3 and Figure 3 (Figure 4 in the new version).

Questions:

1. About the behavior policy

The definition of the behavior policy can be found in “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems” [5] Figure 1. Precisely, behavior policy is the policy used when collecting datasets. It is not the offline RL policy. In our experiment, we used three kinds of policies (safe, unsafe, random) to collect datasets so the behavior policy could be a mixture of these three policies.

2. The training step in Section 4.2

To help the reviewer better understand the method, please see the pseudo-code (Appendix C). Section 4.2 provides a safe RL algorithm, which can be used offline and online to ensure safety. We divided the offline training phase into Sections 4.1 and 4.2, as Section 4.1 primarily addresses value estimation, while Section 4.2 introduces a method for training a safe policy based on the critic training approach outlined in Section 4.1. Dividing into two parts can help readers to understand how we solve different problems in different ways.

3. The reachability estimation function in Sec. 4.2

We appreciate the suggestion to introduce the RESPO in the Preliminaries. The advantage of using the reachability estimation function is to ensure the policy can learn how to reenter the feasible region when the agent is in the infeasible region (shown in Figure 2). It can help the policy learn more effectively and balance the trade-off between its performance and cost (Figure 3 (Figure 4 in the new version) demonstrates the advantage of REF). $u^{\pi}(s)$ do not equal to $1(s \in S_f)$ as it is actually an expectation. We would like to thank the reviewer for pointing out it since this definition could be difficult to understand. So we replace it with $u^\pi(s) = \mathbb{E}\_{\tau\sim\pi}[\max\_{s\_t\in \tau}(1 - 1((s_t|s_0, \pi)\notin\mathcal{S}_f))]$. The value in the expectation is equal to 0 if there exists an infeasible state and 1 if all states are in the feasible region in a specific trajectory $\tau$. When the current policy encounters a local infeasible region, the reachability estimation function can help it reenter the feasible region from the edge of infeasible regions and ensure all the states in this trajectory are safe.

4. The derivation in Sec. 4.2

The deviation is obvious and introduced by prior work [2]. The reviewer can find it on page 5. Briefly, it first converts equation 12,13 (13, 14 in the new version) to $\max\_{\pi\_\psi} \mathbb{E}\_{s} [ V\_\varphi^r (s) \cdot 1 (s \in S_f)- V_\varphi^c (s)\cdot 1 (s \notin S_f) ], \text{s.t.} V\_\varphi^c (s) = 0, D_{\mathrm{KL}}(\pi_\psi||\pi_b) \leq \epsilon.$ Then, we can get equation 14 (15 in the new version).

Equation 14 (15 in the new version) shows the function of the reachability estimation function. Equation 15 (16 in the new version) is the optimization problem and the deviation from equations 12,13 (13, 14 in the new version) to equation 15 (16 in the new version) is in Appendix A. The logic could be $Eq.13,14 \rightarrow Eq.15, Eq.13,14 \rightarrow Eq.16$ (numbers follows the new version).

5. Why would introducing advantage functions simplify the constraints?

It is ambiguous that introducing advantage functions does not simplify the constraints. Instead, introducing the Augmented Lagrangian can simplify the constraint as it can transform constrained optimization into unconstrained optimization. For the advantage function, we aim to capture the influence of actions within this optimization problem. A larger value function does not guarantee the existence of an action that will lead the agent to the desired state, so it's crucial to account for action effects. Proposition 1 provides a straightforward way to integrate the advantage function into this optimization framework. We have revised it in the paper.

6. The limitation and future work section should be moved to the main text.

We thank the reviewer for the suggestion. Due to space limitations, we cannot move them all to the main body. Therefore, we simplified them in the Conclusion and guided readers to Appendix I.

References:

[1] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. "Offline reinforcement learning with implicit q-learning." arXiv preprint arXiv:2110.06169 (2021).

[2] Ganai, Milan, et al. "Iterative reachability estimation for safe reinforcement learning." Advances in Neural Information Processing Systems 36 (2024).

[3] Zhang, Haichao, We Xu, and Haonan Yu. "Policy expansion for bridging offline-to-online reinforcement learning." arXiv preprint arXiv:2302.00935 (2023).

[4] Huang, Weidong, et al. "Safe dreamerv3: Safe reinforcement learning with world models." arXiv preprint arXiv:2307.07176 (2023).

[5] Levine, Sergey, et al. "Offline reinforcement learning: Tutorial, review, and perspectives on open problems." arXiv preprint arXiv:2005.01643 (2020).

Thank the authors for their detailed response. After reading the response, I still have two questions/comments:

1. Algorithm overview: ideally, one should be able to easily follow the paper without referring to the appendix (e.g., pseudo-code). To help the readers better understand the connections between Sec. 4.1 and Sec. 4.2, the authors may consider adding a concise version of the algorithm in the main text. Alternatively, the authors may describe the algorithm in the text at the beginning of Sec. 4. In particular, it would be helpful to formally define the training objective of the offline training stage before delving into Sec. 4.1 and 4.2 in detail.

2. I still have questions regarding the reachability estimation function and related derivations in Sec. 4.2. I have checked the original definitions and derivations in [2]. If I understand correctly, $1(s \in S_f)$ is the notation of the optimal feasible set in RCRL where the environment is deterministic. REF is a probabilistic notation of the optimal feasible set applicable when the environment is stochastic. Therefore, I get confused about how Eqn. (13) and Eqn. (14) can lead to Eqn. (15), given that they adopt different assumptions of the environment. Also, I believe the set $\mathcal{S}_f$ in $u^\pi(s)$ should be $\mathcal{S}_v$ instead, i.e., the constraint violation set. It is different from the optimal feasible set $S_f$.

We greatly appreciate the reviewer’s suggestion. We are willing to revise the paper and make some explanations to solve his concerns.

1. About the algorithm overview

We agree that explaining the algorithm in Section 4 would help readers to easily follow our work. We have updated the paper and revised the beginning of Section 4. In the new version, we mention the function of Section 4.1 and Section 4.2 to clarify their connection. Meanwhile, we emphasize the training objectives, with Section 4.1 detailing the Critic training and Section 4.2 focusing on the safe Actor training. We also provide guidance for readers in finding pseudo-code.

2. About the reachability estimation function

It is true that the RESPO can be used when the environment is stochastic. However, it also can be used when the environment is deterministic. The algorithm was defined in the deterministic case in [1] Section 5.1, and the stochastic case in Section 5.2 so the REF can be used in both cases. Thus, Eqn. (13) and Eqn. (14) can lead to Eqn. (15). In addition, it has two advantages when compared with RCRL (Section 4.2 in [1]):

1. By optimizing $V_h$ and $V$ simultaneously using probabilities rather than separately with indicator functions, it ensures the policy can (re)enter the feasible set, making it easier to achieve the optimal;
2. It is not limited to deterministic MDPs and can be applied to stochastic environments.

We want to leverage the first advantage to solve our problem. Plus, it is also good at handling mixed constraints according to Section 6.2 which we have mentioned in our paper.

For the definition of $u^{\pi}(s)$, we are so sorry that we found that our previous response was mistaken and we have corrected it. Our definition of $u^{\pi}(s)$ should have this form: $u^\pi(s) = \mathbb{E}\_{\tau\sim\pi}[1 - \max_{s_t\in \tau}(1\{(s_t|s_0, \pi)\in S_v\})]$. The value in the expectation is equal to 0 if there exists an infeasible state and 1 if all states are in the feasible region in a specific trajectory $\tau$. And our optimization can be written as equation 15:

$\max_{\pi} \mathbb{E}[V^r(s)\cdot u^{\pi}(s) – V^c(s)\cdot (1-u^{\pi}(s))], s.t. V^c(s) = 0$

In contrast, the RESPO uses the definition of $u^{\pi}(s) := \mathbb{E}\_{\tau\sim\pi} [\max_{s_t\in \tau}1((s_t|s_0, \pi)\in S_v)]$ since its optimization problem is $\max_{\pi} \mathbb{E}[V^r(s)\cdot (1-u^{\pi}(s)) – V^c(s)\cdot u^{\pi}(s)], s.t. V^c(s) = 0$ (refer to [1] equation 3). The value in its expectation is equal to 1 if there exists an infeasible state and 0 if all states are in the feasible region. In summary, our definition is just an “opposite version” of RESPO’s definition so they are the same.

References:

[1] Ganai, Milan, et al. "Iterative reachability estimation for safe reinforcement learning." Advances in Neural Information Processing Systems 36 (2024).

We are very grateful to the reviewer for pointing out some of the weaknesses it considers. We are very willing to respond to each of the weaknesses and also answer the reviewer's questions one by one.

Weaknesses: 1. While the experiments were validated in Safety-Gymnasium and real robotic environments, the number and complexity of tasks remain relatively limited.

The task complexity is indeed a limitation in our paper, especially for real-world experiments. However, few works have applied safe reinforcement learning to image-based robotic arm planning tasks. Therefore, migrating it to the real world is a non-trivial step, although the task is relatively simple. We have already written this in our future work in Appendix I. For the simulation environment, we selected a more complex environment to validate our method: Race2. We illustrate our experimental results as follows:

The results indicate that our method is still better than baselines in more complex environments. We have updated them in Appendix H.6.

2. The motivation for introducing the new offline to online setting in the paper has not been well communicated.

We appreciate the reviewer for mentioning it. This new setting involves safe offline and online tasks. Here is our motivation:

We want to train a safe policy on an offline dataset and enable it to generalize to similar unseen tasks. The most straightforward idea is to use offline safe RL to train the policy. However, offline RL suffers from distribution shift so the offline trained safe policy struggles to generalize to similar safety-critical tasks with nearly-zero constraint violations. The distribution shift will make it harder to deal with unseen constraints in the new environment. Directly applying it to real-world experiments can lead to severe constraint violations. To address this, we propose a new framework FOSP to leverage the knowledge from offline datasets and enhance performance through fine-tuning. It shows that our method can perform well during offline pretraining and improve tasks’ performance while ensuring constraints during online fine-tuning. Furthermore, it can solve some unseen tasks with near-zero violations.

This part has been added to the Introduction.

Questions:

1. What specific problem is this method intended to address? The reviewer hopes the authors can clarify the benefits of this setting. For instance, the reviewer notes that in Table 1, the performance of SafeDreamer (offline) is actually comparable to that of FOSP.

We aim to solve safe generalization tasks. The offline-to-online framework is a common structure for solving real-world tasks. In our paper, we consider safe offline-to-online settings. We need to learn as much knowledge as possible from offline datasets and ensure relative safety of interaction with the environment during online fine-tuning. Safe online fine-tuning can further teach the model how to safely complete unseen but similar tasks. Therefore, its generalization can be improved on some safety-critical tasks that have not been seen before.

The benefits of our method are shown as follows:

1. It can learn better performance with lower cost from the dataset during the offline phase (shown in Figure 13).
2. It can improve performance by online fine-tuning while ensuring near-zero constraint violations (shown in Figure 4 (Figure 3 in the new version)).
3. It can generalize to unseen tasks while maintaining the cost as low as possible (shown in Figure 6, Figure 17, Figure 12).

In Table 1, the cost of SafeDreamer (offline) is comparable to that of FOSP (offline). However, with the same cost, FOSP (offline) can achieve better performance, which is its benefit.

2. How is safety ensured in this method, particularly when dealing with unseen data and new environments? Specifically, how does the online fine-tuning process avoid safety hazards?

The safety of fine-tuning can be guaranteed by Section 4.3. Specifically, we train a new policy online and let it follow the guidance of the offline trained policy. During the exploration, we choose the action from these two policies according to their safety. So, at the beginning of the fine-tuning, the offline-trained policy can recognize some hazards similar to those in the original environment and avoid them. But it could be challenging for it to handle some novel hazards. As fine-tuning progresses, the online policy gradually learns the new constraints and further improves its performance, enabling it to handle novel hazards. We compare the converged offline FOSP and FOSP during online training. The results are shown as follows:

The results indicate that FOSP(online) does not perform as well as the fine-tuning method, even after training for 1M steps. Direct online training suffers from the number of training steps, usually requiring a large number of steps to converge. The offline pretraining combined with the online fine-tuning framework allows the agent to effectively leverage offline knowledge to improve online performance, with relatively short fine-tuning being sufficient to achieve satisfactory results. This outcome also suggests that long-term fine-tuning is not hindered, and the policy can continue to learn throughout the fine-tuning process. We have added them to Appendix H.4.

References:

[1] Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang, Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. Safe offline reinforcement learning with feasibility-guided diffusion model. arXiv preprint arXiv:2401.10700, 2024.

We found that the cost may slightly increase at the beginning of the fine-tuning in the transfer task. However, it will decrease fast during fine-tuning. So online fine-tuning could be effective to avoid hazards. Furthermore, we found that fine-tuning helps reduce costs even in the same environment. This is because, due to dataset limits, the policies trained offline are often suboptimal, and online fine-tuning can further enhance its safety. We have added the results to Appendix H.3.

3. The reachability estimation function questions:

Explanation: Let’s back to the preliminaries (Safe Model-based RL). The cost functions $\mathcal{c}: \mathcal{S} \times \mathcal{A}\rightarrow [0,C_{max}]$. It means the cost values are non-negative, which ensures that $V^c(s)$ := $\mathbb{E}\_{\tau \sim \pi}$ $[ \sum\_{t=0}^{T} \gamma^t c(s\_t, a\_t) ]$ are non-negative. Therefore, we have the following inference [1]:

• $V^c(s) = 0 \Rightarrow \forall s_t, c(s_t) = 0$, so the policy $\pi$ can satisfy the constraints starting from $s$. Moreover, the optimal cost value function $V^c(s) = 0 \Rightarrow \exists \pi, \min_{\pi}V^c = 0,$ meaning that a policy that satisfies the constraint exists. So the fitted value function equal to zero can always hold only if all states that the policy passes through are safe. The feasible part can be determined by $S_f(s)$ = {$s|V^c(s)=0$}.

4. How do the authors view the potential of this method for safe sim2real applications? That is, training under simulated environment data while ensuring safety and improving performance during real-world deployment.

Of course, our framework has the potential for sim2real applications. Although we show the experiments in the offline-to-online setting in this paper, its potential in safe sim2real is undeniable. In the safe sim2real setting, the model should be first trained in the simulation so it does not need to be trained offline. After training in the simulation, it can achieve relatively good performance and have some safety perceptions. Then, it needs to be fine-tuned in the real world. World models are data-efficiency, so they will be effective in using trajectories collected from the real world. Therefore, it can gradually handle this task through a small number of attempts. Additionally, safe fine-tuning can ensure it will perform as safely as possible during the fine-tuning phase. Finally, the model can transfer from simulation to the real world.

We have added it into Future Work to provide some insights for the community.

5. The safety policy expansion mechanism stabilizes performance during the initial fine-tuning phase, but does it impose limitations on long-term fine-tuning? For example, what would the comparison of results be between the FOSP(online) version and FOSP in short-term versus long-term training?

It won’t hinder the long-term fine-tuning. The offline trained policy is often suboptimal and it has been frozen during the online fine-tuning. Hence, the new online policy will outperform the offline trained policy as long as the fine-tuning steps are sufficient. Meanwhile, thanks to the data efficiency of the world model, the new online policy can converge quickly. We compare the FOSP(directly online) and FOSP with short-term and long-term fine-tuning (offline pretrained 1M steps). The results are shown as follows:

We are very grateful to the reviewer for pointing out some of the weaknesses it considers. We are very willing to respond to each of the weaknesses and also answer the reviewer's questions one by one.

Weaknesses:

1. The tasks are relatively simple

We appreciate the reviewer for pointing out this limitation. We agree with the simplicity of the tasks. And the reasons why we devise these tasks are shown as follows:

1. The simulation is a benchmark, which can be used to evaluate our method’s effectiveness.
2. The real-world task is quite easy because it is a preliminary attempt. Experiments need to progress from simple to complex; if simple tasks cannot be accomplished, it is almost impossible to succeed in more challenging ones. Therefore, we first conducted SafeReach to evaluate the performance in real-world scenarios. However, end-to-end visual safe reinforcement learning still has a long way to go before being directly applicable to real-world tasks. There exists some work that uses vision-based RL for real-world tasks by sim2real [1] or combining with IL [2]. Conversely, their methods do not involve constraints in the task, which makes training easier. In addition, they incorporate many technical skills that make it work. To the best of our knowledge, our method is the first to apply safe RL to real-world safe generalization tasks and we have demonstrated its effectiveness and practicality (by these simple tasks). The proposed framework can be a contribution to the community so that subsequent researchers can study more complex scenarios.

Actually, the real-time control or dynamic tasks the reviewer mentioned seem more like works from a different area. They are not the core issue of our research so our method cannot handle these tasks. Since these tasks need the agent to predict the environment, they are more about how to predict what might happen next based on visual observations, such as [3] using MPC to predict. The design of the network (predictor) is more responsible for these tasks rather than the RL controller. So, we think the modification of the forecast will be more effective in improving dynamic tasks’ performance and it is orthogonal to our method, which means the modification can be applied to our framework.

We also provide some more realistic simulation tasks in the updated version (Appendix H.6) to further illustrate our method’s effectiveness.

2. The lack of control or predictability for novel visual observations beyond scene re-configurations

It seems difficult for us to deal with completely novel environments. Even though our method has generalization ability, it is difficult to handle completely different environments. We have tried it in simulation tasks. After pretraining on SafetyPointGoal1, we transferred it to SafetyPointBuildingGoal1, which is a totally different environment from the former one but the task is the same (the environment can be found at https://safety-gymnasium.readthedocs.io/en/latest/environments/safe_vision/building_goal.html). The results are shown as follows (fine-tuning 50k steps):

It is clear that vision-based safe RL cannot work in totally different environments during fine-tuning.

We want to specifically answer it in the question part and we have added the simulation experiment in Appendix H.5.

Questions:

1. Transfer task changes are more visual in nature.

The above simulation experiment results confirm that our method struggles with significant visual changes since our method is based on vision-only inputs. In real-world experiments, we aim to use colors to differentiate obstacles and goals to ensure the model can perceive obstacles and goals through visual inputs. Therefore, it is impossible to maintain performance when we swap the colors of obstacles and goals. As robot needs to distinguish obstacles and goals by their colors, they might mistakenly identify obstacles as goals, leading to direct collisions. It will not understand the constraints settings as we intended. It is highly likely to trick the method into substantial safety violations because it violates our original intention of using colors. A promising solution is adding semantic information to the inputs, telling the agent which color object is the obstacle and which color object is the target.

We have added the discussion to Limitation and Future Works.

2. Release the code

Yes, we have already released our code in the Supplementary Material at the time of initial submission, though it is a bit messy and needs to be organized. Since our datasets are too large to be released, users should collect data by themselves, and then train the FOSP. Here is a guidance to collect the dataset:

• First, create a new folder “dataset”.
• Replace agent.py with agent_sd.py and behavior.py with behavior_sd.py. Run “python FOSP/train.py --configs safedreamer --method safedreamer --run.script collect_dataset --run.from_checkpoint /xxx/checkpoint.ckpt --task safetygym_SafetyPointGoal1-v0 --jax.logical_gpus 0 --run.steps 50000” to collect safe data (unsafe data: replace safedreamer to dreamerv3). To find the checkpoint, users should download it from Hugging Face (from SafeDreamer [4]) https://huggingface.co/Weidong-Huang/SafeDreamer/tree/main.
• Find the npz files and add them into ./dataset/. Add the path to ./embodied/replay/saver.py: self.load_dir. Then follow the Readme.md to train the FOSP.

We will then organize it with the datasets and release it on github.

References:

[1] Lei, Kun, et al. "Uni-o4: Unifying online and offline deep reinforcement learning with multi-step on-policy optimization." arXiv preprint arXiv:2311.03351 (2023).

[2] Haldar, Siddhant, et al. "Teach a robot to fish: Versatile imitation from one minute of demonstrations." arXiv preprint arXiv:2303.01497 (2023).

[3] Bing, Zhenshan, et al. "Safety Guaranteed Manipulation Based on Reinforcement Learning Planner and Model Predictive Control Actor." arXiv preprint arXiv:2304.09119 (2023).

[4] Huang, Weidong, et al. "Safe dreamerv3: Safe reinforcement learning with world models." arXiv preprint arXiv:2307.07176 (2023).