Thank you kindly for your review. We have corrected all typos and addressed all your major comments below. We want to stress at this point that our method does, in fact, guarantee safety, even throughout exploration.

Safety guarantee during exploration: Our method does guarantee safety during exploration. This is, in fact, the whole point and one of the main contributions of our method. Our method explores in a way that recovers the feasibility of the CBF safety filter before the safe set is exited. In other words, the closed-loop system is always in the safe set during exploration. This is illustrated by Figure 1 and shown theoretically in Theorem 3.2. The key idea in the proof is to show that if we collect data with our exploration strategy quickly enough, we must recover feasibility before exiting the safe set. This is achieved by upper-bounding the information that can be gained inside the safe set, then showing that each collected data point contributes sufficiently toward reducing the information within the safe set (Lemma B.9). This is a crucial aspect of our method, and we will make this clearer in the revised manuscript by explicitly stating it in the abstract and introduction.

Specifications for experiments: In the experiments, we set $\beta=10$, $\epsilon=0$ and $h$ equal to the identity function. The high choice of $\beta$ aims to avoid underestimating the model error. Note that $\beta=2$ is otherwise frequently chosen in the literature, e.g., in Berkenkamp et al. (2017) and Wachi et al. (2018). The choice of $\epsilon=0$ corresponds to assuming that the CBF constraint can be enforced robustly, whereas $h$ equal to the identity function is for simplicity.

Berkenkamp, F., Turchetta, M., Schoellig, A., & Krause, A. (2017). Safe model-based reinforcement learning with stability guarantees. Advances in neural information processing systems, 30.

Wachi, A., Sui, Y., Yue, Y., & Ono, M. (2018). Safe Exploration and Optimization of Constrained MDPs Using Gaussian Processes. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).

Sampling time: Although the sampling time required to achieve safety consistently in the examples is high, note that they correspond to a setting with no prior knowledge. Hence, it is reasonable to expect a high sampling time, as otherwise, sufficient information cannot be collected before exiting the safe set.

GoSafeOpt paper: Thank you for pointing out this paper to us. The paper does indeed employ similar tools and has a similar motivation. Some important differences include:

• On a high level, the goal of GoSafeOpt is to expand a set that is already safe. Although our method can also be used for the same end, its main goal is to render the system safe under high levels of uncertainty. This is particularly useful for settings where other safety certificates can potentially fail.
• A further difference that is related to the one above is that GoSafeOpt requires a set of initially safe control parameters at the beginning of exploration to guarantee safety. In contrast, our approach can guarantee safety purely through exploration, without requiring an initially safe set of control parameters.
• In GoSafeOpt, the goal is to tune a parametric controller, and the GP-based safety conditions depend on the controller's parametric structure. In our setting, the GP model is independent of any underlying control structure. We will include this comparison in the revised paper.

Less safety violations at 10Hz: We agree that this is somewhat unexpected since a higher sampling rate should yield a better model faster, which translates to safer control. However, it is important to remember that at lower sampling frequencies, the system can quickly reach the boundary of the safe set before more than a handful of data has been collected. In these settings, the exploratory input is applied for a considerable amount of time before potentially reverting to the safety filter-based control, making its role non-neglegible. Hence, we speculate that the exploratory control is particularly beneficial for safety in this particular instance.

Thank you kindly for your review. Below, you will find our answers to your comments.

Regarding the claim of solvability of the benchmarks: This claim is tied to the controller having a prior mean model of zero, no measurement data, and only a CBF to guide it. Although existing methods can effectively control these systems, the control design in existing works is informed by a prior (non-zero) model of the dynamics. Alternatively, a backup controller is required. Our approach only requires the CBF and a well-specified GP to inform the control design. We will reformulate the sentence and include this discussion in the revised paper.

The choice of kernel is not obvious: We agree with this statement. Choosing appropriate kernels for modeling dynamical systems is generally a nontrivial task, and there exists an extensive body of literature on how to do so while mitigating model misspecification; see, e.g., Berkenkamp et al. (2019), Fiedler et al., (2021), Capone et al., (2022). Crucially, although many kernels can satisfy the requirements of our approach, some kernels can generalize far better than others, significantly reducing the amount of data required to achieve safety. For example, if the system has a linear component, it is typically beneficial to include a linear component in the kernel, and sometimes, a linear kernel can even outperform a universal kernel in this sense. We will include this discussion in the paper.

Knowing a CBF without knowing the dynamics: While it is generally impossible to formulate a valid CBF without any form of model knowledge, doing so is easier than obtaining an actual system model, since the existence of a CBF typically implies the existence of (potentially infinitely) many others. Furthermore, the assumption that the CBF is valid can be relaxed in many ways: the CBF only needs to be valid in parts of the state space that are visited by the closed-loop system. Moreover, our approach can gradually expand the safe set, enabling incremental improvements to the CBF. We also note that the assumption that a CBF is known despite not having an accurate model is common in the literature; see, e.g., Cheng et al. (2019), Choi et al., (2020), Taylor et al., (2020).

Berkenkamp, F., Schoellig, A. P., & Krause, A. (2019). No-regret Bayesian optimization with unknown hyperparameters. Journal of Machine Learning Research, 20(50), 1-24.

Capone, A., Lederer, A., & Hirche, S. (2022, June). Gaussian process uniform error bounds with unknown hyperparameters for safety-critical applications. In International Conference on Machine Learning (pp. 2609-2624). PMLR.

Cheng, Richard, Gábor Orosz, Richard M. Murray, and Joel W. Burdick. "End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks." In Proceedings of the AAAI conference on artificial intelligence, vol. 33, no. 01, pp. 3387-3395. 2019.

Choi, J., Castañeda, F., Tomlin, C. J., & Sreenath, K. (2020, July). Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functions. In Robotics: Science and Systems (RSS). 2020.

Fiedler, C., Scherer, C. W., & Trimpe, S. (2021, May). Practical and rigorous uncertainty bounds for Gaussian process regression. In Proceedings of the AAAI conference on artificial intelligence (Vol. 35, No. 8, pp. 7439-7447).

Taylor, A., Singletary, A., Yue, Y., & Ames, A. (2020, July). Learning for safety-critical control with control barrier functions. In Learning for dynamics and control (pp. 708-717). PMLR.

Thank you kindly for reviewing our paper. Please find our answers to your questions and comments below.

Comparison with other references and baselines: Thank you for suggesting the additional references. Although the methods of Sui et al. (2015), Wachi et al. (2018) and Prajapat et al. (2022) employ similar tools (the most notable being a combination of GP-based LCBs and UCBs for safety and optimistic exploration), there exist some fundamental differences between their methods and ours. Arguably the biggest difference on a conceptual level is that exploration in their approaches is driven by return maximization. In contrast, exploration in our approach is driven by the need to ensure safety. Moreover, their approaches do not allow the (pessimistically) unsafe states to be visited, whereas ours does, using exploration to reduce uncertainty while in those states. Other differences are as follows:

• Their methods require a tabular MDP setting, which scales poorly with the number of state space dimensions and actions.
• They are designed for a discrete-time setting, whereas our approach considers a continuous-time setting.
• Their methods require the transition dynamics to be known perfectly, whereas our approach does not.
• A further difference tied to the previous point is how optimistic exploration is used. The methods of Sui et al. (2015) and Wachi et al. (2018) utilize exploration to expand the safe set and maximize the returns under optimism. However, the safe set is expanded only by visiting safe states, i.e., by incrementally moving toward the unsafe states. Note that this is typically only achievable with perfectly known dynamics. In contrast, our approach expands the safe set by directly exploring the states that are unsafe under pessimism.

After carefully analyzing the method and code of Wachi et al. (2018), we have concluded that to implement their method on one of our baselines, we would require significant changes to their method:

• First, we would need to discretize the dynamical systems. While this is possible for the cruise control example, it is practically infeasible for the quadrotor. This is because the state space is 10-dimensional, and the nonlinear dynamics are strongly state-dependent, meaning that a dense graph is required. However, this implies a graph with 10^20 nodes for just 20 support points per dimension.

• Secondly, we would need to address the fact that the dynamics are unknown in our setting. To this end, we could either use the GP mean to estimate the state-transition dynamics or a pessimistic estimate of the transition dynamics under model uncertainty and the pessimistic safety constraint. Moreover, the state transition graph would have to be updated after every time step, which was not intended under the original algorithm.

Overall, we feel that directly implementing their code on our setting would entail design decisions that would significantly alter the original method, making a fair comparison difficult. However, we will thoroughly discuss these papers in the revised manuscript. Moreover, we will include a comparison with the heuristic proposed by Choi et al. (2023), which corresponds to an equation that attempts to maximize the LCB whenever infeasibility occurs.

Regarding computation time: Our method mainly requires computation time for GP predictions and to solve the optimization problems involving the LCB (exploration) and UCB (safe control) of the CBF. Please note that, due to the nature of the CBF optimization problem, we only require a single GP pass to formulate the required vectors and matrices, after which we only have to solve a convex optimization problem. Hence, the GP computation scales with the data, whereas the CBF optimization does not. However, sparse inference tools can be easily applied to improve scalability. Below, we report the average computation time for each operation as a function of the data using vanilla GPs (we use the GPyTorch toolbox without GPU acceleration):

The GP column corresponds to the average time of GP computations per iteration, which scales with the data, as expected. The Optim column corresponds to the average time taken by the optimization problem, which stays approximately constant with the amount of data.

Thank you kindly for your review. We have addressed your comments and questions below.

Knowledge of CBF and difference to nominal understanding of the system dynamics/backup controller: Although the CBF assumption cannot be ensured in a general setting, it still offers flexibility, and there are ways to relax it in practice. Typically, only a portion of the safe set is visited during control; hence, the CBF function only has to satisfy the corresponding conditions for the corresponding subset of the state space. Furthermore, we can achieve conservatism by initially restricting the safe set to a region that is easy to control and then iteratively expanding it as more data is collected. This potentially allows us to gradually improve the CBF as more system knowledge allows us to expand the safe set. Alternatively, we can include conservatism in the safety requirement by computing the CBF condition from a collection of (potentially valid) CBFs and taking the worst case.

Additionally, we note that the setting where a CBF is known but the dynamics are unknown is an open ongoing research question that has garnered increased attention recently:

• End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks (Cheng et al., 2019)
• Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functions (Choi et al., 2020)
• Learning for safety-critical control with control barrier functions (Taylor et al., 2020)
• Constraint-Guided Online Data Selection for Scalable Data-Driven Safety Filters in Uncertain Robotic Systems (Choi et al., 2023)
• Safe Barrier-Constrained Control of Uncertain Systems via Event-triggered Learning (Lederer et al., 2024)
• Learning-Based Prescribed-Time Safety for Control of Unknown Systems With Control Barrier Functions (Huang et al., 2024)

Note that these works require additional assumptions beyond a known CBF to achieve safety.

Furthermore, we believe our method still has value even in settings where the CBF is not strictly valid. One of our paper's main contributions is that we use exploration to recover safety if all other safety certificates fail. This strongly contrasts with other approaches, which either assume enough controllability to stay within the safe set at all times or require a safe backup controller. Hence, it provides a principled way to design an exploration-based controller that aims to achieve safety whenever other certificates become invalid. We will include this discussion in the revised paper.

Comparison with other baselines: We are currently implementing a comparison with the heuristic proposed by (Choi et al., 2023), which computes control inputs by maximizing the LCB whenever infeasibility occurs instead of exploring the state space. We will also include a comparison with the baseline with a fully known model. We will report the results during the second round of reviews.

Designing a CBF with a zero model prior: In general, it is impossible to design a CBF without any prior model knowledge. However, it is still less challenging than deriving an accurate system dynamics model, and this requirement can frequently be relaxed, as discussed above. Moreover, note that allowing a prior model of zero is not strictly an assumption of our method but rather something that it allows for, e.g., if we want to learn an unbiased model purely from data. Our paper highlights this because it contrasts with other methods, which require more assumptions/backup controllers even in settings where a prior model is available.

CBF design in experiments: The CBF was taken from the cruise control setting from Ames et al. (2021). It restricts the velocity of the vehicle it comes closer to the vehicle in front. For the quadrotor, the first CBF (for avoiding ground collision) is similar to the cruise control setting, whereas the second CBF (for avoiding overrotation) corresponds to a quadratic equation that constrains the velocities and the orientation of the quadrotor.

What happens if the collected data show that the learned dynamics violates the CBF that is assumed given? The CBF is not necessarily learned from the model. Hence, it can violate the learned dynamics. In fact, it can violate the dynamics under pessimism (this corresponds to the CBF LCB), which is when our method explores.

Choi, J. J., Castaneda, F., Jung, W., Zhang, B., Tomlin, C. J., & Sreenath, K. (2023). Constraint-guided online data selection for scalable data-driven safety filters in uncertain robotic systems. arXiv preprint arXiv:2311.13824.

Ames, A. D., Grizzle, J. W., & Tabuada, P. (2014, December). Control barrier function based quadratic programs with application to adaptive cruise control. In 53rd IEEE conference on decision and control (pp. 6271-6278). IEEE.