High-Dimensional Safe Exploration via Optimistic Local Latent Safe Optimization

The presentation has the potential to be improved and the contributions could be better summarized

Thank you for your suggestion, and we will improve the presentation in the next version.

Here we summarize our contribution to as follows:

1. In this paper, we propose a practical method to achieve safe and efficient optimization of high-dimensional functions. To our best knowledge, no existing work focuses on guaranteeing safety during the high-dimensional optimization.

2. OLLSO overcomes the scalability and efficiency difficulties of the previous GP-based safe optimization method by introducing a local search mechanism with optimistic safety region identification, and achieves safe and efficient optimization of functions with hundreds of dimensions.

3. OLLSO is the first algorithm to use distance-preserved VAE to address safety issue in latent space optimization. We derive the theoretical probabilistic safety guarantee of OLLSO, which enable it to handle extremely high-dimension or discrete input space.

4. Our experimental results shows that OLLSO significantly outperforms existing safe BO methods, constrained BO algorithms, and direct combinations with dimension reduction models.

We want to emphasize that OLLSO is a PRACTICAL method that can safely optimize high-dimensional real-world problems. In addition to state-of-the-art simulation performance, our real clinical experiments show that OLLSO successfully optimizes neural stimulation for the control of human muscles. While the majority of our cited papers conduct their experiments only in simulation or synthetic problems, OLLSO is both practical and theoretically verifiable to safely and efficiently optimize real-world safety-critical problems.

Methodology clarification.

We describe the choice of the acquisition function $A$ and the updates of the local region in Section 4.1.

For the acquistion function $A$, we use Thompson sampling[1] as the acquisition function due to its compatibility with the discrete nature of our safety estimation and search space, and its innate ability for batch optimization by sampling the GP posterior.

The local region $L_t$ is a trust region defined by a center (using current best feasible point) and a side length $l$, where we dynamically update the search space based on the sample history. Thank you for pointing out the confusion part and we would improve the presentation and add some illustration figures or psudo-codes to help readers better understanding our paper.

Further analysis about IRVAE.

We show the experiment of analyzing the IRVAE in Section 6.3. And we will improve our presentation in Section 4.1 to lead readers to the experiment part.

In Figure 4, we show that when using IRVAE, the GP estimation difference between original and latent space is significantly smaller than using standard VAE. We further compute point-wise distance of 10,000 random sampled points. The results shows high linear correlation of point-wise distance between original space and latent space ($R^2$ is 0.851 for musculoskeletal model control task and 0.973 for neural stimulation task), indicating good distance preserving ability of IRVAE.

[1] Kandasamy, Kirthevasan, et al. "Parallelised Bayesian optimisation via Thompson sampling." International Conference on Artificial Intelligence and Statistics. PMLR, 2018.

[2] Yang Q, Simão T D, Jansen N, et al. Reinforcement Learning by Guided Safe Exploration, ECAI 2023.

Thanks for your encouraging comments of our work, and we respond your questions below.

The assumptions limit the widespread application.

We argue that the most assumptions we used in the paper are common-used in high-dimensional BO literature. Assumptions 1 and 2 are regularity assumptions of the function, which enables to use Gaussian process to model function distributions. Assumption 3 is low effective dimension assumption common-used in embedding BO works to manifest the structural reward characteristic. Note the low-dimensional manifold are always observed in most high-dimensional structures. The only additional assumption is the distance preserving ability of the latent space embedding, which is crucial to guarantee probabilistic safety and can be satisfied using some linear embedding, or approximately satisfied using isometry regularized VAE. We have demonstrated success of our proposed method in two different complex high-dimensional problems and one real-world application. And we expect to apply OLLSO to more real-word safety-critical problems.

General recipe of applying OLLSO in real-world applications.

We believe OLLSO can be applied in various high-dimensional safety-critical problems. As mentioned above, the regularity assumptions are common-used in BO literature. In practice, we can fit the kernel hyperparameter using the observed data to make GP better model the functions. Many high-dimensional problems often have low-dimensional intrinsic structure, which satisfy the low effective dimension assumption. The distance preserving assumption depends on the embedding way, where using a IRVAE is able to approximately satisfy the condition.

In the real clinical therapy optimization, most dimensions (16 contact configurations) of the original input space are discrete, which is a sparse representation, and has the potential to reduce to a more compact representation. Our experimental result of IRVAE also shows that it can well preserve the distance and reconstruct the original input. Note that this exam of distance preserving is conducted before real evaluation, and we can start the real experiment after ensure the embedding satisfy the distance preserving requirement.

Overall, to apply OLLSO, the first step is to train an IRVAE using pre-collected or synthesized unlabelled data without actual evaluation. After verifying the embedding distance preserving capability, we can safely optimize the objective function following Algorithm 1. It is easy to be applied to other real-world problems.

Safety probabilistic threshold.

Thank you for pointing out the typos. We will fix them in the next version.

In many real-world applications, we can exclude a large number of unsafe positions using prior knowledge. The remaining search region may have a certain probability to be safe even under random search. Therefore we set the probabilistic threshold smaller than 0.5, which is an optimistic estimation of the safety function in the problem formulation, and enables to use UCB to optimistically identify the safe region. We believe this formulation offers a more practical way to efficiently and safely optimize high-dimensional real-world problems. Our experimental result also shows that OLLSO usually achieves safety probability larger than 80% in the whole optimization procedure, and significant less constraint violation compared to other competitive baselines, which empirically demonstrates the safety guarantee when applying OLLSO in the real-world applications.

Discussion with safe exploration in RL

Thank you for listing the related work about safe exploration in RL. We will include the discussion of safe exploration in RL [2] in the next version.

Thanks for your detailed feedback on our work, and it greatly helps us improve our paper. We answer your questions below.

Optimization over probabilistic safety constraints

Thank you for pointing out the confusion in the problem formulation. We actually want to find an optimal $x^*$ with the highest feasible objective function value, whose safe value is above the pre-defined threshold after observation. During the optimization procedure, we want every decision we make to be probabilistic safe. We will further modify our problem formulation in the next version to make it more clear, but it does not influence the final theoretical result, where we prove that every decision we made during optimization satisfies the probabilistic safe requirement of the problem.

Regularity assumption over latent space.

Our regularity assumption over the latent space enables to use GP to model function over the latent space. We agree that the norm-bound can not be preserved over the original space when the decoder is not well-behaved. Therefore, we want to use an isometric regularized autoencoder to achieve approximated distance preserving over the latent space. Our theoretical result(Theorem 1) is built upon both the regularity assumption(Assumption 1) and the distance preserving ability(Definition 1) of the latent embedding.

Assumption about the effective dimension.

We apologize for the not explicitly defining the low effective dimension of the original function. The correct formulation should be

$g(\boldsymbol{x})=g(\boldsymbol{z}U^T)$

where $\boldsymbol{z}=U\boldsymbol{x}$, and $U^T$ is the inverse mapping from latent space to the original space.

$\epsilon$-subspace embedding assumption

We agree that the $\epsilon$-subspace embedding assumption may not hold for standard VAE with non-linear layers. But we argue that IRVAE may approximately satisfied this assumption. In figure 4, we show that the GP estimation difference between original space and latent space of IRVAE is significantly smaller than latent space of standard VAE. We also compute point-wise distance of 10,000 random sampled points. The results shows high linear correlation of point-wise distance between original space and latent space ($R^2$ is 0.851 for musculoskeletal model control task and 0.973 for neural stimulation task), indicating good distance preserving ability of IRVAE.

Assumptions about noise observation.

We propose OLLSO to optimize general safety-critical problems with noisy observation. In this paper, we theoretically derived the safety guarantee under noise-free setting, and we believe under bounded noise assumption, the derivation of safety guarantee under noise-perturbed observation is feasible.

We also empirically demonstrate that OLLSO works well under noisy setting in the clinical experiment, where the observation is perturbed by noise from measuring EMG signal of lower limb muscles.

Methodology clarification.

1. Explanation of safer trajectory after reset.

When resetting the trust region, we preserve all sample points compared to SCBO, which discards all history data. In this way, we have more knowledge about the safety function compared to last reset point, and the future sample procedure will be safer.

2. The sample trajectory

The sample trajectory $\zeta_{t}$ is defined as a set of tuples {$(\boldsymbol{x}_t,y_t^f,y_t^g)$}. The update of the trust region is based on the selection result of last round, which is a process depending on the historical selection trajectory.

3. The setting of trust region length $l$

We directly use the setting of SCBO BoTorch implementation(https://botorch.org/tutorials/scalable_constrained_bo). In concrete, we set the initial $l$ as 0.8, $l_{min}$ is $0.5^7$, and $l_{max}$ is 1.6. We normalize all inputs in the range of $[0,1]$, therefore we can use same setting across different tasks.

4. The upper confidence bound

We define $u_t(\boldsymbol{z})\coloneqq\max C_t(\boldsymbol{z})$, where $C_t(\boldsymbol{z})\coloneqq [\mu_t(\boldsymbol{z})\pm\beta\sigma_t(\boldsymbol{z})]$ is the confidence interval at the position $\boldsymbol{z}$. Therefore, the max operation is performed over the interval instead of $\boldsymbol{z}$.

VAE-related background

Thank you for pointing out the missing discussion of this topic. We will include it in the next version of the paper.

Unlabeled dataset synthesizing

Taking neural stimulation task as an example, we synthesize the unlabelled stimulation parameter by randomly setting the contact configuration and the intensity under clinical prior (e.g. limiting the contact number or intensity range). The procedure is capable of generating large quantity of data without actual evaluation. There are many other real-world applications where we can synthesize data in the same way, such as protein/molecular sequences.

Parsing of results and typos

Thank you for finding typos in our paper. We will correct them in the next version.

The novelty of the method.

In this paper, we propose a practical method to achieve safe and efficient optimization of high-dimensional functions. To our best knowledge, no existing work focuses on guaranteeing safety during the high-dimensional optimization.

We present the novelty of OLLSO in both algorithmic and practical aspects.

In terms of latent space optimization, previous latent BO works neglect the geometry-preserving during unsupervised representation learning, which is crucial to extend safety guarantee from latent space to the original space. Many of them achieve latent space shaping using collected labeled data [1, 2]. We are the first to use distance-preserved VAE to address safety issue in latent space optimization, and derive the theoretical probabilistic safety guarantee of our proposed method.

The local region adaptation of OLLSO is also different from existing trust-region based BO methods[3, 4] which neglect the safety during the optimization procedure. They would also discard all previous sample point when restarting the trust region. OLLSO updates the trust region in a more safety-sensitive way, and keeps all previous samples when restarting the trust region, ensuring a safer optimization procedure with theoretical guarantee.

In terms of the safe optimization part, previous GP-based safe optimization method are limited to low-dimensional problems (typically below 10, [5, 6]). OLLSO addresses this issue by introducing local search mechanism upon optimistic safety region identification, and is able to handle problem with hundreds of dimensions.

Our experimental result also shows that OLLSO is not a direct application to comprise exsiting algorithms. Figure 2 and Figure 3 show that simply combining IRVAE with existing safe BO (CONFIG (L)) or trust region-based constrained BO (SCBO (L)) perform worse than OLLSO in terms of both optimization performance and safety guarantee. Our simulation results shows that OLLSO significantly outperforms existing safe BO methods, constrained BO algorithms, and direct combinations with dimension reduction models.

We want to emphasize that OLLSO is a PRACTICAL method that can safely optimize high-dimensional real-world problems. In addition to state-of-the-art simulation performance, our real clinical experiments show that OLLSO successfully optimizes neural stimulation for the control of human muscles. While the majority of our cited papers conduct their experiments only in simulation or synthetic problems, OLLSO is both practical and theoretically verifiable to safely and efficiently optimize real-world safety-critical problems.

[1] Tripp, Austin, Erik Daxberger, and José Miguel Hernández-Lobato. "Sample-efficient optimization in the latent space of deep generative models via weighted retraining." Advances in Neural Information Processing Systems 33 (2020): 11259-11272.

[2] Grosnit, Antoine, et al. "High-dimensional Bayesian optimisation with variational autoencoders and deep metric learning." arXiv preprint arXiv:2106.03609 (2021).

[3] Eriksson, David, et al. "Scalable global optimization via local Bayesian optimization." Advances in neural information processing systems 32 (2019).

[4] Eriksson, David, and Matthias Poloczek. "Scalable constrained Bayesian optimization." International Conference on Artificial Intelligence and Statistics. PMLR, 2021.

[5] Sui, Yanan, et al. "Safe exploration for optimization with Gaussian processes." International conference on machine learning. PMLR, 2015.

[6] Xu, Wenjie, et al. "Constrained efficient global optimization of expensive black-box functions." International Conference on Machine Learning. PMLR, 2023.

Thanks to the authors for their response. I wIll require more time to make a judgement on the full response, but I believe one point need to be addressed urgently:

Authors' response: Optimization over probabilistic safety constraints

I am unfortunately not satisfied with the response. As clarity of the problem at hand is paramount to any paper, I wish to further understand it to make an informed judgement:

We actually want to find an optimal with the highest feasible objective function value, whose safe value is above the pre-defined threshold after observation.

So if I understand correctly, the safety value $g(x)$ of any configuration is observed in conjunction with the objective $f(x)$ ? If so, why is the safety $g$ probabilistic? (since we know if $g(x) > h$ for sure once we have observed it, hence there is no probabilistic component).

I encourage the authors to be precise and update the paper on this point.

Thank you for your quick feedback. We clarify your further comments below.

Optimization over probabilistic safety constraints.

We argue that the problem setting in [1, 2] is different from our problem setting. They actually address a constrained optimization problem that aims to find a feasible point that satisfies the probability constraint, neglecting safety during the optimization process. Our problem setting requires that each sample satisfies the probability constraint and considers safety throughout the optimization process.

Regularity assumption over latent space.

We think the regularity assumption of the latent space is the prerequisite for using GP to model functions over the latent space. All latent space BO algorithms used GP over the latent space and achieved success on various different high-dimensional problems [3, 4]. Therefore, we believe that it is not a convenient assumption. The regularity assumption over the latent space also does not means the norm-bound is preserved through the decoding step. Since the non-linearity and distortion of the decoder, the function regularity over the original and latent space mat not be able to use the same RKHS to describe. That’s why the $\epsilon$-subspace assumption is introduced to derive our theoretical result.

We also argue that the $\epsilon$-subspace assumption is not a convenient assumption. This assumption can certainly be satisfied by some linear embeddings, such as PCA or random embeddings. In this paper, we use IRVAE to harness the power of deep learning models to learn a better representation while maintaining the approximated isometry. Our further experiment also empirically demonstrates the distance preserving ability of IRVAE as mentioned in the previous rebuttal.

Assumptions about noise observation.

Thank you for your suggestions on the consistency of our paper. As mentioned in the previous rebuttal, our theoretical is built over noiseless setting, but the real clinical experiment shows that our proposed method successfully optimizes the general safe optimization problem under noisy observation. We will improve our presentation in the next version to reduce this confusion.

Methodology clarification.

Thank you for pointing out the confusion. We will add more details to make the paper clearer.

The novelty of the method.

As mentioned above, we argue that the assumptions we use in this paper are not convenient assumptions. The regularity assumptions are the prerequisite of all latent space BO methods, and the $\epsilon$-subspace assumption can be satisfied using some linear projection or approximately satisfied by some distance-preserved autoencoder. We believe our result of safety guarantee is novel.

Our experiment also empirically verifies our theoretical result, where OLLSO consistently outperforms existing constrained BO methods or safe BO methods in terms of both optimization performance and safety guarantee. Our clinical experiment also demonstrates the success of OLLSO in optimizing real-world high-dimensional safety-critical problems. We believe that our practical contribution is novel.

Overall, we believe that our proposed method is novel in both algorithmic and practical aspects, and is both practical and theoretically verifiable to safely and efficiently optimize real-world safety-critical problems.

[1] Ryan-Rhys Griffiths and José Miguel Hernández-Lobato. Constrained Bayesian optimization for automatic chemical design using variational autoencoders. Chemical Science, 2020.

[2] M. A. Gelbart, J. Snoek and R. P. Adams, Bayesian optimization with unknown constraints. Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence, 2014.

[3] Tripp, Austin, Erik Daxberger, and José Miguel Hernández-Lobato. "Sample-efficient optimization in the latent space of deep generative models via weighted retraining." Advances in Neural Information Processing Systems 33 (2020): 11259-11272.

[4] Grosnit, Antoine, et al. "High-dimensional Bayesian optimisation with variational autoencoders and deep metric learning." arXiv preprint arXiv:2106.03609 (2021).

Comparison with random-embedding BO

Thanks for your suggestions on adding random-embedding-based methods for comparison. We follow your suggestion and run HesBO and BAxUS on a musculoskeletal model control task and a neural stimulation task with no safety constraints in simulation. Due to the algorithmic mechanism of HesBO and BAxUS, we cannot directly use the same initial point as OLLSO. Therefore we randomly sample initial points from their corresponding latent space. In HesBO, we set the same latent dimension number as in OLLSO. The following tables are the best objective function values found by algorithms (shown as mean $\pm$ 1 std).

We observe OLLSO still outperforms HesBO and BAxUS across all tasks, even when optimizing under safety constraint. We think using IRVAE enables utilizing the pre-collected unlabelled data to learn a better representation than random projection.

Missing discussion about related works

Thank you for pointing out the missing discussion of distance-preserving dimension reduction for uncertainty quantification. The listed papers are relevant to our work, and we will include them in the discussion part in the next version of our paper. BALLET is a good work incorporating deep kernel learning with level set estimation to tackle high-dimensional problems, and we will try to get insights from this work and conduct regret analysis in the future.

Experiment result clarification

1. Figure 3 (b) shows the clinical result in neural stimulation therapy optimization, where IL (iliopsoas), RF (rectus femoris), TA (tibialis anterior), BF (biceps femoris) are different groups of target muscles on the lower limb, which are the main muscle groups that control motor funuction. The "L" (Left) or "R" (Right) after the underline are the sides of human lower limbs.

2. Figure 3 (c) shows the simulation result using a computational spinal cord model, where the experiment result is detailed in Section 6.3.1. Bar plot in Figure 3 (c) shows the best feasible objective function value (Objective, top), safe decision ratio of all samples (Safe %, middle), and cumulative safety violation (Violation, bottom) after 800 samples.

Thank you for pointing out the confusion part, and we will improve our experiment result presentation in the next version.

The novelty of our work.

In this paper, we propose a practical method to achieve safe and efficient optimization of high-dimensional functions. To our best knowledge, no existing work focuses on guaranteeing safety during the high-dimensional optimization.

We present the novelty of OLLSO in both algorithmic and practical aspects.

In terms of latent space optimization, previous latent BO works neglect the geometry-preserving during unsupervised representation learning, which is crucial to extend safety guarantee from the latent space to the original space. Many of them achieve latent space shaping using collected labeled data [1, 2]. We are the first to use distance-preserved VAE to address safety issue in latent space optimization, and derive the theoretical probabilistic safety guarantee of our proposed method.

The local region adaptation of OLLSO is also different from existing trust-region based BO methods [3, 4] which neglect the safety during the optimization procedure. They would also discard all previous sample points when restarting the trust region. OLLSO updates the trust region in a more safety-sensitive way, and keeps all previous samples when restarting the trust region, ensuring a safer optimization procedure with theoretical guarantee.

In terms of safe optimization, previous GP-based safe optimization method are limited to low-dimensional problems (typically below 10, [5, 6]). OLLSO addresses this issue by introducing local search mechanism upon optimistic safety region identification, and is able to handle problem with hundreds of dimensions.

Our experimental result also shows that OLLSO is not a direct application to comprise exsiting algorithms. Figure 2 and Figure 3 show that simply combining IRVAE with existing safe BO (CONFIG (L)) or trust region-based constrained BO (SCBO (L)) perform worse than OLLSO in terms of both optimization performance and safety guarantee. Our simulation results shows that OLLSO significantly outperforms existing safe BO methods, constrained BO algorithms, and direct combinations with dimension reduction models.

We want to emphasize that OLLSO is a PRACTICAL method that can safely optimize high-dimensional real-world problems. In addition to state-of-the-art simulation performance, our real clinical experiments show that OLLSO successfully optimizes neural stimulation for the control of human muscles. While the majority of our cited papers conduct their experiments only in simulation or synthetic problems, OLLSO is both practical and theoretically verifiable to safely and efficiently optimize real-world safety-critical problems.

[1] Tripp, Austin, Erik Daxberger, and José Miguel Hernández-Lobato. "Sample-efficient optimization in the latent space of deep generative models via weighted retraining." Advances in Neural Information Processing Systems 33 (2020): 11259-11272.

[2] Grosnit, Antoine, et al. "High-dimensional Bayesian optimisation with variational autoencoders and deep metric learning." arXiv preprint arXiv:2106.03609 (2021).

[3] Eriksson, David, et al. "Scalable global optimization via local Bayesian optimization." Advances in neural information processing systems 32 (2019).

[4] Eriksson, David, and Matthias Poloczek. "Scalable constrained Bayesian optimization." International Conference on Artificial Intelligence and Statistics. PMLR, 2021.

[5] Sui, Yanan, et al. "Safe exploration for optimization with Gaussian processes." International conference on machine learning. PMLR, 2015.

[6] Xu, Wenjie, et al. "Constrained efficient global optimization of expensive black-box functions." International Conference on Machine Learning. PMLR, 2023.

Dear Reviewer WSiY,

Thank you again for your valuable feedback. We have posted the additional experiment comparing with random-projection-based BO algorithms and the clarification about the novelty of our work. As the deadline is less than two days away, we look forward to your further feedback to make sure that all your concerns are addressed.

Best,

OLLSO authors

The novelty of our work.

In this paper, we propose a practical method to achieve safe and efficient optimization of high-dimensional functions. To our best knowledge, no existing work focuses on guaranteeing safety during the high-dimensional optimization.

We present the novelty of OLLSO in both algorithmic and practical aspects.

OLLSO is a novel safe optimization algorithm. Previous GP-based safe optimization method are limited to low-dimensional problems (typically below 10, [2, 3]). OLLSO addresses this issue by introducing local search mechanism upon optimistic safety region identification. It is able to handle problem with hundreds of dimensions. The local region adaptation of OLLSO is also different from existing trust-region based BO methods[4, 5] which neglect the safety during the optimization procedure. Those methods would also discard all previous sample points when restarting the trust region. OLLSO updates the trust region in a more safety-sensitive way, and keeps all previous samples when restarting the trust region, ensuring a safer optimization procedure with theoretical guarantee.

OLLSO has a dimension reduction component to handle very high-dimensional or discrete input spaces. Previous latent BO works neglect the geometry-preserving during unsupervised representation learning, which is crucial to extend safety guarantee from the latent space to the original space. Many of them achieve latent space shaping using collected labeled data[6, 7]. We are the first to use distance-preserved VAE to address safety issue in latent space optimization, and derive the theoretical probabilistic safety guarantee of our proposed method.

Our experimental result also explains that OLLSO is not a direct application to comprise existing algorithms. Figure 2 and Figure 3 show that directly combining IRVAE with existing safe BO (CONFIG (L)) or trust region-based constrained BO (SCBO (L)) perform worse than OLLSO in terms of both optimization performance and safety guarantee. Our simulation results show that OLLSO significantly outperforms existing safe BO methods, constrained BO algorithms, and direct combinations with dimension reduction models.

We want to emphasize that OLLSO is a PRACTICAL method that can safely optimize high-dimensional real-world problems. In addition to state-of-the-art simulation performance, our real clinical experiments show that OLLSO successfully optimizes neural stimulation for the control of human muscles. While the majority of our cited papers conduct their experiments only in simulation or synthetic problems, OLLSO is both practical and theoretically verifiable to safely and efficiently optimize real-world safety-critical problems.

[1] Yonghyeon, L. E. E., et al. "Regularized autoencoders for isometric representation learning." International Conference on Learning Representations. 2021.

[2] Sui, Yanan, et al. "Safe exploration for optimization with Gaussian processes." International conference on machine learning. PMLR, 2015.

[3] Xu, Wenjie, et al. "Constrained efficient global optimization of expensive black-box functions." International Conference on Machine Learning. PMLR, 2023.

[4] Eriksson, David, et al. "Scalable global optimization via local Bayesian optimization." Advances in neural information processing systems 32 (2019).

[5] Eriksson, David, and Matthias Poloczek. "Scalable constrained Bayesian optimization." International Conference on Artificial Intelligence and Statistics. PMLR, 2021.

[6] Tripp, Austin, Erik Daxberger, and José Miguel Hernández-Lobato. "Sample-efficient optimization in the latent space of deep generative models via weighted retraining." Advances in Neural Information Processing Systems 33 (2020): 11259-11272.

[7] Grosnit, Antoine, et al. "High-dimensional Bayesian optimisation with variational autoencoders and deep metric learning." arXiv preprint arXiv:2106.03609 (2021).

Thank you for your appreciation of our paper and your extensive comments. We clarify our work based on the weaknesses and questions you mentioned.

Algorithm 1 explained in more mathematical details.

• (Line 1) we train an IRVAE using the unlabelled dataset $D_u^{\mathcal{X}}$ by adding an isometric regularization loss term over the original loss function (Line 1). Our training objective is

  $\min_{\theta} \mathcal{L}(\theta) + \mathcal{F}(\theta)$,

where $\theta$ is the parameters of the IRVAE, $\mathcal{L}(\theta)$ is the original loss function (typically using evidence lower bound in VAE), and $\mathcal{F}(\theta)$ is the additional regularization term proposed by [1] to shape the learned representation towards isometric.

• (Line 2-3) We initialize the sample trajectory as an empty set, and start optimization loop. The sample trajectory $\zeta_{t}$ is defined as a set of tuples {$(\boldsymbol{x}_t, y_t^f, y_t^g)$}.

• (Line 4-5) At each optimization round $t$, we use the trained IRVAE to map original sampled data to the latent space, and update the objective and safety GP model using Eq.2.

• (Line 6) We update the local region $L_t$ based on the sample trajectory $\zeta_{t-1}$. $L_t$ is a trust region defined by a center (using current best feasible point) and a length $l$. We denote successful/failure of the optimization round using the last sample result: A sampling round is considered "successful" if it finds a better reward while maintaining comprehensive safety. Conversely, it is labelled a "failure" if any unsafe points are found or if there is no discernible improvement. The side length is adjusted—increased ($l_t = 2l_{t-1}$) for successes and decreased ($l_t = 0.5l_{t-1}$) for failures—upon reaching a preset threshold.

• (Line 7-8) We use UCB of the safety GP to identify the safe region, and maximize the acquisition function (using TS here) to get next latent sample.

• (Line 9-12) We project the latent sample back to original space, evaluate the function values and update the sampled dataset and sample trajectory.

We will improve our presentation to help readers better understand our proposed method.

Link the theoretical result with the optimization objective.

In Theorem 1, we prove that with proper choice of the confidence bound scalar $\beta$, the selected safety value is lager than the estimated upper confidence bound with a probability larger than $\alpha$ (that is, $Pr(g(\boldsymbol{x}_t) \geq u(\boldsymbol{x}_t))\geq\alpha$). During optimization, we only consider points whose UCB is larger than the safety threshold $h$ as candidates (Line 7 in Algorithm 1). In this way we can guarantee that the selected safety value is lager than the safety threshold $h$ with a probability larger than $\alpha$ (that is, $Pr(g(\boldsymbol{x}_t) \geq h)\geq Pr(g(\boldsymbol{x}_t) \geq u(\boldsymbol{x}_t)) \geq \alpha)$.