Safe Time-Varying Optimization based on Gaussian Processes with Spatio-Temporal Kernel

W1

We thank the reviewer for the insight. We refer the reviewer to Table 1 in the pdf. The main difference between our method and context-based methods lies in how time is handled in the safe sets. Context-based methods require a safe seed to be available for every context or at every iteration, which may be impractical, as time is not a decision variable. Conversely, out TVSafeOPT ensures safety by tightening the definition of the safe set, and thus even just an initial safe seed would be sufficient for the algorithm to find a safe optimum.

W2

We provide a comparison of the performance of our proposed algorithm, TVSafeOPT, with the solution of an approximate optimization problem used in practice (baseline) and SafeOPT, in the PDF. Contextual approaches with time as a context would require a safe seed for every context, which is unavailable for the unknown changes in a compressor. Similarly, detecting changes due to degradation in turbomachinery is notoriously difficult [10]. The performance of event-triggered SafeOPT relies on the chosen event detection method, thus making the comparison of performance in the compressor case study inaccurate.

The results of the comparison are shown in Fig. 3 in the attached PDF.

W3

We added a comparison of safe sets computed by TVSafeOPT (top row in Fig. 1) and SafeOPT (bottom row in Fig. 1). Because TVSafeOPT takes the possible changes in time into consideration, the safe sets computed by TVSafeOPT are contained in the ground truth safe regions while those computed by SafeOPT have multiple violations. .

W4

Technically, the challenge arises in the non-stationary case from the fact that a safe/unsafe action at one moment might become unsafe/safe in the future. This means that a TVSBO algorithm should be aware of such information in order to find a sub-optimal solution while guaranteeing safety. ETSafeOPT [13] hopes to detect the change with an event trigger. However, in general, when the reward or safety functions change continuously or frequently, ETSafeOPT suffers from poor performance and even loses empirical safety guarantee. In contrast, context-based methods [1,3] well handle this issue with prior knowledge on the time change infused through the temporal kernel. However, existing context-based methods usually require an initial guess of safe set for each context value, which is often unrealistic. Our algorithm does not rely on change detection and overcomes the challenge related to the safe set initialization at every iteration. This is done by propagating the initial safe set to the future utilizing the temporal kernel. Theoretically, proving the safety and near-optimality guarantees for TVSBO algorithms is challenging even in the stationary case because the common proof techniques rely on the monotonicity of the lower bounds, upper bounds, and thus the safe sets, maintained by the algorithm. We lose such monotonicity by design of the algorithm. Instead we manage to circumvent this issue by proposing an lower bound of the safe set which is non-shrinking and converging. Furthermore, near-optimality guarantee in the non-stationary case is a very promising yet challenging problem. The first open-end question to answer is what kind of convergence property under what conditions we can prove for the non-stationary case.

Q1

Indeed, there is no technical difference in the two notations, we kept $x$ and $t$ separate to emphasise that $t$ indicates time and, as such, it is not a decision variable.

Q2

Indeed, the Lipschitz constant is usually unknown in realistic optimization scenarios. Its value can be obtained based on information about the capacity of the chosen kernel [7]. Such information, however, may be unavailable, making the choice a challenge. Oftentimes, however, an educated guess allows overcoming the aforementioned challenge. The authors in [12] have shown that in practice, it is often sufficient to choose an upper bound on $B$ [7], $\beta\geq 2$. This is needed to approximate the Lipschitz constant by an optimistic upper bound. Given that our method relies on Gaussian processes, we made a localized approximation of what has been proposed in [8].

Q3

Thanks. We will apply this change.

L1

We refer the reviewer to Q2.

L2

To make our contribution clear, we have now added an overview of the key aspects of existing methods for time-varying safe BO in Table 1 in the PDF. See also Fig, 2,3

[1] F. Berkenkamp, A. Krause, A. P. Schoellig, Bayesian optimization with safety constraints: safe and automatic parameter tuning in robotics,Machine Learning 2021;

[2] A. Holzapfel, P. Brunzema, S. Trimpe, Event-triggered safe Bayesian optimization on quadcopters, L4DC 2024;

[3] C. K"onig, M. Turchetta, J. Lygeros, A. Rupenyan, and A. Krause, Safe and efficient model-free adaptive control via Bayesian optimization, ICRA, 2021;

[4] C. K"onig, M. Ozols, A. Makarova, E. C. Balta, A. Krause, and A. Rupenyan, {Safe risk-averse Bayesian optimization for controller tuning, 2023;

[5] D. Widmer, D. Kang, B. Sukhija, J. H"ubotter, A. Krause, and S. Coros. Tuning legged locomotion controllers via safe Bayesian optimiztion, CoRL, 2023.

[7] J. Bergstra, R. Bardenet, Y. Bengio, B. Kegl, Algorithms for hyper-parameter optimization, NIPS, 2021

[8] F. Berkenkamp, A. P. Schoellig, A. Krause, Safe controller optimization for quadrotors with Gaussian processes, ICRA, 2016.

[10] Y. Li, P. Nikitsaranont, Gas turbine performance prognostic for condition-based manteinance, 2009.

[12] C. K"onig, M. Turchetta, J. Lygeros, A. Rupenyan, A. Krause, Safe and efficient model-free adaptive control via Bayesian optimization, ICRA, 2021

[13] A. Holzapfel, P. Brunzema, S. Trimpe, Event-triggered safe Bayesian optimization on quadrotors, L4DC, 2024

[14] M. Guptca, R. Wadhvani, A Rasool. Comprehensive analysis of change-point dynamics detection in time series data: A review, 2024

We thank the reviewer for the useful feedback, and the positive assessment of the paper. We provide a point-by-point answers to all raised suggestions, comments, and questions.

W1

We thank the reviewer for this suggestion. In the paper we are focused on safety critical systems where satisfying the safety constraints has highest priority over finding the optimum. The focus on safety is illustrated in Fig. 1 in the paper, where the proposed algorithm follows the safe set, even though the safe set changes over time. In particular, at time $t=170$, the algorithms takes decisions within a safe set that is smaller than the initial safe set, further emphasizing the safety at the expense of optimality. We will discuss these aspects in the Appendix C of the revised paper.

W2

We thank the reviewer for raising this question. As our approach is suitable for safety critical conditions, the focus is put on maintaining safety under change, therefore safety considerations ``dictate'' the optima. If the dimensionality of the problem increases, safety constraints might arise across multiple dimensions, from multiple directions at the price of optimality. This clarification will be included in Appendix C.

W3

Indeed, the choice of the safe set and the kernel is part of ongoing research [6] and typically requires some extended knowledge about the underlying functions[7.8]. A common choice for the kernel is a squared exponential kernel that intuitively puts emphasis on the points that are close to each other. The meaning of ``close'' is further defined by hyperparameters, which can be obtained for instance through maximum likelihood estimation [9].

Regarding the choice of the safe set, often a single safe point is already sufficient [Ch. 16, 10]. One of our assumptions is that at least one safe point is available in the initial safe set, and this is most often the case in practice. In most application it is reasonable to assume some prior knowledge is available, which in turn can be exploited in the initialization phase. This assumption is a direct consequence of the underlying principle that both the objective and the constraint functions can be measured. For example, in the compressor optimization case, the safe point is obtained from domain knowledge - an equal distribution of the gas is a safe point, but may be far from optimal if the compressors are dissimilar [10].

We further point out that most of the algorithms in this domain require the safe set to be reinitialized after each iteration (see Table 1 in the PDF and the associated references at the end of this paragraph). In contrast, TVSafeOPT requires only one initial safe set at $t=0$. As indicated, we will dedicate a subsection in Appendix C to address these points.

Q1

In complex setting it is still reasonable to assume that at least one single safe point is available. As we also mention in the answer to Weakness 3 raised from the reviewer, this would be already sufficient for TVSafeOPT to run and to guarantee optimality within the safe region that can either enlarge or shrink along time (see Fig. 1 in the submitted manuscript).

For example, in the compressor optimization case, the safe point is obtained from domain knowledge - an equal distribution of the gas is a safe point, but may be far from optimal if the compressors are dissimilar [11].

Q2

The choice of the kernel and the hyperparameters of the algorithm is a part of ongoing research [6] and typically requires some extended knowledge about the underlying functions [7,8]. A common choice for the kernel is a squared exponential kernel that intuitively puts emphasis on the points that are close to each other. The meaning of ``close'' is further defined by hyperparameters, which can be obtained for instance through maximum likelihood estimation [10]. One way to adjust the hyperparameters is to collect data in advance and optimize the hyperparameters via maximum likelihood optimization, following standard GP regression methods.

Q3

Through observations during the simulations, the computational complexity of each iteration of TVSafeOpt is only slightly heavier than SafeOpt; The sample complexity of the algorithm is similar to SafeOpt [1] in the stationary case and scales as $\mathcal{O}(\varepsilon^{-2})$, where $\varepsilon$ is the accuracy threshold for the optimization. In the non-stationary case, it is a still open problem to bound the sample complexity.

References

[1] F. Berkenkamp, A. Krause, A. P. Schoellig, Bayesian optimization with safety constraints: safe and automatic parameter tuning in robotics,Machine Learning 2021;

[2] A. Holzapfel, P. Brunzema, S. Trimpe, Event-triggered safe Bayesian optimization on quadcopters, L4DC 2024;

[3] C. K"onig, M. Turchetta, J. Lygeros, A. Rupenyan, and A. Krause, Safe and efficient model-free adaptive control via Bayesian optimization, ICRA, 2021;

[4] C. K"onig, M. Ozols, A. Makarova, E. C. Balta, A. Krause, and A. Rupenyan, Safe risk-averse Bayesian optimization for controller tuning, IEEE Rob. and Autom. Letters, 2023;

[5] D. Widmer, D. Kang, B. Sukhija, J. H"ubotter, A. Krause, and S. Coros. Tuning legged locomotion controllers via safe Bayesian optimiztion, CoRL, 2023.

[6] C. Fiedler, J. Menn, L. Kreisk ̈other, S. Trimpe, On safety in safe Bayesian Optimization, arXiv, 2024

[7] J. Bergstra, R. Bardenet, Y. Bengio, B. Kegl, Algorithms for hyper-parameter optimization, NIPS, 2021

[8] F. Berkenkamp, A. P. Schoellig, A. Krause, Safe controller optimization for quadrotors with Gaussian processes, ICRA, 2016.

[9] Y. Sui, A. Gotovos, J. Burdick, A. Krause, Safe exploration for optimization with Gaussian processes, PMLR, 2015.

[10] Y. Li, P. Nikitsaranont, Gas turbine performance prognostic for condition-based manteinance. Applied Energy, 2009.

[11] B. G. Liptak, Instrument Engineers' Handbook, vol. 2, 2005

W1:They extend the Safeopt algorithm from literature. However, it is clear on what are the additional contributions and difference between these two different approaches.

We thank the reviewer for the positive assessment of our paper, and for their constructive feedback. We now provide a table (see Table 1 in the PDF and refer to References) to give a clear overview of the contributions of our proposed approach TVSafeOPT, compared with other methods associated with time-varying safe optimization. In contrast to safe learning methods based on contextual Bayesian optimization, such as Contextual SafeOPT [1], which rely on using time as a context and thus require a safe seed for every context, our TVSafeOPT requires only an initial safe seed. Adaptive Goal Oriented Safe Exploration (A-GoOSE) can also handle a single initial seed, but without providing theoretical guarantees on safety across time [3,4]. Furthermore, compared to event-triggered methods such as ETSafeOPT [2], our method requires the initial safe set to be provided only once, at the initial time instant, $t=0$. In contrast, ETSafeOPT requires a new safe state initialisation after each iteration. Moreover, we are able to provide both safety and convergence guarantees to the optimal solution within the safe set, while ETSafeOPT guarantees only safety.

References

[1] F. Berkenkamp, A. Krause, A. P. Schoellig, Bayesian optimization with safety constraints: safe and automatic parameter tuning in robotics,Machine Learning 2021;

[2] A. Holzapfel, P. Brunzema, S. Trimpe, Event-triggered safe Bayesian optimization on quadcopters, L4DC 2024;

[3] C. K"onig, M. Turchetta, J. Lygeros, A. Rupenyan, and A. Krause, Safe and efficient model-free adaptive control via Bayesian optimization, ICRA, 2021;

[4] C. K"onig, M. Ozols, A. Makarova, E. C. Balta, A. Krause, and A. Rupenyan, Safe risk-averse Bayesian optimization for controller tuning, IEEE Rob. and Autom. Letters, 2023;

[5] D. Widmer, D. Kang, B. Sukhija, J. H"ubotter, A. Krause, and S. Coros. Tuning legged locomotion controllers via safe Bayesian optimiztion, CoRL, 2023.