Enhanced Bayesian Optimization via Preferential Modeling of Abstract Properties

We thank our reviewer's time and effort for providing a detailed review and raising interesting key points.

• BOAP assumes null human query cost_

  • Bayesian optimization (BO) is the workhorse of scientific experimentation and product design. It provides a rich platform for human-AI collaboration because the nature of BO applications ensures that there would be an expert running the optimization process. In BOAP, the expert inputs considered are qualitative and available as pairwise design preferences based on some abstract properties. Eliciting such pairwise rankings from the human expert should not add significant overhead, in contrast to specifying the complex explicit knowledge.
  • BOAP allows an expert to intervene only when they are willing to provide any preferential input thereby allowing them to evolve their knowledge over the course of optimization. Therefore, we have not considered any human query costs in the current version of the manuscript. However, it is an interesting direction to pursue that makes our proposed BOAP framework even more robust. In our experiments, at each iteration, we randomly choose $n/2$ datapoints to generate preferential data with the newly suggested datapoint.

• Formal definitions: Bayes regret, simple regret; Search space for the synthetic experiments

  • We thank our reviewer for the suggestions, and we will provide them in the updated version of the manuscript.

• How would this method compare with actually using the high-level features directly as additional dimensions?_

  • We would like to clarify to our reviewers that high-level features are not known directly to be used as an additional dimension, instead in practice an expert would reason about a system only qualitatively. However, we have conducted additional experiments to directly use high-level features as additional dimensions (BOAP-AD) and we have found that it outperforms BOAP by a small margin we believe this phenomenon is due to the fact that explicit measurements are always better than approximate qualitative measurements.

  ---

  |                 |      BOAP-AD     |       BOAP     |
  

  ---

  | Benchmark - 1D  |     0.01±0.002   |     0.05±0.01  | 
  | Rosenbrock - 3D |     3.16±1.19    |     5.32±0.12  | 
  

  ---

We appreciate our reviewer's patience, time and effort to provide us with detailed feedback and raise interesting key points.

• Why properties are abstract (i.e. immeasurable) and why existing BO approaches are ill-suited

  • In numerous scenarios, experts often reason about a system based on their approximate or qualitative knowledge of the system at different abstraction levels. They reason using intermediate physical properties that may not be measurable directly. Using such high-level abstractions the expert compares designs and the reason why a design is better than another.
  • The existing BO approaches aim at capturing the expert preferential input directly on the objective function, in contrast to the preferences based on the aforesaid intermediate properties. Therefore we propose a human-AI collaborative framework to incorporate such expert preferential data about unmeasured abstract properties into BO.

• Why are the lengthscales fixed for the original input dimensions?

  • We would like to clarify to our reviewer that in the augmented GP ($\mathcal{GP}_{h}$), we still tune the hyperparameters of the original input dimensions by maximizing the log marginal likelihood. We use a modified ARD kernel that uses a spatially varying kernel with a parametric lengthscale function for each of the input dimensions. For each un-augmented feature, we set the lengthscale function to be a constant lengthscale value $l(\mathbf{x})=l$, whereas for each of the augmented auxiliary features we set the lengthscale function to be $l(x)=\alpha\sigma(\mathbf{x})$, where $\alpha$ is the scale parameter and $\sigma(\mathbf{x})$ is the normalized standard deviation predicted using the rank GP.

• How would the non-user input model ever be chosen? How frequently it is chosen?

  • During the initial phase of optimization the un-augmented control arm is pulled as the augmented arm may have insufficient data to reduce uncertainty in posteriors, making the augmented features only slightly better than noise. However, the augmented features become more accurate through the optimization process as we accumulate more data, so we observe the augmented arm to be more accurate at the end of the optimization.
  • Based on the empirical results we could verify that the augmented model was chosen more than the un-augmented arm. In an ideal scenario with accurate preferential data, we have recorded the percentage of times each was pulled and the results are as follows.

  ---

  |            |   Augmented Arm   |   Control Arm   |
  

  ---

  | Benchmark  |     76.23±2.5     |     24.22±1.2   | 
  | Rosenbrock |     71.87±4.6     |     25.67±2.75  | 
  | Griewank   |     63.95±3.1     |     35.73±2.6   | 
  

• {How is the next input chosen along the added dimensions?

  • In the augmented search space we have $d+m$ features in total, here we Thompson sampling method operating in $d+m$ dimensions to find the next promising candidate for the function evaluation. In the case when the augmented arm is pulled, we randomly draw a sample from its GP and find its corresponding maxima $\mathbf{x}_{t}^{\mathfrak{h}}$. The objective function in the augmented space $h(\tilde{\mathbf{x}})$ is a simpler form of $f(\mathbf{x})$ with auxiliary features in the input, and therefore we observe $h(\tilde{\mathbf{x}})$ via $f(\mathbf{x})$.

• How frequently does the user have to be queried for feedback, and how many user queries do a 20-iteration run entail.

  • BOAP allows an expert to intervene only when they are willing to provide any preferential input thereby allowing them to evolve their knowledge over the course of optimization. The preferential feedback is provided by experts at their discretion i.e., when they feel they can make a clear judgement or that the algorithm has gone astray. How the expert intervenes in giving preferential feedback is not controlled by the algorithm.
  • In our studies with simulated experts (zero query cost), at each iteration, we randomly choose a subset of datapoints containing $n/2$ datapoints to compare with the new candidate suggested for function evaluation. Therefore in a $20$ iteration run, we generate $133$ preferences in total.

• Procedure for providing abstract features

  • For synthetic functions, based on their mathematical formulation, we have considered the high-level features as mentioned in Table 1 of the main paper. We treat such high-level features as a continuous function $\omega(\cdot$) and generate the preference data $P$. For real-world experiments, due to the cost and access limitations of experimental facilities we have used a few columns $(\omega)$ in the published dataset as high-level features. We generate preference data by comparing those columns values $(\omega(\mathbf{x}))$ for two datapoints $(\mathbf{x},\mathbf{x}')$.

Thanks to the authors for their clarification.

Why are the lengthscales fixed for the original input dimensions? These, to me, still seem like unconventional choices. I would encourage the authors to carefully motivate why some lengthscales are fixed (which is a rather drastic modeling choice) so that the proposed method can be evaluated on its own merit.

How is the next input chosen along the added dimensions?

I am still not quite following. Along the preferences, conducting TS, to me, seems to equate to choosing a preference (which are determined by the user, not by BO). As such, I am not following how this can be done in practice.

How would the non-user input model ever be chosen? How frequently it is chosen?

I do not believe the right question was answered here. My question, specifically, is: Since the augmented model has a larger capacity (more dimensions), it should in theory yield a larger likelihood at all times if they are compared on the same (subset of the) data. Yet, it does not seem to do so. Why does it not have a larger likelihood at all times? Surely, I must be missing some aspect of the algorithm here, and it would be great if the authors could clarify what this is.

How frequently does the user have to be queried for feedback, and how many user queries do a 20-iteration run entail? This is valuable context, and I believe its inclusion in the paper would strengthen it.

My concerns regarding the theoretical discussion have unfortunately not been addressed. My overall perception of the paper has not improved, and as such, I will retain my score.

We appreciate the reviewer's time and efforts to provide detailed feedback and raise a few interesting points.

• Why this instead of learning the parameters as one would typically do with GP models?

  • Tuning the human augmented GP ($\mathcal{GP}_{h}$) using the traditional loglikelihood/evidence maximization does not account for the expert uncertainties for the augmented features. Incorporating expert uncertainties from the individual rank GPs can be crucial for modelling as the expert feedback may be inaccurate and thus can adversely affect the overall optimization performance.

• why don't we explicitly exploit this compositional structure as done in Astudillo and Frazier (2020)?

  • Our idea is fundamentally different from Astudillo and Frazier (2020) as we try to incorporate expert preferential feedback about the immeasurable abstract properties in modelling the given objective function $f$. To the best of our knowledge, Astudillo and Frazier (2020) capture expert preferences (along with their uncertainties) directly on $f$ and fail to account for auxiliary abstract properties that can be crucial for improving the optimization performance.

• The choice of scale parameter $\alpha$

  • The predictive variance ($\sigma^{2}$) in the rank GP is a measure of uncertainty in the expert model. In our proposed approach, the scale at which this uncertainty is to be considered in modelling the overall objective function is dictated by the scale parameter $\alpha$ and $\alpha\times\sigma$ will be the underlying lengthscale. We tune the scale parameter for each of the augmented dimensions by maximizing the log-likelihood. Alternatively, we could treat $\alpha$ as an overall "average" lengthscale, such that $\alpha$ is the underlying lengthscale when there is no uncertainty and with expert uncertainty the overall lengthscale is $\alpha+\sigma$.

• Different kinds of human errors can affect the performance of BOAP}

  • We would like to refer our reviewer to the additional experiments mentioned in the supplementary material (Section A.4). We have conducted experiments to account for the human expert errors in providing preferential feedback. We have done this ablation study in two-fold. First, we show the robustness of our proposed approach against human errors in the selection of higher-order abstract properties. Second, we account for the human errors in providing the expert preferential feedback by flipping the preference between two input designs with some probability $\delta=0.3$. The ablation results and the experimental details are provided in the supplementary material (A.4.2).

• Expert preferential knowledge can sometimes be misleading

  • We expect that the human expert preferential input is accurate and thus likely to accelerate BO. But, in some cases, this preferential feedback may be inaccurate due to the wrong/incorrect assumptions on the given search space. Therefore, such misguided preferential data given by an expert could potentially slow down the optimization. On the other hand, our method has some immunity to such inaccurate feedback.

• Normalization in Rank GPs

  • The individual rank GPs (${\mathcal{GP_{\omega_i}}}$) operate at different output scales (as dictated by MAP) and thus different scaling levels in their corresponding predictions. Therefore, before we augment the main GP with the mean predictions ($\mu_{\omega_{i}}(\mathbf{x})$) of the individual rank GPs, we min-max scale their outputs to be normalized in the interval $[0,1]$.

We thank our reviewer's time and effort invested in understanding the paper to raise some interesting points.

• GP-predicted values as inputs to another GP? Has anyone studied this model?

  • Using GP predictive distribution to model another GP has been explored in the past. [1] proposed a two-layered hierarchical model for GP regression that partitions the input data points such that the upper layer is responsible for coarse modelling and the lower layer is responsible for fine modelling.
  • [1] Park, Sunho, and Seungjin Choi. \textquotedbl Hierarchical Gaussian process regression.\textquotedbl{} In Proceedings of 2nd Asian Conference on Machine Learning, pp. 95-110. JMLR Workshop and Conference Proceedings, 2010.

• Why predicting the properties fìrst and then predicting $y$ is easier than predicting directly?

  • The abstract properties can be simple physical properties or even a combination of multiple physical properties that an expert uses to reason about why one design is better than another. A well-chosen abstract property captures the input space of a system sufficiently enough that enable the domain expert to better reason about the output of the system in terms of such higher-order abstract properties. The main objective of augmenting abstract properties into the input space is to capture the inherent transformations of the function space $\mathcal{F}$ that significantly reduce the complexity and thereby simplify the task of GP modelling, thus a better overall optimization performance.
  • Furthermore, human expert feedback is not available always, thus we cannot simply use expert-derived feedback as input. However we model this (intermittent) expert feedback using a GP, and that model will act as a proxy for the expert in experiments where no expert feedback is given.

• The theory quoted in this paper from Russo & Van Roy, 2014 may not be directly applicable?

  • At every step of a typical BO, the kernel hyperparameters are updated, so technically speaking the model is different at each step. Nevertheless, the results of papers like Russo and Van Roy, while not precisely matched to reality, are considered indicative of performance, typically in big-O form (no one expects exact predictions, just an indication of expected performance). The only real distinction in this case is that our kernel - which in effect includes the posterior mean/variance of the expert models in the kernel definition - varies more radically in the early stages of optimization. We discuss Russo and Van Roy in this sense - a guide to what we might expect, not a limiting framework.

• -Dynamically setting kernel lengthscales based on a different model at different inputs? Is this provably a p.d kernel still?_

  • Spatially varying lengthscales have already been explored in the past literature. [2] proved the positive definiteness of the spatially varying kernel by showing that it has the required property of spatial variation of the lengthscales by suitably replacing the constant lengthscale with an arbitrary parameterized function of x. We refer our reviewer to \textquotedblleft Non-stationary Covariance Functions\textquotedblright{} Section (2.3.2) of [2] for the detailed discussion on spatially varying lengthscales and the resultant positive definite kernels.

  • [2] Gibbs, Mark N. "Bayesian Gaussian processes for regression and classification", PhD diss., University of Cambridge, 1998.

• No actual human-in-the-loop experiment is conducted

  • Level of expertise required to conduct these experiments is high and access to such real experts is very difficult. Due to the cost and access limitations of experimental facilities, we rely on numerical simulations and simulated data.