A General Framework for User-Guided Bayesian Optimization

We thank the reviewer for their positive review. We have addressed the minor comments (except for the plots, which we will have to edit for the CR).

Hopefully our new additions, mentioned in the global reply further enhances the reviewer's perception of our work. If so, we would greatly appreciate if the reviewer would increase their score, or confidence, to reflect this.

I sincerely thank the authors for doing their best to address my concerns. I can imagine that the authors spent quite a tough few weeks!

It is a little bit sad that making the experiments fairer took a toll on the significance of the proposed methodology. But still, it appears that the proposed framework has benefits, especially in terms of worst-case performance on the Hartmann 6 problem. Furthermore, I find the technique itself novel and interesting, as mentioned in my original review. In light of the revision, I am happy to raise my score.

Some additional comments:

• Still, most of the experiments are less than 10 dimensions. To increase the impact of the work, I recommend higher dimensional problems.
• The reference formats are not very consistent. For example, some proceedings are in capitals, while some are in sentence cases. Some ArXiv papers have the ArXiv code, but some don't (Huang 2022, for example), and some have URLs while others don't. I recommend polishing the reference section.
• Eq (6): Seems like using $x$ is a little confusing here. How about using $y$ to denote the function values?

Lastly, it seems that the authors might have missed my following comment:

Given that the authors rely on GP posterior sampling, an obvious thing that could have been tried is Thompson sampling. Have the authors tried to include it? I would be interested to hear if the authors could run additional experiments with Thomson sampling. (Given that the rebuttal period is almost over, this is a friendly suggestion, not a request!)

We thank the reviewer for their thorough and informed feedback. We generally find the review to be insightful and have done our best to address the weaknesses in our work. In the newly uploaded pdf, we have:

1. Added five additional HPO tasks (6D) from LCBench. Hopefully, this tackles the reviewer's concerns regarding the experimental evaluation. Moreover, all methods now utilize the same $\text{Mode} + 2$ initialization, on all tasks. Lastly, we have added an illustrative example of how priors over preference relations can be used in ColaBO in a newly added Fig. 1 in the intro.
2. Worked on clarity in Sec. 3 throughout. We address specific feedback below, but this includes:
   • The role of $\pi$ and its conditioning - substituted for $\rho$ to separate
   • Densities and integrals - some missing proportionalities and normalizing constants
   • GBI relation removed.
3. Specified contributions to better distinguish ColaBO from other user-guided BO.
4. Minor comments/faults have been addressed. For a discussion regarding each point of feedback, see below:

The mathematical derivation is too informal in some places, affecting clarity. I believe this needs to be improved. For instance, here represents a belief over functions. Then why is Eq. (4) a function that receives a point ? Shouldn't it be a probability distribution over functions? Also, Def 3.1 introduces a conditioning on $\pi$. I found this quite confusing. Is $\pi$ a density? a function? or a random variable?

We see that the confusion is warranted, and have consequently re-hashed section 3.1. Hopefully, this improves the understanding and overall experience. $\pi (f)$ is replaced by $\rho (f)$. Moreover, the domains, co-domains and variable properties (i.e. randomness of $\rho$ is communicated throughout.

The idea of Eq. 4 was to demonstrate how $\pi(x)$ can be expressed as a function not of the optimizer $x^*$, but as a function of $f$. We subsequently introduce notation where $\rho$ is a function of $f$ for some arbitrary function property, e.g. $x_*$ and $f_*$. We have now re-done the notation to clarify the inputs to each quantity and hope that this improves the overall clarity.

In Section 3.3, the authors introduce a tempering scheme based on the number of datapoints. Furthermore, they draw connections with generalized Bayesian inference (GBI). I think the authors should make it clear that the connection is very loose.

We agree with the reviewer, and that section is now entirely removed. In later experiments, we found that the tempering did not substantially improve performance, and since its "principled" nature was debatable, we removed it to obtain an overall cleaner approach. In practice, the data is sufficient to decrease the impact of a poor user belief.

The derivation of importance sampling in Eq. (10) is unclear. is an unnormalized density, but the integral is taken over it. Then, the equality with Eq. (9), doesn't hold since the expectation is not normalized.

Thanks for catching this typo. We missed a normalizing constant on $\pi$, which will not generally be tractable. We have added this normalizing constant and clarified accordingly. We have added a breif comment on the sampling, but will cover it in more detail in the CR when space permits.

We thank the reviewer for their positive review.

Test functions used to evaluate the proposed framework were quite limited, only a restricted set of test functions was employed.

We have added 5 6D HPO tasks from the LCBench HPO benchmarking suite.Hopefully, this alleviates the reviewer's concern and further demonstrates the empirical performance and flexibility of ColaBO.

Besides the likely location of the optimizer or the optimal value, what other forms of prior knowledge would generally be useful?

We have added an additional example using beliefs about preference relations in the new Fig. 1,, where the user specifies pairs of configurations and their belief over which one is likely to be superior of the two. In the figure, we show that this type of prior yields a uniquely shaped model, and further emphasizes the flexibility of the framework.

We hope that the additional tasks and Figure 1, which highlights the versatility of ColaBO, has increased the reviewer's confidence in the quality of our paper. If the reviewer has any additional questions in the remaining time, we would be happy to address them.

We thank the reviewer for their informed feedback. We agree with the reviewer on the outlined weaknesses, and have addressed them, incorporating the following in the newly uploaded manuscript:

1. We have added more HPO tasks (6D) from LCBench. Hopefully, this gives a more complete view of performance of ColaBO. They can now be found in the newly added Appendix A, but will go into Sec. 4 when space permits.
2. Introduced examples (similar to Fig. 1) of ColaBO with max-value priors and priors over preferences to better highlight how various tasks can be addressed within the ColaBO framework.
3. Re-worked notation in Sec. 3. The notation of $\pi(f)$ is gone, and replaced with $\rho(f)$ to better distinguish between them. Moreover, the properties of $\rho$ are clearly specified, and it is defined in what we believe is a more well-defined manner. However, we have re-hashed and re-worded the notation to hopefully improve the reading experience. While time is short, we would greatly appreciate additional feedback on how clarity can be improved here.
4. Previous Eqs. 8-13 have been clarified with regard to $f_*$, $x_*$ and the typo $f_n^*$. Below, we address each of the reviewer’s comments in detail.

Responses to concerns:

Although the empirical performance of ColaBO looks promising in the synthetic task and hyperparameter tuning task, the theory developed in the work is limited.

We respect the reviewer's viewpoint and agree that the theoretical work is not extensive. However, the introduced theory serves the necessary purpose and yields a valid, conceptually simple and broadly applicable approach.

Notably, there is a rather large collection of work in user-guided BO of various kinds (cited in the intro of the paper) which utilizes various types of knowledge, and ColaBO can successfully handle all of these types in a principled manner without altering the approach. As such, one can argue that this one piece of theory produces the general framework that user-guided BO has been missing. The newly added Fig. 1 exemplifies how all of these user beliefs are seamlessly dealt with by ColaBO.

We acknowledge that work on scalability and approximation quality should be addressed moving forward, but ultimately view this line of work as follow-up.

The presentation of the work can be improved. Initially, the authors define in Eq. 3 with input from $\pi(x)$, but it later becomes $\pi(f)$. Moreover, the notations become messier after Eq. 4, making it more challenging to follow. Also, while rejection sampling seems to be an important step in the model, the way it works is not introduced in the paper. Similar for RFF.

We outline RFFs in the first paragraph of 2.3 and couldn’t find space to expand on it further. The exclusion of various sampling approaches (rejection, importance) but will be added to a camera-ready along with RFFs when space will permit it.

We have made our best attempt at addressing your broader concern regarding clarity. Section 3.1 is now presented using slightly altered notation, where each property is properly distinguished and defined.

Given the authors’ claim that this is a general framework for BO to incorporate prior, the paper should present concrete examples to demonstrate its generality. For example, the authors could show how the examples in the second paragraph of the introduction section can be solved effectively within the proposed framework.

We agree that the conventional prior over the optimum is over-emphasized in the experiments we had presented. To the reviewer’s point, we have added Fig. 1 to highlight examples of how each type of prior is incorporated into ColaBO. Hopefully, this serves to highlight the generality of ColaBO, as requested by the reviewer.

Moreover, there are max-value prior tasks in App. C.2, Fig. 10. We will try and highlight these types of tasks in the paper by adding yet another one.