Density Ratio Estimation-based Bayesian Optimization with Semi-Supervised Learning

We sincerely appreciate your constructive comment to improve our work.

The introduction is poor. It does not introduce properly the problem addressed in the paper.

In general the writing of the paper has to be improved a lot. It does not introduce the concept of DRE-based BO right.

We will revise our paper more carefully in the final version.

It is not clear what is the motivation for DRE-based BO. It is also unclear how the threshold value $y^t$ is chosen.

Please, explain how the threshold value $y_t$ is chosen.

We followed the previous work (Tiao et al., 2021; Song et al., 2022). In order to focus on the use of unlabeled points in the process of DRE-based Bayesian optimization, the selection of $y^t$ is built on the previous literature. Please refer to the prior work.

The authors have to better explain DRE-based BO, why does it work, why is it interesting and why it is better than regression based BO.

We have included why DRE-based Bayesian optimization works in Section 3. Moreover, compared to GP-based Bayesian optimization, surrogate model learning is more efficient. In particular, while GP scales cubically with the number of points, learning a classifier is likely to be faster. Moreover, as shown in the experiment Minimum Multi-Digit MNIST Search of Section 5.2, the use of classifiers helps learn the representation of high-dimensional data.

Figure 1 is not explained properly.

Please, explain better figure 1 and why it illustrates the over-confidence problem.

To accommodate your comment, we have updated Section 1.1. Please see Section 1.1.

The proposed method is not very well motivated. It seems it is simply using a better classifier. The authors claim that they use semi-supervised learning techniques to train the classifier.

We do not agree with this comment. Our method offers a way to employ unlabeled points into the procedure of Bayesian optimization. In particular, in the scenario of pool-based Bayesian optimization, the vanilla Bayesian optimization does not utilize unlabeled points in a pool. Our method with semi-supervised learning enables us to utilize them in Bayesian optimization. Therefore, our contributions are more than simply using a better classifier.

Please, explain why the proposed method needs to sample from a truncated multivariate Gaussian distribution.

The search space of Bayesian optimization is generally defined as a hypercube. Thus, we require utilizing a truncated multivariate Gaussian distribution so that the sampled points should be confined in the hypercube.

We sincerely appreciate your constructive comment to improve our work.

What does the figure specifically illustrate?

We illustrated $p(\mathbf{x} | y \leq y^\dagger, \mathcal{D})$. We would like to emphasize that we do not model $y \leq y^\dagger$ and instead it indicates some class, e.g., Class 1, which implies points that might have function values smaller than $y^\dagger$. By following the conventional expression of machine learning, the probability we illustrated is expressed as $p(\mathbf{x} | c = 1, \mathcal{D})$ where the probability of interest is the probability of $c = 1$.

Unfortunately, the efficacy of proposed method in the performance of optimization over iterations is hardly distinguishable in Figs. 5 or 6.

We disagree with this comment. Our methods quickly converge to solutions compared to other baseline methods. In particular, Figure 6 shows the superior performance of our methods.

The bigger problem is that it is unclear from the text on what condition and why the proposed method outperforms the existing approaches.

Based on the use of unlabeled points and semi-supervised learning, our methods can estimate more informative probabilities, thereby easing the over-confidence problem discussed in Figures 1 and 8 and Section 1.1.

How is matrix $\mathbf{A}$ designed? What covariance is taken for this disbiribution, for what? What are lower and upper bounds $\mathbf{l}$ and $\mathbf{u}$? Are they boundaries of search space $\mathcal{X}$? Is it assumed to be rectangular?

Covariance matrix was designed as an identity matrix, and then $\mathbf{A}$ can be calculated. Since the search space of Bayesian optimization is often assumed to be a hypercube, we also used a hypercube space. Thus, $\mathbf{l}$ and $\mathbf{u}$ can be determined by the search space given. We have updated our submission accordingly.

Does fixed-size pool scenario mean some finite number of candidate points $\mathbf{x}\_i$ are selected in advance and that optimized over these points? If true, is the set of points dynamically updated or fixed in advance?

Yes, it is optimized over a fixed number of points. The set of points is not dynamically updated by following the general formulation of pool-based optimization.

What is the definition of simple regret? Is it $f(\mathbf{x}\_n) - f^*$ in the $n$th iteration, where $f^*$ is the minimizer of function $f$. I am wondering why the regret is monotonically decreasing in Fig. 2. Does this plot $\min\_{n} f(\mathbf{x}\_n) - f^*$?

The definition of simple regret is $\min\_{n} f(\mathbf{x}\_n) - f^*$, which is widely used in comparing Bayesian optimization algorithms. We have included this definition; please see Equation (12).

Are they all two? This is crucial information on the difficulty of black-box optimization problems.

Yes, they are two-dimensional problems. We have added the specifics of the synthetic benchmarks; see Section D.2.

How do you define the distance $|\mathbf{x}\_i - \mathbf{x}\_j|$, especially when categorical values are involved such as activation function and learning rate schedule.

To handle categorical and discrete variables, we treat them as integer variables by following the previous literature (Garrido-Merchan & Hernandez-Lobato, 2020). After converting them to integer variables, the distance between two integer variables is easily defined. We have updated Section D.3 accordingly.

We sincerely appreciate your constructive comment to improve our work.

One doubt I have with this paper perhaps stems from unfamiliarity with semi-supervised algorithms in general. My understanding of such approaches is that they tend to assume that the unlabelled training points are nevertheless generated from the underlying x distribution. In most applications of BO, however, this concept is nonsensical: the only x distribution is the points sampled by BO, which are (in some sense) arbitrary (depending on the acquisition function they may cluster around the optimum as time passes, but not necessarily). Am I missing a key point here?

We have solved two scenarios, pool-based Bayesian optimization and generic Bayesian optimization (defined on a continuous search space). Notably, the formulation of pool-based Bayesian optimization has access to a fixed-size pool, which can be used as unlabeled points. The availability of the fixed-size pool makes DRE-based Bayesian optimization naturally built on semi-supervised learning. On the other hand, as pointed out by your comment, for the scenario of generic Bayesian optimization, the generation of unlabeled points may be nonsensical. However, our sampling process based on the truncated Gaussian distribution allows us to focus on the proximity of the query points that have already evaluated. This is aligned with the nature of Bayesian optimization, i.e., the consideration of both exploitation and exploration.

We sincerely appreciate your constructive comment to improve our work.

The presentation could be improved. Specifically, the focus on over-confidence in the beginning is confusing and I only understand the main point until I read section 3.1 where the relation to exploitation is mentioned. Perhaps this should be moved towards the front.

Thank you for pointing this out. We moved Section 3.1 towards the front.

Some limitations of the work should be explicitly mentioned/discussed. For example, assumption 4.1. seems to imply that the method is only intended to work on smooth functions.

We have described the limitations and societal impacts in Sections H and I. Moreover, we think that the limitation on the smoothness of an objective, which is mentioned by the reviewer, is also related to kernel selection. In Bayesian optimization common stationary kernels work well when an objective is smooth. Therefore, Assumption 4.1 does not significantly add the additional limitation of Bayesian optimization.

The over-confident problem of DRE is recently studied by [1,2]. Can the author(s) comment on how the construction of auxiliary distribution in these work is related to the sampling distribution of DRE-BO-SSL?

We cannot find [1, 2] in your comment. Please let us know which papers you mentioned. We can add a response to this concern later.