Informed Initialization for Bayesian Optimization and Active Learning

We thank the reviewer for their review of our work. We appreciate the recognition of the importance of hyperparameter initialization and the clarity of our presentation. Below, we address the specific questions and concerns raised.

 

1. (...) the EPIG objective targets reduction in predictive uncertainty, not space-filling design. Under what conditions are these equivalent? If and when they disagree, should the user prefer the design that reduces predictive uncertainty, and why? If not, why not use an acquisition function that optimizes for space-filling design directly?

   Great question. Assuming space-filling refers to criteria like minimax-distance or similar, the EPIG objective and true space-filling designs are most likely never equivalent. Referring to our designs as space-filling is a misnomer on our part, we will adjust this.

   Nonetheless, space-filling or random designs are often employed to globally reduce uncertainty, a view supported by many related works [we will add citations]. If this is indeed the purpose, then such a criterion should be employed directly.

   It is certainly possible—and advisable—to prefer a true space-filling criterion (e.g., minimax) if strict input space diversity is desired. However, this would, by definition of the criteria, be less helpful for prediction, which we consider the more important measure for Bayesian Optimization performance.

 

2. How are the batches selected? Especially in the large-batch setting, enumerating over every possible batch of locations is generally computationally intractable. It’s common to approximate the optimal batch by greedily optimizing the acquisition function over candidate locations one-at-a-time. How would this practice affect the performance of HIPE?

   All acquisition functions we consider are quasi-Monte Carlo (qMC) acquisition functions, meaning the joint utility (information gain or improvement) is computed as an expectation jointly over samples from $y(X)$ over a batch of $q$ candidates. This allows optimization over the full $q \times D$-dimensional space, enabling arbitrary batch sizes without enumeration. This general qMC-methodology is described in detail in [1, 2], which we cite in Section 4.2 to explain how HIPE fits into this framework.

   We conducted the full joint optimization using the MC formulation since it was feasible, but a sequential approach to acquisition function optimization would be feasible as well. In low-dimensional ablations, we found that joint optimization yielded slightly larger acquisition function values than sequential, but we do not expect that there would be substantial performance differences between them.

 

3. I find the notation for the information theoretic measures a bit confusing. In line 79, $y(x)$ appears to be defined as a given value of the measurements (a scalar or vector). In eqs. 1, 2 and 4-8, is $y(x$) intended to be a fixed value of the measurements, or the random variable characterizing measurements that arise from a particular value of $x$? The latter is consistent with standard definitions of the EIG. If the latter, should the expectation in eq. 1 be w.r.t. this random variable? If not, could you please clarify how to interpret $y(x)$; what does it mean to take the entropy of a fixed value?

   Thank you for carefully reviewing the notation. $y(x)$ and $y(X)$ always denote the random variable characterizing measurements arising from $x$ (or $X$ in the multivariate case). We acknowledge that the notation was overloaded and will clarify this in line 79. The expectation in Eq. 1 is a typo and should indeed be over the random variable $y(x)$. We appreciate the reviewer for spotting this.

 

---

We hope these clarifications address the reviewer’s outstanding questions and concerns. If not, do not hesitate to ask further clarifying questions. We appreciate the constructive feedback and hope that, if satisfied with our responses, the reviewer might consider raising their score to reflect the strengths and contributions of our work.

---

  References:

[1] James T. Wilson, Riccardo Moriconi, Frank Hutter, Marc Peter Deisenroth. The reparameterization trick for acquisition functions.

[2] Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, Eytan Bakshy. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization.

We thank the reviewer for their insightful and constructive feedback. We appreciate the recognition of the importance the topic, and the quality of the presentation. Below, we address the specific questions and suggestions raised.

1. Why don't you take Query-By-Committee and its variant (Riis et al., 2022) as a baseline since it outperforms BALD in the settings used in their paper?

   While BQBC outperforms BALD in the Riis et al. paper, it does not do so in the Hvarfner et al. paper, where BALD performs substantially better on identical benchmarks. BQBC performs identically across the two papers. We suspect there may have been a bug in the Riis et al. implementation of BALD, since it has subsequently been more competitive. Therefore, we consider our choice of BALD baseline appropriate.

 

2. Would Statistical distance-based Active Learning from Hvarfner et al., 2023 be another choice as a comparative acquisition function?

   Statistical distance-based Active Learning (SAL) is indeed a strong candidate in the sequential setting but as far as we know, it has not been tested in a batch setting. However, we have run the version that is publicly available in BoTorch. This variant is runnable as a batched acquisition function. Below, you can see updated AL task performance with SAL included:

   MLL performance (larger is better): HIPE performs best, followed by BALD and SAL, which are both competitive.

   Function	BALD	HIPE	NIPV	Random	SAL	Sobol
   Australian	-2.66 ± 0.08	-2.61 ± 0.22	-7.47 ± 0.80	-3.00 ± 0.17	-4.11 ± 0.73	-3.40 ± 0.24
   car	-3.07 ± 0.45	-2.68 ± 0.02	-6.09 ± 0.65	-3.09 ± 0.14	-4.16 ± 0.82	-3.30 ± 0.27
   hartmann6	-0.48 ± 0.02	-0.31 ± 0.02	-0.43 ± 0.06	-0.57 ± 0.07	-0.48 ± 0.02	-0.63 ± 0.07
   ishigami	-1.74 ± 0.09	-2.59 ± 0.23	-6.94 ± 1.26	-2.45 ± 0.17	-2.50 ± 0.42	-2.32 ± 0.12

 

RMSE performance: NIPV generally performs best. Batched SAL does not perform well on RMSE.

Thanks for the suggestion to include SAL, we will happily include it in the results for the CR.

 

3. I am wondering whether there are any theoretical guarantees to proof for your acquisition function.

   Our acquisition function is designed to produce a well-informed initialization to improve subsequent Bayesian Optimization performance rather than to minimize regret immediately. As such, regret bounds are not currently available. While theoretical guarantees on model predictive performance might be possible, such results are rare in the broader Bayesian active learning literature. We acknowledge this as a limitation and an interesting direction for future work.

 

---

We hope these clarifications address the reviewer’s questions and concerns. We appreciate the constructive feedback and hope that, if satisfied with our responses, the reviewer might consider raising their score to reflect the strengths and contributions of our work.

We thank the reviewer for their detailed and thoughtful review. We appreciate the recognition of the importance of the method, as well as its simplicityekecnerbflrktggenuicujfrfttluvnj, the clarity of our presentation, and the extensive empirical evaluation. Below, we address the specific concerns raised.

 

1. As far as I understood, the novel acquisition function proposed is a linear combination of EPIG and BALD. Therefore, it is not necessary to discuss NIPV in Sec. 2.

   It is correct that our method is a combination of EPIG and BALD. We introduced NIPV because it is strongly related to EPIG and is arguably the more well-known method in the broader active learning literature.

 

2. It is not clear what the authors consider to be ‘optimal’ in this example (Fig. 1). It is not easy to see why HIPE selects better points than NIPV since both have broad coverage.

   We try to motivate this in the text accompanying the figure but agree it should be clearer in the figure itself. The coverage is basically equally good for NIPV and HIPE, but HIPE additionally has the hyperparameter-informative axis alignment like BALD. Thus, by design, HIPE intuitively should be better for hyperparameter learning than NIPV while visually retaining equal coverage.

   We will add helper lines to the figure to emphasize the axis alignment in HIPE.

   Figure 1 is meant primarily as an illustration to provide intuition of the typical behavior of the different initialization approaches; in general it is not straightforward to draw conclusions on the performance without actually evaluating performance (in terms of model fit and/or optimization performance).

 

3. Prop. 1 introduces a connection between EPIG, BALD, and HIPE. However, it is not clear what this connection means for the proposed method. For example, does this influence the convergence properties of the BO?

   Proposition 1 shows that, for a certain choice of beta, HIPE is equivalent to jointly maximizing both EPIG and BALD, justifying the linear combination from a theoretical perspective. However, this has no direct influence on subsequent BO convergence. A formal proof of convergence for BO with dynamically updated hyperparameters (e.g., MAP or fully Bayesian estimates) is still an open problem and beyond the scope of this work.

 

4. In line 209, the authors claim that ‘entropy in Eq. 10 is Gaussian’. Can the authors provide proof or a reference showing that this is the case?

   Our original formulation was imprecise. More precisely, $y(x)∣X,\theta,{D}$ is the predictive posterior at input $x$ for a Gaussian Process with data $D$. This predictive posterior is Gaussian, and the entropy of a Gaussian has a closed-form expression. Since the entropy depends only on the covariance, it is easily computable. This is a standard result in Gaussian Process literature (e.g., Rasmussen and Williams, 2005).

 

5. It would be great if the authors could provide an ablation varying the number of batches and the batch size to see if HIPE performs well for different settings as well (esp. for Sec. 5.2, 5.3, 5.4).

   We have included the terminal performance for 5 batches with a batch size of 8 (initialization in the first batch only) as a table below; We will also include them in a potential CR. We can add additional ones as plots in the Appendix, similar to the active learning ablations. HIPE performs very well in this slightly smaller batch size, but one can naturally expect the importance of initialization to be largest when it constitutes a larger part of the overall budget.

   Function	HIPE	LHS-Beta	NIPV	Random	Sobol
   Fashion-MNIST	84.65 ± 0.07	84.19 ± 0.06	84.29 ± 0.08	84.17 ± 0.04	84.23 ± 0.04
   MiniBooNE	85.47 ± 0.02	85.25 ± 0.05	85.40 ± 0.02	85.24 ± 0.04	85.21 ± 0.06
   car	84.38 ± 0.15	83.10 ± 0.36	83.74 ± 0.19	82.36 ± 0.37	82.98 ± 0.33
   higgs	65.21 ± 0.08	64.56 ± 0.13	64.90 ± 0.11	64.18 ± 0.14	64.29 ± 0.14
   segment	83.86 ± 0.18	83.12 ± 0.14	83.39 ± 0.12	82.70 ± 0.18	82.95 ± 0.24

   Hopefully, this provides evidence that HIPE does well even with smaller batch sizes.

 

6. To further strengthen the evaluation, considering other HPO-benchmarks like [1] and [2] could be beneficial.

   We thank the reviewer for these references and will consider including these benchmarks in future evaluations to broaden the empirical validation.

 

7. in Fig. 2, the rank of BALD and NIPV tremendously changes when considering RMSE instead of NLL. Can the authors explain why this is happening?

   Certainly. Since NIPV spreads out points to maximally reduce integrated variance, hyperparameter estimation gets difficult. On the contrary, BALD tends to stack axis-aligned queries (Fig. 1) making hyperparameter estimation simple at the cost of poor coverage.

   With great coverage, the mean prediction (as with NIPV) is generally fairly accurate in most places, but poor hyperparameter estimation simultaneously can lead to drastic over- and underconfidence that punishes the MLL. BALD is the opposite: rarely over-or underconfident, but the mean predictions are not very accurate.

 

---

We hope these responses address the reviewer’s concerns. We appreciate the constructive feedback and hope that, if satisfied with our responses, the reviewer might consider raising their score to better reflect the strengths and contributions of our work.

We thank the reviewer for their detailed feedback and positive evaluation of our work. We appreciate that the reviewer finds our paper to read well, and that they recognize the potential significance and historical lack of attention for our problem setting. Below, we address each of the weaknesses and questions raised.

 

1. The authors discuss how the presented method may be too expensive for some applications - would the authors consider including compute times…

Certainly. Below is a runtime table for $q=8$ and $N=32$, namely 4 batches of 8 evaluations each with the first batch being initialization. We show runtime across 20 seeds for HIPE, NIPV and random on Hartmann (6D). The "BO loop" components are summed across all batches, meaning that "Model fit during BO loop" involves 3 unique model fittings for the three BO batches. All experiments are run on one A40 GPU. In our experience, this makes acquisition function optimization substantially faster, but accelerates model fitting less. HIPE is slower than NIPV and Random, but the runtime of initialization is still less than 20% of the overall runtime of BO in this setup. Naturally, the proportion of the total runtime allocated to initialization increases with fewer total batches.

Since we use a fully Bayesian surrogate with NUTS and large batch sizes, all steps are relatively time-consuming. Fully Bayesian models tend to be expensive in general, and HIPE shares similar computational demands. We want to emphasize though that in the few-shot settings considered in this work this cost is typically dwarfed by the cost of running evaluations, which in practice can often be on the order of days or weeks (e.g. training large ML models or running online A/B tests).

 

2. Am I understanding correct that using the proposed method requires a candidate set of initial data points?

Our method does not require a candidate set of initial data points. What it does require / allow for is a distribution $p(x_*)$ to inform the method about promising regions (e.g. based on user priors or domain knowledge). In the absence of prior knowledge about promising regions, a uniform distribution provides the best coverage, which is what we used in all experiments. However, users can bias initialization toward a-priori promising regions with a non-uniform $p(x_*)$. From an implementation perspective, we represent $p(x_*)$ as an empirical distribution of sample points. Importantly though, these sample points do not need to be evaluated on the objective function.

 

3. Is there any reasons the authors only show the Sobol results in Figure 5 (right) and not all the other methods?

Only to de-clutter the plot. Since there were 40 dimensions to visualize, we thought we would only include the two most relevant comparisons. The other methods are very similar to Sobol.

 

4. Did the authors look at performance of their proposed method in the BO setting for iterations > 2?

Yes, we have included the terminal performance for 5 batches with a batch size of 8 (initialization in the first batch only) as a table below; We will also include them in a potential CR. HIPE performs very well in this slightly smaller batch size, but one can naturally expect the importance of initialization to be largest when it constitutes a larger part of the overall budget.

 

5. The method is presented only for GPs, which may be a shortcoming as BNNs and NNs are becoming more used surrogates.

We agree that Bayesian Neural Networks (and other types of surrogates) are increasingly used, though less so for Random Forests. HIPE can be applied to BNNs, where uncertainty over θ corresponds to uncertainty over all neural network parameters. However, the computation is more involved than for GPs, as BNNs lack closed-form expressions for posterior variance or conditioning on new data.

Random Forests do not inherently have a Bayesian treatment of model parameters and thus cannot use HIPE directly.

 

6. Do the authors have any insights on why the init scheme seems to be relatively problem sensitive?

Yes, we do. Since HIPE’s EPIG criterion minimizes integrated (average) variance over the search space, it tends to avoid points near boundaries, as average variance is rarely reduced by sampling near edges. Sobol and random methods have no such bias, which explains the performance swings between HIPE and random methods. Differences between NIPV and HIPE are less straightforward.

For some problems, HIPE’s more centered queries are beneficial early on, while for others (e.g., SVM), where optima lie near boundaries, this is less so. However, HIPE compensates by correctly identifying active dimensions and catches up in later batches.

 

7. I think the title is a little confusing...

Does the reviewer refer to the term “Informed”? We are open to suggestions on what a more representative title would be.

 

8. ...whilst I think the abstract is overclaiming.

HIPE is designed to produce the most informed model (in terms of point spread and hyperparameter inference) to enable successful subsequent BO batches, rather than to achieve the best performance after batch 1. From this perspective, we believe the statement is accurate but agree that we can make it more precise to improve clarity.

 

---

Minor comments have been addressed and will be included in the camera-ready version:

1. The missing citation related to BO surrogate quality and calibration has been added.
2. We agree that the surrogate model is equally important as the acquisition function in BO; this is now reflected in line 80.
3. The claim about Scrambled Sobol sequences not achieving desired coverage anecdotal. We will avoid general statements and discuss Fig. 1 specifically.
4. We have rewritten the repeated content in lines 163-166.
5. The discussion around Figure 1 has been expanded to better highlight qualitative differences between NIPV and HIPE, including a modified figure clarifying axis alignment.
6. The grammatical error in line 306 has been corrected.

 

---

We once again thank the reviewer for their constructive feedback and appreciate their recognition of the paper’s clarity, motivation, and the relevance of the problem setting. We hope that our responses have addressed the reviewer’s questions regarding runtime analysis and ablation studies.

In light of these additions, we hope the reviewer finds the additional results compelling and the overall submission strengthened.

We would like to thank the reviewer for their feedback on our work. Moreover, we are appreciative that the reviewer acknowledges the relevance of our problem setting. Regarding the design of the hyperparameter $\beta$, we would like to bring to the reviewer’s attention that we provided a method for setting it in the paper, in Section 4.1, L192. All our responses are outlined below.

1. The current experiment presentation lacks sufficient details to enable reproducibility, particularly regarding the choice and tuning of the hyperparameter beta, which is not explicitly specified.

   We use a principled approach to set the hyperparameter beta, as described at line 192 in the paper, and apply this consistently throughout all experiments. Appendix A1 provides detailed information on the experimental setup, including hyperpriors and software, to support reproducibility. Our full experimental code is available on GitHub, the link to which is in Appendix A1 as well. We are happy to provide further clarifications if needed.

 

2. Could the authors justify why setting β = EIG achieves a balance between the two competing objectives?

   The EPIG and BALD objectives are generally on very different scales, making weighting necessary. Setting $β = \text{EIG(}y(x_*); \theta)$ provides a principled weighting that reflects how much hyperparameter information actually translates to predictive information on the test set. This choice ensures that the hyperparameter learning component (BALD) is weighted according to its actual relevance for prediction, rather than using an arbitrary constant. When hyperparameter uncertainty has little impact on test predictions, $\beta$ will be small, emphasizing space-filling. When hyperparameter uncertainty significantly affects predictions, $\beta$ will be larger, emphasizing hyperparameter learning. This adaptive weighting achieves the desired balance automatically without manual tuning.

 

3. While the authors introduce NIPV, it is not empirically evaluated in combination with BALD, leaving its practical benefits underexplored.

   We acknowledge that NIPV + BALD can have practical value on its own. However, NIPV is not combined with BALD because it does not adhere to an information-theoretic framework like HIPE. Consequently, the same derivation as in Prop. 1 cannot be made, nor can beta be derived in a similar manner to achieve the desired trade-off between hyperparameter learning and space-filling. Given this, combining the two would be more heuristic than principled.

With that said, We have included the NIPV + BALD combination in the table below. While it wins on 1/5 LCBench problems, the BALD term generally dominates the NIPV term, leading to under-exploration.

 

4. Aside from Proposition 1, which establishes an information-gain equivalence, the paper lacks theoretical guarantees (e.g., regret analysis) to support the proposed method's performance.

   Our acquisition function is designed to produce a well-informed initialization to improve subsequent Bayesian Optimization performance rather than to minimize regret immediately. As such, regret bounds are not currently available. While theoretical guarantees on model predictive performance might be possible, such results are rare in the broader Bayesian active learning literature. We acknowledge this as a limitation and an interesting direction for future work.

 

5. What do the red and blue boxes represent in the right subplot of Figure 5?

   The red and blue boxes in Figure 5 (right) represent the interquartile ranges of the estimated lengthscale distributions for different initialization methods. The red boxes show lengthscales estimated after HIPE initialization, while the blue boxes show lengthscales after Sobol initialization. The plot demonstrates that HIPE correctly identifies the last two dimensions (the SVM's global regularization parameters) as most important by assigning them substantially smaller lengthscales compared to Sobol. We will add a legend to clarify this in the camera-ready version.

 

---

We once again thank the reviewer for their service, and hope our rebuttal addresses the reviewer’s questions and concern regarding how the hyperparameter $\beta$ is automatically set. Should the reviewer have any further questions or require additional details, we would be more than happy to provide them.

Thanks the authors to provide the additional experiments and detailed explanation to my concerns. I would raise my score to 3.

However, I still think that the proposed method lacks novelty, as the new acquisition is essentially a linear combination of BALD and EPIG. While a combination approach is not necessarily a problem, the paper does not convincingly demonstrate that such a simple combination yields a significant improvement in most of the experiments. Moreover, even though the authors explain why $\beta=EIG$ is suggested, I still feel this choice to be rather heuristic and not theoretically grounded.