We would firstly like to thank the reviewer for the thorough comments and details. We would then like to comment on the highlighted weaknesses and address the presented questions.

---

Regarding Laplace approximations and their sensitivity to noise

We agree with the reviewer that in the first version of the paper there was not sufficient evidence to support the claim on sensitivity of noise in LLLA models. However, we have now included a set of experiments in the appendix which highlight that VBLL indeed is more robust to noise than LLLA.

---

Regarding a comparison to a GP model with a D-scaled hyperprior, i.e. with mass concentrated near \sqrt{D}

We now included an additional comparison to GPs with $D$-scaled hyperprior on the lengthscales in the appendix. For the construction we closely follow the suggested paper and use a LogNormal distribution with the same hyperparameters. In the results, we can observe that for the lower-dimensional problems, the influence is negligible. For the very high-dimensional NNDraw the inductive bias does help the performance when using logEI. However, for Thompson sampling there appears to be no noticeable difference. Here, the key aspect is using a parametric vs. a non-parametric model. Lastly, on Pestcontrol, a problem with complex input correlations, the D-scaled prior even not only fails to improve performance but also negatively impacts it when using logEI. This result highlights that here the type of kernel is not suitable for the problem at hand. The VBLLs (and other BNN baselines) directly learn the correlations and converge efficiently to the optimal solution and to us, these results indicate that using a D-scaled hyperprior would not alleviate the problems of GPs that are the motivation for using VBLL surrogates.

---

Regarding Question 1

With regards to the comments on runtime discussion, we agree that this was lacking in the previous paper. We have included comparisons of surrogate fit times and BO performance in the revised version of the paper, and refer to the general comment for a more thorough discussion and comments on the accumulated fit times. We would also like to add that VBLL outperforms LLLA on many of the classic benchmarks (Figure 3 (top)).

---

Regarding Question 2

With regards to claims without backing on early stopping, we agree with the reviewer that some phrasings and claims were too strong and will amend this in the revised version. For the early stopping, we have replaced “optimal” with “good empirical performance” the revised version. To support this claim, we have added an additional experiment in Appendix B.2 (Figure 6 (b)), showing that the performance is not significantly impacted by the early stopping, but we can significantly reduce the runtime.

Firstly, we would like to thank the reviewer for the thorough review. We would then like to address the presented weaknesses brought by the reviewer.

---

Regarding Question/Weakness 1

With regards to why VBLL might be more suited than DKL: the primary reason that DKL is challenging is due to the computational complexity of gradient computation with a marginal likelihood objective, as stated by the reviewer. This becomes especially problematic for larger datasets, which was one of the motivations for the VBLL method. We have highlighted this in the appendix of the paper and tried to clarify elements and differences of the various methods presented in this work. Furthermore, DKL still yields a non-parametric model which can become problematic for high-dimensional Thompson sampling (see eg. NNDraw with TS). Essentially one can interpret the VBLLs as a parametric version of DKL.

---

Regarding Question/Weakness 2

With regards to insights on where the method is expected to underperform we mainly see two areas where we suspect other surrogates may be preferred to VBLLs. The first is in settings where data is extremely scarce i.e. where total number of samples is <<100 - here it is likely that methods such as GPs are still dominant. Secondly, we still see that in comparison to some other surrogate types, the computational costs of VBLLs are quite high, and therefore in settings where surrogate model fit times are a constraint (which is not very common in BO, but does occur), other surrogate models may be more appropriate.

Based on the reviewer’s recommendation, we have added some high-level comments on sensitivity to hyperparameters to the main body text but kept the results in the appendix due to a lack of space.

Finally we would like to thank the reviewer for their thorough comments and hope that they have found we satisfactorily have answered their questions.

Thank you for your review. In the following, we hope to address all your questions and the weaknesses you highlight.

Related to the comment on how neural network architecture affects the performance of the presented methods, we agree that this is a highly interesting aspect of the presented methods - we did in fact run experiments to check how neural network width affected the VBLL performance, and found them to be highly robust to varying widths (these results are currently in the Appendix for space reasons). More complex changes in neural networks architecture such as changing of architecture type is a highly interesting research direction, but requires careful selection of datasets, architectures and tuning and we therefore chose not to perform such experiments here. We hope the reviewer agrees that showing that the VBLL methods are robust to neural network width changes provides evidence that these methods are not brittle to architecture changes, which we of course agree is immensely important.

Additionally, we would like to add that we agree that the placement of the comparison of VBLL and Laplace was in an odd place, but have chosen to move it to the appendix due to space constraints, but refer to it already in the Related Work and Background section.

---

Regarding Question 1

Thank you for the suggestion to point out that $w=S^{−1}q$ is the vector of precision-means! We have now added this to the revised version.

---

Regarding Question 2

With regards to the comments on the proof of Theorem 1: We are highly constrained on space due to new results being added to the body of the paper. While the proof is indeed quite simple, the technical details take up enough space that including the full proof is challenging. We will include a brief proof sketch (that basically states exactly what the reviewer said) in the body of the paper. On the complexity results: we agree that the bulk of the complexity results are standard. However, we want to highlight the complexity due to the (as far as we know, novel) use of the efficient rank-1 Cholesky updating approach. We will include a brief description of this feature in the body and move most of the complexity discussion to the appendix to save space.

---

Regarding Question 3

With regards to the question on the necessity of the Wishart prior scale: Thank you for pointing this out! The Wishart prior scale is used in the initialization of the model. Here, we don’t pass the noise variance, as stated in the algorithm, but the Wishart prior scale. We learn the noise online alongside the variational posterior over W. We have fixed this typo in the revised version.

Finally we would like to thank the reviewer for their time and thorough comments.

Thank you for the thorough review. In the following, we aim to address all the mentioned weaknesses.

---

Regarding Weakness 1

We thank the reviewer for the comments related to missing measurements of runtime and computational cost and agree that this indeed would improve the paper. We have now included figures where we plot accumulated surrogate fit times versus BO performance - see general comment for more discussion on this. As outlined in the general comment, we would like to underline that we did mean to claim that VBLL models are always computationally more efficient than other BNN surrogates. Rather, we show that VBLL models often outperform other BNN surrogates especially in the synthetic setting (especially last layer versions), yet can be costly to fit in the BO scheme. To alleviate this, we propose the recursive update scheme presented in the paper which is novel for these model types as well as continual learning scheduling that spares computational time with little cost to BO performance.

---

Regarding Weakness 2 and Question 1

With regards to comparing to more expensive BNN surrogates, we have now included Deep Ensemble (DE) experiments on the single-objective problems. We generally observe that DEs perform well on the structured problems (on par with VBLL), but VBLLs generally outperform DEs on many of the synthetic problems, e.g., Branin.

With regards to comparison to DNGO, the methods presented in [1] are quite similar to LLLA. In [1] the authors train a neural network end-to-end to learn the basis functions (including with a final linear layer) using a MAP estimate, and after convergence replace the linear layer with a Bayesian linear regressor. In LLLA, the full network is similarly trained end-to-end in traditional MAP fashion and after convergence, a Gaussian approximation is placed on the final linear layer parameters (keeping the MAP estimate as the mean of that Gaussian). In essence, the main difference between these two methods is that the mean of the MAP estimate is replaced with the Bayesian linear regression mean in the DNGO method, whereas the MAP estimated mean is kept in LLLA. The computation of the variances in each method are the same [3, Appendix B.1.2]. In addition, others have observed that when DNGO or BLL models are fitted in this fashion of MAP estimation followed by a replaced final layer, the learned features are not fit to provide good uncertainty estimates which cannot be amended by simply replacing the final linear layer with a Bayesian linear regressor (see [2] for details) - VBLLs should in principle not struggle with this problem since features and variational parameters are jointly learned. It is perhaps also for this reason that BLL models have not seen much popularity in recent years: see eg [4], which also does not compare to BLLs in Bayesian Optimization.

The reviewer also commented that VBLL may be less flexible than LLLA, but we would argue the contrary for a number of reasons such as being able to optimize the noise covariance and jointly learning features with the variational distribution in VBLL, which may alleviate problems such as those described in [2].

We hope the reviewer has found that we have satisfactorily addressed their main concerns of the paper. Should this not be the case we are happy to further address any other concerns.

---

References

[1]: Snoek, Jasper, et al. "Scalable bayesian optimization using deep neural networks." International conference on machine learning. PMLR, 2015.

[2]: Ober, Sebastian W., and Carl Edward Rasmussen. "Benchmarking the neural linear model for regression." (2019).

[3]: Daxberger, Erik, et al. "Laplace redux-effortless Bayesian deep learning." Advances in Neural Information Processing Systems 34 (2021): 20089-20103.

[4]: Li, Yucen, et al. “A Study of Bayesian Neural Network Surrogates for Bayesian Optimization.” International Conference on Learning Representations (2024).

(continuation form previous comment)

Regarding wall-clock comparisons of methods:

A couple of reviewers brought up that the paper in its current version does not display any wall-clock or computational costs. We agree that this was lacking, and have now included a figure in which we plot accumulated surrogate fit time versus BO performance, firstly showcasing the fact that VBLLs are expensive to fit, but have high data efficiency, and secondly that the proposed CL scheme improves fit times.

One thing we would like to note is that the focus of this paper is not to claim that VBLL models are extremely efficient in terms of surrogate fit times. One of the four primary contributions is that we showcase that VBLLs as surrogates perform similarly to GPs in the synthetic setting (which is not a small feat, see e.g. [1, Figure 3 top] where no BNN surrogate consistently performs similarly to GPs on standard benchmarks) and moreover that they outperform GPs in non-euclidian problem settings, whilst performing similarly or better than other BNN surrogates. However, the VBLLs were often expensive to fit. Another of the primary contributions is therefore the proposed recursive update and continual learning scheme to alleviate these fit times with little cost in terms of BO performance. We acknowledge that the abstract previously could have been interpreted as VBLLs being cheap to fit, and we have therefore made amendments to it.

Finally, for the reviewers’ curiosity, we would like to note that we suspect that the VBLL surrogate fit times currently are dominated by the variational posterior parameter optimizations. In the problem settings in the paper, the neural networks are (relatively) small networks, and therefore the overall network fitting is dominated by the variational posterior optimizations. In settings where the non-variational parameters of the network grow (such as with large neural networks), we suspect that the gap between VBLL fit times and other BNN methods should drop. To be more precise, consider a neural network with $n$ weights in the last layer, and $m$ weights in the remaining network. Due to the parametrization of the variational posterior we have $n + 0.5n^2$ parameters for the last layer and $m$ parameters for the feature mappings. In larger networks where $m$>>$n^2$, we thus suspect the gap in surrogate fit times for the BNN surrogates should drop. In our BO setup for Ackley (5D) for example, we have $m\approx33.800$ and $n^2\approx16.600$, i.e. the variational parameters make up $33\\%$ of the total number of parameters. In contrast, PaiNN networks [2], a type of graph network often used for regression tasks on chemical compound spaces, would have $m \approx 535.000$ and $n^2\approx 65.500$, meaning the variational parameters would make up approximately $10\\%$, whilst in MACE [3] models for materials discovery, the variational parameters would make up approximately $1.5-6\\%$ of the total parameters depending on how the final layer size was chosen.

---

References

[1] Li, Yucen, et al. “A Study of Bayesian Neural Network Surrogates for Bayesian Optimization.” ICLR, 2024.

[2] Schütt, Kristof, et al. "Equivariant message passing for the prediction of tensorial properties and molecular spectra." ICML, 2021.

[3] Batatia, Ilyes, et al. “A foundation model for atomistic materials chemistry.” ArXiv, 2023.