Large-Batch, Iteration-Efficient Neural Bayesian Design Optimization

Thank you for your feedback. We are heartened to know that you appreciate our method’s simplicity and its potential impact. We provide responses to the raised questions below.

Is there a crossover from where more simple GP-BO methods fail and LBN-MOBO would be best?

In our empirical studies on batch multi-objective Bayesian Optimization (BO), we've identified that a batch size of 100 serves as a threshold, beyond which traditional BO methods begin to encounter difficulties. Interestingly, there are some ingenious adaptations to BO that enable operation with batch sizes approaching 1000. However, for even larger batch sizes, the shift towards Bayesian Neural Networks (BNNs) in place of Gaussian Processes (GPs) and replacing the acquisition function with the 2MD function becomes essential. As we scale up to batch sizes of 10,000, only methods like Monte Carlo Dropout (MCDropout) and Deep Ensembles in combination with 2MD acquisition prove capable of effectively handling such extensive batch sizes.

New title.

We are open to consider renaming the paper and we appreciate to know your feedback in this regards. We think we can drop ‘iteration-efficient’ and ‘design’ from the title.

Thank you for your insightful feedback. Your appreciation of our work, particularly the real-world evaluations and the potential contribution to the Bayesian Optimization community, is highly encouraging. Following your suggestion to release our benchmarks in open-source code, we will certainly consider this moving forward. We provide responses to the raised questions below.

More analysis on 2MD acquisition function.

We have provided the following insights and analyses about our acquisition function:

• In our paper, Section 5.4 is dedicated to analyzing the influence of uncertainty on the performance of the 2MD acquisition function. Through this analysis, we aim to emphasize the critical role that epistemic uncertainty plays in determining the overall quality of the LBN-MOBO approach.

• We have conducted a preliminary evaluation on the feasibility of applying the 2MD Pareto front approach to noisy Native Forward Processes (NFPs). This examination, detailed in a separate supplementary material (Rebuttal PDF-LBN_MOBO_noisy.pdf), explores the impact of incorporating aleatoric uncertainty into the acquisition function. Our findings reveal that relying solely on epistemic uncertainty in noisy scenarios can inadvertently lead the optimization process to select candidates from these noisy regions. In contrast, by taking into account aleatoric uncertainty and striving to minimize it during the optimization process, we can effectively avoid these noisy areas. This insight highlights the importance of considering both types of uncertainty in the optimization strategy when dealing with noisy NFPs.

• We have introduced a new subsection that delves into how the 2MD Pareto front can be paired with any arbitrary acquisition function. This aspect is crucial as it positions 2MD as a generic acquisition function adaptable to various contexts. In Section 5.1, we demonstrated this adaptability by coupling the 2MD approach with a range of Bayesian Neural Networks (BNNs).

• In Section C3, we revisited the evaluations from Section 5.1, this time using a new problem, i.e., the DTLZ5 test suite. This problem has 20 input and 3 output dimensions. (When applying 2MD acquisition function, this translates to a 6D problem.) Although Monte Carlo Dropout (MCDropout) and Deep Ensembles showed the best runtime performance, the quality of the solutions across all surrogates was highly competitive. This finding suggests that the 2MD acquisition approach is capable of delivering strong performance with various surrogate models, indicating its versatility and effectiveness in handling complex multi-dimensional optimization tasks.

Expansion of complementary related work.

We appreciate the reviewer for mentioning these missing related works. We have included them in the complementary related work in Sections A2 and A3.

Thank you for the thorough and thoughtful review. We appreciate the detailed feedback. We provide responses to the raised questions below.

The reason for modifying the uncertainty measurement part to include only epistemic uncertainty and its benefits.

Uncertainty in BO primarily serves as a guide for effective exploration. Epistemic uncertainty represents a lack of knowledge about a phenomenon, which can be mitigated through additional observations. Conversely, aleatoric uncertainty relates to the inherent noise in the system and is not a function of data accumulation.

In our study, we worked with the underlying Native Forward Processes (NFPs) that are essentially noiseless. Consequently, addressing noisy NFPs is earmarked for future exploration (Limitation section). Given this noiseless scenario, aleatoric uncertainty is presumed negligible, leading us to focus solely on the epistemic uncertainty. This decision significantly simplified and stabilized our model's training phase. This means our training process closely resembles that of conventional neural network, avoiding potential complexities and instabilities in training Deep Ensembles [1][2]. This methodological choice not only enhances the robustness of LBN-MOBO but also increases its accessibility and practicality as any neural net architecture can be utilized.

Usefulness of aleatoric uncertainty.

Prompted by the reviewer's insightful question, we explored a toy noisy NFP to see whether considering aleatoric uncertainty helps with the exploration. This example shows that in the presence of significant noise, using only the epistemic uncertainty could encourage sampling of noisy regions (likely due to Deep Ensembles mechanism), which will deteriorate the exploration. Very interestingly, including the aleatoric uncertainty and avoiding the regions with such uncertainty improves the exploration significantly. This has highly appealing implications for BO in the presence of noise and we plan to study it intensively in the future. This is detailed in our supplementary material (Rebuttal PDF_LBN_MOBO_noisy.pdf).

These findings highlight the potential challenges and benefits of aleatoric uncertainty in scenarios with inherent noise, paving the way for future research in this domain.

Comparing 2MD acquisition to the other acquisition functions, except for being more scalable.

The traditional approaches to acquisition, such as the Upper Confidence Bound (UCB), relies on some form of weighted sum of uncertainty and prediction. This weighting is crucial as it dictates the balance between exploration and exploitation within the optimization process. However, the scale differences between prediction and uncertainty necessitate careful tuning of this hyperparameter. Additionally, as the optimization progresses, the requirements for exploration or exploitation may change, potentially requiring ongoing adjustments to this weight. In the realm of multi-objective optimization, the complexity escalates further due to the need to fine-tune multiple parameters to achieve an optimal balance in exploring various objectives and their respective uncertainties.

In contrast, the 2MD acquisition function, as described in the paper (Section 4.2 and Equation 3), adopts a fundamentally different approach. Here, uncertainties and predictions are treated as separate optimization objectives, which are then jointly optimized to construct a Pareto front of potential candidates. This method does not need the summation of objectives, thereby remedies the issue of scale mismatch between objectives. Consequently, our approach is free from the constraints of hyperparameter tuning, simplifying the optimization process. It's important to note that, to our knowledge, there are no alternative methods capable of scaling to the large batch sizes required for addressing real-world problems of the magnitude we are tackling, positioning our algorithm as a unique solution in this domain.

More benchmark algorithms other than BNNs on real problems.

In appendix section C5 you can find complementary experiments comparing LBN-MOBO using other algorithms that are partially or completely capable of handling such large batch sizes.

How is the “time” defined in Figures 1 and 3?

It is the summation of acquisition time plus training the surrogate (a BO cycle) for 10 iterations.

How well does the modified deep ensembles quantify uncertainty, compared to the original deep ensemble?

There is a widely established way of implementing Deep Ensembles where the training of prediction networks is separated from the uncertainty estimation networks [2]. The proposed simplified implementation is identical to that approach when it comes to computing the epistemic uncertainty: we ensemble prediction networks to calculate the epistemic uncertainty.

We appreciate the reviewer's openness and for raising our score. The answer below will hopefully addresses the remaining concerns of the reviewer.

The benefits of using only epistemic uncertainty in noiseless NFPs and the advantages of aleatoric uncertainty in noisy settings.

Given your feedback and the one of Reviewer acfL, we conducted additional experiments to study the effect of including the aleatoric uncertainty in our acquisition function in noiseless NFPs. These experiments are shown in the appended document (Rebuttal PDF_LBN_MOBO_aleatoric.pdf). We used the examples and procedures described in Sections C1.4 and C1.5 of the paper. The outcomes indicate that incorporating aleatoric uncertainty does not affect the exploration process in a beneficial manner—as showcased by the ZDT1 results—and may even delay the convergence, as observed in the ZDT2 and ZDT3 experiments. This is attributed to the nature of the aleatoric uncertainty, that measures inherent noise which cannot be diminished. This follows our original intuition that aleatoric uncertainty cannot be an effective tool for driving the exploration.

However, the noisy example detailed in our supplementary document (PDF-LBN_MOBO_noisy.pdf) demonstrates that for noisy Native Forward Processes (NFPs), incorporating aleatoric uncertainty is advantageous for avoiding (and not exploring) the system's irreducible noise during BO. Motivated by these findings, we are applying LBN-MOBO with both epistemic and aleatoric uncertainties on our printer’s color gamut problem where we intentionally add noise to one the printer channels (in simulation). This new experiment will be incorporated into the revised version of our paper.

More examples to show the advantage of our proposed surrogate over its possible alternatives.

We have included another example with larger input and output dimensions to demonstrate the robust performance and scalability of our pipeline. We used the DLTZ5 test, which has 20 input design parameters and 3 output dimensions, providing a rigorous challenge. The Section C3 in the revised manuscript shows that this problem behaves similarly to our previous tests. While the performance of all methods is comparable, dropout and Deep Ensembles stand out for their significantly higher scalability.

To align these experiments more closely with our real-world scenarios, we conducted additional tests using HMC and SGHMC surrogates (the two BNNs that could scale beyond 1000 sample batch sizes) with large batch sizes of 5000 and 10000 on the 30-dimensional ZDT3 problem (in a setup similar to Sections 5.1 and C1.4). The results, including runtime and objective values, were captured within a 24-hour GPU time frame.

The data in our table illustrates that the Deep Ensemble not only achieves a better Pareto front but also does it so much faster compared to other BNNs. Furthermore, the table highlights the impracticality of using other BNNs in the context of our real-world problems, as we employed batch sizes exceeding 10,000 samples. This finding reinforces the effectiveness and efficiency of our chosen methods in handling large-scale, complex problems.

Implementation details of using 2MD Pareto front with other BNNs

In principle, one of the main advantages of 2MD acquisition function is that it only queries the surrogate models (without the need of gradients for example). 2MD performs by iteratively using these queried information and evolves the solution until a practically good convergence. We included a section and a conceptual figure in the appendix to clarify how it works (Section B2).

Is batch size b > 1000 a normal batch size in real-world problem?

Absolutely! Specifically in the contex of manufacturing, fabrication and simulation (for which we have shown two example experiments in the paper). In many such cases we are capable of producing very large batches of data with almost the same cost of a single sample but iterative lab visits are costly. As a result, LBN-MOBO makes the overall process efficient by reducing the necessity of lab visits (second paragraph of the introduction).

Can LBN-MOBO outperform Evolutionary Computation methods?

Yes. In section C1.4 of appendix we have compared LBN-MOBO against NSGA-II and a few other advanced (not necessarily neural) BO methods on ZDT1, ZDT2, ZDT3, DLTZ1, and DLTZ4 test suits as well as on both our real-world problems (Sections C5.1 and C5.2). In all these experiments LBN-MOBO outperforms all other methods handily.

Why qEHVI and qNEHVI do not include BNNs other than IBNN and DKL?

Please refer to the final paragraph of section 3.

[1] Seitzer, Maximilian, et al. "On the pitfalls of heteroscedastic uncertainty estimation with probabilistic neural networks." arXiv preprint arXiv:2203.09168 (2022).

[2] Nix, David A., and Andreas S. Weigend. "Estimating the mean and variance of the target probability distribution." Proceedings of 1994 ieee international conference on neural networks (ICNN'94). Vol. 1. IEEE, 1994.

[3] Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. "Simple and scalable predictive uncertainty estimation using deep ensembles." Advances in neural information processing systems 30 (2017).

Thank you for the thorough and thoughtful review. We appreciate the raised questions that help us clarify the paper. We provide responses to the raised questions below.

Explanation of the epistemic uncertainty is missing.

We have elaborated on epistemic and aleatoric uncertainty and their differences in Section 4.1.

Why does LBN-MOBO only needs epistemic uncertainty (and not aleatoric uncertainty)?

Epistemic uncertainty represents the lack of knowledge about a phenomenon and can be reduced with more observation. The aleatoric uncertainty is a sign of inherent noise which cannot be reduced. In this paper we assumed that our underlying Native Forward Processes (NFPs) do not feature significant noise and left handling noisy NFPs for the future work (See Limitation section). In the absence of significant noise, the inclusion of aleatoric uncertainty does not affect the solutions. Thus, we decided to modify the Deep Ensembles to calculate only the epistemic uncertainty which is necessary for the exploration. This decision significantly simplifies and stabilizes our model's training [1] phase, as we don’t need to train variance networks typically needed for aleatoric uncertainty [2].

Prompted by the reviewer's insightful question, we explored a toy noisy NFP to see whether considering aleatoric uncertainty helps with the exploration. This example shows that in the presence of significant noise, using only the epistemic uncertainty could encourage sampling of noisy regions (likely due to Deep Ensembles mechanism), which will deteriorate the exploration. Very interestingly, including the aleatoric uncertainty and avoiding the regions with such uncertainty improves the exploration significantly. This has highly appealing implications for BO in the presence of noise and we plan to study it intensively in the future. This is detailed in our supplementary material (Rebuttal PDF_LBN_MOBO_noisy.pdf).

Discussion of the problems inherent to BNNs (e.g., tuning and uncertainty quality).

The reviewer is right about these challenges of BNNs. While we don’t focus on improving on the BNNs side, our simplified approach in Sections 4.1 and 4.2 addresses both concerns to a great extent. Regarding the surrogate model, by excluding the aleatoric uncertainty we have avoided several complexities and instabilities during the training phase [1][2], making it completely similar to training a group of conventional neural networks. Moreover the way we calculate the epistemic uncertainty is identical to the original Deep Ensembles methods and it is among the best (quasi-)BNNs in terms of the trade off between the quality of uncertainty and scalability [3].

Similarity and differences between UCB and 2MD Pareto front.

As reviewer correctly points out UCB and 2MD perform the optimization while exploiting the uncertainties in different ways. UCB works based on a weighted sum of uncertainty and prediction. In UCB tuning is important as it regulates the trade-off between exploration and exploitation. Given the magnitude difference between prediction and uncertainty, one needs to tune this hyperparameter. Moreover, as we progress through the optimization the need for exploration or exploitation might change and as a result we might need to constantly adapt this hyperparameter. In the case of multi-objective optimization, the number of tuning parameters increases as we need to tune and find the right balance in exploring different objectives in the presence of their uncertainties.

In the 2MD acquisition function, however, the uncertainties and predictions remain independent objectives and are jointly optimized to generate a Pareto front of viable candidates (Section 4.2 and Equation 3). Since each objective is optimized jointly with other objectives without any form of summation we do not need to worry about the scale difference and LBN-MOBO does not introduce extra hyperparameters.

Unfortunately, we could not find any implementation of UCB extension capable of handling both multi-objective and batch optimization to compare it against our method. If pointed to such an implementation, we will be happy to compare it with our 2MD algorithm. To the best of our knowledge, qEHVI and qNEHVI are the state-of-the-art algorithms for multi-objective batch Bayesian optimization. We showed that these methods are not suitable for the large batch setting we are proposing (Section 3).

Thank you for your insightful feedback, we hope the answer below will address the raised concerns.

More studies with and without aleatoric uncertainty using the problems in the paper.

Given your feedback and the one of Reviewer 9JiQ, we conducted additional experiments to study the effect of including the aleatoric uncertainty in our acquisition function. These experiments are shown in the appended document (Rebuttal PDF_LBN_MOBO_aleatoric.pdf). We used the examples and procedures described in Sections C1.4 and C1.5 of the paper. The outcomes indicate that incorporating aleatoric uncertainty does not affect the exploration process in a beneficial manner—as showcased by the ZDT1 results—and may even delay the convergence, as observed in the ZDT2 and ZDT3 experiments. This is attributed to the nature of the aleatoric uncertainty, that measures inherent noise which cannot be diminished. This follows our original intuition that aleatoric uncertainty cannot be an effective tool for driving the exploration. If the reviewer sees any reason for further studies, we can repeat these experiments for our 3D printer and airfoil applications as well. Please instruct.

More intuition on using only epistemic uncertainty and its benefits.

Why only epistemic uncertainty?

• Epistemic uncertainty is crucial for exploration as it identifies knowledge gaps. Aleatoric uncertainty, hinting at the intrinsic noise, doesn't aid exploration.
• The training of prediction networks for epistemic uncertainty is fully independent from aleatoric uncertainty process. This ensures that ignoring aleatoric uncertainty introduces no errors in estimating epistemic uncertainty [1].

What are the benefits of using only epistemic uncertainty?

• Training with only epistemic uncertainty, yielding a faster and more stable training without the need to manage aleatoric uncertainty networks (variance networks) [1].
• Omitting aleatoric uncertainty reduces model complexity, leading to fewer hyperparameters and a more straightforward training phase without balancing different uncertainties [2].

Hyperparameter of the BNNs.

The main focus of our paper is on the utilization of the simplified Deep Ensembles as the surrogate model. The main benefit of this surrogate is that it does not introduce any additional hyperparameters and it is sufficient to ensure that the sub-networks in the ensemble are trained accurately. Regarding other evaluated BNNs, we have adopted the implementation and the recommended hyperparameters from [3] and we also made sure that the training accuracy are reasonable. We will report the training accuracies for all methods in the final paper.

Deeper analysis of the method.

Please see an explicit list of our analyses in response to Reviewer s2Pz. In addition, we have observations to believe that the regret (the difference between the ideal and the current solution) grows at most sub-linearly. We will include in the revised paper (at least an empirical study) to show the regret growth in a batch scenario.

[1] Nix, David A., and Andreas S. Weigend. "Estimating the mean and variance of the target probability distribution." Proceedings of 1994 ieee international conference on neural networks (ICNN'94). Vol. 1. IEEE, 1994.

[2] Ansari, Navid, et al. "Autoinverse: Uncertainty aware inversion of neural networks." Advances in Neural Information Processing Systems 35 (2022): 8675-8686.

[3] Li, Yucen Lily, Tim GJ Rudner, and Andrew Gordon Wilson. "A Study of Bayesian Neural Network Surrogates for Bayesian Optimization." arXiv preprint arXiv:2305.20028 (2023).