EARL-BO: Reinforcement Learning for Multi-Step Lookahead, High-Dimensional Bayesian Optimization

Thank you for acknowledging our efforts and for the continued engagement.

[Computational Efficiency]

Regarding computational efficiency comparisons, please see our discussions with Reviewer eu5H. We reproduce from that discussion a key point on computational time here:

Despite strongly outperforming rollout-based methods, we acknowledge the reviewers’ concerns regarding computational efficiency compared to the cheaper (myopic) methods. We do acknowledge this limitation of EARL-BO (and non-myopic methods in general), but note that BO is specifically designed for optimization problems where function evaluations are expensive, such as hyperparameter tuning for ML models [1], engineering design/simulations [2], and chemical laboratory experiments [3]. In these settings, a single function evaluation can take hours or even days, not to mention monetary costs, while the overhead of BO remains relatively small (difficulty of these problems is also mentioned by Reviewer JCQV). Thus, even as EARL-BO introduces additional computational cost relative to myopic baseline BO approaches, its significant improvement in performance can justify this expense. Furthermore, the improved sample efficiency of our method means that fewer function evaluations are required to reach an optimal or near-optimal solution. This results in a (significant) net reduction in total cost when considering the entire optimization process. Future works can exploit the modularity of EARL-BO for faster settings, e.g., by warm-starting RL, alternative RL methods/tunings, using the actor for multiple BO iterations, etc.

[Permutation Invariance]

We agree that the performance gap between the DeepSets and Attention-Only frameworks is limited. Nevertheless, despite the standard deviations in this preliminary study we do already see a consistent performance improvement using DeepSets (noting the logarithmic y-axis): after 10 iterations, mean regret is 7.8 vs 9.1; after 20 iterations, 5.3 vs 6.2; after 30 iterations, 4.3 vs 4.9; after 40 iterations, 3.9 vs 4.4; and after 50 iterations, 3.6 vs 3.9. These slight, but consistent improvements are non-trivial in expensive settings, e.g., see our discussion above.

While we hope to have time to produce more comprehensive results on this study in a revised version, these preliminary results already reveal some interesting conclusions: (1) these results begin to answer ‘How Markovian is BO?’. This is particularly interesting given that EARL-BO may exhibit some synthetic non-Markovian behavior, as the samples can become less reliable further into the lookahead horizon (see discussions on planning delusion with Reviewers euH5 and JCQV); (2) practically speaking, noting the modular design of EARL-BO, these results may suggest improvements for the encoder design in future works.

We believe these are highly interesting areas to discuss in a revised version and are grateful to the Reviewer for suggesting this ablation study and direction of investigation.

[Scalability of EARL-BO]

To further highlight the relative efficiency of the EARL-BO framework, we will include optimization results on the 30-dimensional Ackley function. A preliminary version of this experiment (two random starts instead of ten for now) can be found at the same anonymous repository https://anonymous.4open.science/r/icml_2025_review-B587. EARL-BO significantly outperforms all comparison methods, again demonstrating its scalability to challenging optimization problems in practice.

The computational requirements for this added experiment emphasize our scalability results: the rollout-based method would require even more Monte Carlo iterations to handle this search space, yet using even 1000 iterations already times out in the 3600s budget. On the other hand, EARL-BO runs in approximately 1600s, a moderate increase in time compared to the much less complicated setting of optimization over the 8-D Ackley function. Specifically, compared to 8-D 3-step lookahead EARL-BO, BO iterations in the 30-D problem require approximately 100% added CPU time, while myopic methods, such as EI and TuRBO require 533% and 215% more CPU time, respectively, to make decisions in 30-D compared to the 8-D optimization problem.

We believe these results strengthen our contribution and will include a more comprehensive analysis in a revised version.

References:

[1] Snoek, J., Larochelle, H., & Adams, R. P. (2012). Practical Bayesian optimization of machine learning algorithms. NeurIPS 2012.

[2] Jones, D. R., Schonlau, M., & Welch, W. J. (1998). Efficient global optimization of expensive black-box functions. Journal of Global Optimization, 13, 455-492.

[3] Shields, B. J., Stevens, J., Li, J., Parasram, M., Damani, F., Alvarado, J. I. M., Janey, J. M., Adams, R. P., & Doyle, A. G. (2021). Bayesian reaction optimization as a tool for chemical synthesis. Nature, 590(7844), 89-96.

Thank you for acknowledging our scholarship efforts and pointing us to the omitted references. We will indeed expand our discussion of the broader context using these in a revised version. In particular:

• [1] This paper is a seminal effort in meta-learning for black-box optimization and will be discussed when introducing meta-learning for Bayesian optimization (the authors nicely contrast their method with BO).
• [2-3] These works use similar attention-based architectures to encode datasets, although for other applications as noted. These share commonalities to our method in their encoder, which takes in a full dataset with the aim of learning and encoding of statistics of a full stochastic process (cf. our training is purely task driven). We will discuss these references when introducing the learned representation of the dataset.
• [4] This recent work discusses how meta-learning can be applied to the slightly different setting of movement-cost constrained BO, which the authors nicely compare to non-myopic BO (Section 4). We will include this line of research in our discussion of meta-learning for BO.

[Applicability of EARL-BO]

In response to the reviewers' concerns, we will highlight the relative efficiency of the EARL-BO framework using the 30-dimensional Ackley function. A preliminary version of this experiment (two random starts instead of ten for now) can be found at https://anonymous.4open.science/r/icml_2025_review-B587. EARL-BO significantly outperforms all comparison methods, again demonstrating its scalability to challenging optimization problems in practice.

The computational requirements for this added experiment emphasize our scalability results: the rollout-based method would require even more Monte Carlo iterations to handle this search space, yet using even 1000 iterations already times out in the 3600s budget. We believe non-myopic methods are the best computational comparison point (see discussions with other reviewers). On the other hand, EARL-BO runs in approximately 1600s, a moderate increase in time compared to the much less complicated setting of optimization over the 8-D Ackley function. Specifically, compared to 8-D 3-step lookahead EARL-BO, BO iterations in the 30-D problem require approximately 100% more CPU time, while myopic methods, such as EI and TuRBO require 533% and 215% more CPU time, respectively, to make decisions in 30-D compared to the 8-D optimization problem. We believe these results strengthen our contribution and will include a more comprehensive analysis in a revised version.

Despite strongly outperforming rollout-based methods with less time, we acknowledge the reviewers’ concerns regarding computational efficiency compared to the cheaper (myopic) methods. We do acknowledge this limitation of EARL-BO (and in fact non-myopic methods in general), but note that EARL-BO, and BO in general, is specifically designed for optimization problems where function evaluations are expensive, such as hyperparameter tuning for ML models [1], engineering design/simulations [2], and chemical laboratory experiments [3]. In these settings, a single function evaluation can require hours or even days, not to mention monetary costs, while the overhead of BO remains relatively small (difficulty of these problems is also mentioned the Reviewer). Thus, even as EARL-BO introduces additional computational cost relative to myopic baseline BO approaches, its significant improvement in performance can justify this expense. Furthermore, the improved sample efficiency of our method means that fewer function evaluations are required to reach an optimal or near-optimal solution. This results in a (significant) net reduction in total cost when considering the entire optimization process. Future works can exploit the modularity of EARL-BO for faster settings, e.g., by warm-starting RL, alternative RL methods/tunings, using the actor for multiple BO iterations, etc.

References:

[1] Snoek, J., Larochelle, H., & Adams, R. P. (2012). Practical Bayesian optimization of machine learning algorithms. NeurIPS 2012.

[2] Jones, D. R., Schonlau, M., & Welch, W. J. (1998). Efficient global optimization of expensive black-box functions. Journal of Global Optimization, 13, 455-492.

[3] Shields, B. J., Stevens, J., Li, J., Parasram, M., Damani, F., Alvarado, J. I. M., Janey, J. M., Adams, R. P., & Doyle, A. G. (2021). Bayesian reaction optimization as a tool for chemical synthesis. Nature, 590(7844), 89-96.

Thank you for the continued engagement and interest in our work. We apologize for the unclear presentation of our preliminary results. We do believe “all baselines are evaluated under a unified standard for computational efficiency,” as explained in the following.

[Analysis of Runtimes]

The table of preliminary computational cost analysis reports the average runtime for each Bayesian optimization iteration following the respective frameworks. The notes column was intended to reflect some important hyperparameter choices, rather than to suggest we compare iterations against episodes, estimations, etc. We will include more comprehensive results and analyses in a revised version, including:

1. EI, TuRBO, and SAASBO are much faster, but are “myopic” in the sense they do not aim to solve BO as an SDP;
2. Rollout_EI is run with 1000 Monte Carlo iterations, which is still a small number, given we had to use similar numbers for the 2- and 4-D problems presented in the paper. Monte Carlo methods require a number of samples scaling with the search space and scenario tree (here based on dimensionality and lookahead horizon) to achieve reasonable accuracy, which becomes impractical as dimensionality increases;
3. Modern RL methods employ function approximation techniques such as actor-critic networks, which can learn useful policies from far fewer samples by leveraging generalization, rather than exhaustive sampling (see below on scalability). We found PPO to perform well with only 4000 episodes, but again note that our EARL-BO framework can integrate alternative/better RL algorithms seamlessly, e.g., by warm-starting RL, alternative RL methods/tunings, using the actor for multiple BO iterations, etc.

[Scalability of EARL-BO]

To further highlight the relative efficiency of the EARL-BO framework, we will include optimization results on the 30-dimensional Ackley function. A preliminary version of this experiment (two random starts instead of ten for now) can be found at https://anonymous.4open.science/r/icml_2025_review-B587. EARL-BO significantly outperforms all comparison methods, again demonstrating its scalability to challenging optimization problems in practice.

The computational requirements for this added experiment emphasize our scalability results: the rollout-based method would require even more Monte Carlo iterations to handle this search space, yet using even 1000 iterations already times out in the 3600s budget. On the other hand, EARL-BO runs in approximately 1600s, a moderate increase compared to the much less complicated setting of optimization over the 8-D Ackley function. Specifically, compared to 8-D 3-step lookahead EARL-BO, BO iterations in the 30-D problem require approximately 100% more CPU time, while myopic methods, such as EI and TuRBO require 533% and 215% more CPU time, respectively, to make decisions in 30-D compared to the 8-D optimization problem. We believe these results strengthen our contribution and will include a more comprehensive analysis in a revised version.

Despite strongly outperforming rollout-based methods with less time, we acknowledge the reviewers’ concerns regarding computational efficiency compared to the cheaper (myopic) methods. We do acknowledge this limitation of EARL-BO (and in fact non-myopic methods in general), but note that BO is specifically designed for optimization problems where function evaluations are expensive, such as hyperparameter tuning for ML models [1], engineering design/simulations [2], and chemical laboratory experiments [3]. In these settings, a single function evaluation can require hours or even days, not to mention monetary costs, while the overhead of BO remains relatively small (difficulty of these problems is also mentioned by Reviewer JCQV). Thus, even as EARL-BO introduces additional computational cost relative to myopic baseline BO approaches, its significant improvement in performance can justify this expense. Furthermore, the improved sample efficiency of our method means that fewer function evaluations are required to reach an optimal or near-optimal solution. This results in a (significant) net reduction in total cost when considering the entire optimization process. Future works can exploit the modularity of EARL-BO for faster settings, as noted above.

References:

[1] Snoek, J., Larochelle, H., & Adams, R. P. (2012). Practical Bayesian optimization of machine learning algorithms. NeurIPS 2012.

[2] Jones, D. R., Schonlau, M., & Welch, W. J. (1998). Efficient global optimization of expensive black-box functions. Journal of Global Optimization, 13, 455-492.

[3] Shields, B. J., Stevens, J., Li, J., Parasram, M., Damani, F., Alvarado, J. I. M., Janey, J. M., Adams, R. P., & Doyle, A. G. (2021). Bayesian reaction optimization as a tool for chemical synthesis. Nature, 590(7844), 89-96.