FunBO: Discovering Acquisition Functions forBayesian Optimization with FunSearch

Restrictive Acquisition Function Input: Unlike previous approaches, this framework only uses the model’s marginal predictive distribution (predictive mean and variance) as inputs for the acquisition function. This restriction may limit its applicability compared to more complex but empirically powerful acquisition functions (e.g., Entropy Search, Knowledge Gradient).

Limited Comparison with Recent Meta-BO Work: While the paper includes comparisons with MetaBO [1], adding comparisons with more recent meta-acquisition functions (e.g., [2]) would provide a clearer picture of FunBO’s empirical performance.

Lack of Theoretical Insight: As with most LLM-based models, FunBO primarily relies on an empirical pipeline, without providing theoretical insights into the underlying methodology. It remains unclear to what extent this method will be useful in practice.

Computationally Intensive Training: Similar to other RL-based meta-acquisition function frameworks, FunBO’s iterative learning scheme requires a full BO loop evaluation at each iteration, which is computationally cumbersome.

[1] Volpp, M., Fröhlich, L. P., Fischer, K., Doerr, A., Falkner, S., Hutter, F., & Daniel, C. (2019). Meta-learning acquisition functions for transfer learning in bayesian optimization. arXiv preprint arXiv:1904.02642.

[2] Maraval, A., Zimmer, M., Grosnit, A., & Bou Ammar, H. (2024). End-to-end meta-bayesian optimisation with transformer neural processes. Advances in Neural Information Processing Systems, 36.

My concerns are mostly minor and arguably subjective.

• discretization the learnt AFs take as input a collection of points, does this not suffer the curse of dimensionallity? Typically AFs are continuously optimized by gradient ascent.

My following points focus essentially on the messaging of what is being learnt

• Structure of past functions I personally feel the paper should more clearly explicitly make the distinction between meta-learning a surrogate model and meta-learning an AF. Given an empirical distribution over functions, $\mathcal{G}$, if they all have the same peak, or similar shapes e.g. linearity, unimodailty or peridicity (e.g. use a method like Neural Processes), FunBO does not learn or exploit any such information, at multiple times the paper expresses “structure of past functions” and I feel this is misleading and risks accidentally blurring the boundary between surrogate model and AF. A sentence to clarify this boundary would help especially for a broader non-expert audience e.g.

“Note that meta learning surrogate model explicitly learns structure and features from past functions, e.g. minima in the same region of space, unimodality, linearity, and such common structure can be leveraged when fitting a surrogate model to a new function of the same class. FunBO does not itself explicitly learn any structure from past functions, rather, it takes as inputs an existing surrogate modelling method and history of functions and and outputs the best possible AF. In experiments in this work we use the most popular choice of surrogate modelling method, a GP with an RBF kernel.”

• scope of learnable AFs the AFs only take predictive mean, predictive variance, and incumbent as arguments. Hence expensive exotic methods like Entropy search and KG that require covariance cannot be learnt.

• why does AF vary for different function classes? I realise this is out of scope of this work and the authors may argue this is an empirical observation from past works. Assume we have two different sets of training objectives $\mathcal{G_1}$ and $\mathcal{G}_2$ and we learn two different acquisition functions, $h_1()$ and $h_2()$, then given one set of predictive means, variances, and incumbent $(\underline{\mu}, \underline{\sigma}, y^*)$ and we pass these into each acquisition function, why should we get a different recommendation back? Particularly since $x$ is not an input, and the points are order invariant, the AF has no knowledge of where the points are or and where the peak may be. (Personally, my only intuition is that if the GP is a bad model fit for one function class, the AF may increase exploration as that is all it can do. In which case FunBO is learning AFs that learn to proportionally compensate for quality of GP model fit. Similarly the past works/BO community is a crowd sourced genetic algorithm making the same mistakes and is not specific to FunBO.)

• Conditioned on surrogate model Following from above, as the FunBO inputs are a surrogate modelling method and set of functions, then the output AF is conditioned on the choice of surrogate model, I may have missed it but this should also be explicitly stated. One possible corollary is that changing the surrogate model (new kernel or prior mean function) would also require re-running FunBO and finding the new AF. (e.g. as above with a better suited GP, FunBO can find an AF that trades off explore and exploit appropriately)

• sentitivity for out of distribution In the limitations, the authors state that FunBO was run multiple times to get the out-of distribution results, I thank you to the authors for the honesty and transparency, however I am not confused, this wording makes it sound like cherry picking, is not the average of all repeated runs reported?

I have two main concerns about the approach the paper takes. First, it seems the settings in which the proposed method can be used are where repeated optimization runs can be conducted, so that the LLM can gain information about the search space and generate well-performing acquisition functions. At least, we would need access to objective functions drawn from the same distribution as the one we are targeting. If I'm dealing with a one-off optimization problem, this seems to exclude the proposed method, which raises questions about the usability of the method.

I also find the authors' argument for the desirability of the generated acquisition functions being interpretable not quite convincing. To me, the general-purpose acquisition functions are quite interpretable, including the ones the authors flagged for being "hard to implement and expensive to evaluate" (entropy search and knowledge gradient). This is because the motivations for the acquisition functions are clear (e.g., reducing entropy of the objective function's optimizer/optimum or increasing the expected one-step posterior mean optimizer), so even if the functional forms of these functions are complicated, I still view them as interpretable. The proposed method, on the other hand, limits itself to acquisition functions whose inputs include the predictive mean and standard deviation, which only offers simply ways to address the exploration–exploitation tradeoff and potentially sacrifices performance. On line 399, is that functional form really that interpretable? Say the acquisition function is not doing well, could interpretability help us modify the functional form in a way that increases performance, for example?

The authors could consider including entropy search policies, knowledge gradient, as well as portfolio-style policies that automatically adjust the trade-off parameter of UCB as baselines.

1. The presentation can be improved and more consistent. For example, the algorithm in Figure 1 has some inconsistent parts with the figure in Figure 1. The positions of figures can also be more considerable. For example, Figure 1 is on the top of page 3 and first mentioned at the end of page 4.
2. Algorithm pseudocode would be easier to read than the original code.
3. Figures 1, 4, 5, 6, and 7 can be more professional.

I have a couple questions (detailed in the respective section), which once clarified may solve some of the listed weaknesses.

The main numerical examples in this paper are low-dimensional optimization test functions and hyperparameter-tuning of a small-scale classification problem. I am wondering how this method can be scaled up efficiently to larger-scale problems (or real-life ones), where function evaluations are even more expensive; even ignoring the prompting cost, the evaluation loop is performed on each function of the provided class, which introduced a significant fixed cost to the algorithm. For example, in the context of hyperparameter-tuning, it is not clear from the experiments of this paper whether the proposed method can scale to large-scale, say, computer vision or robot learning tasks, where there may be far more moving parts (dimensions) to the BO problem than tested in this work. Some more evidence demonstrating scale-transfer, a la mu-parameterization (Yang et al. 2022), showing that FunBO training on smaller-scale problems can actually generate AFs that zero-shot transfer to the larger scale problem at hand, would be very enlightening on this front.

Another related issue that I do not see addressed in the paper is whether (or to what extent) the proposed method is scale-sensitive. In many settings (e.g. robot simulations), the scales of various parameters of the problem vary wildly, and are to some extent arbitrary by choice of unit. However, by using an LLM to generate the AF, it is not clear to me that the proposed method is sensitive to these possible variations of scale and thus possible source of covariate shift between the training and test tasks.

Yang et al. "Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer", 2022.