Predicting from Strings: Language Model Embeddings for Bayesian Optimization

We thank the reviewer for the detailed feedback. Inline responses below:

Weaknesses

1. Novelty

   • (a) Comparisons to [1, 2]: While we agree that [1,2] have a similar design, these works have been very specific to chemistry and biology problems, and have also used domain-specific embedders (e.g. "Chem-T5") but have not been tested on traditional and general blackbox optimization. Our novelty comes from understanding the performance of LLM embeddings in more general scenarios.
   • (b) Our emphasis isn't on the specific ICL regressor, as one could indeed interchange it with [3,4]. Ours indeed is a simple reimplementation of (Nguyen, 2022). We will cite [5,6].
   • (c). "Meta-learning isn't novel" - But the term "meta-learning" is quite broad and there are numerous different ways to perform meta-learning for BBO. We don't think the existence of [7, 8, 4] means there are no new methods to be discovered. We also discussed the nuanced comparisons between different meta-learning methods in our related works section, and believe that our method has the distinct advantage of avoiding tabular featurization by representing inputs as raw strings.

2. "LM embeddings providing benefits" - We indeed don't think (and neither do we claim so) that the benefit comes from massively improved performance over traditional algorithms, especially since the model was trained over synthetic data, and the traditional field of blackbox optimization is the most competitive domain with multiple strong "specialist" methods. We believe the real benefit to our work is moving towards a generalist form of regression over multiple different domains.

3. Comparison with [10, 11]: We specifically discussed comparisons to [10, 11] in our related works section. [10] requires a very customized tokenization that only works with tabular data, while [11] is not in-context, making it difficult to use in optimization scenarios. Our method can be seen as "trying to combine the best of both worlds", i.e. allowing free-form strings while still providing strong ICL and uncertainty quantification capabilities.

Thanks for your positive feedback on the simplicity of our method - Responses below:

Weaknesses

1. Recently, (Tang et al, 2024) discovered that LLM embeddings can be quite strong for high-dimensional regression, even for objectives up to e.g. 100 parameters. Along with the T5-XL encoder dimension being 2048, we hypothesize that higher dimensional objective functions should also therefore be regressable in our ICL case as well.

Questions:

1a) Since we trained over synthetic data, there was no need for metadata - we mentioned the possible use of metadata as an option if there are additional "graybox" clues to the objective (e.g. the name of the objective), which is similar to the setting seen by (Chen et al, 2022).

1b) $y$ was sent through a basic MLP (ReLU) layer, whose output dimension is $d$.

1c) In principle, since we're using language model inputs, $x$ does not need to be re-scaled, provided that enough pretraining has been done, similar to OmniPred (Song et al, 2023). Since in this paper we used cheap synthetic data for pretraining where each parameter in $x$ is bounded within $[-5,5]$, at inference, the model is "used to seeing" numbers within $[-5,5]$, and thus during inference time, we would need to apply an affine transformation from any new bound $[a,b]$ to $[-5,5]$.

Thanks for the detailed feedback and the compliments around our paper's readability. Responses in-line, and we'll incorporate many of the clarifying questions and feedback into the paper.

Weaknesses

1. We're using a standard UCB acquisition function, with a coefficient of 1.8. Thus, the acquisition score is (mean + 1.8 * std).
2. Our paper's purpose is to design regressors specifically towards optimization purposes. Note that this subtly means our model does not need to satisfy certain requirements for pure regression (in the data science sense), which is a also true for GP-based methods:
   • $y$-normalization does not need to be strictly invertible or 1-to-1
   • The regressor must be able to take in additional online observations at inference time.
   • Uncertainty quantification is much more important than mean prediction.
3. Zero-padding techniques are used in PFNs4BO, but they're still very specific to tabular data and have size limitation issues when encountering categorical parameters with many possibilities.
4. This is a small detail that isn't too important to our paper's main point - we mapped $y$ to $\mathbb{R}^d$ simply for symmetry with the $x$ embedding, but could've also just concatenated to $\mathbb{R}^{d+1}$ as you mentioned.
5. This is another subtle but small detail - we can't use auto-regressive attention because the $y$-normalization has a clear separation between history and target points. Thus, the attention must be in the form of a Prefix-LM style pattern - i.e. all target points are fully dependent on history, but not dependent on each other.
6. Agreed - it is possible using soft-prompt optimization techniques to perform gradient-based acquisition maximization. In this paper, we used zeroth-order optimization since it's easier for comparisons and implementation.
7. Random search and quasi-random search are standard baselines. Note that Vizier GP is one of the strongest industry-standard Bayesian Optimization algorithms, which our method matches.
8. The regressor did not see any of the test functions during training. The parameter ranges are the same for BBOB (i.e. every parameter is within $[-5,5]$) but note that any other search space can simply be warped to match $[-5,5]$.
   • As for interpolation - we think that is fundamentally what regression (+ some inductive priors) actually is, however.
9. As mentioned in Question 2 - there are nuanced differences between standard regression and Bayesian Optimization, that require the paper be specific to either application.
   • This is common within previous literature (e.g. one paper discusses Random Forests for regression, while paper adding modifications to be useful for optimization).

Questions:

1. RAG is fundamentally about retrieval (i.e. looking over a corpus of already known facts), which is different from regression, which fundamentally about predicting new "numerical facts" from a query. So we think they're fundamentally different.
2. Yes the standard way (using feature based techniques) of representing $\mathcal{X}$ would have finite dimension. But note that $\mathcal{X}$ doesn't intrinsically have a canonical dimension (e.g. we could've used a linear projection to make the resulting feature have higher dimension). When expressed as strings, the notion of "dimension" no longer has meaning, in a sense.
3. One can embed additional information that might help prediction (e.g. if one has the name of the function, this would help prediction significantly) - we meant to say that our proposed model makes graybox optimization much easier.
4. Doubtful - T5 models are trained only over the C4 (i.e. web crawled data) which consists of mostly English text, and is unlikely to contain any substantial numeric data.

Thanks for spending the time to thoroughly read through our paper and its relation in previous work. We address specific issues below, and will take care to incorporate some of this feedback into updated drafts.

Weaknesses

1.Novelty: R1 and R2 are completely at the mercy of service-provided LLMs and their business priorities, because they rely on emergent behaviors of LLMs. This means that performance is a function of completely uncontrollable factors, such as:

• Whether the service allows long-contexts.
• Which data mixtures were used during pretraining / post training
• What the service provider decided for architecture size.

Since LLM providers (at least those we're aware of) aren't interested in use-cases for optimization, it's impossible for an optimization researcher to improve such model capabilities themselves. At times, we've seen e.g. ChatGPT worse than random search due to these issues. In comparison, our work provides a method specifically tailored to optimization and allows tuning to the user's offline (x,y) data, which is the most significant factor to regression performance.

Also, the term "LLMs for Optimization" is an extremely broad phrase, even though there are endless different ways to use LLMs for optimization, similar to "GPs for optimization" also having many different variants. We don't believe R1 or R2 existing should mean that there's no more novelty to using LLMs in various other ways.

2.Design Decision: On fine-tuning via ICL by simply placing strings - this is in all honesty a fine alternative, but as we discussed in our related works section:

• If using raw strings, this requires a presumably large pretrained model (to understand English) with a higher memory consumption, which would easily limit the context window (as seen in R2, where at most ~60 trials could fit). This issue would be exacerbated when each $x$ may also contain additional metadata.
• If using customized tokenizations for better compression (as in Chen et al 2022): This is fairly restrictive to tabular formats only, and has difficulty handling e.g. combinatorial data.

3a) As mentioned in our general response, we're not advocating that our method is SOTA at all when it comes to comparisons against domain-specific optimizers, rather proposing an alternative formulation which allows the use of strings for regression during optimization.

3b) Higher computational complexity: It is indeed true, that the embedding method currently has a higher inference speed. However, there are numerous ways to reduce costs over time, (using efficient attention mechanisms throughout, using alternative acquisition optimizers, etc.). We consider our current paper a promising prototype to start from.

3c) Performance improvements - As mentioned in our general statement, our goal is not to advocate for a much better performance over GP methods (we would not have shown such experiments equalling against GP otherwise), but to understand whether embedding-based regression performs reasonably, since it is a natural design and gateway for more string-based regression methods.

Questions

1. Note that if the model was pretrained on e.g. parameter bounds of [-5,5] only, it's always possible at inference time to warp any other bound down to [-5,5], with e.g. an affine scaling. Eventually however, warpings should not be needed once the model has been trained over multiple different problems with different space bounds.
2. We assume that the synthetic dataset is "infinite sized", since it is cheap to generate. For practical purposes, we generated an offline dataset of 1M trajectories (each with e.g. 300 random trials), where each trajectory's objective was formed by a cartesian product of (BBOB training function, shifting, categorization).
3. The mutations were uniformly random (e.g. for permutations, select two random indices and swap them), and the population size of 50 was indeed used throughout. We believe these hyperparameters were fairly reasonable (we also used a recommended tournament size of $\sqrt{\text{population}}$).
4. The embedding process parallelization should be automatically taken care of via typical computation graph compilation. At inference, historical embeddings can also be cached to reduce redundant computations.