Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings

Thanks for your positive and valuable comments! Below please find our response. Corresponding experimental results can be found at https://anonymous.4open.science/api/repo/UniSO/file/F6Mo.pdf.

R1 Computational cost

Thanks for your suggestions. We have compared the computational cost in Table S1. Both the training data and the computational budget are available and affordable for academic research. We will revise to include this discussion. Thank you.

R2 Instability in metadata quality

Thanks for your insightful question! The metadata in our experiments is a manually crafted summary of the tasks rather than generated by a model, which is concluded from Design-Bench [1] and SOO-Bench [2]. Besides, to assess model performance under varying metadata quality, we conduct experiments with ablation studies on different metadata components (i.e., name, description, and objective).

Table S2 shows the results on in-distribution tasks, zero- and few-shot results on unseen tasks, revealing the effectiveness of each component in metadata. We will add the detailed discussions to the revised paper. Thank you.

R3 The approach projects input embeddings into a shared latent space, but it does not discuss how this scales with increasing dimensionality or dataset size

Thanks for your valuable comment. In this work, we represent inputs of varying dimensionalities using a string format {"x1":val1,"x2":val2,...}. These strings are tokenized and processed through an embedder to obtain embeddings of a fixed dimension. After that, we employ a nonlinear projection to map the embeddings to the shard latent space, which is a common practice [3]. Thus, our approach is capable of handling inputs with different dimensionalities.

While the approach is flexible, it may face information compression challenges when dealing with extremely high-dimensional inputs or large datasets, as the embedded size is fixed. Investigating potential scaling laws in LLMs for universal offline BBO remains an interesting direction for future research. We will add a discussion about this in the revised paper. Thank you.

R4 Loss balancing strategy

Thanks for your valuable suggestion! We conduct experiments by removing this strategy. Results in Table S3 clearly demonstrate that the strategy is beneficial. We use a similar strategy after removing certain loss (e.g., $L_{lip}$): $\frac{\partial L_{total}}{\partial\boldsymbol{\theta}}=\frac{\partial L_{main}}{\partial\boldsymbol{\theta}}+\frac{L_{main}}{L_{con}+\delta}\cdot\frac{\partial L_{con}}{\partial \boldsymbol{\theta}}$. We will revise to make it clear. Thank you.

R5 UniSO-T vs. UniSO-N

Thanks for your insightful question! Their main difference is whether the initial model is pre-trained or not:

• UniSO-N uses a pre-trained T5-small encoder to encode the input strings, following [4-5];
• Since the token representation of numeric y has a different token vocabulary from pre-trained LLM [6], UniSO-T is trained from scratch.

The inferior performance of UniSO-N may stem from the inconsistent bias from different modalities between natural language and numeric representation [7]. Besides, [8] theoretically investigates the issues of different modalities.

To further investigate if training from scratch would improve the performance of UniSO-N, we compare UniSO-N both from scratch and from pre-trained in Table S4, where UniSO-N trained from scratch performs better than that trained from the pre-trained encoder.

To better understand this phenomenon, we analyze the rank correlation between the ground-truth scores and model predictions in the out-of-distribution (OOD) regions during the training, which is an important metric for offline BBO [9]. The results in Figure S1 show that pre-trained models initially show poor performance, and there is only a small improvement later on. How to align different modalities well is a critical future work.

We will add these discussions in our revised paper. Thanks again for your insightful comments!

---

Thank you again for your encouraging comments! We hope that our response has addressed your concerns, but if we missed anything please let us know.

References

[1] Design-Bench: Benchmarks for data-driven offline model-based optimization. ICML, 2022.

[2] SOO-Bench: Benchmarks for evaluating the stability of offline black-box optimization. ICLR, 2025.

[3] A simple framework for contrastive learning of visual representations. ICML, 2020.

[4] Understanding LLM embeddings for regression. TMLR, 2025.

[5] Predicting from strings: Language model embeddings for Bayesian optimization. arXiv, 2024.

[6] OmniPred: Language models as universal regressors. TMLR, 2024.

[7] Position: Why tabular foundation models should be a research priority. ICML, 2024.

[8] A theory of multimodal learning. NeurIPS, 2023.

[9] Offline model-based optimization by learning to rank. ICLR, 2025.

Thanks for your encouraging and valuable comments! Below please find our response. Corresponding experimental results can be found at https://anonymous.4open.science/api/repo/UniSO/file/GL3R.pdf.

R1 Is "expert" in Table 1 a basic MLP regressor?

Yes. It refers to a numeric-input MLP regressor. We will revise to make it clear. Thank you.

R2 Future work like more data or more exotic search spaces

Thanks for your valuable comments! We wholeheartedly agree with your ideas. There are many interesting directions for future work:

• Training with more data with balanced ratios from different trials;
• Applying different modalities as metadata, instead of solely upon human-summarized text;
• Designing efficient numeric representation or tokenization.

We will keep focusing on discovering better paradigms for universal BBO. Thanks for your encouragement!

R3 Advice for better visualization

We agree that adding more styles of presentations will be better for understanding. According to your suggestions, we additionally show the generalization capability of improved UniSO-T method on unseen tasks in Figure S1, which can better show the generalization and effectiveness of few-shot finetuning. We will change other experimental presentations in our paper in the revised version. Thank you very much!

R4 UniSO-T vs. UniSO-N

Please refer to R5 of Reviewer F6Mo due to space limitations.

R5 y-normalization

We normalize y from multiple tasks following [1], as discussed in Section 3.3:

• Apply z-score normalization inside a task to shift the score to a standard normal distribution;
• Perform a normal distribution fitting process on the subset of data points whose values are below the median. To decrease the influence of extreme outliers, we use a transformation from percentiles to z-scores.
• Transform the scoring distribution through sequential application of min-max scaling $y\leftarrow\frac{y-y_{min}}{y_{max}-y_{min}}$ and logarithmic transformation for re-scaling.

It is quite essential for UniSO-N to normalize y, since an MLP regressor requires y normalization to prevent gradient explosion or vanishment. But objective normalization can be problematic when encountering outlier feedback values at test-time [2] (especially when few-shot evaluations are allowed). How to eliminate such issues (e.g., modeling of UniSO-T does not require y-normalization) for BBO is a promising future work.

R6 Is it necessary to start off with a pre-trained checkpoint, or does random initialization work fine too, for the T5-small model?

Thanks for your constructive question! We conduct a more detailed analysis and find that it is not necessary to start off with a pre-trained checkpoint. Pre-trained language model checkpoints contain priors from natural language, which, however, would introduce harmful bias for offline BBO.

From the final performance in Table S1 and OOD rank correlation (an important metric for offline BBO [3]) curve in Figure S2, we find that UniSO-N trained from scratch consistently performs better than that from a pre-trained checkpoint. The pre-trained models initially demonstrate poor performance, and there is only little improvement later on. These biases may come from different modalities between natural language and numeric representation [4], and how to mitigate this gap and align different modalities well is critical future work in our community.

We will add this discussion to our revised version. Thank you again for the insightful suggestion!

R7 T5 model size

Thanks for your insightful question! Due to computational resource constraints, we can only employ models of similar size to T5-Small. As a follow-up experiment, we further train a larger model of UniSO-N, which utilizes a T5-Base embedder. However, the performance results in Table S2 and the loss curve in Figure S3 show that equipped with a larger T5-Base embedder, UniSO-N performs even worse. Since a larger pre-trained model incorporates more prior, there may introduce more unfavourable biases for offline BBO, which also aligns with our observation in our response in R6. As for UniSO-T, we fail to train with a larger model size due to limited computational resources (since UniSO-T occupies more GPU memory than UniSO-N).

Thanks for your great question! We will include more a detailed discussion on this point.

---

Thank you again for your encouraging comments! We hope that our response has addressed your concerns, but if we missed anything please let us know.

References

[1] Predicting from strings: Language model embeddings for Bayesian optimization. arXiv, 2024.

[2] OmniPred: Language models as universal regressors. TMLR, 2024.

[3] Offline model-based optimization by learning to rank. ICLR, 2025.

[4] Position: Why tabular foundation models should be a research priority. ICML, 2024.

Thanks for your valuable comments. Below please find our response. Corresponding experimental results and references can be found at https://anonymous.4open.science/api/repo/UniSO/file/4Ajw.pdf.

R1 EA-based BBO method doesn’t warrant ICML community’s much attention; why don’t you adopt more effective optimizers (e.g., BO) instead of EAs?

We understand your concerns, but there is a misunderstanding. Our work is not an EA-based BBO method. We focus on offline BBO, a recent popular topic in ML community (A paper list can be found in Table S1). However, none of these studies considers the universal settings, which is a long-standing challenge in BBO.

In offline BBO, multiple optimizers (e.g., gradient ascent, EAs) can be applied to search in the model. Since gradients are difficult to compute in string space, we use EA instead, which is a common practice in offline BBO. In fact, any BBO algorithms are applicable. According to your comments, we further add two methods: BO-$q$EI (based on BoTorch) and CMA-ES (based on evosax). The results in Table S2 show that BO-$q$EI performs best and the default EA used in our experiments is still a competitive method.

We will add these discussions to our revised paper. Thank you.

R2 The preference of ML community: Either prefer provable algorithms or using LLM-based heuristics

Thanks for your comments. However, we beg to differ with you on this matter. A position paper at ICML’24 [1] comprehensively discusses LLM for BBO. In our opinion, there are two major branches in LLM for BBO:

• Leveraging general intelligence (e.g., reasoning, coding) of LLM to design algorithms or heuristics via natural language [2-3]. As LLM shows impressive general intelligence, several works utilize LLM to generate optimization code and heuristics by prompt engineering based on natural language. Representative works such as FunSearch [4] and EoH [5] iteratively search for better algorithms and heuristics, and GENIUS [6] employs LLM to find better neural architecture code.
• Leveraging the representation ability of LLM to improve the ability of learned BBO algorithms via tokens or embeddings [1]. This branch mainly focuses on how to efficiently utilize the inner embeddings of LLM. For example, [7] shows that when numeric $y$ is tokenized into a sequence, LLM is capable of universal regression via pre-training at scale; [8-9] show the superiority of LLM embedders for universal regression and BBO tasks, while [10-11] represent input values as tokens and thus learn in-context via LLM to solve BBO.

Our work falls into the second category, especially extending the scope to universal offline BBO for the first time. We deliver a unified representation that enables universal offline BBO possible and propose two effective advancements to overcome the barriers. The first branch you mentioned could help further enhance the performance of our approach. Thanks to the recent rapid development of frameworks [3], we can conveniently test whether this idea works in the future.

Additionally, the performance of our proposed method is not entirely heuristic; it is based on some theoretical foundations.

• $L_{con}$ is used for embedding distribution alignment, which is actually isomorphic to the relaxed variant of InfoNCE, a popular loss in contrastive learning that has been well studied [12].
• $L_{lip}$ aims to enhance smoothness. [13] has proven that it supports the lower bound on the correlation of the embedding space and the objective space. Besides, Figure 4 in [8] shows the importance of the Lipschitz continuity for regression at different model scales and problem dimensions.

Overall, we believe that our work contributes substantially to the LLM for BBO community. We will revise to add these discussions. Thank you very much.

R3 Computational cost

Please refer to R1 of Reviewer F6Mo due to space limitations.

R4 Details of EA

Thank you for pointing this out! We are sorry for not making it clear. We initialize the population as the top-$k$ scoring designs in the dataset, where $k$ is the population size. For continuous tasks, we use simulated binary crossover and polynomial mutation; For categorical tasks, we use uniform crossover and random replacement mutation. These operators are the default operators in [14]. We will add these to our revised paper. Thank you.

R5 UniSO-T vs. UniSO-N

Please refer to R5 of Reviewer F6Mo due to space limitations.

R6 Influence of metadata components

Thanks to your suggestion, we have conducted experiments to verify the individual contributions of metadata components. The results in Table S3 show the effectiveness of each component in metadata. We will add more detailed discussions to our paper. Thank you.

---

We hope that our response has addressed your concerns, but if we missed anything please let us know.

References are provided in Table S4.

Thanks for your valuable comments. Below please find our response. Corresponding experimental results can be found at https://anonymous.4open.science/api/repo/UniSO/file/TeSq.pdf.

R1 Some typos

Thanks for pointing these out. We have revised them and checked them thoroughly. Thank you very much.

R2 Are single-task experts the same as numeric-input experts?

Yes, the single-task experts are numeric-input experts. We will revise it to make it clear. Thank you.

R3 More elaboration on why LLM cannot perform well on BBO via ICL

Thanks for your insightful comments. The no free lunch (NFL) theorem [1-2] states that no single algorithm can universally outperform others on average over a uniform distribution of learning problems (i.e., all problems are equally likely, with no inherent structure or bias). Crucially, LLM’s success in ICL inherently violates this uniformity assumption [3]: real-world tasks (e.g., language, vision) occupy a highly structured subspace (e.g., grammar rules, spatial locality), which LLM exploits through pre-training on non-uniform, human-aligned data distributions. In contrast, general BBO problems—when stripped of metadata—align with the NFL assumption, because the lack of prior knowledge about problem distributions (e.g., smoothness) forces algorithms to treat the space as uniform [1], rendering cross-problem generalization impossible. Thus, we introduce metadata during training, which is an attempt to explicitly break this uniformity by restricting the problem space to a structured, domain-specific subspace, thereby enabling algorithms to leverage these useful biases. We will revise it to make it clear. Thank you.

R4 Why not train an in-context regressor similar to Predicting from Strings paper?

Thanks for the valuable question. Unlike online BBO, traditional offline BBO only allows evaluation of solutions in one batch and does not utilize the in-context information. Following the common practice in offline BBO [4], we use MLP as the regressor. We agree that it is valuable to test the performance of an in-context regressor. According to your suggestion, we consider offline BBO as a one-epoch online BBO and implement the in-context regressor using TNP [5], treating the top-scoring offline solutions as the contexts. Following Predicting from Strings [6], we maximize the UCB acquisition through EA to obtain final candidates.

Results in Table S1 show that UniSO-N + TNP performs worse than UniSO + MLP. This suggests a potential mismatch between ICL and the offline BBO setting. Extending these approaches to universal offline BBO remains an interesting future work. We will include this discussion in our revised paper. Thank you.

R5 What is the encoder in Section 3.3?

UniSO-N: encoding input strings by a pre-trained T5-small encoder; UniSO-T: training a T5 encoder-decoder from scratch and using its encoder to encode input strings; The encoders for metadata of two approaches are the pre-trained T5-small encoder. We will revise it to make it clear. Thank you.

R6 Don’t UniSO-N train with the regressor during finetuning?

Thanks for pointing this out! We are sorry for not making it clear. In fact, during finetuning, we update the embedder via $\frac{\partial L_{total}}{\partial\boldsymbol{\theta}}=\frac{\partial L_{con}}{\partial\boldsymbol{\theta}}+\frac{L_{con}}{L_{lip}+\delta}\cdot\frac{\partial L_{lip}}{\partial \boldsymbol{\theta}}$ without incorporating the main loss. We will revise it. Thank you very much.

---

We hope that our response has addressed your concerns, but if we missed anything please let us know.

References

[1] No free lunch theorems for optimization. IEEE TEvC, 1997.

[2] The supervised learning no-free-lunch theorems. Soft computing and industry: Recent applications, 2002.

[3] Position: The no free lunch theorem, Kolmogorov complexity, and the role of inductive biases in machine learning. ICML, 2024.

[4] Offline model-based optimization: Comprehensive review. arXiv, 2025.

[5] Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. ICML, 2022.

[6] Predicting from strings: Language Model embeddings for Bayesian optimization. arXiv, 2024.