Large Language Models to Enhance Bayesian Optimization

[P5] Reproducibility

We are committed to ensuring full reproducibility of our results. To ensure this, we will release code, prompt templates, and search history/results for all 79 tasks upon acceptance of the paper. This will allow future research to benchmark against our findings without needing access to our exact API endpoints.

While we acknowledge the reproducibility of the results will depend on the underlying LLM, we would like to highlight the generality of our approach, emphasizing that the novelty and contributions are not in the capabilities of any specific LLM, but in the notion of using LLM for model-based BO. Ultimately, as LLMs improve, we believe so will the performance of our proposed approach.

---

[P6] Additional benchmarks

We appreciate your recommendation. Accordingly, we have expanded our analysis to include 24 additional tasks (8 datasets, 3 ML models) from HPOBench in Appendix E.4. In the interest of time, we note that we have only included results from one seeded run, but we plan to include results across 5 runs (which will be updated as they come in). Our expanded analysis reveals LLAMBO achieves strong performance across this extended set of results, further validating the efficacy and generalizability of our approach.

---

[P7] Additional comments about experiments/writing

• Clock time. We have now included average clock time for LLAMBO and baselines in Appendix E.5.
• Uncertainty. We have updated all figures to include 95% CI.
• Random baseline. We have incorporated random search as a baseline in our end-to-end results in Figure 7.
• Typos. These have been fixed.
• Figure 8. Your interpretation is correct, all baselines receive the same initialization points (and we highlighted this in S6).

---

[P8] Questions

1. We sample $M=20$ candidate points, generate $K=10$ MC predictions, and set the exploration hyperparameter to $\alpha=-0.1$. The choice of $M=20$ aligns with the default/recommended number of candidates sampled in popular TPE implementations (including HyperOpt, Optuna). We set $\alpha=-0.1$ based on observations described in Figure 6 as a value that balanced diversity-performance.
2. We utilized default meta-hyperparameters included in baseline optimizers.
3. These details have been included in Appendix D.
4. gpt-3.5-turbo version 0301.
5. The random baseline samples the same set of M=20 points for each seed, irrespective of the number of observed samples. This accounts for the constant regret and diversity across different values of N.
6. To ensure an equitable comparison with other optimizers, we did not employ warmstarting in our end-to-end evaluation. All methods, including ours, were initialized with the same set of points.
7. When evaluated solely as a surrogate model, SMAC exhibits better predictive performance at lower values of $N$. A possible explanation is overfitting in GP with smaller $N$, whereas RF will benefit from the variance reduction effect from ensembling. At higher values of $N$, however, GP demonstrates enhanced uncertainty estimates and predictions, leading to better overall performance in end-to-end evaluations.
8. We apologize for a plotting error in the SMAC curve that incorrectly included initialization points in the initial 5 iterations. This has now been corrected in average results in Figure 7, 20, 21, but also individual task results in Figures 25-27.
9. We used the pip available implementation of HEBO and default meta-hyperparameters. The original HEBO paper reports results from either batch mode (8 points per iteration) for 16 iterations or (1 point per iteration) for 100 iterations, which differs from the experimental settings in our work. Additionally, we noted similar findings that while HEBO was highly competitive given enough iterations, it has been observed to be inefficient in earlier iterations (e.g. Figure 4 of [1]).

[1] Müller, S., Feurer, M., Hollmann, N. & Hutter, F.. (2023). PFNs4BO: In-Context Learning for Bayesian Optimization. <i>Proceedings of the 40th International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 202:25444-25470 Available from https://proceedings.mlr.press/v202/muller23a.html.

---

We thank the reviewer for your help in improving our work. Please let us know if our latest changes have addressed your concerns, and if there is anything else you would like to see.

[P2] Prompt design

Thank you for this suggestion. Our prompt design adheres to the framework outlined in S2.2, including three key components: (1) optimization problem description, (2) optimization history, and (3) explicit task instructions. We would like to re-emphasize that the primary objective of our research as to investigate the feasibility of enhancing BO with LLMs, and not designing highly robust or optimized prompts. Most of our efforts on prompt design were concentrated on (3), ensuring that LLAMBO behaves robustly from a system perspective, consistently generating predictions/samples in valid formats amenable for subsequent processing.

To further shed light on this process, we have further elaborated on the prompt design process in App C.2. This also includes a new ablation study wherein we methodically evaluated the impact of various components, specifically (a) omitting meta-data from context description (c.f. (1)) and (b) specific task instructions (c.f. (3)). Our ablation revealed that standard LLAMBO configuration outperformed other variants, underscoring the significance of each prompt component in enhancing overall performance. Our findings on (a) indicated that meta-data played a hand in improving search performance further, but excluding it did not significantly affect performance. On examining (b), we also found that the task-specific instructions improved validity of proposed candidate points, leading to more efficient search. We elaborate on (a) further in [P4] below.

However, we view combining recent advances in automatic prompt tuning within our framework as a promising direction that could further improve the optimization performance of LLAMBO.

Actions taken. We have improved our manuscript with a (1) description of the prompt design process and (2) prompt ablation study in App C.

---

[P3] Clarity

We appreciate this concrete recommendation, which we have followed by reorganizing the paper to include related works in the main text.

---

[P4] Meta-data leakage

We acknowledge your concern regarding the potential for meta-data leakage, particularly the risks of LLM memorizing task-specific hyperparameters. To mitigate this, our evaluation included three private and three synthetic datasets. It is crucial to note that no semantic information that could be used to directly identify underlying datasets is provided (please see our prompts in App C). Our synthetic datasets, while generated from known DGPs, had controllable input dimensions and sampling distributions, set by us.

Regarding the meta-data in our prompts, we provided only high-level information, inclusively: [# of # of features (# categorical, # numerical), # of samples, and tasks described as either classification or regression]. This information is provided as HPT optimization should benefit from this information, for instance, to avoid overfitting on big-p-little-n tasks.

To concretely address your concerns, we point to three pieces of empirical evidence:

• In our warmstarting experiments (S3), we analyzed a no context zero-shot prompt, where no semantics or meta-data is provided (see App C), and observed the LLM was able to suggest configurations that exhibited stronger correlations (indicating dataset-agnostic knowledge of generalizable correlations in the optimization landscape) that improved optimization performance.
• In S4.2 and S5, we performed ablations that removed the aforementioned meta-data from the prompts LLAMBO (UnInf) and found that, while the performance degraded relative to the full prompt, the performance was consistently superior to considered baselines (see Figure 4, 6).
• [New result] To further address any concerns of meta-data leak, we perform an additional ablation study, where we removed any meta-data from the prompts and analyzed resulting end-to-end performance. This is included in App C.2. Expectedly, this ablation performs slightly worse, as meta-data can improve hyperparameter suggestion (e.g. specific hyperparameters better suited for large-p-small-n to guard against overfitting). However, we note that the ablation was consistently superior against compared baselines, and achieved very similar optimization performance as the full setting. This highlights LLAMBO’s effectiveness despite the lack of metadata, indicating the performance goes beyond mere reliance on data memorization or leakage (as it operates without access to specific dataset information).

We hope this sufficiently addresses your concerns about meta-data leakage.

Actions taken. In response to your feedback, we have conducted an ablation study to assess the impact of including meta-data in prompts on end-to-end performance in App C.

Thank you for your helpful feedback and comments. We aim to address all the individual points in your review here, but please also see the revised manuscript for changes (highlighted in teal).

---

[P1] LLMs and non-sequential data / empirical uncertainty

Modelling non-sequential data. We thank the reviewer for raising this point. Using an LLM for optimization of non-sequential data is central to our paper, and how to best model non-sequential data forms the basis of our investigations. LLMs, while being sequential models, possess extensive prior knowledge, exhibit strong few-shot learning capabilities, and flexibly incorporate contextual information. These are desirable properties for BO algorithms. Our approach effectively taps into these LLM capabilities to enhance BO, albeit necessitating data serialization.

We note that in other research domains, LLMs have been evaluated for processing non-sequential data. For example, [1-3], which demonstrated LLMs’ effectiveness on tabular regression, classification, and generation, respectively (where the data under consideration lacks sequential structure). More pertinent to our study, [4] employs a Transformer to sequentially process optimization results for HPO. Our empirical analysis further substantiates our method's effectiveness: despite data serialization, we can obtain a strong surrogate model (notably in few-shot scenarios, as shown in S4), along with an efficient candidate point sampler (S5) and an overall effective system (S6).

Ultimately, the serialization of non-sequential data within our approach effectively elicits prior knowledge about problem space correlations, and capitalizes on the few-shot learning and contextual understanding capabilities of LLMs, resulting in an improved, sample-efficient search method. This advantage is vital in most BO applications, and we believe motivates our methodological choice. While performing BO with a sequential model is unconventional, we believe the strong performance and potential improvements of this novel approach is a valuable contribution and would inspire future research in this field.

Empirical uncertainty. Just to clarify, we permute the order of in-context examples, not the hyperparameters within each example. While MC sampling for uncertainty estimation may lack the principled grounding of GP/RF, it is straightforward to implement and widely accepted technique. For example, in deep ensemble literature [5], empirical variance is used for uncertainty estimation. Conceptually, if we consider the predictive distribution produced by an LLM as being induced by the weights of the LLM ($\theta_{LLM}$) and the few-shot prompt ($\theta_{prompt}$), i.e. $p(y| x; \theta_{LLM}, \theta_{prompt})$ (note that $(\theta_{prompt}, x)$ represents the complete prompt), our MC sampling is an approximation of the expectation $E_{\theta_{prompt}\sim p(\theta_{prompt})} [p(y|x; \theta_{LLM}, \theta_{prompt})]$, which marginalizes over the distribution of different prompts. Note, this is conceptually identical to the approach in deep ensembles, where $E_{w\sim p(w)}[p(y|x; w)]$. Intuitively, each differently prompted LLM can be considered an ensemble member, where we compute empirical variance from the ensemble. Indeed, we demonstrated that this approach of combining MC sampling with permutations of in-context examples improved calibration of uncertainty estimates, enhancing end-to-end performance.

[1] Dinh, T., Zeng, Y., Zhang, R., Lin, Z., Gira, M., Rajput, S., Sohn, J.Y., Papailiopoulos, D. and Lee, K., 2022. Lift: Language-interfaced fine-tuning for non-language machine learning tasks. Advances in Neural Information Processing Systems, 35, pp.11763-11784.

[2] Hegselmann, S., Buendia, A., Lang, H., Agrawal, M., Jiang, X. and Sontag, D., 2023, April. Tabllm: Few-shot classification of tabular data with large language models. In International Conference on Artificial Intelligence and Statistics (pp. 5549-5581). PMLR.

[3] Borisov, V., Sessler, K., Leemann, T., Pawelczyk, M. and Kasneci, G., 2022, September. Language Models are Realistic Tabular Data Generators. In The Eleventh International Conference on Learning Representations.

[4] Chen, Y., Song, X., Lee, C., Wang, Z., Zhang, R., Dohan, D., Kawakami, K., Kochanski, G., Doucet, A., Ranzato, M.A. and Perel, S., 2022. Towards learning universal hyperparameter optimizers with transformers. Advances in Neural Information Processing Systems, 35, pp.32053-32068.

[5] Lakshminarayanan, B., Pritzel, A. and Blundell, C., 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30.

[P4] Questions

• Color legend. Thank you for this suggestion, and have incorporated a color legend in the revised manuscript.
• No context warmstarting. No context warmstarting is expected to be better than random initialization. In fact, this was one of our initial sanity check to verify that LLMs contained useful prior knowledge about the optimization problem. This strategy relies on LLMs' intrinsic knowledge of hyperparameter correlations that are independent of specific problems. For instance, there are generalizable correlations, such as the interplay between tree depth and the number of estimators in Random Forests, which are crucial for managing the bias-variance tradeoff. Unlike random initialization methods, no context warmstarting exploits these inherent correlations identified by LLMs, thereby enhancing the optimization process.
• Discriminative SM. LLAMBO incorporates both MC sampling and permutation of in-context examples. This is in contrast to the LLAMBO (MC) variant, which exclusively relied on MC sampling. Our comparative analyses demonstrated that permutations improved calibration of uncertainty estimates obtained from MC sampling. We have updated the manuscript to be more clear about this distinction.
• $\alpha$. The candidate sampler generates points, denoted as $h$, from the distribution $p(h|s’, \mathcal{D}\_n)$. This process is conditioned on a desired target value $s’$ and the existing observations $\mathcal{D}\_n$. The target value $s’$ is determined using the parameters $\alpha$, based on the formula $s’ = s\_{min} - \alpha \times (s\_{max}-s\_{min})$. Here, $s\_{max}$ (worst) and $s\_{min}$ (best) represent the extremities of objective values observed during the search, essentially defining the range of these values. Consequently, $s’$ is calculated by adjusting the best observed objective ($s_{min}$) by an amount proportional to the total observed range ($s\_{max} - s\_{min}$). To illustrate, an $\alpha$ value of $-0.1$ corresponds to targeting the 90th percentile of the observed values, while $\alpha = 0.1$ aims to improve upon the best value by the 10th percentile. Conversely, setting $\alpha$ to 0 targets the current best observed value. We set $\alpha=-0.1$ based on observations described in Figure 6 as a value that balanced diversity-performance. We hope this explanation clarifies the role of $\alpha$ in our sampling process.
• End-to-end process. We appreciate this suggestion. The end-to-end procedure iteratively performs three steps: (1) sample $M$ candidate points $\\{\tilde{h}\_m\\}\_{m=1}^M$. (2) evaluate $M$ points using the surrogate model, i.e. $p(s|\tilde{h}\_m)$ to obtain scores $\\{a(\tilde{h}\_m)\\}\_{m=1}^M$ according to an acquisition function. We use expected improvement (EI), $a(\tilde{h}\_m)= E[\max(p(s|\tilde{h}\_m) - f(h\_{best}), 0)]$. (3) select point with the highest score to evaluate next, $h = argmax\_{\tilde{h}\in\\{\tilde{h}\_m\\}\_{m=1}^M}a(\tilde{h})$. We have included these details in App C.

---

We thank the reviewer for your help in improving our work. Please let us know if our latest changes have addressed your concerns, and if there is anything else you would like to see.

Thank you for your helpful feedback and comments. We aim to address all the individual points in your review here, but please also see the revised manuscript for changes (highlighted in teal).

---

[P1] Relevance to ICLR

Thank you for raising your concern. We believe that our work aligns well with the thematic areas outlined in the CfP, particularly with the domains of BO, LLM prompting, and more generally transfer learning and applications of LLMs. We consider our work as a contribution to BO, a key enabler in many ML fields and core topic in ICLR, as evidenced by prior publications [1-3]. Additionally, research on application and prompting of LLMs are also commonly featured at ICLR [4-6]. We believe that our findings not only demonstrate the feasibility of integrating LLMs to enhance BO, but also open up avenues for future research in enhancing optimization algorithms more generally, making it highly pertinent to the themes and community at ICLR.

[1] Wistuba, M. and Grabocka, J., 2020, October. Few-Shot Bayesian Optimization with Deep Kernel Surrogates. In International Conference on Learning Representations.

[2] Sussex, S., Makarova, A. and Krause, A., 2022, September. Model-based Causal Bayesian Optimization. In The Eleventh International Conference on Learning Representations.

[3] Volpp, M., Fröhlich, L.P., Fischer, K., Doerr, A., Falkner, S., Hutter, F. and Daniel, C., 2019, September. Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimization. In International Conference on Learning Representations.

[4] Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q.V. and Chi, E.H., 2022, September. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. In The Eleventh International Conference on Learning Representations.

[5] Zhou, Y., Muresanu, A.I., Han, Z., Paster, K., Pitis, S., Chan, H. and Ba, J., 2022, September. Large Language Models are Human-Level Prompt Engineers. In The Eleventh International Conference on Learning Representations.

[6] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K.R. and Cao, Y., 2022, September. ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations.

---

[P2] Reporting uncertainty

Thank you for this suggestion. We originally omitted error bars to improve visual clarity in interest of limited space. We have since revised the manuscript to include 95% CI on all plots (Fig 2 - Fig 8).

---

[P3] Prompt design

Thank you for this suggestion. Our prompt design adheres to the framework outlined in S2.2, including three key components: (1) optimization problem description, (2) optimization history, and (3) explicit task instructions. Given that the primary objective of our research was to investigate the feasibility of enhancing BO with LLMs, we did not prioritize extensive refinement of the prompts design. Most of our efforts on prompt design were concentrated on (3), ensuring that LLAMBO behaves robustly from a system perspective, consistently generating predictions/samples in valid formats amenable for subsequent processing.

To further shed light on this process, we have further elaborated on the prompt design process in App C.2. This also includes a new ablation study wherein we methodically evaluated the impact of various components, specifically (a) omitting meta-data from context description (c.f. (1)) and (b) specific task instructions (c.f. (3)). Our ablation revealed that standard LLAMBO configuration outperformed other variants, underscoring the significance of each prompt component in enhancing overall performance. Our findings on (a) indicated that meta-data played a hand in improving search performance further, but excluding it did not significantly affect performance. On examining (b), we also found that the task-specific instructions improved validity of proposed candidate points, leading to more efficient search.

Specifically to respond to your question: “do not recommend values at the minimum or maximum of allowable range, do not recommend rounded values”, these are part of (3). We included these instructions to prevent the LLM from suggesting candidate points that were under-precise or snapped to the limits of the search space. We found that this instruction significantly improved the percentage of suggested points that were valid (i.e. conform to search space) and unique (i.e. not duplicated). Please refer to C2 for a more in-depth discussion.

Actions taken. We have improved our manuscript with a (1) description of the prompt design process and (2) prompt ablation study in App C.

Thank you for your helpful feedback and comments. We aim to address all the individual points in your review here, but please also see the revised manuscript for changes (highlighted in teal).

---

[P1] High-dimensional optimization

Thank you for highlighting this concern. Our current work primarily focused on investigating the feasibility of integration of LLMs into BO framework and examining enhancements around various aspects of model-based BO. In this context, our analysis was centered around conventional HPT tasks. These tasks are drawn from widely used benchmarks (e.g. Bayesmark and newly added HPOBench), with dimensionality between $2-8$.

To further bolster our empirical analysis, we included additional results on $24$ tasks from HPOBench, a more recent and more complex HPT benchmark (please see App E.4). We note that in the interest of time, we have only included results from one seeded run, but we plan to include results across runs (which will be updated as they come in). Our analysis reveals LLAMBO achieves strong performance across this extended set of tasks, further validating the efficacy and generalizability of our approach.

With that being said, we agree that it is important to clearly define the scope of our work. Accordingly, we have revised the manuscript in the Introduction and Discussion. We see extending and evaluating high-dimensional BO tasks (e.g. those encountered in neural architecture search) as a promising future step. We see multiple potential avenues to achieve this, including evaluating our current approach with higher-dimensional natural language representations, or applying some search space decomposition techniques.

Regarding the specific limitation related to the LLM’s context window size, our rough calculations indicate that with $25$ iterations, the model we used (gpt-3.5-turbo with $4K$ context window) can support $16$ dimensional problems, which decreases to $8$ dimensions with $50$ iterations. It is noteworthy that the trend in LLM research is moving at pace towards models with larger context windows. Indeed, after the submission of our manuscript, OpenAI has released models featuring context windows ranging from $16K-128K$ ($4\times-32\times$). Therefore, we do not anticipate the context window as a major bottleneck constraining scalability of our approach.

Actions taken. We have (1) revised the manuscript to include a discussion of higher-dimensional BO problems and (2) included additional results on $24$ HPOBench tasks in App D.2 and E.4.

---

[P2] NN baseline

Thank you for pointing out the related work [R1]. This work utilized neural network-based surrogate models to enhance BO’s performance, particularly in high-dimensional problems. Additionally, the method improves computational efficiency by circumventing the need to invert large parameter matrices. In light of your suggestion, we have included a discussion of [R1] in the related works section of the revised manuscript. Furthermore, we empirically compared against it in App E.3. We observed that while STO is competitive against other baselines, especially during early stage of search, LLAMBO is consistently superior and found better solutions more efficiently.

---

[P3] Questions

• Thank you for this question. The use of a low negative $\alpha$ (e.g., $\alpha = -0.1$) aims to target the 'good' regions of the search space, specifically focusing on the top $10$% of observed points. This approach enables the LLM to exploit identified patterns in the existing observations and generate points in proximity to promising solutions. The underlying assumption here is that areas near good solutions are likely to yield similarly high-quality results. This exploitation of the search space could result in identifying candidates with lower regret. We note here that this approach is akin to the sampling strategy of TPE algorithms (which typically uses $\tau=0.25$, which corresponds to constructing density estimators with the top $25$% of observed points). Additionally, this 'good' region dynamically evolves as more points are sampled and evaluated. In our implementation, we set $\alpha=-0.1$ based on observations described in Figure 6 as a value that balanced diversity-performance.
• For details on the dimensionality of the BO tasks used in our experiments, please refer to App D.

---

We thank the reviewer for your help in improving our work. Please let us know if our latest changes have addressed your concerns, and if there is anything else you would like to see.

Thank you for your helpful feedback and comments. We aim to address all the individual points in your review here, but please also see the revised manuscript for changes (highlighted in teal).

---

[P1] Extension to more complex BO tasks

Thank you for highlighting this concern. Our current work primarily focused on investigating the feasibility of integration of LLMs into BO framework and examining enhancements around various aspects of model-based BO. In this context, our analysis was centered around conventional HPT tasks. These tasks are drawn from widely used benchmarks (e.g. Bayesmark and newly added HPOBench), with dimensionality between $2-8$.

To further bolster our empirical analysis, we included additional results on $24$ tasks from HPOBench, a more recent and more complex HPT benchmark (please see App E.4). We note that in the interest of time, we have only included results from one seeded run, but we plan to include results across runs (which will be updated as they come in). Our analysis reveals LLAMBO achieves strong performance across this extended set of tasks, further validating the efficacy and generalizability of our approach.

With that being said, we agree that it is important to clearly outline the scope of our work. Accordingly, we have revised the manuscript in the Introduction and Discussion. We see extending and evaluating more complex BO tasks (e.g. those encountered in neural architecture search) as a promising future step. We see multiple potential avenues to achieve this, including evaluating our current approach with higher-dimensional natural language representations, or applying some search space decomposition techniques.

Actions taken. We have (1) revised the manuscript to include a discussion of more complex BO problems and (2) included additional results on $24$ HPOBench tasks in App D.2 and E.4.

---

[P2] Prior knowledge about HPT

Thank you for raising this question. The effectiveness of LLAMBO relies on the LLM having useful prior knowledge about correlations in the optimization space. This stems from their extensive training on diverse datasets. It is important to note that due to the Internet-scale data involved in pretraining, it is challenging to pinpoint sources of domain knowledge. However, we conducted various empirical investigations to explore the LLM prior:

• One of our very first experiments was to verify that LLMs contained knowledge relevant to the optimization problem. This was examined through the warmstarting experiments in (S3). In the no context setting, we employed zero-shot prompting, where the LLM was tasked to generate warmstarting points without being provided any dataset-specific information. The goal was to assess the LLM's intrinsic understanding of hyperparameter correlations, independent of specific tasks. We observed that correlation patterns were notably higher than in randomly initialized samples (Figure 2) and warmstarting with these points enhanced the optimization process. This suggests the LLM has knowledge of generalizable correlation patterns between hyperparameters.
• Furthermore, in our existing analysis of the surrogate model (Section 4.1) and the candidate point sampler (Section 5), we evaluated a specific ablation setting, referred to as LLAMBO (UnInf). This variant operated without detailed problem context or explicit hyperparameter names, relying solely on raw observations. While this setting showed that the LLM could generalize from past observations, we observed improved performance when problem context and hyperparameter names were included. This suggests that the LLM's generalization performance is not solely reliant on past observations but also benefits from prior knowledge.

To address your question directly: while the LLM may/may not have been explicitly trained on hyperparameter tuning datasets, its broad training may allow it to apply general and learned patterns to this domain, which can be particularly effective when combined with specific past observations provided in the prompt.

[P3] Mechanism

We appreciate this question. It is essential to note that the generalization mechanism of ICL is an area of ongoing research and is not fully understood yet. While a detailed exploration of this mechanism is beyond the scope of our current work, we can shed light on some recent advances and hypotheses in the field:

• Pattern recognition through ICL. We leverage the LLM’s ability to recognize patterns in in-context examples and generalize its predictions. Several hypotheses have been put forward to explain ICL: [1] explains it through the lens of implicit Bayesian inference. According to this hypothesis, an LLM, during its pretraining phase, learns to perform Bayesian inference over latent concepts. At the time of inference, the model infers shared latent structures across examples, enabling it to generalize effectively. [2] presents an alternative view, conceptualizing ICL as akin to kernel regression performed on in-context examples. [3] equates in-context learning in attention-based architectures to gradient-based optimization in simplified settings.
• ICL applied to optimization. In our framework, this ability to recognize generalizable patterns from very few examples is used to propose high-potential candidate points (based on discerned patterns of successful solutions) and perform surrogate evaluation of proposed points (by identifying similarities to existing observations). By iterating between processes, we effectively leverage ICL to explore and exploit the loss landscape.
• Parsing/interpreting numbers. Regarding parsing numbers, prior studies [4-5] have demonstrated LLM’s capabilities in reasoning with numerical data. Specifically relevant to our work, recent research [6-7] has explored the use of ICL for numerical prediction tasks in settings such as tabular regression and classification. These studies support our findings that LLMs can indeed discern and leverage numerical patterns effectively, even in few-shot learning scenarios.

[1] Xie, S.M., Raghunathan, A., Liang, P. and Ma, T., 2021, October. An Explanation of In-context Learning as Implicit Bayesian Inference. In International Conference on Learning Representations.

[2] Han, C., Wang, Z., Zhao, H. and Ji, H., 2023. In-Context Learning of Large Language Models Explained as Kernel Regression. arXiv preprint arXiv:2305.12766.

[3] Von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A. and Vladymyrov, M., 2023, July. Transformers learn in-context by gradient descent. In International Conference on Machine Learning (pp. 35151-35174). PMLR.

[4] Imani, S., Du, L. and Shrivastava, H., 2023. Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXiv:2303.05398.

[5] Wallace, E., Wang, Y., Li, S., Singh, S. and Gardner, M., 2019, November. Do NLP Models Know Numbers? Probing Numeracy in Embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 5307-5315).

[6] Hegselmann, S., Buendia, A., Lang, H., Agrawal, M., Jiang, X. and Sontag, D., 2023, April. Tabllm: Few-shot classification of tabular data with large language models. In International Conference on Artificial Intelligence and Statistics (pp. 5549-5581). PMLR.

[7] Dinh, T., Zeng, Y., Zhang, R., Lin, Z., Gira, M., Rajput, S., Sohn, J.Y., Papailiopoulos, D. and Lee, K., 2022. Lift: Language-interfaced fine-tuning for non-language machine learning tasks. Advances in Neural Information Processing Systems, 35, pp.11763-11784.

---

We thank the reviewer for your help in improving our work. Please let us know if our latest changes have addressed your concerns, and if there is anything else you would like to see.

We wish to thank all reviewers for their insightful reviews and engagement in the rebuttal process.

We are providing a brief update on our latest revisions. Specifically, we have supplemented the previously partial results with complete findings from the prompt design ablation study (App C2) and HPOBench (App E4).

Kind regards,

The Authors of #8133