Remark:

• (W1) means the 1st weakness, (Q1) means the 1st question.
• Due to character limits, we kindly invite you to refer to our responses for other reviewers' similar questions. We will use ref. "Response Title"@Reviewer. Thank you for your understanding!

Genetic algorithm baseline

(W1, Q1) In particular, the paper is missing a genetic algorithm baseline/ablation with random recombination, which should have been provided to demonstrate that the LLM-designed offspring are superior to a random genetic algorithm.

To directly address your concerns, we have now added a genetic algorithm (GA) baseline that performs random crossover and mutation guided by the fitness scores, with the same population size and number of generations as CAKE. This baseline allows us to isolate the impact of replacing random recombination with LLM-guided recombination. We also would like to note that our original ablation already included a Random Sampling baseline, which randomly combines kernels via addition and multiplication without any evolutionary process.

As shown in the table below, Random Sampling performs the worst (average rank: 6.80), confirming that blind recombination is ineffective. The GA baseline improves over Random Sampling significantly (average rank: 2.70), demonstrating the value of evolutionary selection. Most importantly, CAKE + BAKER achieves the best average rank (1.04), substantially surpassing GA. This suggests that the LLM-enhanced recombination in CAKE goes far beyond random recombination by leveraging semantic understanding and domain knowledge that conventional GA operators lack.

To answer why the LLM can generate better offsprings, we analyzed the evolution of the fitness distributions after successive edits by LLM, Random, and GA. We found that CAKE achieves faster and more consistent improvement in fitness. After just 1 edit, CAKE's distribution shifts significantly to the right, indicating rapid progress. After 10 edits, we can observe that CAKE's distribution is sharply peaked at very high fitness values, demonstrating consistent generation of high-quality offspring. In contrast, Random and GA show slower convergence and broader distributions, suggesting less effective exploration of the solution space. These results provide strong evidence that using an LLM results in better offsprings than random recombination or GA.

Try other LLMs

(W2 & Q2) Furthermore, what happens if a different LLM, such as Claude, Llama, Qwen, Deepseek, is used?

We have conducted additional experiments using 3 open-source LLMs:qwen-2.5-7b, deepseek-r1-distill-qwen-7b, and llama-3.1-8b, and 2 commercial ones: gemini-2.0-flash and claude-3.5-haiku. Results show that CAKE consistently works well across different LLMs, with performance generally improving with model capability. Notably, we observe a performance boost with the open-source deepseek-r1-distill-qwen-7b, suggesting that reasoning-based models can further enhance results. Overall, these results indicate that as LLMs become more advanced, our method naturally benefits from their improved capabilities. We hope this comprehensive evaluation of diverse LLMs strengthens our empirical claims and provides more baselines for future research.

Does LLM reasoning matter?

(W3) Does the reasoning provided by the LLM even matter, given that the LLM responses (as seen in the appendix) are generic and not taking the actual BO experiment/trial results into context?

Yes, LLM reasoning does matter, both for performance and interpretability, and it is meaningfully grounded in the actual BO trial results.

We would like to first clarify that the reasoning is not generic. Our prompt design explicitly incorporates the full history of BO trials—including observations, fitness scores, and prior kernel performance—to ensure the LLM’s reasoning is grounded in actual experimental outcomes. As shown in Appendix C.5, the LLM generates context-specific justifications, such as noting that "the observations can be described using a combination of three distinct patterns..." or identifying increasing uncertainty in specific input regions. These are not generic statements but direct responses to the observed BO trajectory. To further clarify this point, we will include additional representative examples of the LLM’s reasoning in the camera-ready version.

(Q3) What happens if the reasoning prompt is removed; would there even be an adverse effect on CAKE's performance?

Removing the reasoning prompt does harm performance, as demonstrated in our ablation study below. The variant without reasoning, where the LLM outputs only a kernel expression, exhibits a drop in performance compared to the full prompt. This proves that the reasoning step is not merely decorative: it actively improves kernel generation. We hypothesize this is because articulating reasoning forces the LLM into a self-reflective process, enabling it to evaluate, refine, and validate its proposals against past results (i.e., functioning as an internal critique mechanism)

Originality of CAKE

(W4 & Q4) Ultimately the paper does not provide convincing evidence why CAKE should outperform a standard genetic algorithm, while at the same time CAKE is not original compared to the cited works on LLM-genetic algorithms.

While we acknowledge that LLM-genetic algorithms has been explored in other domains, CAKE is, to our knowledge, the first to apply LLM for automating kernel search in BO. Unlike generic code or string-based evolution, we believe kernel search introduces unique challenges, as every kernel proposal must be (1) mathematically valid, (2) preserve compositional semantics, and (3) meaningfully explain observed data patterns. Hence, a key novelty of CAKE lies in our use of a generalized kernel grammar as a domain-specific language (DSL) that formally defines valid kernel expressions and operators. By embedding this grammar into the prompt, we enable the LLM to perform structured search while ensuring all outputs are semantically meaningful and mathematically valid. We believe this differentiates CAKE from both traditional GAs and existing LLM-genetic algorithms in two key ways:

1. Unlike standard GA operators, which combine kernels through random crossover and mutation, CAKE leverages the LLM’s learned understanding of function properties and data patterns to reason about plausible and effective kernel structures.
2. By combining the LLM’s generative capabilities with the formal grammar, CAKE can not only produce mathematically valid kernel expressions but also natural language descriptions for these kernels without human expert intervention.

We believe this combination of domain-specific language, LLM-based reasoning, and interpretability represents a significant step forward beyond existing LLM-genetic frameworks, particularly in the context of kernel design for BO.

Remark:

• (W1) means the 1st weakness, (Q1) means the 1st question.
• Due to character limits, we kindly invite you to refer to our responses for other reviewers' similar questions. We will use ref. "Response Title"@Reviewer. Thank you for your understanding!

Query and surrogate fitting cost

(W1) A minor weakness ... to objective evaluation.

We agree that CAKE relies on a sufficiently capable LLM and that each LLM query introduces additional computational cost, as reflected in Table 5. However, as the reviewer rightly pointed out, in most practical BO scenarios, the time spent on surrogate modeling (including LLM queries and GP population updates) is negligible compared to the cost of evaluating the black-box function. For example, training a machine learning model or running a physics-based simulation can take minutes to hours, while the average wall-clock time per CAKE iteration (including LLM inference and GP fitting) is on the order of seconds. Moreover, since the training of these GPs is independent and parallelizable, the kernel population can be trained in parallel across multiple cores or GPUs to significantly reduce the wall-clock time for surrogate model updates.

Our experiments also showed that CAKE achieves significantly better sample efficiency than baseline methods (see Figures 4-6). It can find high-performing solutions in fewer iterations (ref. "Early stage analysis"@2phH), which directly translates to reduced total optimization cost in expensive black-box settings. In this context, the slight increase in per-iteration time is a worthwhile trade-off for drastically reducing the number of costly function evaluations.

Finally, we note that LLM inference can depend on external factors such as internet connectivity and API latency, which can be mitigated through strategies like batching multiple queries or deploying smaller open-source models locally (ref. "Try other LLMs"@esv8). This makes the overhead of using an LLM acceptable, especially when weighed against the benefits.

Prompt alternatives

(Q1) Did the authors consider...helpful and strengthen the paper.

We did not give the LLM direct numerical access to the raw data or kernel hyperparameters because we did not equip the LLM with any external tools or code interpreters (e.g., through MCP). Instead, we provide structured summaries of the optimization state through the designed prompts, where we update the prompt with the collected observations as well as the fitness scores from the previous trials.

We actually experimented with having the LLM suggest the next query point directly, but we observed the LLM struggled with precise numerical sampling in continuous, high-dimensional spaces, especially without external tools. Similarly, we tested a "population update" approach where the LLM proposes a new list of kernels at once. However, this led to unstable or overly aggressive changes. In contrast, we found the evolutionary approach, using LLM-guided crossover and mutation, provides a more controlled and stable kernel evolution.

These experiments led to the current prompt design where the LLM focuses on symbolic reasoning and hypothesis generation, while the conventional BO components handle the numerical optimization parts. We will include a discussion of these alternatives in the camera-ready version to clarify the design rationale.

Data contamination

(Q2) One concern using LLM's...issue for their results?

While we acknowledge that LLMs are trained on vast corpora that may include scientific literature or code related to common optimization benchmarks, we believe that data contamination is unlikely to meaningfully influence our results, for several key reasons.

First, while the LLM may have encountered general descriptions of synthetic functions or high-level details about HPOBench during pre-training, it is highly improbable that the specific kernel expressions or optimization trajectories used in our experiments exist in any public dataset or text. These are dynamically generated based on noisy observations and fitness feedback unique to each BO run. Thus, even if the LLM has prior knowledge of benchmark functions, it cannot memorize the exact solutions or search paths we evaluate.

Second, a supporting evidence is our photonic chip design experiment (see Section 6.3), which is based on proprietary simulations developed in our own work. The data were generated using our own simulator and have never been published or released. This makes it effectively impossible for this task to be present in the LLM’s training data. Yet, CAKE performs strongly on this task, further confirming that its success relies on reasoning over real-time feedback, not data leakage.

Finally, our prompt ablation study (ref. "Does LLM reasoning matter?"@esv8) directly demonstrates that removing the observed BO history (i.e., observations, scores, and past kernels) from the prompt leads to a significant drop in performance. This shows that the LLM critically depends on in-context learning—it cannot achieve strong results using prior knowledge alone.

In summary, while pre-training may provide foundational knowledge, the specific inputs, outputs, and evolutionary paths in our experiments are unique. We will include a brief discussion of these points in the camera-ready version.

Try other acquisition functions

(Q3) Do the results depend ... in performance differences?

We thank the reviewer for raising this question. Our results demonstrate that CAKE is robust across different acquisition functions, including EI, UCB, and Thompson sampling (TS). While each acquisition function embodies a different exploration-exploitation trade-off (i.e., TS exhibits slightly higher standard errors due to its stochastic nature while CAKE-UCB occasionally underperforms or outperforms EI due to its optimism bias), CAKE consistently achieves comparable performance across all variants. As shown in table below, the performance of CAKE-UCB and CAKE-TS remains close to that of CAKE-EI across a diverse set of benchmark functions. To rigorously assess statistical equivalence, we conducted a one-sided paired Wilcoxon signed-rank test (at the 5% significance level) comparing CAKE-EI against CAKE-UCB and CAKE-TS across multiple random seeds. The results indicate that, on the majority of functions, differences are not statistically significant. Overall, these results confirm that CAKE's effectiveness is not sensitive to the specific choice of acquisition function.

Note: Values that are not significantly different from the lowest average regret for each function are bolded.

BAKER justification

(Q4) The BAKER BIC weighting ... function with BIC?

We designed BAKER to balance model fit (via BIC) and the potential to yield high-utility query points (via acquisition value). As shown in our ablation study (see Section 5.4), relying solely on BIC may lead to kernels that fit the data well but propose low utility query points. On the other hand, using only acquisition value can favor optimistic kernels, those that suggest high improvement but are poorly calibrated, leading to repeated sampling in suboptimal regions. By combining both, BAKER balances the trade-off: it promotes kernels that are both well-fitted to the data and has the potential to yield high-utility query points.

Thank you for the replies. I continue suggesting the paper should be accepted and believe the paper will be made stronger by adding some discussion of these points to the text.

Remark:

• (W1) means the 1st weakness, (Q1) means the 1st question.
• Due to character limits, we kindly invite you to refer to our responses for other reviewers' similar questions. We will use ref. "Response Title"@Reviewer. Thank you for your understanding!

Hyperparameter sensitivity analysis

(Q1) Could the authors provide a ... and population size)?

We have added a sensitivity analysis that shows the average fitness of the population for different settings. We found that:

• Increasing the number of crossovers ($n_c$) enhances exploration initially, but beyond $n_c = 5$ , the gains in average fitness diminish while the computational cost (in terms of LLM API calls) increases significantly. Thus, $n_c = 5$ offers an optimal trade-off between effective exploration and efficiency.

• Increasing the mutation probability ($p_m$) enhances population diversity and helps avoid premature convergence, but when $p_m > 0.7$ , excessive mutations disrupt promising kernel structures and destabilize the evolutionary process. Thus, $p_m = 0.7$ provides a balanced level of mutation that supports refinement without sacrificing stability.

• Increasing the population size ($n_p$) improves diversity and allows more exploration, but for $n_p > 10$, the marginal improvement in average fitness diminishes relative to the increased computational burden from repeated GP model fitting. Thus, $n_p = 10$ ensures sufficient diversity while maintaining computational tractability.

Try other acquisition functions

(Q2) Is the proposed method ... Thompson sampling, LogEI, etc)

We found that CAKE is robust across different acquisition functions (ref. "Try other acquisition functions"@8KrH), including EI, UCB, and Thompson sampling (TS). While each acquisition function embodies a different exploration-exploitation trade-off (i.e., TS exhibits slightly higher standard errors due to its stochastic nature while CAKE-UCB occasionally underperforms or outperforms EI due to its optimism bias), CAKE consistently achieves comparable performance across all variants. To rigorously assess the statistical equivalence, we conducted a one-sided paired Wilcoxon signed-rank test (at 5% significance level). We found in most cases, the differences are not statistically significant.

Kernel pool

(Q3) How does the size and ... optimization performance?

We initialize the kernel pool with the standard set of 6 base kernels: {SE, LIN, PER, RQ, M3, M5}. These kernels were selected because they encode a diverse set of functional priors, enabling CAKE to start with diverse representation of potential function structures. It is worth noting that CAKE is designed to be flexible with respect to the initial kernel set. When domain knowledge on specific structures are available, we can customize the initial kernel pool to include relevant kernels for a more informed and efficient search. Otherwise, we use a standard diverse set to avoid biasing the search towards any specific assumptions.

For the subsequent generations, the pool size is controlled by the population size hyperparameter $n_p$. Larger $n_p$ increases diversity but imposes higher computational overhead (due to GP fitting), while smaller $n_p$ risks premature convergence. Empirically, we found that $n_p = 10$ offers a favorable trade-off: it supports sufficient diversity for effective exploration of the kernel space while keeping computational costs manageable.

BAKER formulation

(Q4) Regarding BAKER ... for this design choice.

Our initial motivation for incorporating $\alpha$ into the kernel selection process was to assess the practical utility of each kernel in guiding the search toward high-value regions of the input space. While metrics like BIC measure how well a kernel fits the observed data, they do not directly reflect the kernel's ability to propose high-utility query points, which is ultimately the goal in BO. We acknowledge that raw EI values across different kernels might not be directly comparable. This concern motivated us to conduct an experiment using a normalized variant of BAKER (denoted as BAKER-N). Encouragingly, our experiment shows that BAKER-N improves the performance of our original BAKER.

We believe this is because, after normalization, both $w_k$ and $\alpha$ lie in comparable ranges, leading to a more balanced trade-off between model fit and utility. Given these results, we will update the paper to include BAKER-N in the camera-ready version to reflect this improved formulation. We are grateful for the reviewer's constructive feedback which has helped us improve the performance of our method.

Thank you for your response. The reviewer’s rebuttal has addressed my concerns. I’m glad that our discussion seems to have had a positive impact on the paper, particularly in the BAKER-N formulation. Assuming that these changes will be reflected in the camera-ready version, I am raising my score.

Dear Reviewer fLo9, given the authors' response, please raise any remaining questions and/or concerns in a timely fashion so the authors have a chance to reply. Thanks!

Remark:

• (W1) means the 1st weakness, (Q1) means the 1st question.
• Due to character limits, we kindly invite you to refer to our responses for other reviewers' similar questions. We will use ref. "Response Title"@Reviewer. Thank you for your understanding!

Kernel space

(W1) I think the kernel ... to explore too.

We agree that incorporating more base kernels and operators can enrich the search space. However, in our work, we focus on addition and multiplication as initial proof-of-concept operators as they already yield a rich, expressive kernel space—e.g., enabling polynomial kernels that capture complex non-linear patterns and multi-dimensional kernels that model interactions among input features. Moreover, our method supports a generalized kernel grammar, allowing easy integration of other valid operators (e.g., convolution, composition). We consider this an important and promising direction for future work. We also appreciate your observation that a richer kernel space inherently increases the complexity of exploration. Balancing expressiveness and tractability in kernel search is indeed a challenging problem and an interesting avenue for future research.

Prompt design

(W2) Another point is ... in-context examples?

In designing our prompts, we intentionally adopted a simple yet principled approach to ensure reproducibility and generalization across diverse tasks. Enriching the prompts with more in-context examples (such as successful kernels from similar tasks or synthetic patterns) could potentially improve performance and is a promising direction for future work, particularly in meta-learning or multi-task settings. However, in the current few-shot, single-task BO setting, our current prompt design strikes an effective balance between simplicity and performance (ref. "Does LLM reasoning matter?"@esv8), avoiding the risks of prompt overload or distraction from the task in hand.

LLM-BO Baseline

(W3) For the baselines ... such as LLAMBO, BOPRO, ...

We clarify that methods like LLAMBO and BOPRO operate in a fundamentally different paradigm from CAKE. These methods use LLMs to augment the BO pipeline (e.g., warm-starting or generating candidates) without learning or evolving kernel structures. In contrast, CAKE focuses on improving the GP surrogate by evolving its kernel via an LLM-guided process. Our goal is to enhance the kernel itself, not to replace BO components with LLM heuristics. Thus, comparing CAKE to LLM-augmented BO methods would conflate distinct objectives. For a fair evaluation, we compare CAKE against GP-based BO baselines that share the same underlying framework and goals.

Early stage analysis

(W4) Lines 216-218: you ... stages more carefully.

To rigorously support our claim that CAKE excels in the early stages of optimization, we conducted a quantitative analysis based on normalized improvement:

where $f_0$ is the initial performance, $f_t$ is the best performance at trial $t$, and $f^*$ is the final performance at 100% budget. This measures how much of the total progress CAKE achieves early on. In the HPOBench tasks, CAKE achieves 67.5% of its total improvement within just 25% of the budget, on average. By 50%, it reaches over 83% of its final improvement, and by 75%, it is nearly converged.

Computational cost

(W5) The biggest weakness ... a bottleneck in practice.

First, we note that the computational overhead of maintaining a population of GPs is not unique to CAKE. Compositional and ensemble-based methods also require fitting and managing multiple GP models per iteration.

Second, while CAKE evaluates multiple kernels at each iteration, the training of these GPs are independent and parallelizable. In practice, the kernel population can be trained in parallel across multiple cores or GPUs to significantly reduce the wall-clock time.

Most importantly, as highlighted in Appendix C.2, in most real-world black-box optimization tasks, the cost of evaluating the objective function far exceeds the cost of surrogate model fitting. For example, running a physics-based simulation can take hours, while fitting a population of GPs takes seconds. In such settings, sample efficiency, or reducing the number of function evaluations, is far more critical. We note that our method excels in this regard, as our experiments showed that it could converge to high-performing solutions in fewer iterations.

Try other LLMs

(W6) Finally, it would ... for reproducibility.

We have expanded our experiments to include 3 open-source LLMs (ref. "Try other LLMs"@esv8). Results show that open-source models can achieve competitive performance, with deepseek-r1-distill-qwen-7b even outperforming several commercial models. This shows that strong results are also achievable with open-source models.

Why BIC?

(Q1) Why use BIC ... good the fit is.

Both the MLL and BIC are valid metrics for model evaluation. We deliberately choose BIC because it imposes an explicit penalty on model complexity proportional to the number of hyperparameters, which provides a more direct control of overfitting in our context. While MLL also includes an implicit complexity penalty via the log determinant term, it can sometimes favor overly complex kernels especially when the hyperparameter optimization converges to a local minima. In contrast, BIC’s penalty increases with sample size and hyperparameter count, which discourages unnecessary complexity as more data becomes available.

BoTorch multi-start

(Q1) ... Also you mention that ... sensitive to that as well.

We did employ BoTorch's multi-start optimization procedure to mitigate convergence to poor local optima. We found this improves robustness in estimating the hyperparameters, which in turn helps stabilize the fitness evaluation.

GP retraining

(Q2) I am not sure ... the highest probability?

We do not replay past decisions or evaluate counterfactual trajectories (e.g., "which kernel would have found the best point earlier?"). Instead, we assess each kernel on its current ability to model the entire observed data. After each new observation is collected during the BO, we retrain all the GP models in the current kernel population using the updated dataset. The fitness of each kernel is then computed according to the BIC, which is later used to select the best kernels for the next generation and to guide the LLM in proposing new ones.

Adding domain knowledge

(Q3) Have you tried ... and fitness metrics?

We designed our prompts to be generic to avoid manual prompt engineering and ensure applicability across diverse tasks without requiring domain expertise. However, we agree that incorporating domain knowledge, when available, has strong potential to improve performance. For example, we see this potential in our photonic chip design experiment (see Section 5.3), where physical meaning of the chip's parameters and objectives are well-understood. However, this involves careful prompt engineering and iterative refinement to determine the optimal level/type of domain information to include. Since the photonic chip design experiments rely on computationally intensive simulations (i.e., each evaluation taking hours to complete), we were unable to finish the experiments within the rebuttal timeline. Nevertheless, we appreciate the suggestion and plan to include this in an extended version of the paper.

Shaded regions in figures

(Q4) What is the shaded ... standard error of the mean?

The shaded regions in Figures 4 to 6 represent the standard error of the mean computed over independent runs/trials. We will add a remark in the figure captions to clarify this.

Significance tests

(Q4) ... It would be best ... methods significantly different.

We have performed a two-sided Wilcoxon signed-rank test to evaluate the statistical significance of performance differences between CAKE+BAKER and the top baselines. The test assesses whether the median performance difference across tasks is non-zero, with the null hypothesis being no difference between methods. We focus on ABO as the primary comparison because it is the second-best performing method overall on HPOBench. We test significance at key budget levels: 10%, 25%, 50%, 75%, and 100% of the total evaluation budget, to identify when CAKE+BAKER begins to significantly outperform ABO. As shown in the table below, CAKE+BAKER significantly outperforms ABO as early as 10% of the budget across all models, with $p < 0.05$ or better. Notably, for LR and XGB, the improvement is highly significant ($p < 0.001$) even at this early stage. This early and consistent separation demonstrates that CAKE+BAKER identifies high-quality configurations more efficiently and maintains a statistically significant advantage throughout the optimization process.

* denotes significant difference ($p < 0.05$),
** denotes very significant difference ($p < 0.01$),
*** denotes highly significant difference ($p < 0.001$).

Dear Reviewer 2phH, given the authors' response, please raise any remaining questions and/or concerns in a timely fashion so the authors have a chance to reply. Thanks!

Remark:

• (W1) means the 1st weakness, (Q1) means the 1st question.
• Due to character limits, we kindly invite you to refer to our responses for other reviewers' similar questions. We will use ref. "Response Title"@Reviewer. Thank you for your understanding!

Comparison with KITT [3]

(W1) The use of LLMs for...kernel selection in Gaussian Processes.

(Q1) Authors do not discuss prior...Can we empirically compare?

We summarize the main differences between CAKE and KITT as follows:

• Different objectives: CAKE is designed for adaptive kernel learning in BO, which requires balancing exploration and exploitation. Meanwhile, KITT is developed for regression, which fouced on fitting and prediction accuracy.
• Usability: KITT uses a fixed kernel vocabulary and requires fine-tuning. CAKE leverages LLMs as plug-and-play tools, usable without fine-tuning and accessible to non-experts.

We did consider including KITT during the rebuttal phase, but ultimately could not due to practical and methodological constraints. KITT requires large-scale labeled data (e.g., 200K samples) and significant computational resources (e.g., hours on Tesla V100 GPUs). More critically, it is tailored to regression and has not been adapted or evaluated for BO. Adapting it to BO would require substantial retraining on BO tasks and optimization trajectories. We instead focused on established, directly applicable BO baselines to ensure a fair and reproducible comparison.

Comparison with FKL [4] and KFO [5]

(W2) This paper focuses primarily on...scalability, and sample efficiency.

We first clarify the fundamental differences between CAKE and functional kernel methods (FKL/KFO):

• CAKE performs discrete search over parametric kernel compositions, which preserves symbolic kernel structures for interpretability.
• FKL/KFO operate in continuous, non-parametric function spaces, which sacrifice interpretability for flexibility.

Hence, their kernel search spaces and optimization methods differ significantly from CAKE, making direct empirical comparison challenging and somewhat orthogonal in objectives.

Nonetheless, we appreciate the reviewer's suggestion and conducted additional experiments comparing CAKE with KFO and both FKL variants on UCI datasets. Using the same benchmark and negative log-likelihood as in [5], the results show that CAKE is highly competitive with these state-of-the-art functional kernel methods and outperforms KFO on several datasets.

Try other LLMs

(W3) Authors have only experimented...the empirical claims made.

We have expanded our experiments to include 3 open-source LLMs (ref. "Try other LLMs"@esv8). Results show that open-source models can achieve competitive performance, with deepseek-r1-distill-qwen-7b even outperforming several commercial models. This demonstrates that our approach does not rely on proprietary APIs to work well.

Any theoretical analysis?

(W4) This paper does not provide...create this bottleneck.

We have actually conducted some analysis through the Bayesian regret framework. By assuming that the underlying objective function is a sample path of a zero-mean GP or a function in the reproducing kernel Hilbert space (RKHS) induced by the obtained kernel function, we can leverage existing results to show that the cumulative Bayesian regret can be bounded as the kernel is refined over iterations using the LLM. However, due to the adaptive nature and complex form of the kernel function across iterations, we are still unable to quantify the convergence speed, which our experiments nevertheless show to be significantly faster than that of the state-of-the-art methods. We can include our current analysis results in the Appendix of the updated draft.

Nevertheless, our primary goal is to highlight the novel kernel search method and its superior performance in BO. To this end, we placed a greater emphasis on conducting extensive empirical analyses and ablation studies to rigorously evaluate the contributions of various components, including prompt design and parameter choices. In the Appendix, we present an empirical study of the fitness distribution across iterations, which provides a practical justification for why the LLM can effectively serve as a genetic operator that evolves kernels toward higher fitness. Additionally, our case study on the expressivity and interpretability of the generated kernels offers qualitative evidence of our method's effectiveness.

Robustness with different prompts or LLM variants

(W5) The performance of CAKE...LLM models/variants.

We have added an ablation study (ref. "Does LLM reasoning matter?"@esv8) to assess the contribution of each prompt component:

• Removing context harms kernel adaptation, showing the LLM relies on observed data to generate meaningful structures.
• Omitting task instructions leads to hallucinated or invalid kernels, degrading performance.
• Removing the reasoning step slightly reduces performance, suggesting explanations improve self-consistency and generation quality

These results demonstrate that our prompt design is purposeful, with each element playing a critical role in ensuring correctness and coherence. While CAKE involves some prompt engineering, this is standard in LLM-based methods, and our ablation clarifies which components matter and why.

Regarding the robustness with different LLM variants, CAKE actually performs better with stronger LLMs (ref. "Try other LLMs"@esv8), indicating it naturally benefits from advances in LLM capabilities and remains forward-compatible with future models.

Kernel hyperparameters

(Q2) When LLMs genetically evolve...hyperparameters of the individual kernels?

We did not expect the LLM to pinpoint exact hyperparameter values, as we did not equip it with any external tools to do so. Its role is to evolve the symbolic kernel expression. Once a kernel expression is proposed, its hyperparameters are optimized via standard maximum likelihood estimation using GPyTorch, and fitness is then evaluated under the properly tuned hyperparameters.

Explanation to Eq. (1)

(Q3) The Baker formulation mentioned...will be sub-optimal.

It is indeed possible for a poorly fitting kernel to propose an overly optimistic expected improvement due to overestimated uncertainty or a misspecified mean function. If the kernel selection were based solely on acquisition value, such overconfident but poorly calibrated models could be repeatedly favored, thus leading to suboptimal querying behavior. To mitigate this risk, BAKER computes a weighted score by taking the product of each kernel's weight and its corresponding acquisition value. By maximizing this product across the kernel population, BAKER inherently balances the model fit (as measured by the weight) and query utility (as measured by the acquisition value). Consequently, even if a poorly fitting kernel proposes an high acquisition value, its low weight acts as a natural regularizer to suppress its influence. In this way, kernels with both high weights and high acquisition values are more likely to be selected.

Explanation to Fig.5

(Q4) In Figure 5, SE outperforms...certain kernels or tasks?

The fact that the fixed SE kernel sometimes outperforms the BIC-based adaptive method highlights a key limitation: selecting kernels solely based on model fit does not ensure better optimization performance. This motivates our design of BAKER, which jointly considers fit (via BIC) and utility (via acquisition value). As shown in our ablation study (Section 5.4), using either criterion alone leads to suboptimal results, while their combination in BAKER yields more robust performance across tasks.

Can adaptive mutation probability help?

(Q5) Would an adaptive or decaying...as optimisation progress?

An adaptive or decaying mutation probability could enhance performance by promoting exploration early and exploitation later. Our sensitivity analysis (ref. "Hyperparameter sensitivity analysis"@fLo9) shows that while higher $p_m$ increases diversity and prevents premature convergence, overly high values can disrupt promising kernels. This trade-off suggests a decaying schedule may offer better control. In this work, we use a fixed $p_m = 0.7$ for the sake of reproducibility as it provides a robust and balanced performance across tasks. Although we did not implement adaptive $p_m$ due to time constraints, we see it as a promising direction for future improvements.

Minor Comments

• Typo: In Line 242: "In robot pushing task.."
• Typo: In Line 308: "..Observations are scarce."

Thanks! We will correct them in the camera-ready version.

Dear Reviewer RoKJ, given the authors' response, please raise any remaining questions and/or concerns in a timely fashion so the authors have a chance to reply. Thanks!