Context-Aware Kernel Search for Bayesian Optimization with Large Language Models

Thank you for your time and effort in reviewing our work. Important changes have been highlighted with blue color in the manuscript. We respond to the questions and comments point-by-point as follows.

The presentation of the algorithm places too much emphasis on kernel search. However, BO is just one application of kernel design in Machine Learning. Therefore, I recommend the authors refine the presentation of this BO-specific method.

Thank you for your feedback. We would like to clarify that our algorithm is not exclusively an application of BO. Instead, our algorithm serves as a general framework for kernel design that can automate surrogate modeling (with GP) in BO. We also emphasized this point in the conclusion part, where we hope that this work can represent a solid step towards a fully automated BO system that can be used by non-experts on arbitrary objectives.

The theoretical analysis is too simplistic and could apply to any adaptation method of kernel design, such as symbolic regression. LLM-related analysis may better describe the performance.

Thank you for your insightful feedback regarding our theoretical analysis. We would like to first mention that our current analysis focuses on the regret bounds and information gain, which are the commonly-used tools for analyzing the performance of BO algorithms. However, we appreciate your suggestion to include an LLM-related analysis. In response, we have added a new section in the Appendix, where we provide empirical evidence that LLMs are indeed familiar with kernel design and can function effectively as good crossover and mutation operators. In Appendix F.1, we have conducted an additional experiment to analyze the fitness distributions of an initial population of kernels before and after applying crossover and mutation operations. Our results show that the initial population has a wide distribution with a peak at lower fitness values; after crossover, the distribution shifts toward higher fitness values, indicating the LLM's capacity to effectively combine kernels and enhance overall fitness. Furthermore, we found that applying mutation further shifts the distribution toward even higher fitness values, suggesting that refining the kernels yields improved outcomes. In Appendix F.2, we have included additional analysis reasoning the outputs from the LLM during the crossover and mutation steps. Based on the outputs (see Figure 9), we found that the LLM demonstrates a strong understanding of the relationships between different kernels and their use cases to address specific data characteristics.

I am curious whether the setting of weights in photonic chip design, where the last three objectives are much more important than the first two, is a commonly recognized practice or carefully designed. This is particularly relevant as CAKES performs outstandingly well on the fourth objective but only fairly on the other objectives.

Thank you for the question. The last three objectives are indeed much more important than the other ones from a practical engineering (e.g., autonomous driving or military laser weapon) point of view. This is because we want the photonic chip to be equipped with large lasing area $(f_3)$ to produce a large light field coverage, higher power $(f_4)$ to transmit signal/beam faster and farther, and have a small divergence angle $(f_5)$ to travel longer distance and focus better. In this case, $f_1$ and $f_2$ are just auxiliary objectives. Since CAKES performed particularly well on the fourth objective, the designed photonic chip is well-suited for LiDAR applications (e.g., for autonomous driving), which demand an outstanding laser power $(f_4)$.

Thank you for your quick reply and questions. We address your questions one by one as follows:

Can the author demonstrate any well-performing kernel using this method? From the information in Figure 9, LLMs recommend a simple combination of base kernels.

Thank you for mentioning this point. First, we would like to clarify that Figure 9 only displays a sample output from a single iteration, particularly in the early stage, of the optimization process. The main purpose of Figure 9 is to illustrate that the LLM has a strong understanding of the relationships between different kernels and their capacity to address specific data characteristics. While the recommended kernel may appear to be a simple combination of base kernels, our method can adaptively refine the kernel and generate more complex kernels in the subsequent iterations. Unfortunately, due to space constraints, we were unable to present all outputs from each iteration across all tasks. Another point that we would like mention is that, there is no single kernel that performs well universally across all tasks. The optimal kernel in BO is problem-specific and may change as new data is collected. This is a key motivation behind our work, as fixed kernels or naive selection strategies often fail to deliver good results across different tasks (see Section 6 and Appendix G). In light of this, as part of our contribution, we proposed a method that leverages the extensive knowledge and learning power of LLMs to adaptively propose kernels, such that the kernels are tailored to the specific characteristics of the data at each iteration.

Can we use symbolic regression to find optimal kernels?

Thanks for bringing up the idea of symbolic regression to find the optimal kernels. While we acknowledge that we only have a basic understanding of it, we recognize that there are significant differences between symbolic regression and our method. Symbolic regression aims to identify the optimal mathematical expression for a fitting function that minimizes fitting error. This process involves both determining the mathematical operations and estimating parameters, which has been proven to be NP-hard [1]. We also afraid that the computational time required for symbolic regression may be prohibitive in both modest and high-dimensional settings. In contrast, our focus is on efficiently identifying the best reproducing kernel Hilbert space (RKHS) for a surrogate function in BO with the assistance of LLMs. While regression and data fitting are not our primary objectives, we prioritize a more robust uncertainty quantification associated with the determined RKHS. This uncertainty quantification is crucial to effectively guide the BO process by balancing exploration and exploitation effectively. Besides, we maintain a practical mindset and aim to balance the accuracy, computation time, and cost. Another notable difference is that we need to make sure the kernel found by our method is a valid kernel, while the search under the symbolic regression has no such constraint on the math operations. Nevertheless, we agree that integrating the idea of symbolic regression (especially the genetic programming) with LLM prompt design may facilitate our kernel search when the kernel operations are not limited to addition and multiplication. Thank you once again for sharing this interesting idea with us; we will certainly consider this new line of research in the future.

We hope this addresses your questions and enhances your impression of our work. We remain open to any further questions or suggestions. If your concerns are resolved, we would greatly appreciate if you could reconsider your score.

References

[1] M. Virgolin and S. P. Pissis, "Symbolic regression is NP-hard," Transactions on Machine Learning Research, 2022.

How do the authors define the values of number of crossovers $n_c$, probability of mutation $p_m$, and population size $n_p$ (line 163-165 in Algorithm 1) in final implementation?

Thank you for your question regarding the parameters used in our final implementation. In our experiments, we set the number of crossovers $n_c = 5$, the probability of mutation $p_m = 0.7$, and the population size $n_p = 10$. We found that these values provide a good balance between maintaining a diverse pool of kernels while ensuring manageable computational complexity for generating and evaluating the kernels. We have added these details to Appendix D.1, along with other additional information regarding the setup of CAKES.

I notice the authors set the maximum number of function evaluation $T$ and number of replications using different random seeds quite small for their experiments. For the benchmark functions (section 6.1), $T = 10d \leq 50$ since $\max d = 5$ for the chosen test functions. For the hyperparameter tuning tasks (section 6.2), $T = 25$. Usually the value of $T$ will be larger (e.g. 50, 100) in BO experiments. Similarly the number of random seeds for each experiment (10 for section 6.1, 5 for section 6.2 and not mentioned for section 6.3) is relatively less than what people usually use (e.g. 20, 25). How many random seeds did the author use for experiment in section 6.3? Is the reason for such not very generous setting due to long generation time of kernel design?

Thank you for your insightful comments regarding the settings used in our experiments. We would like to clarify that the choice of $T = 10 \times d$ is simply a rule of thumb for setting the maximum number of function evaluations, which is also employed by [10]. Regarding the hyper-parameter tuning experiments in Section 6.2, we would like to mention that our setup includes a total of 60 tasks, comprising 12 OpenML datasets paired with 5 machine learning models. For each task, we conducted 5 seeded searches, with each search consisting of 25 trials, resulting in a total of 7500 trials. We believe this setup is comprehensive enough to identify good hyper-parameter configurations. For the photonic chip design in Section 6.3, we used $T = 250$ trials with 6 random seeds, resulting in a total of 1500 trials. We would like to mention that this experiment involves using a physical simulator, which requires a considerable amount of time and cost. We appreciate the reviewer's suggestion regarding the use of a larger number of random seeds. In response, we have increased the number of random seeds for the benchmark function experiments in Section 6.1 to 20 and have updated the results in Tables 1 and 5 accordingly.

References

[1] Simpson, I. Davies, V. Lalchand, A. Vullo, N. Durrande, and C. E. Rasmussen, "Kernel identification through transformers," Advances in Neural Information Processing Systems, vol. 34, pp. 10483–10495, 2021.

[2] Yu, S. Zuo, H. Jiang, W. Ren, T. Zhao, and C. Zhang, “Fine-tuning pre-trained language model with weak supervision: A contrastive-regularized self-training approach,” arXiv preprint arXiv:2010.07835, 2020.

[3] H. Zhang, G. Li, J. Li, Z. Zhang, Y. Zhu, and Z. Jin, “Fine-tuning pre-trained language models effectively by optimizing subnetworks adaptively,” Advances in Neural Information Processing Systems, vol. 35, pp. 21442–21454, 2022.

[4] Kotha, J. M. Springer, and A. Raghunathan, “Understanding catastrophic forgetting in language models via implicit inference,” in International Conference on Learning Representations, 2024.

[5] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in Neural Information Processing Systems, vol. 35, pp. 24824–24837, 2022.

[6] S. M. Xie, A. Raghunathan, P. Liang, and T. Ma, “An explanation of in-context learning as implicit Bayesian inference,” arXiv preprint arXiv:2111.02080, 2022.

[7] C. Han, Z. Wang, H. Zhao, and H. Ji, “Explaining emergent in-context learning as kernel regression,” arXiv preprint arXiv:2305.12766, 2023.

[8] A. J. Smola and B. Sch¨olkopf, Learning with kernels, vol. 4. Citeseer, 1998.

[9] D. Duvenaud, J. Lloyd, R. Grosse, J. Tenenbaum, and G. Zoubin, “Structure discovery in nonparametric regression through compositional kernel search,” in International Conference on Machine Learning, pp. 1166–1174, PMLR, 2013.

[10] G. Malkomes and R. Garnett, “Automating Bayesian optimization with Bayesian optimization,” Advances in Neural Information Processing Systems, vol. 31, 2018.

In line 235-236, the authors mention a brief analysis will be returned by the LLM model. Can the authors provide one actual example of such analysis?

Thank you for your insightful comment. We have added a detailed discussion in Appendix F.2 to provide examples of LLM outputs, including the requested analysis. Specifically, we present sample responses from the LLM for the crossover and mutation steps, which are visualized in Figure 9. In Figure 9a, given the parent kernels LIN and SE along with their BIC values, the LLM suggests combining them using the addition operator $(+)$ to minimize the BIC. The analysis explains the rationale behind this combination: the LIN kernel captures linear trends, while the SE kernel models smoothness. By combining these properties, the LIN $+$ SE kernel effectively represents both linearity and smoothness in the data. In Figure 9b, given the LIN $+$ SE kernel and its BIC, the LLM recommends replacing the SE kernel with the RQ kernel to achieve a lower BIC value. The analysis indicates that while the SE kernel assumes fixed smoothness, the RQ kernel introduces varying smoothness, allowing the model to be more suitable for data with fluctuating noise levels. The above results illustrate that the LLM is familiar with kernel properties and their applications. Based on its responses, the LLM demonstrates a strong understanding of the relationships between different kernels and their capacity to address specific data characteristics, enabling it to make informed suggestions for the kernel selection.

I'm a bit confused by the weighting strategy explained in line 242-243. Does it mean the most ideal kernel with very low BIC (thus low weight) and high acquisition value might not be selected while some worse kernel (higher BIC, lower acquisition value) might? Could the authors further justify the intuition behind such strategy?

Thank you for raising this point. In our weighting strategy, the weight for each kernel $k$ is actually defined as follows: $w_k = \frac{\exp(-\text{BIC}_k)}{{\sum_j \exp(-\text{BIC}_j)}},$

where $\mathrm{BIC}_k$ is the BIC of kernel $k$ and $\mathbb{K}$ denotes the set of kernels. The above definition indicates that kernels with lower BIC values will have higher weights, making them more likely to be selected, and kernels with higher BIC values will be penalized. We recognize that our previous explanation may not have clearly conveyed this rationale, so we have revised Section 3 to include the above explicit definition for improved clarity. Additionally, we want to justify the intuition behind our selection strategy, which is to balance the kernel's ability to fit the data (as indicated by $w_k$) with the expected improvement at the proposed query point (as measured by $\alpha$). This balance is particularly important for the situations where some kernels may indicate a good fit, but the actual improvement may not be as significant as promised.

In Figure 2, what is "Best" (the fourth method in legend)? Does it refer to the "Utility" method in line 411?

Thank you for your careful observation regarding Figure 2. You are correct, the fourth legend refers to the "Utility" method. We have corrected the typo and updated Figure 2 accordingly.

Thank you for your time and effort in reviewing our work. Important changes have been highlighted with blue color in the manuscript. We respond to the questions and comments point-by-point as follows.

This submission does not present adequate contributions in my opinion. The idea of kernel design using transformers and language models has been studied in the field. For example, Simpson et al. proposed their transformer-based architecture for kernel design, and their model is specifically trained using a vocabulary of known kernels.

Thank you for your valuable feedback. We appreciate the opportunity to clarify the contributions of our method, particularly in relation to existing works, including KITT by [1]. We outline the main differences between our method, CAKES, and KITT as follows:

• Our contribution centers on developing a novel method to automate kernel design toward more efficient BO, whereas [1] only considered regression tasks. We believe that the goals of these two tasks differ significantly: BO emphasizes the balance between exploration and exploitation in the optimization landscapes, while regression focuses on fitting and prediction accuracy.
• Our method employs in-context learning through few-shot prompting, allowing it to adaptively generate and refine kernels based on the observed data without requiring extensive (pre)training. This approach contrasts with KITT, which relies on a fixed vocabulary of known kernels and necessitates fine-tuning, thereby limiting its flexibility and generalizability.
• Our approach leverages LLMs as ready-to-use tools that can be used by non-experts and seamlessly integrated into various tasks without the need for fine-tuning. It has been studied that fine-tuning can lead to a loss of general domain knowledge [2-4] and may not be practical or even feasible for every tasks. In contrast, KITT's reliance on fine-tuning restricts its applicability beyond regression tasks.

We hope this clarifies the distinct contributions of our work and addresses your concerns.

The authors claim the "no need for fine tuning" as an advantage for their method, but feels more like the key weakness to me. Considering all the conditioning (line 193-194, 198-199) and numerical information (such as observations and kernel's characteristic) are fed to the LLM model purely through text prompt, CAKES's performance highly (or even entirely) depends on how familiar the chosen LLM model is with GP and kernel design.

Thank you for your comments regarding the effectiveness of our method. We acknowledge that the performance of CAKES does depend on the LLM's familiarity with GP and kernel design. However, we consider this reliance not a weakness but a central aspect of our work. Our primary research question investigates the extent to which LLMs, which are trained on extensive internet-scale data, possess valuable prior knowledge that can be leveraged to enhance kernel design without fine-tuning. We believe that the "no need for fine-tuning" is an advantage because it allows us to utilize LLMs' existing knowledge base, which is derived from a vast amount of statistical information and code, to perform tasks in GP kernel design. Recent studies [5-7] have demonstrated that LLMs can recognize and extrapolate patterns from in-context demonstrations. In Appendix F.1, we have provided an empirical evidence that LLM is indeed familiar with kernel design and can act as a good crossover and mutation operators. Specifically, we have conducted an additional experiment to analyze the fitness distributions of an initial population of kernels before and after applying crossover and mutation operations (see Figure 8). We found that the initial population tend to have a wide distribution, with a noticeable peak at lower fitness values. After performing crossover, the distribution shifts slightly toward higher fitness values, indicating that the LLM can effectively combine the kernels and enhance the overall fitness of the population. We further observed that applying mutation shifts the distribution toward even higher fitness values, implying that refining the kernels can yield improved fitness values.

In addition, the crossover and mutation operators (line 227-234) utilize very standard kernel grammar and simple replacement, which are already well-developed techniques.

Thank you for your insightful feedback regarding the crossover and mutation operators used in our work. While these operators may be considered as standard and already well-established, they are in fact good enough to form a rich and expressive space of kernels. By repeatedly applying these operations, one can construct polynomial kernels, which can be used for capturing non-linear patterns in data. One can also apply these operations independently across each dimension to construct multi-dimensional kernels, which can model interactions between different input features. As we mentioned in Appendix H, while the current work focused on these two operators (addition and multiplication), the kernel grammar can also be expressed using other operators that preserve the closure properties of kernel functions, such as convolution, composition, and affine transformations [8]. In fact, as stated in Definition 1 of Section 3, our method is built based on this generalized kernel grammar, which implies that the other operators can be regarded as direct extensions of our method. We would like to explore these possibilities further in our future work.

Experiments design is not very consistent. In section 6, the authors use 5 different metrics to measure the performance of methods in different experiments. Given the BO settings, I don't see a good reason for varying performance metric between different experiments when the authors could just simply select one. Simple regret is commonly used and more importantly consistent with the regret bound proven in section 4.

Thank you for your valuable comments regarding the consistency of our experiment design. In response to your comments, we have replaced the GAP metric previously used in Section 6.1 with the regret metric, as suggested by the reviewer. Correspondingly, we have updated the results in Tables 1 and 4 to reflect the average regret obtained by each method. In Section 6.2, we have used regret to measure the performance in the hyper-parameter tuning tasks, and further showcased the regret for each individual tasks in Figures 14-18 of Appendix G.2. For Section 6.3, we would like to note that the global optimum is unknown for the photonics chip design problem. Therefore, we employed problem-specific metrics such as the score and hypervolume to evaluate the BO performance. We believe these metrics are more relevant to the specific problem of photonics chip design and provide valuable interpretations for experts and engineers in the field.

The "average rank" metric presented in Table 2 seems to highlight the same information as the "average regret" plots in Figure 2.

Thank you for your feedback. We would like to first clarify that the "average rank" metric actually provides additional insights, as it is calculated based on the performance on individual tasks. In contrast, the "average regret" displayed in Figure 2 presents the aggregated performance across all tasks. In light of the reviewer's feedback, we have decided to move the "average rank" results to Appendix G.2 as additional results to complement the main experiments. We believe this revision makes that our presentation more focused while still providing comprehensive insights.

In line 406-408, the authors mention SE and Matern-5/2 as the two most commonly-used kernels. Why is Matern-5/2 kernel not included in the four base kernels defined in line 203 then?

Thank you for bringing up this important point. Initially, we selected the four base kernels (SE, PER, LIN, RQ) due to their simplicity and ease of combination. These kernels are well-established in the literature, particularly in the foundational work by [9], and provide a solid base for constructing more complex kernels using the grammar operators. In response to your comment, we have revised our manuscript to include two variants of the Matern kernel: Matern-3/2 (M3) and Matern-5/2 (M5). We also conducted additional experiments with these kernels and the new results are given in Appendix G.1 of the revised manuscript. The results and conclusions remain consistent, showing that our method outperforms all fixed kernel approaches in terms of both mean and median regret.

Thank you for the clarifications. With the modifications and additional details, I think the submission is more concrete now. I'm happy to increase my score. One additional comment for authors is to update kernels illustrated in $*Initial Population*$ in Figure 1 to make it more consistent.

Thank you for your time and effort in reviewing our work. Important changes have been highlighted with blue color in the manuscript. We respond to the questions and comments point-by-point as follows.

Limited testing in high-dimensional spaces: The paper does not demonstrate the effectiveness of CAKES on high-dimensional optimization problems. This leaves uncertainty about how well the method scales to more complex search spaces with many variables.

Thank you for your valuable feedback regarding the validation of our method in high-dimensional spaces. In response to your comments, we have conducted additional experiments using the Ackley and Levy functions with higher input dimensions, specifically $d \in$ {10, 20, 30}. We chose these two functions as they are highly non-convex and possess numerous local minima, making them particularly challenging to optimize. We set $T = 10 \times d$ and repeat each experiment using 20 different random seeds. The results, presented in Table 6 of Appendix G.1, demonstrate that CAKES consistently outperforms the other methods by minimizing the regret, even as $d$ increases. This corroborates that our method can scale effectively in more complex search spaces.

Why did you use gpt-4o-mini only? If other large language models such as gpt-4o and llama-3 are used, can the empirical results change? And why did you choose a temperature of 0.7 and a top_p of 0.95 specifically?

Thank you for your feedback regarding our choice of LLM and parameter settings. As we elaborated in Appendix D.1, we selected gpt-4o-mini for our implementation because it provides a good balance of affordability, fast inference, and intelligence. While we acknowledge that other LLMs, such as gpt-4o and Llama-3, may yield different empirical results, we believe that the overall methodology and key insights presented in our work extend beyond the specifics of any particular model. This is further discussed in Appendix H, where we emphasize the general applicability of our method. Regarding the choice of parameters, we specified a temperature of 0.7 and top_p of 0.95 to strike an optimal balance between generating relevant outputs and introducing diversity. As mentioned in Appendix D.1, these settings were determined based on our large-scale experimental results.

References

[1] Wang, C. Gehring, P. Kohli, and S. Jegelka, “Batched large-scale Bayesian optimization in high-dimensional spaces,” in International Conference on Artificial Intelligence and Statistics, pp. 745–754, PMLR, 2018.

[2] Malkomes and R. Garnett, “Automating Bayesian optimization with Bayesian optimization,” Advances in Neural Information Processing Systems, vol. 31, 2018.

[3] K. Kandasamy, J. Schneider, and B. Poczos, “High dimensional Bayesian optimisation and bandits via additive models,” in International conference on machine learning, pp. 295–304, PMLR, 2015.

[4] J. Gardner, C. Guo, K. Weinberger, R. Garnett, and R. Grosse, “Discovering and exploiting additive structure for Bayesian optimization,” in Artificial Intelligence and Statistics, pp. 1311–1319, PMLR, 2017.

[5] D. Ohnishi, T. Okano, M. Imada, and S. Noda, “Room temperature continuous wave operation of a surface-emitting two-dimensional photonic crystal diode laser,” Optics Express, vol. 12, no. 8, pp. 1562–1568, 2004.

[6] M.-Y. Hsu, G. Lin, and C.-H. Pan, “Electrically injected 1.3-μm quantum-dot photonic-crystal surface-emitting lasers,” Optics Express, vol. 25, no. 26, pp. 32697–32704, 2017.

[7] L.-R. Chen, K.-B. Hong, K.-C. Huang, H.-T. Yen, and T.-C. Lu, “Improvement of output efficiency of p-face up photonic-crystal surface-emitting lasers,” Optics Express, vol. 29, no. 7, pp. 11293–11300, 2021.

Thank you for your time and effort in reviewing our work. Important changes have been highlighted with blue color in the manuscript. We respond to the questions and comments point-by-point as follows.

I think that few-shot learning is not correct in this context. The authors didn't fine-tune large language models. It should be few-shot prompting.

Thank you for your feedback regarding the terminology used in our manuscript. We agree that the term "few-shot prompting" is more appropriate in this context. In response, we have revised the manuscript to replace "few-shot learning" with "few-shot prompting" throughout the relevant sections.

More real-world benchmarks are missing. I think that hyperparameter tuning tasks are involved with small simple models. If more sophisticated models are tuned, it would be better.

Thank you for your feedback on the importance of conducting real-world benchmarks and including more sophisticated tasks in our work. In response to your suggestions, we have conducted two additional experiments in Appendix G that involve tuning a controller for the following tasks:

• Robot pushing $(d = 14)$: This task was initially introduced in [1], which involves tuning a controller for two robotic hands to push objects to target positions. The objective is to minimize the total distance from the initial and final positions of the objects to their respective goal positions, thus maximizing the reward. We ran simulations for $T = 100$ iterations with 10 random seeds. The results, summarized in Figure 12 of Appendix G.4, show that CAKES outperformed other methods in achieving higher rewards and minimizing the regret.
• Lunar lander $(d = 12)$: In this task, we consider tuning a controller for a lunar lander in OpenAI Gym. The objective is to land the lunar module between two flags on the landing pad without crashing, while minimizing the engine use. We ran the simulations for $T = 250$ iterations using 50 randomly generated terrains, initial positions, and velocities. As shown in Figure 13 of Appendix G.5, CAKES consistently achieved or exceeded the target objective of 200 points while minimizing the penalties and optimizing the landing strategies.

We believe these additional experiments have demonstrated the effectiveness of our method in more sophisticated, real-world tasks.

More recent baselines are missing. Some baselines such as (Malkomes and Garnett, 2018) should be compared.

Thank you for your feedback regarding the inclusion of more recent baselines for comparison. We focused on GP-based BO methods as the primary baselines because we believe these are the most relevant comparisons to our method, which specifically addresses kernel design. As emphasized in Section 1, one of our contributions is to highlight the urgent need for a more careful treatment of kernel design in BO. We argue that conventional setups, which often rely on fixed kernels or naive selection strategies, do not perform uniformly well across different tasks. Still, we appreciate the reviewer's suggestion to include additional recent baselines, such as those from [2]. In response, we have incorporated comparisons against the following additional methods:

• Bag-of-models (BOM) [3]: This method maintains a collection of kernels, each capturing different structures that best explain the observed data.
• Markov Chain Monte Carlo (MCMC) [4]: This method uses a Metropolis-Hastings algorithm to randomly sample kernels from a grammar based on a proposal distribution.
• Automated BO (ABO) [2]: This method utilizes BO to dynamically select kernels that best explain the observed data.

We provide additional details regarding these baselines and the experimental setup in Appendix G.1. The results, summarized in Table 5, indicate that our proposed method, CAKES, outperforms all the baselines in terms of mean and median regret. Notably, CAKES ranked the first in 12 instances out of 15 and secured a top-two ranking in all instances.

Some baseline results of photonic chip design might be missing. I don't know why only two baseline methods are compared to CAKES in photonic chip design.

Thank you for your valuable feedback. We would like to first clarify the current baselines we included are BO-based methods. To the best of our knowledge, these were the only two GP-based BO methods specifically applied to this problem in the literature. However, we appreciate your suggestion to include additional baselines in this experiment. In response, we have added more baseline methods from the literature, specifically those published in top optics and photonics journals, as detailed in Appendix G.3. The results of these comparisons are summarized in Table 8. Our findings demonstrate that the chip designed by CAKES remains competitive or even superior to these additional baselines.