An Attention-based Approach for Bayesian Optimization with Dependencies

• The main contribution of the paper is to include tree structure aware identifiers with each hyperparameter (Appendix B.1.1) and learning embeddings with the attention module. However, these identifiers are a hand-designed feature and only an artifact of the way tree search space is constructed. Since one of the major goals of the paper is AutoML, this seems to be defeating the main purpose by adding manual feature construction. Please clarify and discuss this important point.

• Some statements in the paper are not entirely incorrect. For example, it is mentioned that "there is no suitable hand-crafted kernel function capable of modeling the similarity between these configurations from" but arc-kernel (reference [1] below) is one such kernel that handles conditional search spaces and is a natural baseline for this problem. Please consider reevaluating them.

  • [1] Swersky, K., Duvenaud, D., Snoek, J., Hutter, F., & Osborne, M. A. (2014). Raiders of the lost architecture: Kernels for Bayesian optimization in conditional parameter spaces. arXiv preprint arXiv:1409.4011.

• I am afraid that Add-tree baseline (which is significantly easier to implement) outperforms the proposed approach in multiple instances in Figure 4 and Figure 15. Please explain this. I also think a simple baseline with imputing missing values for inactive hyperparameters is a simple baseline that is easy to try should be included. If possible, please consider including it.

• Writing should be improved. There are many typos and grammar errors. For example,

1. n noisy observations in Page 1 should be $n$ noisy observations.
2. $d$ is the dimension in Page 1 should be $d$ is the number of dimensions.
3. even different dimensions in Page 1 should be the different number of dimensions.
4. configurations in the same subspace has in Page 1 should be configurations in the same subspace have.
5. black-box object function in Page 2 should be black-box objective function
6. (SE) kernel function in Page 2 should be (SE) kernel functions.
7. quires in Page 2 should be queries.
8. cifer-10 in Page 7 should be CIFAR-10.
9. quires in Page 9 should be queries.

I think there might be other cases. These should be carefully fixed.

• Important references, for example, [1, 2, 3], are missing.

[1] Lyu, Wenlong, et al. "Efficient Bayesian Optimization with Deep Kernel Learning and Transformer Pre-trained on Multiple Heterogeneous Datasets." arXiv preprint arXiv:2308.04660 (2023).

[2] Bowden, J., et al. Deep kernel Bayesian optimization. No. LLNL-CONF-819001. Lawrence Livermore National Lab.(LLNL), Livermore, CA (United States), 2021.

[3] Chowdhury, Sankalan Pal, et al. "Learning the Transformer Kernel." Transactions on Machine Learning Research (2022).

Many papers including the references mentioned have studied Transformer-based or attention-based kernel learning. These are all related to this work. Unfortunately, the absence of the discussion on this line of research degrades the quality of this work. Moreover, the novelty of this work should be also debatable.

• A method to select a query point by considering a tree-structured search space is questionable. Eventually, query points are determined by optimizing the respective acquisition function defined on subspaces according to Algorithm 1. I think it reduces the need to use attention-based kernels. I think there might exist a way to optimize an acquisition function on the entire conditional search space.

• The scale of experiments are relatively small.

1. The most concerning problem is the motivation and problem setup. While the author argues that the proposed embedding and modeling are designed to deal with specific hierarchical dependencies, there is neither a formularized statement of the problem involving the hierarchical structure (only a short footnote in the introduction and an example which is illustrated in Figure 1) nor a clear definition or statement of considered 'hyperparameters' (it seems that the paper deal with the anisotropy of the objective yet this is not clearly defined). The essential concepts need to be more specific throughout the paper. Otherwise, the vagueness undermines the soundness and contribution of the paper.

2. The discussion on related work is limited. For example, the recent advancement in casual-model-based BO methods [1,2] that also explicitly deals with the dependencies on the search space needs to be included.

3. Some results are placed in the appendix but are actually critical while insufficient. The ablation study doesn't show consistent improvement over AttnBO-no-emb in both Figure 13-c and Figure 14-a. Apart from the ablation study, the visualization of the attention maps only uses two random seeds and, therefore, lacks significance.

4. The baselines in the experiment are insufficient compared to the related methods discussed in Section 2. Specifically, Naïve DKL-based BO or VAE-based BO are not compared as baselines. This comparison could serve as the ablation study of the proposed encoder-based AttenBO.

5. The paper's primary contribution resides in the domain of representation and kernel learning, offering a novel solution for the Gaussian process in the context of structured input. The acquisition function is not tailored to this particular setting. The proposed method should be discussed within the broader scope of GP studies rather than being confined to a Bayesian optimization-oriented approach. In that sense, the intermediate evidence of improved GP regression is far from sufficient.

Reference

[1] Aglietti, Virginia, Xiaoyu Lu, Andrei Paleyes, and Javier González. "Causal Bayesian optimization." In International Conference on Artificial Intelligence and Statistics, pp. 3155-3164. PMLR, 2020.

[2] Sussex, Scott, Anastasiia Makarova, and Andreas Krause. "Model-based causal Bayesian optimization." arXiv preprint arXiv:2211.10257 (2022).