HiBO: Hierarchical Bayesian Optimization via Adaptive Search Space Partitioning

W1

W1. Why does HiBO reply on a binary serach tree instead of paritioning each level to multiple clusters? (like R tree) Also does the serach tree need to be a balanced tree?

Thanks for pointing out the design choice of binary search tree. Indeed this can be a valid direction of extending the current work but the effectiveness of our framework does not rely on the number of sub-partitions generated by each partitioning operation.

The partitioning is aimed to differentiate the sampling potential of the generated sub-partitions and provides information of distribution of sampling potential over the search space. This should also be applied to general K-ary cases only if the difference of sampling potential among sub-partitions can be effectively maximized by the clustering- and classification-based partitioning operations. The binary search tree was chosen in our current design for simplicity and computational efficiency, as it minimizes the overhead of managing and evaluating a larger number of partitions at each level.

Regarding the need for a balanced tree, HiBO's framework does not strictly require the tree to be perfectly balanced. Empirically, a threshold on the quantitative metric of the clustering quality to determine whether this partitioning operation can even be set to avoid low-quality splitting and hence result in unbalanced trees. But the core, again, of building this tree is to differentiate the sampling potential of sub-partitions generated by recursive partitioning, whether the tree balanced or not is not the focus of our work as we care more about whether the generated distribution of sampling potential over partitions is informative and effective.

W2

W2. I find it challenging to fully appreciate Rule 1 and Rule 3. I think these rules make sense but the author should perhaps give better insights on these two rules.

Sorry for any confusion resulted from the rules in that part and we are happy to provide some clarification here::

1. Rule 1: Rule 1 is more like an implementation tip for conveniently tracking the depth of the search tree by breadth-first search (BFS). In our algorithm, the tree height corresponds to the number of bifurcations required to form the partitions represented by the leaf nodes. Greater tree depth results in smaller partitions, emphasizing more specific regions for exploitation. For balancing the exploitation and exploration, we implement the logic based on BFS to track the depth when building the tree and stop expanding the tree when the depth reaches the maximum depth.
2. Rule 3: We sincerely apologize for the typo in Rule 3 and thank you very much for pointing it out. The phrase “falls below a threshold” should instead read “goes beyond a threshold.” When the tree depth goes beyond the threshold, it indicates it has encountered multiple times of consecutive failures, the region covered by the partition with the greatest sampling potential becomes limited and the search is suspected to get stuck into a non-promising region. This is similar to the rule applied by TuRBO which restarts the search when trust region size becomes too small after multiple times of consecutive failures. We have modified this statement in the revised version of our paper.

W3

W3. In fact, one may argue that DBMS tunning is suitable for BO becuase the number of important knobs can be small? Perhaps I missed it, but what is the workload in sysbench (read-only, write-only, mixed)?

Thanks for raising the concern on DBMS configuration tuning. The fact is basically correct because DBMS configuration tuning does have limited number of important knobs, but this does not diminish our conclusion that HiBO's effectiveness. The total dimensionality of this task is still high while the sample budget is very limited. This scenario is similar to the sparse synthetic benchmarks with a small subset of effective dimensions (e.g. Levy20-200D) though DBMS configuration tuning is way more noisy and more challenging. Identifying the potential region in a high-dimensionality space for search is non-trivial, reflected by the suboptimal performance of other algorithms in this task.

To further validate HiBO, we added two additional real-world benchmarks in Appendix E, including Mopta08 (124D) and Rover Trajectory Planning (60D), which are not sparse. HiBO outperformed other methods on both benchmarks, and HiBO-GP (HiBO with GP-based BO as the local optimizer) also surpassed vanilla GP-BO. Additionally, experiments on 200-dimensional Levy and Rastrigin tasks indicate that HiBO is effective for fully high-dimensional tasks.

As for the question about workload in SysBench, sorry for missing the detail of the workload. The workload in SysBench is mixed including both read and write requests and the details of the SQL statements in this workload can be found in the newly added Appendix A.3

W1 & W2

W1. The motivation is unclear. It is not evident what the challenges are in introducing structural information of the hierarchy. What are the criteria for optimally utilizing the so-called structural information?

W2. Although this paper dedicates considerable space to discussing the integration of partitioning information, its novelty is limited. Many existing techniques are well-known, which undermines its originality.

Motivation

Thanks for sharing your concern on the motivation and sorry for any unclear expression in our paper about this. What we hope to emphasize is the insufficiency of prior works adopting search space partition for scaling BO to high-dimensional search space. They generate search trees with the tree's structure itself conveying the information of the estimated distribution of sampling potential over the search space, but they only choose the only region with greatest estimated sampling potential for confining the sampling scope into it. We think this is a waste of generated information especially considering the complexity of building this tree. On the other hand, when the initial sample budget is limited, the quality of built search tree has not been good enough yet and confining the sampling scope following the tree can significantly prevent effective exploration over the search space. Our experimental results (e.g., Rastrigin20-200D, Hartmann6-300D, and DBMS configuration tuning) demonstrate LA-MCTS, as the SOTA method of this paradigm, can only perform comparably to plain TuRBO and fall behind HiBO. And we revised our paper to emphasize the motivation in Introduction.

Novelty

Instead of strictly confining the sampling scope, we allow the sampling to span across the regions and aim to leverage the information from the partitioning process for guiding the search in a more flexible and comprehensible way. The guidance provided by the global-level navigator is integrated into the local optimizer's acquisition strategy, which is general for almost all BO-based algorithms and easy to implement. To the best of our knowledge, we are the first to combine the search space partitioning information with the acquisition strategies of BO. Besides, the adaptive control on maximum tree depth proposed in this work is empirically validated to be an effective design to balance exploitation and exploration while reducing the computational cost.

Additionally, we are the first to introduce DBMS configuration tuning as a practical benchmark for evaluating BO algorithms' effectiveness. This benchmark presents a high-dimensional, noisy, and correlated search space, closely resembling real-world scenarios. The inclusion of this task allows us to effectively evaluate HiBO’s sample efficiency on practical high-dimensional optimization problems. From the results presented in Figure 4, HiBO achieves about 28% greater throughput improvement over the default configuration than LA-MCTS within a very limited sample budget (50), indicating the soundness of our motivation and effectiveness of our methods.

Motivation

Thank you for carefully reviewing our responses and for increasing your score. We deeply appreciate your acknowledgment of our efforts to address your concerns and improve the paper.

Following your suggestion, we have further revised the paper to explicitly incorporate the issues raised in the motivation section, ensuring clarity and alignment with your feedback:

• Between line 38 and line 47, we included the challenges of optimization over real-world high-dimensional problems and insufficiency of prior works in the field of DBMS configuration tuning as the example scenario. This contributes to part of our motivation for designing HiBO as an effective approach that can be applied to real-world systems (line 70 to line 72)
• Between line 58 and line 63, we explicitly emphasize the insufficiency of prior works adopting search space partitioning for scaling BO to high-dimensional search space, describing their ineffcient utilization of generated structural information and risk of limiting exploration because of unreliable initial tree and strict sampling scope confinement.
• Between line 66 and line 70, we explicitly included the motivation of designing the adaptive mechanism to control the maximum tree depth, which corresponds to the statement we put on Section 4.3 (line 281 - line 297)

Based on the revised introduction emphasizing the motivations, the contributions are correspondingly modified (line 70 - line 87).

Scalability

Pointers

Besides the motivation-related contents, we additionally included an extra Appendix G for addressing the concerns of Scalability as suggested by Reviewer D9Dm. We discussed the scalability of this algorithm and think there can be two different interpretation of the "scalability": the scalability with respect to increased dimensionality of the problem and the scalability with respect to the large scale dataset. As the previous response provided, the scalability w.r.t large scale dataset is not commonly discussed for BO-based algorithms towards optimization within limited sample budget and the clarification is put in Appendix G.2 (line 1120 - line 1140).

But we further included another part of experiments on evaluating HiBO and TuRBO on Levy benchmark with increasing dimensionalities in Appendix G.1 (line 1077 - line 1119), described as following.

Result Analysis

Given that the primary modification to the original local optimizer lies in the global-level navigator, our analysis focuses on comparing the performance and timing costs of HiBO and TuRBO on the synthetic Levy benchmark for 500 iterations across varying dimensions, as shown in Figure 9. Figure 9 (Left) depicts the ratio of mean regret (RMR) between TuRBO and HiBO ($\overline{regret}\_{TuRBO} / \overline{regret}\_{HiBO}$), while Figure 9 (Right) illustrates the ratio of optimization time (ROT) for HiBO relative to TuRBO ($Time_{HiBO} / Time_{TuRBO}$). Regret is defined as the difference between the optimal objective value and the value achieved by the algorithm at a given iteration, with mean regret representing the average regret over the entire optimization process. Lower mean regret indicates better algorithm performance.

Figure 9 (Left) indicates that RMR increases steadily from ~1.03 to ~1.22 with dimensionality increasing from 50 to 1000, suggesting that TuRBO's mean regret on Levy grows more significantly than HiBO's as dimensionality rising. In other words, HiBO’s performance improvement over plain TuRBO scales effectively with increasing problem dimensions.

On the other hand, as Figure 9 (Right) shows, although ROT remains above 1 due to the additional time cost brought by the global-level navigator of HiBO, ROT decreases from ~1.66 to ~1.18 as dimensionality increasing. This trend suggests that as the dimensionality grows, the local optimizer (i.e. TuRBO) experiences a heavier computational burden and suffers from poorer scalability compared to the global-level navigator. As a result, the relative overhead introduced by the global navigator becomes less significant as the local optimizer becomes the dominant computational component in increased dimensionalities.

These results empirically demonstrate HiBO’s robust scalability with increasing dimensionality, as evidenced by its growing performance advantage over the local optimizer and the decreasing relative computational overhead of its global-level navigator.

Given these updates and the addressed concerns, we would greatly appreciate it if you could consider revisiting your score to the positive side (6 or greater), particularly if you feel the revisions improve the quality and clarity of the work. Thank you very much again for your evaluation and helpful feedbacks.

Dear Reviewer 6Fsx,

Thank you once again for your valuable feedbacks and for increasing your score earlier. As indicated by previous messages sent to you, we have made further modifications to the paper following your additional guidance.

As today is the last day for reviewer to post a message to the authors, we kindly ask if you might have the opportunity to review these updates. If you feel the changes sufficiently address your concerns, we would be grateful if you could consider increasing your score to the positive side.

Sorry for multiple reminders sent here and we greatly appreciate your time and thoughtful contributions to this review process. Please let us know if there are any remaining questions or clarifications we can provide.

Best Regards, The Authors

W2 & Q2

W2 It remains unclear whether and which hyper parameters have to be tuned.

Q2 Which hyper parameters are required (e.g. C_p, see line 263)? How are they automatically chosen/tuned?

Thanks for the question about hyperparameters. We have provided a complete list of hyperparameters with their meanings and scopes in Appendix B in the revised version of our paper. The explanation on the hyperparameters of HiBO include:

• $C_p$: Controls the balance between exploration and exploitation in the UCT calculation. Larger values increase the weight of exploration. Range: [0.5, 5.0].
• $\tau$: Softmax temperature for normalizing partition scores into a positive and normalized range. Range: [0.01, 2]. This hyperparameter is used for controlling the impact of global-level navigator on local optimizer and more details can be found in Appendix D. The two extreme cases of $\tau$'s value can lead to the following effects:
  • Extremely high $\tau$: Results in an overly smoothed distribution, where the difference between biases assigned to different space partitions becomes negligible. Eliminates the impact of global-level partitioning and diminishing differences between partitions;
  • Extremely low $\tau$: Results in a peaky distribution and assigns overwhelming weight to the partition with the highest sampling potential, amplifying the navigator's influence.
• Maximum Tree Depth Limit: The limit of maximum allowable tree depth before the search process restarts after multiple times of repeated failures. It controls the tolerance of consecutive failures in one search. Range: [3, 5].

We did not use advanced algorithms for tuning them but applied naive grid search for searching the best performing combinations. The parameters selected in our experiments are summarized in Appendix A.

W3

W3 Limitations of the paper’s methods could be better discussed.

Thank you for the question about limitations of HiBO may face challenges. From our experiment results and reasoning, we summarized the following two cases:

1. One limitation of HiBO lies in its relatively less pronounced improvement over the local optimizer on dense high-dimensional benchmarks compared to sparse benchmarks. While HiBO demonstrates significant performance gains on sparse benchmarks (e.g. achieving over two orders of magnitude lower regret on tasks such as Hartmann6-300D), its improvement is less evident on dense tasks where all dimensions are uniformly effective (e.g., Levy-200D, Rastrigin-200D). Our interpretation is that in sparse spaces, HiBO efficiently identifies promising regions to guide the local optimizer and it is one of its greatest advantages. While in dense spaces, it requires more samples to achieve accurate estimations across the entire space than in sparse scenarios. But HiBO still converges efficiently towards the optimal point in these tasks and achieves the best in Levy-200D and Rastrigin-200D.
2. HiBO may also struggle in scenarios where certain critical configuration variables are hidden or unobservable during optimization. Such situations can arise in practical tasks where parts of the configuration space are inaccessible due to hardware constraints, legacy systems, or incomplete problem specifications. In these cases, the global-level navigator may build an inaccurate model of the search space, leading to suboptimal partitioning and sampling.

W1 & Q1

W1 The required computational costs and scalability remain somewhat unclear.

Q1 What are the required computational costs? Does the approach scale?

We appreciate the point about computational cost and scalability of our approach and we do understand the concern because the design of HiBO introduces the extra complexity in BO's algorithm framework.

Tuning time cost is more a practical setting that we care in real-world applications. Considering this, we care more about the computational cost in DBMS configuration tuning as it provides a good testcase to observe our algorithm's practical factors. But we do not provide direct timing measurement because it will be meaningless if we do not consider whether the cost is worth it. Therefore, we used the custom metric S-PITR (see Section 6.2.3) as an alternative to comprehensively measure the practical value of our algorithm, which provides the ratio of the tuning effect against the tuning time cost (and also the failure rate). As can be observed in Figure 4 (Right), HiBO achieves the greatest S-PITR score indicating the practical value even considering the tuning time cost.

Specifically, HiBO incorporates an Adaptive Maximum Tree Depth mechanism to balance the exploitation and exploration while preventing excessive computational costs caused by overly deep partitioning trees. For evaluating its effect, we did an additional ablation study about the tree depth settings on the DBMS configuration tuning benchmark and measured the ratio of tuning time against the total time for each tuning iteration, consisting of configuration loading, workload execution and optimization execution time for each setting. We put this part of work in Appendix C in the revised version.

The tuning time proportion can be found in the following table:

From this table, we observe that while HiBO introduces additional computational costs during the optimization phase compared to TuRBO, the increase is minimal: even if we set the tree depth to always be 4, the extra proportion of the execution time within each iteration is within 2% on average. Enabling adaptive control keeps the tuning time cost of HiBO only about 4.3% of the benchmarking time, while it achieves the best performance improvements and S-PITR scores among all selected algorithms in this practical benchmark.

Dear Everyone,

This is a very poorly written review. Similar to CVPR, is ICLR planning on desk rejecting submissions from reviewers due to irresponsible reviews?

Reviewer q24C mentions in their weaknesses for the paper: "Ignores previous work on exactly the same topic of hierarchical Bayesian optimization" and "Ignores previous work on exactly the same application area of database tuning" but never really bothers to cite papers or the related works that are supposedly missing! This is the worst kind of review possible! Reviewer q24C attempts to add a third weakness "Missing rationale and justification on the inclusion and exclusion of related works." which is merely a repetition of the first two mentioned weaknesses and not really a new point, again not supported by any references!

In their questions, in Q1 the reviewer mentions a paper on a far-off topic from this submission and asks why is it not included, and as Q2 the reviewer asks why a non-peer reviewed arxiv paper that was released 4/5 months before the ICLR submission deadline not included in this submitted paper! It seems like the reviewer does not know the ICLR guidelines.

Lastly, not only does the reviewer seem to be oblivious to ICLR guidelines, but the reviewer dares to claim "Ethical Concerns" based on some "advice" from ACM! I'm sorry but this is not an ethical concern! The use of the term "black-box" in machine learning has no roots in racism, but has roots in the history of aviation and the use of the colour black to prevent rust on flight recorders! Moreover, it is a commonly used term and never used with the intention of discrimination. These are superfluous claims at best.

To the best of my knowledge, this is not an LLM-generated or ChatGPT-review, because, in my limited experience with LLMs (ICLR forcing down LLM-generated feedback on reviewers for reviews given to ICLR submissions this year), they would have done a better job reviewing this paper. As a member of the computer science community, in my humble opinion, this review should be discarded, and if possible the actions similar to CVPR could be considered for "Irresponsible Reviews".

Best Regards

Just a simple member of the computer science community who decided to use the "Public Comment" option in openreview

Anyways, thank you very much for your feedbacks and pointing out the concerns. We hope that our responses and the revisions to the paper address your concerns and clarify any ambiguities. If you feel that these responses and changes improve the quality and clarity of the work, we kindly ask you to consider raising your score. We are happy to provide more clarification if you have additional concerns or feedbacks.

W1 & Q1

W1. Ignores previous work on exactly the same topic of hierarchical Bayesian optimization.

Q1. The notion of Hierarchical Bayesian Optimization Algorithm has been introduced by Pelikan and Goldberg in Hierarchical Bayesian Optimization Algorithm. Scalable Optimization via Probabilistic Modeling 2006: 63-90. The present paper does not discuss its relation to this precedent.

Thanks for pointing out the paper published in 2006 using the same terminology 'Hierarchical Bayesian Optimization' to describe their approach. However, after carefully reading their work, we realized that their target problems are not directly related to ours: they aim to provide scalable solutions for nearly decomposable and hierarchical problems while we do not assume any decomposability or hierarchical properties of the problems to be solved. We are only concerned about the general opaque-box optimization problems especially with high dimensionality, and the "hierarchicalness" of our approach lies in the structure of the approach instead of the problem. And correspondingly, the Related Work section in our work mainly include the works about BO-based approaches in high-dimensional search space instead of those for decomposable and hierarchical problems.

I hope this clarifies the rationale of our inclusion and exclusion principles about BO-related works.

Dear Program Chairs,

Thank you very much for the clarification on how "ethical concerns" flagged for papers would be addressed, It is indeed very helpful. I trust in the sound and informed judgment of the ACs and SACs, whatever it might be.

However, the other two very important concerns regarding the review's quality remain unaddressed. I completely understand the workload of all the involved chairs and I am immensely grateful for all the efforts and work invested as they help improve the quality of the conference. Thus, I do not expect a reply, just hoping that the two issues raised in my feedback (other than the issue regarding the alleged ethical concern) on the review in "Low Quality Review" do not go unnoticed.

Thank you very much! To keep the discussions focused on the paper itself and to avoid having to cite multiple references that show that Flight Recordings were indeed historically black and are now painted in bright orange for easy recoverability, I am not going to respond to Reviewer q24C's response.

Best Regards

Thank you for the reply. This distinction of different variants of hierarchicity would be good to clarify in the paper.

Thank you for your detailed feedback and for pointing out the language issues introduced in the revised version. We sincerely apologize for the incomplete sentences in Lines 063–066 and 158–161, as well as the use of the non-standard term "hierarchicalness."

Unfortunately, the deadline for revisions has passed, and we are unable to make further edits to the submitted paper at this stage. But we acknowledge these as valid concerns and will address them in the camera-ready version if the paper is accepted. The first two errors about "while". These errors stem from our misunderstanding that "while" could function like "however" to begin an isolated sentence. We would later replace them with the following two sentences:

• However, incorporating structural information into the original BO algorithm for optimal performance poses challenges, such as designing an effective hierarchy to integrate the module extracting structural information and ensuring the optimal utilization of this information.
• In the contrast, HiBO does not assume any decomposability or hierarchical properties in the optimization problems it addresses.

On the other hand, the non-existent "hierarchicalness" with double quotes is used for emphasizing the property of being hierarchical in a nuanced way, though we now recognize that the standard term "hierarchy" could have conveyed the same meaning effectively. Again, if possible, we guarantee that we would also fix this in the camera-ready version.

While we acknowledge these concerns, it is worth noting that such minor issues can occasionally occur even in papers achieving 10 in ICLR. The existence of the camera-ready stage reflects the importance of providing authors with the opportunity to further refine and polish their work after revisions. We can guarantee that: should the paper be accepted, we will ensure that all remaining language and formatting issues, including those highlighted in your comments, are thoroughly addressed in the camera-ready version.

We have worked diligently to address the major concerns raised in earlier feedbacks from you and other reviewers, and we believe these efforts do improve the paper. We hope that the quality and significance of our work will remain the primary focus of evaluation.

We deeply appreciate your detailed feedbacks and hopefully you could consider revisiting your scores in light of our responses and commitment to addressing all remaining issues in the camera-ready version.

Thank you for your reply and the extra time to re-evaluate our work. We sincerely appreciate for your decision to increase your score—it reflects the collaborative effort to refine this submission. Should the paper be accepted, we will ensure the correct usage of "In contrast" in the camera-ready version as well.

Thanks again for all the detailed feedbacks you provided during this review process, which helped us a lot to improve the clarify and quality of our work.

W1 & Q1

W1. Significance: While the empirical results are promising, theoretical results have not been explored despite a simpler approach.

Q1. What made developing the approach challenging and what kind of theoretical properties could potentially be derived?

We appreciate raising the concern regarding the theoretical aspects of HiBO. HiBO is a dynamic and adaptive BO framework where search space partitioning is data-driven, and the search tree depth is adjusted adaptively based on recent results. These properties make deriving an accurate theoretical analysis challenging, but we can provide a simplified analysis showing how HiBO mitigates over-exploration in vanilla GP-based BO.

HiBO employs a partition-score-guided acquisition strategy that balances exploration and exploitation at a finer granularity. Specifically, the global-level navigator partitions the search space into partitions $\{\Omega_1, \Omega_2, \dots, \Omega_k\}$ and estimates the distribution of sampling potential over these partitions, assigning each partition $\Omega_j$ a score $P_j$ based on Upper Confidence bound for Trees (UCT). Assuming the clustering and classification algorithms perform adequately, partition scores of different partitions are expect to exhibit great difference, and the most promising partition $\Omega^*$ containing the global optimum $x^*$ should be assigned the greatest partition score such that $P^* \geq P_j, \space \forall \Omega^* \neq \Omega_j$

On the other hand, the local BO-based optimizer integrates these scores into its acquisition function:

$\alpha_P(x; D_t) = \alpha(x; D_t) \cdot P_{\text{leaf}(x)}$

where $\alpha(x; \mathcal{D}_t)$ is the acquisition function used in standard BO, $D_t$ is the dataset accumulated til the current $t$-th iteration, and $leaf(x)$ refers to the partition containing $x$. This approach biases sampling toward high-potential space partitions while preserving exploration in under-sampled areas.

Vanilla GP-based BO often suffers from over-exploration due to its reliance on uncertainty $\sigma(x)$ in acquisition functions, particularly in high-dimensional spaces. For example, in the UCB acquisition function defined as below:

$\alpha_{\text{UCB}}(x; D_t) = \mu_t(x) + \kappa \sigma_t(x)$

where $\mu_t(x)$ is the estimated value of objective function at $x$, $\sigma_t(x)$ is the corresponding predicted standard deviation at $x$ and $\kappa$ is a constant controlling the exploration-exploitation trade-off. Regions with high uncertainty $\sigma_t(x)$ dominate this value. This bias causes excessive sampling in uncertain but low-potential regions, exacerbated by the curse of dimensionality, where sparse data leads to uniformly high uncertainties.

Compared to vanilla GP-based BO, HiBO integrates the global-level sampling potential distribution into this acquisition function. Given partition scores are differentiated by their sampling potentials, acquisition function values of samples $x_{low}$ from low-potential regions will be degraded by weighing them with low partition score $P_{\text{leaf}(x)}$. While samples from promising regions can have amplified acquisition function values. As mentioned, when the clustering and classification algorithms perform well, the acquisition function values of samples of most promising partitions $P^*$ can be greatly enhanced, which guides the search towards more promising regions.

A challenging factor for developing this approach lies in how to control the depth of the constructed search tree. At early optimization stages, when the dataset is limited, overly deep partitions may prioritize overly specific but suboptimal regions. Conversely, a constant shallow tree underutilizes the dataset, resulting in insufficient exploitation. To address this, we designed an adaptive mechanism to adjust the maximum tree depth based on recent evaluation results, balancing exploitation, exploration, and computational cost. While this adaptive mechanism complicates theoretical analysis, it empirically enhances HiBO’s performance and efficiency. Please see Appendix C for a more detailed explanation on its role.

W2 & Q2

W2. Limitations: Empirical study could focus more on identifying instances where HiBO struggles.

Q2. What are limitations of the approach and in which settings would it be expected to perform worse?

Thank you for the question about limitations and scenarios where HiBO may face challenges. From our experiment results and reasoning, we summarized the following two cases:

1. One limitation of HiBO lies in its relatively less pronounced improvement over the local optimizer on dense high-dimensional benchmarks compared to sparse benchmarks. While HiBO demonstrates significant performance gains on sparse benchmarks (e.g. achieving over two orders of magnitude lower regret on tasks such as Hartmann6-300D), its improvement is less evident on dense tasks where all dimensions are uniformly effective (e.g., Levy-200D, Rastrigin-200D). Our interpretation is that in sparse spaces, HiBO efficiently identifies promising regions to guide the local optimizer and it is one of its greatest advantages. While in dense spaces, it requires more samples to achieve accurate estimations across the entire space than in sparse scenarios. But HiBO still converges efficiently towards the optimal point in these tasks and achieves the best in Levy-200D and Rastrigin-200D.
2. HiBO may also struggle in scenarios where certain critical configuration variables are hidden or unobservable during optimization. Such situations can arise in practical tasks where parts of the configuration space are inaccessible due to hardware constraints, legacy systems, or incomplete problem specifications. In these cases, the global-level navigator may build an inaccurate model of the search space, leading to suboptimal partitioning and sampling.

W3 & Q3

W3. Significance: Empirical study of proposed approach limited to using TuRBO as a local optimizer.

Q3. What other local optimisers could in HiBO be used instead of TuRBO and do they also get a performance boost when used with HiBO instead of just on their own?

It is good point about the generalization of our algorithm to guide local optimizers based on other algorithms, and sorry for missing this part when submitting the initial version. Considering almost all BO-based optimization algorithms have their acquisition-function-based strategy for selecting next data samples, the hierarchical feature and the acquisition-based augmentation adopted by HiBO can be applied to any other local optimizer. The choice of TuRBO was inspired by its effectiveness and simplicity for implementation.

Considering this point, we just conducted additional experiments to measure the effectiveness of applying HiBO for guiding the vanilla Gaussian-Process-based BO in synthetic benchmarks and the newly added real-world benchmarks (Mopta08 and Rover Trajectory Planning, see Appendix E in the revised paper). The measurements can be found as the additional brown curve named 'HiBO-GP' and the effectiveness of HiBO can be reflected by the gap between it and the curve for vanilla GP-based BO ('GP', blue curve). Among all the synthetic benchmarks, HiBO-GP consistently achieves better results than vanilla GP though to different extents. It can even achieve the second best regret value in 500-dimensional Branin2 task. In the two newly-added real-world benchmarks, HiBO-GP achieves more than 25% less vehicle mass for Mopta08 and about 2 more reward values than vanilla GP-BO on average, further indicating the effectiveness of our design.

Thank you very much for your insightful suggestions. We hope that our responses and the revisions to the paper adequately address your concerns and clarify any ambiguities. If you feel that these changes improve the quality and clarity of the work, we kindly ask you to consider raising your score. We are happy to provide more clarification if you have additional concerns or feedbacks.

Thank you for your engagement and comments on this page. However, it seems this part of your feedback may have been mistakenly associated with our paper, as it references content that appears unrelated to the submission under discussion of this page

Could you kindly verify whether this comment was intended for another paper? If so, it may help to repost your feedback to the appropriate page to ensure it reaches the correct authors :)