Neural Exploratory Landscape Analysis for Meta-Black-Box-Optimization

[Q1, minimal MetaBBO tasks for good zero-shot performance]

The fundamental principle of the minimal number of MetaBBO tasks for good zero-shot performance is that: it should encompass main operating scenarios of existing MetaBBO methods: dynamic algorithm configuration, dynamic operator selection, and dynamic algorithm selection. This is the reason why we choose three MetaBBO tasks as the minimal training tasks in NeurELA. As MetaBBO continues to evolve, we anticipate that this minimal requirement may increase to accommodate newly proposed operating scenarios. However, the training framework of NeurELA does not need to change. Instead, we simply need to augment the training tasks and perform flexible re-training to adapt to the expanded requirements.

[Q2, insights of the learned features]

Following your valuable suggestion, we have added an additional experimental analysis to further explore the relationship between NeurELA features and traditional ELA features. Specifically, we use the Pearson Correlation analysis to quantify the correlation between each NeurELA feature and each traditional ELA feature. The experimental methodology, results, and corresponding discussion have been updated in the revised Appendix B.3, along with Figure 4. From the correlation results presented in the figure, we observe some notable relationship patterns between our NeurELA features and the traditional ELA features:

a) Four NeurELA features (F1, F4, F8, and F16) are novel features learned by NeurELA, exhibiting weak correlation (< 0.6) with all traditional ELA features.

b) Some NeurELA features strongly correlate with a specific feature group in traditional ELA, such as F3, which aligns closely with the Meta-model group.

c) Some other NeurELA features strongly correlate with multiple traditional ELA feature groups, such as F10, which is highly correlated with both the Distribution and Meta-model groups.

d) All NeurELA features show weak correlation with the Convexity and Local Landscape groups, suggesting these groups are less relevant for addressing MetaBBO tasks.

We appreciate the reviewer for this valuable suggestion, which significantly helps improve the interpretability of NeurELA. We hope the above results and discussion could address your concern. Note: we have also added some text content into the revised paper (line 465-468, colorred in blue) to guide readers to check this interpretation analysis.

[Typos]

We sincerely apologize for our oversight during proofreading and have corrected the typos you mentioned. Additionally, we have conducted a thorough and systematic review of all text content, figures, and tables to ensure accuracy and clarity.

We sincerely appreciate the reviewer for acknowledging our NeurELA as a well-motivated approach with rigorous, clear and comprehensive experimental analysis. We have carefully provided following point-to-point responses to clear your remaining concerns.

[W1, analyse zero-shot failure]

The zero-shot failure refers to the case where NeurELA is integrated into the MetaBBO method GLEET for the BBOB problem set (as shown on the left side of Figure 4), resulting in a performance score of 0.64, which is below expectations. There are two main reasons which lead to this unexpected result: a) the unique and complex attention-based neural network design in GLEET. b) the relatively simpler landscapes of the tested problem instances in BBOB set. We locate the reasons by further examining the output landscape features of NeurELA for the tested problem instances in BBOB and found that the layer normalization in our design would narrow the feature value range. The narrowed feature values further go through the GLEET’s attention layers, which are further narrowed by the layer normalization there. This possibly causes the meta-level policy (GLEET’s actor network, a MLP) confused about the decision bound hence causes the unexpected performance. In contrast, when we integrate NeurELA into GLEET for BBOB-noisy set and Protein-docking set, the zero-shot perforamnce is ideal. This is because the problem instances in these sets have intricate landscapes, with significant differences that remain distinguishable even after being narrowed twice. It is also worth noting that fine-tuning NeurELA with GLEET during its meta-learning process could resolve this problem, which forces the learning of the decision bounds, as shown in the left of Figure 4. Furthermore, this fine-tuning process is very efficient since it only consume 20 training epochs to attain similar performance of the original GLEET baseline.

[W2, training efficiency]

We would like to clarify that, although a limitation of NeurELA is the training efficiency when using ES for the neuroevolution of larger models, the experimental results in our zero-shot performance (Figure 3) and in our inference efficiency (Table 1) have demonstrated that, with a relatively small model, NeurELA achieves more effective landscape feature extraction with less computational overhead than traditional ELA features and a comparable level of computational overhead to the original designs in existing MetaBBO methods.

[W3, theoretical justification of network design]

We provide an intuitive explanation why our neural network can work well here. First of all, if we look at the input and output of the traditional ELA and our NeurELA, they are same: the input is sample points and their objective values {Xs, Ys} and the output is the landscape features summarized from the input. The difference is the calculation rules. In traditional ELA, the rules are a series of human-crafted principles. In contrast, the rules in NeurELA are learned under the tailored training objective we proposed in Eq. (4). the learned neural network-based rules are inhererently more compatible with the target operating scenario: MetaBBO tasks. This is demonstrated by our experimental results (Figure 3).

We also provide a straightforward explanation of the two-stage attention-based network design, which we stated at the beginning of Appendix A.1. There are three motivations: a) generalizability, the neural network should be able to handle optimization problems with different dimensions. b) scalability, the neural network should be sufficiently efficient as the number of sample points scales. c) computational completeness, the neural network should involve computation not only across sample points but also across each problem dimension. To address a), we chose attention-based network which could process any number of sample points by the attention mechanism. To address b), we chose attention-based network since attention mechanism holds highly parallelizable property. To address c), we designed the two-stage attention. The first stage is cross-solution attention, which promotes the feature computation among the sample points. The second stage is the cross-dimension attention, which further promotes the information sharing among different problem dimensions. By doing so, the neural network parameters (the attention query, key, value weights) is trained to extract useful features of the optimization status information.

We hope this explanation addresses your concerns and clarifies the design and functionality of NeurELA.

[W3, high dimensional optimization scenarios]

We would like to clarify that NeurELA’s two-stage attention-based feature extractor is inherently capable of handling optimization problems of any dimensionality. As per your suggestion, we conducted additional experiments to evaluate the zero-shot performance of NeurELA on MetaBBO methods and CoCo BBOB problems with 100 and 500 dimensions. The results are presented in the following tables (Deep-ELA is excluded as it only supports up to 50 dimensions). The results demonstrate that NeurELA effectively boosts the performance of MetaBBO methods on high-dimensional problems by leveraging its generalizable and scalable two-stage attention-based feature extractor.

Results on BBOB-100D

Results on BBOB-500D

[W4, interpretability and relation with traditional ELA]

We would like to clarify that in Section 4.3, Figure 5, we have conducted an initial interpretability analysis on what NeurELA has learned compared with the traditional ELA. This analysis revealed that for a given problem, our NeurELA can provide a clear decision bound during the optimization dynamics, whereas traditional ELA cannot, since it is developed to describe static optimization problem properties rather than dynamic ones.

Following your valuable suggestion, we have added an additional experimental analysis to further explore the relationship between NeurELA features and traditional ELA features. Specifically, we use the Pearson Correlation analysis to quantify the correlation between each NeurELA feature and each traditional ELA feature. The experimental methodology, results, and corresponding discussion have been updated in the revised Appendix B.3, along with Figure 4. From the correlation results presented in the figure, we observe some notable relationship patterns between our NeurELA features and the traditional ELA features:

a) Four NeurELA features (F1, F4, F8, and F16) are novel features learned by NeurELA, exhibiting weak correlation (< 0.6) with all traditional ELA features.

b) Some NeurELA features strongly correlate with a specific feature group in traditional ELA, such as F3, which aligns closely with the Meta-model group.

c) Some other NeurELA features strongly correlate with multiple traditional ELA feature groups, such as F10, which is highly correlated with both the Distribution and Meta-model groups.

d) All NeurELA features show weak correlation with the Convexity and Local Landscape groups, suggesting these groups are less relevant for addressing MetaBBO tasks.

We appreciate the reviewer for this valuable suggestion, which significantly helps improve the interpretability of NeurELA. We hope the above results and discussion could address your concern. Note: we have also added some text content into the revised paper (lines 465-468, colorred in blue) to guide readers to check this interpretation analysis.

[Q2, training done on GPU?]

We apologize for the typo. The statement refers to the two-stage attention mechanism being well supported for parallelization on CPUs by pytorch. We have updated it in the revised paper.

[Q3, wall time comparison]

First, the wall time comparison focuses solely on the feature computation time for NeurELA, traditional ELA, and the original design, which is a subcomponent of the entire MetaBBO workflow, to ensure a fair and accurate comparison. Additionally, we kindly direct the reviewer’s attention to the upper part of Table 1, where we have already included results for 1000-dimensional problems—a significantly large scale. In this case, the computation time for traditional ELA is no longer in milliseconds but approximately 300 seconds. This result clearly demonstrates the superior computational efficiency of NeurELA compared to traditional ELA, even in high-dimensional settings.

Finally, we sincerely thank the reviewer for all valuable comments and suggestions, which have significantly helped us improve the paper quality. We hope the above responses could clear your concerns and look forward to your positive feedback in the remaining time of the rebuttal phase.

We appreciate the reviewer for recognizing our paper being well-structured with good quality. Below, we provide point-by-point responses to address your concerns.

[W1, differences with Deep-ELA]

Thank you for raising questions regarding the distinctions between NeurELA and Deep-ELA. Although we have included a discussion on this issue in Section 2 (lines 187–194), we are happy to expand and clarify the distinctions here.

1. Target Scenario: NeurELA is explicitly designed for MetaBBO tasks, where dynamic optimization status is critical for providing timely and accurate decision-making at the meta level. In contrast, Deep-ELA serves as a static profiling tool for global optimization problem properties and is not tailored for dynamic scenarios. NeurELA supports dynamic algorithm configuration, algorithm selection, and operator selection. In contrast, Deep-ELA’s features are restricted to static algorithm selection and configuration, limiting its adaptability in dynamic MetaBBO workflows.
2. Feature Extraction Workflow: Considering the feature extraction workflow, NeurELA distinguishes with Deep-ELA in two key aspects:
   1. First, NeurELA addresses the limited scalability of Deep-ELA for high dimensional problem . Concretely, the embedding in Deep-ELA is dependent on the problem dimension and hence the authors of Deep-ELA pre-defined a maximum dimension (50 in the original paper). To address this, NeurELA proposes a novel embedding strategy which re-organizes the sample points and their objective values {Xs, Ys} to make the last dimension of the input tensor is 2 (Section 3.2, line 267-291). This embedding format has a significant advantage: the neural network of NeurELA is hence capable of processing any dimensional problem and any number of sample points.
   2. Second, NeurELA enhances the information extraction through its two-stage attention-based neural network. Specifically, when processing the embedded data, Deep-ELA leverages its self-attention layers for information sharing across sample points only. In contrast, NeurELA incorporates a two-stage attention mechanism, enabling the neural network to first extract comprehensive and useful features across sample points (cross-solution attention, lines 296–297) and then across problem dimensions (cross-dimension attention, lines 298–299). This design helps mitigate computational bias and improve feature representation.
3. Training Method: The training objective and training methodology in NeurELA and Deep-ELA are fundamentally different. Deep-ELA aims to learn a neural network that could serve as an alternative of traditional ELA. Its training objective is to minimize the contrastive loss (InfoNCE) between the outputs of its two prediction heads (termed as student head and teacher head) by gradient descent, in order to achieve invariance across different landscape augmentation on the same problem instance. In contrast, the training objective of NeurELA is to learn a neural network that could provide dynamic landscape features for various MetaBBO tasks. Specifically, its objective is to maximize the expected relative performance improvement when integrated into different MetaBBO methods. Since such relative performance improvement is not differentiable, NeurELA employs neuroevolution as its training methodology. Neuroevolution is recognized as an effective alternative to gradient descent, offering robust global optimization capabilities.

In summary, NeurELA and Deep-ELA are significantly different works, with distinct target operating scenarios, algorithm design tasks, neural network designs and workflows, as well as training methodologies. We hope the above detailed explanation would clear your concern. We have added this discussion into the revised Appendix B.2 (colorred in blue). To guide readers to this discussion, we have also updated text content in Section 2, line 187-193 (colorred in blue) of the revised paper.

[W2 & Q1, comparison with Deep-ELA]

Following your suggestion, we have added Deep-ELA as a baseline in our experiments. Specifically, we utilized the open-sourced Deep-ELA model (large_50d_v1, https://github.com/mvseiler/deep_ela/tree/main/deep_ela/models), and the testing followed the same procedure used for NeurELA and other baselines. We have updated the results in Figure 3 of the revised paper. Overall, our NeurELA consistently outperforms Deep-ELA and demonstrates substantial and reliable performance improvements across the tested MetaBBO methods.

Dear reviewer #ARe5:

Since the discussion period is extended, we respectifully request you to check the experimental restults and discussion we have added following your constructive suggestions. We have given a more comprehensive discussion on the differences between our NeurELA and a recent work DeepELA to demonstrate the novelty of this work. We have shown the superiority of NeurELA to DeepELA on a) MetaBBO tasks, b) high-dimensional optimization scanarios. Furthermore, we have statistically analysed the correlation of NeurELA features and traditional ELA features to enhance interpretability. If you have any further instructions, we are open to them and would cooperate with you to make this paper better.

Best regards, the authors

We appreciate the reviewer for acknowledging our NeurELA as a novel landscape analysis framework with reasonable motivation and effective performance. For your remaining concerns, we provide following point-to-point responses to address them.

[W1 & Q1 & Q2, increased computational cost]

We would clarify that although certain computational cost is required to train NeurELA (through neuroevolution on multiple MetaBBO methods and downstream BBO problems), we have demonstrated in the experiments (Figure 3, zero-shot performance) that the trained NeurELA can be seamlessly integrated into existing MetaBBO methods to provide effective dynamic landscape analysis, without further re-training. That is, NeurELA can be regarded as a feature extractor exactly the same as the traditional ELA. We also provide the inference wall time comparison in Table 1 to compare the computational cost required to obtain the landscape feature by our NeurELA and traditional ELA, where the results show that NeurELA require less processing time than traditional ELA, particularly for the high-dimensional problem, this is due to the attention-based neural network which facilitates highly paralleled computation. We believe this explanation could clear your concern in Q1 and Q2.

[W2 & Q3, training convergence]

We have included the training convergence curves of various ES baselines in the revised Appendix, as shown in Figure 3. We kindly request the reviewer to review the results in the newly added Appendix B.1 (colorred in blue), which show that under our setting, the Fast CMAES adopted for training NeurELA converges and achieves superior training effectiveness to other ES baselines.

[Q4, usage scope of NeurELA]

We would like to clarify that NeurELA mainly focuses on population-based BBO paradigm since the two-stage attention-based feature extractor we proposed is designed to process the information of a collection of sample points by first promoting the information sharing among all sample points and then promoting the relationship detection across dimensions. By doing this, we can provide accurate and dynamic landscape feature for the subsequent MetaBBO tasks. We believe NeurELA could be integrated into some human-crafted population-based BBO methods, which requires landscape feature for dynamic algorithm configuration or operator selection. This is supported by our interpretability analysis in Section 4.3, Figure 5. We can observe the clear decision bound of our NeurELA features when profiling the dynamic optimization status. This clear decision bound could serve as useful features for those human-crated population-based BBO methods.

[Q5, dimension of NeurELA features]

The dimension of NeurELA features is 16, which we have stated in Section 4, line 347-348. We have discussed the dimensionality of NeurELA features in the model complexity discussion part (Section 4.4, line 503-517), where we compare the feature dimensions 16, 64 and 128. Due to the limitations of the ES baseline in effectively searching for the optimal NeurELA neural network, increasing the feature dimension could not lead to performance improvement.

[Q6, performance metric definition]

We would like to clarify that we have provided the performance metric definition at the beginning of Section 4.1, line 373-377. It is exactly the exponential of the relative performance we have designed in Eq. (3). Under this setting, the performance of the original MetaBBO method is always 1. For our NeurELA and traditional ELA, a performance value larger than 1 indicates that substituting the original feature extraction design by NeurELA or traditional ELA could improve the optimization performance, and vice versa.

We appreciate the reviewer for the valuable comments. We also thank you for acknowledging NeurELA as a novel and interesting work, with superior performance, convincing ablations and positive code sharing. We provide following point-to-point responses to address your remaining concerns.

[W1 & W2, presentation & writing]

Following your valuable suggestion, we have carefully checked and refined the typos in the revised paper, including the text content, figures and tables. We also agree with the reviewer that a more detailed description of the workflow in the previous section would enhance the understanding of the readers, hence, we have added some text content in the beginning of the introduction of the revised paper (line 039-046, colorred in blue) to this end.

[W3, meaning of “epoch” in Figure 4]

We would like to clarify that the “Zero-Shot” mode of NeurELA refers to integrating the pre-trained $\Lambda_\theta$ into the neural network group of a given MetaBBO method to substitute its original feature extraction mechanism. The neural network of the MetaBBO methods still requires the meta-learning process to learn a useful policy on the training problem set, while $\Lambda_\theta$ is frozen. In contrast, for the “Fine-tuning“ mode, $\Lambda_\theta$ is activated and co-trained as part of the meta-learning process. Hence, we can plot the two performance gain curves along with the training epochs.

[W4, computational overhead]

We argue that, as shown in the top part of Table 1, the computational overhead of NeurELA is in the similar level with the original MetaBBO baselines across different problem dimensions. Moreover, as shown in the bottom part of Table 1, when the number of sample points increases, NeurELA consumes significantly less time to compute the features than the original MetaBBO baselines. We kindly invite the reviewer to examine these results. Considering the consistent performance improvements demonstrated in Figure 3, along with the comparable feature computation wall time, we believe this instead underscores the contribution of our NeurELA.

We sincerely appreciate your positive feedback on our NeurELA! Thanks for the time and efforts you have contributed to improve our paper.

Dear AC #Lg5i:

Following the suggestion from the meta-review and our reviewers, we have changed title of this paper from "Neural Exploratory Landscape Analysis" to "Neural Exploratory Landscape Analysis for Meta-Black-Box-Optimization", which further clarify the scope of our paper and could help future readers more. We would like to note that this change would not harm any content consistency with the previous version. We appreciate your contribution on ICLR and particularly on our paper, thanks!

Best regards, the authors