DesignX: Human-Competitive Algorithm Designer for Black-Box Optimization

We want to express our deepest gratitude to the reviewer for recognizing our paper as a novel and interesting work. Besides, we also thank you for the valuable and insightful comments. For your concerns, we provide following point-to-point responses to address them.

[W1&W2&Q1&Q2, Positioning and Related Works Discussion]

We appreciate for the constructive suggestions and agree that key works in related areas should be replenished to enhance the paper’s scientific integrity. To this end, we provide following discussion.

1. L2O: Sharing the same meta learning paradigm, key difference between L2O and MetaBBO locates at the targeted optimization domain: L2O commonly addresses gradient-based optimization problems (so called White-Box), while MetaBBO such as DesignX addresses gradient-free (black-box) scenarios.
2. PbO: While level-0 to level-3 PbO is closely related to some automated algorithm design concepts such as parameter selection, configuration, operator selection, algorithm selection, etc. DesignX shows great similarity with the highest level PbO (level-4): providing designs for every where of a program/workflow. However, two differences remain: a) PbO requires context-based program specification for a specific target scenario, which requires developer to provide these prototypes as much as possible, while in contrast, DesignX provides abstract modular space that includes generic operators adaptable for a wide range of scenarios. b) PbO focuses on improving performance on specific tasks, while DesignX serves as a pivotal step toward foundation model with generalization ability for diverse optimization problems.
3. CASH: CASH methods have been proven effective in many AutoML scenarios, such as selecting machine learning algorithms and corresponding hyper-parameter values for specific tasks. In comparison, DesignX’s target scenarios are a subset of the CASH target scenarios, designing BBO optimizers. However, in terms of methodology, DesignX shows more flexibility in the design process: a) CASH focuses on selecting existing algorithms while DesignX is capable of “creating” new ones. b) CASH determines a parameter setting once for all, while DesignX supports adaptive control.
4. Others: We also have to note that dynamic algorithm configuration, algorithm selection and hyper-heuristic are also closely related to DesignX. Thanks to the long-standing study on them, DesignX draws valuable insights from them and achieves automated end-to-end algorithm design.

Considering PbO and CASH, we decide to change our claim from “the first framework that jointly learns workflow generation and tuning” to “the first deep learning framework that jointly learns workflow generation and tuning” in the revised paper.

We hope the above discussion could address your concerns. We will refine such discussion into the related work of the revised paper.

[W3&Q3, Better Statistical Presentation]

The aggregated results shown in Table 1 and Figure 3 are a deliberate choice to manage the scale of our comparison. Evaluating DesignX against 20 BBO/MetaBBO baselines on thousands of problems generated an immense volume of data, making it impossible to present individual results for all baselines and problems within the confines of the paper. Table 1, therefore, reports average performance on a subset of 20 randomly selected problem instances. Full, disaggregated results have been provided in an Excel spreadsheet within the anonymous GitHub repository (since the time of submission).

In addition, we appreciate the reviewer for this brilliant idea of using rank-based statistical diagrams! Following your suggestions, we have conducted all possible partitions of the tested set. The suggested statistical validation is also conducted and the results show that CMA-ES and our DesignX dominate the other baselines with 95% confidence in all tested cases. We showcase two partitions and the corresponding ranking results of the top 10 baselines below and will add all results to the revised paper (Appendix).

Unimodal & Multimodal

Adequate Global Structure & Weak Global Structure

We can observe that: for easier problems such as those with unimodal or adequate global structure, CMA-ES achieves better performance due to its straightforward convergence. However, for those multimodal and weak global structure problems, our DesignX is capable of discovering more promising algorithms. We thank the reviewer for leading us to a promising way to present the advantages of our DesignX.

For Figure 3, we will provide the same ranking results in Appendix and use a log scale for the x-axis in the revised paper.

[W4&Q4, Ablation Baselines]

We have added CMAES and the SBS as baselines according to your suggestion. We report the average performance of them in the following table.

Results show that 1) DesignX outperforms CMAES, SBS and the ablated baselines. 2) The single optimizer workflow found by SBS cannot defeat the workflows generated by Agent-1 with randomized Agent-2, validating the necessity of generating customized workflows for each problem.

[W5&Q5, Limitation Discussion]

We apologize for the absence of the discussion of limitations. Here we discuss the limitation of DesignX from the four aspects you suggested.

• Computational overhead / resource requirements: We report the average solving time of DesignX, traditional BBO optimizer and MetaBBO basleines below: admit that compared to traditional BBO optimizers, DesignX has higher computational overhead due to additional agents. We report the averaged runtime for one test run on one problem of DesignX, BBO optimizers and MetaBBO optimizers below.

  DesignX	BBO	MetaBBO
  5.5s	2.6s	3.6s

  Indeed, DesignX requires more computational overhead to solve a problem. However, the solving time remains at second-level. Considering that: a) DesignX achieves better optimization performance than baselines, and b) for human users who have limited knowledge of designing optimizers, the time and resources DesignX saves are actually implicit. This reveals certain advantage of our DesignX.

  For hardware requirements, training DesignX requires two 32 cores CPUs and 488G RAM, while using DesinX to infer only requires a single CPU core.

• Conceptual complexity / tunability: Compared with traditional optimizers, DesignX’s architecture and training paradigm are relatively complex. However, we have to clarify that this is the burden of us, DesignX’s developers, and is not in the development loop of DesignX’s users. With DesignX at hand, a user could solve optimization problems without cumbersome algorithm selection and tuning process. For the generalization aspect of DesignX, we admit more future works have to be explored such as a) improving its life-long learning/continual learning through robust fine-tuning strategy; b) in-depth study on relationship between problem space and generalization boundary.

• Evolutionary BBO: Indeed, the primary focus of DesignX is currently restricted within EC methods. However, the methodology of constructing modular space, the universal interface design, and the corresponding training paradigm is generic. Take BO as an example, on the one hand, we could include algorithm modules in BO methods into Modular-EC thanks to our universal interface design. On the other hand, DesignX could also serve as an inspiration for BO developers to construct their BO version DesignX.

• Tuning of hand-crafted baselines: We would clarify that we have tried two baseline settings: a) default settings, that is, following the settings in their original papers, and b) settings fine-tuned by SMAC3. In preliminary experiment we found that the fine-tuned performance is only comparable with the default settings. This is not surprising since the default choices typically have been set by the authors through careful sensitivity analysis. As a result, as stated in Appendix E.1, we use default settings for hand-crafted baselines.

[Paper Formatting Concerns]

We apologize for any confusion. We have addressed the typos you mentioned in the revised paper. The first matrix $\mathcal{W}\_{token} \in \mathbb{R}^{13 \times h}$ in 3.2.1 is a typo. It should be $\mathcal{W}\_{feature}$ indicating the problem feature embedder. The second $\mathcal{W}\_{token}\in\mathbb{R}^{16 \times h}$ is correct, indicating the tokenizer.

Developing DesignX is a long long challenging journey, we really hope this study could appeal more researchers to explore possibility of foundation model for optimization. We hope the above responses could address your concerns and at last turning you back to acceptance side. Thanks for taking your time reading such long long responses. If you have any further questions, we promise timely address them in the next rebuttal phase.

I would like to thank the authors for their extensive reply.

[W1&W2&Q1&Q2, Positioning and Related Works Discussion] That change would close this concern.

[W3&Q3, Better Statistical Presentation]

Thank you for the full set of ranks.

Based on your results, CMA-ES and DesignX are close competitors, but DesignX seems more sample-efficient / to perform better on complicated functions - which is both desirable and interesting.

We think the rank results above would not weaken our claim We have provided the overall and average absolute optimization performance results (based on objective values) in the paper and online

My main concern, at this point, is that I have some trouble reproducing similar results using the data you have originally provided in the excel sheet (anonymous repo):

I found that

• in a pairwise comparison, CMA-ES outperforms DesignX in 2780 of the 3200 test cases.
• average rank (across all 21 methods, averaged across all 3200 functions) is CMA-ES: 3.52 vs DesignX: 7.89. In fact, 4 methods have better average ranks than DesignX: CMA-ES, SADE, LMCMAES, NL_SHADE_LBC.
• based on average performance (objective) across all instances, CMA-ES ranks 1st and DesignX only 10th out of 21 methods. Note that this metric is extremely susceptible to outliers, but still it is a metric used by the authors.

Is this consistent with your observations?

Note: I found the excel sheet somewhat difficult to parse as "value (error)" is provided as text for each method / function pair. I extracted the values and based my calculations on these. My apologies if something went wrong.

[W4&Q4, Ablation Baselines]

We kindly remind the reviewer that the performance metrics have been normalized by a normalization term, which is the optimal found by random search (Appendix E.2). In such case, suppose for a problem, optimal found by random search is 10, while SBS’s optimal is 0.001 and DesignX w/o A2’s optimal is 0.00001, then their normalzied performance is (1-0.001/10) and (1-0.00001/10), which is not significant at all, however, they actually differ by orders of magnitude.

I agree that such differences often matter and that the metric chosen by the authors does not properly reflect them (sensitivity to outliers) - which was my main concern in the evaluation and presentation. Also here, average rank would offer a different (better?) perspective. Could you provide ranks of the ablated versions? (this would also allow non-parametric / rank-based statistical tests)

[W5&Q5, Limitation Discussion]

Thank you for the more detailed runtime data.

We would like to clarify that we did not claim that DesignX becomes the new default sota optimizer. Instead, we think DesignX is the first step to explore the possibility of whether foundational optimization model could relieve the the hand-crafted design process. Currently, DesignX can already be regarded as an alternative alongside sota optimizers such as CMA-ES when a user has little expertise in optimization (considering that our DesignX shows potential in more complex scenarios, and with on average similar running time with CMA-ES, 5.5s v.s. 5.0s). In the future, with the efforts we continuously make to address existing technique bottlenecks, we believe foundational optimization model will finally achieve “fully automated optimization algorithm design”.

I mostly agree with this argument and view this as an acceptable limitation. In my opinion, the original manuscript was not sufficiently transparent about DesignX just being "the first step". If the authors include such argument in the final version, this concern would be addressed.

We are happy that after three rounds of discussion, we have addressed most of the reviewer’s concerns. For the last concerns about the results evaluation/interpretation, we would like to clarify following aspects:

• We first confirm consistent observations with you: 1) “in a pairwise comparison, CMA-ES outperforms DesignX in 2780 of the 3200 test cases”; 2) “average rank (across all 21 methods, averaged across all 3200 functions) is CMA-ES: 3.52 vs DesignX: 7.89”. These results are obtained by comparing averaged per-problem-instance objective values (without normalization), which to some extends, are also reasonable metrics could be used for evaluating BBO optimizers.
• Although in the pairwise comparison, CMA-ES outperforms DesignX on 2780 test cases, this is just one perspective from rank of average objective, however, as we explained in previous rebuttal round, a more reasonable pairwise comparison should be conducted before average. In that case, on 51 independent runs of 3200 problem instances, DesignX outperforms CMA-ES on 68% of 51x3200 columns. The reason that results in such inconsistency is somehow tricky. While DesignX has 68% possibility to outperform CMA-ES with small performance gap, it might has 32% possibility to underperform CMA-ES with larger performance gap.
• We believe the above reason could also explain “average rank (across all 21 methods, averaged across all 3200 functions) is CMA-ES: 3.52 vs DesignX: 7.89”.

As we discuss with the reviewer in these four rounds of discussion, we have a clearer and more objective perspective on how to evaluate/measure BBO optimizers. We believe this discussion is very important to our community, since with different metrics, the comparison results of baselines might go to a totally opposite way. This discussion must be added into our paper to appeal more and more participants rethink their evaluation protocols. Thanks for the patience you spend on our paper, your comments/suggestions indeed help us improve our paper, and more importantly, ourselves. We would like to reflect your valuable contribution for our paper to area chair, you deserve it!

We appreciate the reviewer for your positive and valuable feedback. For the remaining concerns, we address them as below.

[W1, Computational Resource Requirement]

Thank you for your valuable suggestion, we would clarify that the computational resources used for DesignX's training are reported in Appendix E.1: we train DesignX on two Intel(R) Xeon(R) 6458Q CPUs with 488G RAM.

[W2, Abbreviation Issue]

We apologize for the overuse of abbreviations. We will follow your suggestion and include the full names of all abbreviations upon their first use in the revised paper.

[Q1&Q2, Extendibility and Transferring]

Modular-EC is an extensible framework where modules are organized within a hierarchical inheritance architecture and universal interfaces. If more advanced expert-designed BBO algorithms are available, users can decompose algorithms into modules, integrate these modules into Modular-EC by inheriting its base classes and implementing modules using universal interfaces. This approach also works for integrating modules tailored to specific types of BBO problems, such as Bayesian optimization modules for solving expensive problems. With these new modules, similar to ConfigX [1], users can conduct life-long learning to adapt the DesignX model to new modules and solve problems of interest with advanced expert-designed BBO modules.

[1] Guo, Hongshu, et al. "Configx: Modular configuration for evolutionary algorithms via multitask reinforcement learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 25. 2025.

[Q3, Optimal Objective Value]

The training problems are synthetic problems from well-known optimization benchmarks, including the CoCo BBOB and CEC benchmarks. These problems have known optimal objective values. In this paper, these values are 0. If the optimal objective value for the target BBO problem is not yet available (e.g., for realistic benchmark problems), we use the best objective value found by all baselines across all runs as the "optimal objective value".

[Q4, Dimensionality Sensitivity]

In this paper, Modular-EC integrates BBO modules tailored to low-dimensional optimization. For large-scale optimization where dimensionality can exceed 1,000 or even 10,000, DesignX with the current Modular-EC cannot handle these problems. As mentioned in the response to Q1 & Q2, Modular-EC is an extensible framework. We can integrate large-scale optimization modules and conduct transfer learning to enable DesignX to solve high-dimensional problems, which is a future work.

[Q5, Efficiency on Expensive Problems]

BBO and MetaBBO optimizers require objective values during optimization; therefore, expensive problems reduce their efficiency. Since DesignX’s reward is calculated using objective values obtained directly from the low-level optimizer without extra evaluations, its incurred slowdown is equivalent to that of both BBO optimizers and other MetaBBO optimizers. To address expensive evaluations, surrogate-based methods have been extensively studied. DesignX could integrate these optimization modules to enhance efficiency for expensive problems.

We hope the above responses could address your concerns and anticipate your timely feedback.

We sincerely appreciate the reviewer for your insightful comments and valuable suggestions. For your concerns, we provide following point-to-point responses to address them.

[W1, Design Space Extendibility and Coverage]

• Coverage of Modular-EC: we would like to clarify that the operator pool choice and action length choice have provided a wide coverage over existing single-objective optimizers. For operator pool, the 116 submodules are carefully collected by us from decades of literatures and representative BBO optimizers. We provide two DE examples below to showcase Modular-EC could express either simple structure or very complex modern optimizers:

  1. a simple 1997 DE/rand/1/bin combines:
     1. Uniform initialization module
     2. rand/1 mutation module
     3. binomial crossover module
     4. Clip boundary control module
     5. DE-like selection module
  2. a very advanced and complex 2022 NL-SHADE-LBC optimizer uses:
     1. Uniform initialization module
     2. current-to-pbest/1 with archive mutation module
     3. binomial crossover module
     4. Halving boundary control module
     5. DE-like selection module
     6. Non-linear population reduction module

  For action length choice (12), we believe it is enough for existing or any potential operators, e.g., even the latest DE algorithms such as NL-SHADE-LBC (winner of IEEE CEC Congress) holding 3-4 parameters per module.

• Extendability for Outlier Problems: On the one hand, thanks to our proposed universal interfaces and inherentance architecture in Modular-EC, we believe extending it to embrace other novel algorithmic modules is quite easy to achieve. On the other hand, the 16-bit binary module ID spans a huge module space, many of which are still reserved for future module extension. We agree with the reviewer that extending (or more specifically fine-tuning) DesignX for complex optimization domains still require efforts such as novel fine-tuning strategy. For such point, we cite the paper ConfigX [1], where life-long learning ability of similar modular system has been validated. We will regard this as an important future work to enhance the scalability of DesignX.

[W2, Agent Model Comparison]

Agent-1 and Agent-2 use separate GPT-2 models without shared parameters. We will clarify this in the revised paper.

1. Similarity: the internal GPT-2 blocks share same architecture.
2. Difference: First, the input and output of the two agents differ with each other: for Agent-1, input is the problem feature, output is the softmax distribution of submodules; for Agent-2 input is the concat of submodule-id and optimization status, output is the sample distribution of hyper-parmeter values. Second, the feedforward logics are different: for Agent-1, the GPT-2 auto-regressively output submodules; for Agent-2, it is a self-attention among submodules in the generated workflow.

[W3, Designing Time]

We would like to clarify that when we claim “designing such optimizers requires up to month”, we mean that for humans, designing an optimizer and fine-tune its hyper-parameters to achieve maximal perforamence gain on the target problem might consume days to months. Humans may select a BBO optimizer suitable for the target problem according to their expert-knowledge on optimization, design optimizer’s workfow, fine-tune the hyper-parameters and evaluate the optimizer on the target problems. Such process would repeat over and over again until the optimizer achieves desired performance on the target problem, which is really time-consuming.

In contrast, our trained DesignX only consumes seconds to infer an algorithm and optimize the given problem. If we understood your question wrongly, please give us timely feedback so we can discuss with you more on this issue.

[W4&Limitations, Discussion on Limitation]

We appreciate your careful reading and constructive suggestion. We will replenish a limitation discussion section into the appendix of the revised paper to enhance this paper’s scientific integrity.

While Designx has been validated as an effective and novel framework to learn automated algorithm design policy, it still hold two limitations and anticipates further efforts in the future to address them.

1. In this paper, the primary optimization domain is restricted within single-objective optimization. However, optimization domains such as multi-objective optimization requires more complex algorithm modules and corresponding training problems, which challenges the extensibility of DesignX. Thanks to the uniform interfaces we have developed in Modular-EC system, complex modules in such optimization domain can be easily integrated, and life-long learning/continual learning techniques might be a promising way to make DesignX embrace diverse optimization domains.

2. The training efficiency of DesignX is not very ideal. It still takes us 6 days to train DesignX on high-performance PC. Further improvement such as Multi-card parallel mechanism, proximal training objective construction, architecture simplefication/distillation could be regarded as important future works.

We hope the above discussion could address your concerns. We will refine such discussion into the revised paper.

[Q1, Rationale of Transformer Architecture]

Let us explain the rationale behind our Transformer choice for you. First, we actually have conducted prelimary experiments on other neural network architectures, to be specific, MLP and LSTM architectures. Four experiments were conducted:

• Agent-1 (MLP), Agent-2 (MLP): this experiment fails since MLP can not generate algorithm workflow in an auto-regressive fashion.
• Agent-1 (MLP), Agent-2 (LSTM): same as above, fails.
• Agent-1 (LSTM), Agent-2 (MLP): we observe two issues in this experiment, during training of Agent-1, the LSTM encounters gradient explosion due to the MDP of Agent-1 is an reward-delayed one, hence the policy gradient might be too much for LSTM. The parameter control performance of MLP-based Agent-2 is far worse due to MLP can not capture the topology dependency within an complete algorithm workflow.
• Agent-1 (LSTM), Agent-2 (LSTM): Agent-1 still encounters gradient explosion issue while LSTM-based Agent-2 performs better than MLP-based one. However, it still underperforms DesignX.

Second, these preliminary results coherent with a recent MetaBBO work ConfigX [1], where the ablation results also reveal that Transformer architecture achieves superior modelling capabability than MLP and LSTM.

[Q2, Training Hardware]

We indeed train our Design on 2 32-core CPUs with 488G RAM. Our primary consideration regarding GPU deployment for agents involves the trade-off between GPU acceleration of network operations and CPU-GPU transfer overhead. The total number of parameters of DesignX’s Agent-1&2 are 100k. For such relatively small models, GPU acceleration cannot offset the overhead introduced by excessive CPU-GPU transfers. Therefore we do not use CPU-GPU mixing strategy. More importantly, during the testing, for a single testing problem, 1 CPU core is enough to infer the algorithm workflow and do hyper-parameter control in the subsequent optimization episode. However, for future work involving larger training sets or models with more parameters, GPU deployment may be efficiency.

We hope the above responses could address your concerns and anticipate your timely feedback.

[1] Guo, Hongshu, et al. "Configx: Modular configuration for evolutionary algorithms via multitask reinforcement learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 25. 2025.