SOO-Bench: Benchmarks for Evaluating the Stability of Offline Black-Box Optimization

• Lack of solution space exacerbating the OOD problem: In the experiments, parts of the solution space may not be covered by the offline dataset, which restricts the algorithm’s exploration capabilities during the optimization process. As a result, the algorithm struggles to identify potential optimal solutions within the solution space. This limitation often leads to the "OOD" problem, where the algorithm’s performance in new optimization tasks is constrained by the incomplete representation of the solution space.
• The offline dataset’s optimal solutions are already highly competitive: The offline dataset itself contains solutions that are close to the global optimum. With such high-quality data, the algorithms have limited room for further optimization, making it difficult to achieve solutions that significantly surpass the current optimal results.

Q2 About SI metric.

A2 Thank you for your insightful feedback. FS represents the optimal solution found by the algorithm after the optimization is completed. This indicator is intended to indicate optimality.

SI quantifies the stability of an algorithm by comparing its performance to two reference points: an extremely conservative baseline and an ideal algorithm. The conservative baseline represents the scenario where only the optimal offline dataset is obtained, while the ideal algorithm achieves both perfect optimization and stability returning the best possible solution. The SI value is calculated as the difference between the performance of the given algorithm and the conservative baseline, divided by the difference between the ideal algorithm and the conservative baseline. This metric effectively captures the algorithm's stability throughout the optimization process.

Regarding the issue of precocious behavior, we believe that both FS and SI should be considered comprehensively. We do not underestimate the importance of FS. As noted in line 460, when SI is high but FS is low, we do not consider the algorithm to be performing well. Ideally, we aim for future algorithms to achieve stability throughout the entire optimization process while maintaining high FS for maximization problems—this would result in both high SI and high FS.

Regarding stability, we would like to clarify that we typically set a sufficiently large step size to avoid the issue you mentioned. Additionally, we want to emphasize that many previous offline optimization algorithms experience performance degradation as optimization progresses, what we don't want to see. Our proposed indicators SI can reveal this phenomenon.

Even in the situation you described, where SI may be lower, the FS might still be relatively high, and this scenario would be acceptable because our goal is to achieve the maximum performance while ensuring stable performance over time. It is important to clarify that in maximization tasks, algorithms with high FS and SI values are considered advantageous. Conversely, in minimization tasks, algorithms with low FS and high SI values are deemed more favorable.

[1] Trabucco B, Geng X, Kumar A, et al. Design-bench: Benchmarks for data-driven offline model-based optimization, International Conference on Machine Learning, 2022.

Thank you for your efforts in providing valuable comments on our work. We appreciate the reviewer's comment about the points that should be further considered from the aspect of our experimental point of view. We have thoroughly considered the concerns raised by the reviewer and revised our manuscript as follows (we also recommend the reviewer check the global response).

List of changes in the manuscript:

1. Appendix D are revised to add additional experiments according to Q1.

Q1 About experiment setting.

A1 Thank you for your valuable suggestion. In response, we have narrowed down different regions of the solution space, and the experimental results can be found in Appendix D.2, highlighted in blue. However, we would like to emphasize that the goal of offline optimization is to stably surpass the offline dataset during optimization procedure. When there is a significant lack of high-quality $y$ values, the algorithm often faces a serious out-of-distribution generalization problem, which is precisely the challenge that previous offline optimization algorithms have aimed to address. For this reason, we chose to process the y values in our approach, similar to how the first offline optimization benchmark "design-bench" [1] also processed $y$. The different solution distribution results in GTOPX unconstrained scenario can be found in Table 1.

Table 1: Overall the different solution distribution results in GTOPX unconstrained scenario. Results are averaged over five times, and “$\pm$” indicates the standard deviation. $f(x_{OFF})$ means the optimal objective function value in the offline dataset. FS (i.e., final score) means the function value that an offline optimization algorithm finds in the final step during optimization process. FS measures optimality while SI measures stability.

The experimental results in Table 1 show that the FS values of all algorithms are close to the optimal solutions of the offline dataset, indicating that under this experimental setting, it is challenging for the algorithms to further improve upon the current solutions. The main reason why FS is the same but SI is different is that there is a difference in the decimals after FS, although rounding is omitted. This phenomenon can be attributed to two key factors:

Dear Reviewer 8L7a,

We express our sincere gratitude for your invaluable comments and constructive feedback on our paper. We understand that you have a busy schedule, and we kindly remind you that the revision deadline is approaching. During the rebuttal period, we diligently addressed your concerns by providing point-to-point responses, which included the incorporation of new experiments (such as the different solution distributions) , and the clarification on the expression of SI metric. If you have any further questions or concerns, please do not hesitate to let us know. Moreover, if you find our response satisfactory, could you please kindly consider the possibility of updating the rating. Thank you very much for your valuable suggestion.

Thank you for your attention to our work, and we look forward to your response.

Best regards,

The Authors.

Q2: About SI metric.

A2: Thanks for your constructive suggestions. First and foremost, we would like to thank the reviewer for their valuable feedback. Initially, our focus was primarily on algorithm stability, an aspect that is often overlooked in existing research. Our original approach was to evaluate algorithm stability in conjunction with FS to provide a more comprehensive assessment of algorithm performance. After considering the reviewer's suggestions, we realized the importance of accounting for performance improvement, and we have since revised our approach to better address the identification of increasing performance. In the revised version, we have incorporated this idea to ensure that new metric better reflects performance improvement in various contexts.

We greatly appreciate the reviewer’s insight, which has been instrumental in refining our work and making significant improvements to the paper.

Specifically, to provide more direct representations in each aspect, we further propose two independent indicators to represent the optimality and stability, respectively.

Optimality Indicator (OI) compares a given algorithm $A$ to a hypothetical algorithm $A_d$ that returns the best solution in offline static data in every step $t$. The algorithm $A_d$ is considered as a fundamental baseline that is extremely conservative and any effective algorithm is expected to surpass it. Hence, OI indicates the optimality of $A$ by showing how well it outperforms $A_d$. Formally the OI of $A$ is calculated as $OI(A) = \frac{S_A}{S{A_d}}$ , where $S_A$ represents the shaded area under the curve of $A$ and $S{A_d}$ represents the shaded area under the curve of $A_d$. The OI indicates the optimality of algorithm $A$ by showing how well it outperforms the hypothetical baseline algorithm $A_d$.

Stability Indicator (SI) compares $A$ to another hypothetical algorithm $A_o$ that always returns the best solution encountered during the entire optimization procedure of $A$ at every step t. The algorithm $A_o$ is considered as an ideal algorithm that achieves both the best optimality and stability (i.e., consistently returns the best solution). Therefore, SI indicates the stability of $A$ by measuring how close its solution scores are to those of the ideal algorithm $A_o$. A higher SI value implies that in more steps, the scores of the solutions generated by algorithm $A$ are closer to the best solution, indicating higher stability. Conversely, a lower SI value suggests greater variability in the solution scores and lower stability. Formally the SI of $A$ is calculated as $SI(A) = \frac{S_A}{S_{A_o}}$ , where $S{A_o}$ represents the shaded area under the curve of the hypothetical ideal algorithm $A_o$.

Thus, we introduce a novel metric, Stability-Optimality metric (SO) in Definition 1, for offline MBO to assess the model's capability in identifying the stable and optimal one from a set of candidate designs in the whole optimization process.

Definition 1 (SO metric for Offline MBO): For each optimization step $t \in \lbrace \{1, 2, \cdots, T \rbrace \}$,

The function $w(t)$ decreases gradually over time. In the early stages, greater emphasis is placed on achieving optimality, while in the later stages, the focus transitions toward stability. Inspired by F1 score, the SO metric is defined as:

The SO metric for offline MBO can effectively evaluate a model's ability about stability and optimality. A higher SO value indicates better performance in optimality and stability. In the early stages of optimization, the focus is on achieving optimality, as we aim to make significant improvements during this phase. As the optimization progresses and reaches later stages, the priority shifts toward maintaining stability, ensuring that performance does not degrade significantly due to out-of-distribution generalization, even after extensive optimization.

Thank you very much for your careful review and feedback!

Q1: About solution space distribution.

A1: Thank you for your valuable comments. Black-box optimization has been developed and applied across a wide range of fields. However, its widespread adoption in real-world applications has been hindered by the need for frequent evaluations. In many tasks, evaluating the objective function is costly, risky, or even infeasible. Although active evaluations of new solutions may not always be feasible, historical data in the form of solution-function value pairs is often available. Offline optimization, which does not require real-time querying of solutions, aims to leverage this offline data to recommend high-quality solutions for real-world applications. In many domains, such as experimental chemistry, researchers cannot explore the entire solution space. Instead, they typically make slight adjustments based on prior experiments, meaning the solution (denoted as $x$) may remain similar, while the objective value $y$ changes. One of the primary challenges in offline optimization is that this optimization process carries a risk of performance degradation due to the overestimation error in the surrogate model [2, 3, 4, 5]. A well-trained surrogate model can provide accurate predictions for solutions near the offline data, but it may make significant errors when predicting out-of-distribution (OOD) solutions [2, 3, 4, 5]. As a result, optimization algorithms tend to explore unseen OOD solutions in the hope of discovering better-performing solutions (i.e., solutions that outperform the best ones in the offline dataset). However, as the optimization progresses, the risk of performance degradation increases, and the quality of the found solutions may decline. Without active evaluation feedback, it becomes difficult to determine when to stop the optimization and return the final solution. Therefore, the core goal of offline optimization is to stably surpass the offline dataset during optimization procedure.

That is to say, the goal is to identify the best possible solutions (i.e., those with high $y$-values) based on historical data. This goal cannot be effectively achieved by simply adjusting the distribution of the solution space. As noted in Line 37 of the paper, the objective function is typically a black-box function, meaning we lack prior knowledge of which regions contain the global optimum. Excessive exploration of the solution space would not only be resource-intensive but may also fail to identify the global optimum. Additionally, uniform sampling of the solution space could inadvertently cover regions containing the global optimum. If regions near the global optimum are included in the dataset, offline optimization algorithms, which are proposed to solve OOD issue, might fail to distinguish between different solutions.

While obtaining the global optimum may not always be feasible, we can identify regions with relatively high $y$-values and exclude these areas from the dataset. This ensures that the dataset avoids regions near the global optimum, which is essential for constructing a more reliable offline dataset tailored for offline optimization.

Therefore, we believe it is more reasonable to process the dataset based on $y$-values, similar with famous benchmark design_bench in offline optimization [1]. While the approach we propose is relatively simple, it effectively addresses the distribution issue. We also provide guidance on how to adaptively construct a dataset on GitHub [link], where users can customize their datasets.

To better clarify, let’s consider the following three sequences:

A: [0.4, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5]

B: [0.4, 0.5, 0.5, 0.6, 0.6, 0.7, 0.7, 0.7, 0.8, 0.8]

C: [0.4, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8]

As you pointed out, in our previous indicator SI, we considered sequences A and C to be very stable, which led to the same SI value of 0.944 for both. However, we did not account for the actual FS value. In contrast, for sequence B, which is steadily increasing, an SI value of 0.583 at this stage. The new indicator we proposed addresses all the issues you raised previously, balancing both stability and optimality. For the A, B, and C sequences discussed earlier, assuming the total step length $T=200$, consistent with our previous settings, the SO values at this stage are 2.442, 3.017, and 3.705, respectively. This outcome aligns with our intuitive expectations. In summary, Sequence B represents the algorithm with the increasing performance you mentioned. This is the issue you pointed out earlier, which SI can not address. Our newly improved metric SO effectively resolves this problem. Additionally, the precocious behavior you mentioned can lead to poor performance, as seen with Sequence A, which receives a low value with the new metric SO. We are confident that our improved indicator addresses all of your concerns.

We agree that the new metric SO can validate offline optimization approachs on our benchmark with diverse backgrounds strengthens the paper's credibility. Here we only show the GTOPX unconstrained scenario. To further simulate a narrow distribution, the experimental results for the cases where $m=25$% is missing near the worst value and $n=75$% is missing near the optimal value are as follows:

Table 1: Overall results on GTOPX unconstrained scenario. Results are averaged over five runs. $f(x_{OFF})$ represents the optimal objective function value in the offline dataset. FS means the function value that an offline optimization algorithm finds in the final step during the optimization process. SI is our previous metric, while SO, our new metric, evaluates both optimality and stability.

By comparing the results of FS and SI , we found that our new metric SO can take both optimality and stability into consideration. Since GTOPX unconstrained tasks are minimization tasks, SO should be as small as possible. we observed the advantages of the ARCOO algorithm, which can dynamically control the optimization risk during the optimization process. Existing offline BBO methods (e.g., CBAS, ARCOO) perform well across different tasks, except for TTDDEA. This is because TTDDEA, being an evolutionary-based method, relies on high-confidence data augmentation without considering the out-of-distribution issue. All these experimental results validate the importance of steady improvement during the optimization process.

Due to time constraints, we do not have time to revise the full text. We will revise the experiments and expressions of the full text later. Thank you for your reply.

Q3: About typos.

A3: Thanks for pointing these out. About how to generate these solution distributions and the meaning of "Low-high" values, we have explained them in Line 1536-1546. "Low-high" have been modified to "Left-Right”, which means right-$n\%$ of the solution space range and the left-$m\%$ of the solution space range are removed. We have revised them and proofread thoroughly according to your suggestions. Thank you very much.

About how to generate these solution distributions: First, we provide a black-box ground-truth oracle objective function for each task. An initial dataset is obtained by uniformly sampling and evaluating the objective function. Datasets of different difficulty levels are constructed based on this sorted initial dataset. Specifically, the right-$n\%$ of the solution space range and the left-$m\%$ of the solution space range are removed to show different solution space distributions. Through the above steps, an offline dataset with a narrow distribution in real tasks is constructed by removing solutions.

To simulate a more realistic data distribution, we choose a dataset size that is 1000 times the variable dimension. At the same time, to further simulate the narrow distribution, missing the $m\%$ and the $n\%$ are used to construct different solution distributions. In this paper, we select the middle 50% of the data (i.e., $m\%-n\%=50\%$) to construct a simulated dataset as a reasonable baseline. Since the proposed benchmark is highly flexible and customizable, it enables users to modify the data volume, $m\%$, and $n\%$ as needed.

[1] Trabucco B, Geng X, Kumar A, et al. Design-bench: Benchmarks for data-driven offline model-based optimization, International Conference on Machine Learning, 2022.

[2] Han Qi, Yi Su, Aviral Kumar, and Sergey Levine. Data-driven model-based optimization via invariant representation learning, in Advances in Neural Information Processing Systems 35, 2022.

[3] Brandon Trabucco, Aviral Kumar, Xinyang Geng, and Sergey Levine,. Conservative objective models for effective offline model-based optimization, in Proceedings of the 38th International Conference on Machine Learning, 2021.

[4] Ye Yuan, Can (Sam) Chen, Zixuan Liu, Willie Neiswanger, and Xue (Steve) Liu. Importance-aware co-teaching for offline model-based optimization. In Advances in Neural Information Processing Systems 36, 2023.

[5] Taeyoung Yun, Sujin Yun, Jaewoo Lee, and Jinkyoo Park. Guided trajectory generation with diffusion models for offline model-based optimization. In Advances in Neural Information Processing Systems 37, 2024.

If there are any issues, please feel free to reply, and we will respond promptly.

Dear Reviewer 8L7a,

As the deadline approaches, we sincerely hope to address your concerns and discuss the rebuttal with you further. If you have any questions, please feel free to ask directly. Moreover, if you find our response satisfactory, could you please kindly consider the possibility of updating the rating. Thank you very much for your valuable suggestion.

Best regards,

The authors

Table 2 and 3 shows performance of CCDDEA in the different size of the offline dataset and the different solution distribution. They are proposed by other reviewers of this ICLR paper. We also use CCDDEA as the baseline and put the method description and pseudocode of CCDDEA in Appendix B. We found that even in complex settings, such as changing the amount of data or changing the distribution of the solution space, CCDDEA is still a relatively competitive method in terms of FS indicators.

Table 2: Overall results for different sizes of the offline dataset using the CCDDEA method in the GTOPX unconstrained scenario. Details are the same as Table 1.

Table 3: Overall results for the different solution distribution using the CCDDEA method in the GTOPX unconstrained scenario. Details are the same as Table 1.

We sincerely appreciate your efforts in reviewing our paper. We are pleased that the reviewer noticed the strengths regarding the SOO-Bench. We have carefully reviewed the weaknesses pointed out by the reviewer and have revised our manuscript as follows (we also recommend the reviewer review the overall response).

List of changes in the manuscript:

1. The typos are fixed according to Q1.
2. Evaluate new offline optimization algorithm CCDDEA on our benchmark according to Q3.

Q1 About typos.

A1 Thank you for pointing out the typos. We have fixed it in the updated PDF, and the correction is highlighted in blue.

Q2 About FS and SI metric.

A2 Thank you for your questions. In the performance evaluation of offline optimization algorithms, both FS and SI indicators must be considered. In general, for maximization problems, the goal is to make FS and SI as large as possible. However, the trade-off between these two indicators can vary depending on the specific problem. For example, in some tasks, the focus may not be on finding the highest possible solution but rather on ensuring the stability of the solution. In such cases, more attention should be given to the SI indicator. This process is akin to finding the Pareto optimum between the FS and SI indicators, where different preferences will lead to different outcomes. But we want to restate that high FS and SI are the ultimate goal in the maximization problems.

Q3 About submission history.

A3 Thank you for your question regarding the paper’s submission history. Indeed, the NeurIPS score was high, with three positive reviews and one borderline negative review. However, the negative review did not adequately involve in the rebuttal discussion, which ultimately led to the Area Chair's rejection. This outcome took us by surprise. We decided to withdraw the paper only after the rejection. Given the short time between the rejection and the ICLR submission, we didn’t have enough time to incorporate the additional experiments mentioned in the NeurIPS rebuttal. We have now included the relevant experiments in the Section 6.1, Table 2, 5, 9, Appendix B.1 only for GTOPX tasks. We are making further improvements to the manuscript for other tasks now. The CCDDEA results in GTOPX unconstrianed scenario can be found in Table 1.

Table 1: Overall CCDDEA results in GTOPX unconstrianed scenario. The distribution m-n means the missing m% near the worst value and the missing n% near the optimal value. Results are averaged over five times, and “$\pm$” indicates the standard deviation. $f(x_{OFF})$ means the optimal objective function value in the offline dataset. FS (i.e., final score) means the function value that an offline optimization algorithm finds in the final step during optimization process. FS measures optimality while SI measures stability.

As shown in Table 1, although the SI of CCDDEA is relatively poor, this is mainly because the task is difficult and it is easy to overestimate when exploring the OOD area. However, compared with other evolutionary algorithms such as TTDDEA, it is found that CCDDEA can be closer to the optimal solution.

Q2 About benchmark setting.

A2 Many thanks for the constructive comments. We would like to highlight that the key difference between our previous offline optimization benchmark “design-bench" is our ability to adaptively adjust the data distribution and our in-depth thinking on the essential goals of offline optimization (i.e., the goal of offline optimization is to stably surpass the offline dataset during optimization procedure). This feature allows the algorithm to explore performance more thoroughly. In response to your feedback, we have modified the size of the offline dataset. The updated results are shown in blue in Appendix D of the modified version. Refer to Table 1 for results concerning various sizes of the offline dataset in the GTOPX unconstrained scenario.

Table 1: Overall the different size of the offline dataset results in GTOPX unconstrained scenario. Results are averaged over five times, and “$\pm$” indicates the standard deviation. $f(x_{OFF})$ means the optimal objective function value in the offline dataset. FS (i.e., final score) means the function value that an offline optimization algorithm finds in the final step during optimization process. FS measures optimality while SI measures stability.

As shown in Table 1, the experimental results demonstrate that when the data volume is small (N=100), the performance of all algorithms is significantly affected. For instance, the FS values of TT-DDEA exhibit considerable fluctuations, highlighting the difficulty of effectively optimizing the objective function in low-data scenarios. Moreover, under small data volume conditions, the SI values frequently approach 0 or $-\infty$, further confirming that insufficient data volume undermines the convergence and stability of the algorithms. We also observed that BO and ARCOO are minimally affected by data volume and are able to find better solutions, demonstrating their exceptional capabilities.

Q3 About randomness.

A3 Thank you for your valuable feedback. In fact, the fluctuating performance of the baseline methods is due to the different random seeds used by the algorithm, rather than the task itself.

Thank you for your efforts in providing valuable comments on our work. We appreciate the reviewer's comment about the points that should be further considered from the aspect of our experimental point of view. We have thoroughly considered the concerns raised by the reviewer and revised our manuscript as follows (we also recommend the reviewer check the global response).

List of changes in the manuscript:

1. Appendix D are revised to add results for offline datasets of different sizes according to Q2.

Q1 About SI metric.

A1 Thank you for your questions. We address your questions from the three aspects: the relationship between SI and variance, the limitations of SI in reflecting performance independently, and the issues for evaluating performance directly through p-value.

• About the relationship between SI and variance: Using the sequence [0.1,0.5,0.1,−0.3,−0.3,−0.5] as an example, with the best offline dataset value is 0.1, we illustrate their differences. In this case, SI is −0.417. The negative SI value indicates poor stability, as most solutions fail to surpass the offline optimal value in offline dataset. This reflects that the offline optimization algorithm lacks the stability necessary to surpass the offline benchmark effectively.

  By contrast, the variance is relatively small (0.112), indicating that the values are narrowly distributed around the mean. However, this measure does not reveal that most values are below the best offline dataset value, nor does it evaluate whether the algorithm meets the offline optimization goal (i.e., the goal of offline optimization is to stably surpass the offline dataset during optimization procedure).

  This example highlights the difference between SI and variance. Variance measures dispersion but is agnostic to task-specific objectives. SI, on the other hand, evaluates stability in the optimization process. This differences emphasize why SI is better suited for stability assessment in offline optimization. All in all, variance is a single evaluation metric, which does not capture the relationship between algorithms' results and the best offline dataset value.

• About the limitations of SI in reflecting performance independently: SI measures an algorithm's stability by comparing its performance to two benchmarks: a conservative baseline (optimal offline dataset value) and an ideal algorithm (perfect optimization and stability). This metric effectively captures the algorithm's stability throughout the optimization process. However, offline optimization requires not only stability, it also needs to find the best possible solution, so FS is equally important. We believe that both FS and SI should be considered comprehensively. We do not underestimate the importance of FS. As noted in line 460, when SI is high but FS is low, we do not consider the algorithm to be performing well. Ideally, we aim for future algorithms to achieve stability throughout the entire optimization process (i.e., high SI) while maintaining optimality (i.e., high FS for maximization problems and low FS for minimization problems).

• About the issues for evaluating performance directly through p-value: While hypothesis testing, can provide statistical insights into stability to a certain extent, it falls short in certain critical aspects. Specifically, hypothesis testing cannot distinguish solutions that are worse than the offline optimal value in offline dataset, similar to the limitations of variance. For example, in cases where most solutions fail to exceed the offline optimal value in offline dataset, a good p-value might suggest good performance, but this ignores the crucial fact that the algorithm generates worse solutions than the offline optimal value in offline dataset. In contrast, the SI metric is explicitly designed to evaluate stability in the context of optimization by comparing the algorithm's cumulative performance to the offline optimal value in offline dataset and the optimal solution found by the algorithm. This ensures that SI reflects both stablity and success in surpassing the offline dataset during optimization procedure.

  In summary, while hypothesis testing can complement stability analysis, it cannot replace SI, as it overlooks the iterative nature of optimization processes, while SI provides a goal-oriented evaluation of algorithm performance.

Finally, we would like to emphasize that while variance and p-value can provide some insight into stability, they are single evaluation methods. In contrast, our SI indicator offers a more comprehensive assessment by incorporating both the performance of the optimal solution found by the algorithm and the offline optimal value in offline dataset. This alignment reflects the core goal of offline optimization: to steadily surpass the offline dataset during optimization procedure. This makes SI a more comprehensive and meaningful measure compared to variance or p-value.

Dear Reviewer Lkg5,

We express our sincere gratitude for your invaluable comments and constructive feedback on our paper. We understand that you have a busy schedule, and we kindly remind you that the revision deadline is approaching. Your insights have played a pivotal role in refining our research. We have addressed your concerns in our response, including the incorporation of new experiments (such as the different sizes of the offline datasets) and enhancing the clarity of SI metric. Should you have any additional questions or unresolved issues, please do not hesitate to inform us. We are fully dedicated to promptly resolving any remaining concerns within the author-reviewer discussion period. Moreover, if you find our response satisfactory, could you please kindly consider the possibility of updating the rating. Thank you very much for your valuable suggestion.

Thank you for your attention to our work, and we look forward to your response.

Best regards,

The Authors

Dear Reviewer Lkg5,

As the deadline approaches, we sincerely hope to address your concerns and discuss the rebuttal with you further. If you have any questions, please feel free to ask directly! Moreover, if you find our response satisfactory, could you please kindly consider the possibility of updating the rating. Thank you very much for your valuable suggestion.

Best regards,

Authors

We appreciate your overall positive assessment of our contributions and are grateful for your suggestion. We have already updated the new revision based on your suggestions (we also recommend the reviewer check the global response).

List of changes in the manuscript:

1. Section 1 has been revised to clarify the expression of black-box optimization (BBO).
2. Appendix B.1 are revised to add the details of the evaluated algorithms and pseudocodes following Q2.

Q1 About surrogate model.

A1 Thank you for your insightful comment. You are correct that BBO consists of model-based and model-free methods. In line 39, we mention that "most BBO methods involve constructing a surrogate model." In fact, we do not intend to overlook model-free methods, and we have recognized that the phrase "most of" may be inappropriate in this context. We have revised this in the new version of the paper to provide a more accurate expression.

Our paper focuses on "offline optimization", where surrogate models are often employed to approximate the true objective function. As such, we concentrate more on model-based methods. Even model-free method like CMA-ES typically require constructing a surrogate model [1], [2], [3] in offline optimization. This is because it still relies on an approximation of the objective function to express true objective function to achieve "active" sampling.

We have updated the paper to clarify the expression of BBO and ensure that our focus on model-based methods is properly highlighted.

Q2 About the details of the evaluated algorithms.

A1 Thank you for your suggestions. We have added detailed descriptions of the evaluated algorithms and their pseudocodes in Appendix B.1, highlighted in blue.

[1] Huang P, Wang H, Jin Y. Offline data-driven evolutionary optimization based on tri-training. Swarm and evolutionary computation, 2021.

[2] Gong Y J, Zhong Y T, Huang H G. Offline Data-Driven Optimization at Scale: A Cooperative Coevolutionary Approach. IEEE Transactions on Evolutionary Computation, 2023.

[3] Trabucco B, Geng X, Kumar A, et al. Design-bench: Benchmarks for data-driven offline model-based optimization, International Conference on Machine Learning, 2022.

Dear Reviewer x8NF,

We sincerely appreciate the time and effort you have dedicated to reviewing our work. As the deadline for updating our manuscript is rapidly approaching, we would greatly appreciate your timely feedback on the revisions and clarifications we have provided. During the rebuttal period, we diligently addressed your concerns by providing point-to-point responses, which included the incorporation of more details of the evaluated algorithms and pseudocodes, and the clarification on the expression of black-box optimization (BBO) . Would you mind checking our response (a shortened summary, and the details) ? If you have any further questions or concerns, we would be grateful if you could let us know. Moreover, if you find our response satisfactory, could you please kindly consider the possibility of updating the rating. Thank you very much for your valuable suggestion.

Thank you for your attention to our work, and we look forward to your response.

Best regards,

The Authors

Dear Reviewer x8NF,

As the deadline approaches, we sincerely hope to address your concerns and discuss the rebuttal with you further. If you have any questions, please feel free to ask directly! Moreover, if you find our response satisfactory, could you please kindly consider the possibility of updating the rating. Thank you very much for your valuable suggestion.

Best regards,

Authors