MetaMetrics: Calibrating Metrics for Generation Tasks Using Human Preferences

Thank you for your thorough comments. We have addressed your questions and concerns as follows:

Limited Contribution and Lack of Out-of-domain (OOD) Generalization Evaluation

We have presented results for both in-domain and out-of-domain settings, specifically highlighting the robustness and generalization capabilities of our methods in Machine Translation, Image Captioning, and Reward Modeling. In Machine Translation, we demonstrate that our metric can generalize to unseen languages, such as Hebrew-to-English (he-en). Additionally, we illustrate the cross-domain and cross-dataset adaptability of our metric in the areas of image captioning and reward modeling. For a detailed overview, please refer to Tables 2 (Machine Translation) and 4 (Reward Modeling), as well as Figure 2 (Image Captioning), which emphasize these findings.

The authors should also clearly present their experimental settings, such as how they create the training, validation and testing sets. For example, the setting for Question Answering tasks is not clear. It would be interesting to add more OOD and real scenarios to test this metric. See following comments for detailed feedback.

We have included comprehensive details for running the experiments, including task-specific splits, in Section D (Datasets) of the Appendix. To enhance clarity, we have also specified the 30%-70% train-test split ratio in Section 4 (Experimental Setup) for your convenience.

GP model performs similarly to CLIP-S and METEOR in the OOD setting. (The statement about the superiority of MetaMetric-Cap over any other individual metric, in lines 426 and 427, is misleading.) This suggests that the use cases for the trained metric are limited, and its performance may be worse than using a single conventional metric.

We would like to clarify that our metric MetaMetrics-Cap outperforms all other individual metrics in Figure 2 (formerly Figure 3 prior to revision). This figure illustrates performance when trained on one dataset and tested on a different, out-of-domain (OOD) dataset. Detailed results are presented in Table 13 (formerly Table 12), where MetaMetrics-Cap (GP) achieves a score of 0.638, compared to CLIP-S at 0.514 and METEOR at 0.481 on the Flickr8k out-of-domain dataset. A similar trend is shown on Thumb1.0, where MetaMetrics-Cap (GP) achieves 0.298, surpassing CLIP-S at 0.292 and METEOR at 0.281, also in an out-of-domain setting.

Lack of Evaluation in Real Settings: The evaluations are limited by the training datasets with human-annotated preference values. It is unclear whether this metric is useful for real applications, as suggested by the authors in lines 41 and 42. For example, the authors could follow a similar evaluation procedure to DPO and PPO papers, using the proposed metric to evaluate various LLMs - before and after RLHF-tuned - and compare their win rates based on MetaMetric.

Our experiment was conducted in a low-resource setting, requiring only a minimal number of samples during calibration. Specifically, we utilized 30% of the dataset for training while reserving the remaining 70% for testing. We believe this experimental setup effectively simulates real-world scenarios where human-annotated preference data is scarce, demonstrating that only a small portion of datasets needs to be human-annotated.

Reference-based and Reference-free Settings: In line 83, the authors claim to design the metric for two settings: reference-based and reference-free. If I understand correctly, this depends on the choice of metrics rather than the model design. If the chosen metrics require references, then MetaMetric can only be reference-based. In the experiments, the authors only distinguish these two settings for Machine Translation, not for other tasks. It would be better that the authors add both settings for all the tasks.

We would like to clarify that not all tasks utilize reference-free metrics. In particular, for the machine translation task, reference-free settings are commonly used for evaluation. In contrast, other downstream tasks primarily rely on reference-based metrics. Therefore, our reference-free setting is exclusively applied to machine translation.

Presentation Issues

As per your suggestion, we have moved all table captions to appear above their respective tables. Additionally, we have moved Figure 2 to the Appendix, as it serves a supplementary role, and have updated the text accordingly. The previous reference to Table 9 was an error; it has been corrected to refer to Table 1, where the relevant data is actually presented.

The first paragraph is a bit confusing. The authors discuss the shortcomings of BLEU and BERTScore in the main text, but Figure 1 presents BLEU and CLIP-S. Additionally, what criteria are used in Figure 1 to determine if a metric is good or not? For the figure on the right hand side, why are BLEU 0.2 and MeMetric 0.4 considered good, but CLIP-S 0.6 is considered bad? What is the threshold?

The figure compares BLEU and CLIP-S to illustrate metric alignment with human judgments, not to advocate for specific metrics. A "good" metric aligns closely with human ratings, scoring below 0.5 for low-quality instances and above 0.5 for high-quality ones. The apparent discrepancy (e.g., BLEU 0.2 and MetaMetrics 0.4 as "good" vs. CLIP-S 0.6 as "bad") reflects differences in alignment with human ratings.

The authors state, "models that optimize for human preference ... have demonstrated superior performance." What metrics have these models used to demonstrate human alignment? The authors avoid discussing this in their paper, instead comparing their metric with conventional metrics not originally designed for human preference alignment. For example, Ouyang et al. [1] used the Perspective AI score to measure toxicity.

We wish to clarify that the statement in question was made based on the references provided. We added the statement according to the findings on the references. We didn’t intend to avoid any discussion regarding this matter. Let us know if you have further questions or clarification, we are happy to answer your question.

In line 42, the authors mention "evaluation metrics that accurately reflect human subjective judgements." Since human judgements are subjective, is it possible that a metric can accurately measure these varied judgements across different individuals?

Automatic evaluation metrics serve as a proxy for human evaluation, which can be costly and time-consuming. Therefore, using automatic evaluation metrics is essential to offset the expenses and time associated with collecting human evaluations.

In line 53, the statement relating the necessity of human aligned metrics to the complexity of tasks seems implausible. Regardless of task complexity, we always seek metrics that align with human evaluations.

Thank you for bringing this to our attention. We wish to clarify that our intent was to highlight the absolute necessity of human-aligned metrics in navigating the intricate, multi-aspects of human preference inherent in complex downstream tasks. This should not be misconstrued as implying that such metrics are exclusively requisite for only the more sophisticated or challenging tasks. Rather, we concur with the notion that human-aligned metrics carry significant importance across the entire spectrum of downstream tasks, irrespective of their respective levels of difficulty or complexity.

Universality of MetaMetric: For conventional metrics (e.g., BLEU, ROUGE, METEOR), they are used across various natural language generation tasks. Is it feasible to have a single MetaMetric applicable to all natural language generation tasks?

We believe that even conventional metrics have advantages and disadvantages on each task. For example, BLEU is not well-correlated to human preferences in machine translation[1].

[1] Freitag, M., Rei, R., Mathur, N., Lo, C. K., Stewart, C., Avramidis, E., ... & Martins, A. F. (2022, December). Results of WMT22 metrics shared task: Stop using BLEU–neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT) (pp. 46-68).

Choice of Training Methods: Why did the authors choose Bayesian Optimization and Boosting to train MetaMetrics?

In this work, we focus on two optimization methodologies to train MetaMetrics: Bayesian Optimization (BO) and Boosting. BO offers the advantage of interpretability, allowing us to clearly identify which metrics contribute most significantly to the final outcome. Conversely, the Boosting method excels in enhancing alignment and accounting for the compositionality of different metrics, even when dealing with more complex functions. Although we can measure the contribution of each metric, the clarity and distinctness of these contributions are more pronounced with BO compared to Boosting. Certainly, there are many alternative regressors that can be used as substitutes.

To address your concerns comprehensively, we have expanded the details in our robustness section (Section 6.3) and Appendix F.7. We provide an in-depth discussion on our observations and results in both cross-lingual and cross-dataset settings. To further substantiate our claims, we have included new results for the WMT24 cross-lingual evaluation on the English-Spanish (en-es) and Japanese-Chinese (jp-zh) language pairs.

Additionally, for your convenience, we have summarized the cross-dataset results on Image Captioning as follows:

We hope this addresses your concern. Please feel free to reach out with any further questions; we are more than happy to assist.

Thank you for addressing my questions. I appreciate the clarity of your in-domain experiments, where 30% of the dataset is used for training and the remaining 70% for evaluating the MetaMetrics.

Since you are proposing a new evaluation metric, I’d like to confirm the intended protocol for using it. When using the MetaMetrics, it is always the ood setting, which is why I'm very concerned. Below is my understanding. Please correct me if this does not align with your intended approach. Some of my concerns and questions are based on this procedure:

1. Identification of the downstream task, such as Image Captioning.
2. Training MetaMetrics on datasets with available human ratings, evaluating on a held-out set, and selecting the MetaMetrics based on this in-domain test performance.
3. Applying the chosen MetaMetrics to a target ood dataset without human ratings to assess language model behavior against human preferences.

Regarding your OOD experiments in the Robustness section, I have the following questions:

1. It’s encouraging to see experiments on unseen languages. However, I’m unclear about the experimental settings for robustness testing. Could you clarify:
   • What are the language pairs (full dataset or 30%, and from which year) used for training MetaMetrics-MT?
   • How was MetaMetrics-MT selected for evaluation on unseen languages (in-domain test performance on the rest 70%)?
   • Additionally, what correlation metric is used in Table 17?
   • Please also provide a reference for the WMT MQM 24 datasets.
2. Looking at Tables 16 and 17, it’s promising to see that MetaMetrics outperforms many conventional metrics individually. However, when comparing with all conventional metrics, the correlation metrics show marginal or no improvement. For cross-dataset evaluations, similar trends are observed.
3. Thank you for including Table 18. However, it appears to replicate Table 12 with less information. For example, the in-domain test performance, which is crucial for determining the appropriate version of MetaMetrics for OOD evaluations (based on the above mentioned evaluation protocol), is missing. As highlighted in Table 12 (MetaMetrics-Cap section, THumB1.0 column), the GP full model performs best in-domain testing. However, when applied to OOD datasets (MetaMetrics-Cap cross-dataset section), its performance (0.298) is very close to CLIP-S (0.292) and similar to METEOR (0.281).

To summarize, I remain unconvinced that the MetaMetrics will consistently perform well in OOD settings. The beyesian optimization and boosting methods that are not inherently designed with ood generalization in mind. The ood generalization capability of metametrics may heavily depend on the task similarity (e.g., the ja-zh pair, which is dissimilar to any language pair in the training set, shows no improvement as reflected in Table 17).

Thank you again for responding to my questions, and I am happy for further discussions.

Thank you for your response. We'd like to help clarify any misunderstandings you may have.

It seems there may have been some misunderstanding regarding certain aspects of our paper. Let us clarify the correct steps:

1. Identify the downstream task, such as image captioning.
2. Train MetaMetrics using datasets that include human ratings. Evaluate the performance on an in-domain held-out set, selecting the MetaMetrics based on the performance of this same in-domain held-out set (not the test set). The test set is not used at this stage.
3. Evaluate the selected MetaMetrics on an unseen test dataset split that includes human ratings. This is a standard machine learning evaluation. We use Kendall or Pearson correlation to assess language model behavior relative to human preferences. The test dataset split can include both in-domain unseen test data and out-of-domain unseen test data (e.g., cross-lingual and cross-dataset scenarios). For the in-domain setting, use 70% of the dataset as the test split. For the out-of-domain setting, we use the whole dataset as test set.

Let's walk through a step-by-step example of testing in an out-of-domain cross-lingual setting for machine translation. This should help ensure we're on the same page.

When evaluating on WMT23, we train our MetaMetrics using Datasets A_train, B_train, and C_train. During calibration, we fine-tune the MetaMetrics only with the training data. For testing, we apply our metric to the datasets D_test, E_test, and F_test, while ensuring that A_test, B_test, and C_test remain untouched. In this scenario, we do not have any training data for WMT23 or WMT24.

Let's also walk through a step-by-step example for cross-dataset in case you need clarification.

When evaluating the Flickr8k on the cross-dataset setting, we train our MetaMetrics using Datasets B_train and test on A_train and A_test, and vice versa for THumB1.0.

What are the language pairs (full dataset or 30%, and from which year) used for training MetaMetrics-MT?

As mentioned in the above example and F.7.1 Cross-lingual Setting Section in Appendix, here are the breakdown (just to make sure it is clear):

Additionally, what correlation metric is used in Table 17?

Average Results of acc-t and Soft-pairwise accuracy (SPA). We reported the number shared with us by WMT24 organizers and added to the latest version of our paper.

Please also provide a reference for the WMT MQM 24 datasets.

We added the citation of WMT24 shared task paper. To preserve anonymity, we will not specifically mention anything about the WMT24 paper. We will add more information once this paper is accepted.

Looking at Tables 16 and 17, it’s promising to see that MetaMetrics outperforms many conventional metrics individually. However, when comparing with all conventional metrics, the correlation metrics show marginal or no improvement. For cross-dataset evaluations, similar trends are observed.

It appears there may have been a misunderstanding regarding the interpretation of the tables. Let us clarify them for you. In Table 16, we demonstrate that MetaMetrics outperforms all individual metrics. When compared to XCOMET-Ensemble, MetaMetrics still shows superior performance, although the margin is smaller. It's important to note that, according to the WMT23 paper, XCOMET-Ensemble utilizes much larger models and they are larger than MetaMetrics. Additionally, we couldn't access their metric as the authors have not made it publicly available.

In Table 17, the metrics we compared against are not the same as those used to build MetaMetrics. Many of these metrics are not currently accessible, as the shared task competition concluded only recently. We did not incorporate all metrics listed in Table 17 into MetaMetrics; they are included solely for comparison in relation to the WMT24 submission. It's important to note that MetricX-24 differs from MetricX-23, which we used during calibration. MetricX-24 is a more advanced version and is not publicly available. Therefore, we believe it is not a fair comparison to claim there is no improvement, as we did not base MetaMetrics on these comparison metrics. Furthermore, we employed the same MetaMetrics for both WMT23 and WMT24 without any additional calibration or tuning for adaptation.

However, it appears to replicate Table 12 with less information. For example, the in-domain test performance, which is crucial for determining the appropriate version of MetaMetrics for OOD evaluations (based on the above mentioned evaluation protocol), is missing. As highlighted in Table 12 (MetaMetrics-Cap section, THumB1.0 column), the GP full model performs best in-domain testing. However, when applied to OOD datasets (MetaMetrics-Cap cross-dataset section), its performance (0.298) is very close to CLIP-S (0.292) and similar to METEOR (0.281).

Thank you for highlighting this issue. We want to emphasize that in cross-dataset setting even our least performing results are still better than other existing metrics although it is not as far as our best metric.

The beyesian optimization and boosting methods that are not inherently designed with ood generalization in mind.

We have demonstrated that our MetaMetrics effectively generalize across both cross-lingual and cross-dataset scenarios. We chose Bayesian Optimization and Boosting Methods for their strengths in supervised learning, not explicitly for out-of-domain (OOD) generalization. These methods have shown potential for OOD performance if appropriately applied, as demonstrated in our work. Claiming that these method are not inherently designed with OOD generalization is a strong statement without providing any references. While, in our experiments we showed that it is possible to generalize well on OOD.

Dear Reviewer nKfe,

We hope this message finds you well. We kindly want to check if you had a chance to review our latest response and we have addressed all of your concerns, and if you have any further questions or comments we can address to help with your evaluation. Thanks again for your efforts and suggestions in improving our manuscript.

Sincerely,

The authors

Dear Reviewer nKfe,

We hope this message finds you well. We kindly want to check if you had a chance to review our last rebuttal, and if you have any final comments after our last response. Thank you again for your review.

Sincerely,

The authors

Thank you for your insightful review. We have addressed your questions and concerns as follows:

The proposed approach requires training a new combination weight vector once the base metrics have changed, thus limiting the usability of the approach.

In terms of efficiency, the calibration process is lightweight, requiring only CPU resources, and scores for each base metric from previous runs can be reused even if there are changes to the set of base metrics used for MetaMetrics, eliminating the need for additional GPU resources.

In addition, regarding metric robustness, our approach demonstrates adaptability to unseen domains and languages, as evidenced by our results in image captioning and machine translation tasks. This allows us to apply the same weight combination to similar tasks across different domains or languages.

Please provide the training details. From my understanding, when training a combined metametric for reward modeling, you need to run inference on all of the training data using each of the base metrics first, and then train your weight vector to optimize for the human preference. How much GPU hours have you spent on the inference, and the training? How many GPUs did you use for the experiments?

To perform the entire inference process to obtain a score using GPUs, we generally require about a day when employing the A100 40GB for reward modeling, as noted in the Limitations section. We allocate one metric per GPU for LLM-based metrics. In contrast, non-LLM-based metrics are executed on CPUs. Once all the scores are acquired, the calibration process takes approximately 30 minutes to an hour for GP and about 10 minutes for XGBoost on a 16-core CPU.

Could you provide a list of base metrics used for each meta-metric variant in one place? I only found the base metrics for reward modeling in line 474 and it's unclear what's the base metrics used for other benchmarks.

For clarity, we have added a comprehensive list of the base metrics in Appendix Section G. This section is also accompanied by heat maps that illustrate the weights of each feature or metric for both GP and XGBoost.

Thank you for useful review. We have addressed your questions and concerns as follows:

It is very good that the authors have shown their method on multiple datasets, but some more experiments/analysis on the proposed methods would strengthen the paper more. One such thing can be, the authors used Kendall-Tau as the objective calibration function. How were the results when spearman correlation is used as a calibration function?

Following your suggestion, we have added an ablation study in Appendix F.4 using Spearman correlation as the calibration function on XGBoost for the Summarization task. The results show minimal performance differences between the two evaluation functions, which is expected given the similarity of evaluation measurement between Spearman and Kendall correlations.

| Metric             |                SummEval           |     Benchmark LLM     |    Combined   | Avg.  |
|                    | Coh.  | Cons. | Fluency | Rel.    | Faith.| Coh.  | Cons. | Coh.  | Cons. |       |
|--------------------|-------|-------|---------|---------|-------|-------|-------|-------|-------|-------|
| XGBoost (Kendall)  | 0.190 | 0.180 |  0.164  |  0.255  | 0.406 | 0.411 | 0.397 | 0.403 | 0.367 | 0.308 |
| XGBoost (Spearman) | 0.185 | 0.174 |  0.165  |  0.261  | 0.406 | 0.411 | 0.397 | 0.402 | 0.369 | 0.308 |

Although the metametrics have shown they have higher alignment with human preferences, the optimization methods (BO and Boosting) are to be considered black boxes and offer very less explainability. Understanding how these methods find the combination of metrics and interactions between different metrics within the model is not possible. Hence explanation on why they correlate more with human scores is not there.

Optimization methods like BO retain a level of interpretability, particularly in terms of weight allocation, and are well-motivated and transparent in their operation. Although boosting is comparatively less interpretable than BO, it still allows us to assess metric importance through feature importance. Thus, these enable us to understand the contribution of each metric to the overall performance metric.

Thank you for your constructive suggestion. We have addressed your questions and concerns as follows:

Empirical and ad hoc nature of MetaMetrics: The calibration process for MetaMetrics relies on empirical and ad hoc weighting of existing metrics for each dataset and human preference. These weights are highly dependent on the specific dataset and task, which raises concerns about their generalizability to a broader range of datasets or tasks.

To enhance the generalizability of our method, we have applied MetaMetrics to a diverse range of tasks and modalities, including abstract summarization, machine translation, question answering, reward modeling, and image captioning. We also test the robustness of our method in cross-domain and cross-lingual contexts. Specifically, for image captioning, we calibrate our metric in one domain and evaluate it in an unseen domain, as illustrated in Figure 2. This demonstrates that MetaMetrics remains robust in unfamiliar domains. Additionally, we carry out experiments on cross-lingual machine translation tasks by evaluating our model with an unseen language pair (he-en). In this paper, we limit our exploration for a single model per task and the development of a single model applicable to all tasks is left for future research.

Lack of ensembled baselines: While the paper compares MetaMetrics against individual metrics, it does not include ensembled baselines, such as averaged or weighted-average scores of individual metrics. Including these as baselines would provide a fairer comparison and strengthen the evaluation.

Thank you for your suggestion. We have included baselines for both uniform averaging and weighted averaging across all tasks to ensure a fairer comparison. Our method outperforms these baselines across all tasks.

Absence of human studies in experiments: Although MetaMetrics aims to align with human preferences in evaluating model performance, the study does not involve an actual model being evaluated and analyzed through human studies. To comprehensively validate MetaMetrics, experiments involving direct human evaluation of model outputs are necessary.

All of our test sets are all human scores and we use kendall correlation to compute the correlation between our scores and humans. We report all results against humans.

Unclear human preference in certain tasks: In some evaluation tasks, the connection between the ratings and human preferences is not adequately explained. For instance, it is unclear how ratings like sys Pearson, seg Pearson, and acc-t in Machine Translation tasks, or NQ, TQ, NarrativeQA, and SemEval in Question Answering tasks, align with human preferences. More detailed explanations of these ratings and their relationship to human preferences are needed, especially for readers unfamiliar with these tasks.

We have provided a detailed description for each rating, including kendall, sys Pearson, seg Pearson, and acc-t in Appendix Section H to enhance clarity on how ratings are aligned with human preferences.

High computational cost for reward model ensembling: The use of multiple metrics in MetaMetrics can significantly increase evaluation costs, particularly in reward model scoring. For example, in online RLHF training, simultaneously running multiple reward models (e.g., 7 models in Table 4) could make the process computationally infeasible.

MetaMetrics is designed to be embarrassingly parallel, enabling each metric to be executed independently without requiring inter-model communication, which results in exceptional efficiency. Thus, making it much efficient to run inference if there are multiple models used in MetaMetrics. To have a fair comparison, we take models with similar performance, such as the 70B ones like Skywork-Critic-Llama-3.1-70B, TextEval-Llama3.1-70B, and SFR-LLaMa-3.1-70B-Judge-r. Our MetaMetrics (GP) reward model, with only 41B parameters (composed of multiple models where the largest model contains 8B parameters), outperformed these 70B models. Additionally, despite its size, our MetaMetrics reward model allows for parallel execution of individual models with a computational cost equivalent to 8B parameters, enabling us to use considerably less compute compared to the 70B models, which demand much more substantial resources. Additionally, our metric can still run in a GPU with smaller computing requirement such as A100 40GB, while the 70B model requires much larger compute with 80GB GPU with half precision.

Thank you for your insightful comments and for helping to enhance our score. We find your suggestions feasible and are pleased to incorporate the proposed ablation study in the camera-ready version. We also believe this idea will provide researchers with additional motivation to adopt our method when developing reward models.

Thank you for your constructive suggestion. We have addressed your questions and concerns as follows:

This paper assumes that human preference can be approximated by a weighted combination of metric scores. But it seems to be too simple, and might not always hold. For example, human's evaluation of helpfulness might require both the model to perform well in reasoning and knowledge-retrieving, which we assume are separately measured by two benchmarks.

To ensure the generalizability of our method, we apply MetaMetrics to a variety of tasks and modalities, including abstract summarization, machine translation, question answering, reward modeling, and image captioning. We also explore the robustness of our method in cross-domain and cross-lingual scenarios. For image captioning, we calibrate our metric in one domain and test it in an unseen domain, as shown in Figure 2, demonstrating that MetaMetrics maintains robustness in unfamiliar domains. Additionally, we conduct experiments on cross-lingual machine translation tasks by evaluating our model with an unseen language pair (he-en).

Then a weighted summation of these two benchmarks might not be suitable, instead, a multiplication might be more proper.

Following your suggestion, in Appendix Section F.6, we have included a new ablation study comparing multiplicative and linear weighting. Our findings indicate that linear weighting generally perform better or on par to the multiplicative approach. We conjecture that the multiplicative method introduces a higher-dimensional space and GP may not perform well on higher-dimensional space [1,2,3,4,5], especially on Summarization and Reward Model.

| Method                  | Summarization | MT    | MT (QE) | Reward Model |
|-------------------------|---------------|-------|---------|--------------|
| GP (Linear)             | 0.262         | 0.819 | 0.801   | 93.5         |
| GP (Multiplicative)     | 0.238         | 0.822 | 0.798   | 91.2         |
| GP (Combined)           | 0.234         | 0.818 | 0.803   | 92.1         |

References:

• [1] Frazier, P. I. (2018). A tutorial on Bayesian optimization. arXiv preprint arXiv:1807.02811.
• [2] Binois, M., & Wycoff, N. (2022). A survey on high-dimensional Gaussian process modeling with application to Bayesian optimization. ACM Transactions on Evolutionary Learning and Optimization, 2(2), 1-26.
• [3] Li, C. L., Kandasamy, K., Póczos, B., & Schneider, J. (2016, May). High dimensional Bayesian optimization via restricted projection pursuit models. In Artificial Intelligence and Statistics (pp. 884-892). PMLR.
• [4] Nayebi, A., Munteanu, A., & Poloczek, M. (2019, May). A framework for Bayesian optimization in embedded subspaces. In International Conference on Machine Learning (pp. 4752-4761). PMLR.
• [5] Malu, M., Dasarathy, G., & Spanias, A. (2021, July). Bayesian optimization in high-dimensional spaces: A brief survey. In 2021 12th International Conference on Information, Intelligence, Systems & Applications (IISA) (pp. 1-8). IEEE.

Dear Reviewers,

We hope this message finds you well. We would greatly appreciate it if you could respond to our rebuttals, particularly to let us know if we have sufficiently addressed their concerns. Thank you very much for your support and understanding.