IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Models

4. [Re question 3: How to get the embedding of the examples when constructing the graph? And what is the cost?]

As we illustrated in our paper (page 3 in our submission), we obtain the embedding of the examples through Sentence-BERT which follows the same setting as Vote-$k$. The cost of this process is negligible and can be ignored, as it takes less than 5 minutes for all datasets in our experiments.

5. [Re question 5: It would be great to see some results for more datasets (especially more tasks beyond the text classification) and more LLMs.]

Thank you for your constructive feedback. We address your question below.

• Following the same setting with previous selective annotation work Vote-K in the same area, we have demonstrated the results of IDEAL for datasets beyond text classification, including Multi-Choice, Dialogue, and Generation in Section 4.2. We have also conducted experiments for different LLMs in section 4.3.4.
• Additionally, we have performed additional experiments on medium-sized LLMs LLaMA2-chat 7B and 13B using the same setting as Section 4.3.4, and presented the results below. Our observations indicate that IDEAL consistently outperforms the baselines. This demonstrates that our method is a general approach that works well in medium-sized LLMs

------------------------------ LLaMA2-7b-chat ------------------------------

------------------------------ LLaMA2-13b-chat ------------------------------

Thanks for your constructive suggestions. The insightful and constructive suggestions have enabled us to effectively improve our work. Please find our response in the following text.

1. [Re weakness 1 & question 1: Whether it is required to run the influence function for n times to retrieve n shots?]

Thank you for your constructive suggestion. To retrieve n shots, the influence function indeed needs to be run n times. We also included a time-complexity analysis in the Appendix E in the revised version. Here is a high-level overview of the time-complexity analysis for our IDEAL.

• Information diffusion process: The time complexity of quantifying the influence of a specific subset is O(a + b), where a is the number of edges and b is the number of nodes. This is because it involves running an independent cascade diffusion process (BFS-like traversals) through the graph.

• Greedy algorithm: The greedy algorithm iteratively selects the example that provides the maximum marginal gain in influence. When the annotation budget is n, the time complexity is O(n*(a+b)). This is because, in the worst case, the algorithm needs to evaluate the influence for each of the n nodes in the network, and for each node, the algorithm needs to perform a simulation with a time complexity of O(m + n).

Regarding time consumption, we compared IDEAL with Vote-$k$ in Figure 3 in the log scale. We observed that IDEAL achieves a 7.8× lead on average over Vote-k. This time consumption improvement benefits from the fact that IDEAL doesn’t involve any model prediction time cost, which is totally unsupervised compared to Vote-$k$. Therefore, the proposed method is more efficient.

2. [Re weakness 2 & question 2: How to consider the correlations among the selected examples when performing greedy search?]

We appreciate your suggestion and address your concerns as follows.

• Our method does not require consideration of correlations among retrieved data points. In fact, this is precisely the advantage of our method. IDEAL aims to maximize the influence of the entire selected subset, rather than considering each example separately.
• Taking the example you provided, if the influence of A+B is greater than C+B, our algorithm would choose A+B over C+B, even though C has a greater influence than A. As mentioned in section 2.2, IDEAL aims to maximize the influence of the entire selected subset of examples, which avoids an intractable explicit balance between data diversity and representativeness.

Additionally, we also have supplemented more details in the revised paper (section 2.2) based on your advice. Thanks for this suggestion!

3. [Re weakness 3 & question 4 : Compare with more demonstration selection methods in NLP fields including similarity-based or diversity-based.]

Thanks for the suggestion. We address your concern below.

• We kindly argue that the research problem in our paper is the selective annotation problem, which involves curating a core subset from a large-scale data pool to annotate. The “similarity-based or diversity-based” methods you mentioned are demonstrated retrieval methods that are not what we investigated in this research. Our method is agnostic to example retrieval methods and can be combined with all potential retrieval methods.
• In particular, we also investigated different retrieval methods in Section 4.3.3 when combined with IDEAL. We observed that similarity-based methods achieve better performance.

Dear Reviewer Ra1D:

Thanks again for your efforts in reviewing. We are looking forward to your reply. Would you mind checking our response and confirming if there are unclear explanations?

We are highly encouraged if your concerns have been addressed. On the contrary, if you need any more clarification, we can provide it as soon as possible before the discussion deadline.

Thanks!
Authors

Dear Reviewer Ra1D:

Thanks a lot for your efforts in reviewing this paper. We tried our best to address the mentioned concerns and have provided a detailed response. Are there unclear explanations here? We could further clarify them.

Thanks!

Authors

Dear Reviewer Ra1D:

Thanks a lot for your efforts in reviewing this paper! We tried our best to address the mentioned concerns and have provided a detailed response.

As the discussion deadline is very close, we authors want to confirm whether there are unclear explanations and descriptions here. We could further clarify them.

Thanks!

Authors

1. [Re Weaknesses 1: Include discussion with an important baseline in the main paper.]

We appreciate the suggestions. We have revised the paper according to the suggestion and include the discussion of these two papers in the related work section in the revised version. Thanks for your valuable feedback!

2. [Re Weaknesses 2: How the influence score relates to better diversity and representativeness.]

Thank you for your feedback! We would like to clarify that our method is not intended to select examples to achieve better diversity and representativeness. Instead, we aim to investigate a new paradigm to quantify potential selective annotation subsets through an information diffusion process that avoids the intractable explicit balance between data diversity and representativeness, as mentioned in our paper’s abstract. Furthermore, we provide a visualization for the selected examples in Figure 6 in our paper that compares Vote-$k$ and IDEAL. Vote-$k$ takes diversity into consideration, which may lead to the inclusion of outliers.

3. [Re Weaknesses 3: Include experiment datasets that are probably not trained on the experiment LLMs.]

Thank you for your constructive suggestions! We agree that benchmark leakage is a significant research problem that has garnered a lot of attention in the research community [1]. However, the research problem in our paper is selective annotation in In-Context learning, which does not involve the processing of training data selection. Due to the limited rebuttal time, we will take a deep dive into the benchmark leakage problem in In-Context Learning in our future research and investigate its impact on selective annotation problems. We have also added an additional paragraph in Appendix G to discuss the benchmark leakage problems. Thank you for your suggestions!

[1] Zhou, K., Zhu, Y., Chen, Z., Chen, W., Zhao, W. X., Chen, X., ... & Han, J. (2023). Don't Make Your LLM an Evaluation Benchmark Cheater. arXiv preprint arXiv:2311.01964.

4. [Re Question 1: Will the method still be effective on medium-sized LLMs such as LLaMA2-chat 7B/13B?]

Thank you for your valuable suggestions. We conducted experiments and evaluated the results of our method and baselines on LLaMA2-7b-chat and LLAma2-13b-chat using the same setting as Section 4.3.4. Our observations indicate that IDEAL consistently outperforms the baselines. This demonstrates that our method is a general approach that works well in medium-sized LLMs.

---------------------------- LLaMA2-7b-chat ----------------------------

---------------------------- LLaMA2-13b-chat ----------------------------

5. [Re Question 2: Do you use any instruction on ChatGPT?]

Thank you for your feedback. To ensure a fair and easy comparison, we used the same instructions and prompts as the other LLMs used in our experiments. This approach allows us to more strictly show the differences between multiple LLMs and verify the generalization ability of our method on different LLMs. We have elaborated on this procedure in more detail in our revised version (Section 4.3.4).

9. [Re Question 2 & Weakness 4, 5 and Weakness 3 - part 1: Why don't you use all datasets in all experiments?]

Thank you for your insightful comments! We have the following three reasons for not performing experiments on all datasets for all sections in our paper, including ablations.

(1) We intend to keep the same setting as previous work Vote-k for convenient comparisons and analysis, which also does not perform experiments on all datasets in the ablation sections. Moreover, to demonstrate the robustness of our method in different scenarios, we cover diverse datasets in different sections.

(2) Performing selective annotation experiments is extremely costly in terms of both time and API fees. For example, it costs over 120 USD for a single run of GPT-3.5-turbo on Mowz and GeoQ datasets, which is unaffordable considering our limited budget.

(3) We believe that using partial datasets in the main experiments to perform ablation is acceptable and does not result in a loss of generality. There are a large number of research works in both selective annotation [1] and more general In-Context learning examples selection [2][3][4] that only uses a partial dataset to perform the ablation experiments.

We sincerely appreciate your suggestion, and we will try our best to perform experiments on all datasets in the ablation study during the rebuttal period, as shown in the following reply. Thank you again for your suggestion!

[1] Su, H., Kasai, J., Wu, C. H., Shi, W., Wang, T., Xin, J., ... & Yu, T. (2022). Selective annotation makes language models better few-shot learners. arXiv preprint arXiv:2209.01975.

[2] Chang, T. Y., & Jia, R. (2023, July). Data curation alone can stabilize in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 8123-8144).

[3] Nguyen, T., & Wong, E. (2023). In-context Example Selection with Influences. arXiv preprint arXiv:2302.11042.

[4]Li, X., & Qiu, X. (2023). Finding supporting examples for in-context learning. arXiv preprint arXiv:2302.13539.

10. [Re Weakness 3 - part 2: Why have Table 2 for "18" rather than "100”?]

Thanks for your suggestions. In most cases, we choose to use 18 as the annotation budget. This is because all annotated examples can fit the prompt of the large language models within context limits. Therefore, the prompt retrieval stage is ignored and the evaluation results can naturally represent the quality of the selected examples. In section 4.3.2, the main purpose of the experiment is to demonstrate that the selected annotated examples could work well in multiple LLMs. To exclude the influence of other factors, we chose 18 as the annotation budget.

11. [Re Comments 1:Adding a few illustrative examples in the introduction?]

We sincerely appreciate your suggestion. Due to space constraints, we would like to clarify that the performance improvement brought about by the annotated examples, as compared to the performance without annotated examples (zero-shot), is due to the few-shot learning ability of LLMs itself [1][2], rather than our selection annotation method. Therefore, we have not included such illustrative examples in our paper. However, we have provided some selected examples in Appendix C to demonstrate the core example selection ability of our algorithm, which is the primary contribution of our method.

[1] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., ... & Amodei, D. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33, 1877-1901.

[2] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., ... & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35, 24824-24837.

12. [Re Comments 2: please add to the paper a paragraph or two about the actual/expected cost of annotation for the various domains in the empirical evaluation?]

In our revised version, we have added one section in the Appendix to further illustrate the cost of annotation (Appendix F) . Thanks for your insightful suggestion.

We sincerely thank you for your careful reading of our paper and appreciate your valuable feedback in so many details.

1. [Re Weakness 1.1 - Part 1 & Question 3 : Why choose 18 and 100 as the annotation budget?]

Thanks for your suggestion! We would like to clarify that the annotation budget is not arbitrarily selected but based on the following non-trivial reasons.

• Firstly, we strictly followed the same setting as Vote-k, which also used 18 and 100 as the annotation budget, to enable easy comparison with the most important baseline and perform deep analysis.
• Secondly, we included 18 as the annotation budget because all annotated examples can fit the prompt of the large language models within context limits. Therefore, the prompt retrieve stage can be ignored, and the evaluation results can naturally represent the quality of the selected examples.

We appreciate your valuable suggestion and have revised the paper to make it clearer in section 4.2.

2. [Re Weaknesses 1.2 & Question 1: Why don’t include Auto-IDEAL as a major contribution.]

Although Auto-IDEAL has demonstrated better performance, it has two main drawbacks that may hinder its usability in practice. We list the reasons below.

• Auto-IDEAL suffers from a model inference cost, especially in the era of large language models as a service. Specifically, when performing automatic annotation, Auto-IDEAL has to make predictions for all unlabeled examples to achieve automatic annotation. In our experiments on GeoQuery datasets, it costs more than 120 USD for a single run. Considering this, we think it is not as practical as the native IDEAL.

• Auto-IDEAL may fail when the initial examples to be labeled are not very similar to the initial examples labeled, even though they have a similar embedding. Specifically, Auto-IDEAL performs automatic annotation by following the information diffusion process from the initial annotated subset to the examples to be labeled with similar embedding. When the examples to be labeled are not relevant to the initial examples labeled, it may lead to incorrect automatic annotation and poor performance.

Considering these drawbacks, we decided to use native IDEAL as our core contribution but regarded Auto-IDEAL as a potential extension to IDEAL in a case study in our paper. We also illustrate it more clearly in Appendix G in the revised version.

We may leave the potential improvement of Auto-IDEAL as our future work. Thank you for your suggestions!

3. [Re Weakness 1.1 - Part 2: You should provide an automated approach to automatically assess the smallest amount of annotations.]

Thank you for your valuable feedback! We argue that the concept of “the smallest amount of annotations” should not exist in selective annotation problems. Given that we do not know the distribution of the test examples, we believe that the larger the annotated examples, the better the expected performance, as it will be easier to retrieve the most relevant examples from a larger data pool. In an ideal case, all available examples should be annotated, and the algorithm could achieve the best-expected performance. However, in a more realistic setting, users have a limited budget, and the algorithm could select the best subset to annotate under the user’s budget. Regarding your suggestion of finding the smallest subset, we think a more meaningful setting would be if users have an expected performance threshold, and they expect there exists an algorithm that could help them find the smallest subset to annotate to reach the threshold

4. [Re Weakness 2.1: Please also add an additional row to show the best known performance.]

Thanks for your suggestions. The main purpose of Table 2 is to compare the performance of IDEAL with other methods that could select a “coreset” from the large scale unlabeled data to annotate. Regarding the “best-known” performance, we can observe from Table 2 that IDEAL achieves the best performance compared with all baselines. If we misunderstand your suggestion, please let us know. Thanks!

5. [Re Weakness 2.2: Please also add the results when k=1000 even k=10000.]

Thank you for your suggestions. We would like to clarify that the value of K = 10000 exceeds the size of our dataset. To simulate the process of obtaining unlabeled samples, we followed the same setting as Vote-k, which is described on page 4 of Vote-k. In this setting, the sample size was set to 3000. Due to limited rebuttal time and computational resources, we will try our best to conduct additional experiments when k=1000 during the rebuttal period and update the results accordingly once complete as shown in the following reply. Thank you again for your suggestion!

6. [Re Weakness 2.3: please explain why we see IDEAL(100) < IDEAL(18) on RTE.]

Thank you for your suggestion. It is indeed an interesting scenario that was also observed in previous work Vote-k, where large annotation budgets do not necessarily lead to better performance (Vote-K (100) < Vote-K (18) in MWoZ, GeoQ, and DBpedia). However, in most cases (8 out of 9), IDEAL(100) performs better. We believe this particular scenario exists because a large annotated data pool may include some “bad” examples that have a similar embedding with the users’ query but are not relevant actually. Overall, a large annotation budget typically leads to better performance from a statistical perspective.

7. [Re Weakness 2.4: Why Vote-k outperform IDEAL(100) on GeoQ?]

Thank you for your constructive suggestion. Vote-k is insightful and needs a careful balance between diversity and representativeness. In some cases, e.g., on GeoQ (m=100), a good balance may be achieved. However, since the balance is tricky and complex, in most cases, Vote-k cannot work well, which motivates us to develop more advanced algorithms.

We also kindly argue that the comparison of the two methods needs to be considered at the statistical level. IDEAL outperforms Vote-K in 17 out of 18 cases, which could indeed demonstrate the superiority of IDEAL over Vote-K. We totally agree that in an ideal case, as researchers, we should develop methods that are better than baselines in all cases. This is what we aim to achieve in our future research career and the follow-up work of our method. We appreciate your suggestions!

8. [Re Weakness 2.5 & Question 4: Why not include more trials in the experiments?]

Thank you for your suggestion. We included three different runs for each dataset for two reasons:

• To ensure a fair comparison and ease of analysis, we used exactly the same setting as the most important baseline Vote-k, which also used three different runs.

• Some experiments are especially costly. For example, each run of Mowz and GeoQ costs over 130+ USD dollars using GPT-3.5 Turbo. Considering our limited budget, we cannot afford large-scale experiments in some specific datasets. Due to limited rebuttal time and computational resources, we will try our best to conduct additional experiments to run more trials during the rebuttal period and update the results accordingly once complete, as shown in the following reply. Thank you again for your suggestion!

13. [Re Comments 3: Please intuitively explain the "end-to-end" term that you are using in the Abstract, Introduction, and later on in the Appendices.]

Thank you for your feedback. We refer to IDEAL as an end-to-end algorithm because it directly selects a subset of unlabeled data for annotation from a large-scale pool in one step, without any intermediate procedure. In contrast, Vote-k selects a subset of data in two stages. First, it selects a small portion of data for diversity and annotates them manually. Second, these annotated data act as prompts for predictions on all other unlabeled data, and the remaining ones that need to be annotated are chosen based on diverse confidence scores. This approach adds inference costs due to the predictions on unlabeled data. We mentioned the difference between IDEAL and Vote-k in our original submission.

14. [Re Comments 4: please summarize the running time improvements of IDEAL vs Vote-k?]

Thank you for your feedback. In our original submission, we summarized the numerical improvement with respect to time consumption on page 7. “Specifically, under the same hardware conditions, IDEAL achieves a 7.8× lead on average over Vote-k.”. In response to your constructive suggestion, we also have added this information to the caption to further emphasize the advantage of our method in the revised version. Thank you again for your suggestions!

15. [Re Nitpicks: P1 - P5]

Thanks for your suggestions. We have revised the paper according to your suggestions from P1, P2, P4, P5 which is shown in blue color. For P3, we would like to clarify that the iteration doesn’t start from (d).
We deeply appreciate the time and effort you spent to further improve our submission. Thanks!

-------------------------------------- Experiments Work in Progress --------------------------------------

Due to the extremely limited computation resources and rebuttal time, we will do our best to add as many experiments as possible during the rebuttal period and update the results accordingly. While we cannot guarantee that we will be able to complete all the experiments in a short period of time, we will make every effort to do so. We will also update the results in our future version of the paper. Thank you for your valuable suggestions!

—------------------------------ Update : Nov 16 2023 —------------------------------

Completed

Five seeds results for Table 4.2 with annotation budget 18 (Will run five more seeds)

Current In progress

Using all datasets to perform experiments on section 4.3.1

We sincerely thank you for your careful reading of our paper and appreciate the valuable feedback in your comments. The insightful and constructive suggestions have enabled us to effectively improve our work.

1. [Re Weakness 1 & Question 1: Comparisons with Self-adaptive in context learning and TopK.]

Thank you for your constructive feedback! In our revised version, we have included a detailed discussion in the related work section (Appendix A) for Wu et al., 2022 and TopK (Liu et al. 2022 and Gao et al).

We kindly clarify that the research problem in our paper is different from these papers and the methods are also agnostic. Specifically, Wu et al., 2023, is the first work to formally define the problem of self-adaptive in-context learning, which aims to search for the best in-context example and corresponding order with a set of annotated candidate examples. Liu et al. 2022 and Gao et al. 2021 focus on the example retrieval problem which proposes to use the nearest neighbors of a given test query as the corresponding In-Context examples.

On the contrary, the research problem of our paper is the selective annotation problem, which aims to curate a core set from large-scale unlabeled data to annotate. This problem is agnostic to the both self-adaptive In-Context Learning problem and example retrieval.

Additionally, we also provide ablation experiments in Section 4.3.3 to quantify the effect of different example retrieval methods. We believe that all three research problems are meaningful and worth exploring in the research community. Thanks again for your suggestion!

2. [Re Weakness 2 & Question 2: The rationales behind the reverse performance of MNLI in Table 5.]

We sincerely appreciate your constructive suggestions. We address your concerns as follows.

• The performance drop of MNLI in Table 5 is predictable because Auto-IDEAL is an automatic annotation method that relies on half of the selected examples being automatically annotated rather than manually labeled ground truth.

• Auto-IDEAL performs automatic annotation by following the information diffusion process from the initial annotated subset to the examples to be labeled. In some datasets, when the examples to be labeled are not very relevant to the initial examples labeled (but with a similar embedding), it may lead to a performance drop. We acknowledge that this is a potential drawback of Auto-IDEAL. Therefore, we did not regard Auto-IDEAL as the main contribution of our paper but chose a more trivial method IDEAL.

We appreciate your suggestions and have elaborated more specifically on this issue in the revised versions in Appendix G.

3. [Re Question 3 & Weakness 3: It would be great to apply IDEAL to different LLMs.]

Thank you for your constructive suggestions!

• We totally agree that IDEAL is a general selective annotation method. We would like to clarify that we have shown the evaluation results of different methods using multiple LLMs in Figure 5 in our submission. Our evaluations reveal that IDEAL consistently outperforms the baselines across all models tested.

• Additionally, we further performed more experiments on medium-sized LLMs LLaMA2-7B-chat, LLaMA2-13B-chat using the same setting as Section 4.3.4, and showed the results below. We observed that IDEAL still achieves the best performance compared to the baselines.

------------------------------ LLaMA2-7b-chat ------------------------------

------------------------------ LLaMA2-13b-chat ------------------------------

Dear Reviewer Ctph

We have properly addressed all your concerns and provided clarifications on all confusing concepts. Could you please kindly re-evaluate our paper based on the current situation? If you have any further questions, we are also very glad to discuss them.

Thanks!

Authors

Dear Reviewer Ctph:

We sincerely appreciate your efforts in reviewing this paper! We tried our best to address the mentioned concerns about related work, experimental results, and more empirical evidence. The discussion deadline is approaching now. Are there unclear explanations and descriptions here?

We are highly encouraged if your concerns have been addressed. On the contrary, if you need any more clarification, we can provide it as soon as possible before the discussion deadline.

Thanks!

Authors

Dear Reviewer Ctph:

Thanks a lot for your efforts in reviewing this paper. We tried our best to address the mentioned concerns. As the discussion deadline between reviewers and authors is very close, we tend to confirm whether there are unclear explanations and descriptions here. We could further clarify them.

Thanks!

Authors

We sincerely thank the reviewer for the insightful comments. Please find our response to your comments below.

1. [Re Question 1 & Weakness 2: Should the experiment results of other methods be included in the paper?]

Thank you for your valuable feedback. We would like to clarify that the research problem in this paper is a selective annotation problem and we have compared our method with alternative selective annotation methods, excluding vote-k and random search in Section 4.3.2. The evaluation results show that IDEAL outperforms all alternative methods, which could demonstrate the superiority of our method.

2. [Re Question 2: The randomness of the proposed method.]

The proposed method’s randomness arises from two sources: example selection and model prediction.

• The randomness in the example selection process is due to the independent-cascade diffusion process used to quantify the influence of a specific subset. To ensure stability, we repeat the information diffusion process ten times when quantifying the influence of each potential subset (as illustrated in Appendix D.2 and Section 2.2 of the revised version draft). This procedure is a part of our algorithm.

• The randomness in the model prediction process occurs in specific models like GPT-3.5-turbo. When calling GPT-3.5-turbo through OpenAI API and the temperature is not 0, it suffers from randomness.

3. [Re Question 3: What is the influence of the times of information diffusion?]

Thank you for your feedback. In our paper, we repeat the information diffusion process ten times when quantifying the influence of each potential subset (as illustrated in Appendix D.2). We have moved this content from the Appendix to the main paper (Section 2.2).

4. [Re Question 4: Additional elaboration is required to explain why the combination of IDEAL/vote-k and similarity retrieval proves effective.]

Thanks for the suggestion. The experimental results suggest that examples with more similar embeddings are more likely to be relevant. Therefore, the combination of IDEAL/vote-k and similarity retrieval is more likely to perform better than its corresponding combination with a random retrieval method.

5. [Re Question 5: The combination of IDEAL with chain-of-thought prompting.]

Thank you for your constructive feedback. We kindly argue that our method is agnostic to the chain-of-thought technology. Our approach involves selecting a subset of “core” examples to annotate. Once annotated, we can use any prompting technology, including chain-of-thought, to further boost the final performance.

6. [Re Question 6 & 7: Regarding the embedding calculation and corresponding cost.]

Thank you for your suggestions. For ease of analysis, we calculated the embedding using the same settings as previous works Vote-$k$. Specifically, as illustrated in our paper (page 3 in our submission), we obtained the embedding of the examples through Sentence-BERT. The cost of this process is negligible and can be ignored, as it takes less than 5 minutes for all datasets in our experiments.

7. [Re Question 8: Regarding the label distribution of the origin data before example selection]

We present the label distribution of the origin data in the first row of Table 9. From the statistical results, we can observe that our method achieves the smallest ratio between the numbers of the most frequent class and the least frequent class in 2 out of 3 cases. This scenario implies that our method could mitigate the label skew problem in the origin data.

Dear Reviewers:

Thanks again for your efforts in reviewing. The discussion deadline is approaching. To enhance this paper, we authors hope that you can check our response and confirm whether there are unclear explanations. We really want to solve them for you.

Yours truly,
Authors