We want to thank the reviewer for their positive feedback and comments. We will address individual comments below.

W1: In-depth analysis of the limitations

Thank you for raising this concern. This independence assumption is a mathematical simplification that allows us to develop our statistical approach to perform estimation work. Our assumption follows the learning with noisy label literature settings [1, 2, 3]. As it is also studied in the literature, when making no assumption of the transition matrix's dependence on the sample-level features, it becomes extremely hard to identify the correct noise structure of this noisy rating problem [4] and therefore disables the development of corresponding detection approaches. Besides, this assumption can also be viewed as the "average case" estimation of the amount of errors for samples with the same true rating (though we agree with the reviewer that it can imperfect and non-ideal) Empirically, our final experimental results demonstrate the efficacy of our method under this assumption. There could be a valuable future research direction to enhance performance under weaker assumptions, such as group-dependent or sample-dependent. We have included the limitations in the revised manuscript (Appendix A):

• Sample-independent assumption: The sample-independent assumption is critical for deriving the transition matrix $T$ and the true score probability distribution $p$. However, this assumption may be somewhat strong and could inevitably introduce certain data-specific errors. Exploring weaker assumptions, such as group-dependent approaches, could be a valuable direction for future research.
• Base model scale: Our experiments are primarily conducted on pre-trained models at the 7B/8B scale. It remains uncertain how well the method would perform on larger-scale pre-trained models.
• Rating models: Due to cost considerations, we use the more affordable GPT-4o-mini to generate GPT-level scores. It is unclear whether the score curation mechanism works for more powerful GPT models (e.g., GPT-4 or GPT-o1).
• k-NN clusterability. The k-NN clusterability definition implies that similar embedding vectors should correspond to the same rating score or class, a characteristic commonly leveraged in image classification tasks. However, in text-related tasks, highly similar text can convey opposite semantic meanings due to subtle differences, such as a single word change. To address this challenge, powerful embedding models are essential to accurately distinguish these subtle differences and effectively capture the underlying semantic meaning.

[1] Learning with noisy labels, NeurIPS 2013.

[2] Classification with noisy labels by importance reweighting, TPAMI 2015.

[3] Peer loss functions: Learning from noisy labels without knowing noise rates, ICML 2020.

[4] Identifiability of Label Noise Transition Matrix, ICML 2023.

W2: This paper heavily rely on the capabilities of pre-trained LLMs

Thank you for pointing out this concern. We would like to clarify that the characterization of this work is not a weakness. LLM-based data selection procedures have been demonstrated to be both efficient and widely adopted in practice. This is evident from several newly published works, including AlpaGasus (ICLR 2024), DEITA (ICLR 2024), Qurating (ICLR 2024), InsTag (ICLR 2023), LESS (ICML 2024), and Tree-instruct (ACL 2024), to name a few. More details regarding these methods are discussed in Section 2 (Related Work). Beyond academic research, LLM-based ratings are also extensively used in industry for data selection tasks, serving as a cost-effective alternative to manual annotations.

Furthermore, even though the embeddings used are extracted from the newly released model BGE, our transition matrix does not heavily depend on its performance. In Appendix D, we investigate the impact of using SentenceBert as the embedding model. When compared to Figure 3 (BGE), the transition matrix shown in Figure 8 (SentenceBert) exhibits a striking similarity, demonstrating that our transition matrix is largely independent of the specific embedding model used.

W3: The score error correction process seems unnecessary if using a strong LLM evaluator

Thank you for pointing out this. We agree that a strong LLM evaluator could be beneficial for generating accurate ratings. However, due to the higher cost of using a strong LLM evaluator, current open-sourced LLMs (such as LLaMA-3.1 and Mistral) offer a cost-effective and efficient alternative. The score error correction process plays a crucial role in bridging the performance gap between GPT-based and open-sourced LLMs. Our experimental results further highlight the necessity and effectiveness of this score error correction process, particularly for GPT-4o-mini. From the experimental results of GPT-4o-mini, we believe that the phenomenon of score errors is widespread and unavoidable, even for strong LLM evaluators.

Besides, we would like to clarify that GPT may not evaluate perfectly across all tasks. For instance, in certain domain-specific tasks, GPT may lack the necessary knowledge to provide accurate evaluations due to the privacy and restricted access to such specialized data. In such cases, GPT might produce inaccurate evaluations, leading to potential scoring errors. Therefore, implementing a score curation mechanism to mitigate potential score errors remains crucial for ensuring more reliable and accurate evaluations.

Q1: Examples of Typical Errors in LLM-Rated Scores and How DS2 Addresses Them

Thank you for your comment. Here, we present a target sample from our data pool along with its 2NN samples to illustrate this. By analyzing these samples manually, it becomes evident that the target sample is likely misrated and should have a score of 3.

• Target Example(Score: 1): User: 'You need to complete the following task: Calculate 15% of the following number: 100’, Assistant: '15% of 100 is 15.'
• KNN Example 1 (Score: 3): User: ‘Calculate 15% of 500’. Assistant: ‘75’
• KNN Example 2 (Score: 3): User: 'Calculate 50% of 300.”, Assistant: ' 50% of 300 is 150.'

First, DS2 derives the score transition matrix based on the concept of $k$-NN clusterability. Specifically, it aggregates the score information from all $2$-NN clusters and calculates the average score proportion probability, which serves as the consensus information for constructing the score transition matrix [Section 3.2].

The score transition matrix provides statistical transition probabilities but does not identify specific samples that are likely to be misrated. To address this, DS2 computes the $k$-NN agreement score for each sample [Section 4.1, Line 282]. Intuitively, if a sample’s $k$-NN agreement score is low—indicating its score significantly differs from the scores of its $k$-NN neighbors—it is likely to have been misrated. Samples are then ranked by their $k$-NN agreement scores, and the bottom $M$ samples are flagged as misrated, where $M$ is determined based on the error threshold derived from the statistical transition matrix. [Section 4.1, Line 291].

To address imbalances in LLM-based scores, a confidence probability is introduced to adjust the size of the identified misrated sample set. Finally, DS2 curates (replaces) the scores of these misrated samples with candidate scores suggested by the $k$-NN agreement (majority vote) [Section 4.1, Line 306].

Therefore, in the above example, the score for the target sample will be curated to 3 by DS2, consistent with its 2NN neighbors.

Q2: Explore the impact of diversity score in datasets with different distributions using case studies

The importance of diversity on LLM data selection has been extensively explored by previous work [1, 2,3]. Note that our data pool is composed of five distinct subsets, each characterized by varying levels of complexity and diversity. The statistical analysis of diversity scores across subsets, as illustrated in Figure 14, confirms this. More details could be found in Appendix I of the revised version. To evaluate the versatility of diversity score, we further conduct additional contrast experiments here. In particular, we solely rank the samples of subsets based on the diversity score. Then, we select the Top-k and Bottom-k samples independently to construct datasets for LLM instruction finetuning, where k =10000 here. The corresponding performance results are presented in the following table. For cost considerations, we employ LLaMA-3.2-3B as the base model. The experimental settings are consistent with those outlined in our paper. From the table, it is evident that the diversity score is not universally effective across all datasets. To achieve better results, it should be complemented with other specific metrics, such as LLM rating scores.

Table 1. Performance comparison across different subsets. Bottom-k (Top-k) refers to the samples with the lowest (highest) diversity scores, where $k$ is fixed at 10,000.

[1] How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources, NeurIPS 2023.

[2] What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning, ICLR 2024.

[3] Self-Instruct: Aligning Language Models with Self-Generated Instructions, ACL 2023.

Q3: Explore the impact of embedding models

Thank you for your insightful comment. In the Appendix D (line 1036), we utilize the typical SentenceBERT Model to extract the embeddings of data samples. The corresponding score transition matrix as well as the score pattern shown in Figure 8 is similar to the matrix using the BGE model (Figure 3). Therefore, one reasonably claim that the impact of embedding space is limited, the choice of embedding model does not significantly affect the error patterns produced by LLMs, which also highlight the robustness of our approach.

Q4: Concerns Regarding Cosine Similarity Assumptions and Potential Errors in Open-Ended Questions

We would like to clarify that the k-NN agreement score is defined as the average cosine similarity LLM score distance among k-NN samples. The intuition is that when samples with similar embedding vectors have significantly different scores, resulting in a lower k-NN agreement score, the uncertainty of the LLM rating for this example increases. Consequently, such samples are more likely to require curation. Whether a sample needs curation and whether a curated score should be assigned finally is determined jointly by an error threshold and confidence probability.

When the embedding vectors of two responses to the same open-ended question differ significantly, these two samples can be considered independent, as their k-NN samples may not include each other. In such cases, their cosine similarity scores are calculated based on their respective k-NN samples independently.

Q5: Contradiction in Embedding Similarity and Label Consistency for Math Questions

Thank you for bringing up this important point! We partially agree with the claim that these two questions might conflict with our notion. The scores of samples are influenced not just by correctness but also by broader quality metrics, such as rarity, complexity, and informativeness, as emphasized in our prompt template. Focusing solely on correctness with binary scores (0 or 1) treats correct and incorrect answers distinctly. However, when assessing overall quality on a granular scale (e.g., [0–10], later compressed to [0–5]), both questions could score 0 for simplicity and low informativeness. Thus, considering overall quality reduces the direct emphasis on correctness while reinforcing the evaluation framework. To evaluate this, we generate the scores from the above two questions (Example 1 & Example 2) in the same paper setting.

• Example 1: please tell me the answer of below math question '1+1=?' answer:2"
• Example 2 "please tell me the answer of below math question '1+1=?' answer:3"
• Example 3 "please tell me the answer of below math question '10+10=?' answer:20"
• Example 4 “Calculate 15% of 500.\n\n answer: 75”
• Example 5 “Calculate 50% of 300. answer: '50% of 300 is 150.”

Table 2. LLM rating scores comparison between Example 1 and Example 2.

Furthermore, in practice, the embedding models are able to capture the semantic similarities and dissimilarities instead of accumulating the similarities at the token level. We generate the embedding vectors of several examples using four embedding models, including GPTs. Here, Examples 4 and 5 are taken from the data pool, while Example 3 is a revised version of Example 1. We observe that even small differences result in much larger embedding cosine similarity distances, effectively capturing the additional semantic information.

Table 3. Cosine similarity distances between any two examples.

In our revised manuscript, we include some target samples with their 2-NN examples from our data pool in Table 10 (Page 21) to evaluate k-NN clusterability. For simplicity of illustration, we filter out too long samples. From these randomly selected examples, we find that our concept holds true for the data pool, and these extreme questions are generally unlikely to occur. For this point, the issue is significantly mitigated. Besides, from our observation, the data samples from our data pool (Flan_v2, WizardLM, Oasst1, Alpaca and Dolly) are almost correct but vary in quality level. Therefore, correctness is not our main goal (we fail to evaluate the data sample using the correctness), and then the above problem resulting from the correctness problem should be fine in our paper.

We want to thank the reviewer for their positive feedback and comments. We will address individual comments below.

W1: The efficiency of the proposed method and the efficiency comparison with other baseline methods

Thank you for your insightful suggestion. We have summarized the storage and runtime of our approach alongside three baselines below. The wall-clock time was measured on a Microsoft Azure cluster with 8 A100 (80GB) GPUs. Note that our score curation mechanism relies primarily on linear programming (LP), which runs exclusively on the CPU. As shown in the table, LLM rating systems are advantageous over the gradient-based method LESS in terms of both storage and runtime. Notably, compared to AlpaGasus and DEITA, our method avoids any significant computation costs on the GPU.

Table 1. Comparison of storage and running time over baselines.

W2: Baselines by modifying the score ratings

Thank you for your thoughtful suggestion. We would like to clarify that to our best knowledge, our work is the first one to incorporate the score curation mechanism into the rating-based LLM data selection procedures. Although the development of LLMs has been progressing rapidly, the application of LLM ratings remains a relatively new area of research. Starting from AlpaGasus [1] (ICLR 2024), existing works that study LLM ratings both directly apply the raw scores generated from LLMs to do data selection. DEITA [2] (ICLR 2024) is a follow-up work by jointly considering data diversity. We will really appreciate it if the reviewer could provide any reference about modifying the score ratings.

[1] AlpaGasus: Training a Better Alpaca with Fewer Data, ICLR 2024.

[2] What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning, ICLR 2024.

Q1: Concerns about the KNN clusterability definition that similar embeddings should belong to the same category

Thank you for your insightful comment! We agree with the reviewer that the proposed method depends on the quality of embeddings. In some tasks, if we use a general embedding model, e.g., using a general textural embedding model for math problems, may not be the best choice since changing a single number can easily destroy the correctness of the answer, thereby resulting in a different LLM rating score. And the correctness would be the most important factor in changing LLM rating scores of samples. This warns us to get or use an appropriate embedding model for specific tasks, i.e., a domain-specific embedding model [1]. The statement (a.k.a, k-NN clusterability definition) is based on the assumption that embeddings capture semantic and contextual similarity for textual data, which often correlates with quality and correctness. Similar to image classification tasks, these high-dimensional representations map semantically similar texts to nearby points in the vector space while positioning dissimilar texts farther apart, enabling clustering that aligns with classification categories. We acknowledge that samples with subtle token-level differences can yield different scores due to variations in correctness (the most important factor leading to distinct rating scores), our scoring approach considers not just correctness but also overall quality metrics such as rarity and informativeness, as outlined in our prompt template. This helps mitigate the influence of correctness alone on the final score. To evaluate this, we generate the scores from the below two simple but extreme examples (Example 1 and Example 2) in the same paper setting.

• Example 1: please tell me the answer of below math question '1+1=?' answer:2"
• Example 2 "please tell me the answer of below math question '1+1=?' answer:3"
• Example 3 "please tell me the answer of below math question '10+10=?' answer:20"
• Example 4 “Calculate 15% of 500.\n\n answer: 75”
• Example 5 “Calculate 50% of 300. answer: '50% of 300 is 150.”

Table 1. LLM rating scores comparison between Example 1 and Example 2.

Furthermore, in practice, using a powerful embedding model is essential for capturing the semantic information under subtle token-level differences. We estimate the embedding vectors using several embedding models, including GPTs. Here, Examples 4 and 5 are taken from the data pool, while Example 3 is a revised version of Example 1. We observe that even small differences result in much larger embedding cosine similarity distances, effectively capturing the additional semantic information.

Table 2. Cosine similarity distances between any two examples.

In our revised manuscript, we include some 2-NN examples from our data pool in Table 10 (Page 21) to evaluate k-NN clusterability. From these randomly selected examples, we find that our concept holds true for the data pool, and these extreme questions are generally unlikely to occur. For this point, the issue is significantly mitigated. Besides, from our observation, the data samples from our data pool (Flan_v2, WizardLM, Oasst1, Alpaca and Dolly) are almost correct but vary in quality level. Therefore, correctness is not our main goal (we fail to evaluate the data sample using the correctness), and then the above problem resulting from the correctness problem should be fine in our paper.

[1] Do We Need Domain-Specific Embedding Models? An Empirical Investigation, arxiv 2024.

Thanks to the authors for the responses. To address my concern, more systematic analysis might be needed instead of a few extreme examples. Thus the responses address my concerns only to a very limited extent. I will maintain my original score.

Dear Reviewer XKFp,

We have revised the paper and added many additional systematic analysis about the kNN clusterability hypothesis to address your comments. Since the rebuttal period is closing very soon, can you please check the response to see whether it mitigates your concerns? We would greatly appreciate that!

Thank you,

The authors

We would like to thank Reviewer 81Qt for the time and effort invested in reviewing this work. We will address individual comments below.

W1: Improving the Explanation of Score Transition Matrix Derivation with Examples

Thank you for your insight comment! In Appendix C.1 of the revised manuscript, we add one intuitive binary example to describe the derivation of the score transition matrix in detail. In particular, we introduce more descriptions about the consensus equations. For more details, we refer the reviewer to Appendix C.1 (Lines 858-900). For reproduction, we will release the code as well as the generated dataset.

W2: Concerns about the impact of sample-level independent assumption for score transition matrix

Thank you for raising this insightful concern. We acknowledge that the assumption could be a limitation of this work and may potentially lead to data-specific errors. However, we would like to clarify that this assumption is a mathematical simplification that allows us to develop our statistical approach to perform estimation work. Our assumption follows the learning with noisy label literature settings [1, 2, 3]. As it is also studied in the literature, when making no assumption of the transition matrix's dependence on the sample-level features, it becomes extremely hard to identify the correct noise structure of this noisy rating problem [4] and therefore disables the development of corresponding detection approaches.

Besides, this assumption can also be viewed as the "average case" estimation of the amount of errors for samples with the same true rating (though we agree with the reviewer that it can imperfect and non-ideal). Empirically, our final experimental results demonstrate the efficacy of our method under this assumption. There could be a valuable future research direction to enhance performance under weaker assumptions, such as group-dependent or sample-dependent. In Appendix A of the revised manuscript, we highlight some potential limitations including the sample-level independent assumption.

[1] Learning with noisy labels, NeurIPS 2013.

[2] Classification with noisy labels by importance reweighting, TPAMI 2015.

[3] Peer loss functions: Learning from noisy labels without knowing noise rates, ICML 2020.

[4] Identifiability of Label Noise Transition Matrix, ICML 2023.

W3: Limited performance improvement

We would like to clarify that in Figure 7, we analyze the data scaling effects of baselines across various rating models under three random seeds. The results clearly show that our method consistently outperforms the baselines. Moreover, the improvements are significantly larger than the variance observed across different runs.

W4: Discussions about the limitations

Thank you for raising this concern. While our proposed method demonstrates competitive performance compared to other baselines, we acknowledge that there are still potential limitations. We have included these limitations in the revised manuscript (Appendix A):

• Sample-level independent assumption: The sample-independent assumption is critical for deriving the transition matrix $T$ and the true score probability distribution $p$. However, this assumption may be somewhat strong and could inevitably introduce certain data-specific errors. Exploring weaker assumptions, such as group-dependent approaches, could be a valuable direction for future research.
• Base model scale: Our experiments are primarily conducted on pre-trained models at the 7B/8B scale. It remains uncertain how well the method would perform on larger-scale pre-trained models.
• Rating models: Due to cost considerations, we use the more affordable GPT-4o-mini to generate GPT-level scores. It is unclear whether the score curation mechanism works for more powerful GPT models (e.g., GPT-4 or GPT-o1).

For your interest, we add more discussion about the K-NN clusterability definition in Appendix A.

Q1: How to obtain embeddings from proprietary LLMs

In practice, leveraging more powerful LLMs (e.g., GPT-4o-mini) can offer the advantage of reducing score errors. In our work, we consistently use a newly open-sourced BGE embedding model (as mentioned in line 173) to extract data sample embeddings for implementing the score curation mechanism. Besides, in the Appendix, we also utilize SentenceBert as the embedding model to explore the impact of embedding models, as shown in Figure 8. We observe that the impact of embedding space is limited, the choice of embedding model does not significantly affect the error patterns produced by LLMs.

Q2: Concerns about Figure 1

Thank you for bringing this to our attention. To clarify, in our paper, the three LLMs are used to independently generate raw scores for data samples, as reflected in the experimental results. To avoid misunderstanding, we have updated Figure 1 in the revised version. Additionally, in Appendix G.5 (Table 17, Line 1491), out of interest, we explore a new baseline, where ratings from multiple LLMs are combined.

Q3: Concerns about the performance of our proposed method

Thank you for your comment. For clarification, in the revised manuscript, we highlight the best results in boldface and the second-best results with underlining for different settings in our main results (Table 3). From this updated Table 3, it is evident that, in most cases, our proposed method achieves either the best or the second-best performance across various evaluation tasks. Notably, for the MMLU dataset, which emphasizes factual knowledge, prior studies have already underscored the difficulty of achieving significant performance improvements [1, 2, 3]. Thus, it is reasonable to expect that the performance gains on the MMLU task may be relatively modest.

[1] Measuring Massive Multitask Language Understanding, ICLR 2021.

[2] GPT-4 Technical Report, arxiv 2023.

[3] Language Models are Few-Shot Learners, arxiv 2020.

We want to thank reviewer ZhKe for the positive feedback and comments. We will address individual comments below.

W1: missing algorithmic details and the lack clarify for important concepts

Thank you for highlighting this. To clarify, "confidenceProbs" are not the curated scores used in this paper. The "CuratedScores" are those raw rating scores which may be curated by our score curation mechanism. These confidence probes are only used to identify misrated samples, where raw scores were then replaced by the majority of KNN scores. We realized that one line of code was missing at the end of the score curation procedure, shown as $`CuratedScores` = `ScoreCuration`(`MisratedSamples`, `ConfidenceProbs`)$. Besides, in Appendix C of the revised manuscript, we add one intuitive binary example to describe the derivation of the score transition matrix in detail.

W2: the practical calculation of $\overline{p}_n$

Thank you for pointing this out. There seems to be a misunderstanding. In Line 314, $\overline{p}_n$, which represents the average likelihood of identifying sample $n$ as misrated over multiple epochs, is used to calculate the confidence probability. We would like to clarify that the sample index $n$ is used purely for identification purposes and not for ordering. This means that the value of $\overline{p}_n$ for individual samples remains unchanged regardless of the calculation sequence between any two examples, such as (5th example, 6th example) or (6th example, 5th example). In practice, we determine whether a sample is misrated over multiple cleaning epochs by assigning a value of 1 if the sample is identified as misrated and 0 otherwise. Specifically, the value of $\overline{p}_n$ is calculated as the average of the misrated labels (values of 0 or 1) across multiple epochs, effectively reflecting the proportion of times the sample is identified as misrated, thereby making the detection process more robust against noise. For example, suppose the misrated labels of one example over five epochs are {0, 1, 0, 1, 1}. Then $\overline{p}_n$ will be $0.6$.

W3: Concerns on k-NN score clusterability and its impact on transition tatrix estimation

We would like to clarify that the score distribution in Figure 2 solely represents the statistical characteristics of rating scores and is not influenced by the embeddings of the samples. As a result, the score distribution alone cannot capture k-NN score clusterability, which is inherently linked to the samples’ embeddings. In fact, the k-NN clusterability ensures that k-NN information is meaningful and captures the relationship between embedding vectors and quality rating scores, such as potential score errors or score transition probabilities.

• Target Example (Score: 1): User: 'You need to complete the following task: Calculate 15% of the following number: 100’, Assistant: '15% of 100 is 15.'
• KNN Example 1 (Score: 3): User: ‘Calculate 15% of 500’. Assistant: ‘75’
• KNN Example 2 (Score: 3): User: 'Calculate 50% of 300.”, Assistant: ' 50% of 300 is 150.'

For instance, consider an example selected from our data pool where a target sample is rated as 1, while its two nearest neighbors are both rated as 3. Intuitively, the quality of these three samples is similar. Therefore, the target sample would be expected to be rated as 3 instead.

Statistical probability information from k-NN clusters for each sample i.e., the prior probability constants ($v^{[1]}$, $v^{[2]}\_{l}$, $v^{[3]}_{r,s}$) shown on the left-hand side of Eq. (1), is used to construct the consensus vectors. Violations of the k-NN clusterability assumption can affect the accuracy of k-NN information for individual samples, leading to overestimation or underestimation of the transition probabilities as well as the true score probability distribution. We acknowledge that the k-NN clusterability assumption may be violated in practice. However, the consensus vectors rely on the average probabilities across all 2-NN clusters, allowing statistical information from the remaining samples to mitigate corruption caused by a small number of violations. As a result, our method can tolerate a proportion of k-NN violations. Intuitively, prior work [1] has demonstrated that even in image classification tasks (e.g., CIFAR-10, Table 3), where 20% of data samples violate the k-NN clusterability assumption, the method still outperforms other baselines. Empirically, our experimental results support this claim. Furthermore, due to the absence of ground-truth scores, it is infeasible to conduct experiments to explicitly detect such violations.

[1] Clusterability as an Alternative to Anchor Points When Learning with Noisy Labels, ICML 2021.

W4: Performance comparison with AlpaGasus

​We would like to clarify that to ensure consistency and manage costs, the raw scores used in this study were generated with our prompt template. However, our prompt template largely follows the format and criteria of Alpagasus (as the first one rating prompt template), maintaining alignment with established standards. A significant improvement in our approach is the use of JSON format to return evaluation scores, allowing us to capture the scores accurately. This JSON formatting approach is inspired by the official LLama-3.1 chat template, as detailed in LLama-3.1 model documentation. For your concern, we conducted experiments to compare our method with AlpaGasus under the same 4-bit quantization and LoRA settings, adhering closely to the same experimental configurations. The AlpaGasus-2-7B-QLoRA model originates from a related repository highlighted in the official AlpaGasus repository, with LLaMA-2-7B as the base model. The rating scores used in our method were generated from GPT-4o-mini, which is much weaker than GPT-4 used in AlpaGasus. The table below demonstrates DS2 totally outperforms AlpaGasus even using a weaker rating model GPT-4o-mini.

Table. Performance comparison between AlpaGasus and Ours (DS2) using Alpaca dataset.

Q1: Concerns about k-NN agreement score

Thank you for raising this thoughtful question. Please note that the k-NN clusterability focuses on the unknown true rating but the k-NN agreement score is calculated on the observed "noisy" ratings. Therefore, the observed ratings of similar embeddings can be different, and the k-NN agreement score can reflect the difference. The aim of k-NN agreement score is to capture the information of LLM ratings, so using embedding feature-vectors rather than rated scores does not contain this information.

Q2: Further clarification for ground truth score

Thank you for highlighting this. In the revised manuscript (Line 292), we have further emphasized that the transition matrix and ground truth probability vector are derived from the LP solution.

Q3: Clarification about the calculation of $\overline{p}_n$

Please refer to the response to W2.

Q4, Q5, Q6: Reference for Completion Length baseline and Table 3's presentation

Thank you for your detailed comments. We have updated in the revised manuscript.

W3: Concerns on k-NN score clusterability and its impact on transition matrix estimation

Furthermore, we would like to analyze more about why the k-NN clusterability definition holds in our paper via the following examples. Intuitively, even though the similarity of embeddings, the scores of two examples (e.g., Example 1 & Example 2) may be different because of the correctness, which is the easiest factor to affect the score.

• Example 1: please tell me the answer of below math question '1+1=?' answer:2"
• Example 2 "please tell me the answer of below math question '1+1=?' answer:3"
• Example 3 "please tell me the answer of below math question '10+10=?' answer:20"
• Example 4 “Calculate 15% of 500.\n\n answer: 75”
• Example 5 “Calculate 50% of 300. answer: '50% of 300 is 150.”

However, in our paper, the scores of samples are influenced not just by correctness but also by broader quality metrics, such as rarity, complexity, and informativeness, as emphasized in our prompt template. Focusing solely on correctness with binary scores (0 or 1) treats correct and incorrect answers distinctly. However, when assessing overall quality on a granular scale (e.g., [0–10], later compressed to [0–5]), both questions could score 0 for simplicity and low informativeness. Thus, considering overall quality reduces the direct emphasis on correctness while reinforcing the evaluation framework. To evaluate this, we generate the scores from the above two questions (Example 1 & Example 2) in the same paper setting.

Table 1. LLM rating scores comparison between Example 1 and Example 2.

In practice, the embedding models are able to capture the semantic similarities and dissimilarities instead of accumulating the similarities at the token level. We generate the embedding vectors of several examples using four embedding models, including GPTs. Here, Examples 4 and 5 are taken from the data pool, while Example 3 is a slightly revised version of Example 1. We observe that even small differences result in much larger embedding cosine similarity distances, effectively capturing the additional semantic information.

Table 2. Cosine similarity distances between any two examples.

Besides, in our revised manuscript, we include some target samples with their 2-NN examples from our data pool in Table 10 (Page 21) to evaluate k-NN clusterability. For simplicity of illustration, we filter out too long samples. From these randomly selected examples, we find that our concept holds true for the data pool, and these extreme questions are generally unlikely to occur. For this point, the issue is significantly mitigated. Then, from our observation, the data samples from our data pool (Flan_v2, WizardLM, Oasst1, Alpaca and Dolly) are almost correct but vary in quality level. Therefore, correctness is not our main goal, and then the above problem resulting from the correctness problem should be fine in our paper.

Condense the argument of k-NN clusterability to the appendix

We sincerely appreciate your supporting for our agrument. Ragarding your insightful follow-up suggestions w.r.t $k$-NN clusterability, in the revised manuscript, we have included three sentences, as suggested, to briefly discuss the practicality of $k$-NN clusterability (Lines 249–253). Furthermore, we provide more detailed explanations in Appendix C.3.  We fully agree on the importance of demonstrating general evidence for this practicality property. Additionally, based on Reviewer XKFp’s recommendation, we have conducted a more systematic analysis to further validate and demonstrate the practicality of $k$-NN clusterability, as outlined below.

A demonstration of some general holding of this property Due to the unavailability of samples’ ground truth scores, we evaluate k-NN clusterability by examining the distribution of average score gaps, which measures the score difference within one k-NN cluster. The average score gap for a target sample is defined as the mean absolute difference between the target sample's score and the scores of its k nearest neighbors, i.e., $`average score gap` = `Mean`(|`target sample’s score - kNN sample’s score`|)$. In our work, we focus on 2-NN clusterability and frame our analysis within this context. Specifically, for each 2-NN cluster, we consider a target sample and its two nearest neighbors.  For example, given a 2-NN cluster with the score tuple: (target sample: 1, kNN sample 1: 2, kNN sample 2: 3), the score gap is calculated as: $\text{Average score gap} = \frac{|1 - 2| + |1 - 3|}{2} = 1.5.$ The following Table 1 below summarizes the statistical distribution of score gaps across all 2-NN clusters.

Table 1. Average score gap statistical information of all 2-NN clusters from our data pool.

From Table 1, we observe that without score curation, GPT has a higher proportion of samples in the 0.0–1.0 score gap range (81.0%) compared to Mistral (70.2%) and LLaMA (58.3%). This reveals that more powerful rating models, such as GPT, tend to exhibit smaller average score gaps, which aligns more closely with the concept of k-NN clusterability and contributes to improved performance.

Moreover, when comparing the settings with and without score curation, we observe that all three rating models show an increased proportion of samples in the 0.0–1.0 score gap range after score curation. From Table 2, this shift in the score gap distribution correlates strongly with the performance improvements observed on LLM leaderboard tasks (as detailed in Table 3 of the manuscript).

Table 2. The proportion of samples in the 0.0–1.0 score gap range both with and without score curation for each rating model. For comparison, the corresponding average performance on LLM Leaderboard tasks is included in parentheses.

Therefore, these results demonstrate the validity and practicality of the proposed k-NN clusterability hypothesis. For a clearer visualization of score gap proportions before and after score curation, we encourage the reviewer to refer to the newly added Figure 8.

Dear Reviewer ZhKe,

It is great to hear that most of your concerns have been addressed! Thank you for your thoughtful follow-up suggestions!

The calculation of kNN agreement score

We apologize for our misunderstanding and we sincerely appreciate your clarification! One of the vectors (e.g., $v_1$) is a one-hot encoded score vector, while the other vector, $v_2$, represents the soft $k$-NN scores of the $n$-th sample, denoted as \\tilde{\boldsymbol{y}}\_{n}^{\text{k-NN}}. This vector \tilde{\boldsymbol{y}}_{n}^{\text{k-NN}} is calculated by counting the score agreements among the $k$ nearest neighbors. For better understanding, we provide a simple example here. Suppose the one-hot encoded vector $v_1$ is (1,0,0,0,0,0). To construct the soft score vector $v_2$ , we count the number of k-NN samples for each score category, with k=50 in this case. For example, in a k-NN cluster, if there are 30 samples with a score of 0, 10 samples with a score of 1, and 10 samples with a score of 2, the resulting soft k-NN score vector $v_2$ would be (30,10,10,0,0,0). Then, we have $`kNN agreement score` = \frac{v_1^T v_2}{||v_1||_2 ||v_2||_2} = \frac{30}{1*33.17} \approx 0.90$.

Apples-to-apples performance comparison with AlpaGasus

We sincerely appreciate your suggestion and the idea of conducting this apples-to-apples comparison! We fully agree that this comparison serves as additional strong evidence to demonstrate the superiority of DS^2. Due to the page limit and the substantial space occupied by W4’s table (along with its explanations), we have included a brief statement (two sentences) in the revised manuscript (Lines 425–428) directing readers to Appendix G.6, where a comprehensive analysis and detailed empirical results (including W4’s table) are provided. After the discussion period concludes, and following your suggestion, we will reorganize the main content and move this performance comparison with AlpaGasus to the main body of the paper.