Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance

Thank you for your positive and thorough review and enlightening comments. We are happy to learn that you found our paper thorough and convincing. We hope you find answers to your questions and concerns below:

Binary labels: We acknowledge that our experiments include only binary labeling tasks. Although the labeling scheme is designed for factual consistency, the underlying tasks are diverse, including summarization, dialogue, fact verification, and paraphrasing. Please note that our approach can be extended to multi-class datasets, as generally described in Subsection 2.2.

Approved HITs: We agree that tightening the qualifications could have resulted in better annotators and, consequently, more reliable results. However, the scale of ~100+ approved HITs, combined with a high ~95%+ approval rate, is quite standard [1, 2, 3]. We will add these citations to the paper to support our choice.

Fleisses Kappa: First, we double-checked the calculation and confirmed its validity. As to why it has a low score, additional evidence for low agreement between annotators is shown in Figure 5, where there was little consensus on most examples. Furthermore, Figure 2 shows a low F1 score compared to other annotation methods, indicating the poor quality of the annotations. This low quality is reflected in both the accuracy of the annotations and their consistency (IAA). We believe other factors such as task difficulty or annotator fatigue may also contribute to these results.

Thank you for your meticulous reading, which uncovered these typos. We will, of course, fix them.

References:

[1] Kazai, Gabriella, J. Kamps and Natasa Milic-Frayling. “An analysis of human factors and label accuracy in crowdsourcing relevance judgments.” Information Retrieval 16 (2012): 138 - 178.

[2] Hauser, David N., Aaron J. Moss, Cheskie Rosenzweig, Shalom N. Jaffe, Jonathan Robinson and Leib Litman. “Evaluating CloudResearch’s Approved Group as a solution for problematic data quality on MTurk.” Behavior Research Methods 55 (2021): 3953 - 3964.

[3] Chmielewski, Michael and Sarah C. Kucker. “An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results.” Social Psychological and Personality Science 11 (2019): 464 - 473.

References:

[1] Chong, Derek, Jenny Hong, and Christopher D. Manning. "Detecting label errors by using pre-trained language models." arXiv preprint arXiv:2205.12702 (2022). ‏

[2] Pleiss, Geoff, et al. "Identifying mislabeled data using the area under the margin ranking." Advances in Neural Information Processing Systems 33 (2020): 17044-17056. ‏

[3] Swayamdipta, Swabha, et al. "Dataset cartography: Mapping and diagnosing datasets with training dynamics." arXiv preprint arXiv:2009.10795 (2020).‏

[4] Huang, Hsiu-Yuan, et al. "A Survey of Uncertainty Estimation in LLMs: Theory Meets Practice." arXiv preprint arXiv:2410.15326 (2024). ‏

[5] Klie, Jan-Christoph, Bonnie Webber, and Iryna Gurevych. "Annotation error detection: Analyzing the past and present for a more coherent future." Computational Linguistics 49.1 (2023): 157-198. ‏

[6] Wang, Xinru, et al. "Human-LLM collaborative annotation through effective verification of LLM labels." Proceedings of the CHI Conference on Human Factors in Computing Systems. 2024.‏

[7] Kholodna, Nataliia, et al. "Llms in the loop: Leveraging large language model annotations for active learning in low-resource languages." Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Cham: Springer Nature Switzerland, 2024. ‏

[8] Zhang, Ruoyu, et al. "Llmaaa: Making large language models as active annotators." arXiv preprint arXiv:2310.19596 (2023).‏

Thank you for your detailed review. We hope you find answers to your questions and concerns below:

Similar approaches: We respectfully disagree with the assertion that our work lacks substantial innovation. Our research offers a comprehensive and thorough investigation of label errors, addressing both detection and mitigation strategies using LLMs.

We would like to highlight key distinctions between our method and similar approaches. While there is extensive literature on detecting label errors for fine-tuned models, most of these studies focus on detection within the training loop [1, 2, 3], which is not directly applicable to LLMs when fine-tuning is not feasible. Second, most existing work involving LLMs identifies problematic examples by analyzing model uncertainty, typically characterized by high entropy, low confidence in predictions, or external verifiers [4, 5, 6]. Our approach examines the confidence in the incorrect label, but only in cases where the original label disagrees with the prediction. Lastly, unlike prior work, our method leverages an ensemble of LLMs and prompt designs to enhance robustness [7, 8].

Additionally, our contributions extend far beyond the methodological novelty (which constitutes only a small part of our paper) to include a comprehensive end-to-end analysis of detecting label errors, addressing and re-annotating them, and mitigating their impact on downstream tasks.

First, we demonstrate that as LLM confidence increases, their precision in flagging mislabeled examples also rises. This confidence-based approach provides a more nuanced understanding of LLM reliability in error detection, which has not been explored in prior research.

Furthermore, our extensive empirical investigation into the impact of label errors reveals the significant distortion these errors can cause in model performance. By correcting mislabeled data, we illustrate that performance can improve by up to 15%, and we show that many LLM-reported mistakes are, in fact, errors in the original labels. This finding challenges assumptions made in previous evaluations and emphasizes the need to reassess established benchmarks. We also address how to mitigate the negative effects of label errors during model fine-tuning. Our proposed methods, including filtering and label flipping based on LLM confidence, yield up to a 4% improvement in model performance. This practical approach highlights how our strategies can make models perform better and be more robust. Lastly, our work contributes significantly to the broader understanding of employing LLMs in annotation tasks. We comprehensively compare LLM-based error detection with traditional methods, revealing both strengths and limitations. By doing so, we provide a well-rounded analysis that informs future research and offers new ways to improve data quality.

Multi-class or ambiguous labeling scenarios: Our general approach, as described in Subsection 2.2, is not limited to binary labeling and can be extended to multi-class datasets. We specifically chose datasets from the TRUE benchmark to explore various domains and tasks while benefiting from a unified labeling schema, which provided consistency and comparability across our experiments. While we did not explicitly address fuzzy labeling scenarios in this paper, we believe our approach could be adapted to handle such cases by defining terms like “disagree” or “strongly disagree” with more nuance (e.g., using KL-divergence between the label distributions to quantify the disagreement level).

Validation of our approach: Our research includes extensive experiments that validate our method across multiple datasets, covering various tasks and domains. While we acknowledge that using more datasets would strengthen our findings, budget constraints played an important role, given the need to obtain annotations from multiple LLMs, crowd-workers, and experts. We hope that, once published, our study will inspire and motivate other researchers to apply and extend our approach in diverse scenarios (e.g., exploring other datasets, tasks, or, as you suggested, handling multi-class and ambiguous labeling cases), further validating and building upon our work.

Demographic information: We acknowledge the importance of such information and will gladly provide it in a camera-ready version if accepted. We ensure our expert annotators are experienced and knowledgeable in the field.

Few-shot: Our claim that LLMs show considerable agreement with expert annotators in a zero-shot setting for mislabeled samples is supported by our findings, which demonstrate that zero-shot performance is already strong. We acknowledge that few-shot learning generally improves performance, and we believe this would also be true in our case, even if the few-shot exemplars contain some label errors. Several factors contribute to this improvement: few-shot exemplars are often carefully selected, which reduces the likelihood of errors; they are also primarily used to help the model adjust to the domain, context, and format of the task rather than to explicitly teach labeling rules [1, 2]. Moreover, even if the error rate is not zero, most data is correctly labeled, further aligning the model's predictions with expert annotations.

Also, thank you for your suggestions, we will include them in our next revision.

We hope you find answers to your concerns and questions in our response. Should any questions arise, we would be happy to respond.

References:

[1] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064

[2] Cheng, Chen, Xinzhi Yu, Haodong Wen, Jinsong Sun, Guanzhang Yue, Yihao Zhang and Zeming Wei. “Exploring the Robustness of In-Context Learning with Noisy Labels.” ArXiv abs/2404.18191 (2024): n. pag.

Thank you for your thorough response, we are happy you found our paper interesting and clear. We wish to address your concerns:

Sample size and the number of datasets: We annotated examples by multiple LLMs, crowd-workers, and experts, and as you mentioned, budget control had to be considered. However, we disagree that the number of examples or the number of datasets is too small to make reliable conclusions. Please note that, when possible, for all analyses and experiments in the paper, we provide confidence intervals or confidence-based lower bound estimations that our conclusions take under consideration (e.g., Figure 1 and lines 275-278, Figure 4, Table 2). For example, in Figure 1, for the range [0.99,1.0] even the lower bound of the CI is above the 50% line, suggesting that for this bin of examples, we are certain that the LLM is correct more than the original label. In Figure 4 the whiskers show that the differences between the methods cannot be attributed to chance. In Table 1, we present a lower bound estimate of the error rate across the entire dataset (lines 247-253), and even at this conservative estimate, the error rate remains substantial. Although the sample size might be considered small, our use of robust statistical practices ensures the reliability and validity of the findings and conclusions. Additionally, our datasets span over domains and tasks, which supports the generalization of our findings across different contexts.

Crowd-source performance: It is common for crowd-sourced annotators not to be employed in their basic form (i.e., untrained, common crowd-workers). Typically, these annotators undergo a selection process involving qualification tests and targeted training to improve their skills for the task and experience. Indeed, our findings, as shown in Figure 3, support that more experienced annotators provide better-quality labels. However, these trained workers fall on a spectrum somewhere between plain crowd-workers and expert annotators, rather than representing typical crowd-sourcing. Our paper focuses on the edges of that spectrum, using untrained, plain crowd-workers who do not receive task-specific training; nevertheless, note that they still meet basic pre-qualifications, such as having a history of approved HITs and a minimum approval rating. This likely explains the discrepancy noted. We stand behind our collection process (as fully detailed in subsection 3.2 and Appendix B.1), and therefore by the results it produces (e.g., in Figure 2 and Table 3), which show relatively poor performance from crowd-sourcing.

On why LLMs align more closely with expert annotations despite training on crowd-labeled data– LLMs are trained on diverse datasets including expert-labeled data, data annotated by trained crowd-workers, and a substantial amount of unsupervised text. This broad exposure enables LLMs to generalize effectively and recognize subtle patterns, leading to a closer alignment with expert labels. In contrast to open-source models, which are trained primarily on publicly available datasets, closed-source models are more likely to optimize their training data through data cleaning and constructing custom, expert-annotated datasets. Our findings further illustrate this: experienced crowd-workers achieve an F1 score of approximately 0.73 (Figure 3), on par with open-source LLMs that reach around 0.7, while closed-source LLMs perform even better, obtaining an F1 score of ~0.8 (Table 4). This pattern supports your hypothesis: as LLMs rely more on crowd-sourced labels, their performance decreases, aligning less with expert annotations. This underscores our paper's emphasis on the crucial role of data quality.

Guidelines: Upon manually examining mislabeled examples, we can confidently say that most label mismatches originate from errors, not guideline discrepancies. For example, see the annotations in Table 8. The label errors we identified are clear errors and do not stem from any potential misinterpretation of guidelines.

Yet, we acknowledge that some minor discrepancies might exist between the original guidelines and the merged ones, and we will add this as a limitation in the discussion. However, our datasets are taken from a widely used benchmark for factual consistency - TRUE. As such, they all share a common meaning for labels (i.e., “consistency” should mean the same). When using the TRUE benchmark for evaluating models, the same prompt is used for every dataset, or the same fine-tuned model is evaluated on each dataset.

Therefore, having unified guidelines, even if they were not explicitly established before, is reasonable, valid, and should not create major discrepancies from the original separate guidelines. We made efforts to stay true to the core characteristics and intentions of the original guidelines and took a strict approach, as described in TRUE. This is reflected in our careful use of terms like “all” or “any” in our instructions.

I would like to thank the authors for their responses. However, my concerns remain unaddressed for the following reasons:

Sample size and the number of datasets

Regarding the robustness of the statistical tests you performed, since all such tests rely on estimations, the conclusions drawn heavily depend on the sample size used. If the sample is too small, it does not yield meaningful conclusions, regardless of the strength of statistical test employed.

Crowd-source performance

With due respect, I find your argument unconvincing regarding the statement:

"LLMs are trained on diverse datasets, including expert-labeled data, data annotated by trained crowd-workers, and a substantial amount of unsupervised text. This broad exposure enables LLMs to generalize effectively and recognize subtle patterns, leading to a closer alignment with expert labels."

Your argument is not accurate for two reasons:

1. A significant proportion of the datasets are labeled via crowd-sourcing, which outweighs datasets labeled by experts. Therefore, it is more likely that models learn predominantly from crowd-sourced labeled data, which, according to your argument, are not of high quality and models cannot learn the mentioned nuances from these datasets.

2. The purpose of unsupervised learning is to capture general patterns, not task-specific patterns. If your claim were accurate, we would not need supervised fine-tuning or other alignment-focused training paradigms, as the model would already be capable of learning even subtle nuances through unsupervised learning alone. Clearly, this is not the case.

Guidelines

While it is true that any dataset may contain mislabeled data, as you illustrated in Table 8, making broader claims requires strict adherence to consistent guidelines. Without such guidelines, a majority of mislabeled data could be misinterpreted as genuine errors, rather than being true mislabels.

Few-shot

I did not mean using mislabeled demonstrations in in-context learning (ICL). My point was that, in few-shot ICL, where demonstrations are added randomly (considering the simplest case), performance improvements are commonly observed. These improvements are measured based on labels provided by crowd-sourced annotators. However, according to your results, as LLMs outperform crowd-sourced annotators in aligning with expert annotations, a decrease in performance should be expected due to the discrepancy you identified between crowd-sourced and expert annotators. However, this is not the case, as few-shot ICL typically enhances performance in most scenarios. How do you explain this apparent contradiction?