LEMoN: Label Error Detection using Multimodal Neighbors

Thank you for the detailed review and constructive suggestions!

W1: The theoretical justifications provided in the paper contain several significant flaws, limiting their effectiveness as a core claim.

We appreciate your valuable feedback on our theoretical analysis. We acknowledge that there were flaws in our initial proof. However, the underlying intuition – that contrastive multimodal embedding models can detect noisy labels due to increases in embedding distances caused by label noise – remains valid. We have now revised our theorem and proof (Theorem 4.1 and Appendix A) to correct these errors.

Theorem 4.1 [Contrastive Multimodal Embedding Models Detect Noisy Labels] Let $\mathcal{Y} = \mathbb{R}$ and consider a training dataset $\mathcal{D}$. Suppose that $\hat{h}^{\mathcal{X}}\_{\theta}: \mathcal{X} \rightarrow \mathbb{R}^d$ is an embedding function, and $\hat{h}^{\mathcal{Y}}\_{\theta}: \mathcal{Y} \rightarrow \mathbb{R}^d$ is a Lipschitz continuous embedding function with constants $L\_{\mathcal{Y}} > 0$, meaning that for all $y, y' \in \mathcal{Y}$, $\left\|\left\| \hat{h}^{\mathcal{Y}}\_{\theta}(y) - \hat{h}^{\mathcal{Y}}\_{\theta}(y') \right\|\right\|\_2 \leq L\_{\mathcal{Y}} | y - y' |.$ For an input $x \in \mathcal{X}$ and its corresponding positive label $y \in \mathcal{Y}$, let $\eta$ be a random variable drawn from a normal distribution: $\eta \sim \mathcal{N}(0, \sigma^2).$ Define a noisy label $y' = y + \eta$. Let $d_{mm}(u, v) = ||u - v||\_2$, which is proportional to $\sqrt{d\_{cos}(u, v)}$ when $||u||\_2 = ||v|\|_2 = 1$. Then, with probability at least $\delta(\epsilon) = 1 - 2 \Phi\left( -\dfrac{\epsilon}{\sigma} \right)$, the following inequality holds: $d\_{mm}\left( \hat{h}^{\mathcal{X}}\_{\theta}(x), \hat{h}^{\mathcal{Y}}\_{\theta}(y') \right) \geq d\_{mm}\left( \hat{h}^{\mathcal{X}}\_{\theta}(x), \hat{h}^{\mathcal{Y}}\_{\theta}(y) \right) - L\_{\mathcal{Y}} \epsilon ,$ where $\Phi$ is the cumulative distribution function of the standard normal distribution, and $\epsilon > 0$ is a threshold.

Thus, when $L_{\mathcal{Y}}$ is small, the score for the mislabeled sample cannot be much lower than the score for the positive pair with high probability.

The revised proof can be found in Appendix A.1.

In the proof of Theorem 4.2 (line 954), the variance of 1/k E should be expressed as 1/k² VarE, rather than 1/k VarE.

We believe our original proof is actually correct -- there is indeed a $1/k^2$ term, but one of these factors gets canceled out by the summation over k iid random variables. Concretely:

$\\mathbb{E}[Var(S_m(X, Y) | \\zeta_Y)]$

$= \\mathbb{E}[\\frac{1}{k^2}Var\\left(d(X, \\bar{X}_1) + d(X, \\bar{X}\_\{k \\zeta_Y (1-p)\})+ d(X, X'_1) + ... + d(X, X'\_{k - k\\zeta_Y(1 - p)})\\mid \\zeta_Y \\right)]$

$= \\mathbb{E}[\\frac{1}{k}(\\zeta_Y (1-p) \\sigma_2^2 + (1 - \\zeta_Y (1-p)) \\sigma_1^2 )]$

We have added this intermediate step in the proof in Appendix A.2. Please let us know if this addresses your concerns.

W2: While the paper claims novelty in applying multi-modal scoring to label noise detection, this approach has recently been explored in [1].

Q2: How does LEMoN compare to VDC [1] in terms of strengths and weaknesses?

Thank you for this reference! First, we would like to emphasize that VDC is only evaluated on classification datasets, while our method is able to detect label errors in classification and captioning datasets.

Conceptually, our method is also very different from VDC: VDC relies on an LLM to generate questions about the consistency of a label to a given image, uses a multimodal large language model to automatically generate answers for each image and question, and then evaluates how well the generated caption matches the actual label. Thus, VDC entirely relies on prompting LLMs and VLLMs. In contrast, our method does not utilize any prompt engineering, and instead utilizes the neighborhood information in contrastively trained representations of image and text representations. In addition, our method is more principled in that we have theoretical guarantees for the performance of our multimodal neighborhood score (Theorem 4.2).

Third, our method outperforms SimiFeat-R, a method that VDC performs comparably (Table 4 in VDC paper) by a significant margin (3-6 AUROC points across different noise types; Table 2 and I.2 in our submission). In principle, the idea of prompting a VLLM is similar to our LLaVa baseline, which LEMoN also far outperforms.

Fourth, our method uses fully open-source models, while VDC relies on ChatGPT based on GPT-3.5-turbo for their question generation and answer evaluation portions, and they do not evaluate any open-source alternatives.

Finally, in order to achieve the prompting abilities required, VDC utilizes models that are much larger than ours. For example, the InstructBLIP used in VDC contains 7B parameters, which is 46x as large as the CLIP ViT-B-32 (151m parameters) used in our paper. This certainly has negative implications for their inference time, which is important in applying such methods to billion-scale datasets as the reviewer pointed out. As VDC also utilizes ChatGPT, this would be extremely costly as well.

We are currently working on adapting VDC to our problem setup of detecting mislabeled image caption pairs, utilizing only open-source models. Results for this baseline will be added in a future revision.

W3: The provided source code does not include implementations of the baseline methods. As a result, it remains unclear how the hyperparameters for these baseline methods were tuned, particularly given that the authors introduced a new set of synthetic datasets. Q1: Which implementations of the baseline methods were used, and how were their hyperparameters selected?

Thanks for raising this point! We highlight that all the hyperparameter settings for the baselines are in Appendix G. Based on the reviewer’s suggestion, we have updated the supplementary code to include baselines, including the hyperparameter grids used for running each baseline in experiments.py. For all baselines, the hyperparameters were selected based on the validation set F1-score, matching the $\text{LEMoN}\_{\text{opt}}$ setting. For Simifeat, we use the open-sourced implementation directly from https://github.com/Docta-ai/docta/tree/master. We have clarified this in Appendix G in the revised paper.

Thank you for addressing the concerns raised earlier. However, there are still several areas that require further clarification and elaboration. Please consider the following questions and comments:

W1. Theory

1. Line 236 (Theorem 1): The statement implies that the distance from the incorrect label $y'$ can be equal to the distance from the correct label $y$, even in a carefully designed case. It follows that Lemon may NOT be able to distinguish between correct and incorrect labels.

2. Line 839 (Proof of Theorem 2): In the proof, $p$ is described as a probability, meaning the exact number of relevant neighbors is unknown. Consequently, $S_m$ cannot be directly decomposed into two sums with a fixed number of terms in each.

3. Line 844 (Proof of Theorem 2): The variable over which the expectation is computed appears to be missing. Additionally, why are nested expectations of the form $\mathbb{E}[\mathbb{E}[\dots]]$ necessary in this context?

W2. Novelty

• Line 151: The claim that "we believe we are the first to apply it to the setting of label error detection" must be revised in light of VDC.

W3. Hyperparameters

• The appendix currently includes only the ranges of hyperparameters. For reproducibility and transparency, could you provide the exact hyperparameter values used during the evaluation process?

We appreciate your effort and look forward to seeing the updated evaluation results and responses to these questions.

Thank you for your detailed response and additional questions!

W1.1: The statement implies that the distance from the incorrect label $y'$ can be equal to the distance from the correct label $y$.

We agree with this characterization, and admit that the lower bound is weak as a result. We stand by our statement that "when $L_Y$ is small, the score for the mislabeled sample cannot be much lower than the score for the positive pair with high probability."

Additionally, we emphasize that Theorem 4.1 is a relatively minor contribution of our paper. It largely serves to justify prior works which already utilize the CLIP score for label error detection (Kang et al., 2023; Liang et al., 2023). We believe that Theorem 4.2 sufficiently justifies the main methodological novelties of our work (the scores $s_m$ and $s_n$).

W1.2: In the proof, $p$ is described as a probability, meaning the exact number of relevant neighbors is unknown. Consequently, $S_m$ cannot be directly decomposed into two sums with a fixed number of terms in each.

Thank you for pointing this out! We have now clarified this (in L258) by defining $p$ to be the fraction of mislabeled examples in the nearest neighbors set:

Suppose that $\frac{1}{k}|\\{i: (X_{m_i}, Y_{m_i}) \text{ is mislabeled}\\}| = p$ is constant for all samples in the support of $(X, Y)$.

W1.3: The variable over which the expectation is computed appears to be missing. Additionally, why are nested expectations of the form E[E[…]] necessary in this context?

The nested expectations originate from the law of iterated expectations, i.e. $\mathbb{E}[S_m(X, Y)] = \mathbb{E}[\mathbb{E}[S_m(X, Y) | \zeta_Y]]$. Here, we have used the standard notation that an expectation (without subscript) is computed with respect to the joint distribution of all random variables in the expectation.

Conditioned on $\zeta_Y$, the term $k\zeta_Y(1-p)$ (the exact number of relevant neighbors) is a known constant. There is one implicit assumption that we have made, which is that $\zeta_Y$ is distributed such that the random variable $k\zeta_Y(1-p)$ has support only over the integers $\\{0, 1, ..., k\\}$. We have clarified this assumption in L838.

W2: The claim that "we believe we are the first to apply it to the setting of label error detection" must be revised in light of VDC.

To clarify, Line 151 in our paper states: "Although prior works have utilized the idea of multimodal neighbors in other settings, we believe we are the first to apply it to the setting of label error detection." We believe this is still true since VDC does not utilize multimodal neighbors in any form.

We have cited VDC in our updated revision. We are working on adapting VDC to our problem setting now with open-source models, and hope to provide results for it before the end of the rebuttal period.

W3: The appendix currently includes only the ranges of hyperparameters. For reproducibility and transparency, could you provide the exact hyperparameter values used during the evaluation process?

We have added Tables G.1 and G.2 in the appendix, which lists the optimal hyperparameters for all baselines.

Please let us know if this sufficiently addresses your concerns. We very much welcome any additional feedback that can further strengthen the paper!

The paper does not fairly evaluate the proposed method against its closest baseline [3].

For a fair evaluation, the baseline must be implemented using combined DIS and DIV scores across both modalities, consistent with how LeMON is evaluated.

As we have argued both intuitively (in Appendix B) and empirically (in Table I.12), $\Upsilon\_X^{DIS}$ is the only score from [3] that contributes any signal to label error detection. In fact, the average AUROC (across eight datasets) of the remaining three terms $\Upsilon\_Y^{DIS}$, $\Upsilon\_X^{DIV}$, $\Upsilon\_Y^{DIV}$ are, respectively: 49.5 (1.7), 51.8 (5.0), and 49.3 (2.1), where parentheses show standard deviation across the eight datasets. Clearly, these remaining three terms are no better than random chance at detecting label errors.

Regardless, we have implemented the ensembling of the four scores as the reviewer suggested. For the $\text{Comb-Val}$ strategy, as there are four terms, we sweep over weights in $\\{1, 2, 3, 4, 5\\}^4$, following [3], selecting the best combination using a labeled validation set, identically to LEMoN. For the $\text{Comb-Stat}$ strategy, we use the mean and standard deviations, as in Equation (8) in [3]. We find that none of the combined scores significantly outperform $\Upsilon\_X^{DIS}$. This is because in both combination strategies, a non-zero weight is placed on the other terms, which essentially adds noise to the final score without contributing any signal.

We will add this result to Table I.12 in the revision.

Thank you for your response.

Theorem 1 claims that "Contrastive Multimodal Embedding Models Detect Noisy Labels," but the conclusion does not support this claim. The distances in the final inequality can remain equal even with multiple strong constraints.

We have already addressed this point in a previous response. Specifically, we admit that this is a weak lower bound, and that:

We stand by our statement that "when $L_Y$ is small, the score for the mislabeled sample cannot be much lower than the score for the positive pair with high probability."

Additionally, we emphasize that Theorem 4.1 is a relatively minor contribution of our paper. It largely serves to justify prior works which already utilize the CLIP score for label error detection (Kang et al., 2023; Liang et al., 2023). We believe that Theorem 4.2 sufficiently justifies the main methodological novelties of our work (the scores $s_m$ and $s_n$).

As such, the primary contributions of our paper (L87-L97) hold even without this theorem.

Theorem 2 is trivial, as it simply states that random variables with different distributions can be distinguished to some extent.

We completely disagree that a theorem is trivial just because "it simply states that random variables with different distributions can be distinguished to some extent". For example, a large portion of the field of statistical hypothesis testing deals exactly with designing and analyzing methods to distinguish between random variables with different distributions.

Regardless, we note that (a) we derive an analytical solution for exactly the extent to which the distributions can be distinguished, as a function of the parameters of the distributions, and (b) Theorem 2 provides theoretical justification for our proposed method (the scores $s_m$ and $s_n$). For these reasons, we do not believe this theorem is "trivial", and we believe it is unfair to dismiss it as so.

Assumption 2 is not properly validated in Appendix A.3. The appendix presents dataset-level statistics, whereas Assumption 2 pertains to individual image-label pairs.

This is simply incorrect. First, Assumption 2 pertains to the distribution of distances between images and their "paraphrases", not individual image-label pairs. Second, this distribution is precisely what we show in Appendix A.3 (Figure A.1), and we even conduct a test for normality, showing that this assumption does indeed hold.

The paper lacks a thorough analysis of the distinction between cross-modal retrieval and label noise detection.

It uses CLIP, a retrieval model, for label noise detection, but similar approaches have been explored in prior work. For instance, label noise detection via retrieval has been discussed in Bahri et al. [1], and multimodal retrieval has been studied in [2,3,4]. This overlap makes the novelty unclear.

We respectfully disagree with the reviewer’s point that “similar approaches have been explored in prior work" for label noise detection. As we stated in the prior response (L151): "Although prior works have utilized the idea of multimodal neighbors in other settings, we believe we are the first to apply it to the setting of label error detection." Our novelty in methodology does not lie in using CLIP, but rather proposing a novel multimodal neighborhood-based score for the task of label noise detection. As the reviewer themselves notes in their original review, we propose an “original scoring method for label noise detection”.

Furthermore, we have already compared against the baselines the reviewer specified [1,3], and have shown that LEMoN outperforms them: deep kNN [1] in Table 2 and Table 3, and Discrepancy/Diversity [3] in Table 2 and Table I.12. The other references [2,4] that the reviewer has identified do not propose a score that can be used for label noise detection, but focus on improving multimodal matching and retrieval. Thus, they are not directly applicable to this setting. We will note this in our revised paper.

Lastly, the only connection between our work and cross-modal retrieval [2,3,4] is that these methods are multimodal [2,3,4], and may use neighborhood-based strategies [1,4]. Importantly, we already cite and describe how our works differ from prior multimodal neighborhood-based works in L142-L152 and Appendix B (which the reviewer already references). Additionally, works in cross-modal retrieval [2,4] are orthogonal and potentially complementary to our work: better multimodal retrieval could lead to more accurate construction of multimodal neighborhoods, and thus better scores for label noise detection under our framework. Thus, we strongly disagree with the reviewer’s point that our work “lacks distinction between cross-modal retrieval and label noise detection”.

Thank you for the detailed review and constructive suggestions!

W1: Their approach seems to be mainly designed to detect label errors in image-caption datasets. However, the effectiveness of applying filtering to such a dataset seems to be marginal according to Table 4.

We highlight that in Table 4, even training with a fully clean dataset only outperforms training with a fully noisy (i.e. no filtering) dataset by 2-3 BLEU-4 points. Thus, the range of potential improvement for any filtering method is bounded by this difference. Adding on the variance associated with model training (standard deviations up to 1.0), it is difficult for any method to outperform another with significance. This is an interesting result, since it means that some pre-trained captioning models are stable or have small-sized performance drops in the presence of noisy captioning data. We interpret the result from Table 4 to be that both LEMoN and CLIP Sim. are nearly able to recover the performance of clean data for both datasets. In addition, we note that LEMoN performs slightly better in mscoco, partly because its larger dataset size results in a smaller variance.

In addition, we highlight that in the classification setting, LEMoN is consistently among the top-2 methods across all four datasets in downstream performance. Lastly, in addition to filtering out incorrect data, such label error detection methods are also useful in and of itself. For example, identifying mislabeled samples (and potentially annotators responsible for these mislabels) can lead to improvements in labeling processes. Thus, we strongly believe that label error detection using LEMoN has real-world utility. We will also demonstrate the utility of LEMoN to effectively filter data for downstream training in Datacomp below.

W2: However, there can be more diverse types of noise in real image-caption data collected from the web.

W3: Also, according to their appendix, their metric does not show much gain in the CC3M, which is webly collected. I actually feel that the authors had to conduct more analysis on this kind of dataset since their main motivation seems to detect errors on this kind of dataset, rather than image-classification dataset.

Thank you for this suggestion. We have conducted a new experiment of $\text{LEMoN}\_{\text{fix}}$ on Datacomp [1]. We use the small dataset from the filtering track, which originally consisted of 12.8M images. As these images are accessed directly from the web, only 9.96M images were able to be downloaded as of 2024/11/14. We apply $\text{LEMoN}\_{\text{fix}}$ to this dataset using OpenAI CLIP ViT-L/14 embeddings provided by Datacomp. We select the 3.5M images with lowest mislabel scores, and use the default hyperparameters from Datacomp to train a CLIP model, and evaluate it on the same 38 zero-shot classification datasets. We compare with filtering using only the CLIP score to the same number of images.

We find that given the available images, LEMoN outperforms the baseline on average, and on three of four individual evaluations. However, neither method outperforms the scores reported in the original paper due to their dataset being larger.

We have added this table and discussion to Appendix I.10.

Thank you for the detailed review and constructive suggestions!

W1: The current experiments lack the breadth needed to fully demonstrate the impact of adding nearest neighbor terms. It would be beneficial to include a comparison using only single-side nearest neighbor term, and to present the actual values of all three terms for clearer insight.

Thank you for the suggestion. We have conducted a full ablation study of the performance of each of the three terms in LEMoN, for all datasets. The following table shows the AUROC of each score and all possible combinations:

We find that $d_{mm}$ is the most critical term. Of the two nearest neighbors terms, we find that $s_n$ (nearest image neighbors) is more important in general, though this is highly dataset dependent, e.g. error detection in mmimdb relies much more on neighbors in the text space than the image space, while the opposite is true for mscoco.

We have added this table to the appendices, and discussed it briefly in Section 6.3.

Q1: In section 3, a few symbols are not explained, like r, D, k etc. It's better to split the section 3 into multiple sub sections to explain each term separately

Thank you for pointing this out. The symbol $\mathcal{D}$ was defined at the start of Section 3, and we have clarified the notation in Section 3 by adding several explanations:

• Define $B(\mathbf{x}, r) := \\{x' \in \mathcal{X}: d_{\mathcal{X}}(\mathbf{x}, \mathbf{x}') \leq r \\}$, the ball of radius $r$ around $\mathbf{x}$, and $B(\mathbf{y}, r)$ similarly.

• Let $r_k(\mathbf{x}) := \inf\\{ r: | B(\mathbf{x}, r) \cap \mathcal{D}| \geq k \\}$, the minimum radius required to encompass at least $k$ neighbors.

Further, we have added Table C.1 in Appendix C, which provides a summary of all the notation used in this section. Please let us know if there are any remaining issues!

Q2: Is there any explanation that why using pure CLIP similarity performs better than LEMoN on Flickr30k?

We note that due to the large error bars, CLIP similarity does not actually outperform LEMoN with statistical significance on flickr30k (p=0.63 for AUROC and p=0.20 for F1 from paired t-tests). Since LEMoN is a generalization of CLIP similarity, we would expect $\text{LEMoN}\_{\text{opt}}$ to outperform CLIP similarity when sufficient validation data is available. As the dataset size for flickr30k is small, this results in (1) larger variance in test-set metrics as seen in Table 3, and (2) hyperparameters of $\text{LEMoN}\_{\text{opt}}$ overfitting to the smaller validation set. We explore the second phenomenon further in Appendix I.7., where we vary the validation set size.

Thank you for the detailed review and constructive suggestions!

W1: The details of the application on the unimodal dataset need to be clarified. How to define the nearest neighbors of text in unimodal datasets like CIFAR10/100？

Similar to the original CLIP paper [2], to generate the text modality for these datasets, we use the name or description associated with each particular label. For example, class 0 in cifar10 is "airplane", and this is the caption that we associate with all images of that class. The representation corresponding to this text modality is then used to define text-based neighbors. We have clarified this further in Appendix D.1. in our revised paper.

W2: Figure 3 can be improved. These lines overlap too much and are difficult to distinguish.

Thank you – the small size of the image was due to the lack of space. We have now updated the figures in the Appendix I.13 with an enlarged scale. We have also computed the area-under-the test-error vs %data retained curves for each of these figures as Reviewer Ux12 suggested in Appendix Table I.17., where we find that LEMoN has better AUC than all other baselines on cifar10 and cifar100 (i.e., the plots in Figure 3).