Noisy Multi-Label Learning through Co-Occurrence-Aware Diffusion

Thanks for your valuable suggestions. Below, we provide point-by-point responses to the weaknesses and questions you raised, hoping to address your concerns regarding our work.

W1&Q1: The proposed method is limited in terms of novelty. As described in the paper, it is an adopted version of CARD in noisy single-label learning to apply to the multi-label learning. Due to this similarity, it reduces the originality of the proposed method.

A1: Thanks for your comment. Compared with CARD, our work focuses on analyzing and improving practical performance for the specific problem. Based on the characteristics of the noisy multi-label learning (NML), we conduct targeted theoretical analysis and method design, including: (i) introducing feature-conditional strategies for noise-robustness and providing theoretical applicability guarantees, and (ii) developing diffusion-based embedding and coupling of label co-occurrence constraints.

First, the original feature conditioning mechanism in CARD relies on a supervised classifier $f_\phi$ to define the forward noisy distribution as $q(y_T|x) \sim {N}(f_\phi(x),I)$. However, this mechanism becomes invalid in the presence of noisy labels, as only noisy distributions are available during training, which can lead to a biased $f_\phi$ early on. In our work, we redesign the feature conditioning strategy to be robust to label noise with theoretical applicability guarantees, allowing it to be directly injected into the diffusion process. This improves the model's ability to capture the structured correlation between features and multiple labels.

Second, prior work (Chen J et al. Label-retrieval-augmented diffusion models for learning from noisy labels. NeurIPS, 2023, 36: 66499-66517) has not addressed label co-occurrence relationships in noisy multi-label settings. To remedy this, we incorporate label co-occurrence constraints into the reverse diffusion process, which helps the model identify and avoid generating implausible label combinations (e.g., Elephant and Boat), thereby improving both the plausibility and robustness of the generation path.

Finally, we conduct extensive experiments on both synthetic and real-world noisy datasets, demonstrating that our method significantly outperforms existing SOTA methods across multiple evaluation metrics. These results validate the effectiveness of our proposed improvements. In summary, while our approach is inspired by the foundational ideas in CARD, the modeling and optimization mechanisms we proposed are clearly novel and practically valuable for the NML setting, and not a simple application or replication of prior work.

W2: Another concern is that diffusion models, in general, are designed to estimate the distribution of data. In this case, the data is the noisy multi-labels. The nearest neighbor step in the forward pass already provides a rough estimation of that distribution. The backward pass is to provide a "finer grain" version of that distribution. However, it is considered as the clean label distribution that is "denoised" by the diffusion model.

A2: Thanks for your insightful observation. As you correctly pointed out, the label estimation in the forward process does provide a coarse-grained version of the label distribution. However, due to the limited representational capacity of the pre-trained encoder’s feature space, this initial estimation often falls short of ideal performance. As shown in Table 1 (to be added in the final version, the data originates from Tables 1 and 2 in main text and Appendix Figure F.1) below, even when using a powerful encoder like ViT, the estimated multi-label classification metrics, such as OF1 and CF1, are still far from satisfactory. For example, on the VOC 2007 dataset with 50% symmetric noise, the CF1 score barely reaches 55%. In contrast, after applying the proposed co-occurrence-aware diffusion process, our method outperforms advanced NML baselines like HLC (Xia X et al. Holistic label correction for noisy multi-label classification. ICCV, 2023: 1483-1493.), even when using a coarse-grained feature space and initial label distribution from a weaker encoder like ResNet50. This improvement arises not merely from converting coarse to fine label distributions, but more importantly from refining the feature space itself under the guidance of image features to achieve a better alignment with the target multi-label set. In other words, if we draw an analogy between the label diffusion model and a classifier, the diffusion process can be viewed as learning a mapping between extracted features and multi-label targets. However, unlike traditional discriminative models that directly minimize classification loss, our diffusion framework optimizes the L2 distance ($||\epsilon-\epsilon_\theta(y_t,{x},t)||^2$) between the generated denoised label distributions and the noisy ground-truth labels across all time steps. This indirect training objective encourages the model to learn a probabilistic generation path from features to labels, ultimately building a more robust generative mapping. We will include Table 1 and the accompanying explanation in the final version of the paper to further clarify the mechanism and contribution of the diffusion paradigm in this context.

Table 1: Comparison of label pre-estimation (𝑦̄) and full CAD model results (%) using different pre-trained encoders.

M1: Inconsistent English: the majority of the paper follows the American English, but some are in British English. For example, line 24: "expert-labelled" ==> "expert-labeled".

A3: Thank you for pointing this out. We appreciate your attention to detail. We will carefully revise the final version to ensure consistent use of American English throughout, including corrections like changing “expert-labelled” to “expert-labeled” and reviewing similar cases across the paper.

M2: Eqs (4) and (5) have unnecessary line break for a single column paper.

A4: Thanks for your careful review. We agree that the line breaks in Eqs. (4) and (5) are unnecessary, especially for a single-column format, and may affect readability. In the final version, we will revise these equations to be presented in a single line for better clarity and layout consistency.

Thanks for your valuable suggestions. Below, we provide point-by-point responses to the weaknesses and questions you raised, hoping to address your concerns regarding our work.

W1&Q1: Why is a diffusion model preferred over other generative frameworks on the NML problem?

A1: Thanks for pointing out the need to clarify our motivation for choosing diffusion models over other generative frameworks for the NML problem. In real-world multi-label annotation scenarios, inherent uncertainty is common due to complex image semantics, ambiguous category boundaries, and subjective differences among annotators. Consequently, incomplete or conflicting annotations often lead to label noise, which is difficult to be avoided. Therefore, it is more realistic to view the multi-label learning process as an uncertainty-aware generation task conditioned on image features. Instead of directly fitting noisy labels, we aim to learn a distributional mapping from the original features to the “most likely true label set,” thereby enhancing robustness to label noise. To this end, we adopt diffusion models as the label generation framework. On the one hand, prior work (Han X et al.. CARD: Classification and Regression Diffusion Models. NeurIPS, 2022) has compared the modeling ability of diffusion models against other generative approaches such as GANs (e.g., GCDS, Xiao Z et al. Tackling the Generative Learning Trilemma with Denoising Diffusion GAN. ICLR, 2022), showing that diffusion models are more effective at capturing predictive uncertainty in classification tasks. This makes them particularly suitable for modeling the noise and ambiguity in label generation. On the other hand, in text-to-image generation, diffusion models have demonstrated the ability to synthesize images that combine multiple target concepts given composite prompts. This suggests that diffusion models are inherently capable of capturing multiple objectives. This indicates that in NML tasks, diffusion models may be well-suited to handle input images with multiple target objects, as their inherent ability to capture multiple objectives allows them to generate appropriate combinations of labels, i.e., an ideal characteristic for multi-label classification. Taken together, these characteristics make diffusion models a natural and powerful choice for addressing the challenges in NML. We will incorporate a more detailed discussion of these motivation and comparative analysis in the final version of the paper.

W2&Q2: How does CAD perform when trained from scratch or with a weakly initialized encoder?

A2: We greatly appreciate your concern regarding the use of pre-trained models. Please allow us to address your question from two perspectives.

First, it is important to emphasize that our CAD model does not heavily depend on the pre-trained encoder, we can train a feature encoder from scratch using unsupervised approaches like SimCLR (Chen T et al. A simple framework for contrastive learning of visual representations. ICML, 2020: 1597-1607). Table 1 (to be added in the final version) shows the intermediate neighborhood label estimation and final classification results of CAD when combined with different pre-trained encoders include scratched ResNet50. The ResNet50 trained via unsupervised feature learning on the target dataset surpasses the ImageNet-pretrained counterpart in both label pre-estimation accuracy and performance when integrated with CAD, primarily because it effectively constitutes a form of fine-tuning on the target data distribution. On the other hand, from your perspective, ResNet50 can be viewed as a weakly initialized encoder compared to ViT-L/14, and the neighborhood label estimation alone based on ResNet50 performs worse than the baseline NML method across key metrics like OF1 and CF1. Even with such coarse estimations, CAD is capable of refining the generative feature-label mapping and consistently achieving performance improvements beyond the label estimation stage, outperforming the baseline NML method. This clearly highlights the indispensable role of the diffusion architecture in enhancing performance beyond what weak feature spaces can offer.

Table 1: Comparison of label pre-estimation (𝑦̄) and full CAD model results (%) using different pre-trained encoders.

Second, recent studies (Zhu Z et al. Detecting corrupted labels without training a model to predict. ICML. 2022: 27412-27427.; Ko J, Ahn S, Yun S Y. Efficient utilization of pre-trained model for learning with noisy labels. ICLR 2023 and Feng C, Tzimiropoulos G, Patras I. Clipcleaner: Cleaning noisy labels with clip. ACM MM, 2024: 876-885) increasingly explore how to leverage pre-trained models with noise-free knowledge to handle label noise without additional training. These works, aligned with our view, suggest that utilizing simple, accessible, and easily integrated pre-trained encoders is a practical and reasonable strategy. The weights used for both ResNet50 and ViT-L/14 are publicly available and pre-trained on ImageNet—commonly adopted in most classification settings. For instance, using ResNet simply requires setting $pretrained=True$. While it is possible to train a feature encoder from scratch using unsupervised approaches like SimCLR (Chen T et al. A simple framework for contrastive learning of visual representations. ICML, 2020: 1597-1607), we believe that, when high-quality, readily available pre-trained encoders exist, it is both reasonable and efficient to use them. To ensure fair comparison, we selected LRAD and LSNPC as our main baselines, both of which also use pre-trained encoders. Table 2 (The data originates from Appendix Table E.1) shows that under the same latent feature space conditions, CAD consistently makes better use of latent knowledge and refines it into more effective multi-label mappings. We will include Table 1 in the final version to illustrate the model’s non-reliance on specific pre-trained encoders and will add a more detailed discussion on this issue.

Table 2: Results (%) on VOC 2012 and MS-COCO with 30% symmetric noise, using various methods with different pre-trained encoders.

Thanks for your valuable suggestions. Below, we provide point-by-point responses to the weaknesses and questions you raised, hoping to address your concerns regarding our work.

W1&Q1: What happens if the label correlation is less associated with the latent space similarity? What happens when the domain knowledge is very different from pre-trained knowledge?

A1: Thanks for your insightful observation regarding the dependence of the proposed CAD model on the strength of association between label correlations and the pretrained latent feature space.

First, Table 1 (originates from Appendix Table E.1) shows that the contribution of different pre-trained models to the latent feature space used in CAD. Both models are pre-trained on ImageNet. From your perspective, it is reasonable to interpret that ResNet50 exhibits a weaker association between label correlations and its latent space compared to ViT. This is primarily due to the relatively limited representational capacity of ResNet50’s latent space. Consequently, the performance of CAD combined with ResNet50 is weaker than when combined with ViT or other pre-trained models with stronger representational power (e.g., ADDGCN, HLC). However, compared to other methods that also leverage pre-trained models to extract latent space knowledge (e.g., LSNPC, LRAD), our method consistently makes better use of this coarse-grained feature space. Through a diffusion paradigm, CAD refines these features into a generative mapping for multi-label classification, resulting in superior performance despite relying on the same latent knowledge. Furthermore, even when CAD is combined with a weaker feature space such as that of ResNet50, it still shows significant advantages over methods that do not utilize pre-trained models at all. Efficiently leveraging simple, offline, and easily accessible pre-trained models to obtain usable latent feature spaces is one of the strengths of our method and aligns with strategies recognized as effective in the current studies (Ko J et al. Efficient utilization of pre-trained model for learning with noisy labels. ICLR, 2023 and Zhu Z et al. Detecting corrupted labels without training a model to predict. ICML, 2022: 27412-27427).

Table 1: Results (%) on VOC 2012 and MS-COCO with 30% symmetric noise, using various methods with different pre-trained encoders.

Second, regarding the issue of out-of-distribution noise and potential misalignment between the pre-trained model and domain-specific knowledge, one possible solution is to apply unsupervised fine-tuning methods, such as SimCLR, on the pre-trained model before freezing it to obtain the latent feature space. Based on our empirical experiments, combining CAD with the ViT model allows us to effectively utilize a wide range of image datasets, including those involving fine-grained classification tasks. Prior research (Wei T et al. Vision-language models are strong noisy label detectors. NeurIPS, 2024, 37: 58154-58173) has also demonstrated the effectiveness of fine-tuning ViT encoders to address noise detection under different data distributions. We will include a discussion of this limitation, along with potential directions for future work, in the revised "Limitations" section.

W2&Q2: What are the exact theoretical guarantees?

A2: Our theoretical contribution primarily lies in the development of a label diffusion paradigm designed to simultaneously address multi-label classification and label noise. While CARD has introduced a label diffusion model for single-label classification, it does not provide a solution for either multi-label settings or noisy labels. For example, CARD’s feature conditioning mechanism relies on training an auxiliary classifier in advance on the dataset to guide the forward diffusion process with injected noise. However, in noisy label environments, this reliance often infeasible. On the other hand, LRAD addresses label noise via label estimation but does not consider label co-occurrence relationships inherent in multi-label tasks, limiting its effectiveness for noisy multi-label scenarios. In contrast, our work theoretically derives both unconditional and conditional multi-label diffusion processes and proposes a straightforward, noise-robust method for feature conditioning. This leads to a diffusion network architecture that directly aligns with the proposed theoretical framework. To reduce potential ambiguity, we will revise the final version of the paper by replacing “theoretical guarantees” with “theoretical applicability guarantees” in both the abstract and main text. This will clarify that our contribution lies in demonstrating the applicability of diffusion theory to noisy multi-label learning tasks.

W3: The paper does not discuss other possible ways of mitigating the noise issue using the latent neighborhood space, such as other automatic correction models.

A3: Unlike most methods that iteratively correct labels during model training, our CAD model performs a single-step pre-correction process through the latent feature space, without requiring any additional training or fine-tuning. To further validate this property, we introduced two additional label correction modules that also operate in a training-free and one-step manner: SimiFeat (Zhu Z et al. Detecting corrupted labels without training a model to predict. ICML, 2022: 27412-27427) and CLIPCleaner (Feng C et al. Clipcleaner: Cleaning noisy labels with clip. ACM MM, 2024: 876-885.). SimiFeat leverages a frozen feature extractor and label consistency among similar samples, using a Bayesian thresholding strategy to detect and correct noisy labels. CLIPCleaner aligns image and text embeddings using CLIP’s encoders to perform semantic-based label correction. A preliminary comparison of these two modules with our proposed estimation method (𝑦̄) is presented in Table 2 below. SimiFeat demonstrates superior correction performance, while CLIPCleaner appears to be less effective, possibly due to the limited accuracy of prompt-based semantics in multi-label contexts. When integrated with CAD, we observe that more accurate pre-correction leads to further performance improvements of the proposed diffusion paradigm. Our contribution lies in using the diffusion model to refine image features into a generative multi-label mapping based on pre-corrected labels. The integration with different correction modules also demonstrates the generality and extensibility of the proposed CAD. We will include a more comprehensive comparison of automatic correction models combined with CAD in the final version, and expand the discussion in the related work section accordingly.

Table 2: Performance (%) comparison of different automatic correction models and their integration with CAD on VOC 2012.

W4&Q3: The paper does not discuss the possibility of damaging the rare labels, which may occur simultaneously with other unexpected labels. What is the performance on rare labels?

A4: In noisy multi-label settings, rare labels may be mistakenly treated as label noise, which will negatively impact the model’s ability to fit them. In traditional weak supervision scenarios with extreme class imbalance, previous studies have proposed various semi-supervised and active learning approaches to address the challenge of rare label modeling. For example, graph-based label propagation methods (Pimplikar R et al. Learning to propagate rare labels. ACM CIKM, 2014: 201-210) leverage difference-of-convex optimization and k-NN subgraph sampling to improve robustness to extreme imbalance in large-scale graph-structured data. Alternatively, pseudo-labeling and representation-aware active learning techniques can be used to train robust models that mitigate the impact of rare labels (Mullapudi R T et al. Learning rare category classifiers on a tight labeling budget. ICCV, 2021: 8423-8432). Although our method is not specifically designed to address the rare label problem, we think the proposed CAD can partially alleviate this issue by expanding neighborhood search, or strengthening co-occurrence constraints via advanced matrix estimation techniques. The datasets used in our experiments are popular for evaluating NML methods, and they usually do not exhibit extreme imbalance. For instance, in the VOC dataset (excluding the most basic class person), the lowest label frequency (cow, 273 instances) and the highest (chair, 1168 instances) yield a ratio of roughly 1:4. Even if common noise is added to these extreme labels, it is still less likely for rare label situations to occur. In view of this, the proposed CAD does not include strategies specifically designed to address the rare label problem. Alternatively, We will briefly discuss this limitation in the final version of the paper. In future work, we plan to explore both simulated and real-world imbalanced multi-label datasets to better capture rare label scenarios and to develop dedicated solutions that can be integrated with the diffusion paradigm for enhanced performance.

Thank you for your insightful suggestions. Below, we provide point-by-point responses to the weaknesses and questions you raised, hoping to address your concerns regarding our work.

W1&Q1: It would be great if the paper can provide some analysis related to the computational costs of the developed approach and the baselines.

A1: We have provided a more detailed analysis of the training and inference time costs for our proposed CAD method and the baseline methods in Tables 1 and 2 below (The data originates from Appendix Figure H.1 and Table H.1). Table 1 shows that CAD achieves better training efficiency compared to mainstream approaches due to the lightweight diffusion architecture. Table 2 demonstrates that, with the accelerated DDIM sampling strategy, the time cost of generative multi-label inference is on the same order of magnitude as that of discriminative prediction, indicating that CAD is suitable for large-scale data scenarios. We will incorporate this part of the Appendix into the main text in the final version.

Table 1: Time cost (sec) of different method for training one epoch on one NVIDIA A800 GPU.

Table 2: Test time cost (sec) for CAD and discriminative (Disc.) models on different test sets.

W2&Q2: It would be great to provide more case study results to illustrate how the developed key components work in practice. Some intermediate results may also be presented.

A2: In our CAD architecture, the neighborhood label estimation in the forward process serves as an initial coarse label guess produced by a non-diffusion model. The subsequent diffusion training process refines this estimate by incorporating feature and co-occurrence constraints. As shown in Table 3 below (The data originates from Table 1 and Table 2 in main text and Appendix Figure F.1), which includes results from our ablation study, the diffusion training consistently improves multi-label classification performance based on the initial label estimates. This demonstrates that our training framework enables the diffusion model to effectively learn a robust generative mapping from features to multi-label outputs.

Table 3: Comparison of label pre-estimation (𝑦̄) and full CAD model results (%).

Q3: Some discussed previous works (e.g., [20]) provide theoretical results. Might it be feasible to provide some theoretical results similarly in this paper? Some discussions would be very insightful.

A3: Thanks for your constructive suggestion regarding the theoretical aspects of our work. Indeed, [20] provides valuable theoretical insights on using label correlations for estimating the noise transition matrix (LCT). In principle, it is feasible to incorporate such estimation techniques such as T-estimator (Liu T et al. Classification with noisy labels by importance reweighting. TPAMI, 2015, 38(3): 447-461) and Dual T-estimator (Yao Y et al. Dual t: Reducing estimation error for transition matrix in label-noise learning. NeurIPS, 2020, 33: 7260-7271) into our CAD architecture, since our co-occurrence matrix is also estimated by observing label co-occurrence probabilities within a clean subset. However, these methods typically require additional network components and warm-up training. For example, [20] estimates the transition matrix for each class $i$ by considering all other classes $k$ (where $k ≠ i$), which will increase computational overhead. That said, we agree that employing more accurate co-occurrence estimation methods could potentially enhance the performance of our CAD framework. We conducted a small-scale experiment by replacing our co-occurrence constraint with reweighting strategies based on T-estimator and the method proposed in [20]. As shown in Table 4, all these alternatives led to further performance improvements for our method. We will include these experimental results in the final version and incorporate theoretical discussions from the matrix estimation literature to improve the interpretability of our co-occurrence modeling.

Table 4: Comparison of the combined performance (%) of different matrix estimation reweighting methods with CAD under a 30% symmetric noise environment.