Pi-DUAL: Using privileged information to distinguish clean from noisy labels

We thank the reviewer for taking the time to review our paper and for all the valuable feedback. We address their comments below:

Innovations to PI

We respectfully disagree with the reviewer in that the innovations of Pi-DUAL are minor. As appreciated by the other reviewers, the goal of our paper is to build a simple and effective method that requires the minimal changes to existing training pipelines. In this regard, while Pi-DUAL is not the first in utilizing PI features to combat label noise, Pi-DUAL provides a solution that is both simpler and more efficient. Compared to TRAM, it has the following key distinctions:

1. Network structure: TRAM uses one model but with two heads, one for training and one for testing. In contrast, Pi-DUAL uses three distinct subnetworks with clear roles during training and inference: the prediction network for fitting clean target distribution, the noise network for explaining away label noise, and the gate network for identifying noisy samples. These three roles can not be accounted for in the architecture of TRAM.
2. Learning objective: While the learning objective of TRAM is a weighted sum of two objectives (one for fitting the noisy target distribution, and another for transferring the knowledge to the no-PI head), the learning objective of Pi-DUAL is a simple cross-entropy loss over the noisy targets. For a better illustration of the differences between Pi-DUAL and TRAM, we conducted an ablation which trains Pi-DUAL with the loss function of TRAM, and show that this produces much worse results. This experiment has been added to Section C.8 in Appendix.

3. Modeling of label noise. TRAM does not explicitly model label noise. Instead, it learns the conditional distribution of noisy labels with respect to both x and a, to eventually predict with an implicit marginalization over a. On the other hand, Pi-DUAL explicitly models per-instance-level label noise in its architecture, which permits it to identify noisy labels and to effectively reduce the negative impacts from noisy labels. In contrast to TRAM, which performs approximate marginalization over the PI, the prediction network of Pi-DUAL learns the conditional distribution of clean labels with respect to only x. With this explicit modeling of label noise, Pi-DUAL delivers not only better performance, but also a much better interpretability as to how PI helps explain away label noise (e.g., how the noise network overfits to the wrong labels in Figure 2, and how the gate network identifies wrong labels in Figure 3).

With these non-trivial improvements, Pi-DUAL outperforms TRAM++ on all 5 benchmarks achieving up to 8.1 accuracy points more than TRAM++ on ImageNet-PI. Pi-DUAL also delivers much better performance in identifying label noise post-training due to the explicit modeling of label noise.

Architectural ablation studies

We thank the reviewer for bringing the missing ablation studies on the model architectures to our attention, and we hope the following additional ablations help complete our understanding of the method.

Ablations for the model backbones

In the paper we use a WideResNet-10-28 for all experiments on CIFAR. Here we provide the results when replacing the architecture with a ResNet-34 instead. ResNet-34 is a common choice for model backbones in CIFAR-level experiments in the noise label literature (Cheng et al., 2020, Liu et al., 2020, 2022). We have added this ablation in Appendix C. 6.1. The results using ResNet-34 comparing Pi-DUAL and CE baseline are reported in the following table:

From the table, we observe that Pi-DUAL maintains its performance improvement over CE with a different model backbone, exceeding the performance of CE baseline by a notable margin in all three datasets.

Ablations for PI network structure

Following the reviewer’s advice, we have added ablations on the width and depth of PI-related modules in Appendix C. 6.2.

1. Ablations for the width for PI-related modules: In the paper we set the width of the PI-related modules (for both noise network and gate network) by default to 1024 for CIFAR-10H, and 2048 for all other experiments, without tuning. Here we provide the results for different widths for the PI networks on three CIFAR datasets. | Dataset \ Width | 512 | 1024 | 2048 | 4096 | |:---------------:|:--------:|:--------:|:--------:|:--------:| | CIFAR-10H | 72.8±2.9 | 71.3±3.3 | 71.4±3.6 | 71.2±3.8 | | CIFAR-10N | 83.8±0.4 | 83.6±0.7 | 84.9±0.4 | 85.3±0.2 | | CIFAR-100N | 62.3±0.2 | 63.7±0.4 | 64.2±0.3 | 64.4±0.2 |

From the table we observe that the performance of Pi-DUAL benefits from larger network width for the PI-related modules

2. Ablations for the depth for PI-related modules: In the paper we set the depth of the PI-related modules to 3 by default for all experiments, without tuning. Here we provide the results for different depths for the PI networks on three CIFAR datasets. Note that the width of the PI-related modules are set to the default value when tuning the depth. | Dataset \ Depth | 2 | 3 | 4 | |:---------------:|:--------:|:--------:|:--------:| | CIFAR-10H | 68.9±2.0 | 71.3±3.3 | 70.1±3.5 | | CIFAR-10N | 83.4±0.4 | 84.9±0.4 | 85.2±0.3 | | CIFAR-100N | 59.4±1.0 | 64.2±0.3 | 64.1±0.2 |

From the table we observe that the depth of the PI-related modules have to be at least of 3 layers to maximize the performance of Pi-DUAL. We thank the reviewers for leading us to these interesting findings.

Reasons why PI-methods underperform No-PI methods

We appreciate the comment of the reviewer, but we note that the overall goal of our work is to provide a better approach to utilize PI features, and not a deep analysis of prior work. In the paper, we have provided ample empirical and theoretical evidence illustrating the behavior of Pi-DUAL and we have tried to be as thorough as possible in our ablations. Pi-DUAL closes the gap between PI and no-PI methods even in datasets with low-quality PI features, while exhibiting much better scalability to no-PI methods.

However, to answer the reviewer, we would like to emphasize that one of our main findings is that the relative performance between PI and No-PI methods depends heavily on the quality of PI features. For example, CIFAR-10H has fine-grained PI features, and the performance of TRAM++ outperforms the No-PI methods by 15.5 points. On the other hand, the PI feature quality for CIFAR-10N and CIFAR-100N are much worse as they are averaged over a batch of samples, which significantly reduces the amount of information on each sample carried by PI.

To quantitatively illustrate the importance of PI quality in the performance of PI methods, we provide a new experiment where we randomly corrupt a percentage of PI features in the train set and train Pi-DUAL with the corrupted PI features (details in Appendix C.9). We copy the results here for reference:

As we can see, as the quality of the PI gets worse, the performance of Pi-DUAL degrades significantly, highlighting the importance that good PI has on these methods.

Comparison to state-of-the-art

For a fair comparison, we only included in Table 2 the methods which use only one model and that are trained for a single stage. We did not compare to methods like Divide-Mix, because it incorporates techniques from semi-supervised learning, which substantially increases the computational costs (e.g., it uses two networks, mixup augmentation, longer training epochs, etc.). It is worth noting that prior works in label noise literature also do not compare directly to DivideMix, but compare DivideMix to versions of their methods which use semi-supervised learning techniques. For example, the authors for ELR (Liu et al., 2020) compare DivideMix with ELR+, while the authors for SOP (Liu et al., 2022) compare DivideMix with SOP+ (rather than SOP). ELR+ and SOP+ are respectively more sophisticated versions of the original methods augmented with semi-supervised learning techniques, making the comparison more fair in terms of both costs and complexity.

Therefore, in response to the reviewers request for a comparison to Divide-Mix and other semi-supervised methods, and for a full validation of our proposed method and a fair comparison, we follow the common practice, and propose an augmented version of Pi-DUAL, Pi-DUAL+, and compare it to other semi-supervised learning methods. Pi-DUAL+ adds to Pi-DUAL two regularization techniques, label smoothing and prediction consistency regularization from UDA (Xie et al., 2020), where the latter one was a common choice for augmenting label noise training methods (Cheng et al., 2020, Liu et al., 2022). We have added the details for implementation of Pi-DUAL+ in Appendix C.5, and copy the results for Pi-DUAL+ below for reference, comparing it with several prior semi-supervised pipelines for combating label noise:

In this table, the baseline results are obtained using the code of the original papers if they are not available in the public leaderboard. It is worthwhile noting that, on CIFAR-10H, Pi-DUAL+ outperforms the rest of the methods by a margin of over 11%, while it performs on par with the state-of-the-art methods on CIFAR-10N and CIFAR-100N. Importantly, this is because Pi-DUAL+ benefits from the high quality PI features in CIFAR-10H, while for CIFAR-10N and CIFAR-100N, , the PI is of poor quality, which explains why Pi-DUAL+ does not have a performance boost that is as large as the one in the case for CIFAR-10H. We hope it can help in delivering a thorough comparison with the state-of-the-art.

Comparison to semi-supervised methods

We also agree with the reviewer that a comparison of Pi-DUAL with semi-supervised learning methods would be unfair and that is why we did not consider it in our initial manuscript. Prior works in the noisy-label literature also do not directly compare to DivideMix, but compare DivideMix to versions of their methods which use semi-supervised learning techniques. For example, the authors for ELR compare DivideMix with ELR+, while the authors for SOP compare DivideMix with SOP+ (rather than SOP). ELR+ and SOP+ are respectively more sophisticated versions of the original methods augmented with semi-supervised learning techniques, making the comparison fairer in terms of both costs and complexity.

However, in response to the request of Reviewer h1hP to add a comparison to Divide-Mix and other semi-supervised methods, we have designed an extension of Pi-DUAL, referred to as Pi-DUAL+. It adds to Pi-DUAL two regularization techniques, namely label smoothing and the prediction-consistency regularization from Xie et al., 2020. The latter is a common choice to add semi-supervision to methods of the noisy-label literature (Liu et al., 2020, Cheng et al., 2020). The results for Pi-DUAL+ are shown below and are compared with several prior semi-supervised approaches :

In this table, the baseline results are obtained using the code of the original papers if they are not available in the public leaderboard. We can see that on CIFAR-10H, Pi-DUAL+ outperforms the rest of the methods by a margin of over 11%, while it performs on par with the state-of-the-art methods on CIFAR-10N and CIFAR-100N. This is because Pi-DUAL+ benefits from high-quality PI on CIFAR-10H, while for CIFAR-10N and CIFAR-100N, the available PI is known to be of poor quality (Ortiz-Jimenez et al., 2023). We have added this table, as well as a detailed description of Pi-DUAL+ in our paper.

Comparison with results in literature

We absolutely agree with the reviewer that all methods should be compared under their optimal configurations and this has precisely been our intention in all our experiments. All our experiments, including the reimplementation of other methods, are built on the open-source uncertainty baselines codebase (Nado et al., 2021) and follow as much as possible the benchmarking practices standardized by Ortiz-Jimenez et al. (2023).

As detailed in Appendix B, we follow the same hyper-parameter tuning pipeline for all methods, namely we perform a grid search over their main hyper-parameters using a noisy validation set randomly split from the train set. In this regard, we want to highlight that, since Pi-DUAL only introduces one additional hyper-parameter compared to the cross-entropy baseline, it has the lowest tuning budget compared to the other methods that all have at least two extra hyper-parameters.

Moreover, we want to clarify that there are no discrepancies between the results we report and those from the literature, as the reviewer points out. Specifically:

• On CIFAR10-H, our results for TRAM agree with those released by the original TRAM authors in their follow-up paper (Table 1 in Ortiz-Jimenez et al. 2023). The performance gap between Collier et al. (2022) and Ortiz-Jimenez et al. (2023) is due to the fact that they use two different versions of CIFAR10-H. Collier et al. (2022) use the “uniform” version while Ortiz-Jimenez et al. (2023) use the harder “worst” version; see discussion in Sec. 2.1 of Ortiz-Jimenez et al. (2023). All our experiments are trained using the latter version of the dataset.
• In all our tables, to ensure a fair comparison, we report results of the standard ELR and SOP methods which operate without any additional semi-supervised learning techniques. We note that the leaderboard reports separate results for ELR (our baseline) and ELR+ (including semi-supervised learning). For SOP however, we have found that the leaderboard mistakenly reports the results for SOP+ (including semi-supervised learning) instead of SOP, as one can see by comparing to the original SOP paper (cf. Table 4 of Liu et al. (2022)). Taking this into account, we highlight that our results thus perfectly agree with those reported in the literature, and even exceed them due to better hyper-parameter tuning (e.g. our ELR baseline is 5 points better on CIFAR100N than the one in the leaderboard).

We hope these clarifications will make the reviewer increase their confidence in our results. We fully understand the initial confusion of the reviewer due to the benchmarking disparities in the literature. Our goal was precisely to ensure a fair comparison of the different methods under the same settings to be able to draw solid conclusions.

We thank the reviewer for taking the time to review our paper and for all the valuable feedback. We address their comments below:

Differences between Pi-DUAL and TRAM

We respectfully disagree with the reviewer in that the innovations of Pi-DUAL with respect to TRAM are minor and correspond only to a change in architecture. As appreciated by this reviewer, the goal of our paper is to build a simple and effective method that requires minimal changes to existing training pipelines. In this regard, while both TRAM and Pi-DUAL utilize PI features to combat label noise, Pi-DUAL provides a solution that is both simpler and more efficient. Compared to TRAM, it has the following key distinctions:

1. Network structure: TRAM uses one model but with two heads, one for training and one for testing. In contrast, Pi-DUAL uses three distinct subnetworks with clear roles during training and inference: the prediction network for fitting clean target distribution, the noise network for explaining away label noise, and the gate network for identifying noisy samples. These three roles can not be accounted for in the architecture of TRAM.
2. Learning objective: While the learning objective of TRAM is a weighted sum of two objectives (one for fitting the noisy target distribution, and another for transferring the knowledge to the no-PI head), the learning objective of Pi-DUAL is a simple cross-entropy loss over the noisy targets. For a better illustration of the differences between Pi-DUAL and TRAM, we conducted an ablation which trains Pi-DUAL with the loss function of TRAM, and show that this produces much worse results:

3. Modeling of label noise. TRAM does not explicitly model label noise. Instead, it learns the conditional distribution of noisy labels with respect to both x and a, to eventually predict with an implicit marginalization over a. On the other hand, Pi-DUAL explicitly models per-instance-level label noise in its architecture, which permits it to identify noisy labels and to effectively reduce the negative impact from noisy labels. In contrast to TRAM, the prediction network of Pi-DUAL learns the conditional distribution of clean labels with respect to x alone. With this explicit modeling of label noise, Pi-DUAL delivers not only better performance, but also a much better interpretability as to how PI helps explain away label noise (e.g., how the noise network overfits to the wrong labels in Figure 2, and how the gating network identifies wrong labels in Figure 3).

With these non-trivial improvements, Pi-DUAL outperforms TRAM++ on all 5 benchmarks achieving up to 8.1 accuracy points more than TRAM++ on ImageNet-PI. Pi-DUAL also delivers much better performance in identifying label noise post-training thanks to the explicit modeling of label noise.

We thank the reviewer for taking the time to review our paper and for all the valuable feedback. We address their comments below:

Computational cost of Pi-DUAL

In our experiments, we found that Pi-DUAL does not have extra computational requirements compared for hardware to the standard training baseline. The added cost of processing the PI features (which typically have much fewer dimensions) is indeed negligible with respect to processing the whole images. In this sense, training and inference using Pi-DUAL take roughly the same time as training with standard cross-entropy, TRAM++ or HET. For our large-scale datasets (i.e., ImageNet-PI), we used a TPU V3 with 8 cores for all our experiments.

We use the parameter count as a proxy for the memory cost. Since Pi-DUAL models the per-instance level of noise using the PI features, it does not require introducing parameters that scale with the number of samples or classes as SOP and ELR need to. In fact, the extra parameters needed in our experiments to account for the noise and gating networks are comparable to those of TRAM++ and even fewer than the ones introduced by the no-PI method HET.

We have added these comments to the Appendix D of our new version.

Privacy concerns of using PI

We thank the reviewer for bringing this topic to our attention and we will make sure to discuss it in our camera-ready work. However, we note that none of our results require the use of personally identifiable annotator IDs. In fact, cryptographically safe IDs in the form of hashes work perfectly fine as PI. In this regard, we do not think that there are serious concerns about possible identity leakages stemming from our work if the proper anonymization protocols are followed. In addition, we have added an ethics statement to the paper regarding the privacy and safety protection requirements of Pi-DUAL.

Performance without PI

As mentioned throughout the paper, we would like to emphasize that Pi-DUAL has been explicitly designed to leverage PI features. In this regard, its good performance hinges on the availability of good PI features such as in the case of ImageNet-PI or CIFAR-10H (Ortiz-Jimenez et al. 2023).

However, as mentioned by the reviewer, it is still interesting to study the performance of Pi-DUAL when no PI is available, as we show in Table 4 in the paper (last row). In these cases, we can always construct some randomly-generated PI to train PI-DUAL, following the procedure of Ortiz-Jimenez et al. (2023). As shown in Table 4 (and copied below for reference), Pi-DUAL with the randomly-generated PI still outperforms the cross-entropy baseline on the CIFAR datasets, but its performance is much worse than when using the real PI of the datasets.

Hyperparameter tuning strategy

All our experiments, including the reimplementation of other methods, are built on the open-source uncertainty baselines codebase (Nado et al., 2021) and follow as much as possible the benchmarking practices standardized by Ortiz-Jimenez et al. (2023).

As detailed in Appendix B, we follow the same hyper-parameter tuning pipeline for all methods in which we perform a grid search over their main hyper-parameters using a noisy validation set randomly split from the train set. We do this to make sure that all methods are compared under their optimal and fair settings. In this regard, we want to highlight that, because Pi-DUAL only introduces one additional hyper-parameter being tuned compared to the cross-entropy baseline (i.e., the length of the random PI), it has the lowest tuning budget in comparison to every other method. All other methods introduce at least two additional hyper-parameters to be tuned and therefore we dedicate at least twice as much compute to tune them than the compute used to tune Pi-DUAL. Despite these additional hyper-parameter tuning resources, the performance of Pi-DUAL compares favorably to these methods.

Overall, we see the simplicity of tuning Pi-DUAL compared to other noisy label methods as another strength of the method.