PHICO: Personalised Human-AI Cooperative Classification Using Augmented Noisy Labels and Model Prediction

• Presentation of the framework: I believe that the introduction needs to be improved. The framework in Figure 1 includes numerous elements that lack clarity, such as the green, blue, and orange matrices with entries that are occasionally underlined. Additionally, the individual components are insufficiently explained, while the final box (3) remains unexplained. The current design of this figure doesn’t fit well with the introduction, as its many components make it challenging to understand Phico's exact process and contributions.

• Related Work: The related work on LNL and MRL is brief and lacks depth. Expanding this section, e.g., by removing subsections and adding more details and content, could improve clarity. Additionally, some key references are incorrectly cited; for instance, the Crowdlab paper lacks its conference/journal details, and the VIT paper is cited on arXiv despite its acceptance at ICLR. These minor errors accumulate throughout the paper.

• Convergence Proof: One contribution of the paper, the proof of convergence, is presented very briefly in the main text. While placing a lengthy proof in the appendix is acceptable, highlighting this as a contribution requires a more thorough discussion of the proof and its implications within the paper itself.

• More intuitions in the Methodology: The methodology for the first three components (Annotator Profiles, Augmentations, and Personalized Model) is somewhat vague and would benefit from more intuitive explanations. Additionally, the many components make it challenging to follow the authors' arguments and underlying intuitions.

• Applicability in Inference: A crucial concern is the applicability and realism of the framework, especially for making predictions. Currently, the paper states:

  To classify a testing user into one of the K profiles, we first ask the user to label each image in a validation set…

  If I understand this correctly, each new user must annotate a validation dataset before profiling and predicting is possible. This approach seems extremely expensive and unrealistic.

• Minor problems:

  • Misuse of \cite and \citep. For example, in the introduction in the 3rd paragraph, the authors should use parenthesis around the citations.

1. The paper could benefit from clearer writing - both at a high level explaining the problem setting and method, and by providing details about their implementation. In particular how the annotator profiles are constructed.

2. The inclusion of a proof is nice, but in my understanding it primarily relies on established results for gradient descent and fuzzy k-means - so I don’t think there is much theoretical contribution.

3. The ‘alteration rate’ metric, while useful, seems fairly straightforward measure of how labels change given their method - I don’t believe this is particularly novel.

4. The paper does include many comparisons to many previous methods/baselines - but it seems like this process could have a lot of variance so seeing the results run over several rounds would be much more convincing than a single reported experiment.

Major Weaknesses:

• The matching between a test user and a profile (determined during training) seems to require a clean validation set (cf. definition of $\\mathcal{V}$ containing true labels $\\boldsymbol{y}\_i$ in Section 3.3). In practical applications, such a requirement can be highly limiting for two reasons. First, it is unclear how a validation set with clean labels is obtained. If an expert is available, giving such a task to this expert could lead to high annotation costs. Second, if each test user has to annotate this validation set, this also can lead to high annotation costs. For example, having 80 test users annotating a validation set of 200 instances (setup for CIFAR-10N) would correspond to 16,000 additional annotations.
• The related work covers the three domains of human-AI collaboration, noisy label learning, and multi-rater learning. Due to space limitations, I understand that discussing all different approaches in detail is infeasible. However, I think that particularly the multi-rater learning is not adequately represented. For example, CrowdLab, referred to as key development, seems to be a lesser-known approach (there is even no venue given in the references). Instead, there are many well-known [1, 2, 3] and recent [4, 5, 6, 7] approaches which are not mentioned.
• As a result of the sparse discussion on related work in multi-rater learning, the evaluation lacks an appropriate comparison to corresponding approaches. For example, more prominent multi-rater learning approaches could have been evaluated in analogy to the noisy label approaches in Table 4. Further, it is unclear why the multi-rater and noisy label learning approaches were only evaluated on CIFAR-10 with asymmetric label noise and not on the datasets CIFAR-10N, CIFAR10-H, FashionM-H, and Chaoyang. For example, the study [8] shows the evaluation of noisy label approaches on CIFAR10-N, while the study [4] presents results for multi-rater approaches on CIFAR-10N.
• One contribution is the proof of the convergence of PHICO's training. However, this contribution is not adequately presented in the main paper (there are only three lines referring to the proof in the Appendix). Moreover, after inspecting this appendix, the actual novelty of this contribution is questionable. According to my understanding, the first part of the proof uses the already proven fact that fuzzy $K$-means converges, while the second part generally applies to smooth loss functions with Lipschitz continuous gradients. The third part mainly states that if both previous steps converge, the overall training converges. Accordingly, I wonder whether the convergence proof for PHICO actually targets an open problem. As a result, I would only see this convergence proof as an important contribution, if it would provide unknown convergence rates or relaxed conditions, for example.

Minor Weaknesses:

• The computation of the transition matrix corresponds to a class-dependent assumption. Although such an assumption is made by several related approaches in noisy label [9] and multi-rater learning [1, 2], it does not reflect the actual instance-dependent annotation patterns of humans. Moreover, other works [6, 7] already propose solutions to estimate this instance-dependency. Accordingly, I would expect a brief reasoning why a class-dependent modeling has been selected.
• Since it is typical that a new approach cannot resolve all issues regarding a learning task, I would expect a detailed discussion of current limitations. Yet, Section 6 discusses very briefly potential extensions of PHICO (e.g., modeling of evolving annotation patterns). Accordingly, many aspects, e.g., the requirement for a validation set, modeling of only class-dependent annotation patterns, and requiring training users to provide a certain number of labels per class to compute label vectors for clustering, could have been discussed more transparently.
• It seems that no code has been provided for review, and there is no indication of whether the code will be made publicly available.

References:

• [1] Rodrigues, Filipe, and Francisco Pereira. "Deep learning from crowds." Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.
• [2] Tanno, Ryutaro, et al. "Learning from noisy labels by regularized estimation of annotator confusion." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.
• [3] Cao, Peng, et al. "Max-MIG: an Information Theoretic Approach for Joint Learning from Crowds." International Conference on Learning Representations. 2019.
• [4] Zhang, Hansong, et al. "Coupled confusion correction: Learning from crowds with sparse annotations." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 15. 2024.
• [5] Ibrahim, Shahana, Tri Nguyen, and Xiao Fu. "Deep Learning From Crowdsourced Labels: Coupled Cross-Entropy Minimization, Identifiability, and Regularization." The Eleventh International Conference on Learning Representations. 2023.
• [6] Guo, Hui, Boyu Wang, and Grace Yi. "Label correction of crowdsourced noisy annotations with an instance-dependent noise transition model." Advances in Neural Information Processing Systems 36 (2023): 347-386.
• [7] Herde, Marek, Denis Huseljic, and Bernhard Sick. "Multi-annotator Deep Learning: A Probabilistic Framework for Classification." Transactions on Machine Learning Research. 2023.
• [8] Wei, Jiaheng, et al. "Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations." International Conference on Learning Representations. 2022.
• [9] Zhang, Yivan, Gang Niu, and Masashi Sugiyama. "Learning noise transition matrix from only noisy labels via total variation regularization." International Conference on Machine Learning. PMLR, 2021.

1. Figures need improvement: Figure 1 does not do justice to explaining the approach. The caption is incomplete, and the figure itself is not very high quality. I would advise improving the caption to explain the reader what is happening here. I could understand it only after re-reading the paper a couple times.

2. Statistical testing for PHICO: Largely, the performance gains of this approach over the baselines is marginal. Within a couple percentage points in most places. Thus, reporting error bars and statistical testing would go a long way to show that it actually improves performance over the baselines. Currently, the claim is not based in solid statistical reasoning.

3. Essentially, Annotator Profiles just mean dividing Annotators into types. It would be interesting to see what kind of types are discovered. Are there demographic patterns? Are there patterns in terms of their habits? How different are these truly?