Meta-learning Representations for Learning from Multiple Annotators

Thank you for your positive comments and constructive feedback.

adding a graphical model could further enhance clarity for readers.

Thank you for good suggestion. We have added the graphical model of our method in Section A of the revised paper.

The paper assumes isotropic variance in the latent space, simplifying computation but potentially limiting flexibility. Real-world data often exhibit complex, class-specific structures that may not align with uniform variance assumptions, particularly in nuanced classification tasks.

1.How does the choice of isotropic variance in the latent space affect the model's flexibility and accuracy? Would other types of covariance provide a meaningful improvement in capturing complex data structures?

Thank you for the insightful comment. As you mentioned, we used the isotropic variance in the latent space to simplify the modeling and its computation. However, since the latent space can be flexibly learned by a neural network, such simple modeling works well. Indeed, our method outperformed various comparison methods by a large margin, as shown in Tables 1 and 2. We believe that simple models with good performance are more preferable than complex models. Note that isotropic variance in the latent space is also assumed in prototypical networks, a well-known and high-performance embedding-based meta-learning method (Snell et al., 2017).

Nevertheless, we can consider using a full covariance matrix for each class $\Sigma_k \in \mathbb{R}^{M \times M }$ on the latent space. Since the dimension of the latent space $M$ is arbitrarily determined by users, estimating the $M \times M$ matrix $\Sigma_k$ would not be a problem regarding computation costs when $M$ is not large. We have added the following sentence in Section 3.2 of the reviased paper.

"Although we assume the isotropic variance ${\bf I}$ for simplicity, we can use other covariance matrices such as full covariance matrices."

Modeling A as a KKr matrix may lead to over-parameterization, especially with limited data. Without visualization of learned matrices, it's challenging to assess whether A captures meaningful annotator patterns. 2.Can you show some of the learned Ar and demonstrate how effectively they capture the true annotation bias?

We have already visualized the estimated annotator-specific confusion matrices in Figure 8. We found that the estimated confusion matrices can capture the characteristics of each annotator well. For example, in the estimated matrices of expert (a) and normal annotator with high labeling accuracy (d), the diagonal elements are large, meaning they tend to return correct labels. In contrast, the confusion matrices of annotators with low accuracy (b, c, e) have values in the off-diagonal ones, reflecting that these annotators assign noisy labels.

Additionally, assuming A is feature-independent implies annotator errors are detached from data features, suggesting p(y|x) = p(y) . This could prevent the model from learning a classifier grounded in meaningful data representation.

We think that even when $A$ is input-independent (i.e., $p(y|t,x)=p(y|t)$), $p(y|x)$ is not equal to $p(y)$, where $x$ is data features, $y$ is its noisy label, and $t$ is its latent true label.

We roughly explain this below. First, we have $p(y|x)= \sum_t p(y,t|x) = \sum_t p(y|t,x) p(t|x) = \sum_t p(y|t) p(t|x)$, where we used the assumption $p(y|t,x)=p(y|t)$ in the last equality. Next, we have $p(y)=\sum_t p(y,t) = \sum_t p(y|t) p(t)$. Thus, $p(y|x)-p(y) = \sum_t p(y|t) (p(t|x)-p(t))$. Since $p(t|x)$ is not equal to $p(t)$, we cannot conclude that $p(y|x)=p(y)$.

Maximizing likelihood of noisy label data rather than true labels raises concerns. Likelihood maximization is traditionally aimed at learning distributions close to true labels; without this, it is unclear if maximizing the joint likelihood on noisy data allows meaningful inference of latent true labels. The model might be vulnerable to learning patterns in the noise rather than capturing true class structure.

Since we use the latent variable model in which the generation process of noisy labels from latent true labels is modeled (Eq. 1), maximizing the likelihood of noisy labeled data (more accurately, MAP estimation as described in Eq. 2) can lead to estimating true labels when modeling assumption (e.g., input-independent confusion matrices) is vaild. Indeed, this approach is commonly used for crowdsourcing studies and has shown excellent performance (Dawid & Skene, 1979; Chu et al., 2021; Rodrigues & Pereira, 2018; Raykar et al., 2010; Kim et al., 2022) as described in Section 3.

Moreover, since our model is meta-learned such that it can improve the expected test performance after adaptation to noisy labeled data (i.e., inference of the latent variable model), our meta-learned model can infer more robustly against label noises by transferring knowledge on clean labeled data in source tasks.

Thank you for your insightful comments and constructive feedback.

1.The description of this paper is not clear enough. For example, the description of the method proposed in this paper in the last two paragraphs of INTRODUCTION is lengthy and lacks logic.

Thank you for the constructive suggestion. We have made the structure/description of Section 1 more evident by splitting the second-to-last long paragraph of Section 1 into two paragraphs. Specifically, in Section 1 of the revised paper, the third-to-last paragraph mainly explains the general mechanism of meta-learning (i.e., it consists of the inner and outer problems). The second-to-last paragraph focuses on the inner problem of our method. The final paragraph explains the outer problem of our method. By clearly defining each paragraph's subject, we believe that clarity/logic has been improved. If you have any other suggestions, we would appreciate your feedback.

The RELATED WORK, DATA, COMPARISON METHODS sections are full of nonsense and lack structure.

Thank you for your feedback. We would appreciate it if you could provide more specific details regarding the issues of "full of nonsense" and "lack of structure." We want to incorporate the reviewers' comments as much as possible to improve the paper.

In our submitted paper, we made each section have a clear structure. For example, in the related work section, we created three different paragraphs to explain three main related topics, i.e., methods for ordinary crowdsourcing (learning from mulitple noisy annotators), meta-learning methods for crowdsourcing, and general meta-learning methods, and discussed the position of our work in each paragraph. In the data section, we described the overview of datasets, how we created multiple tasks from these datasets, and the process of adding noisy annotations to each task by separating paragraphs. Similarly, in the comparing methods section, we first provided an overview of all comparison methods, followed by two separate paragraphs explaining comparison methods for ordinary crowdsourcing and for meta-learning of crowdsourcing.

As indicated by the generally positive feedback from the other three reviewers on presentation and clarity (e.g., Reviewrs g7k1 and jTSk said that 'Presentation: 3: good' and Reviewr jTSk said that 'Clarity: Overall the paper is well-organized'), we believe the current structure is fine. However, if there are specific concerns, we would appreciate your feedback.

2.The experimental comparison method lacks the latest methods in the last two years.

We have compared many recent state-of-the-art crowdsourcing or noisy label learning methods in our experiments. Specifically, we evaluated the crowd layer (CL) (AAAI2018), the common noise adaptation layers (CNAL) (AAAI2021), few-shot learning from noisy labels (LNL) (CVPR2022), and Label Fusion (LF) (ECCV2022). In addition, since they are not designed for few-shot learning except for LNL, we also created their meta-learning variants (MCL, MCNAL, and MLF) based on the study (Zhang et al., 2023) and evaluated them. We compared 16 comparison methods in total. Our method outperformed these methods.

Although there are more recent crowdsourcing methods such as (Guo et al., 2023; Li et al., 2024) as you mentioned, they all require large amounts of data. Thus, these studies differ from our work, which focuses on a small data regime. Our work proposes and explores a new problem different from traditional crowdsourcing studies, so we believe the current comparison methods are sufficient.

3.The datasets used in the experiment are all in the field of images and lack representativeness.

We agree that adding data other than image data in experiments can enrich our paper. However, in current machine learning research, image data are the standard benchmark, and many existing studies on crowdsourcing, noisy label learning, and meta-learning, including studies from top ML conferences such as NeurIPS, ICLR, ICML, AISTATS, and AAAI (Snell et al., 2017; Rajeswaran et al., 2019; Li et al., 2020; Bertinetto et al., 2018; Guo et al., 2023; Gao et al., 2022; Kim et al., 2022; Zhang et al, 2021), use only image data. Thus, we followed this. In our experiments, we evaluated our method with various conditions/perspectives, such as different numbers of support data, different numbers of annotators, different noise ratios, and different types of annotators. Thus, our experimental results are sufficiently reliable. In the future, we want to evaluate our method with data other than the image domain.

[Zhang et al, 2021] Zhang, Yivan, Gang Niu, and Masashi Sugiyama. "Learning noise transition matrix from only noisy labels via total variation regularization." ICML 2021.

Thank you for your reply.

As you pointed out, our method focuses on the small data regime. In the context of learning from multiple annotators, the small data regime has not received much attention; however, as mentioned in Section 1, we believe there are many potential applications. For example, in domains such as security, healthcare, and science, where data is scarce and annotators must have deep expertise, the issue of small data with a few annotators is crucial (by the way, as mentioned in Section 1, even experts exhibit differences in annotations (Raykar et al., 2010; Mimori et al., 2021).) Thus, we think our work is useful in addressing this challenge.

When many annotated data are available, as you know, there are countless existing studies. Thus, such cases can be handled by the conventional methods. As the no-free-lunch theorem suggests, there is no universal algorithm for all cases. we think that it is important for the machine learning community to work together to increase the available tools that can be appropriately used in the right contexts.

Thank you for your insightful comments and constructive feedback.

Assumption: This paper assumes noise comes from multiple annotators with different error patterns and biases and each annotator’s behavior is consistent within a task, but for real-life data scenarios, it is not accurate, each annotator is domain-specific.

Thank you for the insightful comment. The assumption that each annotator's behavior is consistent within a task is commonly used for modeling in crowdsourcing and noisy label learning studies (Chu et al., 2021; Rodrigues & Pereira, 2018; Raykar et al., 2010; Tanno et al., 2019; Kim et al., 2022). While real-world annotators may exhibit more complex behavior, the effectiveness of our method has been validated on datasets with real-world annotations (LabelMe and CIFAR10H) in Tables 2 and 9. Since this paper proposes a new problem setting (i.e., meta-learning for learning from a few noisy data given multiple annotators), it is reasonable to consider simple models as the first step. To better model real-world annotators and enhance our method's performance further, we can consider assuming input-dependent annotators' confusion matrices, one of the future works described in Section 5.

Novelty: The paper combines well-established methods like GMMs and annotator-specific confusion matrices within a meta-learning framework to handle noisy labels, the novelty is somewhat limited, such as GMM-based clustering has already been widely used for probabilistic class assignments and label noise mitigation.

As you mentioned, GMM-based clustering and confusion matrices have been used in various tasks. However, combining them within meta-learning is novel, which all the other reviewers acknowledge. In particular, the important point is that our method is not just a simple combination; rather, it is cleverly designed to mitigate the high computational cost of meta-learning, a major problem of meta-learning methods described in Section 2. Specifically, by appropriately modeling the inner problem with GMM and confusion matrices, our method can efficiently solve the inner problem with the closed-form EM steps, which leads efficiently to solve the outer problem (meta-learning). The efficiency of our meta-learning are demonstrated in the last paragraph fo Section 4.3 and Section I.11.

Increasing $N_S$ and $R$ significantly improves the model performance. The authors could add experiments with larger $N_S$ and $R$ values to assess at what point performance stabilizes or converges.

Thank you for the good suggestion. We have additionally investigated the performance of our method with large numbers of support data $N_S$ and annotators $R$. In this experiment, we used Miniimagenet since it has many data per class (100). We set $N_S=20$ and $R=7$ during the meta-learning phase. First, we evaluated our method by changing $N_S$ values with $R=7$ in target tasks. The average test accuracies are as follows.

Our method's performance increased as $N_S$ increased, and it seemed to converge when $N_S=120$ or $N_S=160$. Next, we evaluated our method by changing $R$ values with $N_S=40$ in target tasks. The average test accuracies are as follows.

Even if the number of support data is not so large ($N_S=40$), our method performed well with the large number of annotators $R$. This is because many annotators can improve label estimation performance. We have added these results in Section I.10 of the revised paper.

How does the model perform if certain classes have significantly fewer support data points than others? How robust is it to imbalances in the number of labeled examples per class?

Our method can be robust against data imbalance in target tasks, as explained below.

In our model, mixture ratio $\pi_k$ (to be estimated) represents the importance (weight) of class $k$ during adaptation for the inner problem. In Eq. (7) of the EM steps, by setting the Dirichlet parameter's (hyperparameter) $b>0$ sufficiently large values, we can get

$\pi _k = \frac{ \sum _{n=1}^{N _{\cal S}} \lambda _{nk} + b }{Kb + N _{\cal S} } = \frac{ \sum _{n=1}^{N _{\cal S}} \frac{\lambda _{nk}}{b} + 1 }{K + \frac{N _{\cal S}}{b} } \approx \frac{1}{K}$,

where $K$ is the number of classes. This ensures that each class equally influences the adaptation, preventing the minority class from being ignored during adaptation (In contrast, standard classification losses such as the cross-entropy loss can ignore the minority class's data since they minimize the average loss of all data.)

We have added these discussion in Section I.5 of the revised paper.

Thank you for your insightful and detailed comments and constructive feedback.

Computational Complexity: The meta-learning phase involves training over multiple source tasks and iterative EM updates and maybe computationally demanding on larger datasets or in high-dimensional settings.

We agree that meta-learning methods, including our method, generally take more computation cost than ordinary machine-learning methods, especially when data in source tasks are large. However, meta-learning methods can improve the performance greatly as shown in Tables 1 and 2. In addition, the meta-learning time of our method is significantly shorter than existing gradient-based meta-learning methods (MAML) due to the efficient closed-form EM steps, which is demonstrated in Sections 4.3 and I.11. Although our method requires iterative EM updates, Figure 6 shows that our method even with small EM steps (2 or 3) performed best. Even if the input dimension is large, since our EM steps are performed on the $M$-dimensional latent space where users can control the dimension, the computation cost of our EM algorithm is low when $M$ is not large.

Impractical Settings Besides, such high-quality source samples in the source tasks might not be available in practice. Even in most existing test sets (i.e., CIFAR-10, CIFAR-100, ImageNet), there are a lot of label error issues as mentioned in R1. Noting that the testing phase still struggles to find high-quality annotated data, how can we assume perfect annotations are ready in so many source tasks?

Q4: Impractical assumptions In Line 77, the authors mentioned that Since source tasks contain only clean labeled data, This seems to be a harsh assumption. Even in most existing test sets (i.e., CIFAR-10, CIFAR-100, ImageNet), there are a lot of label error issues as mentioned in R1.

This is an important comment. As you mentioned, label noise is inevitable in real-world data. Therefore, assuming completely clean labeled data may be overly simplistic. In the paper, for the sake of convenience, we treated the labels in commonly used datasets for classification tasks (e.g., Omniglot and MiniImagenet) as correct. This assumption is also commonly adopted in experiments of existing studies on meta-learning, crowdsourcing, and noisy label learning (Finn et al., 2017; Snell et al., 2017; Hospedales et al., 2020; Chu et al., 2021; Gao et al., 2022; Guo et al., 2023).

While label noise is inherent in real-world data, the extent of noise can vary depending on the dataset. For instance, the mean labeling accuracy is 69.2% in the LabelMe dataset used as target tasks in our experiments (Chu & Wang, 2021), while it is approximately 96.7% in the standard datasets evaluated in [R1]. Therefore, in practice, to improve performance on datasets with high noise rates such as LabelMe, it would be reasonable to use datasets with relatively low noise rates as the source tasks. Standard datasets can be assumed to have relatively low noise levels and are thus appropriate for use. To clarify this point, we have added the following note in Section 1 of the revised paper:

"For convenience, throughout this paper, we assume that all labels in the source tasks are correct. However, it may be difficult to collect entirely clean data in real-world applications. Thus, in practice, we use datasets that are expected to have relatively low levels of noise as the source tasks."

Annotation Modeling Limitation: By using input-independent confusion matrices, the model does not account for instance-specific annotator behavior, potentially missing nuances in how annotator quality varies by data complexity. Extending to instance-dependent matrices (R2, R3) could improve performance since real-world noise is observed to be more related to instance-dependent label noise (R3).

Thank you for the suggestion. This paper proposes a new problem setting (i.e., meta-learning for learning from a few noisy data given multiple annotators). As a first step in solving this problem, it is better to consider simple input-independent matrices rather than complex models. Even with this simple modeling, we have confirmed that our method outperforms methods that use input-dependent matrices (LF and MLF) in Table 8.

Although the input-dependent confusion matrices could potentially improve performance as you mentioned, they might have drawbacks in terms of their efficiency in solving inner problems. Thus, as described in the conclusion, extending our method to handle input-dependent confusion matrices will challenge future work.