Crowd-Calibrator: Can Annotator Disagreement Inform Calibration in Subjective Tasks?

We thank you for your review, constructive suggestions, and appreciation for finding the combination of soft label with abstention novel.

Inconsistency. Our method especially works better when we aim to achieve higher accuracy, as the coverage is then higher than the baseline. The only case where this trend is broken is our Hate Speech dataset, where for Cov@Acc=0.9 our method gives the best coverage, but not for Cov@Acc=0.95. Additional experiments on the Online Misogyny task (as suggested by Var5, containing roughly similar amount of annotators) show slightly better performance than MaxProb with our Crowd Estimator for (baseline: 0.9205, ours: 0.9228) and AUROC (baseline: 0.7623, ours: 0.7741) and a somewhat similar pattern with more coverage for a higher accuracy. For Cov@Acc=0.85: baseline: 0.8875, ours: 0.8844 and Cov@Acc=0.9: baseline: 0.7094, ours: 0.7125. For Cov@Acc=0.95: baseline: 0.5125, ours: 0.5125.

Soft-labeling for raters. Thank you for pointing this out, we will clarify this in the paper and cite Liu et al. 2019.

We thank you for the positive review and constructive suggestions!

Main takeaway: Our results show that our method is beneficial for seen and unseen datasets when we have access to a small set of samples that have many annotations per instance (~100). Our results also show that our method can work on unseen datasets when we have access to individual annotator samples (> 2000). We will clarify these as the main takeaways of our paper.

Questions w.r.t. scoring strategies: Our method offers flexibility in the choice of the distance metric, KL-divergence, JSD, and TVD (for which we will add the formulae in the Appendix) and aggregation strategy. In general, JSD and TVD give the best results in combination with the aggregation strategies of label distribution or averaged confidences. In all cases, it is beneficial to add entropy to the score of the metric as it prevents the model from still making decisions when both the model and crowds are uncertain of their response, while distance metrics alone would allow the model to make a prediction.

Sensitivity: There does appear to be some sensitivity to the nature of the task, particularly the amount of annotations we can learn from for the Crowd Estimator. We see better performance when the Crowd Estimator learns from 100 annotations per sample in the case of NLI in comparison to estimating the crowd from 24 annotators. We’ll reflect on this in our discussion.

We thank you for your insightful comments and positive review! We answer your questions below.

Comparison to standard soft-label methods. We compare our results to standard soft-label methods (Peterson et al. 2019/Uma et al. 2020) which use human disagreement to create soft labels $\hat{y}\_{soft}$ in Table 1. To make it compatible with abstention, we apply MaxProb. However, the results are far more conservative compared to using hard labels $\hat{y}\_{maj}$.

Related Work. Thank you for the pointers; we will definitely cite these. Deep Learning from Crowd (DLC) is shown to be outperformed by learning soft labels with soft losses in Uma et al. 2020 and Uma et al. 2021, hence we did not include this method. Davani et al’s approach requires data where individual annotators have labeled many samples. This is something that many datasets and the ones we used in the paper---the HateXplain, Measuring Hate Speech Corpus, and NLI datasets—do not have. For the NLI datasets, we do not know who annotated what, we only have access to the distribution of labels per sample. Hence, we could not empirically compare to DwD; we will clarify this in our paper.

$CE_{soft}$ as an evaluation metric. We evaluated the soft label model in Section 2.3 with Jensen-Shannon Divergence (JSD) as it is strictly more principled since unlike $CE_{soft}$ it is symmetrical. When evaluating for selective prediction, we do not use $CE_{soft}$ since our approach aims to improve calibration and selective prediction for which we use the widely used metrics.

Strong Claims. Thank you for pointing this out, we agree and did not intend to suggest we are the first to point out that subjectivity should play a role, but we do want to restate its importance. We will make sure to add the references.

Thank you for your helpful comments and your appreciation of us tackling a crucial issue through a novel approach.

Additional Datasets: We chose our datasets specifically based on their complementary properties (also noted by j6sj) the most common real-world scenario with only limited annotators per sample (HS) and the ideal but rare scenario with many annotators per (NLI).

Thanks for the dataset suggestions; these are similar to HS in having limited annotators per sample. We experimented with the suggested Online Misogyny dataset, and obtained results similar to HS: our Crowd-Calibrator slightly outperforms the MaxProb baseline in terms of AUC (baseline: 0.9205, ours: 0.9228) and AUROC (baseline: 0.7623, ours: 0.7741). We will add this in our paper.

Comparison with Davani et al. (DwD) Davani et al.’s approach requires data where individual annotators have labeled many samples (as highlighted by their GoEmotions experiments). Most datasets including the HateXplain, Measuring Hate Speech Corpus, and NLI datasets do not follow this setting. For the NLI datasets, we only have access to the distribution of labels per sample. Hence, we could not empirically compare to DwD; we will clarify this in our paper.

Chosen evaluation metric. We evaluate with widely used metrics for calibration (Brier’s Score) and selective prediction (Coverage@Accuracy, AUC, and AUROC) since our proposed setup is a soft approach for selective prediction and improved calibration. This does not mean we assume there is a single ground truth; rather we are observing a single sample from the underlying crowd distribution. Our crowd-calibrator abstains such that it does not predict samples that are far off from the crowd and where they both have high entropy.

Usage of “confidence”: Thank you, we will update our paper with the term “softmax scores”.

Weighted Scoring formula. The weighted scoring we used follows:

$r_{non-hs} \cdot \text{score}(AvgConfidence_{non-hs}, \hat{y}\_{maj}) + r_{hs} \cdot \text{score}(AvgConfidence_{hs}, \hat{y}_{maj})$

where $r_c$ is the ratio of votes for the particular class $c$. We will include this in the paper.