Probabilistic Learning to Defer: Handling Missing Expert Annotations and Controlling Workload Distribution

We thank the reviewer for all the minor issues in the initial review. We have accummulated those issues and updated in our revision. In particular:

• For the reference: We apologise for overlooking this matter. We removed the reference in our revision.
• For the statement of workload constraint: We revise the sentence as follows: "Ideally, in many applications where human experts are equally-competent, a perfect balanced workload among experts can be expressed as follows:" in our revision.
• Similar experiment setting: What we meant is to simulate asymmetric label noise similar to (Hemmer et al., 2023). However, instead of using strength and a pre-trained EfficientNet-B1, we simulate label noise based on the common setting of asymmetric label noise in noisy label learning. To increase clarity, we remove the word slightly in our revision.
• Missing conclusion: We created a new section for discussion of our limitations and dedicating the last section for concluding. Please see our revision for this change.
• Typos: We corrected all the typos detected after our initial submission in our revision.

To ensure consistency with the number of testing samples across all datasets, we employ a fixed number of significant digits in our reported results. As the number of testing samples varies from 1,000 (e.g., NIH-AO, Chaoyang, MiceBone) to 10,000 (e.g., Cifar-100), a precision of three or four digits is necessary. To maintain uniformity in Table 1, we have standardised the reporting to four significant digits.

As mentioned in our paper, each L2D method has to be run multiple times to construct its coverage - accuracy curve to analyse the trade-off between coverage and prediction accuracy (or, reliability versus operating cost mentioned in the introduction of our paper). We can also run multiple times at each targeted coverage to obtain a finer estimation of uncertainty. In that case, each point on a coverage - accuracy curve will be represented by two error bars: one along the x-axis denoting the variation of coverage and the other along y-axis denoting the variation of prediction accuracy. Although that provides a fine-grain uncertainty estimation, it is costly and time-consuming. Furthermore, that uncertainty complicates the calculation of area-under-curve. That is why we decide to follow a simple solution, which is to construct the coverage - accuracy curves only.

We thank the reviewer for the comment. We are actively looking at clustering human experts as a solution to address the scalability of our proposed method. As mentioned in the paper, by clustering $M$ human experts into $K$ groups, where $M \gg K$, we can reduce from a large number of $M$ models to a much small number of $K$ models to represent each group of human experts.

We acknowledge that there are some risks associated with that clustering approach. For example:

• Clusterability: which indicates whether data representing the pattern of labelling can be separated and grouped into clusters,
• Optimality: the clustering may over-simplify user labelling behaviour into coarse clusters that do not represent well the users.

These risks can be mitigated by the collection of data that make the labelling patterns of human experts distinguishable. Alternatively, we can try the following options: i) perform a study on the number of clusters, and ii) put in place a rigorous cluster validation with quantitative metrics (e.g., silhouette score) and qualitative checks (e.g., expert domain validation).

We thank the reviewer for the suggestion. We agree with the reviewer that the current setting of L2D assumes a static performance of each human expert, although this is not true in practice.

To accommodate dynamic expert performance, the model could be extended to incorporate temporal dynamics and performance variability by integrating sequential modelling techniques or adaptive mechanisms. For fast-changing performance, such as fatigue, the model could include real-time performance tracking using metrics such as dynamic accuracy, allowing the adjustment of the deferral strategy based on current conditions. Slow-changing performance, like expertise improvement through learning, can be modelled using techniques such as Bayesian updating to adapt expert-specific parameters over time. Temporal modelling (e.g., recurrent neural networks) can also capture patterns in performance changes, enabling the system to make adjustments. By integrating such adaptive mechanisms, the model would better reflect the short-range and long-range evolving nature of human expertise, improving its decision-making capabilities in dynamic scenarios.

In our revision, we included the discussion on the performance change of human experts in Section 7.

We thank the reviewer for the constructive suggestion. As we mentioned in the paper, L2D aims to maximise reliability, while keeping the cost within the acceptable limits. The choice of the ideal operating point that balances reliability and costs typically depends on specific priorities. In high-stakes applications (e.g., healthcare), deferring uncertain cases offers significant advantages, such as improving decision accuracy, enhancing trust by acknowledging AI limitations, and leveraging human oversight to ensure accountability in high-stakes scenarios. Deferred cases can also provide valuable learning opportunities to improve the AI model. However, deferring cases can be costly due to increased annotation budgets, potential expert burnout, and possible delays in decision-making.

On the other hand, minimising deferrals maximises the efficiency of the AI system, allowing for lower costs and quicker decision-making. This approach also reduces reliance on human experts and demonstrates the AI's ability to handle easier cases independently. However, not deferring uncertain cases poses risks, such as potential misdiagnoses and diminished oversight. These risks can have severe consequences for patient safety and undermine trust in the system, especially when high-stakes decisions are involved.

An optimal approach involves adaptive strategies that balance costs, risks, and accuracy. This strategic management allows medical AI systems to maintain high standards of care while remaining cost-effective.

We also included this discussion in section 7 of our revision.

We reported the running time of each method in Table 3 of the Appendices of our original submission. To make it clearer to reader, we revise our paper and explicitly refer to Table 3 as the computational time when discussing our limitations (please refer to line 495 in section 7 of our revision).

We agree with the reviewer that in case of diverse human experts, the number of clusters may be large, which requires the application of further dimensionality reduction (e.g., hierarchical clustering as suggested by the review). Alternatively, we can use a conditional model $h(\mathbf{x}, \zeta)$, where $\zeta$ is the embedding of a human expert, to model each human expert. The embedding of each human expert $\zeta$ can be obtained by extending the sample-wise representation learning to set-wise representation learning (e.g., using contrastive learning coupled with "deep set" (NIPS 2017)). Such a conditional model allows the scalability of our method to a large number of human experts.

We included the discussion on the diverse number of human experts and conditional modelling in our revision and highlighted it in blue.

As mentioned at line 107, $\mathbf{y}$ represents ground truth label, and $\mathbf{t}$ represents annotations of human experts. In our modelling, $\mathbf{y}$ is derived from expert annotations $\mathbf{t}$ and selection $\mathbf{z}$. While $\mathbf{x}$ provides the input context, the model assumes that all relevant information from $\mathbf{x}$ is captured via $\mathbf{t}$ and $\mathbf{z}$, making $\mathbf{y}$ conditionally independent of $\mathbf{x}$. This design prioritises modelling expert-driven uncertainty and subjectivity, aligning with real-world annotation practices.

The confusion at line 150 between $\mathbf{y}$ and $\mathbf{t}$ is due to the need to train the gating model and the classifier at the same time. To simplify, an equivalent procedure is to remove the second term from Eq. (1) (from the submitted paper) by training the classifier using ground truth labels. In that case, the trained classifier is considered as one expert. In the subsequent step, Eq. (1), from the submitted paper, consists of only the first term to train the gating model, and $\mathbf{t}$ is simply the annotations made by experts (either human or the classifier) without being modified. For clarification, we add an explanation in our revision and highlight in blue.

In theory, the log-likelihood of the ground truth label $\mathbf{y}$ evaluated on the categorical distribution parameterised by annotation $\mathbf{t}$ is still valid even though $\mathbf{t}$ is a one-hot vector (the log-likelihood is either 0 or negative infinity at the limit). However, that may lead to numerical instability in practice. Hence, in our implementation, we smooth the annotation $\mathbf{t}$ to avoid such numerical issue.

We thank the reviewer for the constructive suggestion to improve the paper. We revised the paper by scaling the y-axes of subplots in Figures 3 and 5 to increase the clarity in the qualitative analysis. Please refer to our latest revision uploaded to Open Review for this change.

We also provide a link to an anonymous repository of our implementation in our revision. Please refer to that repository for the details of our implementation.

In fact, imposing workload constraint is a form of injecting prior knowledge into the estimation of posterior of latent variables, resulting in a form of regularisation. Different prior knowledge to define workload distribution will result in different regularisation effects. As for the question raised by the reviewer, assigning smaller weight to a particular expert is equivalent to integrate a prior into training, and hence, it is equivalent to a regularisation.

We agree with the reviewer that the area under the coverage - accuracy curve increases with higher missing rates on MiceBone across all methods. This is due to the missing data coupled with the inconsistent performance of human experts in training and testing.

For example, let's consider the naive baseline between two cases: 1) no missing label, and 2) 70% missing labels. Without any regularisation, the naive L2D will often select the best human expert observed in the training set. According to the updated Table 2 presented below (and also updated in our revision in Appendix F), which presents the performance of human experts in training and testing, the naive L2D will most likely select expert 534 in the case of no missing label, and expert 290 in the case of 70% missing label due to their highest performance in each setting (see the updated Table 2 below). This coupled with the inconsistent performance in the testing set (see the column Test in the same table) leads to the observation: lower performance at lower missing rate during testing. For example, 84.45% prediction accuracy when there is no missing annotation versus 85.29% at 70% missing rate. In our revision, we add a discussion in the section of experiments to explain this observation on MiceBone.

The high performance that surpasses the best human expert at low coverage is attributed by the learnt deferral mechanism of the gating model, which selects the correct human expert to make decision. Indeed, this is one of the motivations to study the human - machine cooperation mentioned in the introduction of our paper.

To make it clearer, we note that the human experts considered in each experiment provide complementary label correctness, compared to each other. For example, in Cifar-100, each expert makes mistakes on non-overlapping super-classes, and hence:

$

        \begin{aligned}[b]
            \text{Even though }\Pr(\mathbf{t}^{(1)} = \mathbf{y}) \approx \Pr(\mathbf{t}^{(2)} = \mathbf{y}) \approx 0.75,
            \text{we can have }\Pr(\mathbf{t}^{(1)} = \mathbf{y} \quad \operatorname{OR} \quad \mathbf{t}^{(2)} = \mathbf{y}) = 1,
        \end{aligned}
    

$

    Ideally, a perfect gating model will defer each query to the correct human expert, reaching 100\% prediction accuracy (when their label correctness is complementary). That explains why in Cifar-100 and Chaoyang, the L2D system performs even better than the best human expert at low coverage. Note that, in Figure 3, the results are lower than 100\% is because 1) *imperfect gating model*: the gating model is not perfect and cannot always select the correct human experts, and 2) *missing annotations* introduces uncertainty, resulting in difficulty for the gating model to defer accurately.