Thank you for your encouraging review. We will take your suggestions into account when preparing the final version. Below please find responses to specific questions.

1. Empirical Evaluation: Thank you for the suggestions. As recommended by the reviewer, we will aim to include experiments involving real-world experts, such as those considered in (Mozannar et al., 2023) and (Palomba et al., 2024), in the final version.

But, as you have mentioned, our paper primarily focuses on theoretically principled surrogate losses for learning to defer with multiple experts, grounded in realizable $H$-consistency and $H$-consistency bounds, within the learning theory framework. While we have empirically shown that minimizing our proposed loss functions outperforms the baselines in realizable settings and matches the best-known results in synthetic non-realizable settings—as predicted by our theoretical analysis—we recognize the importance of further empirical exploration. Accordingly, we plan to dedicate future work to conducting a more extensive empirical analysis in real-world settings, particularly in important scenarios where expert domains overlap (as suggested by Reviewer wQs7) and the data is heterogeneous (as suggested by Reviewer 798e).

We will also include a more detailed discussion of the limitations of our empirical analysis and outline potential directions for future work in the final version. Nevertheless, we believe our theoretical contributions are significant and well-suited for acceptance in the theory track of the ICML conference.

2. Deferring More to Experts: We agree with the reviewer that deferring more to experts may not always be preferred in real-world applications, where querying experts can be more costly than relying on an ML model. We would like to clarify that the goal of our experiments on non-realizable data is not to claim that our proposed surrogate losses outperform the baselines in such settings. Rather, our results demonstrate that our surrogate losses match the performance of these state-of-the-art baselines in non-realizable scenarios in terms of system accuracy, as both are supported by consistency bounds.

The strength of our approach lies in the realizable setting, where our surrogate losses are guaranteed to be realizable $H$-consistent—unlike the baselines—as illustrated in Figure 1. Due to the specific realizable form of our surrogate losses, our model also defers significantly more and exhibits a greater tendency to select expert 2 compared to the baselines of Verma et al. (2023) and Mao et al. (2024b). This highlights a key effect of our proposed surrogate: it induces a deferral behavior that is qualitatively different from prior approaches.

3. Theoretical Claims: Thank you for the suggestions. We will revise the notation for the conditional distribution and add further commentary on the proof based on your feedback.

4. Other Comments Or Suggestions: Thank you for the suggestion. We will double-check the arXiv references and update them with their accepted versions, if available, in the final version.

Thank you for your appreciation of our work. We will take your suggestions into account when preparing the final version. Below please find responses to specific questions.

1. Questions: Are there any other interesting properties or improvements the authors want to achieve for their loss family besides those already mentioned in the paper? It would be nice if the authors could discuss a few sentences about future directions.

Response: As suggested by Reviewer 798e, extending our methods to more realistic settings—such as those where experts may not be available during training—is a promising direction for future work. For example, it would be interesting to adapt our framework to handle previously unseen experts at test time, as studied by Tailor et al. (2024, AISTATS), and to incorporate post-processing frameworks for learning to defer under constraints such as OOD detection and long-tail classification, as explored by Charusaie et al. (2024, NeurIPS). We will further elaborate on this potential direction in the paper.

We also plan to conduct a more extensive empirical analysis in real-world settings, particularly in important scenarios where expert domains overlap (as suggested by Reviewer wQs7) and where the data is heterogeneous (as suggested by Reviewer 798e).

[1] Tailor et al. (2024). Learning to Defer to a Population: A Meta-Learning Approach. AISTATS 2024.

[2] Charusaie et al. (2024). A Unifying Post-Processing Framework for Multi-Objective Learn-to-Defer Problems. NeurIPS 2024.

Thank you for your insightful comments. We will take your suggestions into account when preparing the final version. Below please find responses to specific questions.

1. Experimental Designs Or Analyses: Thank you for the suggestions. As recommended by Reviewer m1gt, we will aim to include experiments involving real-world experts, such as those considered in (Mozannar et al., 2023) and (Palomba et al., 2024), in the final version.

Our paper primarily focuses on theoretically principled surrogate losses for learning to defer with multiple experts, grounded in realizable $H$-consistency and $H$-consistency bounds, within the learning theory framework. While we have empirically shown that minimizing our proposed loss functions outperforms the baselines in realizable settings and matches the best-known results in synthetic non-realizable settings—as predicted by our theoretical analysis—we recognize the importance of further empirical exploration. Accordingly, we plan to dedicate future work to conducting a more extensive empirical analysis in real-world settings, particularly in important scenarios where expert domains overlap (as suggested by Reviewer wQs7) and the data is heterogeneous (as suggested by Reviewer 798e).

We will also include a more detailed discussion of the limitations of our empirical analysis and outline potential directions for future work in the final version. Nevertheless, we believe our theoretical contributions are significant and well-suited for acceptance in the theory track of the ICML conference.

2. Relation To Broader Scientific Literature: As the reviewer suggests, we are happy to include an additional discussion of methods based on uncertainty estimation. Previous work has shown that some of these methods can be suboptimal compared to those based on deferral or abstention, as studied in our paper. The reviewer references “uncertainty estimation” and “anomaly detection” methods multiple times; if there are specific publications they recommend, we would gladly discuss them in the final version.

3. Problem Setting: We would like to point out that the same problem setting has been considered in several notable previous works on learning to defer (Mozannar & Sontag, 2020; Verma & Nalisnick, 2022; Charusaie et al., 2022; Mozannar et al., 2023; Verma et al., 2023; Mao et al., 2024b, among others). Within this setting, our paper introduces novel surrogate loss functions and efficient algorithms with strong theoretical learning guarantees. We address open questions regarding realizable $H$-consistency, $H$-consistency bounds, and Bayes consistency for both the single-stage setting (jointly learning the predictor and deferral function) and the two-stage setting (learning only the deferral function with a fixed expert).

We acknowledge that more realistic settings—such as those where experts may not be available during training, as mentioned by the reviewer—are also important. Extending our methods to these scenarios is a promising direction for future work. For example, it would be interesting to adapt our framework to handle previously unseen experts at test time, as studied by Tailor et al. (2024, AISTATS), and to incorporate post-processing frameworks for learning to defer under constraints such as OOD detection and long-tail classification, as explored by Charusaie et al. (2024, NeurIPS). We will further elaborate on this potential direction in the paper.

[1] Tailor et al. (2024). Learning to Defer to a Population: A Meta-Learning Approach. AISTATS 2024.

[2] Charusaie et al. (2024). A Unifying Post-Processing Framework for Multi-Objective Learn-to-Defer Problems. NeurIPS 2024.

4. Application to Large Language Models: Our deferral solution can indeed provide a useful solution in the context of LLMs. Given the significant resources required to retrain LLMs, a two-stage method is a practical and scalable solution. This does not require modifying LLMs or the loss function used to train them.

Our proposed two-stage surrogate loss framework is particularly well-suited for use with pre-trained LLMs. In the presence of multiple LLMs (or experts), using our method, uncertain predictions can be deferred to more specialized or domain-specific pre-trained models, thereby both improving the reliability and accuracy of the overall system and improving efficiency. We will further elaborate on this in the final version.

We have in fact carried out preliminary experiments demonstrating the effectiveness of our deferral method in enhancing LLM performance. However, these results are currently not reported publicly due to IP restrictions.

Thank you for your encouraging review. We will take your suggestions into account when preparing the final version. Below please find responses to specific questions.

1. Claims regarding Verma et al. (2023): Thank you for pointing this out. We agree with the reviewer that Verma et al. (2023) was the first to extend the surrogate losses proposed in (Verma & Nalisnick, 2022) and (Mozannar & Sontag, 2020) to the multiple-expert setting in L2D. We will definitely revise the claims accordingly throughout the paper.

2. Experiments: Thank you for the suggestions. As recommended by Reviewer m1gt, we will aim to include experiments involving real-world experts, such as those considered in (Mozannar et al., 2023) and (Palomba et al., 2024), in the final version.

Our paper primarily focuses on theoretically principled surrogate losses for learning to defer with multiple experts, grounded in realizable $H$-consistency and $H$-consistency bounds, within the learning theory framework. While we have empirically shown that minimizing our proposed loss functions outperforms the baselines in realizable settings and matches the best-known results in synthetic non-realizable settings—as predicted by our theoretical analysis—we recognize the importance of further empirical exploration. Accordingly, we plan to dedicate future work to conducting a more extensive empirical analysis in real-world settings, particularly in important scenarios where expert domains overlap (as suggested by Reviewer wQs7) and the data is heterogeneous (as suggested by Reviewer 798e).

We will also include a more detailed discussion of the limitations of our empirical analysis and outline potential directions for future work in the final version. Nevertheless, we believe our theoretical contributions are significant and well-suited for acceptance in the theory track of the ICML conference.

3. Relation To Broader Scientific Literature: Thank you for the suggestion. As recommended by the reviewer, we will incorporate and discuss literature on the application of deferral-based mechanisms in real-world machine-collaboration scenarios in the final version.

4. Introduction: Thank you for the suggestion, we will seek to revise the introduction accordingly.

5. Figure 1: The behavior for lower values of $q$ is similar to that observed for $q = 0.7$ and $q = 1$. Note that for any $q > 0$, Theorem 3.2 guarantees realizable $H$-consistency. We chose $q = 0.7$ and $q = 1$ because they correspond to standard choices in prior work—$q = 0.7$ is a commonly suggested value within the generalized cross-entropy family (e.g., Zhang & Sabuncu, 2018; Mao et al., 2024b), and $q = 1$ corresponds to the mean absolute error loss, as used in (Ghosh et al., 2017; Mao et al., 2024b).

In this realizable distribution, we observe that our proposed surrogate losses achieve close to 100% system accuracy, while all other baselines fail to find a near-zero-error solution. This demonstrates that our surrogates are realizable $H$-consistent, whereas the compared baselines are not.

6. Table 1: That's a good observation. We are not claiming that our proposed surrogate losses outperform baselines such as Verma et al. (2023) on non-realizable distributions. Rather, our results show that our surrogate losses match the performance of these state-of-the-art baselines in non-realizable settings, as both are supported by consistency bounds.

The strength of our approach lies in the realizable setting, where our surrogate losses are guaranteed to be realizable $H$-consistent, unlike the baselines—as illustrated in Figure 1. Due to the specific realizable form of our surrogate losses, our model also defers significantly more and shows a greater tendency to select expert 2 compared to the baselines of Verma et al. (2023) and Mao et al. (2024b). This highlights a key effect of our proposed surrogate: it induces deferral behavior that is qualitatively different from prior approaches, particularly on the SVHN and Tiny ImageNet datasets.