Adversarial Robustness in Two-Stage Learning-to-Defer: Algorithms and Guarantees

We sincerely thank the reviewer for their encouraging and constructive feedback. We appreciate their recognition of the rigor of our theoretical contributions and the strength of our experimental results in supporting our claims. We also acknowledge and value their observation that, to the best of their knowledge, this is the first work to study adversarial robustness in L2D. Please find our clarifications below.

The authors checked the performance of the proposed methods in classification, regression, and multi-task settings, which sufficiently supported the generalization of their methods. However, it would be better if they could evaluate more datasets for each setting; currently, each task involves a single dataset.

Could you include more datasets in the evaluation of each task?

We would like to emphasize that the primary objective of our paper is to establish a rigorous theoretical foundation for defending against adversarial attacks in the Learning-to-Defer setting. Our experiments primarily serve to empirically validate the proposed theoretical framework, rather than to benchmark performance across a wide range of datasets. Given the theoretical focus and current standards within the L2D community, we believe our existing experiments across classification, regression, and multi-task scenarios sufficiently demonstrate generality (see discussion with reviewer @dgrk).

Nonetheless, exploring more datasets remains a valuable direction for future empirical work.

While the authors do not present many reasoning about why this is important, I believe that evaluating from adversarial perspective for a possibly deployed paradigm is useful.

Thank you for recognizing the importance of evaluating Learning-to-Defer from an adversarial perspective. We strongly agree that such robustness evaluation is crucial. L2D models are increasingly deployed in high-stakes applications, including healthcare [10, 11], or more broadly autonomous decision-making, where adversarial manipulation leading to incorrect deferral decisions can have severe real-world consequences. Specifically, targeted attacks can strategically force the rejector to allocate more queries to a particular agent, thereby causing a harmful bias in the allocation process. On the other hand, untargeted attacks degrade the overall performance and reliability of the entire system, making its behavior unpredictable by maximizing the occurrence of errors.

We further discuss potential misuse of L2D in the discussion with reviewer @689i and will clarify these critical points in our revised manuscript.

References

[10] Strong et al. (2025). Towards Human-AI Collaboration in Healthcare: Guided Deferral Systems with Large Language Models. AAAI25

[11] Joshi et al. (2021). Learning-to-defer for sequential medical decision-making under uncertainty. TMLR21

We thank the reviewer for their thoughtful evaluation and appreciate the recognition of our experiments and theoretical claims. In the following, we clarify our motivations, provide justification for our design choices (e.g., the multi-task emphasis in Section 3), and highlight the technical challenges addressed in deriving SARD. We also discuss how our proposed attacks go beyond existing methods and why they are theoretically and practically significant in the L2D context.

One issue with the paper is that Section 3 begins with the multi-task [...] discussing multi-task learning as an extension.

Our motivation for initially framing Section 3 within the multi-task setting was not to imply that L2D is predominantly studied or especially relevant only to multi-task problems. Rather, we aimed to emphasize the generality of our theoretical results, demonstrating explicitly that the agent costs $c_j$ can take any positive form—classification [5], regression [6], or any multi-task metrics. We intended to highlight that our proofs, losses, attacks, and defense methods hold broadly across these different problem formulations, a fact that is non-trivial [5,6] and significant for demonstrating the versatility of our approach.

In simple words, our approach (SARD) can be used in any setting. We thank you for pointing out this potential misunderstanding and will clarify this in the final version.

The paper does not sufficiently highlight the challenges and technical difficulty of the problem [...] The technical difficulty of deriving SARD beyond this theoretical stacking is unclear.

While the base attack strategies we adapt were initially developed for multiclass classification, a key contribution of our work lies in their novel extension to the fundamentally different setting of L2D. Specifically, we are the first to show explicitly how these attack methods can be repurposed to strategically corrupt query routing decisions, which we believe is not trivial. Importantly, our attacks target the rejector itself (responsible for allocation decisions), rather than the agents performing the tasks, which is an important distinction to multiclass classification. This design choice stems from a core characteristic of the two-stage L2D setting: we do not have access to the internal structure or training procedure of the agents, as also assumed in [4,5].

We provide a concrete example and detailed discussion (see response to reviewer @689i) highlighting why robustness against such novel attacks is critical, given the high-stakes decisions typically addressed by L2D systems.

Regarding technical complexity, we emphasize that the losses, attack strategies, and theoretical results introduced in our paper are entirely novel and require dedicated analysis, as formalized in Theorem 5.7. While previous works (e.g., [7,8,9]) have studied adversarial surrogate losses, their analyses are confined to multiclass classification and do not extend naturally to the L2D. As discussed with reviewer @689i, applying standard adversarial training directly to the L2D loss (Definition 3.1) overlooks the worst-case allocation scenario—captured in Lemma 5.1—and thus leaves the system vulnerable to attacks. Moreover, even in the absence of adversarial perturbations, proving consistency guarantees for L2D is nontrivial, as demonstrated in recent literature [3,4,5,6,12]. Furthermore, Lemma 5.6 and its proof are entirely original and differ from prior results (e.g., [7,8,9]). Notably, our theoretical framework explicitly addresses worst-case deferral loss by introducing adversarial inputs tailored for each individual agent $j \in \mathcal{A}$—a novel and technically challenging aspect that is not present in standard multiclass classification analyses.

We hope this clarifies the challenges involved, and we would be glad to elaborate further if needed.

References

[3] Mozannar et al. (2021). Consistent estimators for learning to defer to an expert. ICML21

[4] Verma et al. (2023). Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensembles. AISTATS23

[5] Mao, et al. (2023). Two-Stage Learning to Defer with Multiple Experts. NeurIPS23

[6] Mao, et al. (2024). Regression with multi-expert deferral. NeurIPS24

[7] Awasthi, et al. (2023). Theoretically Grounded Loss Functions and Algorithms for Adversarial Robustness. AISTATS23

[8] Mao et al. (2023). Cross-entropy loss functions: theoretical analysis and applications. ICML23

[9] Bao, et al. (2021) Calibrated surrogate losses for adversarially robust classification. COLT21

[10] Strong et al. (2025). Towards Human-AI Collaboration in Healthcare: Guided Deferral Systems with Large Language Models. AAAI25

[11] Joshi et al. (2021). Learning-to-defer for sequential medical decision-making under uncertainty. TMLR21

[12] Mao et al. (2024). Principled Approaches for Learning to Defer with Multiple Experts. ISAIM24

We sincerely thank the reviewer for their thoughtful and constructive feedback. We are grateful for your recognition of the rigor in our theoretical contributions and the strength of our empirical validation.

The computational complexity [...] practical deployment considerations.

Thank you for highlighting this important consideration. Let $\mathcal{F}$ denote the computational cost of performing a single forward–backward pass for the rejector model $r \in \mathcal{R}$. For standard L2D approaches involving $|\mathcal{A}| = J + 1$ agents, the complexity is $\mathcal{O}(\mathcal{F} + J)$. SARD, involves performing adversarial training through a PGD, which requires $T$ forward–backward passes per agent. Consequently, the computational complexity of SARD is $\mathcal{O}\bigl((J+1) T \mathcal{F}\bigr)$. We will explicitly clarify this complexity comparison in the revised manuscript.

The paper doesn't explore whether [...] as an alternative solution.

We demonstrate that naively applying standard adversarial training to existing L2D baselines (Definition 3.1) fails to minimize the desired worst-case deferral loss (as shown in Lemma 5.1). As a result, an attacker could still exploit the allocation mechanism if adversarial training is applied to the non-worst-case deferral loss defined in Definition 3.1.

This motivates our design and optimization of a distinct loss function—one that fundamentally departs from the standard adversarial training objective. We will further clarify this in the main body of the paper.

Minor typo on page 5: [...] missing the complete definition.

Thank you for pointing this out. We will correct this.

How does SARD's performance change [...] compare to standard adversarial training approaches?

We did not observe any unexpected behavior from SARD compared to standard adversarial training [7,8,9]. Intuitively, as the adversarial budget increases, the problem becomes inherently more challenging to defend against. From a theoretical standpoint, SARD does not exhibit any particular breakdowns beyond those already known for standard adversarial training.

Did you explore the effectiveness [...] with simpler adaptations of existing methods.

Yes, we explored this explicitly in our experiments. In each considered scenario, we evaluated baseline approaches both on clean datasets and under our novel attacks with a PGD [13]. We demonstrate that existing baseline methods are highly vulnerable to both our attacks, while our proposed approach, SARD, consistently outperforms these baselines by a significant margin under adversarial conditions.

How sensitive is SARD [...] practical guidelines for implementation?

Indeed, SARD's performance depends on these hyperparameters, similar to smoothing approach in adversarial training [7,8]. Unlike previous observations in multiclass classification [7, 8], SARD generally achieves better performance with relatively smaller values of $\nu$. This suggests that our L2D formulation inherently requires a lighter adversarial regularization term, likely due to the increased complexity of the decision boundaries involved.

We will explicitly clarify these practical guidelines.

For real-world deployment [...] benefits would clearly outweigh the slight decrease in clean performance?

L2D frameworks are specifically designed for scenarios where decision-making carries important risks, thereby making robustness an essential consideration. Consider a hospital deploying L2D for cancer diagnosis. Ideally, L2D would allocate cancer detection queries to the most suitable agent available. Suppose the hospital has several agents: a neurologist, a dermatologist, a general practitioner, and a properly trained AI model, each with distinct consultation costs (e.g., neurologist: $5\beta_1$, dermatologist: $5\beta_1$, general practitioner: $\beta_1$, AI model: $0$).

In this realistic scenario, an adversary might deliberately manipulate the L2D system (rejector) through the attacks we have introduced: our untargeted attack (Definition 4.1) could cause misallocation of queries, leading to critical cases, such as complex skin cancer detection, being incorrectly routed from the dermatologist to a less specialized agent like the AI model, increasing the risk of making a mistake. Alternatively, using our targeted attack (Definition 4.2), an adversary might intentionally route straightforward cases to expensive experts (e.g., neurologist) to unnecessarily increase costs ($5\beta_1$ instead of $0$), or maliciously redirect consultations to a collaborating agent motivated by financial gain.

Given these potential vulnerabilities, we firmly believe that the benefits of robustness clearly outweigh minor decreases in clean-data performance in such high-stakes settings. This motivates our approach, especially as L2D frameworks are increasingly adopted in critical applications [10, 11].

See discussion with reviewer @BFHK for references.

We thank the reviewer for their thoughtful and constructive feedback. We are glad that they found our contributions meaningful, and we appreciate their recognition of the soundness of both our theoretical and empirical results.

In the experiments, [...] with real-world expert predictions (e.g., CIFAR-10H) can be included.

We want to emphasize that the primary contribution of our paper is theoretical. The experiments presented are primarily intended to empirically support and illustrate these theoretical results.

Within the L2D community, synthetic experts are widely preferred for theoretical evaluation precisely because they allow controlled, systematic analyses across diverse conditions (e.g., specialized expert behaviors, and critical edge-case scenarios) that are generally not reproducible with currently available real-world expert labels [2,3,4,5,6]. For instance, we introduce synthetic experts to rigorously expose vulnerabilities to our novel targeted attack—a scenario that is difficult to construct using existing real-world dataset, yet remains plausible in practical deployments.

Nonetheless, we acknowledge the value of evaluating real-world datasets and consider this for future empirical exploration.

j-th adversarial margin surrogate closely resembles the structure of Gamma-Phi losses [...] relevant to this study.

Very good question! We confirm that our surrogate can be rewritten in a Gamma-Phi form $\widetilde{\Phi}^{\rho,u,j}_{01}(r, x, j)= \sup\_{x\_j' \in B\_p} \gamma ( \sum\_{j' \neq j} \phi \big( r(x\_j', j') - r(x\_j', j) \big))$ with $\gamma(v)=\log(1+v)$ (assuming $v=1$) and $\phi(v)=\min(\max(0,1 - v/\rho), 1)$. However, despite satisfying the Gamma-PD condition (Definition 3.1 in [1]), our surrogate does not satisfy Definition 3.2 (Phi-NDZ), as $\phi(v)$ is not differentiable on $\mathbb{R}$. Thus, we cannot directly leverage the conclusions of Theorem 2.6 from [1] to prove classification-calibration.

Furthermore, the classification-calibration provided by [1] implicitly assumes the hypothesis class includes all measurable functions $\mathcal{R}_{\text{all}}$, a strong assumption that does not hold in our analysis. Additionally, while classification-calibration implies Bayes-consistency—an asymptotic property—our Theorem 5.7 provides stronger finite-sample guarantees via explicit inequalities, without highly restricting the hypothesis class $\mathcal{R}$.

We will explicitly discuss these distinctions in the revised manuscript.

The training [...] post-hoc estimator for L2D [2].

Thanks for the suggestion—we will include it. In fact, [5] can be viewed as an extension to the multi-expert setting, whereas [2] addresses only the single-expert case.

There are some minor formatting issues in the citations [...] preprints.

Thank you for pointing this out. We will correct this in the revised manuscript

This paper primarily focuses on attacks targeting the deferral rule, whereas attacks on the classification rule [...] combination as a direction for future work.

We agree that investigating combined attacks represents an interesting direction for future research.

However, we would like to emphasize that the motivation for our work stems from the two-stage L2D setting [2,5,6], where the rejector is solely responsible for allocating each query to an external agent. In this setup, the external agents are fixed and not accessible beyond their output predictions. That is, we do not have access to their internal parameters, decision boundaries, or training pipelines; moreover, they may not even be describable as functions (e.g. human decision-makers).

As such, while we acknowledge that attacks on the classification and deferral rules are not mutually exclusive in general, the structure of our problem precludes joint attacks. We cannot meaningfully design or evaluate perturbations that target the expert prediction, as we lack the ability to interact with or analyze the internal behavior of the experts. This naturally restricts our adversarial analysis to the deferral mechanism, which is the only component under the learner’s control.

Moreover, robustness in classification has been extensively studied. In contrast, robustness in L2D systems remains unexplored. Our work aims to address this gap.

We will clarify this in the revised manuscript.

References

[1] Wang et al. (2023). On classification-calibration of gamma-phi losses. COLT23

[2] Narasimhan et al. (2022) Post-hoc estimators for learning to defer to an expert. NeurIPS22

[3] Mozannar et al. (2021). Consistent estimators for learning to defer to an expert. ICML21

[4] Verma et al. (2023). Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensembles. AISTATS23

[5] Mao, et al. (2023). Two-Stage Learning to Defer with Multiple Experts. NeurIPS23

[6] Mao, et al. (2024). Regression with multi-expert deferral. NeurIPS24