Human Expertise Really Matters! Mitigating Unfair Utility Induced by Heterogenous Human Expertise in AI-assisted Decision-Making

We appreciate the reviewer’s recognition of the novelty of our problem and the methodological contributions of our work. Thank you for your valuable and thoughtful comments. Please find our point-by-point responses below:

Regarding the comment “The paper is limited to a fairly simple setting”

We would like to clarify that group-level multicalibration can be applied to non-binary decision settings. The challenge in generalizing from binary decision problems to more complex decision spaces lies in the identifying appropriate utility function conditions in such settings (Section 6.2). This requires a social-research into modeling and understanding human utility preferences and behaviors, which is beyond the scope of this work. Therefore, we chose the binary decision setting to provide a solid foundation for future exploration of more general decision tasks.

Regarding the comment: “Without substantial innovation beyond group-level confidence multicalibration.”

We respectfully disagree with this point. While group-level multicalibration represents an innovation in our work, developing a comprehensive theoretical foundation for identifying and addressing fairness issues represents a more significant and non-trivial contribution. We would like to clarify the detailed novelty includes:

1. The first work focusing on fairness in this context: Our work highlights a previously overlooked fairness issue in AI-assisted decision making, which is critical for real-world applications. As discussed in Lines 89–96, ensuring fair utility is essential for advancing AI for social good and improving societal welfare. Neglecting fair utility can, on one hand, undermine human willingness to adopt AI assistance, and on the other hand, exacerbate utility disparities among decision-makers due to the Matthew Effect, ultimately harming societal welfare. These factors critically influence the feasibility of AI-assisted decision-making systems in practice, which are unfortunately ignored by previous work.

2. A novel calibration objective: We propose a new fairness-driven calibration objective, inter-group alignment, which fundamentally differentiates our theoretical framework and methods from prior work.

3. A tight theoretical framework of fairness issue in AI-assisted decision making: We provide the first in-depth analysis of the fairness limitations in existing confidence calibration methods. We derive the tight upper bound on utility disparity (which is really non-trivial), justifying the need of multicalibration in group-level—a focus absent in previous studies.

Thank you for your feedback. Regarding the two remaining suggestions you still hold, we would like to provide the following clarifications:

Regarding your concern and suggestion that "The current percentile-based experimental results do not demonstrate significant improvement. I suggest that the authors consider conducting statistical tests rather than relying solely on percentile-based analysis."

We have conducted T-tests on the outcomes of $100$ repeated experiments (random seeds 0–99) to assess the statistical significance of differences. The following table presents the results. The key takeaways are:

1）Across all four experiments, our group-level multicalibration consistently demonstrates higher mean utility and lower mean utility disparity, outperforming the baseline multicalibration

2） The T-tests highlight that the differences observed in both utility and utility disparity between the two methods are statistically significant with $p-value<0.05$.

Regarding your concern that our work is "restricted to binary decision-making scenarios" and the suggestion that "extending this approach to multi-class settings would be far more compelling"

We acknowledge that our theoretical results apply only to binary-label decision-making tasks, and extending the analysis of fairness issues in categorical or continuous decision tasks would be an exciting and valuable direction for future work. However, as the first work addressing unfairness arising from human expertise heterogeneity in AI-assisted decision-making, we would like to re-clarify the reasons behind focusing on binary decision-making scenarios in this work:

1. Wide applicability of binary decision-making tasks in AI-assisted decision-making: Binary decision-making tasks are widely used in many real-world applications (e.g., medical diagnostics, loan approval, and hiring decisions). Using these tasks as the focus of our work provides substantial practical value.

2. Broadly Acknowledged Challenge in Multi-Class Scenarios: As clarified in our previous rebuttal, there is a broadly acknowledged challenge in multi-class settings due to the complexity of identifying appropriate utility function conditions. We have included an explicit section, Section 6, describing this challenge. Solid theoretical results depend on robust foundational models. Similar challenges have also been faced in many prior theoretical works [1–5], where binary-label tasks are widely adopted by these previous works [1-5] as a basis for deriving tight theoretical results (which is really non-trivial).

[1] Zhou, Zeyu, et al. "Counterfactual Fairness by Combining Factual and Counterfactual Predictions." NeurIPS. 2024.

[2] Donahue, Kate, and Jon Kleinberg. "Optimality and stability in federated learning: A game-theoretic approach." NeurIPS. 2021.

[3] Lou, Renze, et al. "Muffin: Curating multi-faceted instructions for improving instruction following." ICLR. 2023.

[4] Corvelo Benz, Nina, and Manuel Rodriguez. "Human-aligned calibration for ai-assisted decision making." NeurIPS. 2023.

[5] Li, Zhuoyan, Zhuoran Lu, and Ming Yin. "Decoding AI’s Nudge: A Unified Framework to Predict Human Behavior in AI-assisted Decision Making." AAAI. 2024.

Regrading the weakness 3 in formal setup

Thank you for your thoughtful comments. We address your concerns point by point below:

• Assumption on decision policy: we would like to clarity that the decision policy $P(T=1|h,a)=w\cdot h+(1-w)\cdot a$ in Figure 1 is provided as an intuitive and computationally simple example, not a core dependency of our theoretical framework. To avoid this misunderstanding, we have revised the description in the Lines 84-85.

  Our theoretical framework formally specifies the assumptions on decision policies in Section 2 (Monotone), which accommodates a broad range of decision policies when human decision-makers act rationally. Additionally, in our experiments, we model decision policies using a nonlinear activation function in an MLP, capturing realistic human behavior where the decision policy may be nonlinearly influenced by human confidence and AI confidence.

• Multi-Subgroup Settings: We discussed the challenge in the multi-subgroup setting you mentioned in Section 6.2, particularly the uncertainty in defining utility disparity across multiple subgroups. When the form of utility disparity is well-defined, our algorithm can be extended. We selected a specific form of utility disparity—using the standard deviation of utility distributions across subgroups as an example. For this experiment, we considered 4 subgroups defined by both "gender" and "education". The results, presented in the revised submission (Appendix A.8.2), demonstrate the effectiveness of group-level calibration in multiple subgroups.

• Conversion of Probabilities to Decisions: We clarify that our framework focuses on a binary decision-making scenario (Line 118). Raw probability values are converted to decisions using the rule: $T=1$ if $P(T=1)\ge 0.5$, and $T=0$ otherwise. This same conversion is applied in our experiments. Based on this conversion, we evaluate utility not by using false positives or false negatives but by using accuracy, $\mathbb{E} \left [\mathbf{1} (T=Y) \right ]$. Following your suggestion, we have added the description of conversion in Section 2 (Lines 129-131) of the revised submission.

Regrading the weakness 4 in technical results

Thank you for your suggestions. We address your concerns point by point below:

• Clear Narrative and Key Takeaways: Following your suggestion, we have revised the manuscript to provide a clearer narrative around the importance and implications of our technical results. Specifically:

  1. We reduced the emphasis on explaining the fairness limitations of existing calibration methods and added more discussion on the theoretical insights from inter-group alignment and the upper bound on utility disparity.
  2. Subsections have been added in Section 3 to clarify the connections between different parts and enhance readability.

• Novelty w.r.t. Corvelo Benz & Rodriguez (2023): As mentioned in our response to Weakness 2, our technical results serve for different purpose: fairness, while Corvelo Benz & Rodriguez (2023) focus on utility improvement. We highlights and addresses the fairness limitations of existing calibration methods an propose new fair calibration objective.

  Our experiments also demonstrate that considering only human alignment Corvelo Benz & Rodriguez (2023) may exacerbates fairness issue compared to uncalibrated case (Lines 449–452). This highlights a critical issue for real-world applications and AI for good. We believe our work takes an important step toward advancing fairness in AI-assisted decision-making, a previously unexplored direction.

Regrading the weakness 5 in experiments

We would like to clarify that confidence intervals are not shown in Figure 3 because it focuses on the calibrated confidence rather than final decision, which remains consistent across different random seeds as it does not involve the variability of final decisions modeled by an MLP (as in Figure 2). The purpose of Figure 3 is to illustrate how group-level multicalibration adjusts the AI confidence compared to traditional multicalibration. Specifically, it demonstrates that group-level multicalibration mitigates utility disparity caused by human-only decisions, either by reducing the disparity or creating an offsetting disparity with the opposite sign. To further validate this finding, we conducted additional experiments by changing the demographic feature from "gender" to "education." This adjustment causes the results in Figure 3 to change, but the finding remains consistent. These results are included in the revised submission (Appendix A.8.2).

Regrading the question 1

Please refer to the third point in the response to Weakness 3.

Regrading the question 2

Please refer to the response to Weakness 2.

Regrading the question 3

Please see the second point in the response to Weakness 3.

Regrading the question 4

Please see the first point in the response to Weakness 3.

We sincerely appreciate your valuable and thoughtful comments. Please find our point-by-point responses below:

Regrading weakness 1 in the presentation

Thank you for your helpful suggestions on improving our work presentation. In response, we have made the following revisions in the newest submission:

• Dense Presentation: We have revised Section 3 by adding subsections and restructuring the content to make it clearer and easier to follow.

• Figure 1: We understand your concerns about the difficulty of parsing Figure 1. To address this, we have ensured that the process depicted in the figure is thoroughly described in Lines 78–88.

• Language and Terminology: To sharpen the language and clarify terms, we made the following updates:

  1. Unified references to "human groups," "human populations," and "real-world users" under the consistent term "human decision-maker groups" for clarity. Thus, "AI confidence facilitates fair utility" refers to fair utility with respect to human decision-makers.
  2. Experimental configuration have been moved to Appendix A.7 for further detail.
  3. Following your suggestion, we replaced "inter-group alignment" and "human alignment" in Figure 1 with the more intuitive phrase "Calibrating AI confidence" to better describe the process represented by the red arrow (calibrating AI confidence to a specified value). Providing a detailed explanation of "inter-group alignment" and "human alignment" in Section 1 might introduce redundancy with the comprehensive discussion in Section 3. For this reason, we opted to use a simpler and more intuitive term to ensure clarity.

Regrading the weakness 2 in related work

Thank you for your insightful feedback and for pointing out the works by Rambachan et al. and De-Arteaga et al. These studies explore expert heterogeneity in different contexts and provide valuable evidence supporting the existence of heterogeneous experts in practice. They also underscore the importance of addressing the issues raised in our paper. We have now cited these works in the revised submission (Section 6).

We also appreciate your suggestion to articulate better the distinction between our work and that of Corvelo Benz & Rodriguez (2023). While their work presents inspiring results on improving utility, our work addresses a distinct and non-trivial challenge: identifying and addressing fairness issues in AI-assisted decision-making. This focus introduces significant conceptual and methodological differences. Specifically:

1. The first work focusing on fairness in this context:  Our work highlights a previously overlooked fairness issue in AI-assisted decision making, which is critical for real-world applications. As discussed in Lines 89–96, ensuring fair utility is essential for advancing AI for social good and improving societal welfare. Neglecting fair utility can, on one hand, undermine human willingness to adopt AI assistance, and on the other hand, exacerbate utility disparities among decision-makers due to the Matthew Effect, ultimately harming societal welfare. These factors critically influence the feasibility of AI-assisted decision-making systems in practice, which are unfortunately ignored by previous work.
      
2. A novel calibration objective: We propose a new fairness-driven calibration objective, inter-group alignment, which fundamentally differentiates our theoretical framework and methods from prior work.    
3. A tight theoretical framework of fairness issue in AI-assisted decision making: We provide the first in-depth analysis of the fairness limitations in existing confidence calibration methods. We derive the tight upper bound on utility disparity (which is really non-trivial), justifying the need of   multicalibration in group-level—a focus absent in previous studies.

We have revised the submission to clarify these distinctions in Section 6 more explicitly.

We thank the reviewer for taking the time and effort to review our paper. Please find below the point-to-point responses to the your comments.

Regarding weakness 1 and question 1 about "Multi-calibration implies the two fairness conditions in the paper. Are they equivalent?"

Thank you for your thoughtful comment. We want to clarify the relationship between the fairness conditions and group-level multi-calibration. The two fairness conditions, inter-group alignment, and human alignment serve as the calibration objectives, providing the theoretical foundation for fairness. Group-level multi-calibration, on the other hand, acts as a practical and feasible pathway to achieving these objectives.

These fairness conditions are not equivalent to group-level multi-calibration but rather justify its effectiveness in mitigating unfairness in AI-assisted decision-making systems.

Regrading weakness 2 about "generalization to general decision problems if the confidence is interpreted as a probabilistic prediction of the random variable"

Thank you for your insightful comment. We would like to clarify a misunderstanding: both the human confidence $h$ and AI confidence $a$ in our work are continuous values in the range $[0,1]$ and are random variables interpreted as probabilistic predictions. The binary variable is the final decision T\in \left \\{ 0,1 \right \\} , based on $h$ and $a$. This corresponds to the "action" or "choice" as described in the work of Ziyang Guo et al. (2024). We agree that extending the analysis to utility-fairness issues in categorical or continuous decision tasks would be an exciting and valuable direction for future work. As discussed in Section 6.2 of our paper, generalizing from binary decision problems to more general decision spaces is challenging, primarily due to the complexity of identifying appropriate utility function conditions in such settings.

Regrading the weakness 3 about "Typo"

Thank you for pointing out this typo. We have corrected this in the revised submission.

Thank you for your constructive comment.

We would like to  clarify the condition when our fairness notion is not equivalent to multicalibration: Applying multicalibration in the AI-assisted decision-making context can provide for each human confidence $h\in \mathcal{H}$, the AI confidence $a$ is calibrated to satisfy $\mathbb{E} [P(T=1|h,a)-P(Y=1|h,a)]\le \alpha$ [1].

However, only under a strong assumption, $P(Y=1|a,h)=P(Y=1|a,h,S=1)=P(Y=1|a,h,S=0)$, $\mathbb{E} [P(T=1|h,a)-P(Y=1|h,a)]\le \alpha$ also leads to $\mathbb{E} [P(T=1|h,a,S=1)-P(Y=1|h,a,S=1)]\le \alpha$ and $\mathbb{E} [P(T=1|h,a,S=0)-P(Y=1|h,a,S=0)]\le \alpha$,  and thus becomes equivalent to our fairness notion.

The issue is that, in practice, the assumption often does not hold. This issue may not arise in predictors scenarios;  it stems from human behavior, which is unique to AI-assisted decision-making context. For example, specific human subgroups may exhibit overconfidence in likelihood estimation, leading to  $P(Y=1|a,h)\not = P(Y=1|a,h,S=1)\not =P(Y=1|a,h,S=0)$.  Our fairness notion explicitly accounts for such a scenario.

[1] Hébert-Johnson, Ursula, et al. "Multicalibration: Calibration for the (computationally-identifiable) masses." International Conference on Machine Learning. PMLR, 2018.

I totally understand that you care about utility fairness. Blackwell Informativeness Theorem essentially says multicalibration (calibration conditioned on group identity, and, expertise in your case) implies improved utility for all groups. To me, by Blackwell's theorem, achieving the fairness notion for all decision tasks with binary states and not necessarily binary decisions seems an equivalent condition to multi-calibration (conditioned on group identify and not the one in Benz and Rodriguez), which implies your special case of binary decision. The definitions here are not interesting unless the authors can clarify the statement is not true.

We would like to clarify that multicalibration is typically used as an algorithmic fairness condition to ensure fair predictions across predicted groups, such as ensuring that disease diagnosis results are fair for patients in different racial groups. This notion of fairness is fundamentally different from the fairness we address in this work. Our focus is on the fairness of utility derived by human decision-makers from AI assistance. For example, we focus on how different groups of doctors can achieve fair diagnostic accuracy when assisted by an AI system, rather than addressing fairness for the patient groups being diagnosed. Our work aligns more closely with the concept of "allocation fairness" in collaborative settings, which is distinct from algorithmic fairness. Regarding our fairness definition, it is specifically designed to tackle the unique challenges of ensuring fair utility for human decision-makers in AI-assisted setting, an area that remains critical and underexplored.

Furthermore, under the fairness definition proposed in our work—focused on utility fairness for decision-makers—extensive experimental results (Figures 2, 5, and 7) demonstrate that directly applying multicalibration does not guarantee fairness and may even exacerbate fairness issues. This is an important point to verify that our definition of fairness goes beyond the fairness provided by multicalibration.

Thank you for your feedback and for highlighting the [1].

Regarding the comment about “I think this assumption needs to be discussed.”

In Section 2 (Expertise Disparity (ED) and Utility Disparity (UD)), we provide the background of the research problem, focusing on human decision-makers with heterogeneous expertise. This section captures the assumption discussed in our rebuttal, making our work not equivalent to multicalibration.

Following your suggestion, we have added our above discussion in Section 2 to explicitly highlight the unique aspects of our setting. Please refer to the updated submission for these additions.

Regarding the comment about “The following paper shows a similar observation.”

We would also like to elaborate on the differences between our work and [1]. Beyond the distinct application contexts, there are fundamental differences in problem settings:

• [1]: The problem setup assumes two predictors, each trained on data from different groups.

• Ours: We assume a single AI predictor trained on a dataset containing data from all groups.

Given this difference, [1] makes important contributions by investigating the compatibility between calibration and Equalized Odds (false-positive or false-negative) under their specific setting. In our work (Lines 107–109), equal utility (equal accuracy) involves simultaneously constrains human-alignment and inter-group alignment below a pre-specified threshold. Our theoretical results show that group-level multicalibration provides a practical way to compatibly achieve both human-alignment and inter-group alignment.

These discussions are now summarized in Related Work (Section 6) of the revised submission.

[1] Pleiss, G., Raghavan, M., Wu, F., Kleinberg, J., & Weinberger, K. Q. (2017). On fairness and calibration. Advances in neural information processing systems, 30.

W2 (Clarifying the presentation)

Thank you for your valuable suggestions regarding clarity and presentation. Following your comment, we have made the following revisions to improve the paper's readability and address the specific concerns:

• Added subsections in Section 3 to make the theoretical content clearer and easier to follow.
• Provided additional explanations for Definition 3.5 in Section 3 to help clarify its role and importance.
• Expanded the explanations for Theorem 3.6 in Section 3, including an intuition behind the upper bound to enhance understanding.
• Defined key mathematical terms in Section 2---Human-AI interactive model in AI-assisted decision-making.

Please review our revised submission.

W3 (Clarifying the typos)

Thank you for your comment. We want to clarify the misunderstanding regarding Lines 130–131 (previously referenced as Lines 126–127). Here, we refer to the decision variable $T$, which is binary, taking values of either 0 or 1, rather than the policy $\pi$. Additionally, thank you for pointing out the typo in Line 189 (previously referenced as Line 192). We have corrected this to replace "calibration" with "calibrated."

W4 (Adding explanations in the code)

We have included additional explanations within the code to enhance its readability. For further details, please refer to the anonymous link in the paper.

Q1 (Extending to multi-subgroups)

Thank you for your question. We discussed the challenge in the multi-subgroup setting you mentioned in Section 6.2, particularly the uncertainty in defining utility disparity across multiple subgroups. When the form of utility disparity is well-defined, our algorithm can be extended. We selected a specific form of utility disparity—using the standard deviation of utility distributions across subgroups as an example. For this experiment, we considered four subgroups defined by "gender" and "education". The results, presented in the revised submission (Appendix A.8.2), demonstrate the effectiveness of group-level calibration in multiple subgroups.

We thank the reviewer for finding our work essential and timely and for finding the results interesting. We also appreciate the detailed comments posed by the reviewer. Please find below the point-to-point responses to your comments.

Weaknesses 1 (Clarifying the novelty)

Regarding the comment: “The authors propose to multicalibrate the AI confidence separately for each subgroup of users; if my understanding is correct, this could already be done by previous work just by considering the subgroup information as a feature. This is done in standard multicalibration in tabular data problems.”

We appreciate the reviewer’s comment and would like to clarify a potential misunderstanding. Prior works such as multicalibration do not incorporate subgroup information into the calibration objective, overlooking the critical necessity of considering subgroup-specific nuances to achieve fairness.

Our findings reveal that applying multicalibration alone cannot guarantee fairness (Theorem 3.4). In fact, as demonstrated in Section 5.2, it can sometimes exacerbate fairness issues due to the lack of explicit subgroup handling.

To address these limitations, one of our contributions is to introduce group-level multi calibration. While this approach is conceptually straightforward and builds upon multicalibration, we demonstrate theoretically and empirically why multicalibration in group-level is not just feasible but necessary for effectively mitigating unfairness.

Regarding the comment: "The main novelty in this work is that the authors show that the AI confidence function will not be calibrated when decision policies vary across groups."

We respectfully disagree with this point. While "the AI confidence function may not be calibrated when decision policies vary across groups" is a new finding of our work, developing a comprehensive theoretical foundation for identifying and addressing fairness issues represents a more significant and non-trivial contribution. We would like to clarify the detailed novelty distinct our work from previous include:

1. The first work focusing on fairness in this context:  Our work highlights a previously overlooked fairness issue in AI-assisted decision making, which is critical for real-world applications. As discussed in Lines 89–96, ensuring fair utility is essential for advancing AI for social good and improving societal welfare. Neglecting fair utility can, on one hand, undermine human willingness to adopt AI assistance, and on the other hand, exacerbate utility disparities among decision-makers due to the Matthew Effect, ultimately harming societal welfare. These factors critically influence the feasibility of AI-assisted decision-making systems in practice, which are unfortunately ignored by previous work.
      
2. A novel calibration objective: We propose a new fairness-driven calibration objective, inter-group alignment, which fundamentally differentiates our theoretical framework and methods from prior work.    
3. A tight theoretical framework of fairness issue in AI-assisted decision making: We provide the first in-depth analysis of the fairness limitations in existing confidence calibration methods. We derive the tight upper bound on utility disparity (which is really non-trivial), justifying the need of   multicalibration in group-level—a focus absent in previous studies.

We have revised the submission to clarify the distinctions more explicitly in Section 6.

Regarding the comment, "In addition, the paper is very similar to the recent work of Benz and Rodriguez (2023). Algorithm, experiments, etc look similar."

As outlined above, one of our key innovations is developing a comprehensive theoretical foundation that justifies the necessity of multicalibration at the group level. Based on this, we propose group-level multicalibration. While conceptually straightforward and building upon the framework of multicalibration (Hebert-Johnson et al., 2018), our approach is specifically designed to provide a practical pathway to achieving the fair calibration objectives outlined in Section 3.

In contrast, Benz and Rodriguez (2023) utilized the original multicalibration method to achieve a utility-optimal calibration objective. Apparently, our work pursues distinct calibration objectives. We explicitly highlight the distinctions in the revised submission (Lines 299-304).

Regarding the experiments, we refer to Benz and Rodriguez (2023) as a baseline and follow their experimental setup (as explicitly cited in Lines 375,406,1220). While Benz and Rodriguez (2023) primarily evaluate aggregate utility, we focus is on utility-fairness across subgroups rather than aggregate utility alone. The experimental results from their multicalibration method serve as our baseline, and our results clearly demonstrate significant advantages in fairness over multicalibration.