Conformal Prediction Sets Can Cause Disparate Impact

Thank you for your effort in reviewing our paper. We appreciate that you found our experiments strong and diverse, that the paper was well-written, and that you were interested in how our experiments run counter to the prevailing wisdom. While you didn’t raise any significant concerns, we welcome your suggestions for improvement. We will make some comments on the the related papers you mentioned, and will incorporate them into our related works section.

1 Bondi et al., (2022) is another study involving humans but where a model can defer the final decision to a human. Hence, humans were only engaged on a small fraction of the test dataset, whereas we consider humans as the main decision makers for every example. You noted that Bondi et al. saw human accuracy increase when the deferral information was shared, but not when the model prediction was shared. This is perhaps unsurprising, since the only examples tested were ones where the model deferred, and these are exactly the cases where the model is most likely to be incorrect. Hence, we should not extrapolate such findings to the less constrained setting we used, where any example in the test set is given to a human, including cases where the model is highly confident and likely to help. The point you are trying to make remains valid though; the format of information provided to the user is likely to have a large effect on how helpful it is, but the overall setting can matter just as much. These are both aspects deserving of future research.

2 We noted De Toni et al. (2024) on page 1 as a prior work involving prediction sets and human decisions. De Toni et al. consider a constrained setting where experts (synthetic or human) must choose a class from the prediction set. This upper bounds the possible accuracy of experts to the coverage level of the sets. In our experiments, participants could choose any class and were free to use the prediction set information as they like. Both settings can be useful in different contexts. However, it does limit the transferability of conclusions between the works.

We agree with your comments that conformal prediction is much less useful in binary settings since the range of possible sets lacks the diversity needed to express uncertainty. Essentially, uncertainty becomes a binary notion as well - the prediction set is either a singleton (confident), or one of $\\{0,1\\}$ and $\emptyset$ (not confident). In our reading of the literature, CP is rarely studied in the binary case because of its trivial nature. While we did not present a case study for $m=2$, our three examples with distinct values of $m>=8$ are more closely aligned with standard applications of CP, i.e. to multiclass problems.

3 This is a helpful comment to point out, and Ruggieri et al. (2023) give valuable guidance. We agree that group-conditional modifications, such as one would perform to generate more singleton sets, could be viewed as conditional corrections susceptible to the Yule Effect. Perhaps Mondrian CP as a whole could be viewed in this light, as it generates different conformal thresholds $\hat q$ per group which are meant to correct any groupwise miscoverage. We have not conclusively proven that equalizing the frequency of singletons leads to more fair outcomes. Across our three case studies, singleton frequency difference was correlated to disparate impact (Figure 4), but this is the extent of what we claim. Moreover, we did not propose methodologies that would equalize singleton frequency and then evaluate their fairness outcomes, but for example this could be done by setting the $k_\text{reg}=1$ in the RAPS algorithm (see Equation 12). Devising methods to equalize singleton frequency, and test them for fairness while avoiding adverse effects like Yule’s should be considered in future research.

Importantly, none of the three papers above consider the fairness aspects of their human-AI decision systems, which is where our work breaks new ground.

Q1

I think the definition of conformal and non-conformal set predictors can coincide... Am I correct?

In short, yes. The conformal and non-conformal methods we discussed are not too different in spirit, both having a calibration step using a quantile, and adding classes to the prediction set based on that quantile. The main difference is that for conformal methods, the quantile is based on the desired coverage rate $1-\alpha$, whereas for Average-$k$ the quantile is based on the desired set size $k$. We could generalize the Average-$k$ method to use an arbitrary score function $s$ like CP does. Then, if the desired coverage and set size were carefully tuned such that the quantiles $\tfrac{\lceil{(n+1)(1-\alpha)}\rceil}{n}$ and $1-k/m$ were equal, the methods could coincide (see Sections 2.2 and 2.3 for notation). In practice, the score functions we used were quite different, with the Average-$k$ version (Algorithm 1) being much simpler than either RAPS or SAPS (Appendix A.2).

We are pleased to see that our work sparked your curiosity, and that you found it easy to read and extremely valuable research on fairness with well supported claims. We are also happy to address the few concerns you had.

W1: Thank you for pointing out that our abstract could be made more accessible to a wider audience. We will add additional explanation so that it assumes less familiarity with conformal prediction and fairness.

Q1

I am curious about the relationship between the Fairness in Conformal predictions and Fairness in Classifiers. Let's imagine a 3rd experiment, where the base classifier is fair and the conformal prediction setup is applied without any fairness constraint. What would the results look like?

This is an interesting question which we didn’t directly address in the paper. To be clear, we are claiming that conformal prediction can cause disparate impact, not that it always causes disparate impact. We believe that the scenario you outline is one where disparate impact would not arise.

For example, Figure 2 illustrates the mechanism of unfairness with CP. Notice that we are assuming the base classifier has some bias already (the model accuracy is higher on group $e$ than group $h$). Marginal CP tends to propagate this bias to human accuracy, and Conditional CP can amplify the bias further. However, if the base classifier is fair, and its behaviour does not have dependence on the group information, then there is no reason to believe that CP would generate unfairness. Model accuracy would be the same across groups, as would the coverage level and set size. Since there would be no difference in the properties of the prediction sets, human performance improvement should also be independent of group.

Thank you for your time and effort spent reviewing our work. We were delighted to see that you found it “novel and interesting”, “well written”, and that it could “spark new work on fairness in conformal prediction”. Like you, we found the experimental results in Figure 5 surprising, since they run counter to the prevailing wisdom in the field around fairness for set-predictors. We will gladly address your concerns below.

W1

… I would argue that if a CP treatment improves accuracy on each sub-group (even if at a different amount) then this isn't necessarily a bad thing, for example the minimax fairness would be improved. In this sense, I actually found figure 5 the most interesting observation, as it indicates how Mondrian CP can actually leave the worst-case groups, off worse. In general I would suggest more discussion on the trade-offs of the specific fairness definitions the authors use.

Fairness is a highly contextual notion; the correct fairness standard to apply depends entirely on the circumstances. We can certainly imagine some scenarios where a method would be beneficial if it improves accuracy on all groups, even if the amounts of improvement are not equal. For instance in a setting like medical treatment, it may be desirable to provide the most effective treatment on an individual basis. As long as no groups are receiving worse outcomes from a new type of treatment, we prefer to give every individual the best care possible. In this case, the disparate impact measure we applied would not be as relevant.

Just as well, there are certainly scenarios where unequal improvements would lead to harms for some groups. Consider a realistic setting where resources are constrained, like mortgage lending. A bank has a fixed, finite amount of capital it can distribute as loans, and receiving a loan is a beneficial outcome for individuals. Since capital is constrained, giving a loan to one individual at the margin means not giving a loan to someone else. These people may belong to different groups, like high and low income. Now, the bank uses a model to generate prediction sets which are given to a human underwriter who then makes a decision whether to issue a loan. If the sets increase underwriter accuracy for all groups, but by different amounts, one group will benefit at the expense of another. If accuracy increases more for high-income earners than low-income, high-income earners may receive proportionally more loans at the margin. Even though accuracy increased for the low-income group, they received fewer loans and experienced worse outcomes. This is an example where fairness corresponds to minimizing disparate impact. As you pointed out, our results also showed one case where equalizing coverage of prediction sets actually reduced human accuracy compared to the control on some groups. This is a more serious problem, where it is abundantly clear that some groups are worse off when Mondrian CP is used.

Given that fairness is such a nuanced topic, we do not expect a single definition to cover all cases. However, even in the title we emphasized that our perspective is about disparate impact. In Sections 1 and 2.4 especially, we were clear about the settings we consider, and the fairness definitions we focused on. We will improve this discussion by talking more about the nuances of fairness, and trade-offs of using a specific measure like disparate impact.

Thank you for the interesting answers to my questions, I maintain my score.

We would first like to thank the reviewer for their time spent reviewing our paper and their helpful feedback. We appreciate that you found our work provides an “important perspective” on conformal prediction via human study, and that you agree with our conclusion that equalized coverage has limitations. We understood your main concerns to be around the significance of our hypotheses, the standard of fairness we used, and some aspects of the experimental design. We will address all these aspects below.

W1:

The main weakness of this work to me is that its hypotheses are a priori unsurprising. If some subgroup A requires larger set sizes for the same coverage level, it is tautologically true that there is higher uncertainty for subgroup A and that any class in the prediction set is less likely to be the ground-truth label. Even if a person relied 100% on the prediction set and simply picked uniformly at random among the predictions, one would expect performance to be worse on groups with bigger set sizes.

While we do not disagree with this hypothetical, the example is overly reductive and misses a crucial point of our work. In reality, people using prediction sets as a decision aid do not pick an answer from the set randomly - the hypothetical argument could be a distraction. For example, we have found in practice that decision makers sometimes select a label that was not in the prediction set at all (Section 6.2; Adoption Rate). The way that humans interact with and use prediction sets is complex and we do not yet have a full understanding of it. The works we reviewed in Section 1 have started to investigate these interactions. Given that, without controlled experimentation we cannot say how human performance will look on groups with larger or smaller set sizes.

One of the main points we investigated and provided evidence for was the very notion that human performance will be higher when sets are smaller. While this would obviously be true for the random selection in your example, it might not be true for human decision makers in reality. One possibility is that human performance is more influenced by the coverage of sets than their size, i.e. it is more helpful if the prediction set often contains the correct answer, regardless of whether several incorrect labels are also in the set. If this were the case, then equalizing coverage would lead to more fair outcomes. It is not obvious a priori which factors of a set-predictor will have the biggest influence on users. In order to make claims one way or the other, we need scientific experimentation. This is what we investigated in Section 6.2 and Figure 4. We analyzed four factors which could potentially influence the fairness of outcomes and found that while set size is strongly correlated with performance, coverage is not. While you may have expected this outcome based on your correct intuition, you also have the benefit of hindsight. Other explanations were plausible before our experiments provided the hard data that now supports your intuition.

Consider a practitioner who is new to the field of conformal prediction. The prevailing wisdom in the field since 2020 has been that equalizing coverage is a fair way to produce prediction sets (i.e. since the seminal work by Romano, Foygel Barber, Sabatti, Candès 2020). This has been repeated and reinforced in papers since, including recently in 2024 (see discussion starting L183). There is a real possibility that well-meaning practitioners see this research and deploy it in practice without the same nuanced understanding you have developed, but this would likely lead to increased unfairness as in our case studies. Our work serves as a public warning to consider the downstream fairness impacts of common design choices in conformal prediction.

Thanks for the engagement with the review here; my qualms with the work remain after rebuttal, and I will maintain my score. Of course it looks like this will largely be a symbolic action since this paper will probably get in the conference based on the other scores.

While I understand there’s no way to re-do the research question and experimental setup from scratch, I hope that this feedback will be a useful signal for how some readers might receive the work given the current exposition. My strongest piece of actionable feedback at this point is to really walk back some of the more dramatic claims about increasing unfairness, and instead being more explicit about the nuances of the actual results. I appreciate that the authors repeatedly mention that ‘fairness is complex’ throughout the rebuttal, and some of the examples given in the rebuttal are thoughtful. But in my reading, the way that the current submission is actually written reflects a pretty reductive “numbers not equal == always really bad” perspective that actually obscures the real takeaway.

I think that this work has potential to be quite impactful in shaping the high-level conversation (and, to the rebuttal’s point, a nonexpert’s perception of what is true) about conformal x fairness. Given that, I would really encourage the authors to do some revisions on the writing level, especially in the intro/abstract, to avoid misleading readers.

(Also I still don’t understand Figure 1…)

Thank you for your quick feedback and your kind words that the work could be impactful for the discussion of fairness in conformal prediction! It seems like we have come to an agreement on some points, but you still have qualms about the portrayal of some results. For instance we agree that: Fairness is a complex notion, and different situations require different fairness standards and measurements. Our main result that conditional conformal prediction actually can cause a certain type of unfairness (disparate impact) is surprising based on the prevailing wisdom in the literature.

However, you feel that some of our claims lacked nuance, while noting your interpretation that our discussion seemed reductive and summarized it as “numbers not equal == always really bad”. While we do not agree with this description of our work, we appreciate your opinion and acknowledge that you are trying to help improve the work. Regardless of the eventual outcome of accept/reject, we hope to engage further to understand your perspective and modify the paper in a way that more clearly communicates the findings.

You first recommended that we “walk back some of the more dramatic claims about increasing unfairness” in the abstract and introduction. Based on your initial review, we interpret your position as being about how we applied only disparate impact as the standard of fairness to our results, as compared to other any other form of unfairness such as your setting where increasing accuracy on different groups by different amounts should be viewed as beneficial (as long as no group sees accuracy decrease). Your concern is that it is overly reductive to only consider one type of unfairness, while boiling down judgements on fairness to numerical comparisons. However, a singular focus on disparate impact is very common in the fairness literature (e.g. references [A-E]) because it is widely applied in legal and regulatory systems that judge fairness in the real world. For example, in the United States the primary regulatory body for the banking industry (the OCC) uses disparate impact as its main criteria for determining if a bank’s policies and practices are fair [F]. Our work is perhaps most similar in approach to [B], which takes an interesting method from machine learning (differentially private SGD, which aims to improve privacy), and shows via case studies that it can cause disparate impact. [B] has had enormous impact on the field of privacy in machine learning by raising these concerns, but notably only evaluated fairness in terms of disparate impact, and did so numerically with a definition akin to our Equation 7. Hence, following all these works [A-E], our analysis was framed around evaluating disparate impact, and under this definition it simply is the case that having different values of $\mathbf{OR}_{t,a}$ for different groups $a$ is undesirable.

Still, we are trying to take your feedback to heart, so we will revise the abstract and introduction to refer more specifically to “disparate impact” than “fairness” which is a more general concept. For instance in the abstract we will change:

• “... prediction sets can increase the unfairness of their decisions.” -> “... prediction sets can lead to disparate impact.”
• “... Equalized Coverage actually increases unfairness …” -> “... Equalized Coverage actually increases disparate impact …”
• “... which empirically leads to more fair outcomes.” -> “... which empirically leads to lower disparate impact.”

[A] Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data 5 2, 2017.
[B] Bagdasaryan et al. Differential Privacy Has Disparate Impact on Model Accuracy. NeurIPS 2019.
[C] Feldman et al. Certifying and removing disparate impact. SIGKDD 2015
[D] Zafar et al. Fairness Constraints: Mechanisms for Fair Classification. AISTATS 2017
[E] Zemel et al. Learning Fair Representations. ICML 2013
[F] Office of the Comptroller of the Currency. Fair Lending (link)