Bridging Fairness and Efficiency in Conformal Inference: A Surrogate-Assisted Group-Clustered Approach

Thanks for your careful review. We provide responses to your questions:

Claims And Evidence

1. Fair coverage is crucial in socially consequential domains, where prediction intervals can help guide decision-making related to access to resources, opportunities, or fair treatment. For example, if a mortgage lender’s prediction intervals systematically under-cover minority borrowers, they may face inflated interest rates or higher denial rates. In healthcare, if limited supplies of critical antibiotics demand triaging decisions, under-coverage for certain demographics could result in denying them lifesaving medications. We will include these examples in the paper.

2. If a given prediction model achieves fair coverage in terms of conformalized prediction sets, it does not necessarily lead to fairness in point predictions. If the length of the prediction sets are the same and the prediction sets are symmetric about the point predictions, fair coverage in prediction sets could imply fairness in point predictions. Therefore, fair coverage in prediction sets could be considered as an extension to group fairness (e.g., demographic parity) in point predictions, as coverage in prediction sets additionally accounts for the uncertainty of point predictions (as our primary motivation). Even if a given model does not achieve group fairness in terms of the point predictions, it can achieve fair coverage in the prediction sets, and one could conclude that its prediction is fair with $(1-\alpha)$ confidence given a user-specified miscoverage level $\alpha$. This point will be emphasized in our paper.

Methods And Evaluation Criteria

1. The baseline WCQR from Lei & Candes (2021) was designed for counterfactuals (or individual treatment effects), which essentially aims to account for covariate shifts between groups with observed outcomes and missing outcomes. Therefore, it is directly applicable in our setting to account for covariate shifts between source data (with outcomes observed) and the target data (with outcomes missing). Therefore, WCQR is an appropriate baseline comparator method for SAGCCI. We highlight the additional efficiency SAGCCI gains vs WCQR by estimating the thresholds via the EIF.

Experimental Designs Or Analyses

1. For the presented experiments with the number of groups $M=3$ and the number of clusters $K=2$, SAGCCI and Surro+Group have similar performances when $N\geq 3000$. When we increase $M$ to $10$ and $20$, while fixing $K=5$, Surro+Group would have much worse performance even if $N\geq 5000$, since each group would have fewer samples to fit the non-conformity scores, which leads to deteriorated performance in both AvgSize and CovGap. The new results are presented here.

2. All the considered conformal inference methods are implemented by the split conformal inference strategy -- therefore, their running times are similar. For example, for $N=1000$, the running time of these methods are all around 1 second when 25% of the data has been used for calibration and to construct the conformalized prediction sets. This point will be made clear in the paper.

Essential References Not Discussed

1. Liu et. al., (2022) and Vadlamani et. al. (2025) focus on learning a fair quantile function (termed ConformalFairQuantl) to construct the non-conformity score to deliver reliable fair prediction sets. However, our method is more general and applicable to any non-conformity score, and is not restricted to these conformalized quantile scores. Their methods do not account for the distributional shifts in covariates, and is therefore not directly suitable to our setting, where the outcomes from the target data are missing; see the updated results here, where the performance of ConformalFairQuantl is not satisfactory under our problem setup.

Other Comments Or Suggestions & Questions For Authors

1. We have fixed the typos and remove the subscript in $V_f$ to avoid any confusion. Now, we have a new data set $V= (W,Z,Y)\sim P_{D_0}$. We use the quantile random forest to fit the quantiles $q_{\alpha/2}$ and $q_{1-\alpha/2}$ and the non-conformity scores are constructed by the conformalized quantile residual by $R(W, Y)= \max(q_{\alpha/2}(W)-Y, Y-q_{1-\alpha/2}(W))$.

---

References

1. Lei, L. and Candes, E. J. Conformal inference of counterfactuals and individual treatment effects. Journal of the Royal Statistical Society Series B: Statistical Methodology, 83:911–938, 2021.

2. Liu, Meichen, et al. Conformalized fairness via quantile regression. Advances in Neural Information Processing Systems 35 (2022): 11561-11572.

3. Vadlamani, Aditya T., et al. "A Generic Framework for Conformal Fairness." (2025) The Thirteenth International Conference on Learning Representations.

Thanks for the comprehensive review. We now provide detailed responses to your concerns.

Theoretical Claims

1. $V_{eff}^r$ and $V_{eff}$ are referred to as the variances of the estimators. Therefore, the more predictive the surrogates are, the smaller $V_{eff}$ will be and the larger (more positive) the quantity $V_{eff}^r - V_{eff}$ will be.

2. The quantile functions are trained over the source data $D_1$ and can be used on the target data $D_0$ under Assumption 4.2 (b) $D \perp Y \mid S, X$.

3. Thanks for catching this; we have updated Assumption 4.1 to be $c_0 < P(D=1\mid x)$ instead.

Experimental Designs Or Analyses

1. The improved efficiency of SAGCCI is achieved by approximately guaranteeing group-conditional coverage, a tradeoff between efficiency and fairness. The intention is to produce prediction sets that have good group-conditional coverage AND are not too large to be useful. We will remove overstatements of the superiority of our method and add: "the choice of an appropriate conformal prediction method should be assessed based on the problem setting at hand and the objectives of the analysis."

2. For the presented experiments, SAGCCI and Surro+Group have similar performance when $N\geq 3000$. When we increase the number of groups $M$ to 10 and 20, while fixing $K=5$, Surro+Group performs much worse, even if $N\geq 5000$, since each group would have fewer samples to fit the non-conformity scores, which leads to deteriorated performance in both AvgSize and CovGap. The new results are presented here. This point will be made clear in the paper.

3. When $\sigma_S^2$ increases, surrogate information will be less representative across the groups, since each group tends to have more different surrogates in finite samples. Therefore, the Surro+Group will have larger AvgSize in finite samples since it is influenced more by the surrogates which have larger variability.

4. A more precise illustration is now presented: when $\sigma_S=3$, the proposed SAGCCI reduces the size of prediction sets by over 60% on average compared to Surro+Group at $N=1000$ but increases the CovGap by only 7%. Compared to Surro+Standard, the sizes of the prediction sets are similar, but SAGCCI reduces the CovGap by over 80% at $N=3000$.

5. Age-morbidity groups are of clinical relevance and the fairness evaluation is of real world concern -- we will update the paper to motivate the sensitive groups based on these subgroups rather than on race.

Essential References Not Discussed

1. We now provide more discussion of the proposed method and state-of-the-art methods. Existing fair conformalized predictions (Liu et. al., 2022, Vadlamani et. al. 2025) focus on learning a fair quantile function (termed ConformalFairQuantl) to construct the non-conformity score to deliver reliable fair prediction sets. However, our method is more general and applicable to any non-conformity score, and is not restricted to only conformalized quantile scores. Their methods do not account for the distributional shifts in covariates, and cannot be used in our setting, where outcomes from the target data are missing; see new results here, which show that the performance of ConformalFairQuantl is not satisfactory under our problem setup. The review paper and other fair conformalized predictions will be referenced in the paper.

Answer to Questions For Authors

1. We realized that we were not clear about the implementation of the proposed method. A full algorithm is now presented here. Most of the models (e.g., quantiles and clustering) are fitted on $I_1\cap D_1$, while the density ratio is fitted on $I_1$ with both source and target data.

2. Although the clustering idea is introduced in Ding et al, its application to fairness-related problems is new, as mentioned by Reviewers G4EX ("Relation to Broader Scientific Literature") and uHDb ("Claims And Evidence").

3. New results for a larger number of protected groups are presented here.

4. If one's sensitive information is missing, we could assume these subjects with missing sensitive information belong to one cluster and proceed with our approach.

5. The cluster number is now chosen automatically by the simple heuristic described in Appendix B. To select the optimal cluster number, we could also use human-in-the-loop strategies, such as the Elbow method and other information criterion (e.g., AIC, BIC). This point will be added to Appendix B.

6. Since $S$ is included in the generation of $Y$, higher variance in $S$ indicates that $S$ is more predictive of $Y$. Therefore, there is no upper limit on the maximum value of efficiency gain as long as $S$ is included in the data generation.

We thank the reviewer for the detailed comments and provide itemized responses to your questions.

Theoretical Claims & Questions For Authors

1. The asymptotic coverage in Theorem 4.6 should be interpreted as $P_{V\sim P_{D_0}}(y \in C(W;r_{\alpha}^k)\mid Z = z) \geq 1-\alpha - o(1)$ as $N\rightarrow \infty$ since the constants do not depend on the sample size $N$. Therefore, the $(\alpha, \delta)$-PAC guarantee can be approximately met with negligible errors; see the last paragraph of Section 3.1 in the paper by Yang et al. titled "Doubly Robust Calibration of Prediction Sets under Covariate Shift" published in JRSS-B. The constants are a result of the use of concentration inequalities. We will update our statements to be more precise in the revised paper.

2. Thanks for pointing out that the protected groups may not always be clusterable. We feel this question would fall into a more general concern as "clustering on non-clusterable groups". We have provided a simple heuristic in Appendix B to choose the number of clusters, inspired by Appendix B.4 in "Class-Conditional Conformal Prediction with Many Classes". We could further resort to human-in-the-loop strategies to avoid overly clustering on non-clusterable groups, such as the Elbow method and other information criterion (e.g., AIC, BIC) in a cross-validation setting. This point will be added to Appendix B of our paper.

Experimental Designs Or Analyses & Questions For Authors

1. The number of protected groups $M$ for the presented experiments is $3$ and the number of clusters is $K=2$ as expected. To achieve the tradeoff between efficiency and fairness, $K$ should be no larger than $M$ but larger than $1$ (i.e., $1<K\leq M)$.

2. The reason we only show a subset of these considered methods is to avoid presenting too much information in one figure and to avoid clutter. Figure 2 focuses on comparing SAGCCI with NoSurro+Cluster and WCQR (with surrogates) to show that we have leveraged surrogates in the most efficient way by leveraging the EIF. Figure 3 focuses on comparing SAGCCI with Surro+Group and Surro+Standard to showcase the advantages of using group-clustered conformal inference to achieve a good balance between efficiency and fairness.

3. For the presented experiments, SAGCCI and Surro+Group achieve similar performance when $N\geq 3000$. When we increase the number of groups $M$ to $10$ and $20$ while fixing $K=5$, Surro+Group has noticeably worse performance, even if $N\geq 5000$, when $M=20$ since each group would have fewer samples to fit the non-conformity scores, which leads to deteriorated performance in both AvgSize and CovGap. These new results are presented here. This point will be made clear in the revised paper.

Relation To Broader Scientific Literature

1. When the surrogate outcomes are not unavailable, the benefits of clustering can be shown by comparing the performance of NoSurro+Cluster with NoSurro+Standard and NoSurro+Group, with new results shown here, where $\sigma_S = 0$ indicates there are no surrogate effects. Full comparisons will be added to the Appendix for completeness.

Other Comments Or Suggestions

1. We agree that Figure 1 can be confusing to readers and therefore replace it with a detailed pseudo-code algorithm, which can be found here.

Thanks for the careful review and nice comments! We here provide detailed responses to your comments.

Experimental Designs Or Analyses & Other Comments Or Suggestions

1. The size of the groups was $M=3$. We have conducted additional experiments with a larger number of groups (e.g., $K=10, 20$), where similar conclusions can be drawn; the results are presented here. The considered group-conditional conformal algorithm (i.e., Surro+Group) is extended from Vovk (2012), and is similar to Gibbs et al (2023), where we further account for the covariate shifts across the groups. However, this method neglects the fairness condition and may have inflated CovGap, as seen in Figure 3. We will emphasize this point in the paper.

Other Strengths And Weaknesses & Relation To Broader Scientific Literature

1. Thanks for bringing up the question about overlapping protected groups! Our method is actually applicable to protected groups that overlap with a relatively minor extension. Let $G_{fair}$ represent the set of groups (potentially overlapping) across which we aim to impose fairness, which is essentially a function of the protected groups $Z$ (see Assumption 2.1 in "Fairness with Overlapping Groups"). Then our method could be used on this new set of groups $G_{fair}$. Furthermore, our method could incorporate other clustering mappings instead of k-means, such as Fuzzy k-means or overlapping k-means, which allows for overlapping clustering as well. Thank you for providing more papers on conformal inference for achieving group-conditional guarantees. We will check them and reference them in the revised paper.

2. Although the clustering idea is introduced in Ding et al, its application to fairness-related problems is new, as mentioned by Reviewers G4EX ("Relation to Broader Scientific Literature") and uHDb ("Claims And Evidence"). Furthermore, the derivation of the EIF with surrogates is an interesting theoretical result in its own right. We further quantify the efficiency gain of leveraging surrogates in Corollary 4.5 and establish the coverage guarantee in Theorem 4.6. See the JRSS-B paper "On the role of surrogates in the efficient estimation of treatment effects with limited outcome data" for related theory, which develops the EIF with surrogates to estimate average treatment effects. Our work extends this theory to the conformal prediction setting. This point will be mentioned in the paper.

Questions For Authors

1. We noticed that we were not clear in explaining the split conformal inference strategy in Section 4.4. We have provided a new pseudo-code to illustrate this algorithm. To your question specifically, these two datasets are combined and split into training and calibration folds as $I_1$ and $I_2$. The intersection of the training and source data (i.e., $I_1\cap D_1$) is used to fit the non-conformity scores (e.g., the quantile functions), and the full training data $I_1$ is used to fit the density ratio $\pi_D(X)$. The reason for the random splitting of the combined data is to ensure that the density ratio $\pi_D(X)$ remains exchangeable across folds $I_1$ and $I_2$. Next, these fitted models are used in the EIF evaluated on the calibration set $I_2$ to obtain the estimator for the cluster-specific threshold $r_{\alpha}^k$ over the target data. The full algorithm is provided here.

---

References

1. Vovk, V. Conditional validity of inductive conformal predictors. In Asian conference on machine learning, pp. 475–490. PMLR, 2012.

2. Gibbs, I., Cherian, J. J., and Cand `es, E. J. Conformal prediction with conditional guarantees. arXiv preprint arXiv:2305.12616, 2023.