Personalized Federated Conformal Prediction with Localization

We sincerely thank the reviewer for your thoughtful and encouraging assessment of our work. We are particularly grateful for your recognition of PFCP's theoretical and empirical advantages over existing methods, and the marginal guarantee of our methods under personalized federated setting, as well as your appreciation of our efforts to maintain clear presentation while tackling technically complex problems. Their insightful comments have helped us further improve the manuscript. Below we provide detailed responses to all points raised.

W1: We appreciate the reviewer's insightful observation regarding the relationship between GLCP and existing localized conformal prediction methods.

• We would like to clarify that GLCP was indeed designed to generalize and unify existing approaches.
• When using kernel conditional CDF estimation (NW estimator), GLCP reduces to Localized Conformal Prediction (LCP). With identity score mapping, GLCP becomes equivalent to Distributional Conformal Prediction (DCP).

We will revise and tone down our statement on GLCP accordingly in the revised version.

W2&W4: Thanks for the valuable feedback regarding reporting test-conditional miscoverage error of FedCP and FedCP-QQ, and including a dataset that is suitable for federated learning setup.

• In response, we have supplemented table below with new real data results comparing test-conditional miscoverage rates, including evaluations on: (1) the DermaMNIST classification dataset from MedMNIST that is suitable for federated learning setup, and (2) two additional baselines (CPhet [1] and CPlab [2]). ||Dataset|GLCP|PFCP|FedCP|FedCP-QQ|CPhet|CPlab| |-|-|-|-|-|-|-|-| |Marginal|BIO|0.902|0.903|0.993$\times$|0.970$\times$|0.981$\times$|0.978$\times$| ||BIKE|0.900|0.900|0.885$\times$|0.890$\times$|0.883$\times$|0.885$\times$| ||CRIME|0.900|0.896|0.866$\times$|0.852$\times$|0.864$\times$|0.863$\times$| ||STAR|0.898|0.899|0.897|0.910$\times$|0.899|0.897| ||CONCRETE|0.903|0.905|0.947$\times$|0.963$\times$|0.946$\times$|0.949$\times$| ||DERMA|0.895|0.895|0.830$\times$|0.819$\times$|0.849$\times$|0.867$\times$| |Miscoverage|BIO|0.0315|0.0199(36.8%)|0.0941|0.0753|0.0844|0.0822| ||BIKE|0.0234|0.0184(21.4%)|0.0678|0.0647|0.0690|0.0681| ||CRIME|0.0387|0.0268(30.7%)|0.0426|0.0495|0.0429|0.0439| ||STAR|0.0392|0.0243(38.0%)|0.0500|0.0492|0.0496|0.0495| ||CONCRETE|0.0366|0.0238(35.0%)|0.0582|0.0675|0.0580|0.0600| ||DERMA|0.0307|0.0264(14.0%)|0.0939|0.1004|0.0802|0.0701| |Size|BIO|0.514|0.503(2.1%)|0.999|0.709|0.827|0.805| ||BIKE|4.097|3.988(2.7%)|4.004|4.091|3.967|3.996| ||CRIME|5.183|4.496(13.3%)|3.918|3.772|3.886|3.880| ||STAR|48.50|43.02(11.3%)|42.80|45.30|43.08|42.85| ||CONCRETE|34.49|29.38(17.7%)|34.71|38.97|34.69|35.15| ||DERMA|2.409|2.327(3.4%)|1.233|1.199|1.331|1.426|

• Moreover, we would like to clarify that in our personalized federated learning setting with test data from the target agent distribution, ensuring valid marginal coverage is the fundamental requirement to be satisfied first, as it forms the basis for a reliable conformal prediction set. FedCP, FedCP-QQ, CPhet, and CPlab cannot guarantee this crucial marginal coverage property in our setting.

• Although FedCP, FedCP-QQ, CPhet, and CPlab cannot guarantee marginal coverage, they may occasionally achieve the desired coverage level in certain scenarios (e.g., the STAR dataset). Even in these cases, GLCP and PFCP outperform the other methods in test-conditional miscoverage error.

W3: We sincerely appreciate the detailed comment regarding this experimental detail.

• We would like to clarify that FedCP-QQ requires all agents' data to be i.i.d. to achieve marginal coverage, which fundamentally contradicts our PFCP setting where the target agent and source agents follow different data distributions and the test point is from the target agent distribution.
• In our experimental design, we artificially partition agents such that they follow distinct distributions to reflect the personalized federated learning scenario - while FedCP-QQ employs random sampling from the entire dataset for each agent in their experiments to reflect the i.i.d. case.
• Thus, under our experiment setup, the distributional heterogeneity between target and source agents inherently violates FedCP-QQ's i.i.d. assumption and consequently undermines its marginal coverage guarantees.

Q1:

• The sentence in line 200 was originally intended to describe the existence of two different approaches to estimating $F(s\mid x)$. However, the use of the word "valid" may have been somewhat misleading. We meant they are both consistent estimators of $F(s\mid x)$ under certain conditions theoretically.
• In the revised version, to avoid confusion, we will rephrase it as:

"both $\mathbb{E}\_{S\sim\widehat{F}}\mathbb{1}(S\leq s)$ and $\mathbb{E}\_{S\sim\widehat{F}\_{\rm mix}(s\mid x)}\\{\mathbb{1}(S\leq s)\widehat{r}(S\mid x)\\}$ can serve as estimators of $F(s\mid x)$".

Q2: We appreciate this insightful question on our theoretical results.

• In Lemma 2.6, the first term in the right-hand side of the inequality represents the global calibration error, which is typically small and scales with the target agent's sample size $n$. When $\widehat{F}$ is the empirical distribution estimator, this term approximately equals $\epsilon + \delta + 1/n$.
• The second term quantifies the estimation accuracy of the conditional distribution estimator, indicating that pointwise test-conditional coverage is governed by the local precision of the conditional distribution estimation.
• Theorem 2.8 further establishes that the aggregated conditional distribution estimation is controlled by both the individual conditional distribution estimation accuracies and the precision of the density ratio estimation. In practice, source data is often abundant, enabling accurate estimation of its conditional distribution, in which case the overall precision becomes primarily determined by the density ratio estimation error. When sufficient relevance exists between source and target domains, this density ratio estimation can be performed with higher precision.
• Even though it is unclear whether the bounds in Lemma 2.6 and Theorem 2.8 are sharp, they are sufficient to provide theoretical justification for PFCP's improved performance compared to GLCP, which is the main focus of our theoretical study.

Q3: Thank you for pointing out this typo. The integral should indeed be taken with respect to the score variable $s$ rather than the input variable $x$.

Q4:

• For synthetic data, the randomness is introduced through different random seeds when sampling from the given distributions.
• For real-world datasets, the stochasticity arises from different random seeds used to partition the data into training, calibration, and test sets.

Q5:

• For synthetic data, we generate 100 prediction sets $C_1(x_i),...,C_{100}(x_i)$ for each test point $x_i$, compute the corresponding coverage using its true distribution we get $p_1(x_i),...,p_{100}(x_i)$, and estimate the conditional coverage at $x_i$ by averaging these values. The test-conditional miscoverage is then obtained by averaging across all test points.
• For real-world data, we first partition the feature space into $k$ clusters using $k$-means. During each experimental repeat, we compute the empirical coverage within each cluster, then average these values across $100$ repeats to obtain the test-conditional coverage estimates. The final test-conditional miscoverage metric is calculated as a weighted average across all clusters, with weights proportional to the number of test samples in each cluster.

Minor1: We sincerely thank the reviewer for identifying this omission. Lemma 2.1 follows directly from the basic validity guarantees of conformal prediction theory. In the revised version, we will add this citation.

Minor2: Thank you for the suggestion to improve the clarity of our notation. In the revised manuscript, we will move the definition of the weight $\pi$ closer to its first usage in line 147.

We are more than happy to answer any further questions during the Discussion.

Reference

[1] Vincent Plassier, Nikita Kotelevskii, Aleksandr Rubashevskii, Fedor Noskov, Maksim Velikanov, Alexander Fishkov, Samuel Horvath, Martin Takac, Eric Moulines, and Maxim Panov. Efficient conformal prediction under data heterogeneity. In International Conference on Artificial Intelligence and Statistics, pages 4879–4887. PMLR, 2024.

[2] Vincent Plassier, Mehdi Makni, Aleksandr Rubashevskii, Eric Moulines, and Maxim Panov. Conformal prediction for federated uncertainty quantification under label shift. In International Conference on Machine Learning, pages 27907–27947. PMLR, 2023.

We thank the reviewer for the positive feedback regarding the paper's clarity, methodological presentation, and comprehensive literature review, as well as thoughtful and valuable suggestions. We address the key concerns below:

W1&Q2&W3: We respectfully clarify that our contribution is not limited to introducing a new score function and disagree with the claim that our contribution is mainly about constructing CDF estimators.

• First, we consider conformal prediction under the PFCP setting, where test data comes from the target agent, not the mixture of all source agents. Our framework not only guarantees marginal coverage on the target agent regardless of the source data distribution, but also improves test-conditional coverage for the target agent, which is an objective that existing federated conformal prediction methods have not addressed. This novel formulation, as noted by Reviewer kS3y, distinguishes our work from other FCP approaches.
• Second, the fundamental motivation behind PFCP is to simultaneously guarantee marginal coverage while enhancing test-conditional coverage on the target agent. The key innovation resides in effectively transferring information from source agents to improve the conditional coverage performance on the target agent. This strategic information transfer, achieved through our carefully designed framework, is novel as noted by Reviewer kS3y.
• Third, while the core technical contribution of our work lies in the construction of novel conformal scores, these scores are not simple function mappings. They are carefully designed fusion estimators that incorporate information from other source agents while strictly preserving the marginal coverage guarantee.
• Our theoretical results go beyond standard CP analyses. In particular, Lemma 2.6 establishes a crucial theoretical connection between the CDF estimator and the test-conditional coverage of conformal prediction, which serves as the fundamental basis for analyzing the improvement. Further, Theorem 2.8 provides a non-asymptotic bound demonstrating that PFCP improves conditional coverage over using only local (target-agent) data (GLCP), thus justifying the benefit of leveraging source information through our reweighting scheme. This theoretical analysis proves PFCP attains superiority over GLCP and prior works, as noted by Reviewer 2MYP.

W2&Q1: Thanks for the question regarding the difference in sample size dependence between our work and [2]. The distinction stems from fundamentally different problem settings and objectives. Our PFCP framework has several key differences compared with [2]:

• In PFCP setting, the test sample comes from the target agent, with distribution different from source agents, and we make minimal distributional assumptions: we only require the target agent's n samples to be i.i.d., without imposing any restrictive requirements between source and target agents, nor assuming known density ratios of any form. This is because we operate in a more challenging environment where source and target agents have different distributions (not limited to only label shift or covariate shift);
• [2] focuses specifically on label shift scenarios where $X\mid Y$ distributions are identical across domains. Their Theorems 2.1 and 2.2 show that with known density ratios of $Y$, weighted conformal prediction (WCP) can leverage all $N$ samples. In practice (as noted in their Theorem 2.4), when density ratios must be estimated, marginal coverage error includes an additional term $R$ from estimation error, and this breaks the exact marginal coverage guarantee in our PFCP setting.
• Our method provides strict finite-sample marginal coverage guarantees regardless of source agent data quality because we only rely on the target agent's $n$ samples for coverage calibration and source data only assists in improving conditional coverage. This is formalized in Theorem 2.8, which provides a non-asymptotic bound showing that our approach (PFCP) can improve conditional coverage compared to methods using only local data. The estimation of the density ratio $\hat{r}$, which is crucial for this improvement, depends on data from all source agents.

W4: Thank you for pointing this out. We have now included the size of the prediction sets in the Table below. Since the other baseline methods fail to consistently achieve the marginal coverage requirement, we omit detailed size comparisons for these approaches. Our results demonstrate that the density ratio estimation using calibrated neural networks (PFCP2) yields significantly more efficient prediction sets compared to the logistic regression approach (PFCP1) and GLCP.

Q3: Thanks for raising this important point.

• In our framework, privacy is maintained in the following ways: (i) no agent accesses other agents’ raw local data; (ii) information exchange only occurs during the estimation of $\widehat{F}_{\text{mix}}$ and the density ratio, both estimation optimized via the FedAvg algorithm; and (iii) the estimated CDF involves only conditional distribution of conformity scores which contains no invertible mapping to raw features or labels.
• We acknowledge that although FedAvg is widely used, it may still carry risks of privacy leakage. However, we can directly replace FedAvg with existing privacy-preserving federated optimization techniques, such as differentially private mechanisms [3] or secure aggregation protocols [4], to provide formal privacy guarantees.

We will add more discussion in the revised version and consider this a direction for future work.

Q4: We appreciate the reviewer’s insightful question. It touches upon an important aspect of our method’s performance.

• As stated in Theorem 2.8, our work provides a theoretical guarantee on the improvement of conditional coverage. However, we acknowledge that a formal theoretical result on the reduction of prediction set size is not currently provided.
• Instead, we substantiate this claim through extensive empirical evidence from both simulation studies and real-world datasets, where PFCP consistently outperforms GLCP in producing smaller prediction sets. Please refer to Table 1 in the original manuscript, Figure 3 and Figures 16--19 in the appendix, as well as Table below and Table above provided in our response to W4, for detailed results supporting this observation.

Q5:

• Since all the datasets used in our main experiments are regression tasks, all methods (including FedCP and FedCPQQ) adopt the standard residual score function, defined as $s(x,y) = |y - f(x)|$, where $f$ is the pre-trained regression model.
• For the classification task (DermaMNIST dataset) mentioned by Reviewer kS3y, we use the common classification score function $s(x,y) = 1 - [f(x)]_y$, where $f(x)$ returns the predicted class probabilities and $[f(x)]_y$ denotes the predicted probability of the true label $y$.

We are more than happy to answer any further questions during the Discussion.

Reference

[1] Charles Lu, et al. (2023) Federated conformal predictors for distributed uncertainty quantification.

[2] Vincent Plassier, et al. (2023) Conformal prediction for federated uncertainty quantification under label shift.

[3] McMahan, H. B., et al. (2017). Learning differentially private recurrent language models.

[4] Bonawitz, Keith, et al. (2017). Practical secure aggregation for privacy-preserving machine learning.

Thank you for your positive assessment recognizing both the novelty of the personalization problem in federated conformal prediction and the methodological innovation of our federated conditional distribution estimator for PFCP. We sincerely thank the reviewer for the constructive and thoughtful feedback. Below, we address each concern in detail and provide clarifications and additional numerical results.

W1&Q1&W2&Q2:

• First, we would like to clarify a typo in our citation in line 96. We mistakenly cited [3], and the correct references should be [2] (for label shift) and [4] (for covariate shift). We will correct this in the revised version.
• For the classification experiments (W1&Q1), we have now implemented the code for classification task using the score function $s(x,y)=1-[f(x)]_y$ as in [1]. We conduct additional experiments on DermaMNIST as a representative classification dataset (denoted as DERMA), and will include these new results in our revised manuscript.
• For the additional baselines (W2&Q2), we sincerely appreciate the reviewer's suggestion to include additional baselines. While [2] (CPlab) and [4] (CPhet) share conceptual similarities with our approach, we note they were originally designed for different problem settings. We have implemented both methods for fair comparison.
• The real data results are shown in Table below. The experimental results demonstrate that both GLCP and PFCP consistently achieve valid marginal coverage on the target agent, while the other baseline methods frequently fail to maintain this fundamental guarantee. Furthermore, PFCP exhibits significantly lower test-conditional miscoverage compared to GLCP, with both proposed methods substantially outperforming all remaining baselines.

W3: As stated in line 116 of our paper, GLCP relies solely on the target agent's data, meaning that it does not involve any communication or data sharing with other agents. As we mentioned in line 139, this reliance on limited target data can lead to unreliable or excessively large prediction sets. This observation motivates our proposed PFCP method, which leverages data from other agents to improve efficiency.

W4: We sincerely thank the reviewer for raising the important question regarding the coverage and prediction set size in the personalized setting.

• [1] consider a setting where the test data is drawn from a mixture of source agents’ distributions—implicitly assuming a certain relationship between the source and target distributions. In contrast, we are considering the scenario when the test data is from a specific (target) agent. As a result, the method of [1] does not guarantee marginal coverage under our setting, which is also confirmed empirically in Figure 1 and Table 1.
• The results in the above Table reveal an important trade-off: when baseline methods achieve marginal coverage exceeding the nominal level (0.9 in our experiments), their prediction sets become substantially larger than those produced by PFCP. Conversely, when these baselines fail to maintain the required coverage (falling below 0.9), their prediction sets shrink to unrealistically small sizes. However, such under-coverage scenarios render the size comparisons meaningless, as the resulting prediction sets lack the fundamental validity guarantee that forms the basis of conformal prediction.
• In our proposed PFCP framework, personalization is achieved by leveraging information from source agents through density-ratio weighting, tailored for a specific target agent. This allows for a more accurate estimation of $F(s\mid x)$ compared to GLCP, which relies solely on target agent data. Consequently, PFCP leads to more efficient prediction intervals (i.e., smaller set sizes), as shown in Table 1.

W5: We sincerely thank the reviewer for pointing out the lack of clarity in our explanation.

• The personalization in our method is primarily achieved through the calibration step (Step 4) is performed on the target agent's data to guarantee marginal coverage on the target distribution.
• The incorporation of source data is solely for improving the CDF estimation accuracy. The density ratio estimation emerges as a necessary component precisely because we introduce source data to enhance the conditional distribution estimation while maintaining the personalized calibration.
• This design ensures that while we leverage additional data from source agents to improve the quality of our conformal prediction sets, the marginal coverage on the target agent is strictly guaranteed.

W6: We sincerely appreciate this feedback and apologize for any confusion caused by our notation.

• The complexity arises from the need to precisely represent multiple algorithmic components in our generalized framework. We apologize if the complexity of the notation caused any confusion.
• To better address your concerns, could you kindly specify which particular notational aspects you found challenging to follow? This would enable us to provide more targeted explanations or improvements to the relevant notations.

Q3: We sincerely thank the reviewer for highlighting this important aspect.

• Our algorithm is designed for any type of heterogeneous local data distributions across agents.
• In our synthetic data experiments, we primarily focus on concept shift, i.e., changes in $P_{Y\mid X}$ across agents, as described in Section 3.1.
• For the real-world datasets, both covariate and concept heterogeneity exist among agents.

We are more than happy to answer any further questions during the Discussion.

Reference

[1] Charles Lu, Yaodong Yu, Sai Praneeth Karimireddy, Michael Jordan, and Ramesh Raskar. Federated conformal predictors for distributed uncertainty quantification. In International Conference on Machine Learning, pages 22942–22964. PMLR, 2023.

[2] Vincent Plassier, Mehdi Makni, Aleksandr Rubashevskii, Eric Moulines, and Maxim Panov. Conformal prediction for federated uncertainty quantification under label shift. In International Conference on Machine Learning, pages 27907–27947. PMLR, 2023.

[3] Vincent Plassier, Alexander Fishkov, Maxim Panov, and Eric Moulines. Conditionally valid probabilistic conformal prediction. stat, 1050:1, 2024.

[4] Vincent Plassier, Nikita Kotelevskii, Aleksandr Rubashevskii, Fedor Noskov, Maksim Velikanov, Alexander Fishkov, Samuel Horvath, Martin Takac, Eric Moulines, and Maxim Panov. Efficient conformal prediction under data heterogeneity. In International Conference on Artificial Intelligence and Statistics, pages 4879–4887. PMLR, 2024.

We are grateful for your positive perspective recognizing the importance of combining personalized federated learning with conformal prediction, as well as our method's formal marginal coverage guarantees and clear presentation. We are grateful to the reviewer for the insightful and constructive feedback. These comments have significantly contributed to improving the quality and presentation of our work. Below we provide detailed responses and clarifications to address the raised concerns.

W1: We appreciate this important suggestion.

• First, we would like to clarify a typo in our citation in line 96. We mistakenly cited [1,2], but it should have been [1] and [3], which correspond to their approaches under label shift and covariate shift settings, respectively. We denote these two methods as CPhet [3] and CPlab [1]. They share similarities with our approach but are designed for very different settings.
• They are not compared because they cannot guarantee marginal coverage under our setup when the test data is from the target agent distribution, which is different from the source agents'.
• We have now included these baselines in our experiments. Real data experiment results are shown in the table below:

W2: Thank you for highlighting this.

• Under our setup with test data from the target agent distribution, both GLCP and PFCP consistently guarantee the desired marginal coverage regardless of the distribution of target and source agents.
• In contrast, FedCP, FedCP-QQ, CPhet and CPlab do not maintain such guarantees when the target distribution differs from the source agents’.
• For the performance of test-conditional coverage, we outline the limitations of PFCP as follow:
  • As shown in Theorem 2.8, achieving better conditional coverage than GLCP requires a moderately well-trained federated conditional density estimator, such that $\delta_1(x,\hat{F}_{\rm agg})<\delta_1(x,\hat{F})$, thereby leading to improved conditional coverage. However, if the density ratio is poorly estimated, the aggregation may not result in any improvement.
  • If the source distribution significantly differs from the target distribution, for instance the support of $P$ and any $P_k$ do not intersect, the information from source agents cannot help improve the conformal set on the target agent.
  • When the data on the target agent is extremely limited, insufficient to yield a reliable estimate of the conditional distribution $F(s\mid x)$ — both GLCP and PFCP may not deliver conformal sets with satisfying test-conditional coverage, though PFCP may still outperform GLCP in the sense of test-conditional coverage. Nonetheless, the marginal coverage remains valid in both cases.

We'll add these discussions in the revised version.

W3: We thank the reviewer for raising this important point.

• Personalized setting refers to scenarios where agent-specific feature-label relationships differ substantially. The key characteristic of our problem is that test samples originate from a single designated target agent, and our primary objective is to guarantee the validity of conformal prediction specifically for this target agent. Our coverage guarantees are explicitly personalized to the target agent's distribution and simultaneously improving local (instance-specific) predictive performance.
• Indeed, Assumption 2 serves as a condition to ensure the transferability of information from source to target domains. The fundamental requirement is that the conditional distribution $f(s\mid x)$ in the target domain can be effectively reconstructed through weighted combinations of source conditional distributions $\sum_k\pi_kf_k(s\mid x)$.
• While Assumption 2 assumes a uniform lower bound on the density ratio $\frac{f(s\mid x)}{f_k(s\mid x)}$ for all $(s,x)$ pairs and some source distribution $P_k$, the actual necessary condition is substantially weaker. What we truly require is that the mixture of source distributions contains sufficient information about the target, meaning the corresponding density ratio $\sum_k\pi_kf_k(s\mid x)/f(s\mid x)$ has lower bounded (equation between line 560-561). This much milder condition would still guarantee that the target conditional distribution can be adequately approximated through our federated estimation framework, while being more realistic in practical applications. We will add more discussion on Assumption 2 in the revised version.

We are more than happy to answer any further questions during the Discussion.

Reference

[1] Vincent Plassier, Mehdi Makni, Aleksandr Rubashevskii, Eric Moulines, and Maxim Panov. Conformal prediction for federated uncertainty quantification under label shift. In International Conference on Machine Learning, pages 27907–27947. PMLR, 2023.

[2] Vincent Plassier, Alexander Fishkov, Maxim Panov, and Eric Moulines. Conditionally valid probabilistic conformal prediction. stat, 1050:1, 2024.

[3] Vincent Plassier, Nikita Kotelevskii, Aleksandr Rubashevskii, Fedor Noskov, Maksim Velikanov, Alexander Fishkov, Samuel Horvath, Martin Takac, Eric Moulines, and Maxim Panov. Efficient conformal prediction under data heterogeneity. In International Conference on Artificial Intelligence and Statistics, pages 4879–4887. PMLR, 2024.