Robust Conformal Prediction under Joint Distribution Shift

Thank you for your questions.

My main concerns are w.r.t. practicality and evaluation. A fundamental requirement of the proposed algorithm and D-NTW is the availability of labelled samples from every test domain $Q _{XY}^{(e)}$ in order to obtain the conformal test score distributions, from which then both the likelihood ratios for covariate shifts and D-NTW can be explicitly computed. Beyond existing concerns about the practicality of estimating a likelihood ratio, we now require actually explicitly estimating every test domain distribution, which is prone to much error.

We introduce how we applied kernel density estimation (KDE) to estimate the likelihood ratio in the author rebuttal. Indeed, the process of KDE is prone to error with a limited sample size from each test domain, especially for high-dimension regression. The experiment of the Airfoil dataset has the smallest sample size, 160, of one test domain, and the highest regression dimension, 5. Figure 2 (a) shows the experiment result of that dataset, and the performance is acceptable. Nevertheless, it is still challenging for KDE with higher dimensions and fewer samples.

Regardless, if we can now explicitly obtain $\hat{F}_Q^{(e)}$ for every test domain, I am wondering why the direct solution is not to just compute a conformal quantile q(e) on the basis of this for every test domain and thus optimally adapt to the existing joint shifts per domain. Perhaps the loss of the disentanglement of shift contributions is the motivation?

The reason why we do not optimize based on joint distribution shift but on concept shift is because covariate shift can be addressed by importance weighting, even if estimation error can be introduced by KDE. Intuitively, since a model has limited fitting ability, we want it to focus on the issue (concept shift) that cannot be addressed by importance weighting to avoid wasting its representativeness.

We do not compute a quantile for each $\hat{F}_Q^{(e)}$ and adapt to them, because a quantile value is $(1-\alpha)$-specific, but we want the trained model to be comprehensive, so test coverage approaches $1-\alpha$ no matter how $\alpha$ changes.

In general, I find this requirement to require knowing or estimating the test domain distributions on the basis of labelled test domain data and thus the use of extensive density estimation quite limiting, especially if we consider high-dimensional settings.

For your concerns about high-dimensional settings, higher dimensions will make the KDE less reliable. However, we do not only consider 1-d regression tasks. Line 60 may cause some confusion. In fact, $\mathcal{X}\subset\mathbb{R}^d$. For the airfoil dataset, d=5 in line 463. For three traffic datasets, d=4. For epidemic datasets, d=2.

It would be more interesting to compare to conformal algorithms for shift such as the mentioned [2,3], showing e.g. that those are not able to fully capture the joint shift or are overly conservative, thus compromising on the metric.

To illustrate the benefits of mRCP, we provide experimental results about coverage and prediction intervals in comparison with the WC method [R1] in the author rebuttal.

Have you considered higher-dimensional experimental settings and/or using models that are more complex than an MLP?

High-dimension regression tasks are included in experiments. As we mentioned, for the airfoil dataset, d=5. For three traffic datasets, d=4. For epidemic datasets, d=2. Applying mRCP to models with higher fitting ability may obtain better performance as they can have a better trade-off between residual loss and distribution discrepancy loss.

In the experiments, all the generated test domain shifts are created manually. Have you considered datasets with unknown test shifts?

Except for the airfoil dataset, all three traffic datasets and three epidemic datasets have natural joint distribution shifts without any manual modification. Their joint shifts are unknown, and we can only estimate their covariate shift by KDE as mentioned above.

In sec 5.2. the correlation is assessed via the Pearson coefficient. Could you provide a reasoning on why you believe a linear relationship exists, and/or if you have considered more robust measures such as rank correlation? Given that the estimated score distributions are empirical CDFs, this seems perhaps more intuitive.

We actually only expect a monotonic positive correlation because $\mathbb{E} _{\alpha}[D^{(e)}]$ is the average of discrete $D^{(e)}$ values for $\alpha = 0.1,0.2,...,0.9$ and $d _{NTW}^{(e)}$ is the distributional distance. However, experimental results also show a strong positive linear correlation. That is because both $d _{NTW}^{(e)}$ and $\mathbb{E} _{\alpha}[D^{(e)}]$ approximate Eq. (10) well, so as Eq. (10) increases, $d _{NTW}^{(e)}$ and $\mathbb{E} _{\alpha}[D^{(e)}]$ increases proportionally.

We present the result of Spearman's rank correlation coefficient in the author rebuttal.

For US-States and MLP, all distance metrics except D-NTW show negative correlation.

Please take a look at Figure 3. US-States only have four test domains; thus, the coefficient can be negative if the four points do not show a positive correlation.

Could you comment on the limitations of the made assumption in Eq. 17 that the calibration domain $P_{XY}$ is considered a linear mixture of the "unknown" test domain distributions? This seems like a restriction that is not explicitly mentioned.

mRCP is also applicable when $P_{XY}$ is not a linear mixture of the test domains. The role of $P_{XY}$ is to provide a ‘target’ conformal score distribution for all test score distributions so that they can approach. It is worth in study the performance of mRCP if $P_{XY}$ is a more complex combination of $Q_{XY}^{(e)}$

Reference

[R1] Cauchois M, Gupta S, Ali A, et al. Robust validation: Confident predictions even when distributions shift[J]. Journal of the American Statistical Association, 2024: 1-66.

Thank you for your questions.

section 3.1 and 3.2 introduce some important definitions: while authors provide some explanation, the theoretical understanding of them are very limited.

Section 3.1 and 3.2 introduce the process of quantifying the empirical coverage gap caused by concept shift, and connecting it to a distributional discrepancy metric.

The theoretical analysis of bounding the coverage gap by population distributions with a probability related to the number of calibration and test samples is discussed in Appendix C.

simulation: authors mention the mRCP can achieve a balance between coverage gap and size of residual, further simulations need to be carried out ( I would be interested to see a plot including the avg. coverage vs avg. residual size

To further illustrate the coverages obtained by mRCP, we provide experimental results about coverage and prediction intervals in comparison with the WC method [R1] in the author rebuttal.

the coverage difference works with the empirical distribution function, would it be possible to study the coverage gap in the form of P(...) - P(...)

We developed the upper bound of the empirical coverage gap in population form in Appendix C.

in the paper, a likelihood ratio dQ/dP is assumed to be known, how would you estimate it in practice?

The likelihood ratio is not assumed known and it is estimated by kernel density estimation. Please read the author rebuttal for details.

The factor (n+1)/n, I believe is typically added in the i.i.d case while for covariate shift, the form is not like this if you take a look at the paper "conformal prediction under covariate shift".

Using the factor $(n+1)/n$, the quantile under exchangeability assumption is given by

In Lemma 1 of [R2], the quantile is calculated by

Both forms produce the same quantile value under the i.i.d. case.

[R2] calculates the importance-weighted quantile based on the latter one, whereas we present Eq. (6) based on the former one. However, there is a minor difference when applying these two forms to importance weighting, regarding how $p _i^w$ and $p _{n+1}^w$ are calculated in Eq. (7) of [R2].

In Eq. (7) of [R2], the $p _i^w$ and $p _{n+1}^w$ are functions of the test input $x$, as $\delta _{\infty}$ is included its Lemma 1. However, in our work, the weights of conformal scores are not functions of $x$.

In Section 4.5 of [R3], another form is provided to get rid of the factor $(n+1)/n$ and $\delta _{\infty}$. We think that is a good example to represent the importance-weighted quantile and would like to take it in the later version.

In practice, how would you select the tuning parameter beta?

$\beta$ is the hyperparameter for our proposed method in Eq (19). Therefore, we need to try different $\beta$ values to draw a Pareto front of the prediction residuals (horizontal axis) and coverage gap (vertical axis) in Figure 2. For each $\beta$ value, we train a model and obtain an optimal Pareto solution of (residual, coverage gap) and finally, we get curves in Figure 2. The baseline V-REx is also tuned by a hyperparameter, so we tried different $\beta$ for V-REx as well in Figure 2. We show our selected $\beta$ values in Table 3. The basic goal of selecting $\beta$ values is to make the Pareto solutions more uniformly distributed and dense enough to obtain reliable Pareto curves.

Better theoretical guarantees for section 4 need to be developed.

When a theoretical coverage guarantee under distribution shift is obtained, it is very hard to keep prediction sets small at the same time. For instance, the worst-case (WC) method [R1] holds the guarantee under distribution shift at the cost of inefficient large prediction sets.

The purpose of Section 4 is to get a good trade-off between the coverage and prediction interval size. This good trade-off can be obtained because of two reasons. First, we do not follow conservative methods as mentioned above. Secondly, prediction interval size is highly related to the magnitude of conformal scores which is absolute residual for regression tasks, and Eq. (19) actually balances the residual loss and the NTW loss, which represents the gap to $1-\alpha$.

In Figure 5 of the author rebuttal attachment, even if mRCP coverage is not at least $1-\alpha$, it approaches $1-\alpha$ very close.

Reference

[R1] Cauchois M, Gupta S, Ali A, et al. Robust validation: Confident predictions even when distributions shift[J]. Journal of the American Statistical Association, 2024: 1-66.

[R2] Tibshirani R J, Foygel Barber R, Candes E, et al. Conformal prediction under covariate shift[J]. Advances in neural information processing systems, 2019, 32.

[R3] Angelopoulos A N, Bates S. Conformal prediction: A gentle introduction[J]. Foundations and Trends® in Machine Learning, 2023, 16(4): 494-591.

Thank you for your questions.

It is assumed that the likelihood ratio dQ/dP is known...

The likelihood ratio is not assumed known and it is estimated by kernel density estimation (KDE). Please read the author rebuttal for details.

In this work, the proposed methods have no theoretical guarantee, and the focus is more on empirical illustrations. ... Why should one apply conformal prediction here if we cannot exploit the advantage of the conformal inference framework?

Standard CP can provide coverage guarantee under exchangeability assumption for distribution-free cases. However, prior knowledge about the extent of the shift is necessary to maintain the coverage guarantee when a distribution shift happens. For instance, [R1] holds the coverage guarantee by constraining the test distribution within an $f$-divergence ball centered at a calibration distribution. Similarly, [R3] also develops the guarantee based on the knowledge of how distributions are contaminated.

Furthermore, even if the theoretical coverage guarantee based on prior knowledge of distribution shift is obtained, it is hard to keep prediction sets small at the same time. For instance, the worst-case (WC) method [R1] holds the guarantee under distribution shift at the cost of inefficient large prediction sets.

Experiments in the author rebuttal prove that our method can get a better trade-off between the coverage and prediction interval size.

Evaluating the difference under distribution shift using (5) doesn't seem very reasonable ...

Eq. (5) characterizes the coverage difference under distribution shift. We minimize this coverage gap by Wasserstein distance regularization in Eq. (19) so we can use the quantile of calibration conformal scores to construct prediction sets.

Specifically, we first use importance weighting [R2] with KDE to address covariate shift and approximate a weighted version of calibration conformal scores. Secondly, Wasserstein distance is reduced between test and the weighted calibration conformal score distributions to reduce the coverage gap caused by concept shift, so the coverage on test data approaches $1 –\alpha$.

it is quite strange why they replace this known value with $\hat{F}_{Q/P}(q^∗)$, generating an additional error.

Appendix B justifies the error between $\hat{F} _{Q/P} (q^*)$ and $\hat{F} _P (q)$ can be controlled by the increasing size of calibration data $n$. Secondly, $\hat{F} _{Q/P} (q^*)$ in Eq. (9) makes $D _{concept}$ a function of only one variable $q^*$, thus facilitating the introduction of NTW in the following sections. Please check Figure 1 (b) for the relationship between $\hat{F} _P (q)$ and $\hat{F} _{Q/P} (q^*)$ with given $\lceil{(1-\alpha)(n+1)} \rceil/n$.

I'm a bit confused about equation (13), where again the exact equality seems to hold. The two empirical CDFs $\hat{F} _Q$ and $\hat{F} _{Q/P}$ are step functions where the jumps occur at the same values $v_1$,⋯,$v_n$, so it seems to me that exact equality holds.

Eq. (13) does not hold exact equality. Because $\hat{F} _Q$ is the test conformal score CDF, it jumps over the test conformal scores $V_t$ as defined in Eq.(4), not calibration conformal scores $V_c$.

In that sense, the remaining discussion in section 3.2 sounds a bit odd. For example, the empirical CDFs have values exactly 1 for $v≥\max v_i$. What does the 'long tail' mean in this case?

The purpose of truncation in Section 3.2 is to estimate Eq. (10) by modifying the Wasserstein distance in Eq. (13). In Eq. (10), each pair of $|\hat{F} _Q(v _i)-\hat{F} _{Q/P}(v _i)|$ has the same weight. In Eq. (13), the weight of $|\hat{F} _Q(v _i)-\hat{F} _{Q/P}(v _i)|$ is $v _{i+1}-v _i$, so we want $v _{i+1}-v _i$ is similar for different $i$ as well to approximate Eq. (10). However, the slop of conformal score CDFs tend to be flatter when the CDFs converge close to 1 but not reaching 1, which means $v _{i+1}-v _i$ there are quite large, so we need to truncate that part (long tail) in Eq. (15) to estimate Eq. (10). Table 1 justifies the necessity of truncation and normalization.

Could the authors clarify the motivation behind the need for 'multi-domain' inference?

$P _{XY}$ is a mixture distribution of $Q _{XY}^{(e)}$ for $e\in\mathcal{E}$ as shown in Eq. (17). This can be a natural partition. For instance, $Q _{XY}^{(e)}$ can be the patient data of a hospital $e$, and $P _{XY}$ is the patient distribution of multiple hospitals. $1 – \alpha$ coverage on $P _{XY}$ can not ensure the coverage on $Q _{XY}^{(e)}$, so we expect $1 − \alpha$ coverage on $Q _{XY}^{(e)}$ as well. Or $Q _{XY}^{(e)}$ can be traffic data from a single hour $e$, and $P _{XY}$ is the data collected within a day. Please check Appendix D for the experiment setups and dataset preprocessing details.

how do we know that the probability P((X,Y)∈e) is 1/M for every e in the partition?

$P _{XY}$ is denoted as calibration domain and distribution at the same time. $Q _{XY}^{(e)}$ denotes the conditional distribution of $(x,y)$ given $(x,y)$ is from domain $e$.

For the weight $1/M$, $P_{XY}$ can be the distribution of daily traffic sensor data, and $Q _{XY}^{(e)}$ is the data of an hour $e$, so the weight is $1/24$. Or taking the hospital example, we can fairly give $Q _{XY}^{(e)}$ of each hospital $e$ the same weight if their patient amounts are similar.

Reference

[R1] Cauchois M, Gupta S, Ali A, et al. Robust validation: Confident predictions even when distributions shift[J]. Journal of the American Statistical Association, 2024: 1-66.

[R2] Tibshirani R J, Foygel Barber R, Candes E, et al. Conformal prediction under covariate shift[J]. Advances in neural information processing systems, 2019, 32.

[R3] Sesia M, Wang Y X, Tong X. Adaptive conformal classification with noisy labels[J]. arXiv preprint arXiv:2309.05092, 2023.

Thank you for the authors' responses. However, I feel that many of the answers do not fully address the questions I raised. Below, I have included further questions and clarifications.

1. "The likelihood ratio is not assumed known and it is estimated by kernel density estimation (KDE)"

This is exactly what I meant. In practice, the likelihood ratio must be estimated, even though the theory is often written as if it is known. For instance, the weighted conformal prediction method by Tibshirani et al. provides a comprehensive theoretical framework for the case where the likelihood ratio is known (and clearly states it), and then also mentions the need for estimation in practical scenarios. A similar level of clarity would be beneficial in this work (though I'm not very sure how much it will make sense in this work where there's no concrete theory even for the known-likelihood ratio setting).

2. "However, prior knowledge about the extent of the shift is necessary to maintain the coverage guarantee when a distribution shift happens..."

I understand that distribution-free guarantees are not possible in the distribution-shift setting. My question is about the benefits of applying the conformal prediction framework in this context. Again as an example, Tibshirani et al. introduce weighted conformal prediction with a theoretical guarantee for the known likelihood ratio case, and Lei and Candes further develops this by showing that using an estimated likelihood ratio provides a coverage lower bound of $1-\alpha$-(TV distance based on the accuracy of estimation). This clearly illustrates the advantage of the conformal-type approach, as it requires only a good estimate of the likelihood ratio and offers 'distribution-free' inference with respect to the conditional distribution of $Y$ given $X$.

In this work, the theoretical components are largely intuitive explanations rather than formal mathematical results. What I'm curious is why one should try to provide such intuition for the conformal-type approach. If some kind of distribution-free guarantee is not the goal, couldn't similar intuition be applied to other approaches that might perform better in practice?

3. Regarding equation (5):

So it is true that $\hat{F}_P(q) = \lceil (1-\alpha)(n+1) \rceil/n$ holds exactly?

I was asking whether it is necessary to replace this known value with one that introduces error, rather than the magnitude of the error. $D_\text{concept}$ is still a function of only $q^*$ if we simply plug in $\hat{F}_P(q) = \lceil (1-\alpha)(n+1) \rceil/n$, so the authors' response is not very convincing to me.

---Actually, upon closer inspection, it seems like the replacing term $\hat{F}_{Q/P}(q^*)$ is also exactly $\lceil (1-\alpha)(n+1) \rceil/n$? So it now seems to me that there is actually no replacement or error?

4. Thank you for the clarification regarding the difference between the supports of the two CDFs. As a quick follow-up, is this work more focused on large sample settings rather than small/finite sample settings? (since there are statements of the form "approximately holds when $n$ is large.") If so, could the authors further address point 1 above, concerning the need for a conformal approach whose main advantage is exact finite sample guarantees?

5. Regarding the mixture distribution representation, I'm not seeking examples. I am wondering, e.g., whether it is an assumption (that all mixture probabilities are equal, which seems quite strong), or something controllable through experimental design, etc. In the paper, it says the ``domain" is partitioned into $M$ bins, and then the mixture distribution representation suddenly appears. I'm just asking for some clarities here. The examples that the authors mentioned seem to be natural partitions rather than an experimental design, but then it doesn't seem very reasonable to just assume that all the mixture probabilities are equal.

I appreciate the authors' detailed responses.

I think I have now received answers regarding points 1, 2, and 4 in the follow-up discussion, and I hope there will be improvements and clarifications in the final version of the paper.

I'd like to add some follow-up questions regarding points 3 and 5.

3. I see that for $\hat{F}_{Q/P}$, the exact equality does not hold---I think what I was trying to say is that it is still a known value that is close to $1−α$ (thanks for pointing this out). So, there is an error, and it now returns to the original question.

In my original comment, what I meant was that viewing the empirical version of $F_P(q)$ as $\hat{F}_P(q)$

doesn't sound reasonable, since $\hat{F}_P$

and $q$ are derived from the same datasets (which involves double-dipping). It also applies to $\hat{F}_{Q/P}(q^*)$.

The main question that remains unanswered is whether, if we proceed with this empirical version, it is necessary to replace this known value with something that includes an error. Are there any challenges in applying a similar approach, if we just plug in the exact value?

5. I understand that it might not make much sense to ask if this statement is some kind of assumption, as it is not as though a theorem is based on it. However, I think the following clarifications would be helpful:

• Is the idea of the proposed method indeed based on model (17), which suggests that the sampling distribution is a mixture distribution with equal mixture probabilities?

• If it indeed requires equal mixture probabilities, then 'partitioning the domain into $M$ bins' is not an accurate statement. Is this what the authors agree?

• My concern is that the dataset examples still seem more fitting to a setting where the domain is partitioned according to some natural rule, rather than by experimental design. So I believe some justification is needed here.

Thank you for your questions.

What specific benefits it offers over other state-of-the-art approaches that address differences in the non-conformity score distributions? I suggest that the authors demonstrate the validity and efficiency of the prediction intervals obtained for different alphas on test data.

When a theoretical coverage guarantee under distribution shift is obtained, it is hard to keep prediction sets small at the same time. For instance, as mentioned in lines 41-43 the worst-case (WC) method [R1] holds the guarantee under distribution shift at the cost of inefficient large prediction sets by applying the highest $\lceil{(1-\alpha)(n+1)} \rceil/n$ quantile to all test distributions.

The proposed method can get a good trade-off between the coverage and prediction interval size because of two reasons. First, we do not follow conservative methods as mentioned above. By minimizing the distribution discrepancy via NTW, we make the calibration and test distribution score distributions satisfy exchangeability better, so we can use the calibration conformal score quantile (not the largest quantile of multiple test distributions) to generate prediction sets. Secondly, prediction interval size is related to the magnitude of conformal scores which is absolute residual for regression tasks, and Eq. (19) can balance the residual loss and NTW loss.

We demonstrate the efficiency of the proposed method in author rebuttal.

Split conformal prediction and group conditional split conformal prediction ... The latter can include ... section 4.1 of [Barber et al. 2020, "The Limits of Distribution-Free Conditional Predictive Inference"] or BatchGCP/BatchMVP in [Jung et al. 2022, "Batch Multivalid Conformal Prediction"].

The performance of Split CP (SCP) with a model trained by ERM is highlighted in the author rebuttal.

For the comparison with BatchMVP [R2], first BatchMVP does not decompose distribution shifts between each group into covariate and concept shifts. Secondly, Batch MVP is trained based on a given $\alpha$ value, so the multi-valid coverage guarantee only holds at a specific error level. However, mRCP minimizes the distribution discrepancy between calibration and test conformal scores, so a single trained model can make the coverage on test data approach $1-\alpha$ no matter how $\alpha$ value changes.

For the comparison with Section 4.1 of [R3], in Eq. (10) of [R3], it takes the supremum of quantiles for groups in $\hat{\mathfrak{X}} _{n _1}$ to ensure the coverage for any group $\mathcal{X}\in\hat{\mathfrak{X}} _{n _1}$. As a result, [R3] has the same problem as [R1] and is likely to cause unnecessarily large prediction intervals.

Performance of the covariate shift split conformal prediction...

The result of importance-weighted SCP [R6] with a model trained by ERM is highlighted in author rebuttal.

Models trained by V-REx can be applied to standard SCP or importance-weighted SCP. The result of standard SCP based on V-REx is shown as the blue curves in Figure 2. The result of importance-weighted SCP based on V-REx will be worse than standard SCP. Because V-REx aligns the unweighted score distributions by minimizing its regularization term, weighting conformal scores afterward will disturb the previous alignment and make score distributions more discrepant, thus enlarging the coverage gap.

The same also happens to the importance-weighted SCP with the model trained by DRO.

An adaptive approach such as the one by [Amoukou and Brunel 2023, Adaptive Conformal Prediction by Reweighing Nonconformity Score]

[R4] applies a novel conformal score function to make the prediction set more customized to the input $x$. However, it is based on the exchangeability assumption, having a different problem setup from ours. The position of [R4] is in the first row of Table 2.

the authors should discuss how their work relates to [Barber, Rina Foygel, et al. "Conformal prediction beyond exchangeability"]...

[R5] uses total variation to bound coverage gap. However, total variation measures half the absolute area between two probability density functions, without considering how the probabilities are distributed across the support of the distributions. On the contrary, Wasserstein distance considers how probabilities are dispersed along the support. Therefore, the same total variations can come up with different Wasserstein distances in Figure 1 (c). Wasserstein distance can reflect the overall difference of two conformal score CDFs along the support, but total variation fails to do so.

How do you pick $\sigma$

The experiment results can be different with different $\sigma$ values. $\sigma$ is selected based on the properties of conformal score distributions. Usually, $\sigma$ should be smaller (truncate more) for more concentrated distributions. We choose $\sigma$=0.8 for MLP and 0.95 for physics-informed models, because MLP has a better fitting ability, making conformal scores more concentrated.

Reference

[R1] Cauchois M, Gupta S, Ali A, et al. Robust validation: Confident predictions even when distributions shift[J]. Journal of the American Statistical Association, 2024: 1-66.

[R2] Jung C, Noarov G, Ramalingam R, et al. Batch multivalid conformal prediction[J]. arXiv preprint arXiv:2209.15145, 2022.

[R3] Foygel Barber R, Candes E J, Ramdas A, et al. The limits of distribution-free conditional predictive inference[J]. Information and Inference: A Journal of the IMA, 2021, 10(2): 455-482.

[R4] Amoukou S I, Brunel N J B. Adaptive conformal prediction by reweighting nonconformity score[J]. arXiv preprint arXiv:2303.12695, 2023.

[R5] Barber R F, Candes E J, Ramdas A, et al. Conformal prediction beyond exchangeability[J]. The Annals of Statistics, 2023, 51(2): 816-845.

[R6] Tibshirani R J, Foygel Barber R, Candes E, et al. Conformal prediction under covariate shift[J]. Advances in neural information processing systems, 2019, 32.

Thank you for your suggestions and questions.

Missing References: Important related works, such as "Conformal prediction beyond exchangeability"...

[R1] uses total variation to bound coverage gap. However, total variation measures half the absolute area between two probability density functions without considering how the probabilities are distributed across the support of the distributions. On the contrary, Wasserstein distance considers how probabilities are dispersed along the support. Therefore, the same total variations can come up with different Wasserstein distances in Figure 1 (c). As a result, Wasserstein distance can reflect the overall difference of two conformal score CDFs along the support, which is related to the coverage gap, but total variation fails to do so.

Why is an empirical distribution used in Equation (7) instead of a population distribution, in the definition of a quantity (the coverage gap) which should be a population quantity?

We focus on the empirical coverage gap with finite samples. This is widely discussed and important to practical applications, such as Section 1.2 of [R1]. The empirical coverage gap can be bounded with a probability related to the numbers of calibration and test samples in Eq. (30) of Appendix C. Besides, the upper bound builds a connection between population forms of conformal score distributions and the empirical coverage gap.

Lines 112-115 introduce an assumption that is neither explained nor motivated...

The assumption in lines 112-115 can be presented more rigorously. In Appendix B, we develop the following inequality for Eq. (7).

Since the error can be bounded in Appendix B as follows and can be controlled by increasing the size of calibration data $n$,

we can focus on $\hat{F}_Q(q^{\ast})-\hat{F} _{Q/P}(q^{\ast})$.

Intuitively, since $q$ and $q^*$ are $\lceil{(1-\alpha)(n+1)} \rceil/n$ quantiles of $\hat{F} _{P}$ and $\hat{F} _{Q/P}$ respectively, with a large $n$, it is reasonable to expect the difference between $\hat{F} _{Q/P}(q^{\ast})$ and $\hat{F}_P(q)$ is very small. Please check Figure 1 (b) for the relationship between $\hat{F} _{Q/P}(q^{\ast})$ and $\hat{F}_P(q)$ with a given $\lceil{(1-\alpha)(n+1)} \rceil/n$ value.

The explanation of handling concept shift via covariate shift, particularly in Equation (10), is confusing...

We do not intend to address concept shift by covariate shift in Eq. (10). At first, we approximately address covariate shift by importance weighting [R4] based on kernel density estimation. Therefore, we can estimate the weighted calibration conformal score CDF $\hat{F} _{Q/P}$ from the unweighted one $\hat{F} _P$. The remaining gap between $\hat{F} _{Q/P}$ and the test conformal score CDF $\hat{F} _{Q}$ is caused by concept shift in Eq. (8). Then, we quantify the distribution discrepancy between $\hat{F} _{Q/P}$ and $\hat{F} _{Q}$ by NTW, and minimize it during training.

mRCP can make the coverage on test data approach $1-\alpha$, rather than applying conservative methods, like the worst-case (WC) method [R5] as mentioned in lines 41-43, which is likely to cause overestimated coverage and prediction sets.

We demonstrate a better trade-off between coverage and prediction intervals of the proposed method compared with WC method in the author rebuttal.

Section 2.1 lacks precision. For example, Equation (2) describes a special case for regression...Is it limited to a specific type of conformity scores, such as that in Equation (2), or is it more general?

We focus on regression tasks as mentioned in line 65 with the conformal score function defined in Eq. (2), which is commonly used for regression tasks [R1][R2][R4]. The proposed method is also applicable to other conformal score functions for regression tasks, like the one proposed in localized split CP [R6].

Is there a connection between the coverage difference you considered and that studied in "Adaptive conformal classification with noisy labels" by Sesia et al. (2023)?

[R3] considers CP on classification problems whereas we focus on CP on regression tasks. [R3] aims to maintain the coverage guarantee if the labels of calibration samples are contaminated during sampling, which means the exchangeability assumption is violated if test samples are drawn from the same population distribution. Therefore, [R3] considers concept shift but not covariate shift as the marginal distribution of features does not change.

Our work investigates the coverage difference under joint distribution shift between calibration and test distributions, which means the covariate shift and concept shift can occur simultaneously.

Reference

[R1] Barber R F, Candes E J, Ramdas A, et al. Conformal prediction beyond exchangeability[J]. The Annals of Statistics, 2023, 51(2): 816-845.

[R2] Romano Y, Patterson E, Candes E. Conformalized quantile regression[J]. Advances in neural information processing systems, 2019, 32.

[R3] Sesia M, Wang Y X, Tong X. Adaptive conformal classification with noisy labels[J]. arXiv preprint arXiv:2309.05092, 2023.

[R4] Tibshirani R J, Foygel Barber R, Candes E, et al. Conformal prediction under covariate shift[J]. Advances in neural information processing systems, 2019, 32.

[R5] Cauchois M, Gupta S, Ali A, et al. Robust validation: Confident predictions even when distributions shift[J]. Journal of the American Statistical Association, 2024: 1-66.

[R6] Han X, Tang Z, Ghosh J, et al. Split localized conformal prediction[J]. arXiv preprint arXiv:2206.13092, 2022.