Conformal Prediction with Cellwise Outliers: A Detect-then-Impute Approach

Q1: Author should highlight the purpose of this paper instead of giving the notation in introduction.

We sincerely appreciate this constructive critique of our manuscript's organization. In response to your suggestion, we will comprehensively restructure the introduction in the future revision to:

1. Problem Focus

   • Lead with the critical challenge of cellwise contamination in conformal prediction
   • Highlight gaps in existing methods' ability to handle this data corruption

2. Solution Framework

   • Clearly articulate our methodological innovations
   • Emphasize how we address previously unsolved problems

3. Presentation Strategy

   • Defer technical notation to methodology sections
   • Maintain narrative flow while preserving rigor

These adjustments will prioritize readability and better highlight the scientific significance of our approach.

Q2: Assumption 3.1 is unrealistic in practice.

We appreciate your inquiry regarding Assumption 3.1 and we acknowledge its imperfection.

• We'd like to stress that Assumption 3.1 is theoretically necessary for model-free and distribution-free coverage guarantees and can be satisfied by choosing a small detection threshold in practice.
• Additionally, we also present a total variation coverage gap of our method when Assumption 3.1 is violated. If the total variation between processed calibration data and test data is small, the coverage can still be approximately controlled. Please refer to Reviewer vdr2's Q1 for details and an additional experiment about Assumption 3.1 (due to the space limit).

We would be happy to discuss this further if additional clarification would be helpful.

Q3: Assumption 3.2 may not hold for correlated features.

Thanks for your question and we’d like to make the following explanations. For correlated features:

• Assumption 3.2 can be satisfied if we use one-class SVM method to learn the score $s_j$ for each coordinate.
• In simulations, we also used detection methods do not satisfy Assumption 3.2, such as DDC in Figure 3 and cellMCD in Figure 4(b).
• In Section 5, we provided theoretical results for our method when Assumption 3.2 does not hold.

Q4: A new experiment with setting linear, homoscedastic, light-tailed could be added.

Thank you for your advice. In fact, we have included the experimental result of the setting you suggested in Appendix D.2 but didn't mention it in the main text due to the space limitation. We apologize for any possible reading incompleteness and will add this setting to the main text when we have an additional page in the revision.

Q5: JDI-CP requires O(n) pairwise operations, which could be inefficient for large datasets.

We appreciate this insightful observation about our method's computational aspects.

1. JDI-CP Design Considerations:
   • Primary objective: Finite-sample coverage guarantee
   • Trade-off: Achieves robustness (demonstrated in Figure 7) at some computational cost
   • Reference: Similar trade-offs exist in Barber et al. (2021)
2. Future Directions:
   • Actively researching more efficient implementations
   • Will optimize computational performance while maintaining theoretical guarantees

[1]Barber R F, Candes E J, Ramdas A, et al. Predictive inference with the jackknife+. The Annals of Statistics, 2021.

Q6: This paper seems simply combining DI and CP. Author cites many theories which are proved in prior work like theorem 3.5. Focus too much on detection instead of CP.

Thanks for your insightful comments on the contribution of our approach. We’d like to provide some clarifications to highlight the contribution of our work.

• Firstly, our approach fundamentally differs from naive DI+CP combinations. As discussed and shown in Appendix A.1, such a direct combination cannot achieve coverage control. Therefore, we proposed new techniques to adaptively deploy DI on calibration data and proposed PDI-CP and JDI-CP.
• Regarding Theorem 3.5, this is a negative result we obtained to illustrate the necessity of Assumption 3.1, and we have not found similar results in other papers. Additionally, we have provided a new counterexample under the CQR score, please refer to Reviewer vdr2's Q2.
• Finally, we believe that detection is essential to cope with cellwise outliers in predictive inference tasks. The classic CP methods under exchangeability assumption have been extensively studied. Here, we are concerned with the nonexchangeable problem caused by cellwise outliers, which is challenging and has not been studied before. The detection and imputation steps are used to identify and remove those outliers, so we can construct informative conformal prediction intervals. The detection is key to constructing exchangeable processed features in ODI-CP and JDI-CP, which rebuilds exchangeability between calibration data and test data and enables coverage control.

Hope these explanations are acceptable for you!

Q1:Concerns focus on Assumptions 3.1 and 3.2, assuming detection with FNR=0. Assumption 3.2, crucial for finite length prediction interval, also excludes joint feature space outliers. Figure 2 shows FDR is around 0.4 , indicating unmet assumptions.

Thanks for your valuable questions! We acknowledge the imperfection of Assumption 3.1 and explain as follows.

• Assumption 3.1(FNR=0), also used in Liu et al.(2022) and Wasserman&Roeder(2009), can be met by choosing a small detection threshold in practice and is theoretically essential for model-free coverage guarantee. When Assumption 3.1 doesn't hold, we derived a new bound showing the coverage gap depends on the total variation between processed calibration and test data (see Reviewer vdr2’s Q1 due to space limit). We also added experiments by varying DDC detection thresholds $\sqrt{\chi_{1,p}^{2}}$ (adjusting $p$) based on Setting C in Appendix D.2. When FNR$\neq$0, our method can still approximately control the coverage. |$p$|0.5|0.7|0.9|0.99| |-|-|-|-|-| |FNR |0|0.005|0.008|0.013| |FDR|0.793|0.669|0.340|0.035| |PDI coverage|0.909|0.907|0.901|0.902| |JDI coverage|0.904|0.904|0.899|0.895|

• We'd like to clarify the primary purpose of Assumption 3.2 is to ensure exchangeability of $\\{\hat{\cal{O}}\_i\cup\mathcal{O}^*\\}\_{i=n_0}^{n+1}$(see Lemma 3.3), not for finite length prediction intervals (PIs). In Section 5, we also provide theoretical coverage results when Assumption 3.2 doesn’t hold. In simulations, we also used detection methods that unsatisfy Assumption 3.2, such as DDC in Figure 3 and cellMCD in Figure 4(b), which don't lead to infinitely wide PIs.

• Actually, Assumption 3.1 requires FNR equal to zero, while the frequency of $\tilde{\cal{T}}\_{n+1}=\hat{\cal{T}}\_{n+1}=\varnothing$ in Figure 2 means the false discovery (positive) rate (FDR) is zero.

Q2:Why weighted conformal prediction is less competitive?

WCP requires precise likelihood ratio estimation, but arbitrary cellwise outliers in our setting render this impossible. Figures 3 and 11 highlight this limitation, showing WCP fails to meet target coverage.

Q3:Localized conformal prediction as a special form of WCP is not discussed.

Thanks for your advice and we apologize for overlooking these references.

• LCP controls conditional coverage under i.i.d. test and calibration data, differing from our focus on distribution shift caused by cellwise outliers in test data, which LCP cannot cope with. We’ll add a detailed discussion of LCP and these references in the revision.
• We also compare our method with baseLCP, calLCP (Guan,2023), and RLCP (Hore&Barber, 2023) through experiments. Clean data follows Section 5.1.1 of Hore&Barber(2023), and cellwise outliers are generated via Equation (10) in our paper with $\epsilon=0.03$. Results show LCP methods fail to provide informative PIs. ||Bandwidth|0.1|0.2|0.4|0.8|1.6| |-|-|-|-|-|-|-| |baseLCP|Coverage|0.90|0.90|0.90|0.90|0.95| ||Infinite PI%|0.90|0.90|0.90|0.90|0.81| |calLCP|Coverage|0.94|0.94|0.94|0.94|0.93| ||Infinite PI%|0.88|0.88|0.88|0.87|0.66| |RLCP|Coverage|0.90|0.90|0.90|0.90|0.90| ||Infinite PI%|0.90|0.90|0.90|0.86|0.48| |PDI|Coverage|0.90||||| || Infinite PI%|0||||| |JDI|Coverage|0.89||||| ||Infinite PI%|0|||||

Q4:The form of non-conformity score is restricted to absolute deviation.

For simplicity, we use the absolute residual score, but our method supports various non-conformity scores like CQR score. Please refer to Reviewer vdr2's Q2 for a discussion about CQR.

Q5:Theorem 4.3's coverage lower bound may be vacuous with large calibration size.

After checking our proof, we found the expansion to $\max_{i= n_0,\ldots,n}E_i$ is unnecessary and the last inequality in (B.10) (Appendix B.3) should be removed. The modified coverage gap is given by:\mathbb{P}\left\\{\hat{q}\_{\alpha}^+(\\{R_i^*-S_{\hat{\mu}}\cdot E_i\cdot|\tilde{\cal{O}}\_{n+1}\setminus\mathcal{O}^*|\\}\_{i=n_0}^n)<R_{n+1}^*\leq\hat{q}\_{\alpha}^+(\\{R_i^*+S_{\hat{\mu}}\cdot E_i\cdot|\tilde{\cal{O}}\_{n+1}\setminus\cal{O}^*|\\}\_{i=n_0}^n)\right\\},which will not be vacuous for large calibration size.

Thank you for catching this key technical nuance!

Q6:Dataset with real outlier will be more convincing.

To further demonstrate robustness, we test our method on a riboflavin gene expression dataset(Liu et al.,2022) with confirmed cellwise outliers. Our method maintain coverage above the target $1-\alpha=0.9$ while LCP methods ($h=0.1$) fail to provide meaningful PIs.

Q7:Are other contamination models worth considering?

The Tukey-Huber Contamination Model generates casewise outliers, assuming most samples follow a target distribution $F$, while the others follow an arbitrary distribution $H$:$X\sim(1-\epsilon)F+\epsilon H,$where $\epsilon\in[0,1)$ is the contamination ratio.

Q1: The validity of Assumption 3.1 (sure detection) depends on quality of detection, which may be restrictive in practice.

Thanks for your valuable question! We acknowledge the imperfection of Assumption 3.1 and make the following explanations.

• Sure detection/screening conditions are commonly used in existing works (Wasserman&Roeder,2009; Liu et al.,2022) to ensure all relevant variables are retained. We adopt a similar condition in Assumption 3.1 for model-free coverage guarantee.
• Assumption 3.1 is essential for meaningful prediction intervals (PIs): if outliers persist in $\check{X}\_{n+1}^{\rm{DI}}$, Theorem 3.5 shows PIs can become infinitely wide in expectation.
• Violating Assumption 3.1 impacts our method by introducing a total variation coverage gap: if outliers persist in the test point $\tilde{X}\_{n+1}$ after detection, Lemma 3.3 fails due to the exchangeability between $(\check{X}\_{n+1}^{\rm{DI}},Y_{n+1})$ and $\\{(\check{X}\_i^*,Y_{i})\\}\_{i=n_0}^n$ is broken, where $\\{\check{X}\_i^*\\}\_{i=n_0}^n$ are the ODI features. At this point, there is a total variation coverage gap of the PI:\mathbb{P}\\{Y_{n+1}\in\hat{C}^{\rm{ODI}}(\tilde{X}\_{n+1})\\}=\mathbb{P}\left\\{|\hat{\mu}(\check{X}\_{n+1}^{\rm{DI}})-Y_{n+1}|\le\hat{q}\_{\alpha}^+(\\{R_i^*\\}\_{i=n_0}^n)\right\\}\ge 1-\alpha-\frac{1}{n-n_0+1}\sum_{i=n_0}^n d_{\rm{TV}}(\check{X}\_{n+1}^{\rm{DI}},\check{X}\_i^*).The impact is mild if $d_{\rm{TV}}(\check{X}_{n+1}^{\rm{DI}}, \check{X}_i^*)$ is small; similar bounds apply to PDI-CP and JDI-CP.
• In practice, Assumption 3.1 (TPR=1) can be satisfied when the detection threshold is small; our method still maintains target coverage in simulations and real examples even if some outliers were not completely detected. For your convenience, we also added a new experiment under different DDC detection thresholds $\sqrt{\chi\_{1,p}^{2}}$ (varying $p$) based on Setting C in Appendix D.2, which shows Assumption 3.1 is exactly satisfied (TPR=1) at $p=0.5$. When TPR<1, our method can still achieve approximate control of coverage. |$p$|0.5|0.7|0.9|0.99| |-|-|-|-|-| |TPR|1|0.995|0.992|0.987| |FDR|0.793|0.669|0.340|0.035| |PDI coverage|0.909|0.907|0.901|0.902| |JDI coverage|0.904|0.904|0.899|0.895|

[1] Wasserman, L. and Roeder, K. High dimensional variable selection. The Annals of Statistics, 2009.

[2] Liu, Y., Ren, H., Guo, X., Zhou, Q., and Zou, C. Cellwise outlier detection with false discovery rate control. Canadian Journal of Statistics, 2022.

Q2: Would the conclusion in Theorem 3.5 hold with other adaptive prediction sets like conformalized quantile regression?

Thank you for your question! We prove a similar result for the CQR score, and add new simulation results based on CQR.

• The PI constructed from CQR is $\hat{C}(X)=[\hat{f}^{lo}(X)-\hat{q}\_n,\hat{f}^{up}(X)+\hat{q}\_n]$, where $\hat{f}^{lo}$ and $\hat{f}^{up}$ are the lower and upper quantile regression models, and $\hat{q}\_n$ is the quantile of empirical distribution of CQR computed on the calibration set.
• Following the proof of Theorem 3.5 in Appendix B.2, $Y_i=X_{i,1}+X_{i,2}$ where $X_{i,1},X_{i,2}\sim\rm{Uniform}([0,1])$ for $i\in[n+1]$. Suppose $\hat{f}^{lo}(x)=\beta_1^{lo}x_1+\beta_2^{lo}x_2$ where $\beta_2^{lo}\neq0$, and the test point $\tilde{X}\_{n+1}=(X_{n+1,1},Z_{n+1,2})^{\top}$ where$Z_{n+1,2}=\frac{M+1}{\beta_2^{lo}} \mathbb{1}\\{\beta_1^{lo}\geq 1\\}+\frac{M+2}{\beta_2^{lo}}\mathbb{1}\\{0<\beta_1^{lo}<1\\}+\frac{M-\beta_1^{lo}+2}{\beta_2^{lo}}\mathbb{1}\\{\beta_1^{lo}\leq 0\\}$for some large positive value $M$. If $\check{X}\_{n+1}^{\rm{DI}}$ still contains $Z_{n+1,2}$ and $\hat{C}(\tilde{X}\_{n+1})$ covers the true label, we have$\max\\{\hat{f}^{lo}(\check{X}\_{n+1}^{\rm{DI}})-Y_{n+1},Y_{n+1}-\hat{f}^{up}(\check{X}\_{n+1}^{\rm{DI}})\\}\geq\hat{f}^{lo}(\check{X}\_{n+1}^{\rm{DI}})-Y_{n+1}\geq M,$which means $\mathbb{P}(\hat{q}\_n\geq M)\geq\mathbb{P}(Y_{n+1}\in\hat{C}(\tilde{X}\_{n+1}))\geq1-\alpha$.
• We also conduct an experiment using CQR to construct PIs, where the Baseline used in our simulation can be considered as the optimal method for constructing split conformal PI for cellwise outlier, which masks calibration and test features by $\cal{O}^*$. This experiment will be added in future revisions. ||Baseline|ODI|PDI|JDI| |-|-|-|-|-| |Coverage| 0.905|0.902|0.900|0.885| |Length|4.366|4.291|4.282|5.544|

Q3: How does the contamination rate affect the length of the prediction sets?

• Figure 6 in Section 6.3 shows the coverage and length across cell contamination probabilities with DDC and Mean Imputation, while Figures 12-13 in Appendix D.4 display results for kNN and MICE imputation.
• Results indicate the length of our method remains stable across varying contamination rates, whereas that of Baseline increases with higher contamination. Figure 3 demonstrates our method's competitive length and target coverage, surpassing the classical WCP method.

Hope these explanations can ease your doubts!

Q1: Assumption 3.1 is impractical. How does coverage degrade when detection is imperfect?

Thank you for your insightful question!

• We acknowledge the imperfection of Assumption 3.1 and have obtained a new coverage gap bound in total variation: if there are still outliers in test point $\tilde{X}\_{n+1}$ after the detection procedure, Lemma 3.3 will not hold because the exchangeability between $(\check{X}\_{n+1}^{\rm{DI}},Y_{n+1})$ and $\\{(\check{X}\_i^*,Y_{i})\\}\_{i=n_0}^n$ is broken, where $\\{\check{X}\_i^*\\}\_{i=n_0}^n$ are the ODI features. At this point, there is a total variation coverage gap of the prediction interval (PI):\mathbb{P}\\{Y_{n+1}\in\hat{C}^{\rm{ODI}}(\tilde{X}\_{n+1})\\}=\mathbb{P}\left\\{|\hat{\mu}(\check{X}\_{n+1}^{\rm{DI}})-Y_{n+1}|\le\hat{q}\_{\alpha}^+(\\{R_i^*\\}\_{i=n_0}^n)\right\\}\ge 1-\alpha-\frac{1}{n-n_0+1}\sum_{i=n_0}^n d_{\rm{TV}}(\check{X}\_{n+1}^{\rm{DI}},\check{X}\_i^*).Notice that, we can still have a approximate coverage control if $d_{\mathrm{TV}}(\check{X}_{n+1}^{\mathrm{DI}}, \check{X}_i^*)$ is small. A similar bound can be obtained for PDI-CP and JDI-CP and we will add these results in the revision.
• In addition, we add a new experiment to show the influence of imperfect detection. Please see Q6 for details.

Q2: If choosing a larger detection threshold ensures Assumption 3.1, doesn’t this break exchangeability? How does this affect empirical coverage?

• According to the construction, we know that false positives do not break the data exchangeability in ODI-CP and JDI-CP. Since we use $\tilde{\cal{O}}\_{n+1}$ to approximate $\cal{O}^*$ in PDI-CP, it will break the exchangeability between processed test data and calibration data because $\tilde{\cal{O}}\_{n+1}$ depends only on test data. As we stated in Theorem 4.3, the coverage gap is affected by the number of false discoveries $|\tilde{\cal{O}}\_{n+1}\setminus\cal{O}^*|$.
• In Appendix D.3, we summarized the empirical false discovery rate (FDR) and true positive rate (TPR) of the detection methods in our simulation. We will include this discussion in future revision, using the table from Q6 as an illustrative example. As the threshold decreases, the number of discoveries increases, leading to a higher FDR. Notably, the table demonstrates the empirical coverage of our method remains robust to variations in FDR.

Q3: Theorem 4.3 and 4.4 assume imputation doesn't change the residual distribution.

We apologize for any confusion. Here, $F_{R}$ denotes the distribution function of the ODI-CP residuals $\\{R_i^*\\}\_{i=n_0}^n$, not the residuals from the raw calibration data before the DI procedure. Thus, our method doesn't assume "imputation preserves the residual distribution." We appreciate your attention and will clarify this notation in the revision.

Q4: Have you tested other imputation strategies? How coverage varies under different imputation and detection methods?

• In addition to Mean Imputation, we evaluated two other imputation methods in Section 6.2: kNN and MICE. Figure 5 shows our method maintains robust empirical coverage across all imputation strategies.
• Our analysis in Sections 6.1-6.2 compares coverage and length of PI across various detection and imputation methods. Figures 4-5 show our method consistently maintains robust coverage control across all configurations with stable interval lengths.

Q5: Direct-ODI and Direct-PDI contradict the main method’s justification.

We apologize for any ambiguity and would like to clarify:

• Direct-ODI and Direct-PDI in Appendix A.1 are naive combinations of DI with CP, which are not our proposed methods.
• Figure 8 shows Direct-PDI fails to maintain proper coverage while our PDI-CP successfully achieves target coverage. This empirical evidence confirms simply combining DI with CP (without our proposed modifications) cannot guarantee valid coverage.

We appreciate your careful review and will revise Appendix A.1 for clarity.

Q6: The paper does not study sensitivity to different detection thresholds and imputation choices.

Regarding the sensitivity of our method:

1. Imputation methods: Figure 5 in Section 6.2 demonstrates our method maintains robust performance across different imputation choices.
2. Detection thresholds: We conduct additional experiments by varying the DDC detection threshold $\sqrt{\chi_{1,p}^2}$ (adjusting $p$) based on Setting C in Appendix D.2. As shown in the table, our method demonstrates robust coverage performance when threshold changes. |$p$|0.5|0.7|0.9|0.99| |-|-|-|-|-| |FNR |0|0.005|0.008|0.013| |FDR|0.793|0.669|0.340|0.035| |PDI coverage|0.909|0.907|0.901|0.902| |JDI coverage|0.904|0.904|0.899|0.895|

Please let us know if you would like any additional details.

Q7: Could you provide a math notation table?

Thanks for your helpful advice. We will add a math notation table to the appendix of the revision.

Hope these explanations are acceptable for you!