Regression Conformal Prediction under Bias

• Elementary Theoretical Findings: While the theoretical contributions are sound, the results may be considered elementary. The core findings, such as the impact of bias on symmetric intervals and the lack of bias influence on asymmetric intervals, are straightforward and follow from the basic principles of CP. Though helpful, these insights do not introduce particularly advanced or novel theoretical complexity, which may limit the paper's appeal to a more specialized audience seeking more profound theoretical innovations.
• Simplified Bias Assumptions: The assumption of bias as a constant noise term following the same global distribution across all predictions may not adequately capture more complex forms of bias present in real-world data, such as feature-specific or covariate-dependent biases. The paper also mentions on lines 089-094 different reasons for these biases. However, they result in way more complex biases.
• Limited Discussion on More Complex Non-Conformity Scores: While the paper acknowledges that more complex non-conformity scores may require different approaches, it does not explore these in-depth, potentially limiting the generality of the findings for more advanced CP methods.
• Limited Synthetic Experiment Settings: The paper only evaluates experiments where $n$ is large, and the data is generated from symmetrical distributions. However, this is problematic because the strong point of the symmetric adjustment is that it can better leverage asymmetries (skewness) of the aleatoric uncertainty distribution (noise distribution on the label).
• Wrong Claims: Your statement on line 266 is incorrect. When the number of samples is large and no bias is present, the length of prediction intervals generated by a symmetric adjustment is approximately equal to the ones generated from asymmetric adjustments. See the above point.
• Section 3.1 is too bloated: Given your bias, you could just take the mean of the calibration set's error, to retrieve it.

Inconsistent Notations and presentation: The paper suffers from unclear and inconsistent notations. For instance, in equation (1), it's not explicit which bias is being addressed --- it is not clear how the randomness in the ground truth is being dealt with. Additionally, \hat{Y_i^b} is ambiguously defined; it is not clear whether these quantities are scalars or they are allowed to be a set of values. For instance, in the first paragraph of Section 3 the Y_i is treated as scalars and in the second paragraph, they use Y_i = {Y_{ij}} as a vector. Besides, the score definition proposed by Romano et al.[1] is presented inaccurately, which raises concerns about the foundation of the theoretical analyses. The overall readability of the paper is not good.

Limited Insight from Corollary: While Theorems 1 and 2 seems a bit straightforward and is a direct consequence of the linear assumptions. The utility of Corollary 3.1 is not clear, the RHS of eq. (6) is independent on b, and the LHS is dependent on b? It is not clear how the authors are setting L_{asymb} \leq L_{sym} in the proof.

Missing Theoretical Justification for the Algorithm: The authors do not provide a theorem guaranteeing that the proposed algorithm retains the validity of the coverage guarantee for CP intervals. This omission is significant as it leaves a gap in understanding whether the method meets one of CP's fundamental requirements.

Lack of Coverage Probability in Simulations: The simulations fail to report the coverage probability, which is a critical metric for evaluating the reliability of proposed algorithm. This weakens the experimental validation of the proposed methods.

Comparison with related work: There are some related work which also aims to reduce the length of the conformal intervals [2,3]. It would be nice to compare the contribution of this work with these two related works.

[1.] Romano, Yaniv, Evan Patterson, and Emmanuel Candes. "Conformalized quantile regression." Advances in neural information processing systems 32 (2019).

[2.] Xie, Ran, Rina Foygel Barber, and Emmanuel J. Candès. "Boosted Conformal Prediction Intervals." arXiv preprint arXiv:2406.07449 (2024).

[3.] Liang, Ruiting, Wanrong Zhu, and Rina Foygel Barber. "Conformal prediction after efficiency-oriented model selection." arXiv preprint arXiv:2408.07066 (2024).

• Non-symmetric CP is not new. The paper's contribution would be better introduced by adding an intuitive explanation of why asymmetric adjustments have been theorized to yield longer interval lengths as a consequence of stronger guarantees (Romano et al.,2019) but has also been empirically observed that asymmetric adjustments yield tighter intervals than symmetric ones.
• The authors focus on the setup where comparable bias is present in the calibration and test samples. They should comment on whether it would be more efficient to 1) improve the underlying point-prediction model or 2) consider more flexible conformity scores (e.g. adding and training a free shifting term to the conformity scores).
• Asymmetric intervals can be defined by splitting the calibration samples according to the residual signs and then evaluating two sample quantiles. It would be helpful to see a theoretical comparison of the gap between the proposed method and such a naive approach as a function of the size of the calibration set.

The paper's main theories (Theorem 2 and 3) are under highly stylized models and the statement lacks rigor (please correct me if I misunderstood anything). For example, Theorem 2 is stated for prediction intervals in the generic form of Eq. (2), but it appears that the proof only focuses on the case of residual errors and CQR. For another example, the proof of Theorem 2 seems to use the fact that $f_{\text{hi}}(\hat Y_i^{b--}) = f_{\text{hi}}(\hat Y_i^{0})+b^{--}$, which is not stated as an assumption. The theorem also appears to assume that the bias is sufficiently large in magnitude.

I am also not following the form of CQR used in this paper; in my understanding, CQR fits conditional quantiles of $Y$ given $X$ by minimizing the pin-ball loss, and the resulting fitted conditional quantile function operates $X_i$ instead of the point prediction of $\hat Y_i$ (unless the quantile is obtained in a specific form centered around a point prediction). I am also confused by the description of CQR in lines 194-196 --- what is $n_s$, what is the set of samples $\hat Y^b_i$ and how are they generated. If the prediction is deterministic, does this reduce to the residual error case? Please let me know if I misunderstood anything.