Conformal Prediction via Regression-as-Classification

We learn nonsmooth distribution but other papers don’t.

If you look at Figure 4 in the appendix, we learn pretty smooth distributions out of the box. Interpolation is not used for this. It is only used to make a discrete density function into a continuous one. Our interpolation is a simple linear interpolation, which does no extra smoothing on top. In fact, the smoothness is a strength of our method. If you look at CHR, they learn very spiky distributions.

You are sensitive to hyperparameter tuning.

No tuning was done at all. We hold $K=50$ bins and $\tau=0.2$ across all experiments which span 16 different datasets with widely varying distribution shapes. If tuning was done, we would expect even better results. We will add ablations in the final version of the paper to investigate how our results are dependent on these hyperparameters.

This won’t work on unimodal distributions.

This is not correct. Many of the data such as MEPS are unimodal and we get the best results out of all methods.

There are unclear aspects of the experiments

We repeat the experiments over different seeds and report the average mean over all seeds. The value you are referring to in Table 3 is the standard error (we have clarified this in the uploaded version). Since we use a large proportion for the test set, this suggests that our method performs well across numerous seeds, and small standard errors are reasonable.

There is a tradeoff between coverage and length

We have fixed all experiments at coverage level of $90\%$. We have specified this in newly uploaded versions. This is standard in the CP literature.

How does this differ from Stewart et al.?

Stewart et. al focuses on learning conditional expectation while we focus on conditional density. Using their loss function naively performs worse than our smoothened version of the loss function according to the ablations. We will clarify better in the final version that Stewart et al’s loss is included in the ablations and did not work. Our loss function is tailored to our specific problem, for example with smoothing.

We thank this reviewer for their thoughtful review. We believe that the reviewer has several confusions regarding some of the concepts regarding conformal prediction methodologies relevant to our work. We hope this response clears up all of the confusions and we will greatly appreciate an increase in score!

the main contribution is proposing a new nonparametric method for modelling the conditional density. However, I don't see any contribution w.r.t. conformal prediction

Most papers on CP focus on the choice of score function. Depending if one want to consider heteroscedasticity, bimodality, skewness etc. Our contribution single handedly covers these case contrarily to DCP, CQR, CHR.

experimental and theoretical comparison with other methods for estimating the conditional density

The goal of our work is to improve conformal prediction, rather than proposing a better method for conditional density estimation (which is a challenging problem in itself). This is why we use best performing CP methods as baselines. Showing that better conditional density leads to better CP method would be an interesting direction, but would lead to a different paper.

Why not add additional baselines such as mean-variance networks?

These methods of Conditional density estimation have not been used in CP literature before. While they are reasonable ways to estimate conditional density, our method is very simple. There are many ways to do conditional density estimation like you mentioned (see Rothfuss et al.). However, these are not any simpler and are more involved than our technique. Moreover, the existing CP baselines do not compare with these techniques either. We do compare with methods like KDE or Histogram Regression which are both conditional density estimators and the latter uses neural networks. Relative to relevant papers such as CHR, we compare to the same amount or more baselines, so we believe our experimental setup is sufficient.

The standard split conformal prediction, which is arguably not the most useful type of evaluation for heteroscedastic regression problems.

We are afraid that the reviewer is confusing split conformal methods (which is an alternative to the highly expensive Full conformal methods) and the choice of the score function that account for heteroscedasticity. For example, CQR or DCP explicitly account for heteroscedasticity and comes with both Split and Full version, where the former is preferred for computational efficiency.

Conditional coverage guarantees need to be analyzed in such cases. This can be established via Mondrian conformal prediction

There are several types of conditional guarantees Mondrian CP are designed to be conditional on the class of the input object. In this case, the p-values function itself is modified by including a specific taxonomy (e.g. provided by a classifier or clustering algorithm). These are available in both the Split CP version and the Full CP version (or even other cross-conformal alternatives). Being split or not is about computational efficiency. Heteroscedasticity or other properties are handled by the choice of the conformity score function.

We would like to add that CQR provides a benchmark against variance-based normalization and achieved superior performance.

By conformalizing our density estimation leveraging a (Mondrian or Venn) taxonomy, our method trivially accounts for class conditionality since this is independent of the choice of score function (it is just about the choice of p-value function). For a review of these concepts, please refer to Chapter 4.6.1 of the book 'Algorithmic Learning in a Random World', second edition, by Vovk, Gammerman, and Shafer.

Moreover, most of the baselines in regression CP focus on split conformal prediction. It is a standard technique in the CP literature. We believe it is unfair to reject a paper for using a standard evaluation setup.

References

Jonas Rothfuss, Fabio Ferreira, Simon Walther, and Maxim Ulrich. Conditional density estimation with neural networks: Best practices and benchmarks. arXiv preprint arXiv:1903.00954, 2019.

Dear Reviewer ZE11,

The discussion period is about to conclude, and despite your did not engage with us yet, we are happy that you enjoyed reading our paper and find it convincing for regression problems with multimodal distributions.

Unfortunately, you have the lowest score among the reviewers, and the highest confidence.

We have diligently replied to all your concerns and several confusions, in detail in our rebuttal, and we would be delighted to provide any further clarifications. We believe that not all the limitations you have expressed are well-founded, and this certainly explains your rating. If our response is not satisfactory, please let us know asap, and we will be happy to add further details in the short time available.

Best regards,

Authors

We thank this reviewer for their thoughtful review. We clarify your weaknesses and questions here. If you find this sufficient, we would greatly appreciate an increase in score!

Weaknesses

What is the role of discretization in this work?

Several works in the literature, including Stewart et al. mention that the classification problem tends to lead to stabler training than regression. Stewart et al. empirically demonstrates that R2C estimates the conditional expectation better than regression alone. The reason that this is the case is an open theoretical problem. However, intuitively, the learning problem under R2C is an unconstrained learning problem with only $K$ outputs whereas methods like CHR have to have 1000 output nodes and average among them to achieve smoothness.

Questions

Is the smoothness-enforcing penalty new?

Smoothing-enforcing penalty has existed in other contexts such as Variational Learning. However, in this context of learning conditional densities using Neural Networks, it is new to the best of our knowledge.

How were K and Tau chosen? Is part of the data used for training the conditional distribution?

$K$, $p$, and $\tau$ were fixed at $50$, $.5$, and $.2$ for all experiments. We didn't change that or tune that as that would be unfair to other methods. Tuning these would likely improve the results of our method.

Why should the entropy regularization be expected to be good at learning bi-modal distributions?

It’s true. It is possible that reducing entropy can sometimes help, but as a general tool, entropy regularization helps overall. Empirically, our structure still works well for bimodal distributions.

Is part of the data used for training the conditional distribution?

We are not sure we understand the question. As usual, the dataset was split in two (training and calibration), and indeed, the training set was used to learn the conditional distribution.

What is the correlation between the test errors and the corresponding intervals?

Thanks for your suggestion. We will try to illustrate this plot for a couple of benchmarks for the camera ready version.

Is the proposed approach equivalent to training a parameterized conformity function (see for example in [2] or [3])?

We are very different from the Einbinder paper since their loss function tries to ensure its scores form a smooth CDF during training and is nondifferentiable. We do no such thing and use much simpler differentiable methods. The second Colombo paper proposes how to transform conformity scores to make it more suitable for APS.

Thanks for the additional reference, we will add [1] in our related work section.

References

Lawrence Stewart, Francis Bach, Quentin Berthet, and Jean-Philippe Vert. Regression as classifi- cation: Influence of task formulation on neural network features. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2023.

We thank the reviewer for their thoughtful review. We clarify the weaknesses and questions here. If you find this sufficient, we would greatly appreciate an increase in score!

Weaknesses

Can you connect more to ordinal regression?

Thanks for the suggestion. We completely agree with you that there are strong connections with ordinal regression. We noted on this connection with citations to Weigend & Srivastava (1995) and Diaz & Marathe (2019). More recently, in computer vision, also some investigation is done to understand the effectiveness of such methods (https://arxiv.org/pdf/2301.08915.pdf). We have also added several relevant citations to the ordinal regression literature in the related works in the newly uploaded paper.

Questions

How does this paper compare to conformal ordinal classification papers?

We have added these two new works to the related works in the newly uploaded document. “Conformal Risk Control for Ordinal Classification” talks about different risk types (analogous to coverage) for ordinal classification while we discuss different score functions when coverage is your loss function. “Improving Trustworthiness of AI Disease Severity Rating in Medical Imaging with Ordinal Conformal Prediction Sets” talks about how you can adapt APS style conformal sets to work with ordinal structure, which was later used in the CHR paper.

Why do we focus on modal values?

A key contribution of our paper is its versatility across different distributions via building a probability density function. Consider a bimodal label distribution; the median and mean would be very unlikely answers since they lie in the middle of two valleys. To account for such distributions, we found modal values to be a better fit.

How many predictions are not intervals?

We can do this in the final version of the paper.

Can you do a full coverage curve?

We have fixed the broken reference in the appendix in the newly uploaded version. Moreover, we can work on the full coverage curve for the camera-ready version.

Can this be extended to multi-variate output?

Our method can be adapted to multivariate outputs, and this is an exciting direction for future work. There is a direct naive approach, which scales like $K^d$, where $K$ is the number of bins and $d$ the dimension of the output. Developing more efficient ways which do not scale exponentially with the dimension of the output (but perhaps like $O(K*d)$) is not straightforward, but can be an interesting direction for future work. Thanks for the question!

Raul Diaz and Amit Marathe. Soft labels for ordinal regression. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.

Andreas S Weigend and Ashok N Srivastava. Predicting conditional probability distributions: A connectionist approach. International Journal of Neural Systems, 6(02):109–118, 1995.

Lawrence Stewart, Francis Bach, Quentin Berthet, and Jean-Philippe Vert. Regression as classifi- cation: Influence of task formulation on neural network features. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2023

Jonas Rothfuss, Fabio Ferreira, Simon Walther, and Maxim Ulrich. Conditional density estimation with neural networks: Best practices and benchmarks. arXiv preprint arXiv:1903.00954, 2019.

We thank this reviewer for their thoughtful review. We clarify your weaknesses and questions here. If you find this sufficient, we would greatly appreciate an increase in score.

Weaknesses

The proposed method was not easy to understand before seeing Algorithm 1

Although we understand your concern, we do not want to assume that everyone is familiar with conformal prediction and its connection to our estimated model. Thus, we first expand on what motivates our proposal and connect it to the existing literature before diving into more technical details. We will consider some of your suggestions to clarify the final version of the paper.

Table 1 is very hard to understand

Thanks, we agree with your points. We have fixed these in the newly uploaded version. We have specified that the brackets include the “Standard Error” over the experiments. We also provide definitions for length and coverage. We believed the graphs and terms were standard enough in the literature. To avoid confusion, we will make sure to be more specific in the final submission.

Most results are not reported with error bars making them less reliable. For instance, in Table 1, CHR does worse than CQR but it should not?

We run all of our experiments by averaging over the test set and over 5 different seeds. With every different seed, we retrain the model. The standard error of the results over the different seeds is shown in the brackets. Given that we are running experiments across significantly more datasets than the ones used in the CHR and CQR papers, it is reasonable that upon seeing new datasets, CHR does worse than CQR in some cases, and in some cases, CHR outperforms CQR. For both baselines, we use code written by the authors themselves and do not edit them at all. We report the numbers given by their open-sourced code explicitly.

Have you considered a baseline where you learn a regressor directly using the same neural network architecture, treat that regressor as $\bar{q}$, and apply the same conformalization?

From what we are aware of, learning conditional density for regression problem using neural network require more involved methods with specific prior such as Gaussian mixture parametrization or Bayesian network (see Rothfuss et al.). Here, we are interested in computing a confidence set instead of the harder problem of getting accurate density estimation. As such, we propose to bypass those difficulties by converting the problem into a multiclassification problem where the parametrization is simpler. As you stated above, we rely on “a traditional neural network with softmax output, which everyone understands”. Also, we note that CHR uses a histogram powered by a neural network to estimate the conditional density in a regressor setting, as you suggest. On top of that, we use 6 baseline effective conformal methods in our benchmark. Of course, we can always add more, but we believe that our numerical experiments are revealing enough.

Which aspects of the R2C method are novel?

As is presented, this method of R2C for conformal prediction has not been done before. Smoothing in different contexts has been proposed, such as in ordinal regression, but not in R2C literature. However, Stewart et al. is the standard R2C loss function, and using that directly does not work (see our ablation studies). We will make it clearer that the loss function from Stewart et al. is the one studied in the ablations. They focus on conditional expectation, whereas our loss focuses on conditional density. Our smoothened loss function works better according to the ablations. This use of R2C to learn conditional density alongside smoothing is novel to this paper.

Questions

What is the exact formulation of linear interpolation

We have added this in the newly uploaded version. We clarify it as below (note here the $\hat{y_k}$ and $\hat{y_{k+1}}$ are chosen such that $\hat{y_k} < y < \hat{y_{k+1}}$:

$\bar{q}(y \mid x) =$ $\frac{(\hat{y_{k + 1}} - y) q(\hat{y_k} \mid x)}{\hat{y_{k + 1}} - \hat{y_k}}$ $+ \frac{(y - \hat{y_{k}}) q(\hat{y_{k+1}} \mid x)}{\hat{y_{k + 1}} - \hat{y_k}}$

Why use R2C versus standard regression?

Several techniques exist to use regression techniques to learn conditional densities, as shown in our baselines. However, these methods can suffer from poor stability during training. For example, see Figure 4, and you will see that CHR’s technique using regressors is not smooth. Our loss function results in smoother and more stable densities. In fact, the Stewart et al. paper demonstrates empirically where R2C outperforms regression techniques for estimating the conditional expectation. Why this is the case theoretically is still an open research problem, as far as we know.

Dear Reviewer RG4V,

We appreciate your review and would be grateful if you could let us know if our response resolved the issues you raised. We regret that the review period is ending and we have not had the opportunity to discuss your questions further with you. Consequently, we are concerned that your rating is still below the acceptance threshold.

If our response was satisfactory, please leave a comment and update your score accordingly. We would be pleased to provide further explanations if desired.

Best,

Authors