Utility-Directed Conformal Prediction: A Decision-Aware Framework for Actionable Uncertainty Quantification

Dear Reviewer yMsv,

We thank the reviewer for their kind words and enthusiasm for our paper. We address your concerns below.

Expand Related Work

The discussion of related work is insufficient, focusing exclusively on conformal prediction literature while neglecting other uncertainty quantification methods

We expanded the discussion in the related work section to include uncertainty quantification (UQ) methods as suggested, and explained why conformal prediction is particularly relevant to our use case. Additionally, we included an extra baseline comparison against variational inference (a Bayesian method) and analyzed its performance relative to our approach.

Additional comparisons with other UQ methods

Building on the previous point, the paper lacks comparisons with existing methods… [1] [2] [3] The authors focus on split conformal prediction, but there are other conformal prediction methods available [1,2]. Can the proposed modifications be applied to these other methods, or are they specific to split conformal prediction?

We would like to highlight that while conformal risk control [1] addresses a similar problem, it targets a different setup. CRC allows for controlling the expected value of a set function to a predefined level; if the loss function is the characteristic function of the event that defines coverage (the true label is in the predicted set), then the user will recover true statistical coverage. Thus, CRC allows control over some utility functions but at the expense of coverage guarantees. In contrast, our method is designed to provide both. Moreover, CRC imposes an assumption on the losses it studies—they must monotonically decrease with set size—rendering CRC incompatible with our approach.

The regression setting, where methods like quantile regression might be relevant, presents an interesting extension to our work. However, for now, it lies outside the scope of the topics explored in this draft.

Readability and experimental details

Some sections, such as Figure 2, require improved clarity for better readability. The experimental details required to reproduce the results are missing. Specifically, classifiers trained on these datasets can be sensitive to variations in training hyperparameters.

Thank you for pointing this out! We have added the missing experimental details to Appendix Section B and have improved the readability of Figure 2 to ensure clarity. improved the readability of Figure 2 to ensure clarity. We are happy to make further clarifications if this remains a concern!

Adaptability and conditional coverage analysis

the lack of adaptiveness in split conformal prediction is a known issue [1]. How does the proposed method affect conditional coverage? Is the decision loss minimized uniformly across all subgroups, or are there significant discrepancies? Since the authors emphasize medical applications, it seems essential to assess the method's performance on specific subgroups in such contexts.

To address these concerns, I recommend that the authors report the average size of the conformal sets in their experiments.

Set-size as a function of entropy of the distribution of labels is the standard to measure how adaptive a conformal based approach is. Hence, we incorporated an analysis of utility by set size in the appendix (and thank the reviewer for the suggestion, as lack of adaptivity is something that can render our method useless). The results demonstrate that our method continues to perform strongly while preserving adaptability. Specifically, our method produces larger sets for challenging examples (instances of $x$ with low-confidence labels), maintaining robustness across diverse scenarios. These results are detailed in Section 4 and appendix D.

Dear Reviewer arZB,

We thank the reviewer for their kind words and enthusiasm for our paper. We address your concerns below.

Framing of the paper

I find the framing of the paper as a decision-focused approach misleading and confusing. This is for two reasons. It hides the novelty of the paper, which is its focus on prediction sets with utility. It also makes it seems that the paper is focusing on decision-making

Following the suggestions made by the reviewer we have revised the Introduction section. Now, it should be more clear that from decision-focused learning we only draw inspiration as an example of how there are methods that inform decision problems (optimization problems) via machine learning predictions. It is now explicit that our method is not and end to end procedure to train a model both for accuracy and to maximize some utility function, but rather a conformal-based post processing that provides sets with higher utility and marginal coverage. In order to avoid extra confusion we changed the title of the manuscript as well.

Lack of adaptability and conditional coverage analysis

My second main concern is that the current analysis of the prediction sets is a bit light. The experiments do not show how large the prediction sets are (which is a common utility metric in conformal prediction) nor their achieved marginal and conditional coverage.

We have improved our analysis by reporting two extra set of results in the updated version of the manuscript (see Section 4). The marginal coverage guarantees that we provide for our methods could be masking insidious behavior by over covering certain sets and leaving some other group undercover. This is why we now report statistical coverage by set size, thus allowing us to provide a proxy for conditional coverage across instances of different difficulty levels. Additionally, we report the average value of the softmax function for the true label as a function of set size. We use this metric as a measure of the adaptability derived from our conformalized heuristics; ideally, bigger sets should be associated with pairs $(y,x)$ that are hard for the model, i.e, for pairs $(x,y)$ where the value $p(y|x)$ is expected to be low (and thus lead to inaccurate prediction). The results obtained for both of these new metrics show that our method is robust and adaptive. Our method produces near optimal coverage across all set sizes while also showcasing the desired behavior of yielding larger prediction sets associated with hard examples.

Dear arZB,

Thank you for the feedback! We address your concerns below:

More in-depth evaluation of the approach (benchmark against an end-to-end procedure to generate prediction sets)

I do not give a higher score as I think that there could be a more in-depth evaluation of the approach. In particular, there could be a discussion and comparison with multi-label classification in which the ML model predicts a set directly instead of the current approach, which trains the ML model to predict a single class and then uses a conformal procedure to get a set. A multi-label approach would allow training the ML model end-to-end on the true utility of the predicted set. However, it would likely lose the coverage guarantee of the conformal procedure.

We thank the reviewer for their valuable suggestion. We agree that a comparison with a true decision-focused approach would enrich the discussion in the paper. To address this, we have added such a comparison in Appendix E. Unfortunately, the datasets within the paper focus on the single-labeled setting, which did not allow us to implement the exact approach suggested by the reviewer. Instead, we have tried our best to develop an approach that would be analogous (i.e., an end-to-end solution). We have updated our draft with this experiment and its results. In our experiments, under the constraint of achieving the same level of empirical coverage, our methods outperform the proposed end-to-end solution.

Minor Comments

Thank you for the feedback and catching these errors! We have made the following updates:

• Added all missing references (see Section 1.1).
• Clarified ambiguous sentence on line 177
• Fixed minor LaTex inconsistencies
• Updated our title and method name throughout the draft to address the comments on “decision” and “decision loss”

Unfortunately, we did not have enough space to move the short proofs to the main body yet. We plan to do so during camera ready.

We hope this addresses your concerns! Thanks again and please let us know if you have any other feedback.

We thank the reviewer for their thoughtful feedback and engagement. For the camera-ready version, we will remove Section E of the appendix. Regarding the computation of gradients, we used backpropagation on a loss function that combined the cross-entropy loss with the cost function of interest. Specifically, the cost function was calculated by predicting a prediction set at each step for every element.

Dear Reviewer 2y3o,

We thank the reviewer for their kind words and enthusiasm for our paper. We address your concerns below.

Comparison with alternative uncertainty methods like Bayesian inference

Lacks comparisons with alternative uncertainty methods like Bayesian inference, limiting context on the framework’s unique advantages

Thanks for the feedback! We have added comparisons against variational inference (a Bayesian method) (see appendix C). The details of how we utilize Bayesian estimates of $p(y|x)$ to construct the predicted sets are now included in the Supplementary Material (see Appendix C). Our new results show that variational methods not only fail to provide true statistical coverage, but further, are unable to generate high-utility sets. Even when we apply our conformalization approach on top of these estimates, the results are at best comparable to those from our existing methods. This demonstrates that the additional layer of uncertainty quantification provided by variational inference is superfluous for our use case of interest.

Hyperparameter tuning is expensive

Tuning for separable loss is computationally intensive; more efficient tuning methods would improve usability

We agree with the reviewer that the required tuning for our conformal-based solution is a drawback, as it increases computational complexity. This is why, as part of the original manuscript, we proposed two methods that do not require hyperparameter tuning: our greedy optimizer and the separable penalized ratio method. Both alternatives outperform the base conformal and Bayesian baselines while offering computationally efficient solutions.

Additional examples of potential applications

Focuses heavily on healthcare; additional applications in other high-stakes areas would demonstrate broader applicability

• We would like to highlight that our paper considers an additional example outside the medical setting by conducting experiments on the iNaturalist dataset (see Section 4 and Appendix D). This dataset consists of a large collection of images of living organisms, with labels organized hierarchically by taxonomy. Our classifier is trained to predict the lowest abstraction level in this hierarchy, which is the class of the organism (here, ’class’ refers to the taxonomic rank). These classes are grouped into phyla and further into broader taxonomic categories known as kingdoms. Here, the goal is to produce prediction sets that not only contain the true species but also animals that are close in the taxonomic tree (and thus genetically similar). Hence, any prediction about a specimen, even if is wrong, should be at least a very similar animal. Our framework can assist biologists in classifying new species or identifying specific organisms, as confidently determining the type of creature being studied can guide appropriate conservation measures.

• Another application where this framework is useful is within autonomous driving systems. In this setting, we would like to provide a self-driving system the set of possible courses of action to take. However, ensuring the correct action is in the set is insufficient to deem the set as safe. Rather we would like all actions in the predicted set to be deemed safe, as incorrect choices can be prohibitively costly and detrimental, e.g., a collision or an incident involving a pedestrian. Therefore, it is crucial to provide sets that provide sets of choices that on average yield a low value for $\mathcal{L}(S_{f(x)}) = \max_{s \in \mathcal{S}} \min_{a \in S_{f(x)}} L(a,s)$(for every possible state, the best choice is not very costly) where $L(s,a)$ is a loss capturing the cost of any state-action pair.

Answer to clarification questions

For the penalty-based approach in separable losses, can you elaborate on the efficiency of grid-search tuning for large datasets? Are there faster alternatives?

In the paper, we proved that empirical risk minimization converges to the true optimal hyperparameter for a finite set relatively quickly. For infinite sets, this remains an open question. In practice, performing a random grid search is likely the most practical approach.

Could you clarify how the fixed order of adding elements to prediction sets affects performance for complex utility structures? Would alternative scoring or ordering strategies improve flexibility for non-separable losses?

Using only scores based on $\hat{p}(y|x)$ naturally omits any consideration of the loss. In the non-separable case, the value of the loss depends on the definition of the set itself, leading to a combinatorial optimization problem. In this setting, the order in which elements are added to the set affects both the set's composition and the loss, resulting in different outcomes. Our greedy approach approximates the best possible order, and if the losses are submodular, it can be proven that this procedure is near-optimal.

Dear Reviewer c9L1,

We thank the reviewer for their kind words and their enthusiasm for the paper! We address your concerns below.

Answer to questions

Could the authors please clarify the advantages of the greedy approach?

The greedy approach does not require any hyper parameter tuning, thereby functioning as a better off-the-shelf solution.

Can you provide theoretical bounds for the performance between greedy optimizer vs optimal solution for non-seperable losses ?

Yes we can if the loss $\mathcal{L}$ is submodular, in which case our problem can be described as an instance of sub-modular maximization with a knapsack constrain. These type of problems are known to be efficiently and optimally approximated by greedy-based approaches.

Is there a relationship between base model calibration and decision loss?

There is no relationship that we are aware of between our losses and a calibrated model. However, this is an interesting question to explore. If we had access to the true values of $p(y|x)$, our conformalization step would not be necessary. Of course, the Bayes classifier is conditionally calibrated, which is a stronger property than simple calibration, but it would still be valuable to investigate the properties of our method under such conditions.

Did you consult medical experts about the clinical relevance of your prediction sets?

One of the authors has a medical background and, although the focus on this paper is on the technical development of an algorithmic framework, we validated with this author the coherence of our method's outputs.

Dear rCi4,

Thank you for the feedback! We address your concerns below.

Further explain motivation and illustrative example

"The motivation for the study and its illustrative example could be more clearly developed...In addition, I suggest moving this example to the introduction or background section to establish the relevance of the study more clearly."

The reviewer is correct that we could be more explicit in illustrating how this "hierarchical approach yields a more interpretable clinical result". We have updated our draft to include the following explanation/clarification in the main text (see Introduction). In Figure 1, we observe that although the base conformal prediction method yields a prediction set containing the true label, the prediction set also contains labels (leaves) from different parent nodes (benign epidermal, malignant epidermal, non-neoplastic inflammatory). This lacks clinical interpretability as benign vs. malignant diseases need to be treated/dealt with differently. On the other hand, our approach leads to prediction sets that contains labels (leaf nodes) sharing a common parent node 'malignant epidermal'. This hierarchical homogeneity of the prediction set aids clinicians in making follow-up decisions (e.g., prioritizing follow-up tests, prescribing treatment, etc.). We are happy to make further clarifications if this remains a concern!

Additional examples of potential applications and elaborating on appropriate loss functions for them

"The method relies on applications that require a decision loss function; however, the necessity and specific contexts for using a decision loss function should be more thoroughly explained...The authors might consider providing additional examples of potential applications and elaborating on appropriate loss functions for each

• We would like to highlight that our paper considers an additional example outside the medical setting by conducting experiments on the iNaturalist dataset (see Section 4 and Appendix D). This dataset consists of a large collection of images of living organisms, with labels organized according to a subtree of the natural taxonomic hierarchy. Our classifier is trained to predict the lowest level of abstraction in this hierarchy, which is the class of the organism (here, ’class’ refers to the taxonomic rank). These classes are further grouped into phyla which in turn belong to broader taxonomic categories known as kingdom. Here, the idea is to produce prediction sets that not only contain the true species in it but also animals that are close in the taxonomic tree (and thus genetically similar). Hence, any prediction about a specimen, even if is wrong, should be at least a very similar animal. Our framework can assist biologists in classifying new species or identifying specific organisms, as confidently determining the type of creature being studied can guide appropriate conservation measures.

• Another application example where this framework is useful is within autonomous driving systems. In this setting, we would like to provide a self-driving system the set of posible courses of action to take. However, in this case-study ensuring that the correct action is in the set is not a good enough desiderata to deem the set as safe. Rather we would like all choices in the predicted set to be deemed safe as well given how incorrect choices can be prohibitively expensive and detrimental, e.g, a collision or an incident involving a pedestrian. Therefore, it is crucial to provide sets that provide robust-optimization type of guarantees, that is sets of choices that on average yield a low value for $\mathcal{L}(S_{f(x)}) = \max_{s \in \mathcal{S}} \min_{a \in S_{f(x)}} L(a,s)$(for every possible state, the best choice is not very costly) where $L(s,a)$ is a loss capturing the cost of any state-action pair.

We thank the reviewer again for their time and consideration! We are more than happy to address any other concerns they might have.