Conformal Risk Training: End-to-End Optimization of Conformal Risk Control

We appreciate the reviewer for recognizing the novelty and significance of our Conformal Risk Training method for learning to control the broad class of optimized certainty-equivalent risks. We respond to each point raised by the reviewer.

Empirical evaluation

We have taken note of the reviewer's desire to see evaluation on more datasets. While we did not have time to run more experiments during this 1-week rebuttal period, we are actively looking into other datasets to test our method.

The reviewer also suggested evaluating the effect of varying the calibration set size on tail risk control. Below are 2 tables that show the average task loss and the CVaR on the battery storage problem as the calibration set size changes for post-hoc conformal CVaR Control. Values are given as mean ± std over 10 random seeds for choosing the calibration set.

As the tables show, a larger calibration set is generally better, both in terms of reducing the task loss and achieving a tighter CVaR bound closer to $\alpha$ (i.e., not being overly conservative). The reason a larger calibration set size $N$ helps is because the term $\frac{1}{N+1} \tilde{B}_t(\lambda)$ appears in $\tilde{h}_t(\lambda)$. Increasing $N$ reduces the effect of the upper bound $B$ and enables choosing larger (less conservative) $\hat\lambda$. However, there are diminishing gains as $N$ increases, which we clearly observe when $\delta = 0.9$.

Table 1: Average task loss

Table 2: CVaR

Applying CRC / CORC after fine-tuning

In all of our results, we always apply CRC or CORC after model training with a separate held-out validation set in order to formally guarantee risk control.

Lower bound on test-time risk

We have not derived a lower bound on the test-time risk. If we apply the same derivation from the original CRC paper to OCE risks, and we assume that $B$ is constant, we end up with

where

However, we have $R[L\_{N+1}(\lambda)] \leq \mathbb{E}[\tilde{L}\_{N+1,t}(\lambda)]$, and the inequality is generally not tight unless $t$ is chosen to be the minimizer of the variational form of the OCE risk. Therefore, we cannot directly compare $R[L\_{N+1}(\hat\lambda)]$ with $\alpha - \frac{2 \tilde{B}\_t}{N+1}$.

Comparing CRC and CORC as purely post-hoc methods

We are not 100% sure what the reviewer means with their question, so we will try to answer the question in 2 different ways.

If the reviewer is asking about comparing post-hoc CRC against post-hoc CORC, we would like to point out that if the risk measure is the expected loss, then CRC and CORC are exactly equivalent. On the other hand, if the risk measure is not the expected loss, then CRC is undefined. Therefore, it does not make sense to compare post-hoc CRC against post-hoc CORC in this setting.

If the reviewer is instead asking to see comparisons of post-hoc CRC vs. Conformal Risk Training (with expected loss), or comparisons of post-hoc CORC vs. Conformal Risk Training (with CVaR risk), we note that this comparison is the focus of our experimental section, and we apologize if our experimental section did not make this clear---we will clarify this accordingly in the camera-ready paper.

Dear reviewer, please engage with the author rebuttal.

We thank the reviewer for recognizing our Conformal OCE Risk Control method as a desirable generalization of Conformal Risk Control to work with more general risks such as CVaR. We also thank the reviewer for recognizing our formal treatment of conformal training. We respond to each of the points raised by the reviewer.

Sensitivity to $t$ hyperparameter

As the reviewer points out, Conformal OCE Risk Control is dependent on picking a good hyperparameter $t$ to tighten the OCE risk bound. From a practical / implementation perspective, we recommend selecting $t$ using a separate data split.

From a more theoretical perspective, we can see that $t$ shows up in two inequalities in the Proof of Theorem 1 (Appendix C.2):

The first inequality is tight at the value $t$ that minimizes the variational OCE expression. For example, when $R = \text{CVaR}^\delta$, $t=\text{VaR}^\delta[L(\hat\lambda)]$ makes the first inequality tight. On the other hand, theoretically analyzing the best choice of $t$ to tighten the 2nd inequality is difficult in general. We welcome suggestions, but for now, we leave this to future work.

In the 2 tables below, we show the empirical sensitivity of the average task loss and the empirical CVaR on the choice of $t$ for post-hoc Conformal CVaR Control in the battery storage task. Here, $\Delta t$ is the (absolute) amount by which we change $t$ from the value tuned on the training set. Evidently, the value of $t$ tuned on the training set is not always optimal, but the resulting task loss tends to be close (within 1 stddev) of the task loss from the optimal $t$.

Due to space limitations, we can only show a few choices of $\Delta t$, but we will include a proper figure in the camera-ready.

Table 1: Task loss

Table 2: CVaR

Theorem 3

We thank the reviewer for a very detailed reading of our proof, and we apologize for the heavy notation. If the reviewer can point to specific lines that are unclear, we welcome feedback to improve the clarity of the proof.

We also deeply appreciate the reviewer for identifying our regrettable mistake in applying the chain rule to the composition of differentiable-almost-everywhere functions. We noticed that we actually made the same mistake in both case (i) and case (ii). The mistake in case (i) is as the reviewer pointed out. The mistake in case (ii) lies in our claim that $\tilde{L}_{i,t}$ is continuously differentiable in $\theta$.

To correct this mistake, one option is indeed to assume that $\phi$ is differentiable almost everywhere, but then this excludes CVaR disutility function $\phi_{\text{CVaR}^\delta}(x) = \frac{1}{1-\delta} [x]_+$ which is not differentiable at the origin.

Instead, we will update Theorem 3 to make the more generic assumption that $\tilde{L}_{i,t}$ is differentiable almost everywhere in $(\theta, \lambda)$. And for case (i), we will directly assume that $\tilde{L}\_{i,t}$ has the form

These changes make our theoretical result less elegant, but do not fundamentally change the applicability of the theory to CVaR and related risk measures.

We also note that our mistake does not affect our empirical experiments. Because the disutility function is only non-differentiable at the origin, the likelihood in practice that $c_0(\theta) - t + \sum\_{j=1}^k c\_j(\theta) = 0$ is minimal, if not negligible.

Theorem 3 case (ii)

We apologize for the confusion about replacing $\ell_i$ with $\lambda$. What we mean is that if $\ell\_i(\theta, \lambda)$ are strictly increasing in $\lambda$, then

Once again, we’d like to thank the reviewer for their attentiveness, and hope we managed to clarify our results. Thank you for your response. Some additional questions/points:

The $t$ hyperparameter:

• Between the two inequalities, I think the first one seems by far the most important. Consider a fixed $t$ (i.e., not chosen on a separate split). While the latter becomes tight with fast $O(1/n)$ rates as $n \to \infty$, for the former this need not be the case.

  It's neat that the "optimal" choice of $t$ would be given by VaR. I would like to see a proof of it, though. I would also suggest adding this to the paper: $t$ is not just some arbitrary parameter, but rather one whose optimal choice corresponds to a fairly meaningful quantity, which could perhaps inform a practitioner with little data on how to choose it.

• Thank you for the tables, but I'm afraid I don't quite understand them. What exactly is the 'task loss'? What is the range of values for the task loss, cvar (or rather, the undelying data for the cvar) and for the $t$? And, just to double-check, the $\alpha$ should be the desired upper bound for the CVaR, right? Without this information it's hard to interpret the results.

Theorem 3:

• Thank you for the correction. I think I'm good with case (ii) (though I suggest the be more explicit in the text about the convex conic problem bit). As for case (i), I do see how the proposed assumption would resolve the issue. However, I am concerned about how applicable the theorem then becomes. Can you show me that this assumption does hold for CVaR, and maybe for some other risk measure?
• Regarding clarity, I'm afraid there aren't really any specific lines that are hard to follow, but rather the proof as a whole. It is technical and with lots of notation (to be frank, there are just too many letters), which makes it hard to follow. It also feels like there isn't a clear thread connecting each step. Perhaps the easiest improvement would be to try to reduce the amount of notation, and/or to reintroduce all of them right before the proof (e.g. in a table).

Non-zero gradients: thank you for your response. I think this may be worth briefly mentioning in the main paper.

Paper title: the proposed alternate titles would be definite improvements. If length is a concern, perhaps "optimized certainty-equivalent" could be shortened to OCE. I would perhaps also suggest changing "Conformal Risk Training" to something like "Conformal OCE Risk Training" to further distance from plain "Conformal Training + (Vanilla) Conformal Risk Control".

Dear reviewer, please engage with the author rebuttal.

$t$ sensitivity tables, continued

Following another reviewer's suggestion (Reviewer UbDa), we also implemented sensitivity analysis with a relative change in $t$, in addition to an absolute change in $t$. In doing so, we noticed a bug that affected our sensitivity analysis for absolute changes in $t$. Let $t_0$ denote the optimal value of $t$ tuned on the training set. In Theorem 2, our CVaR risk guarantee only holds when $t \geq B(\lambda_{\min})$, but we forgot to impose this constraint on $t_0 + \Delta t$. We fixed the bug, and the tables below report sensitivity analyses for both absolute and relative changes in $t$. The cases where $t_0 + \Delta t < 0$ are reported as NaN. (In the battery storage problem, $B(\lambda_{\min}) = 0$.)

The following table shows the values of $t_0$ (mean ± 1 std) selected by tuning on the training set:

Results for absolute $\Delta t$, i.e., $t = t_0 + \Delta t$:

Task loss

CVaR

Results for relative $\Delta t$, i.e., $t = (1 + \Delta t) t_0$:

Task loss

CVaR

Especially from the relative $\Delta t$ tables, it is clear that the task loss and CVaR are not very sensitive to changes in $t$. In particular, these results show that the sensitivity to $t$ is less than that suggested by the tables we presented in our initial rebuttal that did not include the needed constraint on $t$.

Theorem 3

We are working on an improved re-write of Theorem 3 to address the concerns you raised, and we hope to share it with you soon. We have a version of the proof that ensures the updated assumption is valid for neural networks with smooth activation functions, and we are working on a generalization to ReLU-like activations.

In terms of improving the clarity of the proof, we are splitting the proof into several lemmas which are easier to understanding on their own.

We thank the reviewer for detailed feedback and questions. Due to character count limits, our response is split across 2 comments.

Non-zero gradients

In our camera-ready revision, we will add a note to Theorem 3 that when the gradient exists, it may be zero.

Paper title

We are not sure about whether NeurIPS allows changing the title after submission. If the Area Chair could clarify the policy, that would be helpful.

$t$ hyperparameter

The reviewer correctly notes that the first inequality does not depend on $N$, whereas the 2nd inequality becomes tight with rate $O(1/N)$. This is evident from Theorems 1 and 2 in the original CRC paper [1], which show that when $L_{i,t}$ are i.i.d. and continuous, then

Thus, for large enough $N$, choosing $t$ to be close to the minimizer of the OCE risk definition will be nearly optimal.

In the case of CVaR, the fact that $t = \text{VaR}$ achieves the infimum in the OCE risk definition is a classical result from optimization (see, e.g., Sections 2-3 in [2]). We refer to Table 1 in [3] for a list of minimizers of other common OCE risks such as mean-variance and entropic risk.

References

[1] Anastasios Nikolas Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. 2023. Conformal Risk Control. In International Conference on Learning Representations, 2023. https://openreview.net/forum?id=33XGfHLtZg

[2] Stanislav Uryasev and R. Tyrrell Rockafellar. 2001. Conditional value-at-risk: optimization approach. In Stochastic Optimization: Algorithms and Applications. Springer US, Boston, MA, 411–435. https://doi.org/10.1007/978-1-4757-6594-6_17

[3] Ayon Ghosh, L. A. Prashanth, and Krishna Jagannathan. 2024. Concentration bounds for optimized certainty equivalent risk estimation. https://doi.org/10.48550/arXiv.2405.20933

$t$ sensitivity tables

We apologize for not clearly explaining the tables we posted in the rebuttal. The "task loss" refers to $f(y, \lambda \hat{z}\_\theta(x))$ from the battery storage example. Let $D = \\{(x_i,y_i)\\}\_{i=1}^{438}$ denote the test set of (input features, electricity prices) examples.

For $\lambda = 1$, the optimal task loss on the test set ranges from -544 to -61 (assuming perfect knowledge of $Y$, and thus perfect decisions $z$). When using electricity prices predicted by models $\hat{y}\_\theta$ trained to minimize mean-squared error, the decision is made as $\hat{z}\_\theta(x) = \arg\min\_{z \in \mathcal{Z}} f(\hat{y}\_\theta(x), z)$, and the task loss ranges from -446 to +69.

Your understanding of $\delta$, and $\alpha$ is correct: $\delta \in (0,1)$ is the quantile level of the tail, and $\alpha$ is the desired upper bound on $\text{CVaR}\_{(X,Y,\lambda)}^\delta[f(Y,\lambda \hat{z}\_\theta(X))]$.

The tables show the mean ± 1 std of the task loss $\mathbb{E}\_{(X,Y) \sim D}[f(Y,\lambda \hat{z}\_\theta(X))]$ and the CVaR $\text{CVaR}^\delta\_{(X,Y) \sim D}[f(Y,\lambda \hat{z}\_\theta(X))]$, respectively, for 10 models $\hat{y}\_\theta$ trained with different random seeds. The random seed sets the model's weight initialization as well as the training vs. calibration data split used for calibrating $\lambda$.

Since $\phi$ is convex and nondecreasing, $\\{\tilde{B}\_t\\}\cup\\{\tilde{L}\_{i,t}\\}\_{i=1}^N$ are convex in $\lambda$ by Lemma 2. Therefore, $\tilde{h}\_t$ is convex in $\lambda$.

Recall from Eq. (7) that

If $\sum\_{i=1}^N \ell\_i$ is strictly increasing (or decreasing) in $\lambda$ but not strictly convex in $\lambda$, observe that we may replace the objective $\frac{1}{N} \sum\_{i=1}^N \ell\_i$ with simply $\lambda$ (or $-\lambda$) in Eq. (7) without changing the solution $\lambda(\theta)$. Thus,

Since $\lambda(\theta)$ is the solution to a convex optimization problem whose objective $\tilde\ell$ is either strictly convex or convex and strictly monotone, the solution is unique. Every convex optimization problem can be equivalently reformulated as a convex conic problem [1]. Then, the implicit function theorem, applied to the KKT equality conditions, gives sufficient conditions for $\lambda'(\theta)$ to exist [2].

Specifically, define the Lagrangian $\mathcal{L}:\Lambda\times\Bbb{R}\times\Theta\to\Bbb{R}$ and the function $g:\Lambda\times\Bbb{R}\times\Theta\to\Bbb{R}^2$ by

The KKT equality conditions for primal-dual optimality of $(\lambda(\theta),\mu(\theta))$ are precisely given by

By applying the implicit function theorem to the KKT equality conditions [2], if

1. The optimization problem for $\lambda(\theta)$ exhibits strong duality;
2. $g(\lambda(\theta),\mu(\theta),\theta)=0$;
3. $g$ is continuous differentiable in $(\lambda,\mu,\theta)$ at $(\lambda(\theta),\mu(\theta),\theta)$; and
4. $\frac{\partial g}{\partial(\lambda,\mu)}(\lambda(\theta),\mu(\theta),\theta)$ is invertible,

then the derivatives $\lambda'(\theta)$ and $\mu'(\theta)$ exist and are given by

Strong duality is assumed in Assumption 7.

The requirement that $g$ is continuously differentiable at $(\lambda(\theta),\mu(\theta),\theta)$ is satisfied if $\tilde{h}\_t$, $\frac{\partial\tilde\ell}{\partial\lambda}$, and $\frac{\partial\tilde{h}\_t}{\partial\lambda}$ are continuously differentiable in $(\theta,\lambda)$ at $(\theta,\lambda(\theta))$. Assumption 7 ensures that this is the case.

Finally, we note that

which we assume in Assumption 7 to be invertible. $\blacksquare$

[1] Agrawal, Akshay, et al. “Differentiable Convex Optimization Layers.” Advances in Neural Information Processing Systems, 2019. Neural Information Processing Systems.

[2] Barratt, Shane. On the Differentiability of the Solution to Convex Optimization Problems. arXiv:1804.05098, https://doi.org/10.48550/arXiv.1804.05098.

We thank the reviewer for recognizing Conformal Risk Training as a promising method for controlling risk by design. We respond to each of the points raised by the reviewer.

Monotone assumption

The reviewer correctly points out that in some real-world risk control settings, the monotonicity assumption does not apply, especially in tasks involving structured or high-dimensional outputs. For example, the Learn then Test (Angelopoulos et al., 2025) paper gives multiple examples of non-monotone losses and develops a method for controlling their expected loss with high probability. However, as we (and many other papers in the conformal literature) demonstrate, the monotonicity assumption nonetheless encompasses a large number of real-world loss functions.

Unfortunately, relaxing the global monotonicity assumption on the loss functions $L_i$ remains challenging. As shown by the original Conformal Risk Control (CRC) paper (see "Proof of Proposition 2" in their Appendix E), even functions $L_i$ that are monotone in expectation cannot be controlled using Conformal Risk Control methods. For a loss $L$ that is approximately monotone, the original CRC paper proposes the idea of "monotonizing non-monotone risks"—i.e. computing the tightest monotone upper-bound of $L$. This idea can be directly translated to our Conformal OCE Risk Control (CORC) method, though we do not explore this idea further.

Exchangeability assumption

While our work uses the standard assumption that the loss functions $L_i$ are exchangeable, this assumption can be relaxed following standard conformal prediction arguments. For example, the CRC paper demonstrates how to apply CRC under covariate shift by using importance weighting, and it is straightforward to adapt that methodology to our CORC method. We can include a discussion of this in our camera-ready paper if the reviewer would like.

Specifically for time series, split conformal methods such as what we use are generally not directly applicable. The online variants of conformal methods (e.g., adaptive conformal inference (Gibbs and Candes, 2021)) generally rely on very different proof techniques compared to the exchangeable data setting. Thus, we believe that developing an online variant of CORC would be a valuable direction of future work.

Sensitivity of hyperparameter $t$, and tightness of CORC bound

The reviewer keenly points out the importance of tuning the hyperparameter $t$ to tighten the OCE risk bound. As seen in the Proof of Theorem 1 (Appendix C.2), $t$ shows up in two inequalities:

The first inequality is tight at the value $t$ that minimizes the variational OCE expression. For example, when $R = \text{CVaR}^\delta$, $t=\text{VaR}^\delta[L(\hat\lambda)]$ makes the first inequality tight. On the other hand, theoretically analyzing the best choice of $t$ to tighten the 2nd inequality is difficult in general. We welcome suggestions, but for now, we leave this to future work.

In the 2 tables below, we show the empirical sensitivity of the average task loss and the empirical CVaR on the choice of $t$ for post-hoc Conformal CVaR Control in the battery storage task. Here, $\Delta t$ is the (absolute) amount by which we change $t$ from the value tuned on the training set. Evidently, the value of $t$ tuned on the training set is not always optimal, but the resulting task loss tends to be close (within 1 stddev) of the task loss from the optimal $t$.

Due to space limitations, we can only show a few choices of $\Delta t$, but we will include a proper figure in the camera-ready.

Table 1: Task loss

Table 2: CVaR

Nested optimization

We believe the reviewer may have a misunderstanding of our method. We never directly solve any optimization problem that involves a CVaR constraint. Instead, our method replaces the CVaR constraint with the sufficient criterion from Theorem 2.

For Conformal Risk Training, while it is true that we have to calculate $\lambda(\theta)$ at every minibatch, we note that this computation is generally very efficient via the bisection-search method shown in Algorithm 1.

Comparisons with other risk-sensitive training approaches (e.g., DRO, robust optimization)

We were indeed inspired by other risk-sensitive training approaches, especially the conditional robust optimization (CRO) methods by Yeh et al. (2024) and Chenreddy et al. (2024). However, our Conformal CVaR Risk Control method fundamentally solves a different problem than robust optimization. Whereas end-to-end robust optimization (RO) aims to learn the easiest, say, 90%-confidence uncertainty set with respect to which to make decisions, our Conformal CVaR Risk Control method aims to control risk for the hardest 10% of outcomes. In other words, RO and DRO methods aim to be robust to the most likely outcomes, whereas our approach focuses on tail risks which encompass the most unlikely, and costly, outcomes. Without strong distributional assumptions on the loss function, RO and DRO methods do not provide any tail risk guarantees.

Literature connections to DFL and Contextual Stochastic Optimization

Our work is very much inspired by the broader decision-focused learning (DFL) literature. Due to the page limit in our submission, we only cited DFL methods that used conformal prediction for uncertainty quantification (e.g., Yeh et al. 2024, and He et al. 2025), and we did not have space in our submission to discuss connections with the broader DFL literature. In the camera-ready revision, we will definitely discuss and cite the Sadana and Mandi survey papers.

Stability of gradient

Here, we briefly discuss the stability of the gradient for the two cases described in Theorem 3. For case (i), where $B$ and $L_i$ are piecewise constant in $\lambda$, the exact gradient may exhibit high variance. Thus, as we note in Appendix D.1, for the image segmentation example we average the gradient over pixels with similar model outputs. This variance reduction technique was first discussed by Hong 2009 and Noorani et al. (2024) for the simpler conformal prediction settting where the risk parameter $\lambda$ is chosen as an empirical quantile. In contrast, for the CORC setting (whether Theorem 3 (i) or (ii)), the risk parameter $\lambda$ is chosen as the solution to an optimization objective. Analyzing stability then requires understanding the sensitivity of the solution of the optimization problem to its parameters. Characterizing this stability is therefore challenging, and we agree this is a valuable direction for additional research.

Efficiency and generalization of Conformal Risk Training vs. post-hoc CRC

Like the original Conformal Training method, the benefit of Conformal Risk Training is largely empirical. In general, we would expect Conformal Risk Training to yield larger improvements when the Conformal Risk Training objective $\ell(\theta, \lambda)$ (e.g., specificity) is substantially different from the model pretraining objective (e.g., cross-entropy loss).

Formatting concerns

We apologize that the reviewer finds Algorithm 2 unclear. If the reviewer could please elaborate on which specific aspect of Algorithm 2 should be improved, we are happy to make changes. As for the figures, we will take the reviewer's suggestion into account when preparing camera-ready figures.

We thank the reviewer for recognizing Conformal Risk Training as a promising method for controlling tail risk by design. We respond to each point raised by the reviewer.

More examples of CRC loss functions

Our paper highlights three explicit examples of CRC loss functions: false negative rate, battery storage operations loss, and miscoverage loss.

In the false negative rate loss function (see Example 1), the decision refers to classifying a pixel as either tumorous (1) or benign (0), based on the model's predicted probability $f_\theta(x)_j \in [0,1]$ for pixel $j$. This decision is determined by the threshold $\lambda$, and the decision loss is measured by the false negative rate $L(\theta, \lambda)$ given in Example 1.

In the battery storage optimization example (see Section 5.2), the decision refers to the vector $z := (z^\text{in}, z^\text{out}, z^\text{net}) \in \mathbb{R}^{72}$ which describes how much to charge or discharge a battery for each hour of the day. The decision is scaled by the parameter $\lambda$ to reduce CVaR tail risk to the desired level.

Furthermore, in Appendix E, we show how standard conformal prediction is a special case of conformal risk control that aims to control the miscoverage loss. Here, the decision refers to constructing a prediction set $C_\theta(X; \lambda)$, whose size depends on the risk control parameter $\lambda$.

The original Conformal Risk Control (CRC) paper (Angelopoulos et al., 2023), the Risk-Controlling Prediction Sets (RCPS) paper (Bates et al., 2021), and many other works provide various examples of monotone CRC losses, just to name a few:

• financial risk from portfolio optimization
• false discovery rate in multilabel classification
• ranking loss
• projective distance

We would be happy to provide a few more examples of these monotone CRC losses in our camera-ready revision to better motivate our work.

Clarifying assumptions 3 & 4

We apologize for the lack of clarity around assumptions 3 & 4. The reviewer correctly points out that under these assumptions, picking $\lambda = \lambda_{\min}$ always achieves OCE risk control of a random loss function $L(\lambda)$. This is why we use the objective $\ell(\theta, \lambda)$ to determine an optimal $\lambda$. Previous works such as Conformal Decision Theory paper (Lekeufack et al., 2024) only consider $\ell$ that are strictly decreasing in $\lambda$; in that case, they are indeed concerned with choosing the largest $\lambda$ such that risk control (of expected loss) is achieved. Our framework is more general by allowing $\ell$ that are either monotone or convex in $\lambda$, in which case the optimal $\lambda$ (with respect to $\ell$) may not necessarily be the largest.

We will revise our paper to clarify this point.

1-d characterization of decision space

As the reviewer points out, the original CRC formulation also assumes a 1-d characterization. Set coverage, FDR control, FNR control, etc. all have fundamental 1-d characteristics.

For controlling CVaR in the battery storage problem, we are motivated by the "decision-scaling" rule: $z\_\theta(x, \lambda) = \lambda \cdot \hat{z}\_\theta(x)$. In this setting, $\hat{z}\_\theta(x)$ represents the decision that some base decision-model suggests. We scale this decision by $\lambda$, where $\lambda$ intuitively quantifies how much we trust the base decision-model. If we fully trust the model, we set $\lambda = 1$ and perfectly follow the base decision-model. If we don't trust the model at all, we set $\lambda = 0$, resulting in a do-nothing decision: $\hat{z}\_\theta(x) = 0$. The amount of trust (which we call "aggressiveness" in our paper) is naturally characterized by a 1-d parameter. We will revise our paper to more clearly explain this 1-d characterization.

Nonetheless, we recognize that not all decisions can be easily boiled down to a 1-d parameter. Notably, the paper "How to Trust Your Diffusion Model" (Teneggi et al., 2023), provides an example of multi-dimensional $\lambda$ in the RCPS setting. We believe that investigating similar generalizations of our CORC method to accommodate multi-dimensional $\lambda$ would be an interesting direction of future work.

I thank the authors for their response. I also have read the other reviewers comments, and I believe the paper has clear and interesting results that is of particular interest to the community. Hence, I raise my score and vote for acceptance.

We thank the reviewer for recognizing Conformal Risk Training as a promising method for controlling risk by design. We respond to each point raised by the reviewer.

Comparison against related methods

The review highlights 3 related works on risk control: Learn then Test (LTT), Risk Controlling Prediction Sets (RCPS), and the Distortion Risk Control method by Chen et al. (henceforth referred to as "Chen"). All 3 methods provide high-probability bounds on risk, as opposed to the stronger certain (marginal) bound that our Conformal OCE Risk Control (CORC) method achieves. As shown in Appendix B of the original Conformal Risk Control (CRC) paper, the marginal bounds from conformal methods are significantly more statistically efficient compared to LTT/RCPS.

We further discuss each of these methods in order:

• LTT: LTT is primarily used to control the expected loss of non-monotone loss functions. While the generic LTT framework in principle supports a general risk measure $R$, one must be able to compute p-values for null hypotheses of the form $H_\lambda:\ R(\lambda)>\alpha$. The Hoeffding–Bentkus inequality p-values described in the LTT paper only apply to expected losses; while a recent work (“Quantile Learn then Test”, Farzaneh et al. 2024) proposes a p-value for $R=\text{VaR}$ (value-at-risk), to the best of our knowledge, we are unaware of p-values for $R=\text{CVaR}$. We note that a conservative approach would be to use LTT to control the expectation of $\tilde{L}_t(\lambda)=t+\phi_{\text{CVaR}^\delta}(L(\lambda)-t)$, which upper-bounds $\text{CVaR}^\delta[L(\lambda)]$. If the reviewer would like to see a comparison between CORC and this approach to LTT, we can try to implement this comparison.
• RCPS: RCPS is used to control risk resulting from set-valued predictors. For our battery storage CVaR example, it is unclear to us how to formulate the problem using set-valued predictors. Furthermore, RCPS requires the loss functions to be monotone in $\lambda$ (with monotonicity in the same direction). Thus, we do not believe RCPS to be suitable for controlling CVaR for our battery storage example, where the loss functions can be either monotone increasing or decreasing.
• Chen: The main result from the Chen method is an asymptotic (as opposed to finite-sample) high-probability distortion risk control bound. The Chen paper does introduce a finite-sample variant, which is more conservative. If the reviewer would like to see a comparison between CORC and the more conservative Chen method, we can try to implement this comparison.