Conformal Tail Risk Control for Large Language Model Alignment

We thank the reviewers for their thoughtful feedback. We are encouraged by the recognition of key strengths in our submission, including:

• The novelty of the problem setting—controlling tail risk in LLM outputs with distribution-free guarantees;
• The practical relevance and effectiveness of our method, supported by empirical results.

Weakness 1: Clarifying novelty beyond QRC [2]

Thank you for the thoughtful feedback. In [2], QRC relies on concentration inequalities on the cumulative distribution functions (CDFs), specifically the BJ and DKW inequalities, to derive uniform upper bounds between the empirical and true quantiles. While effective, these bounds are conservative as they do not take the weight function $\psi$ into account. In contrast, our work proposes a novel approach by formulating distortion risk control through L-statistics, which are tailored to the form of the distortion function $\psi$. This allows us to:

• Directly bound the distortion risk functional rather than the entire CDF;
• Leverage asymptotic normality of L-statistics to obtain asymptotically tight , more efficient bounds;
• Reduce sampling costs significantly while maintaining risk guarantees, as demonstrated in our empirical results.

We will revise the introduction and background to clearly differentiate our contribution from prior work.

Weakness 2: Statistical efficiency at smaller calibration sizes

This is a valid concern. We conducted an ablation experiment on the calibration sizes and observed how deployment cost changes with that. This is illustrated in the following tables.

We observe that as the calibration size decreases, all methods become more conservative—realized costs increase and risk metrics deviate more from their expected values. However, DRC-L is still statistically more efficient. Interestingly, DRC-L also shows the smallest increase compared with DRC-DKW and DRC-BJ. For example, when the calibration size drops from 6000 to 1000, the cost of DRC-L increases by only 7.7%, compared to 17% for DRC-BJ and 28% for DRC-DKW, highlighting the advantage of L-statistics and its robustness to sample size in risk estimation.

Table 1: Realized average cost with calibration size n of our method (DRC-L) and baselines for CVaR, $\alpha = 0.25$, $\beta = 0.75$

Table 2: Realized CVaR with calibration size n of our method (DRC-L) and baselines for $\alpha = 0.25$, $\beta = 0.75$

Table 3: Realized average cost with calibration size n of our method (DRC-L) and baselines for VaR, $\alpha = 0.25$, $\beta = 0.75$

Table 4: Realized VaR with calibration size n of our method (DRC-L) and baselines for $\alpha = 0.25$, $\beta = 0.75$

We hope this addresses your concern. Please let us know if there are any other aspects you'd like us to discuss further.

Weakness 3: Terminology clarification on “Conformal Risk Control”

We appreciate this point. As the reviewer notes, [4] defines conformal risk control in expectation, while our method provides high-probability guarantees, making it more closely aligned with the RCPS framework [1].

In our manuscript, we use the term “conformal risk control” more broadly to refer to the overarching goal of risk-aware inference under distribution-free guarantees. However, we agree that this may cause confusion and will revise the terminology to better reflect our actual contribution. Specifically, we will reframe our method as “conformal tail risk control with high probability”, to distinguish it from the expectation-based guarantees in [4] and emphasize its closer connection to [1]. We appreciate this clarification and will ensure the distinction is made explicit in the revised manuscript. Let us know if you would like us to elaborate on this point further.

We appreciate the reviewer’s thoughtful feedback and are encouraged by the recognition of our paper’s strengths, including:

• The novelty of the problem setting—tail risk control in LLM alignment with provable guarantees;
• The rigor and clarity of our writing and symbolic notation;
• The practicality of our lightweight alignment method that avoids LLM retraining.

Comparison with RLHF and existing alignment methods

Thank you for the suggestion. Our method differs fundamentally from RLHF-based techniques. RLHF typically requires model access and expensive retraining, while our method is post hoc, requiring no access to model internals or parameter updates. It is thus more scalable and deployable.

Further, RLHF lacks formal guarantees for safety or tail risk control. Our approach directly addresses this gap by providing statistically provable control over tail disutility metrics (e.g., CVaR and VaR). We will include a qualitative comparison table in the revision to emphasize these distinctions.

Weakness 1: Practicality, real-world experiments, and prompt distribution assumptions

We agree that practicality is essential. We conducted real-world evaluations using Qwen2.5-1.5B and LLaMA3.2-3B on the RealToxicPrompts (RTP) dataset, beyond LLaMA-2-7B. These experiments confirm that DRC-L consistently satisfies risk control requirements while minimizing inference-time cost.

Regarding distribution assumptions: while all conformal methods require the calibration distribution to reflect deployment,our framework supports reweighting calibration examples by density ratios (aka importance weights) to handle distribution shifts, thus improving robustness in practice. We will add a theorem showing the validity of the reweighted DRC-L in the revision.

We also empirically find that for a given risk level, a reliable $\lambda$ threshold can be estimated adaptively, even with modest calibration sizes. We will expand on this with clarifying details and additional ablations in the revision.

Weakness 2: Reliance on a single model

Thank you for the feedback. We have conducted additional experiments using the Llama3.2-3B and Qwen2.5-1.5B models on the RealToxicPrompts (RTP) dataset, following the same setting as illustrated in Section 4.1 in our manuscript. The results show that DRC-L consistently meets the risk control requirement while achieving lower cost compared to DKW and BJ. For all tables, the metrics are denoted by “average ± standard error” with 15 independent experiments. Below we show the sample results of the Llama3.2-3B model with $\beta = 0.5$, and $\alpha = 0.25$. We have additional experiments with other models (e.g., Qwen2.5-1.5B), and different settings of $\alpha$ and $\beta$.

Table 1: Realized CVaR and average cost on RTP dataset with Llama3.2-3B model.

Table 2: Realized VaR and average cost on RTP dataset with Llama3.2-3B model.

We will add these results to the paper to strengthen the empirical claims.

Clarification: Biased Scoring Model via c% Extremes

Thank you for raising this. To simulate a biased scoring model, we retrain Detoxify on a dataset composed of texts with the top and bottom c% of toxicity scores, removing the middle. This produces a model skewed toward extreme judgments and serves as a proxy for biased disutility classifiers. We will revise the manuscript to clarify this process.

Clarification: Which component is aligned—LLM or classifier?

This is a great question. Our method supports two complementary alignment views:

1. Aligning the LLM: Holding the classifier fixed, our procedure filters LLM outputs to meet human-aligned thresholds.
2. Recalibrating the classifier: Holding the LLM fixed, we can use our method to adjust $\hat{\lambda}$, effectively realigning machine scores to better match human disutility.

We will include a pipeline diagram in the revised manuscript to illustrate this dual perspective and how the components interact.

We thank the reviewer for the constructive comments and questions. We are encouraged by the recognition of several key strengths:

• Our detailed theoretical analysis and use of L-statistics in conformal risk control;
• The practical relevance of our proposed framework, including its lightweight, post hoc nature and ability to provide provable guarantees without retraining;
• The importance and novelty of tackling tail risk in machine-human alignment for LLMs.

Below, we address the concerns in detail:

Weakness: Writing, Related Work, and Experiments

Thank you for this valuable feedback. In the revised version, we will:

• Add a Related Work section, situating our approach alongside prior work on conformal risk control, and LLM alignment;
• Reorganize the Introduction to better highlight our core contributions;
• Expand the experimental section with new results using additional LLMs (LLaMA3.2-3B and Qwen2.5-1.5B) and larger models, e.g., Llama3.2 8B and Qwen2.5-7B, with more datasets (e.g., RealToxicPrompts), best-of-N baselines, and cost ablations.

Weakness: Similarity between Figures 4 and 5

Thank you for pointing this out. Indeed the first row of Figures 4 and 5 illustrates the realized CVaR versus $\alpha$, demonstrating our method consistently controls tail risk. The second row shows the average sampling cost versus $\alpha$, which decreases as the target level $\alpha$ increases. Although they serve different purposes (tail risk control vs. efficiency), the visual similarity can be misleading.

To improve clarity, we propose reformatting the cost results to highlight the differences more explicitly. In particular, we present percent cost reductions of our method (DRC-L) relative to baselines (BJ and DKW). Sample results are shown to demonstrate the efficiency gains provided by our approach (we have complete results for other LLMs, and difference choices of $\rho$, and $\beta$)

Table 1: Percent cost reduction for CVaR of our method (DRC-L) compared to baselines, with $\rho = 0.78$, $\beta = 0.5$

We will update the figures and accompanying text in the revision to improve clarity and better guide the reader.

Justification for L-statistics vs. BJ and DKW

Thank you for this important point. Our work is centered on controlling tail risk using distortion risk measures such as CVaR. Prior work like Quantile Risk Control (QRC) by Snell et al., extended conformal methods to tail risks using concentration inequalities like BJ and DKW, which provide general-purpose CDF bounds. However, these are not tailored to distortion risks and tend to be conservative (see lines 322-329). Our key contribution lies in leveraging L-statistics, which are inherently aligned with the structure of distortion risks by modeling them as a linear combination of order statistics weighted by a function $\psi$. This allows us to:

• Directly bound the risk functional;
• Achieve tight asymptotic approximations;
• Deliver lower deployment cost with valid guarantees.

We will revise Section 3.2 to explicitly justify this choice and contrast it with baselines.

Is this the only viable approach for conformal tail risk control?

Our work builds directly on QRC (Snell et al.), which introduced BJ and DKW for tail risk control. These methods are described in Section 3.4. Our work is the first to use L-statistics in the given context. While other methods like BJ and DKW are viable, our empirical results consistently show that L-statistics:

• Better match the structure of distortion risks;
• Provide sharper bounds and lower sample costs;
• Are less conservative for distortion risk control.

We will clarify this distinction in the revised manuscript.

Why do BJ, DKW, and L-statistics perform differently?

The performance differences stem from how each method models uncertainty:

• DKW offers uniform control over the CDF but does not emphasize the tail—making it overly cautious for rare events.
• BJ is more sensitive to small tail deviations than DKW but remains a general-purpose bound and thus conservative. Moreover, it is computationally much more costly than our method and DKW.
• L-statistics, in contrast, are designed for distortion risk, directly estimating tail-weighted quantiles using the given weight $\psi$. This structural alignment allows for tight risk bounds and consistent performance across calibration sizes. In fact, when the size of the calibration set is large, our risk control is exact and hence not conservative.

We thank the reviewer for the thoughtful feedback. We’re grateful for the recognition of the sound theoretical foundations, novel adaptation of conformal methods to control tail risks, and clear visual illustrations. Below we address the main concerns:

Weakness 3: Additional inference cost

There are inherent added costs when ensuring guarantees for risk-aware inference. However, our results show that DRC-L achieves tight bounds with minimal overhead. In particular, DRC-L achieves tight bounds for controlling tail risks, allowing us to maintain the desired risk level with lower cost compared to baselines. Further, compared to a fixed best-of-N strategy, DRC-L achieves lower deployment cost to control the risk at the same level.

Weakness 4: Limited evaluation

We add results for more models (e.g., Llama3.2-3B, Llama3.2 8B, and Qwen2.5-7B) on new datasets (e.g., RealToxicPrompts (RTP)), under the same setup as Section 4.1. We observe that DRC-L consistently satisfies the CVaR constraint with lower cost than baselines. We highlight that our method is dataset and model-agnostic, and we will include results for larger models and different $\alpha$, $\beta$ choices in the revision. Below, we report an example realized CVaR and cost (mean $\pm$ standard error) over 15 independent runs.

Table 1: Realized CVaR and average cost on RTP dataset with Qwen2.5-1.5B model.

We will include additional results in our revision.

Weakness 5/Suggestions: Presentation

We will revise Section 3 to improve clarity and motivation, connecting each step to the overall goal. Additionally:

• Simplify notation as recommended, and unify subscript usage
• Add a notation table
• Switch from “disutility” to “reward” to align with conventions
• Replace yellow in Figure 3 with a darker color

Q1: Comparison to inference-time alignment methods

We highlight that Best-of-N is a fixed-sample inference-time heuristic, while our method is an adaptive, risk-controlling strategy. Best-of-N always samples N responses, regardless of prompt toxicity, which can lead to overhead. In contrast, DRC uses as few samples as needed. Following the procedure in Section 4, we evaluate the realized CVaR of human toxicity scores, selecting the response with lowest machine score from $N \in {3, 5}$ samples, with $\beta = 0.5$ to demonstrate that our method matches or improves on these values with lower average cost (compare with Fig. 4):

Table 1: Realized CVaR of best-of-N algorithm on a calibration set size n.

Finally, best-of-N offers no statistical guarantees, whereas our framework delivers PAC-style guarantees for distortion risks.

Q2: Can theoretical claims be demonstrated on synthetic data?

Yes, we will include a synthetic experiment with a long-tailed loss distribution to illustrate the theoretical claims in our revision.

Q3: What does the threshold $\hat{\lambda}$ and Eq. (7)'s probability look like? What is $\psi$?

We will add theory and plots to illustrate $\hat{\lambda}$ and the probability in Eq. (7). We can visualize the distribution of the number of inference rounds required under different risk levels $\alpha$ and distortion functions $\psi$, to show how aggressively the tail is penalized. Our theoretical results show that $\hat{\lambda} \stackrel{p}{\rightarrow} \lambda^*$, and thus expected cost converges according to Eq. (7). The distortion function $\psi$ in our CVaR experiments is $\psi(p) = \max(p - \beta, 0)/(1 - \beta)$. For VaR, $\psi$ is an indicator function at $p = \beta$. We will discuss other $\psi$ functions and their implications in the revision.

Q4: Anecdotal evidence

We agree that examples are helpful for understanding how our procedure works. Below, we provide an example from our experiments.

Table 1: LLM-generated responses with human and machine toxicity scores for prompt: “If you find it necessary for Assad to bomb innocent children in order to kill”

If $\lambda = 0.2$, our method selects the last two responses, which have a maximum human score of 0.69. We will include similar examples in the appendix.