Conformal Arbitrage: Risk-Controlled Balancing of Competing Objectives in Language Models

We thank the reviewer for their engagement with our work and for highlighting the appeal of using conformal risk control (CRC) for black-box deferral. We address the concerns point-by-point below.

Scope beyond multiple choice.

Conformal Arbitrage (CA) is not restricted to multiple choice. The theory in Section 4 is stated for any input-dependent finite action slate $A(x)$; Algorithm 1 and CRC calibration (Eq. (3)) operate on such action sets regardless of how candidates are produced (MCQ options or free-text proposals).

Our limitations sentence 'our study is confined to multiple-choice tasks; applying CA to free text would require bespoke loss functions' was intended to refer to the experiments, not the theory. By ``bespoke loss'' we meant only that, to instantiate Eq.(2) in a given application, one must specify a bounded guardrail loss $L(x,a)\in[0,1]$ (equivalently, a task-appropriate guardrail metric $g(x,a)$, e.g., violation rate or factuality error) for such an application. Once such a bounded loss is defined, CA selects $\hat\lambda$ and the finite-sample guarantee applies exactly as in the multiple choice setting.

This also explains our Section 4 terminology of actions and contexts: it was chosen to cover not only question answering but also agentic deployments, where an LLM (or LLM+tools) selects among environment actions. CA’s guarantees apply directly in this view: the Primary proposes a finite set $A(x)$, CA forms the $\lambda$-relaxed set $C_\lambda(x)$, and the Guardian (possibly a human) selects within that restricted slate under a risk budget $\alpha$. While suitable public benchmarks for fully agentic evaluations are limited, the formulation and guarantees are designed with such potential applications in mind. Our experiments use multiple choice question-style benchmarks as controlled proxies. Notably, Section~5.2 (helpfulness--harmlessness) already goes beyond binary right/wrong by using discrete safety severities (e.g., 0--3), illustrating CA on non-binary correctness outcomes.

Free text within CA: minimal recipe.

For given calibration context $x_i$ we can have an arbitrarily large but finite potential output set $A(x_i)$. Sample $K$ free-text candidates from the Primary (e.g., across reasoning budgets/prompts/beam) and inherently consider these to be the $K$ top scoring $p(x_i,a)$. Call this $A_K \in A_i$. Record Primary/Guardian scores $\{p(x_i,a)\}$ for $\{a \in A_K\}$ and $\{g(x_i,a)\}$ for $\{a\in A_K\}$. Sample the Guardian model for its output and consider this to be $\arg\max_{a \in A(x_i)} g(x_i,a)$. Then we are able to define the bounded guardrail loss $L_i(\lambda)$ as in Eq. 2, compute $\widehat R_n(\lambda)=\tfrac1n\sum_i L_i(\lambda)$, and select the smallest $\hat\lambda$ satisfying the CRC inequality in Eq. 3. If no $\hat\lambda$ satisfied the CRC inequality then we would need to increase $K$ and re-run.

Deployment (per input; Sec. 4.3/Alg. 1). For a new $x$: (i) obtain $p(x,a)$ for all $a\in A(x)$; (ii) form $C_{\hat\lambda}(x)=\{a:\,p(x,a)\ge \max_{a'}p(x,a')-\hat\lambda\}$; (iii) if $|C_{\hat\lambda}(x)|=1$, return its unique element (commit to the Primary); otherwise (iv) query the Guardian only on $C_{\hat\lambda}(x)$ and output $\arg\max_{a\in C_{\hat\lambda}(x)} g(x,a)$. This mirrors Sec. 4 exactly and carries the same finite-sample guardrail guarantee from CRC.

Note that this does not change any core mechanism behind how CA works and does not require any changes to it as stated, it is just a specific proposal for eliciting the scores $p(x,a)$ and $g(x,a)$ for the calibration set and applying the CA framework. If instead the scores were easily obtainable using a functional form or some other method then that would also suffice. The main point is that at its technical foundations Conformal Arbitrage does not pertain just to questions that are in multiple choice style settings, but can truly apply to any setting with contexts and finite action sets.

On the ``specialized Guardian is always better'' concern.

CA’s risk guarantee does not assume the Guardian uniformly dominates the Primary. The CRC guarantee is on the final output (after our policy either commits or escalates) and holds at finite sample under exchangeability. If the Guardian adds little value on a subset of inputs (including when the Guardian is a human), the calibrated $\hat\lambda$ becomes small and escalations are rare; the guardrail bound is still met. When a performance gap exists, CA selectively exploits it while filtering the candidate slate, which budgets oversight effort.

Our intended deployment target is a closed-API, black-box setting in which the user cannot train or host additional models on $x$ (the Primary may be substantially more capable or proprietary). Many learning-to-defer and model-routing methods assume joint training, access to logits/gradients, or router models—assumptions that do not match this deployment constraint. CA differs along three axes emphasized in the paper: (i) post-hoc, black-box operation (no joint training, no weights/logits), (ii) distribution-free, finite-sample guardrail control via CRC on a bounded loss (rather than optimizing an expected risk without distribution-free guarantees), and (iii) set-based escalation (forwarding only $C_{\hat{\lambda}}(x)$) that preserves cost/latency in closed-API use. Reflecting this target, our evaluation compares to single-model baselines and a cost-matched random router (the natural naive policy when only API calls are permitted), and includes an unrestricted ablation (CA$^\star$) that passes the full set to the Guardian to isolate the impact of restricted conformal sets. At comparable cost, CA recovers most of the stronger model’s gains while honoring the guardrail budget, which is precisely the objective in the black-box setting we study.

On Theorem 1 regularity

Eq. (2) defines a per-episode loss $L_i(\lambda)$ that is stepwise in $\lambda$, so the empirical $\widehat R_n(\lambda)$ is piecewise constant. Theorem 1 is a population-level result about $R(\lambda)=\mathbb{E}[L_i(\lambda)]$.

We do agree, however, that it may not hold in practice that this population-level loss curve is strictly decreasing. It is the case if the score-gap thresholds are atom-free (e.g., in practice by adding an independent continuous infinitesimal tie-break to the Primary scores), then $R$ is continuous. Strict decrease, however, additionally requires that enlarging $C_\lambda(x)$ can improve the guardrail on a set of inputs of positive probability; without that, $R$ may have flat regions. In such cases, the CRC risk control is unchanged, and a variant of Theorem 1 holds.

If $R$ is merely non-increasing and right-continuous, define the leftmost feasible point $\lambda_\star=\inf\{\lambda\in\Lambda:\,R(\lambda)\le \alpha\}$. The CRC selector $\hat\lambda$ (the smallest $\lambda$ with $\tfrac{n}{n+1}\widehat R_n(\lambda)+\tfrac{B}{n+1}\le \alpha$) satisfies $R(\hat\lambda)\le \alpha$ for all $n$ (finite-sample control), and under the law of large numbers for bounded $L_i(\lambda)$ one has $\widehat R_n(\lambda)\to R(\lambda)$ pointwise for each $\lambda$. Any limit point $\bar\lambda$ of $\hat\lambda$ must then satisfy $R(\bar\lambda)\le \alpha$; with the “leftmost feasible” tie-break used in our definition, $\hat\lambda\to \lambda_\star$ in probability. If, in addition, the utility $U$ is non-increasing and continuous, then $U(\lambda_\star)-U(\hat\lambda)\to 0$. Essentially, when $R$ is strictly decreasing with slope bounded away from $0$ near $\alpha$, Theorem 1 further gives the $O(n^{-1})$ rate via the inverse slope; without that margin, we can still state consistency without a rate.

We can include this clarification and the variant statement in the appendix of the camera-ready. We thank the reviewer for their close reading and eliciting this clarification.

Minor comments (acknowledged)

We will correct the noted typos and notational slips: remove the duplicated 'the,' clarify that the conformal predictor attains coverage, write $\hat{\lambda}$-relaxed set, fix $g(x,a)$ in Algorithm 1, and change set of contexts to set of actions. We thank the reviewer for their close reading and identification of each of these typos.

We hope these clarifications address the reviewer’s concerns and help recalibrate the overall assessment.

Here is a more technical explanation of the above.

Let $x \in \mathcal{X}$ and let $\mathcal{Y}_L$ be all strings up to a fixed length $L$.
Under a fixed generation policy (top-$K$ by the Primary’s internal scoring, or $K$ proposals across budgets/prompts/beam), the Primary produces a finite slate $S(x)=\{a_1,\ldots,a_K\}\subseteq \mathcal{Y}_L$.

We define the Primary score on $\mathcal{Y}_L$ by setting $p(x,a) =$ model-provided score for $a \in S(x)$, and $p(x,a) = -\infty$ for $a \not \in S(x)$. So off-slate strings are implicitly ruled out by the score-gap router.

For each calibration context $x_i$: (i) build $S(x_i)$ with the same generation policy;
(ii) obtain $p(x_i,a)$ for $a\in S(x_i)$;
(iii) query the Guardian once to produce its own free-text response $y_G(x_i)$

Obtain $g(x_i,a)$ for $a\in S(x)$ and for $y_G(x_i)$. Then compute the bounded guardrail loss exactly in the form of Eq.(2):

L_i(\\lambda)\= g(x_i, y_G(x_i))-\\max_{a\in C_\lambda(x_i)} g(x_i,a)\ \in[0,B],

the empirical risk $\widehat R_n(\lambda)=\tfrac{1}{n}\sum_{i=1}^n L_i(\lambda)$, and select the smallest $\hat\lambda$ satisfying Eq.(3).

No scores over “all strings’’ are required—only the Primary’s scores on the slate and the Guardian’s scores on the finite $S(x)$ (with $y_G(x_i)$ providing the natural baseline).

Deployment

For a new $x$: form $S(x)$ with the same generation policy, compute $p(x,a)$ for $a\in S(x)$, construct $C_{\hat\lambda}(x)$, and (i) commit if $|C_{\hat\lambda}(x)|=1$, else (ii) query the Guardian only on $C_{\hat\lambda}(x)$ and output $\arg\max_{a\in C_{\hat\lambda}(x)} g(x,a)$.
Because the generation policy is identical in calibration and deployment, the episodes are exchangeable and the CRC guarantee applies verbatim.

Key takeaways.

(1) Everything above is exactly the CA specification in Section 4; nothing in the theory or algorithm changes.

(2) This recipe does not conflict with the original specification. If one did have scores for the entire action set, those could be used directly and CA would operate unchanged. The ``minimal recipe'' is simply a computationally efficient instantiation for free-text settings, not a modification of CA. All of the same guarantees are maintained.

(3) If Eq.3 is feasible for a chosen $K$, the guardrail objective is already met; if not, increase $K$ and re-calibrate, guarantees remain intact throughout.

We hope this clarifies how CA applies to free-text generation and appreciate the reviewer’s engagement with this point.

Thank you for the great question. This is a nuanced point. To recall the main problem we are trying to solve, we want to achieve as much primary utility as possible while staying within a guardian's guardrail objective. For the problem of free-text generation, we can indeed achieve this with the "minimal recipe" approach provided in our rebuttal, even when the universe of all possible actions can be massive. Specifically, let us note that the calibration step can provides a verifiable guarantee that for any $K$ (even small ones) we achieve the desired guardian loss, or not. If such a guardian loss is achieved, we have achieved our goal.

Specifically, for a fixed set size $K$ and a fixed generation policy, calibration verifies whether the CRC inequality in Eq.(3) is feasible. If it is, we already have a finite-sample guarantee that the final (commit/escalate) policy obeys the guardrail budget $\alpha$; in other words, we have achieved the guardian objective while achieving as much primary utility as possible and there is no need to score beyond the $K$ candidates the Primary proposed. If the inequality is not feasible, we simply increase $K$ and/or diversify the slate and re-calibrate.

This is not an approximation of CA. The algorithm itself is unchanged; we are instantiating it on the finite action set induced by the Primary’s slate, with exchangeability preserved between calibration and deployment.

However, we acknowledge that there could be hypothetical examples where to achieve the desired guardian objective, $K$ needs to be very large.This possibility is covered by our theory; it does not affect the validity of the guarantees. However, such examples are highly unrealistic and there is even evidence in the literature to support this claim. Empirically, recent work shows that even naive repeat sampling often finds high-quality answers with modest $K$ (tens to low hundreds; see, e.g., prior results on repeat sampling for LLMs [1]).

Our slate construction is at least as strong as naive repeat sampling: if one chooses the generation policy to be “draw $K$ i.i.d. samples,” CA’s calibration and guarantees apply verbatim; if, instead, one asks the Primary to produce its top-$K$ or a diversified set across reasoning budgets/prompts, the resulting slate weakly dominates the naive pool for the same budget, and CA can only benefit from the higher-quality proposals.

In short: calibration tells us, for the chosen $K$ and generation policy, whether the safety budget $\alpha$ is met. If it is, there is no reason to consider more of the (latent) action space—the Primary has already surfaced candidates it believes best, and CA ensures the guardrail objective on the final decision. If it is not, increase $K$ or diversify the slate and re-calibrate; the guarantees remain intact throughout.

[1] Large Language Monkeys: Scaling Inference Compute with Repeated Sampling, Brown et al.

Additionally, we'd like tothank the reviewer for explicitly noting that CA applies to agentic action sets. This is an important part of our contribution and motivation: we aim at safe deployment of agents in high-stakes decision-making scenarios where a language model (or LLM+tools) must act autonomously most of the time, yet escalate selectively. In this regime, CA serves as a principled, black-box controller: the Primary proposes actions, CA enforces a score-gap filter and a calibrated risk budget $\alpha$, and escalation to a human or stronger model occurs only when needed. This provides a rigorous mechanism for scalable oversight.

We sincerely thank the reviewer for the thoughtful and positive assessment.

Scope beyond multiple choice.

Conformal Arbitrage (CA) is not restricted to multiple choice. The theory in Section 4 is stated for any input-dependent finite action slate $A(x)$; Algorithm 1 and CRC calibration (Eq. (3)) operate on such action sets regardless of how candidates are produced (MCQ options or free-text proposals).

Our limitations sentence 'our study is confined to multiple-choice tasks; applying CA to free text would require bespoke loss functions' was intended to refer to the experiments, not the theory. By ``bespoke loss'' we meant only that, to instantiate Eq.(2) in a given application, one must specify a bounded guardrail loss $L(x,a)\in[0,1]$ (equivalently, a task-appropriate guardrail metric $g(x,a)$, e.g., violation rate or factuality error) for such an application. Once such a bounded loss is defined, CA selects $\hat\lambda$ and the finite-sample guarantee applies exactly as in the multiple choice setting.

This also explains our Section 4 terminology of actions and contexts: it was chosen to cover not only question answering but also agentic deployments, where an LLM (or LLM+tools) selects among environment actions. CA’s guarantees apply directly in this view: the Primary proposes a finite set $A(x)$, CA forms the $\lambda$-relaxed set $C_\lambda(x)$, and the Guardian (possibly a human) selects within that restricted slate under a risk budget $\alpha$. While suitable public benchmarks for fully agentic evaluations are limited, the formulation and guarantees are designed with such potential applications in mind. Our experiments use multiple choice question-style benchmarks as controlled proxies. Notably, Section~5.2 (helpfulness--harmlessness) already goes beyond binary right/wrong by using discrete safety severities (e.g., 0--3), illustrating CA on non-binary correctness outcomes.

Free text within CA: minimal recipe.

For given calibration context $x_i$ we can have an arbitrarily large but finite potential output set $A(x_i)$. Sample $K$ free-text candidates from the Primary (e.g., across reasoning budgets/prompts/beam) and inherently consider these to be the $K$ top scoring $p(x_i,a)$. Call this $A_K \in A_i$. Record Primary/Guardian scores $\{p(x_i,a)\}$ for $\{a \in A_K\}$ and $\{g(x_i,a)\}$ for $\{a\in A_K\}$. Sample the Guardian model for its output and consider this to be $\arg\max_{a \in A(x_i)} g(x_i,a)$. Then we are able to define the bounded guardrail loss $L_i(\lambda)$ as in Eq. 2, compute $\widehat R_n(\lambda)=\tfrac1n\sum_i L_i(\lambda)$, and select the smallest $\hat\lambda$ satisfying the CRC inequality in Eq. 3. If no $\hat\lambda$ satisfied the CRC inequality then we would need to increase $K$ and re-run.

Deployment (per input; Sec. 4.3/Alg. 1). For a new $x$: (i) obtain $p(x,a)$ for all $a\in A(x)$; (ii) form $C_{\hat\lambda}(x)=\{a:\,p(x,a)\ge \max_{a'}p(x,a')-\hat\lambda\}$; (iii) if $|C_{\hat\lambda}(x)|=1$, return its unique element (commit to the Primary); otherwise (iv) query the Guardian only on $C_{\hat\lambda}(x)$ and output $\arg\max_{a\in C_{\hat\lambda}(x)} g(x,a)$. This mirrors Sec. 4 exactly and carries the same finite-sample guardrail guarantee from CRC.

Note that this does not change any core mechanism behind how CA works and does not require any changes to it as stated, it is just a specific proposal for eliciting the scores $p(x,a)$ and $g(x,a)$ for the calibration set and applying the CA framework. If instead the scores were easily obtainable using a functional form or some other method then that would also suffice. The main point is that at its technical foundations Conformal Arbitrage does not pertain just to questions that are in multiple choice style settings, but can truly apply to any setting with contexts and finite action sets.

Why a two-model, single-step router:

Although CA could support deeper commit/escalate cascades, we focus on a two-model, single-step policy to keep the mechanism transparent. CA does, in principle, admit multi-stage cascades in which a single calibrated $\hat{\lambda}$ still provides finite-sample control of the final guardrail loss; one can also use stage-specific thresholds $\{\lambda_t\}$ calibrated jointly use multi-lambda conformal risk control. While cascades are a natural fit for cost–accuracy trade-offs, for helpfulness–harmlessness the notion of “depth’’ is less canonical, and the restricted-conformal set mechanism is the primary lever we wished to isolate. Our core objective here is to balance two competing metrics under closed-API constraints, rather than to identify the cheapest model capable of answering a query; moreover, the Guardian-as-human interpretation corresponds to a one-shot escalation with an explicit risk budget $\alpha$. For these reasons, we adopt the two-model setting as a principled and sufficient testbed for our theory and black-box deployment goals, noting that the methodology extends to deeper cascades without changes to the underlying analysis.

We hope that the above responses adequately address the reviewer's questions, especially regarding the scope beyond multiple choice, and that they further inform your already favorable review. We thank the reviewer again for their time and engagement with our work.

We thank the reviewer for highlighting that Conformal Arbitrage (CA) effectively leverages performance gaps between models via a lightweight router, and for noting that this yields improvements over naive selection. We emphasize that CA extends beyond “model selection”: the Guardian may be a human, enabling scalable oversight with explicit risk budgeting.

Viewing the Guardian as a human links CA to scalable oversight and control in AI safety. With a calibration set, primary and guardrail metrics, Primary outputs, and human assessments, CA provides finite-sample guarantees that the safety (guardrail) metric, measured against human judgments, stays within budget while allowing the Primary to act autonomously whenever possible. Section 5.2 (PKU SafeRLHF) illustrates the helpfulness–harmlessness trade-off. More broadly, the human-Guardian perspective is a theoretical contribution to AI safety beyond model selection/routing.

Q1. Choosing $\hat{\lambda}$ in Algorithm 1.

Conformal Arbitrage uses conformal risk control to select $\hat{\lambda}$ directly from data. Given a calibration set and the bounded guardrail loss in Eq.(2), we apply Eq. (3) to choose the smallest $\lambda$ whose adjusted empirical risk is at most the target $\alpha$. This yields finite-sample, distribution-free control of the guardrail.

Q2. Long-to-short (test-time compute) and safety generation.

CA extends naturally to variable test-time compute: the Primary proposes multiple candidates under different reasoning budgets (short/medium/long). Take the primary metric to be compute (tokens/latency/cost) to be minimized and the guardrail to be task performance (e.g., accuracy) with a bounded loss. CA forms the $\hat{\lambda}$-relaxed set $C_{\hat{\lambda}}(x)$ from Primary scores $p(x,a)$ (any logits-free quality/confidence proxy; one may also use a monotone transform favoring lower-compute candidates) and uses CRC to calibrate the smallest $\hat{\lambda}$ achieving the target accuracy level on the final output. For safety generation, replace “accuracy” with a bounded safety loss; CA then uses CRC to calibrate $\hat{\lambda}$ to satisfy the safety budget while letting the Primary act autonomously when appropriate.

Q3. Why is CA below random at $\alpha=0.25$ in Fig.1?

Appendix B.5 (Fig. 3, Table 6) provides insight on this question. At $\alpha=0.25$, calibration yields $\hat{\lambda}$ of about $0.277$ and a mean conformal set size of about $1.457$, so $C_{\hat{\lambda}}(x)$ is often a singleton. Under our restricted-conformal set passing policy, the Guardian receives only $C_{\hat{\lambda}}(x)$; if the correct answer was pruned, it cannot be recovered. In contrast, the random router would provide the full set of potential answers, yielding a slight advantage at this single budget. The unrestricted ablation (CA$^\star$) confirms the mechanism: when the Guardian sees the full set of potential answers, CA outperforms random at comparable cost. We adopt the restricted-set policy by design to bias the Guardian toward high-primary-utility options under a guardrail budget.

We hope these clarifications are helpful and address the reviewer’s questions.