We thank the reviewer for their valuable feedback and address the concerns raised below.

Limited novelty compared to prior works: While the MBR framing is clean, many recent UQ methods already combine confidence and consistency (e.g., SAR, Semantic Entropy). The main novelty lies in formalizing their combination via MBR. I would like to ask the authors to explain a bit more their motivation from [9] and [10], and what is the novelty of this formulation.

We acknowledge that there are existing methods which blend confidence and consistency. However, we argue that our method proposes several novel contributions:

While existing approaches have explored combining confidence and consistency and they did serve as an inspiration for our work, they tend to aggregate both confidence and consistency signals over all the sampled outputs, which reduces sensitivity to the expected quality of the particular model output (https://arxiv.org/pdf/2305.19187).

This observation informs our design of CoCoA, which formalizes uncertainty estimation through a Minimum Bayes Risk (MBR) framework, interpreting uncertainty as expected risk based on both confidence and semantic consistency via explicit utility modeling. Within the MBR framework we introduced, semantic similarity and consistency are not used to reweight model confidence scores, but rather a separate signal integrated into the utility function.

To further improve efficiency, we introduce CoCoA Light-a supervised approximation of the consistency component-which enables uncertainty estimation without the need for repeated sampling at inference time, which is another limitation of existing hybrid approaches.

Lack of theoretical justification: The main section for the CoCoA seems to build up a combination of model confidence and semantic inconsistency directly, without explaining the intuition and theoretical justification.

While we don’t have the full theoretical treatment for the choice of the loss function for CoCoA (which is often a heuristic choice in MBR framework), we can consider a case study of the default confidence choice (MSP) that leads to the strongest performance.

In this case, $U(y_*) = - \log p(y_* | x) \cdot \frac{1}{M} \sum_i (1 - s(y_*, y_i)$

Equivalently, $U(y_*) = - \log [ p(y_* | x)^{\frac{1}{M} \sum_i (1 - s(y_*, y_i))}] = - \log [ p(y_* | x)^{ 1- \frac{1}{M} \sum_i s(y_*, y_i)}]$.

This formula is essentially a criterion to select the maximum probability sequence but tempered by a semantic consistency of this sequence with other samples. From the MBR perspective, it is a natural criterion that balances choosing the sequence with high probability and simultaneously from a part of the distribution with highly concentrated meaning (high average similarity).

Measurement using PRR: While CoCoA improves PRR, it's less clear how well it calibrates model uncertainty. My understanding is this metric measures the ranking but not model uncertainty directly.

While calibration is indeed a very important topic in UQ, it requires uncertainty scores to be interpretable as probabilities and the output correctness measure to be binary. Neither is true for a large portion of our experimental setup. Most UQ methods output unbounded values which can not be interpreted as probabilities, while many quality metrics are continuous, especially in NMT and summarization tasks. So straightforward computation of ECE is not possible.

However, we adopted an approach to calibration proposed in (https://arxiv.org/abs/2406.15627) and fitted an isotonic regression to pairs of raw ue scores and response quality, and report MSE between ground truth quality and prediction of the regression here:

Results for Evaluated Sequence – Greedy Sample: Mean MSE between quality and calibrated UQ score across datasets for each task. The best performing method is in bold, and the second-best is underlined. Arrows indicate improvement in CoCoA over the base version.

CoCoA family methods still retain their edge over the baselines in most of the task-model pairs in this setting.

Similarity function bottleneck: Results depend heavily on the semantic similarity function used. The paper acknowledges this but could further explore when and which score should be used for certain tasks and why (which is really helpful if we do not have an additional held out set, e.g. open-ended generation tasks).

This is indeed an important point, we thank the reviewer for bringing this up. Indeed, the choice of the similarity function is the important step here. We hypothesize that for general natural language generation with outputs of limited length (1-2 sentences) any capable NLI/CrossEncoder model will suffice. For longer generations, models that perform similarity estimation on the chunks with further aggregation over the chunks (like AlignScore for example) can be a better choice. For special cases, like code generation, a domain-specific similarity function would probably be needed.

Error bars missing: I am wondering why the variance is not reported in this paper.

Thank you for pointing this out. We only ran a single instance of the experiment. Each run requires generating 2,000 outputs from the model, followed by extensive quality and other downstream evaluations. Due to the significant computational cost, repeating the experiment enough times to produce statistically meaningful error bars was not feasible within our resource constraints. We acknowledge this as a limitation and plan to address it in future work, resources permitting.

We appreciate the reviewer’s feedback and address their concerns below.

Introducing a Framework Rather than a Specifc Method: as is acknowledged by the authors in the limitations, the different versions of CoCoA excel at different tasks. An entire framework of possible options is therefore presented that can accommodate many prior methods and has multiple design choices. The method empirically speaking is largely a combination of two families of prior work methods, each performing better in different scenarios. This erodes the methods empirical utility.

While it is true that our method combines two families of prior approaches, this is not a weakness but an intentional design choice. In the general case, a maximum sequence probability (MSP)-based combination yields strong, reliable performance. The framework makes it flexible: when time and computational resources are available, it can be further optimized to better suit the specifics of a task. This makes it not only effective in general settings but also adaptable to specialized use cases. In this sense, we view the introduction of a framework not as an erosion of empirical utility, but as a practical strength - offering a solid default with room for task-specific refinement.

Treatment of risk combinations: even though the authors attempt to derive their measure from a theoretically grounded standpoint of prediction risk, the resulting CoCoA measures are still quite arbitrary. For instance, the authors do not provide a theoretical explanation for multiplicative combination of the confidence and consistency and instead settle for picking one with greater empirical performance. If the Confidence and Consistency parts both correspond to some sort of risk, I would imagine them being additive rather than multiplicative.

In Appendix C.2, we compare our multiplicative formulation to additive and full-matrix consistency variants: Additive combination often allows one signal to dominate, especially since some information-theoretic confidence scores (e.g., entropy) are unbounded while consistency is bounded in [0, 1]. This imbalance undermines the contribution of consistency and practically requires proper empirical selection of relative contribution of each source. This selection would be task-dependent and would generalize poorly to the OOD settings. On the other hand, since consistency is bounded between 0 and 1, it serves as a natural scaling factor on confidence. This enforces mutual alignment between the two signals without requiring learned weights or tuning.

While additive risk formulations are common in theory, in practice this setting involves differently scaled signals, and the multiplicative form has shown stronger empirical performance with a clear intuitive justification. That said, we agree that deeper theoretical analysis is a promising direction for future work.

Evaluation Nuances: the authors discuss the advantages of PRR over AUROC (even though its not the focus of the paper), yet acknowledge that they have to cut the PRR evaluation at 50% rejection to avoid artifacts.

The choice to cut off at 50% was deliberate, as it rarely makes sense to reject all of the answers in practical settings. By focusing on the 50% most uncertain cases, we target the region where rejection is most relevant and actionable. This provides a more realistic and interpretable measure of performance in scenarios where selective prediction is actually useful.

Furthermore, due to diminishing sample size, the expected quality of the output on the remaining dataset is a heteroscedastic value wrt rejection rate, and extreme rejection can lead to very noisy estimates.

That being said, we report PRR up to 100% rejection here, and CoCoA-family methods retain their edge over the baselines even in this setup.

Results for Evaluated Sequence – Greedy Sample: Mean PRR up to full rejection across datasets for each task. The best performing method is in bold, and the second-best is underscored. Arrows indicate improvement in CoCoA over the base version.

Would introducing weighing for the two terms into the AdditiveCoCoA improve its performance for specific tasks?

Yes, introducing learned or tuned weights could improve the performance of AdditiveCoCoA for specific tasks. However, adding weights would effectively turn the method into a supervised or task-specific approach, as these weights would need to be tuned on downstream validation sets to balance the contributions of each term. This tuning may yield better empirical performance but sacrifices the task-agnostic and unsupervised nature of the original CoCoA formulation. In contrast, our multiplicative formulation avoids this trade-off by enforcing mutual agreement between the two signals without requiring tuning or additional parameters.

How would the evaluation picture change if the PRR rejection cut-off was changed from it's 50% value?

We report PRR up to 100% rejection here, and CoCoA-family methods retain their edge over the baselines even in this setup.

What is the theoretical justification for multiplicative CoCoA combination rather than additive?

The key justification lies in how consistency and confidence are treated as signals of uncertainty within the Minimum Bayes Risk (MBR) framework. Consistency is bounded between 0 and 1, while confidence metrics such as perplexity (PPL) or mean token entropy (MTE) are unbounded. As a result, an additive combination would require careful supervision and tuning of weights to ensure a balanced contribution from each signal. Without proper tuning, one signal could dominate or be underrepresented, reducing the robustness of the final score. By contrast, the multiplicative combination used in CoCoA avoids this issue, as it inherently normalizes the interaction between signals, preserving their relative informativeness without requiring task-specific weight calibration.

Regarding the CoCoA-MSP, the top-performing variant in the tables - $U_{MSP}$ does it use the logarithmic or just the likelihood?

It uses logarithmic value for the confidence: $U_{conf} = -\log(p(y \mid x))$.

though there is a small factual error in line 961, as $-\log(p(y\mid x))$ inverts the order compared to $p(y^*|x)$ because of the negative sign

Indeed, but likelihood formulation uses $1 - p(y \mid x)$ as a $U_{conf}$ so the order is kept between logarithmic and likelihood formulations. We agree that this is confusing, as intuitively confidence implies that higher values represent higher confidence. Following [https://arxiv.org/abs/2305.19187] we use terms uncertainty/confidence to distinguish between types of uncertainty related to the generation process in general, and uncertainty of the particular output $y$. To reduce confusion in this matter, we should probably reverse the consistency term as well, to align the whole expression with the intuitive understanding of confidence. We will do this in the camera-ready version (if accepted), thank you for highlighting this issue.

Perhaps likelihood in range [0,1] and consistency in range [0,1] would lead to a better signal match in the additive setting (rather than log likelihood + log consistency )

Thank you for this important observation. Here we present the results of additive formulation with likelihood instead of logarithm for confidence term:

While additive formulation with normalized likelihood gets close to the the performance of multiplicative CoCoA MSP, its more unstable, and falls short on NMT and SUM. At the same time it requires confidence to be a bounded likelihood, reducing the generality of the multiplicative formulation, where confidence term can be of any nature. For example, using token entropy in additive setting catastrophically reduces performance, while multiplicative formulation retains performance close to other choices of the confidence term.

At the same time, I do not fully understand why the difference in scales would be such a significant issue for the additive combination in the case of CoCoA-Light, since it already incorporates a small trained model whose output could be normalized to match the distribution of the confidence term.

Indeed, but the distribution of the confidence term can be task-specific. For example, generations of different lengths have vastly different likelihoods. Learning this distribution would make the method less generally applicable. For example here we provide an additive formulation with a scaling factor for consistency term selected for best performance on QA tasks:

While having strong performance in-domain (QA), it falls short on summarization.

We thank the reviewer for the insightful comments. Below, we would like to address the concerns raised by reviewer:

Limited Novelty: The core idea of combining confidence and consistency feels somewhat incremental. The paper could be strengthened by a deeper investigation into the relationship and interaction between these two types of uncertainty, beyond the proposed multiplicative combination. The current insight into why this combination is so effective is not fully explored.

On limited novelty:
While our work builds upon established concepts of confidence and consistency, our contribution is threefold.

(1) We focus on selective generation, where the decision to output a response is based on the confidence score of a specific candidate, rather than modeling uncertainty over a potential input. Specifically, we frame our approach as a variant of Minimum Bayes Risk Decoding, which defines a utility function that naturally favors responses that are both confident (according to the model) and consistent (with other sampled candidates).

(2) We show that our MBR-based framework is flexible and performs well across various uncertainty measures.

(3) We demonstrate that the consistency component can be learned, significantly reducing the computational cost associated with extensive sampling.

On discussing potential reasons for the best performance of multiplicative formulation:

We agree that more discussion in the combination strategy is valuable. In Appendix C.2, we compare our multiplicative approach to additive and full-matrix variants. The additive form often lets one signal dominate (e.g., consistency gets overwhelmed by high-entropy outputs). Full-matrix averaging dilutes pairwise relationships by incorporating irrelevant comparisons. The multiplicative form works best because it naturally favours an output if it is both confident and semantically consistent. This enforces alignment between both criteria without introducing extra parameters or reweighting.

Missing Baselines: The evaluation does not include comparisons with "verbalized confidence" methods[1]

We do not focus on verbalized uncertainty methods, as prior work (https://openreview.net/pdf?id=gjeQKFxFpZ, https://aclanthology.org/2025.tacl-1.11/ ) has shown that LLMs tend to express high confidence regardless of actual correctness - particularly in smaller models, like 7-8b models used in our experiments. In such models these methods often yield poorly calibrated scores that do not reflect true reliability.

Incomplete Evaluation Metrics: The evaluation relies primarily on failure detection metrics (PRR, AUROC). While important, these do not measure confidence calibration. Including metrics like the Expected Calibration Error (ECE) would provide a more complete picture of the UQ quality.

While calibration is an important aspect of uncertainty quantification (UQ), it assumes that uncertainty scores are interpretable as probabilities and that output correctness can be measured in binary terms. Neither of these assumptions holds for much of our experimental setup. Most UQ methods produce unbounded scores that lack a probabilistic interpretation, and many of the quality metrics we use - particularly in tasks like neural machine translation (NMT) and summarization - are continuous rather than binary. As a result, standard calibration metrics such as Expected Calibration Error (ECE) are not directly applicable.

However, we adopted an approach to calibration proposed in (https://aclanthology.org/2025.tacl-1.11/) and fitted an isotonic regression to pairs of raw UQ scores and response quality values, and report here MSE between ground truth quality and prediction of the regression here:

Results for Evaluated Sequence – Greedy Sample: Mean MSE between quality and calibrated UQ score across datasets for each task, lower values mean better calibration. The best performing method is in bold, and the second-best is underlined. Arrows indicate improvement in CoCoA over the base version.

CoCoA family methods still retain their edge over the baselines in most of the task-model pairs in this setting.

Limited Model Scale: The experiments are conducted on models up to 8B parameters. The claims would be more convincing if the method's effectiveness were also demonstrated on larger, state-of-the-art models (e.g., Llama 3 70B, even with quantization).

We appreciate the reviewer’s point regarding model size. Due to limited compute resources, our experiments focus on models up to 8B parameters (Similar to https://aclanthology.org/2025.tacl-1.11/, https://neurips.cc/virtual/2024/poster/97746, https://arxiv.org/abs/2503.05318, https://arxiv.org/abs/2406.04306 ).

Our method does not rely on model-specific tuning and is compatible with quantized or cached generation setups, making it feasible to apply to larger models in future work. We agree that testing on 70B-scale models like LLaMA 3 would further strengthen our claims, and we plan to explore this direction in the future.

How would you adapt the CoCoA framework for use with closed-source, black-box models (e.g., via the GPT-4 API) where direct access to token probabilities is not available?

While CoCoA relies on confidence estimates such as token-level probabilities, adaptation to black-box models like GPT-4 is still feasible: GPT-4 and other APIs expose log probabilities for generated outputs. This allows the computation of confidence-based metrics like Maximum Sequence Probability (MSP). In cases where logprobs are unavailable, they can be estimated empirically from the sampled responses, or verbalized confidence can be used instead, which should perform well enough for larger models which are often used in a black-box setting.

The paper omits comparisons with verbalized confidence methods. Could you elaborate on the potential advantages or disadvantages of CoCoA relative to this line of work?

We do not focus on verbalized uncertainty methods, as prior work (https://openreview.net/pdf?id=gjeQKFxFpZ, https://aclanthology.org/2025.tacl-1.11/) has shown that LLMs tend to express high confidence regardless of actual correctness - particularly in smaller models. In such models these methods often yield poorly calibrated scores that do not reflect true reliability.

While the experiments show that a multiplicative combination works well, have you explored other aggregation functions (e.g., attention-based weighting)?

In Appendix C.2, we evaluated alternatives to the multiplicative combination-including additive scoring and full-sample averaging. The underperformance of these alternatives stems from their inability to balance both confidence and consistency signals effectively.

While we did not experiment with attention‑based weighting of token logprobs (if that’s what is being referred to in the question), we stress that any confidence measure can be used in the CoCoA (CCP, TokenSAR, attention-based re-weighting, etc). We show that even using very basic approaches like perplexity and sequence probability can improve over the baseline methods.

How the reasoning-models like QwQ-32B perform on the CoCoA framework?

We have not evaluated CoCoA on reasoning models such as QwQ-32B, so we cannot make claims about their performance. However, if a long reasoning chain is considered as a part of the output, we hypothesize that more complicated measures of confidence and consistency are required. The reason for that is that over long sequences, simple aggregation of log probabilities can be drowned in noise and simple NLI similarity is not well-defined. In general, uncertainty quantification for reasoning setup and long-form outputs in general requires special considerations that we are planning to tackle in further work on CoCoA, but which are out of scope for this particular submission.

We thank the reviewer for their thoughtful comments and the time they invested. Please find below our responses to the concerns raised.

W1. There is some dependency on an external similarity model. They mention task and domain dependency in the limitations section. I wonder if the length of the generated text is also crucial. I might have missed it, but it would be good to comment on the effect of length of the generated output on CoCoA.

We absolutely agree that expected length of the generated sequence should be a major factor when selecting both particular confidence and consistency estimates. One has to be mindful whether this length falls into the domain that the similarity model was trained on. Most modern similarity models are trained on rather short sequences, so longer outputs might require special handling. On the other hand, from the confidence standpoint, some sort of importance weighting for individual token contribution might be needed.

W2. QA (especially long form), summarization and machine translation are somewhat similar tasks. Adding GSM8k is useful. It would be good to see how these methods work on code generation tasks.

We fully agree that code generation is an exciting domain to test UQ capabilities of LLMs. It would require a careful selection of the similarity measure for consistency estimation so we consider this to be out of scope for this submission. However, we definitely will pursue this in our future work.

W3. More details on the CoCoA light training will be useful. What is the size of the training set?

Training set sizes:

The average training set among datasets was around 5k examples. The architecture used was a simple MLP with one hidden layer. We have observed that optimal behavior was reached after only a couple of epochs, so probably the train set size can be further reduced with longer train time. We also highlight that these training examples don’t need to be labelled for correctness, only consistency among samples should be computed for each.

How will you use this approach for large black box models provided by the likes of OpenAI, Claude and Gemini. We can generate consistency scores. Can the confidence be approximated somewhat similar to what you have tried with CoCoA light? Assuming it is allowed under their terms, can small models be learned from such large black box models to provide confidence scores and used in this CoCoA setting?

Some black box models give probabilities for top-1 tokens, which allows us to estimate confidence directly. For models which do not expose such information in their API, one could use empirical probability estimation from the relative frequencies of the outputs in the samples, or opt for verbalized approaches for confidence estimation, which are calibrated better for large models.

In Figure 1, you have illustrated the problem with model confidence. You tend to alleviate this problem by using it together with consistence measures. But is it possible that if the models are large and the generated text is small, the consistency based methods will also miss the variations in the possible outputs? What I mean is most of the samples look like romeo and juliet, and very few look like romeo & juliet or romeo and juliet,. This might happen if the samples are few and based on temperature sampling. In such a scenario how does MBR or CoCoA help?

https://arxiv.org/abs/2406.15627 shows that for very large models, pure consistency estimation methods still perform well. This suggests that CoCoA would be a strong choice in such settings as well. We do agree, however, that the artifacts highlighted in Figure 1 are more prevalent in the outputs of smaller models. Conducting benchmarking with larger models would indeed be a valuable direction for future research.