$\texttt{BetaConform}$: Efficient MAP Estimation of LLM Ensemble Judgment Performance with Prior Transfer

Thank you, Reviewer gw97, for your detailed review and for recognizing our work as economic, well-written, and extensively validated. Below, we address your questions point by point:

Q1: Binary $\mathrm{Match}(\cdot)$ function

A1: Our rationale for adopting the binary $\mathrm{Match}(\cdot)$ function is that many tasks evaluated by LLM-as-a-Judge can be naturally framed as binary classification, resulting in conclusions of "yes" or "no." Additionally, accuracy in scoring-based judgments can be considered binary, where a judgment is deemed incorrect if the deviation exceeds a certain threshold, otherwise correct.

We acknowledge that the current approach does not support ranking and appreciate your helpful suggestion. We plan to explore incorporating ranking as part of our future work.

Q2: Mixture distribution assumption

A2: Our work assumes a typical judging process without adversarial conditions. Adversarial prompts represent special cases of challenging questions designed specifically to induce incorrect judgments from LLMs. Our sampling assumption is based on random availability of samples.

Although our two-peak assumption originated from observations presented in Figure 3, extensive experiments (see Table 1) demonstrate that our assumption consistently yields better performance compared to baseline methods. This supports the validity of our two-peak assumption across various datasets.

We acknowledge the challenge posed by adversarial prompts, which can intentionally create challenging distributions that degrade estimation accuracy. Addressing this will be part of our future research.

Q3: Performance degradation with GPT-3.5

A3: While distribution transfer aims to enhance estimation accuracy, it may also introduce noise. To mitigate potential negative impacts from the transfer, we designed the transfer mechanism (see Equation 13) to carefully balance reliance on source distributions, thus preventing over-dependence and reducing possible degradation.

Q4: Typos

A4: Thank you for pointing these out. We will revise them in the final manuscript.

Thank you, Reviewer Qt2U, for your detailed review and recognizing our work as economic, well-written, and extensively validated. Below, we address your questions point by point:

W1: Motivation for population-level estimation

An intuitive example of population-level estimation is a classification task, where the model’s accuracy is assessed across the entire dataset rather than a single sample. When given new samples, the estimated accuracy is the accuracy on the entire dataset. Following this intuition and standard practice in LLM-as-a-Judge, we adopt population-level performance estimation.

W2 (1), W3 (1) & Q1: Equation on Line 33

Thank you for highlighting this. Our intention was to indicate that our task involves estimating the number of correct judgments for any input question, but the original equation missed this assumption. We will revise the equation on Line 33 as follows:

where $x$ is a sample and $D$ is the dataset. To achieve this goal, we adopt population-level estimation to represent the distribution for any given sample.

W3 (2): Definition of error rate and error margin

We apologize for the confusion. Definitions are clarified as follows:

• Error rate: The fraction of erroneous judgments made by the ensemble majority vote over the total number of samples in the dataset. It intuitively represents how frequently samples are judged incorrectly.
• Error margin: The absolute difference between the estimated and actual fraction of incorrectly judged samples. This metric assesses the quality of our estimated judgment performance relative to the true performance.

Q2 (1): Application of Condorcet's Jury

We reference Condorcet’s Jury Theorem to highlight the inadequacy of assuming a binomial distribution. Given the following reasons, we believe the Condorcet’s Jury Theorem is applicable:

1. $p>0.5$ implies the model performs better than random guessing. This condition typically holds true in our experimental settings.
2. We respectfully disagree with the notion of high correlation in LLM decisions. While LLMs judge independently without intercommunication, decisions are conditionally dependent since judgments are made based on the same question $x$. Thus, the joint judgment distribution $\mathbb{P}(y|x)$ is conditional, not independent.

Q2 (2): Meaning of “Beta”

Thank you for this critical inquiry. The motivation behind our choice stems from addressing empirical shortcomings in simpler Binomial models. A Binomial distribution assumes a single, fixed probability of correctness ($p$) across all questions, which contradicts our empirical observations (Figure 3). Introducing the "Beta" component enables modeling the probability itself as a random variable, effectively capturing variations in question difficulty.

Q2 (3): Further validation of the Beta-Binomial mixture

To further validate our method, we include additional baseline comparisons involving mixtures of three, four, and five Beta-Binomial distributions, as well as a mixture of two Binomial distributions:

The results show that adding more Beta-Binomial distributions provides minimal or no performance gain compared to our two-distribution choice, justifying our decision. Compared to the mixture of Binomial distributions, our method achieves lower error margins, suggesting our practice is effective.

W2 (2) & Q3: Additional metrics for evaluation

To further assess our method's distribution estimation quality, we provide additional comparisons using Wasserstein Distance and KL Divergence:

Table a: Comparison with baselines using Wasserstein Distance

Table b: Comparison with baselines using KL Divergence

In the results above, we select HaluEval, JudgeBench and BIG-bench and report the average numbers across the three datasets. As shown by the results, our method still outperforms baselines under the Wasserstein Distance and KL Divergence. This shows that our model is consistently better in different metrics。

Q4: Typo and minor issues

Thank you for identifying these issues. We will thoroughly address and correct them in our manuscript revision.

Thank you, Reviewer vhVT, for your detailed review and for recognizing our paper as precise, highly efficient, theoretically rigorous, and broadly validated. Below, we address your questions point by point:

Q1: Two-component mixture distribution

A1: While our initial design of the two-component mixture distribution was based on observations presented in Figure 3, we extensively validated our method across 5 models and 11 datasets. The results consistently demonstrate that our approach outperforms baseline methods, highlighting its generalization capability.

Q2: Reliance on the quality of text embedding

A2: We argue that, in practice, ample unlabeled data for computing embeddings is typically available. Our method significantly reduces the annotation effort required from humans to achieve accurate estimations, making it capable of leveraging the unlabeled data. Moreover, in scenarios where only a few dataset samples are accessible, our method can directly perform estimations without transfer, similar to the setting of Table 1.

Q3: Computational overhead

A3: Our method incurs minimal computational overhead. The conformal prediction and EM algorithm are executed solely on a CPU, typically completing within three minutes per dataset. Additionally, adaptive sampling using conformal prediction allows for efficient estimation using only a small fraction (approximately 200 samples) of the entire dataset. Furthermore, generating text embeddings relies on efficient model inference, further minimizing computational costs.

Thank you, Reviewer NToF, for the detailed review and acknowledging our paper as original, well-motivated, theory-rooted, empirically strong, and clearly written. Below, we address your questions point by point:

Q1: The choice of two distributions

A1: We observed that the judge distribution consistently shows two peaks near all-wrong and all-correct responses, as illustrated in Figure 3. Motivated by this, we adopted a mixture of two Beta-Binomial distributions to capture this pattern. Furthermore, our experimental results (Table 1) demonstrate that our chosen mixture distribution outperforms using a single Beta-Binomial distribution.

To further validate this design choice, we compared our two-distribution mixture with mixtures of three, four, and five distributions, following the setup in Table 1 on judgments from Llama-3.3-70B:

The results show that adding more distributions provides minimal or no performance gain compared to our two-distribution choice, justifying our decision.

Q2: Clarification of mathematical concepts

A2: Thank you for this valuable suggestion. We will add further explanations in the revised manuscript. Brief, intuitive explanations for key concepts are provided here:

• Beta-Binomial Distribution and Mixture: Imagine flipping a coin whose fairness you're unsure about. The Binomial distribution describes how many heads you expect if you know the coin’s bias (probability of heads). The Beta distribution captures your uncertainty about this bias. Combining these, the Beta-Binomial distribution models both the number of heads and your uncertainty about the coin’s fairness. We observed that some questions are "easy" and others are "hard" for LLM judges; thus, we use a mixture of two Beta-Binomial distributions to model both question types simultaneously.
• Conformal Prediction: Conformal prediction generates prediction intervals with guaranteed confidence without making strong assumptions about data distributions. It helps determine "how many samples are enough?" by continuously sampling until the prediction interval is acceptably narrow, ensuring efficiency.
• Expectation-Maximization (EM): EM is an iterative algorithm used to estimate model parameters. It alternates between two steps: the Expectation (E-step), where it estimates missing information based on current parameters, and the Maximization (M-step), where it updates parameters to better fit the data. This process repeats until convergence.

Q3: Procedure for distribution transfer

A3: We apologize for any confusion. The distribution transfer involves the following steps:

1. Estimate distributions on both target and source datasets using EM, initially without transfer.
2. Compute text embeddings for the target and source datasets (Lines 230-233, Equation 12).
3. Transfer distributions from source datasets to the target dataset by refining initial estimates using information from the source datasets (Lines 234-244, Equations 13 and 14). Specifically, we perform a weighted sum of source and target distributions as described in Equation 14.

Q4: Ensemble of different models

A4: Currently, our method only considers ensembles composed of the same model, aligning with common practice in LLM judging. We plan to explore ensembles of different models as future work.