Distributional LLM-as-a-Judge

Dear reviewer 3U8S,

Thank you so much for your precious time and your positive feedback. We will now address your concerns.

For Weakness 1: The approach requires datasets with multiple human annotations per sample to obtain empirical approximations, where are not commonly seen in practice.

Thank you for this valuable comment. We acknowledge that requiring multiple human annotations per sample is indeed a necessary condition for our method. However, we would like to offer the following points to alleviate this concern:

1. Fundamental Importance: As discussed in our paper, we believe that modeling the distribution of human judgments is fundamentally important. The natural disagreements among annotators carry valuable information that is often lost when reducing labels to a single point. This richness is particularly critical in high-stakes domains—such as medical diagnosis or policy-making—where capturing the full range of expert perspectives is essential. Therefore, we argue that the cost of collecting multiple human annotations is justified, given the significance of the problems we aim to address.

2. Generalization Capability of Our Framework: A key strength of modern LLMs is learning transferable knowledge. To verify that our framework learns a transferable representation of human disagreement, rather than overfitting to dataset-specific annotation patterns, we conducted an OOD experiment (training on SNLI, testing on the unseen ChaosNLI [1] dataset).

   The results are presented in the table below, where we report mean ± std over several runs and * denotes a statistically significant improvement (p < 0.05). The data confirms our method's strong generalization to the unseen dataset. This suggests that the learned alignment capability is transferable, potentially reducing the need to collect extensive multi-annotated data for every new task.

   Model	Method	KL Divergence (↓)	Accuracy (↑)
   Qwen2.5-7B	Raw model	$3.94_{±0.00}$	$60.3\\%_{±0.0\\%}$
   	Single-point	$1.01_{±0.01}$	$66.5\\%_{±0.9\\%}$
   	Distribution (Ours)	$\mathbf{0.31}^*_{±0.01}$	\mathbf{67.8\\%}^*_{±0.5\\%}
   LLaMA3.1-8B	Raw model	$0.68_{±0.00}$	$57.8 \\%_{±0.0\\%}$
   	Single-point	$1.15_{±0.05}$	$60.3\\%_{±0.1\\%}$
   	Distribution (Ours)	$\mathbf{0.46}^*_{±0.02}$	\mathbf{62.7\\%}^*_{±0.1\\%}

3. Robustness with Limited Annotations: In fact, during the early conceptualization of this work, we were fully aware of this potential limitation. To address this issue, our adversarial training strategy was specifically designed to enhance robustness, even when only a small number of annotations are available per sample. Our original experiments (Table 1 and 2) already demonstrate that our method achieves satisfactory performance under such settings.

For Weakness 2: The approach may incur extra computational overhead in the introduced adversarial training strategy, but the overall efficiency appears undiscussed in the paper.

We thank the reviewer for this important practical point. The computational overhead is introduced by the inner PGD loop (K=5 steps), which finds the worst-case label distribution. Crucially, this PGD optimization computes gradients with respect to the label distribution p(x), not the model parameters θ. During this inner loop, the model's output distribution q_theta(x) is treated as a fixed constant. As a result, these PGD steps are computationally inexpensive as they do not involve any backpropagation through the main language model.

To empirically validate this, we conducted a direct comparison of the training efficiency, benchmarked on the MNLI dataset with the Qwen2.5-7B model on a single NVIDIA A100 GPU. The table below shows our method incurs a modest slowdown of approximately 21%, which we believe is a worthwhile trade-off for the significant gains in alignment quality.

We will add a discussion of this analysis and include these results in the final version of our paper.

We sincerely thank you for your time and insightful comments, which are very important for improving our paper. We hope our responses have addressed your concerns.

References:

[1] Nie, Y., et al. "What can we learn from collective human opinions on natural language inference data." EMNLP (2020): 9131-9143.

Dear reviewer 6xen,

Thank you so much for your precious time and your thoughtful feedback. We will now address your concerns.

For Weakness 1: The using of CE loss and accuracy metric may contradict our paper's goal of modeling distributions.

We thank the reviewer for this fundamental question. These designs are not contradictory, but are deliberate and principled decisions to ensure both training stability and meaningful evaluation.

1. CE loss -- for learning stability:

The reviewer is correct that we use a standard CE loss on the majority-vote label. However, its role is not as a primary objective, but as an auxiliary regularizer for training stability. As we noted in our paper (Line 129-133), this design is directly inspired by the Knowledge Distillation [2, 3] framework, where a "hard" label loss is used to stabilize learning from a "soft" target distribution. Our own results in Figure 3 empirically validate this necessity: on challenging datasets like MT-Bench, a pure KL objective (α=1.0) is unstable and performs worse, demonstrating the crucial role of the CE term as a stabilizing anchor.

2. Accuracy metric -- for meaningful evaluation:

Similarly, Accuracy is a secondary, complementary metric, with KL divergence being our primary measure of success. We include it for two practical reasons: it serves as a valuable indicator of our model's ability to capture the majority consensus, and more critically, it provides a crucial bridge for comparison with prior work that relies solely on this measure. This allows us to demonstrate that we achieve our primary goal without sacrificing performance on this traditional standard.

We will add a more detailed discussion of this rationale to the paper. We thank the reviewer again for this valuable prompt.

For Weakness 2: Not using the highly relevant ChaosNLI [1] dataset.

We sincerely thank the reviewer for this important suggestion. We agree that ChaosNLI is an ideal and challenging benchmark for our distribution alignment framework. Consequently, we have conducted a new set of experiments on the ChaosNLI dataset. The results are presented in the table below, where we report mean ± std over multiple runs. The * indicates that the improvement of our method over the single-point baseline is statistically significant (p < 0.05).

The results on ChaosNLI are fully consistent with our original findings: our method significantly outperforms baselines in KL divergence while maintaining competitive accuracy. We will incorporate these results into the final paper and are grateful for the valuable suggestion.

For Weakness 3: Analyze the method’s robustness to other unseen datasets.

We thank the reviewer for this excellent suggestion. While our paper already demonstrated robustness to label noise (Sec 4.5), we agree that testing for out-of-distribution (OOD) generalization is a complementary and even more challenging evaluation. Therefore, we conducted a new OOD experiment, fine-tuning models on the SNLI training set and evaluating them directly on the unseen ChaosNLI test set:

These OOD results demonstrate our method's superior generalization over the baseline, confirming that our framework learns a transferable representation of human disagreement instead of merely overfitting to dataset-specific patterns. We will integrate this crucial analysis into our final paper and are grateful for the valuable suggestion.

For Weakness 4: The need for statistical significance tests.

We thank the reviewer for this crucial point on experimental rigor. We completely agree that statistical testing is essential, particularly for metrics with small margins like accuracy. We have conducted new multi-run experiments (N=5), focusing specifically on the datasets with the narrowest accuracy margins: SNLI and MNLI.

The table highlights our method's key achievement: yielding substantial and statistically significant improvements in KL divergence, while maintaining competitive accuracy. While these multi-run experiments were limited to SNLI/MNLI due to time constraints, we will include these rigorous statistics for all datasets in the camera-ready version. Thank you again for this feedback.

For Q1: Why not using JS loss?

We thank the reviewer for this insightful question. Our choice of KL divergence is based on two main reasons:

• First, our choice is motivated by its established success as the standard measure in paradigms like Knowledge Distillation [2, 3], which shares the core technical goal of aligning a model's output distribution with a target "soft" distribution.
• Second, KL divergence generally offers more stable gradients for optimization compared to JS divergence, which can suffer from vanishing gradients. [4]

To empirically validate this choice, we conducted a new experiment comparing both training objectives on SNLI using Qwen2.5-7B. The results below confirm that using KL divergence yields stronger performance across all metrics:

For Q2: Typo on Line 204-205.

We thank the reviewer for catching this error. We will correct this typo in the final version.

For Q3: Perturbation taking into account annotator information.

This is a fascinating and thought-provoking question. We agree with the reviewer's excellent intuition: if the annotator pool lacks diversity, the collected distribution may be biased, and a larger perturbation radius ε would be a principled way to account for this unobserved variance. The primary practical obstacle, however, is the lack of annotator metadata in most crowdsourced datasets, and we would like to leave this direction as an important future work.

For Q4: Accuracy of GPT-4o-mini vs. GPT-4o on NLI tasks

Thank you for this sharp observation. GPT-4o-mini indeed achieves slightly higher accuracy on the NLI datasets than GPT-4o. Our hypothesis is that on a mature and relatively well-defined task like NLI, the accuracy of powerful models can be very close.

We sincerely thank you for your time and insightful comments, which are very important for improving our paper. We hope our responses have addressed your concerns.

References:

[1] Nie, Y., et al. "What can we learn from collective human opinions on natural language inference data." EMNLP (2020): 9131-9143.

[2] Hinton, Geoffrey, et al. "Distilling the knowledge in a neural network." arXiv:1503.02531 (2015).

[3] Sanh, Victor, et al. "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter." arXiv:1910.01108 (2019).

[4] Arjovsky, Martin, et al. "Wasserstein GAN." PMLR (2017): 214-223.

Dear Reviewer a38u,

Thank you for your thoughtful feedback. We will address your concerns below.

For Weakness 1: Despite the reduction in KL divergence, the improvements in accuracy are minimal. As such, more motivation is needed to justify modeling the full distribution given the added data annotation cost.

The core value of our work extends beyond improving single-point accuracy. Instead, our goal is to capture the rich, underlying structure of human evaluations—such as the variance of opinions, which signals the level of consensus. This distributional information is valuable in high-stakes applications like medical diagnosis, where understanding the full spectrum of expert opinions is more useful and safer signal than a single, overconfident prediction.

Accordingly, our evaluation is centered on KL Divergence as the primary metric, because it directly quantifies how well our model captures this entire information-rich distribution. Accuracy serves as a secondary metric to ensure our method captures the majority consensus and remains competitive with traditional standards.

We will expand on this motivation in the final paper.

For Weakness 2: The evaluation is underwhelming. First, the margins (particularly accuracy) are slim, and only point estimates are provided. Second, the dataset choice could be improved to better fit the LLM-as-a-Judge narrative. Lastly, the method is only tested on in-distribution tasks so the generaliability of the framework is not clear.

1. On slim margins and point estimates

We have conducted new multi-run experiments (N=5) on the datasets with the narrowest margins: SNLI and MNLI. The results are reported in the table below, where we report mean ± std over multiple runs. The * indicates that the improvement of our method over the single-point baseline is statistically significant (p < 0.05).

The results highlight our key achievement: yielding substantial and statistically significant improvements in KL divergence, while maintaining competitive accuracy. We will include these rigorous statistics for all datasets in the final version.

2. On the choice of datasets

We have conducted new experiments on these modern benchmarks (HelpSteer2 [1] for Qualitative Assessment task, and HelpSteer2-Preference [2] for Human Preference Understanding task). The results for both datasets are presented below.

HelpSteer2

HelpSteer2-Preference

The results on both modern benchmarks provide strong additional evidence for our framework's effectiveness. We will incorporate these extensive new results into the final paper and are grateful for this suggestion.

3. On out-of-distribution tasks and generalizability

We thank the reviewer for this excellent suggestion. While our paper already demonstrated robustness to label noise (Sec 4.5), we agree that testing for out-of-distribution (OOD) generalization is a complementary and more challenging evaluation. Therefore, we conducted a new OOD experiment, fine-tuning models on SNLI and evaluating them directly on the unseen ChaosNLI [3] dataset. The results are presented below.

These OOD results demonstrate our method's superior generalization over the baseline, confirming that our framework learns a transferable representation of human disagreement. We will integrate this crucial analysis into our final paper and are grateful for the valuable suggestion.

For Q1: [3] should be cited in Section 4. The formulations are identical: a CE + KL objective solved in a PGD min–max loop, only this work shifts the KL term from input perturbations to label perturbations.

We thank the reviewer for their sharp observation and the opportunity to clarify this connection.

While there is a structural resemblance in the optimization form, the two works operate in fundamentally different domains with distinct goals. TRADES addresses robustness to external input attacks in computer vision. Our work, in contrast, addresses distributional alignment with human opinions in the paradigm of LLM-as-a-Judge.

This fundamental difference in goals is directly reflected in our loss design. In TRADES, the KL term is an auxiliary regularizer to enforce output smoothness against input perturbations. In our framework, the KL term is the primary objective, designed to directly align model outputs with the target human distribution. Our auxiliary CE loss merely serves to stabilize this primary alignment task, a design validated by our hyper-parameter analysis.

We will revise our manuscript to cite TRADES and add this important contextual discussion. Thank you again for this valuable prompt.

For Q2: Can this approach generalize to unseen datasets, e.g., with a different label space?

As demonstrated in our new OOD experiment (presented in our response to Weakness 2.3), our framework generalizes well. The results confirm that our distribution alignment method's capabilities transfer to the unseen ChaosNLI dataset significantly better than the single-point baseline.

For Q3: In Figure 3, why does the KL divergence increase considerably at alpha=1.0 for MTBench, but none of the other tasks? Is there some feature of MTBench that necessitates the CE term more so than the other datasets?

This is a sharp observation. Our hypothesis is that this phenomenon is tied to the high difficulty and subjectivity of certain tasks. The greater disagreement among annotators leads to more complex target distributions p(x). For such challenging distributions, a pure KL divergence objective (α=1.0) can be unstable.

To verify this hypothesis, we conducted the same analysis on HelpSteer2-Preference. Similar to MT-Bench, this is a benchmark for subjective human preference. We observed the exact same pattern—a sharp increase in KL divergence when the CE loss was completely removed:

The consistent results from both datasets show that the auxiliary CE loss is crucial for stabilizing these difficult alignment tasks.

We sincerely thank you for your time and insightful comments, which are very important for improving our paper. We hope our responses have addressed your concerns.

References:

[1] Wang, Zhilin, et al. "Helpsteer2: Open-source dataset for training top-performing reward models." arXiv:2406.08673 (2024).

[2] Wang, Zhilin, et al. "HelpSteer2-Preference: Complementing Ratings with Preferences." arXiv:2410.01257 (2024).

[3] Nie, Y., et al. "What can we learn from collective human opinions on natural language inference data." EMNLP (2020): 9131-9143.

Dear Reviewer Lp1a, Thank you for your rigorous and constructive review. We present our detailed responses below.

For Weakness 1: No evaluation on tasks more similar to what LLM-as-a-Judge is often used for in practice, ... incoporate a more diverse range of tasks.

Thank you for the feedback on practical relevance. We find that some of our chosen tasks correspond well to the paradigms the reviewer mentioned: SummEval is a 'quality evaluation' task, and MT-Bench addresses 'pairwise preference prediction'.

To further strengthen our evaluation, we have conducted new experiments on the recently proposed datasets, which align more closely with common use-cases. HelpSteer2 [1] provides a state-of-the-art benchmark for 'quality evaluation' of modern LLM generations, while HelpSteer2-Preference [2] offers a more fine-grained benchmark for 'preference prediction' on paired LLM responses. The results are presented below, where we report mean ± std over multiple runs. The * indicates that the improvement of our method over the single-point baseline is statistically significant (p < 0.05).

HelpSteer2

HelpSteer2-Preference

As the results show, our framework's effectiveness is further validated on these modern and practical benchmarks. We will incorporate these extensive new results into the final version of our paper and are grateful for the suggestion.

For Weakness 2: Limited experimental results and minimal evaluation... or raw LLM judges.

Thank you for the feedback. We would like to respectfully clarify that our original evaluation was already covering Overall Performance (Table 1), Component Contributions (Table 2), Hyper-parameter Analysis (Figure 3), and Robustness to Label Noise (Table 4). We have also made three substantial additions during rebuttal:

1. Out-of-Distribution (OOD) Generalization: We conducted a new OOD experiment, fine-tuning models on SNLI and evaluating them on the unseen ChaosNLI [3] dataset. The results below provide strong evidence that our framework learns a transferable representation of human disagreement.

   Model	Method	KL (↓)	Accuracy (↑)
   Qwen2.5-7B	Raw model	$3.94_{±0.00}$	$60.3\\%_{±0.0\\%}$
   	Single-point	$1.01_{±0.01}$	$66.5\\%_{±0.9\\%}$
   	Distribution (Ours)	$\mathbf{0.31}^*_{±0.01}$	\mathbf{67.8\\%}^*_{±0.5\\%}
   LLaMA3.1-8B	Raw model	$0.68_{±0.00}$	$57.8 \\%_{±0.0\\%}$
   	Single-point	$1.15_{±0.05}$	$60.3\\%_{±0.1\\%}$
   	Distribution (Ours)	$\mathbf{0.46}^*_{±0.02}$	\mathbf{62.7\\%}^*_{±0.1\\%}

2. Validation on More Relevant Benchmarks: We have validated our method on new, contemporary benchmarks, as detailed in our response to Weakness 1.

3. Enhanced Statistical Rigor: To improve the rigor of our claims, we have incorporated significance tests into all new experimental results presented in this rebuttal. We are committed to this standard of validation and will update the tables from our original manuscript to include these statistics.

For Q1: Highly relevant paper not yet cited: Nie et al. (2021)... a substantial weakness for the paper.

We sincerely thank the reviewer for pointing out this highly relevant work and apologize for the oversight in our initial submission. While we share the foundational goal with Nie et al. (2021), our work is distinguished by a fundamentally different methodology, which we designed for the modern LLM-as-a-Judge paradigm and validated across a broader scope of tasks. Our key distinctions are:

1. Novel Adversarial Robustness Mechanism: our work introduces a key innovation not present in Nie et al.: adversarial training on the label distribution. We identified that the empirical human distribution is a noisy approximation of the true underlying distribution due to limited annotations. Our adversarial strategy is the first to mechanistically address this challenge by training the model to be robust against worst-case perturbations. This significantly enhances alignment fidelity and robustness, a critical aspect that Nie et al. acknowledge as a problem but do not solve.

2. Modern LLM-a-Judge Context with Broader Scope: Our work is situated in the modern LLM-as-a-Judge paradigm, where we validate our framework on powerful modern LLMs across a broader range of tasks beyond NLI (e.g., quality scoring and preference understanding).

We will revise our manuscript to include a thorough discussion of Nie et al. and take this opportunity to broaden our discussion of related literature.

For Q2: If you set a dataset size budget... which method performs better?

This is an excellent and insightful question. To empirically investigate the trade-off in data collection under a fixed budget, we conducted a new experiment on the SNLI dataset. We fixed a budget of approximately 8,000 total annotations and compared three distinct strategies:

• Strategy 1: 8,000 samples, each with 1 random annotation.
• Strategy 2: 2,667 samples, each with 3 random annotations.
• Strategy 3: 1,600 samples, each with all 5 annotations.

The results reveal an interesting trade-off between annotation depth and sample breadth. Strategy 2 achieves the best alignment, outperforming both Strategy 1 (which lacks a direct distributional signal) and Strategy 3 (which lacks sufficient sample for generalization). As a secondary benefit, our approach is also far more computationally efficient, dramatically reducing training time per epoch.

We will add this important analysis to our paper and are grateful for the reviewer's prompt, which led to this deeper and more nuanced finding.

For Limitation 1: Not many alternative methods are compared, related works section is very limited and missing key citations

Please see our detailed response to Question 1 above.

For Limitation 2: Limited experimental investigation

As detailed in our response to Weakness 2, we have conducted substantial new experiments (on new SOTA datasets, OOD generalization, etc.) to form a more comprehensive evaluation.

For Limitation 3: Method requires high-redundancy annotations, which are costly.

We agree this is a key consideration. However, we believe the cost is justified by the fundamental importance of modeling human disagreement for trustworthy AI in high-stakes applications. In fact, our robust alignment method is exactly designed for the case with limited annotations.

We sincerely thank you for your time and insightful comments, which are very important for improving our paper. We hope our responses have addressed your concerns.

References:

[1] Wang, Zhilin, et al. "Helpsteer2: Open-source dataset for training top-performing reward models." arXiv:2406.08673 (2024).

[2] Wang, Zhilin, et al. "HelpSteer2-Preference: Complementing Ratings with Preferences." arXiv:2410.01257 (2024).

[3] Nie, Y., et al. "What can we learn from collective human opinions on natural language inference data." EMNLP (2020): 9131-9143.

Thank you for your additional experimental results and comments. I have revised my score.