Split and Merge: Aligning Position Biases in Large Language Model based Evaluators

Response to Reviewer cPgm

Q1

It is in doubt about the semantic meaningfulness of alignment. The method... In many scenarios, alignment may not yield meaningful results.

The paper mentions that the split should preserve the answer order, then it may... Then an oder-invariant alignment will not be able to find an optimal alignment.

Thank you for your question. As we clarified in our global response, our framework is designed to accommodate a range of open-ended questions and responses generated by various LLMs. While Figure 1 depicts point-to-point responses, this represents only a small portion (less than 10%) of the total responses, which are predominantly open-ended.

About the order insensitive answer, we clarify that determining whether two sentences remain semantically equivalent after sequential exchanges is a long-standing and complex problem [5,7], and our work does not attempt to address this issue. It is possible that the requirement of our "order preservation" is so strict that we may miss some cases where the order is not important. However, we argue that "order preservation" is a reasonable requirement for most cases, as it is hard to determine whether the order is important or not.

Moreover, the goal of this work is to address the position bias issue in LLM evaluators, which have been proven to be capable of providing high-quality evaluations compared to human evaluators [1,2]. To achieve this goal, our framework tries to identify a segmentation choice with balanced length (length alignment) or better semantic overlap (semantic alignment). Both alignment methods serve to enhance the consistency of evaluation results, as outlined in Section 4.5.

Q2

How does Portia work for short sequences, which cannot be split into shorter segments?

Thank you for pointing this out. Please first refer to our global response for the discussion of the responses' length.

Moreover, we clarify that we do not have a strict requirement for the length of the responses. As long as the response has content that can be segmented (i.e., >k), our framework is applicable. During our experiments, we found that it is uncommon for responses to be too short to be segmented, and in such cases, our framework would not make any enhancements. This is because, as discussed in our global response, the position bias is typically not present in extremely short answers, and thus no adjustments are necessary.

Q3

Limited applicability: The proposed system, i.e. the pipeline of splitting, alignment... so it is interesting to see if a system that is fine-tuned toward MT benchmark can be applicable to other forms of assessment tasks.

Thank you for your insightful suggestion. As we stated in the global response, the benchmark we select includes a diverse range of open-ended questions from eight primary categories, such as writing, reasoning, and roleplay. The eight distinct LLMs that are considered in this work possess varying capabilities and preferences, resulting in diverse responses. Furthermore, this benchmark has been widely used in previous research [3,4], indicating its applicability to a variety of assessment tasks.

Moreover, we acknowledge that it would be interesting to investigate whether position bias exists in other benchmarks such as alpaca_eval [6]. This benchmark contains shorter answers, with 53% of the responses being less than 412 characters. As shown in the global response, it is difficult for LLM evaluators to exhibit position bias when the answers are shorter. Therefore, exploring position bias in other benchmarks with shorter answers could provide valuable insights, and we leave this as a potential direction for future research.

[1] Cheng-Han Chiang and Hung-yi Lee. 2023. Can Large Language Models Be an Alternative to Human Evaluations?. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631, Toronto, Canada. Association for Computational Linguistics.

[2] Liu, Yang, et al. "Gpteval: Nlg evaluation using gpt-4 with better human alignment." arXiv preprint arXiv:2303.16634 (2023).

[3] Zheng, Lianmin, et al. "Judging LLM-as-a-judge with MT-Bench and Chatbot Arena." arXiv preprint arXiv:2306.05685 (2023).

[4] Wang, Peiyi, et al. "Large language models are not fair evaluators." arXiv preprint arXiv:2305.17926 (2023).

[5] Schwartz, Daniel G. "Axioms for a theory of semantic equivalence." Fuzzy Sets and Systems 21.3 (1987): 319-349.

[6] https://huggingface.co/datasets/tatsu-lab/alpaca_eval

[7] Finch, Andrew, Young-Sook Hwang, and Eiichiro Sumita. "Using machine translation evaluation techniques to determine sentence-level semantic equivalence." Proceedings of the third international workshop on paraphrasing (IWP2005). 2005.

Thanks for your response. I agree with other reviewers and think the major issue of this paper is still the generalizability. The experiments in the global response do not alleviate my concerns: It shows that the variable lengths do not affect the consistency, but the point of "split and merge" seems groundless in this scenario. Moreover, it does not explain how Portia works for semantically different responses.

Response to Reviewer YB2D

Q1

For answers/texts that are significantly different in length. Or with almost the same length but significantly different in symantics or content.

Thank you for pointing this out. As stated in our global response, our framework is designed to accommodate responses that may differ significantly. However, we acknowledge that evaluating responses with similar lengths but significantly different semantics or content can pose a considerable challenge, even for human evaluators. Previous research has demonstrated that LLM evaluators possess the ability to provide high-quality evaluations [1,2,3] while facing the position bias issue. Our framework addresses this issue by splitting and merging responses, and our experimental results show that with our framework's assistance, LLM evaluators can provide more consistent evaluation results. Furthermore, our framework has the potential to improve the agreement rate between human and LLM evaluators.

Q2

If the answers/texts have different orders, changing the order of one of the text may affect the evaluation of the generation quality.

Thank you for pointing this out. As emphasized in Section 3.1 of our framework, the preservation of the original sequence of information presented in the answers is a crucial consideration in our design. Instead of altering the content of the generated responses, our framework adds split boundary prompts (as exemplified in Table A.2) between each segment of the merged response. This approach allows LLM evaluators to receive the same information as the original responses for evaluation purposes.

Q3

Besides, is it possible that the merging operation makes it harder (longer context, in a comparing way) for the LLMs to understand and evaluate the output?

Thank you for your question. In Section 2 of our paper, we introduce the concept of pairwise comparison as a means of evaluating open-ended questions. This approach involves presenting two responses side-by-side for evaluators to compare, providing a longer context than the single-wise comparison. Pairwise comparison is a widely used method in recent research [1,2,3] due to its ability to generate high-quality evaluations.

Our framework, which includes split boundary prompts of several tokens in addition to the standard pairwise comparison, is designed to further improve the evaluation process. We present experimental results in Section 4.4 that demonstrate the effectiveness of our framework in enhancing the agreement rate between human evaluators and LLM evaluators. These findings suggest that our framework does not impede the ability of LLM evaluators to comprehend and evaluate the output.

[1] Zheng, Lianmin, et al. "Judging LLM-as-a-judge with MT-Bench and Chatbot Arena." arXiv preprint arXiv:2306.05685 (2023).

[2] Wang, Peiyi, et al. "Large language models are not fair evaluators." arXiv preprint arXiv:2305.17926 (2023).

[3] Zeng, Zhiyuan, et al. "Evaluating large language models at evaluating instruction following." arXiv preprint arXiv:2310.07641 (2023).

Response to Reviewer SWLn

Q1

The method is not clearly described, making it hard for me to properly evaluate the soundness and contribution of the paper. ... I am willing to adjust my review based on the author's response.

Thank you for pointing this out. We appreciate your valuable feedback and take steps to address the concerns you raised. Specifically, we make revisions to the description of the semantic alignment part in Section 3.2 to provide a more comprehensive and understandable explanation. We clarify that $n_s$ represents the index number of the current segmentation, and its maximum value is equivalent to the total number of computation operations $Cal$ as discussed in Section 4.3. Additionally, we emphasize that the depth of loop traversal is dependent on the chosen value of $k$, and hence, we have used $n_s$ to denote the extent of traversal.

To further enhance the clarity of our framework, we include two detailed algorithm descriptions of the function $partition$ in Appendix F. These descriptions provide a step-by-step process of semantic alignment when $k$ is equal to 2 and 3, respectively. We believe that these revisions will provide a more comprehensive and lucid explanation of our framework. [Please refer to our revised paper for all the details. We have marked major changes in red.]

Q2

The models used in this paper are not very well explained. I cannot see which Llama-2 model is used, the Llama-2 model or the Llama-2-chat model.

Thank you for your question. We apologize for any confusion caused by the lack of clarity regarding the specific Llama-2 model used in our paper. We would like to clarify that the Llama-2-chat models are used in our experiments, and necessary revisions are made to Appendix G of our manuscript to reflect this.

Q3

The experiment setting in Appendix B is not clear. The paper seems to require that the scores in different scales need to have a linear mapping relationship... Refer to [1, 2] for more experiment details.

Thank you for your insightful feedback and suggestions. We would like to clarify that the primary focus of our paper is on pairwise comparison, as it is particularly relevant to our study's scope. Since single-wise comparison does not require comparing two responses together, there is no position bias. We acknowledge that single-wise comparison is a valid approach for evaluating open-ended questions, and it has been employed in previous research, including the works the reviewer mentioned [1,2], as well as other recent studies [3,4,5].

We apologize for any confusion our description of the experiment setting in Appendix B may have caused. Our intention is to emphasize that the absolute scores do not strictly follow a linear mapping relationship in different scales, which may weaken their significance. We understand that our statement regarding the inconsistency of single-wise comparison may have been overstated in Section 2.

In our revised manuscript, we provide a more balanced discussion of both single-wise and pairwise comparison methods in Appendix B, acknowledging the merits of each approach. We emphasize the primary focus of our paper and ensure that our descriptions are clear and accurate, reflecting the limitations and scope of our study.

Q4

Missing references: [1] and [2] are the earliest two works that use LLMs as evaluators and should be included in the related works.

Thank you for bringing to our attention the missing references [1] and [2] which are the earliest works that use LLMs as evaluators. We appreciate your suggestion and include these references in the related works section of our revised manuscript to provide a comprehensive overview of the relevant literature.

Q5

What happens if the responses to be rated are very short, or what if the two responses that need to be rated have a very different length?

Thank you for your question. Please refer to our global response.

[1] Cheng-Han Chiang and Hung-yi Lee. 2023. Can Large Language Models Be an Alternative to Human Evaluations?. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631, Toronto, Canada. Association for Computational Linguistics.

[2] Liu, Yang, et al. "Gpteval: Nlg evaluation using gpt-4 with better human alignment." arXiv preprint arXiv:2303.16634 (2023).

[3] Zheng, Lianmin, et al. "Judging LLM-as-a-judge with MT-Bench and Chatbot Arena." arXiv preprint arXiv:2306.05685 (2023).

[4] Wang, Peiyi, et al. "Large language models are not fair evaluators." arXiv preprint arXiv:2305.17926 (2023).

[5] Zeng, Zhiyuan, et al. "Evaluating large language models at evaluating instruction following." arXiv preprint arXiv:2310.07641 (2023).

Response to Reviewer RHzc

Q1

The method design is a bit ad-hoc and does not seem to be generalizable enough for evaluating open-ended generation, ...., since generation evaluation usually has to consider the answers' coherency (potentially spanning over long-range contexts that should be evaluated as a whole).

Thank you for pointing this out. In our global response, we clarify that our framework is applicable to open-ended questions, where the responses can be largely different from each other.

Furthermore, we clarify that our framework does not involve splitting a response into multiple segments and querying LLM-based evaluators multiple times for separate evaluations. This approach may compromise the coherence of the response and incur considerable extra cost. Instead, our framework splits and merges the response, and the evaluator is queried only once with the merged response.

To facilitate the evaluators' ability to distinguish between the two responses and avoid misidentification, we include split boundary prompts (as exemplified in Table A.2) between each segment of the merged response. As a result, the LLM-based evaluators receive the same information as the original responses, in the same order, and with clear boundaries. This assists the evaluators in evaluating the responses as a cohesive whole.

Q2

The experiment evaluation only focuses on the consistency rates and does not provide evidence of its impact on the accuracy (e.g., whether the calibrated prediction corresponds better to human judgments). Such evaluations are obviously necessary because any naively deterministic method (for example, a system that always prefers the longer answer) will have a consistent rate of 100% but will not be useful in practice.

Thank you for your question. We would like to clarify that in Section 4.4 of our study, we have presented evidence regarding the impact of our framework on assessment accuracy. Specifically, Table 3 displays the results of our evaluation, demonstrating that our framework is effective in enhancing agreement rates between human and LLM evaluators. The table reveals that the average human agreement on original LLM assessments increases by 11.25% after using our framework. Further analysis can be found in Section 4.4. These quantitative results shall provide compelling evidence that our framework successfully augments the assessments of all LLM evaluators, resulting in greater accuracy.

Q3

I expect to see stronger baselines included (there are several methods aiming at enhancing LLM-as-evaluators, though they may not be completely addressing position bias). I'd encourage the authors to also discuss the new studies that came out after the ICLR submission deadline such as Zeng et al. (this is not a weakness but a suggestion)

Stronger baselines

Thank you for your suggestion. Following your advice, we compare our framework with various baselines, including the vanilla method, MEC, BPC, and HITLC. Specifically, the VANILLA method asks evaluators to simply output their preference without any explanation, while the prompts used in our baseline are presented in Appendix A.1. MEC, BPC, and HITLC are the methods proposed in [2], which require multiple turns of querying or human effort.

We present the results in the table below, where the agreement rate between humans and corresponding LLM evaluators for GPT-4 and GPT-3.5 are provided.

From the table, we can observe that our framework outperforms all methods except HITLC, which requires human effort at an extremely high cost. Given that our framework is a fully automated method with low cost, we believe that it serves as a strong baseline for future research.

New studies

Thank you for your valuable suggestion. We have considered it and have made the necessary revisions to our manuscript. In particular, we have included the findings of [3] that position bias exists in the evaluation of instruction-following models in Section 1.

[1] Zheng, Lianmin, et al. "Judging LLM-as-a-judge with MT-Bench and Chatbot Arena." arXiv preprint arXiv:2306.05685 (2023).

[2] Wang, Peiyi, et al. "Large language models are not fair evaluators." arXiv preprint arXiv:2305.17926 (2023).

[3] Zeng, Zhiyuan, et al. "Evaluating large language models at evaluating instruction following." arXiv preprint arXiv:2310.07641 (2023).

Dear Reviewers,

We thank you again for your constructive reviews and appreciate the time and effort you have invested in discussing with us.

We would like to clarify one final point: our system, PORTIA, is specifically designed to address the position bias issue, rather than aligning the reasoning capabilities in LLM-based evaluators. Therefore, if an LLM evaluator lacks the capability to adequately judge a pair of semantically different responses, this falls outside the scope of PORTIA's intended purpose. PORTIA is solely focused on rectifying inconsistencies that arise when the order of a pair of semantically different responses is altered, leading to a change in the judgment of the LLM evaluator.

In this context, we believe that PORTIA is generalizable to handling open-ended answers because, regardless of their semantic differences, we can always mix the two previously individual answers (as illustrated in Figure 1). Such a mixing operation effectively eliminates the position bias in pairwise LLM-based evaluation, as indicated in our evaluation data and supplementary experiments.

Best regards,

Anonymous Authors