xFinder: Large Language Models as Automated Evaluators for Reliable Evaluation

Thank you for your detailed review and constructive feedback on our research work. Below are our detailed responses, which we hope will address your concerns.

Answer 1 for Weakness 1

Missing Related Work: The work [1] is also highly related to this work ...

Thank you for your valuable suggestion. The paper you recommended is indeed highly valuable, as its proposed EVOUNA dataset makes significant contributions to the automated evaluation of open-ended question-answering tasks. Similar to our work, it aims to enhance the accuracy and reliability of automated evaluations. We have updated the Related Work section of our paper to include a citation to this work, further enriching the discussion of related studies.

Answer 2 for Weakness 2

Annotation Agreement: Human rechecking is ... between annotators?

As you rightly pointed out, manual review is critical for the annotation of the KAF dataset. In Section 4.2, we briefly introduced the multi-round annotation and manual review procedures for the KAF dataset. Regarding the inter-annotator agreement, we have provided additional explanations in Appendix B.2 of the revised paper. Furthermore, the annotation guidelines have been included in Appendix B.2.1 of the updated version.

Answer 3 for Weakness 3

Human/Case Study: Except for those numbers in the experiment tables ...

Thank you for your suggestion. While our paper provides a detailed performance comparison between xFinder and other methods, conducting further case studies is indeed valuable. In fact, we have presented some failure cases of RegEx-based evaluation frameworks in Figure 2 and Appendix D.1 of the paper. Additionally, in Section 3, we formally defined xFinder's approach to these tasks to facilitate a more systematic understanding of its performance.

For extraction paradigms based on LLMs, such as GPT-4, our analysis identified three main failure modes:

• Despite being explicitly prompted to extract key answers from the LLM output, GPT-4 often attempts to directly answer the evaluation question itself, leading to extraction errors.
• Some LLMs exhibit weaker instruction-following capabilities and include multiple key answers in their outputs for evaluation tasks. In such cases, we label the response as "[No valid answer]." However, GPT-4 may still extract one of the key answers, resulting in incorrect extraction.
• Although GPT-4 has relatively strong instruction-following abilities, its extracted content can occasionally contain redundancies, leading to mismatches with the Gold Label. For instance, when the expected Gold Label is "A," GPT-4 might extract "(A) computer savvy."

The Judge Model approach combines both extraction and matching steps. This multi-step reasoning poses challenges for LLMs [2]. Furthermore, Judge Models directly determine the correctness of the evaluated LLM's output or provide a score, which can introduce reliability issues. Even when a Judge Model produces a correct evaluation result, analysis might reveal that it does not genuinely understand the evaluated question, and its result could simply be a random guess, coincidentally correct.

To address these issues, we decompose the evaluation process into two distinct steps: key answer extraction and matching. By designing the KAF dataset and xFinder model, we aimed to enhance the reliability of the evaluation process. Experimental results demonstrate the effectiveness of this approach.

Answer 4 for Weakness 4

Writing: I recommend adding more content about experimental settings ...

Thank you for your suggestion. The first paragraph of Section 5 in the original paper provides a description of the experimental setup. However, due to the extensive scope of our experiments, we provided additional details in the appendix for reference. To improve readability, we have revised the first paragraph of Section 5 in the latest version of the paper to more clearly explain the experimental setup (the revised content is highlighted in blue).

Answer 5 for Question 1

Why does the key extraction task need to have short text and alphabet option categories ...

We specifically included both short text and alphabet option settings because restricting evaluation tasks solely to the alphabet option may introduce certain reliability issues (refer to lines 51–68 of the paper). We explored and validated these issues in our study (refer to lines 523–528 of the paper). Moreover, this dual-setting approach extends the applicability of xFinder and helps enhance the reliability of evaluation results. Additionally, other studies have also investigated and analyzed the potential impact of using the alphabet option on the reliability of evaluation results [3, 4].

Answer 6 for Question 2

Does the order of the xticks in the bump charts affect the comparison between alphabet option and short text?

Thank you for your question! The xticks are solely used to represent the ranking positions of different evaluation methods on the 10 tested LLMs, and their order does not affect the comparison results between alphabet option and short text. The task format (e.g., alphabet option and short text) itself influences the rankings, which are independent of the xticks' order.

Answer 7 for Question 3

What is the trade-off between xFinder and simply adding one or more regular expression patterns in RegEx ...

Regarding the trade-offs between xFinder and RegEx-based methods, it is true that strong LLMs have advantages in learning patterns and executing instructions, but many weaker LLMs still exhibit limited instruction-following capabilities. Moreover, it is important to distinguish between the evaluation performance of LLMs and their instruction-following abilities. The improvements brought by xFinder primarily stem from its adaptability to complex answer patterns and its generalizability, rather than relying solely on specific pattern recognition.

First, while RegEx can perform well with simple and fixed patterns, it often requires manual creation of specific patterns for various unique cases when dealing with the complex and irregular answer formats of many LLM responses. This approach is not practical and lacks scalability. In contrast, xFinder leverages the semantic understanding capabilities of LLMs to extract key answers from context, demonstrating strong generalization abilities even in the face of highly variable answer patterns. Additionally, xFinder goes beyond simple pattern matching by understanding the relationship between questions and answers, thereby improving Extraction Accuracy. Compared to the pattern-dependent RegEx methods, xFinder demonstrates higher reliability and evaluation accuracy, as evidenced by our experimental results.

In terms of efficiency, we recognize that LLM-based methods incur higher computational costs compared to RegEx-based approaches. However, xFinder's advantage lies in reducing the need for manual adjustments and minimizing dependence on complex regular expressions. This is especially beneficial in scenarios where answer patterns are diverse, as it effectively improves Extraction Accuracy and enhances the reliability of evaluation results. Furthermore, we analyzed xFinder’s evaluation efficiency (refer to Table 5 in the paper), showing that it can evaluate 200 samples in an average of just 10.67 seconds. While this efficiency still falls short of RegEx-based methods, it surpasses existing Judge Models and is sufficient to meet the needs of most researchers and evaluators.

Thank you again for your valuable feedback. If you have any further questions or comments, please let us know at any time.

[1] Wang, C., Cheng, S., Guo, Q., et al. (2024). Evaluating Open-QA Evaluation. Advances in Neural Information Processing Systems, 36.

[2] Yang, S., et al. (2024). Do Large Language Models Latently Perform Multi-Hop Reasoning? ACL 2024. Retrieved from https://aclanthology.org/2024.acl-long.550.

[3] Balepur, N., Ravichander, A., Rudinger, R. (2024). Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question? arXiv preprint arXiv:2402.12483.

[4] Li, J., Hu, R., Huang, K., et al. (2024). PertEval: Unveiling Real Knowledge Capacity of LLMs with Knowledge-Invariant Perturbations. arXiv preprint arXiv:2405.19740.

Thank you for your detailed review and valuable comments. Below are detailed responses to your concerns.

Answer 1 for Weakness 1

Although the KAF dataset is used to validate xFinder’s performance, the paper lacks... generalizability to... different datasets and error types.

We would like to clarify our exploration of xFinder's generalization ability across different datasets. In the KAF dataset, we specifically designated a Generalization Set to evaluate xFinder's generalization capabilities. For this Generalization Set, we used evaluation datasets completely different from the Training Set used for fine-tuning xFinder (e.g., OpenbookQA and SIQA). Additionally, we incorporated responses generated by various LLMs (e.g., Llama3-8B-Instruct and Qwen1.5-MoE-A2.7B-Chat) and designed multiple distinct prompting templates. These efforts ensured that the Generalization Set contained a diverse range of LLM responses.

This setup provided a comprehensive test of xFinder's ability to generalize across different datasets and various LLM responses. As shown in Table 2 of the paper, xFinder's performance on the Generalization Set demonstrates its strong generalization ability. Details about the composition of each part of the dataset are provided in Appendix B.1.

Answer 2 for Weakness 2

The paper does not include sufficient analysis on the impact of xFinder on... outcomes, such as a comparison with other extraction methods.

Regarding xFinder's impact on final evaluation results, we conducted extensive experiments in our paper. In Table 3, we compared the Judgement Accuracy of xFinder, other extraction-based LLM evaluation frameworks, Judge Models, and GPT-4. The results show that among the extraction-based frameworks, the highest Judgement Accuracy achieved by OpenCompass is only 88.7%, while PandaLM's Judgement Accuracy is 51.9%. The 33B JudgeLM achieves a Judgement Accuracy of 78.13%, and GPT-4 as a Judge achieves 84.2%, both significantly lower than the 97.61% achieved by xFinder.

In Table 4, we analyzed the discrepancies between Extraction Accuracy and Judgement Accuracy across various methods. The baseline method with the smallest discrepancy, OpenCompass, has a 14.32% gap between its Judgement Accuracy and Extraction Accuracy, highlighting the significant unreliability of traditional RegEx-based extraction methods. In contrast, our xFinder-llama38it shows a gap of only 2.43% between its Judgement Accuracy and Extraction Accuracy, effectively reducing errors caused by inaccurate answer extraction.

Additionally, in Section 5.3, we presented the evaluation results of different extraction-based frameworks across various datasets (further comparisons between xFinder and baseline methods are provided in Appendix D.3). The results demonstrate significant inconsistencies among the evaluation outcomes produced by different frameworks, indicating the unreliability of current LLM evaluation results and corresponding leaderboards. These findings further highlight the robustness and reliability of xFinder's evaluation results.

Answer 3 for Weakness 3

There is a lack of detailed analysis of the KAF dataset’s quality, such as inter-annotator agreement metrics.

To ensure the quality of the KAF dataset, we implemented rigorous multi-round annotation and manual review procedures, employing different annotation strategies for the Training Set, Test Set, and Generalization Set.

1. Training Set: To enhance annotation efficiency, we adopted a semi-automated annotation strategy. Specifically, we used GPT-4 with different prompts (refer to Appendix F.2) to generate two sets of annotations. We then applied the Self-Consistency approach to identify data items with inconsistent annotations. These items were subsequently manually annotated to ensure accuracy.

2. Test Set and Generalization Set: For these sets, we conducted two rounds of manual annotation on all data items to ensure label accuracy and consistency.

For data that requires manual annotation, each data item is annotated by two different annotators. If the two annotators produce different results for the same item, the authors will recheck the annotations and make the final decision.

Details about the dataset annotation procedures can be found in Section 4.2 of the paper. Additionally, we have included guidelines on manual annotation in Appendix B.2.1 of the revised version of the paper. We believe that these measures effectively ensure the quality of the KAF dataset, providing a reliable foundation for training and evaluating xFinder.

Thank you again for your valuable feedback. If you have any further questions or comments, please let us know at any time.

Thank you for acknowledging our work and providing valuable suggestions. Below are detailed responses to your specific comments and questions, aiming to address your concerns.

Answer 1 for Weakness 1

The techniques may not be applicable to responses where the answer is not a short, extractable phrase.

As you correctly pointed out, the primary motivation behind designing xFinder is to address the unreliability of evaluation results caused by inaccurate answer extraction in LLM evaluations. Therefore, the first version of xFinder was designed to support four major types of mainstream evaluation tasks: alphabet option, short text option, categorical label, and Math option.

At the same time, we recognize that these task types may not fully encompass all real-world application scenarios. However, we believe that xFinder has strong transferability. In future work, we plan to enhance the diversity of the dataset to include a broader range of task types. For example, we aim to incorporate tasks such as long-text comprehension and generation, open-ended questions, and complex reasoning tasks. By adding these new task types, we hope to further improve the applicability and versatility of xFinder.

Answer 2 for Weakness 2

Although the results are promising, I suspect the technique might be replaced by stronger LLMs used as judges ...

Thank you for your valuable feedback! We understand that future, more advanced LLMs as Judges, combined with improved prompting techniques, may further enhance evaluation performance. However, as shown in Table 3 of the paper, the accuracy of GPT-4 as a Judge in our experiments is still significantly lower than that of xFinder. While LLMs continue to improve, their use as Judges remains limited by efficiency and cost constraints. Even with excellent performance, their evaluation efficiency is relatively low, and the high computational cost makes them less practical for evaluators.

In contrast, our efficiency analysis demonstrates that xFinder achieves high accuracy while maintaining low evaluation costs. On average, xFinder evaluates 200 samples in just 10.67 seconds (refer to Tables 5 and 6 for efficiency and cost analysis).

Additionally, as you noted, as a contribution to the field of datasets and benchmarks, the KAF dataset and xFinder provide a solid foundation for reliable evaluation and can serve as a valuable resource for future research on automated evaluation methods. We also fully agree on the potential of combining xFinder with other techniques and plan to explore broader application scenarios in the future to further enhance its generality and practicality.

Answer 3 for Question 1

What is the prompt you used for GPT-4 as Judge (CoT)?

Thank you for your question! We apologize for the oversight in the initial draft, where we omitted the prompt details for GPT-4 as Judge. In the latest version of the paper, we have added the prompts used for both GPT-4 as Judge and GPT-4 as Judge (CoT) in Appendix F.3.

Thank you again for your valuable feedback. If you have any further questions or comments, please let us know at any time.

You are a diligent and precise assistant tasked with evaluating the correctness of responses. Think step by step as you make your evaluation.
—
We request your feedback on whether the model's response correctly answers the user question above. Follow these steps to make your evaluation:
1. Think step by step: Read the user question carefully.
2. Think step by step: Review the reference answer and understand the key points it covers.
3. Think step by step: Compare the model's answer with the reference answer.
4. Think step by step: Determine if the model's answer addresses the key points in the reference answer and correctly answers the question.

First, provide your reasoning in detail. Then, clearly state your judgement as either "Correct" or "Incorrect."

Please present your response in the following JSON format: 
{{
    "reasoning": "Your step-by-step reasoning here.", 
    "judgement": "Correct or Incorrect"
}}


—
Question: {question}

Reference Answer: {reference}

Model's Answer: {answer}

Thank you for acknowledging our work and providing valuable suggestions. Below are detailed responses to your specific comments and questions, aiming to address your concerns.

Answer for Questions

To be perfectly honest ... would it possible to prompt GPT-4 or other very big LLM using ICL with the constructed data, and how would it perform?

Thank you for your thoughtful review of our work! We are glad to provide clarifications regarding your concerns about the novelty and comparisons to other LLM evaluation methods.

First and foremost, our study focuses on enhancing the reliability of LLM evaluations rather than proposing a new model architecture. This work falls within the domain of datasets and benchmarks, aiming to support the iterative improvement of LLMs through more robust evaluation methods. Through our analysis, we identified significant reliability issues in current evaluation frameworks, which can prevent researchers from accurately understanding the limitations of the evaluated LLMs. This, in turn, may lead to biased advancements in LLM development. Thus, improving the reliability of LLM evaluations is of paramount importance.

There have been numerous studies proposing automated evaluation methods based on Judge Models [1, 2, 3]. However, previous research has highlighted issues with the generalization ability and fairness of these fine-tuned Judge Models [4]. Specifically, Judge Models often suffer from overfitting to the datasets used for fine-tuning, resulting in biases when applied to real-world evaluation scenarios.

Additionally, the Judge Model approach combines key answer extraction and matching into a single process, introducing a "skipping step" problem. Such multi-step reasoning poses challenges for LLMs [5]. The Judge Model directly determines the correctness or assigns a score, which might be unreliable. Even if the Judge Model's evaluation result is correct, we might find that it did not truly understand the problem itself, and the evaluation result might just be a random guess that happened to be correct. The experimental results in Table 4 of our paper corroborate this issue. For example, when using GPT-4 as a Judge, there is a significant gap between its Extraction Accuracy when splitting extraction and matching and its overall Judgement Accuracy when combining these steps.

To address this, we decompose the evaluation process into two steps: key answer extraction and matching. We designed and constructed the KAF dataset and the xFinder model to support this approach. Experimental results demonstrate that xFinder excels in performance, achieving significantly higher Judgement Accuracy than existing Judge Models (refer to Table 3). Moreover, xFinder demonstrates higher evaluation efficiency compared to current Judge Models (refer to Table 5), substantially improving the reliability of LLM evaluations.

Regarding the question of using datasets to prompt models like GPT-4 for evaluation, we conducted relevant comparison experiments, as shown in Table 3 of our paper. The results indicate that even GPT-4's performance lags significantly behind xFinder (a model with only 0.5B parameters). Furthermore, prompting such powerful LLMs as evaluators is inefficient, incurs high costs, and is impractical for widespread adoption in real-world evaluation tasks. We believe this further demonstrates the practicality and value of xFinder.

Thank you again for your valuable feedback. If you have any further questions or comments, please let us know at any time.

[1] Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023. Retrieved from https://openreview.net/forum?id=uccHPGDlao.

[2] Wang, Y., et al. (2024). PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization. ICLR 2024. Retrieved from https://openreview.net/forum?id=5Nn2BLV7SB.

[3] Zhu, L., Wang, X., Wang, X. (2023). JudgeLM: Fine-Tuned Large Language Models are Scalable Judges. arXiv preprint arXiv:2310.17631.

[4] Wang, Y., et al. (2024). Large Language Models are not Fair Evaluators. ACL 2024. Retrieved from https://aclanthology.org/2024.acl-long.511.

[5] Yang, S., et al. (2024). Do Large Language Models Latently Perform Multi-Hop Reasoning? ACL 2024. Retrieved from https://aclanthology.org/2024.acl-long.550.