Evaluating LLMs at Detecting Errors in LLM Responses

We appreciate your insightful comments and valuable feedback!

The LLMs employed to generate the response (GPT4 and Llama2) could impact the reliability of the benchmark.

I would appreciate it if you could elaborate on this point. We select GPT-4 and Llama 2 70B because they are widely adopted, and the study of error detection on their responses is well-motivated.

The ReaLMistake benchmark contains a relatively small number of samples compared to other benchmarks with binary labels (e.g., SummEdits and BIG-Bench Mistake).

While we agree that larger is better, we consider the size of our benchmark to be reasonable for the following reasons:

(1) Unambiguity: ReaLMistake includes unambiguous errors with high annotation quality. We expect that a relatively small benchmark can reliably evaluate error detectors.

(2) Diversity: Our benchmark includes diverse instances in three tasks and four error categories.

(3) Annotation Cost: SummEdits and Big-Bench Mistake use synthetic data or relatively simple tasks, which make human annotation easier. Our work focuses on natural mistakes by LLMs, whose error annotation is much more difficult.

In Table 2, I notice that the dataset is biased [...]

Our benchmark includes errors in LLM responses in the natural distribution. We consider that the label distribution is not an issue if a sufficient number of instances are included for each label. When a balanced dataset is required, we can sub-sample instances with the majority label or up-sample instances with the minority label.

Our experiments suggest that our benchmark includes a reasonable number of instances for each label because (1) our experiments capture performance differences between different models and (2) the state-of-the-art models are still far from human performance.

In Table 4, is it suitable to use your defined "Random" recall score as the baseline? [...]

We select our random baseline to simulate a situation in which we know the estimated error frequency of LLM responses. For example, when providers of LLM-based services try to detect mistakes in LLM responses, they often have the estimated error frequency of their LLMs and use it as a baseline.

We also note that there is no universally reasonable random baseline. For example, a random baseline with 50% error probability is relatively strong for error detection on GPT-4 responses (with around 50% errors) but very weak for Llama 2 70B responses (with around 75% errors).

We sincerely appreciate your insightful comments and valuable feedback!

It would be interesting to understand what factors make an error hard to identify. The results in Table 4 are fairly coarse grained. Does the dataset support more fine-grained evaluations?

Our analysis shows that error detection on MathGen (which includes errors in instruction following) is relatively easy for strong models, while the other tasks are difficult for all models. It suggests that instruction-following errors are relatively easy for detectors, but other types of errors (especially context-faithfulness and knowledge) are challenging.

ReaLMistake includes information that can be used for more detailed analysis: error categories and human explanations about the errors. We will provide more analysis in future versions.

[...] it would be useful to evaluate prompting strategies more comprehensively, beyond what is described in Section 4.3.

Beyond Section 4.3, in Section 4.4 and Appendix A.3, we also evaluate a prompting strategy that provides detailed instructions for evaluation steps in prompts, which is motivated by G-Eval. However, it does not improve performance.

We agree that more analysis of prompting strategies, such as decomposing responses, is a promising direction for future research.

It seems like it should be possible to improve recall at the cost of lower precision. How large is that cost?

As shown in Figures 9 and 10, we observe that the recall of strong models (e.g., GPT-4) is not sensitive to prompt engineering. Therefore, as mentioned in Section 4.3, we conclude that there is no easy way to improve the recall of strong models, even at the cost of lower precision.

The errors are generated from a limited number of templates on three tasks. It is uncertain how generalizable the results will be to other types of errors, and those that occur in the wild. The paper is nonetheless a good starting point.

ReaLMistake is suitable for evaluating error detectors because it includes unambiguous errors in LLM responses, collected using well-designed tasks. In addition, to enable our benchmark to provide generalizable analysis, we include three diverse tasks and four error categories widely observed in LLM responses.

We agree that creating more diverse benchmarks is a promising direction for future research. However, collecting unambiguous errors is difficult in tasks in the wild, which are often open-ended and subjective.

We appreciate your insightful comments and valuable feedback!

(1) The source tasks are limited to synthetic tasks like question generation and fact verification. It would be more interesting to see such a dataset constructed on more diverse instructions such as ShareGPT/Alpaca/OpenAssistant.

We design realistic and meaningful tasks in ReaLMistake to collect unambiguous errors in LLM responses, which are useful for evaluating error detectors.

We agree that creating error detection benchmarks with more diverse instructions is desirable. However, collecting unambiguous error annotations in general tasks is challenging, as they are often open-ended and subjective.

(2) The negative correlation between the LLM capacity and error detection recall, though interesting, is a bit unintuitive. The authors also did not offer a reasonable guess of why this happens -- which makes me wonder if this is due to suboptimal designs of the prompt.

We consider there is no easy way to improve the prompts since strong LLMs consistently show low recall on multiple prompt designs (Sections 4.3, 4.4). Here are possible explanations for the negative correlation.

• Strong LLMs can correctly classify responses without mistakes as "no error", resulting in high precision. However, they are conservative about detecting mistakes, resulting in low recall.
• Weak LLMs do not have strong error detection capabilities and may return almost random predictions for difficult cases. As a result, recall can become higher than the conservative strong LLMs, although it results in low precision.

We will improve the explanation in future versions of our paper.

Would it make the model perform better if they are prompted with the error type definitions and are prompted to provide the error type as well?

ReaLMistake is designed to evaluate general-purpose error detectors, which do not assume any specific target task domains or error types. This setting is motivated by a real-world situation in which providers of LLM-based services try to detect any type of mistake in LLM responses automatically.

In the situation of targeting specific tasks, we agree that error detectors would be improved when designed for specific error types (e.g., prompted with the error type definitions, as you suggested).

The experiments would be more insightful as well if a per-type analysis is provided.

Thank you for your suggestion. We will provide a more detailed per-type analysis in future versions.

We appreciate your detailed comments and valuable suggestions!

My main reservation about the paper, which could be corrected if claims were slightly toned down [...]

Thank you for your suggestion. We agree that it is technically possible to study ambiguous errors by annotating each instance with multiple annotators, which can be a future work. We will discuss this point and tone down our claim.

In this work, we choose to use unambiguous errors because of the following advantages:

(1) Not affected by annotators' subjectivity.

(2) Reliable evaluation: Unambiguous errors can reliably evaluate error detectors because of the small noise.

(3) Binary labels: We can collect binary labels, making the evaluation of binary predictions easy.

(4) Annotation cost: We do not need to collect many annotations for each instance.

It is not clear whether the explanations were agreed upon between the annotators.

Thank you for your suggestion. We agree that this is a promising direction for further verifying that the errors in ReaLMistake are unambiguous.

[...] Table 1 suggests ‘MT-Bench, PandaLM, LLMEvalˆ2 Multiple NLP Tasks’ has more criteria and in multiple domains

As per the above, it is not clear how novel the approach is [...]

We would like to clarify that those benchmarks are for evaluating pairwise (ranking) evaluators, which compare multiple responses, while our benchmark is for evaluating pointwise evaluators. Pairwise benchmarks cannot be directly used to evaluate error detectors because they include relative evaluation between multiple responses.

This table intends to show that benchmarks for pointwise evaluation are lacking, although there are diverse pairwise benchmarks. We will improve it in future versions.

I would reduce the strength of the claims around ambiguity [...]

Thank you for these suggestions. We will definitely improve the use of the terms to describe the properties of our benchmark more precisely.

Please calculate and report Cohen’s kappa [...]

We report F1 because we evaluate human performance, not inter-annotator agreement. Each instance in ReaLMistake is annotated by one annotator and verified by another, and after the benchmark is finalized, human performance is evaluated using error classifications by other annotators.

For reference, when we consider the labels in ReaLMistake and human classification (for human performance) as two annotators, Cohen's kappa is 0.80 for GPT-4 and 0.96 for Llama 2 70B.