See What LLMs Cannot Answer: A Self-Challenge Framework for Uncovering LLM Weaknesses

Thank you for your valuable review.

Seed Instance Collection

The full list can be found at https://anonymous.4open.science/r/Self_chal_anonymous-9205/seed_instance.json. In this research, we are motivated by the failure of GPT4, and we collect those failure cases from:

(1) 4 papers that study the failure of ChatGPT(4) [1,2,3,4] (67.7%), where we focus on QA data.

1. Why Does ChatGPT Fall Short in Providing Truthful Answers?
2. A Categorical Archive of ChatGPT Failures
3. Is ChatGPT a General-Purpose Natural Language Processing Task Solver?
4. Towards Benchmarking and Improving the Temporal Reasoning Capability of LLMs

For example:

The school in which the Wilmslow Show is held is designated as what?

Who did Jessica Valenti work for in 2005?

The original sources of those papers include HotPotQA, TSQA, NQ, BigBenchHard, BoolQA.

(2) failure cases from online (twitter) and our usage (32.3%). For example:

If knaves always lie and knights always tell the truth, and you ask a person if they are a knight and they say no, what are they?

For cases presented via screenshot, we manually convert them into text. All data are then tested using our GPT-4 API.

We will further clarify those sources and provide full seed instances in our revision.

Evaluation in Limited Tasks

First, we acknowledge that we did not involve traditional NLG tasks, e.g. MT, due to the fact as you mention. We will discuss it as a limitation.

Second, 8 tasks (patterns) are summarized from 189 data. With more data collected, we believe there would be more interesting tasks (patterns). In fact, the Self-Chal benchmark involves arithmetic reasoning tasks such as Patterns C and D. For example:

... If a student starts with a paper containing 5 grid squares and each subsequent paper's number of squares is doubled, what will be the total number of characters written on the 4th paper?

Using More LLMs for Experiments

We use Llama3-70B-instruct, which summarizes 21 patterns. However, we find Llama3 is less effective compared with GPT4: (1) one G-4 pattern can cover multiple L-3 pattern; (2) L-3 patterns cannot be used to generate challenging queries; (3) multiple G-4 patterns are not shown in L-3 patterns.

We benchmark more LLMs on Self-Chal. Please also see Response to xmpu.

Suggestion on clarification

We will follow your suggestion to revise our paper. We use gpt-4-32k-0613.

We would really appreciate it if you could reconsider our paper.

Due to the limited space (2500 characters) for rebuttal, we respond to the main questions. Here, we take the opportunity to engage further discussion.

Figure 1 clarification

We have modified the Figure 1: https://anonymous.4open.science/r/Self_chal_anonymous-9205/Figure_1.pdf (please click View raw to see) and will add a clarification on the caption as below.

Figure 1:  The overall Self-Challenge framework. We first summarize initial error patterns from seed failure instances (Step 1). Then, we perform pattern evaluation (Step 2) to evaluate the quality of summarized patterns, and obtain corresponding human feedback; pattern optimization (Step 3) to modify the original pattern based on human feedback. We frame Step 2 and Step 3 iteratively. We present the difference between the Initial Pattern and the Optimized Pattern in underlined text.

Word sorting

Thank you for your suggestion. We will clarify it in our revision.

We hope that our answers have been satisfactory and that we've addressed your concerns effectively. Please let us know if you have any further questions or require clarification on any points. We are happy to engage in further discussion.

Thank you for your valuable review.

Self-Challenging of Llama3-70B-instruct

The full patterns summarized by Llama3 can be found at https://anonymous.4open.science/r/Self_chal_anonymous-5FE3/README.md

Generally, we find (1) Llama3 summarized patterns are mostly covered by GPT4. For example, P-B in our paper covers P-8 and 10, P-E covers P-15, 17 and 19. (2) some GPT4 patterns are not found by Llama3. For example, P-C (counting), P-D (logical), P-H (text manipulation).

We further optimize P-5 and find Llama3 cannot generate queries that induce Llama3 to hallucinate.

We will include full results in our revision.

Benchmarking other LLMs

Generally, all LLMs show poor results. Similar trends can be consistent with previous research [1, 2] that LLMs can be homogeneous as they are trained with similar techs (e.g., transformers) on similar data (e.g., Wikipedia).

1. Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?
2. The Platonic Representation Hypothesis

Human Performance

Due to space limit, please see response to nH6u, where we provide estimated human performance.

Evaluation Protocol

We acknowledge that the validation on 115 data is small. However, this is a supplement to our validation on full 1835 data.

Diversity

Following your suggestion, we evaluate the diversity of our data using Distinct score. We report dist-3 and dist-4 here. The higher, the more diverse.

We compare BigBench word-sorting task, which is similar to pattern H.

It shows our data are diverse and are of slightly higher diversity compared with similar task.

We would really appreciate it if you could reconsider our paper.

We acknowledge that relying on GPT-4 to generate instances for evaluation can lead to potential biases, in particular data homogeneity [1, 2]. Although we enrich the input pattern with additional domain information and evaluate their level of diversity by calculating the Distinct score [3], there still remains a potential problem that queries under the same pattern and domain can be similar to each other. This problem can be solved by decomposing the current patterns into more fine-grained and detailed sub-patterns, or providing larger domain text as a source for query generation, which we leave for future work.

Recently, researchers have explored using LLMs as annotators for creating and augmenting data of their interest, using both zero-shot and few-shot learning[1, 2]. For example, [3], [5], [9],[11], and [12] use LLMs such as ChatGPT and GPT-4 to label raw text for text classification tasks, and [6] and [13] use LLMs to generate output texts for text generation tasks, such as machine translation and text summarisation.  Additionally,  [4] finetunes GPT-2 with small data to annotate new training data for text classification tasks. [7], [8] and [10] use LLMs to augment dialogues with few-shot learning. Our work falls in the above studies that use LLMs for data augmentation. However, different from those that mostly use LLMs to augment analogous data or label output text, we use LLMs to obtain diverse queries for a specific task under the guidance of patterns (instruction) generated by LLMs themselves.

As merely relying on GPT-4 to evaluate model response can be less effective [14, 15], we evaluate model performance by comparing their responses and gold responses.

[14, 15] show that although LLMs can show superior performance compared with humans, they can have limitations in basic tasks.  

[15] use LLMs to generate complex and multi-constrained instances, by drawing on common-sense reasoning. Compared with them, we focus on more general QA-format tasks. 

Thank you for your valuable review.

Human Performance

We provide 3 estimated human performance (binary-label) on Self-Chal benchmark. (1) We measure untrained annotators’ performance by calculating their first trial annotation on 20 instances (before training). 5 annotators were involved (1 did not pass our follow-up test annotation). Their Fleiss' Kappa score is 0.56 (moderate agreement). The average acc by majority vote is 81.00%. (2) We measure their trained performance by evaluating their test annotation on 100 instances, where we provide gold-labels. Their averaged acc are 90.20% (w 1 who did not pass) and 92.50% (wo). (3) We measure 4 annotators' formal annotation performance by comparing their first round of formal annotation, and the final data (gold) after checking and revision. The acc is 96.73%.

Unanswerable Questions

Yes, there are questions that cannot be answered by one gold response. For example: Please combine two names of the centers and institutes at Stanford University in an alphabet order.

Such questions are filtered out of our benchmark, and are not used for evaluation.

Tokenization Issues

Thank you for acknowledging the tokenization tasks are interesting. From a practical view, such tasks can be closely related to applications such as poetry generation, spelling checking, and Teaching Aide in the area of NLP4Education. We also emphasize that such issues reveal the problems of underlying tokenization mechanisms, which can further influence the performance of downstream tasks. Also, it reveals that the current tokenization design in LLMs is different from humans.

Lastly, we agree that our methodology mainly relies on prompting LLMs, which has been adapted by previous study [1, 2, 3, 4]. We also highlight that this framework is novel in that we propose a new semi-automatic evaluation protocol that can help humans better mine and understand the error patterns in LLMs.

1. NLG Evaluation using GPT-4 with Better Human Alignment
2. Better patching using LLM prompting, via Self-Consistency
3. Tree of thoughts: Deliberate problem solving with large language models
4. Chain-of-thought prompting elicits reasoning in large language models

We thank you for your appreciation of our work.

Thanks for your valuable review.

‘Self-Challenge’

We will revise it by adding Human-in-the-loop, referring to the process of LLM finding questions that can challenge themselves with the guidance of humans.

Annotation

For the unsatisfactory for each data, we check both label and response:

1. Incorrect: If any error (in particular factual) is found in GPT4 response, annotators label it as correct (or vice versa); annotated responses contain errors, in particular annotators use incorrect GPT4 responses as gold responses without modification
2. Unreadable: annotated response is oversimplified particularly if annotators modify GPT4 responses, but annotated responses need GPT4 response as context to understand
3. Unnatural: annotators use GPT4 response, but it contains words such as 'As an AI…'

The binary-label annotation is in Appen C (first para).

Why GPT4

We used GPT4 as it is regarded as the most powerful LLM on textual tasks.

Please also see Response to xmpu. We conduct experiments using Llama-3-70B-instruct. But it is less satisfying compared with GPT4.

Human involvement

Yes, our work involves the process of human-in-the-loop. In particular, after generating/optimizing error patterns, we evaluate the pattern and obtain feedback from human evaluation (as mentioned in Sec2.2 and 3.2). This process cannot be automated as LLMs do not know what is correct and is incorrect.

4 domain words

Those 8 patterns are not summarized from 4 domains, but from 189 seed data, and are then optimized under 4 domain words with human feedback. Here the smooth of 4 domains is compared with 1.

We will clarify it as a limitation.

Pattern selection

We select 8 qualified patterns from the initial 14 ones. As in Sec 2.2, we evaluate the pattern by evaluating the new queries generated by them. The new queries should meet 2 criteria. 1. they should contain important features of the pattern, e.g. an Assumption of Existence (AOE) query should contain non-existent content. 2. the new queries should be challenging enough, e.g. an AOE query should induce LLM to hallucinate.

Pattern requirement

The requirement of identify specific ... refers to the features in pattern (Fig5, Appen E), which is completely generated by the LLM.

More LLMs

Please see Response to xmpu, we benchmark more LLMs.

For all above issues, we will clarify & highlight them in our revision. We would really appreciate it if you could reconsider our paper.

Due to the limited space (2500 characters) for rebuttal, we respond to the main questions. Here, we take the opportunity to engage further discussion.

Evaluation metrics

Following [1], we modify our evaluation prompt, asking LLMs to explain why the model-generated response is correct or not [1], and then give a label for the response. In particular, the evaluation prompt is:

Please first read the below question, its correct answer and a model-generated answer:

Question: {question}

Correct Answer: {answer}

Model-generated Answer: {model_response}

Please explain whether and why the model-generated answer is correct or incorrect and add a tag \"correct\" or \"incorrect\" at the end of your response."

Similarily, we show the precision, recall, and f1 scores of this metric by comparing its outputs with the human-annotated labels.

On full 1835 instances. Precision 94.57, Recall 78.06, F-1 85.52 On 115 instances. Precision 90.57, Recall 73.85, F-1 81.36

We see that compared with direct prompting, asking LLMs to first generate an explanation and then a label gives higher precision and lower recall and F-1 scores. We assume that this can be because the LLMs can generate inappropriate explanations for those questions (in particular those questions are challenging to LLMs), and subsequently give a wrong label.

For example, given the question:

Suppose there is a rule that patients with Long QT syndrome can only participate in sports with a prime-numbered amount of players on the field. If a patient wants to join a sports team where the number of players is doubled each season, which sport should they choose to maximize their chances of participation in the first three seasons?

And the GPT-4 answer is:

To maximize the chances of participation in the first three seasons, we need to find a sport where the number of players on the field is a prime number in each of the first three seasons. Let's denote the initial number of players as x. Then, the number of players in the first three seasons will be x, 2x, and 4x.

We need to find a value of x such that x, 2x, and 4x are all prime numbers. Let's analyze the possibilities:

1. If x is an even number, then 2x and 4x will also be even numbers. The only even prime number is 2, but if x = 2, then 2x = 4, which is not a prime number. So, x cannot be an even number.

2. If x is an odd number, then 2x will be an even number. As mentioned earlier, the only even prime number is 2. However, if 2x = 2, then x = 1, which is not a prime number.

Based on this analysis, there is no sport that meets the criteria of having a prime-numbered amount of players on the field for the first three seasons when the number of players is doubled each season.

which is correct as labeled by both human and our evaluation metric.

However, the above prompting evaluation gives an explanation:

​​The model-generated answer is incorrect. The explanation provided by the model is accurate in stating that if x is an even number, then 2x and 4x will also be even and therefore cannot be prime (except for the number 2). However, the model then incorrectly states that if x is an odd number, then 2x will be an even number and cannot be prime, which is true, but it incorrectly concludes that there is no sport that meets the criteria.

The fundamental error in the model-generated answer is the assumption that there could be a prime number x that, when doubled or quadrupled, results in other prime numbers. This is impossible because, except for the number 2, all prime numbers are odd, and multiplying an odd prime number by 2 or 4 will always result in an even number greater than 2, which cannot be prime.

The correct answer clearly states that it is impossible for x, 2x, and 4x to all be prime numbers because 2x and 4x will always be composite if x is greater than 2. Therefore, there is no sport that meets the criteria of having a prime-numbered amount of players on the field for the first three seasons when the number of players is doubled each season.

Incorrect

where the explanation incorrectly reasons the model then incorrectly states that if x is an odd number ... will always result in an even number greater than 2, which cannot be prime.

We will include the above case and result as a supplement material in our revision.

Wang, Zezhong, et al. "Self-guard: Empower the llm to safeguard itself." arXiv preprint arXiv:2310.15851 (2023).