Ask Again, Then Fail: Large Language Models’ Vacillations in Judgement

Thank you so much for your kind words! Your appreciation means a great deal to us. We will provide you with a detailed explanation of your concerns.

Q1: I didn't understand the second sentence in footnote 1.

A1: We are sorry for the ambiguity. When generating a response, the model usually provides a lot of thought process, ending with an answer. However, there is currently no particularly effective way to automatically evaluate these intermediate thought processes, so we can only assess the model based on the final answer it provides. To enable automated evaluation, we instruct the model to output the final result in a specified format (i.e., Answer:). We hope our response is helpful to you.

Q2: Modification Rate (M. Rate) was not clear to me.

A2: Sorry for the confusion. We hope to explain the concept of Modification Rate (M. Rate) through an example. Suppose there is an evaluation test set with 1000 samples, and the model answered 10 correctly in the initial question and answer. We continue to ask follow-up questions for these 10 samples, and after the follow-up question, the model only answered 5 samples correctly. So M. = 10/1000 - 5/1000 = 5%, and M. Rate = (10 - 5) / 10 = 50%.

The rationale for employing both M. and M. Rate to assess the judgement consistency of LLMs primarily stems from the fact that in scenarios where initial performance is poor, the potential for further decrease in model performance is constrained. Consequently, relying solely on M. might not provide an accurate reflection of the model's judgement consistency. For example, in the above example, although the model's performance only decreased by 5% after follow-up question, 50% of the samples answered correctly in the first round were answered incorrectly in the second round, indicating that the model's judgement consistency is low. Therefore, considering these two indicators together can provide a more accurate and comprehensive reflection of the model's judgement consistency.

Thank you for your valuable feedback. We hope our response has resolved your confusion.

The results of ChatGPT, PaLM2-Bison, and Vicuna-13B under emotional disturbance.

The results of GPT-4-1106-preview, UltraLM-13B-v2.0, XwinLM-13B-v0.2, and Zephyr-7B-Beta under emotional disturbance.

The results of ChatGPT, PaLM2-Bison, and Vicuna-13B under irrelevant information disturbance.

The results of GPT-4-1106-preview, UltraLM-13B-v2.0, XwinLM-13B-v0.2, and Zephyr-7B-Beta under irrelevant information disturbance.

Q3 (from weakness 2): While disturbances like questioning, negation, and misleading are considered, the study might benefit from exploring a wider range of disturbances to provide a more comprehensive understanding of LLMs' judgement consistency challenges.

A3: Thank you for your comprehensive and careful consideration. Exploring more different types of disturbances indeed helps us more thoroughly evaluate and understand the judgement consistency of LLMs in various scenarios. In addition to the three types of disturbance mentioned in the paper, we have designed two types of disturbance based on scenarios that may occur in actual teaching[5][6]: emotional disturbance and irrelevant information disturbance.

• Emotional disturbance is expressing dissatisfaction with the model's response using sentences containing strong emotional implications. Specifically, we use “I am very disappointed with your answer, please respond again.” as a follow-up question.

• Irrelevant information disturbance refers to confusing the model's judgement by adding some irrelevant information to the original question. We sample 500 samples from GSM-IC-2step[7] and GSM-IC-mstep[7] as the experimental dataset, and concatenate "I have added some information to the question, please answer it again." with the new samples that have added irrelevant questions as follow-up questions.

Following the setup in the paper, we evaluated the judgement consistency of ChatGPT, PaLM2-Bison, Vicuna-13B, GPT-4-1106-preview, UltraLM-13B-v2.0, XwinLM-13B-v0.2, and Zephyr-7B-Beta in these two new disturbance scenarios, and the experimental results are shown below.

From the experimental results, it can be seen that whether it is the three types of follow-up questions proposed in the paper or the two new types of disturbance proposed, the model's judgement consistency is generally low when facing these disturbances. Adding new disturbance further verifies the universality of this issue.

Note 1: GSM-IC[7] is constructed based on the validation set of GSM8K by adding an irrelevant sentence to each sample, and is divided into two datasets, GSM-IC-2step and GSM-IC-mstep, according to whether the intermediate steps are more than 2 steps.

[5] Humphries S. Please teach me how to teach: The emotional impact of educational change. The emotional rollercoaster of language teaching, 2020.

[6] Tofade et al. Best practice strategies for effective use of questions as a teaching tool. American journal of pharmaceutical education, 2013.

[7] Shi et al. Large language models can be easily distracted by irrelevant context. International Conference on Machine Learning. PMLR, 2023.

Q2: Can you provide insights into how the proposed mechanisms and solutions could be practically applied in real-world scenarios, especially in fields where LLMs are extensively used, such as customer support or healthcare?

A2 (also response to weakness 3): Thank you for your constructive feedback. We agree that discussing how this mechanism can be integrated with practical applications can indeed help strengthen the impact of our research.

Currently, LLMs mainly appear as virtual assistants in real life. Considering that they may be questioned by users or have disagreements with users during the interaction process, we believe it is necessary to use the mechanism we proposed to evaluate the model's judgement consistency in the face of interference before they are officially put into use. If the judgement consistency is low, the mitigation methods in the paper can be considered to improve their judgement consistency in the face of interference to some extent. This can not only enhance the user experience and satisfaction but also improve the reliability of the model-generated content in some fields where virtual assistants participate in actual decision-making.

Here are some potential impacts and applications of our proposed mechanism and mitigation methods in real-life scenarios:

• Customer Support: LLMs are widely used as virtual bots in the customer support field, primarily for answering user questions, solving problems, and providing advice. In this process, users may question the bot's responses or disagree with the bot-generated answers. For this application scenario, the quality assurance and monitoring team of virtual bots can use our proposed mechanism to evaluate the judgement consistency of customer support virtual bots when facing user interference. After comprehensive and reliable analysis of the results, the development team can implicitly concatenate the mitigation methods from the paper as model input after the user's question to improve the judgement consistency of virtual bots when facing interference, thereby enhancing the quality and reliability of customer support services and increasing user satisfaction and trust.

• Healthcare: LLMs can serve as virtual assistants in healthcare, assisting in areas such as diagnosis, medical image review, and drug development. For example, when an LLM serves as a virtual medical assistant in reviewing medical images and submitting the results to doctors for diagnosis, our proposed mechanism can be used to repeatedly evaluate the consistency of its judgements with different interference questions. If the consistency reaches a preset threshold, the judgement can be submitted as auxiliary material to the doctor; otherwise, we may reasonably suspect that the judgement's reliability is low, and the mitigation methods from the paper can be used to improve judgement consistency. If the consistency still fails to meet the preset threshold after applying the mitigation methods, the patient case can be marked to remind the doctor to exercise caution. It is important to note that the final judgement should be made by the doctor, and the judgements and recommendations provided by the virtual medical assistant serve as reference information to support the doctor.

It is important to note that although our proposed mechanism and mitigation methods can assess and improve the model's judgement consistency to some extent, considering the complexity of real-world situations and the high requirements for consistency in some application scenarios, more efforts are needed in model training strategies and more comprehensive evaluations in the future to completely solve this problem.

Thank you for your insightful suggestions, and we hope our response has been helpful to you.

Q1: Is there room for future research to address this limitation?

Based on the latest evaluation results we have added above, it can be observed that the issue of fluctuating judgement consistency in models when subjected to user interference is still very significant, thus there is ample room for further research. We believe that some preliminary research directions for the future include:

• From evaluation perspective, exploring more evaluation methods and metrics, such as designing prompts with other types of interference, can more comprehensively assess the judgement consistency of LLMs in various scenarios. In addition, the impact of different base models, training strategies, and optimization algorithms on the model's judgement consistency issue can be evaluated and compared.

• From training or fine-tuning perspective, on one hand, explore other training or fine-tuning strategies, such as adversarial training, reinforcement learning, etc., to improve the robustness of LLMs when facing interference; on the other hand, research how to combine our evaluation methods with existing model training and optimization techniques to enhance the judgement consistency of LLMs. Our work aims to draw attention to this issue through systematic and comprehensive evaluation, providing inspiration and assistance for future efforts in addressing this issue through model training or fine-tuning in the future.

• From alignment perspective, explore how to alleviate the issue of LLMs tending to please and flatter users, thus improving judgement consistency, when facing questioning, negation, or disagreement from users, by alignment, such as aligning the model's thinking process after being disturbed with the human's thinking process after being disturbed.

The results of GPT-4-1106-preview.

The results of UltraLM-13B-v2.0.

The results of XwinLM-13B-v0.2.

The results of Zephyr-7B-Beta.

Thank you for the insightful comment! We will address your concerns as follows:

Q1: Considering the rapid advancements in AI technologies, how might the results differ when applied to newer or upcoming LLMs? Is there room for future research to address this limitation?

A1 (also response to weakness 1): Your concern is indeed very necessary. Considering the rapid development of large language models, the latest LLMs may have improvements in various aspects, and we also believe it is necessary to explore whether this issue remains universal on the latest LLMs. With limited computing resources, we have evaluated the judgement consistency of several of the latest and most capable closed-source and open-source models, such as GPT-4-1106-preview[1], UltraLM-13B-v2.0[2], XwinLM-13B-v0.2[3], and Zephyr-7B-Beta[4], on the benchmarks MultiArith, StrategyQA, and CoinFlip, as per the experimental setup in the paper. We report the experimental results below.

The experimental results show that even the most advanced LLMs generally exhibit noticeable fluctuations in judgement consistency when faced with user questioning, negation, or misleading inputs. Consequently, we posit that this challenge will persist in the realm of LLMs, even with the advent of newer, more advanced models in the future. This issue is universal across all LLMs and is currently underemphasized, which underscores the importance of our research. Given this context, it is unlikely that newly developed models will be able to fully address these challenges in the near term.

Note 1: We chose models based on AplacaEval Leaderboard rankings and our computational resources we could afford.

Note 2: Due to the costs associated with calling the GPT-4 API, we only sampled 100 samples from the test sets of each of the three datasets for evaluating the judgement consistency of GPT-4. For all other models, the number of samples used for evaluation strictly adhered to the evaluation settings outlined in our paper.

[1] https://openai.com/blog/new-models-and-developer-products-announced-at-devday

[2] https://huggingface.co/openbmb/UltraLM-13b-v2.0

[3] https://huggingface.co/Xwin-LM/Xwin-LM-13B-V0.2

[4] https://huggingface.co/HuggingFaceH4/zephyr-7b-beta

Dear Reviewer,

I hope you're doing well. The discussion period is soon coming to an end. Thank you very much for your suggestions. We hope that we have addressed your concerns through the additional experimental results provided.

If you still have any further reservations or suggestions, please don't hesitate to share them. Your insights are invaluable to us, and we're keen to address any remaining issues.

Best regards!

Authors

Thank you for the valuable feedback!

Q1 (from weakness 1): The overall novelty of this work is a bit limited given that prior work (many of which are also cited by the authors) has investigated the "sycophantic" behavior of LLMs, and the proposed methods in the paper are quite similar to the ones in prior work. For example, the paper by [Turpin et al.] which the authors seem to miss studies LLM's behavior when there exists bias in the context, where one of the settings is exactly about putting human user's belief (in a wrong answer) in the context, which is close to the type L (leading questions) prompt explored in this paper. Similar findings are also present in [Perez et al., 2022] as cited. [Wang et al., 2023a] as cited explores using another conversational LLM conditioned on a wrong solution to engage in a debate with the original LLM; the "follow-up" responses by the simulated user there also share many similarities with the ones proposed (expressing disagreement, doubt, different opinions, etc.).

A1: Sorry for the oversight. We will add [Turpin et al.] to the related work, and we appreciate your detailed and friendly reminder.

Thank you for your insightful comments. Although our work indeed intersects with several studies you mentioned regarding the reliability of large language models, it distinguishes itself in several key aspects:

• Novel Research Perspective. Unlike [Wang et al., 2023a], which designs debate-like dialogues with invalid solutions for each sample, and [Turpin et al., 2023], which introduces bias features into model inputs for multiple-choice questions (like modifying the order of options), our Follow-up Questioning Mechanism is closer to scenarios that ordinary users might encounter in real-life use of LLMs. Furthermore, the simpler and more conversational follow-up questions in our mechanism are more general and in line with the habits of everyday users than the templates or methods designed for each sample in other approaches.

• Comprehensive Scenario Design. As you mentioned, the research methods of the related work [Wang et al., 2023a][Turpin et al., 2023] only resemble one type of question in our proposed Follow-up Questioning Mechanism (leading questions), neglecting questioning and negation, common dialogue scenes in interactions between users and LLMs. Moreover, our experimental results show that questioning and negation also cause significant fluctuations in the judgement consistency of LLMs when facing interference.

• Beyond Sycophancy, Further Discoveries. Our study not only corroborates the sycophantic behavior mentioned by [Perez et al., 2022] but also reveals a new finding: the model may become cautious and neutral in the face of interference, a behavior not extensively covered in previous studies. As analyzed in the Error Analysis (refer to pages 6 to 7), we categorized errors into four types through human observation, where error categories 2, 3, and 4 can be attributed to sycophancy. However, it's worth noting the existence of category 1 (Unable to answer), refer to Figure 5 in the paper. In cases of error category 1, the model opts for a cautious and neutral stance, avoiding direct answers. This behavior is crucial for understanding the practical usability of LLMs, as it reflects an attitude distinctly different from sycophancy when faced with challenges, negations, or misleading information, rather than just flattery.

Our research aims to demonstrate through comprehensive analysis that conversational large language models show unreliable judgement when faced with disturbances like questioning, negation, and misleading. Although our study shares some thematic overlaps with the cited works, it contributes new perspectives and insights into the reliability and practical application of LLMs. Together with these insightful related studies, our work is vital for the future development and real-world deployment of LLMs.

We hope our clarification can address your concerns.

Q2 (from weakness 2): While there are some discussions/insights about the issue in the paper, overall, as an analysis/evaluation type work, I feel the contribution could be strengthened if more fruitful thoughts/speculations about the underlying cause of the observed issues (and potential ways of mitigating them) are included.

Thank you for your constructive suggestions! Here are our responses:

We believe that the potential reasons for the occurrence of this issue may primarily include the following:

• Misalignment of thought processes (as mentioned in the first sentence of section 4 of our paper). When humans encounter questioning, negation, or disagreement, they typically rely on their own experiences and knowledge to reevaluate their perspectives, engaging in deeper contemplation of the issues at hand. In contrast, the model's response is solely based on the information it has seen in the training data, lacking genuine thought processes and only attempting to generate the most probable response for the given input.

• Limitations of training data and training process. Large language models are typically trained on vast amounts of data, which may contain errors, biases, or incomplete information. This can lead to challenges when these models encounter real-world scenarios that differ from their training data. Specifically, if LLMs don't effectively learn to handle skepticism or disagreement during training (e.g., SFT or RLHF), they may struggle in similar real-life interactions. Additionally, the lack of exposure to dynamic, real conversational interactions during training could hinder their ability to navigate complex dialogue situations, such as those involving in-depth questioning or deep thought.

• Sycophancy and user-centric influence. Through error analysis, we have found that sycophancy behavior is the primary cause of decreased judgement consistency in the model. This behavior is closely related to the model's preference learning during the training process, as larger models tend to generate answers that users want to hear. Furthermore, models designed for user interactions usually need to focus on user experience. Therefore, when confronted with skepticism or disagreement, the model often starts by expressing apologies and may even seek compromise to avoid potential conflicts.

• Limitations of the autoregressive model structure. The model is likely to generate apologies or admit mistakes first due to sycophancy. Since the model relies on autoregressive methods when generating responses, it may make incorrect judgements in subsequent responses in order to maintain semantic consistency with the earlier apology, and it may even modify the original question to make responses sound plausible (refer to Error #2 in the error analysis).

Regarding potential mitigation methods for this issue, we believe they include but are not limited to the following (from low to high cost):

• Alignment of thought processes. We can design prompts to simulate the human thought process when facing interference, thus enhancing the model's judgement consistency. For example, as proposed in the paper, few-shot prompting mitigation method can align the model's "thought process" when dealing with interference with that of humans facing similar interference by designing demonstration examples.

• Trade-offs between stubbornness and sycophancy. We can stimulate the model to simulate the emotional responses that a person with a specific character might have by designing the model with a certain personality. For instance, setting the system prompt as "You are a highly confident, self-assured, and opinionated intelligent assistant." can enable the model to maintain its judgement when confronted with skepticism or disagreement, mitigating issues of poor judgement consistency.

• Emphasis on data quality and realistic interaction training. We can rigorously purify our pre-training and supervised fine-tuning datasets, eliminating any incomplete, biased, or incorrect contents (despite the potentially higher costs). Additionally, we can collect dialogue data under scenarios of skepticism, negation, and misleading contexts. The collection methods can include manual annotation, distillation from more powerful models, or context distillatio using the model itself[1]. Furthermore, we can collect preference data by gathering multiple responses in the face of distractions and then ranking them. This collected dialogue or preference data will be integrated with existing dialogue (or preference) datasets for training, strategically enhancing the model's resilience and effectiveness in responding to distractions such as questioning, negation, and misinformation.

Thank you for your insightful comments. We hope our response can address your concerns.

​[1] Bai et al., Constitutional ai: Harmlessness from ai feedback.

Q2 (from weakness 2): The qualitative analysis misses some rather important details such as the proportion of each error category. While there are some discussions/insights about the issue in the paper, overall, as an analysis/evaluation type work, I feel the contribution could be strengthened if more fruitful thoughts/speculations about the underlying cause of the observed issues (and potential ways of mitigating them) are included.

A2: Thank you for your valuable suggestions. In the qualitative analysis, we have presented the proportions of each error type in the form of bar charts (refer to Figure 5). To provide a more intuitive representation, we now present the proportions of each error type in tabular form.

Based on the results of error analysis, we can categorize the model's behavior into two categories: sycophancy and caution. Error#2, Error#3, and Error#4 can be attributed to sycophancy behavior, while Error#1 represents the model's cautious and neutral stance, which is in stark contrast to sycophancy. By examining the proportions of different error types, we can observe that sycophancy behavior is the primary reason for the model's poor judgement consistency when facing skepticism, denial, or misleading input. However, caution and neutrality also contribute to fluctuations in the model's judgement consistency when dealing with interference to some extent.

The proportion of four types of errors on StrategyQA.

The proportion of four types of errors on CoinFlip.

The proportion of four types of errors on MultiArith.

Dear Reviewer,

I hope you're doing well. The discussion period is soon coming to an end. Thank you very much for your valuable feedback. We hope that we have addressed your concerns through careful comparison with other relevant works, along with additional analysis and discussion supplements.

If you still have any further reservations or suggestions, please don't hesitate to share them. Your insights are invaluable to us, and we're keen to address any remaining issues.

Best regards!

Authors