Large Language Models Often Say One Thing and Do Another

C5: Furthermore, it is difficult to read the comments of the experiments in the findings. For example, see ‘Exp. 1’ on line 326. I understand that the authors wanted to make the work more discursive, but they should pay more attention to style. An example could be constructing subsections by structuring the discourse by placing the RQ next to the answer. The title of the section should describe the analysis (e.g. line 334 should be a subsection called ‘inconsistency’ or something similar).

Thanks for your constructive suggestion. Based on your suggestion, we have undertaken a logical restructuring of Section 4. This section is organized into three subsections, each corresponding to one of the research questions (section 4.1, 4.2, 4.3 in new PDF). Each subsection now contains multiple sub-conclusions and one overall conclusion. Each sub-conclusion is presented as a paragraph, with the title of the paragraph describing the analysis (for example, "Inconsistency between LLMs' words and deeds"). The first sentence of each paragraph states sub-conclusion (for example, “Inconsistency between words and deeds is a common phenomenon across LLMs and domains”), followed by a detailed description and analysis of the experimental results. The overall conclusion (at the end of section 4.1, 4.2, 4.3 in new PDF) summarises the aforementioned sub-conclusions and offers answer to the corresponding research question. We believe this new structure will facilitate better reading and comprehension of our work.

C6: The evaluation is made on a limited dataset (1325 examples). This is not a major problem, but it does undermine the scientific soundness of the paper. For example, look at the well-known QA tasks (SIQA, PIQA) or the mathematicians (GSM8K). all of these have significantly more instances than 1k. It would be interesting to observe these phenomena on a larger scale (e.g. 4k).

Thanks for the suggestion.

However, the construction of the dataset is quite labor-intensive. Each test case pair is created based on different beliefs which require manual selection from various datasets (as shown in section 2.3.1). Additionally, a certain proportion of these crafted test cases require rewriting and all test cases need to be double-checked (as shown in section 2.3.4). This makes it challenging to rapidly expand the size of our dataset in a short period of time.

Despite this, we provided an analysis of the correlation between the data size and the evaluation results in the last discussion of section 5. Specifically, we randomly sampled the dataset five times at various sample ratios and compared the results on the subsets with the results on the complete test set. The consistency scores on subsets of the test set at different sample ratios are listed in Appendix B.1 in the new PDF. We can observe that: 1) models’ consistency scores largely stabilize when the data size exceeds 50% ( as shown in Figure 10). 2) there are no statistically significant differences ($p > 0.05$) between the evaluations performed on the subsets and the entire dataset (as shown in Table 8).

Therefore, we reasonably infer that evaluations based on 1,000+ test cases are stable and consistent and the phenomena observed would remain consistent on a larger scale.

C7: The construction of the examples could be biased by the constructor (an LLM constructs, to be precise does augmentation, of the examples on which it is prompted)

We understand your concern about the potential bias from the LLM constructor. However, in our study, we primarily rely on the exceptional language generation abilities of the LLM to transform manually constructed word questions into deed questions. Moreover, the test cases constructed by the LLM are just the initial versions, which are then manually reviewed and rewritten where necessary. This is a common practice in benchmark construction, such as SOTOPIA[1], an evaluation benchmark for social intelligence that prompts GPT-4 to generate characters, relationships, and scenarios in test cases; and Mercury[2], a code efficiency benchmark that constructs test case generators using GPT-4.

To alleviate your concern, we also constructed test cases using qwen2.5-72b-instruct following the same procedure presented in the paper. The results show a persistent and significant inconsistency between words and deeds across LLMs and domains, demonstrating that our findings are not dependent on the constructor. Moreover, the Pearson correlation coefficient between the scores obtained from the original and new test cases is 0.95 > 0.8, further verifying the robustness of our results.

Q1: What is the role of the consistency score? As a non-multiple-choices question answer, your score should be able to capture all possible generations. Did you test and or ablation test for this?

Thanks for your question. We’d like to clarify that all our test cases are multiple-choice questions (as shown in the examples in Table 1). Therefore, we utilize regular matching to extract options from the model's free-text responses. The consistency score refers to the proportion of test cases where LLMs’ choices to the word question and deed question is consistent (as described in section 3.2.2).

Q2: How much impact do systemic demonstrations in SFT have on DPO?

Thank you for your question. I'm a bit confused by "systemic demonstrations in SFT". In terms of the SFT process, we utilize standard system messages and 0-shot questions for training the model. We believe this process is set realistically and should not affect the experimental conclusions regarding DPO.

Q3: What are the limits of the transferability of your study? What if another LLM was used instead of GPT-4 to build the samples?

Thanks your thoughtful question. We believe our work has great transferability, considering the consistency of the results based on multiple LLM constructors as shown in reply to C7, and various robustness analyses in Section 5. If a different LLM, other than GPT-4 was used to build the samples, it's likely that similar results would be obtained.

Reference

[1] SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents. ICLR, 2024.

[2] Mercury: A Code Efficiency Benchmark for Code Large Language Models . NeurIPS, 2024.

Dear Reviewer,

Thanks for your detailed and constructive comments, which have been really valuable in enhancing the quality of the paper. Below you will find our point-by-point response to your comments or suggestions. We hope the revisions and explanations will be satisfactory and address your previous concerns.

C1: RQs are described (this is a good thing). However, these should be dealt with in the course of the contribution and perhaps discursively introduced in the intro. The ‘answers’ delivered by the authors in the intro to lines 49,53,73 should be revised (this is not a note on the content but on the style ‘To answer...’ is not a proper formula to be repeated three times).

Thanks for your valuable suggestion.

In the revised manuscript, we gradually introduce the research questions by discussing the prevailing background and existing research gaps, naturally leading to the issue of LLMs’ consistency between words and deeds (line35-42 in new PDF). This makes the logical progression of the questions much clearer, moving from the current situation (Are LLMs consistent in words and deeds?) to the cause (If not, what role does alignment play in this context?) and lastly to the method (can common knowledge generalization methods facilitate consistency between LLMs' words and deeds?).

For the answers to the research questions, we also abide by the progressive relationship between the research questions, allowing for a more natural transition (line 51-83 in new PDF).

C2: The summary of contributions on line 82 makes sense but why use line 49 to line 77 to answer the RQs and then repeat lines 82-90?

We have reorganized our contributions, focusing on succinctly describing the practical and theoretical significance of the paper, such as proposing a novel word-deed consistency evaluation benchmark, instead of reiterating our answers to the research questions (line 87-97 in new PDF).

C3: The experimental setting is described in a sufficient and comprehensible manner, even if it is sometimes difficult to follow the discourse due to the positioning of the figures.

We apologize for any unpleasant reading experience you may have had.

However, the figures and tables in the experimental setting section all serve to present the details of the experimental setting of the paper, such as the specific prompt, results of grid search of learning rates. Considering that these details are not the core issues of our research and there is a strict space limit for the paper, we had no choice but to put them in the appendix.

In order to offer a clearer reading experience, we have revised the original text, placing them at the end of the paragraph and emphasizing that the figures and tables present experimental details, placed in the appendix (as shown in section 3.3).

C4: The fact that the authors decided to organise the section into Discoveries and Discovery Discussions is not wrong as many papers do this. However, I do not think it is the best way. Indeed, the contribution shows a lot of analysis but it is really limiting to organise the presentation of results in this way.

We respect your opinion, however, we believe the structuring into ‘Discoveries’ and ‘Discovery Discussions’ is justified as the two sections discuss two distinct analyses. Specifically, the ‘Discoveries’ (section 4) is essentially the answers to research questions. We have followed the logical progression of these questions and elucidated the experimental conclusions separately. The ‘Discovery Discussions’ (section 5) is our robustness analysis of the results. Here, we have conducted a series of critical analyses to eliminate the influence of factors unrelated to word and deed differences, primarily to ensure the reliability of our results. Therefore, we strongly felt that it was necessary to discuss these in two separate sections.

Dear Reviewer 8NCb,

With the discussion phase ending in 2 days, we sincerely hope to receive your response. If you have any questions or concerns regarding our explanation, please do not hesitate to contact us. Your response is of great significance to us for improving this work, and we look forward to hearing from you.

Warm Regards, :-)

C5: This work lacks ablation studies. Like studying which components of the models or alignment techniques contribute most to the word-action inconsistency. This would help clarify whether the problem is primarily due to data, model architecture, or training techniques.

Due to the complexity of LLM training and considering the limited reply time, it is challenging for us to thoroughly study the impact of all factors during the training process on LLMs’ consistency of words and deeds.

To simplify, we started from two base models, performed supervised fine-tuning on three datasets, and aligned models by two alignment techniques to investigate the impact of different factors*. The final word-deed consistency scores of the models are shown in the following table.

We can observe that:

• Data, model architecture, and training techniques all influence the consistency between words and deeds. We also noted interactions between these factors.
• The problem of word-deed inconsistency is common across models, data, and alignment methods.

Therefore, we infer that the root cause of word-action inconsistency may lie in the pre-training stage or the basic model architecture, indicating that it is a universal issue.

*In the SFT stage, we adopted a learning rate of 5e-6. In the alignment stage, we tried learning rates of [5e-6, 1e-6, 5e-7] and chosen the best performing ones. For CoH, we include 'p,n,pn,np' four types of feedback and use default other parameters.

Reference:

[1] https://huggingface.co/datasets/stingning/ultrachat

[2] https://huggingface.co/datasets/teknium/OpenHermes-2.5

[3] https://huggingface.co/datasets/HuggingFaceH4/no_robots

[4] Direct preference optimization: Your language model is secretly a reward model. NeurIPS, 2023.

[5] Chain of Hindsight aligns Language Models with Feedback. ICLR, 2024.

Q1: In Table 3, both LLaMA 2 and Mistral base models show relatively low average Consistency Scores (CS), whereas ChatGLM3 base has a noticeably higher CS. Does this suggest some fundamental differences in the architecture, training methodology, or other factors that set ChatGLM3 apart from the other two models?

Thanks for your insightful question.

As the training data and details of models aren't publicly available, we can only speculate based on the technical reports and experimental results of the models.

We hypothesize two possible reasons:

• From the experimental results in Table3, it appears that ChatGLM3 has a significant advantage in value domain. Hence, we infer that this may be due to ChatGLM3 pay more attention on value-related training in the pre-training phase, such as removing texts contrary to mainstream values during data selection. The original text from the technical report states: "cleaned data in the pre-training stage by removing text containing sensitive keywords and web pages from a pre-defined blacklist."[1]
• The training objectives of ChatGLM[2] differ from those of LLaMA2[3] and Mistral[4]. GLM is trained by optimizing an autoregressive blank infilling objective, , while LLaMA and Mistral are trained on the next token prediction task. This differential in model structures could potentially contribute to the noted inconsistency.

Q2: In Figure 5, the consistency rate for the second epoch suddenly drops from over 50% in epoch 1 to 0% in epoch 2. Is there anything unusual about this drastic drop? Could it be related to the specific alignment process, or does this indicate a possible issue in the model's training or evaluation setup?

Thanks for your valuable feedback. In Figure 5, the y-axis represents the final consistency score of questions newly aligned in this round. The sudden drop in the consistency rate for the second epoch signifies that no new questions were aligned after the second round.

Q3: Given that LLaMA 2 is somewhat outdated, would it be possible to try the newer LLaMA 3.1 Instruct model? It would be interesting to see if the consistency issues observed in LLaMA 2 are still present in LLaMA 3.1 instruct.

Thanks for your suggestion. We’ve included the results of LLaMA 3.1 instruct, and you can refer to the results in the reply to C4.

Q4: Have you analyzed how sensitive the consistency scores are to the prompts?

Thanks for your question.

We have analyzed the sensitivity of the LLM's choices to prompts in section 5, finding that LLM's choices are relatively consistent across paraphrased prompts, a conclusion in accordance with previous work. The consistency score analysis naturally tends to be stable if the LLM's choices are stable.

To further solidify our experimental results, we supplemented an analysis of the sensitivity of the consistency scores to the prompts. Specifically, we carried out five paraphrases of the test cases based on GPT-4, and separately evaluated models’ consistency scores. The specific scores and standard deviations are as follows, showing that the consistency scores, in general, remain relatively stable even when the prompts are changed.

Reference:

[1] Chatglm: A family of large language models from glm-130b to glm-4 all tools.

[2] GLM: General Language Model Pretraining with Autoregressive Blank Infilling. ACL, 2022.

[3] Llama 2: Open foundation and fine-tuned chat models.

[4] Mistral 7B.

Dear Reviewer,

Thank you for your insightful and constructive feedback. Below you will find our point-by-point response to your comments or suggestions. We hope our revisions and explanations will fully address your previous concerns and be satisfactory.

C1: The paper looks a little bit rough in places. For example, Table 4 and Figure 8 are poorly formatted, extending beyond the usual layout boundaries, which affects readability and professionalism.

Thanks for your kind reminder. We have thoroughly reviewed the manuscript and made adjustments to the formatting, particularly the placement of figures and tables. The formatting of Table 4 and Figure 8 has been improved.

C2: While the paper effectively identifies the issue of inconsistency between words and deeds in LLMs, it does not provide any in-depth analysis or reasoning behind why this inconsistency occurs. This leaves a gap in understanding the reason cause of the problem.

Thanks for your thoughtful review.

But we’d like to clarify that in our paper, we provided an analysis of the reasons behind the inconsistency. Specifically, in section 4.1, we offer an initial hypothesis for the reason: for base models, it’s mainly due to a lack of strong belief and for aligned models, it may stem from the difficulty of generalizing one-sided alignment to the other side. Furthermore, in section 4.2, we verified the reasons behind the inconsistencies in the aligned model's words and deeds.

We have restructured the organization of our findings in section 4, to present our experimental conclusions more clearly.

C3: Although some alignment techniques are explored, they are not shown to fully resolve the inconsistency, and the paper does not provide clear insights for overcoming this challenge.

Thanks for your insightful comment.

We’d like to acknowledge that the main goal of our study is to establish a benchmark for quantifying LLMs’ inconsistency between words and deeds, and provide some widely-used baselines to identify the universality and challenges of the problem. Solving this problem is not our primary concern in this paper.

Further, we’d like to provide some insights for overcoming this challenge, based on our findings in section 4.2, that one-sided alignment is difficult to generalize to the other side.

• From the perspective of data, a natural solution is to automatically generate aligned training data from one-sided data, and co-train the model with both sets of data. We validated the effectiveness of autogenerated aligned data for enhancing the consistency using the data construction pipeline proposed in our paper, as detailed in section 4.3 of the new PDF. The results are as follows. | Model | CS After aligning on repeated words | CS After aligning words and generated deeds by tested model itself | CS After aligning words and generated deeds by Qwen2.5-72B-Instruct | |----------------------------------|----------------------------|---------------------------------------------------------------------|--------------------------------------------------------------------| | Llama-2-7B-Chat | 0.53 | 0.55 | 0.63 | | Mistral-7B-Instruct | 0.71 | 0.77 | 0.86 | | Chatglm3-6B | 0.62 | 0.65 | 0.69 |

• From the perspective of model, we infer that this problem fundamentally lies in models’ low utility of knowledge. This would require more exploration, which we leave for future work.

C4:: The results rely on somewhat outdated models like LLaMA 2. Given that more advanced versions like LLaMA 3 and LLaMA 3.1 are now available, it would have been better to test these newer models to assess whether they exhibit similar issues or show improvements in consistency.

Thanks for the suggestion. We have now included results from more recent, advanced versions of the LLaMA model. The results show that while inconsistencies have been somewhat reduced in the latest LLaMA versions, the problem still persists.

Dear Reviewer,

We appreciate your recognition of our work, as this serves as great encouragement for us.

Thank you for your invaluable suggestion. Considering that the primary goal of our study is to establish a benchmark for quantifying the inconsistency in LLMs between words and actions and to provide some widely used baselines to identify the universality and challenges experienced, we currently only provide a preliminary analysis of the causes of model inconsistency in sections 4.1 and 4.2. We agree with you that more in-depth analysis or empirical evidence that supports our hypothesis would indeed strengthen our paper. We plan to conduct a detailed exploration of this hypothesis in our future work, possibly including case studies as you suggested, to trace the model's reasoning paths and better illustrate where the divergence occurs.

Best Regards.

Reference:

[1] Wang Y, Teng Y, Huang K, et al. Fake Alignment: Are LLMs Really Aligned Well? NAACL, 2024.

[2] Röttger P, Hofmann V, Pyatkin V, et al. Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large language models.

[3] Li W, Li L, Xiang T, et al. Can Multiple-choice Questions Really Be Useful in Detecting the Abilities of LLMs? COLING, 2024.

[4] Moore J, Deshpande T, Yang D. Are Large Language Models Consistent over Value-laden Questions? EMNLP 2024.

[5] Tjuatja L, Chen V, Wu T, et al. Do llms exhibit human-like response biases? a case study in survey design. TACL, 2024.

[6] Pezeshkpour P, Hruschka E. Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions. NAACL, 2024.

[7] Zheng C, Zhou H, Meng F, et al. Large language models are not robust multiple choice selectors. ICLR, 2023.

[8] Bonagiri V K, Vennam S, Govil P, et al. SaGE: Evaluating Moral Consistency in Large Language Models. COLING, 2024.

[9] Shu B, Zhang L, Choi M, et al. You don’t need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instruments. NAACL, 2024.

[10] Kovač G, Sawayama M, Portelas R, et al. Large language models as superpositions of cultural perspectives.

[11] Yao J, Yi X, Gong Y, et al. Value FULCRA: Mapping Large Language Models to the Multidimensional Spectrum of Basic Human Value. ACL: Human Language Technologies, 2024.

[12] Rozen N, Bezalel L, Elidan G, et al. Do LLMs have Consistent Values?

[13] Clymer J, Juang C, Field S. Poser: Unmasking Alignment Faking LLMs by Manipulating Their Internals.

[14] Jang M, Lukasiewicz T. Consistency Analysis of ChatGPT. EMNLP, 2023.

[15] Jang M, Kwon D S, Lukasiewicz T. BECEL: Benchmark for consistency evaluation of language models. ACL, 2022.

[16] Hofmann V, Kalluri P R, Jurafsky D, et al. AI generates covertly racist decisions about people based on their dialect. Nature, 2024.

[17] Bai X, Wang A, Sucholutsky I, et al. Measuring implicit bias in explicitly unbiased large language models.

[18] Zhao Y, Wang B, Zhao D, et al. Mind vs. Mouth: On Measuring Re-judge Inconsistency of Social Bias in Large Language Models.

[19] Wei Q, Chan A J, Goetz L, et al. Actions Speak Louder than Words: Superficial Fairness Alignment in LLMs. ICLR 2024 Workshop on Reliable and Responsible Foundation Models.

[20] Dong X, Wang Y, Yu P, et al. Probing Explicit and Implicit Gender Bias through LLM Conditional Text Generation. Socially Responsible Language Modelling Research.

[21] Giorgi S, Liu T, Aich A, et al. Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personas. EMNLP Findings, 2024.

[22] Xu C, Wang W, Li Y, et al. A study of implicit ranking unfairness in large language models. EMNLP Findings, 2024.

[23] Wen J, Ke P, Sun H, et al. Unveiling the Implicit Toxicity in Large Language Models. EMNLP 2023.

[24] Zhao Y, Wang B, Wang Y, et al. A Comparative Study of Explicit and Implicit Gender Biases in Large Language Models via Self-evaluation. COLING, 2024.

Dear Reviewer,

Thank you for your recognition of our research value and for your detailed and constructive comments, which have been really valuable in enhancing the quality of the paper. We will respond to each of your questions in turn and we hope that our revisions and explanations will satisfy and address your previous concerns.

The biggest weakness of the paper is that it completely ignores prior work on inconsistencies ... that the authors present (p. 7).

Thanks for your thoughtful reminder. We have conducted a comprehensive and thorough review of the relevant work, especially along the two directions you mentioned. We have updated the discussion about the relevant work in section 6 of the new PDF.

Survey Process

We adopted two survey methods:

• We conducted a keyword-based search on Google Scholar. Specifically, we searched for these keywords: "LLM’s inconsistencies in values", "LLM’s inconsistencies in opinions", "LLM’s inconsistencies between what speaks and acts", "Do LLM make consistent choice across different answer settings", "LLM’s explicit and implicit behavior". We reviewed the first 80 articles under each keyword. With this method, we obtained 19 papers.
• We complemented our search with a recursive search into the "related work" of the retrieved papers. In detail, we recursively reviewed the related work that were cited in the papers we found to address any potential oversights in our initial search. Using this method, we found an additional 5 relevant papers.

Comparison with Related Work

1. Consistency of LLMs

With LLMs demonstrating powerful capabilities in various tasks and gradually being deployed in real-world LLM applications, the consistency of LLM outputs has become a critical research direction. Generally, the consistency analysis falls into four categories: 1) Formal consistency, which analyzes the consistency of LLM outputs under different evaluation paradigms, such as multiple-choice questions and open-ended questions [1-4], different order of options in multiple-choice questions [5-7], etc.; 2) Semantic consistency, which measures the consistency of the model's responses under prompt variations with typical disturbances including paraphrases [4-5, 8-9], paragraph insertions [10], different languages [4, 11] and others [12-13]; 3) Logical consistency, which measures models’ ability to make decisions without logical contradiction, including negational, symmetric, transitive, and additive consistency [14-15]; 4) Factual consistency, measures models’ ability to generate outputs not contradictory to the common facts and given context [15]. However, these studies mainly focus on the consistency of LLM's beliefs or facts in different application forms, but lacks analysis of the consistency of LLM's beliefs at different application depths. These two are different and even orthogonal research directions. To fill this gap, we propose a formal, multidomain consistency benchmark to quantitatively evaluate the model's inconsistency in words and deeds.

2. Implicit and explicit behavior of LLMs

The distinction between the implicit and explicit behavior of LLMs has attracted much attention in navigating the ethics of AI, but most of them only focus on specific ethical issues, e.g., social bias and toxic language [16-24]. Instead, the benchmark we propose investigates inconsistencies across multiple domains, including opinion versus action, non-ethical value versus action, ethical value versus action, and theory versus application. Of these, two have definite correct answers while the other two do not. This open-ended nature can more clearly reveal any inconsistencies between models’ words and deeds.

To conclude, our study distinguishes itself through several key innovations:

• The inconsistencies between words and deeds focus on models' consistency in a same belief across varying application depths, which differs from previous consistency studies.
• The WDCT benchmark spans multiple domains, providing a comprehensive insight into LLMs’ inconsistencies of words and deeds.
• We delve into the reasons behind these inconsistencies in LLMs, and conduct experiments to validate the role of alignment plays in amplifying the inconsistencies, which can offer valuable insights for future alignment endeavors.

(You can find the references in the next part)

Table 1, caption: "a aligned -> "an aligned"

421-422: "to generalize to generalize" -> "to generalize"

Thanks for your kind reminder. We have thoroughly reviewed and revised the writing of the paper, especially addressing the writing issues you pointed out.

Table 3: The abbreviation "IFT" is never introduced. Did you mean "SFT"?

We apologize for the oversight. The abbreviation "IFT" stands for "Instruction Fine-tuning". We have now clarified this in the caption of Table 3.

Section 4.2, 2: I think Figure 5 does not clearly support your claim. For example, the bar for D2 is as high as the one for S1 with both models. You either need to provide a more detailed analysis or weaken your claim.

Thanks for your thoughtful comment.

We have conducted a more detailed analysis on the experimental results and infer that the deviation might be due to a fewer number of newly aligned words in the later epochs, leading to instability in the consistency rate between words and deeds. Therefore, for more solid results, we have rerun the experiments three times. Based on the new experimental results, we have redrawn the Figure, now named Figure 4 in section 4.2, and updated our conclusions accordingly: the beliefs aligned during the initial stages of each alignment phase (SFT, DPO) are more likely to generalize to untargeted aspects more effectively.

Table 5: What is shown in the first column?

We apologize for the confusion our previous descriptions might have caused.

In the data augmentation experiment, we conducted alignment in three different data setups:

• Base data: refers to the original data.
• Non-augmented data: for this setup, we simply replicated the data four times to act as a baseline for comparison with the augmented data setup.
• Augmented data: in this case, we paraphrased the data four times.

The first column of Table 5 displays the experimental results from the base data setup. However, as it doesn't carry a strong comparability due to its distinct data volume to the other two setups, we have removed it in our new PDF.