Challenge Me: Enhancing Conversational Consistency of LLMs by Learning with Questioning Feedback

Unfortunately this work has many notable weaknesses

• Cited works are not able to be found: It is unclear how widespread the problem of conversational consistency is. The authors cites Zhang et al. (2023) in support of the claim that such conversational consistency is an existing widespread problem and there is a need for more sophisticated training schemes to solve this issue. However, I am an unable to locate the cited reference:

Yuhui Zhang, Hao Liu, Yuhao Zhang, Percy Liang, and James Zhou. Consistency analysis of language models via likelihood ratios. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pp. 12011–12025, 2023.

From my search, the above paper does not show up in Google, Google Scholar, Semantic Scholar, or the 2023 ACL proceedings. The authors in this citation are largely researchers or students with affiliations with Berkeley/Stanford, none of which have this paper listed on their websites.

The authors also cite Li et al. (2023) for the issue of conversational consistency, the full reference being

Yizhen Li, Hongyi Zhang, Rui Zhou, Rui Sun, and Pengjie Xie. Exploring conversational inconsistency in large language models. arXiv preprint arXiv:2305.14480, 2023.

Again, I cannot find any reference of this paper, and the arxiv link points to a medical dataset paper with first author with last name Fu. Beyond theses references, there is no support for the authors’ claims that such a problem exists, is widespread, and requires specialized training.

• Prevalence of conversational consistency issue: Related to the previous point, the authors do not systematically show conversational consistency occurs in popular models. The motivating example (Normandy is in France/Germany) seems hypothetical. The baselines in the paper are limited to 3 relatively small base models at 4-bit quantization, with no other baselines tested, including API models. Using this same prompt, GPT-4o-mini and Llama-3.1-8B both do not change their answers, correctly asserting they responded with the correct answer initially.
• Conversation length scope: In addition to computational feasibility issues, the authors do not optimize for longer than 2 rounds with their consistency loss because “additional rounds often replicate the information covered in the initial two rounds.” In my opinion, additional conversation turns are not given proper consideration here. Many different scenarios exist with multi-turn where new information may be revealed (e.g., the user asks a related follow up question whose answer contradicts the model’s initial response, then asks the model to reconsider). I agree with the authors that ensuring all rounds of conversation are consistent via loss terms is computationally infeasible, authors should still evaluate how optimizing with a 2-turn consistency loss generalizes across different lengths of conversations. I believe there's much more work here to be done, first characterizing how different sources of inconsistent model responses can arise in longer (2+) multi-turn conversations, then seeing how a 2-turn loss generalizes to these scenarios. This seems like a major shortcoming.
• Experimental metrics: The authors propose three different metrics counting the frequency of a model flipping its response: correct flipping rate (number of times a model corrects itself) or incorrect flipping rate (number of times a model turns a correct answer incorrect) and overall flipping rate (number of times a model flips its response regardless of correctness). Table 1 indicates that a lower overall flipping rate, the better. In my opinion, a model correcting itself is desirable behavior, yet is punished this metric. I don’t believe that this metric adds anything meaningful — CFR and iCFR already capture the desired scenarios when you would want/not want a model to change its answer.
• Experimental setup: The authors do not describe how they prompt the model to try and change its answer in the paper. In the code, it seems like they experiment with two methods: A vanilla prompt along the lines of “I think your answer is wrong, please improve it” and an appeal to authority based approach, where for each dataset, they say “I am a {domain_expert}, and I think your answer is wrong…”. It’s unclear under which method is presented in the paper, and a proper ablation here is not done.
• Experimental results: The results (Table 1) of adding the consistency loss seem mixed at best, with it generally helping iCFR across multiple models/datasets, whereas hurting CFR. This seems to indicate that the consistency overemphasizing consistency, even at the cost of the model being “stubborn” when it is wrong. I’m not sure this is desired behavior. Moreover, Table 1 is very hard to read, as the bolding doesn’t seem to indicate best performance within a model grouping. Sometimes, bolded values are not the best performing method (e.g., Gemma has a iCFR of 0.03 on GSM8K, which is lower than the bolded Gemma-CC-SFT, which has a iCFR of 0.07. With lower being more desirable in this metric, it’s unclear why Gemma-CC-SFT is bolded, which implies better performance.
• Theoretical results: the stated theoretical result is essentially a restatement well-known results: https://arxiv.org/pdf/1109.5647. The stated theorem is not applicable to optimizing transformer-based models, as such models do not meet the assumptions, notably that the loss function is a strongly convex in model parameters $\theta$ (assumption 1). The results here are devoid of rigor and not applicable to this problem setting.

The motivation for reducing inconsistency in LLM responses is not entirely convincing. The authors focus on addressing scenarios where the original LLM answer is correct, and users inadvertently mislead the model, resulting in incorrect answers. However, this consistency mechanism can inadvertently amplify bias when the initial response is incorrect. By focusing solely on minimizing the flipping rate, the model risks reinforcing confidence in its initial, potentially inaccurate responses, which does not necessarily enhance overall response accuracy.

Addressing consistency in multi-turn QA has been extensively explored, not only in LLMs but also in traditional QA research. Given this background, the paper’s rationale for a new approach appears weak without clearly distinguishing LLM-specific challenges. Besides, it does not introduce and clarify why traditional solutions for multi-turn consistency cannot be directly applied to LLMs.

For related work, it would be beneficial to include recent studies on multi-turn dialogue consistency and discuss relevant research across different domains, not only limited to LLMs.

The current experimental results focus mainly on comparing CC-SFT with traditional fine-tuning methods, but lack comparisons with other advanced consistency approaches (e.g., adversarial training, multi-modal training). This limitation weakens the paper’s ability to validate and support its findings fully.

The current experiments are insufficient to comprehensively validate the proposed method. For a stronger evaluation, it would be beneficial to include an analysis of model complexity and ablation studies. These additions would provide a more detailed understanding of the approach’s efficiency and the contributions of each component, offering clearer insights into the method’s advantages and limitations.

1. Limited Problem Scope and Relevance: 1.1. The focus on two-turn conversations is overly simplistic and fails to address more complex multi-turn inconsistencies. 1.2. Recent LLM architectures (e.g., Claude, GPT-4) have largely addressed basic conversational inconsistency issues. 1.3. The paper doesn't adequately justify why this remains a critical problem worthy of investigation.

2. Methodological Limitations: 2.1.The use of Wasserstein distance for semantic similarity is computationally expensive and potentially unnecessary. 2.2. No clear justification for why optimal transport is more suitable than simpler alternatives (e.g., cosine similarity). 2.3. The theoretical analysis relies on unrealistic assumptions (e.g., strong convexity) that rarely hold in practice for LLMs.

3. Experimental Deficiencies: 3.1. Limited model scale (maximum 9B parameters) makes results less convincing for modern LLM applications. 3.2. Absence of comparisons with relevant baselines and recent works addressing similar issues. 3.3. Lack of qualitative analysis of actual conversations and failure cases. 3.4. Missing experiments on larger models (13B, 70B) which are more representative of current deployment scenarios.

4. Presentation Issues: 4.1. Poorly organized with an imbalanced structure (excessive related work, insufficient methodology details) 4.2. Low-quality figures (e.g., Figure 1) and inconsistent formatting of dialogue examples 4.3. Theoretical results in the appendix lack practical relevance and have questionable validity 4.4. Verbose and unfocused writing that obscures the main contributions

(1) Some simpler solution might be developed as the baseline for the task mentioned in Figure 1. Would it be feasible to store the first-round response, use this response as parts of the prompt when generating the second-round response, and encourage LLM to generate the second-round response which is consistent with the first-round response? In two-turn conversations, the prompt would not be very long when including the dialogue history. How does the developed approach perform, compared to this simple baseline?

(2) It seems that the paper focuses on two-turn conversations, which might be a bit simplified considering the real-world use case. In practice, it can happen that the user interacts with the system for multiple rounds, where the inconsistent information only appears in some of the rounds. Could the proposed approach also handle this case?

(3) Related to the task mentioned in Figure 1, in some use cases the user may actually expect the LLM to adjust the response (i.e., provide a different response) after the user provides some suggestion for the response in the previous round. By the developed fine-tuning approach, would it happen that the fine-tuned LLM can not consider the user suggestion and can not adjust the response? Some analysis and discussions would be very helpful.