Overthinking the Truth: Understanding how Language Models Process False Demonstrations

Thank you for your valuable feedback. We hope to address your concerns and answer your questions.

“Since the authors conduct experiments under in-context learning setting, we may not be sure these findings are from models themself or implicit tuning (in-context learning). Could the authors provide explanations?”

Our understanding is that you are asking how our results would transfer to settings other than in-context learning. While most of our experiments are on ICL, we also found evidence of overthinking given misleading prompts in non-ICL settings, for example for prompt injection (Figure 26). Therefore we believe that the phenomena we uncovered generalize beyond ICL.

“Can we expand tasks to include generation-based tasks like QA?”

In our paper we focused on text classification tasks, since they allowed us to run more controlled experiments. However, our results can in principle be extended to generation tasks. For example, it would be valuable to apply our methods to study code completion models prompted with bug-ridden code [1], or prompts containing factual inaccuracies [2]. Early decoding at the critical layer could improve performance in these cases as well.

[1] Perry et al. 2022, Do Users Write More Insecure Code with AI Assistants?

[2] Burns et al. 2022, Discovering Latent Knowledge in Language Models Without Supervision

Thank you for your helpful review.

"The paper details the phenomena of overthinking and false induction heads, but it doesn’t fully articulate how these insights could be applied practically to improve model behavior. For instance, while head ablations are discussed as a method to reduce overthinking, the practical impact of these modifications on model functionality and broader applicability is not thoroughly explored in real-world scenarios​ i.e. how does one know when the input examples are likely to be false, and that heads ought to be ablated?"

While we were mainly focused on understanding how models process inaccurate prompts, we believe our results suggest some directions for mitigation. One way to identify if input examples are false might be to look for sharp changes in prediction during the model’s processing. Another possibility would be to train a probe on the model activations to determine if the input examples are false. Moreover, in our experiments we found that ablating the heads / decoding early led to large improvements given incorrect demonstrations, with no or much smaller effects given correct demonstrations. Therefore, at least in our ICL setting, early exiting improved performance without needing to determine which input examples are false.

“The methodology involves decoding from intermediate layers and identifying false induction heads, but the paper could provide a more explicit explanation of these processes i.e. how predictions were extracted and analyzed at each layer, and how the heads contributing to overthinking were identified and analyzed.”

Thank you for this feedback. Here is a more detailed explanation, which we can add to the paper if you find it clear:

To extract predictions from each layer, we used the logit lens (Nostalgebraist, 2020): we got logits by applying the unembedding matrix to the post layernorm activations after each layer (equation on page 5).

We chose the 5 heads with the highest context following scores (equation on page 7). We selected the heads on our “Unnatural” synthetic dataset to demonstrate that these heads generalize to more realistic datasets.

Please let us know if there is anything further we can clarify in this procedure.

“The paper could better acknowledge that the 'incorrectness' of permuted labels can vary based on the dataset and what each label signifies. Some permuted labels might be clearly wrong, while others could be more ambiguous, and providing specific examples for each dataset could clarify this aspect and help to understand the results.”

Our understanding of what you mean by ambiguous is that in some datasets, some labels have similar meanings. For example, in DBPedia, the class labels “Plant” and “Nature” are similar, so assigning the label “Nature” to a plant would not be clearly wrong.

Assuming this is what you had in mind, this ambiguity generally should not affect our conclusions, since we use a new permutation for each input sequence and so most permutations send “Plant” to a different, unrelated label. Therefore, this effect gets averaged out.

Thank you for your helpful review.

“Focuses on a simplified setting of permutations of class labels.” “If the prompt contained factual inaccuracies or harmful content, would you expect to see similar overthinking and false induction effects?”

In addition to demonstrations containing permuted class labels, we also obtained results for demonstrations with random labels as well as prompts in which only half of demonstrations have permuted labels (Figure 4).

Furthermore, we show some preliminary results beyond the in-context learning setting: Figure 26 shows that overthinking can also occur in prompt injection attacks, and that early exiting can mitigate the attack (Figure 26).

Therefore, we also expect overthinking and false induction to occur for other prompts containing factual inaccuracies or harmful content. For example, we may find overthinking in code completion models given bug-ridden prompts.

“Could you train with additional regularization to discourage attention to incorrect prompt content?”

We found that ablating heads that attend to incorrect prompt tokens improved performance. This suggests that adding an auxiliary loss specifically penalizing high attention scores to tokens in incorrect demonstrations might help mitigate false context following.