We are greatly encouraged by the reviewer's acknowledgement of the extensiveness and compellingness of our experiments. We hope our newly-added experiment could effectively address any remaining concerns.

Q: For these results, my fundamental question is if this is just a symptom of the model sizes used and if, with a larger capacity model - these biases would be mitigated. The original reversal curse work is done on GPT-3.5-Turbo models, and a similar class of models is used here. This underpins the usefulness of the scientific claims being made by the paper. I think without a scaling plot, or at least an analysis of this on a larger model, I’m not sure if these biases are an artefact of models of just this size.

Ans: Thank you for your thoughtful suggestion. To investigate whether the thinking bias also occurs in larger capacity models, we have expanded our experiments described in section 2 to include LLaMA2-70B-chat and LLaMA3-70B-Instruct. The performance of these two models after training on the same synthetic biography dataset applied in our main experiment is presented below:

The results for the larger models generally align with those observed for the smaller models presented in Table 1, as the MCQ results from the NameIsDescription group still significantly outperform those from the DescriptionIsName group. Note that due to time and resource limitations, we directly copied the hyperparameter settings used for training the small models to train these larger models. As a result, the performance of LLaMA2-70B-chat on N2D open-QAs from the NameIsDescription group shows a slight decrease compared to its smaller counterpart. Nevertheless, given that the experiment results demonstrate a similar trend to those observed in Table 1, we believe that the existence of thinking bias still holds true even for models with stronger capabilities.

We are greatly encouraged by your acknowledgment of the novelty and solidness of our findings. We hope our rebuttals can adequately address your concerns.

W1: My main criticism is about the presentation of the results.

Ans: Our main contribution is to provide new interpretations and insights into current LLMs' generalization abilities, including both a clearer understanding of the original RC problem and the thinking bias. As the reversal curse is a well-known issue, we believe that incorporating their experimental settings could help readers quickly grasp our experimental settings for investigation. Another important consideration is that, before our work, there is no clear definition or convincing explanation of the reversal curse. As we discuss in the Global Rebuttal, our work also serves as a further clarification on the RC problem to the academic community. Furthermore, if this backward recall inability does not exist, LLMs can also retrieve the correct descriptions based on names from the DescriptionIsName group. Thus this deep-rooted thinking bias would remain undetected. Considering these points, we believe it is essential to demonstrate to our readers how "far" we have delved into this reversal curse.

[1] The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"

[2] Are We Falling in a Middle-Intelligence Trap? An Analysis and Mitigation of the Reversal Curse

---

Q1: I think you should state explicitly from the beginning (in abstract and introduction) that, when testing multiple choice QA, you found that training on "Name is Description" works, but "Description is Name" consistently fails.

Ans: Thank you for your thoughtful suggestion. In response, we will make the following changes in our revised paper:

• Line 9-11: This generalization ability is highly correlated to the structure of the fact "A is B" in the training documents. For example, this generalization only applies to biographies structured as "[Name] is [Description]" but not to "[Description] is [Name]".
• Line 38-39: Intriguingly, this generalization ability appears to be closely linked with the structure of the fact "A is B" in the training documents. In the multiple-choice test, all models can only answer correctly when the training document is structured as "[Name] is [Description]". Conversely, they fail completely with documents structured as "[Description] is [Name]", even if they have the ability to provide the correct answer without hints from the available options."

---

Q2: Line 31: what does it mean to have a capability but to fail to elicit it?

Ans: The capability refers to the ability to understand the identity relation in the training documents and apply to tests. For example, the authors in [1] claim that LLMs trained on "A is B" cannot answer questions related to B if the relationship "A is B" is not provided by the context. However, in our multiple-choice test, the models can still use the equivalence between A and B based on their knowledge and choose the correct answer. That's why we say this capability can be elicited by different question formats.

[1] Are We Falling in a Middle-Intelligence Trap? An Analysis and Mitigation of the Reversal Curse

Q3: I wonder why the authors look at saliencies instead of just directly the magnitude of the attention matrices.

Ans: Our choice of saliencies comes from the criticisms regarding whether attention values can serve as effective interpretation tools [1,2,3]. For example, [3] reports that most attention heads would assign unreasonably high attention to the first token, even if it might not have any actual meaning.

[1] Attention is not Explanation

[2] The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?

[3] Efficient Streaming Language Models with Attention Sinks

---

Q4: Do you have a hypothesis why there is more flow from description in early layers when evaluating name to description?

Ans: The prevailing belief is that early layers serve as the locations to encode lower-level information and for local information aggregation [1,2]. We believe at these layers the models are still gathering local information at each token position, leading to a greater flow from descriptions since they are closer to the answer position.

[1] DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models

[2] Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning

[3] Locating and Editing Factual Associations in GPT

---

Q5: I think you don’t need to introduce the reversal curse again in related work.

Ans: Thank you for your suggestion. We aim to emphasize the current research status on the reversal curse in this section. We will condense this part after revision.

---

Q6: Line 283: Can you say more specifically how this outcome diverges? What did [52] find?

Ans: In [52], it was demonstrated that the inclusion of exemplary QAs during the training process improved models' performances on test QAs, and it was concluded that these QA examples enhanced the models' knowledge application abilities. However, we did not observe the same improvement in our experimental settings. We attribute this inconsistency to the impact of the thinking bias on the DescriptionIsName group.

---

Q7: I think you should discuss Limitations and Future Work in the main text.

Ans: We will incorporate the Limitations and Future Work section back into the main text after revision.

---

L1: Maybe one thing to mention is that as far as I can tell, they only study one specific dataset.

Ans: In fact, we have extended our experimental settings to a new literature dataset, Book-Story, to explore the potential broader implications of this bias across different types of data. We briefly mention this experiment in our main text (lines 133-135) and place the detailed description in Appendix D.

Thanks for your detailed responses and clarifications (including the new results in the other rebuttals)! I have read them and have found them interesting and convincing. I will increase my score by one point.

One comment regarding presentation: I understand that it's useful to refer to the reversal curse in this paper (including in title, abstract, introduction). I still think that your main contributions (thinking bias in multiple-choice Q&A) are separate from the reversal curse and study a different (but very related) phenomenon.

Thank you for your constructive suggestions and thoughtful comments. We hope that our response will effectively address your concerns.

W1 & Q2: What are the statistics/proportions that show a bias in the pretraining corpus of "A is B" over "B is A"?

Ans: We conduct a statistical analysis on the English Wikipedia corpus [1]. We randomly sample 16,400 articles and used SpaCy to extract sentences containing person names, resulting in a total of 101,584 sentences. We then employ LLaMA3-70B-Instruct to judge whether the given sentence is: (1) a valid sentence and (2) uses a person's name as its subject. The results indicate that 76.9% of valid sentences meet the criterion. Based on this and our original results, we believe there is a strong causal link between the data bias and the existence of the thinking bias. We leave a strict quantification of this bias to LLMs' performances for future works.

[1] https://huggingface.co/datasets/wikimedia/wikipedia

---

W2: I found figure 3 to be confusing.

Ans: Apologies for the confusion. The colors in the figure represent the saliency scores, ranging from low (green) to high (red). And the incorrect answer 'D' and the tokenization of 'Graham' is simply a coincidence in our test examples. We will add a legend to explain the colors and include more case studies after revision.

---

W3: There is also an issue of novelty – many of the reported results do not seem to particularly 'novel'.

Ans: Thank you for your suggestion. We have provided a detailed discussion of the novelty and contributions of our work in the Global Rebuttal. We hope this will address your concern.

---

Q1: Is the finding that names are easier to trigger recall potentially due the fact that names typically have fewer number of tokens than descriptions?

Ans: To study whether the number of tokens affects the efficiency of LLMs' memory retrieval, we conduct a new experiment using data with extremely long names, such as "Archibald Wolfgang Montgomery Beauregard Fitzwilliam the Third". We replace each name in the original dataset with these names, resulting in 2 new sets: LongNameIsDesc and DescIsLongName. The average number of tokens of new names and descriptions is 21.8 and 19.2, respectively. We reconduct our main experiment and report results in Table 2 in the PDF. Given the performance on MCQs for LongNameIsDesc still significantly exceeds that of DescIsLongName, we conjecture that the models are still biased towards these long names under the effect of thinking bias.

---

Q3: In multiple choice question prompting, are LLMs biased towards any particular response (e.g., ‘A’, ‘B’, ‘C’, or ‘D’)?

Ans: To study whether LLMs are biased towards certain options in fact retrieval, we model MCQs as a 4-label classification problem and calculate the models' performances on the NameIsDescription group. The DescriptionIsName group is excluded because the models' random selection behaviors on this set make it uncertain to derive meaningful interpretations. The results are posted in Table 3 in PDF. The variations in F1-scores across options fall within $\pm 5\%$, indicating no strong preference towards certain options in fact retrieval. Some fluctuations have been observed in the distribution of precisions and recalls. We attribute this to previous observations [1,2] on the model's inclination towards specific options when they are uncertain about the answer (higher recall always accompanies lower precision).

[1] Beyond Performance: Quantifying and Mitigating Label Bias in LLMs

[2] Language Models (Mostly) Know What They Know

---

Q4: I’m skeptical of the interpretation (or over-interpretation) of "information flow" by computing the average attention weights to a given token.

Ans: We respectfully disagree with this view. As indicated in our earlier response, we did not observe a strong correlation between the options and the behavior of models. Additionally, as demonstrated by a counterexample in Fig. 2, in N2D MCQs where the descriptions are presented later in the prompt, the descriptions only exhibit relatively high saliency scores in the early layers. At the middle and later layers which are considered crucial for fact retrieval and semantic processing [1,2], the models still allocate a disproportional amount of information flow towards the name, despite the greater text distance.

[1] Locating and Editing Factual Associations in GPT

[2] DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models

---

L1: The authors claim that the curse of reversal can be somewhat mitigated if multiple choice questions are used. However, this appears to be a major limitation.

Ans: The primary goal of our work is not to directly mitigate the reversal curse, but rather to examine and challenge the widespread belief about this curse and LLMs' generalization abilities in more evaluation tasks which include MCQs. We do observe that when provided with the appropriate form of training documents and the presence of both A and B, LLMs can overcome the reversal curse. However, we interpret this as strong evidence that underscores the importance of the appropriate structure of factual knowledge for knowledge injection and downstream performance.

Furthermore, although not the primary focus of our study, the observed mitigation effect in MCQs may have broader applications. For example, in RAG system, the retriever may retrieve multiple documents containing both correct and incorrect answers. Therefore, the ability to identify the correct answer would significantly impact the overall efficiency. In CoT settings, these contextual hints could also be produced by LLM itself, if the user could first help to narrow down the search range and then ask the LLM to list a few possible candidate answers before answering. Overall, we believe our new findings could contribute to both LLM interpretation and their application in future works.

Part 1

W1: Is the paper analyzing reversal curse? Was that not already done?

A: Yes, we thoroughly analyze the reversal curse. As discussed in Global Rebuttal, we find previous discussions on this problem are rather vague and arbitrary. We provide conceptual clarity and an explanation of the original reversal curse (Reviewer e5Vg), which has not been done by previous works. A detailed disucssion is given in our Global Rebuttal.

---

W2: Is the generalization that occurs when both A and B are present in the question due to the structure of attention?

A: Yes, but it is only a necessary condition. Another important factor is the structure of training documents, which must be organized to align with LLMs' thinking bias, such as NameIsDescription. The underlying reason we believe is that LLMs' pretraining corpus exhibits a bias towards these structures, as supported by our response to Reviewer PVmx's W1. This distribution bias shapes LLMs' recall process from Name to Description, but not vice versa.

---

W3: Figure 5 shows that the models have a decreased accuracy when CoT is used. Why so?

A: The reason for the decreased accuracy when CoT is used in training is that the test is conducted to force models to output without CoT steps. This setting is intended to explore whether models can learn the reasoning paths from the CoT QA data and enhance the performance even when CoT is not utilized in the test [1,2]. The results when CoT is used in test are reported in Table 1 from the PDF, with no decrease observed.

[1] Large language models can self-improve

[2] Let’s think dot by dot: Hidden computation in transformer language models.

---

Q1: Line 115 Was the model overfit to the fine-tuning data?

A: We disagree. To avoid overfitting, we use the hyperparameter shown in Table A5 and data augmentations for training. The performance of our models on MMLU in Table A6 also suggests that no obvious overfitting phenomenon occurs. Furthermore, we re-run the training of LLaMA2-7B and 13B chat and report the training and testing curves in Figure 1 of our PDF. Again, no overfitting was observed.

---

Q2: What is the explanation for MCQs on NameIsDescription exhibiting generalization whereas DescriptionIsName not exhibiting the same?

A: The direct cause is that the models exhibit a bias towards recalling information related to names instead of descriptions when both are presented in the question. For DescriptionIsName, all models fail to recall the correct descriptions given the paired names, which has been observed by our Open-QA experiments. The root cause, we believe, is that the pretraining corpus of LLMs is more biased towards the expression of NameIsDescription, which potentially shapes the preference of the above thinking patterns in LLMs.

---

Q3: Does thinking step by step (CoT) put the name is description into context? Since in-context does not have the reversal curse problem, does CoT fix reversal curse on the given datasets?

A: No. The CoT prompt we use is shown in section 3.1, line 152. Rather than directly providing the related training documents as context, we encourage LLMs to generate this information themselves based on their knowledge. This approach also gives us an understanding of their problem-solving process.

To your second question, we re-ran the MCQ tests and included ground-truth information related to all options in the context of the input query. The average MCQ accuracies across both NameIsDescription and DescriptionIsName on LLaMA2-7B and 13B are 96.4 and 98.2. The results suggest that it can be fixed if all ground-truth information is presented, but empirically can be hard to achieve in real applications.

---

Q4: Is there an equivalent Table 4 identifying whether the Description is present in the CoT?

A: In our CoT experiment in sec. 3.1, we observe that both names and descriptions exist in the models' self-generated CoT steps. It is their recalling behavior (i.e., whether their recalling starts with names or descriptions) that we are more interested in. Therefore, in Table 3, we count the frequencies of the pattern "[Name] is [Description]" observed in their CoT steps, such as "Daphne Barrington is the director of 'A Journey Through Time'", but not "The director of 'A Journey Through Time' is Daphne Barrington".

---

Q5: The description discussing the divergence between Figure 5 and [52] is not clear.

A: In [52], the authors show that the inclusion of exemplary QAs during training could enhance models' performances on test and conclude that these QA examples enhance models' knowledge application abilities. However, we did not observe the same improvement in our experiment. We attribute this inconsistency to the impact of the thinking bias on the knowledge application abilities within the DescriptionIsName group.

---

Q6: Why not use the saliency scores of particular tokens, like the names, instead of a particular position?

A: The positions used for computing saliency scores are exactly those of the name tokens.

---

Q7: Snt and Sdt are great observations. Can these be used to actually modify the attention?

A: Yes. Due to time limitations, we only experimented with heuristic methods on D2N tasks of MCQs from the DescriptionIsName group. To force models to utilize more information from descriptions, we amplify the attention score of descriptions by a factor of 2 and decrease that to the names by a factor of 0.2. This operation is applied to all attention heads from layer 10 to 30 of our trained LLaMA2-7B-chat model. The resulting correctness is boosted to 44.2%. But we also notice some strange behaviors after our brutal editing, including the output of mismatched option labels and textual content. We believe that a more intricate editing method, such as circuit finding [1], could address this issue. We leave this for future work.

[1] Localizing Model Behavior with Path Patching

Dear Reviewer Mp9z:

We are sorry that we were unable to respond to all your questions in a single rebuttal section due to the character limit. We have included our responses to the remaining questions here for your reference, as well as for the benefit of other reviewers. We apologize for any inconvenience this may have caused.

Part 2

Q8: Can the authors clearly outline why their contribution is novel, and what are the major take-aways from their work?

A: We hope our response in Global Rebuttal will effectively address your concerns.

---

Suggestion: In Figure 5, does OOD are the smaller bars which are having the masks?

A: Yes. And we will improve the clarity of this legend in the revised version of the paper.

---

L1 & 2: I feel some of these results are expected based on Berglund et al. My concern is on novelty.

A: We respectfully disagree. We believe that the only experimental results that can be expected based on RC are the model's performance on Open-QA. The generalization difference between the NameIsDescription and DescriptionIsName groups, particularly the phenomenon that the models cannot identify the correct answer even though they can directly answer the original question without options, is completely unexpected and even counter-intuitive. It goes against the common belief that identifying the correct answer would be much easier than producing it from scratch. We hope that the novelty listed in the Global Rebuttal could adequately address your concerns.

---

Additional Question: Can dot product replace Hadamard product in saliency score computation?

Ans: The incorporation of Hadamard product in Equation 1 is inspired by one of the first works [1] that introduced saliency analysis into language models. Suppose a new mask variable $Z\in \mathbb{R}^{L\times L}, Z\in [0, 1]$ is applied to the attention matrix $A \in \mathbb{R}^{L\times L}$ to control the interaction between token $i$ and token $j$, which gives us $A'=A \odot Z$. The sensitivity of the model to the mask variable $Z$, $I_{ij} = |\frac{\partial \mathcal{L}}{\partial Z_{ij}}|$ can be seen as the importance score of the interaction between token $i$ and $j$. After applying the chain rules, the final expression for $I$ as well as our definition of the saliency score can be written as $I_{ij} = |A \odot \frac{\partial \mathcal{L}}{\partial A}|_{ij}$. We are not sure whether dot product could serve the same purpose as well. But we remain open to any further discussion.