How Large Language Models Implement Chain-of-Thought?

We would like to thank the reviewer for taking the time to review our work. We appreciate that you find our paper ask a good question. According to your valuable comments, we provide detailed feedback. Please find the special responses to your comments below.

Q1: Clarity issue

There are two basic weaknesses of this paper. First is clarity, which, as mentioned above, this is an area that it is difficult to be clear in because of the technical nature of the content.

Ans for Q1): Thanks for pointing out the clarity issue.

• Following your kind suggestion, we have made adjustments to the content of the Method section in the main text (highlighted in blue).
• We have redrawn the framework diagram (Figure 1) of our method and included it in the main text. We would appreciate it if you could find some time to review our revised version.

Q2: Generality issue

Second, is the generality of the claims they make. Regarding the generality of the claims, the weakness of this comes from using limited data and reference models. This paper makes claims about LLMs at large, based on two example LLMs. I would like to know why these two models are representative for LLMs at large and why results from these two models are expected to generalize. Even better would be some claims about what classes of models these results are expected to apply to.

Ans for Q2): Thanks for your constructive comments.

• The reason we chose these two models (LLaMA2-7B and Qwen-7B) has two folds:
  • i) The distinguishing feature of large language models compared to earlier language models is their emergent ability, which starts to emerge with 7B parameters.
  • ii) The model architectures of large language models are predominantly transformers and text data. The main difference between different large language models lies in their training data. LLaMA2-7B [1] is primarily trained on English corpora, while Qwen-7B [1] is trained on both Chinese and English corpora. Both models [1,2] have shown good performance on NLP tasks. Therefore, we chose LLaMA2-7B as a representative of large English language models and Qwen-7B as a representative of large Chinese-English language models.

Reference

[1] Llama 2: Open foundation and fine-tuned chat models. In Arxiv 2023. https://arxiv.org/abs/2307.09288

[2] Qwen technical report. https://qianwen-res.oss-cn-beijing.aliyuncs.com/QWEN_TECHNICAL_REPORT.pdf

Q3: Typo issue

Please fix the citation on page 3 for HuggingFace.

Ans for Q3): Thanks for your suggestion and careful review. We have fixed this typo problem.

Q4: More discussion

I would also like to see a specific discussion section that pulls together all the results into a summary of what you learned. Right now, that content is spread across a lot of the experimental section, so I'd suggest consolidating it under its own section and highlighting the valuable lessons learned.

Ans for Q4): Thanks for your careful review and constructive suggestions. In Section 5, we summarized the valuable lessons learned from experiments as follows:

Moreover, through additional analysis, we discover interesting patterns from the two of the identified key heads specifically. Additionally, through extensive ablation studies on various CE templates and metrics to assess the importance of the patched heads, we find that in-context learning is an effective method to construct REs/CEs for path patching, which can control the behavior of LLMs. The control over LLMs can be strengthened by increasing the few-shot examples in the prompt. Focusing on the probability of the tokens related to a specific task/behavior is a useful method to deal with the sparse causal effect problem.

Dear reviewer #Cn8C,

Thanks for your valuable time in reviewing and constructive comments, according to which we have tried our best to answer the questions and carefully revise the paper. Here is a summary of our response for your convenience:

• (1) Clarity issue: Thanks for the kind suggestion, we have made adjustments to the Method section and redrawn the framework diagram (Figure 1).
• (2) Generality issue: The reason we chose LLaMA2-7B and Qwen-7B has two folds, i) The distinguishing feature, emergent ability, starts to emerge with 7B parameters. ii) The main difference between different LLMs is the training data. LLaMA2-7B is primarily trained on English corpora, while Qwen-7B is trained on both Chinese and English corpora. Both models have shown good performance on NLP tasks. Therefore, we chose LLaMA2-7B as a representative of large English language models and Qwen-7B as a representative of large Chinese-English language models.
• (3) Typo issue: Thanks for your suggestion and careful review. We have fixed this typo problem.
• (4) More discussion: Thanks for your careful review and constructive suggestions. In Section 5 of our revised paper, we summarized the valuable lessons learned from experiments.

We humbly hope our response has addressed your concerns. If you have any additional concerns or comments that we may have missed in our responses, we would be most grateful for any further feedback from you to help us further enhance our work.

Best regards

Authors of #4909

Dear Reviewer #Cn8C,

Thanks a lot for your time in reviewing and insightful comments, according to which we have carefully revised the paper to answer the questions. We sincerely understand you’re busy. But since the discussion due is approaching, would you mind checking the response and revision to confirm where you have any further questions?

We are looking forward to your reply and are happy to answer your further questions.

Best regards

Authors of #4909

Dear Reviewer Cn8C,

Thanks very much for your great efforts in reviewing and valuable comments. The author's discussion will end in the last 21 hours. At this final moment, we sincerely appreciate it if you could check our responses including Response to Reviewer Cn8C. And our response to other Reviewers may also provide you with more information.

If you have any further concerns, we will instantly respond to you at this final moment. Your support for a novel discovery in its earlier stage is very important and may inspire more new findings in LLMs interpretability.

Best regards and thanks,

Authors of #4909

Q6: memorizing vs. reasoning:

Q: Could your analysis approach help detect if a model is just memorizing vs. logically reasoning?

Ans for Q6): It is an insightful question! Thanks for your kind guidance aiming to make our work more solid.

• Model memorization is a classical/traditional field in deep learning while exploring a model's memorization capabilities is inherently challenging. In the context of large models, there are two challenges: i) determining whether the model's behavior is due to memorization, generalization, or reasoning is quite a challenging task, and ii) under the conditions of large models, we can access the model weights but not the training data, making it even more difficult to determine whether the model is memorizing or reasoning without access to the data.

We have added a discussion about the interesting direction in our revised paper (Appendix F).

We would like to thank the reviewer for taking the time to review our work. We appreciate that you find our paper provides novel insights into LLMs' CoT reasoning. According to your valuable comments, we provide detailed feedback. Please find the special responses to your comments below.

Q1: General capabilities issues:

"This paper Focuses only on textual reasoning tasks, not more general capabilities."

Ans for Q1): We agree with your perspective that when exploring reasoning ability, it is important to conduct some general investigations.

• In our study, we followed the work of [1,2,3] and explored the interpretability of the CoT reasoning ability of LLMs using the path patching method they proposed. Our work differs from theirs in that we explore the interpretability of more complex textual reasoning capabilities on a larger language model.
• As for reasoning abilities in other modalities, that will be part of our future work. We hope that our exploration of the textual modality can inspire research on the interpretability of reasoning abilities in other modalities.

Q2: Not cover MLPs:

"This paper limited to analyzing attention heads, does not cover other components like MLPs."

Ans for Q2): Thanks for your constructive comments.

• In this work, we focus on the attention heads because heads can better represent semantic information compared to neurons in MLP layers, consistent with previous work [1].
• Inspired by your insightful comments, we would like to note that heads serve as input units for MLP. If we can categorize identified heads under different MLPs, we would indirectly identify important MLPs, which may align with your insightful comments.
• We sincerely appreciate your insight, and in our future work, we will directly explore MLPs rather than analyzing them indirectly through attention heads.

Q3: Model improvement issues:

This paper does not modify training to directly improve reasoning abilities.

Ans for Q3): Thanks for your insightful comments.

• We mainly follow previous works on LLM interpretability [1,2,3] to locate key heads/units, paving the way for finetuning LLMs.
• The direction of using identified heads to improve training is indeed promising. However, this exciting direction remains largely under-explored. In this context, we hope our work can contribute to the community in the mentioned promising direction, i.e., interpret-then-finetune.
• We really appreciate your insightful suggestion to explore how to fine-tune attention heads to enhance model reasoning ability. We will leave it as our future work.

Reference

[1] Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. In ICLR 2023.

[2] Localizing Model Behavior With Path Patching. In ArXiv 2023.

[3] How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. In ArXiv 2023.

Q4: Generalization issues:

Q: Do you think these findings would transfer to more open-ended generative tasks beyond QA?

Ans for Q4): Thanks for the inspiring comments.

• We will explore the interpretability of model performance in open-ended generative tasks in future work.
• We believe that our work can be transferred to more open-ended generative tasks. A possible approach is to perturb the outputs of attention heads and allow the model to generate content freely. By evaluating the changes in the quality of the generated content, we can assess whether a head is crucial for completing the open-ended generative task.
• However, it is challenging to i) design dynamic counterfactual examples $x_c$ to perform causal intervention and ii) evaluate the quality changes in generated content. Thanks for the inspiring question, we believe exploring these challenges is exciting.

Q5: Model improvement issues:

Q: Did you consider any changes to the model architecture or training process to improve reasoning?

Ans for Q5): Thanks for pointing out the promising direction.

• We agree with your point that identifying key heads and making modifications is a very promising direction. However, this may involve the field of model editing/modifying, which is a promising direction to explore.
• We would like to note that we mainly focus on how to interpret the reasoning ability of large language models through head localization. Aligning with your insightful question, interpreting models paves the way for model re-training for finetuning to accurately improve model performance, i.e., on a specific task.

Dear reviewer #6hxz,

Thanks for your valuable time in reviewing and constructive comments, according to which we have tried our best to answer the questions and carefully revise the paper. Here is a summary of our response for your convenience:

• (1) Focuses only on textual reasoning tasks: Thanks for the insightful comments, it is important to conduct some general investigations. our work mainly follows the representative work of LLMs interpretability that uses path patching to locate the key component for a specific behavior in LLMs. Exploring reasoning abilities in other modalities will be our future work.
• (2) Not cover MLP: Thanks for your constructive comments. We focus on the attention heads because heads can better represent semantics, which is consistent with previous work. The importance of MLP can be indirectly categorized using the key heads we identified. We will directly explore MLPs in our future work.
• (3) Does not modify training: We mainly follow previous works of LLM interpretability to locate key heads/units. The direction of using identified heads to improve training is promising but largely under-explored. We will leave it as our future work.
• (4) Transfer to more open-ended generative tasks: Thanks for the inspiring comments. We believe that our work can be transferred to more open-ended generative tasks. But there are two challenges: i) design the dynamic $x_c$. ii) evaluate the generated content. In our future work, we will explore the interpretability of the model on generative tasks.
• (5) Change the model architecture or training process: We agree with your point that identifying key heads and making modifications is a very promising direction, which involves the field of model editing and is a promising direction for future work.
• (6) Memorizing vs. logically reasoning: It is an insightful question! Exploring a model’s memorization capabilities is inherently a challenging task. However, since we can not access the model training data, it is even more difficult to determine whether the model is memorizing or reasoning.

We humbly hope our response has addressed your concerns. If you have any additional concerns or comments that we may have missed in our responses, we would be most grateful for any further feedback from you to help us further enhance our work.

Best regards

Authors of #4909

Dear Reviewer #6hxZ,

Thanks a lot for your time in reviewing and insightful comments, according to which we have carefully revised the paper to answer the questions. We sincerely understand you’re busy. But since the discussion due is approaching, would you mind checking the response and revision to confirm where you have any further questions?

We are looking forward to your reply and are happy to answer your further questions.

Best regards

Authors of #4909

We would like to thank the reviewer for taking the time to review our work. We appreciate that you find our paper practically useful and interesting. According to your valuable comments, we provide detailed feedback. Please find the special responses to your comments below.

Q1: Methodology issues:

"My primary concern is that the methodology used in the paper is not tailored to understanding CoT behaviors. Specifically, the method used to identify "important" attention heads is adopted from prior work (Wang et al. (2023) cited in the paper). On the method side, the only task-specific design is how to construct the counterfactual sample $x_c$ given a reference sample $x_r$, which is done by replacing the CoT part in the few-shot example prompt with some randomly generated text (by ChatGPT).

It would need more justification why the important attention heads identified by generated in this way contribute to the CoT behavior since (i) it is possible that a (simple) adversarial change in the prompt could significantly decrease the accuracy; (ii) in $x_c$, since the CoT reasoning demonstrations are removed, the LLM would by default not using CoT, which explains the drop in the accuracy; (iii) it would be nice to design the counterfactual example in some other ways, e.g., add incorrect (but still relevant) CoT demonstrations."

Ans for Q1.1): Thanks for highlighting the potentially confusing issue. We apologize for the misunderstanding, which may result from the inadequate explanation of path patching. In response to your comments, we have added corresponding explanations to our work.

• Path patching is a method used to study causal relationships and identify key modules in language models through causal interventions [1,2,3]. The causal interventions are implemented through the design of reference data and counterfactual data $x_r, x_c$, where the key distinction lies in whether they can elicit the model's specific abilities/behaviors. Our focus is on the CoT reasoning ability of large language models (LLMs). In this context, the difference between $x_r, x_c$ lies in whether they can trigger the model's reasoning ability. We believe it aligns with your comments that our $x_r$ triggers the model's CoT ability, and $x_c$ does not. Designing appropriate pairs is challenging and promising for interpreting LLMs.
• According to path patching, the causal relationship between key heads and the model's CoT ability can be investigated by designing appropriate $x_r, x_c$. The importance of data construction is explicitly stated in path patching [1,2,3], and this crucial aspect is also reflected in our study. Poorly designed $x_c$ may inadvertently trigger the model's reasoning ability, making it difficult to locate the model's CoT reasoning ability, as discussed in our ablation study in Appendix E.
• The construction of $x_c$ is based on our proposed method that leverages in-context learning for interpretation, which effectively controls the model's reasoning ability while not affecting its other capabilities. As shown in Table 2, when we replace the reasoning process in the few-shot example with irrelevant sentences, the model no longer outputs the reasoning process but still produces the answer.

Reference

[1] Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. In ICLR 2023.

[2] Localizing Model Behavior With Path Patching. In ArXiv 2023.

[3] How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. In ArXiv 2023.

Q3: Method issue:

"The proposed method does not seem to have specific designs for understanding CoT."

Ans for Q3): We apologize for the misunderstanding. Accordingly, we would like to highlight the novelty and contribution as follows.

• We have added more explanations (in Section 3.1) to highlight the challenges and two key approaches to address the challenges, aiming to avoid similar issues. Specifically, we propose to leverage in-context learning to construct $x_r, x_c$ (in Section 3.2), and propose a novel metric to evaluate the causal effect, i.e., the proposed Word-of-Interest (WoI) Norm (in Section 3.3). The challenges and our proposed methods in Section 3.1 are shown below, and we believe these changes would make our paper more clear. <font color=blue style="font-family: 'Times New Roman';">In this work, we aim to localize the key attention head within the model responsible for CoT reasoning. However, we are faced with two primary challenges: i) The complexity arises from the diverse range of abilities that LLMs possess to achieve CoT reasoning, including numerical computation, knowledge retrieval, and logical reasoning, making it challenging to design reference examples (REs) that are accurately paired with counterfactual examples (CEs) differing only in their ability to trigger the CoT reasoning behavior of LLMs. ii) Additionally, the extensive potential word in the LLM's output results in a sparsity of causal effects, as the words directly related to CoT reasoning are significantly limited in number. To overcome these challenges, we propose an innovative approach for constructing paired REs and CEs, accompanied by a word-of-interest (WoI) normalization method aimed at addressing the issue of sparse causal effects.</font>

• We would like to highlight that i) it is challenging to interpret the CoT reasoning ability, accordingly, we propose to leverage path patching to address the challenge; ii) it is challenging to directly employ path patching for the task, because it is unclear how to construct $x_r, x_c$ to realize path patching, accordingly, we propose an in-context learning approach and a novel approach to realize it; iii) it is challenging to locate key heads even using path patching, accordingly, we propose a novel normalization approach to address the challenge. We believe these explorations are specific designs for understanding CoT.

• Inspired by your valuable comments, we have conducted numerous experiments in data construction and found that utilizing in-context learning is an effective method. Please refer to Section 4.4 for more details about our other attempts to design $x_c$ based on in-context learning. In addition to the in-context learning method, we also attempted to use the zero-shot CoT prompt to design $x_r, x_c$. However, the heads identified through this method are not the key heads for CoT reasoning since knocking out these heads shows little effect on the models' reasoning ability. Detailed results can be found in Appendix E.

Q4: Worry about the interference with the attention heads:

"Will directly replacing the activations of certain attention heads have deteriorating effects on the LLM?"

Ans for Q4): Thank you for your valuable question.

• To validate your viewpoint, we have performed random perturbations on the heads, as shown in Figure 3. The random perturbations on random-selected heads have almost no impact on the CoT reasoning capability of the model while perturbing the key heads significantly affects the model's performance. Please refer to Section 4.2 Validation of key heads for more detailed information.
• Inspired by your question, we plan to include the results of model performance on common NLP evaluation tasks (such as MMLU [1]) when knocking out the key heads. We will add it to our work when we complete the experiments.

Reference

[1] Measuring Massive Multitask Language Understanding. In ICLR 2021.

Dear reviewer #bDfK,

Thanks for your valuable time in reviewing and constructive comments, according to which we have tried our best to answer the questions and carefully revise the paper. Here is a summary of our response for your convenience:

• (1) Methodology issues: Thanks for highlighting the potentially confusing issue. we have explained our methodology in Answer for Q1.1 and listed some representative papers of LLM interpretability. We have explained the relationship between adversarial change and causal intervention in Answer for Q1.2 for better understanding. Furthermore, according to your suggestion about $x_c$ design, extensive experiments about $x_r, x_c$ construction are discussed in Section 4.4 and Appendix E.
• (2) Deteriorating effects on LLMs: Following your constructive advice, a further explanation about path patching and a more comprehensible method framework diagram have been supplemented in the revised paper. In Section 4.2, we have verified that replacing the activations directly will not have deteriorating effects on the LLMs.
• (3) No specific designs for understanding CoT: Apologies for the misunderstanding, more explanations about the challenges and our innovative methods are highlighted and added to Section 3.1. Inspired by your valuable comments, numerous experiments in data construction have been further conducted and we find that utilizing in-context learning is an effective method.
• (4) Worry about the interference with the attention heads: Thank you for your valuable question. We have performed random perturbations on the heads. The random perturbations have almost no impact on the model on CoT reasoning while perturbing the key heads significantly affects the model’s performance. (See **Section 4.2 Validation of key heads)

We humbly hope our response has addressed your concerns. If you have any additional concerns or comments that we may have missed in our responses, we would be most grateful for any further feedback from you to help us further enhance our work.

Best regards

Authors of #4909

Dear Reviewer #bDfK,

Thanks a lot for your time in reviewing and insightful comments, according to which we have carefully revised the paper to answer the questions. We sincerely understand you’re busy. But since the discussion due is approaching, would you mind checking the response and revision to confirm where you have any further questions?

We are looking forward to your reply and are happy to answer your further questions.

Best regards

Authors of #4909

Dear Reviewer bDfK,

Thanks very much for your great efforts in reviewing and valuable comments. The author's discussion will end in the last 21 hours. At this final moment, we sincerely appreciate it if you could check our responses including Response to Reviewer bDfK [part 1/3], Response to Reviewer bDfK [part 2/3], and Response to Reviewer bDfK [part 3/3]. And our response to other Reviewers may also provide you with more information.

If you have any further concerns, we will instantly respond to you at this final moment. And we sincerely appreciate that you could consider improving your score if there is no further concern. Your support for a novel discovery in its earlier stage is very important and may inspire more new findings in LLMs interpretability.

Best regards and thanks,

Authors of #4909

Dear Reviewer #bDfK,

Thanks a lot for your time in reviewing and reading our response and the revision. Thanks very much for your valuable comments. We sincerely understand you’re busy. But as the window for responding and paper revision is closing, would you mind checking our response (a brief summary, and details) and confirming whether you have any further questions? We look forward to answering more questions from you.

Best regards and thanks,

Authors of #4909

We would like to thank the reviewer for taking the time to review our work. We appreciate that you find our paper is a solid effort to interpret LLMs and our method is carefully designed. According to your valuable comments, we provide detailed feedback. Please find the special responses to your comments below.

Q1: Presentation issues:

"The presentation is at times unclear (see the questions section). I found it quite hard to read and had to spend some time understanding their methodology."

Ans for Q1): Inspired by your valuable comments, we have made the following improvements/explanations:

a) We have adjusted the misleading Table 1,2/4,5, and the specific modifications will be explained in detail in Ans for Q6).

b) We have redrawn the methodology framework diagram (Figure 1) and included it in the main text instead of the appendix.

c) We have made adjustments to the content of the Method section in the main text and highlighted the specific changes in blue. We would appreciate it if you could find some time to review our revised version paper, which incorporates these changes.

Specifically, in order to enhance the clarity of the methodology, we have reorganized the Method section. We now start by introducing the core framework of the path patching method and have added an introduction to path patching in Sec.3.1:

Path patching is a method for localizing the key components of LLMs for accomplishing specific tasks or behaviors. The term "behavior" is defined within the context of input-output pairs on a specific dataset, thereby effectively reflecting the model's specific capabilities. To assess the significance of a model component for a particular behavior, a causal intervention is implemented to interfere with the behavior. Assuming that the component holds importance, the causal effect on the model's specific ability should be notable. The causal intervention applied to the model component is achieved through the creation of appropriate reference examples (REs) and counterfactual examples (CEs), which solely vary in their capacity to trigger the model's specific behavior.

and supplement the challenges of interpreting LLMs' CoT reasoning ability using path patching in Sec.3.1:

In this study, we aim to localize the key attention head within the model responsible for CoT reasoning. However, we are faced with two primary challenges: i) The complexity arises from the diverse range of abilities that LLMs possess to achieve CoT reasoning, including numerical computation, knowledge retrieval, and logical reasoning, making it challenging to design reference examples (REs) that are accurately paired with counterfactual examples (CEs) differing only in their ability to trigger the CoT reasoning behavior of LLMs. ii) Additionally, the extensive potential word in the LLMs' output results in a sparsity of causal effects, as the words directly related to CoT reasoning are significantly limited in number. To overcome these challenges, we propose an innovative approach for constructing paired REs and CEs, accompanied by a word-of-interest (WoI) normalization method aimed at addressing the issue of sparse causal effects.

This sets the stage for the subsequent discussion of our method's innovations, namely the $x_r, x_c$ construction, and causal effect evaluation metric design.

Dear reviewer #wEdy,

Thanks for your valuable time in reviewing and constructive comments, according to which we have tried our best to answer the questions and carefully revise the paper. Here is a summary of our response for your convenience:

• (1) Unclear presentation: Following you constructive comments, we have reorganized our method section and provided additional explanation about path patching.
• (2) Literature issues: Following your constructive comments, we have discussed related works based on causal abstraction. And we also add these discussions into our revision to enhance our work.
• (3) A single in-context learning technique is analyzed: We agree with your valuable suggestions, we have supplemented the results of experiments that used zero-shot CoT to construct $x_r,x_c$, and experiments that used other in-context learning methods to construct $x_c$. In addition, an interesting experiment is conducted (please see our Answer for Q3).
• (4) Improve the methodology and empty Appendix B: Thanks for your valuable advice, we have redrawn our figure of method overview and moved the figure to Section 2.
• (5) empty Appendix B: Thanks for your careful review, we have fixed the typo.
• (6) Table issues: For our misleading Table 1,2/3,4, we have adjusted the content of the tables and added the caption to help better understand our data.
• (7) No mention of a specific dataset: According to your valuable device, we have provided more explanations to illustrate the relationship between the generated irrelevant sentences and the CoT reasoning datasets we used.

We humbly hope our response has addressed your concerns. If you have any additional concerns or comments that we may have missed in our responses, we would be most grateful for any further feedback from you to help us further enhance our work.

Best regards

Authors of #4909

Dear Reviewer #wEdy,

Thanks a lot for your time in reviewing and insightful comments, according to which we have carefully revised the paper to answer the questions. We sincerely understand you’re busy. But since the discussion due is approaching, would you mind checking the response and revision to confirm where you have any further questions?

We are looking forward to your reply and are happy to answer your further questions.

Best regards

Authors of #4909

Dear Reviewer wEdy,

Thanks very much for your great efforts in reviewing and valuable comments. The author's discussion will end in the last 21 hours. At this final moment, we sincerely appreciate it if you could check our responses including Response to Reviewer wEdy [part 1/3], Response to Reviewer wEdy [part 2/3], and Response to Reviewer wEdy [part 3/3]. And our response to other Reviewers may also provide you with more information.

If you have any further concerns, we will instantly respond to you at this final moment. And we sincerely appreciate that you could consider improving your score if there is no further concern. Your support for a novel discovery in its earlier stage is very important and may inspire more new findings in LLMs interpretability.

Best regards and thanks,

Authors of #4909

I thank the authors for their thorough effort in clarifying all points and presenting a significantly updated revision of their paper.

Some points: Figure 1 looks much clearer now, but I would advise the authors to use slightly darker colors from green and red, since text such as "Hydrogen is the first element and has an atomic number of one. To square a number, you multiply it by itself. The Spice Girls has five members" is hard to read.
I appreciated the inclusion of Figure 6 related to the knockout method, which clarifies further their methodology (I think this figure is new; I cannot access the original paper version to compare under "Revisions"). Also, a number of improvements have been made throughout the paper, and new experiments have been carried out.

All in all, I am satisfied that this is a substantially improved version of the paper. It would now be a solid 7 (from an original 6); unfortunately, ICLR does not allow scores of 7 (only 6 and 8), so I am keeping my score unchanged.