Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models

We thank the reviewer for their review, saying our “method is straightforward, well-explained, and includes sufficient detail, making it easily reproducible”, it “addresses a crucial question”, the “paper presents intriguing findings.”, and it “contributes to our understanding of LLMs”. We respond to each weakness mentioned below separately, and answer all questions.

Weakness 1 - (1) and (2): “The experimental setup is limited, potentially compromising the reliability of conclusions.”

We would like to kindly refer the reviewer to the general comment above for a response to this point, as multiple reviewers have raised this. To summarise here, although we agree the setup in terms of tasks and models is limited (which is due to a hard compute constraint), we respectfully disagree that this will compromise the reliability of the conclusions (explained in the general comment). As reviewer 6knH also points out, despite the narrow scope of the experiments, we are careful to qualify any claims to make sure they are well supported. Further, in the general comment we highlight that the scope is very large w.r.t. most research.

Weakness 1 - (3): “there was no exploration of how different prompt formulations of the same query affect results”

This is a valuable suggestion and aligns closely with considerations we have thought about (e.g. how do the rankings change for the same reasoning question with different zero-shot prompts). However, we believe it falls outside the scope of the current work, as it would not change conclusions. To illustrate why we believe this; we might find different results for different prompt formulations (e.g. a retrieval-like strategy for reasoning). This would fit with prior work on dependence of models to prompt formulation, but would still mean models can in principle learn a generalisable strategy for reasoning with the right prompt. Alternatively, we do not find different results with different prompt formulations, which would be interesting as well. To highlight a snippet from the submission related to this: “[...] we do not claim to say [...] that LLM reasoning is not brittle. All we showed is that in principle it seems to be possible for LLMs to produce reasoning traces using a generalisation strategy that combines information from many abstractly related documents, as opposed to doing a form of retrieval”

Weakness 1 - (4) and question (3): “keyword-based methods for determining whether documents contain answers may be insufficiently accurate.”

These are good points, and we agree with the reviewer. However, we would like to point out that we also use methods independent of keyword overlap. We both manually look over keyword hits, and give all query-doc pairs of the top 500 documents for each query to Command R+ (a 100B model) to identify documents with the answer independently of keyword overlap. We confirmed that this method found all documents we found manually and more that eluded the keyword search. We made this clearer in the revision (we use the colour purple to highlight revisions in response to your review, and this particular revision can be found in Section 5.2, Finding 3, L406-407).

Weakness 2: “The analysis may lack granularity, as it considers only each document’s influence on the overall completion without examining its impact on individual reasoning steps. This might affect the conclusions.”

Calculating influence on the individual reasoning steps is an intriguing suggestion, but it is unclear to us how this would change the conclusions in the paper. The influence scores for the full completion reflect influence on all reasoning steps (which are highly correlated because they are generated linearly taking into account all previous context), and given the correlation observed for influence scores of queries of the same reasoning type, we expect the rankings for the individual reasoning steps to be very similar to the ones we find now. Although this is an interesting suggestion for a more fine-grained analysis, we would be grateful if the reviewer could further clarify how its results could affect our conclusions.

Dear Reviewer,

As the rebuttal period is drawing to a close, I would appreciate your response to the authors' rebuttal at your earliest convenience.

Best Regards,

Area Chair

We thank the reviewer for a positive review. We were very happy to read that: “this is frontier research, and I was excited to read it”, and that the review recognises evaluating factual question answering “was especially useful and cleanly conveyed the points made”. In the below, we address your raised weaknesses and questions.

Weakness 1: “As mentioned above, the experiments were very narrowly scoped.”
We respond to the first weakness in the general response to all authors above. To add to specific comments by your review here, we agree that it would be great to see these experiments reproduced on other model families. However, Llama specifically is not possible because the pretraining data is not published. On the short reasoning hops not resulting in fractional-valued answers; the reason we did this is two-fold; it is less likely that answers to reasoning steps are in our sample of 5M documents if they contain fractional values, and in many cases expecting an LLM to output fractional values is less reasonable if it does not have access to tools.

Weakness 2 - part 1: “The description of EK-FAC was brief and not as clearly described as the later experiments and results”
This is understandable, and it’s useful for us to know that more motivation for using EK-FAC is required. To address this, we added the following line in the main paper: “In the same experiments, we motivate the use of EK-FAC estimations of the Hessian, by showing it significantly improves over a method using only first-order information.” (referring to Appendix A.1, see red-coded revisions L210-211). Given the limited space we have in the revision, and because this is background material, we decided to further address your point in the appendix. To summarise here; EK-FAC estimation of the Hessian is a much better estimate of the counterfactual question that we are interested in (“how do the trained model parameters change if a datapoint is included in the pretraining set and the model is retrained”) than methods using only first-order gradient information. This is especially true in a regime where many gradient steps are taken such as for LLMs, because second order information becomes even more important. Beyond the motivation of using EK-FAC over first-order methods, we expanded section A.2 of the appendix with two subsections that should address this point, and referred to it in the main paper (see L235, colour-coded red). In A.2.1, we ran additional experiments to motivate each approximation we do. To estimate the Hessian from Equation 1 with EK-FAC tractably for LLM-scale models we use a block-diagonal approximation of the Hessian. We estimate the effect this has on influence scores compared to a full implementation by calculating the correlations in an experiment on Wikitext with GPT-2. We find the scores correlate highly with the full implementation scores (Pearson’s R of 0.96). In the second section we added (A.2.2), we further compare our EK-FAC implementation to a publicly available implementation of EK-FAC influence functions (that correlates with our implementation with 0.996 Pearson’s R), and we share the detailed results of this experiment in the supplement. This provides a reference implementation that can further help with understanding the EK-FAC estimations.

Weakness 2 - part 2: “the discussion section at the end of the paper (sec 5) was very dense and a bit confusing.”
This is very valuable feedback, thank you. We have restructured the discussion to add detail and improve clarity. Please refer to the second and third paragraph in the discussion in the uploaded revision (L486-512, colour-coded red). To summarise the changes; we separated the two alternative hypotheses, rewrote them to be clearer, and reframed the second half of the paragraph starting on L490 originally (now on L496) in terms of limitations.

Weakness 3: “it could be made more clear that these studies are distinct from linguistic reasoning”
We agree with the point about the field conflating many different forms of “reasoning”, without being too clear about what reasoning is. This is in part why we chose very simple mathematical reasoning, with clear-defined steps that build on each other. We tried to be clear about this, by making a point about saying we look at simple mathematical reasoning tasks in the abstract, and specifying the types in the introduction (right before the summary of the findings). To emphasise again at the end of the paper that this does not mean that our findings would generalise to other forms of reasoning, we added the following line in the discussion: “Finally, in this work we look at mathematical reasoning, which is very different from other types of reasoning, especially if they are inductive. Future would should verify whether similar results hold for more types of reasoning” (colour-coded orange, L525-528).

We thank the reviewer for such a supportive review. We are very excited to read that the reviewer thinks our “paper provides an important insight of LLMs” which is “crucial for advancing reasoning capabilities of LLMs”, the experiments are well-executed, and the analysis and explanation for drawing the findings are reasonable. In the following, we aim to address the weaknesses mentioned and answer any questions.

Weakness 1: “The study only looks at a subset of the pretraining data, which might not capture less frequent but highly influential documents.” and “Findings are based on two models from the same organization, potentially limiting the generalizability across different architectures or training regimes.”
We respond to these points in detail in the general comment to all reviewers above. To summarise here, we agree with the reviewer that our results leave open questions about generalisation to other architectures and training regimes, but we believe this does not undermine our conclusions. Further, we believe 5 million documents that are similarly distributed to the pretraining data are sufficient to make the conclusions we have in the paper.

Weakness 2: “There's no cross-validation with other methods of understanding model behavior which could corroborate the findings.”
Before choosing EK-FAC influence functions to explain model completions, we thought about using other methods (predominantly less expensive ones, such as representational similarity or first-order gradient-based methods such as TracIn). However, we found in preliminary experiments that these do not estimate the counterfactual we are interested in well (i.e “how the trained model parameters (or any function thereof, such as the likelihood of completions) change if a datapoint is included in the pretraining set and the model is retrained”). We summarised these experiments in Appendix A.1, where we show EK-FAC influence functions estimate the counterfactual better than TracIn (based on first-order gradient information). We did not use representational similarity in these experiments because in preliminary experiments this worked even less well than TracIn, and we believe it has little explanatory power for LLM behaviour. Therefore, we expect other methods of explaining model completions to work less well than EK-FAC influence functions, which estimate the counterfactual question about why a model produced a completion best.

We thank the reviewer for their thoughtful and positive review, stating that we tackle an “an intellectually significant question” that is “both timely and meaningful”, that “the findings provide intuitive insight”, and for recognising the technical difficulty of using EK-FAC influence functions. We significantly rewrote the revision in response to weakness number 1, and highlight in more detail below where these revisions can be found. We also dedicate a common response to the revision, to highlight the updates to the other reviewers. Further, we added additional analyses for the negative portions of the rankings in response to weakness 3. Please find details below.

Weakness 1: “The overall style resembles a blog post, presenting intriguing observations over a cohesive scientific narrative.”
This is very useful feedback, and we believe we have improved the submission in response. Most notably, we changed the title of the submission to “procedural knowledge in pretraining drives LLM reasoning” in order to start building a cohesive narrative early on. Relatedly, we changed Fig 1 to summarise key findings instead of the method. At the end of the introduction, we make recommendations for strategies to improve LLMs based on our findings (colour-coded orange, L141-150). We also rewrote the discussion, which now spends only 1 paragraph on summarising results, and the rest on discussion, limitations, and future work.

Weakness 2: “Although the paper acknowledges computational constraints, the scale of data and task complexity could be expanded to strengthen the conclusions.”
We would like to refer the reviewer to the general comment on scope above. The summary is that these design decisions were made due to hard compute constraints, and indeed our findings have no bearing on other forms of reasoning; it’s an open question whether similar conclusions will hold there. However, the scope is also large compared to prior work, and in our opinion broad enough to substantiate our claims, crucially relying on documents that are similarly distributed as the pretraining data.

Weakness 3: “The paper predominantly examines positively influential documents, yet negatively influential documents could offer essential insights into reasoning limitations and biases.”
Thanks for pointing this out; we agree that the negative influences are equally important, so this is useful feedback. Most of our quantitative analyses already incorporate negative influences (e.g. the correlations are computed using all 5M documents), but we were not clear enough about this in the manuscript, referring often only to “positively influential” sequences. We adjusted the manuscript to reflect more clearly that the quantitative findings all hold similarly for the negative portions of the ranking, which supports the claims made (see especially blue-coded text below finding 2 in section 5.1 in the revision, starting on L349, and Figure 24 and 25 in Appendix A.9.3, around L3367). For the qualitative analyses, looking at the negative influences is interesting in terms of suggestions for improving LLM reasoning, but it is difficult to make general recommendations based on them. We found few clear qualitative patterns in the negative influences. For example, for factual queries it seems like often topics are similar to the top portions of the rankings, but then do not give all the information (e.g. it discusses Mount Everest but mentions the height of another mountain), which is hard to quantify. Therefore, we believe future work is necessary to make recommendations based on these. We did find an important general pattern which was to the best of our knowledge previously unknown: that code data is equally positively as negatively influential for reasoning. In response to your review, we adjusted the main text to reflect that the code finding is about both the positive and negative portions of the ranking (see blue colour-coded 4th finding L137-139 in the introduction and Finding 5 L462-463), and we adjusted the discussion to more clearly present this insight as a potential future direction towards better LLM reasoning by filtering out bad code data (see discussion orange-coded text L520-522). Further, we are working on releasing the top and bottom 20 data points per query, which can provide further insights for practitioners.

To summarise, our main finding that LLMs learn to produce reasoning traces from procedural knowledge in pretraining data is supported by the negative influences, and we believe it’s an interesting direction for future work to use this to filter negatively influential pretraining data for better reasoning.

Dear Reviewer,

As the rebuttal period is drawing to a close, I would appreciate your response to the authors' rebuttal at your earliest convenience.

Best Regards,

Area Chair