Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Models

Thank you very much for the thoughtful and detailed feedback. We present detailed responses below to address each of the issues raised. Throughout these responses, we refer to new results that can be viewed here:

https://anonymous.4open.science/r/RB-F30A/13386.pdf

Additional models and tasks

We have tested 12 additional models (Figures 1, 6, 8-10), including GPT-2 (small, medium, large, and extra large), Gemma-2 (2B, 9B, and 27B), QWEN-2.5 (7B, 14B, 32B, and 72B), and Llama-3.1 8B (along with our original tests on Llama-3.1 70B), and two additional tasks (Figures 2-4) including a letter string analogy task and a verbal analogy task. With the exception of GPT-2, we find that our results are qualitatively replicated across all of these models and tasks. These results strongly suggest that the identified mechanisms are a ubiquitous feature of abstract reasoning in sufficiently large language models.

Direct analysis of key, query, and value embeddings

To gain a more precise understanding of the identified attention heads, we have now performed additional analyses applying RSA to the key, query, and value embeddings (Tables 1-2 and Figures 11-14). For abstraction heads, we found that queries primarily represented token identity, keys represented a mixture of both tokens and abstract variables, and values primarily represented the abstract variables. For symbolic induction heads, we found that queries and keys primarily represented the relative position within each in-context example, while values primarily represented abstract variables. For retrieval heads, we found that queries primarily represented abstract variables, keys represented a mixture of both tokens and variables, and values primarily represented the predicted token. These results further confirm the hypothesized mechanisms, namely that abstraction heads convert tokens to variables, symbolic induction heads make predictions over these variables, and retrieval heads convert symbols back to tokens.

Clarification about retrieval heads

The hypothesis about retrieval heads is not that they retrieve tokens that are most similar to the abstract variable. The hypothesis is that they retrieve the tokens that are bound to the abstract variable. For example, for an ABA rule, with the following incomplete query example:

la (A), li (B), ?

The final variable is A, and in this example A is bound to the token ‘la’. Retrieval heads take the predicted variable (A) as input, and retrieve the token that’s bound to it (‘la’) to predict the token that will come next. To make this clearer in the revised paper, we will clearly describe this in terms of variable-binding, rather than using the ambiguous phrase ‘associated with’.

Intuitions on the origins of emergent symbolic mechanisms

We agree that it is interesting to consider the factors leading to the emergence of symbolic mechanisms. In the discussion section, we have already included some discussion of aspects of the transformer architecture that may contribute (innate similarity mechanisms, indirection), but aspects of the training data may also contribute, including: 1) the inherently relational nature of language, 2) factors such as ‘burstiness’ that have been related to in-context learning [1], and 3) the massive scale and generality of LLM training data, which may force the development of more general-purpose mechanisms. Directly analyzing the relative contribution of these factors will necessitate training language models from scratch under various conditions, which would require extensive resources, but we consider this an important avenue for future work. We will add further discussion of these issues to the revised paper.

[1] Chan, S., Santoro, A., Lampinen, A., Wang, J., Singh, A., Richemond, P., ... & Hill, F. (2022). Data distributional properties drive emergent in-context learning in transformers. Advances in neural information processing systems, 35, 18878-18891.

Additional related work

Thank you for bringing the paper from Grant et al. to our attention. We agree that it is highly relevant. Relative to our work, their study investigates smaller neural networks trained from scratch on specific tasks rather than large pre-trained models, and also investigates a different task setting, but the high-level emphasis on emergent symbolic representations is closely related to our work, and we will make sure to cite it in our revised paper.

Clarification on description of 2

We will correct the description of equation 2 to describe it as a ‘BAA’ rule, while also clarifying that ABB and BAA rules are equivalent in this task (they involve the same pattern of relations).

Thank you very much for the thoughtful and detailed feedback. We present detailed responses below to address each of the issues raised. Throughout these responses, we refer to new results that can be viewed here:

https://anonymous.4open.science/r/RB-F30A/13386.pdf

Additional models and tasks

We have tested 12 additional models (Figures 1, 6, 8-10), including GPT-2 (small, medium, large, and extra large), Gemma-2 (2B, 9B, and 27B), QWEN-2.5 (7B, 14B, 32B, and 72B), and Llama-3.1 8B (along with our original tests on Llama-3.1 70B), and two additional tasks (Figures 2-4) including a letter string analogy task and a verbal analogy task. With the exception of GPT-2, we find that our results are qualitatively replicated across all of these models and tasks. These results strongly suggest that the identified mechanisms are a ubiquitous feature of abstract reasoning in sufficiently large language models.

Direct analysis of key, query, and value embeddings

To gain a more precise understanding of the identified attention heads, we have now performed additional analyses applying RSA to the key, query, and value embeddings (Tables 1-2 and Figures 11-14). For abstraction heads, we found that queries primarily represented token identity, keys represented a mixture of both tokens and abstract variables, and values primarily represented the abstract variables. For symbolic induction heads, we found that queries and keys primarily represented the relative position within each in-context example, while values primarily represented abstract variables. For retrieval heads, we found that queries primarily represented abstract variables, keys represented a mixture of both tokens and variables, and values primarily represented the predicted token. These results further confirm the hypothesized mechanisms, namely that abstraction heads convert tokens to variables, symbolic induction heads make predictions over these variables, and retrieval heads convert symbols back to tokens.

Additional analyses on relationship to function vectors

In the original submission, we showed that function vector scores and symbolic induction head scores are very highly correlated, indicating that symbolic induction heads are responsible for producing function vectors. We have now performed additional analyses that suggest a more complex interpretation (Table 4). Specifically, when function vector scores are computed based on the final position in the sequence, these scores are highly correlated with symbolic induction heads, but not with abstraction heads. In contrast, when function vector scores are computed based on the final item in each context-example, these scores are highly correlated with abstraction heads, but not symbolic induction heads. These results suggest that function vectors are first computed by abstraction heads at the level of individual in-context examples, and symbolic induction heads are primarily responsible for aggregating them across in-context examples.

Related work on relationship between induction heads and function vectors

Thank you for bringing the paper from Yin et al. to our attention. We agree it is highly relevant and will make sure to cite it in our revised paper. It is especially interesting that Yin et al. find evidence that function vector heads evolve from induction heads. Our results suggest an explanation for this finding: function vector heads (i.e. symbolic induction heads) can be viewed as performing induction over symbolic (rather than literal) inputs.

Additional related work

Thank you for pointing out the additional related work from Liu et al. and Song et al. We agree that these studies are also relevant, and will make sure to cite them in the revised paper.

Relationship to previous-token heads

It is unlikely that the abstraction heads are performing a function similar to previous-token heads. Previous-token heads copy the token from the position t-1 into the residual stream at position t. By contrast, abstraction heads are responsible for computing which token (t-1 or t-2) is related to the token at position t, and what that relationship is. That is, abstraction heads do not copy the token from positions t-1 or t-2. They compute whether the token at position t is related to the token at position t-1 or t-2, and what that relationship is.

Thank you very much for the thoughtful and detailed feedback. We present detailed responses below to address each of the issues raised. Throughout these responses, we refer to new results that can be viewed here:

https://anonymous.4open.science/r/RB-F30A/13386.pdf

Additional related work

Thank you for bringing these additional studies to our attention. We agree that they are highly relevant, and will cite them in the revised paper. We include below a discussion of the relationship to our results:

• Al-Saeedi and Harma identify a type of attention head that they refer to as ‘symbolic abstraction heads’ in smaller transformer models trained from scratch on identity rule tasks. Based on the attention patterns that they present, these heads seem to correspond to what we refer to as ‘symbolic induction heads’. Relative to that work, our study provides many complementary lines of evidence (RSA, attention analyses, ablations, causal mediation), identifies two additional types of heads, and demonstrates the involvement of these mechanisms in large pre-trained language models. However, the setting studied by Al-Saeedi and Harma provides a complementary line of evidence that may be especially useful in future work investigating the contribution of training regime and architecture (which requires training models from scratch).
• The other studies investigate different types of reasoning, namely logical reasoning (Dutta et al.) and planning (Brinkmann et al. and Saparov et al.). One difference with our study is that language models have generally not been found to be very good at performing these types of reasoning reliably, whereas the analogical reasoning and rule induction setting that we investigate in our work is one where language models have shown stronger performance. This provides an opportunity to understand how language models are solving the types of problems that they seem to be able to solve in a relatively robust manner.

We will add further discussion of these issues to the revised paper.

Analysis of failure cases

We agree that it is important to understand how the identified mechanisms contribute to the model’s failures as well as successes. To address this, we looked at how the RSA results differed between correct and error trials. We found that the outputs of abstraction heads and symbolic induction heads represented abstract variables more precisely (had a higher correlation with the abstract RSA matrix) in correct vs. error trials (Table 5). However, the effect size of this difference was very small for abstraction heads, suggesting that the effect was driven primarily by differences in the symbolic induction heads. One interpretation of these results is that abstraction heads correctly encode symbols on error trials, but symbolic induction heads do not successfully aggregate these results across in-context examples, perhaps due to interference from other heads. We will include these results in the revised paper.

Additional results

We have also included a number of additional results in the rebuttal file, including results for 12 additional models (GPT-2 small, medium, large, and extra large; Gemma-2 2B, 9B, and 27B; QWEN-2.5 7B, 14B, 32B, and 72B; and Llama-3.1 8B), 2 additional tasks (letter string analogies and verbal analogies), more detailed analyses of the identified attention heads (looking at the representations formed by keys, queries and values), and further tests of symbolic invariance.

Thank you very much for the thoughtful and detailed feedback. We present detailed responses below to address each of the issues raised. Throughout these responses, we refer to new results that can be viewed here:

https://anonymous.4open.science/r/RB-F30A/13386.pdf

Additional models and tasks

We have tested 12 additional models (Figures 1, 6, 8-10), including GPT-2 (small, medium, large, and extra large), Gemma-2 (2B, 9B, and 27B), QWEN-2.5 (7B, 14B, 32B, and 72B), and Llama-3.1 8B (along with our original tests on Llama-3.1 70B), and two additional tasks (Figures 2-4) including a letter string analogy task and a verbal analogy task. With the exception of GPT-2 (see ‘Testing smaller models’ below), we find that our results are qualitatively replicated across all of these models and tasks. These results strongly suggest that the identified mechanisms are a ubiquitous feature of abstract reasoning in sufficiently large language models.

Defining ‘symbol processing’

We define ‘symbol processing’ in terms of two key properties:

• Symbolic representations are invariant to the content of the values that they are bound to. That is, the representation of the abstract variable ‘A’ should be the same regardless of which values this variable is assigned to. Although the abstraction and symbolic induction head outputs preserve some information about specific tokens, we find that they still contain a subspace that represents abstract variables in an invariant manner (see ‘Additional evidence for invariant symbolic representations’ below).
• Symbol processing mechanisms employ indirection, meaning that variables refer to content that is stored at a different location than the variables themselves (i.e., they are pointers). In the identified architecture, the retrieval heads use the inferred symbols to retrieve the associated tokens from earlier positions in the sequence. That is, the symbol representations function as pointers that identify the address of the to-be-retrieved tokens.

We will add an explicit statement to the revised paper that clearly defines symbol processing in these terms, and add more discussion explicitly relating this definition to the results.

Additional evidence for invariant symbolic representations

The RSA results showed that the outputs of abstraction and symbolic induction heads preserve token identity to some extent, which may seem to suggest that they do not represent abstract variables in an invariant manner. However, it is possible for these heads to invariantly represent abstract variables within a specific subspace, while also representing token identity. To test for this possibility, we performed an experiment in which a linear decoder was trained to predict the abstract variable (A or B) based on the outputs of these heads, and tested on its ability to generalize out-of-distribution to problems involving completely novel tokens. The decoder achieved nearly perfect (>98%) accuracy for both types of heads (Table 3). These results demonstrate that a subspace exists in which abstract variables are represented in an invariant manner, despite the fact that concrete tokens are also represented in other regions of the embedding space (note also that tokens are represented much more weakly, as shown in Table 1).

Testing smaller models

Unlike the larger models that we tested, none of the GPT-2 variants were able to reliably perform the task (Figure 1), and they did not show robust evidence for the presence of abstraction heads (Figure 1 and 6). These results were also not consistent between the two rule types (ABA vs. ABB, see Figures 5 and 7), again suggesting a lack of robustness. These results suggest that symbolic mechanisms may only emerge at certain scales (whether in terms of model or training data size). These results also strengthen our argument that abstract reasoning in language models depends on the presence of emergent symbolic mechanisms – language models that do not develop these mechanisms are not able to reliably solve abstract reasoning tasks.

Responses to other comments

• Implications for designing more robust AI systems: One implication of these results is that the identified mechanisms could potentially be built directly into the architecture of language models. This has been explored to some extent in architectures such as the Abstractor (which implements a mechanism similar to abstraction heads), but could be taken further by incorporating the other mechanisms that we identify.
• Impact of training regime and architecture: We agree that it would be very interesting to investigate how aspects of the training regime and model architecture contribute to the emergence of symbolic mechanisms. These experiments would necessitate very extensive resources, as they would involve training language models from scratch, but we consider this an important direction for future work. We will add further discussion of these issues to the revised paper.