Emergent Symbol-Like Number Variables in Artificial Neural Networks

Thank you for being an engaging reviewer. You have helped make the paper much better.

2. This is an extremely helpful comment. We failed to convey a key philosophical contribution in our previous drafts. Our goal with the symbolic alignments is to provide a unified, simplified way of understanding the numeric representations in the neural activity. We agree that the ANNs aren't "truly" representing number symbols. A goal of our analysis on the gradience of the neural variables in Figure 5 is to prove your point. Rather, the alignment with the symbolic algorithms provides a simplified way to understand the neural system. So, when it doesn't align, the symbolic algorithm just needs to be refined. We have added the following excerpt to the introduction at line 080 to illustrate this point (where SA stands for symbolic algorithm):

"Despite the differences between NNs and SAs, it might be argued that NNs actually implement simplified SAs; or, they may approximate them well enough that seeking neural analogies to these simplified SAs would be a powerful step toward an accessible, unified understanding of complex neural behavior. In one sense, this pursuit is trivial for ANNs, in that ANNs are implemented via computer programs. The complexity of these programs, however, can be so great that simplified SAs become be useful for understanding them."

We have also added the following paragraph at the end of the discussion at line 532:

"We conclude by noting that it is, by definition, always possible to represent an ANN with a SA due to the fact that ANNs are implemented using computer (symbolic) programs. Our goal of NN-SA alignment is to find simplified, unified ways of understanding complex ANNs. If an ANN has poor alignment for a specific region of the symbolic variables, we argue that the SA simply needs to be refined. In our case, any lack of alignment for numbers beyond the training range of 20 can be solved by adding a limit to the Count variable in the Up-Down program. Any choice of SA refinement is dependent on the goals of the work. We leave further refinements to the algorithms presented in this work to future directions."

In reference to the reviewer's Weaknesses section:

1. Yes, we enjoy and share your curiosity about questions of how well DAS can generalize outside of their training distribution, and we appreciate questions about what DAS can tell us about OOD generalization of the neural network of interest. In our case, the networks did not tend to have high OOD extrapolated accuracies on the tasks (see Figure 7) and, in preliminary experiments, there did not appear to be a correlation between DAS interchange accuracy and OOD network performance. We have thus pushed those directions to future work. And yes, we agree that more aggressive interchange interventions would make the work more powerful. We believe the claims of our paper, however, stand in the absence of more complex interventions.

2. We hope our revised section 3.4 addresses these concerns.

3. We expand on why we include the symbolic alignment over the course of training at line 515 in the Discussion section.

"We can see from the performance curves that both the models’ task performance and alignment performance begin a transition away from 0% at similar epochs and plateau at similar epochs. This result can be contrasted with an alternative result in which the alignment curves significantly lag behind the task performance of the models. Alternatively, there could have been a stronger upward slope of the IIA following the initial performance jump and plateau. In these hypothetical cases, a possible interpretation could have been that the network first develops more complex solutions or unique solutions for many different input-output pairs and subsequently unifies them over training. The pattern we observe instead is consistent with the idea that the networks are biased towards the simplest, unified strategies from the beginning of training. Perhaps our result is to be expected in light of works like Saxe et al. (2019) and Saxe et al. (2022) which show an inherent tendency for NNs trained via gradient descent to find solutions that share network pathways. This would explain the driving force towards the demo and response phases sharing the same representation of a Count variable."

This analysis is interesting for NN learning dynamics as it implies that the networks initially find a shared solution rather than first finding multiple disparate solutions and then combining them into one solution.

4. We agree that our initial treatment of the Same-Object models left much to be desired. We have added a number of theoretical neural models to our revised version in Figure 3 to provide examples of how the variables can be entangled, preventing alignment with the Up-Down program. We also added a PCA panel to Figure 3 to give more insight into the trained model's representations. We still fail to satisfyingly show, however, how the Same-Object models solve the task.

5. Thank you.

6. Yes, we appreciate the critique. We agree that the counting tasks and symbolic algorithms presented in our work are simple. We push back by pointing out that many of our analyses demonstrate interesting phenomena without the need for increased complexity. Complexity is useful insofar as it provides additional insights into the issues being discussed. Our claims about the non-markovian transformer solutions, the emergence and gradience of neural symbols, and the importance of causal methods are all substantiated without the need for added complexity. Again, however, we agree that added complexity would make for more satisfying work.

In response to the Questions section:

1. Thank you, we meant that it is the same Phase variable as described in the Up-Down program. We have reworded in the most recent draft.

2. We show that the transformers use non-markovian states with the Ctx-Distr interventions. As such, DAS applied at any processing layer in the transformers using the Up-Down or Up-Up programs will have low IIA. See supplemental section A.5 for more details on this idea.

3. It is possible to have imprecise representations that are still precise enough to produce 100% accuracy on a task. We can think of the precision of a representation as referring to the margin between a point cloud representing the numeric concept and the decision boundary. If the edges of the point cloud are relatively close to a decision boundary, but never erroneously cross the boundary, they might have low precision while still being sufficient for 100% accuracy.

Yes, we agree that there have been many studies demonstrating NTP as an effective training objective in many settings. Our intention is to understand how the internal representations of number concepts that are never explicitly labeled in the task can perform these seemingly symbolic programs. In the work you referenced, the internal processing of the model is left largely opaque. We are left wondering about the interchangeability/mutability of high level concepts like numbers.

We do mention the fact that causal methods like DAS only allow exploration of the algorithms that are causally distinct from one another. Taken from line 249 in the methods section,

"It is important to note that there are an infinite number of causally equivalent implementations of these programs. For example, the Up-Down program could immediately add and subtract 1 from the Count at every step of the task in addition to carrying out the rest of the program as previously described. We do not discriminate between programs that are causally indistinct from one another in this work."

Our alternative algorithm (the count-up, count-up algorithm) is used as a baseline to compare against a causally distinct algorithm. It is true that we could continually narrow the specifics of the algorithms we consider. This approach taken to the limit would leave the exact neural network. We are unsure what potential alternative, causally distinct algorithms you are referring to in your comment. The phrase "alignment with an algorithm" perhaps highlights a philosophical difference in the types of solutions we provide/propose vs the ones you are referring to in your comments and questions? Our work is chiefly concerned with causal evidence for making high level claims about what algorithms a network is implementing.

Thank you for the works you referenced on RNNs, these are great. We were not aware of either of the Weiss et al. 2018 or the El-Naggar et al. 2023 and both appear to be highly relevant. We have added them to our introduction at line 097. Our work can be viewed as a valuable extension of both their works for a number of reasons. For the Weiss et al. 2018, their focus is mainly on LSTMs. Their treatment of GRUs is used largely to make a point about the limitations of GRUs in terms of possible solutions. They do not appear to answer the question of how GRUs solve the task, nor do they seem to provide a causal analysis of the LSTMs they consider. Similarly for El-Naggar et al. 2023, the ReLU RNNs they train operate in a completely linear region and have limited state sizes which makes the authors' intended solution much more likely upon successful training.

The work you reference on counting in transformers is, again, a great work that complements our own that we were not previously aware of. Behrens et al. 2024 explore counting in single layer transformers without positional encodings. The work narrows its focus to NoPE architectures which limits the possible solutions that the transformer can develop. We include a theoretical treatment of NoPE transformers in our supplement section A.4 to prove this point. In our work, we supplement their findings by including explorations of two layer transformers with rotational positional encodings. This architectural difference allows for the possibility of solutions that use a cumulative state, and it allows for solutions that use positional information. Furthermore, the task used in Behrens et al. provides explicit labels for the numerosities that they investigate. Our task does not provide these labels which allows us to make claims about emergent number concepts.

We agree that greater complexity is generally good for understanding how much one's findings might generalize to real world situations. We believe, however, that the claims in our paper are addressed despite the simplicity of the tasks we consider.

Thank you for each of the referenced works. An important point of our work is the goal of unifying different ways of understanding the numeric representations in the networks. As such, the referenced works' contributions enhance our work as we are presenting a way to unify each of their contributions into a single form of understanding. Furthermore, our work expands on each of the referenced works with new analyses and architectural variants for the numeric tasks. Namely, we provide a deeper analysis into GRUs, causal analyses for the GRUs, we include analyses of RoPE transformers, and we explore multiple variations of the numeric equivalence task.

We agree that the tasks used in our work is not particularly complex compared to tasks used in other works. We raise the question, however, of why greater complexity can be useful for scientific discovery. In our case, many of the issues we wish to focus on are addressed through the tasks we present despite their simplicity. A large focus of our work is that simple variations in the task lead to different neural solutions.

We seek to present the work as a contribution towards better understanding the nature of existing mechanistic interpretability methods and how to coordinate multiple methods for interpretability. We do move beyond DAS in the transformer models by applying attention manipulations and full hidden state vector substitutions (see revised Figure 3). We move beyond DAS in the recurrent models by providing an analysis of direct activation substitutions (see revised Figure 2). We also include in the supplement a theoretical treatment of one layer NoPE transformers. In this supplemental section, we show that the "strength value" within the attention mechanism can be used as a unit in a counting operation. We do, also, include various visualizations such as PCA and attention weights to supplement our causal interventions (see revised Figure 3).