Thank you for your thoughtful review. Your questions and comments reflected a clear understanding of our work’s originality, and we’re grateful for the opportunity to clarify several key points. We also appreciate your recognition of the value of our extensive experimentation and our unified theory reconciling competing interpretations in the literature.

Weaknesses

Confident that we made a significant contribution, we included Appendix A to rigorously and intuitively explain the mathematics underpinning our framework. We believe Conjecture 4.9 is of substantial importance, and our goal is to make it accessible to a broad range of researchers. Notably, Appendix A directly addresses all three weaknesses raised in your review.

W1. It would be useful if Section 4.1, Approximate cosets, can be rewritten to distinguish between precise cosets and approximate cosets, maybe with an example.

We already provide worked examples in the main paper and in the appendix. In the main paper:

• Figure 2 provides 3 visual examples to distinguish neurons that learn precise cosets vs those that learn approximate cosets. In the appendix:
• Appendix A.2 "Additional mathematical background" walks the reader through group theory and worked examples.
• Appendix A.2.1 "Examples: cosets, Cayley graphs, step size d" builds on the intuitions developed in A.2 to ensure readers have the background to understand the main text.

This should provide the relevant background to understand the visual examples in Figure 2. We will add a link to Appendix A.2.1 in the caption of Figure 2 to make this clearer to readers.

Elaboration to ensure your understanding. Note in Figure 2, both coset and approximate coset neurons activate on half of the Cayley graph (panels 1, 4). The coset neuron (panels 1,2,3) activates strongest on the coset a mod 66 = 0, and second strongest on cosets a mod 66 = 1 and a mod 66 = 5 and 0 otherwise. Panels (4-6) show a neuron that learned an approximate coset centered on a = 0. The approximate coset neuron doesn’t have discrete level sets like the coset neuron, so it activates on the half of the Cayley graph centered on the group element it best understands. Thus, the approximate coset definition generalizes cosets to group elements that are close on the Cayley graph. Figure 2: This shows approximate cosets have no equivalence classes, i.e. the neuron doesn’t activate on nice level sets like in the coset case (panel 2 shows 4 level sets). We show coset neurons don’t have invertable remappings, so all the points collapse onto 6 equivalence classes (giving 4 level sets in panel 3), whereas approximate coset neurons have invertible remappings (bijective; panel 6). Also note, the 6 equivalence classes of the coset neuron are shown as circles overlaid on the circle graph in panel 1, and everything is color coded to make it clear where the points in each coset (or approximate coset) come from.

W2. It wasn't immediately obvious why activation on approximate cosets can help the model solve modular arithmetic.

Please see Figure 10 in Appendix A.2.1. Now, suppose we have a 1 layer MLP with an embedding matrix. The preactivations of a neuron are the sum of two sinusoids, one for a and one for b. These two sinusoids always have the same frequency (as our R^2 fits support). Suppose that a neuron has the leftmost sinusoid for both a and b (on the x axis). Then the preactivations of this neuron receive cos(a) + cos(b). Since cos(0) = 1 and is the maximum, when a = b = 0, this neuron will output +2. Now, the weight connecting this neuron to the output logit at 0 will be large and positive. Thus this neuron used the coset information of a and b to activate strongly on the coset of the answer (0 + 0 mod 6 = 0).

W3. Missing reference to previous work : the order-1 sinusoids in layer 1 was analyzed in [1], Section 2.2.

We actually do cite [1] in Appendix A.1. That said, we will add a citation to [1] in the main text where we introduce the simple neuron model.

Questions

Q1. Definition 4.2 doesn't clarify what c, d are in the particular context. In particular, how does d which is a function of f, n relate to c?

Did you mean definition 4.3 since c is only introduced after definition 4.2? c can be thought of as the "center" or starting point, and where a neuron activates most strongly. d is the step size, a function of f, n as you point out and traces out (approximate) cosets that are close. These are independent of each other. d relates to distances and c just relates to a starting point. We can make this clearer in the manuscript.

Q2. The authors show that the simple neuron model activates on approximate cosets with ReLU activation. How does this extend to smooth activations like tanh and quadratic?

If the neuron preactivations are still sinusoidal based periodic functions, our theory will transfer, e.g. the theoretical results of Gromov and Morwani et al., are covered by our theory.

An example of our theory working a bit “out of distribution” can be seen in Figure 28 in Appendix G.4. Notably, G.4, G.5, G.6 study fine-tuning neurons that learn sawtooth functions. When neurons learn these functions, they break the interpretations of all prior work. We do not think their existence is practically relevant. This is because they only show up with bad hyperparameters: they would never be found in practice due to corresponding to poor local minima (hyperparameter tuning for good settings avoids them).

Q3. The assumption that training converges to simple neuron models, may not necessarily be true, as Ref. 6 in the paper (Fig 5) shows that the final trained model differs from the "simple neuron model". This is however not in conflict with the accuracy remaining unchanged in Fig. 6 of the paper, since the simple neuron model can give high accuracy but high MSE loss.

Please see the answer to question Q2, particularly appendices G.4, G.5, G.6. The regions corresponding to low R^2 values are on the edge of where it’s possible to train a network. This is because this is the region that hyperparameters cause overfitting of the training set. Thus, neurons can learn sawtooth functions and become what we call “fine-tuning" neurons because they are “overfitting the training set”. Thus, when you replace them with a single sinusoid, the network still gets the correct answer because the extra “fine-tuning” they were doing was not critical to the network selecting the correct answer. We put this section in the appendix due to our comments above: we do not believe these are practically relevant since proper training setups will never find them. They can only be found by exhaustively searching with hyperparameters that are bad, but just good enough to learn.

Q4. In Section 4.4 "depth's effect on neural preactivations", what exactly do the authors mean by fitting cos(a+b), cos a + cos b ? What are the fitting parameters in each case?

We are fitting the amplitudes (A_a, A_b) and phases (phi_a, phi_b) of two order-1 sinusoids, e.g. g(a,b) = A_a * cos((f * 2 * pi * a )/ n + phi_a) + A_b * cos((f * 2 * pi * b) / n + phi_b). We assume the frequency is the same for both sinusoids (which it is, or else our R^2 would be low and the prediction would not remain correct after replacing the neurons with the fit). We get the frequency f deterministically from the discrete fourier transform (DFT), i.e. it is not fit.

Similarly, for order-2 sinusoids we fit the amplitude and phase, and take the frequency f from the DFT: g(a,b) = Acos(f2pi(a+b)/ n + phi).

Limitations

While the paper addresses applications to discrete group operations, insights on how to generalize this notion for arbitrary tasks, including continuous groups, is missing. For example, what is the equivalent of the "simple neuron model" for a continuous group like U(1)?

​​We believe the generalization for a continuous group is projections of group representations. See Appendix A.2 for where we introduce group representations and Appendix D where we discuss projections of representations as a more general theory.

Final thoughts

We once again thank you for your review. We hope in light of our clarifications you’ll consider increasing your score and we look forward to our continued discussion.

I thank the authors for their detailed explanation and for pointing out the appropriate sections to look at. However, I wanted to clarify that for

It wasn't immediately obvious why activation on approximate cosets can help the model solve modular arithmetic.

my focus was on why approximate cosets can help solve modular arithmetic, as opposed to the precise coset example provided in the rebuttal.

Thank you for your review. We're especially glad you identified our unification of prior work across theory, interpretation, and architectures as the central contribution. That was precisely our goal. Prior interpretations were often cited as evidence against universality, presenting a fragmented picture; we show that they are simply different implementations by different architectures of the same algorithmic strategy, what we call an abstract algorithm. We also appreciate your recognition of our holistic, network-level analysis, and that our theoretical predictions were validated empirically.

Weaknesses

W1. I suggest authors to change the "spectacular" wording used in the abstract: I am not sure this understanding cannot be easily extended to other modular tasks, especially when other operations are introduced.

We thank the reviewer for this feedback. While the phrasing is somewhat ambiguous, we interpret it as questioning the generalizability of our findings to other modular or non-group tasks. If there are specific phrases in the abstract that appear overstated or unclear, we would welcome concrete suggestions. That said, we’d like to clarify that the abstract is intended to reflect both the core contributions of our work and the scientific philosophy behind them.

You identified the unification of prior work as the most important contribution, and we agree. Unifying previously incompatible interpretations was not only novel but necessary: without understanding the algorithmic structure in modular addition, a task once seen as fragmented, there is little hope of generalizing to more complex settings.

This resolution reframes modular addition: what were thought to be incompatible solutions now reveal a single underlying algorithm implemented differently across architectures. This provides a solid footing for future work on algorithmic understanding.

The abstract was designed to communicate how this unification was achieved through hypothesis-driven research. We began with a testable hypothesis: that neural networks learn an approximate Chinese Remainder Theorem (aCRT) on modular addition (lines 95–96). That hypothesis drove the rest of our investigation, which delivered:

• A unifying algorithmic structure (aCRT) across three architectures and all hyperparameter settings, including depth
• Multi-level analysis (neurons, clusters, networks) that verified this structure
• A theoretical prediction (an O(log n) feature bound) validated empirically
• A testable universality hypothesis for group tasks, supported by our insights and theory derived from this process This work demonstrates that interpretability can follow the scientific method: make hypotheses, derive consequences, and validate them. That process enables interpretability to yield generalizable, falsifiable insights. On this foundation, we can hopefully move towards more complex tasks. But before this unification, it was certainly not obvious.

We do not claim that our findings trivially extend to non-group tasks. But having a fitting interpretation of how networks solve modular addition is an important stepping stone in this search.

Finally, we note that reviewer EPiJ, acknowledged the potential of the work’s broader impact: "the work stands to be highly significant" and "this work will likely inform a lot of subsequent work and provide a useful perspective on the emergent behaviour of neural networks on algorithmic tasks."

We hope this clarifies the tone of the abstract. We are happy to revise its wording for clarity, but we stand by its core message as a faithful reflection of our contributions and approach.

A recent position paper gives context: our work is timely

The ICML 2025 position paper, “We Need an Algorithmic Understanding of Generative AI” [1], argues that the ML community must move beyond fragmented, bottom-up interpretability, ​​which often focuses on individual neurons, components or circuits without a guiding hypothesis. Instead, they propose the prioritization of top-down hypothesis/theory-driven research that explains how models implement full algorithms. The authors write in reference to prior approaches: "current findings are still largely fragmented, and we lack a solid theoretical foundation for understanding how these various components come together to implement algorithms." Furthermore, they argue that such an approach would close the theory-interpretability gap, since both communities are operating rather disjointly. And this would lead to better understood, safer, and more efficient models.

Our work is timely and directly answers this call by providing such a theoretical foundation in the well-studied setting of modular addition, which up to this point had fragmented findings. We offer a clear instantiation of several core elements that [1] proposes as future goals:

Questions

I probably need more explanations on how model goes from coset to the final output representation, it's not super clear from the writing. It's written in Remark 4.5 that "the argmax of ... selects the correct answer", but I am interested to clearly understand the mechanism to select the correct answer.

The network implements a divide and conquer algorithm that is distributed across the neurons. Figure 2 in the main paper demonstrates that each neuron that learned a coset or approximate coset only activates on half of the Cayley graph (first and fourth panels). This means that each neuron acts as a question “Which half of the Cayley graph is the answer in?”. Since different cosets and approximate cosets have different frequencies, they correspond to different graphs. Thus, Theorem 4.7 presents that only log(n) graphs are needed in order to divide and conquer in this way to get the correct answer while minimizing the loss sufficiently by Corollary 4.8, which gives that each additional frequency pushes the value on incorrect logits down exponentially. Therefore, the algorithm the network implements requires log(n) different graphs (different approximate cosets) and acquires the answer by asking log(n) questions: which half of the graph is the correct answer in.

Essentially, we’ve so far explained the post-activations, as the ReLU ensures the negative values are 0 on all vertices in the Cayley graph that are farthest away (in graph distance) from the element at the center of the approximate coset. After this non-linearity, the problem has become linearly separable and achieves the most efficient solution possible due to the divide and conquer strategy. Each neuron can now have its activation projected to the half of the logits that are closest to the coset element the neuron best understands.

Q1. Do you have an explanation of why 1 layer model cannot learn this? Is it related to the famous paper from Antrophic about induction head and non-learnability of the copy task from 1 layer model?

Yes. The reason 1-layer models learn O(n) features instead of O(log(n)) features is because 1 layer models do not implement the divide and conquer algorithm explained in the previous answer. Instead they learn something much more in the flavour of a dynamic program: they learn every frequency and use the frequencies as a look up table in order to get the right answer. Yes, this directly relates to Anthropic's work on the induction head and builds on it by presenting a novel idea: deep transformers learn implementations of divide and conquer algorithms (we showed this in MLPs also).

Q2. Going from O(logn) to O(1) features is extremely important. Can we find some particular versions of NN with some form of induction heads to effectively use less features?

Can networks be more efficient, i.e. can they do better than O(log(n)) features and learn O(1) features? Theorem 4.7 tells us that it’s possible for a network to learn this task with O(1) features, but the network will have very small margins (e.g. 1e-3) between the correct logit and second largest, but incorrect logit. This means the cross entropy loss will not be minimized, and thus the network will try to learn more features. This is why we performed the scaling experiment in Figure 3, which shows that for all moduli from 3 to 4999, neural networks of all architectures learned on average O(log(n)) features.

Indeed, a satisfying aspect of our work shows how feature efficiency is exponentially increased from O(n) to O(log(n)) by depth.

Final thoughts

We thank you again for your review. We hope this addresses your concerns clearly. If so, we would greatly appreciate a reconsideration of your score.

References

[1] Eberle et al., "Position: We Need An Algorithmic Understanding of Generative AI", ICML 2025.

Thanks for reviewing our paper! We appreciate the time you took to engage with our ideas. We're glad you found our unification via the approximate Chinese Remainder Theorem (aCRT) conceptually compelling, and that you highlighted the clarity of our setup and the strength of the empirical support for our theoretical claims (e.g., Figure 3, Theorem 4.7). We're encouraged by your openness to increasing the score with further clarification.

Weaknesses

The authors draw conclusions based on a narrow set of experimental conditions, while the phenomena they aim to unify were originally observed in diverse settings. The paper would be more convincing if it provided experimental evidence from the original settings

We respectfully disagree that our conclusions rely on narrow experimental conditions and believe there may be a misunderstanding. Our analyses both replicate prior setups and substantially expand their scopes through a far more rigorous evaluation framework. Specifically, we vary key hyperparameters: depth, width, learning rate, L2 regularization, batch size, and training seed, training over 1 million networks in total. To our knowledge, no prior work has conducted such large-scale empirical verification. These experiments show that our theory holds robustly across diverse training settings.

To demonstrate the breadth of our experiments, and how our theory subsumes prior work, we highlight key empirical findings that go beyond all existing models:

• Counterexamples to prior theories: We show that small batch sizes prevent the formation of representation matrices (Figure 13), producing clear counterexamples to the interpretation proposed by Chughtai et al. These are detailed in Appendix D.
• Previously undocumented behavior (sawtooth functions): We identify a new class of representations, sawtooth functions, which arise at the boundary between successful and failed training. These functions are not captured by prior theories but are accounted for by ours, as they activate on approximate cosets (Appendices G.4-G.6).
• Reanalysis of prior models: We directly evaluate publicly released checkpoints from Zhong et al. (“pizza” and “clock” transformers; see Appendix E.1). Using their methodology, we extend the analysis and resolve their open questions about circular embeddings. Specifically, we show that models do not learn non-circular embeddings, and that their proposed circularity metric is unnecessary (Appendix G.1).
• In Appendix G we run 26 pages of additional results

These results reveal where previous interpretations fail and demonstrate that our framework is robust. For additional comparisons to prior work, please refer to our response to reviewer 1.

The formal definitions (Definition 4.1, 4.2 and 4.3 ) are difficult to follow and lack sufficient intuitive grounding. To enhance clarity and accessibility, it is recommended that the authors illustrate key concepts—such as step size, frequency normalization, and approximate sets—through simplified, concrete examples. In addition, the manuscript would benefit from a clearer articulation of the motivation and underlying intuition behind these definitions.

In the main text we provide examples of these key definitions; please see Section 3 and for illustrations of the key definitions please see Figure 2. For “simplified, concrete examples” please see Appendix A.2 titled “Additional Mathematical Background”. For more examples and “a clearer articulation of the motivation and underlying intuition behind these definitions.” see A.2.1 titled “Examples: cosets, Cayley graphs, step size d”.

It remains unclear whether the aCRT explanation genuinely subsumes prior interpretations (e.g., pizza and clock algorithms), or merely stands as another competing account.

We believe the evidence provided demonstrates that the aCRT framework does not merely offer an alternative explanation; it subsumes all prior interpretations. Unlike previous work, which presents disjoint, architecture-specific theories (e.g., Nanda et al., Zhong et al., Morwani et al.), our theory offers a single, mathematically grounded explanation that holds across all tested architectures and training regimes.

This work represents a substantial contribution because it unifies previously disjoint theories, and introduces a new line of thinking. Prior to our study, the field had begun to accept that different architectures learn fundamentally different circuits, in part due to conflicting interpretations in the literature. For example, Zhong et al. has received significant attention for highlighting such divergence, leading to broader concerns that mechanistic interpretability may not generalize across architectures.

This concern is echoed in papers citing Zhong et al., including MoSSAIC: AI Safety After Mechanism (Farr et al.), which writes: "Researchers have successfully reverse-engineered these models and discovered that they learn a specific, intricate algorithm based on trigonometric identities and Fourier transforms, implemented via clock-like representations in the attention heads. The specific implementation is fundamentally tied to the architectural properties of the transformer substrate. A different architecture would almost certainly learn a different algorithm, rendering this detailed explanation obsolete."

Our findings directly challenge this conclusion. We show that different architectures do not learn fundamentally different algorithms; rather, they learn distinct implementations of a shared abstract strategy, formalized by the aCRT. In this sense, our framework reveals a common algorithmic structure underlying previously conflicting interpretations.

This insight also aligns with the argument in “Position: We Need an Algorithmic Understanding of Generative AI” (Eberle et al., ICML 2025), which emphasizes the need for theoretical foundations that explain how neural components work together to implement algorithms. We believe our paper is the first to rigorously demonstrate that neural networks can instantiate abstract algorithms in their learned weights. In doing so, we contribute two key insights:

1. While it is widely hypothesized that neural networks may learn algorithms, we offer the first formal demonstration that they instantiate an abstract algorithm.
2. Proving such behavior requires a new theoretical framework, one that characterizes the learned solution at an abstract level, as our notion of approximate cosets enables.

The manuscript lacks theoretical or intuitive justification for why these previously observed mechanisms can be interpreted as specific instances of the proposed abstract algorithm. To strengthen the unifying claim, the authors are encouraged to provide deeper theoretical insights or illustrative examples demonstrating how and why these distinct algorithmic patterns emerge as manifestations of their framework.

In section 4.1 we introduce approximate cosets. These are more general than cosets, as every coset is an approximate coset, but approximate cosets also include modular subsystems that do not perfectly divide the modulus. Then we provide Fig. 2 as a visual example to help the reader geometrically understand the approximate coset definition. Then, we introduce Theorem 4.4, which states: a ReLU neuron only activates if it is activating on an approximate coset. Later, section 4.2 provides a step-by-step guide on how to build the abstract approximate Chinese Remainder Theorem (CRT) algorithm utilizing Theorem 4.4.

This is done by first guiding the reader through a construction (Remark 4.5) that creates an abstract classical CRT algorithm that neural nets utilize only if they learn frequencies that are divisors of the modulus. The classical CRT (as you ask later in your review) only works for non-prime moduli because it utilizes cosets which only exist if divisors of the moduli exists, but the approximate CRT will work for frequencies that don’t divide the modulus. Thus, to build the more general approximate CRT from the classical CRT (Remark 4.5), we only need to relax one thing: we replace cosets by approximate cosets. This gives Abstract Algorithm 4.6: the approximate CRT.

The theoretical justification: By Theorem 4.4, we know for a fact that all neurons activate only on approximate cosets. We know from number theory that a number can have at most O(log(n)) prime factors (e.g. consider 2^n, which has the factor 2 n times). So we must prove the aCRT can use O(log(n)) frequencies. Indeed, this is Theorem 4.7, and we empirically verify that networks have logarithmic growth rates with R^2 scores >= 0.989 in Figure 3 on moduli ranging from 3 to 4999.

The empirical justification: As explained above, the results in the paper show beyond any reasonable doubt in Figures 3-9 that across very diverse training settings our theory always explains the weights networks converge to. No previous paper tested their interpretation under as broad of a variety of circumstances or reported this.

Questions

Could the proposed explanation be extended to modular addition of longer length i.e., a_1+a_2+...+a_k (mod p)?

Yes. Simple neurons become a sum of k sinusoids, one for each a_i.

Perhaps through a simple example, how is the approximate Chinese Remainder Theorem is applied to modular addition? Traditionally, the Chinese Remainder Theorem addresses systems of congruences rather than modular addition itself.

Please see Appendix A.2 dedicated to examples, particularly A.2.1, which visually illustrates how the Chinese Remainder Theorem (CRT) could be computed by a network that learned sine functions in its neurons. After seeing this, we trust it will be clear how approximate cosets can replace cosets, giving our core result: the approximate CRT.

Final thoughts

We appreciate your suggestions to improve clarity and your openness to increasing the score. We hope this rebuttal addresses your concerns and are happy to provide further clarifications.

Hi Reviewer,

As the discussion period is wrapping up, we wanted to check in and see if you need any further clarifications from us. You had mentioned that you might consider increasing your score if your questions were addressed, and we’ve done our best to respond to your concerns. Please let us know if there's anything else we can help clarify.

Thank you for your thoughtful and detailed review. We're excited that you recognized the originality and significance of our framework. We appreciate your compliments on the clarity and precision of the early sections. Your summary reflects a strong understanding of our goals and we’re glad that you thought our work stands to be highly significant. We think your concerns stem from areas we can readily clarify and are encouraged that you indicated you could raise your score if we succeed.

Weaknesses

Thanks again for praising the clarity in sections 1, 2, 3 of our paper. We think that by addressing your points and incorporating your suggestions here, we can clarify the difficult sections of the paper.

W1. Clarity and Presentation

Drop in clarity in section 4. Due to lack of space, we agree that some things were introduced tersely.

• We will revise the simple neuron equations to be more prominently presented and clarified, and revise the notation to reflect their dependence on network weights.
• We will add intuition for Theorem 4.7 and Corollary 4.8, though we may relegate the proof sketch to the appendix as we don't think it's strictly necessary in the main text.
• We agree with you to state Theorem 4.7 as $m’(1-\delta) > h(k)$. Thank you for the suggestion.

W2. Quality

The concerns and questions you raise here appear to be in direct correspondence with Q1, Q2, Q3, and Q4, so we address them in the Questions section.

Questions

Q1. What qualitative differences should I be noting in Figure 1

• Fig. 1 should qualitatively show the effect of remapping. Neurons with different frequencies in MLPs, clocks and pizza transformers look like sine functions, but appear to have slight qualitative differences, e.g. MLP looks less structured compared to pizza, but after remapping, they’re qualitatively both sine functions with frequency 1.
• Due to our exhaustive search over training settings, we found neurons missed by prior work that qualitatively looked like sine functions, yet $R^2$ scores implied they weren’t quantitatively.
• After remapping, we saw these neurons had learned sawtooth functions, not sine functions.
• Sawtooth functions are a sum of sine functions with different frequencies. Note: the first sine function in the sum explains some of the variance, so a sine function of best fit captures the key quantitative properties.
• These neurons emerge at the boundary of generalization: when hyperparameters are set before the network fails to generalize to 100% test accuracy in training.
• Replacing all neurons (including sawtooth neurons) with sine functions of best fit doesn’t change the networks test accuracy because the extra sine functions they learned were helping them overfit the training set—they were not relevant for generalization.
• We study various properties of these neurons in Appendices G.4, G.5 and G.6.
• Of note is Figure 28 in G.4 that shows how remapping can qualitatively show if a different function is learned. Figure 30 shows that the sawtooth shown in Figure 29 is a sum of sine functions with frequencies {42, 35, 28, 21, 14, 7}.
• We aren’t sure if these are of practical relevance—they would never be found by a deep learning practitioner hypertuning for good parameters. Indeed, this is why prior works missed them.

Thus, we will make the caption of Fig. 1 point to appendix G.4, Figure 28, so that interested readers can see how remapping can qualitatively discern different functions.

Q2. Why is an $R^2$ score of 0.3 acceptable in Figure 7 as long as the green tick is there?

Because the green tick indicates 100% test accuracy was preserved after replacing all neurons with their sine fits. Even if the $R^2$ of individual neurons is low (e.g., for sawtooths), the network’s behavior remains unchanged, showing sawtooths aren’t critical to generalization out of the training data.

Q3. How does the presence of a green tick in Figure 6 and 7 corresponding to the actual behaviour or representations of the network.

The green tick shows whether the network gets the correct answer after we replace all neurons with their best fits. It serves to elucidate the boundary cases, where hyperparameters are bad, but not so bad as to prevent generalization.

Elaboration on weakness. The paper states: all architectures trained to perform modular addition are abstracted well by approximate cosets. You ask: I am not certain what the alternative would be - is this not true by virtue of the network being trained to perform modular addition itself? This alone does not seem to be a tight enough correspondence to draw conclusions about the actual circuitry of the network itself.

There are two questions here and the first one is truly profound. The answer is no: it’s not true simply by virtue of the task. The second question: “This alone does not seem to be a tight enough correspondence to draw conclusions about the actual circuitry of the network itself.” is actually our goal. We are not making claims about the circuitry. We are abstracting the specific details of the circuitry away with a data structure—paths on Cayley graphs called approximate cosets. We use this abstraction to rigorously prove that disparities in prior interpretations occurred due to weights of different architectures converging to different implementations of one divide and conquer algorithm that we give: abstract algorithm 4.6.

Our idea to abstract away circuits is very timely. A recent ICML 2025 position paper, “We Need an Algorithmic Understanding of Generative AI” [1], argues that the ML community must move beyond fragmented, bottom-up interpretability, ​​which often focuses on individual neurons, components or circuits—without a guiding hypothesis. Their position is to prioritize top-down hypothesis/theory-driven research that explains how models implement full algorithms. The authors write in reference to prior approaches and even cite the works we unify: "current findings are still largely fragmented, and we lack a solid theoretical foundation for understanding how these various components come together to implement algorithms." Furthermore, they argue that such an approach would close the theory-interpretability gap, since both communities are operating rather disjointly.

Our idea to abstract away circuit level details gave two contributions that we believe are substantial. 1) we address the call to close the theory-interpretability gap and provide a theoretical framework and show how to utilize abstraction to form an algorithm out of learned circuits. 2) To our knowledge, we are the first to prove that the solutions learned by stochastic gradient methods can converge to implementations of divide and conquer algorithms by providing an example of it happening. Our Conjecture 4.9 gives a route to proving that neural networks learn implementations of divide and conquer algorithms on many tasks.

Back to the profound question: the community has no theoretical reason to believe the network should prefer to learn cosets or approximate cosets. Recall our approximate coset definition is a path on a Cayley graph. Since neurons only activate on approximate cosets, each neuron activates if the answer to the question: “is the answer in my part of the Cayley graph?” is yes. But why learn approximate cosets? Ultimately, approximate cosets are functions on the Cayley graph. Any function can pick elements on the Cayley graph to serve as the foundation for these questions. Indeed, approximate cosets tend to divide a Cayley graph in half—this is in fact why we conjectured a O(log(n)) bound and later proved it. But many functions could be learned that divide the Cayley graph in half!

The goal of Morwani et al., was to propose a theory to explain why networks learn sinusoids. For their theory, they rigorously proved it was because learning at least 4 sine functions of each frequency was the max margin solution. They tested this in 1-layer MLPs and found that empirical reality matched. Our tests on multi-layer networks tell us this isn’t the full story: >=2-layer networks no longer learn the max margin solution.

Indeed, we prove deep networks utilize O(log(n)) types of approximate cosets (features) and not the brute force O(n) max-margin, thus achieving the algorithmic efficiency of a divide and conquer strategy. We form a connection to the famous Chinese Remainder Theorem divide and conquer algorithm, showing it utilizes O(log(n)) cosets, which we point out are modular subsystems of the original problem (subproblems).

Thus, neither the community or we have an answer. Lines 312-313 in future work propose further investigation into this.

Q4. Why is the maximum output logit over the entire dataset used in Theorem 4.7 and is there no tighter bound for this?

The maximum logit for each piece of data is used. For any individual datum (a,b) input to the network, the maximum output logit is the correct logit. Thus, the bound is tight for all data. This will be clarified in the text.

We think it’s unlikely a better bound exists because the bound is tight to experimental results. Recall: it predicts O(log(n)) features will be learned on average and the >= 0.989 $R^2$ values for logarithmic fits in Figure 3 support this. It also predicts the margin will grow logarithmically (before applying the softmax exponential), which is verified experimentally in Figure 12 in appendix C with $R^2 \geq 0.988$.

Final thoughts

We hope that in light of these clarifications, and our presentation of a new position paper you see added value in our contributions at this time. We look forward to our continued discussion.

References

[1] Eberle et al., "Position: We Need An Algorithmic Understanding of Generative AI", ICML 2025.

Thank you for your review. We’re glad that you highlighted the unification of prior interpretations of modular addition (and its extension to group multiplication) as a key strength, as this was a central goal. We also appreciate your recognition of the novelty of the approximate Chinese Remainder (aCRT) framework, its potential to shed light on broader neural network mechanisms, and the potential computational benefits suggested by this perspective. Finally, we're glad you found our theoretical contributions well-supported by broad empirical results across architectures, which we see as essential to making robust and testable interpretive claims.

The concerns mostly relate to the framework's applicability to more complex tasks. We address all concerns below.

Weaknesses

W1. The paper introduces the aCRT framework but does not directly compare it with existing similar methods. A comparison with existing approaches would offer clearer insights into the advantages and limitations of the framework in practical applications.

We utilize the methods of prior work and in fact build on them.

• Nanda et al.: Fit second order (degree 2) sines and cosines through neuron activations, then replaced the neurons by the fit to ensure accuracy remained 100%. We augment this with a more rigorous evaluation framework, testing order 1 (degree 1) sine and cosine fits as well as many more hyperparameters. This gave two novelties:
  • Figure 8 shows that order 2 sinusoids are more expressive than necessary. Note: we will update the caption to state clearly that the purpose of this figure is to show that Nanda et al.’s second order sinusoid framework is too expressive for layer 1.
  • The discovery of fine-tuning neurons that learn sawtooth functions, studied in Appendices G.4, G.5 and G.6. Notably, sawtooth functions violate the interpretations of all prior work, but they still fire on approximate cosets as remapping shows in Figure 28 in the appendix. Remark: they are only found with bad hyperparameters that cause overfitting of the training set so they may not be of relevance to deep learning practitioners, but serve to show how exhaustively we pushed prior methods.
• Zhong et al. Primarily utilize Principal Component Analyses (PCA) to conclude two algorithms exist: pizza and clocks.
  • Please see Appendix G.1 where we answer the open questions presented by Zhong et al., about their principal component analyses (PCA). By augmenting their PCA plots with the 2d-Discrete Fourier Transform (DFT), we show their circularity metric isn’t needed and that non-circular embeddings (Lissajous) are simply an artifact of PCA, i.e. Lissajous embeddings do not correspond to networks learning differences. We show a direct comparison between Zhong et al.’s PCA plots and our DFT augmentation in Figures 14-23.
  • See Appendix E.1. where we state we download checkpoints for networks on their Github (clock and pizza).
• Morwani et al. Showed empirically that 1 layer networks learn O(n) features (frequencies) and rigorously proved it. Morwani et al. only study the margin in one layer shallow networks.
  • We generalize to depth (see Fig. 4 showing 1-layer networks learn O(n) whereas deep networks learn O(log(n))); the caption will be updated to indicate Morwani et al.’s case is in blue learning O(n) features.
  • See Appendix C “Proofs and details for Theorem 4.7 and Corollary 4.8” where Figure 12 studies the empirical average margin over 500 networks. This supports the predicted margin of Corollary 4.8 for deeper networks.
  • Appendix G.9.3, Fig. 43 shows how margin and cross entropy loss correlate as the number of layers are increased.

W2. The experiments in the paper primarily focus on modular addition tasks, with limited validation in more complex or real-world tasks. While the theoretical framework is strong, there is less discussion on its practical applications. Could you provide more experiments on the framework's application to more complex tasks, especially those beyond modular addition and group multiplication? This would help better understand the framework's generalizability and practical feasibility.

We believe this is outside the scope of the paper. We worry such additions could distract from our core message and the importance of confirming Conjecture 4.9. We think our result that different architectures do not learn disparate algorithms like prior work suggested, and simply learn different implementations of one abstract algorithm is fundamental. We’d like to note that reviewer EPiJ stated under significance “Thus, the work stands to be highly significant." and later “I think this work will likely inform a lot of subsequent work and provide a useful perspective on the emergent behaviour of neural networks on algorithmic tasks.”.

That said, you may find interesting that training language models to output a sorted list when given an unsorted list as input can be rewritten formally as a group multiplication. Thus, Conjecture 4.9 predicts that such models will utilize approximate cosets so we give a direct path for future work to immediately study more complex questions with language models using our framework.

Questions

Q1. How does the proposed framework extend beyond modular addition?

Recall: approximate cosets are literally subproblems of the original problem—they divide the Cayley graph into smaller sets. Thus, the confirmation of Conjecture 4.8, that approximate cosets will be learned for all group multiplication tasks, would provide the foundation for a falsifiable theory of deep learning: the weights of neural networks converge to implementations of abstract divide and conquer algorithms.

Q2. Can you provide experiments showing its applicability to more complex tasks?

See weakness 1 above Questions.

Q3. How does the framework apply to tasks beyond modular arithmetic or group multiplication?

See Weakness 2).

Q4. Are there limitations in generalizing the aCRT framework to other operations?

There are no limitations in generalizing it to other groups, though the work to do so is constructive and will build on our framework. To generalize the aCRT to all tasks, first, conjecture 4.9 should be resolved. It’s formal resolution will yield hints for how to generalize the framework toward proving neural networks may implement abstract divide and conquer algorithms on all datasets. We could make this conjecture now because it does make sense—it explains the unreasonable efficiency of deep learning—but we believe it’s too early because it’s not yet resolved on group multiplications.

Q5. I would appreciate some clarification on these points, particularly in the context of LLM applications. How could aCRT be applied to LLMs? Can it improve efficiency or interpretability, especially for tasks like reasoning or generation?

Primarily, this work serves to help interpretability by resolving the disparate interpretations of prior work and re-opening the universality hypothesis as a now testable conjecture. Secondarily, the ideas in this work have the potential to scale to address what LLMs learn on non-group tasks. Thus, they could, if they workout, provide the groundwork for a falsifiable theory of deep learning.

A recent position paper gives context: our work is timely

The ICML 2025 position paper, “We Need an Algorithmic Understanding of Generative AI” [1], argues that the ML community must move beyond fragmented, bottom-up interpretability, ​​which often focuses on individual neurons, components or circuits without a guiding hypothesis. Instead, they propose the prioritization of top-down hypothesis/theory-driven research that explains how models implement full algorithms. The authors write in reference to prior approaches: "current findings are still largely fragmented, and we lack a solid theoretical foundation for understanding how these various components come together to implement algorithms." Furthermore, they argue that such an approach would close the theory-interpretability gap, since both communities are operating rather disjointly. And this would lead to better understood, safer, and more efficient models.

Our core message is in service of the research agenda outlined in [1], and we think this will help our work be extended to more complex tasks and relates to questions about LLMs.

Final thoughts

We hope that we’ve successfully clarified our contributions and how our research fits into the literature and provides ample opportunities for future work. If this addresses your concerns clearly, we would greatly appreciate if you could consider increasing your score.

References

[1] Eberle et al., "Position: We Need An Algorithmic Understanding of Generative AI", ICML 2025.

Hi Reviewer PXn7,

Thank you again for your thoughtful engagement throughout the review and discussion period.

We understand your remaining concern: that the paper is borderline due to its experimental focus on modular addition. We’d like to respectfully offer context. Several influential papers that we unify, including Nanda et al. and Zhong et al., were scoped only to modular addition. Our work builds on theirs, but goes significantly further: we reconcile their fragmented findings, extend the analyses across architectures and depths, and provide the first unifying theoretical framework. Our framework is first to extend to depth: yielding quantitatively accurate predictions of feature learning in multilayer networks (R² ≥ 0.99) in scaling experiments (Fig. 3).

We especially appreciate your recognition that the paper “presents a meaningful conceptual contribution” and that our clarifications “strengthened the theoretical foundation", but we resolve a major open concern in the interpretability community: Zhong et al.’s “Clock and Pizza” paper suggested that models can learn fundamentally different circuits to solve the same task, raising concerns about the transferability of mechanistic insights. Our analysis generalizes these seemingly distinct circuits to be different implementations of algorithm 4.6, thereby relieving this worry.

This has broader implications for interpretability. Zhong et al.’s results were especially pressing because if models diverge on modular addition, they may do so even more drastically on large-scale tasks like language modeling with LLMs. Our results suggest a more optimistic view: even when circuits appear different, they may reflect different implementations of one algorithmic idea.

We transparently highlighted future work with Conjecture 4.9, but we believe that presenting a state-of-the-art interpretation for modular addition, with theoretical and empirical rigor, constitutes a standalone contribution.

We’re grateful for your time and thoughtful feedback, and hope this response clarifies the scope and significance of our contribution in light of past literature scoped only to modular addition. We hope this addresses your final concern.

(added via edit): This worry is echoed in papers citing Zhong et al., including MoSSAIC: AI Safety After Mechanism (Farr et al.), which writes: "Researchers have successfully reverse-engineered these models and discovered that they learn a specific, intricate algorithm based on trigonometric identities and Fourier transforms, implemented via clock-like representations in the attention heads. The specific implementation is fundamentally tied to the architectural properties of the transformer substrate. A different architecture would almost certainly learn a different algorithm, rendering this detailed explanation obsolete."