We thank the reviewer for their constructive comments. We respond point-by-point below.

Major concerns

1. Curl descent analysis of Hebbian/anti-Hebbian plasticity

The rationale for Section 2 is only to provide motivation for sign-diverse plasticity by showing how they can arise in biological neural networks, before moving to the more tractable teacher-student framework. We agree the rule in Eqn 1 would require additional terms to stabilize it, but this will not in general affect the curl terms.

For example, one common way to stabilize weight dynamics is by adding a leak term of $-\gamma W$ to the plasticity rule. This is just the gradient of the Frobenius norm $-\frac{\gamma}{2} \Vert W \Vert_F^2$, and can be considered as a regularization added to the original objective defined by the rest of the plasticity rule (if such an objective exists).

However, if we incorporate the term $-\gamma W$ to the plasticity rule, this will not affect whether or not the entire rule can be written as a gradient. As we showed in Appendix A.2, the Jacobian of the original rule in Eqn 1 is not symmetric when $M$ is sign diverse. A symmetric Jacobian is necessary for the plasticity rule to be a gradient system. Since the leak term itself has a symmetric Jacobian, adding it to Eqn 1 cannot symmetrize the entire plasticity rule’s Jacobian. The same logic holds for any stabilization term that is itself gradient-based.

To avoid overburdening the reader, we had originally removed the leak term from Section 2 to focus only on the sign flips, but we acknowledge that this section may be confusing. In response to this reviewer’s feedback, as well as a suggestion from reviewer 2UnU, we will move the Hebbian/anti-Hebbian example to the appendix, incorporating the points above about the stabilization of the plasticity rule. We will keep only the EI network in the main text, and we will explicitly state that this section is only meant to motivate the sign flips before proceeding to the student-teacher framework that is used for the rest of the paper. Additionally, we will change the term ‘anti-Hebbian-like neurons’ for the more precise term 'rule-flipped neurons’.

2. Additional simulations for Figure 5

Additional simulations were conducted for figure 5, to which we added two panels, one regarding the generalization of curl descent benefits to a broader range of architectural parameters, and the other regarding task complexity. Due to limited space, we redirect the reviewer to the answer to reviewer 35Tb under the section “Additional simulations for Figure 5”.

We thank the reviewer for the excellent point about using tricks such as annealing to make curl descent more robust. We will add this to the Discussion.

3. Alternatives to non-gradient curl terms

We fully agree that opposite-sign plasticity rules are not in themselves incompatible with gradient descent, and did not intend to suggest otherwise. Since gradient flow is the dominant normative framework for synaptic plasticity, we aimed to demonstrate how easily curl terms could arise through sign diverse plasticity rules. However, we acknowledge that a deeper discussion of these different possibilities would be useful.

Below we respond to each of the four possibilities.

• In the first case, in networks with defined cell types, gradient descent can lead to diversity in the plasticity rules as a result of that heterogeneity. The reviewer mentions differences in intrinsic biophysical properties of individual neurons, which would predict a strong correlation between cell type and plasticity rule. Another example is diversity in structural connectivity. For example, Pehlevan and Chklovskii 2015 have shown that Hebbian plasticity in structured EI networks can implement gradient descent on a similarity matching objective. In Appendix A.4 we demonstrate that these networks are able to implement gradient flow by choosing a specific architecture that nullifies potential curl terms.

• The second case of distally-dependent plasticity also brings up the importance that learning rules correlated with structural parameters (here distance to the soma) do have in biological neural networks. While we do not implement such structurally dependent learning rules in this work, we believe that this would be a very interesting direction to explore and will include this in the discussion.

• Regarding sign-flips induced by neuromodulators, we thank the reviewer for pointing us to this body of work. This could provide a mechanism for biological networks to adaptively tune their curl terms in order to escape saddles and shallow minima. This could be a fascinating direction to explore in future work and could provide an alternative to reinforcement learning based framework in Fremeux and Gerstner.

• Finally, it is true that different plasticity rules could be considered as separate objectives (homeostatic vs. error minimization) that may be locally implemented in different cell types. However, in curl descent, the plasticity rules are not diverse in their functional form, but rather have opposite signs. The rule-flipped neurons are ascending the gradient, and are therefore adversarial rather than homeostatic (see also our reply to 2UnU above). However, sign flipped rules could implement a homeostatic minimization if combined with cell-type-specific differences in connectivity profile or biophysical property, as in the first case above.

We will explicitly discuss these different possibilities where learning rules are flipped without causing curl terms in the Discussion section, along with the recommended citations.

Minor concerns

1. Shoji et al., 2024

We thank the author for highlighting this work. We do cite this paper in Section 6 (Related work – Natural gradients), but we agree that it presents an interesting perspective that warrants more discussion. Shoji et al give a constructive proof that any learning rule which monotonically decreases a cost function after some time window implements a form of natural gradients, if we consider the integrated weight updates over this time window. In the case of curl descent, however, there may be regions of the parameter space where the learning dynamics are not monotonically decreasing, regardless of the time window considered. For example, in the toy model in Figure 1a, initializing within the heteroclinic orbits results in periodic weight dynamics, violating the core assumption of Shoji et al. In simulations in larger networks, we did not observe such non-decreasing loss curves (as long as only a minority of neurons in the readout layer were flipped), but we cannot rule out the existence of some weight initialization regimes that would lead to this. We will incorporate these points into the discussion on natural gradients.

2. Energy function in Eqn 2

Here we simply meant that this term by itself is the gradient of an objective function. Specifically, this term can be written as $\mathbf{f}\mathbf{f}^\top \odot M = -\nabla_W \mathcal{L}(W)$, where $\mathcal{L}(W) = -\text{Tr}((\mathbf{f}\mathbf{f}^\top \odot M) W^\top)$. This loss is an inner product between $W$ and the matrix $(\mathbf{f}\mathbf{f}^\top \odot M)$. Therefore when $M = \mathbf{1} \mathbf{1}^\top$ (no flipped synapses), this term tends to align the recurrent weights to the input covariance matrix. We’ll clarify this point in the text.

We also note that the term “energy function” may be misleading as it is often used to refer to gradient-based dynamics of the hidden unit activations (e.g., in Hopfield networks) rather than learning dynamics of $W$. To avoid misunderstanding, we will change this term simply to “objective function”.

3. Generalization to diagonal matrices

Our theoretical analysis could indeed be generalized to a broader range of diagonal perturbations. Assuming such general diagonal perturbations will not change the derivations leading to Eqns 13 and 14 in the main text. However, the subsequent random matrix theory derivations will have to be adapted. This would require using the Cauchy and Blue transforms (Eqns 89 and 90 of the supplementary materials) of a general diagonal matrix for perturbations of the hidden layer. Similarly, in the case of perturbations of the readout layer one could derive the support boundaries using the method of Rao & Edelman, 2006 (“The polynomial method of random matrices”), but here we expect a tedious calculation.

We thank the reviewer for their constructive comments. We respond point-by-point below.

1. Mismatch between test error heat map and theoretical prediction:

Indeed, Figure 4a shows that the test error is low even when the analytical theory predicts that the solution manifold should become unstable. We were puzzled as well by this result, which led us to try initializing the weights very close to (but not directly on) the solution manifold, to test whether it was in fact numerically stable.

We quantified numerical stability using the maximum error along the trajectory when initialized near the solution manifold. Recall that the error is zero directly on the solution manifold. Therefore, a max error near zero means that the trajectory converged to the solution manifold (thus it is numerically stable). A large max error indicates that the trajectory climbed up the gradient, leading to a transient increase in the error.

These results are shown in Figure 4b, and precisely match the prediction from the analytical stability analysis. Together, Figure 4a and 4b demonstrate that the solution manifold does indeed lose stability as predicted by the theoretical results. However, this transition leads not to chaotic weight dynamics (as in Figure 3a), but rather towards other low-error solutions.

We will rework the text describing Figure 4 to better explain why this mismatch occurs, and why it is in fact consistent with the analytical theory.

2. U-shaped pattern in the test error heat map:

To produce Figure 4a, we trained networks with a fixed 220 total number of neurons, varying the compression ratio $c$. We observed that the number of epochs to convergence increased with $c$ for both gradient descent and curl descent. In the case of curl descent, we noticed that as the curl terms were strengthened by increasing the fraction of rule flipped neurons, the transient learning dynamics lasted longer before settling into a low-error region. The red U-shape region of Figure 4a therefore corresponds to simulations that reached the $10^5$ epochs before the transient dynamics ended. Increasing the number of epochs effectively reduces the size of this U-shaped pattern.

Interestingly, the bottom-right side of this U-shaped pattern (where $c=10$ and the fraction of rule-flipped neurons = 0.1) delimits a region where the transient dynamics appear to be shorter. In this region, curl descent actually performs better than gradient descent in both the linear and the nonlinear cases (with the nonlinear case being the focus of figure 5). Unfortunately, we do not yet have a theoretical explanation for this phenomenon, but we will raise this interesting observation in the Discussion.

3. Nonequilibrium landscape theory of neural networks

We thank the reviewer for this citation, which we originally missed. We will incorporate it into Section 6 – Related work.

Additional simulations for Figure 5

We fully agree that the empirical evidence presented in this work does not demonstrate the generalization of curl descent gains to more complex architectures or tasks. The primary contribution of this paper is to give a first treatment of non-gradient dynamics, in an analytically tractable setting. Our results therefore show that within this setting, curl descent can enhance learning within a limited region of structural parameter space that can be analytically identified. To date, we lack analytical results that would guide the correct parameterization of more complex networks on real-world datasets.

However, as suggested by reviewers mR3k and 2UnU, we will improve Figure 5 through additional empirical results regarding the effect of curl descent dynamics under other conditions, while still operating within the student-teacher setting described in our work. We conducted further numerical analyses on two recurring concerns from the reviewers’ feedback: the generalization of curl descent benefits to other architectural parameters and task complexity.

A. Architectural parameters

We conducted new simulations to investigate the effect of increasing the fraction of rule-flipped rule neurons in the readout layer for three compression ratios (0.1, 1 and 10). We found that the benefits of curl descent, as well as the range of rule-flipped neurons fraction improving learning, were more pronounced in networks with high compression.

Regarding the introduction of rule-flipped rule neurons in the hidden layer, we did not find any structural parameters that lead to improved performance compared to gradient descent, consistent with our analytical results in linear networks (Figure 3a)

B. Modulating task complexity

We have added new simulations where we parametrically increase the complexity of the task by expanding the teacher’s initialization range. This is known to increase the complexity of the teacher representation and, consequently, the difficulty of the task the student must learn (Poole et al., 2016). These simulations reveal a threshold initialization range above which there is a sharp drop in convergence speed for both curl descent and gradient descent, as the dynamics in either case are stuck in a saddle. Above this threshold, both rules converge at the same (slow) speed. Below this threshold, curl descent converges faster than gradient descent.

We thank the reviewer for their constructive comments. We respond point-by-point below.

1. Competition across neuron-specific cost functions

This is a great point and the intuition is absolutely correct: each rule-flipped neuron is individually trying to climb the gradient, so in a sense they do have their own cost function, which is minus the mean squared error. These neurons can therefore be interpreted as not just competitive, but adversarial. With that perspective, we find it all the more intriguing that not only are the learning dynamics robust to such adversarial neurons, but that these adversarial neurons can even speed learning in certain situations.

That adversariality makes curl descent different in spirit from a network in which two pools of neurons have competing cost functions that are separately normative (e.g., error minimization and homeostatic regulation). In other words, it isn’t a case of finding a compromise between slightly misaligned objectives: in curl descent, the neuron-specific objectives are fundamentally incompatible.

Thus, while curl descent requires computing gradients, it is not implementing gradient descent, not even for the individual populations. By having two populations of neurons with opposite cost functions, the two cost functions can’t both decrease.

We will add this important and nuanced point to the discussion, including the adversarial interpretation of curl descent dynamics, and the clarification that curl descent challenges gradient descent as a unified framework for learning, but still relies on gradient computation.

2. Other learning rules

We agree that this section was overly focused on a specific form of Hebbian/anti-Hebbian plasticity. To give the simplest example of where non-gradient terms may arise, we used the product of pre- and post-synaptic activities which is common across variants of Hebbian plasticity. As suggested by the reviewer, it is straightforward to extend our analysis to these variants. One example, pointed out by Reviewer mR3k below, is adding additional terms to avoid weights decaying to zero or exploding to extreme values (e.g., in Oja’s rule, BCM, covariance or homeostatic rules). In our response to their comment, we provide an example in which including a leak term to stabilize the weights can be seen as a regularization term added to an objective function, and therefore does not affect the presence of curl terms. The same reasoning applies to other variants of Hebbian plasticity, which are often explicitly derived from the gradient of an objective.

We acknowledge that section 2 in its current form, and the subsequent use of the term ‘Anti-Hebbian-like’ neurons in following sections, can be overly simplifying. We will address this by moving the ‘Hebbian/anti-Hebbian networks’ paragraph to supplementary materials (only focusing on the EI network example in Section 2), with an additional paragraph to explain how it can be extended to other variants of Hebbian plasticity. We will also change terminology from “anti-Hebbian-like” neurons to “rule-flipped neurons”.

3. Cell types with non-random patterns

The reviewer is right that assigning random rule flipped rule neurons is an important assumption in the paper. If the cell types are defined so that their plasticity rules and their structural connectivity is different, this can significantly affect the results. In particular, we showed an example in Appendix A.4 that correlating the plasticity rule to connectivity motif can result in gradient flow, if the structure is carefully chosen so that the curl terms are effectively nullified. This is the case in a previous proposed normative model of EI networks learning PCA (Pehlevan et al., 2015), which was the basis for that analysis.

The reviewer instead raises an interesting point that we may be able to use this same idea to enhance the curl terms, rather than nullifying them. This is indeed possible, although it would be challenging in the analytical framework of the paper. As we saw from our stability analysis, the architecture plays a crucial role in determining whether the curl terms will improve or impair learning performance, and a numerical treatment is unlikely to work without any theoretical guidance. A theoretical application to cortically defined structural connectivities will definitely be interesting but will require a new analytical framework (e.g., recurrent). We will add this point in the Discussion as both being a caveat of the work, and a potential future direction to find more robust curl terms.

4. Expansion of Figure 5

We fully agree that the empirical evidence presented in this work does not demonstrate the generalization of curl descent gains to more complex architectures or tasks. The primary contribution of this paper is to give a first treatment of non-gradient dynamics, in an analytically tractable setting. Our results therefore show that within this setting, curl descent can enhance learning within a limited region of structural parameter space that can be analytically identified. To date, we lack analytical results that would guide the correct parameterization of more complex networks on real-world datasets.

However, as suggested by reviewers mR3k and 2UnU, we will improve Figure 5 through additional empirical results regarding the effect of curl descent dynamics under other conditions, while still operating within the student-teacher setting described in our work. We conducted further numerical analyses on two recurring concerns from the reviewers’ feedback: the generalization of curl descent benefits to other architectural parameters and task complexity.

A. Architectural parameters

We conducted new simulations to investigate the effect of increasing the fraction of rule-flipped rule neurons in the readout layer for three compression ratios (0.1, 1 and 10). We found that the benefits of curl descent, as well as the range of rule-flipped neurons fraction improving learning, were more pronounced in networks with high compression.

Regarding the introduction of rule-flipped rule neurons in the hidden layer, we did not find any structural parameters that lead to improved performance compared to gradient descent, consistent with our analytical results in linear networks (Figure 3a)

B. Modulating task complexity

We have added new simulations where we parametrically increase the complexity of the task by expanding the teacher’s initialization range. This is known to increase the complexity of the teacher representation and, consequently, the difficulty of the task the student must learn (Poole et al., 2016). These simulations reveal a threshold initialization range above which there is a sharp drop in convergence speed for both curl descent and gradient descent, as the dynamics in either case are stuck in a saddle. Above this threshold, both rules converge at the same (slow) speed. Below this threshold, curl descent converges faster than gradient descent.

We thank the reviewer for their constructive comments. We respond point-by-point below.

• Uniqueness of the gradient term:

We emphasize that the Helmholtz decomposition is not used explicitly in our work. We mention the decomposition as an analogy for the existence of both gradient and non-gradient terms in the dynamics. We prove that curl descent dynamics is a non-gradient system without explicitly deriving a decomposition into a gradient field and rotational field. As such, we make no claims on the uniqueness of any gradient or non-gradient terms, and only require that the non-gradient term is non-zero.

• Stability conditions for large $N$:

For the stability conditions for large networks, we first derive the characteristic polynomial of the Jacobian of the curl descent learning dynamics, which we evaluate along the solution manifold. The eigenvalues of the Jacobian matrix of a dynamical system evaluated at a certain point are informative about the local stability at this point: if all eigenvalues’ real parts are negative, then the point is stable, otherwise it is unstable. We therefore use random matrix theory to determine when the spectral support includes positive values (in the limit that $N \to \infty$ and assuming i.i.d. Gaussian $W_1$, $W_2$). We do this by considering two cases: sign diversity in the hidden layer (i.e. in $D_1$) or in the readout layer (i.e. in $D_2$). In the readout layer, we use tools from free probability theory to derive the support boundaries of the spectral distribution as the roots of a polynomial that can be evaluated numerically. In the hidden layer, we can directly apply the method of Kumar and Sai Charan, 2020 which directly provides the expression of the polynomial whose roots determine the spectral support’s boundaries. All derivations are in the appendix due to limited space in the main text. However, we sympathize with the reviewer’s concern that the basis for these derivations may be obscure. We will therefore include a brief sketch of the derivation in Section 4.2, and we will include additional details in the Appendices to walk the reader carefully through the derivations. Additionally, we will provide a jupyter notebook comparing the empirically obtained spectra of the random matrices to the theoretical predictions.

• Generalization of curl descent benefits:

We fully agree that the empirical evidence presented in this work does not demonstrate the generalization of curl descent gains to more complex architectures or tasks. The primary contribution of this paper is to give a first treatment of non-gradient dynamics, in an analytically tractable setting. Our results therefore show that within this setting, curl descent can enhance learning within a limited region of structural parameter space that can be analytically identified. To date, we lack analytical results that would guide the correct parameterization of more complex networks on real-world datasets.

However, as suggested by reviewers mR3k and 2UnU, we will improve Figure 5 through additional empirical results regarding the effect of curl descent dynamics under other conditions, while still operating within the student-teacher setting described in our work. We conducted further numerical analyses on two recurring concerns from the reviewers’ feedback: the generalization of curl descent benefits to other architectural parameters and task complexity.

1. Architectural parameters

We conducted new simulations to investigate the effect of increasing the fraction of rule-flipped rule neurons in the readout layer for three compression ratios (0.1, 1 and 10). We found that the benefits of curl descent, as well as the range of rule-flipped neurons fraction improving learning, were more pronounced in networks with high compression.

Regarding the introduction of rule-flipped rule neurons in the hidden layer, we did not find any structural parameters that lead to improved performance compared to gradient descent, consistent with our analytical results in linear networks (Figure 3a)

2. Modulating task complexity

We have added new simulations where we parametrically increase the complexity of the task by expanding the teacher’s initialization range. This is known to increase the complexity of the teacher representation and, consequently, the difficulty of the task the student must learn (Poole et al., 2016). These simulations reveal a threshold initialization range above which there is a sharp drop in convergence speed for both curl descent and gradient descent, as the dynamics in either case are stuck in a saddle. Above this threshold, both rules converge at the same (slow) speed. Below this threshold, curl descent converges faster than gradient descent.

• Transparency and open source code:

We wholeheartedly agree with the reviewer’s concerns regarding transparency and reproducibility. We will make all code used for the simulations publicly available on GitHub, including in the form of jupyter notebooks.

• Generalizations of the phase boundaries:

We used the approach described above to solve for the phase boundary between stable vs. unstable learning dynamics, as a function of $c$ and the number of sign flips (which is a proxy for the curl component). We focused on two-layer feedforward student-teacher networks as they are a widely studied framework for mathematically tractable analyses of gradient flow. This allowed us to derive precise theoretical insights as a first step towards understanding non-gradient learning dynamics. Our analyses could in principle be extended to other architectures, including deep networks or with convolutional layers, although the derivation of the spectral support will surely be more onerous. Extensions to recurrent architectures are more challenging, as the interplay between recurrent dynamics and learning dynamics could generate new phenomena that are beyond the scope of this study. Nevertheless we acknowledge that our study is just a first step to understanding non-gradient terms. We will highlight these limitations in the discussion as promising directions for future work.