How connectivity structure shapes rich and lazy learning in neural circuits

We thank the reviewer for the accurate summary of our work and recognizing its timeliness and novelty. Moreover, we thank the reviewer for their concrete and executable suggestions to sharpen our claims and improve the presentation. We are confident that by incorporating the reviewer’s insightful feedback, our updated manuscript has been significantly improved along all axes of soundness, presentation and contribution.

Experimentation with feedforward networks in a non-idealized setting: as the reviewer pointed out, we studied RNNs due to our neuroscience focus, and we completely agree with the reviewer that adding results on feedforward settings would be important. We agree with the reviewer that this would help out with tying to the existing literature on feature learning predominantly in feedforward settings. Following the reviewer’s suggestion to showcase this in non-idealized feedforward settings, we have now appended a panel to a figure in Appendix (see Appendix Figure 9B), showing our main claim (higher rank initialize lead to effectively lazier learning) also apply to to nonlinear feedforward networks trained on a real-world dataset. Although we trained the network on MNIST (instead of something like CIFAR), it is sufficient for the purpose of departing from the idealized setting. As originally mentioned in Limitations & Future Work, more comprehensive examination across wider range of architecture, including feedforward settings, is left for future work. We have also added a sentence in Simulation Results alerting interested readers to this appendix figure.

Tracking task-kernel alignment during training: We thank the reviewer for the insightful comment. In line with the reviewer’s suggestion of tracking impact on not just the final trained network but also during the learning process (and related comments by other reviewers), we have now added Appendix Figure 14 tracking the tangent kernel alignment as well as the alignment of the kernel and task throughout training. Figure 14A examines the kernel alignment (alignment of the current kernel to the initial one) in the idealized linear setting with 2D input in Fig. 2 in Atanasov et al. across different initialization schemes: random rank-1 initialization, random full-rank initialization and initially aligned low-rank initialization. We see that low-rank random initialization lead to greater movements, i.e. growing to be more dissimilar to the initial kernel (lower alignment value). Figure 14B looks at the task kernel alignment, $y^T K y / |y|^2 TrK$. Regardless of the initializations, the network kernel moves to be more aligned to the task over training. Low-rank initialization, without being specifically tuned to the task, is the latest to achieve alignment.

Figure 14C examines the kernel alignment (alignment of the current kernel to the initial one) in a non-idealized setting: sMNIST task. Again, random low-rank initialization leads to more movement over training, i.e. more dissimilar to the initial kernel. For Figure 14D, with multiple (10) output targets, we track the centered kernel alignment (CKA) of the NTK and class label, e.g. used in Baratin et al. (2021). We again saw that all initializations grow to be more aligned with the class label through training, but higher rank initialization gets to a higher alignment value faster and with less kernel rotation. We have already trained for nearly 20000 SGD iterations and we are not sure if training longer would eventually lead to similar final alignment values across initializations. We note that although achieving higher CKA over training can indicate greater degree of feature learning, a hallmark of the standard rich regime, this does not contradict our findings centered on effective learning regime that is defined based on the amount of change post-training (see Introduction). To incorporate these new findings in the main text, we have added sentences in Simulation Results to bring these Appendix Figures to the attention of an interested reader.

Please continue to the comment below

Unclear task selection criteria: We thank the reviewer for this valuable comment. The primary criteria for our task selection were neuroscience relevance, aligning with our core objectives and benchmark popularity, to ensure we weren't perceived as choosing idiosyncratic tasks that merely optimize our results — a rationale that also informed our decision to include sMNIST. That said, there are many tasks that fit the criteria due to the wide range of tasks that different species can solve, and the type of tasks we examine fall far short of the pool of all relevant tasks, so we listed the examination of wider range of tasks and architecture as limitation and future work in our initial submission. We remark that because of Theorem 1 and our intuition explained at the beginning of Discussion, we anticipate that our conclusion should hold across a wide range of settings despite having not exhaustively explored all relevant settings. In response to the reviewer’s suggestion, we have revised our Discussion section to reflect this and make propositions for related future work.

Despite that, the reviewer’s feedback made us realize that the strength of our empirical results would be improved if we can show at least one additional task (with neuroscience relevance) that is significantly different from the structure of the existing tasks. For that reason, we have added the sequence generation task inspired from Bellec et al. (2020) (Appendix Figure 15) and a sentence in Discussion alerting interested readers to that figure. We remark that exploration across a broader range of tasks is left for future work, as mentioned in our initial submission, and in particular, it would be interesting to explore the harder neuroscience tasks in Mod-Cog introduced by Khona et al. in the future (we have added the citation of Mod-Cog to that discussion in our updated Limitations & Future Directions).

Citing Canatar et al.: We thank the reviewer for catching this missed citation. We worked hard to be comprehensive in our citation, and yet, we still somehow let this highly relevant reference slip through our attention. We have thus added it to exactly where the reviewer suggested.

Tabula rasa: we thank the reviewer for bringing this to our attention. "Tabula rasa" means "blank slate”. However, Gaussian or Uniform random initialization, with non-vanishing variance, is not exactly a “blank state” in the strictest sense, as they can still exhibit inherent bias. We have thus replaced “tabula rasa” with “random” in the offending sentence.

Citations to Allen and MiCrONS: we thank the reviewer for catching these and we have now fixed them.

Sorting eigenspectrum plots and effective rank: We thank the reviewer for the comment. The eigenvalues were not sorted because we used numpy.linalg.eigvals(), which does not guarantee sorted eigenvalues and we didn’t ensure that once we were able to clearly observe the faster decay trend in the eigenspectrum of the experimentally-driven connectivities was still visible. As per the reviewer’s comment, we have now sorted the eigenvalues in the eigenspectrum plots.

As for the effective rank measure — which captures the proportion of eigenvalues on the order of the leading one — we chose it because it has been used before (e.g. Murray et al., ICLR’23) and would correspond to the area under the curve of the eigenspectrum plots scaled relative to the leading one, thereby informing us regarding the eigenspectrum decay. Low rank matrices would have fewer eigenvalues on the order of the leading one. That said, singular values are used more generally for capturing the effective rank. Hence, we have also added plots that measure effective rank using singular values (Appendix Figure 13) and a sentence in Simulation Results referring to that figure. These plots all support our main point here: these example experimentally-driven structures exhibit lower effective rank compared to null.

Combining Figure 2 and 3: We thank the reviewer for this suggestion and we have now combined them into one figure. This indeed gave us a lot more space to address the reviewers’ important points.

Improving Discussion Section: In response to the reviewer’s crucial comment regarding more concrete experimental explanation or prediction, we have now updated the manuscript, and the paragraph reads as: “We study the impact of effective weight rank on learning regime due to its potential implications in neuroscience. In particular, learning regimes are indicative of the amount of change undergone through learning, which bear consequences for metabolic costs and catastrophic forgetting (McCloskey & Cohen, 1989; Plaçais & Preat, 2013; Mery & Kawecki, 2005). These considerations are especially pertinent when we examine the various learning regimes evident in neural systems. For instance, during developmental phases, neural systems undergo resource-intensive, plasticity-driven transformations characterized by significant synaptic changes. These transformations are in contrast to the more subdued adjustments observed in mature neural circuits (Lohmann & Kessels, 2014). Based on this understanding, we predict that a circuit's alignment with specific tasks is likely established either through evolutionary processes or during these early developmental phases. As such, the specialization of a neural circuit (e.g., ventral vs dorsal (Bakhtiari et al., 2021)) likely stems from its engagement with tasks that share overlapping computational requirements, ensuring that each circuit is optimally configured for the tasks it is predisposed to perform. Conversely, circuits with high-rank structures, due to their inherent flexibility, are expected to be less specialized, potentially engaging in the learning of a broader range of tasks. With this in mind, our framework could be used as a tool to compare the connectivities across brain regions (and species) to predict their function and flexibility… ”

For deep learning, we note that while low-rank initialization is not common practice, low-rank adaptation and other update parametrizations have recently been popular in large model training schemes. See for example, Low-rank Adaptation (LoRA) in Hu et al. (2021). As such, we argue that the impact of low-rank structures on learning dynamics is quite relevant for current AI practices, and our study contributes to examining how having low-rank structures impact the effective learning regime. The methods and results we present here have the potential to be adapted and built upon to study rankedness and learning regime in several settings, and we are keen to see future work stemming from these ideas. We also note that our study has the tangential benefit of informing the deep learning community about potential applications of their models in other fields such as neuroscience. However, due to the significantly heavier balance of Section 4.1 in discussing neuroscience implication, we have removed “Deep Learning” in the section heading. We have updated Section 4.1 reflecting these points.

Upon re-reading Section 4.2, we agree with the reviewer that Section 4.2 is jammed with too many ideas. As per the reviewer’s suggestion, we have now broken 4.2 into smaller paragraphs for readability. We also thank the reviewer for pointing out the long sentence involving neuroscience implications and Tishby’s proposal. This sentence suffers the same problem as the entire section that the reviewer pointed out: too many ideas jammed together, which compromises clarity. We, thus, have removed the mention of Tishby’s proposal since it is creating confusion and distraction from our main point in the sentence, which is asking for deeper exploration into the neuroscience implications in the future.

Please continue to the comment below

We thank the reviewer for their positive feedback on the presentation and soundness. More importantly, we thank the reviewer for their concrete suggestions to further improve upon these axes as well as to better articulate our contribution. Below, we explain how we address the reviewer’s concerns.

More explanation/intuition of the main theoretical result: we thank the reviewer for this important comment. Our intuition is indeed in line with that of the reviewer’s. As per the reviewer’s suggestion, we have added a sentence explaining the intuition right after the Theorem statement. The sentence reads as “The intuition of Theorem 1 result is that, when two random vectors are drawn in high-dimensional spaces, corresponding to the low-rank initial network and the task, the probability of them being nearly orthogonal is very high; this then necessitates greater movement to eventually learn the task direction“. We have also updated the proof in the Appendix with more explanations for clarity.

In addition, we followed the reviewer’s suggestion and updated the main text with an outline of key proof steps; roughly these steps are: 1) The kernel alignment (Eq. 9) consists of three factors: $Tr(K^{(0)} K^{(f)})$, $\| K^{(0)} \|$ and $\| K^{(f)} \|$. We first derived the expression for each of them, which then gives us an expression for the kernel alignment. 2) Since we are interested in the expected alignment over tasks, we take the expectation of the expression found in step 1 over $\beta$ (i.e. the task definition). 3) We write the resulting expression in step 2 in terms of the singular values of the weights, and we show that its optimized when the singular values $s_i$ are distributed evenly across dimensions.

Better articulation of neuroscience and ML implications: this is another important comment from the reviewer, and other reviewers have also raised this concern. We apologize for having not made these points clear in our initial submission. In response to this, we have updated Section 4.1 in the manuscript and the paragraph reads as: “We study the impact of effective weight rank on learning regime due to its potential implications in neuroscience. In particular, learning regimes are indicative of the amount of change undergone through learning, which bear consequences for metabolic costs and catastrophic forgetting (McCloskey & Cohen, 1989; Plaçais & Preat, 2013; Mery & Kawecki, 2005). These considerations are especially pertinent when we examine the various learning regimes evident in neural systems. For instance, during developmental phases, neural systems undergo resource-intensive, plasticity-driven transformations characterized by significant synaptic changes. These transformations are in contrast to the more subdued adjustments observed in mature neural circuits (Lohmann & Kessels, 2014). Based on this understanding, we predict that a circuit's alignment with specific tasks is likely established either through evolutionary processes or during these early developmental phases. As such, the specialization of a neural circuit (e.g., ventral vs dorsal (Bakhtiari et al., 2021)) likely stems from its engagement with tasks that share overlapping computational requirements, ensuring that each circuit is optimally configured for the tasks it is predisposed to perform. Conversely, circuits with high-rank structures, due to their inherent flexibility, are expected to be less specialized, potentially engaging in the learning of a broader range of tasks. With this in mind, our framework could be used as a tool to compare the connectivities across brain regions (and species) to predict their function and flexibility…”

For deep learning, we note that while low-rank initialization is not common practice, low-rank adaptation and other update parametrizations have recently been popular in large model training schemes. See for example, Low-rank Adaptation (LoRA) in Hu et al. (2021). As such, we argue that the impact of low-rank structures on learning dynamics is quite relevant for current AI practices, and our study contributes to examining how having low-rank structures impact the effective learning regime. The methods and results we present here have the potential to be adapted and built upon to study rankedness and learning regime in several settings, and we are keen to see future work stemming from these ideas. We also note that our study has the tangential benefit of informing the deep learning community about potential applications of their models in other fields such as neuroscience. However, due to the significantly heavier balance of Section 4.1 in discussing neuroscience implication, we have removed “Deep Learning” in the section heading. We have updated Section 4.1 reflecting these points.

Please continue to the comment below

Thank you to the authors for the detailed responses and thorough revisions. I agree with the authors and other reviewers that they improve the paper, and I now recommend that the paper be accepted (and am updating my score accordingly).

2. “I am still not convinced of the concrete applications of this work...”

Although LoRA focuses on parameter updates rather than initializations, the exploration of how the rank of these updates influences learning regimes is a crucial area for future research. Our results provide an initial toolkit that can potentially be adapted for different contexts (e.g. with low-rank updates and structure enforced).

While different rank initializations, to our knowledge, have been sparsely examined (an example work can be found in Vodrahalli et al. (2022)), our study indicates that they should be examined more. Indeed, our result shows that the learning regime can be influenced by initialization rank (or other structure that has an impact on rank), and that the learning regime can significantly impact the nature of the solutions learned, specifically with respect to generalization (George et al., 2022). Consequently, it's conceivable that inductive biases in the form of ranked initialization could be employed to influence the generalization characteristics of neural networks, an aspect we have identified as a potential direction for future research. We thank the reviewer for pressing us on this point, which we recognize to be important for the ML community. As such, we add this discussion point to our discussion section.

For concrete neuroscience predictions, continuing the previously added/modified discussion paragraph mentioned in the previous response, “... With this in mind, our framework could be used as a tool to compare the connectivities across brain regions (and species) to predict their function and flexibility ”, we predict that the effective rank of a circuit, as determined from connectomic datasets, could indicate its level of functional specialization. Low-rank connectivity suggests a higher learning cost for diverse tasks, potentially rendering these circuits less adaptable for generalist roles. However, we acknowledge that the link between rankedness and specialization could be confounded by other factors — such as the underlying plasticity rule (also recognized by the reviewer) and the task rank that have been mentioned in Discussion, underscoring the need for deeper explorations in the future.

An additional concrete prediction of our theory would be that for neural circuits learning a new task, connectivity rank would influence how much change in task-dependent neural activity is expected before vs after training. Existing brain-machine interface experimental protocols already allow to explore this question, by measuring changes in activity during learning following readout perturbation, which was done by substituting a new mapping connecting neural activity to BCI output positions (i.e. re-initializing the readout weights). In previous work (Sadtler et al. (2014) and Golub et al. (2018)), the authors show that learning following a perturbation differs whether or not the perturbation was within neural activity manifold, or outside of it, necessitating some realignment to the task. One could apply this analysis to test the prediction by examining how the learning differs depending on the rank of the new readout. We have updated Discussion reflecting these points.

Overall, we note that the study of learning regimes is of great interest to the neuroscience community as it can inform ways by which neural circuits may exploit synaptic plasticity to perform credit assignment. A number of recent publications explore this topic, including Bordelon and Pehlevan (2023) that explores the impact of biologically plausible learning rule on learning dynamics and representation in different learning regimes, and Flesch et al. (2022) that compare experimental data to expected neural activity changes in distinct regimes. Overall, much like for the deep learning advantages outlined above, there is an interesting question about the characteristics of solutions that are discovered by different learning regimes, with some being more prone to spurious correlations and therefore, with distinct generalization properties (George et al., 2022) to be further examined in the future. It is likely that different brain regions employ different learning regime strategies and our contribution helps the community effort in building a toolbox to explore this question.

In summary, we hope we convinced the reviewer that the question of learning regime and connectivity is of great interest from different perspectives, and that our results help contribute to better understanding impacts on both biology and AI.

We thank the reviewer for their additional concrete and important comments to further strengthen our manuscript. Please note that we replaced the headings of subsections 4.1 & 4.2, in the previous manuscript version, with bolded text in the updated manuscript to yield extra space for incorporating the discussion points below.

1. “While I appreciate the authors' additional analysis in Figure 14, the results in Figure 14D appear to be in tension with the intuition underlying Theorem 1 would predict. Although the authors say that their findings are not contradictory (obviously they are not, assuming all are true!) it makes the findings of the paper harder to interpret for me. If the authors have a clear understanding of why high-rank initializations result in greater target alignment during learning, despite being overall in a "lazier" regime, an explanation would be very helpful.”

We apologize for not having articulated how this new result integrates into our existing one, and we hope our intuition below will better clarify and suggest how this new result in Figure 14 is actually in line with the main finding of Theorem 1. The intuition is in line with the intuition that the reviewer asked us to articulate in their previous comment, “in high-D spaces, when you draw two random vectors corresponding to the task and the low-rank initialization, they have a high chance of being nearly orthogonal to each other”. This necessitates more movement for the low-rank initialization to be aligned to the task; consequently, this need for greater movement would require more training (as seen in Figure 14B). If training is stopped early (e.g. due to resource limitations), as likely the case in Figure 14D, they would achieve less final alignment within the training window.

Regarding “If the authors have a clear understanding of why high-rank initializations result in greater target alignment during learning, despite being overall in a "lazier" regime”, we would like to remark that cannot be concluded from Figure 14D alone. First of all, these different initializations all lead to similar final alignment in the setting in Figure 14B. Second, we mentioned in our previous response to this reviewer that “... we are not sure if training longer would eventually lead to similar final alignment values across initializations…” Based on the upward trend at the end of Figure 14D, we predict that the alignment should continue to rise if we train longer. We note that for sMNIST, we concluded our training upon reaching 97% accuracy, a criteria informed by both published results and our computational resources. We also experimented with halving and doubling the training iterations and observed similar trends. Whether they would all converge to similar alignment values at the end of training under a wide range of settings ties to one of the proposed future directions (see below).

In response to the reviewer’s feedback, we have updated the caption of Figure 14 clarifying these points. Also, since this discussion is in line with our proposed future direction on deeper investigation of the link between rankedness, the learning regime and the consequent impact on representation (including kernel task alignment) and generalization properties of converged solutions under a wide range of settings, we have updated the corresponding sentence in Discussion alerting the reader to Appendix Figure 14.

Please continue to the comment below

Biological constraints enforced during training: we agree with the reviewer that enforcing constraints (e.g. Dale’s law) would be important for biological relevance. We initially didn’t do this to test our theoretical prediction, which assumes gradient-descent update without constraints. Following the reviewer’s comment, we have added a new figure (see Appendix Figure 17) with Dale’s law enforced for the EI experiments and observed similar trend as before: EI initialization led to greater changes post-training. We have also added sentences in Simulation Results alerting readers to this appendix figure.

Continuous RNN settings: we thank the reviewer for this important comment. We have now added Appendix Figure 18 to show our main conclusions — that higher rank random initializations lead to effectively lazier learning — also hold for the case when $\tau_m > dt$ and with longer sequence length. Specifically, we kept $\tau_m=100$ but changed $dt$ from $100$ to $20$, which increased the sequence length by a factor of 5. We have also added sentences in Simulation Results alerting readers to this appendix figure.

Balance Dale’s Law: we thank the reviewer for this question and we apologize that it wasn’t clear. We used 80% excitatory and indeed balanced the initialization according to standard practices. We have updated Setup & Simulation Details reflecting this point.

High variance due to variable eigenspectrum in low-rank network: However, our newly added Appendix Figure 19, which maintains a relatively stable leading eigenvalue by fixing the dominant eigenvalue across comparisons, still results in high variance for low-rank initializations. This suggests that there may be other factors contributing to this high variance, and it would be interesting to examine this in the future. This point is also reflected in the sentences added to Simulation Results, as a part of responses to the reviewer's earlier comment about the dynamical regime as a possible confounding factor.

Minor fixes: we thank the reviewer for catching these and we have fixed them.

We thank the reviewer for their insightful contextualization of this work, effective articulation of our intuition and providing constructive and executable suggestions to both improve the content and presentation of this work. Below, we explain how we address all of the reviewer’s concerns.

Confounding factor from dynamical regime: We thank the reviewer for this insightful comment/question. Indeed, it is plausible that the learning regime in RNNs could be influenced by the initial dynamical regime. With stable dynamics (non chaotic) , for instance, activity typically settles into steady states in absence of inputs over time steps, potentially needing more weight adjustments during training to effectively propagate information to the last time step. The study of learning dynamics and dynamical regime of RNNs is rich and involved and there are multiple ways to control for dynamics stability. A common method is through the leading weight eigenvalue of the connectivity matrix, which directly influences the top Lyapunov exponent dictating the rate of trajectory expansion.

As such, we now add details about a new control where we compare the leading weight eigenvalue (Appendix Figure 19), and we observed a consistent trend: higher-rank initializations lead to effectively lazier learning post-training. We note that a deeper exploration of the relationship between learning regime and various notions of the dynamical regime is a promising avenue for future work but falls outside the scope of the present paper, and we express this view in the discussion. We thank the reviewer for this insightful comment/question, and in response, we have updated the manuscript that added sentences in Simulation Results to reflect these points and alert interested readers to Appendix Figure 18.

Evidence that the brain uses low-rank connectivity: we thank the reviewer for pointing out this important and subtle point. In response to the reviewer’s important point, we have toned down our claim throughout the manuscript (Abstract, Introduction and Discussion in particular): there are evidence (e.g. Song et al.) suggestive of effective low-rank structures in the brain, rather than hard evidence. We also removed the citation to Mastrogiuseppe & Ostojic here, as the reviewer pointed out. We originally had a sentence in between the two citations but it got lost in the editing.

The extent to which the brain utilizes low-rank structures remains an open question, given the significant variation in neural circuit structures across regions and species. On top of that, as the reviewer pointed out, while local connectivity statistics can offer some predictive insight into the global low-rank structure, this relationship is not always immediately apparent (Shao & Ostojic, 2023). Our theoretical results contribute to this discourse by providing tools that link connectivity features, particularly effective rank, with learning (see Section 4.1). We have updated the manuscript reflecting this discussion in Related Works and Limitations & Future Directions.

Please continue to the comment below

We thank the reviewer for their accurate summary of our work. We also thank the reviewer for their insightful comments to enrich the study and concrete suggestions to improve the presentation. We explain below how we address the reviewer’s comments.

More fine-grained description of the impact of initialization schemes: In line with the reviewer’s suggestion of tracking impact on not just the final trained network but also during the learning process (and related comments by other reviewers), we have now added Appendix Figure 14 tracking the tangent kernel alignment as well as the alignment of the kernel and task throughout training. Figure 14A examines the kernel alignment (alignment of the current kernel to the initial one) in the idealized linear setting with 2D input in Fig. 2 in Atanasov et al. across different initialization schemes: random rank-1 initialization, random full-rank initialization and initially aligned low-rank initialization. We see that low-rank random initializations lead to greater movements, i.e. growing to be more dissimilar to the initial kernel (lower alignment value). Figure 14B looks at the task kernel alignment, $y^T K y / |y|^2 TrK$. Regardless of the initializations, the network kernel moves to be more aligned to the task over training. Low-rank initialization, without being specifically tuned to the task, is the latest to achieve alignment.

Figure 14C examines the kernel alignment (alignment of the current kernel to the initial one) in a non-idealized setting: sMNIST task. Again, random low-rank initialization leads to more movement over training, i.e. more dissimilar to the initial kernel. For Figure 14D, with multiple (10) output targets, we track the centered kernel alignment (CKA) of the NTK and class label, e.g. used in Baratin et al. (2021). We again saw that all initializations grow to be more aligned with the class label through training, but higher rank initialization gets to a higher alignment value faster and with less kernel rotation. We have already trained for nearly 20000 SGD iterations and we are not sure if training longer would eventually lead to similar final alignment values across initializations. We note that although achieving higher CKA over training can indicate greater degree of feature learning, a hallmark of the standard rich regime, this does not contradict our findings centered on effective learning regime that is defined based on the amount of change post-training (see Introduction).

In addition, to examine how the leading component changes relative to the rest, we track the evolution of the kernel effective rank, inspired by Baratin et al. (2021) in Appendix Figure 15. Here, the effective rank is the ratio of the trace to the dominant one, thereby capturing how much the leading eigenvalue evolves relative to the rest. We found that the kernel effective rank is getting closer to the target/task dimension over training.

Please note that we focused the additional analyses on the network kernel rather than individual layer weights, as the former is more reflective of the network’s function as a whole. To incorporate these new findings in the main text, we have added sentences in Simulation Results to bring these Appendix Figures to the attention of an interested reader.

Feedforward theory and RNN experiments: We agree with the reviewer’s comment. Note that the new addition of feedforward experiments in non-idealized setting (in response to Reviewer rK41) would be a step closer to what the reviewer suggests about closer theory and experiment setting.

Improving presentation: We thank the reviewer for this suggestion. We fixed several more verbose sentences in addition to the one the reviewer pointed out, including the very first sentence in Introduction. We have also fixed the typo that the reviewer pointed out for Figure 3B (now in Figure 2) in the original submission.

Please continue to the comment below

I would like to thank the authors for the detailed reply. I have also read the other reviewer comments. I think the presentation and clarity has been improved in the revised manuscript. I also appreciate that the authors have done a large and commendable amount of experiments including the additional ones during the rebuttal period, which have addressed some of my concerns.

However, I still am of the sentiment (similar to reviewer SwWy) that the paper does not clearly explain why low rank initializations lead to "richer" learning. For example, this finding appears to be dependent on the choice of the initial gain $g$ in the recurrent weight matrix (in Fig 8d in Appendix, for gain < 1 weights appears to change more with increasing rank), which suggests there is a more nuanced explanation than only the rank of the initial weight matrix. This also appears to differ to the feedforward theory which was true when $\sigma << 1$, which also sets the scale at initialization. Related (and minor): it would be also helpful to clarify the details of the initialization for the new feedforward experiments done in the revision.

Dear Reviewer RNHT,

As the author-reviewer discussion period nears its end, we would appreciate any feedback on whether our responses have addressed your concerns. If there are additional questions or points we can address, please let us know. We hope our recent updates have resolved the initial concerns and are committed to further improving our paper through this constructive dialogue.

Thank you, Authors