Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networks

We thank the reviewer for their constructive and detailed feedback on the manuscript.

First, we address the reviewer's question regarding the optimization reasons behind an artificial network seeking robust solutions. Our analysis is general—we study the entire solution manifold and derive broad statements about neural computations and their representations without relying on any specific learning or optimization algorithm. An investigation into why and when an algorithm converges to a particular region of the solution manifold is beyond the scope of our study. However, as noted by the reviewer, extensive work in the machine learning literature has explored this question, albeit without focusing on their relation to neural representations. We have now expanded our Discussion to address this important aspect in more detail, including relevant references concerning implicit regularisation (e.g., Smith et al., 2021, as suggested) and neural collapse (e.g., Zhu et al., 2021).

Next, we address the reviewer's second question regarding the visualizations in Figure 2. Our aim was to follow well-established standard representations found in the literature (e.g., the Wikipedia article on “Kernel (linear algebra)”). Nevertheless, we recognize the need for additional clarity. We have revised the figure caption and expanded the accompanying explanations to explicitly clarify that: (1) the orange areas represent the general vector space in which representations reside, (2) open black circles denote the image of linear transforms, with the surrounding area corresponding to their kernel, and (3) filled black circles indicate the subspaces occupied by the input, hidden, and output representations of the training data, respectively. We welcome any further suggestions for improving the clarity of our visualizations.

We appreciate the reviewer's suggestion for additional experiments. Could the reviewer specify which experiments they believe would most effectively strengthen our paper’s message? We are open to incorporating further empirical evaluations, even if complex, to reinforce the practical implications of our theoretical findings.

Furthermore, as suggested, we have tempered our claims regarding noise as the cause of representational alignment between artificial and biological systems, instead framing them as hypotheses that open up an interesting field of investigation. In particular, it remains unclear if, why, and when brains may operate in these regimes. However, we emphasise that our results suggest that assuming they do without empirical validation could lead to misleading comparisons and conclusions about decodability and representational similarity.

We have also made the following revisions based on the reviewer’s comments:

• Corrected the typo in line 152.
• Clarified in the caption of Figure 3c that the elephant is deliberately engineered.
• Expanded the caption of Figure 5 to explain that "duplicate" refers to networks extended by duplicating hidden neurons, a transformation that preserves the input-output mapping.

We once again thank the reviewer for their time and valuable insights. We remain open to further suggestions, particularly regarding simulation studies that could further illustrate the scope and significance of our contributions.

We would like to sincerely thank the reviewer for their time and detailed feedback. Below, we address the key points raised in the review.

The reviewer asks whether our analysis provides insights beyond the conclusion that internal representations cannot be interpreted in isolation. First, we emphasize that this is not widely understood, as many subfields of neuroscience, machine learning and their intersections do center on analysis of representations in isolation. For example, as we outline in great detail in Section 4, a significant body of research in computational neuroscience assumes that internal representations between systems (models and/or brains) can be meaningfully decoded and compared. Our work analytically demonstrates that this assumption does not necessarily hold—in particular, we identify in which regimes representations are arbitrary with respect to the task, and instead dependent on other parameters like parameter initialization.

However, this is only one part of our findings. We further identify specific regimes on the solution manifold where representational similarities are not arbitrary. We derive precise analytical conditions under which internal representations in fact are informative and comparable without requiring knowledge of other parts of the network; in particular, when they are task-specific. Additionally, we show that these task-specific stable representations coincide with regions of the solution manifold that are robust to noise, linking identifiable representational structure to a desirable computational property. This goes beyond the claim that “understanding internal representation is a hard problem” because we establish precisely when it is hard.

It is not clear to us why the reviewer dismisses these results as “intuitive”, “preliminary”, and “not novel”. To ensure a constructive discussion, we kindly ask the reviewer to provide references to prior work that establishes the same conclusions as we do in the manuscript.

Additionally, to clarify our contributions, we have revised the manuscript by adding an explicit contributions section that is detailed in the response to Reviewer rKc5.

Minor:

We would like to thank the reviewer for catching the typo in Assumption 2.2. Indeed, the greater-than-or-equal sign should have been a less-than-or-equal sign!

We appreciate the reviewer’s explicit feedback on Figure 2. Our visualizations follow well-established standards found in mathematical visualizations (e.g., the Wikipedia article on “Kernel (linear algebra)”). However, we are happy to improve and expand on the explanations of the image and kernel of a matrix in the main text. Further, we have revised the figure caption and expanded the accompanying explanations to explicitly clarify that: (1) the orange areas represent the general vector space in which representations reside, (2) open black circles denote the image of matrices, with the surrounding area corresponding to their kernel, and (3) filled black circles indicate the subspaces occupied by the input, hidden, and output representations of the training data, respectively. We welcome any further suggestions for improving the clarity of our visualizations.

Further, the reviewer notes that they did not carefully review the supplementary material due to its mathematical density, but still conclude that the results “might be sound” as they do not disagree with the paper’s results and conclusion. This aligns precisely with ICML guidelines, which state that reviewers are encouraged (but not required) to consult supplementary material and that key claims should be understandable from the main text. Furthermore, rigorous mathematical derivations are essential for theoretical contributions in machine learning, just as extensive simulations are standard for empirical work. Appendices that extend over many pages are common in ICML theory papers, and theory of machine learning and applications to (neuro)science are subject areas in ICML’s yearly call for papers. Throughout our 8-page supplement, we have made substantial efforts to fully provide all our assumptions, proofs and derivations to ensure our results are fully reproducible, testable, and clearly presented.

Finally, we sincerely thank the reviewer again for their time and effort. We hope that, in light of our clarifications, the significance of our contributions and the value of our thorough theoretical analysis (including the detailed appendix) become evident. We remain open to any further suggestions or points for clarification and are hopeful that the reviewer may reconsider their assessment accordingly.

We thank the reviewer for their thorough evaluation of our work, and for providing explicit feedback!

First, we address the reviewer’s concern about the novelty of our analysis. To clarify this, we have added a dedicated "Contributions" section to the revised manuscript and reproduce it here in detail. As stated in the manuscript, deep linear networks perform “…multistage computations that give rise to hidden-layer representations.” Their overparameterization through depth, rather than width as in linear regression, introduces nontrivial representational degeneracies not present in linear regression. Our work provides a complete analytical characterization of these degeneracies and their implications for decodability and comparability of neural representations. Further, we study the consequences of our analysis on use cases in neuroscience, where neural networks are a commonly employed model of learning, in Section 4.

In particular:

• Definition 3.1 directly follows from Laurent and Brecht (2018).
• Theorem 3.3 resembles least-squares linear regression, but its derivation in the context of two-layer linear networks requires extra care (e.g., does not hold if the network is bottlenecked).
• The constraint satisfaction problem in Definition 3.4 has been studied in Saxe et al. (2019), but only under a set of strong assumptions ($\Sigma_{xx} = I$, $N_i = N_o$, and that $\Sigma_{yx}$ has full rank) as stated in the manuscript.
• In contrast, Theorem 3.5 imposes no additional assumptions about the task's statistics or network structure (beyond the network not being bottlenecked).
• Definition 3.6 stems from our analysis of parameter-noise robustness (Section 5), but can be related to notions from implicit regularisation (e.g., Smith et al., 2021) and neural collapse (e.g., Zhu et al., 2021).
• Accordingly, Theorem 3.7, which shows the parameterization of minimum representation-norm solutions, is novel to the best of our knowledge.
• Theorems 3.8 and Corollaries 3.9–3.11 are novel and central to our analysis, analytically revealing the degrees of freedom within neural representations and defining when representational similarities are fixed and can be used for functional comparison.
• For theorems and corollaries regarding robustness to noise in Section 5, similar results exist for linear regression (e.g., minimal norm confers noise-robustness) but again require careful treatment in the multi-layer setting (e.g., cross-correlation terms in the case of parameter noise).

In summary, to our knowledge, this is the first work to make an analytical and exact connection between different regions of the solution manifold of deep linear networks and the identifiability and comparability of neural representations, as well as their correspondence to optimality in noise robustness. If the reviewer is aware of any further prior work that has similar results, we would be delighted to include it and provide appropriate reference.

Minor:

We would like to thank the reviewer for catching the typo in Assumption 2.2. Indeed, the greater-than-or-equal sign should have been a less-than-or-equal sign!

Regarding terminology, we made an effort to be precise while avoiding overly lengthy terms like "minimum-readout-and-representation-norm" . We recognize that "minimum-representation-norm" is only partially descriptive, as it omits the fact that we are also minimizing the norm of the readout weights. While minimising the norm of the readout-weights has no influence on the hidden-layer representation, we chose the constraint satisfaction problems in Definitions 3.2, 3.4 and 3.6, such that they align with the analytical results on noise robustness in Section 5. We believe our terminology is sufficiently descriptive but would be grateful for any suggestions the reviewer may have for a more concise naming convention.

Regarding Corollary 5.4, the parameter noise is scaled by the norm of the inputs and the size of the output layer (see the first and second summands in Equation 22). By inversely scaling the variance of the noise, we ensure the equation is factored consistently, independent of these measures. Without this adjustment, the solution would be identical up to a fixed scaling factor. Thus, this scaling is simply a matter of convenience. We have added a corresponding comment to the manuscript.

Regarding the scaling in Figure 5E, the initial phase of the sigmoidal curve corresponds to test error (near-zero noise), while the ceiling aligns with random guessing (high noise). We lack analytical insights into the shape of this curve due to the non-linear setting. However, we would be happy to include additional simulations if the reviewer has a specific question in mind.

Again, we would like to thank the reviewer for their time and effort, and we remain open to any further suggestions or points for clarification.

We thank the reviewer for their thorough evaluation of the manuscript and supplementary material, and sincerely appreciate their effort and positive feedback! We would like to kindly ask the reviewer, if time permits, to elaborate on the specific strengths and significance of the work to help the Area Chair better understand the basis of their positive assessment.

Regarding the $\stackrel{!}{=}$ notation in the supplementary material, it was used to indicate that an equality is not derived but rather an assumed condition from which a separate conclusion is derived. However, since this notation is not widely used, we have replaced it with explicit language such as “such that”, “with the requirement that”, or “we would like to show” for clarity.

Please do not hesitate to let us know if there are any further points or suggestions to address.