Weak Correlations as the Underlying Principle for Linearization of Gradient-Based Learning Systems

As much as I appreciate the importance of the research question and the mathematical elaborateness, the paper exhibits the following problems:

(1) Clarity / Presentation is immature: While the posed claims of the paper and the introduced methodological notions (NTK, random tensor asymptotic behaviour) are crystal clear, the presentation of logical reasoning and evidence to support the claims is rather hard to follow. Even after studying the supplementary material, the overall synthesis of the many involved steps, aspects and "directions" is incoherent/confusing. Most notably, "m(n)" is used to establish the main theorems, but the definition of "m(n)" is nowhere to be found (also not in the supplementary), except for a prosaic description of "the typical parameter of the linearization/correlation decay where m(n) → ∞". This is in contradiction to the seeming mathematical rigour and elaborateness.

(2) The paper presents no experiment(s). The work would strongly benefit from numerical examples to more clearly support the claims and illustrate the implications & significance of this work. This should at least include 2 manufactured toy examples, where (i) linearization is demonstrated in correspondence with weak correlations, as well as (ii) the opposite of that. Also the transition regime is interesting. Or even better, maybe even real-world applications can be found for demonstration

The key weakness that needs to be addressed before I can recommend this for publication is a clear presentation of what the method gives and doesn't give, in particular in comparison to existing methods (and a discussion of how this improves on the existing literature).

The paper also takes of tangents with notation and results that cannot be particularly relevant to the main goal. For instance, there are double factorials in the main, but they appear nowhere in the final result (and again, given that it is a 'soft' result, they cannot be of any importance for the whole result).

Also, some things are not correctly written (e.g. the NTK does not converge to the target function as written somewhere).

In my opinion this work fails on providing and/or communicating anything new on the topic.

1. Discussion on some very important related work is missing, which this work should have compare with.
2. Several statements are unsupported, definitions are missing, there are several inaccuracies and the paper is overall very hard to follow.
3. The mathematical notation is cumbersome and, for no apparent reason, completely different from many related papers.
4. Crucial points are relegated to the appendix, without which the main text is severely incomplete.

The main reference missing is “On the linearity of large non-linear models: when and why the tangent kernel is constant”, NeurIPS 2020 by Liu, Zhu, Belkin (https://arxiv.org/abs/2010.01092), but there are many other papers following this one that have studied the question of why neural networks can be linearised, also in relation to the model derivatives. This line of work is not discussed at all.

i) As a main contribution, the author list the case of “wide neural networks with two distinct activation functions”, but the only thing they say about this case in the main text is one sentence on page 9, relegating all about this claim in the appendix (we are not even told what “neural networks with two distinct activation functions” mean in the main text). ii) Section 2 is completely unmotivated and its relevance remains unclear until much later. For example, in the first three paragraphs of section 2.2 it’s unclear what is the goal and the challenges in reaching the goal. iii) The function \Epsilon is supposed to be a generic convex function, but there seems to be an (unstated) assumption that it depends on the difference between F and y, which is true for the square loss but not for many other commonly used loss functions. iv) (x,y) are called, respectively, label and images, but it should be the other way around. v) A “limiting parameter” in introduced on page 5 but never explained in the main text vi) Equation 9 only applies to gradient flow, not to gradient descent. After reading Theorem 3.1 it becomes clear that Equation 9 is a definition, but until then it just looks like a mistake. vii) Below Equation 12, Why can the parameter indices viewed as random variables? They are not random variables, and they are not drawn from a uniform distribution. Instead, all indices are summed over all the parameters. If they were a sample from a uniform distribution, there would be some noise. viii) No intuition is given here about the relevance of the quantity introduced in equation 12. ix) I don't understand Definition 3.2. "O" is supposed to be limiting order. What is "O" there? x) n0 not defined in equations (16) and (17) xi) There should not be a Delta in the second expression of equation (17) xii) Inequality 19 seems to be crucial for obtaining the results. However, the statement "nearly all realistic scalable systems satisfy" is not justified. xiii) What does "typical parameter of the linearization/correlation decay" mean? xiv) “This scenario is a little more complex but can be dealt with.” How is this scenario dealt with? It seems the reader here just needs to trust the authors without any explanation or justification. xv) “These systems can be interpreted as non-linear dynamical systems.” Any reference for this statement?

i) The Jacobian of the model is called “”derivative matrix and is transposed with respect to the Jacobian that everyone uses. ii) I have never seen a gradient with a subscript “T” to denote the transpose of the operation result. iii) In equation 12, the gradient with several subscripts and superscripts is just a (high order) partial derivative. Why re-inventing the notation?