The Underlying Scaling Laws and Universal Statistical Structure of Complex Datasets

Thank you for your thoughtful review and insightful questions regarding our work. We are pleased that you found the paper to be clearly presented and accessible to a wider audience, as this was one of our goals. We also agree that the onset of universality properties in real-world datasets should motivate some ubiquitous modeling assumptions.

Moving forward, we aim to address the limitations and questions you raised to strengthen the contributions of this research. We have incorporated several changes to our manuscript, in the hope that clarifying the issues raised here will lead you to consider admission.

1. Earlier works and context: Thank you for bringing these important works to our attention. We clearly should have situated our observations within this broader literature, rather than presenting them without the proper foundations. To address this weakness, we plan to refer directly to these works in the main text. Our work differs in focusing specifically on relating these heavy-tailed behaviors to random matrix theory. Moreover, we'd like to comment that the observation of power law data is valid for natural images, but does not seem to hold for dynamical system simulations, which often display multi-scaled behavior.

2. Effects beyond the bulk on the learning process: Thank you for raising an excellent point that we glossed over a key aspect regarding the role of outliers in feature learning and classification performance. Focusing only on the bulk spectrum is an oversimplification, as outliers often encode crucial information as you rightly pointed out. To address this weakness more comprehensively: We agree that for classification tasks, such as Gaussian mixtures, the class information resides in the outliers rather than the bulk. More generally, the relationship between spectral properties (bulk vs outliers), feature learning dynamics, and generalization is complex and not fully captured by only considering the bulk. While the bulk may characterize certain neural scaling properties under distribution shift, it does not explain how networks learn discriminative representations or classify datasets that differ primarily in their outliers/eigenvectors rather than bulk spectral shape. We should have acknowledged these limitations. Analyzing the interaction between bulk, outliers and learning dynamics/generalization remains an important open question, as recent works have started to demonstrate how outliers impact early gradient steps and network collapse. Truthfully, we have preliminary work investigating how datasets with identical bulk spectra but different outliers/eigenvectors and means may lead to different performance, among other configurations we find to be interesting. This future work should provide avenues for addressing some of these open questions. We have included discussion on these limitations and future work more carefully in the revised manuscript.

Questions:

1. Bulk spectrum and network analysis: Thank you for bringing up this point. We agree that we could have better justified how our results can be explicitly used for studying neural network dynamics and generalization. While the bulk universality provides a solvable model for aspects like neural scaling laws, as you note, the relationship between feature learning and generalization is complex, involving both the bulk and outliers. As a simple example of how the observations in this paper affect network analysis we have added Appendix D., where we study a simple teacher-student model trained on power law scaling data, and show that even in this simple example, our results are necessary for a full analysis. We appreciate you pushing us to better justify the relevance and limitations of our results.

2. Error universality: You make an excellent point that is directly relevant to our work but that we failed to sufficiently discuss - the relationship between bulk universality and recent developments in analyzing error universality from a RMT perspective. It is an oversight on our part not to connect our observation that the bulk spectrum follows RMT to the implications this has for analyzing generalization error using tools from RMT. As you note, for simple tasks like regression, knowing the bulk spectral density Characterizes the error through allowing computation via RMT theory, even for multi-modal distributions. To address this major gap, we will expand our discussion on universality to include a connection to error universality, incorporating the suggested works. We will explain how our finding of bulk universality motivated by RMT enables bridging to these works analyzing error from an RMT perspective, even allowing characterization of the error beyond just the population regime.

Notation: We understand that this notation may be non-standard, as it is borrowed from the physics literature, and we will be sure to clarify that $X_{ia}$ is simply the transpose of the standard Data matrix $X^T$.

Thank you for your thoughtful and constructive feedback. We are encouraged by the strengths you identified in our work, namely that the motivation and use of random matrix theory tools were found to be clear and compelling. The observation that our findings could bring useful attention to understanding representation in datasets from the perspective of classical random matrix ensembles is promising. To strengthen the technical & empirical rigor of the results, we address each of the points raised in order, hopefully leading to the acceptance of the paper.

1. This point could be construed in two different ways: First, the algorithm for finding $\alpha$ for a given dataset - this is done by computing the empirical Gram matrix, diagonalizing it, arranging the eigenvalues in decreasing order and fitting a power law to the bulk, we regret not describing this process explicitly in the main text, and will correct it in the revised paper. From a theoretical point of view, there is no a priori way to know the correct $\alpha$ for real data, but we observe empirically that natural image datasets display $\alpha=0.2-0.6$. Furthermore, it need not have been the case that a single correlation scale exists, as suggested by a single $\alpha$, and for instance chaotic fluid images exhibit multiple scalings. We add a discussion on the implication of power law scaling for NNs in a new appendix (D.) where we apply our results in the context of a simple learning problem.

2. Thank you for bringing up this important subject, as we are currently working on a paper which explores the relations between outliers, bulk and tail, both for eigenvalues and eigenvectors when used for training NNs. As shown by [1] and verified by our results, the outliers can also be described by a Gaussian model, but simply not the CGD that we suggest, as the outliers describe the most shared features in the data, and do not hold the local correlation structure of the bulk. They are crucial in classification tasks, and in particular their effect, as well as that of the different class mean values are the most important for linear classifiers. Our approach focused on the case where one studies improved performance by using more data, long after the outliers have been well described, and the only improvement gains must be squeezed out of the bulk, as described in our related work section. We agree that this discussion is important, and will be included in the revised manuscript.

3. We appreciate the feedback from the ML perspective regarding our RMT claims, and agree that the results can be better stated. To strengthen our claims, we have revised Appendix B. to include a detailed derivation for the spectral density, and compare it with numerical simulations and the bulk of real data.

Questions:

1. This is a very interesting question, and in practice we believe that the local spectral properties converging to GOE rather than a Poisson-GOE intermediate ensemble, coupled with the correct prediction for the bulk spectral density is sufficient proof. Nevertheless, one may wonder if there exist datasets which are sufficiently constrained to produce Poisson statistics, but as those datasets would require a strictly diagonal matrix with i.i.d entries we have not found such an example.

2. Thank you for this question, it is indeed possible and we are in progress of completing a paper which takes the synthetic model as a basis for network analysis and deals precisely with this question. We also consider the effects of outliers and higher/lower moments in this analysis. Regardless, we point to Ref.[1,2] in our manuscript for examples that require only power law scaling for generalization.

3. The number of outliers itself should be data dependent, and is not necessarily related directly to the number of classes, it may be related to the nearly constant values that are shared between all images.

4. This question is apt, and we found that different classes can have either very similar, or quite dissimilar spectra, with somewhat different exponents. We study the effects of these differences in an upcoming work.

5. $X_{ia}$ is the design (data) matrix composed of $M$ columns, each representing a sample, of dimension $d$, implying that Eq.1 is simply $1/M X^T X$, the standard definition of the empirical Gram matrix.

6. Yes.

7+10) We apologize for the confusion. The covariance used to generate samples is composed of the singular values of the Toeplitz matrix, and is therefore PSD, and so the formula (24) holds for it. This is now more clearly explained in the text.

8. The exponent is that of the modular hamiltonian, but in this instance is simply of $\Sigma_M$. This has been clarified in the revised text.

9. The linear part of equation 9, $2\tau$ is the universal part. The onset of this ramp is a known indicator of universality, as shown for real data in Fig.3.

10. We apologize for the mistake, it has been corrected in the revised manuscript.

Thank you for your important comments and questions, which have led us to consider carefully how to better put our results in the correct context for the ML community. We are glad that you agree that the providing better minimal theoretical models for data can lead to a better understanding of neural network behavior. We have made a number of changes which we hope address or clarify some of the issues you’ve raised, and which we hope would lead you to consider admission. We address these in more detail now, by the order in which they were raised.

1. Explaining how the universality in data affects the learning behavior of ML models: We thank you for highlighting to us the importance of relating our results to the learning behavior of ML models. We attempted to address this point by appealing to works done on neural scaling laws, in which the Gram matrix spectrum is a key part of the analysis, but is limited by either taking the population limit ($d/N=\gamma\to0$), or taking a phenomenological distribution for the eigenvalues. Our work verifies that the Gram matrix of real world datasets can faithfully be described by the RMT regime for a finite number of samples $N\sim d$, demonstrated by the local properties we studied. The implication is that given a power law exponent $\alpha$, we know not only the spectrum, but also its degeneracy, given by the spectral density $\rho_\Sigma$ which we derive in Appendix B. In order to further strengthen the connection between our results and the ML setup, we have added a new appendix - Appendix D., in which we give a concrete example of how one may use our results to obtain theoretical predictions. Namely, we solve a simple Teacher-Student model with power law correlated data, and show that the training dynamics as well as the convergence depend on the spectral density of the Gram matrices studied in the main text. We obtain analytical expressions for the training and generalization losses.

2. Eigenvector distribution and relation to learning in the GOE regime: Thank you for stressing the importance of the eigenvector behavior, which we agree is very important in the context of network learning. Indeed we did address the fact that the basis in which the covariance matrix is given is very important to the learning dynamics and convergence ("Additionally, the interplay between eigenvectors and eigenvalues in neural networks merits further exploration, as both components likely play crucial roles in the way neural networks process information"), we chose to focus only on the eigenvalue properties in this work. In ongoing work, which will be presented soon, we will discuss in detail the relationship between eigenvalues, eigenvectors, and network performance in multiple settings. To address the specific question "...whether such eigenvector statistics also fit into the universality perspective..." we would like to make two points. First, eigenvectors are important e.g. for classification problems since they encode information about the relative embedding of the objects with respect to a given fixed basis. Second, while the eigenvectors are fixed for a single realization of the covariance matrix, they are expected to be random and rather delocalized, when considering an ensemble of such realizations. This is not the case for the eigenvalue distribution, which admits its RMT prediction even using a single realization of the Gram matrix. It is therefore more subtle to assign to them universal characteristics. This implies, in particular, that the eigenvectors information used for training may not be valuable for generalization, depending on how delocalized the eigenvectors are, which also depends on the spectral density (see Bun et al. https://arxiv.org/abs/1502.06736).

3. Discussion regarding related works: We appreciate that our discussion of related works may be lacking. We have made revisions to the manuscript to better put the paper in the correct context within the literature. Firstly, we have added the suggested references, and in particular our citation of (Couillet and Liao 2022) is meant as a proxy for the many theorems therein, but we have included the appropriate source in the revised manuscript. Secondly, we are fully aware of many of the suggested works, and agree that they should be placed in the discussion on universality and its applications in machine learning. For example, the Gaussian equivalence property will certainly hold for the datasets we studied, as they have the correct symmetry property from the RMT perspective.