The Underlying Universal Statistical Structure of Natural Datasets

Dear Reviewer FARz,

We appreciate your thoughtful reading of our manuscript. Your main concern is the statement that natural data matrices behaving as Gaussian at sufficiently large dimensions is already known in the literature. We argue that this is not the case and we will attempt to explain our estimation of the distinction between what was known and our work below.

Weakness and Questions

We believe there is a subtle misunderstanding between us and the reviewer, which may come from the focus of our work. The large body of literature which concerns Gaussian universality concerns the learning curves of neural networks when taking gaussian data as a surrogate for real data, or the equivalence between neural networks in certain limits and Linear noisy networks applied to Gaussian data [1]. None of the works cited by the reviewer deal directly with the statistical structure of the data itself. This is a key difference in our opinion. We agree that the studies of generalized linear regression rely on the properties of the covariance matrix which we study, but they are an indirect measurement. Our work then serves two purposes: (1) it is an agnostic, and much stronger validation of the conjectures used in the literature to model the data (2) the r statistics (as well as the other local statistical measures) can offer a systematic path towards explaining the discrepancies between predictions for gaussian and real world data, since $r^{(n)}$ statistics explicitly measure deviations from Gaussianity (3) the local statistics inform you of the number of samples required to reach the RMT regime, showing a non-trivial universal behavior which is not discussed in the previous works (4) the fact that we find level repulsion could have implications for the localization of eigenvectors, which certainly does have an effect for learning tasks, particularly classification tasks. Note that some of the works you cited are already discussed in our text, particularly in App. F, but we should highlight these in the main text.

Regarding specific questions:

1. ”How does your analysis and measurements extend the empirically known Gaussian universality principle?” – as we replied in the above paragraph, please see our points (1,2) and (3).
2. ”Do the specific measurements you proposed shine light on any specific training phenomena? E.g., learning problems like linear regression or neural networks?” - We have not studied the direct effects of our results on models beyond App. F, but it is clear that this is the key question that will be studied in future work. The intuition is that by studying the higher level spacing statistics on real data we can systematically describe the deviations from Gaussianity. By focusing on the effects of these deviations for learning curves, we should be able to make better predictions for learning curves as described in (2) above. Additionally, see point (4) in the previous paragraph.

Methods

The reason for this definition of entropy is a common practice in quantum chaotic systems – we wish to interpret the data matrix as the equivalent of a physical hamiltonian whose complexity can be analyzed using its spectrum. Then, the Renyi entropy definition corresponds to a version of entanglement entropy, measuring the structural complexity of the data.

Theoretical claims

We use the “correlated” term simply due to the Toeplitz matrix assumption. The reason for this model is that it gives a simple “real-space” interpretation of the known spectrum. In that sense the source for anisotropy comes from correlations, offering some direction towards understanding why the power law appears at all.

Experimental design

We thank the reviewer for bringing this work to our attention. We will incorporate the proposed methods into our work. Still, we would argue that since the errors reported in Simon et al. with direct methods are at most ~5%, our results can be trusted, since we do not focus on the precise estimation of $\alpha$, but rather showing that it is generically between 0 and 1 for natural data. We believe our observations regarding convergence trends and spacings are also reasonable within this 5% error margin.

Relation to broader literature

”Line 78 "and many advances have been made by appealing to the RMT framework." Give examples?” - We relegated some of the relevant citations to the appendices due to space limitations, we will revise our text to better reflect the contributions in the main text.

Comments

Thank you for the comment regarding Eq.1, it will be fixed.

We hope that our replies are sufficient to re-evaluate the merit of our work and change your score of our paper. We welcome any further questions and comments

Dear Reviewer ZEdS,

Thank you for your careful review and positive appraisal of our work, tending towards acceptance. We’re glad you found the work interesting and well supported by empirical evidence. Below, we address the remaining issues you raised, as well as questions and comments.

Methods and Evaluation

” I don't see how this can be used in practice, since the exponent which is in principle unknown, is estimated itself at finite M. This looks more like an qualitative heuristic estimation.” - We agree that defining the asymptotic exponent itself at finite $M$ is suboptimal, and therefore its value may not be entirely trustable, but this is truly the best one can do on any real-world finite dataset. The important part is the universal convergence towards this exponent, and the interesting fact that this convergence occurs in two sharply separated phases, as seen in Fig 5 (middle).

Weaknesses and Questions

• ”…level-spacing statistics of the Gram matrices seems not to play (to the best of my knowledge) any role for theoretical analysis of ML.” - This is inaccurate, please see Appendix F for more details. Briefly, the role of the level spacing metric is to confirm the spectral density distribution, which is critical for analysis of even linear regression in the empirical limit (the resolvent is required to find the optimal network solution, which depends directly on the data covariance matrix’s eigenvalue distribution).
• ”…whether a 2d model would not be more appropriate to interpret the data?” - In order to capture the eigenvalue statistics a 1-d model is sufficient, and any higher dimensional model can work, provided that there is a component that decays as a power law. For instance if we were to take a fourier type model, as long as the spectrum had a component that decayed with the magnitude $|\vec{k}|$ as a power law, the same type of results would hold. Our goal was simply to provide the simplest possible model.
• “equation (9) to check, should it not be continuous at tau=1?” Yes it should, thank you for bringing this to our attention, we will correct the formula in the revised version.

We hope that our replies are sufficient to raise your confidence in our work and possibly raise your scoring of our paper. We welcome any further questions and comments.

Dear Reviewer kWZr,

Thank you for your careful review and positive appraisal of our work, deeming it acceptable at ICML. We’re glad you found the work interesting and well supported by empirical evidence. Below, we address the remaining issues you raised, as well as questions and comments.

Related work

1. We apologize for not bringing the connection between neural scaling laws, learning curves and the spectral properties of the data more to the forefront in this section. We provide a concrete example for this connection in Appendix F, but we will clarify these connections in the related work section as well in the revised version.
2. Thank you for pointing out these references, we will update the revised manuscript to include them both.

Weaknesses

1. Highlighting the connection to real world DNN training on CGD - We understand the reviewer’s suggestion and agree it has considerable merit, however, our intent in this paper was to bridge a long line of existing semi-empirical works (including ones that train networks on gaussian versions of real world data, see Ref [1,2] and similar works), and purely theoretical works that make the assumption of gaussian data. We will include Ref [1,2] and several other works that perform such experiments to better connect our results to the already large existing literature.

Comments

1. Line 153 right column, above eq 2: 'with' is misplaced. - Thank you for this comment, the typo will be fixed.

2. I find the index notation confusing, e.g. $X_{ia}\in \mathbb{R}^{d×M}$; usually $X_{ia}\in \mathbb{R}$ as in the matrix element. - We accept your comment, and will augment the final version accordingly.

Questions

1. Lines 434-436: “accomplishing learning tasks for which the spectrum is insufficient and eigenvector properties are needed, requires a different scale of data samples.” - would it not be fair to say that for most typical learning tasks eigenvector properties are needed? - We agree that there are many cases, such as classification tasks, the eigenvectors are crucial, and we study to what extent in a future work. However, for teacher-student settings, many other regression based settings, the eigenvectors are not necessarily important, and the spectrum is sufficient to predict performance and dynamics.

2. & 3. What information is NOT captured by the CGD? why does the scaling law behavior begins around i=10? - First, the CGD does not capture the top ~10 eigenvalues, since these eigenvalues contain the least “universal” properties of the data, and constitute the most particular aspects of every different dataset, also corresponding to the most informative eigenvectors. Therefore, the CGD, which describes the universal part of the spectrum, is shared between datasets and is only valid for the bulk of eigenvalues. Second, as stated in the main text, the CGD does not capture the eigenvector structure, as we do not assume a special basis for the CGD model. It would be interesting to explore which combinations of bases and spectra would render the CGD model a more complete description of particular datasets.

4. Lines 189-191: “For real-world data, we consistently find that $\alpha>0$” - is there a simple reason for this? - This is a very good question, which is currently still open. One possibility is that these values of power law spectra can emerge from hierarchical data structures with varying levels of rare features [3]. Answering this question could also shed light on certain aspects of neuroscience, see [4] for the emergence of power laws in human visual cortex activity data. We would also like to note that $\alpha>0$ seems to be prevalent for natural datasets, but not necessarily for other types of data. For instance, simulations of physical systems can give rise to negative exponents (see for instance [5] for the case of turbulence).

References:

[1] Refinetti et al., Neural networks trained with SGD learn distributions of increasing complexity (2023)

[2] Székely et al., Learning from higher-order correlations, efficiently: hypothesis tests, random features, and neural networks (2024)

[3] Cagnetta et al., How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model (2024)

[4] Gauthaman et al., Universal scale-free representations in human visual cortex (2024)

[5] Levi et al., The Universal Statistical Structure and Scaling Laws of Chaos and Turbulence (2023)

Dear Reviewer LgA8,
Thank you for your positive reading of our manuscript, deeming our paper acceptable.

Below, we address your comments/questions:

Essential References Not Discussed:

We believe the list offered is likely a mistake, since all of these references are explicitly cited in our related work section under “Random Matrix Theory”. Please let us know if we have missed something.

Questions

1. How robust is the observed universality across modalities other than vision (e.g., natural language or audio)? - The results appear to be quite universal, also across different modalities. In particular, we studied the RMT properties of language data, and found similar results. These results will be postponed to future work.
2. Could different neural architectures affect the emergence of universal spectral structures? - Since the essence of our work deals directly with the data itself, the notion of a neural architecture doesn’t explicitly enter into our analysis. However, it is clear (and known) that performing different nonlinear transformations on data, can dramatically change its RMT behavior, since it essentially changes the moments of the underlying data distribution, leading to different sample complexities and training dynamics.
3. What are the practical implications or potential limitations when applying these universal properties to realistic neural network training scenarios? - The practical implications could range from different sampling techniques based on PCA (and eigenvector sampling) and obtaining scaling laws for sampling required to reach ergodicity, to a more detailed study of the structure of data required for successful learning, moment by moment (second, third etc.).

We hope that our replies are sufficient to raise your confidence in our work and accept the paper. We welcome any further questions and comments.