Models of Heavy-Tailed Mechanistic Universality

We thank the reviewer for their positive assessment of our work and for taking the time to verify the correctness of our proofs. We appreciate that the reviewer finds the experiments to be sound and suitably validate our model class. We agree about the high writing density; there are many individual components and motivations in our discussion. In line with other reviewer comments, we are considering alterations to the second section to better emphasize relevant details and move away from some of the statistical physics (see comments to Reviewer RfHj for example). We will include all experimental details in the main text by adding a separate experiment section with the following:

5+1 phases of learning experimental details: For Figure 5, we train a MiniAlexNet for the CIFAR10 classification task with different batch sizes. This MiniAlexNet is a simplified version of AlexNet, which contains six layers: the first three are convolutional layers, followed by max-pooling layers, and the last three are fully connected layers. The histograms in Figure 5 are the histograms of the eigenvalues of $WW^T$ where $W$ is the trained weight matrix in the first fully-connected layer with input dimension $192 * 4 * 4$ and output dimension 1000. Here $W$ is initialized by the centered normal distribution with variance $\sqrt{1/\mathrm{fan_in}}$. For all the histograms in Figure 5, we trained the network using SGD with momentum, with a learning rate of 0.01 and a momentum parameter of 0.9 for 200 epochs. For each histogram, we repeat the experiments 3 times for the average. The red dot curves are the numerical simulations of the density function of the HTMP with different $\kappa$ and $\gamma=1000/(192 * 4 * 4)$.

To obtain NTK spectral densities, we consider larger neural networks that are trained to near-zero loss (all $>99.8\%$) on a subsampled dataset of 1000 entries through 200 epochs of a cosine annealing learning rate schedule with 200 epoch period, starting from a learning rate of 0.05 with a batch size of 64. Each model is comprised of the following number of parameters:

• resnet9 (4.8M parameters)
• resnet18 (11.1M parameters)
• vgg11 (9.2M parameters)
• vgg13 (9.4M parameters)
• lenet (62K parameters)
• logistic (30K parameters)
• densenet121 (7.0M parameters) The output layer of each model is altered from their ImageNet counterparts to classify with ten classes (for CIFAR-10, SVHN, and MNIST datasets).

We also recognize the relationship of the amount of training data to be a limitation. It is possible to complete the scaling law by including the power law dependence on the number of model features, although this differs from model size in a strict sense. Unfortunately, without a precise parameterization, it is difficult to establish such a law, but along with dependencies on individual model properties (e.g. depth, see comments to Reviewer Weri), we consider this of prominent interest in follow-up work.

We thank the reviewer for their positive assessment of our work, and for providing several references whose inclusion greatly enhances the working document! In particular:

• Thamm et al. do an excellent job of highlighting the phenomenon we are interested in and provides further evidence for our case (their Figure 4 shows a proportion of eigenvectors are non-uniform)
• We thank the reviewer for providing the neural scaling law paper by Bordelon et al. 2024, and we admit that this paper considered the power law assumption for the Fourier spectra of the kernels at initialization. This feature learning setup is similar to our power law assumption of the feature map and is the motivation of our neural scaling law section. They also consider a similar setup to the simultaneous training of weights and features in our Proposition 3.1. Very cool! We will modify the literature review in our neural scaling law section.

The other references seem to provide excellent examples of more fine-grained analyses with other precise models of training and architectures. We agree these provide excellent context and a better illustration of the state-of-the-art in the theory of feature learning.

In addition, we include the following references:

• (Pillaud-Vivien et al., 2018) also assume power law features.
• (Mezard & Montanari, 2009) for statistical mechanics of learning
• (Yang et al., 2023) "Test Accuracy vs. Generalization Gap" instead of Martin & Mahoney, 2021a for review papers on robust metrics

We recognize that we did not summarize the 5+1 phases of learning. We include the following at line 363:

In a sequence of papers, Martin & Mahoney observe 6 classes of empirical behaviors in trained weight matrices, comprising a smooth transition from a random-like Marchenko-Pastur to a heavy-tailed density, before experiencing rank collapse. Excluding rank collapse, the five primary phases are: (a) Random-Like: Pure noise, modeled by a Marchenko-Pastur density. (b) Bleeding-Out: Some spikes occurring outside the bulk of density. (c) Bulk+Spikes: Spikes are distinct and separate from the Marchenko-Pastur bulk. (d) Bulk-Decay: Tails extend so that the support of the density is no longer finite. (e) Heavy-Tailed: The tails become more heavy-tailed, exhibiting the behavior of a (possibly truncated) power law. The transition from (a) to (e) is also seen in (Thamm et al., 2022). This smooth transition between multiple phases is a primary motivation of this work. We find that this behavior is displayed by a combination of a nontrivial covariance matrix to capture the spikes, and the HTMP class with decreasing $\kappa$.

To answer your questions:

1. The depth and architectural effects of the structures are challenging to analyze due to the adopted general approach. These aspects can be studied empirically, necessitating further research. We admit that the model depth and other architectural details in this model will strongly affect the prior distribution of the feature matrix $\Phi$ and also the final feature matrix. In this work, we want to provide a new random matrix model (high-temperature MP law) which may resemble the spectral properties of the feature matrix $\Phi$. By tuning the parameters in HTMP, we can mimic the spectra of the feature matrices in different scenarios, which may be at initialization, may have been well trained, or may be associated with very complicated architectures. Although we do not have a clear picture of the relationship between the architectural details and spectra of the feature matrices, our random matrix model can potentially be viewed as an equivalent simplified model to study the feature matrices with very different architectural details.
2. Yes, we are assuming that features equilibrate more rapidly than the loss. In this way, we are examining trajectories at the end of training (the kernel learning regime in (Fort et al., 2020) "Deep learning vs. kernel learning") once the features are mostly trained. We elaborate more on this in the current version.
3. Interesting! So the input data is uniform, but this example still seems to assume a power law Fourier spectrum in the target function. Unless we are missing something, this still assumes heavy tails in the data (in the labels in this case). However, we have found that (Liao & Mahoney, 2021) "Hessian Eigenspectra" also shows how nonlinearities can influence the spectrum, although they do not prove scaling laws from this. In this case, conditions for the nonlinearity to exhibit heavy tails are still unclear, so we consider this a case of examining individual models.

We thank the reviewer for their careful examination of our work and for the thoughtful feedback. We appreciate that they find our framework to be ambitious, and understand the concern about empirical verification of individual claims within our theory. One of our fundamental assumptions---that the eigenvectors are not uniformly distributed---was verified by a reference provided by Reviewer Weri; (Thamm et al., 2022), and we will include the reference to this. However, our primary goal in this paper is the development of a single theoretical model designed explicitly to display observed asymptotic spectral tail behavior (Theorem 4.3). Our discussion is used to motivate the construction of a variational family with the desired characteristics. We study this model to compare its overall predicted tail behavior against empirical phenomena. We cannot claim that this model represents specific architectures or matches empirical phenomena identically; our analysis cannot replace other fantastic work that is being conducted in this direction. To help focus our claims, we add the following to line 53:

Our central objective is to identify a plausible random matrix class that exhibits inverse Gamma law spectral behavior, and the smooth transition from Marchenko-Pastur to heavier-tailed densities observed in Martin & Mahoney (2021) and Thamm et al. (2022).

and replacing 63--65:

• construct a parametric family of spectral densities which includes the Marchenko-Pastur law, but allows for heavy-tailed spectral tail behavior in line with empirical observations (Figures 1 & 5, Appendix I)

We replace lines 125--135 with the following:

However, PIPO also fails as a standalone learning theory, as it does not account for implicit model biases. If data alone influences tail behavior, models trained on the same dataset should exhibit similar power laws (Section 4.1), which contradicts empirical findings (Yang et al., 2023). At present, theoreticians analytically examine the interactions of individual models in the presence of heavy-tailed data (Maloney et al., 2022), but such analyses are intractable at scale. Originally conjectured by Martin & Mahoney (2021), here, we look for a third alternative: a universal, model-agnostic mechanism that can give rise to different heavy-tailed spectral behaviors from the same dataset. We construct a sequence of hypotheses that lead to such a mechanism, which we refer to as "eigenvector entropy". Matrix models in the literature typically have eigenvectors that are Haar-uniformly distributed (maximum entropy) (Anderson et al., 2010); this is also represented by delocalization (Bloemendal et al. 2014). Breaking this property, reducing entropy, our framework provides a family of variational approximations (HTMP) with the right qualitative behavior: arbitrary power law spectra, and inverse Gamma laws that can arise from model design alone.

Regarding more specific comments:

• $M$ should be positive-definite, not $\Phi$ (typo)
• "Near ballistic" has been changed to "power law exponents $\alpha < 3"
• Donsker-Varadhan now cited
• Hanin & Nica discuss heavy-tailed log-normals; not power laws, but can appear indistinguishable, see (Clauset et al., 2007). We now make this clear.
• "Free eigenvectors" replaced with "eigenvectors $v_1,...,v_N$ with $v_1,...,v_d$ Haar-uniform and $v_{d+1},...,v_N$ fixed.
• The approximation (6) is proposed in accordance with patterned matrix models whose spectral densities we calculate in Appendix D. It is not exact, but designed to replicate qualitative phenomena.
• See response to Reviewer Weri regarding 5+1 phases.
• For further citations to cover claims made about the wider literature, see the response to Reviewer Weri.
• Section 2.5 of (Anderson et al. 2010. An Introduction to Random Matrices.) showed that eigenvectors of GOE/GUE are Haar uniformly distributed; we now include this.
• For general Wigner or sample covariance matrices, (Bloemendal et al. 2014. Isotropic local laws for sample covariance and generalized Wigner matrices) used local law results to prove the delocalization for all eigenvectors, which means all the entries of the normalized eigenvectors are $1/\sqrt{N}$ order. For more details, we refer to local law lecture note by (Benaych-Georges & Knowles 2016).

To answer your questions:

1. Yes, but this assumption is not needed if $M = \Phi \Phi^\top + \frac{\gamma}{2\tau} I$, so we take this in the updated version.
2. Thank you for the suggestion; this now reads "For more general losses..."
3. Linear regression with equal numbers of parameters and datapoints is one example, since $\min_\Theta L = 0$.
4. The Appendix includes $L^2$ regularization, but otherwise the two are equivalent. We now add $L^2$ regularization to the activation matrices section.

Please let us know if you have remaining questions about claims; space constraints prevent us from addressing further comments, but we are happy to provide evidence in the discussion period.