Comparison to works [1,2,3]:

We are aware of the three and believe that these are quite orthogonal to our setting.

• [1] is on learning linear combinations of polynomial single index models. This is squarely within the traditional “feature learning” regime where the dynamics depend sensitively on information/generative exponent.
• [2] considers training a student network on a target two-layer relu teacher network with relevant subspace of dimension $r$, in the heavily overparametrized regime where the student width is exponential in $r$ (the exponential dependence would thus also appear in the runtime). While their analysis focuses on the “local convergence” regime where the student already achieves small loss, the setup and phenomenology are entirely different from ours.
• [3] studies learning a single relu using an overparametrized student network, showing the convergence rate is exponentially slower than if the student itself were also a single relu. Again, our focus is not overparametrization. Furthermore, the target function we are learning is far richer than a single relu.

Our focus is on low-rank fine tuning, which is a novel setup not considered in any of the works mentioned, and their analyses unfortunately do not say anything in our setting. Therefore, we do not directly compare our work to these papers, but we are happy to add them to the Related Work section. To reiterate, our focus is on training a student on a teacher given by a low-rank perturbation of the initialization, and as we show, this is a genuinely new regime that operates strictly between kernel dynamics and full feature learning.

Assumptions on parameters:

First, we do not make any assumptions about $\lambda$ and are stating our results in terms of $\lambda_{max}/\lambda_{min}$. We are also not assuming anything about $\xi$ except implicitly that it is known to us, which can be relaxed by replacing the $\xi^2$ terms with $\xi \tilde \xi$. For simplicity of presentation we focus on known $\xi$. Furthermore, we demonstrate via simulations that our findings are not sensitive to the assumptions we make on $c$. We refer the reviewer to the comments left by reviewer DDTa, who noted, “However, I do not view this as a significant drawback for the authors, as the dynamics under this simplified setting already reveal interesting phenomena.” Given that our work is the first theoretical work on fine tuning beyond the linear regime, we respectfully suggest evaluating our work based on the fact that we have rigorously identified interesting and robust phenomena that are genuinely different from what was known in the two-layer literature.

Rank-1 setting:

Finally, we would like to comment about the rank-1 modeling. If the base network weights are 0 and rank > 1, this already subsumes the regime of multi-index models. While this regime has seen significant attention in recent years, the truth of the matter is that we still know fairly little. Single-index models (where the target network has 1 neuron) have certainly been widely studied and are quite well-understood, but the works that go beyond that (i.e. multi-index models) actually make far more restrictive assumptions than we do on the target function (e.g. Abbe et al, 2023).

Hence, our understanding for the multi-index setting is already a bottleneck to understanding the rank>1 setting, and we think the reviewer should keep this in mind when contextualizing our work. Regardless, even in the rank-1 setting, our modeling is more general than much of the multi-index model literature. In particular, our assumptions on the weights and the activations of the base layers are more general than many of the works that consider full training (e.g. Abbe et al, 2023 or [1]).

Regarding “LoRA application in practice,” we do not think this is a serious criticism. First, it is unclear why the reviewer is mentioning “weight LoRA within the attention layer.” as we focus on MLPs. The interaction between LoRA across layers is indeed an interesting open question; we are not able to solve every possible question in our setting, but there is ample evidence that even in our setting, there is a rich phenomenology to understand and we make serious strides toward doing so. The reviewer is correct that our setting is closely related to the broader literature on training dynamics for shallow networks. We would like to point out that papers in this direction remain an ongoing, highly active area of learning theory research which continues to be featured in NeurIPS/ICML/ICLR. Our work offers fresh insights into how the fine-tuning setting is genuinely different from the regime of learning from scratch. It is also one of the first works on fine-tuning to go beyond expressivity and kernel methods. Thus we believe the rank-1 setting deserves study, and criticism of the reviewer about rank>1 is not fair given that even the special case of 0 base network weights is not well understood.

As the discussion period is coming to an end, we wanted to check if you had the opportunity to read our rebuttal where we addressed your concerns. Your time is greatly appreciated and we are happy to answer any remaining questions.

Rank-1 setting and student-teacher setup

Related to Weakness 1, note that even in the original LoRA paper, it was observed that rank-1 perturbations actually perform quite well for downstream tasks for GPT-3. More generally, the fact that LoRA is empirically successful is in fact justification for why there doesn’t seem to be an issue with approximation power, making it reasonable to model the data as coming from a low-rank perturbation of a pre-trained model. For this reason, the student teacher setup is not a priori restrictive, though it is of course interesting to study richer settings, e.g. agnostic.

The reviewer’s objection might be the focus on rank-1 perturbations (since if we allow arbitrarily large ranks and second layer training, we can learn any target function). We have points related to this:

1. Note that for learning rank > 1 perturbations, this is already at least as hard as learning multi-index models from scratch, as setting the base model weights to 0 recovers that setting. There are still many open questions in that setting, despite many years of intense study. We certainly intend to attack this direction in future work, and investigating the case when the rank of the perturbations of teacher and student models differ is part of this open direction. Nevertheless, our work makes clear that even the rank 1 setting is highly nontrivial and offers useful conceptual takeaways.
2. The rank-1 setting already reveals interesting phenomena and meaningful separations between (see reviewer DDTa’s comments) training from scratch and fine tuning, as some of the quantities that dictate the complexity of training from scratch (such as the information or generative exponent) do not affect training complexity in the fine tuning regime.

Generalization error

Related to the reviewer’s comment about relating learning u to generalization error, in appendix D3 we show that once u is learned, there is a simple algorithm that does linear regression on the second layer of a 2-layer network that can learn the target function to \epsilon error over the dataset. Since this aspect of learning is more straightforward, we focus the discussion in the main text on learning the perturbation direction u as there is a simple algorithm for learning the target function afterwards (see Theorem 10, appendix D3). However, in the camera ready version, we can elaborate a bit more on this in the main text if this was not clear.

Organization

Related to the division of the sections, we believe this is a fair comment and we are looking to make the organization of the sections and presentation of the results more clear for the camera-ready version. We are planning to separate the intro and contributions section from the technical discussion.

Frozen $c$

Finally, the reviewer notes that freezing $c$ makes a significant difference and makes training faster. We are guessing that there might be some confusion about Figure 1 since the scales of the two figures are not the same, and freezing $c$ has a longer time scale. Once we adjust for this scaling, the difference turns out not to be so dramatic, and the later stages of training with a frozen $c$ is just longer by a factor of 2. For the camera ready version, we will make the figures a bit more clear by using the same scaling for the time-iteration axis (in Figure 1), as we have noticed this might be causing some confusion. Nevertheless, we claim the difference between the two settings is not in asymptotic order, but rather a constant factor. Since we are interested in the complexity of fine-tuning in the dimension parameter $d$, and number of neurons $k$, this difference is not significant for us.

As the discussion period is coming to an end, we wanted to check if you had the opportunity to read our rebuttal where we addressed your concerns. Your time is greatly appreciated and we are happy to answer any remaining questions.

We thank the reviewer for the extensive and detailed feedback. We will respond to all the points:

1. As the reviewer noted, our paper aims to shed light on the practical performance of fine tuning (specifically low-rank fine tuning), and some of the assumptions are to facilitate the analysis. Regardless, we reveal an interesting separation between fine-tuning and training from scratch.
2. We agree that there are works that can go beyond the information exponent (e.g. via sample repetition or smoothing), but they require modifications to the training algorithm such as repeating samples. This is the main reason for our emphasis on the information exponent, but we will add discussion about these recent works in our related works section. Note Our algorithm does not require reusing samples, so it already establishes a separation between fine tuning and learning single index models without additional modifications. Regardless, we can compare our complexity results to general SQ algorithms as the assumptions we have on the activation are quite general and encapsulate smooth, lipschitz activations. As the reviewer has noted, Damian et al, 2024 shows even such functions can have high generative exponents, thus leading to runtime scaling with a large polynomial in the dimension. Note that their results are for single index models and not for multi-index models as in Abbe et al, 2023, so we can only compare learning single index models to fine-tuning for results that go beyond CSQ. However, if they were to generalize their results to multi index functions with similar constructions, our separations should generalize as well. Overall, we will implement the suggestion of the reviewer to our related works section, and discussion of results.
3. For the camera ready version, we can aim to offer results similar to ones in Arous et al, as our analysis should be tight given the variance bounds are tight. The main limitation we have for showing tightness of analysis is finding tight bounds on the variance of the gradients (similar to the single index setting). Because our setting involves many more degrees of freedom than single index models, how the w_i’s and lambda_i relate to each other can affect the variance, which in turn affects the tightness of our analysis. Hence, we do not immediately provide a tightness guarantee as this would require more carefully bounding the variance and our goal is to investigate the dimension dependence, which is tight since the information theoretical limit is linear in the dimension.
4. Yes, it shouldn’t be too hard to prove lower bounds on the error when only the second layer is trained. This might be easy to show when $\xi = \Omega(\sqrt{k})$, and we can add this result for the camera ready submission.
5. We do not claim one can immediately obtain $d^{\Omega(k)}$ complexity for learning the target network in our setup, but we claim the moment tensors or the activation do not sensitively affect the complexity. We will reword this passage to make this clearer. We cite the existing works to provide exposition on the complexity of learning 2-layer networks, but it is an interesting direction to build up on our analysis and show examples of networks that are very hard ($d^{\Omega(k)}$) to learn from scratch and significantly easier to fine tune. We are optimistic that such a construction could exist in the angularly-separated + rank-1 setting.
6. While the reviewer points out an interesting connection, for us, the setting we study is ultimately not about expressivity because we are focused on training dynamics at the input layer, not the output layer. However, note we do not make assumptions about the randomness of W, which is a different setting that relies on the random-feature model.

Finally, we are aiming to make some revisions to the paper given the reviewer’s comments. Concretely,

1. We will add more discussion about the lower bounds (e.g. generative exponent) and algorithms that can go beyond the information exponent, and mention that our assumption on the activation holds in that setting.
2. Establish lower bounds on only training the second layer without learning the new direction of perturbation to also show separations between low-rank fine tuning and adapting the second layer.
3. Comment on the tightness of the finite sample analysis.

As the discussion period is coming to an end, we wanted to check if you had the opportunity to read our rebuttal where we addressed your concerns. Your time is greatly appreciated and we are happy to answer any remaining questions.

Novelty of results and proof

Related to the novelty of our proof technique and the strength of our results, while we do use some tools from existing work such as hermite analysis and the drift martingale decomposition (as do essentially all papers in this literature), there is a significant gap between the works that use these and our setting. To give the reviewer a sense for this, note that in the well-studied single index model setting (Arous et al, 2021) there is only one parameter (the overlap between ground truth and estimate) that controls the landscape, and the behavior of the population gradient is easily shown to be monotone. This is because they are training a single neuron to learn a function depending on a 1-dimensional projection. Their main contribution is the drift-martingale analysis which connects their population gradient analysis (which is quite simple with single index models) to the finite sample analysis (e.g. stochastic gradient descent, rather than population gradient descent). We readily use this tool, but it is entirely orthogonal to the main thrust of our work, which is in analyzing the population gradient.

There is a sizable jump in complexity when going from lower bounding the population gradient in the traditional single index setting versus in our fine-tuning setting. In our setting, the population gradient is much more complex and affected by many variables such as $\lambda, c, \xi$, angles between $w_i$, and the overlap $\langle u_t, u\rangle$. Because of having significantly more parameters that affect the optimization landscape, we need to make some assumptions, simply due to having more degrees of freedom in our model. As reviewer DDTa notes, these assumptions are not so limiting, and indeed compared to what is known in the multi-index setting, we actually make relatively weaker assumptions on the activations and structure of the target function. In particular, our contributions include but are not limited to:

• Reducing the analysis of training dynamics in MLPs to lower bounding a non-linearity term ($h(m)$) in the population gradient. Note that the idea generalizes to other settings, where one can follow our proof structure to even analyze dynamics that involve more than 1 variable (e.g. 2).
• Analyzing the structure of the highly non-linear, non-convex function $h(m)$, which is significantly more involved compared to previous work. This differs from previous analyses in many ways
  • The angularly separated setting involves various moment tensors whose effect can be arbitrary. We provide anti-concentration results (e.g. probabilistic lower bounds, appendix B) relating the inner products of these moment tensors to the magnitude of the population gradient.
• We work with a much more general class of activations than many of the work that focus on two-layer networks.
  • Other than the generic assumptions about non-pathology (e.g. linear polynomial growth, existence of weak derivative) we make a mild assumption about the decay of the hermite coefficients of the activation. This is much more general than much of the work that considers polynomial, smooth, or specifically ReLU activations.

Strength of conclusions

Now, focusing on our results, we show separations between fine tuning, training from scratch, and linearized methods. which we actually believe to be a fairly strong conclusion. A priori one would expect that fine-tuning is more tractable than pre-training in practice simply because it occurs in a linearized regime, as was suggested in prior work. We give the first proof that this is not the case, and that it operates in a genuinely new regime not previously studied in the learning theory literature. The fact that the dynamics are genuinely nonlinear but not dependent on the particulars of the activation function is quite surprising.

Rank-1 setting

Finally, we would like to comment about the rank-1 modeling. If the base network weights are 0 and rank > 1, this already subsumes the regime of multi-index models. While this regime has seen significant attention in recent years, the truth of the matter is that we still know fairly little. Single-index models (where the target network has 1 neuron) have certainly been widely studied and are quite well-understood, but the works that go beyond that (i.e. multi-index models) actually make far more restrictive assumptions than we do on the target function (e.g. Abbe et al, 2023).

Hence, our understanding for the multi-index setting is already a bottleneck to understanding the rank>1 setting, and we think the reviewer should keep this in mind when contextualizing our work. Regardless, even in the rank-1 setting, our modeling is more general than much of the multi-index model literature. In particular, our assumptions on the weights and the activations of the base layers are more general than many of the works that consider full training.

As the discussion period is coming to an end, we wanted to check if you had the opportunity to read our rebuttal where we addressed your concerns. Your time is greatly appreciated and we are happy to answer any remaining questions.