On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective

We thank the reviewer for their review. We are glad that you found the paper timely, precise, and clear. Below, we will respond to each of the points raised in the review.

• Magnitude of difference. In the paper, we focused on the conditions under which weak-to-strong generalization can happen. The exact magnitude of the improvement is dependent on the specific parameters of our model and can be smaller or larger, but the key takeaway is that a student can systematically improve upon a teacher by being regularized differently.

• Regularization in real scenarios. We use ridge regression as a simple, analyzable proxy for the much more complex set of implicit and explicit regularizers in large neural networks. In a real-world scenario, the choice of architecture and optimization algorithm can also play a similar role. For example, if the student has an architecture that is better suited for the underlying tasks, e.g., it has some invariances that also exists in the target, it can correct some mistakes made by the weak teacher. This is very similar to setting 1 that was studied in the paper. Also recall that in Burns et al., it is seen that early stopping of the student is crucial for weak-to-strong generalization. Our paper propose a potential explanation for this phenomenon. A better inductive bias of the optimization algorithm can also play a similar role.

• GPT-4 has access to some ground truth about natural language that remains out-of-reach for GPT-2? We hypothesize that a possible explanation of weak-to-strong generalization in GPT-4 and GPT-2 is that the tasks studied by Burns et al. [2024], e.g., the chess puzzles, have "leakage". The more extensive pre-training of GPT-4 (compared to GPT-2) on other tasks, combined with the fine-tuning on the data generated by GPT-2, can help in chess puzzles as well.

• Broader impact and take-aways of the paper. Weak-to-strong generalization is a new generalization paradigm and is often considered a mysterious phenomenon and the mechanisms that enable it are not entirely clear. Our paper aims to demystify it. In this paper, we provide a theoretical grounding for weak-to-strong generalization. We show that weak-to-strong generalization is not unique to neural networks; even very simple linear models can exhibit it. We believe that this is an interesting finding that enables more in depth theoretical study of the phenomenon in a more controlled setting. We move beyond empirical observation to provide a rigorous theoretical analysis, identifying concrete and analyzable mechanisms—such as the student compensating for the teacher’s under-regularization or better leveraging pre-trained knowledge—that explain how and why a more capable student model can learn effectively from a weaker teacher.

We thank the reviewer for their review. We are glad that you found the analysis interesting, well-written, and detailed. Below, we will respond to each of the points raised in the review.

• Is learning the easy component trivial? The easy component is a single-index function with unit information-exponent. It is known that Learning this function using SGD requires $\Theta(d)$ samples, compared to $\Omega(d)$ samples required for functions with an information-exponent greater than one. Thus, learning the easy component is still non-trivial (please see Ben-Arous et al. (2021) for more details).

• Weak-to-Strong generalization in setting 3. We thank the reviewer for this comment. We note that the magnitude of the rank-one components in the updated weight matrices is controlled by the step-size of the gradient update. Thus, the step-size should be large enough for weak-to-strong generalization to happen. Also note that in the paper, we show that with $n = \Theta(d)$ samples, the updated weight matrix of the student will become aligned to the easy and hard components. With weights aligned with the target, even a very few $o(d)$ number of samples is enough for generalization. We will clarify these points in the final version of the paper.

• Novelty of the contributions. We argue that we provide novel phenomenology that was not known in prior theoretical and empirical works on the subject. We show that weak-to-strong generalization is not unique to deep models and even very simple models can exhibit this phenomenon. We believe that this on its own is a solid contribution that enables the study of waek-to-strong generalization in a more controlled setting. In setting 1 and 2, we study the case of ridge regression in detail. The analysis of the pairs $(\lambda_s, \lambda_t)$ for which weak-to-strong generalization happens is completely new and complements the prior work that do not study explicit regularization. In particular, we believe that the phase transition happens between under- and over-parameterized students, is very surprising and was not expected a priori. The analysis of setting 3 is also generalization of the results of Ba et al. [2022]. We show that the student model can learn the easy direction without "forgetting" the hard direction learned during pre-training. In fact, from a mathematical perspective, Theorem 11 is of independent interest and is actually a necessary component to study feature learning beyond the one step setup studied by Ba et al. [2022], Moniri et al. [2024], Cui et al. [2024], etc.

We thank the reviewer for their insightful comments. We are glad that you found our paper timely, well motivated, and a nice addition to the literature. Below, we respond to the points raised in the review:

• More general regression settings. We agree with the reviewer that it is in principle possible to analyze this problem under more general assumptions on the input covariance. However, we note that the resulting expressions in this case will be very complex, and will not have a closed form; i.e., they will involve a fixed point equation. Thus, the results will become much harder to interpret. Moreover, we believe the study of the case of general covariance matrix is not central in our phenomenology of weak-to-strong generalization. In our view, studying the simplest possible settings that exhibit weak to strong generalization without additional clutter makes it easier to understand the fundamental mechanisms that enable it without unnecessary distractions.

• Results of section 3 and weak-to-strong generalization. We thank the reviewer for this comment. We completely agree that setting 1, in which the student and teacher are trained in a completely similar way, and the setting 3, where the weights of the student model are already aligned to some target components, are fundamentally different. But, we believe that setting 3 (similar to setting 1) actually does correspond to a form of weak-to-strong generalization. For instance, note that in Burns et al. [2024], the teacher model is GPT-2 and the student is GPT-4 and these model are indeed trained in a completely different way. GPT-4 already has learned many more capabilities in pretraining compared to GPT-2. Also note that in even in the setting of Burns et al. [2024], the strong student should also be only fine-tuned (and not fully trained) on the outputs of the weak teacher, otherwise, the pre-trained capabilities of the strong student beyond that of the capabilities of the weak teacher will be eventually forgotten.

• Introduction. We thank the reviewer for this comment. As suggested by the reviewer, we will adjust the introduction and explicitly mention our mathematical assumption alongside the motivation. Regarding the faithfulness of the motivation, we argue that the motivation of setting 3 is faithful to the mathematical assumption in the following way: the hard direction has been learned during the pre-training phase, and the network weights are already aligned to the hard direction, because it is a direction that is relevant for many different tasks and can be learned from them during pre-training. However, the easy direction is problem specific and could not be learned by the pre-training on other tasks. However, we will make sure that the actual mathematical assumption is also included in the introduction.

• Constants. The constants in Theorem 11 are not universal constants. The reviewer is right and $c_h$ does in indeed depend on the parameter $\tau$. We will adjust the notation in the final version to avoid this confusion.

• Choice of parameters in numerical experiments. We solely made this decision for the choice of $\sigma$ to enhance visualization. We will also include the plot for $\sigma = 1$ in the final version of the paper.

We thank the reviewer for their insightful comments. We are glad that you found our analysis of the weak-to-strong generalization in linear and nonlinear models interesting and well-structured. Below we respond to each of the comments raised in the review:

• Effect of noise level. We thank the reviewer for this constructive suggestion. To see the effect of the SNR, we first fix a teacher regularization parameter $\lambda_t$. As a corollary of Theorem 5, we know that in the high-SNR case where $\sigma_\epsilon^2 \leq \lambda_t/ \gamma_t$, weak to strong generalization cannot happen. However, the low-SNR case is completely different. In the low-SNR case ($\sigma_\epsilon^2 > \lambda_t/ \gamma_t$), if the student in under-parameterized, there is always a $\bar\lambda>0$ such that weak to strong generalization happens for $\lambda_s \in (0, \bar\lambda)$. Moreover, in the over-parameterized case $\gamma_s >1$, weak-to-strong generalization happens if we have

This means that in the overparameterized case, weak-to-strong does not happen in the ultra-low SNR regime; i.e., in the regime with

We will discuss these in detail in the final version of the paper.

• Discussion of double descent. In the final version, given the extra space, we will move the main parts of the discussion on double descent from the supplementary to the main body of the paper.

• Beyond synthetic datasets. We thank the reviewer for this suggestion. From a theoretical side, we note that the limiting traces that appear in the expressions of our paper exhibit universality; i.e., the actual distribution of the of entries of the matrices does not matter and they can be replaced with a Gaussian matrix with matching first and second order moments and still the limit will be the same. This fact can be proved by a standard Lindeberg exchange argument. Thus, our results in theory do hold much more broadly than the Gaussian case. Experimentally, similar kinds of behavior is also seen with real datasets like MNIST. The new NeurIPS rebuttal rules for this year prevents us from sending new plots, but we will include a detailed discussion of this universality and numerical experiments in the final version of the paper.

• Information is lost. In the case where $\gamma_s > 1$, we have $d > n_s$; i.e., the student model is over-parameterized. Information-theoretically, having access only to $n_s < d$ samples generated by the teacher, no algorithm can fully recover the $d$ dimensional vector $\hat\beta_t$. This is what we mean by "some information is lost". In the final version of the paper, we will clarify this point to avoid confusion in the final version of the paper.

• Limit of $\zeta \to 0$. When $\zeta \to 0$, setting 2 reduces to setting 1. This can be seen by comparing the limiting expression of $\mathcal{L}_s - \mathcal{L}_t$ in Theorem 4 and 8. In setting 1, we have $\mathcal{L}_s - \mathcal{L}_t \to \Delta$ and is setting 2, we have $\mathcal{L}_s - \mathcal{L}_t \to \Delta - \zeta^2 \Delta_g$ where $\Delta_g \geq 0$. In particular, having $\zeta > 0$ enables the student to outperform the teacher in setups (i.e., values of $\gamma_s,\gamma_t, \dots$) in which a student model with $\zeta =0$ cannot.