Quantifying the Gain in Weak-to-Strong Generalization

Thank you so much for reading our paper, and for your review and comments. We are really glad that you like our work!

With regards to the points you bring up:

It's unclear why the authors chose the regression setting, when the original work and likely scenario of use is classification setting. I think this difference from the original work should be made clearer.

Regression with the least squares loss, while arguably being one of the most basic tasks in statistics and machine learning, also turns out to be a setting where we can extract the essence of the weak-to-strong generalization phenomenon in an intuitive, geometric manner — as we show, the phenomenon in this setting effectively boils down to the Pythagorean theorem for projection onto convex sets (a well-studied and standard fact from convex analysis). We do believe that a similar intuition and explanation (in terms of “projections” onto “convex” sets) may also apply to the classification setting (with say the binary cross-entropy loss). It is also worth mentioning that the theory in several previous studies [1,2] on out-of-distribution finetuning is also focused on regression. Nevertheless, we will emphasize more on the difference in our setting (regression vs classification) in the next revision.

[1] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine tuning can distort pretrained features and underperform out-of-distribution.

[2] Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts.

The empirical confirmation in realistic data is limited to one setting, which is an unfamiliar dataset to me and likely most of the LLM and NLP community. It would be beneficial to demonstrate the phenomena across more settings, especially more standard ones.

The molecular prediction task seemed natural for our setting and also stood out to be a standardized task [1] in computational chemistry for regression over sequential data, with a tractable pre-training dataset, and a good variety of fine tuning tasks to demonstrate the applicability of our results. Additionally, the community had also previously developed specialized transformer architectures for these tasks [2]. That is why we decided to go with this dataset.

As can also be seen, for example, in Burns et al., 23, a major bulk of NLP/LLM tasks seem to be tailored to classification/generation; nevertheless, we would be happy to include results on a suitable NLP regression task (suggestions welcome!) in the next version.

[1] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning

[2] Benedek Fabian, Thomas Edlich, Héléna Gaspar, Marwin Segler, Joshua Meyers, Marco Fiscato, and Mohamed Ahmed. Molecular representation learning with language models and domain-relevant auxiliary tasks.

Giving some intuition as to why the theoretical result holds from the proof would be beneficial. In particular, pointing out that the inequality achieved relies on the distance measure not being a metric, otherwise the triangle inequality for metrics would force the inequality to be the other way round, if my understanding is correct?

You are indeed right that if the distance measure (average squared distance) was a metric, we would get the reversed inequality. An intuitive reason for the inequality being the in other direction is the textbook cosine law on the sides of a triangle: $a^2 = b^2 + c^2 - 2bc \cos(\theta)$, where $\theta$ is the angle between sides $b$ and $c$. Convexity guarantees that this angle is obtuse, which makes the cosine negative. Hence, $c^2 \le a^2 - b^2$. Here, $a^2$ is the weak model loss, $b^2$ is the weak-to-strong misfit, and $c^2$ is the strong model loss.

It's likely that in the setting from the original paper, the weak labelling is within the function class of fine-tuned strong models (where fine-tuning is over all parameters), at least on a naïve interpretation. It would be beneficial to discuss whether the authors think their theoretical results is the reason why WTSG happens in this setting as well (given the assumptions of their theory break), or whether they think WTSG happens in that scenario for a different reason.

This is a great question, as in such a case, our theory (which works in the setting of regression) would indicate that no weak-to-strong generalization should be exhibited. This is also what the authors of Burns et al, 23 refer to as perfect student-supervisor agreement/imitation. To mitigate this (Section 4.3.2 in Burns et al., 23), notice that the authors of that paper suggest incorporating an auxiliary "confidence" loss in their finetuning optimization objective to enable weak-to-strong generalization—this serves as a regularizer, and avoids the student (strong) model to naively overfit the weak labels. Note however, that our theory heavily uses the fact that the minimizer of the (unregularized) mean squared error is in fact the projection of the weak model onto the convex set—this is no longer true with a regularization term. However, there is a natural duality between regularized objectives and constrained optimization problems. An interesting future direction then would be to characterize the regularization terms that constrain the space of optimization to still contain the target function, but exclude the weak model itself, so as to provably mitigate student-supervisor imitation.

---

Please let us know if we can help answer any other questions!

Thank you for your comment. We do believe that the performance gains via WTSG in classification settings ought to also be characterized to some extent by the misfit between weak and strong models; however, other terms also seem necessary (capturing, among other factors, the case where the strong model can represent the weak model). Indeed, the gains in accuracy with naive finetuning without auxiliary losses in the classification settings considered by Burns et al. 23 already suggest that such terms should be necessary, and that the phenomenon of accuracy gain via WTSG in classification is more nuanced than in regression (e.g., not only the quantity, but even the quality of misfit can matter here). Thus, our theory does not directly explain the gains in performance in these cases, and a different analysis seems to be required.

As for initial thoughts about a theoretical explanation for gains in classification: the phenomenon in classification that we want to capture is the following: the weak model is presenting the strong model with labeled data; however, the labels are really only "pseudolabels", and not true labels. The strong model, while learning from the weak model, is nevertheless still rectifying the wrong "pseudolabels". The self-training framework of [1], where the student and teacher are the same model, does indeed capture the case where the student model can in principle fully represent the weak model. Under certain assumptions (like expansion and separation), and along with consistency losses, this theory does explain performance gains of the self-trained student model, and thus seems to be a promising avenue to explain gains in WTSG as well. It would be interesting to see if the high-level analysis in our work (projection onto the space of strong models that are already implicitly biased towards the ground truth) could be combined with such an analysis to explain performance gains, even when no consistency losses are involved. Again, all these are really fascinating directions for future study!

[1] Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data.

Thank you so much for reading our paper, and for your review and comments. We are really glad that you like our work!

With regards to the points you bring up:

The results of WSCM20 are not properly contextualized. Their analysis is not limited to a self-training scenario and applies for any student model learning from an arbitrary teacher, including a student that is more powerful than the teacher.

Thank you for pointing this out. We will clarify that the analysis of [WSCM20] applies to arbitrary student-teacher settings (albeit under expansion assumptions and a consistency loss) in the updated version!

missing a discussion of and citations to relevant work in other semi- or un-supervised settings that bound generalization error in terms of the disagreement between two classifiers

Thank you for pointing out these references on disagreement-based analyses. We will be sure to include them in the revision.

L71 "Next, we imagine that the weak model sees data labeled by the target function $f^* \circ h^*$, and after finetuning, learns some arbitrary function $f_w \circ h_w$." Is this assumption really necessary? Can't we just assume we are given a weak predictor with some error rate? This limits the setup significantly to weak teacher models that are fine-tuned on ground-truth data.

You are right, our results broadly hold under access to any weak predictor. Note that we do mention in our statement of Theorem 1 that $h_w$ is any weak representation, and $f_w$ is any (arbitrary) predictor (so that $f_w \circ h_w$ is any arbitrary predictor). Nevertheless, we will rephrase the sentence in the introduction to make it clear that the weak predictor can really be any arbitrary predictor.

What happens in the current theory if technically the strong model hypothesis class $\mathcal{F}_s$ technically contains a function that can exactly fit the weak classifier $f_w \circ h_w$, but due to regularization this doesn't occur? This is often the case in practice, where regularization terms are required to avoid overfitting to the weak labels. Can the theory be modified to include a regularized version of $\mathcal{F}_S$?

This is a great question. In the setting considered in our paper, if $f_w \circ h_w$ can be represented exactly as $f_s \circ h_s$ for some $f_s \in \mathcal{F}_s$, and we were to perform naive finetuning without any regularization, the strong model will exactly converge to $f_w \circ h_w$, and we will see no weak-to-strong generalization. As also suggested in the work of Burns et al., 23, auxiliary “confidence” losses serve as regularizers to mitigate this phenomenon. Note however, that our theory heavily uses the fact that the minimizer of the (unregularized) mean squared error is in fact the projection of the weak model onto the convex set—this is no longer true with a regularization term. However, there is a natural duality between regularized objectives and constrained optimization problems. An interesting future direction then would be to characterize the regularization terms that constrain the space of optimization to still contain the target function, but exclude the weak model itself, so as to provably mitigate student-supervisor imitation.

---

Please let us know if we can help answer any other questions!

Thank you so much for reading our paper, and for your comments. We address your concerns ahead:

W1..presentation very confusing..notation $d_P$ to denote the mean-squared distance..

The only reason we introduced the notation $d_P(f,g)$ was for ease of reading: it is convenient to have some notation for the long-form term $E_P(f(x)-g(x))^2$. We explicitly state in line 133 that $d_P$ is the “avg squared distance wrt $P$" (this is already not a valid "distance" in the context of metric spaces, which indeed must satisfy the triangle inequality).

We want to assert that, if anything, we have tried to be as explicit and upfront about the simplicity and intuitive nature of our result as possible. There was no intent to make anything confusing. Nevertheless, we will add a line clarifying that $d_P$ does not satisfy the triangle inequality. We will also consider changing $d_P$ to $MSE$,just as you use in your review, and denoting the sides of the triangle as $A^2, B^2, C^2$ in Fig 1.

Thm 1 just a combination of law of cosines and convexity

This is indeed the right intuition; formalizing it to the space of functions requires a proof, which is what we include. To us, the fact that weak-to-strong generalization (WTSG) (a seemingly complex, and increasingly important phenomenon going ahead) in the concrete setting of regression can be directly traced to the Pythagorean thm for convex sets (a std fact in convex analysis) is both surprising and satisfying. The main contribution of our work is in modeling this phenomenon and formalizing the question. In this capacity, we view simple explanations as a strength and not a weakness, as they help us understand the system better.

...proof complicated by explicit use of representations throughout

The only reason we frame our results as “finetuning on top of representations” is because this is predominantly the language used for understanding AI models today. Your suggestion about clubbing $f_s \circ h_s$ to $\phi_s$ is well-taken; however, if we simply denote the composition by a single function, the result appears more abstract, and the link to WTSG is obfuscated. If anything, our intention was for the reader to appreciate that the working of these models can be elicited at a level where one can instantiate standard mathematical tools.

W2. result of the paper holds only for the MSE loss.

Indeed, and this is something we repeatedly make clear to be the scope of the paper (line 67, 133, 309..). This is a first step towards understanding the phenomenon of WTSG, and extending to other settings is an interesting future direction.

...qualitative conclusion clearly doesn't hold for classification...possible for strong model to differ from weak model on points where weak model is correct

It is a priori not clear that the qualitative conclusion of our work (that the gain in WTSG is effectively characterized by the WTS misfit) does not extend to settings beyond regression. It may very well be possible that an inequality like Eq 2 (with suitable error terms) holds in the classification setting, with $d_P$ measuring (say) the avg binary cross-entropy loss. Moreover, in such a case, it may still be possible that the strong model makes a mistake on a point where the weak model does not; the overall gain in performance will likely stem from the fact that the probability under the data distribution on such points is low, whereas the probability on points where the strong model improves is high. In any case, this is an interesting direction beyond the scope of the present work.

W3...Appendix E in Burns et al., 23, where they note that the strong model improves upon the weak supervisor when it is unable to simulate the weak labels

This is precisely what our main theorem is also saying! Any strong model that suffers a large misfit error when trained (without auxiliary losses) on weak labels (i.e., it is unable to simulate the weak labels well) exhibits non-trivial WTSG! Relatedly, Fig 8 in their paper shows that adding auxiliary losses in the objective helps reduce student-supervisor agreement (alternatively, increase misfit), and thereby improve WTSG.

W4...in general, high WTS disagreement will not always imply WTSG

The study in Appendix E.1,E.2 in Burns et al. 23 is more to do with the qualitative structure of WTS disagreement in the setting of classification. Namely, they ask: do different forms of disagreement (e.g., random errors vs correlated errors), for the same weak model accuracy, lead to differing WTSG?

While this nuance arises in the setting of classification, in the setting of regression that we consider, projection onto the convex set only results in movement towards the true function assuming realizability---in this sense, any misfit is "in the correct direction”, and its full quantitative benefit is realized. Thus, the qualitative difference among different forms of misfits does not manifest for us. While it is true that in other settings, the nature of disagreement might matter, the general principle of decreasing student-supervisor imitation (alternatively, increasing misfit) to foster WTSG, either via auxiliary losses or otherwise, does constitute a significant theme in the work of Burns et al., 23.

Even under the assumptions of convexity etc of Thm 1...

As our theorem asserts, for the regression setting under the squared loss and with a convex space of finetuning functions, it is (mathematically) true that for a given $d_P(\phi^*, \phi^w)$, a higher $d_P(\phi^w, \phi^{ws})$ (where $\phi^{ws}$ is the projection/minimizer of loss) will indeed lead to lower $d_P(\phi^*, \phi^{ws})$ (this is verbatim the inequality). In fact, this is also (nearly exactly) corroborated by all our experiments.

---

We will definitely add a summary of the discussion above, in relation to Appendix E in Burns et al., 23 in the final revision. We hope that our response helps address your concerns. Please let us know if we can answer any more questions.