A transfer learning framework for weak to strong generalization

Thank you for the suggestions on empirical results to add. Below we include (our best efforts to recreate) the mentioned OAI baselines on the persona experiment.

For bootstrapping we use the model chain weak -> gpt-3.5-turbo -> gpt-4o-mini. Unfortunately, it is not possible for us to directly implement the auxiliary loss method of OAI since we cannot access the model weights to train with a custom loss. As a proxy, we used a data doubling method, where each training question has an answer from the weak teacher and an answer produced by the strong model. We believe this is a reasonable way to emulate the auxiliary loss method, which adds a term that punishes the model for deviating from its own predictions. If the reviewer has a different idea in this vein they wish us to test, we are happy to. If, on the other hand, this is satisfactory, then we will add these results and discussion to the submission.

Our findings are below, "cont" denotes the accuracy while "style" measures the amount of concept transfer. Bootstrapping offers some safeguarding against accuracy degradation, but generally still allows for too much corruption of the strong model. The auxiliary loss improves accuracy but reduces the style transfer (this is expected in light of our theory, one can think of this method as strengthening $\eta$ in Prop 3.2). Overall, our methods offer a nice balance of (mostly) transferring the concept with minimal degradation.

TTQA:

TAE:

Thank you for the helpful comments and questions. We discuss specific concerns in the following.

in practice, stronger models may produce outputs that are completely beyond the capabilities of weaker models.

We wish to clarify that our framework allows for this to occur. While the target model must be in the convex hull of the source model, the weak model does not have to satisfy this property. For example, the weak model may be a version of the (aligned) strong model that is missing some features or underfit in some way.

strength of convex hull relationship and how we can determine it in practice We use the convex hull assumption to encode the intuition that the strong source model might possess some latent ability to accomplish the target task; it needs only to be directed to do so by the weak model. This assumption allows us to reason statistically about methodologies to try, but is not a pre-requisite for implementing our refinement methods.

Practically, our suggestion is to try our refinement approach, and if it fails, it may suggest that the assumptions that lead us to this approach do not hold.

If we find that the source model cannot fully cover the target distribution, what are some feasible solutions? In light of the above discussion, we believe the heart of this question is enquiring about settings where our methods can fail. We agree that weak to strong generalization is an on-going open problem, some alternative methods are suggested in the references below, but they are certainly not uniformly successful (and several are extensions of our simple ICL method or baselines we beat in our experiments). [1] Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. 2023 [2] Yang et al. Weak-to-Strong Reasoning. 2024

Thank you for the helpful comments and concerns. Below we address specific issues.

Strong assumptions that are sometimes not elaborated enough.

• The assumption that the weak model has the correct prior is likely necessary in theory; without this, the learner has no access to either the correct model or the correct prior over the concepts. Philosophically, this is also reasonable as when applying weak-to-strong generalization to achieve its main objective in practice, i.e., allow a human to align a strong LLM to their interests, it is a natural assumption since the human's prior is the prior they wish to transfer onto the LLM (i.e, human is "perfectly aligned").

• We wish to clarify that the assumption that betas are orthonormal should hold in the target too. In all cases, the orthonormality is not necessary for the results, but it substantially simplifies the terms in bounds, generally improving interpretability. What is important is some distinguishability between the $\{\beta_k\}$, orthonormality is a simple way to impose this.

• Likewise, identity covariance of $\phi^*$ is not necessary. In the revised draft we have simplified things by making population assumptions on the features rather than fixing a design matrix.

• The assumptions in case 2.1 were meant to represent simple cases where the weak model is some underfit (e.g. missing some features) version of the strong model (in practice, the weak model is indeed much smaller). These were not necessary, so we have dropped them from the results.

Insufficient exploration of simpler weak-to-strong generalization techniques than the proposed one summarized above

For a more thorough exploration of the first auxiliary method (from Appendix D), see the table in the response to reviewer DyBd. We have added more baselines and included a side-by-side comparison with the ICL method in the main text.

We make a general note here that we don't believe one method is better than another; rather, these are multiple ways we thought of for implementing a refinement type procedure. The best choice is likely highly dependent on the task at hand.

Elaboration of Theorem 3.2.

We have simplified the lower bound in the new draft. Note that before $\eta$ did not appear because we minimized over $\eta$ for the final lower bound; in the new version, we have removed this final step and left the dependence on $\eta$ explicit.

Elaboration of lines 305-308 In order to clarify the presentation and provide a better connection to prior work on statistical models for LLM's we have made changes to the presentation of this area. Namely (i) We have laid out directly the assumption we need for Theorem 4.3 and in appendix A added two references for prior works ([1], [2]) where this assumption holds. (ii) To appendix B, we have added a calculation for the case where this assumption does not hold.

[1] Xinyi Wang et al. Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning. 2023

[2] Reese Pathek et al. Transformers can learn regression mixture models. 2024.

Clarification of figures

Yes, the weak models (llama 7b, gemma 2b, falcon 7b, mistral 7b) remain constant left to right. In a given row of plots the order goes gpt-3.5-content, gpt-3.5-style, gpt-4o-mini content, gpt-4o-mini style. Note here gpt is the strong model.

In the case of the math experiments, no labels from falcon 7b are used.

Thank you for the helpful comments, we address your concerns below.

The connection between the theoretical model and practical analysis is weak.

We agree that the validity of our transfer learning framework is dependent on prior results demonstrating that ICL can be modeled as implicit Bayesian inference. In order to strengthen the connection between theory and practice, we have done the following:

(i) To appendix A We have added discussion on two more references that consider latent bayesian/ mixture models of LLM's. The first [1] proposes a simplified version of Xie et al's work. The second [2] shows that LLM's can directly learn the mixture of regression setting we work in.

The theory in our work can be seen as specifying an $f$ in [1] and studying ICL inference when the examples are ``weak". If the latent concept model is less convincing, an alternative interpretation of the theory is an analysis of the MSE of labels produced by the architecture in [2] (with the complication that examples are produced by a weaker model).

(ii) To provide a more direct connection to Xie et al, we have now included an analysis where the source model is a mixture of Hidden Markov Models. This is an extension to the main result of Xie et al where ICL examples come from a weaker model. This setting is a bit more complex but is slightly closer to a real LLM (in particular delimiter tokens can be treated formally).

[1] Xinyi Wang et al. Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning. 2023

[2] Reese Pathek et al. Transformers can learn regression mixture models. 2024.

Improving assumptions made to bias the weak model

These are not strictly necessary for the analysis, and we have dropped them from the results in the paper.

empirical results are missing some key baselines.

(i) "If we ignore the labels from the weak model, and directly use the stronger model to label data...would that curate a good data set too?"

The motivation of the weak-to-strong generalization setting is some form of (weak) human providing examples to teach the model. So we have chosen to focus on methods which utilize the human/weak model data in some manner. The necessity of this data is clear in the style experiments where throwing away all weak data will clearly lead to poor performance, as there will be nothing left influencing the strong model towards the desired style.

Beyond this, completely ignoring the weak model and generating synthetic data from the strong model to improve itself is out of scope of the weak-to-strong generalization problem setting. It has different goals (i.e. improving the model's overall capabilities through its own synthetic data, as opposed to aligning the model to the specific interests of a weaker human teacher) and many prior works have already studied this problem, e.g.

Wang et al. Self-Instruct: Aligning Language Models with Self-Generated Instructions, 2023. NVIDIA, Nemotron-4 340B Technical Report, 2024

That said, we do agree that the use of some plain self-training data (along with weak data) is an interesting approach. In the response to reviewer DyBd, we have included a comparison to a method which mixes in such data.

(ii) "As the performance of the weak model improves (still subpar compared to the stronger model), naive finetuning outperforms their ICL version of finetuning in multiple tasks"

While it is true that for some weak models, naive fine-tuning can improve the style transfer, we disagree that this should be interpreted as naive fine-tuning outperforming our ICL method. As an example, consider the explanation technique experiment. While naive fine-tuning leads to simpler explanations, the explanations themselves become incorrect as the strong model capability is reduced. In general, we do not interpret this as a more successful outcome. Finally, we also note that extensions to our method discussed in the appendix eliminate this "style gap".

Other Questions

• We have simplified the lower bound in the new draft so that the role of the bias of both the source and weak models is clear.

• Labels from one source refer to labels from only one weak teacher. We have clarified this in the draft.

• Statistically, the lower bound is avoidable because inferring the target discrete concept from the weak data is easier than trying to learn the target model. The separability makes this inference possible.

• We agree that the tightness of theorem 4.2 is an interesting future question. To address this properly would require establishing the minimax rate of estimating $f_Q$ in our set up. This is of interest to us but is non-trivial, and we hope to consider it in future work. It is quite possible that our method is not optimal in the minimax sense. That said, the statistically optimal method likely does not have any analog that can be implemented for a real LLM, which is one advantage of our refinement-based estimates.