Weak-to-strong Generalization via Formative Learning from Student Demonstrations & Teacher Evaluation

Q1: The paper introduces various notations, including $r^{\text{weak}}$, which first appears in line 114 but is only defined in line 159. This lack of clarity in the early definition of terms can lead to confusion and make the reading process difficult. It would benefit from defining all key symbols and notations clearly in the "Preliminaries" section to ensure that readers can follow the theoretical developments without distraction.

Answer: Thank you for pointing this out. We believe you were referring to line 144 instead of line 159. This is indeed a mistake on our part — the definition of $r^{\text{weak}}$ should appear at line 144, and we will revise the manuscript accordingly in our camera-ready version.

Q2: One limitation of the paper is that the student model is restricted to having only a single parameter size, which limits the exploration of how different model sizes or architectures might affect the performance of the EVE method. The impact of the student model’s architecture and parameter size on the EVE approach remains unclear, and further investigation into this area could provide a better understanding of how the model size influences the efficiency and scalability of the method.

Answer: Thank you for your valuable feedback. Our evaluation protocol follows the existing works (e.g., Burns et al.) and employs a similar set of teacher and student models. However, we'd like to confirm that we indeed extended our experiments to qwen2.5-7B as the student and llama-3.2-1B as the teacher in Section 5.2. The results consistently show superior performance compared to baselines, reinforcing that EVE's can scale well to larger models.

Q3: The proposed EVE method introduces substantial computational overhead, especially when generating data from the strong student model. This is due to the need for repeated sampling and the additional feedback loop required for evaluation. While the method is effective, the extra computational cost may pose challenges for scaling the approach, particularly for large language models or applications requiring high-performance computing.

Answer: Thank you for your thoughtful comment. We would like to clarify that generating data from the strong student model is not unique to our EVE method—this step is also present in refinement-based approaches ([1,2]), where strong models are used to sample responses. However, a key limitation of such refinement methods is the lack of systematic evaluation of the generated data, which can lead to suboptimal or noisy supervision.

EVE explicitly addresses this gap by using weak teachers to evaluate the generated data using importance weighting. While this adds some computational overhead, it is a principled trade-off that ensures higher data quality and more reliable weak-to-strong learning.

Q4. Why do the results of GPT2-Large in Figures 3 and 6 differ significantly from GPT2 and GPT2-Medium?

Answer: As discussed in our Section 5 (for Figure 6) and in the caption of Figure 3, when there is a large gap between the teacher and the student (i.e., the teacher is much weaker than the student, as in the case of GPT2 and GPT2-Medium), the W2S performance of EVE is significantly better than that of SFT. This is because the weaker teacher generally can make more mistakes than a stronger one, making the strong student, trained with SFT, also more susceptible to these mistakes. As the teacher becomes stronger, approaching the strong student, the stronger teacher's feedback will become more reliable, leading to a smaller performance gap between SFT and EVE.

Q5. What is the specific computational overhead introduced by EVE, and how does it compare to baseline methods?

Answer: Thank you for your question. EVE introduces minimal computational overhead compared to baseline methods. A key design feature of EVE is that it can pre-compute the rewards from the teachers in training data before training. This eliminates the need to load or query the teacher model during the training process, which can incur additional computational cost. As a result, the computational overhead of EVE's is similar to supervised fine-tuning (SFT) — using the same memory footprint and requiring no additional forward passes through teacher models -- but with an additional step of loading the reward scores.

To verify this claim, we use Llama-3.2-3B on TL;DR summarization task with a batch size of 128, we measure the time for each training step over 100 steps and GPU memory of SFT and EVE's for comparison.

Q6. Can EVE be combined with Refinement? The authors could experiment with a hybrid model that incorporates both EVE and Refinement, where the weak teacher first generates data and the strong student refines it using the Refinement method, with EVE's evaluation guiding the student's learning process. This would help explore whether combining these methods can lead to more robust generalization, especially when faced with unreliable or limited weak supervision. A detailed analysis of the results from such a hybrid approach would provide useful insights into the synergy between the two methods. Could the authors discuss the feasibility and potential benefits of combining these two methods? Would such a hybrid approach further improve performance, especially in cases where the weak teacher is considerably weaker than the strong student?

Answer: Thank you for the insightful suggestion. In fact, EVE can be naturally viewed as a generalization of the Refinement framework (see Section 2.1). While Refinement improves weak teacher outputs by leveraging the strong student as a generator, EVE augments this setup by using weak teachers as an explicit evaluation function to guide and filter the learning process. This added evaluation signal makes EVE's more flexible and robust.

Q1. The experimental section is insufficiently comprehensive, focusing solely on two instruction-following benchmarks. While these datasets are relevant for instruction-following tasks, the evaluation lacks diversity in task types, such as complex reasoning, code generation, limiting the assessment of EVE's broader applicability.

Answer: Thank you for your valuable feedback. We agree that expanding the range of evaluation tasks can provide a more holistic assessment of EVE’s generalization capabilities. However, we would like to clarify that our focus on controlled-summarization and instruction-following benchmarks is consistent with the predominant experimental settings in prior works on weak-to-strong learning and alignment, which primarily evaluate on summarization and instruction-following tasks [1,2,3,4]. Our goal was to provide a direct and fair comparison under these widely adopted settings.

Q2. The comparison with baseline methods (e.g., Direct Preference Optimization, DPO) is limited, omitting other relevant approaches like variants of RLHF or traditional supervised learning, which weakens the evidence for EVE’s superiority.

Answer: Thank you for the comment. We would like to clarify that our experiments do include a comparison with a standard supervised fine-tuning (SFT) baseline, where EVE consistently outperforms this method, demonstrating its effectiveness even against traditional supervised approaches.

Moreover, to the best of our knowledge, prior works in the weak-to-strong (W2S) alignment works typically adopt Direct Preference Optimization (DPO) as a post-alignment algorithm [1]. In contrast, full-scale fine-tuning using standard RLHF methods—such as PPO—is often omitted due to the high computational cost, especially when scaling to large student models.

Q3. The manuscript has textual errors and inconsistent formatting, e.g., “KL divergence” is italicized in lines 147 and 153 but uses regular font in lines 102, 156, and 176. The authors should carefully review the typesetting.

Answer: Thank you very much for pointing this out. We will correct these issues in the camera-ready version.

Q4. Additionally, some "proofs" (e.g., lines 146 and 152) are obvious and better termed "derivations" for accuracy.

Answer: Thank you for your suggestion and we will revise accordingly. The result presented in Theorem 4.2 provides a key insight that motivates our proposed method: it demonstrates that imitating a weak teacher can be interpreted as a reward maximization problem. This perspective highlights the potential risk of overfitting to suboptimal teacher behavior. As a result, we introduce KL regularization as an effective regularization to mitigate the influence of teacher errors by encouraging the learned policy to stay closer toward its initial knowledge.

Q5. The checklist section "5. Open access to data and code" is unanswered, suggesting oversight.

Answer: We sincerely apologize for the oversight. We will definitely release our code publically and update the checklist accordingly in the final version.

Q6. The "7. Experiment statistical significance" section notes the absence of multiple trials and significance testing, raising doubts about the authors' thoroughness and whether the results are coincidental. These issues indicate a lack of careful review, undermining the paper's credibility.

Answer: Our evaluation protocol exactly follows prior influential works in LLM Alignment and Weak-to-strong generalization [3,5,6], which also omit statistical significance testing do to the time-consuming and compute-intensive nature of training LLMs (e.g., running 5 experiments per setting is extremely difficult). Nevertheless, the consistent improvements of EVE across different experiments makes the coincidence of EVE's good performance statistically less likely.

---

[1] Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models. NeurIPS24.
[2] Aligner: Efficient Alignment by Learning to Correct. NeurIPS24 Oral.
[3] Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization. ICLR25.
[4] A transfer learning framework for weak to strong generalization. ICLR25.
[5] Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS23 Oral.
[6] Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms. NeurIPS24.

Q1: Prior W2S mechanisms such as pseudo‐label correction and coverage expansion (see [1]) are neither cited nor ablated, despite their relevance to weak-to-strong learning theory. In particular, without a precise mathematical specification of the "weak teacher" distribution and the "strong student" model family, it is unclear how the method's forward-KL objective differs in practice from the pseudo-label correction paradigm. Similarly, coverage expansion methods explicitly encourage the student to assign non-negligible probability mass to low-scoring but valid outputs, covering the underrepresented regions of the output space. Because the method neither references nor empirically compares to these formulations, I find it unclear whether its gains stem from the forward-KL directionality alone or inadvertently incorporate aspects of label correction and coverage regularization.

Answer: Thank you for this insightful comment. We would like to clarify your concerns on the role of forward‑KL vs label correction and coverage. [1] prove that if the data distribution and student class satisfy an expansion condition, then the student can correct teacher mistakes (pseudolabel correction), and perform well on regions where the teacher abstains (coverage expansion) If our gains are due to forward‑KL alone, without these expansion conditions, then we might be inadvertently inducing label correction or coverage implicitly via regularization or data neighborhood structure. Our current results do not disentangle these contributions.

Q2: Computational cost is unreported: while the authors note limited resources for > 7B models, no GPU-hour or memory‐footprint estimates are given.

Answer: Thank you for your question. EVE introduces minimal computational overhead compared to baseline methods. A key design feature of EVE is that it can pre-compute the rewards from the teachers on the training data before training. This eliminates the need to load or query the teacher model during the training process, which can incur additional computational cost. As a result, the computational overhead of EVE's is similar to supervised fine-tuning (SFT) — using the same memory footprint and requiring no additional forward passes through teacher models.

To verify this claim, we use Llama-3.2-3B on TL;DR summarization task with a batch size of 128, we measure the time for each training step over 100 steps and GPU memory of SFT and EVE's for comparison.

Q3: No paired‐sample significance tests are reported for gains presented as mean

Answer: Our evaluation protocol exactly follows prior influential works in LLM Alignment and Weak-to-strong generalization [3,5,6], which also omit statistical significance testing do to the time-consuming and compute-intensive nature of training LLMs (e.g., running 5 experiments per setting is extremely difficult). Nevertheless, the consistent improvements of EVE across different experiments make the coincidence of EVE's good performance statistically less likely.

Q4: Instruction‐following performance is currently assessed solely through LLM‐as‐judge metrics (e.g., AlpacaEval scores) without any reporting of human‐annotator agreement rates or the number of examples evaluated. This is problematic because automated judges can inherit the same biases and blind spots as the models they evaluate, leading to overestimates of true quality. Without details on sample size, it is impossible to gauge statistical power or confidence intervals for the reported gains.

Answer: Thank you for the valuable feedback. However, we note that using GPT-4 as an evaluator has become a widely adopted practice in LLM Alignment and Weak-to-strong literature [2,3,4,5], due to its strong correlation with human preferences in many tasks. Given the high cost and scalability challenges of human evaluation, we follow a similar evaluation protocol as prior works [2,3,4,5], which includes standardized benchmarks such as AlpacaEval 2.0 and IFEval and GPT-4-based assessments. This setup remains consistent with the current evaluation standards in the field.

Q5: What are the computational costs of the method on a typical dataset?

Answer: Thank you for your thoughtful comment. We would like to clarify that generating data from the strong student model is not unique to our EVE method—this step is also present in refinement-based approaches ([5,6]), where strong models are used to sample responses. However, a key limitation of such refinement methods is the lack of systematic evaluation of the generated data, which can lead to suboptimal or noisy supervision.

EVE explicitly addresses this gap by using weak teachers to evaluate the generated data using importance weighting. While this adds some computational overhead, it is a principled trade-off that ensures higher data quality and more reliable weak-to-strong learning.

Q6: Several plots and tables do not clearly label which model is used (as the "strong student") in the experiment (e.g., Figures 2 and 3). Figure 3’s PGR‐over‐epoch curves are never referenced in the text

Answer: Thank you for pointing out the typos. Figure 3 demonstrates the problem of overfitting to the weak teacher's mistakes in SFT and how EVE could fix this problem, and fits under Section 4.3. We will revise and add the missing discussion to our camera-ready version.

---

[1] Theoretical Analysis of Weak-to-Strong Generalization. NeurIPS24.
[1] Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS23 Oral.
[2] Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models. NeurIPS24.
[3] Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model. ICLR25 Spotlight.
[4] A transfer learning framework for weak to strong generalization. ICLR25.
[5] Weak-to-Strong Reasoning. EMNLP24.
[6] Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision. ICLR25 Spotlight.

Dear reviewer 5hvm,

Thank you for your response. While [1] can be seen as related -- which we will appropriately cite and discuss in our work as mentioned in the rebuttal, our work is orthogonally different in contributions, scopes, and motivations. We will further elaborate our rebuttal's response w.r.t these 3 points:

• Contributions: [1] does not provide any new training methods, but rather a theoretical framework to encourage W2S generalization. Our work, on the other hand, theoretically reveals the limitations of existing W2S learning from the lens of reward maximization (we will elaborate more in Objectives), proposes a new W2S learning method, and finally rigorously analyze and validate the method in an extensive set of models and benchmarks.

• Scopes: It's important to note that [1]'s theoretical analysis is only limited to classification, where they assume the class conditional set $\mathcal X_i$ to provide an error bound, which it not available in open-ended generation task, which is the focus of our work. Consequently, it is non-trivial to derive and ensure the equivalent coverage expansion and pseudolabel correction in generative task. Note that, the related W2S works on generative tasks [2,3,4,5] do not also provide any connection to the theoretical error bound in [1].

• Objectives: [1] assumes the strong student naively learn to imitate the teacher's predictions, while our work view W2S generalization as reward maximization. Specifically, we generalize refinement and use the teacher to refine the weak responses as an effective regularization that avoids imitating weak teacher errors. Notice that, the Forward KL minimization $\text{KL}\left(\pi^{\*}||\pi\_\theta\right)$ is between the optimal policy RLHF $\pi^\*(y|x) \propto \pi^{\text{strong}}\_{\text{ref}}(y|x)\exp\left(r^{\text{weak}}(x,y)\right)$ and the learned strong student $\pi_\theta$, while [1] studies the Forward KL minimization $\text{KL}\left(\pi^{\text{weak}}||\pi_\theta\right)$, which is well known to imitate errors.

For these reasons, while it is interesting to connect our work to [1], it is extremely difficult and complex, which deserves several independent works, including either theoretical focus (just like [1]) and empirical focus to validate the theoretical results. We believe that this should not be viewed as a limitation of our work, but rather a potential extension to both [1] and ours, and we will add this discussion and cite [1] to our final version accordingly.

As explained to the reviewer on statistical significance tests, we follow the existing protocol [6,7,8], and admitted the difficulties in computational requirements. Nevertheless, we have been running statistical significance testings for the "main experiments" only since the start of the rebuttal and will provide the results shortly during the discussion.

---

[1] Theoretical Analysis of Weak-to-Strong Generalization. NeurIPS24.
[2] Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision. ICLR25 Spotlight.
[3] Weak-to-Strong Reasoning. ACL24.
[4] Bayesian WeakS-to-Strong from Text Classification to Generation. ICLR25.
[5] Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models. NeurIPS24.
[6] Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization. ICLR25.
[7] Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS23 Oral.
[8] Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms. NeurIPS24.

---

Thank you for your response. Please see our response below:

Statistical significance testing: As provided in the global response, we have now included statistical significance testing for all experiments. This is a significantly more effort than any existing W2S papers (see [1-13] in the response), which do not have any significance testing due to computational reasons of training LLMs. The details of significance testing (sample size, standard deviations, $p-value$ in the response, and evaluation details are standard as in the paper) is also provided for reproducibility.

Lack of a precise mathematical definition: In our previous response to the reviewer, we greatly clarified why [1] is not related to our work, and a similar theoretical analysis is unsuitable and extremely complex in our setting. We'd also like to emphasize again that a large number of related and even less related papers [2-9] to ours do not have any similar theoretical analysis (or any theoretical connection) as in [1]. Nevertheless, our work already provided the relevant theoretical analysis on reward maximization, which is the core study of this work.

We hope that our effort and explanation to clarify the misunderstandings about our paper would convince the reviewer of the novelty and significant contributions of our work!

---

[1] Theoretical Analysis of Weak-to-Strong Generalization. NeurIPS24.
[2] A transfer learning framework for weak to strong generalization. ICLR25.
[3] Alice: Proactive Learning with Teacher’s Demonstrations for Weak-to-Strong Generalization. 
[4] Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization. ICLR25.
[5] How to Mitigate Overfitting in Weak-to-strong Generalization?. ACL25.
[6] Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model. ICLR25 Spotlight.
[7] Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision. ICLR25 Spotlight.
[8] Weak-to-strong Reasoning. ACL24.
[9] Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning. ACL24.

---

Q1: Experiments stop at a 3B parameter student. There is no evidence shows that method can be scalable

Answer: Thank you for the comment. We believe that there are some misunderstanding. We'd like to clarify that our experiments are not limited to 3B parameter student models. In fact, we include evaluations with 7B-scale student models in the paper (see Section 5.2), which demonstrate both the scalability and effectiveness of our approach. The 7B results consistently show superior performance compared to baselines, reinforcing that EVE's can scales well to larger models.

Q2: Evaluation relies on synthetic rewards and GPT-judge metrics, with minimal human or real-world validation. It is suggested to add more LLM judgers.

Answer: Thank you for the valuable feedback. However, we note that using GPT-4 as an evaluator has become a widely adopted practice in LLM Alignment and Weak-to-strong literature [3,4,5,6], due to its strong correlation with human preferences in many tasks. Given the high cost and scalability challenges of human evaluation, we follow a similar evaluation protocol as prior works [3,4,5,6], which includes standardized benchmarks such as AlpacaEval 2.0 and IFEval and GPT-4-based assessments. This setup remains consistent with the current evaluation standards in the field.

Q3: Hyper-parameters are tuned using ground-truth validation labels, breaking W2S premise.

Answer: Thank you for your feedback. However, we believe that there is also some misunderstanding. We would like to clarify that we do not perform any hyper-parameter tuning using ground-truth validation labels. In fact, our ablation studies show that EVE's is robust to a wide range of reasonable $\beta/\lambda$ values.

Moreover, prior W2S works [1,2] also introduce additional hyper-parameters, which are either tuned per model or fixed to a specific value. We adopt the latter strategy: we empirically set the ratio $\beta/\lambda = 1.0$ and use this same value across all experiments. This approach ensures consistency and avoids reliance on label-based tuning, in line with the W2S paradigm.

Q4: The LLM sets used for the two tasks are different, and there is no explanation on why to do so.

Answer: Thank you for your feedback. For the instruction-following task, we follow prior works that study weak-to-strong generalization by selecting models that are sufficiently capable and proficient at language understanding. In contrast, GPT-2 lacks the necessary capabilities for instruction-following and thus is not suitable for this setting. Therefore, we adopt a stronger model to more meaningfully study generalization in this task.

Q5: Suggest to add more recent works in the related works.

Answer: Thank you for your suggestion. If the reviewer can kindly let us know which relevant work is missing from our paper, we'd be happy to address them and cite them accordingly.

---

[1] Bayesian WeakS-to-Strong from Text Classification to Generation. ICLR25.
[2] Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision. ICLR25 Spotlight. 
[3] Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS23 Oral.
[4] Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models. NeurIPS24.
[5] Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model. ICLR25 Spotlight.
[6] A transfer learning framework for weak to strong generalization. ICLR25.