Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictions

We thank the reviewer for their positive feedback on our theoretical analysis, extensive experiments, novelty and broader impact.

Cor 5.1-2: why label agnostic

As noted in L306 right, once we factor the label-dependent term out of the operator norm (becoming the label variance C), it can be treated as constant for a fixed dataset. When varying the models, the only factor that changes on RHS of 5.1 is $||P_s(I-P_w)||$, independent of labels. Thus the trend of RHS can be predicted w/o labels. Same for 5.2

L110,163

Thanks for the suggestions. We’ll incorporate them in the revision.

L175 Left: L180 typos

This is a typo. We didn’t intend to use Einstein notation. We’ll add the missing summation after $\frac{1}{n}$.

“Conditions d and e closely related”

One possible way to make (d) and (e) more “unified” is to rephrase e as a variant of the cross-sample statement but with the size of one sample taken to infinity. We’ll note this in the revision.

Thm 3.6

We’ll include the following discussion. Thm 3.6 highlights the generality of Def 3.3, showing more examples can be constructed. Any representation composed of a part satisfying Def 3.3 with δ=0 (eg, from a very low-rank distribution) and a high-dimensional sub-Gaussian will, as a whole, satisfy Def 3.3. Eg, Example 3.4 can be extended by concatenating sub-Gaussian.

discussion on hyperparameters

(1) How we tuned the hps: See Sec D.2 for detailed hp values. We select the hps that maximizes correlation with test Err_w2s. Our metric is computed on the w2s split, consistent with theory.

(2) Cross-model hp transfer: We note that, although each model could technically require different hps, in experiments we let all weak models share hps for simplicity and still achieve strong results, suggesting that our approach is not very sensitive to hps. Further, we present a new experiment demonstrating that hps selected using one group of models (ie, as a validation set) generalize to other models. We randomly split the weak models into two groups, select hps based on one group, and evaluate them on the other. We repeat this 20 times and report the results on 5 datasets https://anonymous.4open.science/r/icml2025figures/table.png. Correlation remains high with low std, indicating that hps selected using a few models can reliably generalize to new ones. Additionally, we note that a small number of labeled data should suffice for hp tuning, as they are only used to measure test performance and not to compute our metric.

(3) intuitions for hp selection

• For β: β captures the effect of regularization and should be set higher when stronger regularization is used. This could explain why the optimal β in Exp III is much higher than in II: in III finetuning for one epoch only (following prior work) introduces very early stopping and thus strong regularization.

• For α: The choice of α depends on the underlying dimensional structure. While the relationship can be complex, one intuition is that larger models tend to require a higher α. For small models whose dimensionality is relatively low compared to the sample size, most components can concentrate well. There may be few or no non-principal ones, so a small α suffices to filter them out. In contrast, larger models have high-dimensional yet low-rank representations (Huh et al 2021), where only a few top components concentrate with finite data. There are more non-principal components, and moreover, their magnitudes can appear inflated in the finite sample due to MP law—necessitating a larger α. These intuitions align with the reviewer’s observation from Fig 6 Exp II that the strong model requires a larger α than the weak ones

“explicitly targeting the label agnostic property; common crawl”

We note that all experiments demonstrate the label-agnostic property. In Figs 2-4, our label-agnostic estimator (x-axis) strongly correlates with Err_w2s (y-axis). For the second half of the comment, if the reviewer was asking whether our metric computed on Common Crawl (CC) could predict performance on arbitrary tasks, this is not feasible: label-agnostic≠task-agnostic. Predicting performance on task A still requires unlabeled data from A. It is not reasonable to expect that CC could be used to indicate performance on any tasks. If the question was instead about predicting performance on CC itself, the bottleneck is evaluation: since CC lacks labels, we can’t compute W2SG performance as ground truth to compare our metric against.

beyond human comprehension regime

We will revise the sentence to reflect: (1) in the future AI may surpass humans in certain tasks; (2) even today, AI can outperform average humans on certain tasks—yet W2SG shows that average humans can still help improve AI in those tasks. In particular, our results indicate which humans (or weaker LLMs) can best teach a strong AI —and interestingly the answer is not necessarily the strongest human or the strongest weak LLM.

We thank the reviewer for finding our theoretical claims clear and convincing, and for acknowledging the novel insights of our paper.

“...relative representation structures of the weak teacher and strong student matter…Without controlling for weak supervisor quality, it’s hard to know whether this is a confounder …”

(a) A new experiment. We note there are cases where the weak teacher’s performance cannot indicate W2S performance. We conducted a new experiment, where we control the weak supervisor’s error to lie within a narrow range, and observe that the weak supervisor’s error correlates poorly with the W2S error, while our metric shows a strong correlation.

Since in our main experiments in the paper, the weak supervisors’ errors span a relatively large range, it is difficult to find a sufficient number of weak supervisors with similar errors. Therefore, we explicitly construct weak supervisors with similar error levels in the following way: We take a single checkpoint of a weak supervisor from Experiment I (the one with hidden size 256 pretrained for 5 epochs), and generate 20 modified versions of it by randomly masking out 100 coordinates of its features each time. We then run the weak-to-strong pipeline for each of these 20 weak supervisors. We compare the correlation between $Err_w$ and $Err_{w2s}$, and between $| P_s(I - P_w) |{\mathrm{op}}$ and $Err_{w2s}$ across all three datasets. As shown in the table below, the weak supervisor’s error correlates poorly with the weak-to-strong error, while our metric maintains a strong correlation. This further demonstrates that the detailed relationship between the weak teacher and the strong student plays an important role in weak-to-strong generalization—beyond what can be explained by the weak supervisor’s error alone.

(b) We illustrate our intuition with a simple example. Suppose a downstream task consists of 40% advanced linear algebra and 60% advanced calculus. We have two weak pretrained models: model A specializes in basic linear algebra, and model B in basic calculus. Assume fine-tuning mainly builds on existing knowledge rather than learning from scratch. Then, after fine-tuning, model A would likely achieve ~40% performance and model B ~60%, reflecting their alignment with the task. Now consider a strong student pretrained only on linear algebra. According to our main theory, model A—being aligned with linear algebra—should be a better supervisor for this student leading to better W2SG performance, even if its own performance is lower. Our proposed metric captures this alignment and should correlate with W2SG performance, whereas weak supervisor performance alone does not. This highlights a case where our metric offers meaningful insight beyond what weak model performance alone can explain.

(c) Practical perspective: Measuring the weak supervisor’s error requires access to labels. In contrast, our metric is label-agnostic. The fact that it achieves such a high correlation while using less information is already impressive.

“The writing throughout the first part of Section 3 could be more clear. Lots of notations and intermediate results are presented without a lot of guiding intuitions towards the final claims.”

Thanks for the suggestion. Due to space constraints and the large number of results, we focused on conveying the most important messages and were unable to include more detailed explanations. We will add more intuitive explanations in the revised version.

If there are no further concerns, and given the reviewer’s largely positive feedback, we would greatly appreciate it if the reviewer would consider raising the score.

We thank the reviewer for finding our claims well-supported, our explanations insightful and intuitive, the theory and experiments extensive and convincing, and the proposed metric novel. We respond to the comments below.

the distinction between Sec 4 and Wu & Sahai (2024)

The main differences are twofold:

(1) Wu & Sahai (2024) show that benign overfitting can happen, but they do not extract general insights about when and how it occurs. In contrast, we identify a single key quantity driving benign overfitting in W2SG—namely, $||P_s(I - P_w)\frac{1}{\sqrt{n}} y||$ in Thm 4.1—which characterizes how much of the label aligns with the intersection between what is missed by the weak model's principal kernel and captured by the strong model’s principal kernel. When this quantity is small, the strong model can avoid repeating the weak model’s mistake, regardless of the extent of overfitting, thereby achieving error mitigation. This very mechanism is not revealed in Wu & Sahai.

(2) Our Thm 4.1 is stated in a very general setting, whereas Wu & Sahai focus on a highly specific distribution with detailed assumptions, making it more of a toy example than a realistic scenario. E.g., there is no evidence that neural network representations follow exactly their assumed bi-level ensemble structure and that labels depend 1-sparsely on representations. In contrast, our assumptions (discussed on page 4) cover a wide range of realistic cases, supported by literature suggesting that neural network representations often exhibit such properties.

further explanation of assumptions in Def 3.3; and intuition or practical implications of condition (c)

Due to space limitations, we only provided explanations for kernel-wise isotropy and small cross-sample inner products on $\mathcal{V}^\perp$, as we believe these two are the most involved, while the others are relatively natural and self-explanatory. Here, we provide further explanation for all the items, which we’ll include in the revised version.

• (a) is a basic condition that ensures reasonable magnitudes of representations and labels.

• (b) states that representations are well-concentrated in the subspace $\mathcal{V}$, both in terms of their covariance and their correlation with labels. This is why the representations on $\mathcal{V}$ are referred to as the principal representations—they are the part where the empirical distribution closely aligns with the underlying population distribution.

• (c) implies that kernels constructed using only the components in $\mathcal{V}^\perp$ exhibit a certain level of uniformity across all directions, with the degree of this uniformity controlled by $\delta$. In the paper, we discuss two extreme cases—one with very small $\delta$ and one with very large $\delta$—to aid understanding. Importantly, this assumption is not made solely for analytical convenience; it is also general and applicable to realistic settings. For example, high-dimensional sub-Gaussian noise satisfies this condition with a small $\delta$—a scenario highly relevant to deep neural networks with large internal dimensions—since these vectors tend to be orthogonal to each other in the high-dimensional limit. More concrete instances can be found in Examples 3.4 and 3.5 and Thm 3.6, with their significance and relevance discussed in the right column of page 4. Thus, (c) is a key condition that allows us to capture all these diverse scenarios. It is not just analytically useful, but also practically relevant to real-world scenarios.

• (d) holds either when representations on $\mathcal{V}^\perp$ are nearly orthogonal across samples or when their magnitudes are small.

• (e) means that the representations on $\mathcal{V}^\perp$ have small magnitudes in the population.

Typos in Def 3.3 (b)

As the reviewer correctly pointed out, the expression $|| \frac{1}{\tilde{n}} \Pi_{\mathcal{V}} h(\tilde{x_i}) \tilde{y_i} - \mathbb{E}[ \Pi_{\mathcal{V}} h(x) y ]||$ contains a typo—we forgot to include the summation after $\frac{1}{\tilde{n}}$. We will fix it in the revised version.

If there are no other concerns, and given that the reviewer’s feedback is largely positive, we would greatly appreciate it if the reviewer could consider raising the score.