The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gains

Questions:

  2. What if we pair models from two families, such as Llama 3B over Qwen 1.5B?

Great question! Our original motivation for using a weak and weaker model from the same series was that they are trained with largely similar data and architecture and differ primarily in parameter count. This setup increases the likelihood that the larger model is consistently stronger, thereby reducing the risk of generating noisy preference pairs where the smaller model may produce better outputs than the larger one.

That said, nothing fundamentally prevents us from pairing models across families. Motivated by your question, we ran additional experiments using cross-family model pairs: (1) Llama-3B over Qwen-1.5B (as you suggested), (2) Llama-3B over Qwen-0.5B, and (3) Qwen-3B over Llama-1B. We follow the hyperparameters of our analysis experiments (Appendix D.6). The results in Table R4 below show that cross-family pairings yield consistent improvements in average performance. However, the gains are generally smaller than with same-family pairings. We speculate this is due to increased noise in the preference pairs: for instance, on MATH, Qwen-1.5B outperforms Llama-3B, so training with Llama-3B over Qwen-1.5B introduces a delta in the wrong direction and subsequently hurts MATH performance after tuning. We will add these results to our paper. Thank you for the suggestion!

Table R4: Results of training Tülu-3-8B-SFT on preference data from cross-family model pairs

――――――――――――――――――――――――

Thank you again for taking the time to review our work. We look forward to further discussion!

Thank you again for your thoughtful review! We truly appreciate your time and engagement, and we’re thrilled to hear that all but one of your concerns—scaling to 405B—have been addressed.

We agree that validating delta learning at frontier scales would be exciting future work. However, we would like to reiterate that our results at the 8B scale are robust, reproducible, and clearly demonstrate the promise of delta learning. As you noted, like most academic researchers, we do not have access to the compute required to train 405B models. We hope you’ll consider these constraints in light of COLM’s reviewing guidelines, which recognize that meaningful innovation can come without experiments at the “upper limit of scale,” and that “limiting” research to only those with massive resources may stifle progress [1]. While not all 8B-scale results may hold at frontier scales, "some results will." We are optimistic that delta learning falls into this category.

Given that your other concerns have been addressed, we sincerely hope you’ll consider increasing your score. We look forward to further discussing with you—thank you for your time!

[1] https://colmweb.org/ReviewGuide.html

Thank you again for your feedback! Based upon your review, we plan to revise our paper by moving our discussion on scale limitations (Appendix B, L551-552) to our main text. We also plan to revise the discussion to include the following text:

In this work, we have shown that models can learn surprisingly well from the delta between paired weak data points. At the widely-deployed 8B scale, we have used this insight to greatly simplify open post-training recipes. We are optimistic that our work may open future research on what other deltas---beyond the overall model quality deltas explored in this paper---may be informative, and what tasks such deltas might benefit. Moreover, our work focuses on the 8B scale; it remains an exciting open question to validate delta learning at larger frontier scales, and to understand how the properties of what makes a delta useful might change as models get larger. We leave these questions to future work.

We would love to hear your thoughts, as well as any further feedback you may have for strengthening our paper. We eagerly await your response to our comments!

EXAMPLE 1:

Prompt: Dani brings two and half dozen cupcakes for her 2nd-grade class. There are 27 students (including Dani), 1 teacher, and 1 teacher’s aid. If 3 students called in sick that day, how many cupcakes are left after Dani gives one to everyone in the class?

Chosen response (Qwen-2.5-3B-Instruct):

First, let's determine the total number of cupcakes Dani brought. Since a dozen is equal to 12, two and a half dozen cupcakes is calculated as follows…

So, the number of cupcakes left is:

$\boxed{4}$

Rejected response (Qwen-2.5-1.5B-Instruct):

To determine how many cupcakes are left after Dani gives one to everyone in the class, we need to follow these steps…

So, the number of cupcakes left after Dani gives one to everyone in the class is:

$\boxed{6}$

Note: the correct answer is 4.

――――――――――――――――――――――――

EXAMPLE 2:

Prompt: when were we first settled by humans

Chosen response (Qwen-2.5-3B-Instruct):

The first humans to settle in what is now known as the world are believed to be the Homo sapiens, who originated in Africa around 300,000 years ago. However, the first modern humans to leave Africa and settle in other parts of the world are thought to have occurred around 50,000 to 70,000 years ago.

Specifically:

1. Out of Africa: The first major migration of modern humans out of Africa occurred around 50,000 to 70,000 years ago. This migration involved groups moving into Europe, the Middle East, and eventually Asia.
2. Europe: The first known settlement in Europe by modern humans is believed to be in the Levant (modern-day Syria, Lebanon, and Israel), around 50,000 years ago.
3. Asia: The first known settlement in Asia by modern humans is believed to be in the Altai Mountains in Siberia, around 45,000 to 50,000 years ago.
4. Americas: The first known settlement in the Americas by modern humans is believed to be around 15,000 to 17,000 years ago, with the earliest evidence found in the Beringia region, which is now part of Alaska.

So, while the first humans originated in Africa, the first major settlements outside of Africa occurred relatively recently in human history, with the earliest known settlements in Europe, Asia, and the Americas occurring around 50,000 to 17,000 years ago.

Rejected response (Qwen-2.5-1.5B-Instruct):

The exact date when humans first settled on Earth is not known, as there is no evidence of human presence on the planet before the formation of the solar system. However, scientists believe that humans originated in Africa and then spread to other parts of the world over a period of millions of years. The earliest known human fossils date back to around 2. 5 million years ago, and the earliest known human settlements date back to around 100, 000 years ago.

――――――――――――――――――――――――

EXAMPLE 3:

Prompt: Classify whether the given answer is correct or not… …Kai wanted to save money so he went to the car dealership and haggled… Question: How would you describe Kai? Options: (A) wasteful (B) extravagant (C) thrifty Answer: C Output:

Chosen response (Qwen-2.5-3B-Instruct):

The given answer C (thrifty) is correct based on the context provided. Kai's actions of wanting to save money and haggling to buy a used car demonstrate frugality and careful financial management, which aligns with the definition of being thrifty.

Rejected response (Qwen-2.5-1.5B-Instruct):

Yes

Table R3: Statistics of the chosen and rejected responses in our weak preference datasets, with statistics of responses from the Tülu 3 preference dataset shown for reference

Thank you for a detailed response to my concerns.

Regarding my last comment (on the preliminary versions of this paper), I came across a preliminary version of this work in a workshop presentation. When I wrote my review, I had not noticed that the workshop was a non-archival venue.

Overall, I think this is a great work. I will raise my score to 9.

Concerns:

  2. The paper has strong empirical evidence, but could stand out with mechanistic understanding of why delta learning works.

Spurred by your feedback, we studied delta learning in a controlled theoretical setting. We prove that delta learning succeeds with high probability for homogeneous logistic regression (homogeneous meaning no bias, so that the decision boundary passes through the origin). We believe the proof offers insight into why delta learning can work, and we sketch key ideas in the problem setup and proof here. We will update our paper to include the longer formal derivation. If you have interest, we would also be more than happy to discuss further details through OpenReview during discussion.

Problem setup. We assume inputs $x \in \mathbb{R}^d$ are drawn from an isotropic Gaussian with mean 0 and identity covariance, and that there exists some (unobserved) ground-truth model with weights $\theta^\*$ achieving perfect classification accuracy. Fix an arbitrary student model $\theta_0$ that we aim to improve. We construct preference pairs $(x, y_c, y_r)$, where $y_c, y_r$ are chosen and rejected pseudo-labels (respectively) generated by teacher models $\theta_c, \theta_r$. The teacher models are randomly sampled from the set of all logistic regression models, under the constraint that their cosine similarities with the ideal model $\theta^\*$ are some arbitrary but pre-determined constants $-1 < \alpha_r < \alpha_c < 1$:

$$\cos(\theta_c, \theta^\*) = \alpha_c, \cos(\theta_r, \theta^\*) = \alpha_r.$$

In other words, we pick a random $\theta_c$ satisfying $\cos(\theta_c, \theta^\*) = \alpha_c$, and likewise for $\theta_r$. Intuitively, higher cosine similarity with $\theta^*$ implies better average performance over the data, so $\alpha_c > \alpha_r$ implies that $\theta_c$ is a stronger model than $\theta_r$ in expectation.

We train the student model with SGD to prefer the stronger teacher’s labels $y_c$ over the weaker teacher’s labels $y_r$ using the simple loss, $\mathcal{L} (x,y_c,y_r; \theta) = p_\theta (y_c | x) - p(y_r | x)$. For simplicity, we assume training with the population gradient here; we have also shown an extension to finite empirical gradients via a standard martingale analysis and will present the full results in our final paper.

Theorem (informal). Fix a failure probability $\delta \in (0,1)$. So long as the dimension $d$ of the model exceeds a certain (reasonable) threshold, then with high probability at least $1-\delta$ (over the sampling of teacher models) the training procedure improves the student, even when both teachers are weaker than the student itself. In particular, the threshold depends on the student model's initial alignment with $\theta^\*$ (denoted $\alpha_0$), the size of the strength gap $(\alpha_c -\alpha_r)$ between the teacher models, and the failure probability $\delta$:

$$d > 2\ln \frac{4}{\delta} \Biggl( \frac{2|\alpha_{0}|\cdot |\|\theta_{0}\|\| } {(\alpha_{c}-\alpha_{r}) (1-\alpha_{0}^{2})} \Biggr)^{2}+1.$$

For realistic choices of these parameters, $d \approx 2000$ suffices, i.e. the threshold is not prohibitively high.

Our result says that for any arbitrary student model, almost all pairs of teacher models exhibiting a performance delta suffice to generate preference data that will improve it. As a caveat, we have no guarantees on the magnitude of the improvement. In particular, our result requires early stopping; it does not imply that we can keep training the student to get arbitrarily large gains.

Proof (informal). The key idea is that by Stein’s Lemma, the gradient update induced by the loss moves the student in the direction of $(\theta_c - \theta_r)$—towards the stronger teacher $\theta_c$ and away from the weaker teacher $\theta_r$. Since $\theta_c$ is, by assumption, better aligned with the optimal model $\theta^\*$ than $\theta_r$, the difference vector $\theta_c - \theta_r$ is itself positively aligned with $\theta^\*$, regardless of how low the absolute alignment of $\theta_c$ may be. In other words, the delta between the two teachers yields a directionally correct signal, even when both teachers are individually weak. In high dimensions, the noise components of the update become negligible and this directional signal dominates. Thus, with high probability, training improves the student, even when the student is already better than either teacher.

We will discuss these results in our updated paper. Thank you for helping strengthen our work!

Concerns:

  3. What semantic properties of the delta (beyond magnitude) matter remains an open question.

This is an exciting open question! We mainly focus on defining the delta learning hypothesis and showing its utility, and view further studying what facets of the delta matter as future work. Still, we are optimistic that our controlled experiments on bolded section generation (Section 3.1) offer early evidence that deltas can be targeted to the exact semantic aspects we aim to improve. In other words, we speculate that deltas may be most effective when semantically aligned with gaps in model behavior. One promising future work may curate deltas by synthesizing targeted corruptions along specific axes—e.g., adding unsafe language to generate rejected responses for preference tuning to improve safety.

We also manually inspected our best weak preference data (Qwen-3B over 1.5B) to qualitatively understand what semantic deltas were present. Overall, there was no single axis by which Qwen-3B responses are better, but we observed a few interesting deltas for future study:

• For verifiable prompts (e.g. math prompts), we find pairs where Qwen-3B responds correctly but Qwen-1.5B does not
• For knowledge-seeking prompts, Qwen-3B often gives more detailed responses
• For prompts allowing brief answers, we find pairs where Qwen-3B generates a chain-of-thought, while Qwen-1.5B gives just the answer

We will add this discussion to our next revision.

Questions:

  1. How does delta learning perform on safety?

Please see our response above. Thank you!

  2. Is delta learning a complement to SFT, or a substitute?

Great question! Without strong data, SFT often hurts while delta learning can still succeed (Figure 4); we are thus optimistic that delta learning can be a more scalable substitute. That said, in a pilot study applying delta learning directly to Llama 3 base pretrained models, we found the trained models often failed to follow instructions. While we did not optimize the delta learning data in this study, the results hint that some SFT may be needed to at least teach basic chat formatting. It remains an exciting open question as to what skills are better instilled with SFT versus delta learning. We will revise to include this discussion to help motivate future work.

  3. What is the degradation with an incorrect delta?

Thanks for the question! To test this, we tuned Tülu-3-8B-SFT (i.e. the base model from our post-training study) on incorrect deltas constructed by swapping the chosen and rejected responses in our weak preference datasets. We try

• Qwen-1.5B over Qwen-3B responses
• Llama-1B over Llama-3B responses

We follow our analysis studies’ hyperparameters (Appendix D.6). Results in Table R2 show large degradation across all tasks under both incorrect deltas (-6.3 points avg. with Qwen, -11.6 with Llama). We will include these results in our next revision.

Table R2: Performance degradations under an incorrect delta

  4. How do we ensure that preference tuning avoids learning from low-quality responses of weak models?

Great observation! While we empirically show that preference tuning with weak data yields state-of-the-art performance across standard benchmarks (Table 4), this does not ensure gains across all model behaviors. Conceptually, delta learning requires a useful delta between the chosen and rejected responses to drive learning. On behaviors where both the chosen and rejected models are equally weak, there is little signal to learn from. Moreover, it is possible for a a given Model X to outperform Model Y in one behavior while underperforming in another. In such cases, no matter how the models are paired, the resulting preference data contains a negative delta in at least one behavior. We observed this directly in the jailbreaking resistance results discussed in response to your Concern 1.

Overall, delta learning is not a magic solution and has potential pitfalls. Even so, it shows strong promise, enabling us to greatly simplify open post-training recipes. We are optimistic that future work may curate deltas in a more fine-grained and targeted manner than our simple approach of pairing outputs from two models. More principled curation may maximize the informativeness of each delta and reduce unintended regressions. We will revise our paper with these points—thank you for helping us strengthen our work!


We look forward to further discussion!