Can DPO Learn Diverse Human Values? A Theoretical Scaling Law

We thank the reviewer for their thoughtful comments. We respectfully address the concerns below.

1. Generalization to unseen values

Thank you for the thoughtful feedback. We agree that analyzing generalization to unseen values is an important direction. However, our current focus on generalization within the same value distribution is deliberate and novel for several reasons. First, even in the current setting, understanding how DPO generalizes under value diversity remains an open and non-trivial challenge. Our work provides the first theoretical characterization of how training dynamics and sample complexity scale with the number of distinct human values—a critical step toward a rigorous understanding of preference optimization. Second, many existing generalization results are asymptotic or independent of training dynamics; in contrast, our framework directly captures the finite-step behavior of DPO, which better reflects real-world LLM fine-tuning practices.

We view our results as foundational groundwork and agree that extending the theory to OOD generalization is a valuable future direction. To extend our results, one promising path forward is to model relationships between seen and unseen value clusters—e.g., when unseen values lie within the span or convex hull of the training set, or when they share representational structure in the embedding space. This could enable interpolation-based generalization bounds or transfer guarantees based on geometric alignment (e.g., via cosine similarity or shared directions). While formally extending our current framework to this setting is non-trivial, we view it as a natural next step built on the foundation laid by this work. We've discussed this direction in the Limitations section and plan to explore it in future research.

2. Modeling assumption

Thank you for the thoughtful comment. This modeling assumption is well supported by a growing body of recent work on the linear representation hypothesis [1],[2], which posits that meaningful structure in LLMs can often be accessed or manipulated through linear operations on learned representations. This makes our assumptions not only analytically tractable but also well-grounded in the inductive biases of real-world models.

[1] Park, Kiho, Yo Joong Choe, and Victor Veitch. "The Linear Representation Hypothesis and the Geometry of Large Language Models." International Conference on Machine Learning. PMLR, 2024.

[2] Jiang, Yibo, et al. "On the Origins of Linear Representations in Large Language Models." International Conference on Machine Learning. PMLR, 2024.

In particular, a number of recent studies have leveraged linear probing, steering vectors, and representation engineering to analyze and intervene in LLM behavior (e.g., [Zou et al., 2023; Yan et al., 2024]). These works provide strong empirical evidence that linear structure over frozen embeddings is both informative and practically useful for understanding high-level model behaviors. Our theoretical framework builds on this evidence and provides the first generalization guarantees in this setting for preference optimization—an area where formal understanding remains limited.

Moreover, the technical novelty lies not just in the derivation of generalization bounds, but in the construction of a scaling law that quantifies how sample complexity grows with value diversity—an insight that has not been formalized before. Our proof techniques go beyond standard generalization theory by directly tracking reward margin dynamics under finite-step updates. We believe this offers a fresh perspective on preference learning and opens up new avenues for understanding alignment at scale.

3. Notation

Thank you for the careful read! We will update the notation accordingly.

We thank the reviewer for their thoughtful comments. We respectfully address the concerns below.

W1. Constrained setting

Thank you for the thoughtful comment. We address it in three parts:

(1) Empirical support for our modeling assumptions.

As we discussed in L147-L154, our modeling assumption is well supported by a growing body of empirical work on the linear representation hypothesis (Park et al., 2024; Jiang et al., 2024), which posits that meaningful structure in LLMs can often be accessed or manipulated through linear operations on learned representations. A number of recent studies have leveraged linear probing, steering vectors, and representation engineering to analyze and intervene in LLM behavior (e.g., [Zou et al., 2023; Yan et al., 2024]). These works provide strong empirical evidence that linear structure over frozen embeddings is both informative and practically useful for understanding high-level model behaviors. Our theoretical framework builds on this evidence and provides the first generalization guarantees in this setting for preference optimization—an area where formal understanding remains limited. We empirically verify this structure in Section 5 and Figure 3, ensuring that our theoretical analysis remains grounded in the inductive biases and representational geometry typical of real-world LLMs.

(2) Consistency with related theoretical literature. Our theorem setup is also consistent with recent literature on the theoretical understanding of alignment:

• Prior theoretical works on alignment dynamics adopt comparable simplifications to extract insights (e.g., Razin et al., 2025; Im & Li, 2024).
• Recent empirical work [Qi et al., 2025] finds that DPO updates disproportionately affect the first few tokens, supporting our focus on the single-token case as a key driver of alignment.
• Our setting also incorporates the embedding structure associated with distinct value clusters, which has been observed empirically (Park et al., 2024; Jiang et al., 2024). Moreover, clustering structure persists post-DPO training [Im & Li, 2024], suggesting that our assumptions capture the relevant inductive biases of LLM training.

Taken together, this places our contribution in line with other rigorous analyses that start from simplified but well-motivated cases, and it highlights that our results provide non-trivial and practically relevant insights into how sample complexity scales with value diversity.

(3) Clarification of limitations. We do agree with the reviewer that our limitations section should have more explicitly acknowledged this, and we will revise the paper to explicitly state this modeling constraint in the limitations section.

[1] Park, Kiho, Yo Joong Choe, and Victor Veitch. "The Linear Representation Hypothesis and the Geometry of Large Language Models." International Conference on Machine Learning. PMLR, 2024.

[2] Jiang, Yibo, et al. "On the Origins of Linear Representations in Large Language Models." International Conference on Machine Learning. PMLR, 2024.

[3] Razin, Noam, et al. "Unintentional unalignment: Likelihood displacement in direct preference optimization." International Conference on Learning Representations (2025).

[4] Im, Shawn, and Yixuan Li. "Understanding the Learning Dynamics of Alignment with Human Feedback." International Conference on Machine Learning. PMLR, 2024.

[5] Qi, Xiangyu, et al. "Safety alignment should be made more than just a few tokens deep." International Conference on Learning Representations (2025).

2. Direct Verification of Bound

We appreciate the suggestion. We have verified the theoretical scaling law by comparing the generalization error as a function of the number of concepts $K$:

Our empirical results (Table above) show that with fixed $Q$, the test error increases approximately linearly with $K$. The linear fit with $R^2 = 0.97$ confirms that the dependence on $K$ matches the theoretical prediction. This is precisely what the bound in Theorem 4.3 anticipates: adding more distinct human value clusters increases the statistical burden proportionally, unless compensated by more samples per cluster. We will include more extensive verification with runs across varying $Q$ in the revision.

3. Different Model Families

We appreciate the concerns about whether results hold for different model families and have added results for Mistral-7B-v0.3 and Qwen3-8B-Base.

In Theorem 4.2, we show that the rate at which the reward margin increases can decrease as the number of clusters or concepts increases in training. In Theorem 4.3, we show that the test error grows linearly with the number of clusters or concepts. The empirical results are consistent with our theorem across models, that the training reward margin grows more rapidly for smaller $K$, given the same number of training steps, and the test error increases linearly. Fitting a linear model gives an $R^2$ of 0.95 and 0.99 for Mistral and Qwen, respectively.

Mistral-7B-v0.3

Qwen3-8B-Base

We thank the reviewer for their thoughtful comments. We respectfully address the concerns below.

1. Idealized Assumptions

We appreciate the reviewer’s careful reading and would like to clarify why these assumptions are well-motivated and empirically supported, and we also extend our results to relax them:

1. Orthogonality of concept directions.
   While concepts in real-world LLMs are not perfectly orthogonal, there is strong empirical evidence that they can be well-approximated as such. Recent work on the linear representation hypothesis [Park et al., 2024; Jiang et al., 2024] shows that LLMs often encode disentangled semantic and behavioral features along nearly linear and often approximately orthogonal subspaces. We empirically verified this in Sec. 5 (Fig. 3), where subtracting the shared component yields embeddings with near-zero cosine similarity across distinct values, supporting the approximation.
2. Gaussian clusters with isotropic noise.
   We model preference distributions as Gaussian mixtures to enable tractable analysis of reward dynamics. This is standard in theoretical work on representation learning, and our empirical validation shows that LLM representations of values indeed form well-separated clusters with approximately spherical variation, consistent with this assumption.
3. Shared component orthogonal to concept directions.
   This assumption is not arbitrary: prior work (e.g., Park et al., 2024) identifies shared representation components in LLMs (e.g., frequency, style) that can be factored out to reveal concept-specific directions. Our empirical analysis confirms this: removing a shared direction from persona embeddings in Llama-3.1 leaves residual vectors that are approximately orthogonal across values (Fig. 3).

Extension to approximate orthogonality.
That said, we do appreciate the suggestion to consider approximate (rather than exact) orthogonality. We have extended our results to allow cluster means to have dot products of magnitude at most $\delta \leq \tfrac{1}{64(Z+2)}$. Under this relaxed condition, we recover the exact same generalization bound with the same probability as in Theorem 4.3. This strengthens the robustness of our theoretical framework by showing that our guarantees hold even when concept directions are only approximately orthogonal, which more closely matches real-world LLM representations.

Broader perspective.
We emphasize that these assumptions are not intended as literal descriptions of every aspect of LLM geometry, but rather reasonable approximations that make rigorous analysis possible. The fact that our theoretical predictions (Theorem 4.3) align with empirical scaling laws (Sec. 5, Fig. 5) and now extend to approximate orthogonality provides strong evidence that our framework captures the essential structure relevant to preference optimization, even in noisy, entangled settings.

2. Simplified Model

Thank you for the thoughtful comment. We address it in three parts:

(1) Empirical support for our modeling class.

As we discussed in L147-L154, our modeling assumption is well supported by a growing body of empirical work on the linear representation hypothesis (Park et al., 2024; Jiang et al., 2024), which posits that meaningful structure in LLMs can often be accessed or manipulated through linear operations on learned representations. A number of recent studies have leveraged linear probing, steering vectors, and representation engineering to analyze and intervene in LLM behavior (e.g., [Zou et al., 2023; Yan et al., 2024]). These works provide strong empirical evidence that linear structure over frozen embeddings is both informative and practically useful for understanding high-level model behaviors. Our theoretical framework builds on this evidence and provides the first generalization guarantees in this setting for preference optimization—an area where formal understanding remains limited. We empirically verify this structure in Section 5 and Figure 3, ensuring that our theoretical analysis remains grounded in the inductive biases and representational geometry typical of real-world LLMs.

(2) Consistency with related theoretical literature. Our theorem setup is also consistent with recent literature on the theoretical understanding of alignment:

• Prior theoretical works on alignment dynamics adopt comparable simplifications to extract insights (e.g., Razin et al., 2025; Im & Li, 2024).
• Our setting also incorporates the embedding structure associated with distinct value clusters, which has been observed empirically (Park et al., 2024; Jiang et al., 2024). Moreover, clustering structure persists post-DPO training [Im & Li, 2024], suggesting that our assumptions capture the relevant inductive biases of LLM training.

Taken together, this places our contribution in line with other rigorous analyses that start from simplified but well-motivated cases, and it highlights that our results provide non-trivial and practically relevant insights into how sample complexity scales with value diversity.

(3) Theory holds for full model fine-tuning. Although our theorems are derived in the fixed feature map setting, in Sec. 5 (Fig. 5) we empirically confirm that the same behavior predicted by Theorem 4.3 also appears under full-model fine-tuning.

3. Practical Guidance

Thank you for raising this concern. Our primary takeaway for practical usage is raising the necessity of more data per value/group as diversity increases, which we observe in Section 5. While this is derived in an ideal setting, in order to have clear guidance and insights on the impact of diversity, we believe this is necessary. In cases where values are overlapping, there is a wide range of scenarios that require different analyses, such as cases when the two values should be treated as distinct or cases when the values should be treated as one broader value. Providing clear insight and guidance under these different settings, especially on the role of value diversity itself, is an important problem we believe should be further studied in the future. Furthermore, we believe our bound is sufficiently strong to apply to most practical datasets, as even with 50 values, < 1000 samples per value is enough to achieve a guarantee of at most 5% generalization error, and many datasets have tens or hundreds of thousands of samples.

4. Multi-token Decomposition

In Sec. 4.3, we derive the exact decomposition of the reward gradient (Eq. 5) into interpretable terms: a token co-occurrence factor, a probability factor, and an output distribution correlation factor. This goes beyond a "hand-wavy" analogy — it is a precise gradient dynamics equation for multi-token responses, proved in Appendix C. Importantly, the structure of Eq. 5 mirrors that of the single-token case, establishing a formal connection between the two regimes.

Moreover, recent empirical work [Qi et al., 2025] finds that DPO updates disproportionately affect the first few tokens, supporting our focus on the single-token case as a key driver of alignment.

Extending convergence and generalization bounds to multi-token responses is highly challenging: dependencies across positions create coupled dynamics that resist closed-form analysis. We therefore positioned Eq. 5 as a first step toward that goal, showing that the core structural factors in the single-token case also govern the multi-token regime.

5. Empirical Support

We provide results for a small subset in the main paper for visual clarity. In addition to the subset shown in Fig. 3, we computed cosine similarities across all personas in the Anthropic dataset (see Appendix D). The same pattern holds: after subtracting the shared component, the off-diagonal similarities are near zero, supporting approximate orthogonality across diverse values.

---

References

[1] Park, Kiho, Yo Joong Choe, and Victor Veitch. "The Linear Representation Hypothesis and the Geometry of Large Language Models." International Conference on Machine Learning. PMLR, 2024.

[2] Jiang, Yibo, et al. "On the Origins of Linear Representations in Large Language Models." International Conference on Machine Learning. PMLR, 2024.

[3] Razin, Noam, et al. "Unintentional unalignment: Likelihood displacement in direct preference optimization." International Conference on Learning Representations (2025).

[4] Im, Shawn, and Yixuan Li. "Understanding the Learning Dynamics of Alignment with Human Feedback." International Conference on Machine Learning. PMLR, 2024.

[5] Qi, Xiangyu, et al. "Safety alignment should be made more than just a few tokens deep." International Conference on Learning Representations (2025).

We thank the reviewer for their thoughtful comments. We respectfully address the concerns below.

1. Empirical Scope

We appreciate the concerns about the scope and have added results for Mistral-7B-v0.3 and Qwen3-8B-Base. For the dataset, we choose the Anthropic Persona dataset as it allows for building datasets of distinct concepts, and there are a limited number of such preference datasets. One other such dataset is Stanford Human Preferences, but many of the categories have significant overlap or could even be considered subsets of each other, restricting the ability to clearly test the impact of diverse values.

Mistral-7B-v0.3

Qwen3-8B-Base

2. Multi-token extension

While we do not provide formal guarantees, we provide the multi-token decomposition to provide a basis for future studies. In particular, our goal in providing this decomposition is to demonstrate some connections with the single-token case but also shed light on the key factors that would need to be considered to tackle a broader multi-token understanding.

3. Orthogonal Concepts

We appreciate this question, and it would be possible to extend our theory to account for approximately orthogonal directions. We extend our results to account for the means to be approximately orthogonal with two values having means with dot product with magnitude at most $\delta \leq \frac{1}{64(Z+2)}$ while maintaining the exact same bound with the same probability.

We also note that the persona datasets are not synthetically generated and do not perfectly satisfy the orthogonality constraints and Gaussian distribution assumption. Despite this, we can still observe the expected scaling behavior.

4. Logarithmic Verification

We verify the scaling law by comparing the test error with the number of concepts $K$, and find that it closely follows a linear model. Based on this, we can expect additional samples per concept $Q$ to be helpful in reducing the test error for larger $K$ to those of smaller $K$. We will include more extensive verification with runs across varying $Q$ in the revision.

Linear Fit $R^2 = 0.97$

5. Ambiguous Clusters

Thank you for raising this question. We agree that in practice, values may not be so clearly defined, and it can be ambiguous to define individual values, and these may also vary across annotators. While our current results do not directly address this, these are important aspects of human preferences that should be further studied, potentially through extending our current framework. In particular, for annotators who have differing notions of the same value, we can build this in by adding a noise model that captures this variance across individuals.

Dear Reviewer,

Thanks for your participation in the reviews! Now that the author-reviewer discussion period is close to wrapping up, please take a look at the author's response and see if any of your concerns have been addressed or if you would like to maintain your stand. It would be helpful to respond directly to the authors in addition to filling in the mandatory acknowledgement checkbox.

Please leave a response to the authors based on their rebuttal content as we proceed to the reviewer discussion phase shortly. Thanks!

• AC RJJA