Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment

Thank you for your thorough review and insightful feedback on our manuscript. We appreciate the time you took to evaluate our work and provide constructive comments. We believe addressing your points will significantly strengthen the manuscript.

1. Intransitivity Usefulness:

Q1: The paper would benefit from providing more evidence on why better modeling of intransitive human preferences is useful...

Thanks for your suggestion. Modeling intransitivity is motivated by observing that human judgments can be complex, context-dependent, and exhibit cycles, which simpler Bradley-Terry (BT) models cannot capture. GPM's multi-dimensional space aims to represent these richer structures. While noise is a factor, capturing potentially genuine complex preferences (Appendix F examples) is vital for alignment. Distinguishing noise from true intransitivity is important for future work.

2. Line 114 Citations:

Q2: Additionally, in line 114, the authors cite two papers,...

The two references (Tversky, 1969; Agranov & Ortoleva, 2017) are among the earliest to study the prevalent phenomenon of humans exhibiting non-transitive preferences. We use them to motivate moving beyond transitive models like BT. We will clarify the scope of these references in our revision.

3. Cyclic Data:

Q3: However, the cyclic preference data created by the authors...

This synthetic dataset demonstrates GPM's expressiveness where BT models inherently fail (random guessing) due to their transitivity assumption. Appendix E details its construction from Ultrafeedback; we'll add a summary to the main text.

4. Boundedness for Theorem 3:

Q4: For example, in theorem 3, the preference score needs to be bounded for the convergence result to hold,...

Thanks for the feedback. Our GPM implementation ensures boundedness. We apply L2 normalization to embedding vectors $v_{y \mid x}$, making them unit length. Since the score is $s\left(y_i \succ y_j \mid x\right)=v_{y_i \mid x}^{\top} D(x) R^{>} D(x) v_{y_j \mid x}$, and $R^{>}$ is magnitude-preserving, scores are bounded if eigenvalue scales $\lambda_l(x)$ are bounded. This is guaranteed as we use a softmax function on the gate outputs determining eigenvalues. Thus, theorem conditions hold. We'll clarify.

5. Cyclic Experiments:

Q5: Regarding the cyclic preference experiments (Table 1), ...,

Synthetic data provides a clear proof-of-concept for modeling intransitivity. We agree that evaluating cycle prevalence in real benchmarks and comparing generalization vs. BT models are valuable future work. Appendix F shows potential real-world examples.

6. Table 3 (SPPO+GPM vs. GPO):

Q6: First, what is the difference between SPPO+GPM and GPO?... Second, according to the LC win rate metric...

SPPO+GPM uses win rates from GPM scores; GPO uses raw scores s(yi>yj∣x) directly. GPO maximizes expected score; win-rate methods maximize expected probability. Raw scores might offer a richer signal (any real number) vs. win rates (0 to 1), potentially explaining GPO's performance despite win rates offering stability. We'll clarify. We acknowledge GPM/GPO doesn't always beat BT/SPPO on Length-Controlled (LC) win rate. GPM/GPO models tend to produce longer responses ("Avg. Len"). As LC Win Rate controls length bias, this might impact results. The appendix shows length-normalized GPO (LN-GPO) results. More discussion will be added.

7. Section 3.2 (Preference Modeling):

Q7: Section 3.2 on preference modeling does not accurately represent the existing literature...

Sec 3.2.1 introduces PairPMs for general preference modeling, distinct from standard BT models (Sec 3.1). Lines 188-190 illustrate PairPM examples and potential issues, not all reward model training. We'll revise for clarity, distinguishing standard models (scalar head, BT loss) from PairPMs. We'll add a discussion on Stiennon et al. (2020) and Lambert et al. (2024).

8. Notation (K vs. d, Complexity):

Q8: Some of the notation is a bit confusing... number of responses is K and the embedding dimension is 2k...

$K$ is response count; 'd' is embedding dimension (formerly $2k$) . $K, d$ are unrelated. Complexity concerns model forward passes: GPM/BT require $\mathcal{O}(K)$ passes (one per response) for embeddings/rewards. PairPM requires $\mathcal{O}\left(K^2\right)$ passes (one per pair). GPM offers linear scaling of model calls w.r.t. K, matching BT, improving over PairPM. We'll clarify notation/complexity. The $P \in \mathbb{R}^{2 k \times 2 k}$ setup in Thm 2 illustrates spectral decomposition; our method (Sec 4.2)/Thm 1 doesn't require $d=K$. We'll clarify this distinction.

9. Automatic Subspace Discovery:

We appreciate your interest. You asked for empirical evidence of interpretable directions/eigenvalues. Excellent suggestion for future work. Analyzing embeddings/eigenvalues via probing/visualization could yield insights. Figure 3 is a first step; systematic investigation is needed. We'll add this as a future direction.

Thank you for your thoughtful and detailed review. We appreciate your constructive feedback and the opportunity to clarify aspects of our work. We address your points below:

Below, we try our best to address the points you raised:

1. GPM Architecture, Training, & Efficiency:

Q1: The authors did not elaborate on the details of GPM's model architecture, and training objectives... hard to verify the claimed computational efficiency...

• Architecture/Implementation: We describe the implementation of GPM in Section 4.2, detailing the eigenvalue scale gate and the eigenvector embedding head used to generate the preference embeddings.
• Training Objective: The optimization objective for training GPM is explicitly defined in Appendix A.2 (Equation A.1). It involves minimizing the standard cross-entropy loss used in preference learning, based on the predicted preference probability ($\sigma(s(y_w>y_l∣x)$), where s is the GPM score from Eq 4.1) and the observed preference data $P_D(y_w>y_l∣x)$. This objective directly mirrors the typical objective used for training BT-based reward models.
• Computational Efficiency: GPM achieves $\mathcal{O}(K)$ query complexity (Sec 4), matching BT models and improving on $\mathcal{O}(K^2)$ pair-based models, by processing responses individually.

2. GPM Performance on RewardBench:

Q2: Although GPM is claimed to have stronger expressiveness, it does not seem to show a consistent bonus on RewardBench...

• Thank you for your observation regarding RewardBench performance. While performance can vary across specific sub-tasks and embedding dimensions (as shown in our ablation studies in Section 6.1 and Appendix E.1), GPM consistently outperforms the BT reward model on average across several tested base models, as shown in Table 2. Specifically, GPM showed average improvements of +7.44% (Gemma-2B-it), and +1.34% (Llama-3.1-8B-Instruct) over the BT baseline. Significant gains were often observed in the Chat and Chat-Hard categories. We appreciate the suggestion to analyze complex cases further and will add this to the appendix.

3. Clarity on Cyclic Preference Setting:

Q3: ...The remaining issue can be on cyclic preference, which I fail to understand the settings. A step-by-step explanation... would make it more clear.

• These datasets were constructed using Ultrafeedback data, where we identified instances exhibiting cyclic preferences based on ratings across different criteria (e.g., helpfulness, honesty, instruction following). For example, a cycle might emerge where Response A is rated higher on honesty than B, B higher on helpfulness than C, but C higher on honesty than A. We provide details on the construction in Appendix E and results demonstrating GPM's near-perfect accuracy on these tasks in Section 6.2 and Table 1. Figure 3 also visualizes the learned embeddings for these cyclic cases.

• We will add a more detailed step-by-step explanation of the dataset construction to the appendix in the revised version, as you suggested.

4. Preference Embedding Significance vs. ArmoRM:

Q4: ...similar to ArmoRM... each head in ArmoRM has a clear meaning... while heads in GPM do not. Therefore, simply adding all scalars...

GPM's design prioritizes expressiveness and generality. The core idea is that complex, real-world human preferences might not always decompose neatly into a few predefined, interpretable scalar dimensions. They can exhibit intransitivity and context-dependency that require a more flexible representation.

• Theoretical Grounding: Our multi-dimensional embeddings, combined with the skew-symmetric operator $\mathbf{R}^{\succ}$ and inner product $< \mathbf{R}^{\succ} v_{y_i|x},v_{y_j∣x}>$, provides a principled way to model any skew-symmetric preference relationship, including cycles. Theorem 1 and Theorem 4 provide theoretical grounding for this expressiveness.
• Expressiveness and Generality: While ArmoRM's interpretable heads offer clarity, this structure inherently limits its capacity. Such models based on aggregating scalar rewards, even multi-dimensional ones, generally cannot capture arbitrary preference structures, particularly those involving intransitivity (like cycles) or complex mixtures of transitive and intransitive components.
• Automatic Learning vs. Annotation: Furthermore, ArmoRM relies on aggregating scores from heads associated with predefined metrics (like helpfulness, truthfulness), which often require costly, explicit human annotation for each metric. In contrast, GPM aims to automatically discover and learn the relevant multi-dimensional preference representations directly from a single, standard pairwise preference signal ($y_w \succ y_l$). The dimensions in GPM are learned end-to-end to capture relevant preference factors without needing pre-defined semantics or multi-metric labels. The eigenvalue scale gate further allows the model to dynamically weight these learned factors based on the context.

We sincerely thank the reviewer for their thoughtful and constructive comments. We especially appreciate the feedback regarding Table 3 and acknowledge that the acronyms used were not sufficiently explained in the current draft.

1. Clarification of Table 3:

Q1: First of all, it's full of acronyms (e.g. LC. WR, WR, Avg Len), and the acronyms are not explained in the caption and main text. I suppose WR probably means "win rate", but it's also unclear to me if it means the win rate over what. I can't understand the results, unfortunately.

To clarify:

• LC. WR stands for Length-Controlled Win Rate, a metric from AlpacaEval 2.0 that adjusts for length bias in generation.
• WR denotes the Win Rate, i.e., the proportion of times our model's response was preferred over the baseline (GPT-4-turbo)'s, as judged by GPT-4-turbo.
• Avg. Len refers to the Average Length of generated responses in tokens. The win rates are computed using pairwise comparisons between model outputs (e.g., our trained models vs. GPT-4-turbo) on the same set of prompts, with GPT-4-turbo as the evaluator.

2. Why These Acronyms Were Used:

Again, I don't fully understand the experiment settings because of writing issues.

AlpacaEval is a widely used benchmark in nearly all recent LLM alignment papers (e.g., SimPO, SPPO, MagPie, Nemotron), and the format and metrics (including LC. WR) are standard in the literature. Due to space constraints, we opted to use the established abbreviations. That said, we agree that clearer exposition would benefit broader readability, and we will revise the caption and surrounding text to explicitly define these terms.

3. On the Role of This Table:

Q3: Table 1 presents a comparison of Bradley-Terry (BT) reward models and General Preference embedding models (GPM) on cyclic preference datasets. The results seem convincing and support the claim. The main problem is with Table 3, which shows the main result.

We emphasize that our primary contribution is the development of a new, expressive architecture (GPM) for modeling general preferences, which addresses the limitations of traditional reward models like Bradley-Terry. The GPO-based alignment results in Table 3 are supplementary and serve to illustrate one possible downstream use case (Section 6.3). We believe that the clarity issues in Table 3, while important to correct, should not detract from the significance of our core contributions.

4. Additional Revisions:

Q4: I'm not aware of critical misses. However, I think it'd be better to add references to more prior work in preference optimization, such as: Mitchell, A note on DPO with noisy preferences & relationship to IPO, 2023 Liang et al., Robust preference optimization with provable noise tolerance for LLMs, 2024 Furuta et al., Geometric-Averaged Preference Optimization for Soft Preference Labels, 2024.

We also appreciate the reviewer’s suggestions on related work. We will incorporate a discussion of the following relevant papers in the final version:

Mitchell, A note on DPO with noisy preferences & relationship to IPO, 2023

Liang et al., Robust preference optimization with provable noise tolerance for LLMs, 2024

Furuta et al., Geometric-Averaged Preference Optimization for Soft Preference Labels, 2024.

We sincerely thank you for your time, insightful feedback, and constructive comments on our paper. We appreciate the recognition of GPM's novelty, theoretical grounding, and universal framework potential.

We would like to address the specific points raised:

Q1: Generalisability and further results on complex cyclic preferences.

A1. We appreciate the suggestion regarding generalisability. As shown in Section 6.2/Table 1, GPM achieves near-perfect accuracy on cyclic preference datasets where BT models perform near-random guessing. This demonstrates GPM's effectiveness in capturing the intransitivity inherent in complex preference structures, a key limitation of BT models.

Q2: Need for more diverse, real-world scenarios/context, noting benchmark challengeability.

A2. Thanks for your suggestion. We tested GPM on RewardBench (covering Chat, Chat-Hard, Safety, and Reasoning) and assessed downstream alignment using AlpacaEval 2.0. GPM consistently outperformed BT models across these benchmarks and different base models (Gemma-2B/9B, Llama-3.1-8B). GPM-integrated methods also showed improved win rates. While benchmarks have limitations, these consistent gains across diverse tasks suggest practical advantages.

Q3: More ablation/real-world scenarios are needed to make really strong claims.

A3. We acknowledge your point that further studies could strengthen our claims. However, we believe the current results already provide significant support. The demonstrated superiority in modeling cyclic preferences, consistent outperformance on the diverse RewardBench tasks, and improvements in downstream alignment tasks collectively offer strong empirical evidence for GPM's effectiveness and advantages over traditional BT models. We have also included ablation studies on embedding dimensions (Section 6.1, Table 2) and GPM architecture design (Appendix E.1, Table 4).

Q4: Curiosity about embedding dimension analysis for competing/conflicting preferences and how GPM manages trade-offs.

A4. Thanks for your feedback on analyzing embedding dimensions in scenarios with competing preferences and how GPM handles trade-offs. This is an excellent question that touches upon a core aspect of GPM's design. As discussed in Section 4.2, the multi-dimensional embedding space allows the model to automatically discover subspaces corresponding to various preference facets (e.g., helpfulness, honesty, style). The eigenvalue scale gate, which computes context-dependent eigenvalues {λ(x)}, then modulates the influence of these different dimensions based on the specific prompt. This mechanism allows GPM to dynamically weigh competing preference aspects and manage trade-offs based on context. Our ablation studies on embedding dimensions (Section 6.1, Table 2 and Appendix E.1, Table 4) further explore how performance varies with dimensionality across different tasks and models, indicating that the optimal configuration can depend on the specific trade-offs required.

Q5: Adding references on representation learning.

A5. Thank you for suggesting a more extensive review of representation learning. We have acknowledged this connection briefly in Appendix D and agree that it's a relevant area. We will expand on this connection in the revised version.