General Preference Modeling with Preference Representations for Aligning Language Models

Q1: Accuracy of GPM on Cyclic Preference Data

Thanks for your feedback. We appreciate your concern and have thoroughly investigated our experimental setup. The high accuracy is due to the GPM's ability to fully capture cyclic preferences by design, as it models preferences in a more expressive latent space. The BT model, on the other hand, cannot model cyclic preferences effectively, resulting in lower accuracy. We have ensured that there is no information or data leakage in our experiments. The dataset used for testing was kept separate from the training data. However, we acknowledge that the cyclic preference dataset used is relatively small (around 200 samples), which might contribute to high accuracy. To address this, we plan to construct a larger and more diverse cyclic preference dataset to further evaluate and validate the model's performance.

Thank you again for your valuable feedback and for recognizing the contributions of our work. We appreciate your consideration and hope that our clarifications might encourage you to view our submission even more favorably.

Dear reviewer, thank you for your thoughtful review and for recognizing the strengths of our work. We appreciate your constructive feedback and the opportunity to address your concerns.

1. Complexity of Training and Tuning Processes

Thanks for your feedback. We understand your concern that introducing a latent space and preference representation learning might add complexity to the model. However, in practice, the changes to the training procedure are minimal. Our General Preference Model (GPM) replaces the scalar output in the reward model's (RM) training with a vector output. When the embedding vector dimension is small (e.g., 2, 4, 6, or 8, as used in our experiments), the training process remains very similar to standard RM training.

The added components, such as the eigenvalue scale gate and the eigenvector embedding head, do not significantly increase computational complexity or require extensive tuning. The model can be trained using the same frameworks and optimization techniques as traditional RMs, ensuring ease of implementation and scalability.

2. Interpretability of Latent Space Embeddings and Preference Scores

We appreciate your point regarding the interpretability of the latent space embeddings. On the contrary, our model provides opportunities to understand and analyze the preference structures more deeply. By embedding responses into a latent space, the GPM allows the preference model to automatically discover different components within human preferences.

As detailed in Section 4.2 (originally presented in Appendix A.2) of our paper, the eigenvalue scale gate computes context-dependent scaling factors based on the prompt, adjusting the importance of various aspects such as helpfulness, instruction-following, and creativity. The eigenvector embedding head generates embeddings that capture nuanced characteristics of the responses relevant to human preferences.

This structure enables the model to adapt dynamically to varying user preferences and makes it possible to analyze the embeddings to understand which aspects are being prioritized in different contexts. Therefore, the GPM enhances interpretability by providing a richer representation of preferences.

3. Comparison with Other State-of-the-Art Methods

Thank you for suggesting a more comprehensive comparison. We have extended our experiments to include additional state-of-the-art (SOTA) models to demonstrate the effectiveness of GPM across different architectures and sizes. Below is the updated table comparing our GPM with the Bradley-Terry (BT) models and other SOTA models on RewardBench.

Table: Comparison between BT Models, GPM, and Other SOTA Models on RewardBench

• Across all base models, GPM consistently outperforms BT RM. For example, with Gemma-2B-it, GPM achieves an overall score of 77.38%, which is a 9.11% improvement over BT RM.
• In addition, our model with sizes 8B and 9B surpassed previous SOTA methods including RLHFlow/ArmoRM-Llama3-8B-v0.1, Nvidia/Nemotron-4-340B-Reward, OpenAI/GPT-4-0125-preview and Google/Gemini-1.5-pro-0514.
• Our Model's Strengths:
  • Our GPM achieves competitive performance using smaller, more efficient models (e.g., 2B and 8B parameters), highlighting its effectiveness in resource-constrained settings.
  • Our work specifically addresses the challenge of modeling cyclic and complex preferences, which is less explored in existing models. This focus contributes to advancements in understanding and handling intricate human preferences.
  • The ability of GPM to improve performance across various model sizes demonstrates its scalability and adaptability.

Thank you for your response. I have reread the article and believe this work has value for the research community. I will keep my score.

2.a. Training and evaluation datasets for RMs in Table 1

A: Thank you for your feedback. Indeed, the Reward Models (RMs) in Table 1 were both trained and evaluated on the same cyclic preference dataset constructed from Ultrafeedback. We have updated the main text to explicitly state this, ensuring transparency in our experimental setup. This clarification helps in understanding the context and significance of the results presented.

2.b. Training of BT RM

A. Thanks for your attention. The BT RM was trained using standard reward model training configurations, employing the logistic sigmoid loss function $\mathcal{L} = -\log\sigma(r_w - r_l)$, where $r_w$ and $$r_l$ are the scalar rewards for the winning and losing responses, respectively. This method allows the BT RM to learn scalar rewards that model transitive preferences. We have revised our paper to include a detailed description of how the BT RM is trained, including the loss function and training settings.

2.c. Explanation of ''1st'' and ''2nd'' columns in Table 4

A: Thank you for bringing this to our attention. The “1st” and “2nd” columns in Table 4 represent training using SPPO and GPO methods, respectively. We have revised the caption of Table 4 to explicitly explain this, making the table self-contained and more understandable.

3. Under-discussed Experimental Results

A: We appreciate your feedback on the need for a more detailed analysis of our experimental results. Regarding the MT-Bench results in Table 4, we acknowledge that the gains achieved by GPM+GPO over SPPO+BT RM are marginal. This suggests that while our method shows significant improvements in certain benchmarks (e.g., AlpacaEval 2.0), its impact on MT-Bench is less pronounced. We now think that MT-Bench may not be the most suitable benchmark for highlighting the strengths of our approach, as it focuses on general language capabilities rather than alignment with complex human preferences. Therefore, we have moved the MT-Bench results to the Appendix as supplementary material. In the main text, we have revised our claims to more accurately reflect the experimental outcomes, emphasizing the gains observed in AlpacaEval 2.0.

Additionally, we have expanded our discussion of Tables 2 and 3 in the main text, providing a more thorough analysis of the results. We delve into why the GPM outperforms the BT RM in modeling general preferences (transitive and cyclic part) and how the embedding dimension affects performance. This enhanced discussion helps to contextualize our findings and reinforces the contributions of our work.

Q1. How is the von Neumann winner related to GPO?

A: As mentioned earlier, we realize that the introduction of the von Neumann winner may not clearly aid the understanding of GPO. Our GPO method builds upon the iterative framework using multiplicative weight updates, similar to the approach in Freund and Schapire (1999) of the form:

This iterative process allows our method to converge asymptotically, akin to Theorem 4.1 in Wu et al. (2024). We have moved the mention of the von Neumann winner and explanation of the convergence properties of GPO with established iterative optimization frameworks in Appendix C.

Q2. I am not sure what the bold font in Table 4 represents. Could you clarify?

A: Certainly. In Table 4, the bold entries indicate that the GPM outperforms the corresponding BT RM under the same training settings. This formatting is consistent with Tables 2 and 3. We have revised the caption of Table 4 to clearly explain the meaning of the bold font, ensuring that the table is self-explanatory.

Q3. How is BT RM trained?

A: The BT RM is trained using standard reward model training configurations. Specifically, we use the logistic sigmoid loss function $\mathcal{L} = -\log \sigma(r_w - r_l)$ to learn scalar rewards based on pairwise preferences, where $r_w$ and $r_l$ represent the rewards for the preferred and less preferred responses, respectively. This approach enables the BT RM to model transitive preferences effectively. We have added this explanation to the main text, and we hope it may provide readers with a clearer understanding of the training methodology for BT RM.

Q4. Line 1307 (Appendix B.2): “For the Bradley-Terry (BT) model, the temperature parameter β was set to 1, following standard practice (Rafailov et al., 2024)”. Can you clarify what “standard practice” refers to?

A: Thanks for your feedback. We have revised the citation and now reference Ouyang et al. (2022) ("Training language models to follow instructions with human feedback") and Christiano et al. (2017) ("Deep reinforcement learning from human preferences"), for standard reward model training practices.

Q5. Could you explain the experimental results on MT-Bench in Table 4 in more detail?

A: We appreciate your request for a deeper analysis of the MT-Bench results. As noted earlier, the improvements achieved by GPM+GPO over SPPO+BT RM in MT-Bench are marginal and within the margin of sampling variability. This suggests that our method may not offer significant advantages in this particular benchmark.

MT-bench is a multi-round conversation benchmark while our training data for both the preference model and LLM policy are primarily single-round question-answering or instruction following. The performance of these two different tasks may not be highly correlated. We believe that this phenomenon is common in language model alignment methods.

Since we have moved the key modeling technique parts (scale gate and embedding head) which consume 1 page, we have moved the MT-Bench results to the Appendix. We revised our claims in the main text to reflect the experimental findings accurately.

Once again, we sincerely thank you for your constructive feedback. Your insights have been instrumental in helping us improve the quality and clarity of our work.

Dear reviewer, thank you for your thoughtful and detailed review of our paper. We appreciate your recognition of the strengths and contributions of our work, as well as your valuable feedback on areas that need improvement. We have carefully considered your comments and have made substantial revisions to address your concerns. Below, we provide detailed responses to each of the points you raised.

1. a. The von Neumann winner and its relation to GPO

A: Thanks for your feedback. To clarify, the von Neumann winner concept arises from game theory and relates to finding optimal strategies in zero-sum games. In our context, the policy optimization problem can be framed as a two-player zero-sum game between policies, and the von Neumann winner corresponds to a policy that performs optimally against any opponent.

However, upon reflection, we realize that introducing the von Neumann winner may have caused unnecessary confusion in understanding GPO within a limited page. Therefore, we have moved the mention of the von Neumann winner from Section 5 to Appendix C, which now focuses on the iterative framework of GPO, explaining how it builds upon existing methods like multiplicative weight updates and how it converges asymptotically. Here is our newly added theorem (in Appendix C):

Consider the optimization problem defined by the GPO loss (Equation 12) and assume it is realizable. Let $\\{\pi_{\theta_t}\\}\_{t=1}^T$ denote the sequence of policies generated by GPO, and define $\bar{\pi}\_T = \frac{1}{T}\sum_{t=1}^T \pi_{\theta_t}$ as the average policy. Given that the preference score $s$ is bounded within $[-\rho, \rho]$, by setting $\beta = \Theta\left(\sqrt{T}\right)$, we have:

b. Structure and clarity of Section 5

A: Thank you for pointing this out. We have restructured Section 5 to improve clarity and focus. Specifically, we have moved the general review of previous work and discussions on computational efficiency to the Related Work section. This allows us to dedicate Section 5 entirely to introducing and explaining the GPO method. The revised Section 5 now begins with a clear formulation of the policy optimization problem and proceeds to detail our proposed solution without redundancy.

c. Key modeling techniques (Eigenvalue Scale Gate; Eigenvector Embedding Head) are in the Appendix rather than in the main text.

A: We agree that these components are crucial to understanding our General Preference Model (GPM) and should be presented prominently. We have moved the descriptions of the Eigenvalue Scale Gate and the Eigenvector Embedding Head from the Appendix into Section 3 of the main text. This includes detailed explanations of how these components function and their roles in enhancing the expressiveness and computational efficiency of the GPM. By bringing these details into the main text, we aim to provide readers with a complete understanding of our model’s architecture and capabilities.

Dear Reviewer,

Thank you for your follow-up comments and for acknowledging the improvements in clarity and presentation in our revised paper. We greatly appreciate your constructive feedback, which has helped us further refine our work.

We have carefully considered the points you raised and have made the following revisions to address your concerns:

1. Analysis of Results in Table 2 (Line 453).

A: We appreciate you bringing this to our attention. We have revised the text to more accurately reflect the experimental results. The updated passage now reads:

“The results are presented in Table 2. On RewardBench, using the Gemma-2B-it base model, GPM achieves an average score of 77.38%, which is an improvement of 9.11% over the BT model’s average score of 68.27%. Specifically, in the Chat task, GPM improves performance from 71.51% (BT RM) to 78.49%, and in the Chat-Hard task, from 64.69% to 66.23%.
For the Llama-3.1-8B-Instruct base model,  GPM achieves an average score of 91.86% (embedding dimension 6), representing a 1.30% improvement over the BT model’s average score of 90.56%. In the Chat task, GPM improves from 88.55% (BT RM) to 93.30%, and in the Chat-Hard task, from 85.75% to 86.18%.
Using the Gemma-2-9B-it base model, GPM achieves an average score of 92.33% (embedding dimension 4), showing an improvement of 0.87% over the BT model’s average score of 91.46%. In the Chat task, GPM boosts performance from 91.62% (BT RM) to 93.58%, and in Chat-Hard, from 85.96% to 87.72%.
These results indicate that GPM outperforms the BT model across various base models and tasks, particularly in the Chat and Chat-Hard tasks. Note that BT RM is a special case of GPM when the embedding dimension d = 1 (see Section 4).”

This revision acknowledges that while GPM generally outperforms BT, the improvements are more pronounced in certain tasks and base models, particularly for smaller models like Gemma-2B-it.

2. Discussion on Embedding Dimension Effects (Line 487).

A: We agree with your assessment and have revised the text to accurately describe the nuanced effects of embedding dimensions. The updated passage now reads:

“We conducted ablation studies to assess the impact of varying the embedding dimension in GPM.  As shown in Table 2, the performance of GPM varies with the embedding dimension. For the Llama-3.1-8B-Instruct base model, an embedding dimension of 6 achieves the highest average score of 91.86%, compared to 91.37% with dimensions 2 and 8. In the Chat-Hard task with the same base model, the highest score of 87.50% is achieved with an embedding dimension of 6, compared to 87.06% with an embedding dimension of 8.
Similarly, for the Gemma-2B-it base model, the highest average score of 77.38% is achieved with an embedding dimension of 8, showing an improvement over lower dimensions, such as 74.94% with dimension 4. Nevertheless, for some tasks and models, increasing the embedding dimension beyond a certain point does not yield additional benefits and may even lead to slight performance declines. These results suggest that the optimal embedding dimensions vary across different base models and tasks.”

This revision reflects the observed trends in the data.

3. Inclusion of More Performant Models in Table 2.

A: Thank you for this valuable suggestion. We have updated Table 2 to include more performant models, such as Llama-3.1-Nemotron-70B and Skywork-Gemma-2-27B-v0.2 (Top 2 models), to provide a more comprehensive comparison. We have also changed the heading to “Other Models Reported on RewardBench” to accurately reflect the content.

We have chosen these models for comparison because they are among the publicly reported results on RewardBench and provide relevant points of reference in terms of various model sizes and architectures, such as BT reward models (Skywork-Gemma-2-27B-v0.2, Llama-3.1-Nemotron-70B), PairRM (ArmoRM-8B), regression loss-based reward models (ArmoRM-8B), and generative reward models (GPT-4o, Gemini-1.5).

Furthermore, we have added a footnote to clarify our reproduced results:

“We report our reproduced results, which may be different from the officially reported results of Skywork-Reward-Llama-3.1-8B-v0.2.”

This transparency ensures that readers are aware of potential discrepancies between our reproduced results and the original reports, acknowledging that exact replication can be challenging due to differences in experimental setups (using different training codebases and different GPUs).

We hope that the updated manuscript meets your expectations and contributes valuable knowledge to the field.

Thank you once again for your thoughtful feedback. We hope our responses fully address your concerns. If so, we wonder if you could kindly consider raising your rating score? We will also be happy to answer any further questions you may have. Thank you very much!

Dear Reviewer, thank you for your thoughtful review and for recognizing the clarity and novelty of our work. We appreciate your constructive feedback and are glad to have the opportunity to address your concerns.

1. Practical Benefits of Capturing Cyclic Preferences

We understand your interest in seeing how modeling cyclic preferences can lead to better performance in downstream applications. Cyclic preferences are indeed common in complex human judgments, especially when multiple evaluation criteria conflict with each other.

In real-world scenarios, users often have conflicting priorities such as instruction following, helpfulness, honesty, and creativity. For example, a response that strictly adheres to instructions might lack creativity, while a more creative response might deviate from the instructions. This inherent conflict can lead to cyclic preferences.

Theoretical work by Balduzzi et al. (2019) has shown that every finite function game decomposes into transitive and cyclic components, suggesting that cyclic preferences are an inherent part of multi-criteria evaluations (Open-ended Learning in Symmetric Zero-sum Games, ICML 2019). Empirical studies, like the one by Lee et al. (2023), have also observed that annotator disagreements often arise from differing priorities among correctness, helpfulness, and style (Common Law Annotations: Investigating the Stability of Dialog System Output Annotations, ACL 2023 Findings).

To demonstrate the practical benefits of capturing these preferences, we conducted experiments on downstream tasks such as AlpacaEval 2.0. Our General Preference Model (GPM) consistently outperformed the Bradley-Terry (BT) model with different base models. As shown in Tables 3, 4, 5, and 6 of our paper, GPM achieved higher win rates, indicating better alignment with complex human preferences.

By effectively modeling cyclic preferences, our GPM enhances language model alignment, leading to improved performance on tasks where such preferences play a significant role.

2. Discussion and Analysis on Representation Vector Generation and Generalization

Thanks for your feedback. We agree that providing more discussion and analysis on the generation and generalization of the representation vector would strengthen our paper.

We already conducted ablation studies to assess the impact of key architectural components, particularly the eigenvalue scale gate and L2 normalization in the embedding head in the initial submission. Our findings indicate that generally removing the L2 normalization or the eigenvalue scale gate leads to performance degradation using base model LLama-3.1-8B.

We restate the results of ablation studies here for convenience:

Reward normalization, achieved through L2 normalization, is crucial for stable and effective learning. This aligns with findings in recent works, such as "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models" (https://arxiv.org/abs/2402.03300).

Regarding generalization, our method shows consistent improvements across both 2B and 8B models. While hyperparameter tuning can optimize performance, the underlying approach does not require specific recipes for different use cases, indicating good generalization capabilities.

To further demonstrate the robustness and generalizability of GPM, we conducted experiments using two additional base models: Gemma-2-2B-it and Gemma-2-9B-it. These experiments aim to show that our approach is effective across different model architectures and sizes.

Q2. Clarification on Preference Bias and Use of GPT-4-o-mini

Thanks for your feedback. You asked why we used GPT-4-o-mini as the evaluator to avoid preference bias when using GPT-4-turbo. Recent studies have identified that language models can exhibit self-bias when evaluating similar models. For instance, the paper "Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement" (https://arxiv.org/abs/2402.11436) discusses how models can favor responses similar to their own outputs.

To mitigate this bias, we used GPT-4-o-mini, which is less similar to the models being evaluated, providing a more neutral and fair assessment. Additionally, we provided results using GPT-4-turbo and DeepSeek-v2 evaluators, which show consistent improvements, reinforcing the robustness of our findings.

Q2. Results Length and Choice of Metrics

Thanks for your feedback. You inquired whether our results using GPM are generally longer than the baseline and why we focused on Win Rate (WR) over Length-Controlled Win Rate (LC WR) in Table 3.

Our GPM may produce slightly longer responses, but the performance gains are not solely due to increased length. The additional length often reflects more comprehensive and informative content, enhancing the overall quality of the responses.

Reward normalization, through L2 normalization in the embedding head, helps control output length and ensures stable training. This is consistent with findings in recent research emphasizing the role of reward normalization in improving performance.

Regarding the choice of metrics, both Win Rate (WR) and Length-Controlled Win Rate (LC WR) are important. WR captures overall preference, including content quality, coherence, and relevance, while LC WR mitigates length bias by controlling for response length. Our results show consistent gains in both WR and LC WR. The gains in LC WR, though slightly smaller, are still significant, demonstrating that our model’s improvements are due to better content quality, not just longer outputs.

We conducted further experiments to control for length bias by adjusting hyperparameters, such as the L2 normalization in the embedding head and the beta coefficient in the GPO loss function. The results, presented in the tables below, indicate that with proper tuning, we can manage output length without compromising performance.

Table 1: AlpacaEval 2.0 Results with Scale Gate and Embedding Head L2 Normalization

Table 2: AlpacaEval 2.0 Results Without Scale Gate and Embedding Head L2 Normalization

Table 3: Ablation Study of Coefficient Beta on GPT-4o-mini Evaluator

These tables show that with and without L2 normalization, our GPM still outperforms the BT model. In addition, we can adjust the hyperparameter $\beta$ to control the alignment strength and the average output length. We can control the output length through proper configuration, and performance improvements are not solely due to longer responses.

We appreciate your valuable feedback, which has helped us enhance our paper. We hope that our responses have addressed your concerns comprehensively. Given the strengths you’ve acknowledged in our work, we kindly ask you to reconsider your rating.

Dear Reviewer e2YS, we wanted to express our appreciation once more for your thoughtful review of our paper.

We have provided detailed responses to your comments and questions, and we hope they have addressed all your concerns. As there are still a few days left in the discussion period, we are eager to engage in any further dialogue that might help clarify our work.

Please feel free to let us know if you have any additional questions or if there are points you’d like to discuss further. Your insights are invaluable to us, and we are committed to ensuring that all your concerns are fully addressed.

Thank you for your time and consideration.

Dear Reviewer e2YS,

Thank you for your continued engagement. We have conducted additional experiments to improve the LC WR performance of GPM + GPO.

As presented in the previous response, we varied the hyperparameter β in the GPO objective function to control the alignment strength and the average output length. By tuning β, we can balance the trade-off between model performance and response length.

Implementing Length-Normalized GPO Objective:

To further address length bias, we conducted experiments using a length-normalized GPO (LN-GPO) objective, inspired by the length normalization term introduced in SimPO (https://arxiv.org/abs/2405.14734). The LN-GPO learning objective is:

Here are the preliminary results with β = 5:

These results indicate that with length normalization, GPM + GPO achieves higher Win Rates and comparable LC WR scores relative to BT RM + GPO.

We emphasize that length bias can be effectively controlled by tuning hyperparameters like β and incorporating length normalization into the GPO objective. By doing so, we can improve LC WR for GPM + GPO, demonstrating gains in both metrics.

We have included these updated results and analyses in the revised version of our paper to provide a comprehensive evaluation of our model under LC WR metrics.

We hope these updates address your concerns. Given your earlier positive remarks on the clarity and innovation of our approach, we kindly request that you reconsider your rating. Specifically, we feel the current rating of 3 may not accurately reflect the strengths of our work, as indicated in your comment:

 Strengths:
 This paper is well written and is very clear.
 This paper provides a novel approach by embeds responses into a latent space to capture intricate preference structures efficiently. The preference representation learning and skew-symmetric operators is innovative and well-suited for addressing limitations of traditional methods.
 Presentation: 3 good

We will also be happy to answer any further questions you may have. Thank you very much!

Dear Reviewer e2YS,

Thank you for your continued engagement and valuable feedback. We have carefully addressed your concerns and made substantial revisions to the manuscript. Here is how we have addressed your specific points:

1. Attributing Performance Gains to Capturing Cyclic Preferences:

   • We conducted ablation studies varying the embedding dimension k in our General Preference Model (GPM). The Bradley-Terry (BT) model is a special case of GPM when k = 1 modeling only transitive preferences.

   • By increasing k to higher dimensions, GPM captures cyclic preferences. We observed measurable performance gains as k increased from 1 to higher values, suggesting that the ability to model cyclic preferences contributes to improved performance.

   • This provides both quantitative and qualitative evidence supporting the causal relationship between modeling cyclic preferences and performance gains.

2. Demonstrating Gains Using Length-Controlled Win Rate (LC WR):

   • We implemented length normalization in our GPO objective (LN-GPO) to mitigate length bias. Our newly added experiments show that GPM + LN-GPO achieves higher LC WR and higher WR relative to BT RM + LN-GPO, demonstrating gains even when controlling for response length. We have included these results and analyses in the revised manuscript.

We believe these revisions comprehensively address your concerns. We kindly invite you to review our updated manuscript.

If you find that your concerns have been satisfactorily addressed, we would greatly appreciate it if you could consider adjusting your rating.

Thank you again for your time and thoughtful consideration.

Best regards,

The Authors

Dear reviewer, thank you for your thoughtful and detailed feedback on our submission. We appreciate the opportunity to address your concerns and clarify several points in our work.

1. Relevance of Cyclic Preferences in LLM Alignment

Thanks for your feedback. While annotation processes aim for consistency, real-world applications often involve multiple, conflicting criteria that can lead to intransitive (cyclic) preferences. For instance, consider the interplay between instruction following, honesty, truthfulness, and helpfulness:

• A response that strictly follows instructions might sacrifice helpfulness if the instructions are incomplete or misleading.
• Prioritizing honesty might lead a model to admit uncertainty, which could reduce perceived helpfulness or failure to follow instructions.
• Emphasizing truthfulness might result in blunt or overly detailed responses that are less engaging or helpful.

These conflicts aren't merely hypothetical. They reflect inherent challenges in aligning models with complex human values. Importantly, this phenomenon is theoretically underpinned by game theory. According to Theorem 1 by Balduzzi et al. (2019), "every finite function game decomposes into a sum of transitive and cyclic components." This means that in most cases, cyclic preferences are not just possible but inevitable (Open-ended Learning in Symmetric Zero-sum Games, ICML 2019).

Building on this, our own Theorems 4.4 and 4.5 (Theorem A.3) in the paper demonstrate that our method can effectively model both the transitive and cyclic components of preferences. By doing so, our approach is equipped to handle almost all possible preference structures, ensuring robustness in a wide range of scenarios.

Moreover, recent empirical studies support the presence of cyclic preferences in LLM outputs:

• Lee et al. (2023) found that annotator disagreements often stem from differing priorities among correctness, helpfulness, and style (Common Law Annotations: Investigating the Stability of Dialog System Output Annotations, ACL 2023 Findings).
• In large-scale deployments, aggregating feedback from users with varying backgrounds and needs can lead to non-transitive preferences.

Another cause of non-transitive preference probability can arise from a mixture of user groups. Say we have 1/2 of users with transitive preference: P(A>B)=1, P(B>C)=2/3, P(A>C)=2/3. The other 1/2 of users with transitive preference P(B>C)=2/3, P(C>A)=1, P(A<B)=2/3. The overall preference probability is: P(A>B) = 2/3, P(B>C)=2/3, P(C>A)=2/3, leading to cyclic preferences.

By explicitly modeling cyclic preferences, our method enhances the alignment of language models with nuanced and sometimes conflicting values inherent in human judgments.

2. Computational Complexities

Thank you for highlighting the need for precision regarding computational efficiency. You're correct that the BT model achieves an O(K) complexity when evaluating K responses and doesn't require embedding computations. We intended to emphasize that our General Preference Model (GPM) maintains this O(K) complexity while accommodating cyclic preferences—a capability that the standard BT model lacks without additional computational overhead. To clarify:

• BT Model: Efficient with O(K) complexity but limited to modeling transitive preferences.
• Pairwise Preference Models: Can handle cyclic preferences but typically require O(K²) complexity due to the need to evaluate all pairs.
• Our GPM: Achieves O(K) complexity similar to BT but extends expressiveness to handle both transitive and cyclic preferences efficiently. The compute overhead of embedding interaction is no more than pair-wise vector inner-product, which is negligible. We will revise the manuscript to accurately reflect this comparison, ensuring that we present a fair and balanced view of GPM's computational advantages without overclaiming.

3. Cyclic Criteria in the Ultrafeedback Dataset

We appreciate your request for a more detailed explanation of the cyclic preference criteria used in our ultrafeedback dataset. The cycle "instruction following ≻ honesty ≻ truthfulness ≻ helpfulness ≻ instruction following" was crafted to simulate realistic trade-offs that occur in human evaluations:

• Instruction Following vs. Helpfulness: A model may strictly follow instructions but provide unhelpful responses if the instructions are flawed or incomplete.
• Honesty vs. Truthfulness: A response might be honest about the model's limitations but not entirely truthful if it withholds certain information, and vice versa.

These criteria are not artificially constructed but mirror genuine conflicts that arise in complex tasks. By incorporating them into our dataset, we aim to evaluate how well models can navigate these trade-offs. To strengthen our justification, we will include an analysis of how these criteria align with overall user satisfaction scores.

4. Alignment Performance and Length-Controlled Win Rate

Thank you for emphasizing the importance of controlling length bias in evaluation metrics. We agree that the length-controlled (LC) win rate provides a more equitable assessment of model performance. In response to your suggestion, we will move the LC win rate results from Appendix Table 6 into the main body of the paper. We will update our discussion to highlight these results, acknowledging that while the performance gains of GPM over the BT model may be modest in some cases, the key advantage of GPM lies in its ability to model general preferences without sacrificing alignment performance.

We are grateful for your constructive feedback, which has helped us identify areas for improvement in our manuscript. Thank you again for your valuable insights and for helping us improve our contribution to the field.

Thank the authors for their detailed responses. I have read through the responses. However, I'm not satisfied with some points of the response. I hope the authors could carefully revise the paper.

• It would be good to just show some real examples to demonstrate the exists of cyclic preference in chatbot scenario. In my opinion, given the same condition (criterion, question and response candidates), it is hard to find response A is preferred over B, and B is over C, while C is preferred over A. This situation may only exist if the condition has been changed (e.g., user changes its requirement, would like a more safety response instead of helpfulness).
• In terms of the computational cost, the revised manuscript still emphasizes the previous reward models account for N^2 complexity, which implies that the proposed method has an advantage, which is overclaimed.
• Could you give me some examples from the ultrafeedback dataset? I would also like to the correlation between each aspect with the overall score.
• As the general alignment performance of GPM is not clearly better than existing reward models, can we imply that the cyclic preference issue is not prevalent in general scenarios? Can the authors provide more explanation or insights?

Dear reviewer, thank you for your continued engagement with our work. We appreciate the opportunity to have a constructive discussion here.

1. Real Examples Demonstrating Cyclic Preferences in Chatbot Scenarios

We understand your concerns about the presence of cyclic preferences in chatbot settings under consistent conditions. To illustrate that cyclic preferences can occur even without changing the criteria, we'd like to share examples from the Ultrafeedback dataset (https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized).

Example 1:

Prompt: "Provide a detailed history and significance of the Mississippi Arts Center in Jackson". We have three responses within the Ultrafeedback dataset:

Response A:

The Mississippi Arts Center, located in Jackson, Mississippi, is a vital cultural institution that has played a significant role in the state's arts scene for decades. It is a facility of the Mississippi Department of Archives and History and is home to the Mississippi Arts Commission, Mississippi Craft Center, Mississippi History Museum, and the Eudora Welty House Museum.
The Mississippi Arts Center was established in 1968 as the Mississippi Arts Commission. Its primary purpose is to support and develop the arts in Mississippi through a variety of programs and initiatives. Over the years, it has provided grants to arts organizations, schools, and individual artists; managed a state artist roster; developed arts education programs; and promoted cultural tourism in Mississippi.
The physical Mississippi Arts Center building opened in 2004 and houses various facilities and spaces for arts exhibitions, education, and programs. The Mississippi Craft Center is a featured part of the arts center, displaying and selling the works of local artisans and craftsmen. It also offers educational programs, classes, and workshops for both children and adults.
The Mississippi History Museum is another important component of the arts center. The museum explores the state's history from prehistoric times up to the present day, featuring exhibits on Mississippi's Native American culture, the Civil War, and the Civil Rights Movement.
In addition to these facilities, the Mississippi Arts Center is also home to the Eudora Welty House Museum, which is dedicated to the life and work of Eudora Welty, a renowned short-story writer and photographer from Mississippi. The museum features tours of her home, which has been restored to its original condition.
The Mississippi Arts Center in Jackson holds significant importance as a vital cultural institution and resource for both the local community and the state of Mississippi. It plays a crucial role in promoting the arts, preserving history, and providing educational opportunities related to Mississippi's rich cultural heritage. As a result, it has become an essential part of Jackson's cultural landscape and a must-visit for those interested in exploring Mississippi's unique history and arts scene.

Response A provides extensive information about the Mississippi Arts Center but mixes in details about other institutions like the Mississippi Arts Commission and the Mississippi History Museum, leading to partial inaccuracies.

• Instruction Following: 2 (Partially addresses the instruction but includes unrelated institutions)
• Honesty: 5 (Confident and detailed)
• Truthfulness: 3 (Some inaccuracies due to mixing institutions)
• Helpfulness: 4 (Informative but somewhat confusing)

Response B:

The Mississippi Arts Center (MAC) is a state-funded, non-profit art museum located in Jackson, Mississippi. It is the largest art museum in the state, and its mission is to "inspire creativity, enrich lives, and connect communities through the arts."
The MAC was founded in 1938 as the Mississippi Fine Arts Commission. It was originally housed in the Old Capitol building in Jackson. In 1976, the MAC moved to its current location in the former Woolworth's building on Capitol Street.
The MAC's collection includes over 15,000 works of art, including paintings, sculptures, drawings, prints, photographs, and decorative arts. The collection represents a wide range of artists from Mississippi and the American South.
The MAC also offers a variety of educational programs, including art classes, workshops, lectures, and tours. The MAC's Education Department serves over 100,000 people each year through its programs and outreach activities.
The MAC is a significant cultural institution in Mississippi. It is a place where people can come to learn about the arts, to create art, and to experience the power of art to enrich lives.
Here are some additional facts about the Mississippi Arts Center:
The MAC's building was originally a Woolworth's department store. It was designed by architect James Gamble Rogers, who also designed the Woolworth's buildings in New York City and Washington, D.C.
The MAC's collection includes works by some of the most important artists ...

Dear reviewer pX7m,

We appreciate your engagement with our work and have provided additional responses to your follow-up questions. As the discussion period is ending soon, we wanted to check if there are any remaining concerns we can address. We sincerely hope that with these revisions and clarifications, you might consider raising your score for our submission.

We’re keen to have a constructive discussion and look forward to your reply.