Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching

We thank Reviewer rouN for the comments and questions. Below we provide our responses.

---

Q1. However, I do not understand why one would consider a payoff function other than the identity one.

A. Thank you for the question. First and foremost, we would like to clarify that the main form of $\Psi$ we consider—emphasized in the introduction (lines 70–71)—is:

$$\mathcal{P}_\theta(y \succ y') = \Psi(\mathcal{P}(y \succ y')),$$

where $\mathcal{P}_\theta$ denotes the empirical preference model used in practice.

One key practical motivation for studying non-identity transformations $\Psi$ is to analyze how empirical preference models used in real-world NLHF implementations (which may be distorted or biased) affect the outcome of alignment. Our theoretical framework allows us to examine the implications of using such transformed preferences and to identify conditions under which desirable alignment properties are preserved or violated.

In practice, we do not have access to the ideal ground-truth preference model $\mathcal{P}$. Instead, we rely on finite data and limited optimization to train an empirical model $\mathcal{P}_\theta$, which inevitably introduces bias and noise. As a result, although the identity map of a ground true preference model in our framework satisfies properties such as Condorcet consistency, Smith consistency, and diversity considerations in theory, it remains unclear whether these properties hold when the empirical model replaces the ideal one. To address this, we introduce a general mapping \Psi that represents the transformation from the ideal preference model to the empirical one:

Specifically, if we treat $\Psi$ as a random function, then $\Psi(\mathcal{P}(y \succ y’))$ serves as a stochastic estimate of the ideal preference, denoted by $\mathcal{P}_\theta(y \succ y’)$, thereby capturing the practical uncertainties inherent in NLHF.

In addition, our analysis also applies to scenarios where an additional mapping is introduced into the NLHF framework. Therefore, our results provide a general theoretical framework for analyzing such extensions.

---

Q2. For this reason, I cannot recommend the paper for acceptance without a strong argument, or an example, or a simulation, of alternative payoff functions that can be useful in practice.

A. This framework enables us to analyze practical NLHF and derive several key insights:

• To achieve Condorcet consistency, the requirement on the empirical preference model $\mathcal{P} _\theta$ is relatively mild. It only needs to preserve correct pairwise comparisons. Specifically, if the true preference model satisfies $\mathcal{P}(y \succ y') > 1/2$—meaning that more than half prefer response $y$ over $y'$—then the empirical model should also satisfy $\mathcal{P} _\theta(y \succ y') > 1/2$. Importantly, the exact proportion of preference is not required to be highly accurate. This indicates that some degree of bias in $\mathcal{P} _\theta$ is acceptable, as long as the direction of pairwise preferences is preserved, highlighting the robustness of practical NLHF.

• When there is noise in the empirical preference model, it becomes difficult for practical NLHF to satisfy all the desired properties because when $\mathcal{P}(y \succ y’)$ is close to 1/2, even small amounts of noise can easily lead the empirical model to make incorrect pairwise comparisons. In such cases, although the empirical preference model may still be a good approximation of the ideal one—with only minor noise—it can nevertheless cause NLHF to fail in satisfying all these properties. This makes us consider several modifications to solve this issue:

  • If the pair $(y, y^\prime)$ corresponds to a preference value close to $1/2$, we exclude it from the alignment process.

  • We can incorporate regularization terms into the training process of the preference model to encourage it to output higher values when the true preference exceeds $1/2$, and lower values when it falls below $1/2$.

  • If we assume that the empirical preference model closely approximates the true preference and only contains noise, we can use Bayesian inference to estimate the probability that the true preference exceeds $1/2$, given an empirical value near $1/2$. Based on this estimate, we can adjust the preference values to increase the likelihood of making correct pairwise comparisons.

Exploring practically useful modifications and developing efficient alignment methods that incorporate them is left for future work.

Moreover, our general framework introduces additional payoff functions for game-theoretic LLM alignment, enabling the achievement of several desirable properties.

---

Q3. Could you elaborate on lines 201–202, where you state that continuous mappings are commonly encountered in practice? This may not hold true, for example, when there are few data points.

A. The continuity referred to here is a property of the mapping $\Psi$ itself and is independent of the number of data points. Even with limited data, the mapping can be defined to be continuous.

---

Q4. In line 259, do you mean to say "human preferences does not satisfy"? Is this a typo?

A. Thank you so much for this valuable question! This is not a typo, but it might be confusing here. We have revised the sentences as follows in our manuscript: When human preference does not satisfy BTL model, this formula can be viewed as a natural generalization of standard RLHF. And our results imply that this generalized formula satisfies Smith consistency.

---

Q5. Why does the symmetry of the payoff matrix 5.2 make it difficult to interpret? I do not understand your comment on line 326.

A. Here, symmetry refers to the condition $\alpha_{ij} = \alpha_{ji}$. However, we interpret $\alpha_{ij}$ as the payoff when response $i$ beats response $j$, and $\alpha_{ji}$ as the payoff when response $j$ beats response $i$. From this perspective, increasing $\alpha_{ij}$ should correspond to decreasing $\alpha_{ji}$. For instance, they are typically expected to satisfy an anti-symmetry condition such as $\alpha_{ij} + \alpha_{ji} = 1$. Therefore, we find this construction unintuitive and difficult to interpret.

---

Q6. You show in your impossibility result that preference matching is impossible. Could you think of how it is possible to achieve this approximately, with some relaxation? You could mention this in the discussions.

A. Incorporating regularization terms into the training objective may be a promising direction for enabling preference matching. Exploring specific strategies to overcome the current impossibility results is left for future work.

---

Q7. Minor typographical errors … The no-tie assumption is pretty significant. Even an individual labeler, nevermind in aggregate, may wish to indicate a tie on a specific pair of responses especially if they are very similar. I believe this assumption should be highlighted in the limitations section. Further, the work is purely theoretical. A potential avenue for future work, which should be mentioned in the discussions, is to perform simulations with different payoff functions to observe their effect on (the ease of) identifying the Condorcet winner, Smith set, or capturing diversity in human preference.

A. Thank you very much for pointing out the typos and highlighting important limitations. We have corrected the typos and added a more detailed discussion of the no-tie assumption, along with additional future directions, to the limitations section of our manuscript.

We thank Reviewer 7tZ6 for the comments and questions. Below we provide our responses.

---

Q1. The motivation for generalising NLHF is unclear. What problem is this paper trying to solve? Why do we want to generalise NLHF with a mapping? Does this provide better understanding about performance guarantees in practice? The paper mentioned PO, but the motivation of introducing is very clearly articulated in that paper as a generalisation of RLHF and DPO.

A. Thank you for the question. First and foremost, we would like to clarify that the main form of $\Psi$ we consider—emphasized in the introduction (lines 70–71)—is:

$$\mathcal{P}_\theta(y \succ y') = \Psi(\mathcal{P}(y \succ y')),$$

where $\mathcal{P}_\theta$ denotes the empirical preference model used in practice.

One key practical motivation for studying non-identity transformations $\Psi$ is to analyze how empirical preference models used in real-world NLHF implementations (which may be distorted or biased) affect the outcome of alignment. Our theoretical framework allows us to examine the implications of using such transformed preferences and to identify conditions under which desirable alignment properties are preserved or violated.

In practice, we do not have access to the ideal ground-truth preference model $\mathcal{P}$. Instead, we rely on finite data and limited optimization to train an empirical model $\mathcal{P}_\theta$, which inevitably introduces bias and noise. As a result, although the identity map of a ground true preference model in our framework satisfies properties such as Condorcet consistency, Smith consistency, and diversity considerations in theory, it remains unclear whether these properties hold when the empirical model replaces the ideal one. To address this, we introduce a general mapping \Psi that represents the transformation from the ideal preference model to the empirical one:

Specifically, if we treat $\Psi$ as a random function, then $\Psi(\mathcal{P}(y \succ y’))$ serves as a stochastic estimate of the ideal preference, denoted by $\mathcal{P}_\theta(y \succ y’)$, thereby capturing the practical uncertainties inherent in NLHF.

In addition, our analysis also applies to scenarios where an additional mapping is introduced into the NLHF framework. Therefore, our results provide a general theoretical framework for analyzing such extensions.

---

Q2. The analysis for preference matching in Section 5 seems detached from the two consistencies introduced in the earlier sections of the paper.

A: Thank you for the question. Let us clarify the connection between preference matching and the two consistency properties discussed earlier in the paper. Our analysis is organized around two hierarchical goals in human preference alignment:

Primary Goal of Alignment: Aligning with majority preference:

To improve the overall performance of LLMs, it is critical to align the model with the most preferred response—specifically, the Condorcet winner (if it exists), or more generally, the Smith set. This is why we focus on whether NLHF satisfies Condorcet consistency and Smith consistency in Sections 3 and 4.

Secondary Goal of Alignment: Preserving minority preferences:

After aligning with the majority-preferred responses, a natural secondary goal is to ensure that minority preferences are not disregarded. This is where preference matching becomes important: it captures the idea that an aligned LLM should reflect the full distribution of preferences—including those of minority groups—rather than collapsing entirely to majority views.

In Section 5, we analyze preference matching as a formal property that supports this secondary goal. Although this analysis is distinct from Condorcet and Smith consistency, it complements them by addressing fairness and diversity in generation after majority alignment is achieved.

Below, we further clarify the connection between preference matching and preserving minority preference. First, we would like to clarify the distinction between two concepts of preference matching: the original concept, which is independent of a reward function, and the equivalent concept discussed in our paper.

Original concept of preference matching:

A policy $\pi$ is said to satisfy preference matching if

$\frac{\pi(y)}{\pi(y')} = \frac{\mathcal{P}(y \mid y')}{\mathcal{P}(y' \mid y)}.$

This definition does not rely on an explicit reward model. Its intuitive meaning is clear: for instance, if 70% of people prefer $y$ over $y'$, and 30% prefer $y'$ over $y$, then ideally, the LLM should generate $y$ and $y'$ with probabilities in a 7:3 ratio.

However, not all human preferences can be represented by a policy satisfying the above condition. In fact, it has been shown in previous studies that this is possible if and only if human preferences can be modeled by a BTL model. In that case, the original concept of preference matching is equivalent to the following formulation.

Equivalent concept of preference matching:

A policy $\pi$ is said to satisfy preference matching if

$\pi(y) = \frac{\exp(r(y))}{\sum_{y'} \exp(r(y'))},$

where $r(y)$ is a latent utility score associated with response $y$.

We agree that introducing the equivalent concept directly—without first discussing the original one—can be misleading. We have revised this part in the paper to improve clarity.

In our paper, we reflect this hierarchy of objectives. In Sections 3 and 4, we first address the primary goal—aligning the policy with the Condorcet winner and Smith set. Then, in Section 5, we consider a secondary yet important goal—ensuring alignment with minority preferences through preference matching.

---

Q3. Lack of empirical study and practical relevance. While this submission is clearly a theoretical paper, its theoretical claims should be demonstrated by empirical results. These would help show, for example, the benefits of satisfying the two consistency properties in Sections 3 and 4. Alternatively, the paper should at least demonstrate, in practice, what is at risk if these consistencies are violated. Otherwise, it is hard to assess the practical relevance of the findings.

A: Thank you for the thoughtful suggestion. We would like to emphasize that this line of work—studying the axiomatic foundations of aligning LLMs with human feedback—is inherently theoretical in nature. Several papers in this area have focused purely on theoretical contributions and have been accepted at NeurIPS. For example:

• [1] Noothigattu et al. (2020) propose axioms for learning from pairwise comparisons without empirical validation.
• [2] Ge et al. (2024) develop axioms for alignment from human feedback, again focusing on theoretical properties.

We appreciate the reviewer’s concern about practical relevance. However, our paper focuses on analyzing the theoretical properties of existing alignment approaches, rather than proposing a new algorithm. As such, empirical experiments are not essential for validating our contributions.

Given this context, we believe that pure theoretical work of this nature aligns with the NeurIPS standard.

References
[1] Noothigattu, R., Peters, D., & Procaccia, A. D. (2020). Axioms for learning from pairwise comparisons. NeurIPS, 33, 17745–17754.
[2] Ge, L., Halpern, D., Micha, E., Procaccia, A. D., Shapira, I., Vorobeychik, Y., & Wu, J. (2024). Axioms for AI alignment from human feedback. NeurIPS, 37, 80439–80465.

---

Q4. The empirical relevance of findings presented.

A. This framework enables us to analyze practical NLHF and derive several key insights:

• To achieve Condorcet consistency, the requirement on the empirical preference model $\mathcal{P} _\theta$ is relatively mild. It only needs to preserve correct pairwise comparisons. Specifically, if the true preference model satisfies $\mathcal{P}(y \succ y') > 1/2$—meaning that more than half prefer response $y$ over $y'$—then the empirical model should also satisfy $\mathcal{P} _\theta(y \succ y') > 1/2$. Importantly, the exact proportion of preference is not required to be highly accurate. This indicates that some degree of bias in $\mathcal{P} _\theta$ is acceptable, as long as the direction of pairwise preferences is preserved, highlighting the robustness of practical NLHF.

• When there is noise in the empirical preference model, it becomes difficult for practical NLHF to satisfy all the desired properties because when $\mathcal{P}(y \succ y’)$ is close to 1/2, even small amounts of noise can easily lead the empirical model to make incorrect pairwise comparisons. In such cases, although the empirical preference model may still be a good approximation of the ideal one—with only minor noise—it can nevertheless cause NLHF to fail in satisfying all these properties. This makes us consider several modifications to solve this issue:

  • If the pair $(y, y^\prime)$ corresponds to a preference value close to $1/2$, we exclude it from the alignment process.
  • We can incorporate regularization terms into the training process of the preference model to encourage it to output higher values when the true preference exceeds $1/2$, and lower values when it falls below $1/2$.
  • If we assume that the empirical preference model closely approximates the true preference and only contains noise, we can use Bayesian inference to estimate the probability that the true preference exceeds $1/2$, given an empirical value near $1/2$. Based on this estimate, we can adjust the preference values to increase the likelihood of making correct pairwise comparisons.

Exploring practically useful modifications and developing efficient alignment methods that incorporate them is left for future work.

Moreover, our general framework introduces additional payoff functions for game-theoretic LLM alignment, enabling the achievement of several desirable properties.

We thank Reviewer g65k for the comments and questions. Below we provide our responses.

---

Q1. The discussion on Condorcet and Smith consistency is interesting, but I do not see its broader implications. The widely used identity mapping already satisfies both properties, and it's unclear how this discussion provides practical insights or inspiration.

A. Thank you for the question. First and foremost, we would like to clarify that the main form of $\Psi$ we consider—emphasized in the introduction (lines 70–71)—is:

$$\mathcal{P}_\theta(y \succ y') = \Psi(\mathcal{P}(y \succ y')),$$

where $\mathcal{P}_\theta$ denotes the empirical preference model used in practice.

One key practical motivation for studying non-identity transformations $\Psi$ is to analyze how empirical preference models used in real-world NLHF implementations (which may be distorted or biased) affect the outcome of alignment. Our theoretical framework allows us to examine the implications of using such transformed preferences and to identify conditions under which desirable alignment properties are preserved or violated.

In practice, we do not have access to the ideal ground-truth preference model $\mathcal{P}$. Instead, we rely on finite data and limited optimization to train an empirical model $\mathcal{P}_\theta$, which inevitably introduces bias and noise. As a result, although the identity map of a ground true preference model in our framework satisfies properties such as Condorcet consistency, Smith consistency, and diversity considerations in theory, it remains unclear whether these properties hold when the empirical model replaces the ideal one. To address this, we introduce a general mapping \Psi that represents the transformation from the ideal preference model to the empirical one:

Specifically, if we treat $\Psi$ as a random function, then $\Psi(\mathcal{P}(y \succ y’))$ serves as a stochastic estimate of the ideal preference, denoted by $\mathcal{P}_\theta(y \succ y’)$, thereby capturing the practical uncertainties inherent in NLHF.

In addition, our analysis also applies to scenarios where an additional mapping is introduced into the NLHF framework. Therefore, our results provide a general theoretical framework for analyzing such extensions.

---

Q2.1. In game-theoretic LLM alignment, there is no explicit reward function, so it is unclear why the Nash solution should correspond to a preference-matching policy.

A: Thank you for the question. We would like to clarify the distinction between two concepts of preference matching: the original concept, which is independent of a reward function, and the equivalent concept discussed in our paper.

Original concept of preference matching:
A policy $\pi$ is said to satisfy preference matching if

$\frac{\pi(y)}{\pi(y')} = \frac{P(y \mid y')}{P(y' \mid y)}.$

This definition does not rely on an explicit reward model. Its intuitive meaning is clear: for instance, if 70% of people prefer $y$ over $y'$, and 30% prefer $y'$ over $y$, then ideally, the LLM should generate $y$ and $y'$ with probabilities in a 7:3 ratio.

However, not all human preferences can be represented by a policy satisfying the above condition. In fact, it has been shown in previous studies that this is possible if and only if human preferences can be modeled by a BTL model. In that case, the original concept of preference matching is equivalent to the following formulation.

Equivalent concept of preference matching:
A policy $\pi$ is said to satisfy preference matching if

$\pi(y) = \frac{\exp(r(y))}{\sum_{y'} \exp(r(y'))},$

where $r(y)$ is a latent utility score associated with response $y$.

We agree that introducing the equivalent concept directly—without first discussing the original one—can be misleading. We have revised this part in the paper to improve clarity.

---

Q 2.2. Moreover, I am not convinced that learning a preference-matching policy should be the primary goal in this setting. More explanation is needed to clarify this point.

A: Thank you for the question. We agree that the primary goal of training LLMs is to improve performance. However, other aspects are also critical in alignment. Since LLMs are generative models, maintaining generation diversity is an important consideration. An ideal alignment algorithm should achieve both high performance and diversity, ensuring that the model does not collapse onto majority preferences alone.

Preference matching plays a key role in this regard: it preserves generative diversity and protects minority opinions, which are often suppressed under majority-focused objectives.

In our paper, we reflect this hierarchy of objectives. In Sections 3 and 4, we first address the primary goal—aligning the policy with the Condorcet winner and Smith set. Then, in Section 5, we consider a secondary yet important goal—ensuring alignment with minority preferences through preference matching.

---

Comment 1. I find the overall contribution of the paper to be limited. The core challenge in RLHF lies in designing practical and effective algorithms for LLM post-training. However, the theoretical analysis presented in the paper offers limited insight into how such algorithms could be developed or improved.

A: Thank you for the comment. First, although our paper is primarily theoretical, it also offers practical insights. This framework enables us to analyze practical NLHF and derive several key insights:

• To achieve Condorcet consistency, the requirement on the empirical preference model $\mathcal{P} _\theta$ is relatively mild. It only needs to preserve correct pairwise comparisons. Specifically, if the true preference model satisfies $\mathcal{P}(y \succ y') > 1/2$—meaning that more than half prefer response $y$ over $y'$—then the empirical model should also satisfy $\mathcal{P} _\theta(y \succ y') > 1/2$. Importantly, the exact proportion of preference is not required to be highly accurate. This indicates that some degree of bias in $\mathcal{P} _\theta$ is acceptable, as long as the direction of pairwise preferences is preserved, highlighting the robustness of practical NLHF.

• When there is noise in the empirical preference model, it becomes difficult for practical NLHF to satisfy all the desired properties because when $\mathcal{P}(y \succ y’)$ is close to 1/2, even small amounts of noise can easily lead the empirical model to make incorrect pairwise comparisons. In such cases, although the empirical preference model may still be a good approximation of the ideal one—with only minor noise—it can nevertheless cause NLHF to fail in satisfying all these properties. This makes us consider several modifications to solve this issue:

  • If the pair $(y, y^\prime)$ corresponds to a preference value close to $1/2$, we exclude it from the alignment process.

  • We can incorporate regularization terms into the training process of the preference model to encourage it to output higher values when the true preference exceeds $1/2$, and lower values when it falls below $1/2$.

  • If we assume that the empirical preference model closely approximates the true preference and only contains noise, we can use Bayesian inference to estimate the probability that the true preference exceeds $1/2$, given an empirical value near $1/2$. Based on this estimate, we can adjust the preference values to increase the likelihood of making correct pairwise comparisons.

Exploring practically useful modifications and developing efficient alignment methods that incorporate them is left for future work.

Moreover, our general framework introduces additional payoff functions for game-theoretic LLM alignment, enabling the achievement of several desirable properties.

In addition, from a technical perspective, we introduce novel proof techniques that go beyond prior work. Specifically, we develop tools to analyze general non-symmetric games directly—without relying on the symmetry assumptions that underlie previous results in NLHF. We believe these techniques can serve as a foundation for future theoretical and algorithmic advancements in preference-based alignment.

---

Q3. A key motivation behind game-theoretic LLM alignment is to move beyond the Bradley-Terry model assumption and consider general preferences. Given this, why is it necessary to focus on the preference-matching policy and design a payoff matrix such that the Nash solution corresponds to it?

A: Thank you for the question. We respond to this point in part in our response to Question 2, but we are happy to elaborate further here.

First, we emphasize that NLHF is designed to handle general preferences, not limited to any specific parametric form such as the BTL model. The axiomatic approach studies the desirable properties that a solution should satisfy when certain conditions hold—this is common in the theoretical analysis of social choice and decision-making systems.

For example:

• In general preferences, a Condorcet winner may not exist. NLHF operates on general preferences, but when a Condorcet winner does exist, it is desirable for NLHF to return it. This property is referred to as Condorcet consistency.
• Similarly, in general preferences, preference matching policy may not exist. NLHF operates on general preferences without such policy. However, when the preference-matching policy exists (i.e., can be represented by a BTL model), it is desirable for NLHF to return the preference-matching policy.

This is why we study (though do not exclusively focus on) the performance of NLHF under the BTL assumption. Doing so allows us to characterize what alignment behavior should look like in idealized, structured settings—while still preserving the generality of the overall framework.

We thank Reviewer ua5U for the comments and questions. Below we provide our responses.

---

Weakness 1. This work is intended to be a fully theoretical paper that can guide... However, our theoretical results indeed provide practical implementation guidelines…

A. This framework enables us to analyze practical NLHF and derive several key insights:

• To achieve Condorcet consistency, the requirement on the empirical preference model $\mathcal{P} _\theta$ is relatively mild. It only needs to preserve correct pairwise comparisons. Specifically, if the true preference model satisfies $\mathcal{P}(y \succ y') > 1/2$—meaning that more than half prefer response $y$ over $y'$—then the empirical model should also satisfy $\mathcal{P} _\theta(y \succ y') > 1/2$. Importantly, the exact proportion of preference is not required to be highly accurate. This indicates that some degree of bias in $\mathcal{P} _\theta$ is acceptable, as long as the direction of pairwise preferences is preserved, highlighting the robustness of practical NLHF.

• When there is noise in the empirical preference model, it becomes difficult for practical NLHF to satisfy all the desired properties because when $\mathcal{P}(y \succ y’)$ is close to 1/2, even small amounts of noise can easily lead the empirical model to make incorrect pairwise comparisons. In such cases, although the empirical preference model may still be a good approximation of the ideal one—with only minor noise—it can nevertheless cause NLHF to fail in satisfying all these properties. This makes us consider several modifications to solve this issue:

  • If the pair $(y, y^\prime)$ corresponds to a preference value close to $1/2$, we exclude it from the alignment process.

  • We can incorporate regularization terms into the training process of the preference model to encourage it to output higher values when the true preference exceeds $1/2$, and lower values when it falls below $1/2$.

  • If we assume that the empirical preference model closely approximates the true preference and only contains noise, we can use Bayesian inference to estimate the probability that the true preference exceeds $1/2$, given an empirical value near $1/2$. Based on this estimate, we can adjust the preference values to increase the likelihood of making correct pairwise comparisons.

Exploring practically useful modifications and developing efficient alignment methods that incorporate them is left for future work.

Moreover, our general framework introduces additional payoff functions for game-theoretic LLM alignment, enabling the achievement of several desirable properties.

---

Q1. Can your results be extended beyond NLHF? For example, can your results be extended to GRPO?

A: In our analysis, we consider a general preference-based game-theoretic alignment method, where NLHF is a special example with \Psi being the identity. However, as we understood, although the reward can be interpreted as preference in Group Relative Policy Optimization (GRPO,[2]) as it is either 0 or 1, GRPO is inherently not a game-theoretic alignment method. Therefore, our results cannot be directly extended to GRPO.

---

Q2. Can you give specific examples of \Psi(t) and run the experiments to verify your theory

A: As noted above, \Psi being the identity is the traditional NLHF. Another special example of \Psi that is worthy to note is \Psi(t)=log(t/(1-t)), which is a natural game-theoretical extension of traditional RLHF, as we pointed out in line 258-259 in the paper. As all our theorems are proven with mathematical rigor, there is no need to run experiments to verify the theorems. Nevertheless, we recognize that testing the empirical performance of different \Psi on language models is important, but it is beyond the scope of this theoretical paper.

---

Q3. Recent work [1] has shown that preference embedding, an approach that embeds responses into a latent space to capture intricate preference structures can handle preference cycles (e.g., the Condorcet cycle discussed in your paper). Do your theoretical results, particularly the impossibility of preference matching, have any implications for this work? For instance, does your analysis suggest inherent limitations even in the preference embedding model, or could your analysis be adapted to accommodate such generalizations?

A: Thank you for your insightful question. In [1], the authors proposed the General Preference embedding model (GPM) for modeling preference, which can then be used in preference-based LLM alignment, including non game-theoretical approaches (like \PsiPO or the GPO proposed in this paper), and the game-theoretical approach discussed in our paper.

Theorem 4.2 in our paper, as discussed in line 266-269, suggests that the preference model used should be skew-symmetry to guarantee Smith consistency and previous preference models like PairRM are not skew-symmetry. The GPM in [1] is designed to be skew-symmetry, therefore our result implies that using GPM is better than PairPM in preserving Smith consistency.

Theorem 5.1 (the impossibility of preference matching) applies to any game-theoretical alignment method, regardless of the preference model used. Therefore if we use the GPM model and a game-theoretical alignment method, this limitation will persist. We have added the above discussions into our paper. Thank you again for bringing [1] to our attention.

References

[1] Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment, ICML 2025. [2] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.