Axioms for AI Alignment from Human Feedback

Could you cite the places where you obtained the axioms of social choice theory?

We consider canonical axioms from the social choice literature. See, for instance, "The Handbook of Computational Soical Choice," Chapter 2. We are happy to add further references in a revision.

Since you did not include where these axioms are from, are you missing any investigations of other axioms? If so, why did you choose the ones you investigated and why leaved the others out?

Indeed, there are a great number of axioms in the social choice literature one could consider. Our choice of PMC and PO was due to these serving as very natural starting points for this kind of investigation. In particular, Pareto optimality is viewed as one of the most fundamental properties expected of social choice rules, and is satisfied by practically all reasonable social choice rules, whereas PMC related to Condorcet consistency, one of the most extensively studied axioms. We also considered several others (see some discussion of these in Appendix B). In any case, we believe that an extensive study of other social choice axioms is an important subject for follow-up work.

The sampling strategy over voters and candidates (prompt/responses) is far from obvious and not discussed in the paper. Any additional comments about how such a sampling strategy would look like would be appreciated.

We believe you are referring to this passage in the discussion: "However, the complete rankings are not necessary for computing this rule, rather, all we need to know are PO dominance relationships and majority directions. We can therefore apply the rule whenever we can determine this information at least approximately through, e.g., sampling."

We had the following model in mind: Each participant is shown a certain number of randomly selected pairs of candidates. We should be able to estimate all pairwise margins within a small error as long as each pair is compared at least $\Omega(\log m)$ times in expectation.

The only information needed to compute LCPO is for each pair of candidates in the dataset (i) which a majority of voters prefer and (ii) whether all voters have this preference. Determining these exactly is impossible with only approximate pairwise margins. However, running LCPO on the estimated margins would get an "approximate" LCPO winner, in the sense that the output ranking would be the winner on a profile differing on only a small fraction of voters.

What about non-linear reward function classes?

This is a great question. We first note that linear reward models are more general than they may first appear, as the "feature vectors" can be output embeddings of a pretrained LLM (and we just learn a linear layer of a reward model from pairwise comparisons). Consequently, even a linear model can in principle encode highly non-linear characteristics of the raw input (e.g., text).

Nonetheless, expanding beyond linear reward functions is an important direction for future work. A significant challenge is that once we consdier nonconvex model classes (e.g., even small multi-layer perceptrons), they seems difficult to analyze theoretically, as we now run into an issue of local optima and computational intractability of computing globally optimal solutions.

We appreciate your discussion of limitations. You raise excellent points on the challenges here (and some relate to discussions we have had amongst ourselves!). With regard to your direct questions:

Do you think the PO and PMC axioms (possibly along with majority consistency, winner monotonicity etc.) are in any sense the "gold standards"?

We view both PO and PMC as relatively minimal requirements, more so than majority consistency and winner monotonicity. For example, PO (at least in traditional social choice) is a basic property that is satisfied by essentially all natural rules. Nevertheless, we agree that many other axioms can be considered, and it is indeed not self-evident that enforcing PO outweighs the tradeoff of not respecting large margins. We do not view our axioms and results as the final word. Rather, we hope the paper offers a starting point for having such conversations.

Could there be other similarly reasonable axioms that are contradictory with PO/PMC?

This seems quite plausible. We were quite surprised by the fact that PO and C1 are incompatible given how easy they are to achieve in traditional social choice. It does suggest that there may be other strong impossibilities yet to be discovered.

I appreciate the authors' response. The authors' replies to my questions are reasonable, and I strongly encourage the authors to include these discussions in the paper. I will leave my original score unchanged.

Could the authors comment on how only trying to recover part of the full ranking would impact their results?

We believe our work addresses this concern. The prompt/response datasets are derived from sampling an LLM, ensuring the corresponding feature vectors are already in a reasonable region of feature space simply by virtue of being in the dataset. Our axioms are designed to guarantee good performance specifically on these vectors.

We could have instead chosen even stronger notions requiring the conditions to hold across the entire feature space. For instance, PO could be defined such that for any two feature vectors $\mathbf{x}, \mathbf{x}' \in \mathbb{R}^d$, if all voters prefer $\mathbf{x}$ to $\mathbf{x}'$, then $\theta^*$ should as well. This is (i) impossible to achieve with trivial counterexamples, and (ii) unnecessarily stringent for precisely the reasons you mentioned.

This idea extends to RLHF beyond LLMs, as long as the dataset defines a reasonable region for the reward approximator to perform well in.

The proposed social choice-based rule seems to me to be a bit artificial and only created to enforce Pareto optimality. It is not clear if such rule could easily be implemented in RLHF for instance and obtained via learning. For instance, in RLHF, this rule would require to consider all trajectories, which is impractical for any complex problem. [...] Could you the authors comment on their proposed rule could be implemented in practice?

We believe that the problem you're alluding to may not be an issue. In our desired use case, all "trajctories" are sequences of tokens which can in principle be embedded into a fixed-dimension feature space. In existing implementations, we are given a dataset of pairwise comparisons over token sequences which are interpreted as feature vectors. In common variants of RLHF, a reward function is learned to maximize the Bradley-Terry (BT) likelihood over a dataset of such comparisons. LCPO is just an alternative to this step, where instead of maximizing likelihood we solve a small number of linear programs. Just as in the case of BT, LCPO can be used on a dataset which contains only a small subset of possible pairwise comparisons. Once the reward function is obtained using LCPO, it can be used for reinforcement learning in precisely the same way as the standard RLHF pipeline.

I feel that the linear social choice paradigm may be too restrictive. I believe that the authors focus on linear models, because this is indeed a natural machine learning model. However, I don't think it is necessary to assume that the rankings of voters (i.e., humans in AI alignment) have to be linearly representable. In my opinion, the paper would be stronger if the results were presented without this latter assumption.

While the linearity assumption is admittedly a limitation, we note that linear reward models are more general than they may first appear, as the "feature vectors" can be output embeddings of a pretrained LLM (and we just learn a linear layer of a reward model from pairwise comparisons). Consequently, even a linear model can in principle encode highly non-linear characteristics of the raw input (e.g., text).