Selective Preference Aggregation

Response Thank you for your feedback! We address them and include tables at https://tinyurl.com/2ybsfs95

Is the focus on the algorithmic framework for selective aggregation, or on its application to a specific problem domain?

The primary contribution is the proposed algorithmic framework. We will revise the text to state this. The explored problem domains serve as illustrations of the framework's capabilities in different contexts.

The approach of simply abstaining from comparisons does not address situations where a complete ranking is essential …

We appreciate this concern. Many relevant real-world tasks (e.g. RLHF, content moderation, search ranking) do not require total orders. When necessary, the dissent parameter can be raised to 0.5, and further until graph disconnect (with loss of guarantees). If there is still no total order, there is no majority preference between certain items or a cycle is formed. If a total ordering is truly necessary, other methods can be applied within tiers to produce local complete orderings. This process distinguishes comparisons supported by the algorithm's guarantees (between tiers) from those less well-founded. Online, the approach highlights where more information could resolve disagreement. SPA serves as a robust first step: identify consensus comparisons first, then apply targeted methods (experts, more users) where disagreement persists. We’re happy to add this to the text.

The paper glosses over the trade-offs involved in choosing the dissent parameter.

We have several strong points to consolidate a new subsection in section 3 (Algorithms), which are already in the paper: The path algorithm (Appendix) allows us to avoid preselecting a tau value. Importantly, computing all possible rankings for any $\tau$ < 0.5 incurs no additional asymptotic cost.

Theoretical guarantees can guide its selection. The discussion starting at line 281 gives an example of how we might select tau based on assumptions about noise or missing preferences.

There is little discussion of alternative robust ranking methods, and key references (e.g., for baselines like ORPO or CPO) are not adequately introduced when they first appear.

We're happy to add discussion of ORPO/CPO/other methods. ORPO focuses on alignment during the training process. SPA is less constrained - it may be possible to use it in tandem; SPA could filter for high-quality responses with ORPO used for fine-tuning.

CPO more closely resembles SPA, but its purpose is to contrast “near-perfect but flawed translations”. While comparing top tiers in SPA could have a similar purpose, our method captures preferences across all items. As a result, we feel rank aggregation methods are more appropriate baselines.

The selective rankings are claimed to be more robust and transparent … yet the overall performance differences are marginal and not convincingly argued.

We appreciate this and will highlight SPA's benefits. SPA provides practical benefits through transparency and robustness. It reveals underlying consensus strength (e.g., Sushi's weak majority via path, App D.3). Existing methods obscure this. As noted, SPA offers stronger robustness guarantees (see dGCS). SPA exhibits no inversions while others vary significantly - this stability is crucial in domains like RLHF, preventing reward signals from flipping due to small sample changes. SPA consistently yields lower disagreement rates (Table 1) compared to baselines (often 0-6% for SPA vs. 4-12%+ for baselines).

The overall novelty is limited; the idea of abstaining from forced arbitration is not revolutionary and may offer little benefit in many practical applications. The main contribution, the idea of using selective rankings to “agree to disagree” is not clearly motivated ….

Beyond abstaining, SPA reveals preference strength, crucial for trustworthy AI. For instance, methodologies aiming to learn from demonstrations [Brown et al. 2019] or generative reward models [Mahan et al. 2024] use steps with simple majority rules. Our rankings are always supported by a majority of users, clarify where preferences conflict, remain stable under noise, and can be adjusted. Our response to sHCm further details future use cases in ML workflows like RLAIF and personalized models.

• Mahan, et al. "Generative reward models." arXiv preprint arXiv:2410.12832 (2024).
• Brown, et al. "Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations." ICML. PMLR, 2019.

More extensive and realistic experiments are needed ….

We appreciate the feedback. Table 8 (in our response) shows performance in scenarios with and without consistent user preferences. Only SPA reveals user contradictions. See also our responses to dGCS (ablation) and fgRQ (dropped comparisons).

We hope that our response resolves any misunderstandings! Thank you again for your time, and we look forward to resolving any remaining concerns that you may have.

Thank you for your time and feedback! We appreciate your feedback and your detailed suggestions for improvement, including further datasets to improve our work. We provide tables at https://tinyurl.com/2ybsfs95

However, the experiment in Section 6 is less compelling … The reported prediction error should then correspond to the evaluation set. In our revised setup, we use only a subset of users and items (80/20 train/test split) to create rankings for the given items. We switch to using a pre-trained model (bert-mini) as a starting point, and replace pairwise majority with the Copeland method for consistency with our experiments. We then note the total per-user error, as well as how well a model trained on these rankings generalizes to new users in Table 10.

We also provide a table of model generalization to new items (with train users), new users (on train items), and new users on new items. We binarize to pick the threshold with the Maximum TPR s.t FPR is capped to 10% (Table 11). We hope to spend further time exploring other pre-trained setups and architectures to ensure higher performance for test set items and users, and to demonstrate SPA’s performance advantages with other choices of threshold.

Also, I am not sure I fully understood how SPA was adapted to Section 6.

We made an error in our description of the adaptations, now corrected. DICES uses annotators who rate an item toxic/non-toxic, but the level of toxicity beyond that is not clarified. In our new setup, in order to avoid excessive (and incorrect) levels of “ties”, we have only included preferences where there is distinction (toxic vs non-toxic). We then scale the weights of each preference pair to the same total weight, to make each item-pair equally important.

How is "distinctions among conversations" measured?

“Distinctions between conversations” is used to specify the greatest number of tiers (greatest comparability). We have added text to our manuscript to make that distinction clear. There are several tiers created - the chosen threshold determines which are grouped under toxic/non-toxic.

The definition of label error (line 420, left column) introduces a variable t … Could the authors clarify?

$t$ in this instance represents the majority threshold, at which point a majority of users rate a conversation toxic. We have revised the text to make this clear as well.

Finally, I am confused about the use of the word "consensus" throughout the paper. In plain English, "consensus" seems synonymous with "quasi-unanimity", which suggests using . But this is not how it seems to be used, e.g. lines 323, right column, or line 420, left column.

We agree that the term was not well-defined and should be clarified. In this setting, we use “consensus” to refer to the true majority vote of annotators.

Additionally, one interesting feature of pol.is … especially when the communities are not predefined, and have instead to be learned)?

This is a fascinating direction for potential future work! We take the question to mean that we want to find cases where multiple distinct communities agree (please correct us if needed!). One could imagine applying this approach hierarchically in such a setting - finding a selective preference aggregation for each community individually, then treating each of those tiered rankings as an individual “judge” and finding a preference aggregation across communities. If communities need to be learned, one could imagine assigning each individual a weight based on an estimated probability of belonging to each community (see our response to reviewer dGCS for further details on adding weights to our approach). SPA could also be used as part of the identification process; identifying groups of users who agree with each other in areas where overall disagreement is high (within a tier, for instance) can be used to help determine these communities.

I would also be interested in the authors' thoughts on how SPA could be adapted for highly sparse comparisons, as is the case in RLHF and recommendation AIs (see e.g. [3]).

In future work, we plan to explore different ways of adapting SPA for use in larger/sparser datasets. The most straightforward is to build on existing work using LLMs to make judgments (RLAIF). We could use multiple LLMs to make judgements to avoid amplifying model biases; one could also use lightweight models and “recruit” larger models only when preferences conflict to save on computational cost.

SPA could also be used with modeling. SPA could be used to limit uncertain responses from existing models, or could be used to help create models that better model users. Comparisons within tiers could be used to train a base model for accuracy, and identified groups (see previous response) could be used to train personalized models with high accuracy for each group.

We hope our responses clarify your points! Thank you for your feedback and suggestions.

Response Thank you for your time and feedback! We include supplementary tables here: https://tinyurl.com/2ybsfs95

The authors claim that the algorithm is fast and scalabile however this is not convincing … … 60-100 seconds for 500 items seems rather slow.

As you’ve noted, our empirical runtimes were slower than our asymptotic analysis suggests. That is due to some inefficiencies that we have since resolved in our implementation. We provide a table of updated runtimes (in seconds) in Table 6 and have updated the figure in our manuscript. (Note that the naive method remains slow because it does not leverage the path approach’s tricks.)

We’re happy that you agree our proven asymptotic runtime of O(n^2*p) is reasonable. Indeed, many competing methods can be quite computationally intensive - for instance, Kemeny-Young is NP-Hard and must be solved with integer programming in many contexts. Our algorithm’s asymptotic scaling - which is linear in the number of input pairwise preferences - gives a strong characterization of our algorithm’s scalability, independent of any quirks in implementation or hardware.

Often, there are (of the order of) n*log n pairwise comparisons available…, for missing pairwise comparisons for bigger datasets.

We appreciate your point regarding the lack of missing pairwise comparisons. We provide Table 7 highlighting the number of comparisons at $\tau$_max with 5% of all pairwise comparisons dropped. We also refer you to values of Δ-sampling, which serve as a median change with 10% of samples dropped, equivalent to 10% missing comparisons.

SPA_0 and SPA_min very often do not produce actual rankings …

To clarify, we do feel these values (even when creating minimal tiers) highlight key information. At SPA_0, any tiers > 1 indicate unanimous preference among users. The dissent rate at which we see SPA_min can be informative as well - in certain datasets such as Sushi, this value is high (> 0.4), which reveals high levels of underlying disagreement. This information can also be found essentially for free, since Algorithm 2 gives these tiered rankings as part of the process of finding more granular rankings like SPA_max.

The datasets are way too small …

We appreciate the reviewer's suggestion. Existing larger datasets are generally sparse; we discuss future adaptations with reviewer sHCm that would enable SPA. We note that the number of potential pairwise preferences processed in existing scenarios is already substantial - over 7.5 million for DICES (Section 6) – demonstrating SPA's ability to handle a considerable volume of preference data. SPA’s design and theoretical guarantees (Section 4) ensure predictable behavior at any scale. Several applications remain at small values of n; reviewer dGCS points out potential applications of SPA in RLHF at n < 5, and we note applications that exist at the current scale. If the reviewer wants further clarity on larger scale data, we are willing to conduct experiments on synthetic data.

The distinction with ranking with ties is not very clear … this distinction theorically.

We use 'incomparable' when no strict pairwise preference exists between items. Items in the same tier are thus 'incomparable', which can arise from various situations like cycles, evenly split preferences, or judges explicitly marking items as equivalent (see Fig 3). Therefore, 'incomparable' (in the same tier) does not necessarily mean 'equivalent'. For example, if ⅔ of judges think A > B, ⅔ think B > C, and ⅔ think C > A, that does not necessarily mean A = B = C. Perhaps 0 judges think any of those items are equivalent. Users can state equivalence, and our ranking considers it. However, the distinction between abstention/disagreement and asserted equality is a key benefit within our tiers.

Also the empirical analysis should benefit from a direct comparisons with ranking with ties methods.

Regarding comparison to tie-handling methods: standard baselines like Borda or MC4 do produce ties when scores are equivalent, but by design these instances are unlikely. Our mechanism —abstaining based on exceeding a pairwise dissent threshold — differs from methods inducing ties via score equivalence.

what is exactly Comparisons(T )? Also T_mathcal is undefined, I guess it is the family of tiered rankings

We define $\Comparisons{\tierset} := \sum_{i,j \in \intrange{n}} \indic{\aggprf{i}{j}{\tierset} \neq \dnc{}}$ in Section 2, although we acknowledge issues with formatting in our submission that we have now fixed. Please let us know if you would like further clarification on our definition of Tiered Rankings (T_mathcal) in Definition 2.1.

What the colors in figure 2 represent?

The colors denote items in the same tier.

Perhaps the Bradley-Terry model….

We have added a reference, and include an additional experiment in our response to reviewer mXsg.

We hope these clarifications address your concerns! We are happy to resolve any further questions.

We thank you for your response! We provide tables at https://tinyurl.com/2ybsfs95

In RLHF settings where the number of items n is typically small (<5) …This would clarify whether SPA’s advantages (e.g., robustness, transparency) persist in the small-scale preference comparisons characteristic of RLHF.

Certainly! SPA's ranking robustness primarily hinges on the number of judges (p) and their agreement (1 - $\tau$). High consensus is needed for admitted pairwise preferences. More judges clarify patterns and enable rankings at lower dissent. Low granularity suggests the need for more judges. Deep disagreement might warrant other methods (like expert input), highlighting SPA's transparency. (See reviewer responses mXsg/FgRQ for details). We provide an example in Table 9. Increasing p (10 -> 30) allows more tiers (3 -> 4) at lower dissent (0.5 -> 0.333). Limited rankings (dissent < 0.5) can signal the need for more judges and identify items with strong agreement.

SPA assumes uniform weighting of user preferences. However, in real-world applications (e.g., expert-driven labeling), users may have heterogeneous weights (e.g., experts’ preferences matter more). Does SPA can be directly applied in such a scenario? and analyze how it affects tiered rankings and guarantees (e.g., stability under missing data)?

This is possible. The most trivial way is to include expert preferences as equivalent to multiple judges. We can also use arbitrary nonnegative weights on each judge. The guarantees then become relative to not overruling a certain sum over weighted preferences of judges.

Just set

$\\textrm{Disagreements}(T) := \\max_{i,j \\in [n]} \\sum_{k=1}^p w_k \\mathbb{1}[\\pi_{i,j}^{k} \neq 1, \\pi_{i,j}(T) = 1]$

Where $w_k$ is the weight of a given judge, renormalized so that $\sum_{k\in\intrange{p}} w_k = p$. We can maintain many of the guarantees premised on the disagreement constraint - for example, stability under missing data. Some theory does change, for example, $|\mathcal{W}|$ is no longer necessarily linear in p.

Key theorems are valid under stated assumptions but require clarification to resolve inconsistencies …may not hold for non-random missingness.

Proposition 4.2 does hold even for adversarial, non-random missingness - that’s one of the more fascinating ramifications of our approach, actually. The proposition holds because indifference is a conservative assumption to make - it counts as the same or more disagreement with any possible comparison than would any other possible value. We never make comparisons that would be invalid given the true values, regardless of the missingness mechanism.

It may be possible to model missing preferences to extend coverage. See our response to reviewer sHCm (future work).

SPA’s linear runtime is untested on large n (>10^4), limiting real-world applicability.

We appreciate your prior note that there are applications for n<5; please also see our response to FgRQ and sHCm, which address the scale of experiments and use cases. We would like to note that in many domains (e.g., fine-tuning), a limited amount of examples (50-100) is enough to improve performance. From Meta: “A general trend we’ve seen is that quality is more important than quantity … documentation suggests even a 50- to 100-example dataset can potentially make a difference.”

• https://ai.meta.com/blog/how-to-fine-tune-llms-peft-dataset-curation/

SPA’s 0% inversion rate under Δ-Gaming (Table 1) supports robustness claims, but the absence of p-values or confidence intervals undermines statistical significance. Robustness claims (e.g., Δ-Gaming) lack significance tests, making it hard to validate improvements.

We would like to clarify the nature of the Δ-Gaming (Δ-Adversarial) metric reported in Table 1. SPA exhibited zero inversions across 100 simulations, each with 10% adversarial preference flips. This 0% reflects the maximum observed inversion rate under the worst-case condition, demonstrating strong stability against manipulation. As such, CIs/p-values on this reported maximum are not directly applicable. We refer you to Appendix C for guarantees on stability and robustness.

The proposed methods are theoretically sound, but evaluation criteria and baseline selections are outdated and contextually limited.

We wish to clarify our intent with the choice of metrics. Our chosen evaluation criteria (e.g., Disagreement Rate, Abstention Rate, Δ-Adversarial robustness) directly quantify the core characteristics of SPA – its ability to manage the trade-off between coverage and dissent, and the resulting robustness gained through abstention. If the reviewer feels specific metrics would enhance the evaluation, we are open to revising/adding these where possible.

We hope our responses clarify the points raised and are happy to address any remaining questions.