Representative Ranking for Deliberation in the Public Sphere

We thank the reviewer for their comprehensive and thoughtful comments.

The reviewer seems to be evaluating our paper primarily as a contribution to social choice. However, the main goal of our work was not to contribute to social choice, but rather to facilitate deliberation online (particularly on social media platforms). Since this area is lacking any representation axioms, we ground our algorithmic approach in social choice, specifically JR, and demonstrate the efficacy of this approach both theoretically and empirically under real-world conditions. We believe this is an important and novel contribution that can open up new directions in the research on prosocial recommender systems. This goal explains many of the decisions we made, and we will try to make this context clearer in our revision.

Our focus on JR

The reviewer asked why we did not focus on stronger variants of JR. Practically speaking, a representation constraint is unlikely to be implemented on a social media platform if it also comes at a substantial decrease in user engagement or a substantial increase in the toxicity of comments. This is why in this initial work we started with the basic JR axiom, which is most likely to be compatible with the optimization of other score functions (engagement, civility classifiers, etc).

However, in response to the reviewer’s concern, we also checked how many of the JR committees in our experiments satisfied EJR+ (and will add this to the paper). Across the 48 JR feeds, only one JR feed did not also satisfy EJR+. The paper that the reviewer mentioned by Boehmer et al. makes similar observations (e.g., "All four proportional rules [including three rules not guaranteed to satisfy EJR+] return committees satisfying EJR+ for all tested instances.’’).

Overall, we believe focusing on JR was a reasonable first step in our work due to (i) the need to demonstrate that the representation axiom is compatible with other arbitrary score functions, and (ii) the observed alignment of JR and EJR+ in real-world settings despite the theoretical gap. Nonetheless, we very much agree with the reviewer that studying stronger axioms would be a promising direction for future research.

Theoretical results

We respectfully disagree with the reader’s assessment of our theoretical results. Coming back to our motivation of integrating JR into online platforms, the goal of our theoretical analysis was to show that, in natural settings, JR is compatible with the optimization of other score functions the platform may care about. While we agree that it is not surprising that the worst-care price of JR becomes higher for arbitrary score functions, these worst-case results primarily serve as motivation for our more novel contribution where we show that, in natural settings, the price is still low (Section 5). This result is crucial, as high prices would likely deter the incorporation of representation constraints on these platforms.

Theorem 5.1 could be re-written in terms of n/k-justifying sets—that is essentially the proof of the theorem. However, our goal was to show a natural condition for why one might expect a small n/k-justifying set, and thus a low price of JR, on social media. In the social media context, when bridging interventions (diverse approval) are done, users are already partitioned into non-overlapping groups that are supposed to be distinctive in their preferences (these groups are typically learned by basically clustering the vote matrix). Theorem 5.1 says that if those groups are also cohesive in the JR sense, then we have a low price. We will clarify this reasoning and motivation, while acknowledging the prior related results, in our revised text.

We also note that, in practice, each group may not necessarily be fully cohesive in the JR sense. Thus, Theorem 5.1 was also primarily meant to serve as intuition for the extended Mallows model result (Theorem 5.4).

Set selection and ranking

The reviewer is puzzled about the focus on set selection for comment ranking. We do not see a discrepancy here. In recommender systems, a very common method for showing a diverse set of items (e.g. videos that span your interests) is to re-rank items greedily so that the set of top items is diverse (see Algorithm 1 in “Fairness and Diversity in Recommender Systems: A Survey” by Zhao et al). Moreover, real-world recommender systems proceed in multiple steps that are actually set selection problems (e.g. retrieval -> early stage-ranking -> late-stage ranking). In our revision, we will clarify the connection between set selection and ranking.

Related Work & Impact Statement

We appreciate the many references that the reviewer shared. We will incorporate them into our paper with a more comprehensive section on related work in social choice. We will also add an impact statement. Lastly, we will correct the typos the reviewer found.

Once again, we thank the reviewer for their thoughtful consideration of our paper.

We appreciate the reviewer’s careful reading of our paper and thank them for their constructive comments. We will correct all the typos that they identified.

With regards to the Perspective API bridging score $f_C$, it is the average of the scores for the seven available bridging attributes:

• “Affinity”: References shared interests, motivations or outlooks between the comment author and another individual, group or entity
• “Compassion”: Identifies with or shows concern, empathy, or support for the feelings/emotions of others.
• “Curiosity”: Attempts to clarify or ask follow-up questions to better understand another person or idea.
• “Nuance”: Incorporates multiple points of view in an attempt to provide a full picture or contribute useful detail and/or context.
• “Personal Story”: Includes a personal experience or story as a source of support for the statements made in the comment.
• “Reasoning”: Makes specific or well-reasoned points to provide a fuller understanding of the topic without disrespect or provocation.
• “Respect”: Shows deference or appreciation to others, or acknowledges the validity of another person.

The score for each individual attribute is a probability between 0 and 1. We provide results broken down by each individual attribute in Appendix E.7.

Additional references for the Perspective API bridging classifiers:

• https://developers.perspectiveapi.com/s/about-the-api-attributes-and-languages?language=en_US
• https://medium.com/jigsaw/announcing-experimental-bridging-attributes-in-perspective-api-578a9d59ac37
• Saltz et al. Re-Ranking News Comments by Constructiveness and Curiosity Significantly Increases Perceived Respect, Trustworthiness, and Interest. arXiv 2024.
• Schmer-Galunder et al. Annotator in the Loop: A Case Study of In-Depth Rater Engagement to Create a Prosocial Benchmark Dataset. AIES 2025.

We thank the reviewer once again for their consideration of our paper.

Thank you to the reviewer for their comprehensive review. We address their questions below.

Connection Between Theoretical and Empirical Results

The reviewer asks about how our theoretical results connect to the empirical findings. Even without considering Theorem 5.4, the bound from Theorem 5.1 already provides a close upper bound to the observed prices in practice. We will highlight this connection in our revised text. For Theorem 5.1, the bound holds if there exists an $n/k$-justifying set of size less than or equal to gamma (which the partition of gamma cohesive groups implies). In all ten questions, the GreedyCC algorithm finds an $n/k$-justifying set of size less than or equal to two (see Table E.3). Our bound from Theorem 5.1 would then imply that the price for all three score functions is at most 8/(8-2) = 1.33. The average empirical prices that we observe (see “Enforcing JR comes at a low price” in Section 6) are 1.05 for engagement, 1.06 for diverse approval, and 1.18 for Perspective API.

Handling Partially Observed Votes

The reviewer also asks how the theoretical results hold when each user only reads a very small portion of items. In practice, the full approval matrix will need to be inferred from the partially observed votes—which is what we do in our Remesh experiments (see Footnote 10). Since this is a standard recommender system task, we do not focus on this approval inference in our paper. However, when implementing this on a real platform, it is very important to ensure that the inferred approvals are faithful to users’ actual approval. We touch on this issue in our discussion section: “on many platforms, users’ approval will need to be inferred from some form of engagement such as upvotes or likes. If the chosen engagement significantly diverges from actual user approval, the validity of the process could be compromised.”

Experimental Results and Stronger Proportionality Axioms

With regards to experimental results, the reviewer asks how our experiments with GreedyCC may hold up against stronger proportionality axioms. In response, we have checked how many of the JR feeds in our experiments also satisfy EJR+ (Brill and Peters, 2023), a much stronger extension of JR. We found that 47 out of 48 feeds also satisfy EJR+ (and will add this result to the paper).

I. Diverse Approval

• 5/10 DA feeds are JR, all of them are also EJR+
• 10/10 JR feeds are also EJR+

II. Engagement

• 3/10 engagement feeds are JR, all of them are also EJR+
• 9/10 JR feeds are also EJR+

III. Perspective API

• 10/10 Perspective API feeds are JR, all of them are also EJR+
• 10/10 JR feeds are also EJR+

This suggests that, although GreedyCC is only optimizing for coverage, it nevertheless satisfies stronger axioms in practice. This result has also been corroborated in other empirical research which has found that JR and EJR+ tend to empirically coincide despite their large theoretical gap (e.g. “Approval-Based Committee Voting in Practice: A Case Study of (Over-)Representation in the Polkadot Blockchain” by Boehmer et al (2024)).

Potentially important as a future direction: The “one member approves one item” condition in JR is too weak (as the authors have acknowledged in the paper). How can we improve this, either theoretically or empirically?

Empirically, the results just described suggest that our approach may satisfy axioms stronger than JR in real-world datasets. We concur with the reviewer that theoretically studying and guaranteeing stronger extensions of JR would be an interesting direction for future research.

Other Comments

“Have you tried other simple heuristics other than JR, and how well do they work?”

We have not tried other heuristics that are weaker than JR, since the GreedyCC algorithm on its own is already quite simple, and as noted above, it has the advantage of satisfying stronger axioms empirically as well.

“It would be helpful to visualize where these methods lie (with & without GreedyCC) on the quality-representation tradeoff plane.”

This is a great idea, and we will add these figures to the appendix. Right now, the trade-offs can be understood through the representation results + the prices of JR shown in Figure 2, but we agree that an explicit visualization would make this clearer.

We thank the reviewer again for their thoughtful comments.

We appreciate the reviewer’s thoughtful comments and expertise. The reviewer’s main question is about their interpretation of Sections 5 and 6, which we address in the following. In Section 5, we first introduce novel theoretical results for the price of JR for arbitrary score functions, and then evaluate these results in section 6. Please note that the aspect of arbitrary score functions is key here for the theoretical results and connects both sections.

Our main contribution is to introduce JR into the problem of facilitating deliberation online, particularly on social media platforms. In the social media context, JR is unlikely to be adopted if it also comes at a substantial decrease in user engagement or a substantial increase in the toxicity of comments. Therefore, we analyzed the price of JR, not only with respect to engagement (like Elkind et al did), but also with respect to arbitrary score functions.

Our theoretical results in Section 5 hold for arbitrary score functions, which is why it logically follows to present them before our experiments where we evaluate the price of JR not just for engagement but also for diverse approval and the content-based bridging classifiers. In this way, our experiments also go beyond Elkind et al who only studied engagement. Although the worst-case price for arbitrary score functions can be arbitrarily high (as we showed in Sec 4), the observed price for all three score functions is remarkably low. As the reviewer points out, our theoretical results in Section 5 may explain these findings—a connection we intended to highlight and will clarify in our revision.

Our empirical results also contribute to the literature on prosocial interventions in recommender systems, such as bridging-based ranking. The most common way to operationalize “bridging” is via diverse approval. However, surprisingly, we found that the content-based bridging classifiers provided much better representation than diverse approval. Indeed, all the Perspective API bridging feeds always satisfied JR by default. This is surprising as there is a line of literature showing that toxicity classifiers (including Perspective API’s) can be biased against various groups (e.g. Sap et al 2019, Lee et al 2024). Yet, it appears that the newer bridging classifiers target attributes with broad appeal, suggesting that these content-based classifiers may be a promising direction for future research in prosocial ranking.

We will also make clear that the bound in Theorem 5.4 is not about any particular algorithmic strategy.

Thank you again to the reviewer for your careful review of our paper.