A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models

Thank you for your time in reviewing our survey. Regarding the points you raised, we'd like to clarify the following --

• [W1] Re. experiments.
  • Thanks for mentioning this. However, we’d like to clarify that our work is a survey paper, and thus, our focus in this work is not empirical experiments. To this end, similar survey papers in the prior iterations of COLM share our scope, e.g. [1-4] in prior surveys in COLM (see below).
  • Meanwhile, personalized/pluralistic alignment is an emerging research direction with no consensus on best evaluation dataset practices (Table 1), making direct comparisons between methods challenging. Regardless, we will add Table 1 and relevant discussions to the paper to provide further information about the relationship between current works to our readers.

Prior surveys in COLM:

[1] LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models, COLM 24

[2] What Are Tools Anyway? A Survey from the Language Model Perspective, COLM 24

[3] Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models - A Survey, COLM 24

[4] A Survey on Deep Learning for Theorem Proving, COLM 24

Thank you for your valuable insight to our work. Regarding the points raised --

• [W1] Despite the above strengths, it is sad to see that the authors did not clarify how this topic was different from other problems in definition or experimental setting, as certain works involved in this survey can be shared in the aspect of method design.
  • Thanks, compared to other personalization tasks (e.g., replicating user behavior), while some evaluations transfer to this line of work, a common distinct setting/evaluation is preference-ranking with LLM as a judge mimicking user or user groups with specific preferences, typically conditioned on synthetic persona or preferences [e.g., 5, 6, 8, 11, 12, 14, 17, 18].
  • However, as discussed in section 6 and shown in Table 1, currently, evaluation in the field varies significantly, and we hope our work can accelerate convergence of our community on evaluating personalized preference alignment, by delineating relevant works.
  • We will add this discussion to section 6, and point to the discussion in problem setting.

[W2] The authors focused on analyzing specific works and techniques. However, it would further strengthen the survey if they could provide deeper insight into this problem in Sections 1 & 3.

• Please see the response to W1, where we will add a discussion about the uniqueness of the problem setting compared to adjacent fields and provide further discussion re. the problem itself.

[W3] The authors are recommended to add references to more related works.

• Thanks. We will make reference to the suggested general methods and other further related work as fit.

Thank you for providing valuable feedback to our work. Regarding your comments/concerns:

• [W1] Limited synthesis across methods While many methods are described in detail, the paper sometimes reads like a catalog rather than a deeply analytical synthesis. For example: What are the trade-offs between training-time and inference-time personalization in real deployment scenarios?
  • Thanks - while we share your view that the community would benefit from head-to-head comparison between various methods, we wanted to point out that due to the emerging nature of the field, current works are frequently parallel to each other and employ different benchmarks, baselines, and evaluations [Table 1, 1-18]. Regardless, current works show (a) personalized methods are better than non-personalized ones (e.g., DPO, SFT, Zero-shot) [5, 8, 11, 18]; and (b) training-time interventions are generally more effective [e.g., 2, 3, 4, 12, 14]. We will add this discussion along with Table 1 to the appendix. [W2] Ambiguity in scope definitions. The term pluralistic preference alignment is introduced early, but its distinction from personalized alignment is somewhat vague. Does “pluralistic” simply refer to accommodating diversity across users, or does it imply multi-objective or group-aware alignment?
  • Thanks. We want to clarify that pluralistic and personalized preference alignment are working towards the same general goal, i.e., building a system that exhibits behavior that is preferable to each individual user (section 3.1). While the two terms differs slightly in pitch, e.g., pluralistic alignment emphasizes “building a system that satisfies diverse preferences across the whole user population”, the term “personalization,” i.e., “for each user, being able to cater to his/her preference” actually pitches towards the same objective from a different angle.
  • We’ll add this comment to section 3.1 to highlight this.
• [W3] Minor writing issues..
  • Thanks for the catch, we will clear the citations and fix the typos accordingly.

Thank you for your valuable feedback. Please find the responses to each individual comments as follows:

• [W1] The comparative analysis lacks focus on performance and efficiency aspects of existing methods on different benchmarks.
  • Thanks for mentioning this point. While we share your view that head-to-head comparison of methods on a standard set of benchmarks is important, we wanted to clarify that, as we discussed in the paper (line 62+, introduction; line 304+, limitations of dataset & evaluation), since personalized/pluralistic alignment is an emerging field, there are current no universally acknowledged benchmarks yet, making a direct comparison between methods challenging.
  • To make this more concrete, we select representative methods mentioned in our work and summarize their evaluation benchmarks and baselines in Table 1. As shown in the table, while [1-4] conduct evaluation on the LAMP benchmark, benchmark selection generally exhibits great diversity [e.g., 5-18]
  • For methods that are evaluated on the same benchmark (e.g., LAMP), contemporary works often compare with the same set of baselines (e.g., RAG/PAG proposed as baseline in the LAMP benchmark paper itself) due to being developed in parallel. Based on self-reported performances, Per-PCs [3] is a more efficient method than OPPU [2]; while performance-wise it outperforms Hydra [4] on some subtasks while being lower on others. We will add this information, relevant discussions here, and Table 1 to the appendix of the paper. As we discussed in section 6, it would be beneficial for our community to converge towards unified benchmarks for comparison between methods (as you mentioned). To this end, part of the motivation/contribution of this work is to clearly delineate the status of personalized preference alignment (which itself is an emerging field), to help our community to build upon each others’ work and accelerate the progress towards unified evaluation/comparisons.
• [W2] There is no dedicated discussion of ethical, privacy or other potential risks and implications of personalized preference alignment.
  • Thanks. We will highlight this more explicitly in the final version, and add a discussion for relevant matters such as bias and privacy in the appendix, so readers can easily trace the relevant literature (currently in related work section, line 70-71, kirk2024a). Please see response to Q3 for further details.
• [W3] Discusses limitations and a few future work directions but does not elaborate on them. It would be useful to further discuss the quality gaps and challenges that remain to be addressed in this field.
  • Thanks. For Continuous personalized alignment (future work paragraph 1), recent work [Zhao ICLR 25] shows LLMs frequently fail to correctly remember and apply user preferences, even when the preference is explicitly stated.
  • To this end, better long and complex instruction-following models (future work paragraph 2) seems a natural remedy, though, as mentioned, there are limited data and methods geared towards addressing this problem in personalized alignment settings, and it would be interesting to see if current success such as data synthesis [e.g., 8 in Table 1] holds.
  • We will add these discussions to the future directions section.
• [Q1] Reading through sections 4 and 5 it is not clear to the reader which methods are better performance-wise or efficiency-wise and where are the gaps. Could you expand on that?
  • Please see response to W1, and general response to reviewers.
• [Q2] In the future work directions, what are the anticipated challenges and potential ways to address them?
  • Please see response to W2.
• [Q3] What are the ethical and privacy concerns that you foresee with personalized preference alignment?
  • Personalized/pluralistic alignment in LLMs naturally requires collecting user-specific information such as user profiles and personal descriptions and/or their behavioral data. To this end, this direction shares privacy concerns with other adjacent domains, such as recommender systems. Similarly, personalization can limit individual users’ access to diverse content, similar to the filter bubble effect in other personalization domains.
  • As mentioned in response to W3, we will add a dedicated discussion to these issues.

Table 1: Representative methods and their setting

• Hyperlinking to [12]'s Arxiv site seems to break openreview table rendering.