"Methodology details"

Thank you for pointing out the missing details. We will update them clearly in the manuscript, as follows:

1. We assume that our prior is a multi-variate Gaussian with mean $\mu$ and covariance $\Sigma=\text{diag}(\sigma\sigma^T)$, where $\mu, \sigma \in \mathrm{R}^d$. In all experiments, they are initialized from a standard Gaussian. However, in our control experiments, we observed that using a learned Gaussian i.e. setting $\mu$ and $\sigma$ to learnable parameters under the ELBO objective (Eq. 3) improved performance and stability during training.
2. The humans are simulated using Oracle reward functions that are included in Appendix B.1. We randomly sample a user type and query a batch of annotations from the given user to generate a single data point.
3. The accuracy of the reward model is "1" if the LM rewards model assigns a higher reward to the response preferred by the given user over the alternative response, and "0" otherwise. We report the average accuracy of the model over the eval set consisting of multiple prompt and response pairs, labeled by diverse users.
4. VPL-SPO is required as a part of policy optimization while currently, we focus on preference modeling for LLMs in this work. VPL and VPL-SPO will model the preferences with similar accuracy. The only difference would be the scale of rewards that affects policy optimization.

--

[1] Swamy et al. (2024). A Minimaximalist Approach to Reinforcement Learning from Human Feedback.

[2] Siththaranjan et al. (2023). Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF

[3] Zhao et al. (2023). Group Preference Optimization: Few-Shot Alignment of Large Language Models.

[4] Ivison et al. (2023). Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2.

[5] Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model.

[6] Cui et al. (2023). UltraFeedback: Boosting Language Models with Scaled AI Feedback.

[7] Sorensen et al. (2024). A Roadmap to Pluralistic Alignment.

[8] Shen et al. (2023). The Trickle-down Impact of Reward (In-)consistency on RLHF.

[9] Gao et al. (2022). Scaling Laws for Reward Model Overoptimization.

[10] Ivison et al. (2024). Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback.

[11] Meng et al. (2024). SimPO: Simple Preference Optimization with a Reference-Free Reward.

[12] Sharma et al. (2023). Towards Understanding Sycophancy in Language Models.

[13] Ouyang et al. (2022). Training language models to follow instructions with human feedback.

[14] Bai et al. (2022) Constitutional AI: Harmlessness from AI Feedback.

the model being biased/presenting sycophancy [12] to satisfy user beliefs; second, the model reinforcing users with antisocial/unethical/criminal preferences, which often presents preferences that are very far from the rest of the society.

Thank you for bringing up this insight, and we will include the following discussion in the social impact statement.

In pluralistic alignment, we assume that some differences in user preferences reflect divergent but equally valid perspectives on which moral or cultural values an LLM should align to; for example, individuals from one culture may hold collectivist moral values, while another culture is more individualist. Both value systems should be respected, and as such LLMs should be able to recognize and adapt to the values held by a particular user. However, the personalized model could potentially either be sycophantic or align with adversarial users, which is undesirable. This raises very interesting questions, such as: At what point should the LLM embrace a more universal set of values? How can we detect when such a point has occurred in a particular conversation? The probabilistic framework of the user distribution could allow us to identify low probability or rare behavior, and also the distributional nature of reward functions can help us point out responses where the users are divergent (maybe signifying disagreement). Additionally, a model could flexibly switch between adhering to the user's personal preferences and conforming to a more universal perspective on topics where it could be biased, or is sensitive to jailbreak [14]. Taking inspiration from Constitutional AI [14], we can allow a system designer to specify the topics for which the LLM should not engage in user personalization. Overall, this presents an exciting future research direction toward building safe personalized LLMs.

Pets dataset is simple

We acknowledge that Pets dataset is a simple synthetic dataset. Therefore, we present the improved capabilities of our method with additional experiments on a larger and more diverse UltraFeedback dataset (UF-P-4), with four different users (In Table AM:1). The Pets dataset provides a sanity check over VPL to show that the model is able to adapt to multi-modal preferences in imbalanced language datasets.

“this raises concerns if variational inference is required here - perhaps a base LLM can already separate these users in the feature space, but there is no such a baseline in the work.”

We include an additional baseline in Table AM:1, "VPL+Ground Truth User", where we replace the predicted latent user vector by the ground truth one-hot user vector to adapt the model to different users at training and test-time. We show that VPL provides comparable performance to this condition (63% vs 66%). Here, a key advantage of VPL is that it does not assume access to explicit user types but learns to encode and cluster users directly from preference labels, while achieving similar accuracy.

there is only preference modeling and no RLHF/alignment

We present experiments on learning policies using VPL reward modeling in simulated control domains. We acknowledge that our language experiments are focused on preference modeling, but we primarily follow and compare to prior work, which also focuses on preference modeling from diverse datasets alone [2, 3], without including experiments for LLM fine-tuning. We believe that preference modeling alone is interesting, as modeling diverse users may also provide increased interpretability in highlighting potential reasons for preferences [7, Sec. 2]. We believe that further exploring how to best apply VPL to downstream tasks and larger, noisier datasets is an interesting and exciting avenue for future research. Additionally, prior work has shown that improving RM performance can yield improved downstream performance, both when used in RLHF training [8, 9, 11] and when used in best-of-N settings [9, 10].

“the requirement of filtering out the context where the users agree..”

In the additional experiments with four users (Table AM: 1 of the rebuttal PDF), we filter out instances only where all users disagree. So, the context can still contain queries where at least two users overlap. Thus, VPL works in cases where different users agree on some responses, but not all of them.

We will add the following to the limitations sections of the paper: “In LLM preference modeling, VPL assumes that the queries used to generate context inputs for posterior inference contain some useful information to identify the users. In our work, we filter out the instances where all users agree i.e. avoiding degenerate contexts that provide no information about the users.”

Thank you for your feedback. We appreciate that you found the method interesting and agree that pluralistic alignment is a challenge that is mostly overlooked by the RLHF community. We have conducted 5 additional experiments to demonstrate scalability and include the results in the rebuttal PDF, and address your concerns below.

“The major concern is on the evaluation setup. [...] not sufficient to claim scalability in the LLM setup for the proposed method.”

In our LLM experiments, we adopt the widely used UltraFeedback dataset to expose pluralistic alignment. The original dataset has 4 attributes along which fine-grained preferences are generated, and in the original submission, we used two of these attributes to indicate diverse users (similar to prior work [2]). We present additional experiments in Table AM:1 where we compare VPL and the baselines, in a scaled and more diverse setting where all four attributes are present in the preference dataset. Effectively, in these experiments, we go beyond the prior work [2] that includes two users (harmlessness and helpfulness), and include all four attributes in the UltraFeedback dataset. This dataset presents a standard benchmark, which is similar to past works in preference modeling with diverse users [2, 3], as well as state-of-the-art preference modeling works that use either UF or HH-RLHF [4,5,6]. By using all available attributes in UF as distinct users, we have created a challenging benchmark for pluralistic alignment from the best available real RLHF language datasets.

The reviewer suggested : “Although the paper brings a diverse [...] hidden context is composed of very few variables with simple distributions”. So, to demonstrate VPL in the presence of more users, we include new experiments with a large number of users and hidden variables, in Figure AM:2. We supplement the Maze and Habitat experiments with 10 and 5 users (Figures 8 and, 4 respectively), with new Habitat-Rearrange experiments, where we have ~ 100 users. Each user has a ranking over five different locations for storing objects in their home. Here, the space of users and variables is combinatorial (as total possible orderings are 5!). In Figure AM:4, we observe that VPL infers the user distribution and adapts the downstream model, demonstrating its effectiveness with a large user population. Additionally, in Figure AM:3, we test the generalization of VPL to unseen users at test time and show that it can effectively infer and follow unseen goals at test time, outperforming the baselines.

“Another crucial concern is that the proposed reward scaling technique is not equivalent to the learned latent reward [...]”

We appreciate the reviewer for taking the time to craft this example. To test the given MDP under the scaling from SPO [1], and standard version (BTL) of preference learning [13], we simulated the MDP you suggested and report the value function of states $s_1, s_2$.

Under a Markovian oracle, both methods fail to generate the correct value function i.e. $V(s_1) < V(s_2)$. This points to a larger issue in preference learning, which all RLHF methods face and is not unique to our scaling. Essentially, in preference-based learning, we only have binary labels, and no notion of reward scale; so we assume how the scale of rewards affects the preferences (e.g. BTL or SPO). Consequently, under the assumed scale, a cumulative reward estimate (i.e. the value function) might not have the same ordering as the ground truth, which is what we observe in the above table.

We adopt a common assumption in preference learning, particularly in [1] that inspired the scaling: the oracle providing preference labels is non-Markovian (i.e. it prefers all states in a trajectory with higher returns over those in a lower return one). As a result, in the given MDP, the states $s_1, s_3, s_5, s_7, s_9$ would be preferred over the states $s_2, s_4, s_6, s_8, s_{10}$. This generates the accurate value functions and policy i.e. $V(s_1) > V(s_2)$, and the policy under this scaling is invariant.

“data requirements during test-time inference [...] it looks like it requires 100 samples, so it is questionable if this is sample-efficient enough for interacting with humans in the real world”

We would like to clarify the training and evaluation setup here: VPL uses only a random sample of 2-8 questions from the subset of 100 questions in the LM experiments to predict the latent distribution, which provides us with a sample efficient way to infer the user distribution at test-time, as opposed to using all 100 samples.

“How does VPL handle irrational users? (i.e., users whose preferences are inconsistent across context/responses)”

In Figure AM:2 of the rebuttal PDF, we include experiments with varying number of labels queried from each user at test time. We also add noise to the context i.e. we flip the labels with a certain probability to simulate inconsistent and noisy users. The performance degrades with increasing noise, but VPL still outperforms the baseline at 25% noise level. With more labels, VPL performs better at higher noise levels, but at 50% noise, the context provides equal information for all user types. Consequently, it fails to identify the particular user and the performance coincides with the baseline (that has no mechanism for personalization). This shows that VPL is able to handle irrational users i.e. users whose preferences are inconsistent across context/responses.

“comparison with several baselines on multi-objective RL or aligning with diverse preferences. ”

We thank the reviewer for providing additional references for our work. We will incorporate the additional references in the related works section as follows:

[1, 4] aims at trading conflicting alignment among diverse users with different objectives through techniques like Pareto-optimal optimization or multi-objective RL. The goal of such methods is to optimize the reward model to maximize worst-case performance over the different groups. In contrast, our work does not aim to optimize against the diversity but rather solve the model misspecification and learn reward models that can infer the context and specialize to a particular user. This ensures the model can align to all user groups, rather than trade-off among them.

[2] introduces an approach that uses explicit clustering of human groups and learns individual reward functions for the different clusters. Our work instead relies on variational inference and latent conditioned rewards to infer and model diverse humans directly from the preference data. [2] further introduces an additional method that assumes a single reward function for all the clusters, but adopts a probabilistic approximation to the reward model, similar to DPL [6]. We include DPL as a baseline in all the LLM experiments in Table 1 in the original submission and Table AM:1 in the rebuttal PDF, and show that our method outperforms DPL across multiple datasets.

We included a reference to [5] in L112 of the original submission.

[3] proposes a pluralistic alignment framework, using smaller LMs that are trained on community or user-specific data. Further, it uses responses from the smaller community LMs to adapt a larger LLM to provide responses covering all or just one specific user. Meanwhile, our approach adopts an unsupervised method (no access to explicit user distributions) to identify the latent and condition the preference model towards the specific user preferences.

prior and posterior structure

Here, we provide additional details about the structure of the prior and posterior of our model. We assume that our prior is a multi-variate Gaussian with mean $\mu$ and covariance $\Sigma=\text{diag}(\sigma\sigma^T)$, where $\mu, \sigma \in \mathrm{R}^d$. In all experiments, they are initialized from a standard Gaussian. However, in our control experiments, we observed that using a learned Gaussian i.e. setting $\mu$ and $\sigma$ to learnable parameters under the ELBO objective improved performance and stability during training. The posterior is an MLP that takes in the annotated samples $**S** \sim (s^i_1, s^i_2, y^i)_{i=1}^N$ and predicts the latent distribution $\mathcal{N}(f(**S**), g(**S**))$.

Also it is done on majorly simulated tasks [...]

Our work presents an algorithmic solution to the problem of personalization in RLHF. We present extensive experiments across simulated control and language experiments, which we believe present realistic setups and benchmarks. So, we believe real-robot experiments to be beyond the scope of this algorithmic paper and leave on future work the challenge of deploying this method to real-world robot systems.

--

[1] Chakraborty et al. (2024). MaxMin-RLHF: Towards an equitable alignment of large language models with diverse human preferences.

[2] Park et al. (2024). RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation.

[3] Feng et al. (2024). Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration.

[4] Boldi et al. (2024). Pareto-Optimal Learning from Preferences with Hidden Context

[5] Jang et al. (2024). Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging.

[6] Siththaranjan et al. (2023). Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF

[7] Wu et al. (2023). TidyBot: Personalized Robot Assistance with Large Language Models.

[8] Graham et al. (2013). Moral Foundations Theory: The Pragmatic Validity of Moral Pluralism.

[9] Cui et al. (2023). UltraFeedback: Boosting Language Models with Scaled AI Feedback.

[10] Bhattacharjee et al. (2020). Is More Autonomy Always Better? Exploring Preferences of Users with Mobility Impairments in Robot-assisted Feeding.

We thank the reviewer for providing useful feedback regarding our work. We appreciate that the reviewer believes that our introduced approach is natural towards aligning models to diverse preferences. The reviewer brings up useful concerns regarding the scale of the experiments, demonstrating the effectiveness of VPL in realistic benchmarks and drawing bridges to practical scenarios for this problem and solution. We address these questions by providing additional experiments, and outline those and other responses below.

“The experimental setup is restricted to a simulated environment and tasks.”

Thank you for suggesting that we scale our experiments to more realistic tasks and environments. Following this, we include five additional experiments in the rebuttal PDF:

1. In Table AM:1, we scale the language experiments to UltraFeedback dataset with four users, which is comparable in scale and diversity to state-of-the-art prior work on preference modeling [6]. In these new experiments, we go beyond the prior work [6] that includes two users (harmlessness and helpfulness), and include all four attributes in the UltraFeedback dataset. By using all available attributes in UF as distinct users, we have created a challenging benchmark for pluralistic alignment from the best available RLHF language datasets.
2. In Figure AM: 1, we expand our robotics experiments to additional tasks in the Habitat simulator. We include a new Habitat-Tidy task inspired by TidyBot [7], that presents a realistic setting for diverse users in the real world.
3. To test VPL with a large number of users, in Figure AM: 4, we demonstrate that it outperforms the baselines in the Habitat-Rearrange task (when scaled to ~ 100 users).
4. We also show that VPL is able to generalize to new users at test-time i.e. adapt the model to align with unseen goals in Figure AM: 3.
5. In Figure AM:2, we show that this learned model is robust to noise or inconsistency in user context labels when modeling diverse user preferences.

Overall, the additional experiments and analysis, along with the existing results cover a diverse number of realistic tasks with increasing scale and diversity.

Can the authors motivate a practical scenario of the setup, where this setup can make sense?

Consider real-world tasks such as autonomous household robotics. Each person has strong individual preferences for things like how a user arranges things in their home [7], which must be modeled by a robot to effectively assist in cleaning the home (such as one person may prefer storing shirts in the drawer, while another may prefer them on the shelf). In assistive robotics for the disabled [10], different users have different physical constraints and preferences regarding feeding methods and food choices. To succeed at these tasks, the robot must efficiently infer and adapt to each user's unique preferences. Our simulated control experiments, as well as the additional experiments shown in Figure AM:1 in the rebuttal PDF, test whether VPL is a promising method for this type of real-world robotics task.

LLMs must interact with a large, diverse, global population of users. Within this population, individuals from one culture may hold collectivist moral values, while another culture is more individualist [8]. Or in certain cultures, it is morally acceptable to eat a certain type of fish while pregnant, but in some others, it is a taboo. Both of these value systems should be respected, and as such LLMs should be able to recognize and adapt to the values held by a particular user.

In our experiments, the UltraFeedback dataset contains multiple attributes that users may prefer, such as harmlessness and helpfulness [9]. E.g. if the prompt is “teach me how to dive into a river” - users can prefer the models to be either helpful (i.e. provide the steps to learn diving) or harmless (i.e. warn the user that it is dangerous and provide no instructions). Here, the LLMs should infer and personalize to the values and preferences of each user, and VPL presents a step in that direction.

In summary, these directions present some practical cases where an AI model has to personalize and cater to multiple users. And, our method presents a practical approach to this problem where each user answers a few questions to personalize the downstream model.

“The setup is not extremely practical. For ex: If I am the LLM company when a new user [...] any other way to do that or we need to incorporate the active learning strategy every time for each new user/group?”

We thank the reviewer for bringing up this concern. We would like to present an important clarification that the active learning procedure needs to be performed only once after the model has been trained to generate the most discriminative queries i.e. using active learning we get a set of N query pairs. Beyond active learning, we can instead select random queries (potentially leading to inefficient performance). For each new user, we get labels over the N pairs, which enables us to estimate the posterior $q(z | (s^i_1, s^i_2, y^i)_{i=1}^N)$. Our experiments show we get accurate performance in as little as 2 test-time queries (see Figure AM:2 in the rebuttal PDF and Figure 5 in the original submission). However, if we do not have access to any test-time queries for a new user, we have access to the prior $p(z) \sim \mathcal{N}(\mu, \sigma^T \mathrm{I})$, and can sample from the mode of the prior, where the model would respond according to the majority group (as in existing preference learning methods). Thus with no additional information, VPL is as performant as a vanilla RLHF model.

“This process seems quite costly as it requires multiple full passes through the dataset. A complexity analysis should have been included.”

Thank you for this comment, we would like to clarify that our approach doesn't require multiple passes over the entire dataset for each new user. In our active inference techniques, we use a sampling-based method inspired by [3] to actively query the model. Given a dataset of D queries $(s^i_A, s^i_B)_{i=1}^{|D|}$, we sample $S$ query batches of size $Q$, where $Q$ is number of annotated samples per batch we get from a user. Here, $Q \in [2,8]$, so we need to perform O(S * Q) passes over the model with batch size $2^Q \sim [4, 256]$. Furthermore, this process only needs to be performed once after the model is trained to obtain the most discriminative set of queries for the given model. Finally, whenever a new user interacts with the system, we need to get labels on the actively inferred queries (usually 2-4) but do not require any additional passes over the query dataset.

“The imbalance may affect the approach's effectiveness.”

Thank you for your attention to this detail. In our control and language experiments, particularly the control experiments and the pets dataset are imbalanced i.e. the preference dataset contains an unequal number of preferences from individual groups or users. As a result, the baselines can achieve > 50% accuracy, converging to the preferences of the majority user groups. So, VPL works in the presence of imbalanced datasets. While recent works [7, 8] focus on achieving Pareto-optimal performance across the groups, VPL treats each user individually via latent conditioning and does not suffer from this problem. VPL is able to personalize to the minority groups as well (see Figure 11 in Appendix A).

One missing piece in the experiment is an analysis of the latent code's negative impact on the language model. An important feature of language models, especially LLMs, is their versatility. They are not supposed to only answer questions related to pets. One question that needs to be answered is whether conditioning on a pets-related latent code would lead to catastrophic forgetting or distortions in responses to other tasks/questions unrelated to pets.

Thank you for raising this insightful question. VPL introduces a latent bottleneck during the reward learning process, but since the base architecture of the policy and the reward model does not change otherwise, we do not believe this should majorly affect the performance of the model. We will leave a thorough investigation of this question to future work, but will note this as a potential limitation in the limitations section.

--

[1] Guan et al. (2023). Relative behavioral attributes: Filling the gap between symbolic goal specification and reward learning from human preferences.

[2] Wu et al. (2023). Fine-grained human feedback gives better rewards for language model training.

[3] Sadigh et al. (2017). Active Preference-Based Learning of Reward Functions.

[4] Peng et al. (2024). Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Input.

[5] Siththaranjan et al. (2023). Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF.

[6] Ivison et al. (2023). Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2.

[7] Boldi et al. (2024). Pareto-Optimal Learning from Preferences with Hidden Context

[8] Chakraborty et al. (2024). MaxMin-RLHF: Towards an equitable alignment of large language models with diverse human preferences.

Thank you for the useful feedback. We appreciate that the reviewer recognizes the importance of the problem, and the realistic setup introduced to study it. The reviewer raised useful questions about the effectiveness of the user encoder with varying labels, the generalization performance, and the scale of experiments. Overall, the concerns raised are very interesting and we provide additional experiments to demonstrate the usefulness of our method in real-world settings. We respond to all the concerns individually below:

[1] and [2] below also learn multi-modal reward

Thank you for pointing us to these references. While these methods present conditional reward models, we highlight the specific differences between our work and the references. We will update our related section to contrast our contribution with these works as follows:

[1] presents an approach to learning a reward model conditioned on a feature vector that represents the preferences for the desired behavior. Further, in [1] the users provide feedback on the individual features of a trajectory. However, in our work, we do not assume access to such explicit features. Instead, we propose a method to encode and predict latent distributions directly from binary preferences in an unsupervised setting. Moreover, [1] does not include experiments on language models and only validates the method on simulated locomotion experiments, while we scale our approach to state-of-the-art preference datasets for LLMs, along with diverse and realistic simulated experiments.

[2] introduces a method to use dense rewards over multiple specific attributes to align LLMs. This treats the problem of pluralistic alignment as a multi-objective RL problem i.e. learn different reward models for each user or objective individually. Whereas in our work we aim to encode and model the multi-modal user preferences, and then condition the same model i.e. steer it to align with the particular user.

“this work doesn't significantly extend beyond them in terms of scalability.”

In addition to the differences highlighted above, we also scaled our experiments to demonstrate VPL. Following this, we include five additional experiments in the rebuttal PDF:

1. In Table AM:1, we scale the language experiments to the UltraFeedback dataset with four users. In these new experiments, we go beyond the prior work [5] that includes two users (harmlessness and helpfulness), and include all four attributes in the UltraFeedback dataset. By using all available attributes in UF as distinct users, we have created a challenging benchmark for pluralistic alignment from existing real RLHF language datasets.
2. Our method is more general and focuses on inferring the (potentially unknown number of) different preferences without explicit user information and learning a latent conditioned model to adapt to the particular end user. Meanwhile, [1] uses explicit features for modeling different users or goals, and [2] learns different reward models for the different features. The probabilistic nature of VPL enables us to do active learning as well (Figure 5 in original submission), which makes this a more efficient model for diverse preferences.
3. In Figure AM: 1, we expand our robotics experiments to additional tasks in the Habitat simulator. We include a new Habitat-Tidy task inspired by TidyBot [7], that presents a realistic setting for diverse users in the real world.
4. To test VPL with a large number of users, in Figure AM: 4, we demonstrate that it outperforms the baselines in the Habitat-Rearrange task (when scaled to ~ 100 users).
5. We also show that VPL can generalize to new users at test-time i.e. adapt the model to align with unseen goals in Figure AM: 3.

Overall, the additional experiments and key differences highlight that our proposed approach presents a scalable and general method for pluralistic alignment over the included references. In summary, [1,2] focuses on learning personalized policies or reward models given the explicit classes or users. Further, these methods are tested only over language or control experiments. In our work, we present an initial step towards learning a probabilistic framework to predict the latent user distribution directly from binary comparisons for language models and learn personalized policies that are tested in dynamic and realistic control settings.

“how encoder’s performance improves as the # of user samples increases”

We present additional experiments, where we vary the number of labels at test-time, for LLMs (in Figure AM:2) and control experiments (in Figure 5 in the original submission). It shows that the performance of the model improves with the number of labels, reaching peak performance at 8 labels. However, actively selecting the queries could allow the model to perform similarly with fewer queries (see Section 4.3 and Figure 5). Furthermore, using more labels makes the model robust to small noise levels as observed in Figure AM:2.

“how well the encoding generalizes”

To test the generalization of VPL to new users, we include experiments in Figure AM:3, where the reward model and policy are trained on users preferring 10 different goals in the maze, while at test-time the agent interacts with users preferring 5 unseen goals that are sampled in-distribution. We compare the performance to an oracle baseline, that uses the exact goal information during training and test time, to analyze how well the encoder can infer the distribution over unseen users. In Figure AM:3, we observe that VPL significantly outperforms the baselines, and has performance at par with the oracle baseline. This demonstrates that VPL can interpolate between different users, even slightly outperforming the oracle that is trained on only the set of 10 goals, while VPL uses the prior to optimize the policy over the set of all possible in-distribution goals.

We appreciate the reviewer’s feedback and are glad that they agree with the motivation of the problem and the novelty of the approach. We address the concerns raised as follows:

“Having a single-modal dataset [...] would help further ground the work and increase the experimental depth.”

This is an interesting analysis and we thank the reviewer for suggesting this. We ran additional experiments on the UltraFeedback dataset, where we considered the preferences of a single user (preferring the model to be “honest” over all other attributes) to analyze the single-modal case as suggested. The standard BTL model gives a 77.04% eval accuracy while our VPL model gives a 77.16% eval accuracy. Our model matches the baseline performance, indicating that there is no drop in performance when using VPL compared to traditional RLHF over an unimodal dataset.

“evaluation outside of designed control environments.”

We present experiments on learning policies using VPL reward modeling in simulated control domains, which we have expanded and made more realistic in the rebuttal PDF. We acknowledge that our language experiments are focused on preference modeling, but we primarily follow and compare to prior work, which also focuses on preference modeling from diverse datasets alone [1, 2], without including experiments for LLM fine-tuning. We believe that preference modeling alone is interesting, as modeling diverse users may also provide increased interpretability in highlighting potential reasons for preferences [3, Sec. 2]. We believe that further exploring how to best apply VPL to downstream tasks and larger, noisier datasets is an interesting and exciting avenue for future research. Additionally, Prior work has shown that improving RM performance can yield improved downstream performance, both when used in RLHF training [4, 5, 7] and when used in best-of-N settings [5, 6].

--

[1] Siththaranjan et al. (2023). Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF.

[2] Zhao et al. (2023). Group Preference Optimization: Few-Shot Alignment of Large Language Models.

[3] Sorensen et al. (2024). A Roadmap to Pluralistic Alignment.

[4] Shen et al. (2023). The Trickle-down Impact of Reward (In-)consistency on RLHF.

[5] Gao et al. (2022). Scaling Laws for Reward Model Overoptimization.

[6] Ivison et al. (2024). Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback.

[7] Meng et al. (2024). SimPO: Simple Preference Optimization with a Reference-Free Reward.