PersonalLLM: Tailoring LLMs to Individual Preferences

We sincerely appreciate the reviewer for the time and care taken in reviewing our submission and offering feedback on how we might improve our paper. We are encouraged that they agree with the value and novelty of our method of simulating diverse personas for methodological development. Below, we respond to your particular comments. Noted changes can be viewed in our updated submission PDF.

It is stated in the paper that the goal is not to match preferences of a distribution of real users, but rather to generate diverse preferences that are more heterogeneous/diverse. I think that this requires more justification since random preferences would give even higher diversity but may not be useful.

We agree with the reviewer that diversity in and of itself is not the only important criteria for a simulated benchmark. By building our simulated personas on top of reward models trained with human preference feedback, our simulated reward models inherit some reasonable biases about human preferences, while still exhibiting the desired diversity. We believe that our analysis in Section 3 shows that our simulated users achieve a good tradeoff between offering reasonable representations of human preferences while overcoming the technical bottleneck in creating diverse preference targets.

In line 76 it says “in doing so we are able to simulate an entire user base”. On the other hand it says in line 102 that “We do not claim our simulated personal preference models provide a high-fidelity depiction of human behavior”, so this may be a bit confusing and you may want to rephrase these statements. After reading the first one I was hoping for some evaluation of how realistic the simulated users are. This is actually done in “Comparison to Human Preferences” in Section 3, so I guess you are doing some of that?

We appreciate this concern, and have updated Line 76 to more clearly reflect the nature of our preference models.

If the goal is to obtain high coverage rather than matching the distribution of users, perhaps this can be made explicit and possibly evaluated against real user behavior? Perhaps some measure of support instead of Wasserstein? It would be also interesting to compare the results in Figure 5 to those from standard personas baselines. Actually, if the goal is coverage then random preferences should give better coverage, but are probably not very useful, so just optimizing coverage doesn’t seem to be a good objective. Can you please clarify the objective here?

We adopted the methodology of (Santurkar et al., 2023) for evaluating our simulated user base on OpinionQA, in order to measure how well human preferences are represented by our simulated users. We felt that this made for the strongest basis of comparison, and also allowed including the baseline results from other LLMs without reproducing outputs. With respect to coverage, this was roughly our goal in evaluating across the 60 demographic groups. We aimed to ensure that the preferences exhibited by our simulated users were reasonable with respect to many different segments of the population, and found positive results. In consideration of the reviewer’s concern, we will attempt to expand these comparisons before the camera-ready version.

Regarding Figure 5, based on the reviewer’s suggestion we have extended these experiments to the persona prompting baseline. These new results can be seen in Figure 8.

Another potentially interesting baseline is to have each user choose one of the rewards, a hard choice instead of a weighted sum. There will only be 10 user “types”, so it may be interesting to see how the results change in that case.

We agree that this is an interesting setting, and may reflect many applications where users exist in tight “clusters”. We have clarified in line 277 that this can be achieved by lowering the alpha parameter for the Dirichlet distribution for the sampling of weightings.

Sometimes there are long multi-line sentences that could be simplified to improve readability and flow. It is easier to read a paper that has no sentences that span more than 2 lines.

We have shortened these sentences, and others that we felt were too long. Thank you for this suggestion.

Line 254: choosing only 500 personas may be too little if the goal is to achieve heterogeneity, especially since 1000 users are sampled for PersonalLLM. Can you please include results with 1000 personas? It may actually be interesting to see how the results change when increasing the sample size for both persona and PersonalLLM.

Based on this concern, we have updated Figure 4 (and Figure 8) with results from 1,000 randomly sampled personas, to make for a better comparison with the PersonalLLM simulated user population.

Hello, as the discussion period will be ending in a few days, we wanted to follow up and see if there are any remaining questions we can answer or any other changes we can make to address the reviewer’s concerns. Thank you again for the time and consideration.

Thank you for the clarifications.

We thank the reviewer for the time and consideration offered in reviewing our paper. We are encouraged to see that they see our work as original and clearly presented, and that they feel our testbed may make a significant contribution to the field of LLM personalization. Below we respond to particular points of feedback.

The paper uses reward models from a leaderboard (as opposed to fine-tuning to persona data or something), which means that the reward models are all high-quality, but may result in reward models which are less distinct from each other than they might otherwise be. The paper clearly justifies this as not preventing their resampling method from reaching higher diversity than persona-based prompting, but are there other sources of high quality reward functions that might be more different from each other?

We agree with the reviewer that it is worth considering other methods for producing a large and diverse set of high-quality reward functions, given the acknowledged shortcomings of our approach. We are not aware of any such methods at this time but hope that researchers take inspiration from this work and are able to develop more faithful and diverse simulators in the future. We have further acknowledged this concern and the need for further work in the “Future Directions” section of our paper.

Similarly, were the leading LLMs used to sample the 8 preferences prompted with personas? The different LLMs might be somewhat more similar to each other than they need to be, but of course resampling the dataset could be quite expensive, and the dataset is quite valuable as is.

We hit our budget constraint as an academic lab producing the LLM responses for our dataset, and were not able to probe the effects of persona prompting on these responses from these strong models. We agree that this would be a very interesting direction for future research, and hope this is enabled by the release of our dataset and pipeline code.

It's worth it to double-check that including the LLM responses in a dataset is within the relevant terms of use -- my impression is that generally they are, but it should be double-checked.

Thank you for this suggestion. We checked this before submission but did not explicitly state this in our dataset card in the appendix. We have added this explicitly in Section A.2.1.

Thanks for double-checking the terms of use! I stand by my positive assessment, and appreciate the authors explaining their constraints.

We thank the reviewer for their consideration of and feedback on our submission. Please see below for responses to specific questions and comments.

The paper is unclear about what the preference is in the data; is it user preference of items in recommender systems or any replacement of NLP tasks or others?

Our testbed is meant to explore user preferences with respect to different possible LLM responses to a user query. We have attempted to convey this in the first 3 paragraphs of the introduction, especially lines 40-46, as well as Figures 1-3. If the reviewer has any further suggestions to clarify this point we would be happy to update the submission.

The paper is uclear about how the PERSONALLLM is formulated, the author presented the reward model, but how it is trained/built up.

We detail our approach to producing simulated personal reward models in Section 2.2. To summarize, we address the challenge of simulating diverse user preferences using a set of strong, open-source RLHF reward models (sourced through RewardBench https://huggingface.co/spaces/allenai/reward-bench). We generate simulated users by sampling weighted combinations of these models, defining user-specific preferences as weighted sums of reward scores from a selected set of 10 high-performing models, such as Llama3 and Mistral. This approach enables scoring of any (prompt, response) pair through simple weightings, providing a scalable framework for studying personalization.

The author illustrates the heter preference PERSONALLLM involves in which differs from the home ones, but how these two preferences demonstrate is not clear.

One of our main goals in creating PersonalLLM was to create a set of preference models and data such that the preference models would offer heterogeneous preferences over the responses in the data. In order to verify our approach, in Section 3 we examine whether populations of personal preference models sampled via the method outlined in Section 2.2 do in fact display heterogeneous preferences over the prompt/response pairs in our dataset, and compare to the popular persona prompting baseline. In Figure 4 and the resulting analysis, we find that our method produces heterogeneous preferences over our dataset of prompts and responses, considerably more so than persona prompting an LLM. For example, under our method the most popular response to a query receives a majority user vote for only about half of the prompts, while that figure is closer to 90% for the persona prompting baseline. Also, for roughly 60% of prompts, at least 5 different answers are chosen as the best by at least 1 user under our set of simulated preference models; for LLM persona prompting, it is roughly 30%, meaning that for most data examples, at least half of potential responses are not preferred by a single user. Finally, our ensembled preference models have a much more diffuse set of preferences over the response-generating LLMs than persona prompting.

What is the relationship between PERSONALLLM and recommender system? Is it a replacement of existing ones or a more general preferenc-based system incuding RS? Why?

Our work on PersonalLLM is inspired by classic recommender systems in several ways. First, we aim for PersonalLLM to allow for the simulation of a large number of users, enabling the study of the full personalization paradigm for applications such as search engines and recommender systems wherein a historical database of user data is leveraged to personalize new interactions. We also build on the existing paradigm of using simulated rewards for developing recommender systems. Further, the setting in Section 4.2 resembles typical recommendation systems, but “actions” are now defined over the space of natural language outputs instead of a fixed set of items. We attempt to highlight this throughout the submission, but we will make sure to emphasize it further in the camera-ready version.

We thank the reviewer for the time and care taken in reviewing our submission. We are encouraged that they felt we had taken an impactful direction, and that they recognized the importance of using meta-learning to address the data sparsity issue in these settings. Below, we respond to particular points of feedback.

The personal preference models used to simulate diverse user preferences are not convincing enough to represent real users. First, it is difficult to verify whether the linear combination of scores from reward models aligns with the distribution of user rewards in the real world. Second, the candidate responses generated by LLMs may not cover real-world user-specific responses, making it challenging for LLMs to learn user-specific preferences or align with user-specific backgrounds. For instance, users may have particular preferences or habits that general reward models inherently struggle to account for when providing accurate rewards.

Our goal is not to produce a fully realistic simulation of human behavior, but instead to create a challenging simulation environment that can serve as an empirical foundation for innovation in LLM personalization. In this case, this primarily means creating a diverse enough set of preference models such that different users prefer different responses. In Section 3 we show that creating diverse preferences is challenging with existing approaches and that we resolve this technical bottleneck with our simulation method. Given that our testbed is the first to enable the study of settings where a large historical database of user data can be leveraged to personalize new chat outputs for new users, we believe that PersonalLLM represents a meaningful contribution towards advancing the personalization of language-based agents.

The paper lacks an overarching figure that illustrates the construction logic of the dataset and what the samples within the dataset look like.

We appreciate the reviewer pointing this out, and have added a new Figure 7 that illustrates the construction logic of the dataset. Further, we have added an example of what a data sample looks like to Appendix A.

The comparison of the paper with other relevant personalized LLM benchmarks, such as the LaMP dataset.

Thank you for pointing this out. We have added LaMP, as well as other relevant LLM personalization benchmarks, to our related works section.

Some related concepts are not clearly explained, such as 'interaction history', 'preference data', and 'user data,' which are not well defined.

Thank you for the suggestion, we have attempted to clarify these terms in the paper.