LoRe: Personalizing LLMs via Low-Rank Reward Modeling

We thank the reviewer for the positive assessment of our work. Below we provide answers to your concerns:

Trade-off between performance and complexity: If we understand correctly, the reviewer is asking if the model complexity (number of parameters – controlled by the rank) affects the unseen performance. We present the analysis in point 3 later. We apologize if we misunderstood the question, and are happy to provide answers post clarification.

Integration into Multi-Objective Alignment: We agree with the reviewer’s point. We didn’t focus on response generation due to two key challenges in evaluating personalized response generation:

(i) Lack of accessible annotators: The datasets we use were constructed with human annotators whose preferences are no longer available. This makes it infeasible to evaluate newly generated responses with human-in-the-loop feedback.

(ii) Subjectivity of evaluation: Personalized preferences are highly individual, and evaluating generated responses without access to the original users introduces significant ambiguity and bias. This makes standard qualitative evaluation unreliable.

As a practical and scalable alternative, we adopt a Best-of-N (BoN) sampling strategy [1] to assess whether LoRe reward models guide useful downstream generation. For each prompt and user, we sample multiple responses from GPT-4o and rank them using either the LoRe reward model or baseline models (ref, BT, VPL, PAL). We then use GPT-4o (conditioned on prior preference examples) as an LLM judge [2] to compare response quality. On a subset of 20 users from the PersonalLLM dataset, LoRe-ranked responses win 68% of pairwise comparisons, indicating strong downstream utility.This experiment serves as a fast and informative proxy to full multi-objective alignment.

Importantly, seminal prior work highlights that response generation quality depends heavily on accuracy of reward models:

“The effectiveness of RL depends heavily on the accuracy of the reward model... If the reward model assigns high scores to summaries that are not aligned with human preferences, the policy trained with RL will not improve quality.” ([1], Section 6.3)

“We evaluate models trained with weaker reward models and find that performance degrades significantly, confirming that reward model accuracy is critical.” ([3], Section 5.2)

Discussion on Rank Selection: We refer the reviewer to our general response, where we present a table on how the performance varies as a function of the rank. As we discuss in the general response, varying the rank helps us trade off model performance and generalization.

[1] "Learning to summarize with human feedback." Stiennon, Nisan, et al.

[2] "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." Zheng, Siyu, et al.

[3] “Training language models to follow instructions with human feedback” Ouyang et. al

We thank the reviewer for appreciating our work. We hope to address your questions below:

Basis Coverage and Representation Power (Q1):

(a–b) If the training data reflects a diverse set of user preferences, the learned basis reward models can, by design, span the subspace of preferences present in the data. This is analogous to classical matrix factorization, where the low-rank basis captures the main modes of variation. In LoRe, individual user reward functions are parameterized as linear combinations over this basis, allowing us to approximate any reward function expressible within the training span.

(c) When training data exhibits low preference diversity—e.g., when users have highly similar taste – the learned basis will be low-rank, and may not generalize well to users with orthogonal preferences. However, this limitation is not specific to our method—it arises in all personalization approaches that rely on fixed training distributions. To address this, LoRe naturally supports continual learning: the basis can be expanded or refined as new users arrive in practice (e.g., in streaming or batched updates). In our experiments, due to current preference datasets having limited user count, we trained the basis on a subset and evaluated few-shot personalization on held-out users to mimic this real-world setting.

On Training an LLM Policy Using the LoRe Reward Model (Q2): While training an LLM policy directly with LoRe-based rewards would be ideal, there are two key challenges that make evaluation difficult:

(i) The annotators who originally provided the preference data are no longer available, making it infeasible to obtain human feedback on newly generated responses;

(ii) Qualitative evaluation is inherently subjective and biased, particularly in the absence of the original users who provided the feedback.

As a practical and scalable alternative, we adopt a Best-of-N (BoN) sampling approach [1] to approximate policy optimization. We generate multiple responses using a base LLM (GPT-4o), then rank them using the user-specific LoRe reward model. This allows us to assess the reward model’s effectiveness in selecting high-quality personalized responses. For comparison, we also rank responses using baseline reward models (reference, BT, VPL, PAL).

To evaluate the selected responses, we prompt GPT-4o (with the user’s prior preference examples in context) to serve as an LLM judge [2]. On a held-out subset of 20 users from the PersonalLLM dataset, responses ranked highest by LoRe win 68% of the pairwise comparisons against those selected by the baselines.

[1] "Learning to summarize with human feedback." Stiennon, Nisan, et al.

[2] "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." Zheng, Siyu, et al.

We thank the reviewer for finding LoRe simple yet effective. We hope to address your concerns below:

Personalized response generation: We can generate personalized responses using our learned reward models via a Best-of-N (BoN) sampling strategy [1]. Specifically, for a given user and prompt, we sample N candidate completions from GPT-4o and rank them using the user-specific reward model learned by LoRe. The top-ranked response is then selected as the personalized output. We will add some examples in the appendix of the paper. While it is inherently difficult to qualitatively assess personalization from a few examples, especially without access to the original annotators, we provide a quantitative evaluation using an LLM-as-a-judge setup as discussed next.

Human Evaluation: We are unable to conduct new human evaluations, as the annotators who provided the original preference labels were contracted independently by the dataset creators and are no longer accessible. As a scalable alternative to costly and complicated task of building a new human evaluation dateset, we can use an LLM-as-a-judge setup [2], where the model is shown a user’s past interactions and asked to select the response that best matches their preferences.

To assess downstream utility of the learned reward models, we ran an LLM-judge (prompted GPT-4o to choose the better response given the user’s past responses) evaluation on a subset of 20 users from the Personalized LLM (PersonalLLM) dataset. Responses generated via BoN sampling and ranked using LoRe’s reward models were preferred 68% of the time over the best of existing baselines: reference model, BT, VPL, and PAL. This shows strong promise that our learned reward models meaningfully improve downstream response generation and align well with user preferences.

Discussion on selection of rank: We refer the reviewer to our General Response, where we detail the process of selecting rank. The chosen values of rank are highlighted in bold in the table for the different settings.

L144 clarification: We model the weight vectors as $w_i$ = softmax($q_i$), where $q_i$ is the learnable parameter, via gradient descent on the objective in Eq. 10. This way the weight vector is automatically projected onto the probability simplex.

[1] "Learning to summarize with human feedback." Stiennon, Nisan, et al. [2] "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." Zheng, Siyu, et al.

We would like to thank the reviewer for their constructive feedback on our paper. We are glad that you found our work on LoRe promising, and we appreciate your suggestions on areas where we can improve clarity and further strengthen our contributions. Below, we address the points raised in your review, and are happy to make any more clarifications during the discussion period:

Clarity of Methodological Advantage: Thanks for this question. We want to illustrate this with an example first. Suppose there are 2 responses y_1 and y_2 in response to a prompt x, and suppose there are 2 users: user 1 who likes y_1 more than y_2, and user 2 who likes y_2 more than y_1. The conventional monolithic reward takes as input a prompt and response and maps to a scalar output. Thus it’ll always conform to one of the user’s preferences (say rank either y_1 over y_2, or rank y_2 over y_1 but never both). Thus simply scaling up parameters doesn’t resolve the issue. One potential workaround is for the reward model to take as an additional input, the previous feedback pairs in the context of the reward models. However, the pre-trained reward models are usually trained without such additional history of feedback, and would need to be trained from scratch to be able to capture the user’s preference history directly in the reward function. Second, the inference costs are much higher and build up as the user accumulates more history. Our method parametrizes the history into a low dimensional weight vector. Another variant of this is already studied by the baselines PAL, VPL where they learn a function to produce a latent code of the user preference, and pass in their monolithic reward model. We found LoRe outperforming these methods (also refer to detailed discussion on analysis later in our response).

Overfitting: To avoid overfitting in the Bradley-Terry model, we monitor validation accuracy on a held-out cross-validation set across training epochs and select the model checkpoint with the best performance. This approach has also been utilized in OpenAI’s seminal paper on RLHF [1], where cross-validation is used to avoid overfitting. We will add this to the appendix.

Sample Size needed for personalization: The reason why LoRe is sample efficient is that the only learnable parameters for a new user is the low-dimensional weight vector w_new. This design choice goes hand in hand with being able to adapt with few-shot examples.

Positioning of Related Work and Multi-Objective Alignment Section: We will move the section on multi-objective alignment to the related work section, as suggested by the reviewer, to maintain a smoother flow of our contributions. Our original intent was to present a thorough comparison with baselines like VPL and PAL after introducing the necessary preliminaries. We placed this discussion before the experiments to clarify the baselines’ mechanisms and improve interpretability when analyzing performance results.

[1] “Training language models to follow instructions with human feedback” Ouyang et. al