On Orchestrating Personalized LLMs

Thank you for your encouraging comments and constructive feedback. We address individual points below.

The method has some limitations addressed in the paper: having to train on all preference pairs (which may make scaling to a large number of preferences less feasible), having to retrain the model if new preference types are introduced, and having a larger memory footprint compared to parameter merging methods at inference time.

Thank you for the comment. We would like to provide some thoughts on the limitations.

Train on all preference pairs: Note that only a single PCM is trained with RL where we uniformly randomly sample preference combinations during training. This can be considered as an RL training procedure with diverse initial state distributions (i.e. preference combinations in addition to instructions are provided to the PCM as prompts). The training ensures that one RL procedure can learn to perform well on average across all preference combinations. We note that we do not need to perform training or maintain a separate PCM per preference combination which is indeed not tractable.

Large memory footprint: We agree with the reviewer that the memory footprint of MPD is larger than weight merging baselines but the inference time is actually more efficient as discussed between Line 489-506. One potential memory footprint reduction technique is to use lower precision for the expert models while still keeping full precision of PCM. Because the output probability is weighted averaged / ensembled, the output responses from MPD should be more robust to lower precision.

Results shown are always winrate over all prompts, which makes it details on where and how the method is improving unclear beyond a handful of qualitative examples. I would be curious if breaking down results per-preference dimension (shown in Table 1) and comparing with baselines would yield any interesting observations - perhaps some methods are better than others at particular preference dimensions?

Thank you for the great suggestion and below we provide the win rate statistics of MPD against two baselines on the individual preference dimensions. It can be seen that MPD indeed performs differently: 1A (elementary) and 2A (concise) consistently outperforming baselines, whereas 1B (knowledgeable) being slightly worse than baselines.

Is preference prompting used for the MPD (Uniform) setting (for generating responses)? I don’t quite understand how this could yield better personalization otherwise.

Yes, the reviewer is correct. We used the same preference prompting for Personalized Soup and MPD to ensure a fair comparison. Intuitively, it is possible that MPD (Uniform) outperforms Personalized Soup. The difference is that Personalized Soup averages model parameters and MPD averages model outputs. The results show that averaging model outputs performs better and it is not very surprising since averaging outputs might be more interpretable than averaging parameters.

How many expert models are trained? Is it one per row in Table 1, or one per A/B pair? I found this a bit unclear.

Sorry for the confusion. There are six expert models trained (one per row in Table 1). We will explicitly mention this in our revision.

I am curious about how well an expert-only baseline would do, where we compare just using one of the expert models for generation - do we gain anything by merging the output distributions token-by-token, or does this perform similarly to picking the best output from relevant expert models?

Thank you for the suggestion. In our preliminary experiments, we found that when our actual reward is only based on one of the preference dimensions, the PCM would learn to allocate a weight of 1 to that dimension and 0 to the other dimensions after training. This suggests that the expert is indeed very strong in personalization for the individual preference dimension and our RL-based approach will learn to converge to the correct expert.

Thank you for the detailed review and constructive feedback! We address individual points below.

Limited Preference Dimensions and Generalizability: The selected preference dimensions appear arbitrary and lack sufficient justification. The paper does not adequately address the generalizability of MPD to a more diverse and nuanced range of human preferences. Given the limited number of dimensions explored, it remains unclear how MPD would perform with a substantially larger set, which would be necessary to capture the true complexity of human preferences.

We first highlight that there is no standard benchmark or preference dimensions commonly used in related work [1, 2, 3]. For a fair comparison, we closely follow the experimental setup of [1] which is one of the first works using preference experts for multi-objective personalization. As discussed in [1], the three preference dimensions correspond to (expertise, informativeness, style) with two opposite options for each dimension (A and B as seen in Table 1). We also note that other related work such as [3] also only experiments with three preference dimensions. We chose to follow the setting in [1] for a fair comparison to their approach.

Scalability of Preferences and Combinatorial Explosion: The necessity of enumerating all preference combinations during training introduces a scalability bottleneck. While the paper acknowledges this limitation and briefly touches upon handling new preferences, the proposed approach does not convincingly demonstrate MPD's scalability to a higher-dimensional preference space. The combinatorial explosion inherent in an increasing number of dimensions raises concerns about the feasibility of training MPD with a realistically large preference set.

How does MPD's performance scale with an increasing number of preference dimensions? What strategies are being considered to mitigate the combinatorial explosion of preference combinations during training?

Thank you for the great question and we fully acknowledge that MPD is trained on all preference combinations. We would like to highlight that, however, only a single PCM is trained with RL where we uniformly randomly sample preference combinations during training. This can be considered as an RL training procedure with diverse initial state distributions (i.e. preference combinations in addition to instructions are provided to the PCM as prompts). The training ensures that one RL procedure can learn to perform well on average across all preference combinations. We note that we do not need to perform training or maintain a separate PCM per preference combination which is indeed not tractable. In contrast, Personalized Soup indeed needs to perform model merging at inference time for all possible combinations, which can be intractable.

Besides, as discussed briefly on Line 511, other multi-objective approaches such as [2] that require training are also trained on all preference combinations. In fact, the training of MPD is more efficient than other approaches since it does not calculate gradients for the multiple expert models. The gradient is only calculated on the much smaller PCM model.

Justification for Tulu-7B and Model Size: The rationale for selecting Tulu-7B as the base model is insufficiently explained. An evaluation of MPD using more widely adopted, powerful LLMs (e.g., Llama 3, Qwen2.5, Anthropic, GPT, etc,) would significantly strengthen the paper's conclusions. Furthermore, the relatively small size of the tested model (7B) raises questions about the robustness and generalizability of the learned merging strategy. The observed win rates near 50% suggest a limited effect size, potentially attributable to the PCM's capacity.

What was the rationale for choosing Tulu-7B as the base model? Have experiments been conducted with other similarly sized LLMs (e.g., Llama 3 8B, Qwen2.5 7B) or larger, more powerful models (Gemini, Llama 70B, Sonnet, etc)? If so, how do the results compare?

We used Tulu-7B because we want to have a direct and fair comparison with [1] from which we also adopted the experimental setup. To ensure PCM is not limiting the performance, we also carry out an experiment where we directly finetune the base Tulu-7B model with the same reward function and RL learning algorithm on all preference combinations. This should be considered as an upper bound of MPD. MPD has a 49.15% win rate against this approach, suggesting that MPD is achieving strong personalization results.

Thank you for your valuable and insightful comments. We address individual points below.

Significance of the Problem: The manuscript describes a solution for a "black-box method" using trained expert models for each preference dimension. However, the necessity and practicality of these black-box expert models are questionable as they do not seem to exist in practice. As described in the manuscript, the expert models are not pre-existing and are instead trained by the authors following the procedure from Jang et al. (2023). This approach contradicts the concept of using already available black-box models and raises questions about the real-world applicability of the proposed method.

Thank you for the great question! As discussed in Appendix A.1 Line 706, we trained the expert models ourselves only because [1] did not release trained checkpoints for the expert models. We also believe that black-box methods are more advantageous since it implies that we will not compute the gradient of expert models which makes the approach much more feasible if the expert models are too large or too expensive to be finetuned.

We also give our thought on if expert models naturally exist. We are under the impression that there do not exist many pre-trained expert models in academia. However, we do believe expert or personalized models exist in the industry and have real-world applicability. For example, Character AI has a diverse set of characters, each has some unique personality and is tailored towards the audience with different personalization goals.

Lack of Technical Novelty: Based on the approach described, I do not find significant innovation in this paper, especially in comparison to existing literature cited as [1-3], particularly [3]. The content in Section 3.1 looks similar as in [1], and Section 3.2 seems to extend existing work on RLHF and REBEL without introducing substantial new insights or advancements. This overlap with prior works might limit the perceived contribution of the current study.

Thank you for the references! We do not believe Section 3.1 looks similar to [1] or [2] since both work merge model weights instead of model outputs. We would appreciate it if the reviewer could elaborate on which specific parts are similar.

For Section 3.2, we actually intend to make MPD training simple without introducing any novel RL components (this is not a new RLHF paper). In fact, our experiments demonstrate that any online RL algorithm such as PPO and REBEL can be applied for MPD training. Our main contribution is a framework and proof-of-concept where a small PCM that controls how token-level output probability from experts are merged can achieve strong results for multi-objective personalization.

The main distinction between MPD and [3] is that [3] directly merges the output from expert models without any training of the merging weights. In our manuscript, this is an ablation of MPD, namely MPD (Uniform) where we uniformly merge the outputs from the experts. In Table 3, the results demonstrate that by further training a small PCM model, we can improve the win rate by more than 6%, suggesting the effectiveness of our approach compared to [3].

Thank you for your constructive feedback and pointing out the scalability of our approach! We address individual points below.

Details about the architecture of the Preference Control Model (PCM) are limited and could be expanded.

Sorry for the confusion. In Section 4.2, we describe the architecture of PCM, which is a 160M Llama-based model. The small size is attributed to a small hidden size and fewer number of layers compared to Llama 2. The last linear layer of this model is replaced with a much smaller linear layer of size number of preference dimensions ($n = 3$).

The MPD training approach appears conventional, lacking novelty in this aspect.

We actually intend to make MPD training simple and any online RL algorithm such as PPO and REBEL can be applied for MPD training. Our main contribution is a framework and proof-of-concept where a small PCM that controls how output probability from experts are merged can achieve strong results for multi-objective personalization.

The code is not open source, which limits reproducibility. The reasons for its performance advantages over Personalized Soup are not clearly explained.

We will make sure to open source our code in the final version of our paper. Both Personalized Soup and MPD use the same set of experts that specialize in individual preference dimensions. The difference is that Personalized Soup averages model parameters and MPD learns to dynamically mix model output distribution at the token level (i.e., mixing weights can be different at different decoding steps). Our results demonstrate that dynamically mixing the model output distribution at token level performs better. Personalized Soup averages the model parameters uniformly regardless of the prompt; in contrast, MPD learns to mix outputs based on preference and prompts. The ability to adapt to different prompts and mix at token level dynamically makes MPD outperforms Personalized Soup.

In Figure 1, which components represent the expert models? Are these expert models distinct or identical?

Sorry for the confusion; the expert models are denoted as $M_i$ (LLM personalized for X) in Figure 1. These expert models are frozen throughout the training of MPD and can either be identical or distinct as long as they have the same tokenization. In our experiments, these experts have the same architecture.

Do the expert models utilize different prompts to reflect varied preferences?

Thank you for the question! No, each expert model always uses a fixed preference prompt regardless of varied preferences. The preference prompt for each expert can be found in Table 1.

If the expert models are identical, what advantages does MPD offer over using a single model for preference alignment?

Thank you for the great question. The expert models have identical architecture but not identical weights. Compared to the single instruction-tuned base model, as seen in Table 2, MPD offers significantly better personalized results than Preference Prompting. Compared to alternative RLHF approaches that directly finetune the base model, MPD is much more lightweight and can be used with blackbox base models since it only finetunes a small PCM module and does not assume weight access of base models.