Aligning to Thousands of Preferences via System Message Generalization

Dear reviewer bEUV, we deeply appreciate your valuable feedback.

We provide responses to your comments in the weaknesses (W) and questions (Q) section.

Details of human annotation/evaluation (W1)

We employ 14 undergraduate students aged 20-24 from various majors, consisting of 6 males and 8 females. Hiring annotators with diverse backgrounds and preferences is an important aspect of human evaluation. However, in this study, human annotators assess the quality of the {system message, user prompt, response} set according to clear criteria, regardless of their personal preferences. They also evaluate which model better reflects a given preference in response generation. The Stage 1 and Stage 2 human evaluation process is detailed in Appendix F.1. Therefore, while individual preferences of the annotators are valuable, they do not apply in our specific context.

Analysis of response verbosity (W2)

In Appendix G.1 Figure 8, we plot the length distribution of responses and reference answers on Multifaceted Bench. It indeed shows that Janus’s responses are longer than other LLMs including Mistral 7B Instruct v0.2 and GPT 3.5 Turbo. Since the distribution of Janus responses and that of reference answers made by GPT-4-Turbo-0125 are similar, it can be seen as the result of supervised learning. Still, Table 3 shows that Janus outperforms models including the same Mistral 7B Instruct v0.2 and GPT 3.5 Turbo on AlpacaEval 2.0 when compared using the length-controlled (LC) win rate. The LC win rates are a debiased version of the win rates that control for the length of the outputs and are reported to improve leaderboard correlation [1]. Since Janus exhibits high performance consistently across benchmarks including length-controlled measures, we stress that verbosity does not play as a critical factor for evaluators to prefer Janus responses over those from other models.

Hacking the Chatbot Arena (Q1)

Could you please elaborate on your question regarding our method 'hacking' the Chatbot Arena? Are you suggesting that this hacking is a result of biases in the datasets or models, or are you referring to biases in the evaluation methods?

[1] Dubois et al.. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. arXiv 2024.

Dear reviewer bEUV, thank you for the clarification!

Our training dataset and method does improve the model in preference-based benchmarks like AlpacaEval 2.0 (Table 3).

However, our method is not centered on how to extract user preferences from the human population. Instead, we created training data simulating user preferences by distilling GPT-4 Turbo’s knowledge, which itself was effective in achieving both personalization and general helpfulness.

Also, since Chatbot Arena is updated via real-time user votes, we do not see it feasible to fit the arena users’ preferences as well.

We emphasize that our method cannot be used to mine or fit existing users’ preferences. The intended use of our method is to align to diverse individuals’ preferences and generalize to unseen system messages.

We hope this answers your question!

Dear reviewer k2dN, thank you for the important remarks that will help us better shape our contributions. We will address your concerns in weaknesses (W1, W2) and questions (Q1, Q2) below.

Redefining the scalability of our approach (W1)

Creating a synthetic dataset can indeed be costly, and it is challenging to consider the values of every individual in the world when generating data. However, this does not necessarily mean that the process is not scalable. According to previous studies [1, 2, 3], training with a sufficiently diverse and large dataset can enable language models to address problems requiring skills they have not explicitly learned. As shown in Table 2, all the preferences used in the benchmark are ones that Janus has not encountered during training. Nonetheless, Janus's superior performance compared to the baselines suggests that it possesses significant generalizability. We believe that our approach can overcome the scalability problem in individualized alignment by previous personalized RLHF approaches (Section 2).

Discussing the role of system messages for individualized alignment (W2, Q1)

The core of our methodology is to verbalize preferences in the model input and train a model on many of such instances to improve generalization. Our method is an extension of instruction tuning, where we control preferences in the instruction to give stronger signals to the model on fine-grained preferences than on general helpfulness and harmlessness dimensions. In this aspect, your comment on using different instructions in the user prompt to achieve individualized alignment is valid. Applying our training recipe using user prompts instead of system prompts would yield similarly strong results.

Nevertheless, we tried to separate the preference-specific instructions in the system message, since there is much underexplored potential in it for reaching alignment targets. From an application standpoint, the best scenario would be when the user specifies their preferences by themself in the user prompt, but in reality, users would not take much effort to do so. A way to improve user experience would be the developers inferring preferences (possibly based on past conversation history), verbalizing them in the system message, and aligning the model response with the user’s hidden interests. A model can be trained to assign highest privilege to the system message for more granular preference steering [4]. However, there is little work that experiments with diverse system messages (Section 2), so we aimed to conduct a study on the benefits of varying system messages for the same user prompt.

Typos (Q2) Thank you for the suggestions. We will go over the draft carefully and revise any inaccuracies.

[1] Sanh et al.. Multitask Prompted Training Enables Zero-Shot Task Generalization. ICLR 2022.

[2] Longpre et al.. The Flan Collection: Designing Data and Methods for Effective Instruction Tuning. ICML 2023.

[3] Kim et al.. Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models. ICLR 2024.

[4] Wallace et al.. The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions. arXiv 2024.

Dear reviewer k2dN, thank you for raising the discussion. We took your concerns carefully and will provide detailed comments below.

First, as per your main concern on our method’s similarity with instruction tuning, we agree with the following statement:

aligning with varying preferences through system message generalization seems somewhat similar to using diversified instructions in instruction tuning, aiming to enable generalization to other instructions.

System message itself is an instruction that the model should follow, so the basis of system message generalization is indeed in instruction tuning. With sufficiently many diversified system messages, our method takes advantage of the instruction tuning recipe to generalize to unseen system messages (i.e., individualized alignment targets).

However, our focus on the content differs from previous instruction tuning works. While previous works scale inputs, input-free tasks, or tasks per input (three paradigms described in [1]) in instructions, we aim to scale meta-instructions, instructions that guide how to respond to subsequent instructions. Our motivation was that different people expect different responses on the same instruction, so such meta-instructions that set preferences for task execution should also be taught to the model. Including this in instruction tuning helps the model approach various types of prompts and dynamically adjust response strategies, as shown in Section 5.1 and 6.2. Method-wise, our hierarchical data generation strategy enables careful curation of those meta-instructions to specifically simulate user preferences instead of presenting irrelevant challenges or hinting solutions with respect to the instruction. To the best of our knowledge, there is no work that shares a similar motivation with us or has publicly released a training dataset containing meta-instructions.

[1] Lou et al.. MUFFIN: Curating Multi-Faceted Instructions for Improving Instruction Following. ICLR 2024.

Lastly, we would like to elaborate on why we have set meta-instructions in the system message and how to further utilize them. While some meta-instructions can only be defined by a specific user message, it can also be more general. For example, the following system message in our dataset is associated with a user prompt giving a specific hyperbola equation, but is also applicable to other kinds of problems.

You are a mathematics coach, specializing in presenting complex mathematical concepts to users with an intermediate understanding. Your teaching method is known for its succinctness, offering clear, direct explanations and step-by-step walkthroughs without overcomplicating the information. To facilitate learning, incorporate visual aids like charts and diagrams, making abstract concepts tangible and easier to understand. Your approach always considers the users' math anxiety, striving to create an engaging, supportive environment that reduces stress and fosters confidence. Your aim is to make math accessible and less intimidating, ensuring users can grasp the essentials of mathematical theories quickly and effectively, with patience and creativity at the core of every interaction.

In this sense, meta-instructions can be effective when set programmatically on relevant user messages in the form of system messages. This (i) allows reflecting the users’ common needs on various instructions and (ii) lowers user burden. In practice, OpenAI has already shipped similar features like custom instructions [2] and memory controls [3] in ChatGPT; a user can specify their preferences in the custom instruction by oneself (e.g., When I ask you for code, please just give me the code without any explanation on how it works.) or ChatGPT will memorize the details itself from the conversations. When user preferences are gathered or summarized from interactions, it can be decided to be included as part of the system message without user specification for seamless chat experience.

Exactly how to infer user preferences or how to manage user controls on system messages is application-dependent and beyond the scope of our work (noted in broader impact in Appendix I). In the line of works on dialogue summarization [4, 5], we look forward to future work to explore the challenges of inferring preferences from conversations. The focus of our work lies more in representing user preferences in the form of meta-instructions and training models to follow meta-instructions. Also, from a technical perspective, training meta-instructions in system messages with accompanying gold responses would make the model (with appropriate system tokens) better distinguish between meta-instructions and instructions and tailor responses to the higher-level guidance.

We greatly value your feedback and will improve our writing to shape the storyline better.

[2] OpenAI. Custom instructions for ChatGPT. OpenAI Blog 2023.

[3] OpenAI. Memory and new controls for ChatGPT. OpenAI Blog 2024.

[4] Zou et al.. Unsupervised Summarization for Chat Logs with Topic-Oriented Ranking and Context-Aware Auto-Encoders. AAAI 2021.

[5] Zhao et al.. TODSum: Task-Oriented Dialogue Summarization with State Tracking. arXiv 2021.

Dear reviewer uZmQ, we appreciate your thoughtful comments.

Our responses to the issues raised in the weaknesses (W1) and questions (Q1) sections are detailed in the global responses (G2: Diversity, G4: Bias).

Diversity of User Preferences (Q1)

We measured the ROUGE-L score among user preferences in the training data and found an average similarity of about 0.2. Compared to previous works, this indicates sufficient diversity in the preferences present in our dataset. Furthermore, the model trained with this data, Janus, generates more diverse responses compared to baselines. This demonstrates the diversity of our dataset.

Bias and representativeness of human preferences (W1)

We tested Janus’ performance on three social bias benchmarks and discovered that Janus exhibits a bias level comparable to LLaMA 3 8B Instruct across all tasks and surpasses Mistral 7B Instruct v0.2 in most tasks. No significant bias issues are observed in Janus relative to other models.

We also followed experiments in previous work to measure the similarity between the human distribution and model distribution on survey questions and personality tests. Results using the Jenson-Shannon distance show that Janus shows 2x smaller decrease in similarity to Mistral 7B Instruct v0.2. The similarity is especially evident in terms of entropy distributions. These findings suggest that Janus is reasonably calibrated to the human population and that Multifaceted Collection can be representative of diverse human preferences to a large degree.

Please see more information in G2 and G4 of our global response.

Dear reviewer eXeU, we deeply appreciate your constructive feedback. Here we address the weaknesses (W) and questions (Q) that you raised on our work.

Discussing the potential risks of allowing flexible preference specification and available safeguards (W1,Q1)

We agree that training a model with flexible preference specifications may increase its vulnerability to misuse, such as jailbreaking. To address this, we incorporated safety considerations into our data generation process by including a harmlessness dimension in every system message. As discussed in G3, our method does not significantly compromise safety, as demonstrated through evaluations of both the dataset's safety and the model’s likelihood of generating toxic responses to harmful prompts. In future work, we plan to enhance our approach by filtering toxic data from the dataset, increasing the proportion of safety-related data in the training set, and utilizing AI moderation technologies like LLaMA-Guard and Prompt-Guard.

Demystifying the factors that contribute to performance gains (W2, Q2, Q5)

(1) Number of training data

We conducted an ablation study to examine the effects of data scaling. Janus 7B was trained on a dataset of 197k instances, where each user prompt is paired with 3 different system messages and responses. To test the effects of scaling, we reduced the number of system messages and responses per user prompt to 2 and 1, resulting in datasets of 132k and 66k instances, respectively, and trained separate models. The results indicate that, across five benchmarks, the scaling effect is evident: increasing the amount of training data leads to higher scores.

(2) Hierarchical data generation strategy

We prompted GPT-4-Turbo-0125 to generate preferences freely (a preference set of four values or a single detailed preference description) and qualitatively compared them to our original hierarchically generated ones. On ten samples, we observed that free-form preference generation can create more topic-specific preferences, but oftentimes they deviated from what we expect preferences to be. Specifically, some generations included preferences irrelevant to the goal of the user instruction (e.g., a preference for nutritional benefits in a math problem illustrated with apples) or attempted to resolve the user request and hint solutions (e.g., explicitly instructing correct implementations for a coding problem).

These problems arise because the model needs to understand and elicit preferences from the human user side, not the assistant side. Our hierarchical data generation strategy allows sufficient control over synthesizing what users would expect in the response. We have decided that preferences for any response would differ under the style, background knowledge, informativeness, and harmlessness dimensions based on various existing literature (See Appendix A.2). Our method of providing the dimensions in context, coupled with manually crafted seed examples, is instrumental in obtaining high-quality, individualized preferences.

Please see our global response for further verification of our synthetic data and method.

Effectiveness of our method as model scales (W3, Q3)

We explored how variations in the type and size of the base model affect performance. Initially, we used Mistral 7B v0.2, but we also experimented with LLaMA models. The results in the table indicate that LLaMA 2 7B performs similarly to or slightly worse than Mistral 7B v0.2. However, increasing the model size from 7B to 13B (LLaMA 2 7B vs. 13B) results in a clear performance improvement. Notably, the latest model, LLaMA-3 8B, surpasses the larger LLaMA 2 13B in benchmark scores (LLaMA 3 8B vs. LLaMA 2 13B). Therefore, both model size and the capabilities of the base pre-trained model significantly impact performance when applying our method.

Consistency of Janus in maintaining preferences in multi-turn scenarios (Q4)

To assess whether Janus effectively adheres to diverse preferences in multi-turn scenarios, we conducted an evaluation focusing solely on the multi-turn questions from the MT-Bench, excluding single-turn questions. The results indicate that Janus consistently outperforms the baselines, Mistral 7B Instruct v0.2 and LLaMA-3 8B Instruct (6, 6.6 vs 6.8). This is consistent with the trends observed in Table 3, suggesting that Janus remains highly competitive in multi-turn settings.

Dear Reviewers,

We appreciate for your time and effort on reviewing the paper.

As we near the end of the discussion period, we kindly remind you of the upcoming deadline.

We are eager to discuss any aspects of the paper that may require further clarification.

Thank you once again for your valuable feedback.