MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models

Thank you for reviewing our paper and recognizing its strengths. We hope the following responses can help address the concerns.

Weakness 1: MetaAligner has a simple methodology. It triggers 2 novel capabilities:

1. instance-level alternation of the alignment objectives without re-training;
2. generalizable multi-objective alignment to align unseen objectives in a zero-shot manner.

We designed dynamic objectives reformulation (Sec. 3.1) to build training data with various combinations of target objectives which are used to train MetaAligner in a conditional weak-to-strong correction manner (Sec. 3.2). With the dynamic multi-objective capability triggered by the above modules, We further extend it to generalizable inference (Sec. 3.3) with the in-context learning ability of MetaAligner. All modules are centered on achieving the two capabilities.

We didn't show the ablation results because MetaAligner has a simple and intuitive structure, and most components are essential for the models. We agree that showing the ablation results for $D_e$ can be useful evidence for proving its effectiveness. Due to time limits during rebuttal, we will add these results in the future version of our paper.

Weakness 2: We admit that the MetaAligner can introduce extra inference costs (we discussed in limitations, appendix A.2). But during training, MetaAligner has fewer trainable parameters and lower costs:

1. All policy model weights are fused and only MetaAligner parameters are updated. When using LLaMA2-7B as policy (Table 3), MetaAligner-1.1B has much fewer trainable parameters than the baseline methods, but with comparable or better performance. MetaAligner-(7B,13B) have comparable or more trainable parameters, but with much better performance.
2. MetaAligner models can be trained once and used for all policy models (see Table 2), while the baseline algorithms have to be trained from scratch for each new policy model, as they update policy parameters.
3. The efficiency of MetaAligner improves with larger policy models. Note that we used LLaMA2-7B in Table 3 as the policy model due to limited computation resources. With larger policy models (e.g. LLaMA2-70B), the training cost for baseline methods vastly increases, while the training cost for MetaAligner remains constant.

Weakness 3: Compared to the user-specified weights for all objectives, MetaAligner’s weight-free specification of desired objectives and zero-shot alignment on unseen objectives are more useful features that no other methods achieve.

1. The optimal weight assignment is hard to determine because the Pareto front for multi-objective alignment is usually hard for humans to perceive and quantify[2]. We believe users are more concerned with properly improving certain objective combinations than quantifying their weights by themselves. With MetaAligner, the users can simply identify their desired objectives, and the model can modify the response accordingly with an implicit optimal Pareto front learned during training. We believe our proposed solution is more user-friendly in practical scenarios.
2. Compared to previous methods that require users to assign weights to every trained objective, MetaAligner allows users to customize their own combinations of objectives for each instance, which is more flexible.
3. MetaAligner allows users to specify new objectives by providing text descriptions, and perform zero-shot alignment on these unseen objectives, a feature no previous methods have achieved.

Weakness 4:

1. We believe the core of the reorganization process is not the prompt, but identifying target objectives for each chosen-rejected pair, the key to the dynamic multi-objective alignment.

2. MetaAligner shows 2 key advantages over previous SFT methods: (1) MetaAligner achieves unlimited simultaneous alignment objectives at the instance level, with the dynamic multi-objective alignment training; (2) MetaAligner achieves zero-shot alignment for unseen objectives, with its in-context learning ability.

3. Prompt-based inference is the optimal way to leverage the in-context learning ability of MetaAligner to achieve generalizable alignment. It's also widely used in previous works[1][2][4]. Following their settings, we provide empirical evaluations on generalizable alignment (Sec. 4.4). Providing theoretical guarantees is hard and beyond the scope of most related works due to the low interpretability of deep models.

Weakness 5: MetaAligner can have extra inference costs, but these are reasonable trade-offs for savings during training. Due to word limits, please refer to our response to Weakness 1.

Question 1: Enhancements in one objective can affect performance in certain objectives due to their controversial nature, known as "alignment tax"[2]. For example, aligning on "Fair" with MetaAligner-(7B, 13B) benefits its win rates, but harms performance on objectives such as "Readable" and "Factual" compared to when "Fair" is unaligned. This phenomenon is expected and unavoidable. We will include these analyses in the future paper.

Question 2: MetaAligner training is simple. We searched learning rates between 5e-6 and 5e-5. All hyperparameters are presented in Table 5 in the appendix.

Question 3: We tried to ensure a fair comparison. For MORLHF, we set uniform preference weights, as the weight search process for over 2 objectives can be costly. The weights determination process of MODPO is presented in appendix G.2. RiC doesn’t involve assigning weights, but a preference-to-reward mapping process (appendix G.1).

References: [1] Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment.

[2] Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment.

[3] Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards.

[4] SALMON: Self-Alignment with Instructable Reward Models.

We sincerely thank the reviewer for reviewing our paper and recognizing the strengths of our work. We also thank the reviewer for willing to consider raising the scores. The following parts contain our point-to-point responses to the weaknesses and questions. We hope they can help address the reviewer's concerns.

Weakness 1: We apologize for not including more details about Aligner in the main body due to page limits. Aligner is an SFT-based single-preference alignment method, which trains the model to correct the weak responses to approach the strong responses. The advantages of MetaAligner over Aligner are as follows:

1. MetaAligner is the first work to explore dynamic multi-objective alignment which enables instance-level alteration of the alignment objectives without re-training and unlimited simultaneous alignment objectives. In contrast, Aligner can only perform single-preference alignment. To achieve this capability, MetaAligner is trained to approach chosen responses from rejected responses, considering dynamic target objectives. This training paradigm is not explored by Aligner or any other alignment methods.
2. MetaAligner is the first work that can perform generalizable multi-objective alignment, which can align unseen objectives in a zero-shot manner, while Aligner or any other previous multi-objective alignment works can only perform alignment on objectives that they were trained on. To achieve this capability, we innovatively leverage the in-context learning ability of MetaAligner to understand the unseen objectives and plan for alignment, allowing the users to flexibly describe their alignment objectives in a natural language format, another key difference from the methodology of Aligner.
3. Different from Aligner, We conduct experiments on datasets in both general (HH-RLHF, UltraFeddback) and mental health (IMHI) domains, comprehensively showing the effectiveness of MetaAligner. We also conduct experiments to prove MetaAligner’s generalizable alignment ability to unseen objectives and evaluate its accuracy in objective-wise alignment, which is not covered by Aligner.

We will include more of these explanations in the future version of our paper.

Weakness 2: We apologize for the complexity of Figure 2 and will revise it to provide a clearer depiction of MetaAligner. We will discard the MORLHF and MODPO illustrations, and reorganize the MetaAligner pipeline from left to right as follows: (1) dynamic objective reformulation, introducing the building process of the dynamic dataset; (2) conditional weak-to-strong correction; (3) generalizable inference. We will simplify the current complex illustration within each stage. Due to time limits during rebuttal, we will include the modified figure in the future version of our paper.

The extension to multi-objective alignment occurs at both the methodological and dataset levels. Methodologically, we are the first to propose dynamic multi-objective alignment to enable instance-level alternation of the alignment objectives, which is achieved by objective-aware weak-to-strong corrections. We are also the first to perform generalizable multi-objective alignment to align unseen objectives, which is achieved by MetaAligner’s in-context learning ability. At the dataset level, we use Algorithm 1 to reorganize the dataset, training MetaAligner to handle various combinations of alignment objectives, which is crucial for its flexible and zero-shot alignment capability.

Weakness 3: Different from Aligner which utilized LLMs to correct the weak answers to generate correction pairs, we leveraged existing response pairs within alignment datasets (e.g. HH-RLHF, UltraFeedback) to supervise weak-to-strong corrections without needing to explicitly generate correction pairs.

Specifically, MetaAligner is trained to generate improved outputs using weak responses as references and strong responses as supervision, guided by the target objectives from the dynamic multi-objective dataset. The target objectives of each response pair are obtained via their preference annotations in the original dataset. More details are described in Section 3.1.

Weakness 4: MetaAligner and Aligner can have fundamentally different evaluation processes. Aligner uses a single-preference approach, generating one correction per policy output and evaluating it across "Helpful," "Harmless," and "Honest" dimensions. In contrast, MetaAligner can explicitly incorporate target objectives in the input, allowing it to generate different corrections for the same output based on specified objectives, a key advantage over Aligner. We didn't include the objective-specific evaluation results because as Section 4.5 demonstrates, MetaAligner usually provides better results when controlling on the target objectives than full alignment.

We agree that including comparisons with Aligner in different evaluation settings would further enhance our arguments. We provide our latest results, comparing the performance of MetaAligner-7B and Aligner-7B on the LLaMA2-7B-chat policy model:

For UltraFeedback we have:

These results show that though Aligner is comparable to MetaAligner on objectives such as “Helpful” in HH-RLHF, it significantly underperforms MetaAligner in fine-grained objectives such as “Humour” and “Truthful”. These results show the effectiveness of the dynamic multi-objective alignment method. Objective-specific alignment further improves the performance of MetaAligner.

These results will be included in future versions.

Dear Reviewer E7pS,

This is a gentle reminder that we have provided a detailed response to your concerns. Please note that the discussion period is ending soon. We'd appreciate it if you could find time to check the response and further consider your evaluations of our paper. We look forward to our further discussions based on these new clarifications.

Best wishes,

On behalf of the authors of Paper 5185

We sincerely thank the reviewer for reviewing our paper, which provides careful assessments and valuable comments. We also thank the reviewer for recognizing the strengths of our work. The following parts contain our point-to-point responses to the weaknesses and questions. We hope they can help address the reviewer's concerns.

Response to Weakness 1: We adopted the win rate evaluation method because it is a widely recognized approach in SOTA benchmarks such as AlpacaEval [1], Arena-Hard [2], and MT-Bench [3]. For instance, Arena-Hard uses win rates against GPT-4 outputs to measure model performance, a practice followed by many recent alignment studies [4][5]. This method is considered both effective and robust.

In our evaluation of MetaAligner, we calculated win rates for each policy model's output against ground truth answers both before alignment: $W_b$ and after alignment $W_a$. The performance of MetaAligner was quantified by the difference in win rates: $W_a-W_b$, as shown in Table 2.

When comparing MetaAligner performance based on the same policy model (Table 3 and Figure 3), we reported win rates after alignment ($W_a$) directly, because all methods used the same policy model ($W_b$ are the same), ensuring a fair comparison. We believe these methods provide a reliable assessment of MetaAligner's performance advantage.

Response to Weakness 2: We agree that including prompt engineering-based aligners would further demonstrate MetaAligner’s effectiveness. This approach involves obtaining an initial response and then prompting the same model to refine it. However, this method often requires aligner models with strong in-context learning capabilities, leading to high inference costs due to larger model sizes or expensive commercial models. MetaAligner offers a cost-effective advantage by employing a smaller model (e.g., MetaAligner-1.1B) for the refinement stage, reducing inference costs while maintaining competitive performance through supervised fine-tuning on the alignment dataset. One MetaAligner model can also be applied to different policy models.

To further address the concern, we take time to add a new prompt-based baseline using LLaMA2-70B-chat with one-time self-refinement. For HH-RLHF, the results are as follows:

For UltraFeedback we have:

According to the results, MetaAligner-7B can significantly outperform LLaMA2-70B-chat performance on 6 out of 7 objectives, with only 10% in inference cost. MetaAligner-1.1B can also achieve better or comparable performance on 4 objectives, but with only 1.5% in inference cost. These results prove the significant advantage of MetaAligner over the refinement-based methods. We will include these results in the future version of our paper.

Response to Question 1: As noted in our response to Weakness 1, our evaluation method measures performance based on relative differences in win rates before and after alignment, ensuring that the capability of the policy model does not affect evaluation accuracy (see Table 2). While strong models like LLaMA2-Chat-70B and ChatGPT may start with high win rates (high $W_b$), their influence is neutralized by subtracting these rates from the aligned results ($W_a-W_b$), isolating the contribution of MetaAligner. This approach ensures that our assessment reflects the genuine impact of the MetaAligner modules.

Response to Question 2: We thank the reviewer for carefully assessing the statistics and pointing out this mistake. We mistyped one of the figures when drawing the heat map with Python codes. We have rechecked and the percentage for “helpful” should be 0.58. We will correct this error in the future version of the paper.

References:

[1] Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv: 2404.04475

[2] From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline. arXiv: 2406.11939

[3] Judging llm-as-a-judge with mt-bench and chatbot arena. NIPS 2023.

[4] SimPO: Simple Preference Optimization with a Reference-Free Reward. arXiv: 2405.14734

[5] Low-Redundant Optimization for Large Language Model Alignment. arXiv: 2406.12606

The authors thank the reviewer for considering our response and raising the score. We are glad that our response addresses most of the reviewer's concerns. We will review and modify the figures and data in the future version of our paper.

Regarding Figure 1, we will simplify the figure to clarify the model structure. We refer to our response to Weakness 2 of reviewer E7pS for a detailed outline of the modification plan.

We sincerely thank the reviewer for reviewing our paper and providing valuable comments. The following parts contain our point-to-point responses to the weaknesses and questions. We hope they can help address the reviewer's concerns.

Response to Weakness 1: We'd like to clarify our novelty and contributions compared to Aligner and other alignment works:

1. MetaAligner is the first work to explore dynamic multi-objective alignment which enables instance-level alteration of the alignment objectives without re-training and unlimited simultaneous alignment objectives. In contrast, Aligner can only perform single-preference alignment and other previous multi-objective alignment works can only perform alignment on fixed objectives. To achieve this capability, MetaAligner is trained to approach chosen responses from rejected responses, considering dynamic target objectives. This training paradigm is not explored by Aligner or any other alignment methods.
2. MetaAligner is the first work that can perform generalizable multi-objective alignment, which can align unseen objectives in a zero-shot manner, while Aligner or any other previous multi-objective alignment works can only perform alignment on objectives that they were trained on. To achieve this capability, we innovatively leverage the in-context learning ability of MetaAligner to understand the unseen objectives and plan for alignment, allowing the users to flexibly describe their alignment objectives in a natural language format, another key difference from the methodology of Aligner.
3. Different from Aligner, we conduct experiments on datasets in both general (HH-RLHF, UltraFeddback) and mental health (IMHI) domains, comprehensively showing the effectiveness of MetaAligner. We also conduct experiments to prove MetaAligner’s generalizable alignment ability to unseen objectives and evaluate its accuracy in objective-wise alignment, which is not covered by Aligner. We have provided related codes, models and data in the submitted paper, and they will be released for public usage.

Response to Weakness 2: While we did not explicitly use the term “preference descriptions” in our paper, we understand that the reviewer may refer to the dynamic combinations of multi-objective descriptions for each chosen-rejected pair. For each chosen-rejected pair, we first identify the target dynamic objectives via their preference annotations in the original dataset. Then the “preference descriptions” are obtained via a random shuffle of these target objectives, followed by a concatenation of their textual descriptions. More details are illustrated in Sec. 3.1 and Algorithm 1. We also provided examples of the building process in Appendix C.3 to further clarify the process.

Response to Weakness 3: We appreciate the opportunity to clarify our choice of datasets:

1. Tasks of MATH and HumanEval are not well-suited for evaluating multi-objective aspects, such as "Harmlessness," "Humor," and "Fairness," because they do not engage with the human-centered elements that require subjective judgment and contextual interpretation. Because of these factors, objective datasets are rarely used in multi-objective preference alignment research [1][2][3][4][5];
2. While MT-Bench and AlpacaEval are popular for single-preference alignment, they lack support for multi-objective alignment, which is central to our work. The absence of multi-objective statistics on their public leaderboards limits their applicability to our research. Additionally, these benchmarks offer a limited number of testing queries, with only 80 for MT-Bench and approximately 800 for AlpacaEval 2.0;
3. We utilized the testing splits from HH-RLHF and UltraFeedback, both of which are widely recognized in multi-objective preference alignment research ([1][5]). These datasets provide extensive testing queries (15,000 each), ensuring a comprehensive evaluation. We also included the IMHI benchmark to assess performance in the mental health analysis domain, with 2,400 testing queries available. All our testing data is publicly accessible, and our methods are reproducible, ensuring the reliability of our results.

Response to Weakness 4: While SPO is indeed relevant, it was first submitted on May 21, 2024. Given that the NeurIPS 2024 main conference paper deadline was on May 22, 2024, it was not feasible to incorporate SPO into our study within such a limited timeframe. Our paper includes comparisons with representative multi-objective alignment methods such as MORLHF, CDPO, and RiC, which effectively demonstrate the strengths of our approach. We acknowledge the value of including SPO and plan to add it as a baseline in future work. However, given the timing constraints, we believe the absence of SPO should not be considered a weakness at this stage.

In our work, we selected MORLHF, MODPO, and SFT-based methods as baseline works, which are widely used baseline methods in previous works[1][3][4][5]. We also do not observe a significant decrease in the number of our compared baseline methods compared to these works.

Response to Question 1: We guess the reviewer refers to how the objective combinations are formed, please see our response to Weakness 2, where we detail the process of creating dynamic multi-objective combinations.

References:

[1] Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment. arXiv preprint arXiv:2402.10207.

[2] Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment. arXiv preprint arXiv:2402.19085.

[3] Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization. arXiv:2310.03708

[4] Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards. arXiv preprint arXiv:2402.18571.

[5] SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling. arXiv: 2405.12739

Dear Reviewer XHXG,

This is a gentle reminder that we have provided a detailed response to your concerns. Please note that the discussion period is ending soon. We'd appreciate it if you could find time to check the response and further consider your evaluations of our paper. We look forward to our further discussions based on these new clarifications.

Best wishes,

On behalf of the authors of Paper 5185

The authors sincerely thank the reviewer for considering our response. Here we provide brief responses to the reviewer's new comments. We hope the reviewer can further consider these points:

Response to Comment 1: As discussed in Sec. 3.3, the authors mentioned "This simple pattern can theoretically lead to unlimited simultaneous alignment objectives". It is worth noting that according to our detailed experimental results in Sec. 4.4, training on 4 objectives already effectively expands alignment performance on up to 10 objectives, which significantly supports the above statement. We believe they are strong evidence of the generalizability of MetaAligner.

Response to Comment 2: Please note that though we incorporated the training method of Aligner, all modules are centered on achieving the two capabilities.

1. instance-level alternation of the alignment objectives without re-training;
2. generalizable multi-objective alignment to align unseen objectives in a zero-shot manner.

We designed dynamic objectives reformulation (Sec. 3.1) to build training data with various combinations of target objectives which are used to train MetaAligner in a conditional weak-to-strong correction manner (Sec. 3.2). With the dynamic multi-objective capability triggered by the above modules, We further extend it to generalizable inference (Sec. 3.3) with the in-context learning ability of MetaAligner.

Response to Comment 3: The authors would like to clarify that by mentioning our experiments on the mental health analysis domain, we didn't mean to list it as a novelty, but as an empirical contribution that Aligner or other alignment methods didn't consider.

Response to Comment 4: The authors would like to clarify that testing on UltraFeedback and HH-RLHF also thoroughly evaluates the basic LLM capabilities on alignment.

1. The testing data covers most categories reflected in benchmarks such as MT-Bench, including Writing, Reasoning, QA, Extraction, etc.
2. Our testing data has much more samples than the mentioned benchmarks, which allows a thorough evaluation of the capabilities of the LLMs.