PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model

Thanks for your thoughtful review and valuable feedback. We address your concerns as follows.

[Experimental Designs Or Analyses]. GenARM is the only baseline in evaluation ... include comparision with policy-guided approaches [2,3,4,5].
[5] Rewarded soups (NeurIPS 2023)
[Weaknesses 2]. Comparison with policy-guided approaches.
[Other Comments Or Suggestions]. difference between personalization and managing trade-offs.
[Relation To Broader Scientific Literature]. (2) guided-generation process and weak-to-strong guidance are already explored in [1,2,3].

The goal of our PARM is fundamentally different from [2,3,4,5]. Here are the key distinctions (summarized in Table R1 at https://anonymous.4open.science/r/4525-TCL8/rebuttal_to_TCL8.pdf):

• PAD [3] only aligns personalized preferences but fails to handle trade-offs among multiple dimensions. Specifically, PAD can handle discrete preference combinations ("helpful", ..., "helpful and harmless", ...) but fails to handle continuous preference combinations (like "60% helpful and 40% harmless"). This limitation is also acknowledged in PAD's rebuttal (see their reply to Weaknesses 1 raised by Reviewer NqpU, https://openreview.net/forum?id=e7AUJpP8bV).
• MOD [2], CLP [4], and RS [5] require training $k$ policy LLMs ($k$ is the number of preference dimensions). This is computationally infeasible, especially when the policy LLM is large (e.g., 65B). Additionally, MOD [2] directly combines the logits from multiple trained policy LLMs at inference, causing significant computational overhead, and it does not contain a guided-generation process and weak-to-strong guidance. In contrast, we keep the policy LLM frozen and use a smaller reward model to guide the larger frozen policy LLM (e.g., 7B guides 65B in our experiment). This avoids training multiple large 65B models as in [2,4,5] and only requires training a 7B reward model, making our method more efficient and promising for users with limited computational resources.

As suggested, we conduct additional experiments to compare PARM with MOD [2] and RS [5]. The experimental setup is the same as in Section 5.1. Results in Figure R1 and Table R2 (in the anonymous link above) show that PARM achieves better Pareto front than RS and MOD, and significantly outperforms them in terms of HV and MIP, demonstrating its effectiveness and high alignment quality.

[Relation To Broader Scientific Literature]. (1) ... preference-aware adapter ... would be of great value if the empirical advantages can be well supported.
[Weaknesses 1]. the necessity of preference-aware adapter design is not well demonstrated. ... SVD-LoRA is comparable with PBLoRA.

We appreciate your recognition of the originality of our proposed preference-aware adapter (PBLoRA), but we argue that we've provided sufficient evidence for its design necessity in our paper. We clarify the evidence for the necessity of designing PBLoRA as follows.

• PBLoRA outperforms SVD-LoRA significantly: (1) The Pareto front of PBLoRA entirely covers that of SVD-LoRA (yellow vs. purple curves in Figure 3). (2) PBLoRA achieves 11.4% and 59.9% improvements in HV and MIP compared to SVD-LoRA (Table 3).
• General framework: As detailed in Lines 220-226 of our paper, SVD-LoRA is a special case of PBLoRA, which means PBLoRA has a greater exploration space and can achieve better results.
• Ablation evidence: In Section 5.3, we systematically analyze different configurations of PBLoRA, showing the effectiveness of each component.
• PARM with a single PBLoRA addresses the inefficiency and misalignment issues in GenARM using $k$ LoRAs. Please refer to our reply to Weaknesses 1 raised by Reviewer 3zGf for details.

[Questions 1]. In Figure 2(b), why would PARM be much better than GenARM in solely optimizing humor, while only comparable in optimizing helpfulness?

Humor is a narrower objective than helpfulness. PARM learns from multiple preference dimensions jointly, benefiting from shared knowledge, while GenARM trains separate reward models for each preference, preventing it from leveraging others data to enhance humor. This difference leads to PARM’s superior performance on the Humor dimension. Conversely, helpfulness, being a broader dimension, can be effectively learned even with GenARM’s isolated approach, resulting in comparable performance on this dimension for both models.

[Questions 2]. Why does PARM show better numerical performance than GenARM in Table 1,4, while only slightly better than GenARM in Figure 1,4?

HV measures the area under the Pareto front. In the Figures, it is clear that PARM has a larger area than GenARM, leading to a higher HV.

MIP measures the uniformity of the solutions on the Pareto front. Obviously, the distribution (the markers in the figures) of PARM is more uniform, so its MIP is higher.

Thanks for your thoughtful review and valuable feedback. We address your concerns as follows.

[Essential References Not Discussed]. Discussions on controllable text generation are missing.

Controllable Text Generation (CTG) generates text from LLMs with specific attributes or constraints. Our PARM is a specific CTG implementation for multi-objective test-time alignment. Compared to the mentioned method PPLM (ICLR 2020), which uses multiple attribute models and requires forward and backward passes during generation, PARM employs a single reward model to dynamically adjust text during inference, achieving lower computational costs.

We will expand this discussion and include the CTG references in the revision.

[Weaknesses 1]. The significance of the research problem is unclear, as human preference alignment in LLM generation can often be achieved through prompt engineering without additional models.
[Claims And Evidence]. prompt engineering is not considered a baseline.

Prompt engineering alone is insufficient for aligning LLMs with complex human preferences. Most successful alignment methods (like RLHF) require post-training, and our method can be viewed as a test-time alternative to post-training to reduce training computations.

As suggested, we conducted an additional experiment to compare our method with a prompt-based baseline whose instruction is "Please ensure your response is X" (adapted from Personalized Soups (NeurIPS 2024 workshop) and PAD (ICLR 2025)), where "X" is "helpful", "harmless", or "a% helpful and b% harmless". The experimental setup is the same as in Section 5.1.

As shown in https://anonymous.4open.science/r/4525-isvK/rebuttal_to_isvK.pdf, prompting has little effect on both two preferences. Moreover, prompting cannot achieve precise control over preferences trade-offs and its results fail to form a Pareto front, demonstrating that prompting is not a good choice for aligning LLMs with complex human preferences. In contrast, our method has a much better Pareto front, enabling precise control over multiple competing preferences simultaneously.

[Weaknesses 2]. introduces additional inference costs and requires extra training data ... impractical compared to simpler alternatives.
[Claims And Evidence]. relying on an additional 7B or smaller LLMs for preference guidance ... computational overhead.

As discussed in our reply to Weaknesses 1, the prompt-based method, a simpler alternative, is not effective enough to align LLMs with complex human preferences. Recently, many advanced alignment methods (e.g., RLHF, GenARM) have emerged, and our work extends them to multi-objective test-time alignment.

Training data problem: we would like to clarify that obtaining a multi-objective preference dataset is not very difficult and can be derived from a traditional single-objective dataset (see our reply to Weaknesses 2 raised by Reviewer 3zGf).

Inference cost problem: we would like to clarify that the increase in inference cost is small and can be effectively mitigated through distributed deployment as follows:

• Addressed by distributed deployment. Our method uses a single reward model to guide the frozen policy LLM. Since both models can generate the next token in parallel, the inference time remains the same as using the frozen LLM directly.
• Without distributed deployment, our method is still practical for two reasons: (i) our method enables weak-to-strong guidance, such as using a 7B reward model to guide a frozen 65B LLM in our experiments. It increases inference time by about 30% compared to directly using the 65B LLM. We believe that the increase in inference time is worthwhile since our method significantly improves preference alignment without extensive training of the large policy LLM (Figure 4). (ii) Compared with GenARM, our method significantly reduces inference costs. Please refer to our reply to Weaknesses 1 raised by Reviewer 3zGf for details.

Hence, by leveraging a smaller reward model to guide a larger LLM, PARM offers a practical and efficient method for achieving multi-objective alignment without requiring extensive training. This is especially beneficial for users lacking resources to fine-tune the policy LLM.

[Questions]. Can a much smaller language model achieve comparable performance while significantly reducing inference latency?

There is a trade-off between the capacity of the reward model and its guiding effect. An extremely smaller reward model can reduce inference costs but may also compromise the guiding effect. Note that in our experiments, the reward models are already very small compared to the frozen LLMs (e.g., 7B vs. 65B and 1.1B vs. 7B). Moreover, as discussed in our reply to Weaknesses 2, the inference cost problem can be effectively mitigated through distributed deployment. Thus, we leave the exploration of much smaller reward models for future work.

Thanks for your thoughtful review and valuable feedback. We address your concerns as follows.

[Weaknesses 1]. The proposed model is the extension of the existing model ARM and in particular the GenARM. The main difference is that the proposed model is to condition the massive of model parameters on the k-dimensional preference vector alpha. Given this, the contribution of the paper sounds limited.

While PARM builds upon the foundation of ARM and GenARM, it introduces significant innovations and improvements that address key limitations of GenARM. We clarify it in detail as follows.

Unlike GenARM, which requires multiple ARMs for different preference dimensions, PARM uses a single ARM conditioned on the preference vector, resulting in several advantages:

• More Parameter-efficiency: GenARM requires storing $k$ separate ARMs, while PARM needs only a single one, making it approximately $k\times$ more parameter-efficient (Table 2 in our paper).
• Faster Inference: PARM is significantly faster during inference (Table 2), as it computes rewards from a single ARM rather than $k$ separate ARMs in GenARM.
• Better control of trade-offs with preference vector: In GenARM, the $k$ ARMs are trained independently on different preference dimensions without awareness of each other, leading to potential conflicts when their rewards are combined during inference. PARM, on the other hand, is explicitly trained to manage trade-offs between different preferences (as detailed in Section 4.3). As a result, our model's behavior aligns more directly with the specified preference vector, making control more intuitive and predictable. This leads to better alignment with user-specified preferences and simplifies usage, as demonstrated by the significantly higher HV and MIP scores in Tables 1 and 2. Examples 1 and 2 in our paper further demonstrate how PARM effectively balances competing objectives like helpfulness and harmlessness according to the specified preference weights.

Therefore, our PARM provides a more efficient, scalable, and controllable method for multi-objective test-time alignment of LLMs. We believe these contributions are substantial and address important challenges in the field of preference-aligned language models.

[Weaknesses 2]. As the proposed model works for multi-objectives, training material needs to be sufficient for training the model, which may be bottleneck in some setting where training data are difficult to obtain.

We appreciate your concerns regarding the potential bottleneck of training data for multi-objective models. We want to point out that multi-objective dataset is not very difficult to obtain, as it could be derived from a traditional single-objective dataset as follows.

Specifically, we can take standard preference datasets $\\{x, y^1, y^2, z\\}$ (which are widely available, $y^1$ and $y^2$ are two different responses to the prompt $x$, and $z=1$ if $y^1$ is better than $y^2$, otherwise $0$) and extend them to multi-objective datasets $\\{x, y^1, y^2, z_1, \dots, z_k\\}$ by adding preference labels $z_i$ for different dimensions. These additional labels can be obtained using GPT judges or publicly available reward models/classifiers. For example, in our Helpful Assistant experiment (Section 5.2), the humor preference labels were obtained using a public classifier (as detailed in footnote 10), following the approach of previous works [1,2].

[1] MetaAligner: Towards generalizable multiobjective alignment of language models. NeurIPS 2024.

[2] Rewards-in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. ICML 2024.

[Weaknesses 3]. To achieve the goals of multi-objectives learning, there are a number of strategies. And one of them is the studied strategy in this paper, i.e., introducing an autoregressive reward model. Why don’t the authors consider fine-tune the original model?

While fine-tuning the original model is a possible approach, this method requires enormous computational resources that many researchers and practitioners cannot access when the model is large (such as Alpaca-65B).

Our PARM is a test-time alignment approach that addresses this limitation by training a small reward model rather than the original LLM, reducing computation cost largely and making multi-objective alignment accessible with limited computing resources.

Our method enables weak-to-strong guidance, allowing a smaller reward model to guide a larger frozen LLM without expensive training. For example, in our experiments, a 1.1B reward model guides a 7B LLM, and a 7B reward model guides a 65B LLM. This capability is particularly valuable for users who cannot afford to train large LLMs but still need to leverage their capabilities.

Therefore, based on test-time alignment, our PARM provides a practical solution that makes multi-objective alignment more accessible to the broader research community.