PAD: Personalized Alignment of LLMs at Decoding-time

Case 1

• System: Your task is to generate a response by considering the following preference.

• User: Personalized Preference

  None

  User Instruction

  Create a message for an offer letter for an internship at Apple as an AI research intern.

• PAD:

  Subject: Offer of AI Research Intern Position at Apple

  Dear [Recipient's Name],

  We are thrilled to offer you the AI Research Intern position at Apple, as selected through our competitive application and interview process. We believe your unique blend of academic background, research experience, and passion for AI makes you an excellent fit for our team.

  The AI Research Intern program at Apple provides a unique opportunity to work alongside our world-class researchers and engineers to develop innovative AI solutions that transform the way people interact with technology. During your 12-week internship (start date: [start date], end date: [end date]), you will have the chance to:

  • Conduct original AI research under the guidance of our experienced researchers, focusing on [specific area of AI, e.g., computer vision, natural language processing, or machine learning].
  • Collaborate with a talented team to design, develop, and test AI-powered products and features.
  • Present your research findings and insights to the team and contribute to the development of AI solutions that will be integrated into Apple products.
  • Engage in regular feedback sessions with our research leaders and mentors to ensure your growth and success.

  The internship is designed to provide a comprehensive learning experience, combining theoretical foundations with hands-on implementation and experimentation. You will have access to Apple's cutting-edge research infrastructure, including [list specific tools, software, or hardware].

  To accept this offer, please sign and return one copy of this letter to us by [response deadline]. Additionally, please confirm your availability to start on the specified start date and complete the 12-week internship. Please note the following details:

  • Internship duration: 12 weeks (start date: [start date], end date: [end date]).
  • Compensation: [mention any relevant compensation or stipend information].
  • Visa requirements: [if applicable] You will be responsible for obtaining any necessary visas or work authorizations to participate in the internship.

  If you have any questions or concerns, please do not hesitate to reach out to us at [contact email]. We look forward to welcoming you to the Apple AI Research team and working together to shape the future of AI research! Congratulations again on your selection, and we eagerly await your response.

  Best regards,

  [Your Name]
  AI Research Intern Program Manager
  Apple Inc.

Question 1: Why Best-of-N decoding is worse than other decoding strategies in helpful, humor, and diversity? Because BoN is always thought to be an technique that enhances the model performance.

Your comment is correct that Best-of-N (BoN) decoding can enhance model performance, as demonstrated in Figure 4 where BoN consistently outperforms Base using conventional greedy decoding strategies. However, the discussion here pertains to the performance of PAD when integrated with different strategies, which differs from traditional experimental setups. For Greedy and Stochastic strategies, PAD calculates the reward from the personalized reward model at each token, and then uses weighted scores to select the next token under the two strategies. In contrast, for BoN, the model first generates N responses, from which the best one is chosen based on scores from the personalized reward model. Intuitively, for Greedy and Stochastic strategies, PAD can continually adjust the output of the base model during generation, providing more external information and thereby achieving better performance. In contrast, BoN is still limited to expressing the model’s internal knowledge. It is noteworthy that PAD's superior performance over BoN also confirms the superiority of our approach.

Reference

[1] Args: Alignment as Reward-Guided Search

Thank you for your constructive comments. We have carefully considered your suggestions and would like to provide the following clarifications. We have also updated the manuscript, with the modified contents highlighted in orange for your review.

Weakness 1: According to Table 2, Preference Prompting that simpliy add preference information to the prompt without training the model can achieve good performance (GPT4 winrate) comparing to other methods that requires training.

Thank you for highlighting this aspect. In our experiments, we also found that Preference Prompting serves as a strong baseline. This may be attributed to the fact that models through Supervised Fine-Tuning (SFT) are well-aligned on these personalized prompts, thus achieving more effective results than training-based methods. However, compared to Preference Prompting, PAD demonstrates superior alignment performance on both pre-defined and customized preferences, confirming its overall superiority.

Weakness 2: I think a case study can strengthen this paper. For example, showing how apply different personalized configurations to the same instruction steers the genenrated responses to varied directions.

Thank you for your insightful suggestion. We agree that incorporating a case study would substantially enhance the paper by illustrating how our method can guide the generation of responses in various directions based on different preferences. In response to your suggestion, we conducted a case study on a single instruction while employing four distinct personalized preferences using the PAD method. Due to space limitations, we put the cases in the last four comments. You can also refer to Figure F.4 (Pages 25-28) in the revised manuscript for content of the case study. For all scenarios, the base instruction was: "Create a message for an offer letter for an internship at Apple as an AI research intern." The personalized preferences applied were: "none" (vanilla generation), "expert", "comprehensive", and "humorous". The analysis revealed noticeable variations in the model's outputs according to the different personalized configurations. Compared to the vanilla generation, the response under the "expert" configuration adhered more strictly to formal offer letter conventions and language. The "comprehensive" configuration produced a response that included a greater level of detail, such as codes of conduct, and even exceeded length constraints. Conversely, the response under the "humorous" configuration was notably more casual and engaging.

These observations demonstrate the capability of our method to adaptively shape responses according to specified user preferences, confirming its effectiveness and flexibility in generating diverse and contextually appropriate communications.

According to your suggestion, we have included a new section Appendix F.4 (Lines 1323-1466 in Pages 25-28) in the revised manuscript that presents the case study. We appreciate your feedback, which has undoubtedly enhanced both the quality and clarity of our paper.

Weakness 3: What is the benefit of decoding-time alignment? It seems to me that PAD still need to train the backbone model, which seems to violate the motivation of decoding-time algorithm.

Decoding-time alignment integrates alignment directly into the decoding process, thereby eliminating the need for training the base model [1].

We would like to clarify that PAD is a strategy for alignment during the decoding phase and does not require training the policy model. PAD requires training only a personalized reward model. The optimized personalized reward model guides the decoding process to achieve alignment. For further clarification, we have added two algorithm boxes in Appendix A.1 (Line 864-890) that includes detailed pseudocode for the practical implementations of PAD in the revised manuscript. Furthermore, in Section 4.4, we explore the capability of PAD to be applied across different base models (model-agnostic). In these experiments, PAD utilizes the same trained personalized reward model without requiring retraining for different base models. Empirically, we demonstrate that, in contrast to other methods that necessitate retraining for each base model, PAD requires only the training of one reward model to align different base models.

The benefits of PAD are summarized as follows: (1) It does not require training policy models, making it training-free. (2) It utilizes only a single reward model, emphasizing its efficiency. (3) It generalizes to unseen preferences, highlighting its generalizability. A detailed comparison with existing methods is provided in the checklist in Table 1 in our manuscript.

Weakness 3: During inference, the need to load and run an additional LLM for the personalized reward model introduces computational overhead.

Thank you for your insightful comment. Latency and computational overhead are currently focal points in decoding-time alignment research.

Empirically, we explore the effects of different decoding strategies on latency during the decoding phase in Section 4.3, "Effect of Decoding Strategies," where the time costs are reported in Table B3 on page 19. The results are also provided in the following table. 'Base' represents conventional greedy decoding of vanilla base models. Time costs of training-based models are very close to 'Base'. 'Greedy' and 'stochastic' represent guided decoding by PAD algorithm. Our results reveal that generation time of PAD increases by 2-3 times compared with conventional greedy decoding. Additionally, we have quantified the memory overhead of PAD. Decoding with PAD (trained from Llama-3-8B) requires an additional 17,095 MB, which is considerably modest.

Despite this increase in processing time and memory, there is a notable enhancement in performance across all dimensions (e.g., 26% increase as for helpful and 24% increase as for harmless), indicating a reasonable tradeoff between reward optimization and inference speed. Moreover, PAD's advantage lies in its ability to align LLM outputs with diverse personalized preferences during the inference phase. This approach eliminates the need for the exhaustive process of retraining the RL model and allows for rapid adaptation in response to changes in personalized preferences. In addition, the efficiency gap can be further narrowed by employing a smaller reward model or parallelizing the reward computation across multiple candidates.

In summary, while we acknowledge and discuss the computational overhead introduced by PAD during the inference phase, the reduced training costs and superior performance make this a worthwhile trade-off. Additionally, we suggest that more efficient decoding-time alignment could be achieved using existing strategies.

Table. Comparison of the performance and computational cost of various decoding strategies. Base: conventional greedy decoding with only the base model. Greedy: select the token with maximum score with PAD. Stochastic: sample a token from the probability distribution of the top-k candidate with PAD.

Based on your suggestions, we have added a new paragraph in Section 4.3 (Line 512-519) and results in Table C3 (Line 1053) that specifically discusses the Computational Cost for PAD. We appreciate your feedback, which has undoubtedly enhanced both the quality and comprehensiveness of our paper.

Weakness 4: Some minor typos and errors that did not influence my rating: 1) In Table 2, under the "Helpful" RM column for MetaAligner on the HelpSteer dataset, the highest value isn't highlighted. 2) Line 322: "LLama" should be "Llama.

Thank you for identifying these typos. We will correct them in the revised version of the manuscript (Lines 403 and 322).

References

[1] Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment

[2] Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards

[3] Decoding-Time Language Model Alignment with Multiple Objectives

Table. Extended comparisons of PAD with baselines across various base models. The best performance is denoted in bold.

Thank you for your constructive comments. We have carefully considered your suggestions and would like to provide the following clarifications. We have also updated the manuscript, with the modified contents highlighted in orange for your review.

Weakness 1: The paper lacks a Pareto front analysis for scenarios involving multiple rewards. For experiments that combine multiple objectives (like "Harmless and Humor" or "Expert, Informative, and Creative" in Figure 3), including such an analysis would help illustrate the trade-offs between different preferences.

Thank you for your insightful comment. We agree that incorporating a Pareto front analysis is crucial for demonstrating the trade-offs between competing objectives. In response, we have now included a Pareto front analysis in our study. Following the approach outlined in [1], we analyzed the empirical Pareto front in a three-dimensional space. Due to format limitations, please refer to Figure C1 on page 19 in the revised manuscript for the comparisons of the PAD with the baselines. Unlike prior multi-objective alignment works [1, 2, 3], the goal of PAD is to align different preferences rather than managing trade-offs among multiple dimensions. As such, PAD does not accommodate the interpolation of preference weightings. Consequently, our analysis is confined to seven distinct preference combinations: {'Harmless', 'Helpful', 'Humor', 'Harmless and Helpful', 'Harmless and Humor', 'Helpful and Humor', 'Helpful, Harmless, and Humor'}. The results demonstrate that the performance frontier of PAD approaches Pareto efficiency more closely than that of baselines. This finding underscores the effectiveness of PAD in managing scenarios with multiple objectives.

Based on your suggestions, we have added a new Section C.1 (Line 965-995) that discusses the Pareto front analysis for PAD, and the additional experimental results are reported in Figure C1. We appreciate your feedback, which has undoubtedly enhanced both the quality and comprehensiveness of our paper.

Weakness 2: Table 3 only displays PAD's performance. Including results from other baseline methods across different models would be nice.

The primary objective of Table 3 is to explore the capabilities of PAD when directly applied to different base models in a model-agnostic manner. In this setup, we employ a single trained personalized reward model without additional retraining for varying base models. Consequently, most baseline methods that require retraining the base model are inapplicable.

Based on your suggestions, we have further compared two baselines that are also not specific to any base model (Preference Prompting and MetaAligner). The results are provided in the next comment. For each model, the best performance for each metric is highlighted in bold. Notably, MetaAligner achieves significant improvements across multiple dimensions for various models, especially excelling in the 'helpful' dimension for Gemma, which aligns with its model-agnostic originality. In contrast, Preference Prompting offers a more balanced yet modest improvement. However, PAD outperforms the baselines in most scenarios and consistently achieves the optimal overall performance, demonstrating its model-agnostic superiority. Additionally, we acknowledge your concerns regarding validating PAD's superiority across multiple models. Therefore, we plan to further supplement our experiments by varying the base model, comparing PAD with training-based baselines; these experiments are ongoing.

Following your advice, we have added a new Section C.4 (Lines 1063-1070) in the revised manuscript to discuss the comparisons of PAD across various base models, and the additional experimental results are reported in Table C4.

Question 1: Could you include pseudocode or an algorithm box for PAD? This would be helpful for the audience to understand the training and inference details.

Thank you for your insightful suggestion. We agree that providing these details in a structured format can significantly enhance the comprehensibility of the training and inference processes involved in PAD. In response to your suggestion, we have added two algorithm boxes in Appendix A.1 (Line 864-890) that includes detailed pseudocode for the practical implementations of PAD in the revised manuscript.

We appreciate your feedback, which has undoubtedly improved the quality and readability of our paper.

Question 2: How are the prompts for personalized preferences constructed in your experiments? Have you tested the robustness of PAD by using paraphrased versions of the prompts used during training to see if it maintains performance with variations in preference expressions?

We construct it following the method described in [1] by prepending synthetic user prompts to instructions. For example, consider the original instruction x "What is needed for self-sufficient living spaces?", the personalized preference p is the synthetic user prompt "Generate a response that is harmless, helpful and humorous." A Prompt Template is provided in Section A.2 (Lines 851-858) for better illustration.

Following the reviewer's suggestion, we have conducted an exploration of the robustness of PAD towards variations in preference expressions. We created four paraphrased versions of the original user prompt "Generate a response that is harmless, helpful, and humorous" by substituting synonyms for the key words representing preferences:

• Prompt 1: Create a reply that is safe, helpful, and humorous.
• Prompt 2: Generate a reply that is harmless, useful, and humorous.
• Prompt 3: Produce a reply that is harmless, helpful, and witty.
• Prompt 4: Compose a response that is safe, useful, and witty.

Subsequently, we replaced the prompt in PAD framework and conducted tests, with results displayed in the following table. It is evident that Prompts 1 and 2 had virtually no impact on PAD's performance. In contrast, Prompts 3 and 4 resulted in a decrease in the Humor rating, which we hypothesize is due to the term "witty" not closely aligning with the reward model's "humor" objective. Overall, when we paraphrase prompts using semantically consistent objectives like "safe" or "useful", there is no change in performance, confirming the robustness of PAD. Furthermore, considering the consistent improvement over the base model under different prompts, we show PAD demonstrates robustness to variations in preference expressions. Thank you once again for your interesting idea. We have incorporated this analysis into Section C.5 (Lines 1073-1122) of the revised manuscript.

Table. Analysis of the robustness of PAD with paraphrase prompts.

In addition, in Section 4.2, Line 416, we conducted experiments on alignment to customized preferences, which were not seen during the training phase. The results demonstrate that PAD consistently enhances alignment performance across varying preference expressions, confirming its robustness. Furthermore, we provide a theoretical analysis in Section 3.3 on PAD's ability to generalize to a broader range of personalized preferences. The main intuition is that if PAD has previously aligned to a similar personalized preference, it should effectively generalize to new preferences, which further explains the robustness of PAD.

Reference

[1] Personalized soups: Personalized large language model alignment via post-hoc parameter merging.

Thank you very much for addressing my concerns and suggestions. I appreciate the authors' effort. I will increase my rating.

Question 1: During the first training phase, is personalized data p included? If so, in what form is it integrated? Could you provide an example?

Yes, our training data includes personalized preferences. We construct it following the method described in [6] by prepending synthetic user prompts to instructions. For example, consider the original instruction x "What is needed for self-sufficient living spaces?", the personalized preference p is the synthetic user prompt "Generate a response that is helpful."

A Prompt Template is provided in Section A.2 (Line 851-858) for better illustration. During both training and testing, x and p serve jointly as inputs to the model, as also depicted in Figure 1 in our manuscript.

Question 2: Is there a distinction between the training data used in the two stages?

The training data remains the same across both stages, but the focus of the training shifts. In the first stage, we aim to optimize the backbone to learn general features. In the second stage, we focus on optimizing the value head to learn different user preferences. For further clarification, we have included an algorithm box in Appendix A.1 (Line 864-890) of the revised manuscript, featuring detailed pseudocode for the practical training process of the personalized reward model.

Question 3: Theoretically, PAD appears highly similar to DPO, differing only by the additional hyperparameter β and the personalized parameter Wp. Can the first training stage be understood as DPO, with the second training stage merely training a value head?

Thank you for your insightful comment. We wish to further clarify the connections and distinctions between our work and DPO.

Broadly speaking, the primary distinction between PAD and DPO is that DPO optimizes a policy model, whereas PAD trains a personalized reward model. PAD is a strategy for alignment during the decoding phase and does not require training a policy model. It guides the decoding process to achieve alignment through the optimized personalized reward model.

Specifically, the loss function of our personalized reward model (Eq. 9) bears a formal resemblance to the objective function of DPO. Inspired by the works on DPO [7,8], we have extended existing research in two significant ways, which constitute the technical novelty of this paper. First, we formally demonstrate that by rewriting the reward function in the manner of DPO (Eq. 8), we can derive the implicit Q function with the optimized personalized reward model (Eq. 10), making it suitable for guidance during the decoding phase. Secondly, by decoupling personalized preferences from the state in reward modeling, we demonstrate that personalized preferences can be separated from the MDP dynamics (Eq. 4 to Eq. 7), thereby facilitating easier transfer to different user preferences.

In summary, PAD is a decoding-time alignment algorithm that performs decoding under the guidance of a personalized reward model. Within this framework, we incorporate the characteristics of DPO to train the personalized reward model, enabling it to provide token-level guidance. Furthermore, through an innovative personalized reward modeling strategy, we ensure that the personalized rewards are generalizable across different preferences.

We hope this response satisfactorily clarifies the distinctions and connections between PAD and DPO. We appreciate your keen observations and are eager to discuss any further concerns you might have.

Question 4: What does π^ref represent in Equation (13)?

Thank you for pointing out this issue. We recognize that we did not clarify the meaning of $\pi_{ref}$ in this section. In fact, $\pi_{ref}$ here is consistent with earlier mentions (Eq. 3) and represents the reference policy, which is typically used as the initial policy $\pi_{\theta}$ in frameworks such as RLHF or DPO. To be more specific, in this paper, we use Llama-3-8B as the backbone of our personalized reward model. Thus, $\pi_{ref}$ is Llama-3-8B, and $\pi_{\theta}^*$ is the trained personalized reward model. Based on your suggestions, we have provided further clarification on this point in the revised manuscript (Line 280-281).

References

[6] Personalized soups: Personalized large language model alignment via post-hoc parameter merging

[7] From r to Q∗: Your Language Model is Secretly a Q-Function

[8] Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Weakness 4: Lack of comparisons

Thank you for providing additional baselines. Both [1] and [2] are outstanding works that are highly relevant to our study. Reference [1] focuses on steering the output of large language models (LLMs) towards specific styles by adding style vectors to the activations of hidden layers during text generation. This approach is also suitable for steering outputs towards various personalized preferences. Reference [2] can be directly applied to base models to adjust their outputs to align with user preferences.

Based on your suggestions, we have included comparative experiments with Aligner [1] and Steering [2]. For Aligner, we replicated the method using its official repository [3]. To align with preferences for 'Helpful', 'Harmless', and 'Humor', we conducted "training_based" steering using the dataset in RiC [4], while maintaining other steps and experimental settings unchanged. For [2], we utilized the open-source model and the usage code provided in [5]. We modified the prompt to "Edit the following Question-Answer pair to make it more helpful, harmless, and humorous" to standardize the alignment targets. The performance of additional baselines is presented in the following table. It is evident that Aligner excels in the dimensions of 'Helpful' and 'Harmless', likely due to its design specifically targeting these dimensions. However, Aligner shows no improvement in the dimension of 'Humor', showcasing its limited scalability. In contrast, [2] provides a more balanced yet modest improvement across all three dimensions. We have incorporated the results of these additional baselines into Table 2 in the revised manuscript. Overall, the results still demonstrate the superiority of PAD in personalized alignment. Based on your suggestions, we have included the two additional baselines in the revised manuscript (Line 365) and in Table 2 (Line 392-393, 404-405).

Table. Comparison of baseline methods and PAD with additional baselines. The best performance is denoted in bold, while the second-best performance is indicated in italics.

References

[1] Konen K, Jentzsch S, Diallo D, et al. Style Vectors for Steering Generative Large Language Model[J]. arXiv preprint arXiv:2402.01618, 2024.

[2] Ji J, Chen B, Lou H, et al. Aligner: Achieving efficient alignment through weak-to-strong correction[J]. arXiv preprint arXiv:2402.02416, 2024.

[3] https://github.com/DLR-SC/style-vectors-for-steering-llms

[4] Rewards in-context: Multi-objective alignment of foundation models with dynamic preference adjustment.

[5] https://huggingface.co/aligner/aligner-7b-v1.0

Thank you for your constructive comments. We have carefully considered your suggestions and would like to provide the following clarifications. We have also updated the manuscript, with the modified contents highlighted in orange for your review.

Weakness 1: Unclear Description of Training Process

Thank you for your insightful feedback. We recognize the importance of a clear and detailed description of the training process and appreciate your suggestion to enhance this aspect of our paper. We would like to clarify the actual training process as follows:

• Dataset Construction: We utilize training data from four datasets: Ultrafeedback, HelpSteer2, Rewards-in Context, and SafeRLHF. These datasets consist of instructions, multiple corresponding responses, and human ratings across various dimensions, as detailed in Table A.2 (Lines 832-858). To represent personalized preferences, we prepend synthetic user prompts to the instructions. More details are provided in Question 1 of the FAQ section.

• Training of Personalized Reward Model: Utilizing the dataset constructed in the previous step, we perform two-stage optimization. Initially, we employ a single backbone to predict embeddings, followed by an additional value head to predict personalized preferences. In the first stage, we fix the personalized preference as a unit vector and optimize the backbone to learn general features. In the subsequent stage, we freeze the backbone and solely optimize the value head to learn diverse user preferences.

The trained personalized reward model is then employed during the inference phase for guided decoding. In response to your suggestion, we have included two algorithm boxes in Appendix A.1 (Lines 864-890) of the revised manuscript, featuring detailed pseudocode for the practical training process of the personalized reward model. Additionally, we have enhanced the description of the data construction process in Appendix A.2 (Lines 832-858). Further implementation details are provided on page 16 of Appendix A.2 (Lines 860-863). We appreciate your feedback, which has undoubtedly improved the quality and readability of our paper.

Weakness 2a: The definition of variable a in line 287 is unclear, and the inconsistent use of subscript t could lead to confusion.

Thank you for pointing out the typo. In line 287 (in the original manuscript), an instance of a was incorrectly written as a_t, and we have corrected it. In this context, a denotes a candidate action, while a_t represents the action selected at time t. Note that the concept of an action is central to the Markov Decision Process (MDP), which can be interpreted as the next token in a Language Model (LM), as explained in lines 120-126. During the Guided Decoding process, the base language model initially generates probabilities for each candidate $a$. Subsequently, these probabilities are reweighted by the reward, and the candidate with the highest weighted score is selected as the next token $a_t$. Based on your comment, in the revised manuscript lines 287-289, we have corrected the typo in the subscript 't' and further clarified the definitions of $a$ and $a_t$.

Weakness 2b: The term PRM is commonly understood to mean "Process Reward Model," which differs from its use in this paper. Clarification on this term would help avoid ambiguity.

Thank you for your suggestion. To prevent confusion due to its similarity with terms in existing literature, we have renamed it to 'PersRM' in the revised manuscript (e.g., Line 288) in accordance with your suggestion, thereby clarifying its distinct identity.

Weakness 3: Inaccuracy in "Model Agnostic" claim.

We apologize for the confusion in Section 4.4 and would like to further clarify this section. In Section 4.4, our goal was to investigate whether PAD could be directly applied to different base models. In this experiment, PAD utilizes a single trained personalized reward model (the same one) and does not require retraining for different base models, thus constituting a model-agnostic analysis. Empirically, we demonstrated that, unlike other methods that necessitate retraining for each base model, PAD requires only the training of one reward model to align different base models. This finding underscores the model-agnostic characteristic of PAD and further highlights its superiority.

We apologize for the lack of clarity in this section. In the revised manuscript, we have provided additional explanations of the motivation and experimental setup (Lines 523-524).

Thank you for your constructive comments. We have carefully considered your suggestions and would like to provide the following clarifications. We have also updated the manuscript, with the modified contents highlighted in orange for your review.

Weakness 1: Not much details are shared on how the model can generalize to unseen preferences not seen during the training. The w_p is dependent on p being specified so new preferences would need some retraining.

Thank you for your insightful comment. "PAD can generalize to unseen preferences" is one of the main contributions of this paper, and we wish to further clarify it.

Firstly, by decoupling personalized preferences from the state in reward modeling, we demonstrate that personalized preferences can be separated from the MDP dynamics (Eq. 4 to Eq. 7), thereby facilitating easier generalization to different user preferences. Then, we provide a theoretical analysis in Section 3.3 on PAD's ability to generalize to unseen preferences. The main intuition is that if PAD has previously aligned to a similar personalized preference, it should generalize well to the new preference.

In addition, we conducted empirical analysis in Section 4.2, Line 416, where we performed experiments on alignment to customized preferences that were not seen during the training phase. The results confirm the superiority of PAD in generalizing to unseen personalized preferences.

In summary, PAD does not require pre-defined personalized preferences during the training phase to generalize to unseen preferences. Should you have any further concerns, please do not hesitate to contact us.

Weakness 2: While the experiment section compares the performance of the model against the baselines, the latency and cost is not compared which will have practical implication in real world use.

Thank you for your insightful comment. Latency and computational overhead are currently focal points in decoding-time alignment research.

Empirically, we have explored the effects of different decoding strategies on latency during the decoding phase in Section 4.3, "Effect of Decoding Strategies" (in the original manuscript). The empirical results are reported in Table C3 (Line 1053 in the revised manuscript). The results are also provided in the following table. 'Base' represents conventional greedy decoding of vanilla base models. The time costs for training-based models closely align with those of the 'Base' configuration. 'Greedy' and 'stochastic' represent decoding-time alignment by PAD algorithm. Our results reveal that generation time of PAD increases by 2-3 times compared with conventional greedy decoding. Additionally, we have quantified the memory overhead of PAD. Decoding with PAD (trained from Llama-3-8B) requires an additional 17,095 MB.

Despite this increase in processing time and memory, there is a notable enhancement in performance across all dimensions (e.g., 26% increase as for helpful and 24% increase as for harmless), indicating a reasonable tradeoff between alignment performance and inference costs. Moreover, PAD's advantage lies in its ability to align LLM outputs with diverse personalized preferences during the inference phase. This approach eliminates the need for the exhaustive process of retraining the RL model and allows for rapid adaptation in response to changes in personalized preferences. In addition, the efficiency gap can be further narrowed by employing a smaller reward model or parallelizing the reward computation across multiple candidates.

In summary, while we acknowledge and discuss the computational overhead introduced by PAD during the inference phase, the reduced training costs and superior performance make this a worthwhile trade-off. Additionally, we suggest that more efficient decoding-time alignment could be achieved using existing strategies. Based on your suggestions, we have added a new paragraph in Section 4.3 (Line 512-519) and Table C3 (Line 1053) that specifically discusses the Computational Cost for PAD. We appreciate your feedback, which has undoubtedly enhanced both the quality and comprehensiveness of our paper.

Table. Comparison of the performance and computational cost of various decoding strategies. Base: conventional greedy decoding with only the base model. Greedy: select the token with maximum score with PAD. Stochastic: sample a token from the probability distribution of the top-k candidate with PAD.

Dear Reviewer LdS8,

We hope this message finds you well.

We sincerely apologize for any inconvenience our submission may have caused and greatly appreciate the time and effort you have devoted to reviewing our paper. Your insights and comments have been invaluable to us.

We have carefully considered each of your suggestions and have made comprehensive, point-to-point responses and revisions in our revised manuscript. These adjustments are aimed at thoroughly addressing the concerns you raised, and we hope that our efforts resonate with your expectations for the paper.

At your convenience, we would be grateful if you could review our responses and the corresponding revisions. Should there be any further issues or if additional clarification is needed, please know that we are fully prepared to make the necessary adjustments.

Thank you once again for your dedication and assistance.

Best regards,

The Authors