On-the-fly Preference Alignment via Principle-Guided Decoding

We appreciate the reviewer’s constructive comments and detailed review. We also thank the reviewer for pointing out some limitations. We hope that the responses can address the reviewer’s concerns and we welcome further discussion.

"Recent research has focused on alignment during the decoding phase, such as [1][2][3][4], and the authors should discuss these works."

Thanks for the reviewer’s suggestion to include a more detailed discussion of the related works. We have modified the Related Work section accordingly in Section 2.2 (line 149).

"Some content lacks clarity, such as what constitutes a Principle in Alignment?"

Thank you for pointing out the ambiguity in this terminology. In our context, a principle refers to a clear and descriptive guideline that encapsulates the underlying preference or desired behavior. This terminology is similarly employed in [1]. The purpose of principles is to explicitly instruct the model on what types of responses or practices are preferred and aligned with specific goals or values.

To improve clarity, we have revised Section 4.1 to include a Principle Curation part:

Throughout the experiments, we curate principles based on task-specific heuristics. For general alignment tasks that an instruction-tuned model typically handles well (such as helpfulness and harmlessness), we carefully curate the principle $c$ to better illustrate what this universal preference means, thus granting the model a better understanding of these universal principles. For personal preference alignment tasks, the principle $c$ is a direct statement to ask the model to behave in a desired manner (e.g. Please act as if you are an experienced researcher).

"if Principles are set in the system prompt, they can follow the described text…"

Thank you for the insightful suggestion. Adding principles via the system prompt is indeed a form of prompt engineering, where the system role provides global context for how the model should respond. In our experiments with Mistral and Vicuna (our base models), principles were set via system prompts, followed by user/assistant conversations. This approach corresponds to the Principle Prompting (PP) baseline. The results show that while system prompts can guide the model, they often fail to fully enforce the desired principles, where OPAD demonstrates a clear advantage.

Following your suggestion, we extended our experiments to the newer SOTA open-source model, Llama-3.1-8B-Instruct, where principles were also added via system prompts. Our findings are as follows:

On HH_RLHF: Llama-3.1-8B-Instruct, fine-tuned with SFT and RLHF for helpfulness and safety, establishes PP as a strong baseline. However, OPAD demonstrates an edge by further enhancing alignment with the desired principles.

On DSP: OPAD shows a notably greater advantage, excelling in aligning model outputs with target principles for personalized tasks, even when applied to this highly capable base model.

These results further highlight OPAD’s ability to improve adherence to principles, particularly for specialized or personalized tasks, even when using state-of-the-art models.

(Please see part 2 of our response.)

Thank you for your quick response and valuable suggestions for improving our manuscript!

First, we would like to apologize for the typo in our earlier response, where the "CD" baseline is actually the Self-Contrastive Decoding (self-CD) method (this has now been corrected, as well as Bo$N$ for HH-RLHF). For highly personalized tasks such as DSP, where no specific reward models are directly available, we omitted the Bo$N$ baseline for personalized alignment tasks, following the methodology outlined in the main paper. Since the base model is inherently aligned with helpfulness and safety topics and we use a very strong reward model that was trained on the HH-RLHF task to select the best answer, Bo$N$ demonstrates strong performance. However, Bo$N$ is resource-intensive, as it requires sampling multiple responses for ranking, making it less efficient in practical applications. In contrast, OPAD achieves reasonable results while being significantly more resource-efficient.

In response to your suggestion, we have conducted additional and comprehensive experiments with Llama-3.2-1B-Instruct, Llama-3.2-3B-Instruct, and Llama-3.1-8B-Instruct on both HH-RLHF and DSP tasks. These findings have been directly incorporated into the manuscript to enhance its robustness and contribution to the field. A detailed analysis is provided in Appendix C (line 804, highlighted in blue for your convenience). Please note that while the results are currently consolidated in the appendix, we plan to distribute them throughout the manuscript in the final version to better integrate them with the main text.

We greatly appreciate your feedback and please let us know if there are any further changes you would like us to make.

We want to thank the reviewer for the insightful feedback and the detailed comments that helps us better develop and validate the proposed approach. We hope that the responses can address the reviewer’s concerns and we truly welcome further discussion.

"designing well-aligned principles without access to $P_{data}$ seems almost impossible; at least a surrogate $P_{data}$ is needed for guidance and principle evaluation."

We apologize for any confusion caused by our earlier statement regarding “we have no access to $P_{data}$”. To clarify, we meant that we do not have direct access to the training data itself. We agree with the reviewer that designing well-aligned principles requires prior knowledge of the task, goal, or preferences involved. And we have revised the wording in Proposition 1 (Line 219 in blue) to ensure clarity and precision.

"how to design principles without access to $P_{data}$, and how to verify and guarantee this condition holds."

As clarified above, we need prior knowledge of the task, goal, or preferences when designing the principles, which is usually a clear and descriptive guideline that encapsulates the underlying preference or desired behavior. In our experiments, the preference principles are constructed using heuristics, and we have included a detailed explanation of this process titled Principle Curation in Section 3.3 (Line 316) as well as here for your convenience:

In our experiments, we curate principles using task-specific heuristics. For general alignment tasks (e.g., helpfulness and harmlessness), we define the principle $c$ to clearly communicate universal preferences, helping the model understand these concepts. For personal preference alignment tasks, the principle $c$ directly instructs the model to act in a desired way (e.g., "Please behave as if you are an experienced researcher"). The specific principles for each task are provided in Appendix E as well.

"In fact, even if the constraint $c$ partially aligns with $P_{data}$, it could still lead to bad alignment."

Typically, inference-time algorithms (without relying on RAG or additional sources) assume that the model already retains the necessary knowledge to answer the query, and our goal is to elicit it more effectively. The base model generating “health-related statements but with low accuracy” means the knowledge base preserved in the model is not sufficient, and re-training or fine-tuning may be required in this case.

As stated in Appendix C, we want to emphasize that the best use case for our method lies in scenarios where the model retains sufficient domain knowledge, but fails to follow the instructions perfectly. For such models, our approach effectively “reinforces” the principles to guide the model to respond in an aligned and preferred manner without requiring fine-tuning. Conversely, for smaller models that lack the foundational knowledge, or for larger models that already follow instructions well, the benefit of inference-time algorithms diminishes.

"the Methodology section lacks both a complete description of the decoding steps…"

This is a very pertinent point and we appreciate the reviewer for the suggestion to improve the clarity of this manuscript. We have relocated the 'Relation with the residual EBMs' section to the later analysis and provided a detailed decoding algorithm instead to improve clarity. (line 268 in blue)

(Please see part 2 of our response.)

We thank the reviewer for the detailed review and constructive comments. Below are our clarifications to all the questions. We hope that the responses can address the reviewer’s concerns and we truly welcome further discussion.

"In some cases, there is a high percentage of ties instead of clear wins or losses"

The high percentage of ties observed in some cases is primarily due to the inherent positional bias in using LLMs like GPT-4 as evaluators. Studies, such as [1], show that GPT-4 tends to favor the first displayed response, even when the order of candidates is changed. To mitigate this bias, we use a pairwise comparison approach, which is standard in preference evaluation pipelines.

Despite this mitigation, GPT-4 often provides a "tie" judgment or shows inconsistent preferences when the order of responses is switched, especially when the quality gap between the responses is small. This results in higher tie rates.

In instances where the tie rate is high, it indicates that the baseline method is closely comparable to our proposed OPAD method, leading to GPT-4's inconsistent preferences upon order reversal (e.g., BoN in Table 1). However, it's important to note that in most cases, the Win rates for our method still dominate.

Besides, high tie rates are not uncommon in preference evaluation literature, as observed in [2] (40-50% in Figure 2), [3] (around 60% in Figure 5), and [4] (50-100% in Figures 4 & 5).

[1]. Wang, Peiyi, et al. "Large language models are not fair evaluators." arXiv preprint arXiv:2305.17926 (2023).

[2].Gao, Songyang, et al. "Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback." arXiv preprint arXiv:2401.11458 (2024).

[3]. Zeng, Yongcheng, et al. "Token-level Direct Preference Optimization." arXiv preprint arXiv:2404.11999 (2024).

[4]. Shi, Chenyu, et al. "Navigating the overkill in large language models." arXiv preprint arXiv:2401.17633 (2024).

"it would be valuable to demonstrate its performance on larger, more capable models to assess the trend and generalizability of your approach across different model scales"

Thanks for the valuable suggestion. We conducted additional experiments on models of varying capabilities, including Vicuna-13B, Vicuna-33B (more capable), and Pythia-2.8B (less capable), across a general alignment task (HH-RLHF) and a personalized task (DSP).

The results with PP (principle prompting) clearly illustrate key trends in the scalability and generalizability of OPAD. We observe an increasing then decreasing trend in OPAD's relative effectiveness as the model scale increases:

(Please see part 2 of our response.)

(Please read part 1 of our response first.)

1. Performance on mid-scale (less capable) models: OPAD is most effective on mid-scale models (e.g., Vicuna-7B and Vicuna-13B) that retain sufficient domain knowledge, but fails to follow the instructions perfectly. OPAD manages to effectively “reinforce” principles to improve alignment. In practical scenarios, 7B and 13B models are widely used due to their balance between capability and resource efficiency. OPAD can help these models adapt better to task-specific requirements during decoding without requiring fine-tuning.
2. Performance on larger (more capable) models: As model scales up, direct prompting with principles (PP) becomes a stronger baseline. Larger models are inherently better at following instructions, exhibiting diminished marginal benefits from OPAD.
3. Performance on smaller (weak) models: For weak models (Pythia-2.8B), they often lack the foundational ability or knowledge to respond appropriately to user queries, making the “reinforcement” provided by OPAD less impactful. In these cases, fine-tuning is ultimately required.

We have incorporated the findings and analysis directly into the manuscript in Section 4.4, Line 426 (highlighted in blue) to showcase the generalization capability of the proposed method. This addition offers clear guidance on the optimal use cases for our approach. Thank you again for your valuable feedback!

"How does your method perform when LLMs do not achieve sufficient performance, such as smaller LLMs?"

To explore this, we evaluated our method using the SFT-ed Pythia-2.8B model, which is a smaller and much weaker LLM. The result, as indicated above, shows that OPAD is less effective when applied to models that do not inherently achieve sufficient performance, since the reinforcement signal brought by our method cannot fully compensate for the model’s inherent weaknesses. However, compared to other inference-time baselines, OPAD still demonstrates an advantage.

"Would it be more appropriate to rename Section 5 from 'Discussion' to 'Conclusion' for clarity?"

Yes, we have renamed Section 5 to “Conclusion” for clarity. Thanks again for the valuable feedback!

Thanks again very much for the review. We hope that our reply has addressed your concerns. Please let us know if there are any further changes you would like to see or if there is anything else that we can clarify.

Dear Reviewer mD8g,

We hope this message finds you well.

We appreciate your time and effort devoted to reviewing our paper and your insights and feedback have been truly invaluable to improving our manuscript.

The discussion period ends soon and we have made significant efforts to address your concerns with our detailed reply and additional experiments. In particular, we hope that our reply will clarify your major concerns about the scaling effects of the proposed method.

At your convenience, we would greatly appreciate it if you could engage with us and our rebuttal during the remaining time of the discussion so that we can try to address any additional concerns and further improve our manuscript.

Many thanks in advance

Best regards,

The Authors