Inference time LLM alignment in single and multidomain preference spectrum

We begin by thanking the reviewer for their feedback. Here’s how we reflected on the feedback:

1. First, we want to clarify that, we use both GPT-4 and Claude 3 sonnet for different purposes: Claude for data generation, and GPT-4 for evaluation. We also added a clarification why claude-3-sonnet was used over GPT-4 model for synthetic data generation in section 4.1. We did this because claude performs comparably to GPT-4, often ranking among the top foundational models. Additionally, using GPT-4 as an independent evaluator helped avoid the known bias of models favoring their own outputs during evaluations. However, we agree that we should perform the alignment experiment with more open-source LLMs like Llama and Qwen, which we plan to do in the future (added to the conclusion).

2. We corrected the citation inconsistencies and made all citations as “\citep”.

3. We agree that experts might want to have an expert model on one specific domain area, and our single-domain preference tuning idea exactly resonates with that application. However, as we explained in the third paragraph of the introduction, several business needs may require multidomain preference objective. This is specifically true when it comes to organizations operating in overlapping domains, like, balancing an insurance company's need for expert legal responses, generic financial answers, and avoiding medical responses. Handling all of them can be challenging, and deploying separate models could be resource-intensive. Joint training with targeted preferences offers a solution but that is inflexible and requires massive training effort.

4. As per your suggestion, we added a detailed mathematical formulation for Preference Accuracy in section 5.1.

5. We updated the description in Figure 2 .

6. We used Claude-3-Sonnet only for synthetic data generation, GPT-4 for evaluating the generations. However, these models are not open-sourced, and therefore, we couldnot use them to experiment with our objective. Therefore, we used Mistral-7b for model editing and alignment, as an open source candidate model.

7. We also appreciate your suggestion on why we need inference time alignment over conventional approaches, and we added a box in the introduction section to highlight the necessity of inference time alignment. Basically, in contrast to conventional training time approaches, inference time alignment provides flexibility and adaptability by enabling dynamic adjustments to model behavior based on task or user needs without retraining.

Answers to your questions:

1. We checked the valid persona-query pair by manually investigating 50 samples. We estimate that at least 93% of all the samples are valid with 95% confidence (computed based on wilson score interval). We found Claude-3-sonnet reliably follows the instruction provided in our instruction prompt. To choose the number, we first recursively generated 15k queries for all domains, as explained in section 4.1. Next, we found 1-3% of queries and responses being cut short due to timeout and quota limit issues, which were removed. We also found a few queries where language was not English, and those were removed as well. Furthermore, we only considered complete sets of persona-query-three proficiency level responses, and we had to discard a few samples for that as well. Finally, we looked for repetitions of queries, and while some personas were very close, we didn’t find queries to be repetitive. After all these process, we ended up getting 13,000 personas for the medical domain, 12,374 personas for the financial domain, and 12,867 personas for the legal domain. We included this part in the appendix B.

2. There were only human annotators for evaluating the LLM generation quality

3. This is an important question, and we also think this term can be misleading. In our paper, we used the term “ground truth” only once in the section 4.2, to illustrate how we compute LLM and human agreement accuracy. However, we removed it, and rephrased the sentence as “We also calculate the average annotation agreement by each annotator with the LLM generation. “

4. A multidomain preference approach addresses the need for organizations operating in overlapping domains, such as balancing expert legal, financial, and medical responses, where deploying separate models is resource-intensive. Joint training with targeted preferences is inflexible and requires significant effort, making multidomain tuning a practical solution.

5. We found the preference accuracy best describes the alignment quality. Basically it’s a measure where we compute the accuracy for the alignment level of LLMs, where we seek out of N total queries, how many the LLMs would reward the preferred answers. We also added the mathematical description of this metric on Q4 answer. However, if reviewers suggest, we will be happy to explore other metrics.

6. We used the same metric “preference accuracy” for safety and helpfulness.

We would like to start by expressing our gratitude to the reviewer for their valuable feedback. Here is how we have addressed and reflected upon the comments:

1. Rame et al.’s [1] work is closely related to our multi-domain preference alignment. However, their approach focuses on training-time alignment by interpolating weights from models fine-tuned on diverse rewards to achieve Pareto-optimality. In contrast, our work introduces a preference adjustment strategy that operates at inference time, in addition to achieving multi-dimensional alignment. Similarly, while Jang et al. [2] address personalized preference alignment and post-hoc merging, our approach provides a unique capability: preference level adjustment. This feature offers greater granularity and control, enabling dynamic and fine-grained customization of model behavior. We’d like to thank you for suggesting these papers, and we also included these details in our “related work” section.

2. We appreciate the concern regarding the theoretical motivation. While some previous works explored the weight interpolation, our research focuses on the control and tunability aspect of it by a rather “incremental” interpolation. Also, this area was unexplored for LLM alignment objectives.

3. Next, we added the details of dataset creation, especially how the reported numbers were achieved, in appendix B. Basically, we generated queries across multiple domains and rigorously filtered the dataset by removing truncated, non-English, and incomplete entries while ensuring linguistic consistency and uniqueness.

4. We also appreciate the paper restructure suggestion, and we moved the Methodology section before the “Synthesizing Specialized Preference Data”.

5. Finally, we agree that using the term “preference accuracy” can be ambiguous, and therefore, we added the mathematical description of this metric in Section 5.1

We sincerely thank the reviewer for their valuable feedback and address the comments as follows:

1. First, we agree that our method of inference time alignment still requires training the model to align. And you rightly pointed out, when AVs are added to the unaligned model, it returns to the fully aligned model. However, our objective is to achieve the preference alignment tunability in the inference time, which otherwise would require separate training objectives. We mainly replaced the expensive process of retraining with AV-based inference time model editing.

2. The model editing process is done in O(n) times, being much faster than any training, and therefore, inference time also does not increase. Also, model editing is done before inference, and we only need to do it once instead of every inference time. Furthermore, in a multi-domain setting, training time alignment could have m^n training combinations (m=levels, default is 3), whereas inference time alignment only requires n times training, and provides finer-grained, continuous levels.

3. Finally, we agree that we cannot be sure about the generated content in specialized domains being accurate or not. However, we found claude models being used to evaluate responses to medical questions, and it showed superior performance on National Board of Medical Examiners Sample Questions over GPT3.5 [1,2]. However, we will provide an in-depth LLM-generated response correctness evaluation in our future work (this part is added to our limitation section as well). (Please note, in Appendix E, we included some sample generations for the readers)

[1] Abbas, A., Rehman, M. S., & Rehman, S. S. (2024). Comparing the Performance of Popular Large Language Models on the National Board of Medical Examiners Sample Questions. Cureus, 16(3).

[2] Hosseini, P., Sin, J. M., Ren, B., Thomas, B. G., Nouri, E., Farahanchi, A., & Hassanpour, S. (2024). A Benchmark for Long-Form Medical Question Answering. arXiv preprint arXiv:2411.09834.

We’d like to express our gratitude for the kind words and feedbacks. Here’s how we address the comments:

1. We agree that GPT-4 judged metrics can be stronger with human study. However, in a small-scale evaluation, we found that humans have an ~87% agreement with claude-based proficiency level response generation, and thus, a high preference accuracy score can also mean highly in agreement with human judgment, as preference accuracy measures the alignment level to preferred responses.

2. We also agree with the reviewer feedback that we should use more LLMs to validate our claims, and we plan to do it in the next step (also outlined in section 7).

3. We also want to mention that most test-time alignments are prompt-based, be it a prompt engineering technique or including special symbols/signs in the prompt. However, our research objective is to tune the preference level so that the granularity in the preferenc level could be achieved. We did not find the other existing alignment methods are directly comparable to our approach. This issue is also briefly discussed in section 2.

More details on “over-generalization” effect:

Aligning LLMs with multiple training objectives can often be counterproductive, since these objectives can be orthogonal in many cases, due to complex human preference nature. We have seen safety alignment reducing helpfulness [1] and reasoning capabilities [2], indicating the aligned behavior being widespread across all preference dimensions. In our case, aligning a model for responding to medical queries with expert opinion also induces expertise for other domains like finance or legal. This indicates instead of picking up the “domain-specific-proficiency-behavior”, the model picks up the “overall-proficiecncy-behavior”, which makes it difficult to do multi-domain alignment.

[1]Tuan, Y. L., Chen, X., Smith, E. M., Martin, L., Batra, S., Celikyilmaz, A., ... & Bikel, D. M. (2024). Towards Safety and Helpfulness Balanced Responses via Controllable Large Language Models. arXiv preprint arXiv:2404.01295.

[2]Alami, R., Almansoori, A. K., Alzubaidi, A., Seddik, M. E. A., Farooq, M., & Hacid, H. (2024). Alignment with preference optimization is all you need for llm safety. arXiv preprint arXiv:2409.07772.

Thank you. We will address these concerns in the next cycle.