Discovering Factor Level Preferences to Improve Human-Model Alignment

We thank the reviewer for recognizing the strengths of our work, including the presentation of "an explainable framework that enhances understanding of human-model preference alignment" and addressing "a significant gap in current methods by focusing on specific factors influencing preferences." We appreciate the acknowledgment that our approach has "practical implications for developing more human-aligned LLMs." In response to the reviewer's valuable feedback, we have incorporated revisions to further clarify and enhance the manuscript, marked in red. Below, we address specific comments and suggestions in detail.

“might not thoroughly compare with existing methods, leaving questions about its relative advantages”, “... compare quantitatively and qualitatively with existing methods in preference alignment? Such as all kinds of llm-as-a-judge methods / G-Eval / ... etc”

(CLARIFICATION) Existing methods like LLM-as-a-judge and G-Eval primarily focus on evaluating how well a model's outputs meet predefined criteria, aiming to score or rank responses based on absolute quality metrics such as fluency, coherence, and relevance.

PROFILE diverges fundamentally from these approaches because its goal is to interpret what the model values when generating or evaluating better outputs. Rather than simply assessing output quality, we investigate why certain outputs are deemed better by uncovering the underlying preference mechanisms.

• Qualitative Difference: PROFILE seeks to understand preference alignment by uncovering the importance of different criteria (e.g., fluency, coherence) in the model's own judgments between high-quality and low-quality outputs. For example:
  • LLM-as-a-judge evaluates how "fluent" an output is.
  • PROFILE evaluates how fluency is differentially represented in outputs the model prefers versus those it does not, revealing the model's internal prioritization.
• Quantitative Advantage: PROFILE's methodology enables the analysis of relative differences between the factors, highlighting which criteria significantly influence the model's preferences. This offers a novel lens for interpretability not captured by traditional evaluation methods.

"lack sufficient empirical validation due to limited experiments or datasets"

(CLARIFICATION) While we appreciate the reviewer's concern regarding the limited dataset coverage, we want to clarify that we intentionally chose three representative tasks—summarization, helpfulness, and document QA—because the factors in these tasks encompass core aspects of preference alignment tasks, making our findings broadly relevant to similar tasks.

"Can you provide more details on how the framework performs when applied to tasks beyond the three studied, and are there limitations to its generalizability? Such as creative writing, role-playing, and coding."

(CLARIFICATION) We deliberately excluded tasks such as coding and mathematics from our evaluation because these domains are primarily assessed by output "correctness" rather than nuanced preference alignment, which falls outside the core objective of our study.

However, PROFILE's methodology is inherently extensible to other domains such as creative writing and role-playing. The process is straightforward:

1. Define additional factors (e.g., "plot originality" for creative writing).
2. Establish definitions and prompts for evaluating these new factors.
3. Extract factor-level preferences and analyze them using our metrics as outlined in the paper.

This demonstrates PROFILE's potential for broad applicability beyond our initial tasks.

"concerns about the scalability of the proposed framework without fine-grained human annotations, impacting its practicality."

Thank you for your thoughtful review and careful consideration of our work. We appreciate the opportunity to provide clarification.

(CLARIFICATION) Far from being a limitation, the absence of fine-grained human annotations is actually a key advantage of our proposed framework. As explained in lines 122-123 ("Specifically, we experiment with a pairwise preference setting, where preferences are determined by comparing two responses"), our methodology requires only coarse-grained preference information from both human and model responses. As detailed in Section 3.2, PROFILE compares response-level preferences with factor manifestations extracted automatically to uncover each factor's influence. In other words, automatically analyzing the impact of fine-grained factors is the core of our PROFILE methodology.

(REVISION) To address potential misunderstandings, we have enhanced the clarity and organization of Sections 2 and 3 in the draft, providing more precise explanations of our approach. We hope the reviewers will reconsider their concerns in light of these clarifications and revisions.

“The boundary between Receptiveness / Intent Align / Helpfulness is vague and not independent of each other.”

(CLARIFICATION) The distinctions between Intent-Alignment, Receptiveness, and Helpfulness are more nuanced than they might initially appear: Intent-Alignment and Receptiveness differ in their comparative approach: one compares to the source, while the other examines input-output relationships. Because of this inherent difference in the tasks they are considered, they are not measured at the same time, so they are independent of each other. Receptiveness and Helpfulness are defined distinctly as Input-Output vs. Output-only metrics, and have been treated differently in prior research. Specifically, while receptiveness measures “Whether the core question of the input has been answered,” Helpfulness measures “The ratio of facts that provide additional helpful information.”

(ADDITIONAL ANALYSIS) Our additional feature correlation analysis (Appendix G.3, Figure 5) supports these distinctions. We analyzed the correlation of feature score distributions across model-generated samples Receptiveness and Helpfulness showed low correlation across models:

• Highest correlation: GPT-4o at 0.63
• Lowest correlation: Tulu-PPO-13B at 0.19

Since the correlation between the factors is not consistently high and varies depending on the model, these results suggest that even if the factors in GPT-4o exhibit high correlation, it is not due to the inherent characteristics of the factors themselves, but rather to the model's behavior.

Dear Reviewer vTJd,

We understand the discussion timeline is inconvenient for your busy schedule, but we would love to continue our discussion before the time window closes.

We hope that we were able to resolve all your questions and please let us know if there's more.

Best,
authors

Thank you for your detailed response. In the absence of evidence for further generalizability, I will keep my score the same.

We thank reviewer ZMM8 for their feedback, for recognizing the core value of our work in "identifying specific reasons for human-model preference divergence," and for finding our analysis “comprehensive.” We would like to note that key changes in the updated draft are highlighted in red. Below, we address the reviewer’s specific concerns:

"uses GPT-4o for factor quantification while examining human-AI preference alignment, which may introduce new biases"

(CLARIFICATION) We appreciate the reviewer's concern about potential biases introduced by using GPT-4 for factor quantification. Our decision to use model-based extraction was primarily driven by the need for scalability, while building upon established precedent from [1], which demonstrated the reliability of GPT-4 for similar factor extraction tasks.

While we acknowledge the inherent limitations of model-based evaluation, we took several steps to validate and ensure reliability. Detailed documentation of our validation process can be found in Appendix D.3. Specifically,

1. We drew on existing research confirming GPT’s capabilities in atomic fact extraction [1] and factor judgment [2].
2. We validated our prompts through iterative refinement and human verification, with strict reliability thresholds (>80% alignment with human judgment) before incorporating model-based judgments.

We hope the reviewers will reconsider their concerns in light of these clarifications and our comprehensive validation efforts.

"validates the PROFILE framework on a limited set of public datasets, which restricts its demonstration of applicability to other tasks"

(CLARIFICATION) While we appreciate the reviewer's concern regarding the limited dataset coverage, we want to clarify that PROFILE is designed as a generalizable methodology rather than a task-specific framework. We intentionally chose three representative tasks—summarization, helpfulness, and document QA—because the factors in these tasks encompass core aspects of preference alignment tasks, making our findings broadly relevant to similar tasks.

Furthermore, PROFILE's methodology is inherently extensible to other domains. The process is straightforward:

1. Define additional factors (e.g., "plot originality" for creative writing).
2. Establish definitions and prompts for evaluating these new factors.
3. Extract factor-level preferences and analyze them using our metrics as outlined in the paper.

This demonstrates PROFILE's potential for broad applicability beyond our initial tasks.

“lacks sufficient ablation studies to analyze the contribution of each factor”… “impact across tasks”

(CLARIFICATION) We want to emphasize that PROFILE is inherently designed to analyze the individual contribution of each factor, making this factor-level analysis a core feature of our framework.

Our results in Sections 4.1 and 4.2 demonstrate how each factor impacts preference alignment across two scenarios: (1) when models act as generators and (2) when models serve as evaluators. The impact of individual factors is illustrated in Figure 3 (a), (b), and (c), highlighting their contributions across different tasks. For deeper insights, we provide extensive factor-level breakdowns in Appendix G.1, including Table 5 (Summarization), Table 6 (SHP-2), and Table 7 (WebGPT).

We believe this detailed factor analysis serves the same purpose as traditional ablation studies by revealing the contribution of each component. Nevertheless, we welcome specific suggestions for additional ablation studies that could further strengthen our analysis.

“The paper does not clarify how PROFILE enhances model performance or impacts generation quality in practical applications”

(CLARIFICATION) We appreciate the reviewer's concern regarding the practical impact of PROFILE on model performance and generation quality. We believe that the tasks we selected—summarization, helpful response generation, and question-answering—are directly relevant to real-world applications. The datasets used for these tasks are based on real-world Reddit posts, which align closely with practical, user-driven contexts. An example of the task query is “I want to go for a PhD in Philosophy after undergrad: is that stupid?”

By aligning model preferences on these tasks, PROFILE contributes to improving model performance in scenarios that are immediately applicable to practical applications.

References
[1] Hu, Y., Song, K., Cho, S., Wang, X., Foroosh, H., & Liu, F. (2023). Decipherpref: Analyzing influential factors in human preference judgments via gpt-4. arXiv preprint arXiv:2305.14702.
[2] Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W. T., Koh, P. W., ... & Hajishirzi, H. (2023). Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251.

Dear Reviewer,

Thank you. We are glad our updates and clarifications were helpful. If you have any further questions, please feel free to let us know.

Best, Authors

We thank the reviewer for recognizing that our work is "both interesting and important." We appreciate their view that our findings could be "valuable for researchers interested in human alignment, LLMs, and explainable AI" and their positive assessment of our paper's organization. In response to their feedback about writing clarity, we have made revisions throughout the draft, marked in red. Below, we will address the reviewer's specific comments and questions in detail.

“would benefit from more precise notation to improve clarity.”, “L134-135: The notation of score level s and Score(r) is confusing. If Score(r) already equals s, what is the purpose of having the model assign a score again?”

(CLARIFICATION, REVISION) Thank you for highlighting areas in the explanation that could be improved. We have updated Sections 2 and 3 to clarify our definitions and methodology.

To address this, we would like to clarify the scoring process:

We operationally define the preferences of a model through the responses it generates at predefined score levels. Specifically, we prompt the model to generate responses that match a set of predefined scores, represented as R = {r_score | score ∈ {1, 2, 3, 4, 5}}. This systematic prompting ensures that the model's preferences are derived based on its own scoring mechanism. Please refer to the prompt example in Appendix E.1.1 for additional clarity.

We hope this resolves the concern and aligns with your expectations for the explanation.

“The writing quality could also be improved." "... the novelty and key ideas are not clearly articulated.”

(REVISION) We appreciate your valuable feedback on the clarity of our writing. Your comments have been instrumental in helping us significantly improve our manuscript. We have made the following revisions:

• Introduction: Refined the organization and provided a clearer articulation of our methodology's novelty.
• Sections 2 and 3: Improved the clarity of operational definitions and notation.

“A better discussion and analysis are needed. Some findings and conclusions lack depth and specificity.”

(REVISION) Upon review, we realized that the phrasing in the relevant paragraph may have been misleading. To address this, we have revised the discussion in Lines 474–482 to enhance clarity and provide more depth and specificity to the findings and conclusions.

Please let us know if further refinements are needed.

"The so-called "generalizability" conclusion seems to be derived solely from the summarization experiment, which may not provide sufficient support."

We acknowledge that the term "generalizability" has been used in multiple contexts throughout the paper, which may have caused confusion. Below, we clarify the distinct uses of "generalizability" and the revisions made to address potential misunderstandings:

1. Generalizability of our experiments with score-based generation

   • (CLARIFICATION): In the third paragraph of Section 5, we analyzed whether our score-based generation approach—where models generate responses on a 1–5 scoring scale—produces findings that generalize to standard, unconstrained model responses. Specifically, we examined which score-level responses most closely resemble standard responses without predefined scores.

   • (REVISION): To better reflect the focus of this discussion, we have renamed the paragraph from "Generalizability of Our Results" to "Validity of the Score-Based Generation Approach." Additionally, we expanded our analysis beyond summarization to include other tasks, providing stronger empirical evidence to support our conclusions. Please see Table 10 in the Appendix H.

2. Generalizability of the framework

   • (CLARIFICATION) We want to clarify that PROFILE is designed as a generalizable framework rather than a task-specific methodology. We intentionally chose three representative tasks—summarization, helpfulness, and document QA—because the factors in these tasks encompass core aspects of preference alignment tasks, making our findings broadly relevant to similar tasks. We have conducted experiments across all three tasks, with results beyond summarization (Helpfulness and QA) detailed in Appendix Table 7-9. We revised the parts where the writing might misleadingly refer to the generalizability of the results rather than the generalizability of the framework.