Inference-Time Personalized Alignment with a Few User Preference Queries

Thank you for carefully reviewing our paper! We greatly appreciate your feedback. Please see below our responses related to your comments.

About the use of a fixed candidate pool and potential for dynamic generation

We thank the reviewer for raising this point as it aligns closely with a direction we’ve been eager to explore. We see promising opportunities to extend USERALIGN with dynamic pool augmentation, where new candidates are synthesized based on the user’s inferred preference distribution -- specifically targeting underexplored regions of the embedding space. While our current approach uses a fixed candidate pool, it naturally selects responses that lie within the feasible (i.e., consistent) region. Additionally, the pool can be enriched on-the-fly by biasing generation toward examples aligned with previously preferred responses.

One compelling way to bias the generator and enforce feasibility is as follows: once the user selects a few preferred responses, we feed those back into the generator -- e.g., by conditioning it on the last k accepted outputs -- so that it produces candidates aligned with that style or direction. Instead of presenting every new output, we then apply USERALIGN’s version-space filter: only those responses that satisfy all prior pairwise comparison constraints (i.e., lie within the current feasible region defined by intersecting half-spaces) are shown to the user. This ensures that dynamically generated candidates both reflect the user’s evolving preferences and remain consistent with all previous feedback. On the theoretical side, accounting for the non-stationarity due to dynamically generated candidates would introduce significant complexity and, we believe, deserves a dedicated follow-up investigation.

About the real‑world validity and a new user study

Our work focuses on short-term, task-specific interaction sessions, where users typically have clear and stable preferences. In such practical scenarios, assuming consistent and noise-free feedback is both reasonable and realistic. To tackle the reviewers’ concerns about real-world validity of our methodology, we would like to present results from a user study we have conducted. Given the limited time during rebuttal and high costs involved, we managed to conduct the user study for only one domain (food64d). Below we present the details of the study and the results. We will add the details about the user study and results to the updated version of the paper.

As part of this project, we had developed a lightweight web application to expose the methods considered in the paper through an interactive interface. We have now used this platform to conduct a user study with about 500 crowdsourcing participants to evaluate USERALIGN and IIDBest under realistic, noisy conditions in the food64d domain. Each participant was randomly provided with a question from the domain’s question set, and assigned either USERALIGN or IIDBest as their interaction method (blinded to the user). Participants were also placed in one of two personalization conditions: in the with persona condition, they saw a concise persona description (similar to our simulated users experiments) and were instructed to roleplay that persona; in the without persona condition, no persona was shown and participants made selections based on their own preferences.

Each participant interacted with the system in three stages. Stage 1 presented the question and either a persona (in the with‑persona condition) or an instruction to reflect on how the participant would personally like the question to be answered (in the without‑persona condition). Stage 2 consisted of a series of 10 pairwise comparisons, during which participants chose between two options that best fit the persona or their personal preference. In Stage 3, participants compared three final candidates against the zero‑temperature baseline response for their assigned question: the method’s selection at t = 10, the midpoint selection at t = 5, and a candidate chosen by the Random method (again, blinded to the user).

To ensure a fair comparison between methods and to keep the cognitive burden manageable for participants, we fixed the number of allowed comparisons during interaction for both methods to 10. This setting was chosen to allow enough interaction steps while staying within a manageable usage scenario. The candidate pool size was the same as that used in the main experiment, i.e., 20 generations.

For each of USERALIGN and IIDBest, we collected n = 120 evaluation sessions per setting; for Random, which does not depend on interaction steps, we recorded 2n = 240 samples. The tables below report the resulting win‑rates % across both personalization conditions.

• food64d - with persona

• food64d - without persona

About personalization and the use of user attributes or history

We thank the reviewer for raising this point! We are intrigued by this idea and would like to further expand on it. Our current work focuses on short-term, task-specific interaction sessions, where user preferences are assumed to be stable and well-defined. Nonetheless, as the reviewer suggests, incorporating user attributes or interaction history is an interesting direction for future work. Such information could be used to bias the generation of the candidate pool during interaction. However, this would likely require adapting our method to handle the non-stationarity introduced by evolving user profiles. To maintain stability, one could also consider gradually discounting older user comparisons over time.

Thank you for carefully reviewing our paper! We greatly appreciate your feedback. Please see below our responses related to your comments.

1. The experiment results lack a direct 1-1 comparison. All results are computed against a fixed response; however, this does not explicitly tell the quality gap between different methods. For example, a high win rate does not denote a high-quality gap against a fixed response. Therefore, a direct comparison between different methods' generation is beneficial.

We appreciate the reviewer’s concern and have given careful consideration to the design of our evaluation setup. We intentionally chose to compare each method’s selected response against a shared baseline, which serves as a consistent and neutral reference point across all methods. We believe this provides a meaningful and scalable proxy for relative quality: if a method consistently outperforms the baseline more often, it is producing responses that better align with user preferences (or those of the simulated model). This setup allows for fair comparisons across methods while maintaining a consistent evaluation scale.

Our choice follows best practices in the preference optimization literature, where win rates against a fixed baseline are commonly reported. For instance, SimPO explicitly reports win rates against a baseline model for the Arena-Hard benchmark (Meng et al., 2024). Similarly, Rafailov et al. (2023) evaluate their algorithms by win rate against a baseline policy, employing GPT-4 as a proxy evaluator, highlighting that theoretical analyses often rely on such a baseline as a reference point.

Nonetheless, as per reviewer’s feedback, we conducted a preliminary experiment to check whether the performance results seen w.r.t. the baseline also translate into the direct 1-1 comparisons. Across different settings, we have observed that a higher win-rate when comparing to the baseline translates to a high win-rate in direct 1-1 comparisons. In the updated version of the paper, we will report results for direct 1-1 comparisons between methods.

2. Following my question in the following section, the selection of y_2 is a major contribution. Because the current confidence-set-based implementation is quite complex, I wonder if a simpler approach will yield similar performance: we always set the confidence set to $\hat{\theta_t}$. Will the results be significantly affected in practice?

We thank the reviewer for raising this point about how our method could be simplified. After careful analysis, we would like to note that always setting the confidence set to $\hat{\theta_t}$ will not provide us a way to choose a pair of responses. The purpose of maintaining a confidence set is to systematically reduce uncertainty in $\hat{\theta_t}$, which is key to effective preference elicitation.

As discussed in Appendix E.1, our method (USERALIGN) remains computationally efficient, relying primarily on simple convex optimization.

3. Can authors explain how line 8 in algorithm two is computed? To my understanding, authors need to repeat the MLE optimization across the whole candidate set. How is the optimization conducted in a constrained space? How long will the optimization take for this line?

Line 8 of Algorithm 2 involves solving a set of convex optimization problems -- one per candidate in the pool -- with a linear objective and a convex constraint set. These problems are computationally efficient to solve. We discuss computational efficiency in detail in Appendix E.

Thanks for the rebuttal! I appreciate the authors' plan to report the 1-1 comparison in the future.

Regarding the efficiency: I appreciate the analysis in Appendix E, while I still think a better method is to report the wall-time of the optimization (same as the wall-time reported in data generation in Appendix E). Without it, I cannot have a clear picture of how the method's efficiency performs in the real world.

I will maintain my tendency to accept, but not to a higher score, because the current version lacks 1-1 comparison results and wall-time efficiency.

Thank you for carefully reviewing our paper! We greatly appreciate your feedback. Please see below our responses related to your comments.

About the practicality of pairwise comparisons vs. best-of-N selection

We appreciate the reviewer’s suggestion regarding best-of-N selection. Our approach is designed to efficiently infer user preferences and automatically select the best response, rather than relying on the user to manually choose from a full set of N generated options.

Presenting all N candidates at once can significantly increase cognitive load, requiring users to compare multiple options simultaneously and make fine-grained distinctions -- often resulting in less reliable feedback. In contrast, pairwise comparisons reduce this burden by allowing users to focus on just two options at a time, facilitating more consistent and focused judgments. This is supported by findings in the preference modeling literature, where pairwise comparisons are widely used due to their robustness and lower cognitive demand. For example, Ouyang et al. (2022) on InstructGPT and Rafailov et al. (2023) on DPO highlight the effectiveness of pairwise feedback in eliciting higher-quality preference data from human annotators.

Moreover, our method is designed to avoid showing the same pair multiple times, encouraging exploration of new generations while consistently tracking the most promising candidate. This strategy enables both diversity and convergence, making the interaction both efficient and user-friendly.

To empirically evaluate how our approach performs in settings where best-of-N selection becomes infeasible, we conducted experiments with a larger candidate pool (N = 1000). In such high-N settings, asking users to directly select the best response from the full pool would be cognitively unmanageable and practically infeasible. Instead, by relying solely on pairwise comparisons, our method remains both usable and effective. The results, shown below, demonstrate that USERALIGN maintains strong performance even in this high‑N regime, highlighting the method’s scalability and robustness. We will include these findings, along with an expanded motivation for pairwise comparisons, in the updated version of the paper.

• food64d - pool size = 20

• food64d - pool size = 1000

About the new user study and human evaluation

To tackle the reviewers’ concerns about real-world validity of our methodology, we would like to present results from a user study we have conducted. Given the limited time during rebuttal and high costs involved, we managed to conduct the user study for only one domain (food64d). Below we present the details of the study and the results. We will add the details about the user study and results to the updated version of the paper.

As part of this project, we had developed a lightweight web application to expose the methods considered in the paper through an interactive interface. We have now used this platform to conduct a user study with about 500 crowdsourcing participants to evaluate USERALIGN and IIDBest under realistic, noisy conditions in the food64d domain. Each participant was randomly provided with a question from the domain’s question set, and assigned either USERALIGN or IIDBest as their interaction method (blinded to the user). Participants were also placed in one of two personalization conditions: in the with persona condition, they saw a concise persona description (similar to our simulated users experiments) and were instructed to roleplay that persona; in the without persona condition, no persona was shown and participants made selections based on their own preferences.

Each participant interacted with the system in three stages. Stage 1 presented the question and either a persona (in the with‑persona condition) or an instruction to reflect on how the participant would personally like the question to be answered (in the without‑persona condition). Stage 2 consisted of a series of 10 pairwise comparisons, during which participants chose between two options that best fit the persona or their personal preference. In Stage 3, participants compared three final candidates against the zero‑temperature baseline response for their assigned question: the method’s selection at t = 10, the midpoint selection at t = 5, and a candidate chosen by the Random method (again, blinded to the user).

To ensure a fair comparison between methods and to keep the cognitive burden manageable for participants, we fixed the number of allowed comparisons during interaction for both methods to 10. This setting was chosen to allow enough interaction steps while staying within a manageable usage scenario. The candidate pool size was the same as that used in the main experiment, i.e., 20 generations.

For each of USERALIGN and IIDBest, we collected n = 120 evaluation sessions per setting; for Random, which does not depend on interaction steps, we recorded 2n = 240 samples. The tables below report the resulting win‑rates % across both personalization conditions.

• food64d - with persona

• food64d - without persona

About use of pretrained embeddings and feature granularity

In the paper, we demonstrate that our algorithm performs well in different preference spaces defined by pretrained sentence and image embeddings. In the additional user study we conducted, we further validated that these representations are sufficient to capture user preferences in a meaningful and robust way. While pretrained embeddings may have limitations in capturing fine-grained nuances, our method remains flexible -- specialized (e.g., handcrafted or learned) embeddings can easily be integrated to capture additional granularity when needed.

Dear authors, Thank you for clarification and additional experiments!

Through your rebuttal, I acknowledged the proposed method can reflect real-world human preferences.

Though I still have some concerns about the practicality in real-world settings, the inference cost for generating candidate outputs and the need for multiple rounds of user interaction (which also cause latency and human effort).

However, I see that your framework is novel and has potential to be useful for other applications, such as constructing high-quality preference datasets for training or applying to adaptive pool generation (as mentioned in section 6 of the paper). Importantly, theoretical groundness and superior results are promising.

Therefore, I raise my score.

Thank you for carefully reviewing our paper! We greatly appreciate your feedback. Please see below our responses related to your comments.

About the real‑world validity and a new user study

Our work focuses on short-term, task-specific interaction sessions, where users typically have clear and stable preferences. In such practical scenarios, assuming consistent and noise-free feedback is both reasonable and realistic. To tackle the reviewers’ concerns about real-world validity of our methodology, we would like to present results from a user study we have conducted. Given the limited time during rebuttal and high costs involved, we managed to conduct the user study for only one domain (food64d). Below we present the details of the study and the results. We will add the details about the user study and results to the updated version of the paper.

As part of this project, we had developed a lightweight web application to expose the methods considered in the paper through an interactive interface. We have now used this platform to conduct a user study with about 500 crowdsourcing participants to evaluate USERALIGN and IIDBest under realistic, noisy conditions in the food64d domain. Each participant was randomly provided with a question from the domain’s question set, and assigned either USERALIGN or IIDBest as their interaction method (blinded to the user). Participants were also placed in one of two personalization conditions: in the with persona condition, they saw a concise persona description (similar to our simulated users experiments) and were instructed to roleplay that persona; in the without persona condition, no persona was shown and participants made selections based on their own preferences.

Each participant interacted with the system in three stages. Stage 1 presented the question and either a persona (in the with‑persona condition) or an instruction to reflect on how the participant would personally like the question to be answered (in the without‑persona condition). Stage 2 consisted of a series of 10 pairwise comparisons, during which participants chose between two options that best fit the persona or their personal preference. In Stage 3, participants compared three final candidates against the zero‑temperature baseline response for their assigned question: the method’s selection at t = 10, the midpoint selection at t = 5, and a candidate chosen by the Random method (again, blinded to the user).

To ensure a fair comparison between methods and to keep the cognitive burden manageable for participants, we fixed the number of allowed comparisons during interaction for both methods to 10. This setting was chosen to allow enough interaction steps while staying within a manageable usage scenario. The candidate pool size was the same as that used in the main experiment, i.e., 20 generations.

For each of USERALIGN and IIDBest, we collected n = 120 evaluation sessions per setting; for Random, which does not depend on interaction steps, we recorded 2n = 240 samples. The tables below report the resulting win‑rates % across both personalization conditions.

• food64d - with persona

• food64d - without persona

About the diversity of the generated candidate pool

We appreciate the reviewer’s concern about candidate pool diversity. We agree that integrating with separate, domain-specific generation tools that explicitly aim to diversify generations across multiple or complex preference dimensions could make USERALIGN even more powerful. More generally, domains with known axes of variation (e.g., food2d) offer natural opportunities to enhance diversity during candidate generation. However, USERALIGN itself is fully agnostic to the generation process: it operates over any fixed set of candidates and makes no assumptions beyond the ability to embed them. In our experiments, we demonstrate that even when using general-purpose embedding models, USERALIGN remains effective across a variety of domains.

Regardless of the domain, increasing either the diversity or the size of the candidate pool would theoretically enhance alignment performance, as it becomes more likely to include responses closely matching nuanced user preferences. But even with relatively modest pool sizes (20 or 40 candidates), USERALIGN already achieves robust alignment, indicating our current generative strategy involving LLMs reasoning about potential interests provides a reasonable basis.

Finally, as shown in Appendix D, the algorithm remains effective even with unbiased pools, where we did not aim to induce a larger diversity through prompting. Examples of candidate pools are shown in Figure 1 and Appendix Figure 5.

Clarification on i.i.d. fallback behavior

We would like to clarify when USERALIGN resorts to i.i.d. sampling. By default, the algorithm stops once it reaches sufficient confidence and no viable challengers remain. However, in part of our experiments (see Section 5.2), we fixed the number of comparisons to T for consistent evaluation across methods. In those cases, once the preference space was exhausted, we continued by sampling comparisons uniformly at random. In real deployments, the algorithm would terminate early once it converges, without requiring i.i.d. sampling.