Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment

Thank you very much for your thorough review and valuable feedback. We sincerely appreciate your recognition of our work’s strengths, including the importance of personalized alignment, the controlled evaluation setup, and the comprehensive comparison with baselines.

We now address the concerns you raised.

W1 & Q1: Limitations of the profile reward function

We sincerely appreciate this insightful observation. The concern is indeed valid, and we have taken measures to address it in our reward design. Specifically, we adopt a slot-value structured matching approach, rather than relying on surface-level word matching, to mitigate discrepancies caused by semantically equivalent but lexically different expressions. As described in §3.2 of our paper (Page 5, Lines 156–165).

Moreover, during the construction of training and evaluation data, we utilize GPT-4o-mini to convert natural language profiles into structured JSON representations (Appendix D.2). This conversion enables a certain degree of semantic normalization, helping to reduce the sensitivity of the reward to superficial wording differences.

That said, we acknowledge that this method is not foolproof and may still fail to capture all semantically equivalent expressions. To address this limitation, we plan to explore more robust semantic matching mechanisms—such as entailment-based or embedding-based similarity metrics—in future work.

W2 & Q2: Lack of clarity on the response reward design

We apologize for the insufficient explanation and would like to provide further clarification below:

As defined in §3.3 (Page 5, Lines 172–176), the response reward is computed in Equation 4.

That is, a response must first satisfy all five foundational quality criteria (N, R, L, G, F) before any reward is assigned based on the four personalization-related dimensions. This gating mechanism ensures that only responses of sufficient quality are eligible for personalized alignment rewards, thereby increasing the discriminative power of the reward signal and preventing misleading gradients during training due to low-quality outputs.

To further clarify, we provide the full bilingual (English and Chinese) evaluation prompt used by the reward model in Appendix D.4, which includes detailed definitions, judgment rules, and examples for each dimension and criterion. We will additionally include a reference to this appendix in the main text in the camera-ready version to improve clarity and coherence.

W3 & Q3: Lack of clarity on the response reward design

This is an important and insightful concern — thank you for raising it. We have conducted an additional analysis to better quantify context usage in our 70-turn long dialogue setting. Our empirical statistics are as follows:

• On average, user utterances are ~74 tokens per turn.
• The assistant’s response is ~67 tokens, and
• The inferred profile, which is explicitly generated each turn, is ~97 tokens.

This results in an average of ~164 tokens per turn, accumulating to over 16,000 tokens across 70 turns — far exceeding the typical context window limit (8192 tokens). In practice, we confirmed that context truncation does occur in long interactions.

However, we would like to emphasize a key design feature of RLPA that addresses this issue: the inferred profile is not only included in the generation output, but also re-injected in a structured format at the beginning of the next turn. This explicit profile tracking ensures that even when earlier context is truncated, the essential personalization information is retained and reintroduced, preserving alignment quality over long dialogues.

We will include this empirical analysis and clarification in the revised version of the paper to more explicitly address concerns about the effectiveness of our method in long-term interactions.

Once again, we are grateful for your thoughtful review, and we hope our responses address your concerns.

We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate the recognition of our contributions. Below, we address the reviewer’s concerns point by point.

W1: Limited evaluation on only one dataset and model

Thank you for raising this important point. We would like to clarify that our experiments are not limited to a single dataset setting. In addition to the original ALOE benchmark, we introduce an Extended ALOE setup that explicitly targets out-of-distribution (OOD) generalization.

Specifically, Extended ALOE is constructed by sampling from PersonaChat and ensuring there is no overlap in profile field types between the training and test sets. This allows us to evaluate RLPA’s robustness to unseen profile structures and out-of-domain dialogue formats.

To further address your concern, we add the main experiments using the gemma-3-4b model.

• Vanilla ALOE

• Extended ALOE

RLPA consistently outperforms all baselines under both settings. These results strengthen our claim that RLPA is model-agnostic and generalizes well across profile structures and data distributions. We will include these results in the updated version and further discuss future directions for broader model family and dataset evaluation.

W2: Missing definition of profile inference and justification of evaluation metrics

Thank you for pointing this out. We acknowledge that the term “profile inference” was introduced without an explicit definition in the main text, which may hinder clarity for some readers. To address this, we will revise the beginning of Section §3.2 to include the following clarification:

Profile inference refers to the model’s ability to estimate the user’s latent profileat each turn of the dialogue, based on prior conversational history. This predicted profile is then compared to the ground-truth profile in a slot-wise manner.

Regarding the evaluation metrics, we employ three normalized measures—Align. Score, N-IR (Normalized Improvement Rate), and N-R² (Normalized R-squared)—to assess response alignment with user preferences. Their motivations and formulas are thoroughly detailed in Appendix D.3:

• Align. Score is the primary metric and is computed using GPT-4o-mini to assess alignment between the model’s response and the ground-truth user preference.
• N-IR accounts for variance in baseline performance across users and contexts.
• N-R² measures the goodness of fit for alignment behavior across turns, reflecting consistency with the inferred profile.

We agree that briefly explaining the rationale behind these metrics in the main text would improve accessibility. We will update §4.1 (“Metrics”) to include concise justifications for all three metrics.

We appreciate your attentive review and helpful suggestions, which will help us strengthen the clarity and rigor of the final version.

Q1: Alternative reward formats (e.g., free-text profile generation)

We sincerely thank the reviewer for raising this insightful and forward-looking question. While free-text profile generation offers greater flexibility, we opted for the slot-value format in this work based on both theoretical and empirical considerations:

1. Slot-value format is more structured, user-centric, and interpretable

Free-text profiles, though expressive, often result in verbose, redundant descriptions that accumulate surface-level preferences from recent turns, rather than capturing core, long-term user attributes. Consider the following comparison:

• Free-text profile:

Prefers to grab food with friends after surfing; has a favorite beachside taco place with exceptional fish tacos and mango salsa; enjoys pairing food with drinks, especially cold coconut water…

• Slot-value profile

{ "Age": "30", "Interests": ["Surfing", "Collecting sneakers"], "Personality Traits": ["Adventurous", "Ambitious", "Sociable"], "Occupation": "Personal trainer", "Others": ["Has a pet snake named Sly"] }

The slot-based format allows for concise abstraction, supports long-term user modeling, and enables direct reward computation, which is crucial for stable RL training.

2. Slot-based profiles yield better performance in profile-matching tasks

We conducted additional experiments comparing slot and free-text profiles across multiple LLMs (e.g., GPT-4.1-mini, O3-mini, DeepSeek). The results (below) show slot-value formats outperform free-text across all metrics, including Exact Accuracy, Fuzzy Accuracy, MSE, and RMSE:

• Slot

• Free-text

Free-text matching is further challenged by expression variability, redundancy, and noisier semantic signals.

3. Slot-value profiles improve alignment and training stability in RL

We trained RLPA using both formats under the same settings. As shown below, the slot-based version significantly outperforms the free-text version in alignment score (AVG), final step accuracy (k=10), and training stability (N-R²):

This highlights the benefit of slot-value representation in maintaining long-term consistency and efficient optimization during reinforcement learning.

4. Embedding-based or ROUGE-based reward alternatives are less reliable

We also experimented with free-text alignment metrics using ROUGE-L and text embedding similarity. ROUGE-L proved fragile to paraphrasing:

“Occupation”: “Personal trainer” vs. “Fitness coach” → ROUGE-L ≈ 0

Embedding methods (e.g., OpenAI’s text-embedding-3-large) showed acceptable but inferior performance and required manual threshold tuning:

Moreover, the added computation and hyperparameter overheads make them less suitable for multi-turn, online RL optimization.

Conclusion: While free-text profile generation is an interesting future direction, our experiments demonstrate that slot-value formats are more interpretable, scalable, and empirically effective for structured profile modeling and reinforcement learning. We will clarify this design choice and include these new findings in the revised version.

Q2: Reward component for query relevance

In fact, this was explicitly considered in the design of our Response Reward function (§3.3).

Specifically, we employed GPT-4o-mini to score generated responses across five binary dimensions:

• Naturalness (N)
• Relevance (R)
• Logical consistency (L)
• Engagement (G)
• Informativeness (F)

The final reward is computed as the product of all five criteria: (as described on Line 175 of §3.3). This means that a response must satisfy all five aspects simultaneously to receive a positive reward.

Importantly, the Relevance (R) dimension directly evaluates whether the model’s response appropriately addresses the user’s current input, rather than merely repeating or restating user profile information.

Therefore, our reward mechanism is already designed to capture both faithfulness to inferred user profiles and responsiveness to the user query, ensuring a holistic alignment objective.

Comments on Figures

We sincerely thank you for the detailed and thoughtful suggestions regarding the figures. We appreciate your comments on improving the clarity and layout, and we will incorporate all your suggestions in the final version.

Once again, we appreciate your close reading and valuable suggestion, and we hope this clarification resolves your concern.

Please read author rebuttal, thanks.

Dear Reviewer LN6q,

Thank you again for your thoughtful feedback and valuable comments on our paper. We just wanted to kindly follow up and check if our rebuttal and the additional experiments have sufficiently addressed your concerns.

If there are any remaining questions or clarifications we can provide, we’d be more than happy to assist.

We truly appreciate your time and consideration.

We sincerely thank the reviewer for their thoughtful and constructive feedback. We are glad that the reviewer found our work to be significant, methodologically rigorous, and comprehensively evaluated. We appreciate the recognition of our core contributions, including the novel formulation of dynamic personalized alignment as a reinforcement learning problem, the design of our dual-reward RLPA framework, and the robustness of our empirical validation across multiple dimensions.

Below, we address the specific concerns and questions raised:

W1: Simulation-to-Reality Gap

We appreciate the reviewer’s insightful comment regarding the limitation of relying on a simulated user environment for both training and evaluation. We fully agree that, while simulation offers a necessary and controllable setup for developing our reinforcement learning framework, it cannot fully capture the complexity and diversity of real human behavior. We have acknowledged this limitation in Appendix A of the paper and emphasized the importance of bridging the gap between simulation and real-world interactions in future work.

To address this challenge, we plan to incorporate real user interactions in subsequent experiments to better assess the model’s performance in practical dialogue settings. We believe that combining both simulated and real user feedback will enable us to further refine the model and move closer to real-world deployment requirements. Once again, we thank the reviewer for highlighting this critical point and will continue to investigate effective simulation-to-reality transfer strategies in our future research.

W2: Scope of Personalization

Your observation aligns closely with our own thinking. From the outset of developing this work, we recognized the importance of long-term and cross-session personalization. RLPA, by design, focuses on dynamic adaptation within a single continuous dialogue session. More realistic deployment scenarios—such as multi-session interactions and persistent user profiles—indeed require additional engineering support, such as the integration of user-centered databases or memory modules.

That said, we believe RLPA is well-suited to work in tandem with such external systems. Specifically, RLPA can be viewed as a short-term memory mechanism that captures immediate dialogue dynamics, while external user profile repositories can serve as long-term memory—analogous to the relationship between cache and disk in computer systems. This complementary setup has the potential to enable truly adaptive and persistent personalization.

We thank the reviewer again for highlighting this important dimension. Your feedback is highly valuable and will help shape the next stage of our research.

Q1: On the Simulation-to-Reality Transfer

We fully agree that bridging the gap between simulation and real-world deployment is a critical direction for future research.

As described in Section 3 of the paper (Lines 122–151), our current approach uses constructed simulated user models for both training and evaluation. This design choice was made for several practical reasons:

• Controllability and reproducibility: Simulated users allow us to precisely configure user attributes and control the gradual revelation of user preferences throughout the dialogue. This setup is particularly important for building stable and informative reward signals (both profile and response rewards) during reinforcement learning.
• Data efficiency: Unlike human-in-the-loop settings, simulated interactions incur no annotation cost and enable the generation of large-scale, high-quality, and diverse dialogue traces. This supports more effective cold-start learning and long-range adaptation.

That said, we are fully aware of the simulation-to-reality gap, and we have explicitly acknowledged this in the limitations section (Lines 486–490):

In addressing the reviewer’s concern, we would like to highlight that user simulation itself is a promising and underexplored research area. Conceptually, simulated users can be viewed as role-playing agents whose behavior is governed by implicit identity traits, preferences, and personalities that unfold naturally through dialogue. We believe recent advances in role-playing LLMs can be effectively adapted to improve user simulation. Possible directions include:

• Designing user behavior policies through prompt engineering (as we do with step-wise preference revelation);
• Building structured user-profile-based dialogue datasets to train user models via supervised fine-tuning (SFT);
• Applying reinforcement learning to user models themselves, optimizing for behavioral consistency and human-likeness.

We believe that modeling user simulators as standalone research entities will not only improve the fidelity and generalizability of RL training environments, but also facilitate progress in user modeling, intent understanding, and personalized evaluation. Moving forward, we also plan to evaluate and extend RLPA in more open, mixed real-user settings.

Thank you again for your thoughtful and constructive feedback!

Q2: Handling Implicit and Ambiguous Signals

Thank you for highlighting this important challenge. We fully agree that the ability to handle implicit, ambiguous, or even sarcastic user expressions is crucial for real-world personalized dialogue systems.

1. Regarding the concern that our experiments focus mainly on explicit user preferences: while our main evaluation uses structured slot-value profiles (e.g., in the Vanilla and Extended ALOE settings), we intentionally designed our simulated users to reveal preferences gradually and indirectly (Section 3.1, Page 5). This setup encourages the model to infer user intent from weak or partial signals.

Importantly, our reward models do not rely on explicit user statements.

• Profile Reward is based on the accuracy of inferred slot values, rewarding correct inferences even from indirect or stylistically varied cues.
• Response Reward assesses alignment with the inferred profile, emphasizing high-level criteria such as stylistic consistency, goal alignment, and emotional resonance (Section 3.2–3.3, Page 6).

In Section 5.3, our long-term reasoning experiments show that RLPA can accumulate weak signals over multiple turns and improve profile inference progressively—demonstrating robustness to implicit cues.

2. That said, we acknowledge that sarcasm and pragmatic deviations (e.g., irony, humor) are not systematically covered in the current simulated environment. This is a limitation we note in the “Limitations and Future Work” section (Page 13). In future work, we plan to introduce richer pragmatic diversity—including sarcastic and ambiguous utterances—via human inputs or enhanced user simulators, to evaluate and improve RLPA’s capacity in handling such nuances.

We sincerely thank the reviewer for drawing attention to this valuable direction.

Once again, we thank the reviewer for their valuable comments, which have helped us better articulate the contributions and limitations of our work.

Please read author rebuttal, thanks.

We sincerely thank the reviewer for the thoughtful and constructive feedback. We are pleased that the reviewer found our motivation well-articulated, appreciated the comprehensive experimental evaluation, and acknowledged the effectiveness of our proposed reward functions in enhancing personalized response quality.

W1: Insufficient detail on user simulator and profile inference process

We thank the reviewer for raising this important point.

Regarding the user model, we would like to clarify that the design and behavior of the simulated user are thoroughly described in Section 3.1 “Simulated User Design” (Lines 129–150). We also validate these properties through human evaluations (Lines 147–150), which led us to select GPT-4o-mini as the final user model. Additional results and evaluations of the user model are provided in the Appendix H, and also further elaborated in our response to Weakness 4.

Regarding the assistant model, our method is RL-driven—we do not manually design heuristics for inferring user profiles. Instead, the model is encouraged to autonomously discover effective strategies for personalization through simple reward signals. That said, we appreciate the reviewer’s concern and have conducted further analysis of the trained dialogue transcripts.

Interestingly, we observed the following learning dynamics:

• In early training stages, assistant models with few RL updates tend to directly ask for user attributes, which often results in unnatural or impolite behavior from the user model’s perspective.
• After further training, the assistant adopts a more indirect and effective strategy, such as self-disclosure—proactively introducing related topics that implicitly reflect user preferences and observing the user’s response. This not only improves personalization performance (as measured by profile accuracy) but also maintains high response quality and conversational naturalness.

We have included qualitative examples and a brief discussion of this behavior in Appendix A.7 to further clarify how the assistant infers profiles in a realistic, adaptive manner.

W2: Connection between structured profiles and dialogue use unclear

We thank the reviewer for highlighting this crucial point. Below, we clarify two key aspects of our user profile construction and its integration into the dialogue system:

1. Profiles are not drawn from a fixed, finite set, but are dynamically constructed

Contrary to the assumption that our structured profiles are predefined or static, they are in fact dynamically derived from re-annotated open-domain dialogue datasets, namely ALOE and PersonaChat. This is evidenced in the paper in Appendix D.1 (Lines 522–526) and Appendix D.2 (Lines 542–546).

Together, the training and testing sets include 3,821 + 148 profiles, none of which come from a predefined attribute list. The profile schema is thus open and extensible, and there is no reliance on a fixed or enumerated attribute space. This design allows for broader generalization and supports personalized modeling beyond constrained slot-filling paradigms.

2. Profile quality is controlled through a rigorous data construction pipeline

As described in Appendices D.1 and D.2, we employ the following multi-stage quality assurance process:

• Preprocessing: Natural language descriptions are converted to a unified slot format using GPT-4o-mini. The outputs are then manually filtered and deduplicated (Appendix D.1, Lines 533–540).
• Human verification: We randomly sampled 300 rewritten profile instances, which were judged by three annotators for semantic equivalence. The label agreement rate reached 90.7% between system annotations and human judgments (Appendix I.1, Table 7).
• Diversity control: In Extended ALOE (Appendix D.2, Line 545), we deliberately allowed unseen attribute keys to appear in testing, to evaluate the assistant model’s generalization ability beyond training slots.

These procedures ensure that the profile data is both accurate and diverse, while the open schema better reflects real-world personalization scenarios.

W3: Insufficient explanation of evaluation protocol and potential content reuse from ALOE

We sincerely thank the reviewer for pointing out this important issue.

1. On the evaluation protocol clarity

We apologize if parts of our evaluation setup were unclear. In fact, the following sections provide complete definitions and configurations:

• Section 4.1 (Lines 187–202) defines the evaluation metrics.

• Appendices D.1–D.4 (Lines 521–564) describe: Full specifications of the 7 baseline methods, including the prompt templates used, the construction and split criteria of Vanilla ALOE and Extended ALOE and Examples of profile slot formatting and system input/output structures.

We will improve the cross-referencing in the next version to make these details easier to locate.

2. On content reuse from ALOE

We would like to clarify that we have cited ALOE whenever introducing its datasets or evaluation metrics. For example, in Section 4.1 and Appendices D.1–D.4, we explicitly reference ALOE when describing the benchmark setup and metric definitions.

That said, we acknowledge that some explanatory texts describing ALOE may appear stylistically similar to our prior paper. This was not our intention, and we will revise these sections to avoid any perceived redundancy in future versions.

As noted, these reused components serve only as background and standard infrastructure, while the focus of this submission lies in the RLPA framework and its novel reward-driven personalization mechanism. We respectfully ask that the evaluation of originality remain centered on our proposed contributions and empirical findings.

W4: Experimental setup and its alignment with main claims

We sincerely thank the reviewer for this detailed and insightful feedback. We address each point as follows:

1. On the user simulator’s ability to reveal attributes over time

As the reviewer correctly noted, a key premise of our framework is that the user simulator incrementally discloses user attributes throughout the dialogue. We acknowledge that our original human evaluation only covered the first five turns, which was an oversight. To remedy this, we have now extended the evaluation to cover full 10-turn dialogues. The updated human evaluation results are shown below:

2. Clarification on Profile Reward (Equation 3)

We appreciate the reviewer’s insight regarding how limited attribute disclosure from the simulator may affect recall and thus suppress F1 scores. This design is in fact intentional.

By computing the Profile Reward as a cumulative F1 score over the ground-truth profile—even if some attributes remain undisclosed by the user—we incentivize the assistant model to proactively elicit more user information. This creates a form of implicit supervision, encouraging the assistant to maintain profile-seeking behavior over multiple turns.

To prevent the assistant from becoming too aggressive or unnatural in its questioning, we balance this objective with the Response Reward, which measures fluency, coherence, and user friendliness. Together, the two rewards are mutually complementary: the Profile Reward promotes meaningful personalization, while the Response Reward ensures conversational quality and user engagement. This joint optimization results in agents that can extract user profiles effectively and politely, as supported by our qualitative analysis.

3. On the omission of AutoPal as a related work and baseline

We sincerely thank the reviewer for highlighting this important omission. While we compared RLPA with a broad range of personalization baselines, we regret that AutoPal was not included in our literature review section. This was an oversight on our part, and we apologize.

Although AutoPal is not yet published or officially released, we have made our best effort to faithfully reproduce its method based on the available paper description and evaluated it under our experimental setup. The results are as follows:

• Vanilla ALOE

• Extended ALOE

These results further demonstrate that RLPA consistently outperforms AutoPal across both vanilla and extended evaluation settings, validating the advantage of our reward-guided personalization framework.

Minor issues

We thank the reviewer for pointing out the typo in Line 109. It has been corrected.

We are grateful again for the reviewer’s detailed comments, which have helped us significantly improve the clarity and completeness of our manuscript.