Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward

Thank you for your comments and valuable feedback! We sincerely appreciate your recognition of the importance of the problem addressed in our work. We acknowledge your concerns about real-world applicability and address them below with additional experiments and clarifications.

Human evaluation

Thank you for noting the limitation of our work regarding the lack of real human judgment. Following your suggestions, we conducted a human evaluation study of both personalization and conversation quality on the Education Dialogue task. We invited 10 students to participate. Each participant was asked to evaluate dialogues generated by three models across 20 different settings (topic/user type). Evaluations were conducted through pairwise comparisons across two dimensions: personalization and conversation quality, resulting in a total of 200 pairwise comparisons per dimension.

An exact two-sided binomial test showed that human evaluators chose DiffLogAcc over the MTRLHF baseline in 152 / 200 pairwise comparisons on personalization (preference = 0.76, 95% CI [0.69, 0.82]), p < .001. Importantly, increase in personalization did not reduce conversation quality: DiffLogAcc was preferred in 90 / 200 quality judgements (preference = 0.45, 95% CI [0.38, 0.52]), and the same exact test found no significant difference from chance (p = .179). Below, we provide two tables containing the full results of the human study, which correspond to Table 2 and 3. We note that the human evaluation results are reasonably consistent with those obtained through AutoEval. We will include these results in the revised version.

Accuracy of user model and potential impact

Several reviewers raised concerns about whether the method is practical if the user model is not accurate. To address these, we have calculated the accuracy of the user model in both tasks to demonstrate that our approach works without a perfect user model, and will include these results and clarifications in the updated version of the paper. Specifically, we tested the accuracy of the user model on dialogs where human raters can easily identify the user type. While the user model in the Exercise task does get near-oracle performance as the user attributes are clearly distinguishable, in Education the Gemma 7B user model is only 90% accurate (see Figure 4).

We can also create user models by prompting off-the-shelf LLMs, leading to an easy and accessible approach. To demonstrate this, we conduct an additional experiment where we take the conversations we used for the previously described human eval, and prompt ChatGPT to classify them. GPT-4o-mini got 100% accuracy and GPT-3.5-turbo got 95% accuracy, showing that a sufficiently capable LLM with general world knowledge can already serve as an effective user model without extensive prompt engineering when user types are distinguishable by humans.

In real-world industry applications, modeling user preferences is an area of significant focus, so reasonably accurate user models will typically be available, allowing them to train user classifiers directly rather than relying solely on prompting, thereby ensuring higher output quality.

Realism of task and user types

Thank you for raising concerns regarding this. Our datasets are designed to validate the methodology in the absence of open-source benchmarks. While discrete user types are a simplification, we view them as a useful starting point. In real-world settings, domain-specific user type definitions and trained user models are typically available for industrial practitioners. Our framework readily extends to continuous representations (e.g., using the embedding similarity based on different business needs).

Regarding the suggestion to make the evaluation setup more realistic, we acknowledge the importance of broader evaluation. At present, we were unable to find existing benchmarks for assessing personalization beyond the Education Dialogue task, which motivated us to introduce Exercise Recommendation as an additional domain. That said, if the reviewer is aware of other suitable datasets or benchmarks, we would be grateful to incorporate them.

Compute usage

We provide some training details in Appendix A, and here we further supplement it by describing the hardware used and time required. For the training job, we used a slice of 32 Google TPU v4 chips. For the standalone user model (for API calls), we launched four replicas, each on a 4-chip Google TPU v4 device. On the Exercise task, it takes 18 hours for the baseline and 19 hours for CURIO models to train 20k steps. On the Education task, it takes 44 hours for the baseline and 53 hours for CURIO models to train 50k steps. The main computational cost in training the policy model comes from rollout generation during online multi-turn RL. CURIO adds some overhead from user model API calls, but this has limited impact on overall training time.

Number of conversation turns

Most LLM training uses single-turn supervision, whereas our work adopts a multi-turn framework to build cross-turn capabilities. Our main contribution lies in the reward design, which highlights the core difference between multi-turn and single-turn training. From an algorithmic perspective, our method does not fundamentally impose a limit on the number of dialogue turns. However, similar to most existing multi-turn RL works [1,2], we primarily focus on dialogues with fewer than 10 turns, partly because LLMs generally struggle to retain information beyond that range [3].

Moreover, the number of dialogue turns is not fixed in our experiments. Conversations can end early via a special [end of conversation] token. As noted in our results, non-PBRS intrinsic rewards often lead to prolonged conversations, which in turn degrades dialogue quality. This provides evidence to support the need for our proposed PBRS design.

Theoretical guarantees

While we do not formally prove that PBRS reduces sample complexity, Appendix C presents a simplified analogy framing our task as a combinatorial bandit problem, illustrating how PBRS aligns with the efficiency gap between semi-bandit and full-bandit feedback. This provides intuition for its practical benefit. That said, our findings are primarily empirical. Theoretical work on the efficiency of PBRS in RL is limited, with most focusing on optimality. Similarly, the theoretical understanding of RL-based LLM fine-tuning remains under-explored. Our experiments show that baseline and non-PBRS models often fail to learn optimal policies within the training budget.

Base model capability

For exercise, the models are SFTed with data teaching them to ask about all possible attributes about the user. The SFT data is created in a way so that the model only follows the format, but is not aware of whether the question is relevant or not. During RL training, we purposely restrict the number of turns to a very small number, to guarantee the model is actually learning about how to ask only relevant and useful questions. For education, however, the baseline SFT model doesn’t actively ask any questions. It gradually learned that asking questions would lead to higher rewards during the RL fine-tuning process. For this SFT model, we directly use the one given in [1].

RL optimization approach

We acknowledge your point regarding the RL optimization details. Here, we provide additional details and will incorporate this into the revised version. Our framework has two distinct reward signals—extrinsic and intrinsic—with the extrinsic reward only nonzero at the final turn. The KL divergence penalty term against a fixed reference model is included. The total reward combines these components as:

$r_t=\alpha_{\text{ext}} R_t^{\text{ext}}+\alpha_{\text{int}} R_t^{\text{int}}-\beta D_{\text{KL}}(\pi_\theta||\pi_{\text{ref}}).$

We apply GAE to propagate the end-of-conversation extrinsic reward into turn-level signals, requiring a standalone value model to estimate the state-value function $V(s_t)$. The propagated reward per turn is:

$\hat{r_t}=V(s_t)+\sum_{t'=t}^T(\gamma\lambda)^{t'-t}[r_{t'}+\gamma V(s_{t'+1})-V(s_{t'})].$

Policy optimization utilizes the REINFORCE algorithm, though our framework places no restrictions on the specific RL optimization algorithm used. Formally, our policy optimizes:

$\max_\theta E_{\pi_\theta} [\hat{r_t}].$

Other clarifications

…the system prompt is injected artificially to determine the agent's strategy.

To clarify, the system prompts for the policy model contain only general task descriptions and do not prescribe specific personalization strategies. Instead, explicitly helpful questioning behavior emerges through the RL training process.

…the policy needs to be trained for each domain.

CURIO models learn generalizable skills through RL, such as asking informative questions and understanding users. We also believe this approach can extend across domains, especially for broader traits like mood or personality (e.g., Big Five [4]), which we see as a promising direction for future work.

Limitation: scaling-up

Again, we acknowledge some of your concerns that we have not yet shown this for real users at scale. Our work is instead an important first step which, as you rightly noted, shows a novel idea relevant to an important problem, draws a connection to the theoretical literature, and demonstrates empirical evidence through model training at the scale of LLMs.

[1] Shani et al., arXiv:2405.14655

[2] Rahman et al., arXiv:2504.13203

[3] Zhao et al., arXiv:2502.09597

[4] Jiang et al., arXiv:2305.02547

Thank you for your insightful review. Could you please assess how well the authors' response addresses your questions and concerns? Any additional feedback you can provide will be valuable both for the authors and for the decision-making process.

Thank you for your valuable feedback. We sincerely appreciate your recognition that our work addresses a problem that is “especially useful in cold-start settings”, and “show clear gains in both personalization and generalization” of the proposed method. Below, we provide responses to the weaknesses and questions raised by the reviewer.

Human evaluation

However, all experiments are conducted with simulated users and automatic evaluation using LLMs (e.g., Gemini). While this is understandable at this stage, some early human evaluation would have strengthened the paper claims.

We thank the reviewer for noting it. Following your suggestions, we conducted a human evaluation study of both personalization and conversation quality on the Education Dialogue task. We invited 10 students to participate. Each participant was asked to evaluate dialogues generated by three models across 20 different settings (topic/user type). Evaluations were conducted through pairwise comparisons across two dimensions: personalization and conversation quality, resulting in a total of 200 pairwise comparisons per dimension.

An exact two-sided binomial test showed that human evaluators chose DiffLogAcc over the MTRLHF baseline in 152 / 200 pairwise comparisons on personalization (preference = 0.76, 95% CI [0.69, 0.82]), p < .001. Importantly, increase in personalization did not reduce conversation quality: DiffLogAcc was preferred in 90 / 200 quality judgements (preference = 0.45, 95% CI [0.38, 0.52]), and the same exact test found no significant difference from chance (p = .179). Below, we provide two tables containing the full results of the human study, which correspond to Table 2 and 3. We note that the human evaluation results are reasonably consistent with those obtained through AutoEval. We will include these results in the revised version.

Robustness of CURIO compared to RLHF

How robust is CURIO versus common issues of RLHF such as noisy and ambiguous feedbacks?

To better answer this question, we hope to first clarify the RL optimization process of the CURIO method. Note that we have two different reward signals—extrinsic reward and intrinsic reward, where the extrinsic reward is only nonzero at the final turn. We further introduce a KL divergence penalty term against the fixed reference model. We use multiple hyperparameters to balance different reward signals, and the total reward is given by

$r_t = \alpha_\text{ext} R_t^\text{ext} + \alpha_\text{int} R_t^\text{int} - \beta D_\text{KL}(\pi_\theta||\pi_\text{ref}).$

Following the prior work, we apply Generalized Advantage Estimation (GAE) to our total rewards, which aims to propagate end-of-conversation extrinsic reward to turn-level signals. In particular, we need a stand-alone value model to estimate the value function $V(s_t)$ for each turn. The propagated reward signals for each turn is then given by

$\hat{r_t}=V(s_t)+\sum_{t'=t}^T(\gamma\lambda)^{t'-t}[r_{t'}+\gamma V(s_{t'+1}) - V(s_{t'})].$

We then use a REINFORCE policy gradient training approach to optimize the policy model over these propagated rewards. However, our framework does not impose any restrictions on the choice of a specific RL optimization algorithm. Formally, the policy is optimizing the objective:

$\max_\theta E_{\pi_\theta} [\hat{r_t}].$

Returning to the reviewer’s question:

(1) Compared to standard multi-turn RLHF, CURIO relies on the same extrinsic reward model. Therefore, any impact that noisy or ambiguous feedback may have on the quality of the reward model would similarly affect the extrinsic reward used in our approach. In other words, their levels of robustness are comparable.

(2) For the additional intrinsic reward introduced by CURIO, noisy or ambiguous feedback may manifest as the user not clearly expressing their preferences during the conversation. However, since we use a user model to infer user types over the course of multiple turns, as long as the user exhibits identifiable behaviors in the majority of turns, the user model can make effective inferences. This makes the intrinsic reward component relatively robust in practice.

(3) Finally, unlike traditional single-turn RLHF, the task we study involves online multi-turn RL training, which requires an (oracle) environment model to simulate the user and generate rollouts. The quality of this environment model significantly affects training outcomes for both standard multi-turn RLHF and CURIO. There is an increasing body of recent work focused on improving persona simulation quality, which we believe will help address this challenge in the future.

Thank you for your insightful review. Could you please assess how well the authors' response addresses your questions and concerns? Any additional feedback you can provide will be valuable both for the authors and for the decision-making process.

Thank you for your comments and valuable feedback. We sincerely appreciate your recognition that our proposed method is “interesting and well-motivated”, as well as the thoughtful and insightful questions raised. Below, we provide detailed responses to the weaknesses and questions.

Human evaluation

We fully understand your concern regarding the use of simulated models for evaluation. Following your suggestions, we conducted a human evaluation study of both personalization and conversation quality on the Education Dialogue task. We invited 10 students to participate. Each participant was asked to evaluate dialogues generated by three models across 20 different settings (topic/user type). Evaluations were conducted through pairwise comparisons across two dimensions: personalization and conversation quality, resulting in a total of 200 pairwise comparisons per dimension.

An exact two-sided binomial test showed that human evaluators chose DiffLogAcc over the MTRLHF baseline in 152 / 200 pairwise comparisons on personalization (preference = 0.76, 95% CI [0.69, 0.82]), p < .001. Importantly, increase in personalization did not reduce conversation quality: DiffLogAcc was preferred in 90 / 200 quality judgements (preference = 0.45, 95% CI [0.38, 0.52]), and the same exact test found no significant difference from chance (p = .179). Below, we provide two tables containing the full results of the human study, which correspond to Table 2 and 3. We note that the human evaluation results are reasonably consistent with those obtained through AutoEval. We will include these results in the revised version.

Accuracy of user model and potential impact

Several reviewers raised concerns about whether the method is practical if the user model is not accurate. To address these, we have calculated the accuracy of the user model in both tasks to demonstrate that our approach works without a perfect user model, and will include these results and clarifications in the updated version of the paper. Specifically, we tested the accuracy of the user model on dialogs where human raters can easily identify the user type. While the user model in the Exercise task does get near-oracle performance as the user attributes are clearly distinguishable, in Education the Gemma 7B user model is only 90% accurate (see Figure 4).

We can also create user models by prompting off-the-shelf LLMs, leading to an easy and accessible approach. To demonstrate this, we conduct an additional experiment where we take the conversations we used for the previously described human eval, and prompt ChatGPT to classify them. GPT-4o-mini got 100% accuracy and GPT-3.5-turbo got 95% accuracy, showing that a sufficiently capable LLM with general world knowledge can already serve as an effective user model without extensive prompt engineering when user types are distinguishable by humans.

In real-world industry applications, modeling user preferences is an area of significant focus, so reasonably accurate user models will typically be available, allowing them to train user classifiers directly rather than relying solely on prompting, thereby ensuring higher output quality.

Mitigating reward hacking

Thank you for highlighting our discussion of reward hacking. Indeed, the two types of reward hacking described in the paper reflect challenges that are generally present in RL fine-tuning, and are important to address. In our work, we find that both can be mitigated through the careful design of appropriate intrinsic rewards. Specifically, we have shown that using grounded intrinsic rewards can significantly reduce controlling behaviors, thereby enabling genuine personalization. Similarly, PBRS helps the model avoid excessively prolonged conversations, which in turn helps preserve conversation quality. KL divergence is also employed to prevent the policy model from deviating too far from the initial SFT checkpoint, which could otherwise result in low-quality outputs. At the same time, we would like to note that some instances of reward hacking may be partly caused by inconsistencies in the behavior of the environment model. Improving the quality of environment simulation could potentially help mitigate such issues.

Compute usage

We provide some training details in Appendix A, and here we further supplement this information by describing the hardware used for the experiments as well as the time required for different experimental setups, which we will include in the revised paper. For the training job, we used a slice of 32 Google TPU v4 chips. For the standalone user model (for API calls), we launched four replicas of the model, each on a 4-chip Google TPU v4 device. On the Exercise task, it takes 18 hours for the baseline and 19 hours for CURIO models to train 20k steps. On the Education task, it takes 44 hours for the baseline and 53 hours for CURIO models to train 50k steps.

In fact, the primary computational overhead in training the policy model lies in the rollout generation (of both policy and environment models) during online multi-turn RL. The additional resource consumption of the CURIO approach mainly comes from API calls to the user model, which does not significantly increase the overall training time.

Generalizability to other domains

We acknowledge that the experiments are only conducted on two particular domains, and would like to test how well this method performs for more diverse users across more complex tasks. However, we were not able to find extensive datasets of this kind for multi-turn conversation. We found that some multi-turn dialogue datasets use highly synthetic task settings (e.g., LMRL Gym), while others that rely on human annotation tend to be limited in scale or lack scenarios well-suited for defining personalization tasks (e.g., PRISM). If you are aware of specific datasets or benchmarks that would be appropriate for testing our work, we would appreciate any suggestions. We look forward to future developments in this area and the availability of such datasets, which would allow us to apply our methods to a broader range of domains.

Evolving User Traits

This is an excellent suggestion! We are excited about the idea of addressing non-static user types, and believe our approach holds strong potential for this use case as compared to existing methods. Since the user model evaluates the user type based on the conversation rollout, any changes in the user's behavior can be naturally reflected in the evolving probability distribution it predicts. To further enhance the model’s ability to handle potentially changing user types, we could let the user classifier only take in recent turns so that the policy agent is incentivized to ask about the user within a recent time window in order to maintain an accurate user model. While current datasets do not capture preference shifts across multi-turn dialogues, we see this as an exciting direction and hope to design a task and conduct an initial evaluation of this capability in future work.

Thank you for your comments and valuable feedback. We sincerely appreciate your support for our proposed method, as well as your suggestions for potential application scenarios. Below, we provide some updated results as well as responses to the weaknesses and questions raised by the reviewer.

Human evaluation

Following reviewers’ suggestions, we conducted a human evaluation on the Education Dialogue task. We invited 10 students to participate. Each participant was asked to evaluate dialogues generated by three models across 20 different settings (topic/user type). Evaluations were conducted through pairwise comparisons across two dimensions: personalization and conversation quality, resulting in a total of 200 pairwise comparisons per dimension.

An exact two-sided binomial test showed that human evaluators chose DiffLogAcc over the MTRLHF baseline in 152 / 200 pairwise comparisons on personalization (preference = 0.76, 95% CI [0.69, 0.82]), p < .001. Importantly, increase in personalization did not reduce conversation quality: DiffLogAcc was preferred in 90 / 200 quality judgements (preference = 0.45, 95% CI [0.38, 0.52]), and the same exact test found no significant difference from chance (p = .179). Below, we provide two tables containing the full results of the human study, which correspond to Table 2 and 3. We note that the human evaluation results are reasonably consistent with those obtained through AutoEval. We will include these results in the revised version.

Clarification on novelty

To the best of our knowledge, this is the first work to introduce user curiosity as an intrinsic reward for personalization in dialogue systems. The core novelty lies not just in applying the insight from traditional RL to LLMs (which itself poses engineering challenges), but in leveraging a user model to provide an intrinsic reward signal that guides an RL agent to learn to conduct conversations more effectively. That said, we would greatly appreciate it if the reviewer could point us to any related prior work (including those predating the use of LLMs), and we would be happy to include citations to those works and clarify any differences.

Theoretically, few existing works formulate personalization as a POMDP and apply intrinsic motivation to address it, particularly in the language domain. This setting is rarely explored due to the intractability of the environment’s transition dynamics and the difficulty of performing Bayesian belief updates in large action spaces. To make this tractable, we approximate belief updates using a user model (either trained or prompted) which serves as a practical bridge from traditional RL theory to implementation in dialogue systems. The implementation is also non-trivial, which involves 1) multi-turn online RL fine-tuning, which itself is complex and an emerging area of research, 2) incorporating an LLM as user simulator, and 3) another user model in the loop to compute intrinsic motivation.

Clarification on how PBRS is adapted

We frame the personalization task as a POMDP, where the user type is treated as an unobservable state. As a result, the agent must maintain a belief over the user type throughout the interaction. Our curiosity-based intrinsic reward is defined on top of this formulation. Specifically, we treat each utterance in the conversation as a timestep, allowing us to reward the agent for improving its belief about the user between consecutive turns after it has incorporated the user's latest response. Importantly, we have not imposed any restrictions on the specific form of the intrinsic reward function. However, we leverage the theoretical framework of PBRS, which provides guarantees of optimality preservation, to guide the design of our reward signal. The potential functions we choose are crafted to reach their extrema only when the model achieves a goal state—such as maximum prediction accuracy or minimum entropy—thus serving as an additional signal to encourage belief improvement.

Task details and example outputs (Appendix)

Details of datasets, code for Exercise data generation, prompts, example conversations are available in Appendix D, E, and F, which can be found in the supplementary material zip file. We hope that sharing these detailed insights can facilitate reproducibility and support future work in this area.

Different rewards for distinct aspects of personalization

Thank you for this insightful and inspiring suggestion. We completely agree that assigning distinct roles to extrinsic and intrinsic rewards—such as capturing stable user preferences versus situational factors—is a promising direction. In fact, our current setup already reflects this to some extent: the extrinsic reward model is designed to capture more general and stable aspects of personalization, such as conversation quality and overall engagement, while the intrinsic reward encourages the model to adapt to finer-grained user traits, including learning styles that vary across user groups.

Building on your suggestion, we are excited about the possibility of extending the intrinsic reward to model more dynamic and domain-general traits, such as the user’s current mood or learning attitude. We believe this could enhance personalization even further and look forward to exploring this direction in future work.

Thank you for your insightful review. Could you please assess how well the authors' response addresses your questions and concerns? Any additional feedback you can provide will be valuable both for the authors and for the decision-making process.