Dear reviewer d6T5,

Thank you for your detailed, thorough and insightful feedback! We are encouraged by you recognizing our paper's strengths regarding (1) "one of the most comprehensive evaluations" across LLMs and baselines, (2) well-motivated problem setting addressing real-world LLM capabilities, and (3) "exceedingly extensive" experimental analysis.

We appreciate your constructive suggestions and have addressed your concerns with clarifications and additional details.

W1.1: "Terminology confusion around 'CoT'..providing examples to the LLM is generally considered to be few-shot learning (or, previously in-context or demonstration learning). "
We agree that our usage of certain terms could be confusing and have made the following revisions: we have renamed our "CoT" method to "few-shot CoT." In the original CoT paper, CoT prompting is defined as “augmenting each exemplar in few-shot prompting with a chain of thought for an associated answer”. Following this definition, we designed our few-shot CoT prompting, as detailed in Appendix A.4, where we provide manually constructed chains of thought (created using Claude 3.5) to guide the LLM in step-by-step reasoning to adhere to user preferences. Recognizing the potential overlap with few-shot prompting, we explicitly clarified the distinction by renaming it "few-shot CoT”.

We've also clarified that our "self-critic" approach is similar to prior work in intrinsic self-correction, where the LLM itself generates critiques to its own previous generation, citing relevant papers $1$ .

$1$ Large Language Models Cannot Self-Correct Reasoning Yet, Huang et al, ICLR 2024

W1.2: “The evaluation metric of the "generation task" is the accuracy in this paper. If I understand correctly, the generation task in NLP typically refers to translation, summarization, story completion, etc. (hence, metric could be BLEU/ROUGE/METEOR/BERTscore/etc.)”

While traditional NLP generation tasks often employ metrics like ROUGE when clear ground truth responses exist, our benchmark addresses open-ended queries (e.g., restaurant recommendations) where curating ground truth responses is infeasible due to the multiplicity of valid outputs. Using accuracy as our evaluation metric enables clear comparisons of model performance across turn numbers, models, and baselines, which is particularly valuable for understanding preference adherence in open-ended scenarios.

W1.2 cont’d: “LLM-as-a-judge could be done − and is also simpler and cheaper than LLM-as-a-judge − using the answer extraction in $2$ . To be more specific, we can prompt the LLM by asking "Does the response contain any $seafood$ cuisine?" in Figure 1, provided that a user stated that he/she doesn't like $seafood$ , then extract the Yes/No response afterward.).

Thank you for suggesting answer extraction as an evaluation method. We'd like to clarify our "violation preference?" checker in Table 1 already implements this type of binary evaluation. However, we found that relying solely on this violation checker was insufficient for comprehensive evaluation. Our framework includes three additional binary checkers because models can fail in ways beyond simple preference violations – they may provide unhelpful responses, refuse to answer, hallucinate preferences, or exhibit inconsistent behavior across turns. These failure modes are explicitly captured through our additional error types. Therefore, the preference-following accuracy we report provides a more comprehensive assessment of model performance in generation tasks.

W1.2 cont’d: “I personally would prefer to report accuracy in MCQ experiments, whereas LLM-as-a-judge is for error-type analysis only. However, If replacing the generation task performance with the classification task requires substantial revisions, then it's best to explicitly state something like "the accuracy in the generation task is highly correlated to the classification task" either in Figure 1 (on page 2) or in Section 2.5.

We agree with the reviewer's suggestion to explicitly state the correlation between generation and classification task performance. We have added this clarification in Figure 1.
However, we want to emphasize that there is no need to “replace” the generation task with the classification task in our paper. While both tasks show correlated performance, they serve distinct and complementary purposes. The generation task allows for open-ended question answering where models can provide detailed explanations and handle scenarios not limited by predefined options. The classification task, while more computationally efficient, has the inherent limitation that it cannot exhaustively enumerate all possible answer choices. We evaluated both tasks with the same amount of effort in terms of the same set of models, baselines, preference forms, and turn numbers for each. This allows users to choose between the tasks based on their computational resources.

Q7: “In Section 3.1, did authors experiment with placing few-shot examples on top instead of CoT?”

Yes, we experimented with placing reminder and CoT prompts in the system prompt (on top). However, performance was significantly worse in this configuration.

Q8: “In Table 2, is the number of data points 3000 x 5.6% (168) or 1000 x 5.6% (56)? Why is the travel-restaurant topic reported but not others?”

In Table 2, we only report results for explicit preferences, therefore it is 1000 x 5.6% (56). We focused on the travel-restaurant topic due its relevance to real-world dietary restrictions. The topic was not cherry-picked based on results. As conducting experiments on more topics under the 300 turn (~100k tokens) settings is costly due to our computation budget.

Q9: In Section 3.3, why does Claude struggle more with persona-driven dialogue compared to choice-based dialogue?

That’s a good observation. We hypothesize that Claude struggles more with preferences expressed through persona-driven dialogue because it gets distracted by the persona-driven conversation and ignores briefly mentioned preferences, compared to other models. In contrast, choice-based dialogues are less complicated for Claude to process, as they involve just two turns of conversation with a clear preference statement and it outperforms other models.

Q10: In Section 3.4, what is the "structure task"?

"Structured task" refers to the MCQ task, which is more constrained than the generation task, as it requires selecting one answer (e.g., a, b, c, or d).

Q11: In Section 3.6, Could the authors provide an example in "Impact of Multiple Preferences Stated in Conversation"? Does it mean that if a user says "I don't like seafood" (restaurant) in the first turn, then he/she also states "I don't like wearing jeans" (fashion), "I like to read" (music & books) in the following turns?

Yes, your listed example is correct. In the multiple preference experiment, we randomly sample preference from each topic, since we have 20 topics, we can maximally sample 20 preferences to insert within a conversation. We only sample one preference from each topic in this experiment to avoid conflicting preferences that are from each topic.

Similarly, could the authors provide an example in "Effect of Conflicting Preferences Stated in Conversation"? What is the number of data tested in Figures 8 and 9, respectively?

Yes, an actual example of a conflicting preference we used is: “I dislike any form of transportation that involves being on water due to motion sickness” versus “I love taking cruises and sailing trips because being on the water is so relaxing.” Note that one preference is from our dataset, while the other was generated by Claude 3.5 Sonnet to intentionally conflict with the first preference.

In Figure 8, each data point represents an average of 100 conversations. In Figure 9, the results are averaged over 5 topics, comprising approximately 250 data points. Additionally, since Figures 8 and 9 focus exclusively on Claude models, we have conducted further experiments to validate our findings on dynamic preferences. These new experiments include evaluations with Mistral 8x7b and Mistral 7b, and the results are included in Appendix A.16.

Q12: Rephrase to emphasize that the dataset has 1,000 unique user preferences with three levels of difficulty.

Yes, we have clarified this in Section 2.2 and also in our detailed data construction section in Appendix A.13.

Q13: Is the temperature set to 0 for all LLMs?

Yes, we use a temperature setting of 0.

Q14: Did the authors include an Ethics Statement?

We have added an Ethics Statement in the revised paper.

Typos.

Thanks for catching the typos! We will fix them in the revision.

Could try to fit GPT-4o in Section 3.1 since there are only nine models mentioned in this section.

Yes, we have additionally included GPT-4o's results in Table 4.

We greatly appreciate your feedback and believe that these new experiments and clarifications strengthen our paper. We hope this addresses your concerns and please let us know if you have any questions.

W2.1: Missing Data construction details: “...how to first generate the explicit user preference and get the distribution in Figure 2? Are the preferences mined from existing dataset(s) or generated by LLMs? After this, how is the corresponding query generated? Next, what are the criteria for generating the implicit choice-based/persona-driven dialogue? …”

Thank you for this important question. We have added a new section (Appendix A.13) that comprehensively describes our three-component data generation methodology. Here we give a brief summary: we first generated a set of explicit preferences, we started with 20 diverse topics, generated ~10,000 initial preference-question pairs using Claude 3 Sonnet, and implemented rigorous multi-stage filtering using both human labelers and LLM evaluators to select 1,000 high-quality pairs. For implicit preferences, we created two variants: choice-based preferences expressed through multiple-choice conversations, and persona-driven preferences embedded within 5-8 turn conversations using 100 diverse personas. Our quality control measures included validity assessment by human labelers, automated violation rate analysis using 5 different LLMs, and a systematic difficulty rating system using human-labeled examples as in-context examples. During the above construction and filtering process, we have spent large efforts in designing the prompts at each stage.

We will release all data generation prompts in our code repository for reproducibility.

W2.2: “problem formulation in Section 2.1, as it is not formalized in math. $…$ can the PrefEval dataset construction be formalized in a mathematical way?”

We agree that additional mathematical formulation would improve the clarity of our problem setup. We have revised Section 2.1 to include formal mathematical notation that defines the relationships between the key components of our multi-session framework: queries, preferences, turns, sessions, and conversations.

Q1: “In Figure 1, do authors conduct experiments where the user preference (blue box) is in between the multi-session conversations (green box)?”

For most experiments, we place the user preference at the beginning of the conversation and the query at the end. We also conduct experiments where preferences are inserted in the middle of the conversation to evaluate dynamic preference following, as discussed in Section 3.6. And we explored how the position of preference within the conversation affects the final performance in Appendix A.15.

Q2: “In explicit preference form, what is the corresponding "chatbot" utterance when a user states "I dislike eating seafood due to an allergy" in the first turn? Is it an empty string or a predefined sentence?”

The corresponding chatbot utterance is an actual response generated by each LLM, which typically acknowledges the user's preference. We will include additional examples in the appendix for clarity.

Q3: “In the implicit choice-based preference form, is the first user question the same as the query?”

No, the first user question differs from the final query. Repeating the same question would revealing the ground truth to the query so we used different query.

Q4: “When evaluating results, is the multi-session conversation (green box) also presented to the evaluator? If not, why is it excluded in accuracy reporting?”

The multi-session context is not presented to the evaluator. This is a deliberate design choice because our focus is specifically on assessing whether responses correctly follow and implement the user's stated preferences in a helpful manner. Including the multi-session context would not only add unnecessary computational overhead, especially in long-context settings, but would also be irrelevant to the core evaluation objective of preference-following accuracy.

Q5: “In the Introduction, does "prompting" mean "zero-shot, reminder, and CoT" in Section 3.1? Does "iterative feedback" mean "self-critic"? For RAG, is the retrieved context "utterance-based" or "sentence-based"?*”

Yes, "prompting" includes zero-shot, reminder, and few-shot CoT methods. "Iterative feedback" corresponds to self-critique. The retrieval context in RAG is exchange-based, with each turn containing two exchanges.

Q6: “In Section 2.5, is LLM-as-a-judge similar to prior work, and were the error types pre-defined or adapted from existing studies? In Table 1, why the value of (Preference Hallucination Violation, Hallucinate Preference?) is "No"? ”

Our LLM-as-a-judge framework is similar to prior work, but the four error types were newly defined in our study to capture critical failure modes. For Table 1, the "No" value for "Preference Hallucination Violation, Hallucinate Preference?" is a typo, we have corrected this, thanks for catching!

Thank you for your detailed response. Most of the concerns are addressed.

For most experiments, we place the user preference at the beginning of the conversation and the query at the end. We also conduct experiments where preferences are inserted in the middle of the conversation to evaluate dynamic preference following, as discussed in Section 3.6. And we explored how the position of preference within the conversation affects the final performance in Appendix A.15.

Are the experiment in Appendix A.15 explicit preference?

I would suggest mentioning the # data shown in each figure and table (especially when the data is not "large" enough, like Table 2) since sometimes it needs further re-reading to understand (or, guess) the actual number.

The multi-session context is not presented to the evaluator. This is a deliberate design choice because our focus is specifically on assessing whether responses correctly follow and implement the user's stated preferences in a helpful manner. Including the multi-session context would not only add unnecessary computational overhead, especially in long-context settings, but would also be irrelevant to the core evaluation objective of preference-following accuracy.

It would be even nicer, in the final version, if using "small" LLMs (like Mistral 7b or LLaMA 3 8b) to run this experiment and see how they handle this long-context scenario, which is more realistic regarding "lost in the middle" task. (As the discussion period is ending soon, there is no need to report the result this time.)

"Structured task" refers to the MCQ task, which is more constrained than the generation task, as it requires selecting one answer (e.g., a, b, c, or d).

I would recommend change it to classification (MCQ) task.

Q4: “When conflicting preferences are introduced, model performance improves compared to single preferences, which is counter-intuitive. Why does this phenomenon occur, and do other LLMs besides Claude 3.5 Sonnet exhibit similar behavior? (Section 3.6, Page 9). It is recommended that the authors expand their analysis to include more models beyond Claude 3.5 Sonnet. Additionally, they should provide potential explanations for these counter-intuitive results and discuss their implications for understanding LLM behavior in complex preference scenarios.”

To explore this further, we conducted additional experiments extending the analysis to Mistral 8x7b and Mistral 7b, in addition to Claude 3 Sonnet and Claude 3 Haiku, as presented in Appendix A.16 and also in the Table 3 below. Overall, three models (Claude 3 Sonnet, Claude 3 Haiku, and Mistral 7b) demonstrate improved performance with conflicting preference pairs. However, Mistral 8x7b exhibits slightly lower performance with conflicting pairs. This suggests the effect is model-dependent and our findings still holds that conflicting preferences do not necessarily harm performance.

Table 3: Effect of adding conflicting versus non-conflicting preferences on adherence. Results are averaged over five topics using a fixed 100-turn conversation for Claudes and 70-turn for Mistrals.

We hypothesize the increased performance with conflicting pair is due to a "topic-reinforcement effect." Even though the preferences conflict, they focus on the same topic domain, which potentially strengthens the LLM’s memory of the preference context. For instance, if a user states, “I prefer detailed responses for paper summarization” and later specifies, “I prefer concise responses for paper summarization,” the conflicting preferences both emphasize response length as a critical dimension. This repeated focus reinforces the LLM’s attention to that specific aspect, leading to improved accuracy in preference following despite the apparent contradictions.

We greatly appreciate your feedback and believe that these new experiments and clarifications strengthen our paper. We hope this addresses your concerns and please let us know if you have any questions.

Dear reviewer PRWG,

Thank you for your thoughtful feedback and for recognizing our paper's strengths in terms of (1) originality in benchmark creation, (2) thorough evaluation methodology with well-presented experimental design, and (3) significance of our findings for real-world applications. We appreciate your constructive suggestions and have addressed your concerns with additional details and analysis.

W1 and Q1: "The data construction process is not detailed. A major contribution of this paper is the manually constructed personalized preference dataset $...$ However, I did not find a section detailing the dataset construction process." “Regarding the detailed dataset construction process, Multiple methods should be employed to verify the dataset's high quality, which would make subsequent experiments more convincing. I recommend adding a dedicated section that outlines the data preparation process and quality control measures. This should include details on selection criteria, query development, and validation procedures used to ensure data quality.”

Thank you for this important question. We have added a new section (Appendix A.13) that comprehensively describes our three-component data generation methodology. Here we give a brief summary: for explicit preferences, we started with 20 diverse topics, generated ~10,000 initial preference-question pairs using Claude 3 Sonnet, and implemented rigorous multi-stage filtering using both human labelers and LLM evaluators to select 1,000 high-quality pairs. For implicit preferences, we created two variants: choice-based preferences expressed through multiple-choice conversations, and persona-driven preferences embedded within 5-8 turn conversations using 100 diverse personas. Our quality control measures included validity assessment by human labelers, automated violation rate analysis using 5 different LLMs, and a systematic difficulty rating system using human-labeled examples as in-context examples. During the above construction and filtering process, we have spent large efforts in designing the prompts at each stage. We will release all data generation prompts in our code repository for reproducibility.

W2: "Additional experiments fine-tuning for implicit preferences are often more challenging to identify and follow. Experiments in this area are recommended for a more comprehensive understanding of LLMs' preference recognition and adherence capabilities.."

Thank you for this feedback. We have added investigations on how fine-tuning impacts implicit preference following. As detailed in Appendix A.14, while our training data only included explicit preference examples, the fine-tuned Mistral 7B model showed significant improvements in handling implicit preferences. Across both travel and hotel domains, we observed substantial improvements in implicit preference following after fine-tuning, with accuracy gains of 40-60% for persona-driven preferences and 10-40% for choice-based preferences as shown in the table 1 below. These results suggest that training on explicit preferences enhances not only the model's attention to user preferences but also its general preference inference capabilities.

Table 1: Preference following accuracy (%) on implicit settings before and after supervised fine-tuning (SFT). We evaluate the model's ability to follow preferences in two implicit scenarios. We show the results across over 100 preferences instances over 2 topics in the zero-shot setting. We find preference finetuning brings more improvements for Implicit Persona-Driven preferences.

W3: Line numbers

We sincerely thank the reviewer for pointing this out and have added line numbers.

Q1: “A particularly irritating effect of LLM "personalization" / "memory" is that it assumes that some attribute (e.g. I like pizza) should now apply to all my queries, even if not relevant (e.g. high recall, low precision). Am I correct in assuming this is a "preference hallucination" violation?”

We agree that this behavior is undesirable. The scenario you described is more like "task hallucination", where irrelevant preferences are unnecessarily applied. In our framework, "preference hallucination" primarily refers to cases where the LLM, when prompted to follow a preference (e.g., using a reminder), incorrectly invents a completely non-existent preference. This “task hallucination” does not exist in our setup since we designed our tasks (e.g., the queries) to be relevant to the stated preference.

While the hallucination you mentioned falls outside the scope of our work, we recognize its importance in the broader context of personalization and consider it a valuable direction for future works.

Q2: “To validate our LLM-based evaluation method, we manually checked 200 randomly sampled evaluations, with an observed 5% error rate. How was this error rate computed? What was the distribution of errors in the 200 randomly sampled evals?”

The 200 samples were randomly selected, and the error rate was computed by examining preference-following accuracy alone. We acknowledge that this method is not fully comprehensive. In response to this concern, we significantly expanded our manual validation efforts by conducting a more thorough evaluation of our LLM-based evaluator. Specifically, we examined human-LLM agreement rates across four error-type binary checkers and three preference forms, as detailed in Appendix A.17. By sampling responses evenly from six models, five baselines, and all benchmarked turns, we manually validated 100 samples per preference form and calculated human-LLM agreement rates. For each response, we applied four error checkers, resulting in a total of 1,200 error checks. The results, presented in the table below, demonstrate strong alignment between human and LLM judgments, with particularly high agreement rates in detecting helpful responses and hallucinations.

Table: Human and LLM Judge agreement rates across different error checker as well as the final preference following accuracy.

Q3: “How were the implicit personas generated and validated? How much variance is there between the generated personas?”

We generated the personas using Claude 3.5 Sonnet, initially prompting it to create approximately 200 personas. These were then manually filtered down to 100 to ensure maximum diversity. During the generation process, we carefully ensured that no persona contradicted or explicitly revealed any preferences. We plan to release the prompts used to generate the personas used in our dataset for transparency. We have added detailed steps regarding persona generation in Appendix A.13, Step 3.

Q4: “With the additional space in the camera ready, I would move some examples of (p, q, explanation) pairs to the main text.”

Yes, we will add more demonstrating examples in the camera ready.

We greatly appreciate your feedback and believe that these new experiments and clarifications strengthen our paper. We hope this addresses your concerns and please let us know if you have any questions.

Dear reviewer L7qv,

Thank you for your thoughtful feedback and for recognizing the strengths of our work in terms of (1) timely and interesting topic coverage, (2) insightful error analysis, (3) well-motivated interventions, and (4) clear presentation.

We appreciate your constructive suggestions and have addressed your concerns with additional experiments and analysis.

W1: "I'd like to see some analysis of the 'Lost in the Middle' effect...and needle-in-a-haystack comparison.”

• To address this suggestion, we conducted experiments analyzing the impact of preference positioning on preference-following performance, which we believe the position of the preference will affect the retrieval of it. As detailed in Appendix 15 and Figure 23, we evaluated Claude 3 Sonnet and Claude 3 Haiku across four diverse topics by inserting the user preference at different points in a fixed 100-turn conversation. The results show a significant decline in preference-following accuracy when preferences are introduced in the middle of the conversation (around turn 50), compared to preferences placed at the beginning or end. These findings corroborate previous research on the challenges LLMs face with mid-context retrieval.
• Additionally, we revised the main text in Section 4 to explicitly compare our dataset to "needle-in-a-haystack" benchmarks. While those benchmarks primarily test retrieval capabilities from long contexts, PrefEval requires reasoning over preferences, particularly in implicit and persona-driven settings. This distinction underscores the unique challenges posed by our benchmark.

W2: “Second, the long-context integration feels a bit heavy-handed…Were there any measures taken to make sure that the bridge between the pre-existing context and the (p, q) pairs were natural?”

• We understand and agree with the concern about the unnaturalness of inserting preferences into the LMSys dataset when viewed as a single session multi-turn conversation. However , we would like to clarify our setting is multi-session, multi-turn conversations. We believe this also represents the real-world chatbot interactions from the perspective of users shift between different topics across multiple sessions..
• In reality, it is hard for users to maintain a single-topic conversation over long context like 100K tokens. Therefore, to assess long context preference following, we designed our setup as a multi-session conversation. Each conversation is divided into distinct sessions, with each session focusing on a specific topic across multiple turns (please refer to our formalized problem setup in Sec 2.1 of our revised paper). For instance, a conversation might begin with 10 turns discussing a debugging challenge, shift to 2 turns expressing dining preferences while seeking restaurant recommendations, and later move to travel planning that incorporates the preference.
• During preference insertion, we ensure topic transitions occur only after completing a turn. For example, each preference is inserted as a complete session, where the session contains the same number of exchanges from the user and LLM. This makes preference revelation a natural part of the multi-session structure. By modeling dynamic, multi-session interactions, we capture how users naturally engage in diverse, topic-spanning conversations. This setup enables testing of the LLM's ability to maintain and apply user preferences across varied contexts.

W3: “the (p, q) pairs are entirely synthetically generated. This is an unfortunate limitation: a personalization dataset generated with no "real" people … I'd recommend mentioning this in the limitations section.”

We acknowledge that synthetic data cannot fully capture the nuances of preferences expressed by real users, and we have incorporated this point into the limitations section of the paper.

That said, while the (p, q) pairs in our dataset are synthetically generated, they underwent a rigorous selection process from an initial pool of approximately 10,000 generated preferences. This extensive filtering combined manual human evaluation with LLM-assisted ratings to ensure that the preferences and queries are as natural and realistic as possible. As detailed in Appendix A.13, we employed a comprehensive approach, including manual human filtering and leveraging human-provided in-context examples to rate each preference-query pair. This thorough curation process aims to mitigate the limitations of synthetic data.

Great work on the needle-in-the-haystack experiments! I looked over the other authors' concerns, and I'm happy with how you've addressed them, too.

Is there an anonymous dataset link I could look at so I can look at some examples myself? I think that would help! I'm gonna bump my score up since I'd like to see this paper accepted, but I'd still like to see the anonymous link regardless.

Finally, I'd recommend explicitly adding W2 in the final paper as a limitation---so despite your efforts, you might end up with awkward instantiations. But that's fine, since realistic conversations with LLMs also have a ton of topic switches.

Dear reviewer Uqsa,

Thank you for your constructive feedback! We appreciate that you recognize our work's strengths in terms of (1) valuable dataset contribution, (2) presenting insights across major models with various strategies for improving preference following and actionable insights from error analysis, and (3) addressing the important yet underrepresented topic of LLM personalization.

We have conducted extensive additional analysis and experiments to address your concerns. Below, we respond to your specific concerns:

W1: "subjective LLM-based evaluation lacks qualitative and manual validations despite very small manual checks on the responses with 5% error rate."

• We acknowledge this concern and have significantly expanded our human evaluation efforts beyond just preference-following accuracy. We conducted comprehensive manual validation of our LLM-based evaluator (Claude 3 Sonnet) by examining human-LLM agreement rates across four error type binary checkers and three preference forms. By sampling responses evenly from six models, five baselines, and all benchmarked turns, we manually validated 100 samples per preference form and calculated human-LLM agreement rates. For each response, we checked 4 error checkers, resulting in 1200 error checks. The results (shown in the table below and in Appendix A.17) reveal strong alignment between human and LLM judgments, with particularly high agreement rates in detecting helpful responses and hallucinations.

Table: Human and LLM Judge agreement rates across different error checker as well as the final preference following accuracy.

• To further address concerns about subjectivity, our benchmark also includes Multiple Choice Question tasks for each preference, which has a high correlation between performance with generation tasks (in Fig. 11). Our MCQ tasks provide an objective, deterministic evaluation metric that complements and validates our LLM-based evaluations, thereby strengthening the overall reliability of our benchmark.

W2: “Inserting preferences within the LMSys dataset may be unnatural. what strategies were used for maintain semantic coherence of the dataset while introducing preferences and queries.”

We understand and agree with the concern about the unnaturalness of inserting preferences into the LMSys dataset when viewed as a single session multi-turn conversation. However , we would like to clarify our setting is multi-session, multi-turn conversations. We believe this also represents the real-world chatbot interactions from the perspective of users shift between different topics across multiple sessions..

In reality, it is hard for users to maintain a single-topic conversation over long context like 100K tokens. Therefore, to assess long context preference following, we designed our setup as a multi-session conversation. Each conversation is divided into distinct sessions, with each session focusing on a specific topic across multiple turns (please refer to our formalized problem setup in Sec 2.1 of our revised paper). For instance, a conversation might begin with 10 turns discussing a debugging challenge, shift to 2 turns expressing dining preferences while seeking restaurant recommendations, and later move to travel planning that incorporates the preference.

During preference insertion, we ensure topic transitions occur only after completing a turn. For example, each preference is inserted as a complete session, where the session contains the same number of exchanges from the user and LLM. This makes preference revelation a natural part of the multi-session structure. By modelling dynamic, multi-session interactions, we capture how users naturally engage in diverse, topic-spanning conversations. This setup enables testing of the LLM's ability to maintain and apply user preferences across varied contexts.