Tree of Preferences for Diversified Recommendation

W1: Test and validation sets may suffer from the same biases, making it difficult to assess whether the model truly captures users’ latent interests.

We sincerely appreciate your insightful comment. We acknowledge that conventional offline datasets may carry biases in test and validation sets. However, online validation typically requires randomized interventions (e.g., A/B testing), which present practical challenges such as high costs and reproducibility issues.

To rigorously evaluate ToP-Rec under biased test assumptions, we follow established work in debiasing evaluation [1], which aims to conduct unbiased evaluation when offline datasets exhibit selection or exposure bias. We focus on relevance evaluation under these biased assumptions, using two strategies: Inverse Propensity Scoring Recall ($R_{IPS}$) [2] and ATOP [3]. $R_{IPS}$ weights observations using propensity scores, providing an unbiased recall estimator, while ATOP uses item set ranking, also proven to be unbiased for relevance. We report ToP-Rec's performance, compared with both the backbone recommender and comparable baselines. The results show that ToP-Rec retains significant advantages under the assumption that test and validation sets may contain inherent biases.

Since the calculation of category entropy depends solely on the distribution of categories in an item list and is not influenced by unobserved user-item interactions, our diversity evaluation remains unaffected by bias.

W2: The proposed method is highly heuristic in nature and heavily relies on a pretrained LLM without any task-specific fine-tuning, which may limit its adaptability and effectiveness.

Thank you for raising this important point, which inspired us to clarify Top-Rec's design.

Firstly, we would like to emphasize that the key contribution of Top-Rec lies in its general framework rather than the incidental use of LLMs. The tree of preference serves as a structured vehicle for uncovering underexplored user preferences. Our experiments demonstrate that Top-Rec is compatible with a variety of LLMs, including smaller-scale models, which maintain robust performance (Please refer to our response to Q1 from Reviewer K1gm and W1 from Reviewer F6jv). This highlights the framework’s advantages.

Secondly, we utilize pretrained LLMs because Top-Rec's fundamental objective is to harness their rich world knowledge for latent preference discovery, not as specialized solvers for complex tasks (e.g., mathematical reasoning or code generation). Since pretrained LLMs inherently satisfy this requirement, we find them well-suited to our design. Empirical evidence consistently confirms Top-Rec's effectiveness.

Following your insightful suggestion, we further evaluated Top-Rec with fine-tuning. Specifically, we fine-tuned the LLM for user preference inference, which is a key step in the Top-Rec process. Using Twitter training data, we constructed a fine-tuning dataset where the LLM is instructed to infer preferences from users' historical interactions and the constructed Tree-of-Preference. We applied LoRA to fine-tune the LLM used in the paper and compared its performance with the original Top-Rec. The results showed that the fine-tuned version did not lead to significant improvements, confirming that pretrained LLMs are sufficiently effective while offering greater flexibility for diverse scenarios.

W3: The paper does not provide sufficient detail on how the tree is built.

We sincerely appreciate your valuable comment. Regarding how nodes are generated, during the construction of the ToP, we prompt the LLM to generate a multi-level tree structure for user preferences. By instructing the LLM that each node represents a type of user preference and each edge represents a finer preference division, the hierarchical nodes are ultimately generated by the LLM. Please refer to our example prompt in Appendix A.2 and the actual ToP presented in our response to Reviewer K1gm, W2.

Below, we provide key architectural specifications of the tree of preference constructed on Twitter across different runs, including the number of leaf nodes, total nodes, depth, average items assigned per leaf, and average branching factor. To enhance reproducibility, we will revise the paper to include detailed quantitative information about the tree structure, along with a visualization of the ToP.

W4: Hyperparameter analysis does not include discussions related to the construction of the preference tree.

Thank you for reminding us about this issue. We conduct evaluations to assess how the construction of the tree of preference influences performance.

First, we examine how the number of sampled items provided to the LLM for ToP construction affects performance. As shown in the following table, we observe that Top-Rec achieves strong performance with a relatively small sampling ratio, which can be attributed to the fact that our k-means-based item sampling strategy helps maintain the representativeness of the samples.

Second, we evaluate the impact of the ToP structure by adjusting the prompt to modify the requirements for tree depth and branching number. In the two tables below, we report Top-Rec's performance on Twitter with varying tree depth and average branching factor across different runs. We find that shallow structures limit preference resolution, while excessive depth also restricts performance, potentially due to semantic fragmentation. Following a similar logic, the average branching factor exhibits a comparable trend, where relatively fewer or more branches lead to suboptimal performance. We apologize for not clearly articulating this connection in the paper and will include a dedicated discussion of these findings in the revised version.

W5: Availability of rich user profiles.

Thank you for your thoughtful comment. Top-Rec aligns with existing LLM-based recommendation methods [4,5] in terms of its reliance on basic user profile information, without requiring more detailed user data. This basic information is typically easy to obtain and leverage in industrial recommendation scenarios [6,7,8].

More importantly, following the approach in [4], we enhance user data by incorporating item information from the user’s historical interactions. These items serve as an additional, rich source of user information. We collect the textual features of items the user interacted with and provide them alongside the basic user profile information to the LLM for summarization. By focusing on user interests and filtering out noise, we obtain enhanced user textual data.

To evaluate the applicability of our method in scenarios where user profiles are missing, we conducted additional experiments. Specifically, we masked the user profiles and used only the LLM-generated summarization from the historically interacted items as user data. The table below presents the results on Twitter, showing that our method still outperforms strong baselines, demonstrating its robustness.

[1] Bias and Debias in Recommender System: A Survey and Future Directions

[2] Unbiased offline recommender evaluation for missing-not-at-random implicit feedback

[3] Training and Testing of Recommender Systems on Data Missing Not at Random

[4] LLMRec: Large Language Models with Graph Augmentation for Recommendation

[5] LLM4Rerank: LLM-based Auto-Reranking Framework for Recommendations

[6] Large Language Model as Universal Retriever in Industrial-Scale Recommender System

[7] Towards Open-World Recommendation with Knowledge Augmentation from Large Language Models

[8] On Practical Diversified Recommendation with Controllable Category Diversity Framework

W1: Quality of user and item textual attributes.

Thank you for your feedback. Following prior work [1], we enhance the quality of user and item textual data with the aid of LLMs. Specifically, we leverage an LLM to generate user summaries based on their profile and the textual attributes of previously interacted items, as well as to generate item summaries from their textual attributes. By utilizing the LLM, we aim to reduce noise, redundancy, and inconsistencies in the original text, leading to improved summaries that retain key semantic information. We will update the paper to include these refinement details.

W2: Analysis of failure cases and demonstration of ToP and user-item matches.

Thank you for this insightful suggestion. To address cases where diversity gains negatively impact relevance, we conducted a post-hoc failure analysis on Twitter. Our investigation reveals that in 2.1% of cases (45/2,118 users), diversity improvements incurred relevance costs, quantified as an average recall decrease of 17.2%, accompanied by entropy increases of 14.2%. This trade-off is primarily driven by two factors: First, we find that more than half of these cases arise from the user's sparse interaction histories or the presence of uninformative interacted items, which leads to preference misinference. Second, for users with broad but shallow interaction histories, the risks of exploration boundaries may cause diversification to outweigh relevance, leading to a mismatch with their actual preferences.

Due to text-only constraints, we provide a partial illustration of the actual ToP used in the Twitter dataset. We will include visualizations of the ToP structures and visual user-item matches in the manuscript to improve interpretability.

ROOT
├── Political Systems & Governance
│   ├── Governmental Actions and Decisions 
│   │   ├── Policy Decisions
│   │   │   ├── Healthcare Policy
│   │   │   │   ├── ......
│   │   │   ├── Tax & Environment
│   │   │    ......
│   │   ├── Public Statements
│   │       ├── ......
│   ├── Political Figures
│   │   ├── Prime Ministers and Presidents
│   │   │   ├── .....
│   │   ├── Cabinet Members
│   │   ......
│   ├── Political Events
│   │   ├── Parties and Gatherings
......
├── Cultural Arts & Entertainment
    ├── Movies & Television
    │   ├── Streaming Series
    │   ├── Theatrical Releases
    ...
    ├── Music 
    ....

W3: Evaluation metrics of diversity.

Thank you for raising this insightful question. As a widely used metric [2,3,4], category entropy captures the distributional spread of item categories within a recommendation list, which aligns directly with the goal of diversity in our work.

To provide a more comprehensive evaluation, we also implemented two additional diversity metrics: the Gini Index (GI) [5] and Intra-List Distance (ILD) [6]. The following table presents the corresponding results on Twitter, comparing Top-Rec with comparable baselines and the backbone recommender. The results demonstrate that our method consistently outperforms others, highlighting its effectiveness.

Q1: Performance with smaller LLMs and robustness with LLM inaccuracies.

We evaluate our method with three smaller LLMs, including Qwen-2.5-7B-Intstruct, Qwen-3-8B, and Llama-3-8B-Instruct. The following table reports their performance on Twitter, compared with two comparative baselines and the backbone recommender. The results show that even with small LLMs, ToP-Rec can still outperform baselines in most cases, further demonstrating its robustness.

The robustness of ToP-Rec can be attributed to the design of the tree of preference, which acts as a vehicle to help the LLM systematically reason through the user's rationale and uncover their preferences, thereby mitigating the potential impact of hallucinations or inaccuracies.

Q2: Resolution for contradictory preferences.

Thank you for this enlightening question. While a user's preferences may seem contradictory at first, we believe these apparent contradictions can often be rationally explained, provided the user’s choices are driven by reason rather than randomness. For example, a user interested in both budget travel and luxury goods might value practicality and exclusivity in different contexts.

To explore this further, we analyzed how ToP-Rec handles such contradictions in practice. Our findings show that ToP-Rec is generally capable of rationalizing these preferences. For instance, we observed that some users may purchase budget-friendly daily items while also buying higher-end goods. ToP-Rec interprets this as a preference for affordable consumables and premium, long-term items. During item selection, ToP-Rec considers the context of different items, seeks to provide recommendations that align with their preferences.

[1] LLMRec: Large Language Models with Graph Augmentation for Recommendation

[2] Dual-Process Graph Neural Network for Diversified Recommendation

[3] Controlling Diversity at Inference: Guiding Diffusion Recommender Models with Targeted Category Preferences

[4] DGCN: Diversified Recommendation with Graph Convolutional Networks

[5] Post Processing Recommender Systems for Diversity

[6] Avoiding monotony: improving the diversity of recommendation lists

W1 & Q1: Analysis for the ToP generation across different LLM models, prompt variations and initial item sampling.

We appreciate your insightful comment. To improve the stability of the ToP, we employ several designs, including using a k-means sampling strategy during the ToP construction to select a subset of items that is sufficiently representative of user preferences, thereby reducing the impact of randomness. Additionally, we incorporate a refinement mechanism that enhances the balance of the ToP through merge and split operations (detailed in Appendix A.2).

Following your suggestion, we further evaluate the stability of the ToP with different LLMs, different prompt strategies, and varying numbers of initial sampled items. First, we integrate ToP-Rec with LLMs with different parameter sizes and architectures, including Qwen2.5-7B-Instruct and Llama3-70B. We report the stability of the ToP in terms of the number of leaf nodes, depth, and final performance. The following table presents the evaluation on Twitter, in comparison with the backbone recommender and other comparative baselines. The results show that ToP is relatively stable across different LLMs, with the final performance consistently outperforming the baselines.

Next, we apply different prompt strategies, including a simpler version where we remove some specific task instructions, and a more complex version where we provide more detailed task instructions. Similarly, we report the evaluation in the following table. The results show that the more complex prompt leads to a slight increase in the number of leaf nodes in the ToP, while all versions of ToP-Rec outperform the baselines.

Finally, we examine the impact of varying numbers of initial sampled items. Similar to the previous analysis, we report the evaluation in the table below. We observe that ToP remains relatively stable, even with a lower sampling ratio.

W2 & Q2: Detailed quantification of offline computational costs and comparison to simpler active learning.

Thank you for your valuable comment. We’ve provided a detailed breakdown of the main steps in training Top-Rec that contribute to the overall computational costs. For tasks performed on our local device, we provide the computational cost in GPU hours based on a single NVIDIA RTX 3090. For the steps executed via the LLM API, we compute the equivalent computational cost expressed in GPU hours on a single NVIDIA A100. The computational cost analysis for Top-Rec on the Twitter dataset is summarized in the table below.

As suggested, we also compare our selection of influential users with standard active learning by adopting the widely used uncertainty sampling [1]. Specifically, user uncertainty is calculated based on their average item scores, and users with the highest uncertainty are selected, with all other procedures remaining unchanged. The results presented below demonstrate that our method of selecting influential users leads to better overall performance, while maintaining comparable time costs, facilitated by PyTorch’s automatic vectorized computation.

W3: Additional exploration of synthetic item selection.

Thank you for this insightful comment. To further evaluate the effectiveness of our design in synthetic data generation, we conduct a comparison with two more complex mechanisms. First, we adopt the item scoring strategy from [2], which utilizes the Clayton copula function to account for both diversity and relevance. Second, following [3], we utilize an LLM to select items that a user might be interested in from a further filtered item candidate pool. We keep all other settings consistent with the original ToP-Rec and report the performance comparison on Twitter below. We observe that our original design maintains strong performance even when compared to these more complex methods. The main reason is that user preferences can be effectively inferred through ToP, which enhances the quality of the candidate item set and ensures reliable performance in our synthetic data generation.

[1] A Survey on Active Learning: State-of-the-Art, Practical Challenges and Research Directions

[2] Relevance meets Diversity: A User-Centric Framework for Knowledge Exploration through Recommendations

[3] LLMRec: Large Language Models with Graph Augmentation for Recommendation

Q1: Test set contamination which would help Qwen "cheat".

Thank you for raising this valuable concern. Research on test set contamination in LLM-based recommender systems is indeed challenging. We address this by systematically analyzing risks at each stage of Top-Rec.

First, during the construction of the tree of preference and assignment of items to leaves, Top-Rec exclusively utilizes item-side textual information. This process is orthogonal to user-item interaction data in the test set. We believe test set contamination is highly unlikely at this stage.

Second, in the synthetic data generation phase, the LLM indirectly participates in user preference inference. While LLMs possess the capability to memorize the pretraining texts, this memorization ability is unlikely to extend to user-item relations, as such interactions are typically not stored as plain text. Consequently, the feasibility of the LLM cheating in preference inference by memorizing specific interactions remains low.

To quantitatively evaluate this risk, we adopted the LLM pretraining data detection method [1], which identifies whether arbitrary data points belong to the LLM’s training data. Specifically, to investigate potential leakage of test set user-item interaction, we constructed two text corpora (10,000 samples each) describing user-item relationships (e.g., "User A with profile B liked Tweet C with content D"). One corpus contained ground-truth relationships from the test set, while the other contained randomly paired false relationships serving as a control group. We applied a state-of-the-art detector MIN-K% [1] to detect the likelihood of these relationships belonging to Qwen2.5-32b-Instruct’s training data. Results from MIN-K% showed that the proportion of test set relationships identified as non-training data (99.7%) was nearly identical to that of randomly generated false relationships (99.5%), indicating minimal test set leakage risk. We further conducted significance testing on the perplexity distributions of both text groups, which reflect the model’s predictive confidence. We applied two statistical tests: the t-test and the Kolmogorov–Smirnov test. Both yielded p-values above 0.9, showing no significant statistical difference between the distributions of the perplexity scores. This confirms that Qwen’s perplexity for true and false interactions is statistically indistinguishable, providing additional evidence against test set contamination.

Q2: Is the tree of preferences a single, global tree or a per-user tree?

We sincerely apologize for any confusion caused. We would like to clarify that the tree of preference is constructed as a global, one-time structure shared across all users. This design is based on our understanding that, while individual preferences may vary, all users ultimately navigate the same underlying preference space defined by item characteristics. We have revised the paper to clarify this point.

Q3: Does the method require significant user-profile text or can you substitute text from items the user likes?

Thank you for raising this important point.

Firstly, the user profile refers to basic registration information, as shown in Figure 1 (e.g., age, bio). Similar to prior work [2,3], our method incorporates such a profile, but does not rely on a large amount of user-profile text. The profile is not the only source of user information. As you rightly pointed out, we also utilize the text attributes from items the user has interacted with, which provide valuable insights into their preferences. These interacted items serve as another crucial source of user information. Ultimately, we follow [2] to provide both the text attributes from user profile and the items they interacted with to the LLM. By focusing on the user's interests and filtering out noise, we obtain an enhanced user summary from the LLM as the final textual feature.

When the user profile is unavailable, Top-Rec can rely on the text attributes from the interacted items as a substitute to generate the user's textual feature via LLM. To evaluate performance in such scenarios, we masked the original user profile and used only summaries from the interacted items as users' text features, then evaluated Top-Rec on this dataset. The table below presents the results on Twitter, which show that our method still outperforms the comparative baselines.

Q4: How prompt-length limits interact with constraints on how many sample items can be supplied to the LLM and how long the item descriptions can be? And the utility of the method in other contexts.

Thank you for raising this important question. We acknowledge that such constraints warrant consideration, but they do not significantly impact the construction of the tree of preference. This is because ToP relies not on the absolute size of the entire item set, but on the semantic breadth of a sampled subset. Crucially, our approach focuses on identifying a representative subset that captures the underlying preference space of the full item collection.

To achieve this, we designed a k-means clustering-based sampling method (details in Appendix A.2). By clustering items and sampling proportionally from each cluster, we obtain a compact yet representative subset for LLM processing. As an empirical example, Qwen2.5 supports up to 131,072 tokens of context, and in the Twitter dataset, each item averages 27.6 tokens. This results in a maximum theoretical capacity of approximately 4749.0 items per prompt. In practice, our sampled subsets consist of around 650 items (13.7% of the capacity), demonstrating substantial operational headroom.

We also investigate the impact of the number of sampled items on Top-Rec's performance. As illustrated in the table below, we find that Top-Rec performs well even with a lower sampling ratio.

In other contexts, we expect the utility of ToP to remain robust against prompt length constraints. Our sampling method intrinsically mitigates such limitations, and the trend toward longer LLM contexts provides additional flexibility. In extreme cases, potential solutions include partitioned ToP construction based on auxiliary metadata (e.g., item categories).

Suggested rewrites and typos.

We greatly appreciate your valuable feedback. As you suggested, we have carefully reviewed the paper, addressed the imprecise writing, and corrected the typos. Additionally, we have enhanced the paper to ensure a clearer and more rigorous presentation.

[1] Detecting Pretraining Data from Large Language Models

[2] LLMRec: Large Language Models with Graph Augmentation for Recommendation

[3] LLM4Rerank: LLM-based Auto-Reranking Framework for Recommendations