IGD: Token Decisiveness Modeling via Information Gain in LLMs for Personalized Recommendation

We sincerely thank the reviewer for their positive assessment and insightful feedback on our manuscript. We are encouraged that the reviewer found our work to be novel, well-motivated, and supported by extensive experiments (S1–S5). We appreciate the constructive suggestions, which have helped us improve the clarity and rigor of our paper. Below, we address each of the reviewer's concerns and questions in detail.

---

W1&Q1: Why not take a more fine-grained weighting?

A1: Thank you for the question. Our binary-based weighting design is motivated by two key observations about the behavior of Information Gain (IG) in our preliminary experiments:

1. Distinct Zero-IG Token Group: As shown in Figures 2 and 4, tokens with zero IG form a distinct and dominant group in the dataset (>55% of all tokens). These tokens consistently exhibit extremely low training loss and disproportionately high logits. Treating them as a separate class and applying a uniform down-weighting factor $\beta$ is a natural and effective way to handle their outsized influence.

2. Non-linear IG Distribution: As illustrated in Figure 4, the distribution of non-zero IG values is highly skewed and roughly exponential, not linear. This makes it difficult to design a meaningful continuous function over IG values that results in well-calibrated weights. We experimented with a simple linear and monotonic weighting function: $w_t = \beta + (1-\beta) \cdot \frac{\mathrm{IG}}{\mathrm{IG}_{\max}}$ However, as shown in the following tables, this linear function does not outperform our binary approach.

(IGD-T (Linear): best β=0.4, 0.9, 0.8, 0.4 for CDs, Games, Toys, Books)

(IGD-T: best β=0.2, 0.2, 0.5, 0.1 for CDs, Games, Toys, Books)

---

W2: Clarity of Figure 1 and Section 3

A2: Thanks for pointing this out. We will revise the figure to improve its clarity and update Section 3 accordingly.

---

W3 & Q2: What is the candidate item distribution? Is it determined by the frequency of item occurrences in the interaction dataset?

A3: Yes. The candidate item distribution, denoted as $P(\mathcal{I})$, represents the prior probability of any item $\mathcal{I}$, estimated by the frequency of item occurrences in the interaction dataset.

---

W4. Discussion of Concurrent Work

A4: Thanks for bringing this highly relevant concurrent work to our attention. While both our work and Gao et al. conceptualize item generation as a sequential decision-making process, our approaches differ fundamentally in their problem formulation, methodology, and goals:

• Core Idea and Methodology:
  • Flow-guided Tuning (Gao et al.): They introduce a "Flow-guided process reward" to supervise the entire sequence of token generation logits, rewarding paths that are more efficient at identifying the target item. Their contribution is a novel process-level supervision signal.
  • Our Work (IGD): We start from an information-theoretic perspective to identify a fundamental issue: a discrepancy between a token's generation likelihood and its actual "decisiveness" in narrowing down the candidate set. We propose Information Gain (IG) to quantify this decisiveness and identify two specific biases ("Tuning Bias" and "Decoding Bias"). Our IGD strategy is a targeted intervention that re-weights token losses during tuning and adjusts logits during decoding to mitigate these biases.

We hope this discussion clarifies the unique contributions of our work and its relationship to concurrent research. In the revised manuscript, we have added the discussion to the "Related Work".

We sincerely thank the reviewer for their thoughtful and constructive feedback. We are grateful for the positive assessment of our work's quality, clarity, and originality. The reviewer's insightful questions have prompted us to conduct further analyses that we believe have significantly strengthened our paper.

Below, we address each of the reviewer's points with detailed, quantitative evidence.

---

W1: This approach seems to involve more computation. How much training and decoding time it requires in practice?

A1: During training, the additional cost of our method comes primarily from looking up the IG score for each token (performed via a prefix trie) and applying multiplicative weighting. This overhead is relatively small. During decoding, the extra cost involves looking up IG scores and reweighting the logits of candidate tokens based on these scores. This process is also lightweight. We summarize the empirical comparison results in the table below. As shown, the additional cost introduced by our method is generally acceptable.

---

W2: Evaluation on datasets beyond Amazon domain.

A2: Thank you for the suggestion. We have conducted an additional experiment comparing the best-performing baseline (D3) with our IGD method (applied to D3) on a new dataset from Steam, which contains approximately 982K interactions. The results are summarized in the following table. As shown, our method continues to yield meaningful performance improvements. Notably, due to time constraints, we have not included all baselines in this comparison. We will include the full set of results in the revised version of the paper.

---

W3: On Decoding Diversity: As the IG weighting increases, it seems likely that the model would increasingly prioritize informative tokens. Is there any issue with reduced diversity or over-concentration on certain items?

A3: Thank you for this insightful question. Contrary to the potential concern, our proposed method, IGD-D, does not reduce diversity. In fact, our empirical results consistently show that as the weight on informative tokens (controlled by α) increases (in a wide range), the diversity of the recommended items also increases.

Empirical Verification:

We evaluated this effect using two diversity metrics across four standard datasets:

• item_score_entropy (higher is better; indicates more diverse recommendations)
• first_word_repetition_ratio (lower is better; indicates less repetition at the start of recommended items)

The results below demonstrate a clear trend across all datasets. As α increases from 0.0 to 0.4: item_score_entropy consistently rises and first_word_repetition_ratio consistently falls:

CDs

Toys

Video Games

Books

Analysis of the Mechanism:

• Root Cause: From our analysis (Figures 3 and 4), the model overly prioritizes non-informative tokens (such as "The" at the start of many item titles). These tokens tend to receive very high logits, leading beam search to preferentially expand paths containing them. As a result, the generated item sequences often share similar prefixes, which in turn reduces diversity and causes significant repetition.

• Role and Effect of IGD-D: Our IGD-D decoding strategy directly combats this issue by boosting the logits of informative tokens (i.e., setting α > 0). This adjustment encourages the model to prioritize more informative tokens during decoding, thus mitigating the inherent bias toward repetitive, non-informative outputs. Figure 6 ("Entropy difference: prediction vs. ground truth after IGD-decoding") has evaluated the effect from the item entropy perspective.

Dear Reviewer,

With less than 24 hours remaining for the author–reviewer discussion, we would greatly appreciate it if you could take a few minutes to read our rebuttal and consider adjusting your rating.

Thank you, Authors

Dear Reviewer,

Thank you for the follow-up. We are glad our response addressed your concerns. We appreciate your time and support for our work.

Best regards,

The Authors

We thank the reviewer for their insightful feedback and constructive suggestions. We appreciate that they found our method 'interesting' and 'intuitive', and the research problem 'important. To address their concerns about the method's scope, we have conducted several new experiments regarding hyperparameter complexity, dataset scale, and baseline comparisons. We believe these additions substantially strengthen the manuscript.

---

1. Can we naturally extend experiments to more generally language modeling tasks?

A1: We would like to clarify that our proposed IGD method is specifically tailored for LLM-based recommendation. Its computation of Information Gain (IG) depends on a predefined, finite set of items and their frequencies—an assumption that does not hold in general language modeling tasks. This limitation makes it challenging to define item-set entropy or derive meaningful IG values in open-domain settings. Consequently, our method is not directly applicable to general language modeling at this stage. Nevertheless, we are interested in exploring extensions to broader domains in future work.

---

2. Hyperparameter Complexity

A2: Though our method introduces two additional parameters, $\alpha$ and $\beta$, the parameter $\alpha$ is used only during decoding and is tuned over {0.1, 0.2} after selecting a suitable $\beta$, avoiding grid search. Moreover, as shown in our ablation results, even without using $\alpha$, our method still yields performance improvements. Therefore, tuning $\alpha$ can just incur minimal hyperparameter tuning cost.

The primary tuning cost comes from $\beta$, which influences the training process. While we acknowledge that tuning $\beta$ introduces some overhead, our sensitivity analysis indicates that this cost is modest in practice. Across various datasets, our method consistently outperforms baselines over a broad range of $\beta$ values (as shown in the following table). This suggests that identifying a suitable $\beta$ for achieving performance gains is relatively straightforward and does not require extensive tuning.

---

3. More dataset (dataset beyond Amazon) and more baselines (LRURec, TALLRec)

A3: Thanks you for the suggestion. We response the two points, respectively.

More dataset: Following your advice, we conducted an additional experiment comparing the best-performing baseline (D3) with our IGD method (applied to D3) on a new dataset from Steam, which contains approximately 982K interactions. The results are summarized in the following table. As shown, our method continues to deliver meaningful performance improvements.

More baselins: We have added two strong baselines: LRURec, the suggested traditional sequential recommender method, and Tiger, a recent generative recommendation method. We did not include TALLRec in the comparison, as it is not designed for the all-ranking setting used in our experiments. Due to time constraints, we conduct the comparison on the CDs dataset only. The results, presented in the following table, show that our method (applied to the D3 baseline) consistently outperforms these strong baselines.

(We will include the full tables for all datasets in the revised manuscript.)

---

4. Can the proposed method be applied to different decoding paradigms or generalized sampling settings with various logits processing strategies? Additionally, how sensitive is the method to decoding settings, such as temperature?

A4: First, we would like to clarify that our work focuses on recommendation tasks, where the model is required to generate a list of real-world items in each prediction. To achieve this, deterministic constrained beam search (do_sample=False) is commonly adopted, and we follow this setting as well. Under this setup, randomness introduced by sampling is eliminated. While in theory our method can be integrated with various decoding paradigms and sampling strategies, the specific requirements of recommendation systems make deterministic constrained beam search a natural and practical default choice in our framework.

Regarding the sensitivity of our method to decoding settings, we conducted additional experiments using stochastic sampling (do_sample=True) and varied the temperature parameter. We report representative results under nearly deterministic (T = 0.1), low (T = 0.7), and default (T = 1.0) temperature settings by comparing our IGD versions with or without applying the proposed decoding strategy. The results are summarized in the table below.

As shown, our proposed decoding method $w_d$ remains robust and continues to bring performance Improvements (compared to our IGD without using the proposed decoding strategy)— at lower temperatures (T ≤ 0.7).

Dear Reviewer,

We noticed that your rating for our paper is currently hidden, yet we have not received any comments from you. Could you please share your feedback to let us know whether our rebuttal has addressed your concerns?

Thank you, Authors

We thank the reviewer for their constructive feedback. As the questions and weaknesses are closely related, we summarize the concerns and address them collectively, rather than responding to each point in the two sections separately.

---

Q1. Unrealistic Experimental Setup

Q1.1: The current data split, where the test set is much smaller than the training set, may limit the effectiveness of the evaluation.

A1.1: We respectfully clarify that our data splitting strategy is designed to reflect common real-world recommendation scenarios. In many industry applications, models are trained on a large corpus of historical user interactions (the "training set") to predict user behavior over a shorter, subsequent time period (the "test set"). Our chronological 8:1:1 train-validation-test split directly follows this paradigm and is consistent with standard practice in existing works.

However, to further address your concern, we conducted an additional experiment by re-splitting the dataset into a 5:1:4 ratio for training, validation, and testing, with keeping the chronological order. This setup results in a test set comparable in size to the training set and introduces a significant temporal gap between training and testing periods. We compared our method, IGD-Tuning (IGD-T), with the strongest baseline (D3) under this new split. The results, summarized in the following table, show that IGD-T consistently and significantly outperforms D3. This demonstrates that our method (and the computed weights $w_t$) generalizes well and is not simply benefiting from a specific data split.

---

Q1.2. Data preprocessing introduces potential information leakage, as it allows the statistics in the training dataset can be leveraged to test the dataset

A1.2: This appears to be a misunderstanding. Our data preprocessing does not suffer from information leakage. On the contrary, we adopt a chronological data splitting strategy specifically to avoid information leakage from the future into the past.

The reviewer's concern seems primarily to stem from the use of 5-core filtering. First, we would like to clarify that this is not a filtering step introduced by any particular baseline, but rather a widely adopted standard practice in recommendation research. This approach is used in many well-known works in the field:

• "Hidden factors and hidden topics" (McAuley & Leskovec, RecSys 2013): Established the use of 5-core filtering on the now-standard Amazon dataset.
• "Image-based recommendations on styles and substitutes" (McAuley et al., SIGIR 2015): Continued the practice with 5-core filtering for visual recommendation.
• "Neural Graph Collaborative Filtering" (Wang et al., SIGIR 2019): Adopted a 10-core filter for modern graph-based models, confirming k-core is a standard procedure.

Second, we respectfully argue that applying 5-core filtering does not leak training data statistics into the test set. To illustrate this, consider the cold-start item scenario raised by the reviewer: the 5-core filtering is applied to the entire dataset before the chronological data split. As a result, items that appear for the first time during the test period are retained. Consequently, our test sets still include a substantial number of cold-start items—approximately 10% in the Books dataset and 8% in the Toys dataset—which naturally results in distributional drift from training to test. Additionally, significant drift can also be observed from the perspective of item frequency distribution.

Table: Frequency Draft - Top 10% frequent item in the training set and their Avg. Freq. Percentile in Test set.

---

Q2: The Design of the IGD Token Weighting Scheme: Why not use a continuous and monotonic function of IG instead of a binary weighting approach?

A2: Our binary-based weighting approach is not designed arbitrarily—it is motivated by two key observations about the behavior of Information Gain (IG) in our context:

1. Distinct Zero-IG Token Group: As shown in Figures 2 and 4, tokens with zero IG form a distinct and dominant group in the dataset (55% of all tokens). These tokens consistently exhibit extremely low training loss and disproportionately high logits. Treating them as a separate class and applying a uniform down-weighting factor $\beta$ is a natural and effective way to handle their outsized influence.

2. Non-linear IG Distribution: As illustrated in Figure 4, the distribution of non-zero IG values is highly skewed and roughly exponential, not linear. This makes it difficult to design a meaningful continuous function over IG values that results in well-calibrated weights. We experimented with a simple linear and monotonic weighting function: $w_t = \beta + (1-\beta) \cdot \frac{\mathrm{IG}}{\mathrm{IG}_{\max}}$ However, as shown in the following tables, this linear function does not outperform our binary approach.

(IGD-T (Linear): best β=0.4, 0.9, 0.8, 0.4 for CDs, Games, Toys, Books)

(IGD-T: best β=0.2, 0.2, 0.5, 0.1 for CDs, Games, Toys, Books)

---

Q3. Hyperparameter Sensitivity Analysis

A3: Our method introduces two additional parameters, $\alpha$ and $\beta$. The parameter $\alpha$ is used only during decoding and is tuned over {0.1, 0.2} after selecting a suitable $\beta$, thus incurring minimal hyperparameter-tuning cost.

Here, we mainly present a sensitivity analysis of the training-related hyperparameter $\beta$, while keeping $\alpha$ fixed. We evaluate $\beta$ over the range {0.1, 0.2, 0.3, 0.4, 0.5, 0.6}, and summarize the results in the following table.

As shown, our method is relatively less insensitive to the exact choice of $\beta$: it can achieves strong performance across a wide range of values in each dataset. However, the suitable range of $\beta$ may vary across datasets.

Dear Reviewer,

With less than 10 hours remaining in the reviewer–author discussion period, we sincerely request that you review our rebuttal and join the discussion. We have received feedback from all other reviewers, and they have confirmed that their concerns have been addressed. We believe your concerns have also been resolved.

Best regards, Authors