On Softmax Direct Preference Optimization for Recommendation

For Reviewer 6tmb

We appreciate your comments, some of which inspires us to greatly improve our paper. Below we provide the point-to-point responses to address your concerns and clarify the misunderstandings of our proposed method. If you have additional questions, we would be pleased to discuss them with you.

Comment 1: Uncertain contribution

We acknowledge your statement that original DPO has mentioned the utilization of preference models such as Bradley-Terry and Plackett-Luce to model strict ranking relationships among ranked data samples. We claim that S-DPO is a alternated version of DPO deriving a new preference model from the PL model, which is different from the original PL model and has a softmax structure. The the new preference model focuses on partial ranking relationships among preferred items and dispreferred items, which widely exist in recommendation data. Such inherent different in data characteristics makes it nontrivial to properly adapt DPO in recommendation while introducing multiple negatives which is important for recommenders. Besides adapting softmax DPO to recommendation data, we also provide theorectical analysis to connect pairwise DPO with BPR loss and connect S-DPO with softmax loss in recommendation such as InfoNCE, ensuring its superior performance. Also, we further analyze the gradient of S-DPO loss and theorectically prove that mining hard negatives is the reason behind S-DPO to provide effective gradient.

LM-based recommenders aim to directly generate the preferred items by first encoding textual prompt then autoregressively decoding, which is different from traditional recommenders which calculate similarity among user and item numerical representations. With such differences, the training losses between LM-based recommenders and conventional recommenders have connections but are strictly under different settings. So traditional loss functions like BPR and InfoNCE are unsuitable for LM-based recommenders.

For LM-based recommenders, introducing ranking information and multiple-negatives has been largely omitted. Besides, there lacks effective methods to introduce multiple negatives into the training pipeline of LM-based recommenders. To our knowledge, we are among the first to point out that multi-negatives is important in LM-based recommenders and propose to effectively instill partial ranking information into LM-based recommenders through an alternated softmax version of DPO.

Comment 2: Negtive sampling.

You are right that negative sampling is a critical area of exploration in recommender systems. In this work, negative samples were randomly selected. We acknowledge the importance of more sophisticated negative sampling strategies, such as popularity-based and similarity-based negative sampling, which we leave as future work. The type of data you mentioned, where unclicked results serve as negatives, is indeed common in real-world industrial datasets but is challenging to obtain in benchmark datasets like MovieLens. Within our experimental setting, random sampling serves as a simple yet effective strategy.

Comment 3: Gain of performance.

As line 215-217 mentioned, we optimize loss only on item title and find it effective in recommendation data. For other LM-based baselines, we adopt their official implementation for a fair comparison.

Comment 4: Saturation of negative samples.

We believe that the performance will further improve after more training iterations but as mentioned in limitation section that our computation resourse is limited, so we run all experiments for 3 epochs for fair comparson. We also believe that introducing more negatives may bring more gain but we can only explore part of it due to the same limitation in our computation resources.

Comment 5: Efficiency.

We appreciate your point. To address this, we have now included effectiveness and efficiency comparisons between DPO and S-DPO.

$K$ denote the number of negative items.

$C_\mathcal{M}$ denote the complexity of base model $\mathcal{M}$.

Time Complexity

The complexity of S-DPO scales with the factor $\frac{1}{2}+\frac{1}{2K}+\frac{1}{2C_\mathcal{M}}+\frac{1}{2KC_\mathcal{M}}$ compared to DPO. Since $C\_\mathcal{M}$ is usually large for LLMs, the bigger $K$ is, the smaller the factor is, which means the more efficiency S-DPO will posses compared with DPO.

Empirically, When both considering 3 negative examples and trained on goodreads dataset with 4 A100s, it will take 25h for S-DPO and 53h for DPO, indicating the efficiency of our method.

Additionally, we conducted experiments on three datasets compare DPO with multiple negatives and S-DPO on LLAMA2-7b.

LastFM

MovieLens

Goodreads

The results show that S-DPO still achieves leading or comparable performance with less complexity. This can be attributed to S-DPO considering more negative examples during single gradient update, resulting in more effective gradients compared to DPO.

For Reviewer AMu8

Thanks for your time and feedback. To address your concerns, we present the point-to-point responses as follows. Looking forward to more discussions with you.

Comment 1: Uncertain Novelty.

Generative LM-based recommenders has recently been explored which aim to directly generate the preferred items by first encoding textual prompt then autoregressively decoding, which is different from traditional recommenders which calculating similarity among user and item numerical representations. With such differences, the training losses between LM-based recommenders and conventional recommenders have connections but they are strictly under different settings.

For LM-based recommenders, introducing ranking information and multiple-negatives has been largely omitted. Besides, there lacks effective methods to introduce multiple negatives into the training pipeline of LM-based recommenders. To our knowledge, we are among the first to point out that multi-negatives is important in LM-based recommenders and propose to effectively instill partial ranking information into LM-based recommenders through an alternated softmax version of DPO.

Adapting DPO to recommendation is a non-trivial task. Conventional DPO utilizes the Bradley-Terry (BT) or the Plackett-Luce (PL) preference model and focus on absolute ranking relationships among data samples, while we also utilize the PL preference model but focus on partial ranking relationships specifically for recommendation data and derive a new preference distribution from the PL model. Moreover, we also provide theorectical analysis to connect pairwise DPO with BPR loss and connect S-DPO with softmax loss in recommendation such as InfoNCE. Also, we further analyze the gradient of S-DPO loss and theorectically prove that mining hard negatives is one of the reasons behind S-DPO to provide effective gradient.

Comment 2: Uncertain Complexity.

To address your concern, we have now included analysis of efficiency comparison between traditional DPO and our S-DPO, focusing on theoretical analysis and empirical results . This comparison highlights the efficiency gains achieved by S-DPO, supporting its practical value.

Let $K$ denote the number of negative items taken into consideration by S-DPO and $C_\mathcal{M}$ denote the time complexity of the chosen base model $\mathcal{M}$. The time complexity of computing S-DPO loss is $\Theta((K+1)C_\mathcal{M}+K+1)=\Theta((K+1)(C_\mathcal{M}+1))$.

For the DPO loss, the complexity of computing a single positive-negative pair is $\Theta(2C_\mathcal{M})$.

Besides, when an equal number of negative items are added to the DPO training set, the total number of the DPO training set is $K$ times of that of the S-DPO training set. Therefore, let $S_t$ denote the size of the S-DPO training set. The time complexity of training with DPO loss scale to $\Theta(2C_\mathcal{M}\times KS_t)=\Theta(2KC_\mathcal{M}S_t)$. Although this complexity is in the same magnitude as the time complexity of training with S-DPO loss, the latter scales the complexity of training with DPO loss with the factor $\frac{1}{2}+\frac{1}{2K}+\frac{1}{2C_\mathcal{M}}+\frac{1}{2KC_\mathcal{M}}$. Since $C\_\mathcal{M}$ is usually large for LLMs, the bigger $K$ is, the smaller the factor is, which means the more efficiency S-DPO will posses compared with DPO.

Time Complexity

Comment 3: Large scale Dataset

We have used large scale benchmark dataset such as goodreads. For extremely large datasets, it might be a common issue for researchers having difficulties to implementation.

Comment 8: Unclear section

• Synonym. We will replace it with more positive synonym such as "ancillary benefit" in our manuscript.
• Empty section. Methodology starts at Section 3.1.
• Equation (1). We will modify the notation "$>_u$" to "$\succ_u$" to better distinguish the comparisons between items according to the preferences of user $u$ from the comparisons between scores or ranks.
• Lines 174-180. S-DPO treats gradients of different negative (dispreferred) items differently by assigning the gradient of each negative item with an extra term $\frac{1}{\sum_{e_d \in E_d}{\rm exp}(g(e^\prime_d,e_d,x_u))}$=$\frac{{\rm exp}(r_\theta(x_u,e_d))}{\sum_{e^\prime_d\in E_d}{\rm exp}(r_\theta(x_u,e^\prime_d))}$ . This term reflects the relative reward of each negative item compared with other negative items. We can categorize negative items into two categories: (1) Hard negative items , whose reward $r_\theta(x_u,e_d)$ is relatively high, making it more probable to be chosen by LM-based recommenders; (2) Easy negative items, whose reward $r_\theta(x_u,e_d)$ is relatively low, making it less likely to be output. For hard negative items, the extra weight term tends to be larger, leading more decline for likelihood.
• Section 3.2. "Given effectiveness of BPR and InfoNCE loss in recommendation, we argue that sampled-based loss which explicitly compares preferred and dispreferred items such as DPO and S-DPO is more suitable for training LM-based recommenders than only utilizing language modeling loss."
• Rest mistakes will be modified in the revised version, thanks for pointing out.

Comment 9: Complexity.

$K$ denote the number of negative items.

$C_\mathcal{M}$ denote the base model $\mathcal{M}$.

$S_t$ denote the size of the S-DPO training set.

Time Complexity

For Reviewer iyxD I would like to express my gratitude for your detailed review and the valuable feedback provided.

Comment 1: Uncertain Novelty.

We agree with you that introducing multiple negatives is important and has been widely explored in conventional recommenders, as we briefly discussed in section 3.2. However, as your discovery, LM-based recommenders which has different training paradigm compared with traditional recommenders has just been explored, on which multiple-negatives has largely omitted and lack effectively method to introduce into training pipeline. To our knowledge, we are among the first to point out that multi-negatives is important in LM-based recommenders and propose to effectively instill partial ranking information into LM-based recommenders through an alternated softmax version of DPO.

Comment 2: Confusing Statement

• Clarification of Equation 12: Equation 12 is also an equivalent variant of the softmax loss, which can be rewritten as: $E_{(u,i_p,I_d)}\left[{\rm log}\frac{{\rm exp}(f(u,i_p))}{{\rm exp}(f(u,i_p)) + \sum_{i_d \in \mathcal{I}_d}{\rm exp}(f(u,i_d))}\right]$

• We will modify line 225 to "indicating the significant roles of knowledge and reasoning ability in language models for recommendation tasks in semantically informative datasets."

• Untuned LM-based recommenders achieve suboptimal performance because of inadequate instruction-following capabilities (reflected by a low valid ratio and a low hit ratio) or lack of domain knowledge (reflected by a high valid ratio but a low hit ratio). ChatRec, with GPT-4 as its backend, falls into the second case. We will modify line 228 to avoid confusion.

Comment 3: Replicability.

In fact, we have uploaded all the code and prompt used in the anonymous link " https://anonymous.4open.science/r/S-DPO-C8E0" which has been attached in the abstract of submitted manuscripts. We will further include all of our prompts in the appendix.

Comment 4: Evaluation.

• We calculate further HitRatio and NDCG, please refer to supplementary material (Table 2) for results.

• Sorry for leading confusion, the results in Figure 2a is based on LLama2 instead of LLama, which will be modified in the manuscript.

Comment 5: Statistical Test.

We utilize one-sided t-test for all of our statistical test and find the p value less than 0.05 among all shown experiments. Clarification will be included in implementation details.

Comment 6: Comparison.

• It has been shown that 5 epochs is enough for model convergence. Follow your suggestions, we conduct SFT for further 3 epochs and results show that it bring no gain to the final performance.
• We are adding a dedicated hyperparameter table to provide transparency and demonstrate that each model was given an equal opportunity for optimization. Please refer to supplementary material (Table 1).
• Regarding Figure 2b and 2c: We further ensure that the validation setting for DPO and S-DPO are consistent, allowing for a direct and fair comparison. The revised results are now included in the supplementary material (Figure 1). We compare loss, relative likelihood and absolute likelihood to validate the effectiveness of S-DPO
• Regarding Figures 3b and 3c: A very low β overly prioritizes ranking information, compromising the model's ability to follow instructions, as seen in the decreased valid and hit ratios. Conversely, an excessively high β constrains the model too much by the reference model's standards, leading to lower performance (hit ratio@1). A slight increase in valid ratio with β values greater than 1 suggests better adherence to the reference model's constraints, supporting our interpretation.

Comment 7: Related work.

We agree with your suggestion and we will reorganize our revised version. The lack of in-depth discussion is due to space constraints. We will include a more detailed discussion in the appendix.

We sincerely appreciate your valuable comments. Your feedback on direct comparisons and complexity analysis is important for improving our method. We hope our responses have satisfactorily addressed most of your concerns. If this is the case, could we kindly ask you to consider adjusting your score? With the rebuttal period coming to a close, we are eager to discuss any further concerns you might have.

For Reviewer gGah

Your main suggestions about considering additional base models help us substantiate wide applicability of S-DPO.

Comment1: Lack comparison with traditional preference-based losses.

Thank you for your insightful question, which raises a profound and previously unexplored issue: whether traditional preference-based losses can be directly applied to LM-based recommenders.

Following your suggestion, we attempted to train LM-based recommenders using BPR loss and InfoNCE loss. However, due to the significant differences between losses designed for language models (e.g., SFT, DPO) and discriminative preference-based losses, and given the time constraints of the rebuttal period, we have not yet obtained results within a reasonable range. We will continue to adjust parameters and training methods over the next discussion week to provide a more reliable conclusion. We greatly appreciate your question, looking forward to more discussions with you.

Additionally, we want to emphasize that we have compared preference-based losses in traditional recommenders, such as GRU4Rec-BPR.

Comment 2: General applicability of S-DPO

Great point! Following your suggestion, we have extended our experiments to include more base models.

We selected language models with different architectures (LLAMA1-7b, Pythia-2.8b, Mistral-7b), on varying datasets (LastFM, MovieLens), and of different sizes to perform experiments. We compared untuned LMs, LMs with only SFT as the training loss, and LMs with DPO and S-DPO for further training. Due to computational resource and time limitations, we experimented with S-DPO using 3 and 8 negative samples on two datasets.

Results on LastFM

LLAMA1-7b

Pythia-2.8b

Mistral-7b

Results on MovieLens

LLAMA1-7b

Pythia-2.8b

Mistral-7b

The experimental results indicate that DPO generally enhances the performance of SFT across different language models, demonstrating the importance of preference information for recommendation tasks. By incorporating multiple negative examples, S-DPO can achieve further performance improvements on top of DPO. Moreover, as the number of negative examples increases, the model's performance also improves.

Comment 3: Comparison between DPO and S-DPO.

We appreciate your valuable comment. To address this, we have added effectiveness and efficiency comparisons between DPO and S-DPO.

$K$: the number of negative items.

$C_\mathcal{M}$: the complexity of base model $\mathcal{M}$.

Time Complexity

The complexity of S-DPO scales with the factor $\frac{1}{2}+\frac{1}{2K}+\frac{1}{2C_\mathcal{M}}+\frac{1}{2KC_\mathcal{M}}$ compared to DPO. Since $C_\mathcal{M}$ is usually large for LLMs, the bigger $K$ is, the smaller the factor is, which means more efficiency S-DPO will posses compared with DPO.

Empirically, When both considering 3 negative examples and trained on goodreads dataset with 4 A100s, it will take 25h for S-DPO and 53h for DPO, indicating the efficiency of our method.

Additionally, we conducted experiments on three datasets compare DPO with multiple negatives and S-DPO on LLAMA2-7b.

LastFM

MovieLens

Goodreads

The results show that S-DPO still achieves leading or comparable performance with less complexity. This can be attributed to S-DPO considering more negative examples during single gradient update, resulting in more effective gradients compared to DPO.

Limitation 1: Lack of discussion in the recommender system.

We are not sure we fully understand your point. We discussed the LM-based recommender and traditional recommender in related works.

Limitation 2: conducting more experiments and emphasizing S-DPO specifically addresses preferences in sequential recommendation

Thanks for your valuable comments! Following your suggestion, we added more experiments and will emphasize the important role of S-DPO in sequential recommendation.