Response to Reviewer m2Qo (1/2)

Dear Reviewer m2Qo,

Much thanks for your detailed comments. In the following, we provide responses to the concerns you have raised:

[C1] While the paper tests on a few models, the results might not hold for other models or architectures, limiting the generalizability of the method.

[Response] Thank you for the constructive suggestions. Our experimental setup, which follows previous works (SL, BSL, etc.), tests on three typical backbones including basic MF, graph-based LightGCN and SOTA XsimGCL. However, we understand your concerns about generalizability and have subsequently tested our loss on a broader range of recommendation models, different embedding sizes, and different neural layers:

• We conduct additional experiments on various model architectures, including neural-based NCF [a1], neighbor-based Simplex [a2], VAE-based Mult-VAE [a3], and causality-based PDA [a4].
• We conducted sensitivity analysis on embedding sizes.
• We test the performance with different layers in LightGCN.

The results are presented in the following tables in terms of NDCG@20 under IID setting. Our PSL consistently achieved the best performance across all tested scenarios. These results demonstrate the generalizability of PSL, aligning with our theoretical analyses -- the theoretical merits of PSL do not depend on any specific assumptions about model architecture, making it broadly applicable across diverse backbones.

Table A1. Recommendation performance (NDCG@20) with various backbones (NCF, SimpleX, Mult-VAE, PDA).

Table A2. Recommendation performance (NDCG@20) with varying embedding size $d$ in MF.

Table A3. Recommendation performance (NDCG@20) with varying number of layers $L$ in LightGCN.

[References]

[a1] Neural collaborative filtering, WWW'17

[a2] SimpleX: A simple and strong baseline for collaborative filtering, CIKM'21

[a3] Variational Autoencoders for Collaborative Filtering, WWW'18

[a4] Causal intervention for leveraging popularity bias in recommendation, SIGIR'21

Response to Reviewer m2Qo (2/2)

[C2] The method may introduce biases due to the specific choice of activation functions.

[Response] We appreciate your concern. We would like to clarify that PSL does not introduce biases. Although we offer three choices of activation functions, PSL consistently outperforms the compared methods with any choice, with few exceptions, as evidenced in Table 1, Table 2, Table A1, Figure 2, Figures C.1-C.4.

In practice, we recommend using PSL-relu, as it achieve superior performance in most cases. Moreover, the linear ReLU function is simpler than the others, making it potentially easier and more efficient to optimize.

[C3] The reliance on temperature scaling for robustness could lead to parameter sensitivity, requiring careful tuning that might not be practical in real-world scenarios.

[Response] Thank you for raising this concern. However, it is important to note that PSL does not introduce additional hyper-parameters compared to the classic SL, which also utilizes the temperature parameter. SL has been widely applied in practical recommender systems like Taobao and TikTok.

Besides, our empirical experiments on the temperature (presented in the following Table A4) indicate that PSL achieves strong performance requiring a narrow search range for the temperature of $[0.025, 0.05, 0.1, 0.15]$ across four datasets. More importantly, PSL consistently outperforms basic SL at any temperature within this range. This suggests that PSL could serve as a better DCG surrogate loss than SL in practice, without necessitating additional tuning efforts.

In practice, when facing scenarios with significant resource constraints that preclude extensive hyperparameter tuning, we recommend limiting the search for the temperature to only two values $\{0.05, 0.1\}$. At this setting, the performance of PSL may be slightly below the optimum but is still significantly better than BPR and SL. In more extreme scenarios, we may even opt to simply set $\tau=0.05$, as this value yields decent performance in the majority of cases. It is important to note that other advanced loss functions, such as BSL and InfoAdvNCE, are less effective in these scenarios because they require the tuning of more hyperparameters to maintain their performance.

Table A4. Recommendation performance (NDCG@20) with varying temperature $\tau$.

---

We sincerely appreciate your feedback and hope these additional experiments and analyses have addressed your concerns. Please feel free to reach out if you have any further questions or suggestions.

Response to Reviewer ufck (1/2)

Dear Reviewer ufck,

We greatly appreciate your acknowledgement of our contributions and your insightful comments. In what follows, we provide responses to the Weaknesses (W) and Questions (Q) you have raised:

[W1] The proposed PSL loss is derived from the pair-wise form of softmax loss; however, the role of the original softmax loss in the subsequent discussion is not thoroughly addressed, leading to a minor gap in the exposition.

[Response] Thank you for underlining this issue. We will offer more explanations in the next version.

Note that the original form (cf. Eq.(2.4)) is derived by Maximum Likelihood Estimation (MLE), where SL maximizes the probability of positive items:

$\mathcal{L}_{\text{SL}}(u) = -\sum _ {i\in\mathcal{P}_u}\log\left(\dfrac{\exp(f(u, i)/\tau)}{\sum _ {j\in\mathcal{I}}\exp(f(u, j)/\tau)}\right)$

Recent works (e.g., BSL) have rewritten SL into pointwise form for analyses, where positive and negative instances are treated separately:

$\mathcal{L}_{\text{SL}}(u) = -\sum _ {i\in\mathcal{P}_u}f(u,i)/\tau+\sum _ {i\in\mathcal{P}_u}\log\left(\sum _ {j\in\mathcal{I}}\exp(f(u,j)/\tau)\right)$

Although the original and pointwise forms are equivalent, improving SL from pointwise perspective may disrupt the pairwise nature -- i.e., the loss can not be expressed by a function of $d_{uij}=f(u,j)-f(u,i)$ . As discussed in Section 3, ranking metrics are pairwise -- it optimizes the relative order of instances rather than their absolute values. Once the pairwise nature is disrupted, the theoretical relation with ranking metrics is no longer maintained. That's why we derive PSL from pairwise form (cf. Eq.(3.1), Eq.(4.1)).

[W2] While the paper provides some justification for the chosen activation functions, the theoretical rationale for selecting ReLU, Tanh, and Atan over others is not fully developed. The impact of the chosen activation functions on the stability of the training process is not sufficiently discussed.

[Response] Thank you for the constructive comments. According to our Lemma 3, the selection of these three activations satisfies the condition $\delta(d_{uij}) \le \sigma(d_{uij}) \le \exp(d_{uij})$. Admittedly, while the selected activations may not be optimal, they are simple and are sufficiently better than the $\exp(\cdot)$ function adopted in SL, serving as a closer approximation to the ideal $\delta(\cdot)$ function. Our experiments further validate the effectiveness of the proposed PSL.

Activations can affect training stability, specifically impacting the gradient distribution (cf. Eq.(4.3)). In fact, PSL exhibits more moderate gradient distributions compared to SL (cf. Fig.1(b)), thereby enhancing training stability.

[W3] The paper mentions the use of grid search for hyperparameter optimization but does not provide a detailed analysis of the sensitivity of PSL to the temperature factor.

[Response] Thank you for the constructive suggestions. In Table A1, we have conducted sensitivity analysis of temperature $\tau$ on four IID datasets and MF backbone (metric: NDCG@20). Results show that PSL and SL exhibit similar temperature sensitivity, and PSL consistently outperforms SL under the same $\tau$.

Table A1. Sensitivity analysis of temperature $\tau$ (metric: NDCG@20).

[W4] There are a few typos in the paper...

[Response] Thanks for pointing them out. We have made corrections.

Response to Reviewer ufck (2/2)

[Q1] How does PSL perform under conditions of extreme data sparsity? Have the authors conducted experiments to evaluate the effectiveness of PSL in such scenarios, and what mitigation strategies, if any, are suggested?

[Response] Thank you for raising this concern. Note that PSL is theoretically superior to SL and it does not rely on any assumptions on dataset. Empirically, we have conducted extensive experiments on six datasets with varying sparsity (cf. Tab.B.1). The data density ranges from the highest in Amazon-Electronic (0.00208) to the lowest in Amazon-Book (0.00026), with a nearly tenfold difference. We observe that PSL consistently achieved SOTA performance with varying sparsity.

To mitigate the impact of extreme data sparsity, we believe leveraging some cold-start recommendation strategies could be useful like the work [a1, a2]. These strategies can be seamlessly integrated with PSL, requiring only the substitution of their loss functions (e.g., BPR, SL) with PSL.

[References]

[a1] Contrastive Learning for Cold-Start Recommendation, MM'21

[a2] A heterogeneous graph neural model for cold-start recommendation, SIGIR'21

[Q2] Generally, contrastive loss is sensitive to noise (false negative instances). Is it feasible to apply PSL to general contrastive learning?

[Response] Insightful suggestions! PSL has the potential to be adapted as a loss function for contrastive learning (CL). In the context of CL, for a sample $z \sim X$, where the probability distributions of positive and negative samples $z_i^+, z_j^-$ are $P_z^+, P_z^-$, respectively, PSL can be defined as:

$\mathcal{L}_{\mathrm{PSL}}(z) = \mathbb{E} _ {i \sim P_z^+} \left[\log\mathbb{E} _ {j \sim P_z^-}\left[\sigma(f(z, z_j^-)-f(z, z_j^+))^{1/\tau}\right]\right]$

When $\sigma = \exp$ , PSL degenerates into InfoNCE. We can expect PSL equipped with relu or tanh activation functions, could mitigate the noise sensitivity by avoiding gradient explosion (cf. Fig.1(b)). We plan to further explore this promising topic in future research.

[Q3] In the experiments, PSL-ReLU consistently achieves sound performance across all settings. Can authors provide any rudimentary insights or intuitive understandings for the selection of activation function for broader application?

[Response] In practical, we recommend using PSL-relu directly, as it achieve superior performance in most cases. Moreover, the ReLU function is simpler than the others, making it potentially easier and more efficient to optimize.

The reason for the effectiveness of PSL-relu could be explained as follows: as illustrated in Fig.1(a), on the negative half-axis, ReLU is closer to the ideal $\delta$ compared to the others. Note that $d_{uij} = f(u, j) - f(u, i)$ and in many cases the negative score $f(u, j)$ could be smaller than the positive score $f(u, i)$, indicating the $d_{uij}$ are more likely to be located at the negative half-axis. Therefore, PSL-relu acts as a de facto closer surrogate to DCG than others in practice, leading to better recommendation accuracy.

[Q4] The introduction of new activation functions in PSL may alter the training dynamics of the backbone models. Apart from performance metrics, are there differences in the objective function values or convergence rates between PSL and SL?

[Response] Thanks for your question. As stated in the Response to [W2], PSL can in fact enhance the training stability by moderating the gradient distribution. We have provided the loss curves of SL and PSL in Figure R1 (in Author Rebuttal), which show that the training process of PSL is stable and converges fast.

Response to Reviewer kuk6

Dear Reviewer kuk6,

We sincerely appreciate your recognition of our theoretical contributions and experimental setup. Below are our detailed responses to the Weaknesses (W) and Questions (Q) you have raised:

[W1] In the IID setting, PSL demonstrates only marginal improvement even when compared to the standard Softmax Loss (SL).

[Response] Thank you for raising this concern. Admittedly, the improvements of PSL over SL in IID setting are not particularly large, typically ranging from 1% to 3.5%. But we would like to highlight the following aspects:

• The improvements of PSL are consistent across multiply datasets, backbones and metrics. This consistency demonstrates the effectiveness and generalizability of PSL, while some existing advanced losses like AdvInfoNCE and LLPAUC may performs worse than SL in some cases.
• The improvements of PSL are significant -- we have conduct statistically tests with $p < 0.05$ to validate these improvements are statistically significant.
• PSL exhibits much stronger robustness than SL with the distribution shifts or noises. From Table 2 and Figure 2, we observe that PSL shows notable improvements over SL, sometimes reaching improvement levels of 5% and 10% in OOD and noisy settings, respectively. Given that distribution shifts and noises are common in practical RS, we anticipate that PSL could achieve impressive performance in practical.
• The modification required to implement PSL over SL is minimal and practical. PSL only necessitates a simple replacement of the activation function without introducing additional parameters or hyperparameters, making it highly practical for industrial applications. In contrast, losses like AdvInfoNCE and BSL require more parameters and hyperparameters, necessitating higher training costs and extensive tuning efforts.

Overall, we believe that PSL can serve as a superior alternative to SL in practical settings, given its greater effectiveness and robustness, while not incurring extra implementation complexity or hyperparameter tuning efforts.

[W2] The authors argue that SL is highly sensitive to noise, while PSL with an appropriate activation function can mitigate this sensitivity. I would expect that as the noise ratio increases, an activation function with a moderate weight distribution should perform better. In other words, PSL-tanh/atan shall be a better choice to SL/PSL-relu if the noise ratio is high. However, this is not the case according to Figures C.1-4, in which PSL-relu consistently outperforms the other methods.

[Response] Thanks for your insightful question. We have indeed observed that PSL-ReLU achieves slightly better performance than PSL-Tanh/Atan in high-noise scenarios. The reasons may stem from the following two aspects:

• The disparity in the weights of noisy data between PSL-ReLU and PSL-Tanh/Atan is not as pronounced as the disparity between PSL-ReLU and SL. The following Table A1 presents the weights ($\nabla_d \mathcal{L}$) of those extremely noisy instances where $d_{uij}=1$ across different losses. The absolute values between PSL-ReLU and PSL-Tanh/Atan are not so large as those between PSL-ReLU with SL.
• PSL-ReLU employs the widely-adopted linear activation function, ReLU, which has been shown to be more easily optimized and exhibits greater numerical stability than the non-linear Tanh/Atan activation functions.

Both of these aspects contribute to the results. it appears that the second aspect outweighs the first, and PSL-ReLU exhibits slightly better performance even in highly noisy settings.

Table A1. Weights ($\nabla_d \mathcal{L}$) of extremely noisy instances with $d_{uij}=1$ across different losses.

[Q1] Why does BSL only keep on par with SL in both IID and OOD settings? It appears that the connection to Distributional Robust Optimization (DRO) determines the performance in OOD settings, but it is not the case for BSL?

[Response] Thanks for raising this concern. BSL enhances SL by incorporating Distributionally Robust Optimization (DRO) on positive samples. However, this approach exhibits certain limitations:

• BSL adopts a pointwise rather than a pairwise perspective, treating positive and negative instances separately. Given that ranking metrics depend on pairwise comparisons (as discussed in Section 3), this approach disrupts the theoretical alignment with ranking metrics, potentially rendering BSL an unsuitable surrogate for the DCG loss.
• In typical RS, positive samples are typically sparse, with a user often having fewer than 15 positive items. BSL applies KL-divergence-based DRO on these positive instances. Consequently, the support of the perturbed distributions within DRO is confined to these limited positive items. This means the set of uncertainty distributions used in DRO lacks flexibility and may not adequately cover the test distribution, thus compromising the robustness.

Given these two drawbacks, BSL might outperform SL, but the improvement is not substantial. In fact, based on our experience on hyperparameter tuning, tuning the hyperparameters to align BSL closely with SL often yield optimal performance.

Response to Reviewer VFmg

Dear Reviewer VFmg,

We sincerely appreciate your recognition of our work. Your detailed comments are highly valued, and the questions you raised are both interesting and practical. Below are our detailed responses to the Weaknesses (W) and Questions (Q):

[W1] Overall, the softmax loss is utilized in training and the test stage, which means we can utilize the model's output as the probability. The probability can be used in downstream tasks, such as ECPM calculation in online advertisements. So, replace the operation related to ranking metrics (may harm the) practical use.

[Response] Thank you for the insightful comments. It is not straightforward to derive the probability from PSL, but we can construct it by leveraging classic Plackett-Luce ranking model [a1]. The Plackett-Luce ranking model establishes relationships between permutation probabilities with the prediction scores, and SL was firstly introduced in the recommendation community from this perspective [a2].

To elaborate, in the Plackett-Luce ranking model, the probability that an item occupies the top position (i.e., the probability that it is the item the user likes most) can be expressed as:

$P_s(i)=\dfrac{\phi\left(s_i\right)}{\sum_{j \in \mathcal I} \phi\left(s_j\right)}$

where $s_i$ denotes the Plackett-Luce score of an item $i$. Function $\phi(\cdot)$ can be any increasing and strictly positive function. In our PSL, we express $s_i=f(u,i)-b_{ref}$ and $\phi(\cdot)=\sigma(\cdot)^{1/\tau}$. Here a reference term $b_{ref}$ is introduced, which can be set as the prediction value $f(u,j)$ of a specific item $j$ (e.g., the top-$K$ th item, which serves as a threshold score for an item being considered positive). The introduction of $b_{ref}$ is important as PSL optimizes the relative magnitudes of $f(u,i)$ from a pairwise perspective, rather than their absolute values. Introducing a reference term can enhance the stability of the results.

Such Plackett-Luce model possesses some interesting properties:

• When we set $\phi(\cdot)=\exp(\cdot)$, the probability $P_s(i)$ degenerates to the Softmax form.
• The objective of PSL can be viewed as optimizing the log-likelihood of the Plackett-Luce model with specific reference terms. That is, for each positive user-item pair $(u,i)$, we have: $\mathcal{L} _ {\text{PSL}}(u,i) = \log\left(\sum _ {j \in \mathcal{I}} \sigma(d_{uij})^{1/\tau} \right) = -\log\left( \dfrac{\sigma{(f(u,i)-b_{ref}})^{1/\tau}} {\sum _ {j \in \mathcal{I}}{\sigma(f(u,j)-b_{ref})^{1/\tau}}} \right) = -\log P_s(i)$ , where we set $b_{ref}=f(u,i)$.

[Reference]

[a1] The analysis of permutations, Journal of the Royal Statistical Society Series C: Applied Statistics, 1975.

[a2] Learning to rank: from pairwise approach to listwise approach, ICML'07

[W2] The distinction between AUC, Recall, and other metrics should be paid more attention. The motivation behind PSL is prioritized to focus on the DCG metrics.

[Q1] How the PSL affects other metrics, for example, AUC is important in pointwise click scenarios.

[Response] Thank you for the constructive comments. This work prioritizes DCG metric due to its widespread application in evaluating model performance within industrial RS. Furthermore, optimizing DCG remains a challenging problem. We appreciate the reviewers' suggestions to emphasize other metrics. Consequently, we have conducted further experiments to assess the performance of our PSL model using other metrics, including Precision, MRR, and AUC, detailed in the subsequent Table A1, while Recall has been reported in the Table 1 in our manuscript. From these tables, we observe that PSL consistently outperforms other baselines across these varied metrics.

Table A1. Precision@20, AUC, and MRR@20 on IID datasets and MF backbone.

[W3] The experiment design is reasonable, but the analysis needs to be further enhanced.

[Response] Thank you for highlighting this aspect. We will augment our experiment section with more comprehensive analyses, including more detailed comparisons of baseline performances, more analyses across three distinct scenarios, and more analyses of the impact of noise on different losses, etc.

[Q2] In experiments, only results of NDCG@20 are given. Results on more cutoffs should be given in the recommendation.

[Response] Thank you for your constructive suggestions. Table A2 present results of our PSL in comparison with other baselines across various NDCG@$K$ metrics. These results consistently demonstrate that our PSL model outperforms the baselines, thereby validating its effectiveness and robustness.

Table A2. NDCG@$K$ on IID datasets and MF backbone, $K \in [5, 10, 20, 50]$.