Hyperbolic Embeddings in Sequential Self-Attention for Improved Next-Item Recommendations

Thank you for taking the time to thoroughly assess our work and provide valuable feedback. Below we address the concerns related to the listed weakness points. We quote the original point and then provide the details that should address it. We will separate the points into several subsequent comments due to the character limit per single comment.

Point 1

There is no ablation study on the effects of different space curvature values. Varying the curvature and linking performance to estimated dataset hyperbolicity could provide better insights.

While we do not dedicate a separate section to such analysis, we do show the effects of varying the space curvature on the performance. It is depicted in Figure 4. The graph on the left side illustrates the relationship between curvature values and machine precision. It demonstrates that the estimated space curvature values exhibit a considerable range of variation across different machine precision settings. The right part of the graph links the performance of the hyperbolic model to the same machine precision settings. Combining both parts indicates the following trend: higher precision aligns with lower values of $c$ and subsequently yields improved recommendation quality. While this figure was meant to demonstrate the importance of accurately estimating the values of $c$ (which is one of our contributions), it also demonstrates that overestimating these values may degrade the performance and provides some insights into the overall performance of hyperbolic models on different datasets.

In this context, it is also important to note that some prior works, e.g., Khrulkov et al. (2020), did not use the estimated values of $c$ in the final experiments, as they found lower values to provide better results (see the last paragraph of Section 3.1 in their work). According to our calculations, using a better estimation procedure proposed in our work would provide a much better estimate of $c$ that would be much closer to the optimal one (which is an order of magnitude lower) used by the authors in their final experiments.

We appreciate the reviewer's dedicated efforts in evaluating our work and providing actionable suggestions and feedback. We will address the concerns by quoting them and expanding the explanation below the quote. Due to space constraints, our response will be split into several comments below.

Point 1

In my opinion, the contribution is marginal while the only change is the prediction layer (Eqn (6))

While we do understand this concern, we respectfully emphasize that the final results are easier to judge once they're obtained. However, even seemingly simple solutions do not necessarily have an easy path to them. In designing our approach, we experimented with several very different architectures, none of which resulted in any improvement, expect the one presented in the paper. The possibility to boost models performance be simply replacing the final layer with a linear classifier in a non-linear space provides a straightforward practical advantage. However, this is not the only contribution.

We also provide several insights on the conceptual side. We highlight the non-trivial aspects of understanding and modeling the geometry of data, which remain underexplored in the machine learning literature. We showcase the compatibility issue of data with the hyperbolic space geometry and propose a new view on the problem of estimating the curvature of the hyperbolic space. The latter may become helpful in early indication of possible compatibility issues, replacing the need for running expensive experiments. We also expose the non-trivial effects related to the negative sampling in the case of hyperbolic models, which leads to a significant degradation of quality even with respect to the baseline Euclidean model (SASRec). To the best of our knowledge, such effects remain unknown to the community. We believe it layouts a new direction of research that may spark new theoretical results and lead to better understanding of the limitations and applicability of geometric approaches to real-world problems.

Point 2

In Section 4.1, the authors mentioned that “we limit the allowed embedding size values to (32, 64, 128), while the Euclidean models explore higher values from (256, 512, 728)”. More deeper analyses would make the paper stronger

We understand that the choice of ranges of the embedding size may seem non-standard for the typical experiments in the Euclidean space. However, one of the main promises of the Hyperbolic space is its higher representation capacity, starting from the known theoretical fact that any tree can be embedded into the two-dimensional hyperbolic space (Poincaré disk). While this theoretical bound is hard to achieve in pratice due to hardware's finite arithmetic precision, one may still expect to see a representation capacity improvement (lower embedding sizes) in the hyperbolic models over their Euclidean counterparts. We, however, show in our work that it only happens if data is geometrically-compatible. The choice of the reduced embedding sizes helps demonstrating that effect. If we lift the restriction on the smaller embedding size, then even on "bad" datasets the hyperbolic models can be trivially made to perform at the same level as their Euclidean counterparts. It simply takes setting higher values of $\delta$ (lower curvature $c$ respectively) -- in this case the hyperbolic models become practically indistinguishable from the Euclidean ones. This result, however, would prevent us from demonstrating the true behavior of hyperbolic models on problematic datasets, while making the formal results (quality metrics) look much better in comparison. We believe that this result would be too trivial to show and instead focus on the discrepancies in learning abilities depending on the geometric properties of data. We believe that our approach offers a more equitable perspective, helping to more effectively unveil the underlying intricacies associated with training hyperbolic models.

We thank the reviewer for the thorough assessment and extensive feedback on our work. Below we address the outlined weakness points by quoting them and expanding the explanation below the quote. Due to space constraints, our response will be split into several comments below.

Point 1

Limited novelty. The work simply replaces the output layer of SASRec with hyperbolic prediction layers. Besides, the used hyperbolic prediction layers(hyperbolic hyperplane and MLR) have been widely applied in other works. The contribution of this article needs to be re-condensed.

We understand the concern and acknowledge that assessing novelty can be subjective. We believe it's important to note that the journey toward what might appear as a straightforward solution often involves intricate paths and complex problem-solving. While our final results may seem straightforward, achieving them involved rigorous exploration, refinement, and iterative processes that might not be immediately evident.

It's also important to highlight the contextual specificity and the absence of a universally applicable framework for employing hyperbolic models. The application of the hyperbolic geometry in different domains often demands tailored adaptations and nuanced implementations. What might appear as a widely utilized approach in certain studies may not seamlessly translate into another problem set due to the intricacies and contextual nuances inherent in each domain. For example, we conducted several experiments on text-related tasks with the same approach and no immediate improvement was obtained there.

Moreover, our contribution does not limit to the architecture design only. We provide additional experimentally-verified insights into hyperbolic geometry properties with respect to both models and data. We therefore are confident that the approach and methodology we applied in this task provide a nuanced perspective that adds value to the field.

Point 2

The paper measures the hyperbolicity of the dataset simply by δ-hyperbolicity but does not clarify the inherent hyperbolicity of the recommendation dataset. The utilization of hyperbolic space in this setting is questionable.

We totally agree with this viewpoint. Questioning the applicability of hyperbolic geometry is totally reasonable. Which is exactly why we focus a lot on providing additional context on the curvature estimation and compatibility of datasets. This is something that is largely absent in the majority of prior works. This void in understanding and analysis makes it harder to advance the field.

We are not sure what the reviewer meant by the "inherent hyperbolicity of the recommendation dataset". To clarify our approach, measuring Gromov's $\delta$-hyperbolicity is the commonly used standard for understanding geometry of data in practical applications. By definition, it measures the extent to which a manifold deviates from the Euclidean geometry in favor of hyperbolicity. When applied to the recommendation dataset, one can say that it is supposed to uncover the inherent hyperbolicity of data.

On a higher level, the question on whether the dataset is plausible for analysis via the hyperbolic geometry was answered in the seminal work by Krioukov et al. (2010). In that sense, we rely on the common understanding. We acknowledge though, that at least in our experiments the practice diverges from the theory. The emergence of "bad" examples may indicate a blind-spot in the conceptualization of the task via geometric lens or be an artifact of some undiscovered factors. One of our working hypotheses (yet to be tested) on the difference between "good" and "bad" datasets in terms of their hyperbolicity estimation is that the latter ones exhibit more heterogeneous structure. For example, the Movielens-1M dataset, on which the hyperbolic models achieve excellent results, is more homogeneous. This is a "pure" domain of movies, and the items are more likely to coherently gather onto a well-defined manifold. Conversely, the Amazon Arts Crafts and Sewing dataset seem to be more heterogenous with possibly many substructures, ingraining several different manifolds of different properties (e.g., different space curvatures), which prevents the models from learning good data representations. Verifying this hypothesis requires more elaborate analysis, which goes beyond the scope of the current work and presents an important direction for further research.

Point 6

The experimental results are not convincing. The comparison seems to focus on PureSVD-N and EASEr from 2019 (4 years ago). It is a bit misleading since many works have emerged in the past 4 years (see [1, 2] for example).

We partially addressed this concern in the answer to p.5. Our main comparison target is the SASRec extension with CE loss that shows state-of-the-art performance. The referenced by the reviewer MLP model [1] compares to both SASRec (with BCE) and BERT4Rec. As we already stated above, the CE version of SASRec outperforms both of these baselines, and our hyperbolic variant of CE-based SASRec improves the metrics further.

We acknowledge that comparing with MLP model can be interesting. However, the design of this model relies on pairwise ranking loss that also utilizes the negative sampling. As we previously discussed, this sampling may not play well with hyperbolic geometry and we'd argue that final comparison with such models should be performed after the comprehensive analysis of the root cause of the negative sampling impact on hyperbolic models. It links us back to p.3, which we also addressed.

The contrastive learning model from [2] compares only to the weaker BCE-based SASRec and no comparison is made with CE-based BERT4Rec. Overall, the relative improvement in this work looks weaker than the improvement of the full CE-based SASRec studied by Klenitskiy & Vasilev (2023). For example, on the Movielens-1M dataset in [2] one has:

• SASRec's HR@10 = 0.1902,
• CL3SRec's HR@10 = 0.1975,
• the relative improvement 3.8%.

The same dataset in Klenitskiy & Vasilev (2023):

• BCE SASRec HR@10 = 0.2500,
• CE SASRec HR@10 = 0.3152,
• the relative improvement 26%.

The experimental setup differs a bit in these works. However, the resulting difference in the absolute values of the metrics is not significant, while the relative improvement over the original BCE-based SASRec baseline is much more pronounced in Klenitskiy & Vasilev (2023). This allows us to conclude that [2] provides a weaker baseline than the CE-based SASRec.

Additionally, the same remark on negative sampling that we did w.r.t MLP model holds for [2] as well. We envision that CL3SRec won't gain much from the hyperbolic geometry due to the negative sampling utilized in part of its loss. The effects of switching to full CE loss in this case are more intricate, as CL3SRec uses a composite multi-task learning. Adopting hyperbolic geometry would require additional efforts to properly combine all parts of the loss. Performing such work would significantly extend our current approach and make it hard to fit into a single conference paper.

We are grateful to the reviewer for their careful consideration of our work and for the provided suggestions on possible improvements. Below we address the outlined concerns by quoting them and expanding the explanation below the quote. Due to space constraints, our response will be split into several comments below.

Point 1

My first comment is that the originality of this work is quite limited as other works proposed hyperbolic recommenders. The addition of this paper is the extension of SASRec with a hyperbolic layer as stated at the last sentence Section 5. I am not sure how much originality is there.

We recognize that at a superficial glance, incorporating a hyperbolic layer into the SASRec framework may appear as an incremental addition to the existing body of work. However, it is imperative to emphasize the intricate and meticulous process involved in this extension. Our approach involved a nuanced exploration of hyperbolic geometry properties with respect to integration within the sequential learning architectures. The final solution was selected among several competing one and required extensive experimentation. Moreover, our contribution does not limit to the architecture design only. We provide additional empirical insights into hyperbolic geometry properties with respect to both models and data. We therefore are confident that the approach and methodology we applied in this task provide a nuanced perspective that adds value to the field and highlights important issues that inform new directions for further research.

Point 2

As mentioned previously, it would be great if the authors could add some references from recent works of recommender systems on real use-cases just to contrast it with the current SOTA in real applications. Do we expect that such an approach would be viable or would give better online results in a real system?

We appreciate the reviewer's suggestion and recognize the value in contrasting our approach with recent works on real-world recommender systems. We note that our work builds upon the research by Klenitskiy and Vasilev (2023). According to the stated affiliations, the research is conducted within an AI Lab of a large company and provides insights into the ways of significantly improving an already strong sequential learning model. Consequently, our work suggests further improvements to the demonstrated state-of-the-art results. Our approach offers a better quality of recommendations with more compact representation at the same time. We believe, it therefore holds significant practical implications worth sharing with the community.

Our paper delineates the strengths and limitations of our proposed approach's applicability. Furthermore, we offer the complete codebase, enabling straightforward reproduction and adaptation of our solution to specific real-world applications. This comprehensive sharing empowers practitioners to evaluate the practicality of our method, considering its merits and shortcomings in their individual contexts.

It's crucial to underscore that the realm of recommender systems lacks a universal "silver bullet" solution applicable across all domains, and there's an overall agreement in the community that developing a silver bullet is not the end goal [1]. Assessing the viability of a particular model within a specific use case traverses subjective terrain, limiting a scientific approach and potentially harming the field's overall progress.

[1] Ferrari Dacrema, Maurizio, Simone Boglio, Paolo Cremonesi, and Dietmar Jannach. "A troubling analysis of reproducibility and progress in recommender systems research." ACM Transactions on Information Systems (TOIS) 39, no. 2 (2021): 1-49.

Point 5

In the experimental part I would add the following baselines: Most popular Co-occurence baseline, where one would recommend the property that co-occurs more often with the last item in the sequence. This is often a very strong baseline.

We thank the reviewer for the suggestion of additional baselines. In the case of the last-item-cooccurrence model, we agree that it may provide a reasonable quality. It would however add no additional understanding into the difference between Euclidean and hyperbolic sequential learning models, which is one of the main targets in our work.

We do have results for the popularity-based model, though, listed below. One can see that the performance is very poor, which is not surprising.

• Movielens-1M
  • NDCG@10  0.0214
  • HR@10    0.0427
  • MRR@10   0.0151
  • COV@10   0.0529
• Grocery and Gourmet Food
  • NDCG@10  0.0256
  • HR@10    0.0336
  • MRR@10   0.0230
  • COV@10   0.0005
• Office Products
  • HR@10    0.0074
  • MRR@10   0.0024
  • NDCG@10  0.0036
  • COV@10   0.0006
• Arts Crafts and Sewing
  • NDCG@10  0.0096
  • HR@10    0.0150
  • MRR@10   0.0080
  • COV@10   0.0007

We do not report these results to save space and avoid distracting the reader from the main effects.

Point 6

Adding challenging datasets also would be great. In recent years many datasets have been released that can be used for sequential recommendation.

We thank the reviewer for the additional references to rich datasets. This can be a great addition for the deeper analysis of the data compatibility, once the framework for such analysis is developed.

Point 7

Generally the results are no convincing/great. It seems in some cases there is some slight improvement (third digit). And the classification in good and bad datasets is arbitrary. What is the hypothesis that in one dataset of Amazon data the hyperbolic version is better and in another one (like Products) is not better? This seems random.

We respectfully disagree with the assessment regarding the improvement observed. Notably, the improvements were achieved utilizing smaller embedding sizes. Please, see Table 4 in the Appendix for more detailed breakdown. Our results showcase that even if the metrics were to remain at par without any explicit improvement over the baseline, our approach would still offer an advantage in terms of more compact representation while maintaining the state-of-the-art quality. This comparison underscores the practical advantage inherent in our approach.

In terms of the categorization of the datasets, we also recognize the importance of further in-depth analysis to understand what aspects render datasets more compatible with hyperbolic geometry. However, as we discussed in the response to p.2 and p.3 of the Reviewer 9PVb comments, there are several considerable challenges associated with the task. In the absence of better analysis tools that are yet to be invented, we provide an empirical observation and link it to the most visible differences in the $\delta$-hyperbolicity estimation. It is indeed not suitable for uncovering the deep underlying effects, but it is not random.

There seem to be no established theoretical framework for a deeper analysis of data compatibility. We acknowledge that the practice diverges from the theory in our experiments (a circumstance often encountered in general). The theory prescribes that hyperbolic geometry is suitable for any instance of complex networks family, which collaborative filtering datasets belongs to. The emergence of "bad" practical examples (for which the estimation of $\delta$-hyperbolicity is not reliable) may indicate a blind-spot in the conceptualization of the task via geometric lens or be an artifact of some undiscovered factors.

One of our working hypotheses (pending further validation) regarding the distinction between "good" and "bad" datasets in their hyperbolicity estimation is that the latter tends to display a more heterogeneous structure. For example, the Movielens-1M dataset, on which the hyperbolic models achieve excellent results, is more homogeneous. This is a "pure" domain of movies, and the items are more likely to coherently gather onto a well-defined manifold. Conversely, the Amazon Arts Crafts and Sewing dataset seem to be more heterogenous with possibly many substructures encompassing different types of products or consumption patterns, therefore ingraining several different manifolds of different properties (e.g., different space curvatures). This in turn may prevent hyperbolic models from learning better data representations. Verifying this hypothesis requires more elaborate analysis and presents an important direction for further research.

It is worth adding an additional remark on the quality improvement here, that we also made in response to p.2 of the Reviewer WdM2. We deliberately restrict hyperbolic models to the reduced embedding sizes. If we lift this restriction, even on "bad" datasets the hyperbolic models can be trivially made to perform at the same level as their Euclidean counterparts. It simply takes setting higher values of $\delta$ (lower curvature $c$ respectively). In this case the hyperbolic models become practically indistinguishable from the Euclidean ones. This result, however, would prevent us from demonstrating the true behavior of hyperbolic models on problematic datasets, while making the formal results (quality metrics) look much better in comparison. We believe that this result would be too trivial to show and instead focus on the discrepancies in learning abilities depending on the geometric properties of data. We believe that our approach offers a more equitable perspective, helping to more effectively unveil the underlying intricacies associated with training hyperbolic models.