Predicting User Behaviors with Scene via Dual Sequence Networks

W1 and W4:
There is a misunderstanding between our scene feature and multi-behavior feature, so we first show their differences and then introduce the novelty of our work.

The Difference: Scene features are defined as sub-interfaces within an app or website, created by designers to achieve specific functionalities. In Figure 1(a) of the paper, an e-shopping app has many sub-interfaces with each has different functionalities such as text search and image search. Text search enables users to use text query for buying products, while image search enables users to use image query for buying products. In contrast, behavior-type features (e.g., view, click, add to cart, pay) are generated based on user interactions. Modeling the scene and multiple behaviors are two different problems.** In our model and experiments, we only consider one-type behavior**, i.e., ''view'' in Outbrain and ''pay'' in our industrial datasets. As a result, it is not appropriate to compare DSPnet with multiple behavior sequential RS works.

The Novelty: Integrating the defined scene features into sequential behavior modeling constitutes an important and new problem setting derived from our industrial business. Currently, there are limited studies addressing this area because of inaccessible data. Existing scene-based sequential RS [16-19] employ different scene definitions from ours. In Figure 1(b), we have shown that these scene features significantly impact user engagements. So, we propose a new framework specifically designed to incorporate the scene feature in sequential behavior modeling. DSPnet consists of two main and original components: dual sequence learning with sequence feature enhancement and CCR loss, with both are designed to tackle important challenges. The first one is to effectively encode sequential dynamics and deliver these dynamics to scene and item side for predicting behaviors. The second one is to learn representation invariance and improve the model's robustness against skewness. Recognizing the research data gap in this practical area, we intend to release our collected datasets to the community.

W2:
We do not agree that our design presents a contradiction with our motivation. Our design is to better capture the scene-item correspondence in the context of random and noisy user behaviors. Specifically, user behaviors often contain noise and randomness at single behavior level, whereas behaviors at sequence level provide more stable user dynamics. In order to better capture user dynamics and mitigate temporal errors, we preserve the correspondence at sequence level in our dual sequence encoder. Additionally, DSPnet establishes the one-to-one correspondence by the joint prediction objective, where each scene-item pair is predicted simultaneously. By adjusting the time step $T$ of sequences, multiple (scene, item) pairs can be consistently predicted in the sequence. **Our results and theoretical analysis show the superiority of our design.

W3:
(1) We did experiments of SASRec and S3-Rec in the beginning, whose performance are not good. So we only compared to models with more advanced encoders and supervision signals. Also, we have added the results of CARCA in the following table. CARCA contains a cross attention module that easily leads to OOM on large-scale datasets.

(2) We originally considered using scene feature as attributes to perform as strong baselines, like the results of SceneCTC and SceneContraRec in Table 1 of our original paper. We further added the suggested baseline CARCA.

(3) and (4) The scene feature defined in this paper is fundamentally different from multi-behavior feature, representing two distinct problems. Our method and experiments use only one type behavior, so it is not appropriate to compare DSPnet with multi-behavior sequential RS.

Q1:
BERT4Rec adopts the Cloze task, which produces an output shaped [batch_size, sequence_length, num_candidate_items]. While others typically output [batch_size, num_candidate_items]. In our industrial datasets, there are over 7 million candidate items and sequences can be up to 100 items long. This makes BERT4Rec have OOM errors, even using A100 GPUs.

Q2:
In inference, our single-type behavior method predicts the next item a user may interact with from a candidate set and evaluates performance using Recall@k and NDCG@k metrics.

Q1:
Thanks for the response. DSPnet has both relations and differences with the listed Multi-Behavior Sequential RS (MBSRS) works.

The Relation: MBSRS is a significant challenge because it involves modeling both multi-behaviors and behavioral sequences, to consider the existing problems of Multi-Behavior RS and Single-Behavior Sequential RS [9]. In this work, we focus on the sub-problem, incorporating scene features into Single-Behavior Sequential RS.

Technical Difference: MBSRS generally use a one-to-one correspondence encoding between behavior types and items [3-8]. However, our research emphasizes that randomness and noise in user behaviors can introduce misalignment issue between scenes and items (an example is given in following response). To address this issue, we developed a dual sequence encoding approach for sequence-level correspondence in encoding. This approach simultaneously learns reliable user dynamics against misaligned scene-item pairs and delivers the dynamics to both scene and item sides for behavior prediction.

Result Difference: Our sequence-level correspondence encoding approach outperforms the existing one-to-one correspondence encoding style. Table 1 in our paper compares our method with popular Single-Behavior Sequential RS methods (e.g., CARCA, SceneCTC, SceneContraRec) that use the one-to-one correspondence encoding. Additionally, in the following table, we tested replacing our dual sequence encoding with the one-to-one encoding in existing works [3-8].

Table: Performance comparison of DSPnet-- and our DSPnet. DSPnet-- means we replace our dual sequence encoder with one-to-one correspondence encoding in recent multi-behavior sequential RS works.

We will add this discussion, results and the related MBSRS works in the revised version.

Q2:
The one-to-one correspondence of scenes and items within the sequence may face the misalignment problem. For example, in an e-shopping app, a user might see certain products in the "recommendation" interface with the intention to purchase them. However, instead of buying immediately, the user adds these products to the "cart" interface and buys from the "cart" interface later, because of upcoming sales promotions from sellers. In this case, the user's interest and intent are initially reflected in the "recommendation" interface but are incorrectly collected in the ''cart'' interface.

To this end, we establish correspondence at the sequence level, enabling the model to effectively address fine-grained misalignment issues. We also conducted an experiment in which we replaced our dual sequence encoder with a one-to-one correspondence encoder [3-8] that processes concatenated scene-item embeddings as a single input. The results of this experiment are presented in the table above. These findings demonstrate that our dual sequence-level encoder provides superior representation learning capabilities compared to the one-to-one correspondence encoder.

We attribute this superior performance to two factors:

1. The one-to-one correspondence encoder could be sensitive to misalignment errors, while our dual sequence-level encoder learn representations against this misalignment.
2. The one-to-one correspondence encoder struggles to effectively capture the mutual interactions between scene and item when predicting behaviors. In contrast, our dual sequence encoder incorporates sequence feature enhancement module to explicitly captures these complex relationships, learning better user dynamics for behavior prediction.

Reference
[1] Robust User Behavioral Sequence Representation via Multi-scale Stochastic Distribution Prediction, CIKM'2023
[2] Sequential Recommendation with Multiple Contrast Signals, TOIS'2023
[3] Personalized behavior-aware transformer for multi-behavior sequential recommendation, MM 2023.
[4] Multi-behavior sequential recommendation with temporal graph transformer, TKDE 2022.
[5] Multi-behavior hypergraph-enhanced transformer for sequential recommendation, KDD 2022
[6] Multi-behavior sequential transformer recommender, SIGIR 2022.
[7] Knowledge enhancement for contrastive multi-behavior recommendation, WSDM 2023
[8] Multi-view multi-behavior contrastive learning in recommendation, International conference on database systems for advanced applications, 2022.
[9] A Survey on Multi-Behavior Sequential Recommendation. arXiv 2023

Dear Reviewer od31, we sincerely appreciate your valuable suggestions and comments. As the comment phase is drawing to a close, we would like to know if all of your concerns have been adequately addressed. If you have any further questions or require additional clarification on any aspect of our work, we would be glad to provide thorough responses.

W1, W2 and Q2:
Thanks for your suggestion, we give the definition of scene as follows: In online service, scene, created by app or web designers, is the sub-interfaces containing specific themes or functionalities within an App or website. In Figure 1 (a) of our paper, an e-shopping app has many sub-interfaces with each has different functionalities such as ''text search'' and ''image search''. Text search enables users to use text query for buying products, while image search enables users to use image query for buying products. In Figure 1 (b) of our paper, we conclude that items in different scenes have varied themes and largely influence users' engagements. So, we can see the ''scene'' feature widely exists in real-world applications, and it can be generalized to various platforms.

We emphasize that scene feature is totally different from the widely studied multiple behaviors (e.g. view, click, cart, pay). ''Scene'' is created by app or web designers, while multiple behaviors are generated by users. How to model the scene and multiple behaviors are two different problems. In our model and experiments, we only consider one-type behavior, i.e., ''view'' in Outbrain and ''pay'' in our industrial datasets. It is not appropriate to compare DSPnet with multiple behavior sequential RS works.

W3:
The Outbrain dataset was initially developed for click-through rate prediction, using mean average precision as an evaluation metric. In contrast, this paper concentrates on predicting sequential user behavior within scene feature. In this case, evaluation metrics such as Recall@k and NDCG@k are widely recognized and used [1–6].

After considerable efforts of identifying suitable public datasets, we found that only the Outbrain dataset could meet our requirements. Given the scarcity of available data on this problem, we are going to release our collected industrial datasets and put forward this research topic in the community.

W4:
Thank you for your suggestion. In our industrial case, we usually focus on forecasting future items and thus only present these item prediction results in the paper. Following your advice, we have done scene prediction. Because of limited response space, we show the results in Appendix-Table 7 of the submitted revised version. From the table, we see that DSPnet consistently outperforms recent models in scene prediction performance.

W5 and Q2:
Based on the data analysis, we identified the scene feature as a critical factor influencing future behavior occurrences. To the best of our knowledge, studying how to better employ this scene feature is an important and new problem setting in sequential behavior modeling. This area remains underexplored primarily due to the lack of available research data. Consequently, we do not elaborate more works on this research problem. It is important to note that existing scene-based sequential recommendation systems [16-19] employ different scene definitions from ours. [16] explores adaptive sequential RS across different domains (e.g. book, music). [17] defines scene as a collection of predefined item categories. [18] investigates the use of large language models (LLMs) for real-time sequential RS. [19] refers to scenes as 200 predefined topics, such as "weekend spring outing".

Meanwhile, we have also added works published in 2024, including LLMs[5, 6, 7, 8], diffusion models[9, 10, 11] and transformer efficiency [12, 13] for sequential RS .

W6:
We agree that modeling scene sequences can serve as an auxiliary task to enhance item prediction, but DSPnet is specifically designed to integrate scene information into sequence behavior modeling. This specialized design offers advantages over other dual-stream or multi-task learning (MTL) models.

Popular MTL frameworks [14,15] primarily focus on modeling task relationships and utilize various parameter-sharing strategies to address the gradient conflict issue. However, these approaches have two main shortcomings on our problem. First, they are unable to capture the mutual effects between scenes and items when predicting future behaviors. Second, they do not account for the randomness, noise, and skewness commonly present in user behavior sequences.

In contrast, DSPnet proposes dual sequence learning with sequence feature enhancement module to effectively encode sequential dynamics and deliver these dynamics to both scene and item side for predicting behaviors. Additionally, it introduces CCR loss to achieve representation invariance and improve the model's robustness on skewed user behavior sequences.

We will add this discussion in the new version.

Q1:
The number of used scenes is given in Appendix-Table 3. In Outbrain, we take the domain name of website as the scene feature. In industrial datasets, we take the ''text search'', ''image search'' and many sub-interfaces within the app as scene feature.

Dear Reviewer 8nu5, we sincerely appreciate your valuable suggestions and comments. As the comment phase is drawing to a close, we would like to know if all of your concerns have been adequately addressed. If you have any further questions or require additional clarification on any aspect of our work, we would be glad to provide thorough responses.

==============================================================================
W1 and Q1:

Yes. CCR can be integrated with other recommendation models. We have integrated it with the most competitive SOTA SceneContraRec, and the results are provided in the following table. The results clearly indicate that incorporating CCR loss enhances the performance of these sequential recommendation models, demonstrating its effectiveness.

Table: Incorporating CCR with other baseline models.

==============================================================================
Q2:
We analyze the complexity of different models via two components: feature encoding and behavior prediction, both of which are commonly present in sequential behavior prediction models.

The compared methods all use the powerful and popular transformer-based encoder.
Let $B$ be the batch size, $L$ be the number of transformer layers, $|\mathcal{T}|$ be the sequence length of samples, $H$ be the head number, and $d$ be the dimension of each head. The time complexity of the transformer-based encoder is $O(B \times L \times H \times |\mathcal{T}|^2 \times d)$, which is the same for all compared methods. The main difference in complexity lies in behavior prediction. First, we denote $K^{v}$ as the number of candidate items (including positive and negative ones) for prediction. The complexity comparison is listed in the following table.

Table: Time complexity of different models on behavior prediction part.

In this table, $K^{v}$ generally reaches the magnitude of millions in industrial setting, while $|\mathcal{T}|$ is set it to $100$ in our large-scale datasets. Consequently, the complexity of DSPnet is considerably lower than that of CARCA and Bert4Rec and does not significantly increase over SOTA methods such as SceneContraRec. This makes DSPnet well-suited for usage in large-scale industrial datasets. We have successfully deployed it in our system using 16 A100 GPUs.

About hyper-parameter tuning, we make grid search of hyper-parameters on a small scale dataset AllScenePay-1m, and directly use the configurations for AllScenePay-10m and more large-scale datasets. The sensitivity of hyper-parameters are given in Appendix-Figure 6 of our original paper.

Dear Reviewer FSwH, we sincerely appreciate your valuable suggestions and comments. As the comment phase is drawing to a close, we would like to know if all of your concerns have been adequately addressed. If you have any further questions or require additional clarification on any aspect of our work, we would be glad to provide thorough responses.