Learning Effective Language Representations for Sequential Recommendation via Joint Embedding Predictive Architecture

We sincerely thank the reviewer for the positive and constructive feedback. We appreciate your thoughtful suggestions for improving our work. Below, we present additional ablation studies and analyses to address your concerns in detail.

(Q1) it seems that flattering the metadata attributes is an important component in the proposed framework, so will the order of these attributes affect the model outcomes?

Answer: We conducted experiments by changing the order of item sequences during encoding from "Title ... Brand ... Category" to "Category ... Brand ... Title". We then performed pre-training and fine-tuning on Scientific and Office datasets with the same experimental setup as in our paper. The results are as follows:

The results show minimal impact (~1-2% difference), indicating that JEPA4Rec is robust to attribute ordering due to its bidirectional Transformer architecture.

(Q2) I wonder if we can rely on the "knowledge" within the LLMs to encode the items directly for those cases where the metadata attributes are not available

Answer: We conducted experiments using only the Title of items as textual data for encoding, then performed pre-training and fine-tuning on Scientific and Office datasets. The results are:

While using only titles results in some performance degradation, JEPA4Rec still maintains reasonable performance. Additionally, please refer to Section 2 "Robustness to Partial Information" where we tested JEPA4Rec's recommendation capability when masking textual data, demonstrating its ability to work with incomplete information.

(Q3) I wonder if the authors can conduct additional analysis to study the relative contribution of four types of embeddings in the proposed model; additionally, why do the authors choose to simply sum them together? It seems that a more intuitive way is to do a weighted sum.

Answer: We have provided ablation studies on the importance of different token embeddings in Section 3 of our Global Rebuttal. We use simple summation similar to BERT's token embedding approach. To clarify how weighted summation affects JEPA4Rec's performance, we conducted experiments with weighted embedding:

$\mathbf{E}_w = \text{LayerNorm}(\mathbf{A}_w + \mathbf{B}_w + \mathbf{C}_w + 1.25 \times \mathbf{D}_w)$

We assign higher weight to Item Position Embedding ($\mathbf{D}_w$) considering the important role of it in JEPA4Rec's architecture. After pre-training and fine-tuning on Scientific and Online Retail datasets with the same experimental settings, the results are:

The weighted approach shows slight improvement, suggesting that emphasizing different type of embedding information can be beneficial, though the simple summation approach already performs well.

We appreciate your feedback and are happy to clarify further if needed.

Thank you for the insightful and balanced review. We value the reviewer’s comments and respond to the key concerns below.

However, this could still be considered a limited novel work, as the paper needs more details about the challenges that previous works did not tackle, while this paper did.

Answer: We have presented the novelty of JEPA4Rec in our Global Rebuttal. We summarize the key points as follows:

1. Better pre-training framework based on item embedding recovery mechanism
2. Learning at item representation level instead of only token level
3. Experimental validation demonstrating JEPA4Rec's effectiveness through comprehensive experiments

The key challenges we address that previous works did not tackle include: (1) learning generalizable item representations across domains through embedding space recovery, (2) capturing common-sense user preferences through joint embedding predictive architecture, and (3) achieving effective cross-domain transfer with minimal pre-training data.

The related works section is too short.

Answer: We will expand the related work section in future versions to provide more comprehensive coverage of relevant literature.

The claim about "semantic gap between modalities remains a challenge" contradicts works like LMM4Rec that show modality alignment improvements.

Answer: Our statement is based on several previous works [1,2] that utilized different modalities for recommendation systems but did not achieve significant effectiveness improvements. We thank you for the evidence and will cite LMM4Rec in our related work section to provide a more balanced perspective on modality alignment progress.

Why choose 50% masking for the next item instead of other percentages like 80%

Answer: We chose 50% masking for ground truth item tokens based on ablation studies presented in Appendix F (Page 14). Our experiments showed that 50% provides the optimal balance between providing sufficient challenge for the model while maintaining enough information for meaningful learning.

Missing Hit Rate@K metrics, particularly HR@1 for next-item prediction evaluation.

Answer: We thank you for this suggestion. First, we clarify the metric definitions:

• Hit Rate@K: Calculates the share of users for which at least one relevant item is present in the top K
• Recall@K: Measures the coverage of relevant items in the top K
• NDCG@K: Considers both relevance and position of items in the ranked list

For the leave-one-out evaluation strategy, since we predict only 1 ground truth item, Recall equals Hit Rate in this case. We provide additional Hit Rate@1 results:

JEPA4Rec outperforms RecFormer across all datasets, demonstrating the effectiveness of JEPA4Rec in next item recommendation tasks.

Need more detailed context about practical impact.

Answer: JEPA4Rec can be widely applied in e-commerce, especially for platforms with multiple product domains like Amazon. Instead of requiring separate recommendation systems for each domain, we can train JEPA4Rec with moderate datasets from various domains and leverage its knowledge generalization capability for cross-domain recommendations without extensive retraining.

Key practical benefits:

1. Cost reduction: Serving one JEPA4Rec model for multiple domains instead of domain-specific models
2. Enhanced search experience: Superior recommendation performance when revealing product information that customers are searching for (e.g., keywords about title, brand, or category)
3. Robustness to incomplete data: Effective recommendations even when item information is missing or incomplete

Please see our Appendix D Case Study and Section 2 of the Global Rebuttal for detailed applications of JEPA4Rec with incomplete item data.

We appreciate your feedback and are happy to clarify further if needed.

[1] Yuan, Zheng, et al. "Where to go next for recommender systems? id-vs. modality-based recommender models revisited." Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2023.

[2] Liu, Junling, et al. "Is chatgpt a good recommender? a preliminary study." arXiv preprint arXiv:2304.10149 (2023).

We thank the reviewer for the detailed and constructive feedback. We appreciate the thoughtful assessment of our paper's strengths as well as the concerns raised. Below we address the key points raised, particularly around the clarity of Section 3 and the novelty of our contributions.

It is not obvious how each piece fits together, and requires re-reading paragraphs several times to fully understand what it is being explained.

Answer: We provide a comprehensive overview of the JEPA4Rec framework flow:

Framework Components: JEPA4Rec consists of 3 main components:

• Context Encoder ($f_\theta$): Processes masked item sequences
• Target Encoder ($f_{\overline{\theta}}$): Processes complete item information (same architecture as Context Encoder, updated via Exponential Moving Average of Context Encoder parameters, not trained directly)
• Predictor ($g_\phi$): Recovers full representations from partial information

Processing Flow:

1. Input Processing: Item metadata is flattened $\rightarrow$ masking strategy creates partial information $\text{Masked Input} \rightarrow f_\theta \rightarrow h_{CLS}, h_{n+1,M}$ $\text{Complete Input} \rightarrow f_{\overline{\theta}} \rightarrow h_n, h_{n+1}$
2. Representation Recovery: Predictor recovers full representations:

$\hat{h_n} = g_\phi(h_{CLS} \oplus \mathcal{D}_n)$

$\hat{h_{n+1}} = g_\phi(h_{CLS} \oplus \mathcal{D}_{n+1})$

Please note that we use light weight MLP for building $g_\phi$.

3. Training: The mapping loss encourages the predicted item embedding $\hat{h_n}$, which is derived from partial item textual data, to approximate the target embedding $h_n$, which is obtained from the full item textual information. Similarly, it aligns $\hat{h_{n+1}}$ with $h_{n+1}$ in the same manner. We also use MLM loss to help the model learn the semantic information of the dataset, and contrastive loss for the recommendation task.

4. Fine-tuning: Only Context Encoder is used for target domain recommendation with learned $h_{CLS}$ containing rich user preference information.

The section fluctuates between explaining low-level and high-level concepts without clear motives.

Answer: We agree. We will streamline common NLP terminology and move detailed explanations to the Appendix in future versions to focus on our actual contributions.

Claims about learning "common-sense user preferences" are too broad and not justified by experiments.

Answer: Thank you for pointing this out. The term "common-sense" follows the definition from Yann LeCun's work [1]. You are correct that we learn better item representations that create more informative $h_{CLS}$ embeddings (representing user preferences), leading to improved experimental results. Based on surveys of prominent sequential recommendation papers [2,3,4], an accuracy improvement of ~0.01 and average 5-10% improvement per dataset compared to SOTA RecFormer represents substantial progress in the recommendation system field.

Discussion is shallow and ablation experiments are limited despite introducing many components.

Answer: We have supplemented additional ablation studies in our Global Rebuttal, including ablation of sequence-item contrastive task and JEPA4Rec's components. We will provide more detailed discussion for each experiment in future versions.

Novel contributions and originality are not clear, and the differences with standard NLP practices are subtle.

Answer: We appreciate the opportunity to clarify this. We have presented JEPA4Rec's novelty in our Global Rebuttal:

1. Better pre-training framework based on item embedding recovery mechanism
2. Learning at item representation level instead of only token level
3. Experimental validation demonstrating effectiveness through comprehensive experiments

The key novelty lies in applying Joint Embedding Predictive Architecture to textual sequential recommendation, which has never been done before.

Very limited related works section and discussion of framework comparison.

Answer: We agree and will expand this section in future versions.

We will continue to respond to the remaining questions in the next response due to length constraint.

[1] LeCun, Yann. "A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27." Open Review 62.1 (2022): 1-62.

[2] Kang, Wang-Cheng, and Julian McAuley. "Self-attentive sequential recommendation." 2018 IEEE international conference on data mining (ICDM). IEEE, 2018.

[3] Sun, Fei, et al. "BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer." Proceedings of the 28th ACM international conference on information and knowledge management. 2019.

[4] Yue, Zhenrui, et al. "Linear recurrent units for sequential recommendation." Proceedings of the 17th ACM international conference on web search and data mining. 2024.

Based on the authors' rebuttal and comments, I have increased my score from 5 to 7.

I think that the overall paper is good and, although the novelty is sometimes difficult to distinguish from previous works, the authors have done an excellent job at addressing the questions and providing additional information (particularly ablations). The novelty may not be on the individual components, but their combination and thus the framework are the novel part. The experiments are solid and have gone beyond the average paper.

I have some concerns that the additional information provided by the authors will not easily fit into the final version of the paper. I think that the paper requires large edits to some sections to incorporate the clarifications in the global rebuttal, as the framework and some components explained here differ significantly from the paper (i.e., the processing flow is not clear in the paper).

We now proceed to discuss the results presented in Table 1. These results demonstrate JEPA4Rec's superior robustness when making recommendations based on partial item information. Notably, JEPA4Rec maintains stable performance even when 60% of item tokens are masked, with only modest performance degradation (e.g., from 0.1282 to 0.1201 on Scientific dataset). In contrast, RecFormer shows significant accuracy downgrade as the masking ratio increases, with performance declining more substantially (e.g., from 0.1198 to 0.1088 on Scientific dataset). This robustness stems from JEPA4Rec's ability to learn item representations in the embedding space through its mapping loss pre-training, enabling effective inference even with incomplete textual descriptions. JEPA4Rec consistently outperforms RecFormer by ~3-5% on NDCG metric across all masking scenarios.

3. Extended Ablation Studies

We acknowledge the current ablation study section is concise due to page limitations. We provide comprehensive additional ablation studies below:

3.1 Token Embedding Component Analysis

We evaluate the importance of different token embedding types:

• Experiment 1: Removing MLM Loss We remove the MLM loss component to assess how much semantic information learning from textual data contributes to JEPA4Rec's recommendation performance.
• Experiment 2: Removing Token Type Embedding ($\mathcal{C}$) This experiment evaluates the importance of token type embedding in distinguishing different types of textual information within item representations.
• Experiment 3: Removing Token Position Embedding ($\mathcal{B}$) and MLM Loss Token position embedding $\mathcal{B}$ is crucial for language models to determine token positions. We remove both $\mathcal{B}$ and MLM loss to assess JEPA4Rec's recommendation capability when losing components that help learn semantic information from e-commerce text data.
• Experiment 4: Removing MLM Loss, Token Position Embedding ($\mathcal{B}$), and Token Type Embedding ($\mathcal{C}$) This experiment determines JEPA4Rec's performance when relying solely on item representation recovery through L2 Mapping loss.

Note on Item Position Embedding ($\mathcal{D}$): We do not remove $\mathcal{D}$ as it is essential for determining item positions in history sequences and enabling JEPA4Rec to recover complete item representations.

Table 2: Ablation study on the importance of embedding components

1. MLM loss is crucial for performance as it helps the model learn additional semantic information from textual data of items. The performance drop when removing MLM loss (from 0.1282 to 0.1170 NDCG@10 on Scientific dataset) shows its significant contribution to understanding item semantics.
2. All embedding components are important for the model's effectiveness:
   • Token type embedding ($\mathcal{C}$) provides discriminative power for different attribute patterns, with noticeable performance degradation when removed.
   • Token position embedding ($\mathcal{B}$) is essential for understanding token sequence structure.
   • The combined removal of multiple components leads to progressively worse performance, confirming that each embedding type contributes unique and valuable information to the model.