PANTHER: Generative Pretraining Beyond Language for Sequential User Behavior Modeling

W1: Add experiments to validate efficiency claims by measuring training time and resource usage as data volume scales.

A1: Thank you for highlighting the importance of quantitatively validating our efficiency claims. We have conducted additional experiments to quantitatively illustrate how our SPRM module scales efficiently compared to a Transformer baseline.

Specifically, we measured GPU memory usage and inference time as input sequence lengths increased, serving as a proxy for larger per-user data volumes (batch size=128, single A100 GPU). The results clearly demonstrate SPRM's efficiency advantage:

SPRM effectively handles sequences up to 8192 tokens without memory overflow, whereas the Transformer baseline fails beyond 4096 tokens. We will include these experiments in the final camera-ready version to substantiate our efficiency claims comprehensively.

W2:. Clarify how the proposed Pattern Matching Transformer outperforms prototype-based models like Perceiver for user behavior modeling.

A2: Thank you for your suggestion regarding prototype-based models like Perceiver. The Perceiver uses cross-attention to map high-dimensional input data into a compact latent space, with queries (Q) derived from the latent array and keys (K) and values (V) coming from the raw input. In contrast, PANTHER uses convolutional layers to aggregate local patterns and match them with learnable prototypes.

We conducted experiments comparing PANTHER and Perceiver, both with 1 million parameters, on the next transaction prediction task. The results show that PANTHER can outperform Perceiver across all metrics (HR@1, HR@5, HR@10).

Furthermore, we find that PANTHER is significantly more efficient to train. With the same number of training steps, PANTHER's training time is only 17.3% of Perceiver's.

W3: The current simple, frequency-based tokenization risks missing rare but critical fraud patterns due to UNK mappings, and its fixed vocabulary requires frequent updates and retraining to handle evolving transaction behaviors effectively.

Q3: Given that the frequent-token set will challenge as transaction distribution evolve over time, is it necessary to retrain the model in such scenarios?

A3: Thank you for raising this important concern regarding our structured tokenization approach. While it is true that frequency-based vocabularies might map rare events to UNK tokens, our PANTHER pretraining strategy deliberately focuses only on modeling normal user behaviors. Specifically, it compresses historical patterns into personalized embeddings that establish a robust baseline of typical actions per user, rather than explicitly modeling rare or anomalous events. For example, some users regularly engage in large transactions with new contacts for business purposes, whereas others rarely exhibit such behavior—our pretraining method precisely captures these user-specific norms.

In the downstream fraud detection task, the model effectively identifies rare yet critical fraud signals by comparing recent user behaviors, leveraging contextual features and sequential information as detailed in Line 134, against the pre-trained personalized baseline. Significant deviations, such as unusual large payments or unexpected transaction patterns, are flagged as potential fraud.

For distribution shifts, the model's predictive logic stays relatively static and is retrained periodically (e.g., monthly) with vocabulary updates, while we perform regular inference to incorporate new behaviors into embeddings—no full retraining required. We'll clarify this separation in Section 3.2 and expand the limitations section on UNK impacts and adaptive strategies.

Q1: Could the authors provide some statistical summaries of the WeChat Pay dataset to illustrate its scale and characteristics?

A4: Thank you for these insightful questions, which allow us to clarify essential aspects of our dataset and embedding approach:

The WeChat Pay dataset consists of transaction behaviors from approximately 1 billion users collected over the past year. On average, each user has around 300 transactions, encompassing diverse transaction types such as social payments (e.g., peer-to-peer transfers) and commercial transactions (e.g., merchant purchases). Key dataset features include multi-dimensional transaction attributes such as payment amount, payment channel, payment source, merchant category (e.g., retail, gaming), merchant risk scores, and timestamps.

Q2: How are these textual payment behaviors transformed into token embeddings? Does this process utilize a pretrained BERT or large language model, and is this module updated during model training?

A5: In our PANTHER framework, textual payment behaviors are transformed into token embeddings through a hybrid approach. Specifically, merchant names are embedded by prompting a pretrained large language model (LLM) (e.g., "Please introduce the business scope and potential customer base of this merchant"), extracting the final-layer embedding output via average pooling, and discarding the generated textual description. These embeddings are precomputed and stored in a lookup table, remaining fixed during the training process to ensure computational efficiency and embedding consistency.

W1: The paper is not clearly written, the notation and the terms used are not properly introduced:

• HR and NDCG are not defined
• T is not introduced
• \psi is introduced, but how it is trained/computed is not described.

A1: We appreciate the reviewer's feedback. We would like provide a detailed illustration to the notations you mentioned as follows.

• HR and NDCG: Our use of HR (Hit Rate) and NDCG (Normalized Discounted Cumulative Gain) follows the convention of influential works like SASRec [17] and HSTU [12], where these are often treated as standard metrics in sequential recommendation and user behavior modeling tasks. We agree that explicit definitions improve clarity for all readers and will add them to the revised manuscript.
• T: Our notation T consistently refers to the sequence length throughout the paper, and it's explicitly defined in line 187 ("For a sequence of length T"). We acknowledge that its earlier appearance in line 161 might cause confusion. Our intention there was simply to illustrate that the time complexity exhibits a quadratic growth trend. To prevent any ambiguity, we will remove this early mention and rephrase the sentence to "incurring a quadratic time complexity of O(T^2)."
• $\psi$: Our notation $\psi$ in Eq. (3) denotes a general embedding function mapping context attributes to dense vectors. In our experiment, it is implemented by an one-layer MLP and is trained end-to-end on labeled data during downstream fine-tuning.
• Notation table: to further enhance the paper's clarity, we will include a comprehensive notation table in the appendix, listing all key terms and symbols used throughout the manuscript. | Symbol | Description | | --- | --- | | $\mathbf{X}_u=[x_1,x_2,\ldots,x_L]$ | The historical payment behavior sequence for a user $u\in \mathcal{U}$. | | $x_t \in \mathcal{V}$ | The transaction token at time step $t$, $\mathcal{V}$ is the set of possible transaction tokens.| | $\mathrm{Pr}(y = 1 \mid x_{\mathrm{new}}, \mathbf{c}_{\mathrm{new}}, \mathbf{X}_u)$ | Fraud probability of a new transaction $x_{\mathrm{new}}$ with context $\mathbf{c}_{\mathrm{new}}$, given history $\mathbf{X}_u$. $y$ is an indicator for fraud. | | $\mathcal{L}_{\text{gen}}$ | Generative loss function for next behavior prediction.| | $\mathrm{Pr}\_\theta(x\_t \mid x\_{<t}, x\_{\text{profile}})$ | Probability of transaction $x_t$ given history $x_{<t}$, user profile $x_{\text{profile}}$, and model parameters $\theta$. | | $e_{u}$ | Learned user profile embedding for user $u$.| | $g_\phi(\cdot)$ | Real-time fraud detection model that integrates recent patterns, predicted deviations, and user profiles. | | $\psi(\cdot)$ | Embedding function for transaction context features (e.g., a linear layer). | | $f_\mathrm{enc}(\cdot)$ | Sequence encoder (e.g., TextCNN) for short-term user behavior. | | $\Delta(x_{\text{new}}, \hat{x}_{L+1})$ | Deviation between observed new transaction $x_{\text{new}}$ and predicted next behavior $\hat{x}_{L+1}$. | | $\tau = (c_i, a_j, m_k, r_l)\in \mathcal{C} \times \mathcal{A} \times \mathcal{M} \times \mathcal{R}$ | A transaction token composed from contextual and counterparty features. | | $\mathcal{P} \in \mathbb{R}^{m\times d}$ | Set of $m$ learnable prototypes of dimension $d$ in the Sequence Pattern Recognition Module (SPRM). | | $H\in\mathbb{R}^{T\times d}$| Token embeddings of dimension $d$ for an input sequence of length $T$.| | $\mathcal{L}_{\text{CL}}$ | Contrastive loss function for user profile embedding learning | | $(i,j)\in Pos$| Positive user pairs with similar demographic attributes. | | $e_i, e_j$ | User profile embeddings for users $i$ and $j$.| | $\lambda$ | Hyperparameter balancing $\mathcal{L}{\text{gen}}$ and $\mathcal{L}{\text{CL}}$. | | $R_m$ | Risk score for merchant $m$, derived from user deviation scores associated with the merchant.|

W2: P is used in many different meanings, which makes the paper very difficult to parse.

A2: We regret the overloaded use of P, which indeed risks confusion. In the revision, we'll disambiguate by changing notations: e.g., $\mathrm{Pr}$ for probability in Eqs. (1)-(3), $\mathcal{P}$ for prototypes in Section 3.3.2, and Pos for positive pairs in Section 3.4. This should make the paper much easier to parse without altering content.

We believe these targeted revisions will resolve the weaknesses and enhance readability—your input has been invaluable, and we hope this demonstrates our commitment to strengthening the submission.

W3:

• Why only SASRec and HSTU were used for comparison
• Link to HSTU [19]
• Discrepancy in SASRec results

A3: Thank you for highlighting these important points, providing us an opportunity to clarify and strengthen our submission. We believe the confusion arises primarily from a citation error and differences in evaluation protocols, as detailed below. We emphasize that all our results are fully reproducible using the provided supplemental code, including scripts for data processing, model training, and evaluation across internal and public datasets.

1. Selection of Baselines (SASRec and HSTU): We chose SASRec as a widely recognized sequential baseline and HSTU as the current state-of-the-art (SOTA) generative pretraining model, which significantly outperforms other notable methods (e.g., BERT4Rec, GRU4Rec by up to 65.8% in NDCG). Due to page limitations, we prioritized these two highly relevant baselines, focusing on demonstrating clear improvements. Nevertheless, we will incorporate additional results, such as BERT4Rec from the HSTU paper, in the appendix for comprehensive coverage.
2. Correct Reference for HSTU: The correct citation for HSTU [19] is "Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations" by Zhai et al. (ICML 2024, arXiv:2402.17152). We will ensure this reference is properly cited in the final version.
3. Discrepancy in SASRec Results: The noted discrepancy in HR@10 for SASRec on MovieLens-1M (0.2853 reported by our paper/HSTU vs. 0.8245 in [17]) is due to differences in evaluation protocols. Specifically, [17] employs a sampled negative setup (~100 negatives per test item), which significantly simplifies ranking, whereas our evaluation (following HSTU) considers the entire item set (~3,700 items), offering a more realistic and challenging assessment. Notably, HSTU independently confirms our reported performance (0.2853). To avoid confusion, we will explicitly clarify this evaluation distinction with a footnote in Section 4.1.

Finally, we emphasize our methodological contributions tailored specifically for challenging scenarios such as high cardinality, sparsity, and real-time processing needs:

• Token Space Compression reduces dimensionality and noise through structured contextual tokenization.
• Pattern-Aware Convolutional Cross-Attention (SPRM) efficiently captures local and global patterns with linear complexity.
• Contrastive Personalized User Embedding integrates static and dynamic user information to enhance cold-start performance.

These innovations have yielded substantial empirical improvements, achieving gains of 25.6%+ on WeChat Pay benchmarks, 21-29.6% on open datasets, and a 38.6% recall improvement in real-world fraud detection. We believe these clarifications reaffirm the robustness and novelty of our work and look forward to further improving the manuscript based on your valuable feedback.

Dear Reviewer,

Thank you for your continued engagement and for thoughtfully considering our additional comments. We sincerely appreciate your valuable feedback, which has contributed to strengthening our work. We’re pleased to hear that our inclusion of comparisons with TabBERT and other important baselines has addressed your concerns.

Your decision to adjust your score is greatly appreciated, and we are encouraged by your support.

Best regards,

The Authors

W0: Concern regarding the mention of a real-world dataset in the context of the evaluation.

A1: Thank you for raising this concern—we truly appreciate your thoughtful feedback. In preparing the submission, we carefully adhered to NeurIPS guidelines. This practice aligns with numerous accepted NeurIPS papers from industry settings (e.g., those associated with Google, Meta, and Alibaba), where evaluations on proprietary or internal datasets are discussed to highlight practical relevance while preserving the core principles of anonymity. For example, Google's TpuGraphs paper introduces a performance prediction dataset on tensor programs running on Tensor Processing Units (TPUs), their proprietary hardware. We believe this approach balances the need for meaningful context with the integrity of the review process.

W1:Please expand and define HR (Hit Rate) and NDCG on first use, and define them in the paper.

A2: Thank you for this helpful suggestion. We will define HR (Hit Rate) and NDCG (Normalized Discounted Cumulative Gain) on first use in the camera-ready version: "HR@K measures the fraction of instances where the ground-truth item appears in the top-K predictions. NDCG@K evaluates ranking quality by rewarding relevant items appearing earlier, normalized against the ideal ranking." This ensures accessibility for unfamiliar readers.

W2: Encourage ablation studies to assess the impact of key design choices like user embeddings, transformer vs. CNN, and prototype usage.

A3: Thank you for your insightful suggestion regarding ablation studies. We agree that they enhance the robustness of the paper and have already incorporated several in the submission:

• Impact of Personalized User Embeddings: In our paper, Table 1 presents the results of integrating the Unified User-Profile Embedding in the next-transaction prediction task on WeChat Pay data. The variant "PANTHER (SPRM only)" (0.2243 HR@1, +14.9% over the Transformer baseline) represents the model without user profile integration. In contrast, "+ Profile + CL" (0.2452 HR@1, +25.6% overall) demonstrates the added value of personalized embeddings with contrastive learning. For the downstream fraud detection task, Figure 4 shows a similar ablation, where "+ User Profile Embedding" achieves a 16.8% relative recall improvement at Top-0.1% over the DeepFM baseline.

• Impact of Prototypes: Since prototypes are a core component of the SPRM module, the "PANTHER (SPRM only)" variant in Table 1 includes them. In comparison, baselines like Transformer and SASRec do not incorporate this structured approach, highlighting the prototypes' contribution to the +14.9% HR@1 gain.

To address your other points (e.g., Transformer vs. CNN in SPRM and the isolated impact of prototypes on downstream fraud detection), we conducted additional ablation experiments. These assess: (1) replacing the Transformer with a CNN in the model architecture, and (2) evaluating fraud detection with a configuration excluding SPRM (and thus prototypes). We will include these results in the camera-ready version.

W3:Explore the impact of pre-training on multiple datasets and how scaling data and model size affects PANTHER’s performance.

A4: Thank you for this insightful question on scaling laws. The supplementary material (Section B.2) already explores pre-training on multiple transactional datasets via transfer learning. Specifically, pre-training on WeChat Pay and fine-tuning on CCT/MBD-mini results in an average +16.66% HR@1 improvement over direct fine-tuning (Table 6), demonstrating effective cross-context transfer. For data scaling (# transactions/users) and model size —we ran new experiments on WeChat Pay for next-transaction prediction.

Scaling from 0.45M to 44M parameters yields a +8.1% HR@1 and +8.5% HR@10 improvement. The smaller model converges at ~0.1B users, while the medium model converges at ~0.15B users. These results will be added to the supplementary material in the camera-ready version.

Q1: "Could you explain how your model will adapt to a new user that it is not trained on? How are the user embeddings generated?"

A5: PANTHER generates user embeddings by combining static demographic features (e.g., age, location) with dynamic transaction histories through the Unified User-Profile Embedding module, which is trained on a predict-next-behavior task using a subset of users.

For new, unseen users, the model leverages the patterns learned from the trained subset and generalizes them to make predictions, enabling zero-shot adaptation. Supplementary B.1 demonstrates this: untrained users achieve HR@1 = 0.2332, compared to 0.2452 for trained users, resulting in a +301% improvement over cold-start baselines (Table 5). This shows that PANTHER can predict the next behavior for new users based on learned patterns from others.

To further address adaptation for new users with limited data (e.g., very short transaction histories), we conducted experiments where behavior sequences for unseen users were subsampled during inference. These results show robust performance even with minimal data. For instance, with a sequence length of 10, HR@1 reaches 0.3083, surpassing the full-sequence baseline (HR@1=0.2452 for seq=256). This counterintuitive improvement may arise from some users engaging in highly predictable activities—such as frequent game top-ups—that are easier to model with sparse data. As sequences lengthen, performance gradually stabilizes, demonstrating PANTHER’s effectiveness in cold-start scenarios. We will include this analysis and the full table in the camera-ready supplementary material.

Q2: Could you please provide the summary statistics of this dataset? What are the fields? How many users is it from? What is the time range, # of transactions, average length of transaction, etc. What are the different features in this dataset? Will a subset of this dataset be made public? How is the fraudulent data labeled? Can there me false negatives (i.e. fraudulent transactions labeled as normal). How is the train and test split is chosen?

A6: Thank you for these detailed questions about the WeChat Pay dataset, which help clarify its characteristics and our experimental setup. Below, we address each point:

• Summary statistics: The dataset encompasses behaviors from approximately 1 billion users over the past year. The average transaction sequence length per user is around 300, covering both social payments (e.g., peer-to-peer transfers) and commercial payments (e.g., merchant purchases).

• Fields/Features: The dataset includes multi-dimensional transaction attributes such as payment amount, pay channel (e.g., QR code, app), pay source (e.g., bank-linked, wallet), merchant category (e.g., retail, gaming), merchant risk level, timestamps, user demographics (static: age, location). These are compressed via Structured Tokenization for modeling.

• Public subset availability: Due to privacy and proprietary concerns, we cannot release any subset of the WeChat Pay dataset. However, to promote reproducibility, we evaluate PANTHER on fully public open-source datasets (CCT, MBD-mini, MovieLens-1M) in the paper, where it shows strong generalization.

• Fraud labeling: Fraudulent transactions are labeled based on user reports to customer service, followed by thorough investigation to confirm.

• False negatives: Yes, false negatives are possible, as some fraud may go unreported by users. This is a common challenge in real-world fraud datasets, mitigated in our setup by focusing on recall improvements.

• Train/test split: For the fraud detection task, we use a temporal split: the past 60 days of data for training (to learn recent patterns) and the subsequent 10 days for evaluation (to simulate real-time deployment and avoid data leakage).

We will add a dedicated subsection in the camera-ready version summarizing these dataset details for clarity.

Q3:What is the context length of the model? How many tokens can it take at a time?

A7: PANTHER is designed to handle a maximum sequence length of 512 behaviors (tokens) per user during both training and inference. This choice balances computational efficiency—longer sequences quadratically increase complexity in attention-based components—with empirical performance: experiments show that extending beyond 512 yields diminishing returns in metrics like HR@1.

Dear reviewer, Thank you for your thoughtful review and for taking the time to consider our rebuttal. We greatly appreciate your recognition of our work and your supportive feedback, and we are glad that our responses have addressed your questions. Thanks very much!

W1: The paper introduces the function in Equation (3) as part of the hybrid inference model, but its exact form or implementation is not explained.

A1: Thank you for pointing out the need for clarity regarding the function $g_\phi$ in Equation (3) (Eq. 134). The function $g_\phi$ represents a general downstream discriminative model, which can take various flexible forms depending on specific tasks or use cases. Its primary role is integrating embedded context features, recent transaction encodings, long-term user profile embeddings $\mathbf{e}_u$, and behavior deviations to compute a unified risk score.

In our practical implementation, we use a DeepFM structure for capturing feature interactions, combined with a TextCNN for recent transaction sequence modeling, followed by a feed-forward network. The model is trained end-to-end using binary cross-entropy loss on labeled data. We will clearly specify both the general and practical forms in Section 3.4.

W2: Correct the token compression reduction claim from 99.7% to the accurate 97% based on the given values."

A2: Thank you for pointing out the correction regarding the token compression claim. The original reduction from 2 million to 60,000 tokens corresponds to a 97% reduction—not 99.7% as previously stated. We will revise the text to reflect the accurate figure to ensure clarity and correctness.

Q1:Justify the use of outdated baselines (DeepFM, DCN) and consider adding stronger recent models to better highlight PANTHER’s contribution.

A3: Thank you for your insightful question. PANTHER’s hybrid approach is designed to integrate seamlessly with any discriminative model, which is why we chose DeepFM/DCN for this study. While the public datasets (Yelp, MovieLens) used in our experiments are standard for sequence model evaluation, they have limited feature complexity, which restricts the training of more advanced base models. To address this, we will include comparisons with more recent models, such as xDeepFM and AutoInt, in the appendix to showcase PANTHER’s consistent improvements.

In our real-world fraud detection system, we use DeepFM for feature interactions, TextCNN for sequence modeling, and a sequence retargeting model for behavior selection. This setup serves as a baseline, with PANTHER’s generative pretraining outputs yielding a 38.4% recall improvement. Since the core contribution of our work is the generative pretraining, we chose not to focus on this specific baseline setup in detail.

Q2: Include an ablation study on the hyperparameter $\lambda$ to show its impact and justify the chosen trade-off in the optimization objective.

A4: Thank you for the suggestion. We have conducted an ablation study on the hyperparameter $\lambda$, and the results are shown in the table below. As observed, performance remains relatively stable across a range of $\lambda$ values, with minimal fluctuation in HR@1, HR@5, and HR@10. This stability can be attributed to the fact that the contrastive loss converges quickly in the early stages of training, causing its value to diminish rapidly and have little impact on the overall loss thereafter.

We hope this ablation study clarifies the impact of $\lambda$.

Q3. Tokenization for public datasets lacks specifics: How are continuous features (e.g., Yelp’s "review count") bucketized ? Why does MovieLens dataset directly use raw movie IDs ?

A5: For continuous features like Yelp's "review count," we bucketize them using quantiles to ensure approximately balanced bucket sizes across the data distribution.

For MovieLens, we directly use raw movie IDs to maintain consistency with HSTU's experimental setup, enabling fair comparison with their baselines. We'll expand Appendix A with these details for clarity.