From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models

We are grateful for your kind remarks. We hope the following responses can address your remaining concerns.

Concerns about Theoretical Analysis

Response to "Theoretical Claims", and Point 3 of "Other Strengths And Weaknesses"

Following DirectCLR[1], we have explored theoretical justification for our method's ability to mitigate collapse.

Under gradient flow analysis (gradient descent with infinitesimal learning rate), the embedding update process follows a differential equation $\dot{v}_1 = - \frac{\nabla L}{\nabla v_1}$, where $v_1$ is a feature embedding and $L$ the loss. We aim to solve this equation and prove it will be rank deficient during training. However, even for an FM with two features, the solution involves a complex Lambert W function[2]. We are still working on analyzing its rank properties.

[1] Understanding Dimensional Collapse in Contrastive Self-supervised Learning. ICLR 2022

[2] On the Lambert W function. Advances in Computational mathematics, 1996.

Questions about the Hyperparameter Setting

Response to "Experimental Designs Or Analyses"

Our decoder is a one-layer MLP with fixed input and output dimensions. The only tunable hyper-parameter is the nonlinear activations. We evaluated different non-linear activations and found they are crucial for collapse mitigation (see Fig. 7 in Appendix D). Based on our experiments, we recommend ReLU or SiLU for the decoder.

Discussion of Generative Sequential Recommendation with Semantic IDs

Response to "Essential References Not Discussed", and Q3 of "Questions For Authors".

Thank you for your question. The suggested empirical comparison with them is difficult if not imposible since Tiger is mainly designed for the sequential recommendaiton scenario while we focus on the feature interaction scenario.

Concerns about the Ablation Study

Response to Point 4 of "Other Strengths And Weaknesses", and Q1 of "Questions For Authors".

Thank you for your insightful question. Regarding the overfitting issue, we acknowledge that our initial observation was speculative. To further investigate, we conducted a detailed spectral analysis comparing 1-layer and 2-layer MLPs, as shown in https://anonymous.4open.science/r/ICML2025-1748/supp/2layer.png.

The figure shows that both 1- or 2-layer MLPs effectively alleviate the singular value decay (dimensional collapse) observed in discriminative paradigms. However, singular values of the 2-layer MLP decline at a higher rate than the 1-layer MLP, suggesting that the extra layer leads to a more imbalanced embedding space. This aligns with the recommendation performance.

Theoretical or Intuitive Proof for Result 5

Response to Q2 of "Questions For Authors".

Thanks for the question. We only have empirical evidence to justify result 5. We'll revise Result 5 as follows:

"Result5. The field-wise non-linear one-layer MLP is a simple yet effective decoder. Common modifications, such as simplifying the model with field-shared MLPs or removing non-linearities, or increasing complexity through stacking MLP layers or self-attention, lead to inferior recommendation performance."

We apologize for our imprecise statement and thank you for pointing this out, which helps us make the statement more rigorous.

Question on Paradigm Shift for Sequential Recommendation

Response to Q4 of "Questions For Authors".

Many existing sequential recommendals already employ a generative paradigm under the next-item prediction framework. Specially, pre-ranking models such as SASRec follows a self-supervised next-item generative paradigm, while ranking models such as DIN, DIEN follow a supervised next-item generative paradigm. The overhead of generative sequential models such as SASRec and DIN is low for short sequence, leading to a widely deployment of them in industrial systems. The computation cost becomes a challenge for long sequences, but there are many works (such as SIM, TWIN) to employ a two-stage Search & Modeling approach to resolve it.

We are truly thankful for your review efforts. We apologize for the missing information on datasets and notations, and would clarify them as follows.

Dataset Details

Response to "Methods And Evaluation Criteria", "Other Strengths And Weaknesses", and Q1 of "Questions For Authors"

Thank you for your question. The numbers in the table are the number of samples, i.e., user-item interactions.

As for the user or item scales, the user and item features are not annotated in the original dataset. But we can make the following guess:

• Criteo has 13 numerical feature fields and 26 categorical feature fields. All 26 categorical features have been anonymized, but we can guess based on this assumption: user or item ID features are usually with the highest cardinalities. Feature fields with the top-5 cardinalities are (C3: 413,424), (C12: 409,749), (C21: 397,981) (C16: 365,811), (C4: 248,543), where the first denotes feature name and the second denotes the feature cardinality. These features have a relatively high probability of being user or item ID features. Therefore, the scale of users and items could be on the order of ~100K.

• Avazu has 24 categorical features after preprocessing, part of which have been anonymized. Feature fields with the top-5 cardinalities are (device_ip: 2,903,322), (device_id: 820,509), (device_model: 7,259), (app_id: 6,545) (site_domain: 5,461). We infer that 'device' features represent users' devices, which are highly correlated with the number of users. Therefore, the scale of users could be on the order of ~1M. The remaining named features are not likely to be item features, so the item feature may be one of the anonymized features: (C14: 2,556), (C17: 434), (C20: 173). Therefore, the scale of items could be on the order of ~1K.

Besides, we provide more statistics about our industrial dataset: Our industrial model is trained on billions of samples daily, with hundreds of millions of unique users and around 1 million items.

Notations in Eq. 2

Response to "Other Strengths And Weaknesses", and Q2 of "Questions For Authors"

Thank you for your question, and we apologize for missing explanations of Eq. 2. We clarify Eq. 2 as follows:

• Eq. 2 is the formulation of DCNv2, which is a high-order feature interaction model. $L$ denotes the number of cross layers, $l$ denotes the layer index, $N$ denotes the total number of features, $i$ and $j$ denote the indices of features, $\mathbf{v}_i^{(0)}$ denotes the embedding of feature $i$ in the embedding layer, $\mathbf{v}_j^{(l)}$ denotes the embedding of the $j$-th term in the $l$-th layer.
• $M_{F(i) \to F(j)}^{(l)}$ denotes the projection matrix between the $F(i)$ and $F(j)$ field pair in the $l$-th layer.
• $F(i)$ and $F(j)$ denotes the field of feature $i$ and $j$, respectively. In addition, we will further review the entire manuscript to revise any potentially unclear sections and enhance its overall clarity.

We sincerely thank you for your valuable comments. We hope the following responses can address your concerns.

Concerns about Batch-wise Analysis

Response to Weaknesses of "Experimental Designs Or Analyses" & Q1 of "Questions For Authors".

Thank you for your suggestion. The suggested analysis of the entire validation dataset is time-consuming, so we have adopted this batch-wise setting. To ensure the experiment's robustness, we have repeated the analysis experiments with 6 different random seeds when sampling the batches, with results in https://anonymous.4open.science/r/ICML2025-1748/supp/seed.jpg. The trend of embedding spectra is consistent in all batches: On both Avazu and Criteo, the spectrum curve of discriminative paradigms exhibit an abrupt singular decay from ~$1\times 10^{-5}$ to ~$1\times 10^{-15}$, a reduction of $10^{10}$ times. This indicates a severe dimensional collapse issue. But in our generative paradigm, the abrupt singular value decay has been greatly alleviated. This verifies that the generative paradigm substantially mitigates the embedding dimensional collapse issue, forming a more balanced embedding space.

Relationship to Multi-Embedding

Response to "Essential References Not Discussed" & Q2 of "Questions For Authors".

Thanks for your question. The proposed paradigm is orthogonal to the multi-embedding method, and these two methods can be seamlessly combined. We have conducted experiments based on one of the most representative models DCNv2 on the Avazu dataset, with results presented as follows:

• Recommendation performance (AUC) comparison.

  Model\Embedding size	16	16 $\times$ 2	16 $\times$ 4	16 $\times$ 8	16 $\times$ 10
  DCNv2 - DIS	0.79282	0.79402	0.79434	0.79539	0.79577
  DCNv2 - GEN	0.79342	0.79469	0.79534	0.79599	0.79617

  We can observe that, the generative DCNv2 with multi-embedding outperforms discriminative DCNv2 with multi-embedding.

• Embedding spectrum analysis. We have illustrated the spectrum of DCNv2 (DIS), DCNv2 (GEN), and DCNv2(GEN + MultiEmbedding) in https://anonymous.4open.science/r/ICML2025-1748/supp/collapse.png. The results show that:

  • Embedding collapse of DCNv2 (DIS): The spectrum curve of DCNv2 (DIS) exhibits a dramatic decay after the singular value index 250, which indicates a collapsed embedding space.
  • Embedding robustness of DCNv2 (GEN): Different from DCNv2 (DIS), the spectrum curve of DCNv2 (GEN) does not exhibit the abrupt decay. Instead, they decline slowly, indicating a more balanced embedding space.
  • Multi-embedding alleviates collapse: DCNv2(GEN + MultiEmbedding) can lead to a slightly slower decline rate than DCNv2 (GEN), which indicates a more robust embedding space.

We compute the Information Abundance[1] (IA) values of these 3 variants respectively: 9.0082, 13.5031, 13.6999. A higher IA value indicates a less-collapsed embedding space. The IA results are consistent with the embedding spectrum figure. These spectra and IA results both demonstrate that our method and multi-embedding can be effectively combined to achieve both better performance and less collapse.

[1] Guo, Xingzhuo, et al. On the Embedding Collapse when Scaling up Recommendation Models. ICML, 2024.

Typos

Response to "Other Comments Or Suggestions"

Thank you for your corrections and we will modify the corresponding symbols accordingly. In addition, we will further review the entire manuscript to revise any potentially unclear sections and enhance the overall clarity of our work.

We thank the reviewer for their comments. We have addressed the comments in the rebuttal below.

Clarification on the Motivation.

Response to Point 1 of "Claims And Evidence" & "Methods And Evaluation Criteria".

We'll elaborate more on our claim. Our work is mainly inspired by the Interaction-Collapse Theory, that is, direct interactions between ID embeddings in existing CTR models with cross networks lead to dimensional collapse. We resolve this challenge by shifting from the discriminative paradigm that involves direct interactions between raw ID embeddings to a generative paradigm that constructs new embeddings by a decoder network and interacts the constructed embeddings with the raw ID embeddings.

We don't aim to claim the entire feature interaction paradigm as flawed, and we apologize of we made such unintentional and misleading claim. As researchers in this area, we appreciate the promising progresses in the last decade. In particular, we'd like to recognize the positive impact of cross network and DNN as follows, and will add them in the revised version.

The cross network, especially CrossNet in DCN V2, has been proved effective in CTR prediction. A recent work [1] validates that its cross function mitigates dimensional collapse to some extent via field-pair-wise transformation matrix. However, we find that our generative variants can further improve the embeddings' dimensional robustness (Fig 3.c).

Regarding deep networks, DNNs can also mitigate dimensional collapse compared to direct cross networks. In fact, we have a submited paper studying DNNs in feature interaction models from this perspective, showing that non-linear activations greatly improve dimensional robustness. We will add this discussion to the revised version.

[1] Towards Unifying Feature Interaction Models for Click-Through Rate Prediction. 2024.

Concerns about the Cost-effectiveness.

Response to Point 2 of "Claims And Evidence".

We thank the reviewer for pointing out the cost-effectiveness trade-off. In industry CTR prediction, even a 0.1% AUC lift is considered significant [1]. Our analysis shows the ROI (GMV lift/computation cost) for several scenarios far exceeds our release threshold (typically in the dozens; we omit the exact value since it's a commercial secret). Since February, three generative models have passed performance/cost reviews and been fully deployed.

[1] FuxiCTR: An Open Benchmark for Click-Through Rate Prediction. 2020.

Discussion on Essential References.

Response to "Relation To Broader Scientific Literature" & "Essential References Not Discussed".

Thanks for your valuable suggestion. We will specifically discuss these methods in the Related Works section of the final paper. We list the main differences as follows due to space limits:

Our paradigm differs from these works in the sense that we aim to tackle the dimensional collapse issue due to the direct interaction of ID embeddings. We argue that the above-mentioned related works can't achieve this by refinement, denoising or adopting a GNN architecture.

We empirically compared our paradigm with several representative feature refinement models, with results as follows. We observed that some models outperform the discriminative DCN V2 models, but still underperform our generative model.

We also studied the singular spectrum and found that they can mitigate the dimensional collapse on the tail singular valuescompared to the vanilla discriminative DCN V2. However, our generative model leads to more robust values on all dimensions. Refer to the spectrum analysis in https://anonymous.4open.science/r/ICML2025-1748/supp/refinement.png.

[1] MCRF: Enhancing CTR Prediction Models via Multi-channel Feature Refinement Framework.

[2] Enhancing CTR prediction with context-aware feature representation learning.

[3] Fi-gnn: Modeling feature interactions via graph neural networks for ctr prediction.

Clarification of the Discussion Parapragh.

Response to "Other Comments Or Suggestions:"

We will elaborate on this paragraph and revise the manuscript accordingly with the following:

Dimensional Collapse of Raw ID Embedding Interaction. The embeddings of some fields may only span a low-dimensional space due to various reasons, such as the low cardinality of this field. For example, the embeddings of the gender field with values of Male, Female, and Unknown can only span a 3-dimensional space. According to the Interaction-Collapse-Theory, the interactions with these low-dimensional field embeddings may lead to the dimensional collapse of the embeddings of the other fields.