Adaptive Gradient Masking for Balancing ID and MLLM-based Representations in Recommendation

Dear reviewer fBa2,

We sincerely thank you for your careful reading and thoughtful feedback. We truly appreciate the time and effort you dedicated to evaluating our work. Below, we address each of your concerns in detail and provide clarifications where necessary.

Q1: No alignment between item MM and ID represetations

User ID and MM representations are both derived from users’ historical behavior sequences, and thus implicitly encode similar semantics — namely, the user’s preferences. Aligning them helps bridge sparse collaborative signals with rich content-derived features, especially in cold-start scenarios.

In contrast, item multimodal features represent "intrinsic, static content semantics", while item ID embeddings are learned via collaborative signals and are continually updated based on user interactions. These two representations capture fundamentally different information: the former reflects what the item is, the latter who it appeals to. Aligning them may dilute the semantic integrity of content features. Therefore, we align user ID and MM but deliberately keep item ID and MM disentangled.

Q2: Latent Space Between $e^{mm}_u$ and $e^{mm}_v$

In our model, $e^{mm}_u$ is generated from historical behavior sequences consist of $e^{mm}_v$ , which ensures that they are naturally embedded in a shared latent space compatible for downstream interaction modeling. While the UEG module includes an alignment loss to guide $e^{mm}_u$ toward the ID-based user embedding space, we note that the alignment loss is assigned a small weight in the total objective. This design choice ensures that the core semantic structure of $e^{mm}_u$ remains largely intact and does not significantly deviate from the original MM space.

Furthermore, we provide evidence in Table 6: when we remove the alignment loss, the quality of MM representations drops significantly in the zero-shot recommendation task, which is largely determined by the semantic compatibility between $e^{mm}_u$ and $e^{mm}_v$. This degradation suggests that adding alignment process between $e^{mm}_u$ and $e^{id}_u$ does not significantly distort the originally consistent semantic space of $e^{mm}_u$ and $e^{mm}_v$, otherwise adding this alignment process would only result in performance loss in MM representation quality in the zero-shot recommendation task. Even if the alignment might slightly influence the original semantic space of $e^{mm}_u$, the benefit of better aligning user representations outweighs any potential disruption, leading to improved recommendation quality.

Q3: Choice of L2 Norm for Alignment Loss in Equation (3)

To be honest, we chose the L2 norm as the alignment loss just because it provides a simple, smooth, and well-behaved metric that equally penalizes deviations across all embedding dimensions. This decision aligns with many prior works in embedding alignment, such as AlignRec [1].

Q4: Unclear description in Lines 213–219

We apologize for the confusion and clarify here. The “non-uniform adaptive sampling” refers to an importance-based random selection of parameters during training. Concretely, we bias the sampling toward those with higher scores in distribution $\pi$ when we choose which parameters to update during the backforward propagation. In other words, parameters carrying more information are updated more frequently. Therefore, The phrase “directs our focus toward parameters carrying richer information” means that we tend to update these important parameters more often (instead of preserve them).

The second phrase, “probabilistic sampling extends coverage to a broader range of parameters” confused you because the uniform sampling in this occasion is not clear. We apologize for the confusion here. Actully, AGM is fundamentally designed to prioritize the update of more important parameters, and in this context, our “probabilistic sampling” is intended as an improvement over traditional top-k masking strategies commonly used in importance-based sparse updates. Therefore, the sentence“probabilistic sampling extends coverage to a broader range of parameters” means compared to top-k methods, our sampling approach achieves broader coverage of the parameter space.

Q5: $\eta$ in Lines 661 and 667

To ensure the expected decrease in loss, we consider the worst-case alignment where the bias term $b(\theta)$ is in the same direction as the loss gradient $\nabla \mathcal{L}(\theta)$, i.e., $b = k \nabla \mathcal{L}$.
In this case, the inner product and norm simplify as:

By substituting into Eq. (32) (second-to-last line) and neglecting $\sigma$, we obtain:

Dividing both sides by $(1 + k)\||\nabla \mathcal{L}\||^2$, we get:

By conservatively taking $k = 1$, we obtain the sufficient condition:

This justifies the learning rate constraint adopted in the analysis.

The upper bound in Line 661, $\eta \leq \frac{1}{(1 + \nu)L}$, ensures descent of the expected loss under worst-case bias alignment.
In Line 667, this setting serves two purposes:

1. It satisfies the sufficient condition for convergence established in Line 661.
2. It follows the standard choice in stochastic optimization to achieve an $\mathcal{O}(1/\sqrt{T})$ convergence rate for the average iterate. This rate matches the optimal bound under noisy gradient conditions, and the $1/(1 + \nu)$ scaling further accounts for the additional bias variance introduced in our problem setting.

Q6: Confusing notation

We sincerely apologize for the confusion this may have caused. Regarding the specific points you raised: (i) The variable $x_i$ in Equation (8) refers to the current training sample, and corresponds to the tuple ($u_i$, $v_i$). We will clarify this in the revised version. (ii) The notation alternates between indexed and unindexed forms (e.g., $u_i$, $v_i$ vs. $u$, $v$), which may lead to ambiguity. In the revision, we will ensure that all sample-specific variables are consistently indexed, and clarify where variables represent arbitrary users/items versus specific samples.

We appreciate your attention to these important details and will revise Section 3 accordingly to improvev readability and consistency.

Q7: Writing issues

We sincerely thank you for pointing out the writing and formatting issues, including the missing definition of “MM” (Line 4), inconsistent capitalization (Line 50), and other stylistic inconsistencies. We apologize for these oversights. These corrections will be carefully incorporated into the revised version to ensure clarity, consistency, and overall presentation quality.

Finally, we truly appreciate all your valuable feedback. We hope our responses have addressed your concerns, and we would be grateful if you could take these clarifications into account in your overall evaluation. Thank you again for your time and insights — we look forward to any further comments you may have.

[1] AlignRec: Aligning and Training in Multimodal Recommendations

Dear reviewer a6WN,

We sincerely thank you for your thoughtful feedback and positive score. We truly appreciate the time and effort you dedicated to evaluating our work. Below, we address each of your concerns in detail and provide clarifications where necessary.

Q1: Time complexity analysis

First, we clarify that our method adopts a two-stage framework, where the MLLM is fine-tuned offline to extract multimodal features prior to training the downstream model. Since this step is performed once and does not affect online training or inference, so it is reasonable to omit a detailed time complexity analysis for this stage.

We provide a time complexity analysis of the AGM training process below:

For a batch of size $|\mathcal{B}|$, the training time complexity of AGM consists of three components:
1.forward propagation: $\mathcal{O}(|\mathcal{B}|)$.

2.Fisher Information Matrix (FIM) computation: $\mathcal{O}(|\mathcal{B}| \cdot |\mathcal{P}|)$. Here, $\mathcal{P}$ denotes the selected parameters for update, $\theta$ denotes the full parameter set, $|\mathcal{P}| \ll |\theta|$.

3.importance-based sampling via heap-based sorting: $\mathcal{O}(|\mathcal{P}| \log |\mathcal{P}|)$.

Thus, we get the total training complexity per batch: $\boxed{\mathcal{O}(|\mathcal{B}| + |\mathcal{B}| \cdot |\mathcal{P}| + |\mathcal{P}| \log |\mathcal{P}|)}$.

Although AGM introduces lightweight sampling and importance estimation overhead compared to standard training, its overall complexity remains near-linear with respect to the batch size. Therefore, AGM retains comparable scalability to standard models.

At inference stage, AGM incurs no additional complexity compared to conventional models, since neither FIM computation nor sampling is involved. The inference complexity remains $\mathcal{O}(|\mathcal{B}|)$, consistent with standard recommendation architectures.

Q2: More recommendation-related evaluation metrics
In our current experiments, we adopt AUC and LogLoss as evaluation metrics. because our focus is on click-through rate (CTR) prediction, where these two metrics are widely used. We include Recall@10 and Recall@20 results in the table below as a complementary evaluation:

Q3: More details of industrial application

We deploy our method in a large-scale short-video platform in China, which serves over 300 million DAUs. This application scenario involves high-throughput real-time recommendations across various traffic entry points. Given the immense volume and diversity of video content, real-time efficiency and embedding consistency are critical for system performance. Our proposed pipeline effectively addresses these challenges: by caching item-level multimodal embeddings extracted by the fine-tuned MLLM, we significantly reduce computational overhead during training and inference, enabling scalable deployment under stringent latency constraints. This architecture has been verified to support stable online serving and model updates on a weekly basis.

Q4: Robustness to low-quality multimodal representations

We agree that the quality of multimodal features can significantly influence recommendation performance. As discussed in Line 688 of the paper, the effectiveness of AGM depends in part on the quality and diversity of multimodal data.

However, AGM is designed to remain robust even when one modality is degraded or missing. Specifically, the model includes an ID-based embedding branch and an alignment loss between multimodal and ID-based representations. This setup enables AGM to fall back on reliable ID features when multimodal features are noisy or incomplete, avoiding overfitting to poor-quality inputs.

To further validate AGM’s robustness to low-quality multimodal features, we conducted ablation experiments using the three types of multimodal representations described in Table 6—(i) the base MLLM encoder without fine-tuning, (ii) our fine-tuned MLLM encoder, and (iii) our variant without the alignment loss—to evaluate the robustness of AGM to degraded or unaligned multimodal inputs.

Finally, we truly appreciate all your valuable feedback. We hope our responses have addressed your concerns, and we would be grateful if you could take these clarifications into account in your overall evaluation. Thank you again for your time and insights — we look forward to any further comments you may have.

Dear Reviewer 2Y2o,

We sincerely thank you for your thoughtful feedback and positive score. We truly appreciate the time and effort you dedicated to evaluating our work. Below, we address each of your concerns in detail and provide clarifications where necessary.

Q1: Computational cost of fine-tuning

We thank the reviewer for raising the concern regarding the computational cost of fine-tuning large MLLMs. As noted in Lines 575–576 of our paper, we adopt Qwen2vl-2b as the backbone model and fine-tune it for 5 epochs across four datasets, using a batch size of 128 on 4 A100 GPUs. This provides a concrete reference for the resource requirements in our setting.

Additionally, we emphasize that our method adopts a two-stage franework: the MLLM is fine-tuned offline in the first stage to extract high-quality multimodal embeddings, while the downstream recommendation model is trained separately in the second stage. For practical deployment, we cache all item multimodal embeddings, thus eliminating the need for repeated MLLM inference during training or inference. As a result, the computational burden of fine-tuning is incurred only once offline and has minimal impact on online efficiency.

Q2: Evaluation of more datasets

To further evaluate the generalizability of our method beyond e-commerce and short video domains, we include the MIND-small dataset, a widely-used benchmark for news recommendation.

The table below summarizes the performance of AGM*, AlignRec and GUME on MIND-small:

Q3: Comparison with more MLLM methods

We additionally include LIRDRec [1] as a baseline in our evaluation. LIRDRec [1] is designed to learn item representations directly from multimodal content. The following table summarizes the Recall@10 and Recall@20 results across three datasets, which illustrates that AGM* performs on par with or slightly better than LIRDRec and other baselines.

Q4: Quantification of AGM*'s improvement beyond AGM

We provide a clearer quantification of AGM*’s improvement over AGM in Table 2. For example, on the Amazon Baby dataset, AUC increases from 0.6852 (AGM) to 0.6864 (AGM*), and LogLoss decreases from 0.6683 (AGM) to 0.6656 (AGM*). Similar gains are observed across other datasets. We apologize that Table 2 may not have illustrated this improvement clearly enough. In the revised version, we will include a delta row in the table to highlight AGM*’s performance gains more explicitly.

Finally, we truly appreciate all your valuable feedback. We hope our responses have addressed your concerns, and we would be grateful if you could take these clarifications into account in your overall evaluation. Thank you again for your time and insights — we look forward to any further comments you may have.

[1] Learning Item Representations Directly from Multimodal Features for Effective Recommendation

Thank you very much for your response. I have read your response to my comments and other reviewers. The authors address my concerns.

Dear reviwer QngB,

We sincerely thank you for your thoughtful feedback and positive score. We truly appreciate the time and effort you dedicated to evaluating our work. Below, we address each of your concerns in detail and provide clarifications where necessary.

Q1: Supplement code

Due to the NeurIPS 2025 official policy, updating the submitted github repository is not permitted during the rebuttal phase. We will update the repository to include the missing multimodal representation learning module immediately after the rebuttal process.

Q2: Long-tail evaluation

We have added additional experiments to directly evaluate our method’s performance on long-tail items. Specifically, for each dataset, we sort items by their total number of user interactions in descending order. We then partition the items into the following frequency bands:

• High-frequency: top 20% most popular items.
• Mid-frequency: items ranked in the 20%–50% range.
• Low-frequency: the remaining 50% of items.

We present the results in the table below. We select GUME as the primary baseline and conduct evaluations on three datasets. The results show that AGM achieves significant performance improvements on low-frequency items, demonstrating its effectiveness in addressing the long-tail challenge.

Q3: Performance improvement seems relatively small

In the CTR predictions, an improvement as small as 0.001 in AUC is generally considered significant, as even minor gains in offline metrics can lead to substantial performance boosts in online serving scenarios [1]. For example, in [2], the Wide & Deep model achieved only a 0.275% AUC improvement over logistic regression in offline evaluation, but led to a 3.9% increase in online CTR.

Q4: Fairness of the evaluation

We sincerely thank you for your thoughtful question. We understand the concern regarding the fairness of the evaluation and offer the following clarifications:

• Our evaluation protocols follow AlignRec exactly. We clarify that the evaluation protocols used in table 6 follow AlignRec exactly, in both the dataset split and the Intermediate Evaluation Protocol setup. Consistent with AlignRec, we compare the representation quality of different multimodal encoders under the same user behavior-based evaluation.

• AlignRec itself also fine-tunes its MLLM. Different models adopt different strategies for using multimodal features. AlignRec itself includes a fine-tuned MLLM encoder (MMEnc) as part of its pipeline. Similarly, in Table 6, we report results from (i) the base frozen MLLM encoder, (ii) our fine-tuned version, and (iii) our model without the alignment loss. AlignRec’s Table 3 makes an analogous comparison between (i) Amazon, (ii) CLIP, and (iii) their MMEnc, which also shows significant gains from fine-tuning.

• The key difference lies in the fine-tuning strategy employed. Given that both methods generate comparable and consistent multimodal embeddings, the performance gains achieved by AGM over AlignRec can be attributed to the effectiveness of our subsequent representation learning and aggregation mechanisms. Table 2 shows that AGM still outperforms AlignRec, indicating that our performance gains cannot be solely attributed to MLLM fine-tuning. Rather, they result from AGM’s ability to dynamically integrate ID and multimodal features.

Finally, we truly appreciate all your valuable feedback. We hope our responses have addressed your concerns, and we would be grateful if you could take these clarifications into account in your overall evaluation. Thank you again for your time and insights — we look forward to any further comments you may have.

[1] Guo H, Tang R, Ye Y, et al. DeepFM: a factorization-machine based neural network for CTR prediction[J]. arXiv preprint arXiv:1703.04247, 2017.

[2] Cheng H T, Koc L, Harmsen J, et al. Wide & deep learning for recommender systems[C]//Proceedings of the 1st workshop on deep learning for recommender systems. 2016: 7-10.