Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning

Dear Reviewer mGiB:

Thank you for your valuable comments and constructive feedback. Below, we carefully address each of your concerns point-by-point, providing detailed explanations and additional evidence to clarify our approach.

Q1: Memory Complexity (Other Strengths And Weaknesses)

A1: For our method, we utilize the pre-trained weights and the current gradient to evaluate parameter importance for both generalization and specialization, which brings a certain memory cost increase with existing methods. However, our method achieves stable performance across different tuning scenarios, as shown in the following Table. We will add the memory complexity comparison in the final version.

Q2: Parameter Modification Scale (Other Strengths And Weaknesses)

A2: Sorry for the misunderstanding. We plot the parameter modification scale in the following Table. We will add the parameter modification scale and notation table in the final version.

Table: Comparison with relative methods. $M$ means selected parameters. Performance is the harmonic mean of generalization and specialization accuracy.

Q3: Font size and Layout (Other Strengths And Weaknesses)

A3: Thanks for your advice. We will adjust the font size and layout to make it more readable!

Q4: Advantage Discussion for Proposed Method (Other Strengths And Weaknesses)

A4: Our method selects and optimizes the relative downstream important elements to adapt towards the downstream specialization while maintaining the generalization ability. Thus, our method does not bring additional architecture modification and does not couple with a specific model architecture. We will update Table 1 and provide a more detailed advantages discussion for our method in the final version.

Q5: Explanation and Contribution for Selected Datasets (Other Strengths And Weaknesses)

A5: The datasets OKVQA, TextVQA, GQA, and OCRVQA are obviously mentioned as the training datasets in the pre-training stage, making them appropriate benchmarks to evaluate multimodal large language models’ (MLLMs) generalization across diverse tasks. OKVQA examines external knowledge and common-sense reasoning, TextVQA and OCRVQA test understanding of embedded textual information, and GQA assesses compositional and logical reasoning. Together, these datasets comprehensively evaluate MLLMs’ generalization across different reasoning types and practical scenarios. We will provide a detailed explanation of the selected datasets in the paper. Thanks for your comments!

Q6: Typos (Other Comments Or Suggestions)

A6: Thanks for your advice. We will fix the typos and check the final version carefully.

Dear Reviewer CY1z:

Thank you for your insightful review and valuable feedback. Below, we address your key concerns in detail, aiming to clarify and demonstrate the effectiveness of our proposed approach.

Q1: Discussion on Incremental Learning (Essential References Not Discussed & Questions For Authors)

A1: Incremental learning focuses on continuously learning new tasks that are distinct from previously learned tasks, with the requirement to classify all observed classes during the testing phase [1,2,3]. Direct optimization solely on the current distribution can result in catastrophic forgetting of previously acquired knowledge. Existing incremental learning methods broadly fall into three categories: parameter isolation methods [4,5], regularization-based methods [6,7], and replay-based methods [8,9]. In this work, we investigate catastrophic forgetting within the multimodal large language model (MLLM) tuning process, aiming to mitigate the degradation of generalization and enhance specialization performance.

[1] Class-incremental learning: A survey, IEEE PAMI, 2024

[2] A continual learning survey: Defying forgetting in classification tasks, IEEE PAMI, 2021

[3] A comprehensive survey of continual learning: theory, method and application, IEEE PAMI, 2024

[4] Progressive Neural Networks, arXiv, 2016

[5] Dense network expansion for class incremental learning, CVPR, 2023

[6] Overcoming Catastrophic Forgetting in Neural Networks, PNAS, 2017

[7] Memory aware synapses: Learning what (not) to forget, ECCV, 2018

[8] icarl: Incremental classifier and representation learning, CVPR, 2017

[9] Co-transport for class-incremental learning, ACM MM, 2021

Q2: Downstream Dataset Introduction (Other Strengths And Weaknesses)

For downstream evaluation, we selected datasets (Flickr30k, COCO-Caption, IconQA, ScienceQA) that exhibit limited zero-shot performance or were excluded from the pre-training stage. Flickr30k and COCO-Caption focus on text generation tasks within general domains. Specifically, Flickr30k emphasizes everyday activities and events, while COCO-Caption features more complex, diverse scenes involving multiple objects and intricate interactions. Both IconQA and ScienceQA involve multiple-choice question answering tasks. IconQA evaluates abstract and symbolic reasoning using visual questions based on icons and diagrams, whereas ScienceQA assesses multimodal scientific reasoning in educational contexts, incorporating textual and visual content from disciplines such as physics, chemistry, biology, and general science. We will incorporate a detailed introduction to these downstream datasets in our revised paper.

Q3: Writing Clarity and Accessibility (Other Comments Or Suggestions & Questions For Authors)

A3: Thank you for highlighting this. We will simplify the formulations and enhance the clarity of our writing in the revised manuscript.

Q4: Relative forgetting and increase metrics (Questions For Authors & Other Strengths And Weaknesses)

A4: Thank you for the valuable suggestion. We will incorporate relative forgetting and increase metrics into Tables 4 and 5 of the revised manuscript, explicitly reporting both generalization forgetting and specialization improvement to provide a clearer evaluation of method effectiveness.

Thank you once again for your constructive feedback and valuable suggestions！

Dear Reviewer mmMS:

Thank you very much for your affirmation of our work, as well as the insightful concerns and questions you have raised. We have carefully considered each comment and provided responses.

Q1: Value Range Boundary and Normalization Operation Rationale （Theoretical Claims） A1: Sections 2a–2c measure parameter importance for generalization by leveraging the pre-trained weight magnitudes. Specifically, step 2a computes the absolute values of pre-trained weights, yielding a range of [0,$+\infty$). Step 2b normalizes these absolute values to a standardized range of ($-\infty$,$+\infty$). Finally, step 2c rescales the normalized values into the range (0,1) using a sigmoid function. Together, these steps systematically rank parameters according to their importance for generalization, facilitating a meaningful comparison with parameters important for specialization. We will provide detailed explanations and discuss the rationale behind these operations in the revised manuscript. Thanks for your advice!

Q2: Depth Analysis on Full-FT (Experimental Designs Or Analyses & Other Strengths And Weaknesses)

A2: Full-FT directly updates all parameters to adapt to the downstream distribution. However, specialized downstream applications typically have limited data compared to large-scale pre-training datasets. Thus, Full-FT often leads to increased overfitting, especially when training for more epochs, resulting in performance degradation on downstream tasks. We will include a detailed analysis of Full-FT performance in the revised manuscript. Thank you for tips!

Q3: Differences between Regularization-based Optimization and Partial-based Updating solutions (Supplementary Material)

A3: Regularization-based optimization and partial-based updating represent two distinct approaches to addressing catastrophic forgetting in multimodal large language model (MLLM) tuning. The regularization-based approach introduces stiffness regularization terms, carefully balancing generalization and specialization through controlled penalization. In contrast, partial-based updating selects only a subset of parameters to update, preserving the rest. Our work adopts the partial-based updating paradigm, measuring parameter importance for both generalization and specialization, and selectively updating parameters significant for downstream tasks. We will clearly and concisely discuss the differences compared to existing methods in our revised manuscript. Thank you for your valuable suggestion!

Q4: Large Model Merging Paper Discussion (Essential References Not Discussed & Other Strengths And Weaknesses)

A4: Large Model Merging is a promising paradigm that integrates task-specific models into a unified model capable of simultaneously handling diverse downstream tasks [1]. Existing methods typically represent each task expert using downstream-optimized models and rely on various importance metrics to select specific elements for merging [2,3,4]. However, post-merging operations often disrupt the originally optimized parameter space, limiting the specialized capabilities for individual tasks. In our work, we select relative specialization importance elements during the training process, which is compatible with integrates into the training trajectory and squeeze the specialized capabilities into the target parameter space. We will add large model merging works discussion in our final version.

[1] Learn From Model Beyond Fine-Tuning: A Survey, Nature Machine Intelligence, 2024

[2] A Unified View of Delta Parameter Editing in Post-Trained Large-Scale Models, arXiv, 2024

[3] Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models, ICML, 2024

[4] REMEDY: RECIPE MERGING DYNAMICS IN LARGE VISION-LANGUAGE MODELS, ICLR, 2025

Dear Reviewer mG7E:

We sincerely appreciate your time and effort in reviewing our paper. Through the detailed responses outlined below, we seek to fully address your concerns and provide transparency into our proposed approach.

Q1: Essential Reference Discussion and Comparison (Essential References Not Discussed & Other Strengths And Weaknesses & Other Comments Or Suggestions)

A1: We apologize for overlooking the critical reference. Memory Aware Synapses (MAS) [1] addresses catastrophic forgetting in lifelong learning by measuring parameter importance through output sensitivity and applying a regularization penalty to prevent substantial updates to critical parameters. In our experiments, we selected OKVQA as a representative dataset to estimate parameter importance for generalization and compared MAS with different regularization hyperparameters ($\lambda$), as detailed in the table below. The results show that MAS outperforms other regularization-based methods; however, it provides limited trade-offs between generalization and specialization in the context of multimodal large language models (MLLMs). This limitation arises because the regularization terms cannot effectively maintain stable parameter constraints in large-scale models, negatively affecting overall performance. We will expand upon these observations and their implications in our revised manuscript.

Table: Performance comparison with relative methods. We conduct experiments on the VILA architecture and tune on the Flickr30k dataset.

[1] Memory aware synapses: Learning what (not) to forget, ECCV, 2018.

Q2: Distribution Sensitivity Effect on Parameter Importance Difference (Essential References Not Discussed & Other Strengths And Weaknesses & Other Comments Or Suggestions)

A2: Our method leverages pre-trained weights and current gradient updates to separately construct parameter importance rankings for generalization and specialization. In our first module, we identify elements relatively important for downstream tasks based on these rankings. Furthermore, we note that naively updating candidate parameters does not reflect parameter-level variance effectively (Eq. 6). Therefore, we propose the Importance Selection Mask, which dynamically allocates higher update rates to parameters exhibiting greater importance discrepancies, as defined in Eqs. 7-8 and illustrated on page 5. Consequently, our approach dynamically restricts parameter updates when the downstream task distribution closely aligns with the generalization distribution, effectively preserving generalization capability.