Be Confident: Uncovering Overfitting in MLLM Multi-Task Tuning

Dear Reviewer XwCL:

Thank you very much for your valuable comments and constructive feedback. Below, we carefully address each of your concerns point-by-point, providing detailed explanations and additional evidence to clarify our approach and validate its effectiveness.

Q1: Limitation Discussion (Other Strengths And Weaknesses & Questions For Authors)

A1: Open-response and fixed-choice tasks reflect two complementary aspects of Multimodal Large Language Model (MLLM) capabilities: the ability to generate free-form responses to open-ended questions, and the ability to select correct answers from predefined options. These two task types are widely recognized and are also included in the pre-training datasets construction. Solely tuning on fixed-choice tasks would degrade into the single task optimization and limits model application scope. In contrast, open-response tasks are crucial for unlocking MLLM ability to generate free-form expressions, which go beyond the limitations of traditional classification. We have provided a discussion in Page 5. We will provide a detailed limitation discussion in our revised manuscript. Thanks for your valuable suggestions!

Q2: Typos Errors (Other Strengths And Weaknesses & Questions For Authors)

A2: Thanks for your tips. We will fix the typos and check grammar and markup issues in our revised manuscript！

Q3: Rationale for Normal Branch Detach (Other Strengths And Weaknesses & Questions For Authors)

A3: In Eq. 5, we aim to enhance the visual behavioral effect to improve prediction robustness and reduce reliance on language priors. To achieve this, we encourage the noisy prediction confidence $\tilde{\mathcal{I}}$ to align with the normal prediction confidence $\mathcal{I}$, enforcing a one-way alignment constraint. Therefore, we detach the normal branch to prevent it from being influenced by the noisy branch during optimization. We will provide a detailed rationale discussion in our paper. Thanks.

Q4: Clear Keyword Definition (Other Comments Or Suggestions)

A4: Open-Response task means the task of generating a response to a given question such as image captioning dataset. Fixed-Choice task denotes the task of selecting one of several options from the candidate list, e.g., visual question-answering dataset. We will clarify the definition in our revised manuscript. Thanks for your suggestion!

Dear Reviewer SJh5:

Thank you for your thoughtful review and for raising key concerns regarding our work. We aim to address your concerns in our detailed responses below, hoping to provide clarity and demonstrate the effectiveness of our proposed approach.

Q1: Conceptual Discussion (Other Strengths And Weaknesses)

A1: In the context of Multimodal Large Language Model (MLLM) tuning on downstream tasks, empirical evidence shows a tendency toward overfitting on the target distribution. This is primarily due to the mismatch between the model large-scale capacity and the limited size of domain-specific samples. Existing investigations either introduce stiff penalty regularization terms or design partial update masks. However, such approaches typically focus on single-task adaptation and apply uniform anti-overfitting objectives across all examples, without accounting for task-specific differences. Under multi-task settings, different task types often exhibit inconsistent fitting behaviors. Our analysis reveals that open-response tasks are particularly prone to overfitting, largely due to their heightened dependence on textual modalities. To address this, we propose a novel approach that mitigates overfitting on the target distribution by enforcing confidence alignment from noisy visual inputs to their corresponding clean visual patterns. We will provide a detailed conceptual explanation for the proposed approach in our revised manuscript to highlight our novelty. Thanks for your valuable suggestions!

Q2: Figure 3 Explanation (Other Strengths And Weaknesses)

A2: Figure 3 illustrates the performance under both normal and noisy visual input for different methods. Existing methods show minimal performance gap regarding visual quality, which reflects the restricted affect from the visual branch and presents a high reliance on the textual prior information to generate the corresponding output response. We will provide a detailed explanation for Figure 3 in our revised manuscript. Thank you for your helpful suggestion.

Q3: Dataset Construction (Other Strengths And Weaknesses)

A3: For fixed-choice tasks (ScienceQA and IconQA), we use the textual prompt: “Answer with the option’s letter from the given choices directly.” For open-response tasks (Flickr30k and COCO-Cap), we collect five manually written instructions and randomly sample one as the prompt for each caption, as follows:

• "Describe the image as simply as possible with a sentence or phrase"
• "Give a brief summary of what you see"
• "Provide a short description of the image"
• "Write a short description for the image"
• "Briefly describe the content of the image"

We will provide a clear description of the dataset construction procedure in our revised manuscript. We will release the corresponding training and test JSON files in the final version. Thank you for your valuable suggestions!

Dear Reviewer ELAr:

We sincerely thank you for your valuable feedback and hope our responses adequately address your concerns and merit a score revision.

Q1: Theoretical Claims of NRCA and Its Connection to Visual Representation

A1: Denote $I$ and $\tilde{I}$ as the average confidence over ground-truth tokens under clean and perturbed visual inputs (see Eq 4). Let $\tilde{z}_t$ and $z_t$ be the predicted logits at position $t$, and let $\sigma(\cdot)$ denote the softmax function.

The confidence alignment objective of $\mathcal{L}_{\text{NRCA}}$ is:

$$\min \left| 1 - \frac{\tilde{I}}{I} \right| = \min \left| \frac{1}{T} \sum_{t=1}^T \frac{\sigma _{y_t}(z_t)-\sigma _{y_t}(\tilde{z}_t)}{I} \right|.$$

Since $I$ is detached from the computational graph, the optimization reduces to:

$$\min\left| \sum_{t=1}^T \sigma _{y_t}(z_t)-\sigma _{y_t}(\tilde{z}_t) \right|.$$

Assuming $\sigma _{y_t}(z_t) \geq \sigma _{y_t}(\tilde{z}_t)$ for all $t$ (a natural assumption since clean inputs typically yield higher confidence), the objective becomes:

$$\min \sum_{t=1}^T \left( \sigma _{y_t}(z_t)-\sigma _{y_t}(\tilde{z}_t) \right).$$

This encourages the ground-truth token probabilities under perturbed inputs to approach those under clean inputs, i.e.

$$\sigma _{y_t}(\tilde{z}_t) \to \sigma _{y_t}(z_t).$$

Since $\tilde{z}_t=g(\phi(f(\tilde{x}_v)),h_t)$, the loss encourages the learnable modules ($\phi$ and $g$) to produce robust visual features that preserve confidence on ground-truth tokens under perturbations, thereby improving visual representation quality.

Q2: Theoretical Connection Between Language Priors and Open-Response

A2: We consider token-level prediction in autoregressive multimodal generation, where the model learns the probability of a target sequence $y=(y_1, \dots, y_T)$ given a visual input $x^v$ and textual prompt $x^t$. The objective is to minimize the negative log-likelihood $L_{\text{CE}}.$

In open-response tasks such as image captioning (eg Flickr), the output is a multi-token sentence of length $T=k$, with autoregressive factorization:

$$p(y \mid x^v, x^t)=\prod_{t=1}^k p(y_t \mid y_{\lt t}, x^v, x^t).$$

Due to the richness and uniqueness of each output, the model may overfit by memorizing input-output mappings and relying on linguistic priors in $y_{\lt t}$ rather than visual cues.

In contrast, fixed-choice tasks (e.g., IconQA) involve single-token outputs ($T=1$) from a small label set:

$$p(y \mid x^v, x^t)=p(y_1 \mid x^v, x^t),$$

where prefix conditioning is absent and language priors are less effective. The model must therefore rely more on visual input, reducing overfitting.

This analysis complements Appendix C and the observations in Fig 6, further clarifying the connection between language priors and overfitting in open-ended tasks.

Q3: Essential References Discussion

A3: Prior work on modality bias shows that MLLMs often over-rely on textual priors, causing visually misaligned hallucinations[1]. Contrastive decoding compares token distributions across views but adds inference cost[2]. Our method aligns token-level confidence during fine-tuning to improve understanding.

For vision-language alignment, existing methods either build diverse instruction data or use reinforcement learning with designed reward signals, both requiring costly annotation or tuning[3]. Our method avoids these overheads and integrates directly into standard fine-tuning.

[1] Multi-modal hallucination control by visual information grounding,CVPR2024

[2] Mitigating Object Hallucinations in LVLMs through Visual Contrastive Decoding,CVPR2024

[3] Visual-RFT: Visual Reinforcement Fine-Tuning,arXiv2025

Q4: Random Mask Tuning Effectiveness Discussion

A4: Random Mask Tuning applies stochastic masks to parameters, injecting noise to reduce reliance on specific subnetworks and implicitly regularize by limiting parameter co-adaptation. While effective on smaller models (VILA3B), its performance drops on larger ones (LLaVA7B), as shown in Tab 4, suggesting its effect comes from constrained optimization rather than addressing language prior dominance. In contrast, our method perturbs visual inputs and aligns token-level confidence with the clean visual branch, enhancing visual grounding and ensuring consistent performance across model scales. We will include a detailed analysis in the manuscript.

Q5: Visual Attention Patterns

A5: Due to limited space, please refer to Q2 in Response to Reviewer H4RG.

Q6: Fixed-Choice Tasks Performance

A6: The phenomenon of modest gain in fix-choice task is also present in several counterparts designed to alleviate overfitting. Though some of them perform marginally better on fixed-choice tasks, they tend to incur trade-offs and underperform on open-response tasks. By enhancing visual robustness through confidence alignment, our method achieves a better trade-off and consistently surpasses all baselines in overall multi-task performance.

Dear Reviewer H4RG:

Thank you very much for your affirmation of our work, as well as the insightful concerns and questions you have raised. We have carefully considered each comment and provided responses.

Q1: Multi-Task Specialization Definition (Claims And Evidence)

A1: Multi-Task Specialization refers to fine-tuning a pre-trained model for a specific downstream application, which normally has a limited amount of data, compared with the pre-training data scale. For example, the authors of LLaVA claim that it involves $150k$ unique language-image instruction-following samples. However, the specialization application samples are much more limited, i.e., $10k$ for target task. Thus, tuning on the downstream distribution tends to lead to overfitting on the specialization task. And we point out that open-response tasks easily appear the overfitting behavior within the multi-task specialization. We attribute this degradation to language prior reliance and propose Noise Resilient Confidence Alignment to enhance the effect of visual cues in the token prediction. Therefore, our method is applicable to Multimodal Large Language Model (MLLM) training itself but benefits more in the specialization tuning process. We will provide a detailed multi-task specialization definition in our revised manuscript. Thanks for your valuable suggestions!

Q2: Language Prior Reliance Visualization Analysis (Experimental Designs Or Analyses & Questions For Authors)

We assume that for Multimodal Large Language Model (MLLM), input tokens include System Tokens, Prompt Tokens and Visual Tokens and appear different contribution for the prediction output. Thus, we utilize the attention map between the first output token and the input token under the noisy visual input to visualize the contribution of different input token. Besides, recent works [1,2,3] have shown that the MLLM tend to extract object information from the image at the middle layers. Thus, we plot the all layers and middle layers attention allocation for the first output token in the following Table. The results reveal that our method allocates more attention weights on the visual cue. Besides, in Figure 3 (on Page 5), Full Fine-Tuning shows a limited performance gap between noisy and normal visual signals, revealing the reliance on language prior rather than understanding the visual information. Therefore, both the attention allocation analysis and performance comparison demonstrate that our method effectively reduces reliance on language priors and enhances the overall performance of the model.

Table: Attention allocation and performance comparison with relative methods. We conduct the evaluation on the Flickr30k with noisy visual input. We tuning on the Flickr30k+ScienceQA datasets based on LLaVA architecture.

Attributed to rebuttal format constraints, we will include attention map in our final version. Thanks for valuable suggestions!

[1] EAZY: Eliminating Hallucinations in LVLMs by Zeroing out Hallucinatory Image Tokens, arXiv, 2025

[2] From Redundancy to Relevance: Information Flow in LVLMs Across Reasoning Tasks, NAACL, 2025

[3] Cross-modal Information Flow in Multimodal Large Language Models, CVPR, 2025

Q3: Table Design (Other Comments Or Suggestions)

Thank you for your suggestion. We will reorganize the formulation of Table 2 to make it clear and easy to understand!