Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Data

[W2, Q4] Additional Baselines Following the reviewer's suggestion, we implemented an additional baseline to evaluate our data filtering approach. This baseline follows an iterative process:

1. Initial Round: Explanations are generated by prompting LLaVA directly and used as training data.
2. Subsequent Rounds: The fine-tuned model from each round generates the most probable responses, which serve as training data for the next round without filtering.

This process is repeated for four iterations. The results are shown below:

Our results show that while the baseline (without filtering) achieves initial improvements, its performance stagnates and even declines in later iterations. Additionally, the explanation quality (measured by cognition score (CS)) remains lower compared to our method. Due to time constraints, we completed experiments on three datasets. We expect similar results in the remaining datasets and will report the full-scale analysis in the camera-ready version.

[W3, Q3] Cross-Dataset Transfer Cross-dataset transfer is an intriguing direction for future work, though it is not the primary focus of this paper. Our method targets improvements in domain-specific cognition and explanations. Nevertheless, following your suggestion, we conducted experiments to assess cross-dataset transferability (e.g., training on CUB-200 and evaluating on Stanford Dogs). The results are presented below.

The results show a marginal improvement in accuracy when training on CUB-200 and evaluating on Stanford Dogs (16.60%) compared to no training (12.20%). This improvement can be explained by the improved general cognition ability, as indicated in the first table. We leave further exploration of this interesting direction to future work.

[Q5] Benefits of Iterative Process for Explanation Quality Thank you for your insightful suggestions. We have added evaluations of Cognition Score (CS) at each iteration for all six datasets. This metric evaluates the coherence and logical flow of generated explanations. The results are as follows:

These results demonstrate that explanation quality (CS) improves progressively with each iteration, with an average 9.7% improvement between iteration 4 and iteration 1 on six datasets, validating the effectiveness of our iterative approach.

We sincerely appreciate the reviewer's insightful comments and constructive suggestions. Below, we discuss each concern in detail:

[W1, Q1] Justifications for Improvement in Performance Gain We wish to emphasize that existing off-the-shelf LMMs often struggle with domain-specific tasks due to their limited expert-level visual knowledge. This is a new challenge that naive fine-tuning with image-label pairs cannot address. Our method addresses this challenge through fine-tuning LMMs using self-synthesized, high-quality explanations containing carefully selected visual features.

Our method demonstrates two key improvements compared to general models, as indicated in Table 1:

• Enhanced Classification Performance: Our model achieves superior classification accuracy compared to both the untrained base model and traditional fine-tuning approaches (NL and L+GE).
• Improved Explanation Quality: Our model generates more coherent and logically structured explanations, outperforming both the base model and traditional fine-tuning approaches (NL and L+GE).

These improvements stem from two key innovations in our self-synthesized training data:

1. Expert-level visual descriptions generated by language models.
2. Careful visual description selection guided by our designed information bottleneck framework.

The resulting training corpus contains more detailed and semantically rich descriptions compared to conventional image-text pairs used in CLIP or LLaVA pre-training, enabling the fine-tuned LMM to develop domain-specific expertise in visual classification and interpretation.

[W1, Q2] Impact on General Reasoning Our fine-tuning approach enhances domain-specific performance while preserving general reasoning capabilities, as indicated by the MMMU metric in Table 1. Additionally, we evaluated our method on four new general ability benchmarks: MMStar, SEED-Bench-2, MMBench, and MME (Cognition). The results show that training on our self-synthesized fine-tuning data not only preserves but improves general capabilities, achieving a 5.5% overall performance gain compared to the untrained base model.

To facilitate reproducibility and further research, we will release our fine-tuned model weights along with documentation for their use.

As the rebuttal period is ending soon, we wanted to follow up on our response to your review. If you have any additional concerns, we would be happy to address them. Otherwise, we kindly hope you might consider raising the review score.

Thank you again for your time and feedback.

[W4] Acquisition of Label-Level Concepts As stated in Section 3.2 of the main paper, label-level concepts can be obtained through querying domain experts or LLMs (e.g., GPT-4o). For scalability, in our experiments, we obtain concepts by querying GPT-4o. We have added the concept extraction prompts in the revised appendix. This approach minimizes the reliance on manual annotations, addressing scalability concerns. In real-world applications, concepts can be first obtained by LLMs and then verified by human experts to ensure correctness.

[W5] Potential Overfitting to Explanation Format Our method generates diverse explanations with reasonable visual details, as evidenced by cognition scores (Table 1) and example answers (Figure 4, Table 8). Specifically, we incorporate the following mechanisms to prevent explanation format overfitting:

• Rejection Sampling (Section 3.3): Multiple explanations are generated, and only the highest-quality ones are retained for subsequent training rounds.
• Diverse Prompts and High-Temperature Sampling: These encourage variability in generated explanations, reducing reliance on fixed templates.

Additionally, our model retains general QA abilities while improving reasoning capabilities, as shown by its robust performance across diverse datasets. These results confirm the variety of explanation styles in our model.

[W6] Limited Comparison with External Works We want to clarify that our primary baselines (L+GE) represent well-established approaches in the field. The L+GE baseline, in particular, is proposed in recent works by LLaVA [1] and Finer [2].

Moreover, We also implemented an additional baseline where explanations are directly generated by prompting LLaVA. This baseline follows an iterative process:

1. Initial Round: Explanations are generated by prompting LLaVA directly and used as training data.
2. Subsequent Rounds: The fine-tuned model from each round generates the most probable responses, which serve as training data for the next round without filtering.

This process is repeated for four iterations. We report the model accuracy of each iteration and the final cognition score:

Our results show that while the baseline (without filtering) achieves initial improvements, its performance stagnates and even declines in later iterations. Additionally, the explanation quality (measured by cognition score (CS)) remains lower compared to our method. Due to time constraints, we completed experiments on three datasets. We expect similar results in the remaining datasets and will report the full-scale analysis in the camera-ready version.

Overall, we want to emphasize that our paper tackles new problems with limited prior work. We have included all possible external works to the best of our knowledge.

---

References
[1] Liu, Haotian, et al. "Visual instruction tuning." NeurIPS (2023).
[2] Kim, Jeonghwan, and Heng Ji. "Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models." arXiv (2024).

Thank you for your thorough evaluation and valuable feedback. Below, we discuss the weaknesses and answer the questions:

[W1] Generalization to Unseen Categories Our primary focus is on enhancing domain-specific cognition and explainability through fine-tuning, and generalization to unseen categories was not the primary objective. However, our proposed method has demonstrated the ability to improve general cognition, as evidenced by the results presented below. We evaluated our trained models on four new benchmarks: MMStar, SEED-Bench-2-Plus, MMBench, and MME (Cognition). The results show that after fine-tuning on our self-synthesized data, the model not only retains its general abilities but achieves a 5.5% overall improvement over the untrained base model on all four benchmarks.

[W2] Dependence on Base LMM While we used LLaVA-1.5 as our primary model, our framework is model-agnostic and can be applied to different types of LMMs. This adaptability is demonstrated by its successful application to LLaVA-Med on the Chest X-ray dataset, where accuracy improved significantly (e.g., from 62.50% to 98.72%). Additionally, the theoretical grounding provided in Section 3.4 supports the framework's independence from specific base LMMs.

[W3] Label Generation and Testing Set Validity We want to clarify that we use the original labels and train/test split from the original datasets (e.g., CUB-200, Stanford Dogs). Our method mainly generates synthetic explanations for these predetermined labels. Furthermore, we only apply our iterative training to the training datasets, which prevents data leakage. The test sets maintain their original labels and evaluation criteria, ensuring a valid assessment of model performance.

As the rebuttal period is ending soon, we wanted to follow up on our response to your review. If you have any additional concerns, we would be happy to address them. Otherwise, we kindly hope you might consider raising the review score.

Thank you again for your time and feedback.

Thank you for your positive feedback and thoughtful questions. We sincerely appreciate your recognition of our information bottleneck approach and rejection sampling strategy. Below, we discuss your specific concerns in detail:

[Q1] Training Parameters and Performance Gains

We apologize for the confusion. The trainable parameters during fine-tuning consist of two components: (1) LoRA parameters integrated within the LLM. (2) The visual projector layer. The visual encoder remains frozen throughout the fine-tuning process. For efficient fine-tuning, we apply LoRA adapters to all linear layers in the LLM architecture.

The performance improvements primarily stem from two key innovations in our self-synthesized training data: (1). Expert-level visual descriptions generated by language models. (2). Careful visual description selection guided by our designed information bottleneck framework. The resulting training corpus contains more detailed and semantically rich descriptions compared to conventional image-text pairs used in CLIP or LLaVA pre-training. This enhanced semantic richness enables the fine-tuned LMM to develop domain-specific expertise in visual classification and interpretation.

[W1, Q2] Impact on General QA Capabilities

Thank you for your insightful suggestion. We believe that classification tasks, particularly in domains requiring expert knowledge (e.g., medical and scientific applications), are critical for advancing the utility of LMMs. Current models often struggle with specialized tasks, but our method demonstrates improved performance and explainability in these areas, as shown in Table 1. In the case study from Table 2, our approach enables effective and interpretable plant disease identification, illustrating its utility in expert applications.

Most importantly, our proposed approach improves domain-specific performance without compromising general capabilities. The MMMU metric in Table 1 shows that our method maintains general abilities. Additionally, we evaluated our trained models on four new benchmarks: MMStar, SEED-Bench-2-Plus, MMBench, and MME (Cognition). The results demonstrate that after fine-tuning on our synthetic data, the model not only retains its general abilities but achieves a 5.5% overall improvement over the untrained base model on all four benchmarks.

As the rebuttal period is ending soon, we wanted to follow up on our response to your review. If you have any additional concerns, we would be happy to address them. Otherwise, we kindly hope you might consider raising the review score.

Thank you again for your time and feedback.

Thank you for your positive feedback. We appreciate your constructive suggestions and discuss them below:

[W1] Additional Benchmark Evaluations Thank you for the suggestion. We added evaluations on our models with MMStar and SEED-Bench-2 Plus, along with MMBench and MME benchmarks (there are some technical issues with OpenAI API when we test our models with MMVet; we hope to fix the bugs soon and release the results):

The table clearly shows that after fine-tuning on our self-synthesized data, the model not only retains its general abilities but achieves a 5.5% overall improvement over the base model on all four benchmarks.

[W2] Ablation Studies with Unfiltered Data We appreciate your suggestion to include additional baselines to better understand the proposed data filtering mechanism. Following your suggestion, this baseline follows an iterative process:

1. Initial Round: Explanations are generated by prompting LLaVA directly and used as training data.
2. Subsequent Rounds: The fine-tuned model from each round generates the most probable responses, which serve as training data for the next round without filtering.

This process is repeated for four iterations. We report the model accuracy of each iteration and the final cognition score.

Our results show that while the baseline without filtering achieves initial improvements, its performance stagnates and even declines in later iterations, and the explanation quality (measured by cognition score (CS)) remains lower compared to our method. Due to time constraints, we completed experiments on three datasets. We expect similar results in the remaining datasets and will report the full-scale analysis in the camera-ready version.

[W3] Experimental Runtime Reporting We have included a detailed runtime report in the revised appendix. It takes around 2.5 hours to finish one iteration of rejection sampling and fine-tuning for datasets like CUB-200 and FGVC on a single H100 GPU.

[Q1] Impact of Text Encoder We conducted an additional ablation study showing that choosing different text encoders will affect the concept selection result but not significantly. We currently use an off-the-shelf E5 [1] as our embedding model for InfoNCE estimation. For comparison, we test the performance with the BERT-Large and BERT-Base models. The table below demonstrates that E5 outperforms BERT-family models in concept selection accuracy, making it the superior choice for our framework.

[Q2] Qualitative Examples of Data Selection Figure 6 and Table 9 show qualitative examples of our selection strategy. For example, for the Sooty Albatross case study in Figure 6, our approach identifies close-up features like "white crescent-shaped markings around the eyes" in a headshot image, while focusing on broader characteristics like "long, slender wings" and "streamlined body" in a full-body image.

[1] Wang, Liang, et al. "Text embeddings by weakly-supervised contrastive pre-training." arXiv preprint arXiv (2022).

As the rebuttal period is ending soon, we wanted to follow up on our response to your review. If you have any additional concerns, we would be happy to address them. Otherwise, we kindly hope you might consider raising the review score.

Thank you again for your time and feedback.