Reconstructive Visual Instruction Tuning

We thank reviewer rXNr for the valuable time and constructive feedback. We appreciate your comments on the paper being “well-organized and easy to read” and the “straightforward” motivation. In the following, we have done our best to address your suggestions point-by-point.

W1: About the Novelty.

A1: We would like to emphasize that our main contribution is proposing a novel vision-centric supervision for enhanced comprehension capabilities for LMMs, instead of any specific technical modules. The underlying motivation is input images themselves inherently provide rich and detailed information, which is quite important for fine-grained comprehension tasks. As a result, we regard LMMs reconstruct input images as the supervision of those visual outputs.

Actually, Reviewer kvNb uses “novel image-based supervision” to describe our work. Reviewer MCFE regards our work as a “step forward”. Reviewer SsLG says this paper is “very novel”, and Reviewer tGMa thinks our work is “inspiring”.

(1) The idea is not trivial. While utilizing image supervision may seem straightforward, effectively using images to produce meaningful feedback through reconstruction for LMMs remains largely unexplored. The key challenge lies in handling the heavy spatial redundancy of natural visual signals. To address this, we systematically explore various reconstruction targets and objectives, which are definitely specific in-depth designs aimed at enhancing the comprehension capabilities of LMMs. Moreover, the self-attention module of the denoiser is specifically designed to manage the causal dependencies in the original visual outputs, ensuring that the model can effectively process and understand the whole visual content.

(2) Our contribution is not a newly introduced denoising strategy. While denoising is a well-developed strategy in the field of image generation, we are the first to leverage the denoising objective as a reconstruction method to boost fine-grained comprehension for LMMs. Furthermore, denoising is just one type of objective under our framework. We adopt denoising simply because it alleviates the redundancy of natural visual signals. In fact, as demonstrated in Figure 7, vanilla regression still brings significant improvements, highlighting the flexibility and effectiveness of our approach.

W2: When Does the Inherent Details Become Important?

A2: We extend ablations to more representative benchmarks, where Ross manages to bring improvements in most cases. By systematically analyzing the results, we find the improvements are more significant on fine-grained comprehension benchmarks such as HallusionBench, MMVP, ChartQA, and OCRBench, as visual contents for these benchmarks are more crucial. In contrast, observed by Cambrian-1, LMMs can sometimes correctly answer the question without providing the image on knowledge benchmarks such as MMMU and AI2D, where the improvements seem to be less significant.

Examples in Figure 15 are vivid illustrations. Taking a close look at the given image becomes important to answer the question. Under such cases, vision-centric supervision enables LMMs to pay more attention to visual content, thereby enhancing overall performance.

I have carefully read the authors' responses. However, my main concerns still remain:

1. About the novelty of the developed method. The proposed framework is based on mature technologies (image-based supervision, reconstruction framework) from other fields. There is no discussion about it with existing technologies. Although authors claim that utilizing image contexts has not been explored in LMMs and their contribution is the entire system, I do not think the novelty of the idea meats a standard bar. The core technical designs are not new.

2. About the complexity comparison. The provided table seems not fair. The comparison is not apple-to-apple. Compared to LLMs, the vision encoder introduces much larger resource costs. If the authors want to compare their framework with existing LMMs, please provide the comparison with other LMM models like LLaVA, MiniGPT4.

We sincerely thank the reviewer for their prompt and thoughtful feedback.

1. We respectfully disagree with their concerns regarding the novelty of our work.

First, we would like to clarify that the main contribution of our study lies in introducing a novel vision-centric supervision approach to enhance the comprehension capabilities of large multimodal models, rather than focusing on specific technical modules.

Second, while we acknowledge the use of established techniques, we emphasize that the integration and demonstrated effectiveness of image-based reconstructive supervision within the context of LMMs constitute a significant and novel contribution. To the best of our knowledge, no prior research has successfully employed "simple" image-based reconstructive supervision to improve comprehension in LMMs. This challenge arises from the unique difficulty of generating meaningful low-level visual supervision for highly semantic models like LMMs.

This work conducted a systematic study on (1) targets and (2) objectives, with the ultimate goal of handling the heavy spatial redundancy of natural visual signals. Specifically,

• Towards reconstruction targets, we study reconstructing (i) vanilla RGB pixel values, (ii) latent tokens obtained by deep models, and (iii) RGB pixels using a pixel decoder.
• Towards reconstruction objectives, we empirically found that denoising is more suitable than vanilla regression for the ultimate goal as it avoids fitting specific values.

In fact, very few studies have begun to focus on the design of visual supervision for LMMs.

Therefore, our work represents a pioneering step forward, providing a strong baseline for adding visual supervision to LMMs and improving fine-grained comprehension capabilities. We hope our findings will inspire future research and innovations in this area.

We believe that the systematic exploration and integration of these components into a cohesive framework constitute a significant contribution to the field.

2. We would like to clarify that our complexity comparison is indeed fair and apples-to-apples.

We keep all unrelated factors, including the vision encoder, the language model (LLM), and the training data, identical across each two rows in the table. The only difference between the two compared methods is the incorporation of our proposed objective. Therefore, we maintain that the provided complexity comparison is completely fair and apples-to-apples.

Following your suggestions, we estimated the computational costs under the exact same setting as the original LLaVA-v1.5 in the following table. Specifically,

• The visual encoder is set to CLIP-ViT-L/14@336 and is kept frozen.
• The LLM is either Vicuna-7B-v1.5 or Vicuna-13B-v1.5.
• The training data is LLaVA-665K, where the training requires 5197 steps with a global batch size of 128.

The only difference is our Ross incorporates $\mathcal{L}_{\mathrm{LMM}}^{\mathrm{visual}}$ while LLaVA-v1.5 does not. This ensures that any observed differences in complexity are directly attributable to the inclusion of our proposed loss function.

It was officially reported in the LLaVA's GitHub repo [1] that, using DeepSpeed ZeRO-3 on 8xA100, it takes approximately 20 hours for LLaVA-v1.5-13B and around 10 hours for LLaVA-v1.5-7B. Our implementation, under the same conditions, yields similar training times, confirming the reliability of our setup.

With regard to MiniGPT-4, it is hard to fairly compare its computational cost with that of LLaVA (-v1.5) or Ross due to substantial differences in model settings. Specifically:

• Training data: LLaVA-v1.5 utilized 558K and 665K samples for pre-training and instruction tuning, respectively. MiniGPT4 incorporated 5M samples for training.
• Training recipe: LLaVA-v1.5 adopts a two-stage training pipeline, where the first stage is for training the projector, while the second stage is for training both the projector and the LLM. MiniGPT4 has only one training stage for the projector.
• Visual encoder: LLaVA-v1.5 utilized CLIP-L-336 (0.3 B), while MiniGPT4 adopted EVA-CLIP-G-224 (1B).

Given these differences, a fair comparison with MiniGPT-4 is not feasible.

However, the core idea behind Ross, namely vision-centric supervision, represents a general enhancement for visual instruction tuning. We believe this approach could also be applied to MiniGPT-4-like models with minimal computational overhead, as evidenced by our experiments on LLaVA-based models.

References

[1] https://github.com/haotian-liu/LLaVA

Thanks for the authors' further clarification. However, my concerns about the paper's novelty remain.

I indeed recognize the motivation and the "step forward" of this work. However, utilizing existing technologies to address a new setting is not novel. The authors also admit that they just integrate established techniques to develop a simple supervision framework. The designs of reconstruction targets and objectives are not new and exciting.

An interesting high-level idea does not mean that it is novel enough for a high-quality conference. The core technology is also important in supporting its detailed designs with novel inspiration.

Therefore, I believe this paper does not meet the bar for ICLR.

We thank reviewer tGMa for the valuable time and constructive feedback. We are particularly grateful for your kind words about the paper being "well-organized and easy to follow" with a "clear presentation of the key ideas." Your observation that "a simple auxiliary loss can improve the performance of LMMs" is very inspiring and aligns with our goals. We have done our best to address your suggestions point-by-point in our response below.

W1: About Novelty and Insight.

A1: We would like to clarify that "generation" and "reconstruction" are fundamentally different approaches. Previous works (Emu-2) explore “generative objectives”, while our Ross is a kind of “reconstructive objective”. The detailed pipeline comparison can be found in Figure 11 in the Appendix. Specifically, Emu-2 takes outputs corresponding to learnable queries as conditions, while our Ross takes outputs corresponding to visual inputs.

Empirically, our Ross significantly outperforms Emu-2 in comprehension tasks, demonstrating the effectiveness of “reconstruction” over “generation”. Moreover, we have compared reconstruction and generation in Table 2, where using generative objectives actually fails to bring improvements over the baseline. This empirical evidence highlights the effectiveness of the reconstructive approach over the generative one.

Both the distinct pipeline and the superior performance of Ross underscore our contributions and insights.

W2: Pixel-Level Reconstruction may Risk the Langauge Capabilities.

A2: Following suggestions, we evaluate multi-modal benchmarks that mainly require general knowledge following Cambrian-1, including ScienceQA, MMMU, and AI2D. Furthermore, we incorporate representative language benchmarks, including general understanding on MMLU and HellaSwag, and instruction-following on IFEval. Empirical results demonstrate that Ross does not harm language capabilities as it brings improvements in most cases.

W3: Scability.

A3: We appreciate the reviewer’s insightful feedback regarding the scalability of our empirical findings. To address these concerns, we have conducted additional experiments to study (1) the model scaling behavior and (2) the data scaling behavior of Ross.

(1) To study the stability and scalability of Ross across different model sizes, we use the Qwen2.5 series with varying sizes as the base language model while keeping the CLIP-ViT-L/14@336 as the visual encoder. The pre-training data is LLaVA-558K, and the instruction tuning data is LLaVA-665K. The results, shown in the following table, demonstrate that Ross consistently brings improvements over the baseline (LLaVA) across different model sizes.

(To be continued)

We thank Reviewer SsLG for the insightful and positive feedback. We are deeply appreciative of using "very novel" and the recognition of the importance of vision-centric learning in LMMs. The acknowledgment of our "innovative vision-centric supervision method" and the "clever" use of denoising objectives is highly encouraging. We are also grateful for your praise of our "extensive experiments" and "thorough ablation studies". We provide point-to-point responses below.

W1: Computational Costs.

A1: We apologize for the oversight in discussing the computational overhead quantitatively. To address this concern, we have conducted additional experiments in the following table to measure the computational costs and provide a clearer understanding of the practical trade-offs. Evaluations are conducted using 8 A100 GPUs with a global batch size of 128, where the batch size per GPU remains 4 4 gradient steps are accumulated. GPU memories are averaged over 8 GPUs with DeepSpeed Zero 3. As demonstrated in the following table, Ross introduces a marginal increase in training time and GPU memory.

W2: Sensitivity to Hyperparameters.

A2: We appreciate the reviewer’s suggestion to thoroughly discuss the sensitivity of Ross. We study the effectiveness of different schedules of β in the following table, where all methods are equipped with CLIP-VIT-L/14@336 and Qwen2-7B-Instruct. The pre-training data is LLaVA-558K and the instruction tuning data is Cambrian-737K. From the table, we can tell that even with different schedules of β, Ross consistently improves the baseline, demonstrating its robustness to the denoising schedule.

As for the architecture choices, we have analyzed the impact of different visual tokenizers. The results, presented in Figure 10, show that the KL-16 tokenizer outperforms the VQ-16 tokenizer. One intuitive explanation is that KL-16 preserves more low-level details compared to VQ-16, as quantization can lead to information loss. Additionally, Figure 10 highlights the importance of the self-attention module. Since the original visual outputs are causal, modeling inter-token dependencies via self-attention is crucial. The number of trainable parameters for the denoiser is not the primary factor affecting performance.

References

[R1] Jonathan Ho, et al. Denoising diffusion probabilistic models. NeurIPS, 2020.

[R2] Robin Rombach, et al. High-resolution image synthesis with latent diffusion models. CVPR, 2022.

[R3] Alexander Quinn Nichol, et al. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. ICML, 2022.

[R4] Minkai Xu, et al. Geodiff: A geometric diffusion model for molecular conformation generation. ICLR, 2022

We thank Reviewer MCFE very much for the insightful feedback. We are particularly grateful for your acknowledgment of the "step forward" our technique represents. Your recognition of the "significant" improvements in metrics is highly encouraging. Point-to-point responses are provided below.

W1: Clean A/B Experiment with Extended Evaluation Benchmarks.

A1: We apologize that the provided A/B experiment in Table 3 is not comprehensive enough. Following your suggestion, we extend the A/B experiment in Table 3 by incorporating more benchmarks such as TextVQA and DocVQA, providing a more balanced and representative distribution of tasks, where scores of OCRBench are divided by 10 for computing averaged scores.

Empirical results in the following table demonstrate that our proposed vision-centric supervision utilized by Ross leads to significant improvements in most cases. Moreover, we found Ross contributes more significant improvements over fine-grained comprehension datasets, such as HallusionBench, MMVP, ChartQA, and OCRBench.

CLIP: CLIP-ViT-L/14@336; SigLIP: SigLIP-SO400M-ViT-L/14@384; Vicuna: Vicuna-7B-v1.5; Qwen2: Qwen2-7B-Instruct

W2 & Q1: Comparison on High-Resolution Benchmarks.

A2: We appreciate the feedback regarding the need for high-resolution comparisons. To address this concern, we have incorporated the "anyres" technique proposed by LLaVA-v1.6 into our Ross. Specifically, for each image, we employ a grid configuration of 384×{2×2, 1×{2,3,4}, {2,3,4}×1} to identify the input resolution, resulting in a maximum of 5×729 = 3,645 visual tokens. Each 384×384 crop is required to reconstruct the original input via the denoising objective proposed by Ross. In the following table, our ROSS-7B-anyres surpasses LLaVA-v1.6-7B and Cambrian-1-8B in most cases. These results indicate that Ross not only performs well at lower resolutions but also maintains its competitive edge at higher resolutions, making it a robust and versatile method.

We thank reviewer kvNb for the valuable time and constructive feedback. We are truly grateful for your use of the terms "novel" and "makes sense" to describe our work. Your appreciation of our comprehensive analysis and the robust empirical validation is greatly encouraging. Point-to-point responses are provided below.

W1: Potential Unfairness in Comparisons. While the paper includes an ablation study where variables like training data are controlled for fair comparison with other models, its main results table appears to use different datasets compared to competing methods. This inconsistency in data setup might lead to an unfair advantage for Ross, making it difficult to assess the true comparative effectiveness of the approach against state-of-the-art methods.

A1: We appreciate the reviewer's concern regarding the potential unfairness in the comparisons presented in Table 4. We have tried our best to conduct fair comparisons against the baseline, including the same visual encoder, base language model, pre-training and instruction tuning data, where significant improvements are observed consistently, which can be found in Table 3.

We acknowledge that the comparisons in Table 4 might be perceived as unfair due to the use of different datasets. To mitigate this concern, we have performed an additional experiment where we compare Ross with the state-of-the-art method, LLaVA, using the exact same datasets and settings. Specifically, we used the CLIP-ViT-L/14@336 visual encoder and the Qwen2-7B-Instruct language model. Empirical results below demonstrate that Ross consistently outperforms LLaVA under these identical conditions.

To be honest, unfair comparison is actually a common problem in the field of LMMs as there are many challenges in conducting fully fair comparisons. We have listed a series of representative components that lead to this unfairness in the following table. It was relatively hard for researchers to conduct a completely fair comparison against the state-of-the-art methods, not to mention that some methods may contribute at the data level, e.g., ShareGPT4V.