AVG-LLaVA: A Large Multimodal Model with Adaptive Visual Granularity

We thank you for your insightful feedback on improving the quality of our manuscript.

Response to W1

Although multiple pooling operations and router operations have appeared in other fields, their structures are not exactly the same as ours. Importantly, we are the first to propose an adaptive visual granularity selection method for LLMs.

1. Furthermore, our visual granularity scaler and visual granularity router are not the same as previous methods. For instance, the visual granularity scaler stacks 1x2 and 2x1 pooling operations, while the visual granularity router enables the information interaction between image and instruction tokens through a Transformer layer. Then, an MLP layer allows visual and instruction tokens to predict the granularity to be selected, and a Voter enables all tokens to vote for the selected granularity (whereas the router in traditional MOE only contains a linear layer).
2. Additionally, we propose a novel training paradigm, RGLF, which addresses the issue of poor performance due to the difficulty of distinguishing good and bad granularities during direct visual instruction fine-tuning. This aligns router probabilities of multiple granularities with LLM preferences.
3. The ablation experiments on architecture and training in Table 4 also validate the effectiveness of AVG-LLaVA.

Response to W2

In Stage 2, we follow the setup of LLaVA-NeXT [1], training the visual encoder and LLM simultaneously, which are used by most current LMMs. We provide the training costs for each stage. We use a single node with 8 H800 GPUs (each with 80GB of memory) for training, and the costs are as follows:

We have added this result in Table 5. Our computing resources are limited, and training will be faster with more resources in a multi-node, multi-GPU setup. Although the cost is increased compared to LLaVA-NeXT, these costs are justified because they significantly enhance model performance and reduce inference time without requiring additional large amounts of data. When a large number of users are accessing the model, the improvement in inference speed can save a lot of computing resources and bring higher benefits. This trade-off between increasing training costs and reducing inference costs is reasonable. In the future, we aim to explore methods for merging Stages 2, 3, and 4 to reduce training overhead. We also hope to explore LoRA training or methods for freezing certain modules during training.

Response to Q1

Thank you for your suggestion. We will list the performance changes of AVG-LLaVA compared to LLaVA-NeXT in Table 3 in the revised version, as follows:

We use the widely used LMMs evaluation tool, lmms-eval, for testing. The specific throughput inference speeds are as follows:

Response to Q2

Thank you for your feedback. We have added the visualization of visual granularity selected by the router under different instructions in Appendix A.5. As shown in Figure 12 of the paper (see page 20 of the newly submitted pdf), we input the same image with different instructions and then visualize the selected visual granularity on the image, i.e., the number of patches. As can be seen, even for the same image, the router selects different visual granularities for different instructions. For example, when asking about the color of the car, the model does not require such fine-grained visual information (router selects 144 visual tokens per grid), whereas when asking whether there is a cat, the model requires finer-grained visual information (router selects 576 visual tokens per grid).

Reference

[1] LLaVA-NeXT: What Else Influences Visual Instruction Tuning Beyond Data?

Thank you for your response. The main purpose of SPPNet is to handle input images of different sizes in image classification and object detection tasks. It utilizes max-pooling to merge features from images of varying sizes into a fixed number of visual features.

In AVG-LLaVA, the visual granularity scaler is only a small part of the model and not the primary innovation of our work. SPPNet pools image features of different sizes into fixed sizes such as 4×4, 2×2, and 1×1. The only similarity between the visual granularity scaler and SPPNet lies in the use of multiple pooling operations. Beyond the visual granularity scaler, we introduced a visual granularity router consisting of a Transformer layer, an MLP layer, and a voter layer, which is designed to adaptively select the appropriate visual granularity based on the input image and instruction.

In addition, we proposed RGLF, which addresses the challenge of poor performance due to difficulty in distinguishing different granularities during direct visual instruction fine-tuning. RGLF aligns the probabilities of multiple granularities in the router with the preferences of the LLM.

Overall, the similarity between AVG-LLaVA and SPPNet only lies in the use of multiple pooling operations in the visual granularity scaler module, which is a very small part of our work. To the best of our knowledge, our work is the first attempt to design an LMM that can adaptively select the appropriate visual granularity based on the input image and instruction. Our primary contributions are the introduction of the visual granularity router and the novel RGLF training method, enabling an LMM to adaptively select the visual granularity based on the image and instruction. The results in Table 4 (a), (b), (c), (e), and (f) all demonstrate the effectiveness of the visual granularity router and RGLF.

We acknowledge the increased training cost; however, training is conducted offline and only needs to be performed once. We believe this is a worthwhile trade-off, as a moderate increase in training cost can significantly improve inference speed.

We thank you for your insightful feedback on improving the quality of our manuscript.

Response to W1

We acknowledge the concerns about the additional computation costs. We provide the training costs for each stage. We use a single node with 8 H800 GPUs (each with 80GB of memory) for training, and the costs are as follows:

We have added this result in Table 5. Our computing resources are limited, and training will be faster with more resources in a multi-node, multi-GPU setup.

Although the cost is increased compared to LLaVA-NeXT, these costs are justified because they significantly enhance model performance and reduce inference time without requiring additional large amounts of data. When a large number of users are accessing the model, the improvement in inference speed can save a lot of computing resources and bring higher benefits. This trade-off between increasing training costs and reducing inference costs is reasonable. In the future, we also hope to explore methods for merging stages 2, 3, and 4 to reduce training overhead.

Response to W2

Thank you for your feedback. Following your suggestion, we have added the algorithm to Appendix A.1 in the newly submitted version of the paper.

Response to W3 and W4

Thank you for your constructive comments. We randomly sample 50 examples from each of the benchmarks: ScienceQA, ChartQA, MME, and MMB. Then, we conduct a manual review to determine whether the images needed to be carefully examined to answer the questions. Examples that need to be carefully examined require fine-grained visual information; otherwise, coarse-grained visual information is sufficient. The proportion of cases requiring careful image examination is as follows:

Except for ChartQA, most of the other benchmarks only require a quick glance at the image to answer the questions. This indicates the potential to reduce the number of visual tokens. This observation aligns with the trend of granularity selection by the router in these benchmarks, as shown in Figure 5. Our experimental results also indicate that on coarse-grained benchmarks such as ScienceQA, MME, and MMB, using fewer visual tokens can lead to performance improvements.

In addition, we have supplemented the experiments of visualizing the attention maps of model-generated tokens between visual tokens in Appendix A.4. We visualize the attention map between the generated tokens and the visual tokens. The attention weights are calculated by accumulating the attention scores between image tokens and generated tokens across all layers and heads. As shown in Figure 11 (see page 20 of the newly submitted pdf), when the instruction is ``How many sheep are there? Answer the question with a single word,'' the attention weights for the visual granularity selected by the router primarily focus on the two sheep, while the attention weights for other visual granularities are dispersed across the background. This means that selecting the appropriate visual granularity results in a clearer attention map with fewer noise points in the background area, indicating more precise focus on the relevant regions, thereby improving model performance.

In section A.3 of the Appendix, we provide a qualitative analysis to demonstrate the importance of granular selection. Besides, as shown in Table 4 (a), we compare the results of fixed visual granularity and adaptive granularity selection. It can be observed that adaptive granularity selection generally improves model performance. Additionally, as shown in Table 3, adaptive granularity selection can accelerate the model's inference speed.

Response to W5

Thank you for your constructive comments. We follow your suggestion and fine-tune the model three times in Stage 2. The experimental results on general VQA benchmarks and text-oriented VQA benchmarks are as follows:

The experimental results on general multimodal benchmarks are as follows:

1. It can be observed that although three repeated trainings result in improvements on 7 benchmarks (e.g., ChartQA and DocVQA), there is a considerable performance decline on 6 benchmarks (e.g., TextVQA and MMB$^{CN}$). This indicates that repeated training cannot improve the performance on all benchmarks.
2. AVG-LLaVA performs better than LLaVA-NeXT-3epoch on 9 benchmarks, is slightly worse on 2 benchmarks, and has a significant speed improvement, indicating that the advantage of AVG-LLaVA does not simply stem from repeated training.

Response to W6

As you mentioned, in OCR tasks, most of the visual tokens are retained, but we believe this is reasonable because such tasks generally require fine-grained visual information for text recognition. This indicates that the router is capable of distinguishing different inputs and selecting the most appropriate granularity, rather than favoring a single granularity. As mentioned in our response to W5, the performance of TextVQA declines after repeated training. This indicates that the performance improvement in OCR tasks is not solely due to repeated training but also benefits from multi-granularity instruction fine-tuning. Additionally, the results in Tables 3 and 4 (a) show that dynamic granularity selection significantly benefits both speed and performance on other tasks, demonstrating the generalizability of the method.

Response to Q1

It is reasonable for the model to require instructions for granularity selection, as shown in Figure 1. Even for the same image, different instructions may require different visual granularities. To test the model's robustness in granularity selection based on instructions, we applied random dropout and noise perturbations to the instruction tokens input to the router. The experimental results of applying dropout to the instruction tokens on ScienceQA are as follows:

The experimental results of adding Gaussian noise to the instruction tokens on ScienceQA are as follows:

The experimental results above indicate that our granularity selection process is relatively robust to the instructions.

Response to Q2

Thank you for your valuable feedback. We acknowledge the importance of evaluating the model in real-world scenarios with less curated and noisier data to test its robustness. However, at this time, we do not have access to such datasets, as this type of data may involve personal privacy and requires ethical consideration.

We thank you for your insightful feedback on improving the quality of our manuscript.

Response to W1

1. Compared to the Matryoshka model, which requires manually setting the number of visual tokens, AVG-LLaVA can adaptively select the appropriate visual granularity based on the input image and instructions. This makes it more practical in real-world scenarios, as it is infeasible to experiment with every possible granularity due to the high cost. Experimental results show that AVG-LLaVA surpasses the Matryoshka model in both performance and speed in most benchmarks.
2. Furthermore, the hierarchical token merging method used in the Matryoshka model is not unique; for instance, it has been applied in the classic SPPNet [1]. In contrast to the Matryoshka model, we introduce a visual granularity scaler and a visual granularity router, designed specifically for granularity scaling and selection. The architecture also different, with the router consisting of a transformer layer, an MLP layer, and a voter layer, taking multi-granularity visual features and filtered instruction features as input. The primary goal is to enable adaptive visual granularity selection.
3. Additionally, we propose the RGLF training paradigm, addressing the challenge of poor performance in direct visual instruction fine-tuning, where the router struggles to distinguish between different granularities. This allows the router to better select the appropriate visual granularity based on the image and instructions. The ablation studies on architecture and training in Table 4 further demonstrate the effectiveness of AVG-LLaVA.

Response to W2

Thank you for the suggestion. We believe that exploring task-specific fine-tuning or manual selection could reduce the generality of the large multimodal model. These approaches may enhance performance for specific tasks but would require substantial effort for each new task, potentially undermining the model's adaptability and scalability across diverse applications. Moreover, manual selection is impractical in real-world applications, as it requires iterating through all granularities for each sample to select the optimal one, which would result in significant cost overhead. Our approach, focusing on adaptive granularity selection, is designed to maintain the model's flexibility and efficiency while ensuring robust performance across varied tasks.

Response to W3

As shown in Table 3, on the ChartQA benchmark, even though most of the visual tokens were retained, the model's speed only decreases by 3%. Moreover, as mentioned in line 408 of the paper, the parameters of AVG-LLaVA increase by only 1.66%. These observations indicate that the computational cost of the modules we introduced is minimal. On other benchmarks, AVG-LLaVA demonstrates significant speed improvements, especially on AI2D, where it achieves a 2.53x acceleration.

Response to W4

Although AVG-LLaVA shows a slight performance decrease compared to the best baselines on GQA and ScienceQA, it still achieves the third and second best results, respectively. Notably, as shown in Table 3, we reduce the number of visual tokens by 20% and 73.6% on GQA and ScienceQA, respectively, while accelerating by 1.14x and 1.77x. Furthermore, on the other 8 benchmarks, AVG-LLaVA outperformes all other baselines, demonstrating the generalizability of the method. The ablation experiments in Table 4 (a) also compare adaptive and fixed (576) approaches, showing that the adaptive approach outperforms the fixed one with fewer visual tokens.

We thank you for your insightful feedback on improving the quality of our manuscript.

Response to W1

We acknowledge the concerns about the additional computation costs. We provide the training costs for each stage. We use a single node with 8 H800 GPUs (each with 80GB of memory) for training, and the costs are as follows:

We have added this result in Table 5. Our computing resources are limited, and training will be faster with more resources in a multi-node, multi-GPU setup.

Although the cost is increased compared to LLaVA-NeXT, these costs are justified because they significantly enhance model performance and reduce inference time without requiring additional large amounts of data. When a large number of users are accessing the model, the improvement in inference speed can save a lot of computing resources and bring higher benefits. This trade-off between increasing training costs and reducing inference costs is reasonable.

Response to W1

Please refer to the following response to the question.

Response to Q1

Thank you for your feedback. We follow your suggestion and fine-tune the model three times in Stage 2. The experimental results on general VQA benchmarks and text-oriented VQA benchmarks are as follows:

The experimental results on general multimodal benchmarks are as follows:

1. It can be observed that although three repeated trainings result in improvements on 7 benchmarks (e.g., ChartQA and DocVQA), there is a considerable performance decline on 6 benchmarks (e.g., TextVQA and MMB$^{CN}$). This indicates that repeated training cannot improve the performance on all benchmarks.

2. AVG-LLaVA performs better than LLaVA-NeXT-3epoch on 9 benchmarks, is slightly worse on 2 benchmarks, and has a significant speed improvement, indicating that the advantage of AVG-LLaVA does not simply stem from repeated training.

Response to Q2

We randomly sample 50 examples from each of the benchmarks: ScienceQA, ChartQA, MME, and MMB. Then, we conduct a manual review to determine whether the images needed to be carefully examined to answer the questions. Examples that need to be carefully examined require fine-grained visual information; otherwise, coarse-grained visual information is sufficient. The proportion of cases requiring careful image examination is as follows:

Except for ChartQA, most of the other benchmarks only require a quick glance at the image to answer the questions. This indicates the potential to reduce the number of visual tokens. This observation aligns with the trend of granularity selection by the router in these benchmarks, as shown in Figure 5. Our experimental results also indicate that on coarse-grained benchmarks such as ScienceQA, MME, and MMB, using fewer visual tokens can lead to performance improvements.

In addition, we have supplemented the experiments of visualizing the attention maps of model-generated tokens between visual tokens in Appendix A.4. We visualize the attention map between the generated tokens and the visual tokens. The attention weights are calculated by accumulating the attention scores between image tokens and generated tokens across all layers and heads. As shown in Figure 11 (see page 20 of the newly submitted pdf), when the instruction is ``How many sheep are there? Answer the question with a single word,'' the attention weights for the visual granularity selected by the router primarily focus on the two sheep, while the attention weights for other visual granularities are dispersed across the background. This means that selecting the appropriate visual granularity results in a clearer attention map with fewer noise points in the background area, indicating more precise focus on the relevant regions, thereby improving model performance.

The ablation experiments in Table 4 (a) also demonstrate the effectiveness of adaptive granularity.