Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillation

We thank the reviewer for their feedback and comments. We are encouraged that they find our task as "meaningful" and "important". We are glad that they acknowledge our method's "good results on various benchmarks." and that they find our paper easy to follow and well-writen.

We respond to their questions and concerns below:

Pt #1: More Benchmarks

Reviewer's Comment: The proposed model and pipeline look promising but more experiments need to be performed, especially for spatial/depth reasoning. Please try benchmark in SpatialRGPT and SpatialBench

Thank you for your suggestions!

We provide results on the positional and counting tasks (which are aligned with our target tasks) under SpatialBench [2]. As shown in the table below, our VisPer-LM outperforms the baseline LLaVA-1.5 model and both variants of SpatialBot (RGB and RGBD inputs). For SpatialBot, we report results as mentioned by the authors under issue #13 of the official repo due to an update of the model's checkpoint since its publication.

We could not evaluate our model on SpatialRGPT-Bench [1] because the benchmark is designed for grounded spatial reasoning, i.e., it requires the model to be trained with region-aware referring question-answer data which is not present in our LLaVA-1.5 based training set, making it hard for the model to understand and answer the benchmark's questions in way that could help the evaluation of its abilties. We hope the reviewer understands the special nature of this benchmark and our inability to evaluate on it without a special training recipe.

[1] Cheng, An-Chieh et al., "SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models." NeurIPS 2024

[2] Cai, Wenxiao et al., "SpatialBot: Precise Spatial Understanding with Vision Language Models." ICRA 2025

Pt #2: Comparison to MLLMs with non-RGB encoders

Reviewer's Comment: Also, comparing with LLaVA seems to be unfair, because LLaVA is based on RGB image encoder only, and is not specifically trained for spatial/depth understanding and reasoning. Please also compare with: SpatialVLM, SpatialRGPT and SpatialBot.

We respectfully disagree that comparing with LLaVA is unfair just because it only uses an RGB image as input. Most existing MLLM works use RGB encoders including Cambrian-1 [66] which proposed CV-Bench (including the spatial and depth reasoning tasks) for evaluating MLLM's visual perception abilities. We believe it is critical to develop general-purpose MLLMs that are accurate at depth and spatial reasoning with only an RGB image input and our VisPer-LM is a significant advancement in that direction.

Nonetheless, we thank the reviewer for their suggestion and provide comparisons to SpatialRGPT and SpatialBot on the Depth and Relation tasks under CV-Bench below. Interestingly, although our VisPerLM does not show the best performance on either task, it performs best on average. SpatialBot shows the best performance on the Depth task while SpatialRGPT shows the best performance on Relation task but their performance lags on the other task. We believe a plausible reason for this could be the large scale data used by these models, making them specialized for particular tasks despite using additional non-RGB inputs. We could not compare our approach to SpatialVLM [1] since the authors of that paper do not release their code, models or benchmark.

[1] Chen, Boyuan et al., "SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities." CVPR 2024

[66] Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal llms, NeurIPS 2024.

We thank the reviewer for their feedback and comments. We are glad that they recognize the importance of our effort to improve MLLMs at visual perception and that "Section 3 (Preliminary Analysis) provides cool insights that can be useful for future research". We are encouraged that they found the experimental section "comprehensive" and "particularly liked the "data-matched" Cambrian-1 experiment and the comparison against a RADIO-ViT-L based LLaVA".

We respond to their questions and concerns below:

Pt #1: VisPerLM shows improvements on BLINK benchmark

Reviewer's Comment: I wonder how much some of the downstream results may suffer from the chicken-and-egg dilemma when it comes to CV-Bench. All preliminary analyses conduct extensive experiments on all CV-Bench tasks, which inform all design choices of VisPerLM, and the main evaluation for vision-centric capabilities is conducted on CV-Bench as well. I think it is important for this paper to test downstream performance on other vision-centric benchmarks (e.g. BLINK [a]) to ensure this confounder disappears, and check whether the improved vision-perception abilities are stable and consistent;

How much does the downstream performance of VisPer-LM suffer from informed decisions based on CV-Bench? Does improved visual perception persist when evaluating on other vision-centric benchmarks?

Thank you for the suggestion to evaluate our method on the BLINK benchmark! We report results on visual perception tasks under the BLINK benchmark (val set) in the table below. We use CLIP-ConvNeXT-XXL and Llama3-8b as the base visual encoder and base LLM, respectively. We also report results using the data-matched Cambrian-1 RADIO-ViT-L based LLaVA-1.5 models for the reviewer's reference.

We find that our VisPer-LM significantly outperforms all other baselines on average, especially on the Spatial Relation task, demonstrating the effectiveness of our approach.

Pt #2: Effect of system prompt and text tokens on the probing performance

Reviewer's comment: Reading the paper, it is a bit unintuitive that all tokens within LLM decoder blocks are used for probing, i.e., since the goal of the resampler is to probe visual representations, I wonder whether only visual tokens alongside task tokens may be sufficient (or even better), compared to also using text tokens.

Did the authors experiment with other probing setups (.e.g. omitting textual tokens as well as / or tokens from the system prompt)?

We clarify that the goal of our probing setup is to evaluate the representation quality of all the tokens from the LLM as a whole and not just the tokens corresponding to the visual features. This is an important detail as prior works [32, 72] have shown the importance of system and text tokens during the LLM's token generation process. Therefore, in order to establish a correlation between the MLLM's representation quality and downstream performance, we believe it is critical to probe all tokens from a given sequence.

Nonetheless, we agree this is an interesting experiment! Therefore, we share the average probe scores (across all layers) for our VisPer-LM model under different token input settings to the resampler below:

<sys>: system tokens; <img>: visual tokens; <t>: task-specific tokens; <txt>: text tokens

As expected, dropping the system and text tokens, leads to the best depth and seg probing scores. Interestingly, using all tokens leads to the best gen probing performance which we attribute to the language aligned nature of the gen expert features (L#147 in the main text).

However, note that we also conduct a similar experiment while training the model in appendix A Tab. III, i.e., ablating the tokens input to the embedding predictors for the auxiliary losses. We found that using all tokens as input to the embedding predictor leads to the best performance owing to the importance of the system and text tokens for LLM's reasoning.

[32] Omri Kaduri, Shai Bagon, and Tali Dekel. What’s in the image? a deep-dive into the vision of vision language models. arXiv, 2024.

[72] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. ICLR, 2024

Pt #3: RADIO based LLaVA

Reviewer's Comment: About the RADIO-based LLaVA: did the authors train it themselves, or does it come from some previous work I am unaware of?

We trained our own version of RADIO ViT-L LLaVA-1.5 based on Llama3-8b. We will clarify this in the main text. Thank you for pointing this out!

Pt #4: Removing the Gen expert

Reviewer's Comment: Is there any evidence that some vision experts are unnecessary? This question mostly arises from the consistently good probing performance for the image-to-text generation expert, hence I wonder if this can be removed with little to no performance degradation;

This is an interesting experiment! We provide ablations on the different combinations of auxiliary losses during training in appendix A (L#539-545). We share the analysis from the article here for the reviewer's reference and direct the reviewer to Tab. IV in the appendix for the numbers:

Our results reveal that the optimal performance is achieved when all three embedding losses: depth, seg, and gen, are used together. Interestingly, we observe that utilizing only depth or gen embedding losses still leads to notable performance improvements, whereas relying solely on seg embedding loss does not yield significant gains. This suggests that different types of target information contribute uniquely to the distillation process. Investigating how the distillation of one type of target information influences the effectiveness of others presents an intriguing direction for future research.

Pt #5: Fixed seed for experiments

Reviewer's Comment: Were multiple runs executed for these experiments? If so, can the authors please report the standard deviation as well? This would be helpful to understand which performance gains are significant and which are not.

Due to resource constraints, we could not perform multiple runs. However, to ensure experimental validity and significance, we perform all our experiments with a fixed seed as also pointed out under point 7 in the checklist.

Pt #6: Embedding Distillation Layers for Phi3-4k-mini

Reviewer's Comment: How are the sets of layers contributing to embedding distillation from different vision experts selected for the experiments with Phi3-4k-mini (Table 1)?

Because Phi3-4k-mini has the same number of layers as Llama3-8b (32 layers), we use the same set of layers for both the LLMs based on the ablation results from Tab. 4 in the main text. We verified that Phi3-4k-mini based models showed similar probing trends as that of Llama3-8b based models while making this decision.

We thank the reviewer for their reply and time! We appreciate their suggestions on adding more details for a dedicated Baselines section under implementation and will incorporate it in our revised version! We are also grateful for their valuable feedback on the importance of establishing a positive transfer of design choices between different LLM architectures and will spend time doing so!

We provide clarification about the BLINK v/s CV-Bench point from our last comment below:

Reviewer's Comment: Although I see your point, I think this still suggests limited capabilities in abstracting and transferring the knowledge learned by VisPerLM. Judging from this example alone, the BLINK subtask looks exactly like the CV-Bench subtask, just in a different "visual format".

We agree with the reviewer that the BLINK subtask is in a different visual format. However, we would like to point out that it is also in a different language format. We believe that because the LLM is responsible for reasoning over the given visual input and that the model has not seen such "which point is closer?" questions during training, the model is unable to understand the format well to show gains of the improved visual embedding. On the contrary, for CV-Bench, the questions contain object names which are easier for the model to reason over owing to the training samples from the visual genome and COCO based datasets. This observation may point to the over-dependency of MLLMs on the underlying LLMs for visual reasoning.

Nonetheless, we understand the reviewer's point that VisPer-LM should be able to reason well over the BLINK's sample, however, the LLM-driven nature of MLLM makes it a bit hard for the model in our opinion, resulting in accuracy near chance behavior (i.e, 50% as the possible answers are only yes or no for Depth task under BLINK) for all models. We will look more into this though. We thank the reviewer for raising this point!

Reviewer's Comment: Apologies, but I do not understand this point. Although it is of course better to evaluate on benchmarks with as many data points as possible, percentage points remove the dependency from the raw number of samples, don't they? Am I missing something here?

We apologize for our statement being unclear to the reviewer. We meant to share the argument about different number of samples in both benchmarks as a secondary factor and not the primary argument (primary being the nature of benchmarks). We agree that the dependency on % does remove the dependency from the raw number of samples. However, only to an extent. We believe the observations from approx 250 samples from the BLINK benchmark have a higher chance of being noisy than from about 700 samples under CV-Bench. Therefore, we believe the nature of questions is more at play here than the number of samples.

At the same time, I think the results on the counting task are promising, and a nice result to show in the revision of the paper together with the results on Spatial Relationships and Relative Depth.

Definitely! We thank the reviewer for their continued suggestions and feedback. We will add these results to the revised version.

We hope the above clarifications help and appreciate the reviewer's valuable feedback and interest in our work!

We thank the reviewer for their feedback and comments. We are glad that they acknowledge the effectiveness of our approach supported by extensive experiments and analysis.

We respond to their questions and concerns below:

Pt #1: Training Time and Inference Time Efficiency

Reviewer's Comment: The training time for each stage and the efficiency compared to the multi-encoder are not reported. The total loss includes three additional depth, segmentation, and generation tasks, respectively. During training, computing the three target probe features takes time. Compared to the multi-encoder approach, is the training time actually reduced? Specifically, where is the efficiency improvement reflected?

We would like to clarify that by performance-efficiency trade-off in the main text (L#48), we mainly refer to the inference time efficiency. Nonetheless, we report numbers for both inference and training time below to demonstrate the efficient nature of VisPer-LM compared to the multi-encoder baseline.

Inference time: We demonstrate the superior inference throughput of VisPer-LM over the multi-encoder approach in Tab. VIII in appendix A. We report the numbers again below for the reviewer's reference:

Our VisPer-LM has superior inference throughput compared to that of the multi-encoder baseline and similar throughput to that of the single-encoder baseline with much better performance. We record the throughput on a single NVIDIA 80G A100 GPU for a single forward pass on the CV-Bench evaluation set with a batch size of 1. We report the mean and standard deviation across 10 runs.

Training time: We report stage-wise training time for the different approaches below for CLIP-ConvNeXT-XXL as our base visual encoder and Llama3-8b as the llm decoder on 8 A100 GPUs.

As shown in the above table, our VisPerLM reduces the total training time by 8 hours as compared to the multi-encoder approach. Note that we only use the auxiliary losses during the PT stage, resulting in IFT training time comparable to that of the single encoder approach.

Pt #2: All baselines (single and multi encoder) use the same training data and steps as VisPerLM

Reviewer's Comment: Since both increasing the training data and distilling the three target visual information are performed simultaneously, a further comparative explanation of the impact of these two operations is needed.

We clarify that we use the same amount of training data and steps as the baselines for the numbers reported in Tab. 1 in the main text while demonstrating the superior performance of VisPerLM. Moreover, we show the impact of extra training data in Tab. 3 (with an extra VPT stage) in the main text, which also shows gains, showing the scalability of VisPerLM with more data. Therefore, our experiments already include separate experiments to demonstrate the inidividual effect of visual embedding distllation and increasing the amount of data.

Pt #3: The probing experiments explain the multi-encoder performance on the Target tasks

Reviewer's Comment: In Figure 3, it is confusing that the multi-encoder baseline has the best probing performance but achieves lower performance than VisPer-LM. Also, the probing performance of VisPer-LM that training with these three tasks does not significantly improve the quality of single encoder.

Why does the multi-encoder have the highest visual embedding quality in Figure 3, but perform worse than VisPer-LM in Table 1? Doesn’t this contradict the paper’s claim that the quality of visual representations is positively correlated with downstream task performance?

That is a good question and observation! As explained on L#267-269 in the main text, the multi-encoder baselines are slightly better than VisPer-LM on the Depth task in CV-Bench owing to their superior depth representation quality, aligned with our claim that the quality of visual representations is positively correlated with the downstream performance on the target tasks (Depth and Relation under CV-Bench). Note that we probe the LLaVA-1.5 (feat concat.) baseline in Fig. 3.

The multi-encoder baseline performs worse on average than our VisPerLM in Tab. 1 as the former is unable to retain general reasoning ability (on MMStar) same as the single-encoder baseline, unlike VisPerLM, demonstrating the effectiveness of our approach.

Pt #4: Scalability with More training data

Reviewer's Comment: As training progresses and the number of steps increases, does the quality of the visual embeddings and the model’s performance continue to improve accordingly?

Yes, as shown in the second row of Fig. 2(a) of the main text, as we increase the amount of training data (and therefore, the number of steps), the probing score and downstream performance gradually increase as expected. We observe the same phenomenon in row 2 of Fig. 3 as explained on L#156-170.

Pt #5: Gen Probing Performance has low correlation with downstream performance

Reviewer's Comment: In the results shown in Figure 3, for the first and third rows, the performance of VisPer-LM is almost indistinguishable from the single encoder in the deeper LLM layers, and in the generation task, the single encoder even performs slightly better. This suggests that training with these three tasks does not significantly improve the quality of the visual embeddings. Therefore, I have doubts that the performance improvement on the benchmarks is actually due to the use of more data and an extended training stage, rather than an improvement in the quality of the three target visual embeddings.

As explained under Pt #2, we use the same amount of data for training our VisPerLM model as the single-encoder baseline. Therefore, the the auxliary losses are solely responsible for the observed performance and visual representation quality improvement.

Moreover, in appendix D, we find a high positive correlation of 0.98 between the gen/seg probing scores and downstream performance on CV-Bench while only a correlation of 0.37 for the gen probing scores. Therefore, single encoder baseline having slightly better gen probing scores is not a surprise due to the nature of our target tasks (Depth and Relation under CV-Bench).

As explained on L#143 in the main text, prior works [32, 33, 61] have shown that early-to-middle layers have the most significant effect on the reasoning performance in (M)LLMs, and our VisPerLM shows far superior (depth/seg) representation quality in those layers compared to the single encoder baseline, explaining its superior performance. Nonetheless, understanding how different layers contribute to an MLLM's reasoning abilities remains an interesting research direction for the future.

[32] Omri Kaduri, Shai Bagon, and Tali Dekel. What’s in the image? a deep-dive into the vision of vision language models. arXiv, 2024.

[33] Guy Kaplan, Matanel Oren, Yuval Reif, and Roy Schwartz. From tokens to words: On the inner lexicon of llms. arXiv, 2024.

[61] Oscar Skean, Md Rifat Arefin, and Ravid Shwartz-Ziv. Does representation matter? exploring intermediate layers in large language models. In Workshop on Machine Learning and Compression, NeurIPS 2024.

We thank the reviewer for their feedback and comments. We are glad they found our idea “well-motivated and insightful” and our method “novel and clever.” We appreciate their recognition of our method's “excellent trade-off between performance and efficiency” and the “empirically grounded motivation from a strong and novel probing analysis.” We are also grateful for their positive remarks on the our work's execution and extensive ablations.

We respond to their questions and concerns below:

Pt #1: Dimensionality of the "Generation" Probe Target

Reviewer's Comment: In Section 4.1, you note that the target for the "generation" features (Etextgen) has only one token, unlike the 576 tokens for the depth and segmentation features.

Could this significant difference in the prediction target's dimensionality explain the highly stable and distinct probing performance observed for the gen features in Figure 3?

That is an interesting point and perspective! In the main text (L#147-148), we attribute the superior gen probing performance to the choice of $**E**^{\text{gen}}$ (from unCLIP-SD-2.1 [56]) which produces visual features that are already language-aligned owing to its training setup, making it easier for the probes to learn a mapping from the LLM representation (already in the language space).

Therefore, we believe in this particular case, the nature of expert features plays the major role and not the number of tokens. Nonetheless, ablating different sets of expert encoders for probing and embedding distillation remains a promising direction for future work. We thank the reviewer for this intriguing question!

[56] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv, 2022.

Pt #2: Using Multi-Token Generative Target Features

Reviewer's Comment: Following up on Q1… Other works have successfully extracted multi-token features from generative model

E.g., Cambrian-1 uses a 1024-token representation from Stable Diffusion from their code, it seems this is based on methods from "Emergent Correspondence from Image Diffusion".

Have you considered using a similar multi-token generative feature extractor? This would create a much cleaner comparison against your multi-token depth and segmentation experts and help isolate the effect of what kind of knowledge is being distilled from the dimensionality of the distillation target.

Thank you for your suggestion and sharing the exact reference! We share results with the usage of 1024-token outputs from the SD-2.1 as the gen expert features in the table below. We observe that the features from unCLIP-SD-2.1 (used in the main text) performs much better than using the 1024 token representation from SD-2.1 (VAE+UNet). Therefore, we believe that the information captured by the feature embeddings plays a more critical role than their dimensionality and a strong generative prior may even harm the MLLM's understanding performance in some cases, therefore it is important to choose the experts wisely. Nonetheless, future research into comparison of different experts is an intriguing direction.

Pt #3: Impact of VisPerLM

Reviewer's Comment: The final performance gains on the main benchmarks, while consistent, are relatively modest (e.g., ~2.5% average improvement in Table 1)

⇒ While the methodological contribution is strong, the practical impact in terms of state-of-the-art performance is not dramatic

We appreciate the reviewer’s recognition of our methodological contribution's strength. Enhancing MLLMs in visual perception and reasoning remains an active area of research, with several recent works—such as Cambrian-1 [66], which leverages multiple encoders to address the limitations of single-encoder MLLMs, and AM-RADIO [57], which distills knowledge from multiple teacher encoders into a single, enhanced encoder.

We compare our VisPerLM against these approaches in Table 1 of the main text, demonstrating that our method outperforms both under identical data settings, highlighting the effectiveness of our proposed approach.

We believe a 2.5% improvement on average is not modest in an academic setting where we do not use any extra/different data to train the MLLM compared to the baselines. However, we acknowledge the reviewer’s concern regarding the practical impact in terms of state-of-the-art performance. As noted in our limitations, we were unable to conduct experiments at larger scale due to resource constraints. Nonetheless, we believe the current results provide strong empirical evidence supporting a new paradigm for improving MLLMs. We also hope our probing framework serves as a valuable tool for understanding the underlying factors driving performance gains for the community.

[66] Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal llms, NeurIPS 2024.

[57] Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Am-radio: Agglomerative vision foundation model reduce all domains into one. In CVPR, 2024

Pt #4: VisPerLM does not harm general reasoning

Reviewer's Comment: While the distillation method is effective for the targeted perception tasks, its reliance on a curated set of specialized "expert" encoders raises questions about its generality.

It's unclear if this approach is a scalable path forward for improving MLLMs broadly, or if it imposes too much specific structure, potentially at the expense of more general reasoning capabilities that might emerge from simpler, larger-scale training.

We remind the reviewer that VisPerLM's core idea is to improve an MLLM at target perception tasks without any loss of its general reasoning abilities. For example, our approach provides a promising direction to improve current video understanding models at tasks like motion understanding. We leave the exploration of our method with large scale data and more target tasks to future work as noted under limitations.