Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Models

Q1: The origanization of the paper

A1: Sorry for the confusion caused by the paper organization, we will reorganize the paper following your suggestion in the revised version.

Q2: The inference speed of the model

A2: We have discussed the inference speed of our model in the Appendix.D (see L646~L654). We acknowledged that the repetitive model forwarding required by generating heatmaps of different categories will increase the inference burden. Therefore, we parallize the inference procedure using the batch processing technique, and the inference time cost is lower than Griffon, another LMM-based method customized for solving object detection task as discussed in Appendix.D.

Q3: The prediction accuracy of routing tokens

A3: As you suggested, we first modify the task prompts as below.

Original Instruction: "Please find the location of {description}. Respond with {format}."

Modified 1: "What are {description} in the image. Please generate their {format}."

Modified 2: "Generate {format} of {description} in the image."

Afterward, we radomly sample 100 data from each of five special token-related tasks, resulting a testing set with 500 samples in total. Then, we prompt the Lumen with the three types of instructions illustrated in the cell above, and calculate the accuracy of generated special tokens. The accuracy results are reported below:

The results indicate that our model is not sensitive to the change of instruction and can generate the correct task routing token.

Q1: The design of Lumen

A1: As discussed in the Common Question Q1, it is nontrivial to extend the LMM with versatile vision-centric capabities. We further provide proofs below to further validate this claim:

1. Compared with Griffon that refactorizes box coordinates into language answers, our Lumen surpasses it significantly even without meticulously curating high-quality conversations with the aid of powerful GPT-4v as training resources as demonstrated in Tab.1 of the main paper.
2. In the attached PDF file for the rebuttal, we show that even the most powerful LMM, GPT4-v, can not tackle versatile vision-centric tasks well.

Q2: Comparison with mentioned works

A2: There might be a misunderstanding. Our major contribution is that we use the heatmap as the effective intermediate representation to prevent the LMM being trapped by the inductive biases of different vision-centric tasks (i.e, the formulation of boxes, masks and keypoints), rather than the designs of specific modules. And we discussed the benefits of our model in A1 to Reviewer kab5.

Besides, both LISA and NExT-Chat do not prove their dense visual perception abilities like object detection and instance segmentation, while our Lumen can tackle these tasks benefitted from the heatmap representation. And the behavior of our heatmap is not similar to the mentioned Dense Distinct Query (DDQ). DDQ aims at selecting dense object queries distictive from each other from the image feature map to serve for better one-to-one label assignment process for object detection task. As the comparison, our heatmap serves as a unified representation for versatile vision tasks and is optimized without involving any label assignment process (e.g, Hungarian matching) as discussed in Appendix.C.4 (see L641~L645).

Q3: New capabilities enhanced by integrating perceptual data

A3: The integration of vision-centric data can promote fine-grained visual content comprehension ability in VQA task as demonstrated by comparing "FP-C" metric in #2 and #3 of Tab.5, and relevant explanations can be referred to L281~L284 of the main paper. Besides, following your suggestion, we also test our Lumen and the baseline method (i.e, reimplemented LLaVA-v1.5-7B) on HallusionBench[1] as shown below:

The results prove that utilizing vision-centric data can also help our model reduce the hallucination.

• [1] HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models

Q1: VQA performances compared to prior methods

A1: Since our major motivation lies in extending versatile vision-centric capabilities while keeping the general-purpose coversational capabilities of the LMM, we do not meticously select the VQA data for training at present. Thanks for your suggestions, we will continue to improve the VQA capabilities of our model by upgrading the VQA training resources in the future work.

Q2: Comparison between the decoupled learning strategy and the one-stage one

A2: Decoupling task-agnostic and task-specific learnings facilitate the LMM to focus on learning fine-grained multimodal content comprehension, rather than being trapped in learning the diverse specialized decoding rules (e.g, the formulation of bounding boxes, instance masks and keypoints) introduced by vision-centric tasks. In contrast, the one-stage learning paradigm requires the LMM to simultaneously learn these decoding rules lacking in semantics and human conversations requiring semantic comprehension and logic reasoning, which is much more challenging than the decoupled one proposed in our paper.

Q1: Comparison with UniPerceiver-v2 and VisionLLM

A1: There might be a misunderstanding. Our major contribution is that we use the heatmap as the effective intermediate representation to prevent the LMM being trapped by the inductive biases of different vision-centric tasks (i.e, the formulation of boxes, masks and keypoints), rather than the module designs themselves. And the heatmap brings the following benefits that UniPerceiver-v2 and VisionLLM do not possess:

1. Refelcting dense activation of the open-ended user instruction, heatmap can be seamlessly adapted to vision-centric tasks like object detection, pose estimation, with minimal training efforts. In contrast, Uni-Perceiver-v2 involves complex box and mask proposals generation during its visual encoding stage. VisionLLM uses a set of pre-defined query templates (100 in their implementation) like <cls><x1><y1><x2><y2>, and conducts label assignment with the aid of Hungarian matching to handle the unordered nature of dense objects. And both UniPerceiver-v2 and VisionLLM cannot support point-level task like pose estimation.
2. Since heatmaps hide inductive bias of different vision tasks, the general-purpose conversational capability, which is the foundation of a LMM, is not affected when incorporating a large amount of vision-centric data during training as shown in Tab.2 of the main paper. As a comparsion, both UniPerceiver-v2 and VisionLLM do not comprehensively evaluate their VQA abilities.

Q2: Further generalization ability evaluation and explanation to Tab.1

A2: As you suggested, we first provide the quantitative results of our model on the FSCD-LVIS[1] test set for the object counting task below:

Besides, we also evaluate our model on another unseen task ViPBench[2] to validate our model's generalization ability. Please refer to A3 for more details.

As for Tab.1, the gray cells indicate the current method does not support corresponding tasks as addressed in the caption of Tab.1. Meanwhile, we have discussed in Common Question Q1 that it is NOT easy to extend a LMM to versatile vision-centric tasks. On the other hand, by comparing the performances of our Lumen on each task with specialists and generalists along the same column, the capabilities of our model in solving versatile tasks can be validated.

Q3: Further evaluation on more diversed benchmarks

A3: Thanks for your advice. to demonstrate our model's capabilities in handling vision tasks, we further evaluate our model on more benchmarks of vision-centric tasks, including Objects365-v1 val set for object detection, AIC[4] val set for pose estimation, and ReferIt[3] test set for visual grounding as shown in the table below:

where "OD" and "VG" are short for object detection and visual grounding, respectively, representing the type of the specilist model. Since most of existing generalists (e.g, UniPerceiver-v2, mPLUG-v2, Griffon, etc.) only provide detection and grounding results on COCO and RefCOCO(+/g) benchmarks, respectively, we can only list available performances of specialists for comparison in the above table due to time limit. The results indicates that our model can also handle vision tasks on more diversed benchmarks.

Besides, we also evaluate our model on another challenging task ViPBench[2], which requires the model to densely comprehend instances highlighted by visual prompts before answering the questions.

As demonstrated in table above shows, our model signficantly outperforms other methods due to the enhanced dense visual perception capabilities.

• [1] Few-shot Object Counting and Detection
• [2] ViP-LLaVA:Making Large Multimodal Models Understand Arbitrary Visual Prompts
• [3] ReferItGame: Referring to Objects in Photographs of Natural Scenes
• [4] AI Challenger : A Large-scale Dataset for Going Deeper in Image Understanding

Q4: Discussions on the DETR-like decoder

A4: We have discussed the DETR-like decoder selection in Appendix.C.4 (see L641~L645), where we acknowledged that a DETR-like decoder can promote the object detection performances because of its Hungarian matching-based label assignment. However, since our main contribution lies in extending versatile vision-centric capabilities for the LMM with less training efforts while keeping its general-purpose conversational capability, rather than specifically optimizing the object detection task, we do not select a DETR-like decoder in our model.

Q5: Explanation to decoding paths

A5: Regarding the peak point selection operation, the basic workflow can be referred to Sec 3.2 (see L186~L190) of the main paper. For its code implementation, we use torch.nn.MaxPool2d with kernel size of 3 and stride of 1 to efficiently select peak points.

Regarding the lightweight box decoder, we have illustrated its workflow in Fig.5 of Appendix.A.1, as well as detailed descriptions in L555~L564.

Q1-1: The value of the LLM

A1-1: In the workflow of Lumen, the LLM plays a crucial role in jointly comprehending the open-ended instruction and visual content (which is LLM's strength), and its understanding of the inputs is further condensed into the hidden state of the [LOC] token. The V-L dense aligner and task-specific decoder are responsible for efficiently translating such high-level information into the heatmap space and task output space. Experimentally, the LLM's rich knowledge and strong understanding of multimodal inputs are important for tackling vision-centric tasks as verified below:

1. By comparing (1.0*->1.5) in Tab.3c you mentioned, it can be proved that enhanced comprehension ability of LLM can promote vision-centric task performances.
2. As shown in Tab.6, LLM also facilitates our model to generalize to unseen categories well, where the general-purpose content comprehension capability of the LMM plays a crucial role.

Q1-2: The meaning of "specialized datasets"

A1-2: We refer to the "specialized datasets" as the dataset meticuously curated in Griffon (L78), which uses GPT4-v as the data engine to generate dialogues customized for object detection. In contrast, we only sample template questions and answers from pre-defined templates as shown in Fig3, which requires much less efforts.

Q1-3: Comparison with existing LMMs

A1-3: For clarity, we respond your concerns as below:

1. Existing LMM generalists can NOT be easily extended to versatile vision-centric tasks. Please refer to Common Questions Q1 for more details.
2. We use Griffon and InstructCV as comparable baselines because they share similar motivations with us in extending vision-centric capabilities for foundation models in the instruction-following manner. To achieve this goal, they focus on curating specialized datasets with the aid of an external data engine (i.e, a powerful LLM). In contrast, we hide the inductive bias in vision tasks with the proposed decoupled learning paradigm during tuning the LLM.
3. We use InstructCV and Pix2Seq-v2 as comparable baselines in Tab.6 as they are also generalist models similar to our Lumen. As discussed in A1-1, the multimodal comprehension capability of LLM facilitates generalization, and therefore the specialist model (i.e, [41] you mentioned) cannot generalize well even with a heatmap representation.

Q2-1: Explanation to the training phases

A2-1: There might be a misunderstanding. Both phases belong to task-agnostic stage as addressed in Appendix A.3 (see L582). In the phase 1, we focus on promoting dense instance perception ability by using vision-centric data only. In the phase 2, we further add VQA data to reconcile the learned dense perception ability and general-purpose conversational ability. Since most parts of task-agnostic decoding don't need training, we only train the lightweight box decoder on COCO after the phase 1 and 2. Sorry for the confusion, we will clarify this in the revised version.

Q2-2: Explanation to the "complete vision-language interaction"

A2-2: We use Tab.3a to prove comprehensive V-L interaction benefits heatmap generation. Since the V-L dense aligner is not our major contribution, we implement with a lightweight transformer although stacked convolutions can also promote such the interaction.

Q2-3: Explanation to the conclusion in L296

A2-3: We use Tab.3b to prove that the pretrained mask decoder is not the major factor that influences the model performance. By further synthesizing the results of Tab.3a, where the performances decrease evidently due to heatmap degradation, we make such the conclusion that heatmap primarily influences the model performance more evidently than the mask decoder.

Q2-4: Explanation to "FP-C" changes and sparate stages

A2-4: By referring to the role of phase 1 stated in A2-1, "FP-C" drops without phase 1 is because of decreased dense instance perception ability. The motivation of sparate phase training has been addressed in A2-1.

Q3: Explanation to the "fundamental"

A3: These 3 tasks provide scene parsing capabilities from different granularities (i.e, object-level, part-level and pixel-level), which can serve as the foundation for downstream tasks (e.g, human tracking, robot planning, etc). Thus, these 3 tasks are widely explored in previous generalists like Pix2Seq-v2, InstructDiffusion, etc.

Q4: Effects of training less iterations

A4: We have discussed the effects of training iterations in Tab.9d of Appendix C.4 (see L636~L645), which indicates that results of training 50,000 steps are better than those for 10,000 steps.

Q5: Explanation to results comparison between Tab.3b(Maybe a wrong table id here as Tab.4 has no subtables) and Tab.1

A5: The results of "Refer Seg." in Tab.1 are evaluated on RefCOCOg as mentioned in L257~L258, which should be corresponded to the "RCg" in Tab.3b rather than "RC". And since we reduce the training iterations for efficient ablations as mentioned in L267~L268, the result of SAM RCg in Tab.3b is lower than the one in Tab.1.

Q6: Explanation to Fig.4

A6: We first render the peak points selected from heatmaps into the image, and then feed this image to the Lumen again with an instruction of "How many green points in the image?", and the resulting answers are used as the prediction results. We will clarify this in the revised version.

Q7: Discussion of limitations

A7: We have discussed the limitations of heatmaps in the Appendix.D (see L646~L654), i.e., heatmaps for different categories should be generated by forwarding the model separately, which can affect inference speed when evaluating vision-centric tasks like object detection. Therefore, we accelerate this process using parallel techniques.

Dear reviewer #AJBg,

We sincerely appreciate your time and valuable feedback on our paper. Regarding the raised concerns, we would like to make further discussions with you as below:

Q1-1: Experiments to prove the value of the LLM

A1-1: Regarding your concerns on whether experimental results in the paper (i.e, Tab.3c and Tab.6) can prove the value of LLM, we make further explanation as below:

1. We agree that the differences between v1.0* and v1.5 in Tab.3c primarily lie in the advanced VQA data curated by LLaVA-v1.5. By training with these data, the LMM obtain more advanced multimodal content comprehension ability. And the enhanced object detection performance of v1.5 compared with v1.0* in Tab.3c prove that the LMM's multimodal comprehension is a general capability that also benefits vision-centric task. Therefore, we employ the results comparison between v1.0* and v1.5 in Tab.3c to verify the value of LMM on vision-centric task. Meanwhile, a similar phenomenon has also been observed in [1], where the LMM with enhanced multimodal comprenhension ability also performs better on the visual grounding task.

• [1] Shikra: Unleashing Multimodal LLM’s Referential Dialogue Magic

2. When evaluating on unseen categories, the two compared generalist models can be the baseline methods you suggested. Specifically, Pix2Seq-v2 is a vision-generalist that does not involve any LLM utilization, which can be the non-LLM baseline as you suggested. Although InstructCV leverages a LLM to generate instruction data for training a text-to-image diffusion model, its generalization ability is mainly inherited from the CLIP-based image and text encoders of diffusion models, which can be the baseline that uses CLIP embeddings for language comprehension as you suggested. Therefore, by comparing with these two generalists, the significantly enhanced generalization performances of our Lumen in Tab.6 can prove the value of LLM in terms of generalization.

Q1-2: The narration of "specialized datasets"

A1-2: Thanks for your detailed suggestion. We will rephrase the "specialized datasets" as "meticulously curated specialized datasets" in the revised version.

Q1-3: Further explanations on comparison with existing LMMs and generalization facilitated by LMM

A1-3: Firstly, strictly aligning with other LMMs in all aspects is currently intractable, as different LMMs are trained with their respective data and training recipes (please refer to the data composition and strategies of training Qwen-VL and LLaVA for more detailed comparison). Secondly, all training data we used are publicly available and also used in the LMMs we compared, such as RefCOCO, RefCOCO+, and RefCOCOg, which are utilized in both Griffon and Shikra. Besides, for tasks that existing LMMs do not support and cannot be seamlessly extended to, as discussed in our initial rebuttal, we are unable to apply the corresponding datasets to the LMMs we compared. Overall, our Lumen not only achieves better or comparable performance on tasks that existing LMMs can handle, but also extends to novel vision-centric tasks.

For the proof of the generalization facilitated by LMM, generalization evaluation in Tab.6 supports our claim as discussed in A1-1.2 of this dialog box.

Q2-2: Explanation to claim in L290

A2-1: In Tab.3a, we implement the "Conv." and "Trans." architectures with simple standard transformer and convolution layers. Since transformer has a global perceptive field while convolution only has a local one, "Trans." architecture facilitates denser cross-modal interaction (every element can interact with each other in the attention process) than the "Conv." one. Therefore, the better object detection performances achieved by "Trans." indicate that denser vision-language interaction facilitates better dense vision-language alignment. We will rephrase "complete vision-language interaction" in L290 to avoid confusion in the revised version.

Q2-4: Explanation to phase 1

A2-4: It is worth noting that (1) in order to balence vision-centric and the general-purpose conversational (GPC for short) ability, the ratio of VQA data to vision-centric data should be set to 2:1. In our current schedule, we train phase 2 for 10k steps using such data ratio; (2) learning the dense instance perception (DIP for short) ability requires training for even longer, which is 50k steps in current phase 1. If phase 1 and 2 are merged and the above conditions are met, we need to train at least 50k * 3 = 150k steps to ensure both abilities are well learned. The training cost is significantly larger than our current schedule (10k+50k=60k). Having an independent phase 1 without VQA data lets the model focus on learning the DIP ability without affecting the GPC ability, then its learned ability can be maintained in phase 2 with less steps.