Q1. More detailed designs or ablation should be carried out for perception prior embedding design.

A: Thanks for your suggestion. We have added more detailed ablation studies on perception prior embedding strategy. We have conducted ablation studies on different strategies for assigning perception embeddings to each pixel token, and the results are shown in Table R5. The best results were achieved by assigning the corresponding object query to each pixel as the perception embedding to be embedded, using the SoftMax method. The strategy of generating perception embeddings through L1 normalization led to performance degradation, but it still brought a significant performance improvement compared to not using the perception prior embedding strategy.

Q2. More detailed discussion on meta-architecture design should be One question is why you fix the image decoder why not use another decoder with the same architecture but with the same pre-trained weights. Although this operation can increase the parameter costs, I wonder whether joint sharing one decoder can have mutual effects. Thus, more detailed experiments on meta-architecture design should be added.

A: Using another decoder with the same architecture and loaded with the same pre-trained weights to generate segmentation results in a marginal performance improvement, as shown in the table below. Undoubtedly, using an additional decoder can achieve better performance due to the introduction of more trainable parameters. However, the performance gain is marginal, so we chose to adopt a frozen OMG Decoder to achieve a balance between performance and parameters.

Q3. No parameter and Gflops analysis. For example, the author claim they only use one image encoder, one decoder and one LLM. Compared with previous combined approach (GLaMM), this advantage is not well explored.

A: We have statistics on the parameters of the main components of OMG-LLaVA, such as the LLM and visual encoder, and the results are shown in below table. The visual encoder of OMG-LLaVA is only 0.2B, much smaller than the 0.9B of LISA and GLaMM. The number of vision tokens significantly impacts the computational cost of the LLM. Compared to GLaMM, which uses 576 visual tokens, OMG-LLaVA only requires about 276 visual tokens, which is only 50% of GLaMM.

After going through the feedback from other reviewers and the author's detailed replies, I'm more convinced this paper lives up to the NeurIPS standards. I’m especially impressed by the authors’ thorough and detailed response, with the solid experimental results they added. They addressed the key issues I brought up. So, I've bumped my score up to 8, with a clear accept.

Just to sum up my thoughts on this research: The paper introduces OMG-LLaVA, which is a trailblazer in merging three levels of perception into a single LLM system, using just one encoder, one decoder, and one LLM. The way the authors link object queries from a universal segmentation model to the LLM is really smart. It's not just about beating other models like Pixel-LLM and GLaMM—it’s about doing it more elegantly. During the rebuttal, the authors also showed that OMG-LLaVA performs even better with a stronger LLM, pointing to great potential for future work.

At last, the authors should open-source the entire codebase and models (including the stronger LLMs in the rebuttal) for the community soon.

Q1. The proposed method is not very elegant as too many designs customized for different tasks are introduced in the framework.

A: OMG-LLaVA boasts an elegant model architecture and more streamlined workflows across tasks than GLaMM. For instance, in GLaMM, visual prompt encoding and segmentation mask decoding employ entirely different workflows and models. However, in OMG LLaVA, visual prompt encoding and segmentation mask decoding are unified into a single OMG Decoder to generate object-centric queries. Maintaining a consistent workflow while unifying many tasks is extremely challenging and currently difficult. For example, mask decoding and visual prompt encoding are hard to implement solely with LLMs without relying on other modules. We believe that in the future, diverse tasks will be unified under elegant architectures and shared workflows, and OMG-LLaVA is a significant step toward this goal.

Q2. Some important technical details are omitted. For example, to encourage different object queries to attend to different objects, a Hungarian matching algorithm should be applied to assign labels for each individual object query, especially for COCO segmentation, where multiple objects are supposed to be simultaneously segmented. Besides, to generate masks from object queries, the mask decoder should be carefully designed to generate high-quality masks, however, this paper only mentioned a simple FFN is utilized. Is it reasonable? More details should be provided.

A: There is a misunderstanding. The segmentation ability is derived from the frozen perception model (OMG-Seg), not the LLM's output. OMG-Seg is a component of OMG-LLaVA, so there is no need for the LLM to reiterate these perception results. Secondly, the mask decoder of OMG-LLaVA is not a simple FFN layer. The last layer's hidden states of the [SEG] token first pass through a text projector (an MLP layer) and then are fed into the frozen mask decoder (OMG-Seg head) to obtain segmentation masks. During training, the mask decoder remains frozen while the text projector is trained. Since the perception results have already been embedded into the LLM's input via perception prior embedding, generating segmentation queries is not difficult. As shown in Table R4 in our PDF file, we also experimented with adding extra cross-attention layers in the text projector, but the performance was similar to using a simple MLP.

Q3. Some important experimental results are not provided. For example, experimental results on prevalent VQA benchmarks like MMBench, MME, SEEDBench, etc. should be provided to illustrate the general-purpose conversational capability of OMG-LLava.

A: We have evaluated the performance of OMG-LLaVA on multiple image-level benchmarks, including MMbench, MME, SEED-Bench, POPE, and AI2D. The results are presented in Table R2. When jointly co-trained with both image-level and segmentation-based data, OMG-LLaVA significantly outperformed LISA and GLaMM on all image-level benchmarks. However, co-training with segmentation-based data significantly decreases performance on image-level benchmarks, although OMG-LLaVA can mitigate this issue to a large extent through perception prior embedding. We also evaluate the performance of OMG-LLaVA trained solely on the LLaVA dataset on image-level benchmarks. OMG-LLaVA outperformed LLaVA 1.5 on MME, SEED-Bench, POPE, and AI2D by 41, 3.0, 3.0, and 5.1 points, respectively.

Q4. Since comprehension capabilities ranging from image-level to object-level are established, does OMG-LLava possess composite capabilities such as visualizing the intention of LLM during a conventional conversation in the form of segmentation masks? This is very helpful of understanding the working mechanism of LLMs.

A: As shown in Figure R6, OMG-LLaVA can visualize LLM intentions through segmentation masks. While the ability to visualize LLM intentions in diverse conversations was not explicitly included in OMG-LLaVA's training data, the model learned this capability by training on referring expression segmentation data and image-level conversation data. This ability is crucial for AI assistants, as it significantly enhances the user experience.

Q1. The paper needs to include more ablation experiments to help readers better understand the work. For example, what would happen if the visual projector design shared an MLP for pixel- and object-centric visual tokens?

A: Table R4 presents the results of ablation studies on the projector. We observed a substantial performance degradation when pixel-centric and object-centric tokens shared a projector. This phenomenon can be attributed to the distinct semantic and perception information encoded in these two types of tokens. Sharing a projector results in detrimental interference between them. Furthermore, our experiments with additional cross-attention layers in the projector did not yield any performance benefits. For the object-centric tokens, specifically visual prompt tokens and object queries, sharing a projector resulted in marginal performance gains for both segmentation and visual prompt understanding tasks.

Q2. Would the performance of OMG-LLaVA improve with a larger LLM?

A: We have conducted experiments with OMG-LLaVA using various LLMs, and the results are presented in Table R3. Our findings indicate that the performance of OMG-LLaVA increases with the strength of the LLM. Due to time limitations, the performance of OMG-LLaVA models based on larger LLMs, such as Yi-34b, will be reported in the coming days.

Q3. The authors should include more MLLMs in Table 1 for a more comprehensive comparison.

A: We appreciate your feedback. To further emphasize the unique capabilities of OMG-LLaVA, we have compared more MLLMs (about 25 MLLMs). The result demonstrates that OMG-LLaVA provides the most comprehensive capabilities while employing a single visual encoder.

Q1. [Performance] Although this work claims OMG-LLaVA as a generalist model, it cannot outperform prior state-of-the-art models on any specific task.

A: We have updated the performance of OMG-LLaVA, as shown in Table R1. OMG-LLaVA outperforms LISA, PixelLM, and GSVA on the RES benchmarks. GLaMM used the GranD dataset for pretraining. The GranD pretrain dataset contains 11M images, far exceeding the GranD-f dataset used by OMG-LLaVA, which only includes 210K (0.21M) images. Nevertheless, OMG-LLaVA still surpasses GLaMM on the GCG benchmark by 0.6 CIDEr, 1.4 AP50, and 0.1 mIoU.

Additionally, OMG-LLaVA achieves substantial performance gains over GLaMM, LISA, and PixelLM on image-level benchmarks (Table R2). For instance, when jointly co-training with image-level and segmentation-based datasets, OMG-LLaVA achieves 1412 on MME benchmark, outperforming LISA, PixelLM and GLaMM with 1410, 968 and 1389, respectively. When only trained on the LLaVA dataset, OMG-LLaVA outperforms LLaVA-1.5 by impressive margins of 41, 3.0, 3.0, and 5.1 points on MME, SEED-Bench, POPE, and AI2D, respectively.

Q2. [Technical contribution] Overall, OMG-LLaVA seems a direct integration of OMG-Seg and LLaVA.

A: Our most significant contribution is introducing an elegant MLLM architecture that comprises only a perception model and an LLM, capable of understanding and reasoning at the image, object, and pixel levels. The frozen perception decoder can simultaneously encode visual prompts and decode [SEG] tokens. However, our experiments and those in LISA have shown that the frozen perception decoder cannot effectively decode [SEG] tokens. The proposed perception prior embedding is the key component enabling our meta-architecture to work well. Thus, we have three main contributions.

• First, we propose an elegant MLLM architecture with a single visual encoder, a perception head, and an LLM, which can simultaneously handle image, object, and pixel-level tasks.

• Secondly, we introduce perception prior embedding to help the LLM better align pixel-centric visual tokens with object-centric visual tokens, allowing the LLM to generate object-centric tokens that the frozen perception decoder can decode based on the input embedded perception prior.

• Thirdly, we construct OMG-LLaVA, which achieves SOTA performance on pixel-level tasks and comparable accuracy to SOTA on object-level tasks. Additionally, OMG-LLaVA significantly outperforms versatile MLLMs like LISA, PixelLM, and GLaMM in image-level tasks.

From these aspects, we argue that our work is not the direct integration of OMG-Seg and LLaVA.

Q3. [Image-level tasks] Although this work claims OMG-LLaVA can unify image-level, object-level, and pixel-level reasoning and understanding tasks, there is no quantitative evaluation in terms of image-level tasks such as VQA.

A: Table R2 presents our model's performance on various image-level benchmarks, including MME, MMBench, SEED-Bench, POPE, and AI2D. Jointly co-training on segmentation and image-level data can detrimentally impact an MLLM's performance on image-level benchmarks. However, OMG-LLaVA mitigates this negative impact thanks to its perception prior embedding. OMG-LLaVA significantly outperforms GLaMM, PixelLM, LISA, and LaSagnA on these image-level benchmarks. OMG-LLaVA scores 1412 on MME and 47.7 on MMBench, surpassing GLaMM with 1389 on MME and 11.1 on MMBench. By training solely on image-level data, OMG-LLaVA achieves superior performance over LLaVA 1.5 on benchmarks such as MME, SEED-Bench, POPE, and AI2D, attributed to its perception prior embedding.

Q4. [Frozen perception module] In OMG-LLaVA, the “universal perception module” (OMG-Seg) is frozen (Figure 2e) to preserve its learned knowledge. However, Table 3 shows that allowing the OMG decoder to be tuned can actually improve the referring segmentation performance. It is not explained why this module should be frozen in other tasks, or whether this practice would bring further gains.

A: During finetuning, the OMG decoder is duplicated and unfrozen to better decode the [SEG] tokens generated by the LLM. The encoding of object-centric tokens and visual prompts continues to be handled by the original frozen OMG decoder. We also add the experiment of keeping the OMG decoder frozen when finetuning the model. As shown in the last two lines of Table R1, finetuning on the RES datasets while keeping the OMG decoder frozen results in a slight performance degradation compared to the case where an additional OMG decoder is duplicated and unfrozen (78.0 mIoU vs. 77.2 mIoU). However, the performance still significantly surpasses LISA and PixelLM. Moreover, we directly unfreeze the original OMG decoder without duplicating it. In that case, OMG-LLaVA cannot work well due to the reliance of object-centric tokens on a stable mask segmentation capability.

Q5. What is the “pixel shuffle operator” (Line 171)?

A: This paper leverages the pixel shuffle operator to downsample image features. By flattening $S \times S$ neighboring pixel features, the operator reduces the image size from $(H, W, C)$ to $(H/S, W/S, C \times S^{2})$.

Q6. To be consistent with Equation 2.

A: Thank you for the suggestion. We have modified $M \in R^{N_{q}\times HW}$ to $M \in R^{HW \times N_{q}}$.