Emerging Pixel Grounding in Large Multimodal Models Without Grounding Supervision

We appreciate the detailed feedback you provided for our submission. We are encouraged by your acknowledgement of our paper writing, our “motivation of using attention maps for pixel-level grounding is reasonable,” our “good experimental results” for GCG, and “maintaining general VQA performance.” We provide the following clarifications in response to your concerns:

W1. From experiments Table4 and Figure2, it seems that the quality of the attention maps are not good enough for grounding. Please also report the result without SAM. And it seems the model highly relies on SAM, a foundation CV model, to produce good segmentation mask. So the sentence "the grounding ability can emerge in LMMs without grounding supervision. " is a little bit over-claimed. In addition, using SAM may also conflict with the claim "without explicit grounding supervision", as SAM is trained with a huge number of grounding related data.

• In our attend-and-segment method, we only need one point with the highest attention value from the attention map to provide a clue for grounding, so we do not require the entire attention map to be precise. As long as the attention map is able to produce a reasonable starting point for SAM as prompt, our method can generate accurate pixel-level grounding. This simple design also minimizes the number of hyper-parameters. Otherwise, for example, we would need thresholding hyper-parameters to convert the real-valued attention maps into binary masks.

• We already provided results without SAM in Table 1. The point accuracy (PAcc) metric measured how accurately the prompt points (derived from attention maps) could be used to locate objects of interest.

• We would like to clarify that this “no grounding supervision” claim is proper.

  • “Grounding” is the ability to relate language with visual contents, as previously defined in the literature [Ref1, Ref2]. We would like to note that SAM is trained with semantic-free mask annotations, i.e., SAM can perform the segmentation task, but has no grounding ability.

  • In contrast, previous grounding LMMs are extensively trained with grounding supervision that includes correspondences between text and fine-grained visual contents (e.g., object bounding boxes and segmentation masks).

  • Our approach only requires image-level weak supervision, so it is proper to claim that "the grounding ability can emerge in LMMs without grounding supervision.”

We hope this response could clarify the reviewer’s concern, and we would be glad to discuss further details if the reviewer has more questions.

W2. Lack of experiments on referring expression segmentation (RES) benchmark [a,b] and panoptic narrative grounding (PNG) benchmark [c]. RES benchmarks can better validate the effectiveness of the proposed method, as they require LMMs to perform both reasoning and segmentation. PNG benchmark validates the ability of LMMs to segment user-described objects and ground key phrases or words in a text sequence.

We greatly appreciate your suggestions on additional evaluation on other tasks. We have performed experiments on referring expression segmentation (RES) and panoptic narrative grounding (PNG) and achieved stronger performance compared with prior zero-shot methods that are also not trained with explicit grounding supervision. For example, our approach achieves 45.3 average recall in PNG, outperforming the previous zero-shot state-of-the-art method by 6.8 average recall. The details are included in Section D of the updated manuscript. The results on RES and PNG further demonstrate that LMMs can implicitly learn to accurately perform visual grounding without explicit grounding supervision, and our approach is the key to unlock such grounding ability.

W3. As pointed out in the paper, the paper is mostly related to F-LMM (Wu et al., 2024a), however, no direct experimental comparisons are provided.

• First of all, we would like to point out that we are not required to compare with a concurrent and unpublished arXiv paper, according to ICLR policy (https://iclr.cc/Conferences/2025/ReviewerGuide).

• Since F-LMM does not provide results on the instance segmentation or grounded conversation generation (GCG) task, we were unable to provide a direct comparison between our approach and F-LMM. In the updated manuscript, we have included additional comparison with F-LMM on RES and PNG.

• Nevertheless, we discussed the differences between our approach and F-LMM in the related work section, especially in the type of supervision and the visual encoding.

W4. Experimental results in Table1 shows that the AP and AR of DIFF-LMM are very low (only 5.6 AP and 26.0 AR), limiting its application in real scenarios.

• In fact, the real-world scenario was demonstrated in the GCG task (Table 2), in which we have outperformed the previous grounding LMMs like GLaMM.

• We would like to clarify that the results in Table 1 were about analytical experiments on the instance segmentation task, which provided insights that attention maps in LMMs could lead to visual grounding. We did not claim it as a powerful instance segmentation approach in real-world application.

Q1. The method utilizes existing natural language processing tools (e.g., spaCy (Honnibal et al., 2020)) to parse the output sequence into noun phrases. However, in some cases only "noun phrases" may fail to capture the whole semantic meaning in some complex scenario, e.g. "the man in black holding an umbrella". How to handle this problem?

• The LMM is able to understand complex phrases like what the reviewer has suggested, because the large language model (LLM) component can correctly reason the correspondence between elements in complex phrases and visual contents.

• This correct correspondence, as reflected in the attention maps, will not be changed by how we parse the sentences.

• We have updated the manuscript with additional visualization of the attention maps for phrases in complex scenes (Figure 8), which demonstrates the correct correspondence between objects and parts of such complex phrases.

[Ref1] Rohrbach et al. Grounding of Textual Phrases in Images by Reconstruction. ECCV 2016.

[Ref2] Krishna et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV 2017.

• Regarding the reviewer’s concern about the parsing process, we would like to clarify (continued):

  • Our method is still fully automated, consistent with prior grounding LMMs. Our comparison with prior works is fair, except the supervision (prior works require grounding supervision, while ours does not).
    • In our work, we do not rely on any explicit grounding supervision to distinguish words that require grounding, but such words are required in the GCG task. Therefore, we take a minimalist perspective, and adopt a simple NLP tool spaCy to parse noun phrases without human involvement.
    • The usage of spaCy is not always required. For example, the RES and PNG tasks already provide the noun phrase to ground, in which cases we do not need spaCy to parse sentences.
    • Even in GCG, in principle, our method is not tied with spaCy, and a more advanced and automatic approach could have been used. For instance, in our response to Reviewer r6Gw, we showed an example of using a large language model, GPT-4o, to automatically parse the sentence with the following prompt:

    Extract and list all noun phrases that are likely found in the image that the following text describes, without any additional explanation or description:
    
    The image depicts a cozy living room with a large flat-screen TV mounted on the wall. The room is furnished with a dining table surrounded by chairs, and a couch is placed nearby. A woman is standing in the room, possibly preparing to cook or serve a meal... [remaining model response]
    

    GPT-4o would response with the correct set of nouns that may be grounded in the image:

    - cozy living room
    - large flat-screen TV
    - wall
    - dining table
    - chairs
    ... [remaining noun phrases of interest]
    

    Utilizing an LLM would lead to more accurate and advanced sentence parsing, but we chose the simple tool spaCy to avoid unnecessary complexity.
    • We have revised Section 3.2 to better reflect that the usage of spaCy is not always necessary.

• In addition, as the reviewer suggests, we have incorporated the discussion of the previous round into the paper:

  • We have revised the abstract, Sections 1, 3.2, and 4.1 to consistently present our attend-and-segment based on point prompts and avoid confusions about attention accuracy.
  • New results on RES and PNG are included in the main paper (see Section 4.3).
  • Direct comparison with F-LMM is provided along with the results on RES and PNG (same as above, see Section 4.3).
  • We have revised the presentation of the experiment section, to better reflect the analytical purpose of the instance segmentation experiment (Section 4.1 and the beginning of Section 4).
  • We have provided visualized results to demonstrate that LMMs’ attention can correctly capture the semantics in complex scenarios (Figure 8 in Appendix D).

Once again, we sincerely thank the reviewer for providing us with the valuable feedback for strengthening our work. We have updated the manuscript as suggested by the reviewer, and would like to request a re-evaluation of our work. We are more than glad to discuss if the reviewer has remaining concerns, but also would like to kindly remind the reviewer that the deadline for updating the manuscript is very soon, so it would be best if we could address further questions earlier.

[Ref3] Deitke et al. Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models. arXiv 2024.

We appreciate the detailed feedback you provided for our submission. We are encouraged by your acknowledgement of our “well-organised and easy to follow” writing, “all clearly illustrated” method and experimental settings, and alleviating “the need for pixel-level supervision for grounded conversation.” We provide the following clarifications in response to your concerns:

W1. The proposed attend-and-segment paradigm relies on the external tool SpaCy to parse noun phrases, which would simply enumerate all noun words and phrases from a sentence, regardless of whether the words correspond to visual objects in the images. Therefore, the parsing process can be quite noisy and might require prior human knowledge to filter out undesired nouns.

In this work, we used a simple tool spaCy to parse model responses and demonstrate that they could already achieve strong grounding capabilities. We would like to underscore this significant finding of implicitly learned grounding, even using a simple NLP tool without complicated filtering mechanisms.

• For the GCG task (Table 2), we do not filter out any noun phrases for simplicity. As shown by the higher performance on the grounded conversation generation (GCG) task in Table 2, this simple design already achieved strong grounding results.

• In the instance segmentation analysis (Table 1), since we already know the 80 target object categories in MS-COCO, we 1) compute the embeddings of the category labels and the noun phrases with spaCy, 2) compute the cosine similarities between label embeddings and noun phrase embeddings, and 3) remove noun phrases whose similarities with all labels are lower than 0.5. This automated procedure was introduced in Line 323-346, Section 4.1. In other tasks, we can also set such “seed nouns” that can be grounded and remove undesired nouns that are dissimilar to them.

• Furthermore, our method is not tied with spaCy, and we could utilize more advanced tools like large language models (LLMs) to automatically filter undesired nouns. For instance, we may provide the model response (image from https://cocodataset.org/#explore?id=139) to GPT-4o with the following prompt:

  Extract and list all noun phrases that are likely found in the image that the following text describes, without any additional explanation or description:
  
  The image depicts a cozy living room with a large flat-screen TV mounted on the wall. The room is furnished with a dining table surrounded by chairs, and a couch is placed nearby. A woman is standing in the room, possibly preparing to cook or serve a meal... [remaining model response]
  

  GPT-4o responds with the correct set of nouns that may be grounded in the image:

  - cozy living room
  - large flat-screen TV
  - wall
  - dining table
  - chairs
  ... [remaining noun phrases of interest]
  

  which does not include nouns that cannot be grounded (e.g., “The image”). Based on this automatic noun extraction and filtering, we can achieve accurate grounding and interactive conversation with users.

W2. The proposed method is only evaluated on the GCG task, lacking experimental verification under tasks like referring segmentation which has been widely adopted in existing grounding LMMs.

We greatly appreciate your suggestions on additional evaluation on other tasks. We have performed experiments on referring expression segmentation (RES) and panoptic narrative grounding (PNG) and achieved stronger performance compared with prior zero-shot methods that are also not trained with explicit grounding supervision. For example, our approach achieves 45.3 average recall in PNG, outperforming the previous zero-shot state-of-the-art method by 6.8 average recall. The details are included in Section D of the updated manuscript. The results on RES and PNG further demonstrate that LMMs can implicitly learn to accurately perform visual grounding without explicit grounding supervision, and our approach is the key to unlock such grounding ability.

Thanks for the authors' diligent efforts to resolve reviewers' concerns. For now, I am quite split on the final assessment.

On the one hand, my concerns about the parsing approaches have been well addressed, showing quite reasonable parsing results from LLMs like GPT4-O. The authors also provided quantitative results on more grounding benchmarks, including PNG and RES.

On the other hand, I would like to insist on my concerns about using the diffusion unets, or employing any other visual backbone to improve visual grounding. Such design choices indicate that the grounding ability of existing LMMs is too weak to generate reasonable results, which requires additional re-training of an LMM with a third-party visual module. From my experience, the resources in re-training an LMM can be comparable to an LMM with grounding supervision. And the resources used in building modern LMMs are also increasing (as we have seen in Llava-Next/Llava-One-Vision and Cambrian).

I am hanging over between 5 and 6. And I wish to see the feedback from other viewers and ACs.

We appreciate the detailed feedback you provided for our submission. We are encouraged by your acknowledgement of our discovery of the “grounding ability of LMMs without explicit grounding supervision,” our attend-and-segment mechanism being able to “find out the segmentation masks,” and our “impressive results on the grounded conversation generation task.” We provide the following clarifications in response to your concerns:

W1.1. As the core motivation of this work, the efficacy of attention maps from LMMs should be validated. The precision and recall of attention maps representing objects are expected to be reported, which indicate whether they find out the regions of interest and whether the irrelevant objects/background are detected respectively.

We would like to clarify that our attend-and-segment method only uses the point with the highest attention value from the attention map for prompting SAM. In fact, we do not require the entire attention map to be precise, as long as it is able to produce a reasonable starting point for SAM as a prompt. Therefore, we focus on validating the efficacy of the prompt point produced by the attention map in our experiments. In our pilot study of instance segmentation (Table 1), we already validated that our prompt extracted from LMM attention maps could provide effective grounding, even without SAM:

• The PAcc metric computes the ratio of prompt points derived from attention maps that fall into the masks of the correct object category, which exactly reflect “whether the irrelevant objects/background are detected.”

• The mask AR metric reflects the ratio of regions of interest that can be discovered by the attention maps. Note that even if the prompt point falls into the correct region, SAM may not provide a perfect mask, so this AR metric is a lower bound of the actual object recall. Even so, we reach over 45% recall on large objects. In other words, more than 45% large objects can be discovered by the attention maps. We notice that the recall on small objects is low, which is due to the relatively low resolution (336 x 336) in the current LLaVA-based paradigm.

W1.2. Employing a single point with the highest attention value may miss to hit multiple objects or multiple regions of interest. It is unclear how to deal with these situations and how good this design is for capturing the most relevant object or region.

Although a single point usually only locates one object at a time, we observe that our approach is still able to locate multiple objects of the same category in a complex scene.

• Different from grounding LMMs that typically generate one mask per query, our approach is able to generate one grounding mask for every noun phrase in the response. In other words, our approach is not limited to capturing just one object or region in each response.

• Therefore, when generating a description of a complex scene, the LMM can describe multiple similar objects separately, and then our attend-and-segment method produces segmentation masks for all individual objects that are mentioned in the model response. For example, in Figure 8 of the updated manuscript, we visualize examples where the attention map can correctly differentiate one specific object from other similar objects and locate it.

W1.3. The attention maps from different LMMs and different prompts are also worth exploration.

In Table 1, we already compared various LMMs including LLaVA (with CLIP/ConvNeXt CLIP/DINOv2 backbones), our DiffLMM, and Cambrian-1. Different prompts affect the output token sequence, and thus change the final grounding results. Per the reviewer’s request, we try three different prompts in the instance segmentation analysis with DiffLMM with SD-1.5+CLIP backbone (see Table 1 in the main paper):

• Prompt 1 (Standard, the same as in the original experiments): “Describe the image in detail.”
• Prompt 2 (Concise): “Briefly describe the image in a few words.”
• Prompt 3 (Focusing on large objects): “Focus on the large objects in this image and describe them.”

The results are summarized as below:

In general, prompts can change the behavior of the model response, and thus influence how attention maps can be leveraged for object grounding. The second prompt leads to shorter responses and higher precision but lower recall. The third prompt focuses on large objects and is more beneficial for locating such objects.

3. Reviewers hPdE and r6Gw questioned the necessity of integrating diffusion-based features in DiffLMM. We provided clarifications: 1) The main objective of DiffLMM is to improve the grounding ability and preserve the general conversation ability, rather than improving over LLaVA on VQA tasks. DiffLMM addresses the limitation of prevalent grounding LMMs (e.g., LISA and GLaMM), which lose the general conversation ability after grounding-specialized finetuning. 2) The grounding advantages have been shown in Tables 1-4, and the preservation of general VQA is demonstrated by Tables 5 and 7. 3) Our main contribution, attend-and-segment, can be employed with existing LMMs (e.g., LLaVA-1.5, LLaVA-NeXT, and Cambrian-1) without retraining a diffusion-based model.

   Reviewer hPDe suggested a revision of the ablation study to better show the objective, and we have revised Section 4.5 and Table 7 as requested. Reviewer r6Gw replied about the concerns of using a different visual backbone for improving visual grounding and additional training costs. We have provided further explanation: 1) DiffLMM requires minimal changes to LLaVA-1.5 and uses the same training data, so the training cost is much lower than prevalent grounding LMMs (e.g., LISA and GLaMM). 2) Even without any retraining, our attend-and-segment strategy unlocks existing LMMs’ grounding ability, which already achieves better performance in GCG than GLaMM, as shown in Table 2.

4. Reviewers r6Gw and 4QCn had questions about the parsing process with the NLP tool spaCy, which simply enumerates all noun phrases without filtering or complex semantics. We clarified by explaining our straightforward usage of spaCy in the instance segmentation and GCG tasks, and provided examples of parsing model responses with more advanced tools like GPT-4o, and parsing complex phrases involving multiple nouns with spaCy.

   Reviewer r6Gw expressed that our response resolved the concern. Reviewer 4QCn raised follow-up questions on attention map quality and direct comparison with prior methods that automatically generate targets. We have responded with additional discussions to explain that the attention map is indeed focusing on the correct targets, and we adopt the simplest tool to find noun phrases as targets automatically, which induces consistent comparison with prior methods.

In summary, we are deeply grateful for the productive conversations with the reviewers and we believe that we have meticulously addressed all reviewer questions and integrated the results of author-reviewer discussions into the manuscript revision. Our work, as the first work that discovers the implicit grounding ability in LMMs trained without grounding supervision, is a valuable contribution to the LMM research community. We humbly request your favorable consideration of our submission. Again, we appreciate all the reviewers, ACs, SACs, and PCs for the time and devotion.