EVF-SAM: Early Vision-Language Fusion For Text-Prompted Segment Anything Model

1. Novelty:

Using a stronger backbone (BEIT vs. CLIP) and early fusion are both expected to yield better performance, and it's unclear what additional insights this paper provides. Early or late fusions of multimodal inputs have been explored in many earlier referring segmentation works, such as LAVT[1], CoupAlign[2], and MaskGrounding[3], which all reach similar conclusions as this paper regarding the superiority of early fusion. The authors should clarify the unique insights gained from this paper.

2. Clarity:

1. Given that early fusion is presented as a core contribution, a detailed explanation of the process is expected. However, this information seems to be missing from the paper. The only details provided are a high-level diagram in Fig. 3 and unclear wording in line 230, which states, "We leverage the 'early-fusion' vision-language model as the Multimodal Encoder to generate prompt embeddings for SAM." It's unclear whether early fusion is a proposed novelty or part of the original BEIT-3 architecture. It appears to be the latter, suggesting that the main change is swapping CLIP with BEIT-3. Is that true?
2. In lines 237-240, the authors claim that prompt embeddings with special tokens lead to "instability during both training and inference." However, it seems that EVF-SAM also uses a special token (the [CLS] token after projections) as prompt embeddings, and it's unclear how this differs and resolves the previously mentioned problems.
3. For Tab. 7, providing mIoU results for ADE20K would be beneficial, as cIoU is not commonly used for this dataset, making comparisons with existing methods difficult.

3. Unfair comparisons:

1. The authors claim that EVF-SAM is superior to decoder-only Large Language Models like LAVT and PSALM based solely on referring segmentation performance. However, these LLM-based models are capable of more than just referring segmentation, including reasoning segmentation, generic segmentation, VQA, etc. This comparison is unfair unless the authors can demonstrate superiority in these other tasks as well.

4. Insufficient ablations:

1. Based on Tab. 3, early fusion experiments using the widely adopted CLIP model are lacking, and the improvements seen with early fusion in ViLT are modest (average 66.6 vs. 51.7). In contrast, for BEIT-3-Large, the improvement is significant, with performance increasing from an average of 52.3 to 78.0. This suggests that the early fusion mechanism may be particularly effective for BEIT-3-Large but may not be universally beneficial across other vision-language models. Given this, can the strong performance in Tab. 2 still be achieved with the more commonly used CLIP model?
2. The experiments in Tab. 2 are done using different training data, which makes fair comparison difficult. It will be great if the author can provide fairer comparisons using same training data as other works such as PSALM and LISA.

References:

[1] LAVT: Language-Aware Vision Transformer for Referring Image Segmentation, CVPR 2022

[2] CoupAlign: Coupling Word-Pixel with Sentence-Mask Alignments for Referring Image Segmentation, NeurIPS 2022

[3] Mask Grounding for Referring Image Segmentation, CVPR 2024

1. The baseline of LLaVA is not fair enough since LISA mainly focuses on its Reasoning Segmentation ability. So it is necessary to fine-tune LLM-based models like LISA, GSVA with referring expression segmentation tasks to achieve a similar level.

2. The training samples of LISA-7B models are 2 img/gpu $\times$ 8 gpu $\times$ 10 grad-accumulation $\times$ 500 steps/epoch $\times$ 10 epoch = 800k, however EVF-SAM has 128 samples/iter $\times$ 15k steps = 1,920k samples. The number of training samples should also be ablated carefully.

3. The novelty is somehow limited since the architecture and training methods of incorporating VLMs into RES, REC, GRES tasks have been investigated by previous works such as LISA, GSVA, PixelLM, GLaMM, u-LLaVA, PSALM. Using another early-fusion multimodal encoder seems not have enough innovation in terms of methodology. However, considering the abundant experiment results, it is not appropriate to deny the contribution of this paper. The authors are encouraged to provide more theoretical analyses and insights to justify why the early-fusion paradigm encoders bring such significant improvements.

• Lack of novelty

  • The proposed framework seems to be an addition of BEIT-3 as an multimodal encoder to SAM. No new modules nor techniques, but simply adding an existing, well-known multimodal encoder to another existing, well-known segmentation model.

• Omitted references

  • When explaining previous work that explore the text-prompted abilities of SAM (line 051~), authors explain each work but omits references to them. Thus, the reviewer cannot be sure of these findings they draw from these work are valid.
  • Findings used to justify the method also lacks references: ie. these sentences do not have references "However, the uncontrolled length of the answering query introduces instability during both training and inference. Forcing the model to conform to a specific answering template can lead to language drift."

• Unclear writing

  • Authors should better explain what they mean by "matches" in "Fast-SAM (Zhao et al., 2023) matches the similarity of CLIP (Radford et al., 2021) features between the text and Region of Interest (RoI) of image."
  • Authors simply list these short sentences without sufficient explanation in Section 2.2: "PerceptionGPT (Pi et al., 2023) proposes an end-to-end architecture. u-LLaVA (Xu et al., 2023) supports multi-task. PSALM (Zhang et al., 2024) imports mask tokens to LMM input for better performance."
  • Authors claim that traditional RES models are limited since they "They fail to integrate with large-scale foundation models... e.g., SAM, LLaVA," but methods that employ LLMs and LMMs have "heavy computation burden", which seem contradictory. I understand that authors wanted to explain that traditional RES models are not using foundation segmentation/vision models and thus missing out on the generalizablity of these models and that models that use LLMs are inherently computationally heavy, but using LLaVA (which is LLM-based) as an example for the first point as a model that traditional models should try to incorporate and then claiming LLM-based models are heavy is inconsistent.
  • Authors explain that they use a special [semantic] token to alleviate semantic conflict but do not explain what the conflict is. Although I acknowledge the reference to UniLSeg, I think a short explanation would be very helpful.

1. The authors suggest that using a Multimodal Encoder with early fusion enhances SAM’s text-prompting capability. However, the study validates only two multimodal models—CLIP and BEIT-3. The efficacy of multimodal input in these cases could stem from the models’ unique training strategies, such as BEIT-3’s use of shared multi-head self-attention. It remains unproven whether other multimodal models, such as VLMO [1] and CoCa [2], would also benefit from this early fusion and Multimodal Encoder strategy. Further evaluation is needed to determine if these approaches can enhance SAM’s text capability across a broader range of multimodal models.
2. The authors state that “Despite their advantages in terms of fewer parameters and faster inference speeds, these methods either achieve less competitive results or require vast amounts of data due to their lack of integration with foundation models.” However, in Table 2, there is no clear indication of the parameter counts or training data volumes for the comparison methods, making it difficult to assess the efficacy of the proposed approach. To strengthen the results, the authors should explicitly report parameter counts, training data volumes, and inference speeds for the comparative methods.
3. The authors validate their method solely on the referring expression segmentation benchmark. With the introduction of text capability in SAM, there is a risk that EVF-SAM might overfit this specific benchmark, potentially reducing its effectiveness on unseen or less structured data. To demonstrate the robustness of EVF-SAM’s text-prompting ability, additional validation across multiple tasks, such as zero-shot semantic segmentation, zero-shot instance segmentation, and open-vocabulary segmentation, would be added.
4. The paper primarily compares EVF-SAM against methods specific to the referring expression segmentation task. To provide a comprehensive comparison, the authors should also consider benchmarking against other methods aimed at enhancing SAM’s text capability, such as Open-Vocabulary SAM [3], SAM-CLIP [4], and AlignSAM [5].
5. In the related work section, the authors should include a broader summary of research focused on SAM’s limitations and methods for enhancing its various capabilities. Relevant works include [6], [7], [8], and [9].

[1]Bao, Hangbo, et al. "Vlmo: Unified vision-language pre-training with mixture-of-modality-experts." Advances in Neural Information Processing Systems 35 (2022): 32897-32912. [2]Yu, Jiahui, et al. "Coca: Contrastive captioners are image-text foundation models." arXiv preprint arXiv:2205.01917 (2022). [3]Yuan, Haobo, et al. "Open-vocabulary sam: Segment and recognize twenty-thousand classes interactively." European Conference on Computer Vision. Springer, Cham, 2025. [4]Wang, Haoxiang, et al. "Sam-clip: Merging vision foundation models towards semantic and spatial understanding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [5]Huang, Duojun, et al. "Alignsam: Aligning segment anything model to open context via reinforcement learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [6]Xie, Defeng, et al. "Edit everything: A text-guided generative system for images editing." arXiv preprint arXiv:2304.14006 (2023). [7]Sun, Yanpeng, et al. "VRP-SAM: SAM with visual reference prompt." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [8]Yao, Jingfeng, et al. "Matte anything: Interactive natural image matting with segment anything model." Image and Vision Computing 147 (2024): 105067. [9]Yu, Tao, et al. "Inpaint anything: Segment anything meets image inpainting." arXiv preprint arXiv:2304.06790 (2023).

1. The paper has a limited tech novelty and contribution. Early cross-modal fusion between text and image features using cross-attention has been widely explored in popular language grounding methods, such as Grounding DINO, Grounding DINO1.5 [a] and GLIP [b]. There is no clear discussion on the differences to these existing methods. The language grounding tasks and referring segmentation tasks are very close related.

2. The paper only evaluates on the COCO related datasets, more in-the-wild testing on more diverse datasets will also be appreciated, such as using the ReasonSeg proposed by LISA or ReasonSeg-Ins by LISA++.

3. In Table 2, all comparing method are using different text-prompt based encoders, this makes the results comparison not that fair. As BEIT-3 is not the contribution of the paper. If using ViLT as the encoder, EVF-SAM's model performance will decrease from 82.1 to 73.9 on Ref-COCO val set. This means the SOTA performance of EVF-SAM is highly dependent on pretrained BEIT-3 while the comparing methods in Table 2 didnot use it.

4. Besides model parameter comparison, a comparison on the running inference speed in Table 6 will help readers understand better.