OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts

We sincerely thank Reviewer 4 for the positive feedback and valuable insights, especially appreciating your recognition of OpenWorldSAM’s diverse prompting capabilities, our novel tie-breaker design for handling multi-object scenarios, and the efficiency of our training method.

Below, we address your constructive concerns and questions:

1. Multi-scale ambiguous segmentation inherited from SAM2

Concern: OpenWorldSAM inherits SAM2’s issue of predicting multiple masks for single prompts in multi-scale ambiguous scenarios.

Response: Indeed, our approach builds upon SAM2, and thus inherits its architectural characteristics. However, our method specifically addresses instance-level ambiguity, not scale-level ambiguity. Unlike visual prompts (points/boxes), text prompts typically introduce instance ambiguity rather than scale ambiguity (see Figure 3).

The reviewer correctly pointed out that by default, SAM outputs 3 valid masks given one query prompt (multimask_output=True). Importantly, we disable SAM's default multimask_output setting during training, ensuring the model predicts only one mask per query (1 text prompt → $K=20$ queries → 20 mask predictions, instead of $20 \times 3= 60$ masks). We are encouraging the model to predict what exactly the text prompt asks for. Thus, OpenWorldSAM does not inherit the multi-scale ambiguity directly.

Additionally, future improvements to SAM2 will naturally enhance OpenWorldSAM, given our modular, plug-and-play architecture.

2. Inference speed concern

Concern: Reviewer points out the potential slowdown in inference speed.

Response: We conducted detailed profiling to quantify the impact of adding the VLM and our adapter modules to SAM. Below, we present inference timing breakdowns for processing a single 1024×1024 image on an NVIDIA A5000 GPU, averaged over five independent runs:

Single Text Prompt (20 queries):

Summary:

• SAM modules total time: 373.35 ms (81.0%)
• Non-SAM modules total time: 87.21 ms (18.9%)
• Non-SAM overhead: 87.21 ms

Six Text Prompts ($20\times6=120$ queries):

Summary:

• SAM modules total time: 539.72 ms (78.4%)
• Non-SAM modules total time: 148.65 ms (21.6%)
• Non-SAM overhead: 148.65 ms

Takeaway:

1. The profiling results show that adding the VLM and adapter modules results in only a moderate increase in inference time (approximately 19–22% overhead). Most computational cost remains within SAM’s backbone and mask decoder.

2. Mask Decoder scaling: sam_mask_decoder cost grows almost linearly with (K × P).

   • Going from 1→20 queries (same prompt) adds ~41 ms.
   • Going from 1 prompt→6 prompts (120 queries) adds a further ~162 ms.

   Note that one text prompt mimics user clicks 20 times on an image. If automatic mask generation is desired without user intervention, SAM’s built-in auto-mask generator uses a dense 32×32 grid of point prompts, incurring significantly higher costs compared to our text-based prompting approach.

3. Overall overhead: Relative to one vanilla SAM2 call, our pipeline is approximately 39% slower for a single prompt (332 → 461 ms). However, it becomes approximately 3$\times$ more efficient when handling three or more prompts, as the backbone and VLM overhead are amortized.

Thus, our enhancements introduce manageable overhead, maintaining practical usability in real-world applications.

3. Evaluation protocol clarity in Table 1

Concern: Using Oracle-Prompts for OpenWorldSAM but Global-Matching prompts for other methods in Table 1 may be misleading.

Response: We appreciate this clarification. Table 2 already provides both protocols explicitly for X-Decoder, but we agree that clearer labeling and alignment are necessary. We will explicitly state the evaluation modes ("Oracle-Prompt" for OpenWorldSAM, "Global-Matching" for other methods) in Table 1 in the revised manuscript to ensure an accurate, fair comparison. Importantly, OpenWorldSAM remains superior even under aligned evaluation conditions in Table 2.

Additional Questions:

1. Global-Matching prompts for OpenWorldSAM

OpenWorldSAM, like SAM, is specifically trained for interactive, prompt-conditioned segmentation (Oracle-Prompt mode). It is inherently unsuitable for Global-Matching evaluation, which involves prompts that may not correspond to objects in the image, causing issues as negative sampling was not incorporated in SAM’s original training recipe. Thus, evaluating OpenWorldSAM with global prompts would not accurately reflect its intended usage or strengths.

2. Performance with alternative SAM backbones

Due to time and computational constraints, we could not perform additional training with SAM or lighter SAM2 variants. Nonetheless, since our approach is modular, retraining our lightweight adapter to align with other backbones (e.g., SAM2-small/base) is straightforward. Given the minimal performance gaps reported among SAM2 variants in the table bellow (taken from SAM2's Github repo), we expect our method's performance to remain robust across different backbone scales:

3. Combining text and point/box prompts

Currently, OpenWorldSAM does not simultaneously support text and point/box prompts. Our goal was precisely to reduce or eliminate user intervention by using positional tie-breakers that emulate spatial clicks. However, incorporating dual-modality prompts is an interesting direction we intend to explore in future work.

4. Performance on challenging datasets like LVIS

We agree that evaluating on challenging large-scale datasets like LVIS-1203 would further validate our method’s scalability and robustness. Given our strong performance on similarly challenging datasets (e.g., ADE-857), we anticipate competitive generalization to LVIS. Furthermore, incorporating strategies like Grounding DINO's large-scale pretraining (datasets: Objects365, OpenImages, GoldG, Cap4M, COCO, RefCOCO) could significantly enhance our model's capabilities. And VL-SAM’s preprocessing stage where a 17B parameter multimodal reasoning model (CogVLM-17B) is asked to list all the objects in the image could help transition OpenWorldSAM from oracle-prompt conditions toward fully automated open-vocabulary segmentation. We plan to explore these promising avenues in future research.

We deeply appreciate your valuable suggestions and insights, which will significantly enhance the final manuscript.

We sincerely thank Reviewer 3 for the positive and insightful comments, especially appreciating your recognition of our clear exposition, the effectiveness of our proposed tie-breaker module inspired by DETR, and the improved accuracy achieved by OpenWorldSAM over baseline methods.

Below, we address your constructive questions and concerns:

1. Attribution of state-of-the-art results to BEiT-3 vs. proposed methods.

Concern:

Reviewer points out that replacing CLIP/Vicuna with BEiT-3 significantly improves performance, raising a question about the relative contributions of BEiT-3 versus our proposed modules.

Response:

Short answer — both are required.

We fully concur with the reviewer that BEiT-3 primarily contributes semantic reasoning. Our ablation study (Table 4) indicates this, where replacing BEiT-3 with late-fusion CLIP drastically decreases performance. The early fusion strategy of BEiT-3 indeed provides richer semantics closer to SAM's features. Early fusion lets every Transformer layer mix visual and textual evidence, yielding finer localization that late‑fusion CLIP cannot retrieve from a global text embedding.

However, Table 5 already shows that BEiT‑3 alone is not sufficient: if we remove tie‑breakers the model collapses. Tie‑breakers compensate for SAM’s weak semantics by spatially diversifying otherwise identical semantic queries. With language semantics present, removing the positional perturbations causes near‑complete failure: ADE‑150 AP plunges from 17.1 → 1.0 and mIoU from 56.8 → 1.2 (Table 5, E3 vs E1). Tie-breakers are thus essential in enabling multi-instance segmentation. Therefore, both the powerful BEiT-3 encoder and our novel tie-breaker strategy jointly enable our state-of-the-art performance.

2. Clarifying two-stage inference results and visualization discrepancy.

Concern:

The reviewer correctly notes that despite improved visual mask boundaries in our two-stage inference, Table 1 shows no substantial quantitative improvement, and the provided visualization in the appendix is limited.

Response:

Indeed, quantitative metrics (mIoU, PQ, AP) show negligible improvements primarily because:

1. Two‑stage inference only refines mask contours; IoU‑style metrics saturate once coarse localization is accurate, so small contour tweaks seldom raise mIoU/PQ/AP.
2. Amplified errors on hard examples. On incorrectly localized masks from stage 1, refinement anchored to incorrect regions can further degrade metrics.

As the rebuttal submission does not allow visuals, we will add some failure examples to the revised Appendix.

To substantiate the visual improvement despite stable metrics, we conducted a user-study. We ran a blind preference test on 50 random image pairs from ADE‑857 val:

• Participants: 10 graduate volunteers familiar with vision tasks.
• Task & options: For each of the 50 pairs, respondents answered a single forced‑choice question: “Which mask looks more accurate?” Choices: A, B, No discernible difference. Average completion time ≈ 15 min.
• Result: Across 10 × 50 = 500 judgements, Two‑stage preferred: 43.6 % (218/500); One‑stage preferred: 37.2 % (186/500), No difference: 19.2 % (96/500). Excluding “No‑difference”, Two‑stage wins 218/ (218+186) = 54.0 %.
• Takaway: Although the gain is modest, humans still favor the refined masks more often than the originals.

Given that the two-stage inference serves as an optional, low-cost post-processing step, users can conveniently enable or disable it based on their preference.

3. Dedicated multi-instance segmentation evaluation.

Concern:

Given multi-instance segmentation is a core claim, Reviewer 3 recommends a dedicated evaluation.

Response:

We agree on the importance of explicitly evaluating multi-instance segmentation. Notably, our primary results already include extensive multi-instance evaluation through panoptic and instance segmentation benchmarks (ADE20K-150, ScanNet-20). Specifically, OpenWorldSAM achieves new state-of-the-art panoptic (combining instance-level and semantic-level) segmentation performance on ADE20K-150 (35.2 PQ) and strongly outperforms baselines on instance-level metrics (16.9 AP on ADE20K-150). These results directly validate our multi-instance capabilities. Nonetheless, we will further emphasize these findings explicitly in the final manuscript to highlight this strength.

4. Ablation comparison between E3 and E5 and simplifying cross-attention.

Concern:

The reviewer observes that E5 (full three-layer cross-attention, 4.5M parameters) performs only slightly better than E3 (no cross-attention, 1.7M parameters) on ADE-150, recommending additional datasets comparison and exploring simpler cross-attention architectures.

Response:

Thanks for this valuable suggestion. We performed additional comparative experiments across multiple datasets, summarized below:

We observe consistently higher accuracy with E5 (three-layer cross-attention) across datasets, confirming the importance of multi-layer cross-attention. However, a single-layer variant (E5.1) significantly narrows the gap with fewer parameters (2.4M vs. 4.5M), suggesting a practical compromise between parameter count and accuracy. We will include this detailed analysis in the revised Appendix and explicitly discuss potential simplified variants as a recommended direction for practitioners seeking more parameter-efficient adaptations.

Minor formatting suggestion:

Thank you for noting the formatting oversight in Table 1 ("16.9" should be bold). We will correct this in the final manuscript.

We greatly appreciate your thorough and constructive feedback and will incorporate all suggestions to improve our manuscript clarity and robustness.

We sincerely thank Reviewer 2 for the positive feedback, particularly appreciating your recognition of our contributions such as the efficient learnable positional tie-breaker mechanism and the soft prompting Transformer block, significantly improving SAM2’s open-vocabulary segmentation performance in complex, multi-object scenes.

Below, we address the reviewer’s constructive questions and concerns:

1. Semantic richness of SAM’s encoder and the role of cross-attention and tie-breakers

Concern: SAM features have limited semantic richness, raising concerns about the effectiveness of the proposed cross-attention and tie-breaker mechanisms.

Response:

We fully agree with the reviewer that SAM’s encoder alone possesses limited semantic richness. The BEiT-3 encoder indeed plays a critical role by providing robust semantic alignment. The early fusion strategy of BEiT-3 provides richer semantics closer to SAM's features.

However, we argue that our OpenWorldSAM is a general plug-in architecture that aligns SAM’s embedding spaces with VLM’s embedding space while keeping all heavy backbones frozen. Although we instantiate the framework with SAM2 and BEiT-3 in this paper, neither component is required by design; alternative interactive decoders or vision-language encoders can be swapped in with only minor re-training of the adapter (Please refer to Appendix E.1 & Table 5). Appendix Table 5 surveys representative VLM encoders that could replace BEiT-3 in OpenWorldSAM with $\approx$ 5(M) adaptor parameters. For example, early-fusion encoders like BEiT-3 (e.g., VLMo, OFA, Florence-2) require zero architectural change beyond projector resizing and are therefore the most promising immediate swaps. Dual-encoders (CLIP family) need a shallow cross-attention adaptor to overcome missing image context.

Of course, with better VLMs whose embedding spaces are closer to SAM’s encoder space, our model will perform better.

On the role of cross-attention. Cross-attention injects BEiT-3 semantics into SAM’s high-resolution grid. Our proposed 3-layer soft-prompting Transformer (Fig. 4c) performs cross-attention over the high-resolution 64×64 feature map used by SAM2 for mask decoding. This strategy provides every query direct access to BEiT-3’s fused semantics at pixel-level granularity (~16K tokens), rather than relying solely on the coarse global [CLS] embedding discussed in Line 133. Empirically, removing the cross-attention block (while keeping other components intact) degrades performance: ADE-150 mIoU drops by 1.5 points, and ADE-857 mIoU by 0.9 points (Table 5, E3 vs. E5).

On the role of Tie-breakers. Tie-breakers address SAM’s limited semantics by spatially diversifying identical semantic queries. When BEiT-3’s semantics are present, disabling positional tie-breakers leads to a severe performance collapse. Specifically, ADE-150 AP drops from 17.1% to 1.0%, and mIoU plummets from 56.8% to just 1.2% (Table 5, E3 vs. E1). This highlights the critical role of tie-breakers in differentiating multiple instances spatially, especially in the presence of weaker intrinsic semantics from SAM.

2. Operational role and limitations of tie-breakers relative to BEiT-3

Concern: Semantic reasoning primarily comes from BEiT-3 rather than direct language-image alignment, raising questions about the operational role and limitations of tie-breakers.

Response:

Operational role of tie-breakers: BEiT-3 provides robust class-level semantic cues ("what"), whereas tie-breakers supply instance-level spatial cues ("where/how many"). During training, Hungarian matching compels each tie-breaker vector to specialize in spatially distinct regions, thereby enforcing diversity and preventing redundancy.

Limitations: Tie-breakers are fundamentally positional and thus cannot rectify semantic misclassifications arising from BEiT-3. If BEiT-3 misinterprets a semantic concept, tie-breakers cannot correct such semantic errors. Their effectiveness is contingent on BEiT-3’s correct semantic interpretation.

3. Justification for BEiT vs. CLIP and comparisons with EVF-SAM and SAM4MLLM

Concern: The paper would benefit from a deeper discussion of the choice of BEiT-3 over CLIP (especially regarding EVF-SAM) and a comparison to SAM4MLLM.

Response:

We appreciate this suggestion and further clarify below:

Relation to EVF-SAM: In the submitted manuscript, we (1) gave a shout out to EVF-SAM in Related Work L108 (“Inspired by the success of EVF-SAM, we enhance SAM further into the domain of open-vocabulary segmentation…”), (2) identified the problem of EVF-SAM in Figure 3c (“Naive solution [EVF-SAM]: A single language query for ‘zebra’ causes SAM2 to segment only the most salient instance.”), and (3) gave credits to EVF-SAM’s preliminary experiments on replacing CLIP with BEiT-3 in our ablations L292 (“…deep semantic integration is critical for aligning language concepts with visual regions, echoing findings by EVF-SAM”).

Given that EVF-SAM’s primary innovation lies in utilizing BEiT-3, we believe that our multiple citations throughout the submitted manuscript sufficiently acknowledge EVF-SAM's contributions, avoiding redundant discussion on the choice of BEiT-3 over CLIP.

While EVF-SAM also leverages BEiT-3, it is strictly limited to single-instance queries. As illustrated in Figure 3, EVF-SAM segments only the most salient instance when given category-level prompts, ignoring other instances. Our design (tie-breakers plus cross-attention) significantly advances EVF-SAM’s single-instance capability into a robust multi-instance, multi-task model, which is clearly demonstrated by our detailed ablations (Table 5).

Comparison with SAM4MLLM [Chen et al., 2024]: Despite employing a powerful 7B-parameter Qwen-VL backbone with ~29M LoRA-tuned parameters, SAM4MLLM achieves only a minor improvement (+0.5 cIoU) over OpenWorldSAM (74.5% vs. 74.0%) on RefCOCOg. Moreover, SAM4MLLM relies on a two-stage pipeline ("LLM-dialogue → point prompts → SAM"), incurring higher inference latency and failing to demonstrate multi-instance segmentation capabilities or results on challenging multi-instance benchmarks (ADE20K, PASCAL, ScanNet). In contrast, our lightweight 4.5M-parameter adapter with early vision-language fusion via BEiT-3 and positional tie-breakers consistently achieves state-of-the-art results across multiple open-vocabulary segmentation tasks and datasets.

We deeply appreciate your valuable suggestions and will incorporate all feedback into the manuscript.

We thank Reviewer 1 for the strong endorsement of our paper and insightful feedback! We are encouraged that you found our method effective, efficient, and well validated. We address reviewer comments below and will incorporate all feedback.

1. Effect of varying tie-breaker tokens ($K$)

Thanks for highlighting this point. To clarify, setting $K=20$ means for each prompt (e.g., a category name), our model can identify up to 20 distinct objects. For crowded scenes containing more than 20 objects per category, increasing $K$ is straightforward and advisable.

In practice, COCO images typically contain a moderate number of distinct categories and instances (the original COCO paper reports "on average, our dataset contains 3.5 categories and 7.7 instances per image." [1]). The chosen value should match or exceed the maximum expected number of objects per category. For reference, DETR [2] used 100 total queries, aligning roughly with the maximum number of objects per image. Our choice ($K=20$) results, on average, in approximately 70 queries per image (20 queries × 3.5 categories), providing ample coverage for typical scenes.

Further, Zhang et al. (2022) [3] observed that increasing queries initially improved Average Precision (AP), but then plateaued or even slightly declined when queries became excessive, indicating redundancy in higher query counts. However, recall does improve with more queries, since more detection slots increase the chance to find each object.

We conducted additional ablation experiments by varying $K$, pretrained on COCO and evaluated on ADE20K instance segmentation:

Observations. (1) Increasing $K$ from 10 → 20 improves recall and AP; beyond 20 gains saturate, mirroring the behavior reported for DETR‑style object queries; (2) Average Recall with max 100 detections per image (AR@100) improve when increasing $K$ from 10 → 20 → 30; (3) $K=20$ is optimal for balancing precision and recall in standard datasets.

We'll include these findings clearly in the supplementary materials.

2. Query collapse, forgetting, or overfitting on small datasets

We appreciate this concern and provide a detailed analysis below:

A. Query collapse:

During fine-tuning on RefCOCOg, each text prompt corresponds to exactly one target object, making it effectively a small-instance-count dataset scenario. We found our model generalizes robustly: cIoU scores demonstrate strong performance. We explicitly measured query collapse—the fraction of images where multiple tie-breakers predicted identical masks (IoU > 0.9). Results show minimal query collapse (<1%), even when fine-tuning on the smaller RefCOCOg-Mini (2k images):

B. Catastrophic forgetting:

To mitigate forgetting and small-dataset overfitting, we follow established adapter-style fine-tuning strategies. PSF‑SAM [4] observes that fully fine‑tuning SAM on a few hundred images “frequently results in catastrophic forgetting”, and proposes freezing most weights plus a lightweight adapter to preserve upstream skills. For OpenWorldSAM, all heavy encoders remain frozen, and only our lightweight 4.5M-parameter adapter is trained. This approach empirically preserves COCO-level zero-shot capabilities while adapting effectively to RefCOCOg.

Furthermore, we assessed the zero-shot performance on ADE20K following fine-tuning. After RefCOCOg‑Mini fine‑tuning, ADE‑150 mIoU drops by only 4.2 pts (60.4 → 56.2), indicating minimal loss of zero‑shot capability.

C. Overfitting:

We explicitly examined potential overfitting on RefCOCOg-Mini (2k images), finding no evidence of such:

This comprehensive analysis shows that our Hungarian matching objective implicitly enforces query diversity through one-to-one assignment, while the cross-attention layers (lines 155–166) further guide queries towards distinct visual regions. We will clarify this implicit enforcement of diversity in L178 and provide detailed experimental results in supplementary materials.

Thank you again for your insightful questions; we will incorporate these clarifications into the revised manuscript.

References

[1] Lin et al. "Microsoft coco: Common objects in context." ECCV 2014.

[2] Carion et al. "End-to-end object detection with transformers." ECCV, 2020.

[3]Zhang et al. "What are expected queries in end-to-end object detection?." arXiv preprint arXiv:2206.01232 (2022).

[4] Wang et al.. "An efficient fine tuning strategy of segment anything model for polyp segmentation." Scientific Reports 15.1 (2025): 14088.

Additional Questions

1. Clarification on processing multiple text prompts per forward pass

Our implementation supports an arbitrary number of text prompts per image, processed simultaneously in a single forward pass. Specifically, during the forward pass:

• We first tokenize all text prompts for each image simultaneously and repeat the visual tokens from BEiT-3 for each prompt, generating visual-text embeddings in one BEiT-3 forward pass (e.g., embeddings for "giraffe" and "rock" concurrently).
• Tie-breaker embeddings are replicated for each prompt independently, and the resulting queries are concatenated (e.g., concat[queries for "giraffe", queries for "rock"]). This combined stack is then processed through SAM’s prompt and mask decoders simultaneously. Thus, Figure 4(b) accurately represents this concurrent processing.

Evaluation setup in Tables 1–5: For comparability with previous studies (each image is seen in one forward pass), all text prompts per image were processed simultaneously in one forward pass in the reported evaluations.

Impact of batching prompts on accuracy: We empirically tested batching prompts versus separate processing on ADE20K-150:

The minor difference (0.4 mIoU) indicates that batching prompts neither significantly harms nor benefits segmentation quality. The reason is that BEiT-3 already provides a distinct CLS token for each prompt, and tie-breakers are independently replicated per prompt, minimizing cross-interference. The slight improvement from batching arises from the self-attention layers in our soft-prompting Transformer module, promoting query diversity.

How positional information is handled when multiple prompts are batched. Positional tie‑breakers are added independently to every prompt before the prompts are stacked. No extra “prompt‑id” encoding is introduced or needed; Hungarian matching during training ensures that the K queries attached to each prompt specialize on distinct spatial modes for that prompt only.

2. “What exactly is ‘soft‑prompting’ in the soft‑prompting transformer? Is there a trained prefix?”

“Soft‑prompting” here means that each language‑aware query $q_ᵢ$ is refined on‑the‑fly by cross‑attention with the frozen SAM2 level‑3 feature map (Fig. 4c). No static prefix or extra tokens are prepended as in LLM prompt‑tuning; instead the three Transformer layers are entirely learnable and act as an adapter that lets text semantics “look at” high‑resolution vision tokens before the mask decoder sees them.

3. Relationship between tie‑breaker tokens and positional encodings?

Tie-breaker tokens serve as instance-diversity embeddings, fundamentally different from standard positional encodings (learned or sinusoidal) that encode absolute pixel coordinates. While traditional positional encodings map pixels to unique spatial coordinates, our tie-breaker embeddings globally nudge identical semantic queries toward distinct latent spatial subspaces. During training, the Hungarian loss encourages each tie-breaker token to represent distinct, data-driven spatial modes (e.g., differentiating multiple "zebras" in Figure 3). The SAM2 image encoder still provides its standard 2‑D positional encodings, so spatial locality is preserved; tie‑breakers complement these by resolving instance ambiguity rather than spatial positioning itself.

4. Related work discussion

We appreciate the suggestions and will include references to the mentioned concurrent works in the revised Related Work section (§2). Additionally, we will explicitly compare these works quantitatively in Tables 1 and 3:

We sincerely thank the reviewer again for their detailed and constructive feedback, which will greatly enhance the clarity and comprehensiveness of our manuscript.