VL-SAM-V2: Open-World Object Detection with General and Specific Query Fusion

We thank the reviewers for their constructive comments. We address the reviewers' concerns in the rebuttal text below.

Q1: Figure 2 does not state where the text and image features entering GS Fusion originate

We process the text and image features with the open-set model. Specifically, text features are extracted by the BERT encoder, and image features are obtained by a Swin backbone.

We will add the details in the paper.

Q2: Inference cost

We test the inference time and GPU memory consumption of VL-SAM-V2 on a single NVIDIA A800 GPU. We use Swin-T as the backbone and InternVL-2.5-8B as the VLM. We can find that the main inference time and GPU memory are spent on the VL-SAM to generate general queries.

Q3: How does GS Fusion differ from earlier multi-query DETR or prompt-fusion approaches

For multi-query DETR or prompt-fusion approaches, such as GLIP/GLIPv2 and MI-DETR, they employ cross-modality similarity to update queries one by one or utilize the split-then-concatenate method, whereas VL-SAM-V2 utilizes full similarity to update two queries simultaneously.

We will add the discussion to the paper.

We thank the reviewers for their constructive comments. We address the reviewers' concerns in the rebuttal text below.

Q1: Differences between VL-SAMv2 and simply combining the existing VL-SAM and LLMDet methods

In VL-SAM-V2, we propose a general and specific query fusion module to fuse two distinct types of queries with the attention mechanism in a lightweight manner. To demonstrate the effectiveness of the proposed fusion module, we report simple ensemble results of VL-SAM and LLMDet in the following table. The results show that our fusion method significantly outperform simple ensemble method.

Q2: Failure modes

As discussed in the Limitations Section, since we utilize vision-language models to generate point prompts, VL-SAM-V2 may fail to predict correct results when VLMs give incorrect responses (including hallucinations). Specifically, when hallucinations happen, VLMs can give wrong object names or point locations, leading to incorrect predictions. In addition, VLMs may miss objects in the image. For instance, as shown in the last picture of Figure 4 in the paper, VLMs fail to give the "Tree" class. Subsequently, our method can not give a prediction of “Tree”.

Q3: Evaluation on other datasets

We provide the open-set evaluation results on LVIS, COCO and CODA[1]. The results show that our method achieves the best results on various datasets, demonstrating the effectiveness of VL-SAM-V2

[1] Li K, Chen K, Wang H, et al. Coda: A real-world road corner case dataset for object detection in autonomous driving. European conference on computer vision.

Q4: Inference-time computational overhead

We test the inference time and GPU memory consumption of VL-SAM-V2 on a single NVIDIA A800 GPU. We use Swin-T as the backbone and InternVL-2.5-8B as the VLM. We can find that the main inference time and GPU memory is spent on the VL-SAM to generate general queries.

Q5: Ablation on fusion mechanism

We replace our GS Fusion with naive concatenate and report the results in the following table. We can observe that GS Fusion achieves better fusion performance, outperforming naive concatenate by 2.7 AP (rare). The results demonstrate that the fusion architecture is important in our framework.

We thank the reviewers for their constructive comments. We address the reviewers' concerns in the rebuttal text below.

Q1: How is your query fusion different from GLIP/GLIPv2

The key difference is the similarity generation during the attention mechanism. GLIP/GLIPv2 adopts cross-modality similarity while VL-SAM-V2 uses full similarity. Specifically, given two different queries, q1 and q2, the similarity matrix s in GLIP/GLIPv2 is calculated with $s=q_1\times q_2^T.$ Then, q1 and q2 are updated by

$$q_1=softmax(s)\times W_2q_2;~~~~~~~q_2=softmax(s^T)\times W_1q_1,$$

where W1 and W2 are learnable weights. In contrast, we first concatenate q1 and q2:

$$q=concat([q_1,q_2])$$

Then, we calculate self-attention for q and split the updated q into q1 and q2

Q2: Novelty or benefit of denoising point strategy

There are two key differences between the denoising point strategy and DINO denoising training. First, our training strategy directly predicts boxes from noisy points, while DINO predicts box offsets from noisy boxes. Second, the motivation of our training strategy is to reduce reliance on expensive VL-SAM inference. In contrast, DINO denoising training is often used to solve the instability of bipartite matching.

Q3: Open-ended detection performance of recent vision-language models like Qwen2/LLaVA-OneVision

We report the open-ended detection performance of recent VLMs, including InternVL-2.5-8B, QwenVL-2.5-7B, and LLaVA-OneVision-7B, on LVIS. We follow the official spatial understanding inference code of QwenVL-2.5-7B to obtain the detection results. The results show that recent vision-language models perform poor in open-ended multi-object detection.

Q4: Results on COCO settings and other benchmarks

We provide the open-set evaluation results on LVIS, COCO, and CODA[1]. The results show that our method achieves the best results on various datasets, demonstrating the effectiveness of VL-SAM-V2.

[1] Li K, Chen K, Wang H, et al. Coda: A real-world road corner case dataset for object detection in autonomous driving. European conference on computer vision.

Q5: Analysis of failure cases

As discussed in the Limitations Section, since we utilize vision-language models to generate point prompts, VL-SAM-V2 may fail to predict correct results when VLMs give incorrect responses (including hallucinations). Specifically, when hallucinations happen, VLMs can give wrong object names or point locations, leading to incorrect predictions. In addition, VLMs may miss objects in the image. For instance, as shown in the last picture of Figure 4 in the paper, VLMs fail to give the "Tree" class. Subsequently, our method can not give a prediction of “Tree”.

Q6: More detailed ablation analysis

We first provide the ablation studies of the number of queries. We can find that the performance of VL-SAM-V2 improves when the number of queries increases. However, more queries bring more inference time. Thus, we choose 900 queries as the default setting, as it achieves a good efficiency-accuracy trade-off.

Then, we provide the ablation studies of lambda in the following table. The results show that VL-SAM-V2 is robust to lambda_2. For lambda_1, VL-SAM-V2 performs well when lambda_1 is smaller than 1.0. Increasing lambda_1 above 1.0 may introduce false positive points during denoising training, hurting overall performance.

We thank the reviewers for their constructive comments. We address the reviewers' concerns in the rebuttal text below.

Q1: Inference cost

We test the inference time and GPU memory consumption of VL-SAM-V2 on a single NVIDIA A800 GPU. We use Swin-T as the backbone and InternVL-2.5-8B as the VLM. We can find that the main inference time and GPU memory are spent on the VL-SAM to generate general queries.

Q2: Failure case

As discussed in the Limitations Section, since we utilize vision-language models to generate point prompts, VL-SAM-V2 may fail to predict correct results when VLMs give incorrect responses (including hallucinations). Specifically, when hallucinations happen, VLMs can give wrong object names or point locations, leading to incorrect predictions. In addition, VLMs may miss objects in the image. For instance, as shown in the last picture of Figure 4 in the paper, VLMs fail to give the "Tree" class. Subsequently, our method can not give a prediction of “Tree”.

Q3: Confusion between “point prompts”, “noisy points”, and “ranked queries”

Sorry for the confusion caused by the unclear names. "point prompts" denotes the points priority generated from VL-SAM. "noisy points" is the sampled points within ground-truth bboxes by Eq.1 in the paper. "ranked queries" represents learnable queries, which will be matched with points (including "point prompts" and "noisy points") by ranking before sent into GS Fusion.

We will give a more precise definition in the paper.

Q4: Joint attention mechanism improves over naive concatenation or late fusion

We replace our GS Fusion with other fusion methods, including naive concatenate and late fusion. The results of different fusion methods for open-set detection are shown in the table. We can observe that GS Fusion achieves the best fusion performance, outperforming naive concatenate by 2.7 AP (rare).

Q5: COCO or autonomous driving scenarios evaluation

We provide the open-set evaluation results on LVIS, COCO, and CODA[1]. The results show that our method achieves the best results on various datasets, demonstrating the effectiveness of VL-SAM-V2.

[1] Li K, Chen K, Wang H, et al. Coda: A real-world road corner case dataset for object detection in autonomous driving. European conference on computer vision.

Q6: Compared with other segmentation alternatives

Our method can be technically combined with any point/box prompt methods. As shown in the table, replacing SAM with other segmentation methods, including Efficient SAM[1] and GLEE[2], can achieve favorable segmentation performance.

[1]Zhang Z, Cai H, Han S. Efficientvit-sam: Accelerated segment anything model without performance loss. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2]Wu J, Jiang Y, Liu Q, et al. General object foundation model for images and videos at scale. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Q7: Novelty about denoising point training and query ranking

For denoising point training, there are two key differences compared to DETR denoising training. First, our training strategy directly predicts boxes from noisy points, while DETR predicts box offsets from noisy boxes. Second, the motivation of our training strategy is to reduce reliance on expensive VL-SAM inference. In contrast, DETR denoising training is often used to solve the instability of bipartite matching.

For query ranking, we are actually motivated by the anchor assignment in DETR. In DETR, the content queries are selected from features. Thus, the queries are naturally combined with the score. In contrast, our VL-SAM-V2 only has points. To match points with the queries, we follow DETR to predict the score of each point.

Q8: How much of the improvement is due to the method rather than the foundation model

We directly evaluate the open-ended object detection performance of the foundation models (including InternVL-2.5-8B, QwenVL-2.5-7B, and LLaVA-OneVision-7B). We follow the official spatial understanding inference code of QwenVL-2.5-7B to obtain the detection results. As shown in the table, VL-SAM-V2 outperforms InternVL-2.5 by a large margin. These results demonstrate that simply adopting the foundation model can not obtain good detection results. In contrast, adopting general and specific fusion in VL-SAM-V2 can boost the performance, showing the effectiveness of the proposed method.

Dear Reviewer 1JWr,

Could you kindly check whether the authors' rebuttal has adequately addressed your concerns? We truly appreciate the time and expertise you've invested in evaluating the work.

Thanks for your efforts.

Best regards,

Your AC