SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding

We would like to thank the reviewer for their constructive feedback and comments. Below are our itemized responses. We hope that our responses address all questions and concerns and clarify our contributions.

1. [Writing Suggestions] We appreciate the reviewer's suggestions regarding the refinement of the writing and structure of the paper. These suggestions will be incorporated in the next revision.

2. [Comparison with SAM-based Models] Our primary goal with SAM-CLIP is to integrate the abilities of CLIP into the SAM model to achieve enhanced semantic understanding. We demonstrate that SAM-CLIP exhibits superior zero-shot semantic segmentation performance compared to previous methods. To our knowledge, as of the submission date of this paper, the state-of-the-art zero-shot semantic segmentation models were all CLIP-based; thus, we did not have SAM-based models for comparison in this task. It is important to note that the Semantic-SAM and SEEM models, as mentioned by the reviewer, are trained on image datasets with semantic segmentation annotations. Models trained with semantic segmentation annotations are not considered zero-shot in the literature. Therefore, these SAM-based models cannot be compared for the zero-shot semantic segmentation task. We thank the reviewer for bringing Semantic-SAM and SEEM to our attention. Relevant discussion will be included in the revision.

3. [Suitability for Edge Devices] To run an application requiring both SAM and CLIP abilities, one would need to store and deploy both models on the device. However, by merging SAM and CLIP into a single ViT-B model, only one ViT model is required to complete tasks that previously needed two models. Moreover, for processing each image on the edge device, SAM-CLIP only requires one forward pass through the backbone, whereas using two separate models would need 2x forward passes (doubling the compute and memory footprint costs). Therefore, SAM-CLIP is advantageous in terms of storage, memory, and compute costs for edge devices.

   Please note that we are not introducing a new architecture in this work, but proposing to merge existing ViTs into a single one. Hence, existing benchmarks on different devices are valid (e.g., the memory and runtime performance of SAM-CLIP on any device is identical to that of SAM with ViT-B on such a device).

4. [More Details] We thank the reviewer for suggesting elaborating more on the training technical details and distillation background review. These points will be addressed in the next revision. In general, all our design choices (mentioned in Sections 3, 4.1, and Appendix A) are based on the trade-off between forgetting SAM’s original capabilities and learning CLIP’s capabilities. Instance Segmentation (SAM’s capability), zero-shot classification (CLIP’s capability), and zero-shot semantic segmentation (a new joint capability) are the metrics we used to guide our choices.

   Our training process consists of two stages: i) CLIP-head probing and ii) joint training of the SAM-head, CLIP-head, and the ViT backbone using a multi-task distillation loss. In the first stage, only the CLIP-head is trainable, and it is trained using a single CLIP distillation loss (cosine distance between embeddings). At this stage, all image batches come from only $D_{CLIP}$. This stage does not involve early stopping (we train for 20 epochs). The motivation for this step is to have a warm start for the CLIP-head in the next stage where we also allow modifying the backbone, similar to [1].

   In the second stage, the SAM-head and the ViT backbone become trainable, and we have a multi-task objective: CLIP Distillation (Eq. 1) and SAM self-distillation (Eq. 2). The balance between the losses is determined by the coefficient $\lambda$, which we picked to optimize the aforementioned trade-off. Each training step for this stage is performed as follows:

   • Pick a batch of 2048 images from $D_{CLIP}$, run the forward pass, and compute gradients backward from the CLIP Distillation loss (note that only parameters of the CLIP-head and ViT backbone will get gradients after this step).
   • Pick a batch of 32 images from $D_{SAM}$, run the forward pass, and compute gradients backward from the SAM self-distillation loss (note that only parameters of the SAM-head and ViT backbone will get gradients after this step).
   • Apply one optimization step (note that at this point, the parameters of the ViT backbone have accumulated gradients from the above two steps).

   We early-stop after 16 epochs (out of a full training length of 20 epochs) as we observed more severe forgetting (as measured by instance segmentation performance on COCO) after the 16th epoch.

We would like to thank the reviewer for their positive and constructive feedback. Below are our itemized responses. We hope that our responses address all questions and concerns and clarify our contributions.

1. [Clarification on Semantic Segmentation Results] The performance of SAM-CLIP reported in Table 2 is obtained using only the CLIP-head, without any SAM-head refinement -- for a fair comparison, the input images to SAM-CLIP, similar to those in the baseline methods (shown in Table 2), are resized to 448px.

   Figure 3 illustrates that combining the CLIP-head and SAM-head can further enhance performance, although the results in Table 2 do not utilize the SAM-head. The performance of SAM-CLIP with SAM-head enhancement is showcased in Table 5. For example, on the Pascal-VOC dataset, SegCLIP achieves 52.6 mIoU, while our SAM-CLIP, using only the CLIP-head without any refinement, attains 60.6 mIoU (as shown in Table 2). If both CLIP and SAM heads are combined, our SAM-CLIP's performance can further improve to 66.0 mIoU (as shown in Table 5).

2. [Difference from HQ-SAM] SAM-CLIP's training integrates the abilities of CLIP into the SAM model, resulting in a multi-task zero-shot model. In contrast, HQ-SAM focuses on enhancing SAM's original ability and improves its instance segmentation by fine-tuning on high-quality instance segmentation datasets. It is important to note that SAM-CLIP has a completely different goal from HQ-SAM: we aim to make the SAM model multi-task, while HQ-SAM focuses on improving its original task. Consequently, HQ-SAM shows improvement in instance segmentation (e.g., COCO and LVIS), while ours experiences some performance drop, as our goal is not to enhance SAM's instance segmentation ability. We thank the reviewer for bringing the HQ-SAM and MobileSAM papers to our attention. We will include the related discussion in the next revision.

3. [Model Size] We use the SAM ViT-B version as our base model architecture. After the merging process, a single ViT-B image encoder of SAM-CLIP can serve to perform original SAM tasks (prompt-based segmentation), CLIP tasks (prompt-based classification and retrieval), and new tasks of text-to-segmentation (i.e., zero-shot/text-prompted semantic segmentation). Note that all these tasks can be done with a single inference of ViT-B and require loading only one image encoder. We should note that for CLIP-related tasks, SAM-CLIP has an additional light-head (3 transformer layers) which adds 25% to the memory footprint compared to a stand-alone CLIP with ViT-B (12 transformer layers). Thanks for the suggestion. We will clarify this in the next revision.

4. [Dataset Choice and Loss Coefficient] In selecting the dataset, we primarily followed the approach of EVA-CLIP[1], which also distilled a CLIP model, using CC3M, CC12M, and ImageNet-21k. We additionally included YFCC-15M, a subset of the 400M data used to train the original OpenAI’s CLIP model.

   The loss coefficient ratio of 1:10 for the CLIP and SAM heads was empirically determined from three options: 1:1, 1:10, 1:100. We found that 1:10 offers the best trade-off between mitigating the forgetting of SAM’s ability and learning CLIP’s ability. We will clarify this process in the next revision.

5. [Instance Segmentation Without Bounding Box as Prompt] If we understand your question correctly, you are referring to zero-shot semantic segmentation. This task involves providing an image and a set of candidate object class names (e.g., “dog”, “cat”, “human”) and asking the model to perform semantic segmentation. We have provided quantitative results for this task in Table 2. Please let us know if we have misunderstood your point.

References

[1] Fang, Yuxin, et al. "EVA: Exploring the limits of masked visual representation learning at scale." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

We would like to thank the reviewer for their positive and constructive feedback. Below are our itemized responses. We hope that our responses address all questions and concerns and clarify our contributions.

1. [Clarification on Catastrophic Forgetting] We have conducted studies on the catastrophic forgetting of SAM-CLIP versus SAM, as shown in Table 1. The last two columns of Table 1 for COCO and LVIS demonstrate that SAM-CLIP experiences some mild performance degradation compared to SAM in instance segmentation. This indicates that while SAM-CLIP does exhibit some degree of forgetting, it is not catastrophic. To provide a more intuitive understanding of the segmentation quality of SAM versus SAM-CLIP, we present two image examples and their corresponding segmentation predictions by SAM and SAM-CLIP in Appendix A.1 of the newly updated manuscript. Please check out the current rebuttal revision of the paper [PDF].

2. [Model Size] SAM releases three ViT versions of varying sizes: ViT-B(ase), ViT-L(arge), and ViT-H(uge), covering most ViT use cases. In the paper, our experiments were conducted only with ViT-B, but this method can be directly applied to ViT-L and ViT-H as well. The proposed approach can also be applied to subsequent works of SAM, such as MobileSAM [1], to obtain other model sizes for SAM-CLIP.

3. [Performance Gap with the Original SAM] For a quantitative comparison with the original SAM, we have conducted instance segmentation studies. We acknowledge that SAM-CLIP has a small performance gap compared to SAM in terms of instance segmentation ability, as shown in the last two columns of Table 1: (-0.3% and -1.8% mAP on COCO and LVIS datasets, respectively). However, compared to SAM, our SAM-CLIP has broader zero-shot capabilities (Fig 1) and representation quality (Fig 4). Furthermore, in the newly revised manuscript, we provide two image examples in Fig. 5 of Appendix A.1 to qualitatively show that the instance segmentation outputs of SAM and SAM-CLIP have similar quality.

4. [Qualitative Comparison of Segmentation] Thank you for the suggestion on qualitative segmentation comparison. In the newly uploaded revision, we provide segmentation outputs of the original SAM given the same point prompts. One can see from Fig. 6 in Appendix A.1 that SAM tends to segment only a sub-part of the object class pinpointed by the point prompts, instead of segmenting the whole semantic object class. This visual comparison implies that the combination of CLIP and SAM abilities in our SAM-CLIP is quite crucial for (zero-shot) semantic segmentation.

5. [Discussion of More Limitations] We will further discuss additional limitations of our trained model and the proposed method in the next revision, including the above discussion on model size.

References

[1] Zhang, Chaoning, et al. "Faster Segment Anything: Towards Lightweight SAM for Mobile Applications." arXiv preprint arXiv:2306.14289 (2023).

We would like to thank the reviewer for their constructive feedback and comments. Below, please find our itemized responses. We hope that our responses address all questions and concerns and clarify our contributions.

1. [Merging VFMs Other Than CLIP and SAM] Our proposed merging technique, based on an embedding distillation loss and an unlabeled image dataset, can be utilized beyond CLIP and SAM with any image encoder. We will clarify in the next revision how one might use the same approach to merge other vision encoders.

   For instance, one can merge DINOv2[1] with SAM (or SAM-CLIP) in the same pipeline, by substituting the CLIP teacher model with a DINOv2 model. In this paper, we focus on SAM and CLIP because they are both zero-shot models with different capabilities, and merging them provides a vision model with broader zero-shot abilities. In contrast, DINOv2 is a pre-trained image backbone (with strong representation learning ability) that does not operate in a zero-shot manner. However, merging with DINOv2 could enhance representation learning and potentially offer some benefits. For example, one can also merge CLIP and DINOv2 using our pipeline (e.g., using CLIP as the base model and then applying multi-task distillation with CLIP and DINOv2 as teacher models), and possibly obtain a merged model with stronger representation suitable for certain downstream tasks.

2. [Training and Runtime Cost of SAM-CLIP] We believe there is a misunderstanding regarding the training cost of SAM-CLIP. SAM-CLIP is trained on a 41M image dataset, which is 20-30x smaller than the training sets of SAM and CLIP. For instance, SAM is trained on SA-1B, and we use only a 5.7% subset from SA-1B for the SAM self-distillation of our model; the state-of-the-art CLIP model is trained on DataComp-1B (1.4B image-text pairs), and the data we use for CLIP distillation (40M unlabeled images) represents less than 3% of DataComp-1B. Hence, the training cost of SAM-CLIP is quite small compared with the training of SAM and CLIP.

   Regarding deployment, it has been shown that composing SAM and CLIP for semantic segmentation is feasible by using SAM to generate all possible segmentation masks and then using CLIP to provide labels [3]. However, this approach requires loading two models simultaneously (2x memory footprint) and, for each image, needs a forward pass of the SAM backbone (under 1024 resolution) to generate K object segments, followed by a forward pass of the CLIP model for each segment to filter (overall K+1 passes).

   With SAM-CLIP, only one ViT model needs to be loaded (lower memory footprint), and a single forward pass of the ViT backbone is required for each image. Overall, our method offers significant efficiency advantages over the model composition approach in terms of memory and computational costs during inference.

   In the next revision, we will elaborate further on the computational costs of existing training-free approaches for zero-shot semantic segmentation.

3. [Comparison with SAM] We cannot comment on the performance of the SAM model trained with text (using a different pre-training loss than the public SAM) as it is unreleased. The SAM paper [2] refers to it as a “Preliminary Exploration” and does not provide any quantitative results (such as for zero-shot semantic segmentation as we do).

   Regarding capabilities, SAM-CLIP can perform all zero-shot tasks of CLIP (zero-shot classification & image-text retrieval), which SAM (both the public and unreleased versions) cannot. Furthermore, we want to emphasize that our motivation is not solely to improve the SAM or CLIP model but to demonstrate the merging of two foundational models with different zero-shot capabilities into one, with SAM+CLIP serving as a demonstrative case.

References

[1] Oquab, Maxime, et al. "Dinov2: Learning robust visual features without supervision." arXiv preprint arXiv:2304.07193 (2023).

[2] Kirillov, Alexander, et al. "Segment anything." arXiv preprint arXiv:2304.02643 (2023).

[3] "Grounded Segment-Anything." https://github.com/IDEA-Research/Grounded-Segment-Anything (2023)