Personalize Segment Anything Model with One Shot

We sincerely appreciate your valuable reviews and recognition of our work. We hope our response can address your concerns.

---

Q1: PerSAM and PerSAM-F may be constrained by SAM's limited semantics by class-agnostic training

1. Indeed, due to SAM's inherent class-agnostic training, the visual features extracted by SAM's encoder contain limited category-level semantics. This might constrain the category-level discriminative capability for complex multi-object scenes. Observing this limitation, we locate the target object among other objects in test images entirely by feature matching, i.e., the location confidence map. Such a matching strategy only considers the appearance-based class-agnostic similarity, without category semantics.

2. As you suggested, we can leverage other semantically rich image encoders, e.g., CLIP and DINOv2, for PerSAM(-F) to improve the multi-object performance. We conduct an ablation study of different image encoders on DAVIS 2017 dataset for video object segmentation, which contains multiple similar objects within a video. As shown in the table below, applying CLIP and DINOv2 with more sufficient semantic knowledge can improve the results of PerSAM-F for more challenging multi-object segmentation. Encoder|J&F|J|F -|:-:|:-:|:-: SAM|73.4|70.9|75.9 CLIP|74.9|72.8|77.0 DINOv2|76.1|73.2|78.9

---

Q2: Can PerSAM be extended to few-shot segmentation for better performance?

Thanks for your advice! Yes, our approach is not limited to one-shot segmentation, and can accept few-shot references for improved results. As an example, given 3-shot references, we independently calculate 3 location confidence maps for the test image, and adopt a pixel-wise max pooling to obtain the overall location estimation. For PerSAM-F, we regard all 3-shot data as the training set to conduct the scale-aware fine-tuning.

We respectively conduct experiments for 3-shot segmentation on PerSeg dataset and 5-shot segmentation on FSS-1000 dataset. The results are shown in the following two tables. By providing more visual semantics in few-shot data, both our training-free PerSAM and the fine-tuned PerSAM-F can be further enhanced.

---

Q3: Is the fine-tuned PerSAM-F generalized only to a specific object?

Our PerSAM-F can not only be personalized by a specific object, but also generalize to a certain category with the same amount of parameters. As visualized in Figure 13 of Appendix, given a reference cone/armour/crocodile in FSS-1000 dataset, our PerSAM-F can well segment other similar cones/armours/crocodiles in test images. This is because objects of the same category can contain similar hierarchical structures, so the learned scale weights of PerSAM-F by one sample can also be applicable to different objects within the same category. In contrast, for different categories, one needs to fine-tune two sets of scale weights to respectively fit their scale information.

---

Q4: What about using different fine-tuning methods, e.g., LoRA, for multi-object generalization?

Thanks for your advice! As shown in Table 3 of the paper, we have conducted an ablation study using different fine-tuning methods for single-object segmentation on PerSeg. In the table below, we further investigate different fine-tuning methods for multiple objects on DAVIS 2017 dataset. We respectively tune different parameters for different objects within a video. As shown, our proposed scale-aware fine-tuning performs the best with only 2 parameters.

---

Q5: Can PerSAM help DreamBooth achieve better multi-object customization?

Yes. Similar to single-object personalization, we only calculate the loss within foreground regions for DreamBooth with multi-object training samples. As visualized in Figure 20 of Appendix, we show the improvement for two-object customization assisted by our PerSAM. The backgrounds within images generated by DreamBooth are severely disturbed by those within few-shot training images, while the PerSAM-assisted DreamBooth can accurately synthesize new backgrounds according to the input language prompts.

We sincerely appreciate your careful and constructive reviews. We hope our response can address your concerns.

---

Q1: What are the results in Tables 2 and 3 by using the default image encoder features of SAM?

Thanks for your advice! The detailed results of the two image encoders in Tables 2 and 3 are respectively as follows.

As DINOv2 is particularly pre-trained by large-scale contrastive data, it produces more discriminative image features than SAM's encoder. This contributes to a more precise positive-negative location prior for better segmentation results, especially on the challenging FSS-1000 dataset. Despite this, with SAM's original encoder, our PerSAM-F and the training-free PerSAM still obtain better segmentation accuracy than Painter or SEEM on different datasets, demonstrating the effectiveness of our approach.

---

Q2: What's the running speed/memory compared to SAM?

We test the running efficiency on a single NVIDIA A100 GPU with batch size 1. As shown in the table below, our PerSAM and PerSAM-F bring marginal latency and GPU memory consumption over SAM.

---

Q3: How's the performance comparison to SAM-PT?

Thanks for pointing out! We will add the discussion and comparison in the revised version.

1. Method Comparison. Although both our PerSAM(-F) and SAM-PT are developed based on SAM, our approach can be generalized to most one-shot segmentation tasks (personalized/video/semantic/part segmentation), while SAM-PT specifically aims at video object segmentation. One key difference between our approach and SAM-PT is how to locate and associate objects from the previous to the current frame, i.e., propagating the location prompt for SAM across frames. In detail, our PerSAM(-F) simply calculates a location confidence map by feature matching, while SAM-PT relies on an external point tracking network, PIPS [1].

2. Performance Comparison. As shown in the table below, on DAVIS 2017 dataset, SAM-PT performs slightly better than the original PerSAM-F. However, inspired by SAM-PT, we can also incorporate its point tracking strategy (the PIPS tracker) with PerSAM(-F) to propagate the positive-negative point prompt, which effectively enhances the segmentation performance. This demonstrates the flexible extensibility of our approach for applying more advanced trackers in a plug-and-play way. Method|Propagation|J&F -|-|:-: SAM-PT|Point Tracking|76.6 PerSAM|Feature Matching|66.9 PerSAM|+Point Tracking|68.2 PerSAM-F|Feature Matching|76.1 PerSAM-F|+Point Tracking|77.2

Reference:

[1] Particle Video Revisited: Tracking through Occlusions using Point Trajectories. ECCV 2022.

Q4: How does PerSAM differentiate objects with similar appearances in video object segmentation?

For video object segmentation, our approach tries to accurately locate the target object among similar ones by the following three aspects.

1. Discriminative features from the encoder. Due to large-scale pre-training, the SAM's image encoder, or the more powerful DINOv2, can already produce discriminative enough visual features for different similar objects, which is fundamental to the subsequent calculation of location confidence map.

2. Comprehensive location confidence map. We calculate a set of confidence maps for all foreground pixels within the target object, such as the head, the body, or the paws of a dog, and then aggregate them to obtain an overall location estimation. This strategy can comprehensively consider the slight differences in any local parts between similar objects.

3. Temporal cues between adjacent frames. To better leverage the temporal consistency along the video, we prompt SAM's decoder additionally with the object bounding box from the last frame. As different objects have different trajectories, such temporal constraints can better differentiate similar objects by spatial locations.

As visualized in Figure 12 of Appendix, our method can precisely segment the dancing man in front of a crowd (the 2$^{nd}$ row) and differentiate different fishes within a group (the last row).

---

Q5: Can PerSAM also work on self-driving scenarios with many similar vehicles/cars?

Yes. In most cases, our model can segment the designated cars with distinctive appearances in dense traffic.

1. As visualized in Figure 15 of Appendix, for the user-provided target (e.g., a red car, a truck, and a bus), our PerSAM-F can well locate and segment them under severe occlusion or surrounded by similar cars.

2. We conduct an experiment for one-shot segmentation on Tokyo Multi-Spectral-4$^i$ [2] dataset, which contains 16 classes within outdoor self-driving scenarios. As shown in the table below, our PerSAM-F can surpass existing methods that require in-domain training, indicating our generalization capacity in city traffic scenes. Method|In-domain Train|mIoU -|:-:|:-: PFENet|$\checkmark$|14.0 PGNet|$\checkmark$|17.5 V-TFSS|$\checkmark$|26.1 PerSAM|-|18.4 PerSAM-F|-|25.6

---

Q6: More failure cases visualization of PerSAM

Thanks for your advice! There are mainly two kinds of failure cases of PerSAM(-F).

1. As shown in Figures 2 and 8 of the paper, the training-free PerSAM might fail to segment objects with hierarchical structures, e.g., the hat on top of a teddy bear, or the head of a robot toy. Such scale ambiguity can be effectively alleviated by the fine-tuning of PerSAM-F.

2. After solving the scale ambiguity issue, the three types of failure cases of PerSAM-F are shown in Figure 16 of Appendix: (a) different people with the same clothes, indicating our approach is not very sensitive to fine-grained human faces; (b) the key appearance of the target object is occluded by in test images (the red chest of the bird), indicating that we still need to improve our robustness when there is too large appearance change in test images; (c) discontinuous objects that SAM cannot tackle, for which we can replace SAM with stronger segmentation foundation model for assistance.

Reference:

[2] Visible and Thermal Images Fusion Architecture for Few-shot Semantic Segmentation. JVCIR 2021.

We sincerely appreciate your detailed and insightful reviews. We hope our response can address your concerns.

---

Q1: Experiments for one-shot segmentation on additional (more challenging) datasets

Thanks for your advice! It is very necessary to evaluate PerSAM in more challenging scenarios.

In our paper, we have experimented on six datasets of different tasks, i.e., PerSeg (personalized object segmentation), DAVIS 2017 (video object segmentation), FSS-1000 and LVIS-92$^i$ (semantic segmentation), PASCAL-Part and PACO-Part (part segmentation). On different benchmarks with various domains, our PerSAM(-F) without in-domain training can attain competitive performance to existing specialist models.

As you suggested, we further present the performance of our approach on two additional datasets for one-shot segmentation: COCO-20$^i$ [1] and Tokyo Multi-Spectral-4$^i$ [2]. (Note that, ADE20K and CityScapes have no corresponding few-shot benchmarks.)

1. COCO-20$^i$ is constructed from MS-COCO by dividing the diverse 80 classes evenly into 4 folds. We directly test our method on the validation set without specific in-domain training. As shown in the below table, our PerSAM(-F) achieves favorable segmentation performance over a wide range of object categories, comparable to previous in-domain methods. Method|In-domain Train|mIoU -|:-:|:-: FPTrans|$\checkmark$|47.0 SCCAN|$\checkmark$|48.2 HDMNet|$\checkmark$|50.0 PerSAM|-|47.9 PerSAM-F|-|50.6

2. Tokyo Multi-Spectral-4$^i$ is sampled from Tokyo Multi-Spectral containing 16 classes within outdoor city scenes, similar to CityScapes. Different from existing methods, we only take as input the RGB images without the paired thermal data, and do not conduct in-domain training. As shown in the table below, our approach still exhibits good generalization capacity in street scenarios. Method|In-domain Train|mIoU -|:-:|:-: PFENet|$\checkmark$|14.0 PGNet|$\checkmark$|17.5 V-TFSS|$\checkmark$|26.1 PerSAM|-|18.4 PerSAM-F|-|25.6

---

Q2: Comparison with zero-shot or text-guided segmentation methods, like DenseCLIP

Thanks for your advice! It would be beneficial to discuss the relationship between PerSAM and existing zero-shot text-guided segmentation models. However, please note that DenseCLIP cannot perform zero-shot text-guided segmentation for comparison with our method. We first explain the reason, and then show the comparison with other zero-shot text-guided methods.

1. Why DenseCLIP is not comparable? This is because DenseCLIP is a conventional closed-set method, which cannot conduct zero-shot or text-guided segmentation. Specifically, DenseCLIP only utilizes CLIP's text embeddings as axillary semantics for image features, and still requires fully supervised training for segmentation on limited "seen" classes. Therefore, it cannot be directly utilized for text-guided zero-shot segmentation on "unseen" classes, or one-shot segmentation given reference data.

2. Comparing with OVSeg [3] and Grounded-SAM [4]. We select two popular text-guided open-set segmentation methods for comparison with our PerSAM(-F) on three benchmarks: PerSeg and COCO-20$^i$. OVSeg leverages MaskFormer to first generate class-agnostic mask proposals, and then adopts a fine-tuned CLIP for zero-shot classification. Grounded-SAM utilizes a powerful text-guided detector, Grounding DINO [5], to generate object bounding boxes, and then utilize them to prompt SAM for segmentation. Instead of giving a one-shot reference, we directly prompt them by the category name of the target object for text-guided segmentation, e.g., "cat", "dog", or "chair". As shown in the table below, our PerSAM-F consistently achieves competitive results in the two scenarios. This indicates that utilizing PerSAM with a class-agnostic one-shot reference is on par with recognizing the category and then segmenting it with text-guided methods. Method|Prompt|PerSeg|COCO-20$^i$ -|:-:|:-:|:-: OVSeg|Category Name|76.5|37.8 Grounded-SAM|Category Name|93.2|49.9 PerSAM|One-shot Data|89.3|47.9 PerSAM-F|One-shot Data|95.3|50.6

Reference:

[1] Feature Weighting and Boosting for Few-shot Segmentation. ICCV 2019.

[2] Visible and Thermal Images Fusion Architecture for Few-shot Semantic Segmentation. JVCIR 2021.

[3] Open-Vocabulary Semantic Segmentation With Mask-Adapted CLIP. CVPR 2023.

[4] https://github.com/IDEA-Research/Grounded-Segment-Anything.

[5] Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. arXiv 2023.