Unlocking the Power of SAM 2 for Few-Shot Segmentation

Evaluation on LVIS-92$^i$.

Thanks for this precious suggestion, conducting evaluation on LVIS-92$^i$ that includes 920 classes can indeed show the excellent generalizability of our method. We select both Matcher and SINE (A Simple Image Segmentation Framework via In-Context Examples, NeurIPS'24) for comparisons. Since SINE is trained with COCO (80 classes) and directly test on LVIS-92$^i$, we directly take our trained FSSAM on COCO-20$^0$ (trained with 60 classes, under 1-shot setting) to perform evaluation on LVIS-92$^i$. Following Matcher and SINE, each fold comprises 2,300 testing episodes, and the 1-shot results are shown as follows:

We can observe our FSSAM consistently outperform Matcher and SINE in all folds, showing excellent generalizability (COCO-20$^i$ $\to$ LVIS-92$^i$). We will include these comparisons in a newer version.

Is fine-tuning SAM 2 necessary?

Yes, fine-tuning is necessary, and the reasons are as follows:

1. Original SAM 2's memory attention is trained for same object matching, while the matching of FSS is between different query and support FG objects, which we call it incompatible FG-FG matching. As we can observe from the table, with (w/) or without (w/o) fine-tuning (FT) show prominent performance gap.
2. FSSAM can consistently outperform SAM 2 by large margins, since we design modules to resolve the issue. However, FSSAM still requires fine-tuning, because the pseudo mask prompt is inaccurate, e.g., it covers incomplete FG regions and unexpected BG regions, which is different from SAM 2's mask prompt.

Besides, in our response to your previous question, our FSSAM shows excellent generalizability, since our performance on LVIS-92$^i$ (the model is trained on COCO-20$^0$ and directly tested on LVIS-92$^i$) consistently outperform other methods by large margins.

Typo in Line 31.

Thanks for your careful checking! We have corrected this typo.

Can FSSAM use a trainable resnet instead of DINO-v2? What will happen if FSSAM unfreezes the DINO-v2 backbone and fine-tunes it?

Kindly remind DINOv2 is only responsible for generating pseudo mask prompt without learning, so it can be replaced as pretrained ResNet50, and there is no need to make it trainable. When we replace DINOv2 as ResNet50, the 1-shot mIoU on PASCAL-5$^i$ is 74.1, which is still much better than other methods (in Table 1). kindly note that FounFSS also uses DINOv2, and our performance is 4.2% better.

For the second question, we can fine-tune DINOv2, but we need to design an extra learning objective to encourage better pseudo mask prompt, which may raise inductive bias issue (i.e., degrading the generalizability), since the setting of FSS is training on some base classes while testing on unseen classes. Besides, the training cost would be much heavier, since DINOv2-B has 86M parameters.

Evaluation on LVIS-92$^i$.

Conducting evaluation on LVIS-92$^i$ (920 classes) can show the excellent generalizability of FSSAM. We select both Matcher and SINE for comparisons. Since SINE is trained with COCO (80 classes) and directly test on LVIS-92$^i$, we directly take our trained 1-shot FSSAM on COCO-20$^0$ (60 classes) for fair evaluation. Following baselines, each fold comprises 2,300 testing episodes, and the 1-shot results are as follows:

We can observe our FSSAM consistently outperform Matcher and SINE in all folds, showing excellent generalizability (COCO-20$^i$ $\to$ LVIS-92$^i$). We will include these results in a newer version.

(Major concern) Compare with SINE [2] on PASCAL-5$^i$ and COCO-20$^i$.

SINE CANNOT be fairly compared on these datasets:

1. FSS models are trained on some base classes, then tested on unseen classes, i.e., the testing samples are out-domain ones.
2. In Section 4.2 of SINE, authors mention "SINE is trained with all data of COCO". Specifically, COCO and PASCAL have 80 and 20 classes, and COCO's classes include PASCAL's classes. Since SINE is trained on whole COCO, it has learned to deal with all classes of PASCAL and COCO during training, i.e., the testing samples are in-domain ones. Since SINE has been trained with the test classes (in each fold of FSS), its scores will naturally be higher than FSS methods. Unless we remove COCO from SINE's training set, SINE can never be fairly compared on PASCAL-5$^i$ and COCO-20$^i$.
3. That being said, fair comparisons can be conducted on LVIS-92$^i$, and the results are included in previous question, where our method surpasses SINE by 6.1% mIoU, under the consistent out-domain setting of "training on COCO and directly testing on LVIS-92$^i$".

Failure cases and limitations.

There is one failure case/limitation. Kindly remind our method relies on pseudo mask prompt to resolve the incompatible FG-FG matching issue. In some very difficult examples (all baselines cannot deal with such cases), e.g., the FG and BG are very similar and are quite difficult to distinguish, the generated pseudo mask prompt may be misleading, e.g., the real FG is completely uncovered. This motivates us to further design an error correction module, and we leave it as a future direction. We will include some failure cases in a newer version.

Insufficient visualization.

Thanks for this suggestion! We will include some baselines' results for comparisons in a newer version.

Can iteration number (in IMR) be adaptive?

The iteration number can theoretically be adaptive, yet there exist some challenges, and we uniformly fix it as 3 (see Table 4). Let's recall (1) the impact of IMR, and (2) when to use more iterations.

For (1), IMR will make the incomplete FG regions in pseudo mask prompt complete, but at risk of introducing unexpected BGs (noises). Generally, using more iterations can make FG regions more complete, but introduce more noises, so trade-offs should be made to determine the iteration number.

For (2), we dive into the test samples, and find that difficult samples, containing either multiple FG objects or complex BG, require larger iteration number. The initial pseudo mask prompt of these difficult samples can only cover limited FG regions, so more iterations are required to make FG regions complete (to be a better prompt).

Hence, whether the iteration number can be adaptive depends on if such difficult samples can be automatically identified or not. Unfortunately, determining the difficulty of each case is not trivial, i.e., we cannot either measure the number of FG objects in query images or judge whether query FG and BG are easy to distinguish or not, and we leave it as a future direction.

Can we use point prompt?

Yes, but it's not a good choice compared to pseudo mask prompt, since:

1. Although we can find some points (in query) with largest similarities to support FG features as point prompts, the overall framework is not easy to optimize, i.e., it's hard to determine if a point is the best candidate or not.
2. 1 point prompt can only correspond to 1 entity. When there are multiple FG objects in a query image, e.g., 20 people, it's very difficult to automatically find 1 point for each people. If any people is not assigned with point prompt, he will be classified as BG.
3. Instead, pseudo mask prompt can be regarded as a special case of point prompt, which can (1) make optimization easier, and (2) include sufficient points (to cover each FG object).

Larger number of parameters than other methods and result in additional computational cost.

Thanks for this comment. Our parameter number is actually much smaller than most of foundation-based FSS methods. We select some methods and summarize their parameter number, learnable parameter number, as well as the 1-shot mIoU scores on PASCAL-5$^i$ as follows:

The first 3 rows correspond to classical FSS methods that use ResNet50 as the pretrained backbone, and the remaining methods refer to foundation-based FSS methods that use DINOv2 and/or SAM. For our finalized model, we use DINOv2-B (86M) and SAM 2-S (46M) without extra parameters (kindly remind our proposed modules are parameter-free), and fine-tune part of SAM 2's parameters.

It can be observed from the table:

1. Among foundation-based FSS methods, our parameter number is much smaller than most of them, while our performance is consistently much better.
2. Compared to classical FSS methods, though we use more parameters, the difference is not as large as expected, while the performance gap is quite prominent, so we believe the additional cost is worthy.

For computational complexity, the designed modules will introduce additional linear complexity to the original foundation model, which have already been described in "Memory Complexity" of Section 4.2 and 4.3.

Therefore, the computational burden of our FSSAM is reasonable and acceptable.

Substantive discussion about the most important finding. Is the main finding that SAM 2 can be used for FSS, or can a memory-based video segmentation model be used for FSS through the proposed module?

Our main finding is memory-based video segmentation model can be used for FSS through the proposed modules, and we focus on one of the most representative and powerful models, i.e., SAM 2.

In Table 1, baselines Matcher, VRP-SAM and FG-SAM uniformly deploy SAM-L for FSS, where SAM-L (641M) is much larger than the one we deployed, i.e., SAM 2-S (46M).

As shown in the first row of component-wise ablation study in Table 3, naive SAM 2-S cannot outperform any of these baselines, and we attribute it to the facts that (1) our SAM 2-S (46M) is much smaller than their SAM-L (641M), and (2) there exist an incompatible matching issue (as introduced in Section 1).

After using our designed modules to resolve this issue, our FSSAM can use much fewer parameters to outperform these foundation-based FSS methods by large margins, showing the effectiveness of our design, serving as our main finding.

Larger backbone cannot guarantee consistent improvement.

We would like to make the following notes to Table 7:

1. We study 3 versions of SAM 2, including S (46M), B (80.8M) and L (224.4M). According to the official Github of SAM 2, SAM 2-B (80.8M) originally CANNOT outperform SAM 2-S (46M) in 2 out of 3 datasets, so it's reasonable that adapting SAM 2-B (80.8M) for FSS cannot show comparable performance as SAM 2-S (46M). When we remove SAM 2-B (80.8M), the improvement is stable, e.g., the mIoU is increased from 81.0 (SAM 2-S, DINOv2-B) to 81.5 (SAM 2-L, DINOv2-B).
2. DINOv2 is only responsible for generating cosine similarity-based pseudo mask prompt, whose features will not be directly used in other modules, thus upgrading DINOv2 from from B (86M) to L (300M) cannot guarantee better performance, i.e., the pseudo mask prompt generated by DINOv2-B is good enough.

In summary, using larger backbone (SAM 2) can guarantee improvement.

Computational burden and learnable parameters.

Thanks for your valuable suggestion! We agree it would be better to include the parameter number for further comparisons. We select some methods and summarize their parameter number, learnable parameter number, as well as the 1-shot mIoU scores on PASCAL-5$^i$ as follows:

The first 3 rows correspond to classical FSS methods that use ResNet50 as the pretrained backbone, and the remaining methods refer to foundation-based FSS methods that use DINOv2 and/or SAM. For our finalized model, we use DINOv2-B (86M) and SAM 2-S (46M) without extra parameters (kindly remind our proposed modules are parameter-free), and fine-tune part of SAM 2's parameters. It can be observed: (1) Among foundation-based FSS methods, our parameter number is much smaller than most of them, while our performance is consistently much better; (2) Compared to classical FSS methods, though we use more parameters, the difference is not as large as expected, while the performance gap is quite prominent, so we believe the additional cost is worthy.

For computational complexity, the designed modules will introduce additional linear complexity to the original foundation model, which have already been described in "Memory Complexity" of Section 4.2 and 4.3.

Therefore, the computational burden of our FSSAM is reasonable and acceptable.

Comparison with versions of these methods that incorporate learnable parameters.

This comment locates in "Methods And Evaluation Criteria", sorry but we are confused about "different versions", do you mean using SAM 2 and DINOv2 with different sizes or something else?

If our understanding is correct:

1. FSSAM is built upon SAM 2 and DINOv2 without additional parameters, the designed modules are parameter-free, so the variants in Table 3 have same (learnable) parameters.
2. All parameters of DINOv2 are frozen. For SAM 2, we fine-tune its memory encoder, memory attention and mask decoder.
3. Different SAM 2 mainly differ in image encoders, and the learnable parameters of FSSAM are uniformly 11M, regardless of which size is used.
4. We have studied the impacts of different sizes in Table 7. For your convenience, we summarize some statistics as follows.

After making trade-offs between computational burden and performance, we use SAM 2-S (46M) and DINOv2-B (86M). For reasons why using larger SAM 2 and DINOv2 cannot guarantee performance gain, please refer to our reponses to Larger backbone cannot guarantee consistent improvement of Reviewer pnf2.

Same symbol $A_{QQ}$ in different modules.

Thanks for this comment. They uniformly refer to the mutual similarities between two features, but we do agree it would be much better to differentiate them in different modules, e.g., additionally include module names as superscripts.

Inconsistency between Figure 2 and detailed description in Section 4.

Thanks for pointing out this issue, we will modify them accordingly.

Confusing relationship between IMR in Figure 2 and Figure 3.

We guess that the confusion comes from the inconsistent outputs of IMR modules in Figure 2 and 3, we will unify them in a newer version.

In "Parameter study on IMR", performance decreases when the iteration n is 4, yet it increases in fold 5$^3$.

The test classes of fold 5$^3$ comprise of potted plant, sheep, sofa, train, and tv/monitor, and we find that the samples of this fold are more challenging than those in other 3 folds (e.g., the performance of fold 5$^3$ is consistently the worst among all folds for all methods in Table 1). Specifically, many samples from this fold include: (1) multiple tiny objects, and (2) complex background (BG), making it hard to distinguish FG and BG. Therefore, the initially generated prior masks in fold 5$^3$ CANNOT cover sufficient FG regions, and requires larger n in IMR to complete FG regions.

Kindly remind with the increase of n, more unexpected BG regions will also be activated, acting as some noises to prevent from boosting the performance. Therefore, n should not be too large, otherwise, the performance (e.g., that of fold 5$^0$, 5$^1$ and 5$^2$) will decrease.

In Table 4, we conduct experiments to verify the impacts different iteration number, and we uniformly set it as 3 for all folds currently. We think a better way is adaptively determining the iteration number n for each sample, and we will leave it as a future direction.

Discrepancy between the data in Table 1 and 4.

Thanks for your careful checking! We will correct the value in Table 1.