SANSA: Unleashing the Hidden Semantics in SAM2 for Few-Shot Segmentation

We sincerely thank the Reviewer for the positive and constructive feedback. We are encouraged to see appreciation for the clarity of the paper, the motivation behind adapting SAM2 to the few-shot segmentation setting, the flexibility to support diverse prompts, and the strong results achieved in both performance and efficiency. Below, we address the remaining concerns in detail, providing clarifications and additional analyses. We hope these additions further strengthen the paper and improve its presentation.

W1: Unclear whether qualitative results shown in the bottom of Figure 1 are SAM2 zero-shot outputs or ground-truth segmentations.

We appreciate the Reviewer observation and clarify that the bottom part of Figure 1 shows ground-truth masks. These examples are meant to visually contextualize the two types of scenarios discussed in our toy experiment: low semantic shift / high visual similarity (e.g., medical scans, left) and high semantic shift / low visual similarity (e.g., cruise ship vs. rowboat, right). The red masks simply indicate the object of interest. We will revise the caption accordingly.

More broadly, the right part of the figure motivates our core research question: Can the semantic capabilities observed in other vision foundation models (VFMs) also be found in SAM2? A growing line of research [21, 37, 67, 69, 70] has demonstrated that semantic structure is encoded in the features of frozen VFMs such as DINOv2 and Stable Diffusion. We conducted this toy experiment to test whether similar semantic properties could be observed in the frozen SAM2. The aim of Figure 1 is thus to set the stage for the rest of the paper: to highlight that, unlike in other VFMs, semantic structure in SAM2 does not naturally emerge in zero-shot.

W2: In Figure 2 (PCA), it is not explained whether the colors carry semantic meaning.

We thank the Reviewer for the helpful observation and agree that clarifying the caption would improve interpretability. In Figure 2 we apply Principal Component Analysis (PCA) to the extracted feature vectors and map the first three principal components (PC1, PC2, PC3) to the red, green, and blue channels, respectively. Consequently, each point RGB color encodes its position in the 3D PCA-projected space. This approach is widely used to analyze the global structure of high-dimensional embeddings [43, 54, 70]. Importantly, the colors do not correspond to ground-truth class labels and should not be interpreted categorically. Rather, they reflect spatial proximity in the projected space: points with similar colors lie close in the 3D PCA space. This allows us to qualitatively assess what types of variation are captured by the leading axes. In the case of frozen SAM2, although some hue variation exists (e.g., surfboards appear slightly violet and people greenish-blue), the overall color distribution lacks clear separation across the four object categories (dog, frisbee, person, surfboard). This suggests that the principal components capture a mixture of low-level or instance-specific signals, rather than pure semantics. In contrast, SANSA yields clearly distinct and consistent colors (e.g., orange, yellow, violet, light blue), indicating that semantic structure dominates the main axes of variability.

A more comprehensive evaluation is reported in Figure 4a, where we use a 2D PCA projection with ground-truth color labels to visualize class-level clusters.

W3: Details regarding the training: specific values for hyperparameters such as r and J. Do these values depends on whether the model is trained under the 1-shot or 5-shot setting?

We thank the reviewer for the comment and are happy to clarify. We use a training sequence length of J=3, meaning each episode consists of one annotated reference followed by three unlabeled target image. The number of annotated references during training is fixed to r=1. At training, these values do not change. At inference time, we simply provide either one (r=1) or five (r=5) annotated examples based on the benchmark protocol (as in Table 1 of main paper). This unified training strategy enables seamless support for variable r-shot inference (e.g., 1-shot or 5-shot), without retraining or architectural modifications.

Details on the training configuration, including the fixed value J=3, are included in Section §G.1 Implementation Details of the Supplementary Material. We appreciate the suggestion and will revise the text to also clearly state that r=1 is used during training.

W5: If the initial prediction is inaccurate, is there a risk of cascading errors or convergence to suboptimal solutions (e.g., local minima)?

We appreciate the reviewer thoughtful question. The sequential conditioning mechanism is used exclusively during training to promote robust and generalizable representations. By requiring the model to propagate semantic information across multiple unlabeled targets, this objective discourages overfitting to individual image pairs and encourages the emergence of shared, semantic structure, forcing the network to disentangle high-level concepts from low-level visual cues.

To directly address the concern about potential cascading errors or instability from inaccurate early predictions, we perform an ablation on the training sequence length J, which defines how many unlabeled target frames are segmented after the annotated reference. When J=1, no sequential conditioning or mask propagation occurs. If our formulation were prone to cascading errors, we would expect performance to degrade for larger J. Instead, we observe that all settings with J>1 consistently outperform the baseline with J=1, demonstrating the stability and benefits of our training strategy.

Table R5: Ablation on Mask Propagation, † Newly added for rebuttal.

These results provide strong empirical evidence that our sequential training formulation is not vulnerable to cascading errors, but instead enhances the quality and generality of the learned features.

Lastly, we clarify that this sequential conditioning mechanism is used only during training. At inference time, our method follows the standard few-shot segmentation protocol: each target image is segmented independently, using only the annotated reference(s), without mask propagation or reliance on previous predictions. We thank the reviewer for this insightful observation and will incorporate the ablation and its analysis in the revised manuscript.

W4: The construction of the training image set is only briefly mentioned.

We appreciate the Reviewer suggestion to clarify the training data construction process. We followed the standard protocol adopted in few-shot segmentation literature [16, 17, 20, 39, 40]. For each training episode, a class is randomly selected from the training set, and images containing at least one instance of that class are sampled uniformly at random. No filtering is applied based on object size, instance count, or other properties. The order of images within episodes is shuffled during training. Since no special constraints are introduced, we did not emphasize this aspect in the main paper. However, we agree that adding this clarification can improve completeness, and will revise the manuscript accordingly.

We once again thank the Reviewer for the time dedicated to evaluating our work and for the constructive questions and observations raised. We hope our responses have clarified all points and helped improve the overall quality of the paper.

We thank the Reviewer for the constructive feedback. We are glad to see that the strong performance, efficiency gains, and our in-depth analysis, including PCA decomposition, linear probing, and feature visualizations, were appreciated. In the following, we address the concerns raised with new experiments, expanded theoretical insights, and a clearer articulation of our contributions. We hope these additions clarify the novelty and significance of our work.

W1: The core contribution is essentially applying AdaptFormer to SAM2: the authors should more clearly claim their main contributions.

We thank the Reviewer for the opportunity to clarify the core contributions of our work. The novelty lies in a key insight: SAM2, though trained for class-agnostic object tracking, encodes a latent semantic structure (L51-L56) that can be disentangled and repurposed for few-shot segmentation (FSS).

• From a scientific standpoint, we are the first to identify, analyze, and validate the presence of semantic structure in SAM2 features. We show in Supp. §C.2 that a simple linear probe on frozen features recovers class-level information, providing evidence that semantics are already embedded and do not need to be learned. Based on this, we propose a new perspective on few-shot segmentation: instead of relying on separate models (DINOv2 for matching and SAM for segmentation), we show that SAM2 alone can perform both, enabling a unified and streamlined solution. More broadly, this repositions SAM2 as a standalone foundation for a wider class of pixel-level semantic tasks.
• From a technical standpoint, we i) repurpose SAM2 memory mechanism by feeding it pseudo-videos, reinterpreting the temporal dimension as a collection of semantically related images. We ii) introduce a novel training protocol that promotes concept propagation across multiple unlabeled frames, encouraging the emergence of robust semantic correspondence. We iii) adapt the feature space to shift from low-level visual similarity to class-level alignment. Notably, the architectural simplicity of our design is a consequence of our hypothesis: if semantic structure had to be introduced through architectural modifications, it would undermine our claim that such structure already exists in SAM2.
• From a practical standpoint, SANSA achieves SOTA results across FSS benchmarks, in-context learning scenarios, and part segmentation tasks. It does so while being 3× faster and up to 5× smaller than prior methods. Crucially, our model supports low-effort prompts (e.g., points), making it well-suited for real-world applications such as data annotation.

We will revise the manuscript to better frame and position our contributions.

Q2: Why does AdaptFormer work explicitly better than other adaptation methods? Q4: Is the semantic disentanglement specific to your approach rather or a general property of any adaptation of SAM2 features?

Building on the previous point, our results are not meant to imply that semantic disentanglement arises specifically because of AdaptFormer. We emphasize that our goal is not to engineer a dedicated adapter (L56-L60), but to investigate the representational structure in SAM2: if semantic disentanglement depended on the design of the adapter, it would suggest that the semantics are induced by adaptation, rather than being already embedded in the pretrained model.

In this context, we detail what makes an adaptation strategy effective, expanding on results reported in the paper with additional experiments in Tab. R3.

• As shown in Tab. 4 (main paper), basic adaptation strategies (i.e. LoRA, Adapter, AdaptFormer) yield similarly strong improvements of +25% w.r.t. frozen SAM2 (32.2%).
• Importantly, LoRA is a linear adapter. The slight gains (~2%) observed with Adapter and AdaptFormer, both of which introduce a single ReLU non-linearity, suggest that simple non-linear transformations can help refine this structure, but do not fundamentally reshape it.
• In Tab. 5, we study the effect of adapter capacity. Reducing the bottleneck dimension from 768 → 576 → 384 leads to progressively better generalization (58.2% → 59.6% → 60.1%). This indicates that smaller adapters are better at capturing the relevant structure in the pretrained features, avoiding overfitting to spurious correlations.
• To further test this hypothesis, we introduce a new experiment with a recent SOTA adapter, MONA [A], which include depthwise convolutions and LayerNorm to increase functional complexity. Despite this, MONA underperforms (56.9%), suggesting that added complexity leads to overfitting rather than better representations.

In summary, semantic disentanglement is not unique to AdaptFormer, but also does not arise from arbitrary adaptation. Our results suggest that effective adaptation requires simplicity and constraint: low-capacity, bottlenecked modules are best suited to uncovering SAM2 latent semantics. In contrast, more expressive adapters tend to overfit.

Table R3 Finetune vs Adaptation. † Newly added for rebuttal

[A] "5%> 100%: Breaking performance shackles of full fine-tuning on visual recognition tasks." CVPR2025.

Q3: How does SANSA differ from standard fine-tuning approaches beyond the efficiency perspective? W3: Theoretical insight into why adaptation strategy works or how it differs from other fine-tuning?

Our choice to keep SAM2 frozen is not motivated by efficiency, but by the need for generalization, which is the core challenge in few-shot segmentation. All recent state-of-the-art methods [35, 67, 72, 37] do not finetune pretrained Vision Foundation Models to avoid overfitting when learning from limited data. SANSA follows this paradigm. As shown in the Tab. 1 of the Supp. (Sec. §C.3) and in Tab. R3 above, full or partial fine-tuning of SAM2 consistently results in worse generalization. We further added two new experiments (training from scratch and full fine-tuning), which confirm the same trend.

From a representation learning perspective, SAM2 encodes rich, reusable features that capture both appearance and latent semantics. Lightweight bottleneck adapters such as AdaptFormer or LoRA apply low-rank transformations that reorient the embedding space toward task-relevant semantics while preserving the underlying pretrained structure. This can be viewed as a form of semantic disentanglement under a strong inductive prior, i.e. the pretrained model [B, C]. In contrast, directly fine-tuning the model parameters risks erasing these priors, particularly in few-shot or continual learning settings, where data is limited and distributions shift at test time [C, D]. Our empirical findings (Tab. R3: Finetune vs Adaptation) align with this interpretation: adaptation successfully leverages and restructure existing structure, thus promoting generalization.

[B] "Fine-tuning can cripple your foundation model; preserving features may be the solution." TMLR2024
[C] "Fine-tuning can distort pretrained features and underperform out-of-distribution." ICLR2022
[D] "Ranpac: Random projections and pre-trained models for continual learning." NeurIPS2023

Q1, W3: The paper lacks ablations on adapter placement (to justify the "last two layers" choice), architectures, or hyperparameters.

We thank the reviewer for raising this important point. While prior works typically insert adapters throughout the entire network to maximize representational capacity [E,F,G,H], we purposefully limit adaptation to the last 2 hierarchical blocks of the backbone, which are known to encode high-level semantics. This design choice stems from our core objective: disentangle semantic structure. In response to the reviewer suggestion, we conducted a systematic ablation on adapter placement (Tab. R4) to better justify our design choice.

Table R4. Adapter Placement

The ablation confirms that, for semantic disentanglement, it is beneficial to adapt the latest stages of the backbone.

These results complement our ablations on the core design elements, including Adapter architecture (Tab. 4, main paper; expanded in Tab. R3), Adapter capacity (Tab. 5, main paper), Adaptation vs. fine-tuning (Tab. 1, Supp; expanded in Tab. R3). We hope the additional experiment fully addresses the reviewer concern.

[E] "Side Adapter Network for Open-Vocabulary Semantic Segmentation." CVPR2023
[F] "Exploring Temporally-Aware Features for Point Tracking." CVPR2025
[G] "AdapterFusion: Non-Destructive Task Composition for Transfer Learning." EACL2021
[H] "Vision Transformer Adapter for Dense Predictions." ICLR2023

W2: Dense prose; challenging for readers less familiar with SAM2 or FSS.

We sincerely thank the reviewer for this helpful observation. We recognize that clarity is essential, particularly when introducing concepts that may be unfamiliar to all readers. We will revise the manuscript in Sections 3.1 and 3.2 by smoothing transitions and simplifying technical descriptions where needed. While other reviewers found the presentation to be clear, we appreciate that individual perspectives vary, and we are committed to ensuring that the final version is accessible to a broader audience through careful revision.

Thank you for the detailed rebuttal. I think such contributions to representation learning are sufficient after reading the all rebuttals. Therefore, I have increased my score to 4.

We thank the Reviewer for the positive and encouraging feedback. We are glad that the motivation to adapt SAM2 to few-shot segmentation was found to be both interesting and technically sound, and the experiments to be comprehensive. We also appreciate the positive remarks on the clarity of the analysis and presentation. We hope that the additional insights and empirical evidence provided below further strengthen the paper and address the remaining concerns.

Q1: Instead of feature adaptation, why not retrain the model with the whole weights?

This is an interesting point, which directly relates to a fundamental aspect of few-shot segmentation (FSS). The core challenge in FSS is generalization: the ability to transfer to novel, unseen categories from only a handful of labeled examples. For this reason, recent state-of-the-art FSS methods [35, 37, 67, 72] rely on frozen Vision Foundation Models (VFMs) such as DINOv2 and Stable Diffusion. While these models provide rich, transferable features acquired through large-scale pretraining, it has been shown that in low-shot settings [A, B, C] retraining or fine-tuning leads to degrading the pre-trained representations and ultimately poor generalization.

We explicitly analyze this point in Section §C.3 of the Supplementary Material, where we compare our adaptation with several fine-tuning strategies, including decoder-only, QKV-only, and full-backbone tuning. These strategies update between 4M and 210M parameters, yet all fall short of our proposed adaptation.

In response to the reviewer suggestion, we additionally include two further baselines: i) full fine-tuning of the entire model, starting from SAM2 pretrained weights; and ii) training the SAM2 architecture from ImageNet initialization. Both configurations yield substantially lower performance. This confirms the theoretical expectation that, in few-shot regimes, i) full fine-tuning degrades the generalization acquired during pretraining due to overfitting on limited categories, while ii) training from a generic initialization lacks the semantic priors necessary to generalize to novel classes.

Table 1 from Section §C.3 of the Supplementary Material, † Newly added for rebuttal

[A] "Fine-tuning can cripple your foundation model; preserving features may be the solution." TMLR2024
[B] "Fine-tuning can distort pretrained features and underperform out-of-distribution." ICLR2022
[C] "Ranpac: Random projections and pre-trained models for continual learning." NeurIPS2023

W1: As SAM2 is designed for object tracking given videos with spatial relationships, there lacks a theoretical analysis and insights of the adaptation from continual frames to spatially independent pseudo-videos in the feature space.

We thank the Reviewer for the insightful comment, which raises an important point about the adaptation of SAM2 to spatially independent inputs in the few-shot segmentation scenario.

Although SAM2 is designed for tracking in temporally coherent videos, its architecture does not impose any prior on spatial continuity. In fact, as detailed in Eq. (2), masks are propagated through the video based on a cross-attention between frame features, that allows all-to-all matching. Therefore, spatial coherence is not structurally enforced, but rather an emergent bias that arises from pretraining on videos, encoded in the feature space. Our core finding is identifying, within such feature space, the presence of latent semantic structure (cf Linear probing, Section §C.2 Supp) entangled with spatial and appearance-related information (2D PCA, Section 3.4 Main). Our adaptation disentangles these signals to isolate the semantic information, enabling matching and segmentation across spatially independent inputs.

We agree that providing in-depth analyses and insights on the effect of this adaptation is important to ground our results. Indeed, Section 3.4 of the main paper is devoted to study how this adaptation reshapes the feature space, by showing the principal component analysis of the feature space before and after the disentanglement. Please note that, due to space constraints, a more thorough investigation with additional insights and theoretical justifications is presented in the Supplementary (Section  §C: Emergence of Semantic Representations).

While our analysis primarily focused on the emergence of semantic representations, we appreciate the Reviewer suggestion to further examine the effect of adaptation from continual frames to spatially independent pseudo-videos in the feature space. To this end, we have performed a new analysis adopting the spatial coherence metric from [70], which computes the average first-order gradient of the displacement field produced by matching: low values indicate smooth, object-consistent matches, while high values reflect noisy or fragmented correspondences. Through the lens of this metric, we investigate the extent to which SAM2 relies on spatial correlations in the data, and how our adaptation mitigates this dependence. We consider three settings, with progressively looser spatial relationships across image pairs:

• i) Video frames: pairs of consecutive frames from DAVIS [D], where both temporal and spatial continuity are preserved, e.g., a Labrador appears in the bottom left in the first frame and slightly to the right in the second.
• ii) Flipped frames: same DAVIS frame pairs, but the second image is horizontally flipped, disrupting spatial alignment while preserving object identity, e.g. the Labrador appears in the same pose but mirrored, now on the opposite side.
• iii) Independent images: pairs of unrelated images from the same category, sampled from TSS [E], with no spatial or temporal coherence, e.g. the first image shows a Labrador, and the second a Dalmatian.

Table R2 Spatial Coherence across varying continuity conditions
(Lower is better)

Results show that SAM2, as expected, performs well under i), degrades under ii), and struggles under iii), confirming that its coherence partially relies on a continuity in the video frames. In contrast, SANSA maintains coherence across all scenarios, showing its ability to abstract semantic correspondences without relying on spatial continuity.

[D] "A benchmark dataset and evaluation methodology for video object segmentation." CVPR16
[E] "Joint recovery of dense correspondence and cosegmentation in two images." CVPR16

We sincerely thank the Reviewer for these insightful suggestions. We will incorporate the new analysis into Supplementary Section §C, which focuses on the theoretical study of the adapted feature space. We believe this addition offers a valuable complementary perspective and further deepens the conceptual understanding of our approach.

Thanks authors for the detailed and comprehensive rebuttal. The additional experiments and analysis addressed my concerns. I would like to increase my score to 5.

We thank the Reviewer for the thoughtful and positive feedback. We are glad that the proposed solution was found to be both interesting and novel, and that the feature analysis, comprehensive ablations, and state-of-the-art results, across standard benchmarks and in-context settings, were recognized as strengths of the work. We also sincerely value the constructive questions raised. In the following, we did our best to provide clarifications and additional analyses to address these points as clearly and thoroughly as possible.

W1,Q1: "Do we really need masks? With SAM2 strong segmentation ability one doesnt need fully fine-grained mask for the few shot samples. How does the performance change when just prompts are provided in few shot samples (lets say just points instead of masks)?

We thank the reviewer for this valuable question. We fully agree that reducing annotation effort is desirable in practical applications. Motivated by this, our main paper includes an evaluation of SANSA under different prompt types (points, boxes, and even scribbles), reported in Figure 6 (top table), reproduced below. Implementation details are provided in §F.3 of the Supplementary, along with extensive qualitative results in Figure 5 of the Supplementary Material.

As shown below, SANSA achieves state-of-the-art performance across all prompt types, significantly outperforming VRP-SAM, also targeting promptable few-shot segmentation, by +15.0% with points, +5.8% with scribbles, and +4.6% with boxes. Furthermore, SANSA exhibits strong robustness, with a performance drop of –6.7% from full-mask to point supervision, compared to –15.5% for VRP-SAM.

We also highlight that several recent works [35, 67, 71] built on SAM cannot operate with point- or box-level prompts, as they rely on external encoders (e.g., DINOv2) for feature matching, and use SAM to produce the segmentation mask given the matched coordinate locations. VRP-SAM purposefully trains an additional decoder to support such prompts. In contrast, SANSA is the first to directly adapt the SAM2 feature space so that the same features can drive both feature matching and segmentation, enabling to fully repurpose SAM2 promptability for few-shot segmentation with low-effort prompts (L255-L264).

Table from Figure 6 (top table) of the Main paper

W3,Q2: Is the performance gain due to stronger SAM2? Prior works like VRP-SAM are based on older SAM and hence maybe limited in terms of performance.

We agree with the Reviewer on the importance of proper baselines. We address this point in Table 4 of Section §E of the Supplementary Material, reproduced below, where we isolate the effect of upgrading from SAM to SAM2 in recent Segment Anything-based methods. Recent state-of-the-art [21,20,42] models adopt a modular design: they use external encoders for feature matching (e.g. DINOv2), and treat SAM as a promptable segmentation decoder operating at the image level. As shown in Table 1, replacing SAM with SAM2 results in minimal performance gains (< 2% mIoU). This is consistent with results reported in the original SAM2 paper [27], which was designed for video segmentation and reports similarly small improvements in promptable image-level segmentation.

These findings demonstrate that our performance improvements do not arise from simply using SAM2. Instead, the key novelty lies in how SAM2 is used: our method adapts the feature space, revealing a previously unexplored semantic structure that supports both matching and segmentation within a unified framework.

Table 4 from Section §E of the Supplementary Material, † Newly added for rebuttal

W3, L1: Is the work limited to segmenting a single class?

We thank the reviewer for raising this point which gives us the opportunity to further clarify the discussion provided in Section §A.1 of the Supplementary. Our model can perform multi-class segmentation. Indeed, SANSA builds upon SAM2, which already supports tracking and segmentation of multiple objects by storing multiple object embeddings in the Memory Bank. In the same way, our framework naturally handles multiple categories by storing separate reference representations, predicting each class mask independently, and aggregating them into a final segmentation map [27].

That said, the current setup is shared by all existing methods [53, 35, 67], where categories are segmented independently, without leveraging potential relationships between them. In the Section §A.1 of the Supplementary, we highlight this as a broader limitation of the current few-shot segmentation paradigm. This paradigm is effective for evaluating per-class generalization; however, exploring how to incorporate cross-category reasoning could be a promising direction for future research.

W2: Did the work also trying integration of negative prompts?

We thank the reviewer for this insightful suggestion. While our main experiments do not include negative prompts, so as to ensure a fair comparison with prior works, we find this direction highly compelling and took the opportunity to explore it further. The key challenge lies in adapting the notion of negative prompting, originally designed for interactive segmentation, to the setting of few-shot segmentation.

Delving into this analysis, a key distinction is that negative prompts in the SAM2 paper are geometric cues (e.g., clicks) provided at test time to correct segmentation mistakes. Inspired by the reviewer suggestion, we propose to broaden the concept of negative prompting by introducing semantic negative prompts, namely, annotated examples of visually similar but semantically different categories, provided alongside the reference as contrastive context (e.g., including a photo of a horse to help disambiguate the target class zebra).

We thus explored two main strategies: geometric negative prompts (for test-time correction), and semantic negative prompts (for improved reference conditioning). Our experiments include both training-free and lightweight training-based variants, and are summarized below:

• (Training Free) Geometric Negative Prompts: In this setting, negative clicks are provided at test time on the target image, mimicking interactive correction. Since SANSA reuses SAM2 prompt encoder, we can directly leverage SAM2 built-in negative point mechanism.
• Semantic Negative Prompts: We then explore a novel interpretation of negative prompting in FSS, providing an annotated reference image of a semantically different (but potentially confusing) category. We implement it in two variants:
  • Training Free Negative Prompts: We encode the negative reference into SAM2 memory bank, assigning it an empty mask. This signals to the model that this region contains no object matching the current task.
  • Training-based Negative Prompts: We extend SAM2 prompt encoder with a small MLP that encodes the average feature of a negative object into a learnable negative prompt token, similar to SAM2 handling of negative clicks.

Table R1: Geometric vs. Semantic Negative Prompts

We believe this analysis to be highly interesting and potentially impactful. However, we emphasize that it opens a broad set of questions, particularly for semantic negative prompting, for which standardized baselines are currently lacking. We plan to explore these directions more rigorously in future works.

Once again, we sincerely thank the Reviewer for their time and careful consideration. We hope our responses have provided the necessary clarifications and addressed all points raised.