Segment Anything with Multiple Modalities

[Q3] In my understanding, MM-SAM is trained with single or paired modalities, it can be generalized to multi-modality input during inference, is it right? What is the relation with Imagebind?

Response: Thank you for the comment. MM-SAM is trained with paired modalities to align non-RGB representations with SAM's RGB latent space, enabling both single-modal segmentation and multi-modal fused segmentation.

MM-SAM is a flexible framework capable of generalizing to various combinations of data modality pairs (as shown in Table 1) and extending to three-modality combinations (discussed in Section 4.4.2 and Table 4(b)). However, it requires training the SFG module on paired data for selective fusion, meaning it cannot generalize to arbitrary multi-modal inputs during inference without prior training. Nevertheless, extending MM-SAM to new modalities remains cost-efficient and straightforward as SFG leverages pseudo-labeling without requiring human annotations.

Regarding its relation to ImageBind [1], while MM-SAM and ImageBind share certain broad goal of bridging multiple modalities, their tasks, motivations, and approaches differ significantly:

• Different Tasks: ImageBind is designed for CLIP, focusing on image-level understanding for vision-language modeling. In contrast, MM-SAM is tailored for SAM, addressing pixel-level segmentation tasks (dense predictions) in visual foundation modeling. Due to the task and architectural differences between CLIP and SAM, adapting ImageBind to SAM is not feasible.
• Different Insights: While ImageBind emphasizes universal embedding alignment across modalities for the entire model, MM-SAM introduces a novel insight by demonstrating that cross-modal representation unification can occur within a specific part of the architecture—SAM’s image encoder—while remaining fully compatible with its prompt encoder and mask decoder.

In summary, although both methods involve cross-modal knowledge transfer, MM-SAM is uniquely designed to extend SAM’s capabilities for multi-modal segmentation, addressing challenges specific to dense segmentation tasks and sensor fusion.

[W1&Q2] If trained MM-SAM is an all-in-one model? Each modality maintains an independent image encoder LoRA?

Response: Thank you for your comment.

MM-SAM is not an all-in-one model but a flexible framework (Lines 94-97, 317-318) that can generalize to various multi-sensor combinations, as demonstrated with seven different modalities in both SAM and SAM2 in our experiments. We believe such flexibility enables easy extension to new modalities and diverse downstream tasks, as highlighted by you in the Strengths.

In addition, each non-RGB modality is processed through its own patch embedding module, which handles modality-specific inputs; each modality also has a set of LoRA parameters that capture the unique characteristics of that modality. These parameters are lightweight and computationally efficient, ensuring that the overall model remains resource-efficient, as demonstrated in Table 1 of the manuscript. We will update the manuscript for a clearer description.

[W2] Clarify the insights of UCMT, WMMF and pseudo labeling.

Response: Thank you for the comment. We clarify that UCMT does not involve pseudo-labeling. Instead, it employs unsupervised feature unification as defined in Equation (1) using paired data.

Regarding pseudo-labeling, we emphasize its necessity and value in MM-SAM. For many sensor modalities and tasks, such as SAR in remote sensing, collecting large annotated datasets is highly impractical due to the significant costs and labor involved. MM-SAM incorporates pseudo-labeling and achieves superior performance without requiring annotations, thereby greatly reducing the burden of data collection and broadening its application scope, as valued by Reviewer c6Lh and you.

We also highlight that the primary contribution of MM-SAM does not lie in the individual technical modules, but in the innovative extension and expansion of SAM toward cross-sensor (single non-RGB modality) and multi-sensor (multiple modalities) segmentation tasks, achieved in a label-free manner. However, UCMT and WMMF are carefully tailored to address this new challenge with unique insights and considerations distinct from previous work. These insights, detailed in Lines 302–318, are summarized below:

• Novel insight: SAM’s pre-trained RGB representation space can effectively support non-RGB modalities, which forms the foundation of MM-SAM’s technical modules.
  • Cross-sensor segmentation: Building on this shared representation, UCMT enables single-modal segmentation (non-RGB data) through unsupervised cross-modal representation unification for knowledge transfer.
  • Multi-sensor fusion: This unified representation across modalities significantly mitigates the challenges of data heterogeneity. To this end, WMMF learns fusion weights only without altering features of different modalities, enabling superior multi-modal segmentation.
• Generalizability: UCMT and WMMF support both SAM and SAM2 and have been validated across seven different sensor modalities, as shown in Tables 2–5. To our knowledge, few studies on multi-modal segmentation provide such comprehensive evaluations, highlighting the generalizability and robustness of the MM-SAM framework.

MM-SAM represents a significant paradigm shift from conventional multi-modal segmentation methods. It eliminates the need for intensive manual annotations that previous methods need for learning shared representation. We believe this innovation could support a wide range of applications, such as robotics, remote sensing, and autonomous driving, where sensor suites are crucial for robust perception and broader deployment. We will include this discussion in the updated manuscript to highlight the originality and significance of MM-SAM better.

We sincerely thank you for your valuable feedback, particularly for your recognition of MM-SAM's unsupervised training approach, extensibility, simplicity, strong results, and the clarity of our manuscript. Below, we provide detailed, point-by-point responses to address your concerns.

[W1&Q1] More descriptions about prompt input.

Response: Thank you for the comment. We follow [A] and use bounding boxes as input prompts in the training and inference. We will clarify this issue in the updated manuscript.

[A] Segment anything in high quality. NeurIPS 2023.

Dear Reviewer X7NB,

We hope this message finds you well. We have submitted a detailed response addressing your valuable feedback and appreciate the opportunity to clarify and strengthen our work.

We kindly request that you review our response and reconsider our submission in light of the clarifications and supporting evidence provided. We are dedicated to ensuring that we fully meet your expectations.

Thank you once again for your time and consideration.

Best regards,
The Authors

[W4] More fusion details.

Response: Thank you for the suggestion. As described in Lines 264–265, we adopt a confidence-based fusion strategy to combine single-modal mask predictions from RGB ($M_I$) and X-modal data ($M_X$). The confidence of each pixel $(x, y)$ is defined as the maximum absolute value of logits in the respective mask predictions across modalities. Specifically, we compare the confidence scores at corresponding pixel locations in the RGB and X-modal predictions and select the higher one as the fused prediction, formulated as $M_F(x,y) = \max(M_I(x,y), M_X(x,y))$. This approach ensures that the fused prediction leverages the most confident predictions across modalities at each pixel location, effectively utilizing the complementary strengths of different modalities. We will include this explanation in the updated manuscript to enhance clarity.

We sincerely thank you for your valuable feedback, particularly your recognition of the novelty of our approach, as well as the effectiveness and comprehensiveness of our experimental results. In the following, we address your concerns in detail.

[W1] Originality of MM-SAM and difference from suggested studies.

Response: Thank you for the feedback and the shared prior studies.

We would clarify that the primary contribution of MM-SAM does not lie in the individual technical modules but in the innovative extension and expansion of SAM from single RGB segmentation toward cross-sensor (single non-RGB modality) and multi-sensor (multiple modalities) segmentation tasks, achieved in a label-free manner, as recognized by all reviewers.

By addressing under-explored challenges, the framework of MM-SAM demonstrates significant originality, as detailed in Lines 302–318 of the manuscript. Below please find a brief summary:

• Novel insight: We show that SAM’s pre-trained RGB representation space can be effectively shared across non-RGB modalities, forming the foundation of our technical modules.
  • Cross-sensor segmentation: Building on this insight, we develop a straightforward cross-modal feature unification to transfer knowledge from SAM’s pre-trained RGB latent space to other modalities. We adopt LoRA [1] for parameter-efficient tuning (PEFT) due to its wide usage and lightweight design (Lines 231–232), though other PEFT methods such as Adaptformer and VPT could also be applied, as demonstrated in Fig.5 (a).
  • Multi-sensor fusion: The unified representation across modalities significantly mitigates the challenges of data heterogeneity. To this end, we design the SFG module, which learns fusion weights only without altering features of different modalities, enabling superior multi-modal segmentation.
• Generalizability: Extensive experiments, as shown in Tables 2–5, demonstrate MM-SAM’s compatibility with seven different sensor types across both SAM and SAM2 frameworks. To our knowledge, few studies on multi-modal segmentation provide such comprehensive evaluations, highlighting the generalizability and robustness of MM-SAM's framework.

MM-SAM introduces a paradigm shift in multi-modal segmentation by aligning representations from SAM’s RGB latent space for multiple sensor modalities without human labels. This contrasts with the suggested studies with fundamentally different scopes, objectives and tasks:

• LoRA-based studies [2, 3] fine-tune SAM within RGB domains, including remote sensing [2] and blood cell classification [3], using fully supervised frameworks that rely on large-scale human annotations.
• For Fusion modules: [1] uses gated fusion for multi-scale feature integration in fully supervised RGB semantic segmentation. [2] applies selective gating for relation classification in NLP, which is a fundamentally different domain. [3] introduces SA-Gate to recalibrate RGB and depth features in a fully supervised way.

We thank you for your suggestions. We will review the suggested studies in the updated manuscript to better illustrate MM-SAM’s originality and its contributions to the field.

[W2] The ablation of the SFG module.

Response: Thank you for the comment. The SFG module is designed specifically for fusing multi-modal features, and removing it would result in the loss of multi-modal segmentation capabilities.

To examine the effectiveness of SFG as suggested, we conduct an ablation study and compare three different fusion strategies. The math notations are consistent with those in the manuscript:

• Max fusion: The maximum prediction probabilities from RGB and non-RGB feature embeddings are selected at each position: $\hat{e} _ {F _ i} = \max(e_{I_i}, e_{X_i})$. Here, embeddings ($e_I$, $e_X$) are encoded from RGB images and X-modal data $(I, X)$, respectively, with $i$ denoting the patch index.
• Avg fusion: The average of feature embeddings from both modalities is used as the fused embedding: $\hat{e} _ F = \frac{1}{2}(e_{I_i} + e_{X_i})$.
• SFG fusion: The SFG module adaptively learns fusion weights $\omega_i$, as implemented in MM-SAM, following Equation (2): $\hat{e} _ F = \omega_i e_{I_i} + (1 - \omega_i) e_{X_i}$.

We also include SAM’s performance using only RGB inputs for reference. Several points can be concluded from the table below:

• Max fusion: Naively selecting maximum probabilities leads to suboptimal fusion segmentation performance.
• Avg fusion: This fusion strategy can be considered as a special case of SFG by assigning equal weights ($\omega_i = 0.5$). This fusion strategy consistently improves segmentation over SAM, confirming that weighted fusion benefits cross-modal integration. This aligns with our core insight: unifying representations in SAM’s RGB latent space enables effective multi-modal integration, even with simple fusion methods.
• SFG fusion: Unlike Avg fusion, which treats all modalities equally and overlooks their complementary strengths, SFG dynamically adjusts to sensor-specific inputs, as shown in Fig. 4. By leveraging the strengths of each modality, SFG delivers the best segmentation robustness and accuracy across all datasets.

The results further validate the significance of our insights and the effectiveness of the SFG design. We thank you for your suggestions and will include this analysis and discussion in the revised manuscript.

[W3] More results or analysis for SAM’s performance on uncommon data distributions such as remote sensing.

Response: We emphasize the use of pseudo-labeling in MM-SAM because collecting large-scale annotated datasets for many sensor modalities, such as SAR in remote sensing, is challenging and resource-intensive. The pseudo-labeling in MM-SAM allows for achieving competitive performance without requiring manual annotations, greatly reducing the cost and effort of data collection and broadening its applicability to diverse domains, as highlighted by Reviewers c6Lh and X7NB.

On the other hand, MM-SAM is flexible and can operate with annotated data when available. Specifically, pseudo-labels in WMMF’s SFG module can be replaced with ground-truth annotations to train fusion weights. To evaluate this, we conducted experiments on both MM-SAM and MM-SAM2 using ground-truth labels (“GT”) and compared them to the original pseudo-label-based results (“PL”). The results of four remote sensing benchmarks are summarized in the table below:

• DFC23 (RGB+SAR) and Potsdam (RGB+DSM): Models trained with ground-truth annotations achieved slightly higher performance compared to pseudo-label-based training.
• DFC18 (RGB+MS-LiDAR/HSI): Models trained with pseudo-labels performed slightly better than those trained with ground truth.

These results demonstrate that pseudo-labels provide comparable supervision to human annotations in fusion learning, highlighting MM-SAM’s superiority in single-modal prediction and SFG’s effectiveness in achieving adaptive fusion. These findings further validate the practicality and robustness of MM-SAM’s design, particularly in scenarios where annotated data is scarce or unavailable. We thank you for your suggestions and will include this analysis and discussion in the revised manuscript.

Dear Reviewer HTYB,

We hope this message finds you well. We have submitted a detailed response addressing your valuable feedback and appreciate the opportunity to clarify and strengthen our work.

We kindly request that you review our response and reconsider our submission in light of the clarifications and supporting evidence provided. We are dedicated to ensuring that we fully meet your expectations.

Thank you once again for your time and consideration.

Best regards,
The Authors

Based on the experiment results and limitations of the paper as well as the answers of the rebuttal, i decide to keep my score.

We sincerely thank you for your valuable feedback, particularly for your recognition of the novelty, simplicity, versatility, and efficiency of our approach, as well as the effectiveness and comprehensiveness of our experimental results. Please find our responses to your concerns as follows.

[W1] Clarification of originality and contributions of individual techniques in MM-SAM.

Response: Thank you for your feedback. We would first clarify that our main contribution lies not in the individual technical modules but in the innovative extension and expansion of SAM toward cross-sensor (single non-RGB modality) and multi-sensor (multiple modalities) segmentation tasks, achieved in a label-free manner, as you recognized.

As detailed in Lines 98-99, MM-SAM is the first study to tackle this challenge, with meticulous technical designs and novel insights that operate under distinct logistics and considerations compared to prior works (Lines 302–318). We summarize its uniqueness as compared with prior studies as follows:

• Novel insight: We show that SAM’s pre-trained RGB representation space can be effectively shared across non-RGB modalities, forming the foundation of our technical modules.
  • Cross-sensor segmentation: Building on this insight, we directly implement cross-modal feature unification to transfer knowledge from SAM’s pre-trained RGB latent space to other modalities. A straightforward parameter-efficient tuning in UCMT enables effective cross-modal segmentation without requiring human annotations.
  • Multi-sensor fusion: The unified representations across modalities significantly mitigate the challenges of data heterogeneity. To this end, we design the SFG module, which learns fusion weights only without altering features of different modalities, enabling superior multi-modal segmentation.
• Generalizability: We conducted extensive experiments, as shown in Tables 2–5, to demonstrate MM-SAM’s compatibility with seven different sensor types across both SAM and SAM2 frameworks. To our knowledge, few studies on multi-modal segmentation provide such comprehensive evaluations, highlighting the generalizability and robustness of MM-SAM's framework.

MM-SAM represents a significant paradigm shift from conventional multi-modal segmentation methods. It eliminates the need for intensive manual annotations that previous methods need for learning shared representation. Hence, MM-SAM has great potential to support a wide range of applications, such as robotics, remote sensing, and autonomous driving, where sensor suites are crucial for robust perception and broader deployment. We will include this discussion in the updated manuscript to highlight the originality and significance of MM-SAM better.

[W2] More comparison with segmentation-specific methods.

Response: Thank you for the comment. We clarify that Table 10 in the appendix includes comparisons to state-of-the-art methods designed for segmentation tasks (sourced from paperwithcode), using widely adopted datasets such as SUN RGB-D and MFNet. While MM-SAM’s primary goal is to extend SAM for cross-sensor and multi-sensor segmentation, these comparisons provide a broader evaluation of MM-SAM’s performance. We copy the Table 10 below for your quick reference.

It is important to note that the benchmarked methods, including the suggested 2DPASS, achieve high performance by leveraging full supervision and access to large-scale, densely annotated datasets. In contrast, MM-SAM is designed for label-free adaptation. Additionally, these approaches focus on semantic segmentation, whereas MM-SAM focuses on promptable segmentation as SAM. These fundamental differences make direct comparisons less equitable.

Nevertheless, MM-SAM demonstrates highly competitive performance despite its unsupervised nature. As shown in the tables, MM-SAM significantly outperforms state-of-the-art supervised methods on SUN RGB-D and MFNet. Even on the suggested SemanticKITTI, MM-SAM achieves a strong result (66.4 vs. 72.9 by 2DPASS), which is particularly noteworthy given that MM-SAM requires no manual annotations as its training ground truth.

Dear Reviewer c6Lh,

We hope this message finds you well. We have submitted a detailed response addressing your valuable feedback and appreciate the opportunity to clarify and strengthen our work.

We kindly request that you review our response and reconsider our submission in light of the clarifications and supporting evidence provided. We are dedicated to ensuring that we fully meet your expectations.

Thank you once again for your time and consideration.

Best regards,
The Authors