RMP-SAM: Towards Real-Time Multi-Purpose Segment Anything

Q1: Many architectural elements were adopted from other works, it is not clear to me if there are already similar architectures as proposed here, or where exactly is the innovation (except the jointly training).

Firstly, we integrate common components from K-Net and YOSO into our network architecture. However, we propose utilizing shared decoders and independent adapters for multi-task co-training within these structures. Our main contribution is the introduction of a new benchmark designed to support various segmentation tasks through a single real-time model. To evaluate this benchmark, we extended several existing real-time segmentation models to handle panoptic, video instance, and interactive segmentation within a unified framework. Finally, we present our model, RMP-SAM, a dynamic convolution-based approach with an improved task adaptation design. RMP-SAM achieves an optimal trade-off between performance, task versatility, and speed. We note that expanding the image along the time dimension is merely a specific implementation strategy and not our core contribution.

Q2: Architectural level comparison

Thank you for your reminder. We have provided a more detailed architectural-level comparison in the related work section. Our architecture incorporates common network designs such as YOSO's lite neck and K-Net's cascade decoder. Unlike K-Net, we unify various tasks into mask prediction by applying Hungarian matching between the predicted and ground truth masks. In contrast to the MaskFormer series, which relies on a masked-attention mechanism, we have developed an efficient and lightweight dynamic convolution framework alongside a decoupled adapter for semantic-aware segmentation and visual prompt-aware segmentation. Most importantly, we have successfully unified multiple tasks within the same architecture, which represents our core contribution. Please the ablation parts and appendix on the effectiveness of decoupled adapter.

Q3: The tables, especially table 3, are difficult to read because nothing is in bold print and you have to search for the trade-off here. A plot like Fig. 1b would be more useful.

Thank you very much for your reminder. We have modified the structure of the table by bolding some of the results for easier reading and comparison. Please revisit our paper to review these changes.

Q4: The references to the appendix could be a little more precise and there is no reference to Table 2.

Thank you very much for your correction. We have checked the references in the appendix and added the connection to Table 2.

Q5: What does the dot size in Fig. 1b indicate?

Thanks for your suggestion, we have made it clear in our refine draft. We adopt different dot sizes as different model parameters (model size). Large dots mean larger mode sizes.

Q6: The abstract says "generalization ability of these models across diverse scenarios", a learnable classifier with CLIP text embeddings is also used and “segment anything” is in the title. Is there a connection to open-vocabulary?

As you mentioned, our approach does resemble open vocabulary learning. Replacing the learnable classifier with CLIP text embeddings is a standard operation in open vocabulary learning. For our project, utilizing CLIP text embeddings rather than a learnable classifier is essential for unifying multiple tasks. While our model's ability to recognize unseen objects is important, our primary objective is achieving task unification.

Dear authors,

after reading the other reviews and the answers, I decided to raise my score. Thank you for the replies.

Q1: Detailed information about joint training.

Thanks for your questions. We agree that joint co-training can be complex and can lead to performance degradation compared with single tasks. We answer your questions in two aspects: technical parts and training data parts:

Technical parts: We first unify multiple segmentation tasks for joint co-training under a single paradigm. All training targets are represented as one entity label and mask for all three cases, where the entity can be a thing, stuff, class-agnostic mask, or their corresponding labels. We unify label taxonomies across datasets and replace the learnable classifier with CLIP text embeddings. Hungarian matching is then applied between the predicted and ground-truth entity masks to assign object queries to video/image entities. In addition, we present a decoupled decoder design to better balance the semantic-aware masks (panoptic segmentation and video instance segmentation) and prompt-aware masks (interactive segmentation). Please see the ablation parts of this design.

Training data parts: To balance the performance across tasks, we adjust the proportion of training data for each dataset. In our setup, the training data is sampled with a ratio of 1:25:1 for COCO-panoptic, Youtube-2019, and COCO-SAM datasets, respectively. Adopting this data balance method, we can still achieve strong performance of VIS.

At last, when we compare with SAM-like model, we find that co-training is not consistently effective, as performance degradation may occur due to the increased complexity and challenges of multi-task joint co-training. To address this, we first pretrain our model on the COCO dataset. After co-training, we finetune the model using the SAM dataset (5%) to enhance interactive segmentation performance. Thus, the results are reported in other tables (see the Tab.4, Tab.10,11,12).

We have updated these detailed processes in our update draft, and we will open source the training code for these settings.

Q2： This paper ignores some related methods, making it difficult to assess the model's performance relative to existing SOTA approaches. For example, some universal methods obtain better results than RMP-SAM using ResNet50. The authors should make a comprehensive comparison with other methods.

Thanks for your suggestion. We agree that the scope of our research is orthogonality to the works[1]-[3].

Firstly, since our method is not designed for specific models and our method is trained on multiple purposes tasks, directly comparing our method with previous works[1][2] may not be fair. As our core goal is unifying multiple tasks in real-time scenarios and adopting a multi-task co-training strategy, our method may not perform the best on a single task.

Secondly, The work[3] also designs a unified model for video segmentation tasks. However, it cannot perform SAM-like segmentation in real time. We adopt joint image-video data co-training rather than image or video data. Thus, we can keep SAM-like segmentation, panoptic segmentation, and video instance segmentation in one model and run in real time.

Thirdly, all these methods focus on achieving strong results rather than real-time design. The work[2] uses extra transformer encoders after heavy Mask2Former architectures, which brings more computation costs.

At last, following your suggestion, we have added a comparison with previous SOTA methods on YTVIS--2019. We follow the work[1][2], with the modification of replacing pre-training on COCO with our co-training on COCO, COCO-SAM, and YTVIS-2019. Then, we follow the works[1][2] and finetune the pre-trained model on YTVIS-2019. Nevertheless, we achieved 47.2 results when using the ResNet-50 backbone. To provide more comparison, we use ConvNeXt-L as our backbone and train a model based on the joint co-training described in the paper, achieving a result of 62.2. The results indicate our method can still achieve comparable results with these expert models.

According to your suggestion, we have updated this detaild comparison in our appendix. Please check our updated draft.

[1] Tube-Link: A flexible cross tube framework for universal video segmentation. ICCV 2023.

[2] Dvis: Decoupled video instance segmentation framework. ICCV 2023.

[3] Univs: Unified and universal video segmentation with prompts as queries. CVPR 2024

Q1: Novelty and technical contribution.

Our main contribution is introducing a new setting that supports various segmentation tasks within a single real-time model. To our knowledge, there are no previous works in this direction. Most efficient models[1]-[3] are working on a single task and verifying their method on a single dataset.

Thus, we benchmark several existing real-time segmentation models (including Mask2Former), extending them to handle panoptic, video instance, and interactive segmentation within one unified framework. However, most works cannot achieve the best speed and accuracy trade-off on multiple tasks.

Next, we present our model, RMP-SAM, a dynamic convolution-based approach with an improved task adaptation design. Our key designs include a shared convolution-based decoder with mask pooling to accelerate the decoding process and a decoupled adapter to decode the semantic-aware masks (image masks, video tube masks) and visual prompt-aware masks (interactive masks). With these designs, our method, RMP-SAM, can achieve the speed and accuracy trade-off on our proposed benchmark.

RMP-SAM achieves the best trade-off between performance, task versatility, and speed. We note that expanding the image along the time dimension is merely a specific implementation strategy and not the core contribution of our work.

[1] ICNet for real-time semantic segmentation on high-resolution images, ECCV-2018.

[2] You Only Segment Once: Towards Real-Time Panoptic Segmentation, CVPR-2023.

[3] Faster Segment Anything: Towards Lightweight SAM for Mobile Applications, arxiv-2023

Q2: Differences with SAMv2 should be further clarified, especially in terms of claimed semantic labels?

Thanks for your suggestion. Our updated draft provides a more detailed comparison with SAMv2. Here are detailed comparisons:

(1) For functionality, SAM-2 mainly focuses on video object segmentation and interactive segmentation. Compared with SAM, it adds mask tracking ability on given visual prompts, while our method mainly explores multi-purpose and multi-task real-time segmentation. We unify panoptic segmentation, video instance segmentation, and interactive segmentation in one model with the requirements in real time.

(2) For data scale and diversity, SAM-2 builds a large scale dataset and mainly has one purpose: Video Object Segmentation. Our RMP-SAM only involves a small set of public data sources and has multiple purposes: segmentation, mask labeling, mask tracking, and panoptic segmentation.

(3) For goals, SAM-2 aims at the production level with large-scale datasets (including internal datasets) co-training. Our RMP-SAM aims at efficient model design and performs well under real-time constraints.

(4) Last, our work is concurrent with SAM-2 since our work is also inspired by pioneering SAM.

We have updated these responses in our refined draft.

Q1: Compared with other SAM-like methods with more detailed metrics and more stronger detectors.

Thank you for your suggestion. In the updated draft, we have provided a more detailed comparison with other SAM-like methods in Table 10-12 and Figure 5. Please check our paper for a more detailed comparison. The results show that our model achieves comparable or better performance across various detectors and metrics while significantly reducing FLOPs. Moreover, our method is able to support panoptic segmentation and video instance segmentation, which other SAM-like methods cannot, highlighting our core contribution: a multiple-purpose real-time segmentation model.

Q2: Testing the efficiency across different GPU platforms.

We have evaluated several methods from Table 2 across multiple GPU platforms, including A100-40G, A10-22G, and 3090-24G. Unfortunately, we currently do not have access to a V100 GPU and are unable to provide its corresponding results.

All model results for each specific GPU were generated on the same machine. The FPS and GFlops values were calculated using an image with a 1333 x 800 pixels resolution. We report these results with the ResNet-18 backbone.

As shown in the above table, our method can achieve faster speeds at A10 and 3090 and comparable speeds at A100-40G.

Q3: The latency visualization results compared with other SAM-like methods.

Thanks for your suggestion. We have provided visualization results in our updated draft. Please check the Figure 5 in the appendix for this. In particular, we report the GFlops, parameters, and performance comparison with these methods.

Q4: The result of TopFormer in Tab-3 and bold the results in all comparison tables.

Thanks for your suggestion. We have provided the results of the TopFormer and modified the table's structure by bolding some of the results for easier reading and comparison. Please revisit our paper to review these changes.