SAM 2: Segment Anything in Images and Videos

How would existing SOTA VOS methods perform after finetuning on the SA-V dataset? The paper uses SAM+Cutie as a baseline; however, Cutie has not been trained on large-scale data like SA-V. Therefore, I am interested in Cutie's performance after training with the SA-V dataset. After finetuning with SA-V, would SAM+Cutie surpass SAM2? This would not affect the paper's significant contributions, regardless of the results, as SAM2 is an end-to-end model.

Thank you for your suggestion! We followed your recommendation and fine-tuned the best available Cutie model ("cutie-base-mega") on our SA-V dataset.

After fine-tuning Cutie on the SA-V dataset, the semi-supervised VOS J&F results improved from 62.8 to 69.0 on SA-V test (still largely underperforming the 76.0 J&F obtained by SAM 2 on SA-V test, reported in Table 17a). Also, we found that after fine-tuning, the SAM+Cutie baseline (denoted as "SAM+Cutie (SA-V ft)" in the table below) still largely underperformed SAM 2 on our full benchmark of 17 zero-shot video datasets. Below are the full results:

Table: Zero-shot accuracy across 17 video datasets using different prompts (same setting as Table 4 in our submission).

We are also exploring joint training of Cutie on our SA-V dataset along with previous open-source datasets and will include these results in the final version of the paper.

I suggest the authors should cite some works from VIS [1, 2, 3, 4], VSS [5], and VPS [6, 7, 8] fields in the related works section, as these areas are also highly relevant to this paper.

We appreciate these suggestions and will ensure that all of these works are cited in the final version of the paper. Thank you for your insightful review. Please let us know if you have any further questions, and we will be happy to address them.

Thanks for the authors thoroughly addressing all of my questions. I decide to raise my score to 10.

The claimed improvement in running speed is mainly due to the usage of the Hiera image encoder, which may not be viewed as a unique contribution of this study.

While the use of Hiera is indeed an important factor in improved running speed, we actually use a simpler and faster version of Hiera than prior work (see Appendix D.1 for details on the image encoder). Our version is ~1.5x faster than Bolya et al., ICLR, 2024.

Importantly, our goal is not just improved running speed but improved running speed while maintaining or improving on accuracy compared to SAM 1 (on images). To this end, we found other modeling improvements over SAM 1 to also be important, such as the use of a feature pyramid network in the mask decoder, supervising all IoU predictions, and $l_1$ losses for IoU, etc. (see Appendix D.1 and D.2).

It is the combination of these modeling changes that results in a 6x improvement in running speed (with accuracy >= SAM 1).

The primary contribution lies in a large-scale dataset and pre-trained models, while the technical contribution is relatively limited.

Our technical contributions are along three axes: model, data engine, and task.

[Model] The model is a first-of-its-kind unified architecture for images and videos capable of real-time inference on videos with a step-change in performance. Besides the modeling innovations discussed above (improvements in image encoder and training objectives) which benefit both images and video, our design has several key video-specific innovations: (a) occlusion head + ambiguity handling, (b) memory attention mechanism with spatial and object pointer memories, (c) simple, scalable memory encoder, and (d) support for prompting on any frame. In more detail:

• (a) Introduces a way to model ambiguity (and occlusion) in the target object, which is fundamental to the promptable visual segmentation task. Dropping this results in significantly worse quality masks containing visual artifacts. Moreover, the occlusion head allows principled memory selection and can help with longer occlusions (+2.8 J&F on SA-V test, see response to reviewer DQTY).
• (b) We introduce a novel memory attention module with spatial memories and object pointers. Object pointers significantly help performance (+3.8 on SA-V val), while modeling spatial memories with RoPE allows precise modeling (+1.2 on SA-V val) of high-resolution spatial information.
• (c) Contrary to prior work, we do not use a separate image encoder for memories and frame embeddings. This design allows the quality of spatial memories to scale with the quality of the image encoder (unlike prior work). Without this design, we observed worse results from scaling the image encoder in early prototypes.
• (d) SAM 2 natively supports interactivity on any frame of the video without requiring a separate image segmentation model. Our analysis in Table 1 shows the improvement in annotation speed from a unified SAM 2 compared with a decoupled image segmentation model + tracking model.

[Data Engine] The development and training of the SAM 2 model were powered by the availability of a large-scale diverse dataset. However, it is important to note that the developed SAM 2 model also plays a crucial role in the collection of this dataset. SAM 2 was integrated into the annotation tool and could be used online during the annotation task. The model actively contributed to the data collection and refinement, including enabling identification and targeting of annotations on the failure cases of the model. The model advancements and the development of an efficient data engine together enhance the quality and scale of the collected SA-V dataset. This iterative process of model improvement and data collection creates a feedback loop that drives continuous improvement in both the dataset and the model performance. This highlights the technical contributions that underpin the joint development of the large-scale SA-V dataset and SAM 2 model.

[Task] We generalize existing tasks by introducing the promptable visual segmentation task for images and videos as well as relevant online and offline metrics to measure performance in this general setting. This task was specifically aligned with our data engine requirements, such that better performance on the task would directly result in a better model for our data collection. Having a clear, general task and relevant metrics is the foundation that made everything else possible.

We appreciate the reviewer's insightful feedback. If there are any other questions or areas where you would like more information, please feel free to reach out, and we will gladly provide further details.

Has SAM 2 attempted stability testing for results on ultra-long videos? Is object tracking in long videos more prone to errors?

Thanks for raising this question. Longer videos have a larger discrepancy from the training setting, and it is expected there will be some performance degradation (vs shorter videos). Indeed, we can see such a degradation below where we bin videos by duration and report performance on these bins.

While SAM 2 was not specifically designed for long videos (an important and challenging research direction on its own), it nevertheless shows a step-change in performance on long VOS benchmarks such as LVOS v1 and v2 (e.g. > +16 J&F on LVOS v2 for SAM 2.1 (L) vs previous best reported performance).

Moreover, with some further modeling improvements, we are able to mitigate degradation on long videos, see table below where we measure the mean (SEM) performance across varying video durations of the LVOSv2 dataset. Notice that the gains are larger for longer videos.

Table reports performance on the semi-supervised VOS task on LVOS v2, binned by video duration.

Finally, note that since SAM 2 is an interactive model, it also allows corrections in a later frame to recover performance.

If the memory bank is of fixed size, will it lead to forgetting when dealing with long videos?

This is a good point that warrants further discussion. The memory bank for SAM 2 indeed has a fixed maximum capacity. However, this is also the case with nearly all practical models, from prior work in video object segmentation to LLMs (due to their finite budget of e.g., GPU memory). Prior works often use various ad-hoc heuristics (e.g., [1]) to decide how to use the limited model capacity effectively.

In contrast with ad-hoc approaches, SAM 2 is carefully designed to mitigate such issues:

• By default SAM 2 always retains all prompted frames in the memory bank, which allows it to remember the prompted information regardless of the video length and alleviates forgetting.

• SAM 2 is an interactive model and can therefore be prompted on any frames in the video. So even when SAM 2 makes a mistake in long videos, it can be easily fixed with additional prompting.

• SAM 2’s occlusion head (which predicts whether the object being tracked is present on a frame) allows a principled way to select memories to be placed/retained in the memory bank. Adopting such a strategy further improves the performance, e.g., we find SAM 2.1’s performance improves by +2.8 J&F from 78.2 to 81.0 on SA-V test in the semi-supervised VOS setting.

• Finally, SAM 2 has two kinds of memories, spatial memories and object pointer memories - the latter are $\approx10^3$ times smaller than spatial memories, which allows us to retain them over longer temporal context windows and further mitigates forgetting in long videos.

[1] A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression, Devoto et al, EMNLP 2024.

We thank the reviewer for their insightful review and suggestions for improvement. Please let us know if anything else is unclear, and we are happy to answer.

How is the annotation quality of the SA-V dataset, and how do you ensure the quality of the annotations? What is the difficulty level of this dataset (such as the movement of objects in the video, occlusions, etc.) compared to previous datasets?

To ensure annotation quality we include a rigorous quality verification step in the data engine (detailed in Section 5.1). A separate set of annotators are tasked with verifying the quality of each annotated masklet, ensuring every masklet in the SA-V dataset is "satisfactory", i.e., it correctly and consistently tracks the target object across all frames. Unsatisfactory masklets were sent back to the annotation pipeline for refinement.

We use the disappearance rate (the percentage of annotated masklets that disappear in at least one frame and re-appear) as the proxy for occlusion and the difficulty level of datasets. The SA-V manual subset has the highest disappearance rate (42.5%) compared with all previous datasets. The comparison of SA-V with respect to previous datasets on dataset size, video duration, and disappearance rate is provided in Table 3.

We compare the mask size distribution, normalized by video resolution, with previous datasets including DAVIS, MOSE, and YouTubeVOS, in Figure 10 (a). More than 88% of SA-V masks have a normalized mask area less than 0.1, and the SA-V dataset contains more small objects compared to MOSE and YouTubeVOS. In addition, the SA-V dataset also contains annotations for object parts as well as whole objects, compared to previous datasets which focus on whole objects across common categories.

We also find the SA-V benchmark to be challenging in terms of absolute J&F accuracy. As summarized in the table below (from Table 6 in the paper), the J&F accuracy of SAM 2 models is lower on SA-V val compared to DAVIS 2017 val and YouTubeVOS 2019 val. This suggests that the SA-V benchmark is a challenging testbed.

More experiments should be conducted. For example, more interactive VOS methods should be compared. [1*] Modular interactive video object segmentation: Interaction-to-mask, propagation and difference-aware fusion. CVPR 2021 [2*] Memory aggregation networks for efficient interactive video object segmentation. CVPR 2020

Thanks for your feedback! We have also compared with MiVOS [1] in Table 14 of our appendix, under the DAVIS interactive benchmark and following the setting in CiVOS ([A]: https://arxiv.org/abs/2203.01784). It was shown that SAM 2 (under click inputs) outperforms MiVOS under the same type of click inputs, and achieves comparable performance to the MiVOS variant under a strong type of scribble inputs. SAM 2 also largely outperforms MANet [2] (which has a lower performance than MiVOS [1]).

After our submission, we found that SAM 2 could perform better on this benchmark under slightly different eval parameters. The table below is a summary:

Table: Performance of SAM 2 and other models on the DAVIS interactive benchmark (same setting as Table 14 in our submission).

As noted in appendix F.2 of our submission, we found that SAM 2 often tends to segment object parts (e.g., a person’s arm) on the first click while the DAVIS dataset mainly contains whole objects (e.g., an entire person), which could unfairly penalize SAM 2’s J&F performance on this benchmark for being more flexible.

More VOS datasets (e.g., VIPOSeg[4*]) should be included in this paper. [4*] Video Object Segmentation in Panoptic Wild Scenes. IJCAI 2023

Thanks, this is a great suggestion! Please note that SAM 2 has been evaluated on 17 zero-shot benchmarks for VOS in Table 4 and Figure 15. Following your suggestion, we also evaluated SAM 2 on the VIPOSeg dataset and found that SAM 2 has a strong performance on this benchmark:

• In a zero-shot manner on this dataset, SAM 2 already achieves G=78.4 on VIPOSeg val (where G is the overall metric defined in [4], higher is better), on par with the G=78.2 performance of PAOT (the best model in Table 2 of [4] trained on this dataset).

• When fine-tuned on the VIPOSeg training set, SAM 2's performance is further improved to G=79.7 on VIPOSeg val.

• In addition, SAM 2 is more robust in crowded scenes of VIPOSeg as measured by the decay metric defined in [4] (lower is better). On VIPOSeg val set, SAM 2 achieves λ=0.68 decay (without fine-tuning on VIPOSeg) and λ=0.67 decay (when fine-tuned on VIPOSeg train set), both outperforming the λ=0.70 decay of PAOT (the best model in Table 2 of [4] trained on this dataset).

We will add these results on VIPOSeg to the final version of the paper.

It would be even better if the training code and data engine could be publicly available.

We agree. We are releasing training code and the code for a web demo which resembles our annotation tool for the SAM 2 data engine. In Appendix E2 we provide further details on the annotation protocol and tasks (Figure 11). We are also releasing inference and fine-tuning code, together with the models under a permissive license.

Although this paper uses a simpler structure and performs well, it is still possible to use previous structures, such as Cutie [R1], to achieve even better performance with SAM2 data. It would be better if the structure could be explored.

This is a valuable discussion point. The Cutie model is specifically designed for the much narrower semi-supervised VOS task for videos, whereas the SAM 2 model is intended for broader capabilities and is capable of interactive promptable segmentation on images and videos.

That said, we anticipate that using our SA-V dataset to train VOS models (including Cutie) should yield significant improvements.

Following your suggestion, we conducted an experiment to evaluate this.

We implemented a custom SA-V dataloader in the Cutie codebase to train it on our SA-V dataset. After training on SA-V, Cutie achieves 69.0 J&F on SA-V-test compared to 62.8 J&F for the official Cutie checkpoint .

This 6.2-point boost shows the impact of our large-scale SA-V dataset in enabling the "track anything" capability, as evaluated by the SA-V benchmark. Although Cutie's performance on SA-V-test greatly benefits from this SA-V training data, it still falls short of the 76.0 J&F achieved by SAM 2 (Table 17a).

We observed that purely training on SA-V degrades the performance of Cutie on benchmarks that have their own in-domain training sets, MOSE and DAVIS, which become zero-shot if not training on these. We are currently experimenting with jointly training Cutie on our SA-V dataset as well as previous open-source datasets and will report the results as soon as they come available and will report them in the final version of the paper.

SAM2 cannot recognize segmented objects like previous models [R2, R3]. It would be better to discuss this since it may limit the application of this paper. It would also be better to discuss the difference with [R4], which supports image and video segmentation and can recognize the objects.

Thank you for the pointers! While these are indeed interesting works, our focus is on the promptable visual segmentation task (PVS), and therefore, recognition is beyond the scope of this current work.

However, the progress we have made on the PVS task can certainly facilitate significant advances in recognition! Indeed, R2, R3, and R4 were built upon SAM 1 (or the SA-1B dataset) and delivered advances in recognition. We believe that SAM 2 can enable similar advances for (images and) video.

Please note that among these works (R2, R3, and R4), only R4 has video capability, and SAM 2 significantly outperforms it on the VOS task (e.g., [R4] has only 76.9 J&F on DAVIS 2017 vs 90.9 J&F of SAM 2). As such, these works are complementary to SAM 2, and we will discuss them in the paper.

We would like to thank the reviewer for their insightful review. Please let us know if there are any other questions or points you would like us to address, and we will be happy to do so.

Thanks for the authors' comments on the weaknesses I previously mentioned. I would like to keep the score at 8, and I also look forward to the experiments mentioned by the authors in their response. It is a good paper.

All the experiments mentioned in our first response have finished now. Following your suggestion, we aim to evaluate if the SA-V data can also benefit existing models, beyond SAM 2. To evaluate this, we use SA-V to train a recent semi-supervised VOS model, Cutie [1], which SA-V was not collected for, to assess SA-V's generalization capabilities. For this, we implemented a custom dataloader to jointly train Cutie on DAVIS, YouTubeVOS, MOSE and our SA-V dataset.

To assess the generality of SA-V, we compare the performance of Cutie models trained with and without it, both for the existing VOS task with mask or box prompt on the first frame, and interactive video segmentation, identical to the setup in §6.2. We report results in the following table.

Table: Our SA-V data has a broader utility beyond SAM 2 and helps zero-shot generalization in other models. Including SA-V in the training mix improves the performance of Cutie on SA-V test and provides zero-shot gains on LVOS and 17 zero-shot datasets. The final performance of Cutie + SA-V still falls short of the full SAM 2 shown in the last row.

As a baseline, the first row of the table shows the J&F accuracy of the official Cutie model trained on DAVIS, YouTubeVOS, and MOSE (referred to as "w/ MOSE" in [1]) as evaluated by us on SA-V test and our zero-shot benchmark of 17 datasets (using 5 clicks or bounding box as input to SAM+Cutie or ground-truth mask input to Cutie; see §F.1.3 for details). Performance on MOSE, DAVIS, and LVOS benchmarks is from officially reported numbers.

In the second row of the table, we train another Cutie model on our data mixture consisting of SA-V, DAVIS, YouTube-VOS, and MOSE. We use the same data mixture ratio as for SAM 2 and follow the official Cutie training recipe.

Including SA-V in the training mix brings a large performance improvement on the SA-V benchmark. Although integrating SA-V into the training mix does not improve the performance on MOSE and DAVIS in-domain benchmarks, it boosts the zero-shot performance as shown by gains of +1.5 on LVOS and +2.0 on our 17 zero-shot benchmark suite under mask input, and similar boost for the out-of-domain click and box prompts.

Finally, the last row shows the accuracy of SAM 2. Note that SAM 2 significantly outperforms the Cutie models under both training settings, suggesting that the SAM 2 architecture design is another crucial aspect of the final performance.

Thank you for your great suggestion, this really makes the paper stronger and shows the generalization of the data beyond the SAM 2 model architecture.

[1] Putting the Object Back into Video Object Segmentation, Cheng et al, CVPR 2024

Dear reviewers, we have updated the paper draft to incorporate these SAM 2.1 enhancements, along with the updates provided in the rebuttal.