SAM2Long: Enhancing SAM2 for Long Video Segmentation with a Training-Free Memory Tree

We sincerely thank the reviewer for your insightful comments and recognition of this work, especially for acknowledging that:

1. The paper is easy to follow.
2. The approach of constructing multiple memory banks to mitigate error accumulation is logical, and it leads to improved performance in VOS.

Thanks again for your very constructive comments, which have helped us improve the paper quality significantly! Below we would like to provide point-to-point responses to all the raised questions and weaknesses:

Q1: Limited new insights.

A1: We respectfully disagree with this point. While improving the memory bank is indeed a key aspect of video tasks, the specific enhancement of the memory module in the state-of-the-art video model SAM 2 has not been previously explored. Our work is the first to propose an improvement to the memory mechanism in SAM 2. We introduce a tree structure to search at the video level during inference time, with the insight that such test-time computation can significantly enhance the performance of SAM 2.

Q2: Complexity and generalization..

A2: In terms of complexity, the increase in FLOPS compared to the baseline SAM2 is minimal, as detailed in the table below. We decompose the computation process into four key modules and report the results for SAM2-Large as a reference:

As shown, the increase in FLOPS is modest across different pathway configurations, and the parameter count remains unchanged.

Regarding generalization, our method has been rigorously tested across multiple benchmarks. We evaluated our approach on four different model sizes of SAM2 across six benchmarks. Notably, for short-term video datasets, our method shows no negative impact on performance. However, for long-term video object segmentation (VOS) benchmarks, such as SA-V and LVOS, we observe significant and consistent improvements.

Q3: Incomplete ablation study

A3: Thank you for your feedback. Could you please clarify which specific part of the strategy has not been verified? We are happy to supplement the ablation study in a later version based on your suggestions.

Q4: Limitations

A4: While our method shows promise, its performance is ultimately constrained by the capacity of SAM2, as we do not modify any learnable parameters within SAM2. This limitation presents an opportunity for future improvements. Additionally, we have not yet addressed multi-object scenarios. Exploring the semantic interactions between multiple objects within the same frame could provide valuable insights for achieving more accurate segmentation.

Please don’t hesitate to let us know if there are any additional clarifications or experiments that we can offer!

We sincerely thank the reviewer for your insightful comments and recognition of this work, especially for acknowledging that:

1. The proposed approach is conceptually sound.

2. SAM2Long demonstrates superior performance over the strong SAM 2 on standard VOS benchmarks, without requiring additional training.

3. The approach does not lead to a significant increase in computational costs.

Thanks again for your very constructive comments, which have helped us improve the paper quality significantly! Below we would like to provide point-to-point responses to all the raised questions and weaknesses:

Q1: Limited insights and readability of the method..

A1: We respectfully disagree with the reviewer’s assessment that our contribution lacks significant insights for the video object segmentation (VOS) community.

Our contribution is the novel design of a constrained tree memory structure, introduced during test-time inference rather than at training. This enhancement allows SAM 2 to dynamically explore multiple memory pathways in a resource-efficient manner, mitigating error propagation and improving robustness over long sequences.

Although heuristic methods have been applied in some contexts, the idea of leveraging test-time search via a memory tree to address challenges like error accumulation and long-term object tracking is unique to our work. The insight that this structure can significantly improve performance without additional training or external parameters adds value to the VOS community. Furthermore, our ability to manage memory diversity through occlusion-aware pruning and selective memory updates offers an advantage in handling difficult video scenarios—an area not addressed in prior SAM 2 work.

Thus, we believe our approach represents a meaningful research-driven enhancement to SAM 2’s memory system, contributing both theoretically and practically by addressing the limitations of existing methods for complex, long-term video object segmentation.

Regarding reliability, we acknowledge that SAM2Long inherits limitations from SAM 2, but it is important to recognize that no model can address every possible edge case. However, it is important to highlight that SAM 2 currently outperforms all prior methods on the SA-V and LVOS benchmarks. Our approach pushes these boundaries even further, yielding an average improvement of 3.8 points. This advancement is a clear demonstration of how we have enhanced the state-of-the-art model, representing a significant contribution to the field.

Q2: Overfitting to mask prompt.

A2: We respectfully disagree with the reviewer’s assessment. It seems there may be a misunderstanding of our method. The prompts (e.g., boxes, points, masks) and interactive mode mentioned by the reviewer are orthogonal to and independent of our contribution. Specifically, our approach does not alter the nature of how different types of prompts are processed. Whether they are point-based interactions or box-based inputs, all prompts first pass through a prompt encoder and mask decoder to generate masks. These masks are then stored in a memory bank.

Our contribution primarily focuses on improving the memory mechanism, which is compatible with a variety of prompt types, as seen in the original SAM2. The memory mechanism enables the model to retain predicted masks for more efficient and effective segmentation, allowing for enhanced performance when revisiting previous predictions. Our approach does not limit SAM2's versatility but instead enhances the handling of memory, which in turn improves the segmentation process.

Q3: Limitations and Analysis of Failure Cases.

A3: While our method demonstrates significant promise, its performance is inherently constrained by the capacity of SAM 2, as we do not modify any learnable parameters within SAM 2. This limitation provides an opportunity for future improvements. For instance, the model may still track the wrong object during segmentation if it remains occluded, as the uncertainty handling encourages storing all possible candidates in the memory frame selection. This introduces a trade-off between exploring potential solutions and avoiding incorrect associations, which can be addressed in future work to further enhance robustness.

Q4: No Analysis on long-term and occlusion-heavy scenarios.

A4: We encourage to take a look at Figure 1(b). There is a plot showing that our method has greater resilience to elapsed time compared to SAM 2, maintaining superior performance over longer periods.

We sincerely thank the reviewer for your insightful comments and recognition of this work, especially for acknowledging that:

1. The structure of constrained tree memory is a novel contribution.
2. The method is robust to real-world application.

Thanks again for your very constructive comments, which have helped us improve the paper quality significantly! Below we would like to provide point-to-point responses to all the raised weaknesses:

Q1: Unclear Source of Memory Bank Updates.

A1: The memory bank updates depend on the cumulative score from each branch. If the cumulative scores from a single branch are the highest among the available options, subsequent frames will continue using the memory from that branch. As illustrated in Figure 2, in the first frame update, both memory banks remain unchanged because the scores are all equal at 1.6. However, in later frames, memory bank 2 accumulates two higher mask scores (2.3 and 2.4). Consequently, we use the memory masks from this branch (shown in pink) as the memory for the current frame. As a result, we replace the original memory bank 1 (blue) with memory bank 2 (pink).

Q2: Mask Storage and Calculation Method.

A2: We handle memory storage separately for each target. Each target maintains its own individual memory bank. All threshold values are empirically tuned based on performance on the SA-V dataset. We observed minimal impact on performance with slight variations in threshold values. For instance, as shown in the full version of Table 6, there is little change in performance (measured by J&F) as $\delta_{\text{iou}}$ varies from 0 to 0.7:

Q3: Lack of Comparative Experiments.

A3: We did not include these datasets because they are relatively short in duration. For instance, MOSE has an average video length of only 12 seconds, and YouTube-VOS is even shorter. These shorter datasets make it challenging to demonstrate the long-term capability of our method. However, based on the new checkpoints released by SAM 2.1, we have evaluated the performance and report the results below:

Our results show no performance drop on these datasets; in fact, we observe a slight improvement on the MOSE benchmark. This indicates that our method does not negatively impact performance on any type of video and is especially effective for long-term settings.

Q4: Unreasonable Parameter Design in Ablation Experiments.

A4: Due to page limits, we couldn’t include the full table and instead chose a few representative results to convey key insights. Thank you for the great question. While we could achieve better results by performing a grid search to optimize parameters for each specific dataset, this approach is not practical for real-world applications. Therefore, we did not focus on finding an optimal result for any particular dataset; our design aims to show general trends. For instance, setting the confidence threshold too high would result in fewer frames being stored in memory, causing these frames to be too distant to effectively support predictions for the current frame. In our findings, IoU prediction and confidence are closely aligned with ground truth, demonstrating robust generalization due to the extensive training dataset included in SAM 2.

Please don’t hesitate to let us know if there are any additional clarifications or experiments that we can offer!

We sincerely thank the reviewer for your insightful comments and recognition of this work, especially for acknowledging that:

1. SAM2Long achieves strong results compared to baseline SAM 2.
2. An intriguing approach that addresses the challenges of segmentation and tracking in long videos.

Thanks again for your very constructive comments, which have helped us improve the paper quality significantly! Below we would like to provide point-to-point responses to all the raised questions and weaknesses:

Q1: Full evaluation of the computational efficiency.

A1: Thank you for your feedback. The number of parameters remains identical to SAM2 since no new parameters are introduced.

For FLOPS, we decompose the computation process into four key modules and report the results for SAM2-Large as a reference:

This detailed evaluation demonstrates that the additional overhead introduced by our method is minimal in terms of computational cost, while still yielding improved performance.

Q2: Ablation on Modulation.

A2: We intentionally did not modulate the features significantly, as we found that larger ranges, such as [0.85, 1.15], lead to a noticeable performance degradation, as shown in the table. We hypothesize that this is due to the dot-product attention mechanism, which is followed by a softmax operation. This softmax step amplifies the effects of any modulation, causing instability when the range is too large. As a result, we remain cautious in tuning the memory features to avoid performance loss.

Please don’t hesitate to let us know if there are any additional clarifications or experiments that we can offer!