Long-RVOS: A Comprehensive Benchmark for Long-term Referring Video Object Segmentation

Thank you for the constructive comments. We provide our responses as follows.

Q1. How do long descriptions, temporal references, or complex event sequences impact performance?

• Regarding long descriptions. We evaluate the impact of varying description lengths and present the results in Table R1. As description length increases, slight performance declines are observed across models. However, our ReferMo consistently outperforms existing methods across different description lengths.

Table R1. Overall J&F results for different description lengths.

• Regarding temporal references. In Lines 308-311, we demonstrated that most RVOS models exhibit a strong bias toward static attributes, with significantly weaker performance on Dynamic types (involving motion and temporal elements).

• Regarding complex event sequences. We categorized the samples into single-event, two-event, and multi-event groups based on the keywords (e.g., then, finally, ultimately) in descriptions. As shown in Table R2, performance across models declines as the event number increases, yet our ReferMo consistently outperforms existing methods. Also, these results highlight the significance of our long-term benchmark for evaluating the capabilities of RVOS models in understanding complex event sequences.

Table R2. Overall J&F results by event complexity.

Thanks for your suggestions and we will include them in our final version.

Q2. Why SOTA methods fail in Long-RVOS.

The reasons are multifaceted. We'd like to clarify that, in Introduction (Lines 37-46), we discussed the limitations of current SOTA methods in handling long-video scenarios, including their inefficiency in integrating long-term information and capturing fleeting cues, and the training-inference gap due to GPU limitation. Additionally, in Section 5.2 (Lines 295-317), we empirically demonstrated the insufficient vision-language understanding in recent SAM2-based models, the potential static bias, and ineffective long-term consistency. These discussions and analyses can provide clear insights and are far from simply "implying the lack of long-term modeling".

To further address your concern, we present results at various object occlusion rates in Table R3. As the occlusion rate increases, RVOS models exhibit significant performance declines, though the decline extent varies across models. In comparison, our ReferMo model remains robust in most cases (with occlusion rates ranging from 0 to 0.75) but struggles in extremely high-occlusion scenarios (>0.75), as it relies solely on the keyframes. These findings also highlight the significance of our benchmark, which includes much more occlusion scenarios than existing benchmarks, as demonstrated in Table 5 of the Appendix.

Table R3. Overall J&F results at various object occlusion rates.

Q3. Why not submit to dataset track.

We'd like to clarify that we chose the Main Conference track because, in addition to introducing a new dataset, our work establishes a promising and effective baseline method for long-term RVOS. We also confirmed that our submission completely complies with the NeurIPS rules. Furthermore, we will definitely release the dataset and code to enable community verification and use, as committed in the paper.

Q4. It's a good baseline, but it doesn't introduce new tricks or concepts.

We appreciate your recognition of our method's effectiveness for the new challenge. However, we'd argue that the novelty of our method lies not in inventing individual components (e.g., motion information and local-global structure), but in establishing a new paradigm for tackling long-term RVOS.

1. Our method decouples RVOS into keyframe object identification and non-keyframe mask propagation, enabling efficient training and inference on realistic long-form videos. This is meaningful and significantly different from existing methods that perform dense cross-modal reasoning on every video frame.

2. We propose introducing low-res motion frames to bridge the training-inference gap in temporal receptive fields. This solution has not been studied before and is far from straightforward. It can also provide insights for other video fields that require dense computations (e.g., video super-resolution and long-video generation).

Combining these elements, we believe the novelty of our method and its value to the field have been demonstrated.

Q5. The metrics are not newly invented.

We respectfully argue that while the two metrics are established in other video tasks, their first application to RVOS field is still novel and significant. As noted in Lines 47-53, current benchmarks focus mainly on spatial metrics and overlook the temporal/spatiotemporal evaluation. By introducing these metrics to RVOS, we address a long-standing gap in the field. Besides, our work consistently aims to advance the RVOS field towards more practical scenarios, rather than "inventing" some completely new metrics.

Dear Reviewer,

The authors’ rebuttal has been posted. Please check the author's feedback, evaluate how it addresses the concerns you raised, and post any follow-up questions to engage the authors in a discussion. Please do this ASAP.

Thanks.

AC

Thank you for your appreciation of our work and insightful suggestions. We provide our responses as follows.

Q1. The version of SAMWISE.

Thank you for pointing it out. SAMWISE is an impressive work in adapting SAM2 for RVOS. We used the commit 08eb4ed (Apr 7), which was the latest version before the NeurIPS deadline (May 15). Following your suggestions, we re-evaluate SAMWISE with the latest codebase (Jun 28) and achieve higher results, as shown in Table R1. Due to formatting limitations in markdown tables, more detailed results will be updated in the final version.

Please note that all results for SAMWISE in this rebuttal are evaluated with the latest version.

Table R1. J&F Results of SAMWISE with latest codebase.

Q2. How does ReferMo handle cases where the target object is absent in the keyframe?

We agree that this is a common challenge for keyframe-based methods. When the target object is absent in a keyframe, ReferMo is expected to predict an empty mask for SAM2, resulting in an empty mask sequence for that clip. To address your concern, we present the results under varying object occlusion rates in Table R2. Despite relying soly on keyframe reasoning, ReferMo demonstrates significant performance advantages in most occlusion cases (0 to 0.75). Even in extremely high-occlusion cases (>0.75), it still outperforms most existing methods, except for SAMWISE.

Therefore, although it depends solely on keyframes, ReferMo remains sufficiently robust in most non-extreme cases. We sincerely thank you for raising this point. We will include these results in our final version and make further efforts to address this challenge in future work.

Table R2. Overall J&F results at various object occlusion rates.

Q3. Potential applicability in streaming scenarios.

Thanks for your suggestions. Extending RVOS to streaming scenarios is indeed valuable. Intuitively, ReferMo can be adapted to streaming videos by replacing global temporal attention with causal attention. However, an additional post-correction mechanism is also crucial for object correction. We will add further discussions (including comparisons with streaming models like SAMWISE) in the final version.

Q4. Ablation with other propagation strategies.

We follow your suggestion to replace SAM2 with other propagation models (e.g., Xmem++[R1] and Cuite[R2]) in Table R3, which shows that SAM2 contributes 1.1-2.5% J&F gains to overall performance. Notably, even when combined with these traditional propagation models, our ReferMo still outperforms existing SAM2-based RVOS methods, validating the robustness of our approach.

Table R3. J&F comparison of ReferMo variants and baselines.

[R1] Bekuzarov, Maksym, et al. "Xmem++: Production-level video segmentation from few annotated frames." ICCV, 2023.

[R2] Cheng, Ho Kei, et al. "Putting the object back into video object segmentation." CVPR, 2024.

Thanks for your thoughtful and encouraging comments. We provide our repsonses as follows.

Q1. Encourage extending the dataset with reasoning referring expressions.

Thanks for your suggestions. We fully agree that reasoning-based RVOS is a promising direction, as discussed in our Limitation section. In this work, we chose to begin with description-based RVOS because it is commonly used in current video applications and this task remains far from being solved. We will follow your suggestions to further extend Long-RVOS to reasoning scenarios in future versions.

Q2. How are short clips matched with full-length referring expressions?

This is handled implicitly through the attention mechanism. Specifically, our model utilizes cross-attention to aggregate the vision and text features. During this process, text tokens irrelevant to the clip's content naturally receive lower attention weights, thus preventing the model from being significantly misled by the irrelevant text segments. Take Figure 5 as an example, the normalized attention weights in the 4-th encoder layer for "boxing moves" rise from 0.04 to 0.26 when this action occurs.

Q3. Why do SAM2-based methods underperform on Long-RVOS?

While the recent SAM2-based methods excel in tracking and segmentation, they are typically limited in vision-language understanding, as discussed in Lines 298-307. In long videos with numerous distractors and events, it becomes challenging to accurately identify the referred target. Once the target object is misidentified, SAM2 will propagate the error throughout the entire video sequence, thus significantly degrading the performance. In contrast, our ReferMo identifies targets at keyframes as anchors and applies SAM2 within individual clips, effectively mitigating the potential error propagation and enhancing robustness in long-video scenarios.

Q4. Average annotation time and inter-annotator agreement.

We used a rigorous multi-round workflow with separate teams for annotation and validation. The annotation time varies by case complexity: normal cases take 3-5 minutes per object, while hard cases take 8-12 minutes. The validation team consists of four experts. If an annotation is rejected by at least one validation expert, it will be returned back to the annotation team for re-annotation. Thank you for pointing it out. We will include these details in the final version.

Dear Reviewer,

The authors’ rebuttal has been posted. Please check the author's feedback, evaluate how it addresses the concerns you raised, and post any follow-up questions to engage the authors in a discussion. Please do this ASAP.

Thanks.

AC

Thank you very much for your recognition and positive feedback! We truly appreciate your valuable input throughout the review process. We will incorporate all the discussions and suggestions from the rebuttal into our final version.

Thank you for your constructive feedback and appreciation of our work. We provide our repsonses as follows.

Q1. Mask annotation quality.

Thanks for your valuable feedback. We fully agree that ensuring high-quality masks is important, and we've made many efforts to achieve this goal. As detailed in Lines 161-171, we developed an interactive mask annotation tool and conducted an iterative check-correction workflow with human annotators to minimize errors. Despite these efforts, given the significantly longer video duration in Long-RVOS (60.3s vs. 13.8s in SA-V), it remains very challenging to ensure perfect masks across every frame. We respectfully believe that brief imperfect masks spanning 2-3 frames within a 60-second video (with 300+ frames) typically do not have a significant impact on training and evaluation. We will follow your suggestions to further refine the annotations and scale up the dataset in future versions.

Q2. Why ReferMo outperforms ReferDINO in the Static category.

ReferDINO often fails on object-invisible frames because it tends to identify and segment the objects most relevant to the text in every frame. Instead, our ReferMo decouples the task into object identification in keyframes and mask propagation within clips via a pretrained tracker, leading to better robustness in long-term scenarios. Please see Figure 7 in Appendix for their qualitative comparison.

Dear Reviewer,

The authors’ rebuttal has been posted. Please check the author's feedback, evaluate how it addresses the concerns you raised, and post any follow-up questions to engage the authors in a discussion. Please do this ASAP.

Thanks.

AC