ReferEverything: Towards segmenting everything we can speak of in videos

Thank you for your review and valuable feedback. Please refer to our general response for a discussion of the computational requirements of our method (in short, the computational requirements of REM are on par with other approaches in the literature). We address the rest of your comments below.

W1: REM doesn’t tailor video diffusion models to the task of RVOS
Thank you for bringing up this important topic. Changing as little of the video diffusion model as possible is the core idea of our work and the reason behind the top performance in out-of-domain evaluation achieved by REM, not its limitation. This is discussed in detail in Section 3.2 and validated through ablation analysis in Section 5.3.

Notably, VDIT [Zhu et al., ECCV’24] uses the same pre-trained video diffusion model as REM, but “tailors it to the task of RVOS”. It significantly underperforms compared to our method on all datasets precisely because of the modifications to the model architecture that it introduces.

Finally, the simplicity of our method is highlighted as one of its main strengths by reviewers NxrS and bXbA.

W2: Presentation requires further improvement
We welcome concrete suggestions on how to refine the manuscript further and will incorporate any concrete feedback provided. However, we would like to note that reviewers NxrS, bXbA, and hMhL have commended the clarity and readability of our paper.

Q1: Referral vs Referring
We thank the reviewer for pointing out this discrepancy and have updated the manuscript accordingly.

Dear Reviewer BAgM,

Thank you for your time and effort in reviewing our manuscript. We have carefully addressed all of your comments in our previous response. If our revisions and clarifications have resolved your concerns, we kindly ask that you acknowledge this and update your rating. Should you have any additional questions or require further clarification, we would be more than happy to address them. If additional experiments or results are needed, we kindly ask for sufficient time to ensure a thorough response.

Thank you for your review and suggestions. An ablation analysis of our method is reported in Section 5.3 in the manuscript. This includes a variant of the ablation suggested here, where we replace the frozen VAE with a CNN mask decoder from [Zhao et al., ICCV’23] (second to last row in Table 4). This variant performs significantly worse than our default version, especially on the out-of-domain Ref-VPS benchmark. This result supports our claim that preserving the representation of a pre-trained video generation model is crucial for achieving strong generalization in referring video segmentation.

Following your suggestion, we have evaluated a variant with an MLP mask decoder adopted from the popular SegFormer model [Xie et al., NeurIPS’21]. The results are reported below and added to Table 4 in the manuscript. Consistent with our findings for the CNN decoder, this variant also underperforms compared to the default configuration with a frozen VAE decoder.

We hope this additional analysis addresses your concern. If you have any further suggestions for ablations, we would be happy to explore them.

Dear Reviewer hMhL,

Thank you for your time and effort in reviewing our manuscript. We have carefully addressed all of your comments in our previous response. If our revisions and clarifications have resolved your concerns, we kindly ask that you acknowledge this and update your rating. Should you have any additional questions or require further clarification, we would be more than happy to address them. If additional experiments or results are needed, we kindly ask for sufficient time to ensure a thorough response.

Thank you for your detailed review and suggestions. Please refer to our general response for a discussion of the computational requirements of our method (in short, the computational requirements of REM are on par with other approaches in the literature). We address the rest of your comments below.

W1: Boundaries of dynamic concepts
Firstly, we would like to clarify that Figure 1 shows REM predictions, not ground-truth. We have updated the caption of the figure to make this clear. Secondly, we fully acknowledge that segmenting dynamic concepts like smoke or waves introduces inherent challenges due to ambiguous boundaries. To address this, Ref-VPS incorporates Ignore masks in the annotations, which mark regions of uncertainty that are excluded during quantitative evaluations. This approach ensures that the metrics focus on evaluating the most reliable regions of the masks, avoiding arbitrary penalties for ambiguous boundaries. We have added examples of our ground-truth annotations, including Ignore masks in Figure B in the manuscript together with a corresponding discussion. We appreciate the reviewer highlighting this critical aspect, and we have ensured that the manuscript includes detailed descriptions and examples of our evaluation strategy.

Q1: Noise level for the diffusion model
This is a great point! To preserve input fidelity while leveraging the benefits of pre-training, we do in fact set the noise level to 0, as mentioned in lines 272–275 of the manuscript. This ensures that the input images retain as much information as possible without being corrupted. Following the reviewer's suggestion, we have further highlighted this in the implementation details section (line 1117).

Q2: How many frames does the model process at once?
Thank you for pointing out this missing detail. We set the number of frames to 72 at inference time, a value determined by GPU memory constraints. We have updated the implementation details section in the manuscript accordingly (lines 1124-1125).

Q3: Typos
We thank the reviewer for the detailed feedback and have updated the manuscript accordingly.

Dear Reviewer NxrS,

Thank you for your time and effort in reviewing our manuscript. We have carefully addressed all of your comments in our previous response. If our revisions and clarifications have resolved your concerns, we kindly ask that you acknowledge this and update your rating. Should you have any additional questions or require further clarification, we would be more than happy to address them. If additional experiments or results are needed, we kindly ask for sufficient time to ensure a thorough response.

Thank you for your detailed review and suggestions. Please refer to our general response for a discussion of the computational requirements of our method during inference (in short, the computational requirements of REM are on par with other approaches in the literature). We address the rest of your comments below.

W1: Motivation of the paper
We are extending the task of Referring Video Segmentation (RVS) outside the domain of objects. As outlined in lines 36–44 of the paper, the RVOS community has focused on a limited subset of RVS, constrained to objects. Our work removes this artificial restriction, extending the task to include any spatio-temporally localizable entities in videos that can be described through natural language. Importantly, this extension does not alter the task itself but rather expands its applicability. Therefore, we believe the existing evaluation protocol remains adequate. However, we welcome any suggestions for additional evaluations that could enhance clarity or usefulness.

W2: Generalization to out-of-domain datasets other than Ref-VPS
Please note that we did evaluate REM on 2 existing out-of-domain datasets (BURST and VSPW) in the paper (see Table 2 and corresponding discussion in lines 405-419). Specifically, BURST is a large-scale video segmentation benchmark that focuses on object and scene diversity. It features 482 object categories, compared to 23 categories in YouTube-BoundingBoxes, and is labeled with segmentation masks. Our method outperforms the state-of-the-art methods on BURST by 9.5 points in terms of region similarity. In Figure E in the appendix we further investigate the performance of our method and the baselines on this dataset, demonstrating that our approach is indeed a lot more effective at capturing rare object categories in the tail of the distribution. These results provide strong evidence of REM's superior generalizability compared to prior approaches.

W3: Training costs
REM takes 174 hours to train on 4 A100 GPUs. Other methods do not report their training costs, so we have estimated them ourselves (excluding i/o time) given the same compute budget and report the results together with the memory consumption per GPU below, as well as in Table D in the manuscript:

Our costs are on par with most prior works. Notably, UNINEXT – the state-of-the-art RVOS approach, takes 6.3 times longer to train than REM, since it utilizes more than 10 datasets with object supervision to achieve top object segmentation results. In contrast, REM effectively capitalizes on Internet-scale pre-training, allowing it to achieve competitive performance with UNINEXT on traditional RVOS dataset and significantly outperform it out-of-domain. All at a fraction of the training cost.

Q1: What is the definition of a concept and how do you handle ambiguity?
We use the term concept to refer to spatio-temporal entities that are semantically meaningful but cannot always be defined with a single word (i.e. they are more abstract than classes). The term goes back at least to [Ghorbani et al, NeurIPS’19; Kim et al., ICML’18], but, to the best of our knowledge, no formal definition still exists. We have added a reference to the best available definition in line 99 of the paper.

Ambiguity is indeed an important aspect when annotating dynamic processes and we take it seriously. As mentioned in our response to reviewer NxrS above, and shown in Figure F in the manuscript, we have annotated all ambiguous regions with Ignore masks so that the evaluation is only performed on confident foreground and background pixels. Hence, the challenges of Ref-VPS are in fact due to generalizability, not due to the arbitrary nature of our annotations.

Q2: Meaning of “universal” in line 157
Yes, in this context we meant “universal” as not limited to one domain (e.g. not limited to common objects). We have clarified this in the manuscript to avoid confusion.

Q3: What are the advantages of visual representations learned with contrastive objectives?
These models indeed tend to be more lightweight (e.g. ViT used in CLIP vs the U-Net used by most diffusion models). We have updated the corresponding paragraph in Section 2 to mention this. Please note that, as discussed in the general response, the computational requirements of our model are on par with the state-of-the-art referring segmentation methods.

Q4: What is $t$ usually set to?
This depends heavily on the task and the architecture, but most works use low values of $t$ (between 0 and 100).

Q5: Meaning of “physical nature of the event” in line 314
For example, events like glaciers melting over time or soil erosion are not typically captured on TikTok due to the large temporal spans of such physical changes. We have added a clarifying text to the manuscript in lines 313-314.

Reference
[Ghorbani et al, NeurIPS’19] Towards Automatic Concept-based Explanations: https://arxiv.org/abs/1902.03129
[Kim et al., ICML’18] Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV): https://arxiv.org/abs/1902.03129

Q6: What architecture is REM built on top of?
We thank the reviewer for pointing out this missing detail. As described in our ablation in Section 5.3, we chose ModelScope T2V of Wang et al., as our base video diffusion architecture for all results in sections 5.1 and 5.2. Following the reviewer's comment, we have now explicitly mentioned this in the implementation details (lines 1116-1117).

Q7: Are the evaluations in Section 5.2 zero-shot?
As mentioned in lines 363-364 in the manuscript, all the evaluations in Section 5.2 are indeed 0-shot. Following the reviewer's comment, we have reiterated this at the beginning of Section 5.2.

Q8: Ablation with VAE decoder fine-tuning
We thank the reviewer for suggesting this insightful experiment. The results are reported below. The proposed variant with a fine-tuned VAE decoder performs worse than our proposed approach with a frozen VAE decoder, but better than a CNN decoder trained from scratch on our Ref-VPS benchmark. This result further supports our claim that preserving the representation of a pre-trained video-diffusion model is key for achieving strong generalization in referring video segmentation.

Q9: Typos and writing suggestions
We greatly appreciate the reviewer’s attention to detail and have implemented all suggested corrections to improve the manuscript’s readability.

Dear Reviewer bXbA,

Thank you for your time and effort in reviewing our manuscript. We have carefully addressed all of your comments in our previous response. If our revisions and clarifications have resolved your concerns, we kindly ask that you acknowledge this and update your rating. Should you have any additional questions or require further clarification, we would be more than happy to address them. If additional experiments or results are needed, we kindly ask for sufficient time to ensure a thorough response.

Dear Reviewer bXba,

Thank you for the constructive discussion and for updating the score. We sincerely appreciate it.

To clarify, while REM is trained only on segmentation data from Ref-COCO and Ref-YTB, it leverages a video diffusion backbone trained on Internet-scale data with a self-supervised, generative objective. This approach is consistent with methods like VDIT, which uses the same backbone. However, our method demonstrates significantly stronger generalization compared to VDIT, as shown in our experiments on Ref-VPS, BURST, and VSPW (relative improvement of up to 28%). These results highlight the effectiveness of our approach in unlocking the generalization potential of pre-trained video diffusion models.

Thank you for your detailed review and suggestions. Please refer to our general response for a discussion of the computational requirements of our method (in short, the computational requirements of REM are on par with other approaches in the literature). We address the rest of your comments below.

W1: Performance on Ref-YTB
We would like to emphasize that our model is not specifically designed for video segmentation and instead re-uses a video generation model’s architecture in its entirety. Despite this, it performs on par with specialized approaches like MUTR on datasets for which they were designed and heavily tuned. In particular, it clearly outperforms MUTR in terms of region similarity on both Ref-DAVIS and Ref-YTB. More importantly, it dramatically outperforms these approaches out of the domain (relative improvement of up to 51% with respect to MUTR), both on object-centric datasets like BURST and on non-object centric ones like Ref-VPS.

VLMO-L also capitalizes on Internet-scale pre-training, but our approach is more effective at doing so on RVOS datasets. Unfortunately, neither pre-trained models, nor training details are available for this (unpublished) method, so we cannot compare to it out-of-domain. That said, based on the fact that VLMO-L shares a similar architecture with VDIT, with the main difference between the two methods being that the former adopts a VLM backbone, whereas the latter is based on a video diffusion one, we can expect that VLMO-L would also struggle to generalize.

To sum up, our method offers unique out-of-domain generalization, while being highly competitive with specialized approaches in the narrow domain of traditional RVOS benchmarks. Moreover, the performance of REM is likely to further improve with the advancement of video generation models, as suggested by our analysis in Section 5.3.

W2: Impact of synthetic data
We would like to clarify that no synthetic data was used in training REM. Instead, as described in lines 355–357 of the manuscript, we follow prior work and train our model on real-world Ref-COCO and Ref-YTB datasets only. Please let us know if you have further questions regarding our training protocol.

W3: Temporal consistency evaluation
We thank the reviewer for this insightful suggestion. Unfortunately, accurately evaluating the temporal consistency of video segmentation methods is notoriously challenging, because it is hard to distinguish between predicted mask changes that are due to the method’s inconsistency and the changes that are due to the true target deformations. Notably, the temporal consistency metric proposed in the original DAVIS dataset was only applied to videos with no significant object deformations and no occlusions and was eventually phased out by the dataset’s authors.

Recognizing these limitations, we implemented a straightforward consistency metric by computing the average difference of IoU between the model's prediction and the ground truth mask in consecutive frames. Formally,

where $N$ is the number of samples and $T_n$ is the number of frames in the n-th sample. Lower numbers indicate better temporal consistency on this metric, and it’s easy to see that simply outputting empty masks would result in the perfect consistency score of 0. Hence, as with any temporal consistency metric, it should always be considered jointly with a prediction accuracy metric.

We report region similarity and temporal consistency on Ref-VPS and Ref-DAVIS (the two datasets that extract frames at 24 fps) below, as well as in Table B in the updated manuscript. The results demonstrate the superior temporal consistency of REM on both object-centric and non-object-centric datasets. Notably, UNINEXT - the state-of-the-art RVOS approach, shows the worst temporal consistency out of all methods. MUTR achieves a strong temporal consistency score on Ref-VPS precisely because it often outputs empty predictions, as can be seen from its low region similarity score.

Q1: How does REM handle ambiguous or overlapping concepts?
This is a great question! Our Ref-VPS indeed contains some ambiguous scenarios and we took great care in labeling them accurately (see our response to Reviewer NxrS below as well as the new Figure B in the manuscript). We have added a few examples of REM outputs on ambiguous or overlapping concepts in Figure F in the appendix and compared our method with VDIT - the strongest baseline on this benchmark. As you can see, although for many of these samples, no perfect prediction exists, the outputs of our model are both more accurate in the confident regions and more consistent. For example, in the first row, our method only segments the clearly visible regions of lava once it is hit by a wave, whereas VDIT segments the entire wave as well. In the second row, REM consistently segments all the glowing water, whereas VDIT only covers a few patches.

Dear Reviewer QTA2,

Thank you for your time and effort in reviewing our manuscript. We have carefully addressed all of your comments in our previous response. If our revisions and clarifications have resolved your concerns, we kindly ask that you acknowledge this and update your rating. Should you have any additional questions or require further clarification, we would be more than happy to address them. If additional experiments or results are needed, we kindly ask for sufficient time to ensure a thorough response.