Learning Spatial-Semantic Features for Robust Video Object Segmentation

We sincerely thank Reviewer eHC5 for reviewing this paper.

Q1: In Table 1, why is there no separate ablation study for the spatial block and semantic block?
   To save space, we place the detailed ablation study in Appendix Table 7. Table 7 shows that spatial dependence modeling significantly improves performance across all VOS datasets, with a 3.0%+ J&F gain on MOSE val. This improvement comes from better detail modeling during feature extraction. Adding semantic information further boosts performance, reaching 73% J&F on the LVOS test, demonstrating its value in enhancing target semantic understanding. We provide some experiments in the table below. For more details, please refer to the Appendix.

Q2: Some detail issues: In Figure 3, the blue results are not clearly marked and the position of * is not aligned.
   We have fixed these details in the revised version.

Q3: How many trainable parameters and total parameters does the model have?
   In our full version, the total parameters are 226.8 million and the trainable parameters are 54.1 million. The trainable parameters are mainly in the Spatial-Semantic Blocks, the target association module, and the decoder.

Q4: Why does "DepthAnything" achieve the best results?
   DepthAnything provides more powerful representation with pixel-level and depth information compared to other pretrained models, which enables efficient target feature representation across diverse scenarios. This is also validated on instance segmentation in the DepthAnything paper.

Q5: If the backbone of the Cutie model is replaced with ViT, would it achieve good results?
   The table below compares Cutie with different backbones and our proposed model. The results show that replacing Cutie's backbone with ViT improves performance slightly (+0.3% on YT19 and 0.2% on MOSE). This is because ViT provides stronger representation capabilities compared to ResNet. However, simply replacing the backbone with ViT fails to produce multi-scale and the associated semantic information. To address this issue, the proposed Spatial-Semantic Block jointly models semantic and multi-scale spatial information, achieving notable improvements across different datasets (+1.4% on YT19 and 4.5% on MOSE).

Dear Reviewer,

Thank you for the comments on our paper. We have provided a response and a revised paper on Openreview. Since the discussion phase ends on Nov 26, we would like to know whether we have addressed all the issues, and we look forward to resolving any additional questions or concerns you may have.

Thank you again for your time and effort.

Best regards

Q7: JointFormer trained on MEGA datasets.
   JointFormer has not released its source code, so we tried our best to reproduce the code based on the paper. Unfortunately, the reproduced results differ from those reported in the original paper, making it impossible for us to compare it with our method on the MEGA dataset. Although it was not possible to train JointFormer on the MEGA dataset, a comparison can still be made using models trained only on the YT+DAVIS dataset. Our model achieves better performance than JointFormer on the large-scale datasets YT19 and YT18 with gains of 1.4% and 1.3%, respectively.

Q8: What is the individual impact of directly fusing features versus applying Deformable Cross Attention for further enhancement in the spatial-semantic block?
   To save space, we place the detailed ablation study in Appendix Table 7. Table 7 shows that spatial dependence modeling significantly improves performance across all VOS datasets, with a 3.0%+ J&F gain on MOSE val. This improvement comes from better detail modeling during feature extraction. Adding semantic information further boosts performance, reaching 73% J&F on the LVOS test, demonstrating its value in enhancing target semantic understanding. We provide some experiments in the table below. For more details, please refer to the Appendix.

We sincerely thank Reviewer ZJce for reviewing this paper.

Q1: Writing
   We have rewritten the long sentences to shorter ones and polished the manuscript based on the suggestions to make it concise and clear, which can be seen in the revised version.

Q2: Model Details: N blocks
   Our model includes 4 spatial-semantic blocks (i.e. N=4), each interacting with the ViTb (12 layers) model every three ViT layers. This implementation detail is included in the Implementation Details section. As mentioned in the manuscript, we will release our source code and model to the public.

Q3: Argmax operation
   We use the Argmax operation to select the index with the highest similarity and do not perform back-propagation. The backpropagation of the relevant part is achieved through gradient flow via skip connections.

Q4: Will the performance on new datasets degrade when the ViT backbone is not fine-tuned?
   Our model was trained on the YT+DAVIS dataset and the MEGA dataset. Without any training on the LVOS dataset, it achieved SOTA results directly during testing (66.5% and 73.0%). This demonstrates that our model's performance does not degrade when transferred to new datasets.

   The designed SS block is specifically aimed at better-adapting features from upstream pretrained models to the VOS task. The semantic embedding module integrates semantic information, while spatial dependence modeling facilitates the interaction of multi-scale spatial features. Through this block, we effectively adapt a pretrained model to generate the multi-scale requirements of VOS tasks without the need for fine-tuning.

   In our ablation study (Table 1, Row 4), we directly replaced the feature extraction module with ViT and used FPN to extract multi-scale features. Although this experiment involved full fine-tuning of ViT, it did not significantly improve performance.

   We attempted to perform full fine-tuning of our complete model on NVIDIA A100 GPUs (40GB). However, the training encountered Out-Of-Memory errors, indicating that full fine-tuning requires more computational resources and training time.

Q5: When faced with more than 8 targets, can 8 queries adequately represent the different targets?
   The table below provides query numbers with additional values of 4, 10, and 64. The results show that:    (1) Increasing the number of queries does not consistently improve the model's performance. Using larger numbers of queries may degrade the performance slightly, as using too many queries will introduce more noise during training and slow down the convergence speed.

   (2) The model achieves the best performance when the number of queries is set to 8. This is primarily because the number of queries has a close relationship with the number of targets. The table below presents the target number statistics across different datasets, which are mostly concentrated between 1 and 5. Setting the query number to 8 effectively covers almost all sequences in the datasets, as verified by the experimental results.

Q6：Lack reference of the XMem in Table 3.    These two XMem settings represent the same method with different configurations, where the one marked with * is pretrained on static images. We have added the citation in the revised manuscript.

Dear Reviewer,

Thank you for the comments on our paper. We have provided a response and a revised paper on Openreview. Since the discussion phase ends on Nov 26, we would like to know whether we have addressed all the issues, and we look forward to resolving any additional questions or concerns you may have.

Thank you again for your time and effort.

Best regards

We sincerely thank Reviewer dBZh for reviewing this paper.

Q1: Missing ablation study about the number of frames used in the long-term memory bank.
   We present the experimental results of using different numbers of memorized frames in the following table. The experiments are conducted on the LVOS Val dataset, which shows greater sensitivity to the number of stored frames in the long-term memory.

   The table shows that the performance increases progressively with an increasing number of stored frames and the improvement saturates beyond a certain point. Considering the used Memory and run time also increase significantly, we set the Memory number as 20 (corresponding to 100 frames, storing one image every 5 frames), which achieves a good balance between accuracy and efficiency.

Q2: A wider range of query counts should be explored to more thoroughly validate the effectiveness of the proposed method.
   In the table below, we also provide query numbers with additional values of 4, 10, and 64. The results show that:

   (1) Increasing the number of queries does not consistently improve the model's performance. Using larger numbers of queries may degrade the performance slightly, as using too many queries will introduce more noise during training and slow down the convergence speed.

   (2) The model achieves the best performance when the number of queries is set to 8. This is primarily because the number of queries has a close relationship with the number of targets. The table below presents the target number statistics across different datasets, which are mostly concentrated between 1 and 5. Setting the query number to 8 effectively covers almost all sequences in the datasets, as verified by the experimental results.

Thank you for your response, which has basically resolved all my questions.

Dear Reviewer,

Thank you for the comments on our paper. We have provided a response and a revised paper on Openreview. Since the discussion phase ends on Nov 26, we would like to know whether we have addressed all the issues, and we look forward to resolving any additional questions or concerns you may have.

Thank you again for your time and effort.

Best regards

We sincerely thank Reviewer TGSj for reviewing this paper.

Q1: Deformable convolution or simple position encoding.
   Compared to a simple position encoding mechanism, the deformable cross-attention module provides fine-grained spatial details and local features (using dynamic offsets), which better supports the association with the semantic features (especially for objects with complex structures). Besides, sparse attention with linear complexity is more suitable for VOS than global attention with quadratic complexity.

   In the table below, we replace deformable cross-attention with traditional cross-attention. The results demonstrate that using deformable cross-attention significantly enhances the model's performance, especially in datasets with complex targets (+1.2% gains over the MOSE dataset).

Q2: Why does adding spatial cues suppress emphasis on the target instance while enhancing object instances with the same semantics?
   During the spatial dependence modeling process, deformable attention is employed to allow the model to focus more on detailed information, which inevitably enhances the features of objects of the same class. This will not affect the target association across frames, since the proposed method develops a discriminative query propagation module to distinguish targets in the target association phase. In summary, the spatial cues contribute to more accurate predictions and the discriminative query propagation module ensures correct target associations.

Q3： Advantage compared to sam2.
   Compared to SAM2, our approach has the following new designs.

   We introduce spatial-semantic information into VOS by explicitly modeling semantic and spatial information through the design of the SS Block, significantly enhancing the model's understanding of the target. Our model achieves comparable performance against SAM2 on various benchmarks, even when trained on a very small-scale dataset.

   To handle target association in long-term tracking scenarios, we propose a discriminative query mechanism, which contributes to the comparable performance against SAM2 (+7.9% in LVOSV2 and +1.1% in LVOS V1). The discriminative query generation effectively extracts and memorizes the salient feature points of the target, facilitating accurate identification and association during long-term tracking. The results on the LVOS dataset further validate this capability (83.7% on LSVOSv2-val and 73.0% on LVOS-test).

Dear Reviewer,

Thank you for the comments on our paper. We have provided a response and a revised paper on Openreview. Since the discussion phase ends on Nov 26, we would like to know whether we have addressed all the issues, and we look forward to resolving any additional questions or concerns you may have.

Thank you again for your time and effort.

Best regards