SyncVIS: Synchronized Video Instance Segmentation

In the beginning, we want to thank you for the detailed, insightful, and constructive comments.

Novelty

To sum up, as other reviewers have mentioned, our method provides an architectural design that is intuitive and effective (Reviewer 3JCg) and innovative (Reviewer 2sLS), demonstrating a high level of versatility (Reviewer MsYp). Our SyncVIS explicitly introduces video-level query embeddings to synchronize video-level embeddings with frame-level query embeddings. The synchronized video-frame modeling paradigm promotes the mutual learning of frame- and video-level embeddings by selecting key information from both embeddings and updating gradually via the synchronous decoder structure.

DVIS++

In offline mode, our method is lower than DVIS++, but our best performance on ResNet-50 backbone is in online mode, which is higher than DVIS++, by adapting to CTVIS. As mentioned in our paper and credited by Reviewer MsYp, our method demonstrates a high level of versatility on many state-of-the-art VIS methods. DVIS++ is built on DVIS, which incorporates a denoising training strategy and contrastive learning paradigm as well as an improved DINOv2 backbone. These adjustments, however, are orthogonal to our improvements to the modeling of spatial-temporal representations as well as our optimization strategy.

In the referring tracker and the temporal refiner, with the addition of our video-level embeddings and our synchronous modeling structure, our method can still bring significant improvements (2.1 AP) by adapting to DVIS++ in offline mode.

Implementation details

As for the implementation details such as training steps, we list these parameters in the configs file in the link provided in the Appendix. For example for the Youtube-VIS 2019, as the batch size is set to 8, the max_iter is 140000, and we use AdamW optimizer with a base learning rate of 5e-4 and a weight decay of 0.05. The backbone multiplier is 0.1.

Computation resources

Most of our experiments are conducted on 4 A100 GPUs (80G), and on a cuda 11.1, PyTorch 3.9 environment. The training time is approximately 1.5 days when training with the Swin-L backbone.

Motion information in video queries

In the DETR-style architecture, when video queries are associated with features across time via the decoder, they can effectively model instance-level motion through the cascade structure. In Mask2Former-VIS, the use of video queries alone enables the capture of instance motion.

A similar finding has been reported in the Seqformer model, which notes that "a stand-alone instance query suffices for capturing a time sequence of instances in a video." However, as the number of input frames increases, relying solely on video queries becomes insufficient for simultaneously tracking the movement of all instances. Such limitations motivate us to propose such synchronous designs to address the shortcomings in motion modeling.

Ablation of $N_k$

The ablation study of N_k is provided in Table 8 and the related analysis starts in L.324. The ablation results reveal that when selecting top $N_{k}=10$ embeddings to aggregate, the model performance reaches its optimum. When $N_{k}$ gets larger than optimum, the redundant query features will dilute the original compact information. On the other hand, getting too small will lead to the insufficiency of the injected information.

Efficiency

The relevant results are provided in L.242-243. We list the model parameters and FPS of SeqFormer (220M/27.7), VITA (229M/22.8), and our SyncVIS (245M/22.1). Our model performs notably better with comparable model parameters and inference speed.

We appreciate the reviewer's thorough review and valuable input. Your feedback has been instrumental in helping us enhance the quality and clarity of our work.

Analysis of baseline method

The analysis of CTVIS and VITA are included in the Introduction and Related Works (L. 37, L. 104). We choose to analyze Mask2Former-VIS because it's a basic and representative VIS model that both VITA and CTVIS are built on, while VITA utilizes frame-level queries to segment each frame independently and then associate frame-level queries with video-level queries and CTVIS adopts the frame-level queries to build consistent contrastive items and memory banks. 1) VITA: Even though VITA adopts video-level queries and manages to associate them, we argue that this process is sensitive because the wellness of video-level queries heavily relies on the learning of frame-level queries. Since frame-level queries (decoded by frame features) lack global motion information, this asynchronous unidirectional aggregation will cause potential information loss to the output video-level queries. 2) CTVIS: CTVIS utilizes contrastive learning to strengthen the representations of frame-level embeddings and uses frame-level embeddings to maintain a memory bank to model temporal relations. However, it mainly focuses on building discriminative frame-level embeddings, and hardly models the long-term spatio-temporal object representation. That is, CTVIS lacks an explicit synchronous association between video- embeddings and frame-level embeddings. As proved in YoutubeVIS 2019 & 2021, SyncVIS achieves 57.9 and 51.9 in the online mode, while CTVIS is 55.1 and 50.1, which is lower than our results.

We will revise our paper in the final version.

Comparison with TCOVIS

In TCOVIS, the global instance assignment aims to match predictions from all frames in the video clip with GTs. It considers all frames as a whole in searching for the optimal objective. Our method, on the other hand, divides all video frames into smaller sub-clips for matching with GTs.

1. TCOVIS: By matching predictions across all frames, TCOVIS is capable of associating the video frames from the beginning to the end of a minute-long video in the Youtube-VIS 2022 validation dataset, but it may overlook the fine-grained local details that is crucial for distinguishing multiple similar instances. Thus, when handling multiple instances and movements in consecutive frames, TCOVIS fails to simultaneously track and segment many more instances in crowded and occluded scenes. On OVIS, TCOVIS is 46.7 while our SyncVIS is 50.8, outperforming by a large margin.

2. SyncVIS: By dividing video into several sub-clips, our optimization strategy aims to reduce the increasing optimization complexity as the input frame number grows (L. 188, L. 260) because our synchronous modeling paradigm can model temporal association better with more input frames. To realize this target, our strategy is to divide the video into several sub-clips that could make optimization easier while retaining the temporal motion information.

3. YoutubeVIS 2022: Our SyncVIS mainly focuses on solving challenging scenarios with multiple instances and occlusions, which is usually overlooked for previous asynchronous methods because of their query-sensitive designs. When handling minute-long videos, our video-level embeddings are insufficient for modeling the associations across such a huge amount of input frames. Therefore, the performance on Youtube-VIS 2022 is not the SOTA result.

   However, our SyncVIS manages to find a balance between modeling long-range video and keeping high performance on crowded and occluded video scenes in shorter lengths. Even though TCOVIS has better results on Youtube-VIS 2022 validation dataset, its performance on another challenging VIS dataset, OVIS, is 4.1 points below our SyncVIS. TCOVIS manages to model the extra-long videos in Youtube-VIS 2022 validation dataset, but it somehow neglects the much more common cases, which are combinations of long videos and complex scenes with more instances and occlusions.

We will revise this part to the final version of the paper.

Computation resources

Most of our experiments are conducted on 4 A100 GPUs (80G), and on a cuda 11.1, PyTorch 3.9 environment. The training time is approximately 1.5 days when training with the Swin-L backbone. We will specify this in the final version of the paper.

Optimization setting

Since the two datasets have different sizes, and that 2021 has more videos than 2019, we use different training iterations and learning rates for different datasets. in practice. We will revise our paper in the final version to state this difference (As for the image size, the number should be 448, and the learning rate of the Swin-Large backbone on YoutubeVIS 2019 is 5e-4, and we will correct this in the final version).

Improvement upon CTVIS

It's true that building a memory bank contributes to the temporal modeling of instances. However, the forward passing of CTVIS is still asynchronous. While a single frame produces the frame embedding, there is no explicit video-level embedding to interact with the frame-level instance embedding. In our design, we add a set of video-level embeddings that gradually update with the frame-level embeddings. The explicit video-level embedding can directly provide long-range information and temporal information to the frame-level embedding, which enhances the quality of Hungarian matching and the contrastive learning module afterward.

We are grateful to the reviewer for taking the time to provide such detailed and constructive criticism. Your suggestions have been invaluable in strengthening our paper.

Object motion modeling

• Synchronous design for robust modeling

Our proposed SyncVIS employs image encoding as well as video encoding in a synchronous manner to model object motion while compensating for the potential "motion information loss" in the "image encoding" stage. In our design, frame-level embeddings are assigned to each sampled frame, and responsible for modeling the appearance of instances, and video-level embeddings are a set of shared instance queries for all sampled frames, which are used to characterize the general motion (because they encode the position information of instances across frames, and thereby naturally contain the motion information).

SeqFormer has the observation that "a stand-alone instance query suffices for capturing a time sequence of instances in a video." It decomposes the decoder to be frame-independent while building communication between different frames using instance queries, which also aim to model motion across frames. Nevertheless it follows an asynchronous manner, which is less robust in modeling temporal associations than our synchronous design because the aggregation is unidirectional instead of a mutual enhancement and the motion loss in image encoding is not compensated. As proved in YoutubeVIS 2019 & 2021, SyncVIS achieves 54.2 and 48.9 in the offline mode, while Seqformer is 47.4 and 40.5, which is much lower than our results.

• Experiment

As shown in Table (also in the Table 6 of the main paper), by implementing a synchornous structure, SyncVIS outperforms the asynchronous unidirectional structure by 1.6 AP. This further proves that our design is more robust than the design in SeqFormer in modeling object motion.

Synchronized Embedding Optimization

In the synchronized embedding optimization, we aim to enhance bipartite matching between prediction and GTs. L194-L199, the illustrations of $t_i$ and $t_j$ are trying to explain how the optimization strategy works in a divide-and-conquer way and illustrate our motivation of dividing the whole training sample videos into sub-clips.

Setting of $T_s$

The results do not contradict the conclusion. In optimization strategy, our main goal is to reduce the increasing optimization complexity as the input frame number grows (L. 188, L. 260). To realize this target, our strategy is to divide the video into several sub-clips that could make optimization easier while retaining the temporal motion information. Longer Sub-clips could provide the model with more temporal information, but their optimization complexity also rises polynomially.

The other important factor is the dataset. OVIS, compared to YoutubeVIS 2019, has more occluded and crowded scenes (5.8 objects per video for OVIS while 1.7 for YoutubeVIS 2019). Thus, the complexity of OVIS for each frame is three times that of the YoutubeVIS 2019. In order to further reduce the complexity for better optimization, dividing into smaller sub-clips can accelerate the optimization. Also, keeping the size of two is able to maintain the temporal information.

Thanks to the author for the response. All my doubts have been answered in the author's rebuttal. I agree with the phenomena observed on OVIS and hope that there will be more detailed explanations about Synchronized Embedding Optimization in the revision.

In the beginning, we want to thank you for the detailed, insightful, and constructive comments.

Qualitative results

In the Appendix, we provide many illustrations comparing previous base models with our SyncVIS. We select some cases under different scenarios, which include settings with multiple similar instances, settings with reappearance of instance, settings with different poses of instance, and settings with long video (Fig. 4 - Fig. 7). In the Fig. 8 of the Appendix, we provide visualizations of implementing different designs of embeddings, which further illustrate our synchronous design is capable of better segmenting and tracking multiple similar instances, while the asynchronous design ends in ignoring some instances and segmenting objects incompletely.

Mask2Former-VIS on OVIS dataset performance

• Complexity of OVIS

OVIS is known for having more occluded and crowded scenes than the YoutubeVIS dataset (5.8 objects per video for OVIS while 1.7 for YoutubeVIS 2019), and thus poses more challenges to the model's temporal modeling capacity on multiple instances.

• Mask2Former-VIS

Mask2Former-VIS, on the other hand, is a typical offline VIS method that models the whole video sequence with only video-level embeddings, which is extremely insufficient for handling occluded and crowded scenes. Even if we increase the input frame number to bring more temporal information, the modeling is still harder for Mask2Former-VIS. Even though we experimented with different hyperparameter settings for OVIS scenario, the results are still unsatisfying due to the innate restrictions of Mask2Former-VIS. The following-up work of Mask2Former-VIS, such as Seqformer, also suffers from the typical asynchronous design of offline methods, and its performance is also below 15 AP.