Context-Aware Video Instance Segmentation

Our approach leverages surrounding context to improve object tracking, particularly in heavily occluded scenarios. For example, in [this figure], consider the "purple elephant'' in the leftmost part of the first frame. Pixels near its boundary overlap with the "light green elephant's rear.'' In the second frame, similar "light green elephant rear'' pixels remain nearby, enabling an easy match. In the third frame, pixels around the "purple elephant" are surrounded by pixels from the "light green elephant's rear and legs." This context persists across subsequent frames, enabling precise tracking using this information. In contrast, traditional methods like MinVIS and CTVIS, which rely solely on instance features, often fail under heavy occlusion. As shown in Table 1 of the main text, our method outperforms the conventional methods by a large margin, especially on the OVIS dataset. This difference is evident in the t-SNE visualization (see Fig. 4 in Appendix), where our context-aware modeling robustly distinguishes objects even in challenging occlusion scenarios.

Attention map. We appreciate your suggestion to analyze the surrounding features through attention map visualization. However, these features are incorporated within the attention operations of CAIT, where an N$\times$N attention map is computed to represent object-matching relationships. Visualizing this map does not directly clarify the role of surrounding features. Instead, we have provided detailed analyses of model predictions and t-SNE visualizations (Fig 4, Appendix), which more effectively demonstrate the impact and benefits of our surrounding feature modeling.

VISAGE, proposed by Kim, Hanjung, et al., employs appearance features for object matching, specifically relying on low-level feature maps extracted from the backbone encoder. In contrast, our approach utilizes high-level feature maps derived from a pixel decoder, ensuring temporal consistency in the semantic features of the same object across a video. Moreover, these features are not utilized during inference in our method. Given the distinct motivations and differences in training and testing methodologies, our approach cannot be regarded as a simple re-implementation of VISAGE. Furthermore, our method demonstrates superior performance compared to VISAGE on the OVIS, YTVIS19, and YTVIS22 benchmarks as shown in Table 9. To provide additional context, we will include a comparison with VISAGE, along with the accompanying table, in the final version.

Table 9. VISAGE vs Ours

The concern that context-aware features, which include instance features, could be used as the value in context-aware matching is understandable. However, during segmenter training, the instance features specifically drive the segmentation prediction. Therefore, it is more effective to use instance features as the value for matching. Our experimental results also confirm that using instance features leads to better performance, as shown in Table 16. Thank you for highlighting this potential confusion. We will clarify this reasoning and present these findings in the final version.

Table 16. Ablation study on value feature

We have clarified the previously confusing notations in Table 13. This clarification will be reflected in the final version.

Table 13. Confusing notations

Videos often contain objects of varying sizes, and for smaller objects, using an excessively large context area can introduce noise, leading to inaccurate matching as shown in [this figure]. To better understand this effect, we analyzed the impact of different filter sizes to identify the optimal value. Our findings indicate that the overall trend remains consistent, regardless of variations in the number of frames used during training or changes in the backbone architecture, as shown in Table 14-15.

Table 14. ablation study on filter size with R50 backbone

Table 15. ablation study on filter size with Swin-L backbone

When conducting the ablation study under the same setup as the main experiments, we observed similar trends. In the initial version of the paper, we presented results from a tracker model trained with fewer iterations (40k) to simplify reproducibility. To address potential confusion, we now report the ablation study results for the model trained with the full number of iterations (160k) in Table 10-12. In the final version, we will revise the structure to include the ablation study results from the main experimental setup in the main section, while relocating the minimal setup results to the appendix for reference.

Table 10. Detailed ablation study

Table 11. Ablation study on filter type

Table 12. Ablation study on context alignment

Thank you for pointing out the aspects we overlooked. We have uploaded a revised version of the paper, addressing your comments, including the response to your latest remarks on the comparison with VISAGE. Below is our detailed reply:

(1) Difference

VISAGE applies contrastive loss to low-level features extracted from the backbone and utilizes these features during inference. In contrast, our method applies contrastive loss to feature maps from the pixel decoder, targeting semantic-level representation learning. We conducted additional experiments with the same experimental setup of our ablation study (Table 4-(a) in our revised main text) to evaluate the effectiveness of VISAGE's loss in representation learning. As shown in Table 17., the appearance-level loss results in a minimal improvement of +0.5 AP over the baseline, whereas our method achieves a performance gain of +1.7 AP over the baseline. This demonstrates that our PCC loss is significantly more effective for representation learning. Therefore, VISAGE’s contrastive loss and our PCC loss differ fundamentally in both purpose and function. This distinction has been clarified in the Introduction section and the Appendix of our revised paper.

Table 17. Appearance loss vs PCC loss without CL

(2) Performance

We apologize for any confusion caused by Table 10. The performance in the 5th row (35.3) corresponds to object matching using instance features ($\hat{Q}$), while the 6th row (37.6) reflects matching with context-aware instance features ($Q$). To avoid confusion, we have revised the main text to explicitly indicate whether context-aware matching is used (see Table 4-(a) in the main text). Additionally, the 4th row of Table 10 shows that our method achieves 37.2 AP without relying on PCC loss, which is 1.0 AP higher than VISAGE. This result highlights the superiority of our context-aware modeling over the VISAGE approach.

1. Incorrect response

Thank you for pointing out our incorrect response to Reviewer rJtA's comments. Following your feedback, we have explicitly addressed the updates made in response to Reviewer rJtA's comments.

2. Distinctiveness of PCC loss

To clearly demonstrate the distinctiveness of the PCC loss, we conducted additional experiments and revised Sections 4.2 of the main paper and A.3 of the appendix. Our response is as follows:

In VIS methods, object feature representation learning is crucial because tracking relies on matching object features across frames. Existing approaches have primarily addressed this by employing contrastive learning between object features ($\hat{Q}$). These object features predict masks based on the similarity measure with the feature map extracted from the pixel decoder. In other words, object features and the pixel decoder's feature map are semantically intertwined. Consequently, ensuring pixel-level consistency across feature maps between frames can further support the learning of robust object representations.

Experimental results in Table 19 demonstrate that while the appearance loss shows minimal improvement, the PCC loss exhibits strong synergy with contrastive learning. (Table 17 highlights the effectiveness of the PCC loss in representation learning even without contrastive learning, further validating its unique contribution to improving object feature representations.)

Table 19. Synergy with the standard contrastive learning

We have clarified the previously confusing notations in Table 4. and will explicitly denote that the average filter is applied to the height and width dimensions. This clarification will be reflected in the final version.

Table 4. Confusing notations

An example of a model that utilizes full context is CAROQ, proposed by Choudhuri et al. in CVPR 2023. CAROQ defines the context feature as a memory bank of multi-level image features extracted by a pixel decoder (see Sec. 3.4 in CAROQ). This method uses features from all frames, which increases computational complexity and can result in lower performance. Moreover, in practical applications, CAROQ may encounter memory limitations due to its reliance on full-context features.

Interestingly, such a full-context-dependent approach performs similarly to or worse than simpler models like MinVIS, which rely solely on instance features as shown in Table 5. In contrast, our method integrates surrounding features directly into the instance features, enabling efficient and effective matching. As illustrated in Figure 4 of the appendix, this strategy significantly enhances the distinction between instances within the video.

Table 5. CAROQ (full-context) vs other models

We provide a detailed performance analysis by reporting the AP in Table 6. for both the segmenter and tracker when each component is applied, focusing on the impact of the PCC loss and CAIT components. Following the suggestion of Reviewer Fndi, all experiments were conducted under the same setup as the main experiments: training the segmenter with 3 frames over 40k iterations and training the tracker with 5 frames over 160k iterations.

Table 6. Detailed ablation study

Aligning with the recent trend in VIS methodologies prioritizing tracking accuracy, our approach also focuses on enhancing tracking performance over segmentation accuracy. To achieve better object matching, we introduce instance surrounding features, providing a more effective matching method. We provide an additional comparison of VIS results and highlight the moments in the video where severe occlusion happens. (See [this figure]) Our model accurately tracks objects during occlusion by utilizing the surrounding context. In contrast, instance-centric models struggle with severe occlusion, often failing to track objects accurately.

We compared the inference speed of our approach against recent state-of-the-art methods, GenVIS and DVIS, to evaluate the computational cost. The inference speeds were measured under identical conditions on a 2080ti GPU. As shown in Table 7., our method requires an additional time cost of 5.5ms and 6.7ms compared to GenVIS and DVIS, respectively. However, this cost is justified by the performance gains of +5.7AP and +4.5AP, demonstrating a reasonable trade-off between increased computation and improved accuracy.

Table 7. Trade-off between inference speed and performance

Our approach achieves superior performance compared to existing methods on YTVIS22, a well-known dataset featuring long video sequences as shown in Table 8. While long-term consistency was not the primary focus of our method, we will include these results as a reference in the final version.

Table 8. Comparison on YTVIS22 dataset with R50 backbone.

Please note that our method does not rely solely on context. By incorporating both context and instance features, our approach shows robustness even in scenes containing fast-moving objects where context changes rapidly. (See [this figure])

Dear Reviewer 5Yhw,

Thank you for taking the time to review our work and for providing insightful comments and feedback. Your input has been instrumental in helping us refine our research.

We have carefully addressed the concerns you raised and included updated results and detailed responses for each point. We hope our revisions clarify the key issues you highlighted, but we are happy to engage in further discussion to ensure all concerns are thoroughly addressed. Please feel free to share any additional questions or areas where further clarification may be needed. Your feedback remains invaluable in strengthening our work.

Thank you again for your thoughtful review. We look forward to furthering this constructive discussion.

As suggested by Reviewer Fndi, we conducted additional experiments comparing the performance of max and median filters on the surrounding features. All experiments in Table 2. followed the same setup as the main experiments: training the segmenter with 3 frames over 40k iterations. The results show that the average filter consistently outperforms both max and median filters, further supporting its effectiveness in our approach.

Table 2. Further study on filter type.

Since YTVIS22 differs from YTVIS21 only in its test set, the results can be easily verified using the provided pretrained model. As shown in Table 3., our approach also achieves superior performance compared to existing methods on YTVIS22. While long-term consistency was not the primary focus of our method, we will include these results as a reference in the appendix of the final version.

Table 3. Comparison on YTVIS22 dataset with R50 backbone.

Dear Reviewer Ln3A,

Thank you for taking the time to review our work and for providing insightful comments and feedback. Your input has been instrumental in helping us refine our research.

We have carefully addressed the concerns you raised and included updated results and detailed responses for each point. We hope our revisions clarify the key issues you highlighted, but we are happy to engage in further discussion to ensure all concerns are thoroughly addressed. Please feel free to share any additional questions or areas where further clarification may be needed. Your feedback remains invaluable in strengthening our work.

Thank you again for your thoughtful review. We look forward to furthering this constructive discussion.

We apologize for any confusion regarding the concatenation operation in Eq. (7). To clarify, we concatenate the instance feature $\hat{Q}$ and surrounding features $\tilde{Q}$ along the channel dimension, resulting in an $N \times 2C$ representation. This is then processed by an MLP to map it back to $N \times C$. Importantly, the number of queries remains unchanged throughout this process. Any observed performance improvement is therefore not attributable to an increase in the number of queries but rather to the effective integration of instance and surrounding features. We will update the main text to ensure this distinction is clearly explained.

Our model is designed to improve tracking accuracy by achieving precise object matching across frames rather than focusing on segmentation performance. Consequently, if Mask2Former produces inaccurate segmentation results, performance may decrease. However, even in scenarios with imprecise mask predictions, our proposed context-aware modeling can robustly track objects, as demonstrated in [this figure].

We adhere to the evaluation criteria and dataset settings used in the most recent VIS works, including GenVIS, DVIS, CTVIS, TCOVIS, and VISAGE. Consistent with these approaches, our work focuses on Video Instance Segmentation (VIS), specifically enhancing tracking performance by leveraging prediction masks. As a result, we have not conducted comparisons on MOT datasets, as they fall outside the primary scope of our work.

1. Comparison to CAROQ.

CAROQ, proposed by Choudhuri et al., defines the context feature as a memory bank of multi-level image features extracted by a pixel decoder (see Sec. 3.4 in CAROQ). This approach uses features from all frames indiscriminately, which increases complexity and may lead to lower performance. Moreover, it can cause memory issues in practical applications due to memory limitations. In contrast, our method adopts a memory-efficient approach by focusing on the surrounding features of each object during tracking, enabling effective object matching between frames with reduced memory overhead. While both methods utilize the term "context", our definition and application of context differ fundamentally. Thus, CAROQ’s methodology is only tangentially related to our work.

To further clarify the distinction, the below table highlights the superior performance of our approach over CAROQ on multiple benchmarks, demonstrating the effectiveness of our method.

Table 1. CAROQ vs Ours.

2. PCC loss.

VIS methods, including IDOL, CTVIS, and DVIS++, typically apply contrastive loss between instance features to enhance temporal consistency. However, as far as we know, no distinct approach applies contrastive loss at the pixel level across frames. The ablation study comparing a traditional contrastive loss with our PCC loss is reported in Table 3-(a) of our main text. If you were referring to another closely related loss, it would be helpful if you could provide the relevant reference.

(Updated) Following Reviewer Fndi's comment, we have added an explanation in the revised version to clarify the differences between the loss used in VISAGE (ECCV'24) and the PCC loss.

Apologies for any potential confusion caused. We initially misunderstood and mistakenly responded under the assumption that there were no losses similar to the PCC loss. In the revised version, we have addressed the issue raised by Reviewer Fndi regarding the similarity between VISAGE and the PCC loss. The updated explanation is as follows:

VISAGE employs contrastive loss on appearance features extracted from feature maps of the backbone encoder, which are also utilized for object matching during inference. This approach specifically aims to achieve more accurate object matching using appearance features. In contrast, our proposed PCC loss operates on feature maps extracted from the pixel decoder, targeting representation learning at the semantic level. To investigate whether VISAGE's appearance-level contrastive learning also contributes to representation learning, we conducted additional experiments. Using the same baseline architecture and a basic loss function, we tested PCC loss and VISAGE’s appearance loss separately, as shown in the Table 18. The results indicate that our method achieves a +1.7 AP gain over the baseline even without contrastive learning between object features. When combined with contrastive learning, it demonstrates further synergy, achieving 28.9 AP. In contrast, the appearance-level loss results in marginal performance improvements, with gains of only +0.5 AP and +0.1 AP in both cases. These results highlight that the proposed PCC loss facilitates the learning of object feature representations, distinguishing it from existing losses.

Table 18. Appearance loss (VISAGE) vs PCC loss

Dear Reviewer rJtA,

Thank you for taking the time to review our work and for providing insightful comments and feedback. Your input has been instrumental in helping us refine our research.

We have carefully addressed the concerns you raised and included updated results and detailed responses for each point. We hope our revisions clarify the key issues you highlighted, but we are happy to engage in further discussion to ensure all concerns are thoroughly addressed. Please feel free to share any additional questions or areas where further clarification may be needed. Your feedback remains invaluable in strengthening our work.

Thank you again for your thoughtful review. We look forward to furthering this constructive discussion.

(1) Main Focus of VIS

Recent advancements in segmentation architectures have significantly improved the quality of segmentation masks. However, tracking objects in dynamic videos remains a challenging task. As a result, the VIS field has shifted its focus from refining segmentation quality to enhancing tracking accuracy. To address this, we developed a framework specifically designed for videos with frequent occlusions, aiming to achieve more reliable tracking.

(2) Evaluation on MOT

Our research builds on the latest VIS methods, particularly those employing Mask2Former, as shown in the code we submitted. Given the segmentation-oriented nature of this baseline, conducting experiments on MOT benchmarks within the limited discussion period proved challenging. However, based on your suggestion, we tested the effectiveness of our context-aware association strategy by incorporating it into MOTRv2 (CVPR’23), a state-of-the-art MOT model. We conducted experiments on the DanceTrack benchmark. Since this dataset does not provide segmentation masks, we generated edge maps for the entire image using Sobel filters (see [this figure]) and derived boundary maps for individual objects from the predicted bounding boxes. As shown in Table 19, our method outperforms the baseline on the benchmark. These results highlight the versatility of our approach, demonstrating its potential to be applied beyond video segmentation tasks like VIS and VPS, extending its effectiveness to MOT as well.

Table 19. Comparison on DanceTrack val

We have uploaded the revised paper based on the feedback from the discussion. We are truly grateful for the valuable comments provided by all the reviewers, which have significantly improved the quality of our paper. If there are any further suggestions or feedback during the remaining discussion, we will carefully consider them and make any necessary updates. We sincerely appreciate your efforts.

The revised sections are highlighted in Plum color and are as follows:

1. Comparison with Previous Works
   1-(1) We added an explanation of the differences between PCC loss and VISAGE in the introduction section and appendix. (rJtA, Fndi)
   1-(2) We included the differences with CAROQ in the related works section. (rJtA)

2. Method Section Organization
   2-(1) We added a table for the confusing notations. (5Yhw, Fndi)
   2-(2) We provided an explanation of the dimensions of the operations in Eq. (6) and (7). (rJtA, 5Yhw)

3. Updated Experimental Results and Analyses
   3-(1) We updated the performance of CAROQ and VISAGE in Table 2. (rJtA, Fndi)
   3-(2) We updated the ablation study on other filter types (Ln3A)
   3-(3) We updated the ablation study with the same experimental setup as the main table in Table 4, and moved the previous ablation study table to the appendix. (Fndi)
   3-(4) We added an ablation study for the values used in context-aware matching. (Fndi)
   3-(5) We added experiments on computational cost. (5Yhw)
   3-(6) We added the following analyses in the appendix:

   • Potential issues arising from inaccurate masks. (rJtA)
   • VIS results of the t-SNE figure on the video. (5Yhw)
   • Performance report on YouTube-VIS 2022. (Ln3A, 5Yhw)
   • VIS results on videos with fast-moving objects. (5Yhw)
   • Analysis of filter size. (Fndi)