SMITE: Segment Me In TimE

Thank you for appreciating the importance of our work and providing insightful feedback.

W1: "time and memory analysis" : We have included discussions on “Time and Memory Analysis” in Section 7.3. For longer video processing, please refer to our discussion in Section 7.5 of Appendix.

W2: "comparison with few-shot VOS" : Thanks for the suggestion ! Kindly refer to Section 5 and Tables 5 and 6 where we have already included comparisons of our approach in few-shot setting with existing baselines. In addition to this, we have now added “Comparison on DAVIS” in Section 7.3 of Appendix to include additional experiments on DAVIS dataset.

W3: "limited ablation study" : Thanks! We have now updated Table 2 with detailed network ablation on entire SMITE-50 dataset and provided related discussion in L455-463. It is to be noted, that the low-frequency regularization (LP Reg) helps maintain spatial structure and prevents segment drift when tracking is applied to $S_{WAS}$. Without tracking, LP Reg has minimal effect since no segment drift occurs in WAS maps (thus change of energy in $E_{reg}$ is almost zero), so we do not ablate LP Reg separately without tracking. Additionally, we have also included ablations in appendix Section 7.2.

Q1: "time and memory analysis and any frame limitation?" : Please refer to the general response above and the “Time and Memory Analysis” section. We do not have any frame processing limit as we split the video into chunks for longer videos. Please refer to Section 7.5 of Appendix for further details.

Q2: "no inflated U-Net during training" : Very good question! Using standard Unet during training and Inflated-Unet during testing will not impact our results. This is because our training is composed of single images and we process them as videos of length one. Thus, it never utilizes the temporal attention in the inflated UNet. Therefore, using inflated Unet or standard Unet during training makes no difference to our final results.

I thank the authors for the detailed reply and updates to the paper.

W1: "time and memory analysis" -- "We have included discussions on “Time and Memory Analysis” in Section 7.3" -- do you mean Section 7.5? This confirms my suspicion that the algorithm is slow but this is fine. How does this running time and memory usage compare to existing methods?

"However, it is possible to adjust settings to use only 15 GB, though this increases the inference time." -- by how much?

For long video processing (next paragraph) -- "This segmentation has proven not to compromise the final results" -- proven by whom?

I have no further questions on other aspects.

Thank you for appreciating the importance of our work and providing insightful feedback.

W1: "video extension of SLiMe (no fine-tuning) - SLiMe (single frame) + tracking" : Please see the general response above. As per your suggestion, we have now included two new baselines: a) Baseline-II is to use SLiMe and CoTracker simultaneously, and b) Baseline-III, which is the modification of SLiMe by only changing its UNet to the Inflated UNet. Our method outperforms both of these baselines in Table 1.

W2: "limited ablation study; input-object similarity analysis (e.g., horse → zebra, dog), low pass reg. step is dropped" : We have now included detailed network ablation in Table 2. In addition, we have included ablation studies in Appendix Section 7.2. Additionally, we have added a new discussion in Section 7.2 on “Cross-category generalization” and added a visual result in Figure 10 where our model trained on horses can successfully generalize to camels and giraffe which has large topological changes compared to horses. The low-frequency regularization (LP Reg) helps maintain spatial structure and prevents segment drift when tracking is applied to $S_{WAS}$. Without tracking, LP Reg has minimal effect since no segment drift occurs in WAS maps (thus change of energy in $E_{reg}$ is almost zero), so we do not ablate LP Reg separately without tracking. We have updated it in L458-463.

Q1: "figure 3 differences (c vs. d) unclear" : Kindly note that Figure 3 is a video when opened using Adobe Acrobat. In addition to this, we have provided a new video “Ablation Video” in the supplemental material to better show this.

Q2: "improve readability" : Thank you for pointing this out. We have now corrected based on your suggestions and also corrected lines L254-55 and L339. The loss terms used in Section 4.2 is used during training and in Section 4.4 is used during inference optimization. We have symbolically distinguished the training losses as $L$ and testing losses as $\varepsilon$.

Q3: "small technical note, $\epsilon_{\theta}$" : Thank you for your suggestions, we have now addressed it in the revised paper.

Q4: "why not average voting on masks as post-process? - why not apply average voting on $S_{WAS}$?" : Great question! Applying tracking and voting directly in the pixel space as post-processing, can lead to segmentation drift. This explains why Baseline II (SLiMe+CoTracker) which is added to the updated paper performs slightly worse than Baseline I. It is also visible in Figure 12 qualitatively. To address this, we have introduced a temporal voting mechanism on the attention maps enhanced with low-pass regularization within an optimization framework. Since this approach is quite effective, we did not test other possible voting scenarios that might be also promising. Hence we can treat it as one of our future work !

Q5: "justify using text embeddings for indescribable parts" : That is an excellent question! Yes, for segments which are hard to explain with text, we learn embeddings corresponding to the visual segments, since such asymmetric segment descriptions are outside the text vocabulary. In fact, a word may not exist for a part but a text vector embedding through an optimization may be found which is not a regular readable text. This is similar to Textual Inversion [1] or other personalization techniques, where a known word for a particular image may not exist but an optimized vector embedding can represent a particular image.

[1] Rinon Gal, et al., An image is worth one word: Personalizing text-toimage generation using textual inversion. arXiv 2022.

Thank you very much! We appreciate your comments and positive feedback.

Thank you for appreciating the importance of our work and providing insightful feedback.

W1: "limited comparison (add few-shot, zero-shot methods)" : Please refer to Section 5 and also Tables 5 and 6 where we had already included comparisons on PumaVOS dataset in few-shot setting with Xmem++. In addition, we have now added Tables 8 and 9 on DAVIS-2016 dataset. Please see the general response above and refer to the “Comparison on DAVIS” in Section 7.3 of Appendix.

W2: "time analysis" : We would like to clarify that we do not claim our method is capable of real-time segmentation in the paper. Please refer to the "Time and Memory Analysis" section in the Appendix of the paper and also the general response. Please also see our response to reviewer askr under Q1 for more details on time analysis.

W3: "limitations" : We already had some limitations included in Section 6 in the original submission but we now have expanded the limitations to better clarify our method's shortcomings.

W4: "random videos, random references?" : We do not use any particular method to hand pick or select videos or annotations. As long as references are of high quality with most of the necessary parts visible, our technique is able to perform reasonably well.

Thank you for appreciating the importance of our work and providing insightful feedback.

W1: "clarity issue" : Thanks for your input! For better clarification of our idea, we have now updated the text in lines L267-288 to discuss tracking integrates into the attention map and added lines L154-158 to explain how SLiMe works. We also added minor clarifications about the regularization terms in L344-347.

W2: "expand dataset or analyze generalization" : Thanks for your input! While SMITE-50 dataset is planned to be expanded into more classes in the future, it is a challenging dataset and we believe it has its own merits (e.g., number of segmentation parts per video) as compared to the existing video segmentation benchmarks like DAVIS and PumaVOS having more parts per video and many multi-part videos as shown in the table below.

For generalization to other categories, please refer to "Cross-category generalization" in Section 7.2 of Appendix. As per request, we have additionally tested our proposed SMITE model on DAVIS dataset in Section 7.3.

W3: "time analysis" : We would like to clarify that we do not claim our method is capable of real-time segmentation in the paper. We have now added "Time and Memory Analysis" section in the Appendix of the paper and also the general response.

Q1: "experiment with other tracking methods" : In our experiments, we have used the state-of-the-art tracking algorithm CoTracker2 [2]. In addition to this, we have tested on the recently released CoTracker3 [1] and found no significant performance differences. While our method can leverage the benefit of any available tracking algorithm, it will inherit its drawback to some extent. Our low-pass regularization helps partially correct tracker errors since it encourages to respect the semantic segmentation obtained from the diffusion model.

Q2: "explanation of low-pass regularization" : Since applying tracking directly (using a voting mechanism with the aid of a tracking method as a post-processing step), may suffer from gradual drift in segmentation, we added low-pass regularization. The low-frequency component captures the global structure of segments, helping to prevent segment drift. This regularization ensures consistent boundaries and maintains the overall segmentation structure during tracking (refer to Figure 8 for more clarity). As a result, we have two different energy functions: one pushes the prediction toward removing flicker and achieving more consistency across time (tracking), while the other pushes toward preserving correct semantics and avoiding any possible drift (low-pass regularization).

Q3: "rapid movements, occlusions, and lighting changes" : In fact, this is one of the advantages of our method. We have provided several examples to showcase these capabilities. Please see example videos featuring rapid movement (e.g., a running horse in 0:47 to 0:52 of Qualitative Comparison video in the supplementary) and occlusion (e.g., a hand occluding a face 0:15 to 0:21 of Qualitative Comparison video in the supplementary). Our tracking coupled with temporal voting, which accounts for occlusions, is capable of handling such scenarios. Since we use a Diffusion Model with strong semantic knowledge for images and videos under different conditions, our method performs robustly under different lighting conditions. For example, in a camouflage scenario (e.g., Figure 14), although we trained our model on usual turtle examples, our method still performs well in a environment with challenging lighting condition.

[1] Nikita Karaev, et al. CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos. arXiv,2024.

[2] Nikita Karaev, et al. Cotracker: It is better to track together. arXiv,2023.

I have read the authors' response and appreciate the clarifications provided. The authors have adequately addressed my concerns regarding the clarity of the methodology, providing a more detailed explanation of the tracking mechanism and its integration with the attention maps. The added information on SLiMe also helps improve overall understanding.

While the dataset size remains a concern, the authors' justification regarding the number of parts and multi-part focus is reasonable. The inclusion of cross-category generalization analysis in the Appendix and the added experiments on the DAVIS dataset further strengthen the paper. The response regarding inference time clarifies the scope of the work.

The authors have also adequately responded to my questions, particularly elaborating on the low-pass regularization and providing examples demonstrating robustness to rapid movements, occlusion, and lighting changes. The rationale for bidirectional tracking is well-explained and justified.

Overall, the revisions have improved the paper and addressed my concerns. I am raising my score.

Thank you for appreciating the importance of our work and providing insightful feedback.

W1: "ablation for joint vs. two phases": We would like to clarify that we do not perform two fully separate training stages. Initially, we optimize the text embeddings while the cross-attention components are frozen. Then, we shift the focus to optimizing the cross-attentions of the UNet. During this phase, we continue updating the text embeddings, but at a slower pace (every 10 steps). As such, the initial step serves more as a warm-up phase. Whereas, joint training refers to optimizing the text embeddings and cross attentions together from scratch.

Please refer to the “Training Ablation: Joint Training vs. Two-Phase Training” section in Appendix Section 7.2 of the updated paper. Table 7 demonstrates that two-phase training outperforms joint training, achieving 75.14 mIoU compared to 62.17 mIoU, due to better convergence. Additionally, two-phase training leads to a more stable training process with less loss oscillation during training (Figure 9).

W2: "smaller eye region ; Hard vs. soft voting" Thank you for pointing this out. We discovered a minor issue during the visualization of this specific category of results, particularly those shown at 1:46 of submitted video. It is common practice [1] to use a small Gaussian blur for improved visualization, but we unintentionally set it to a larger value, which affected small segments such as the 'eyes.' However, please note that this only impacted the visualization; the quantitative results were produced without Gaussian blur. We have now updated the video to exclude the Gaussian blur, and now the eyes are segmented appropriately. In addition to this, we have also provided a comparison image in Supplementary file to show that our model segments out the eye accurately wrt the ground-truth eye segment.

W3: "shape-agnostic within class, ability to segment unseen samples": Please refer to the “Cross-category generalization” experiment in Section 7.2. As discussed, our method can handle shape variations such as horse to camel or sedan to SUV as illustrated in Figure 9. We have added a discussion in Appendix and added Figure 10 to show this capability.

Q1: "Use multi-object annotations as input?": Thanks for your suggestion. Yes, our model can handle multi-object annotations together. We have now included a discussion on “Multi-Object Flexible Granularity” in Section 7.2 and added a new visual result (Figure 11). As shown, SMITE is able to handle multi-object segmentation.

Q2: "failure case": Please see the general response above and refer to Section 6 and also Figure 13.

[1] Koichi Namekata, et al. Emerdiff: Emerging pixel-level semantic knowledge in diffusion models. ICLR, 2024.

Thanks to the authors for the efforts to resolve my initial concerns and curiosity. The authors specifically clarified the proposed pipeline and presented additional insights to what all the reviewers suggested. I will maintain my recommendation for acceptance.