Order-aware Interactive Segmentation

Dear Reviewer DpQL,

Thank you for your encouragement and detailed suggestions. We have revised the paper accordingly. We are thrilled that you found our work “promising” and with “superior performance”!

3. (a) Unfair comparison: Depth Anything V2 requires much more (about 6x) pretraining images than SAM and other backbone

To address your concern about unfair comparison, we have provided 2 new tables (Tables B and C in the previous answer) that show our results using other backbones. The tables highlight that our performance gains are mainly from our proposed order-aware and object-aware attention modules instead of the specific DepthAnything V2 backbone.

3. (b) Unfair comparison: “MM-SAM (Table 5 in Appendix) with Depth Anything V2 achieves similar performance with OIS w/o order and object attention (Table 4).”

Please note that our main contribution lies in the object-aware and order-aware attention modules. We would respectfully like to point out that it is irrelevant to our work if OIS w/o order and object performs similar to MM-SAM. Our original OIS (that contains both the order and object-aware attention modules) obtains an NoC score of 3.80 and a 5-mIoU score of 92.76 on the DAVIS dataset as compared to an NoC score of 6.22 and a 5-mIoU score of 88.19 for MM-SAM with depth maps from DepthAnything V2. We think this is a significant improvement.

3. (c) Freeze the backbone of SAM and train an additional depth head ... train the object and order attention to verify that the improvements don’t come from better backbones.

Thanks for this insightful suggestion! We have adopted the MAE pretrained ViT-B backbone to be consistent with the current SOTA: SegNext, InterFormer, and SimpleClick. The results from Table C (in answer 2 (b)) indicate that our performance improvements are mainly from the object and order-aware attention modules instead of a better backbone.

Our experiments with SAM backbone on our OIS model are in progress, and the results will be included in the revised version of our paper upon completion.

4. Limited novelty

Please see our response to all reviewers titled “Highlighting Our Technical Novelty”, especially points 1 and 2.

5. Same order map for all positive points may be too naive... the depth values of different positive points may be different.

Thanks for bringing up this important point. We have often encountered cases where the objects in a scene have variable depth values. While we completely agree with this comment, our powerful object-aware attention module is able to negate any discrepancy caused due to the simple order-map construction for positive clicks for objects with variable depth (Please see Fig. 17 and Section A.9 of our revised paper for more details). Preliminary experiments on using a more complex construction of order maps for positive clicks (one order map for every positive click) did not show a large improvement in the accuracy (please see the Table D below). Hence we chose to stick to a simpler design. However, please note that our model’s flexibility allows it to accommodate different order maps constructed for each positive click in the future for a more complex dataset if needed, just like our current design choice of using different order maps for different negative clicks.

Table D: Performance comparison of different order map designs for positive clicks.

Dear Reviewer pFaJ,

Thank you for your thorough review and valuable suggestions. We have tried to address each of your concerns below.

1. The concept of order

In this paper, we have clearly defined “order” as “the relative depth between objects in a scene”. It does not relate to the sequence of clicks. However, we are happy to revise the paper if you have a suggestion for a different word that represents the relative depth between objects.

2. (a) Ablation study: sequence of object and order-aware attention modules

Our choice of this sequence is intuitive. The object-aware understanding module is placed before the order-aware understanding module to ensure that the model first develops a clear notion of the target object and locates it accurately. Following this, the order-aware understanding module is used to refine the predictions by eliminating background regions and incorporating missed foreground regions with the help of the order maps.

We further study the significance of this sequence on the DAVIS dataset in the following Table A. The results show that having the order-aware understanding module first, followed by the object-aware understanding module slightly decreased performance than our current sequence (object-aware understanding module first, followed by the order-aware understanding module), indicating the effectiveness of our sequence choice.

Table A: Performance comparison on DAVIS dataset with different sequence ordering of object and order-aware understanding modules.

2. (b) Ablation study: different backbones

To show the robustness of our model to different backbones, we conduct an ablation study on the HQSeg44K dataset, and tabulate the results in Tables B and C. We replaced the DepthAnything V2 backbone with an MAE pretrained ViT backbone to be consistent with prior SOTA, SegNext, InterFormer, and SimpleClick. Table B shows that our method still significantly outperforms the other methods with MAE ViT backbone. Table C highlights that the performance gain from our proposed approach exceeds the performance gains from using a powerful backbone by a substantial margin. This represents the effectiveness and importance of our proposed method.

Table B: Comparison of performance using the same backbone with other SOTA methods on HQSeg44K dataset.

Table C: Comparison of performance improvement of order and object-aware attention with the same backbone on HQSeg44K dataset.

Dear Reviewer pFaJ,

This is a kind reminder that the discussion period will close on November 26, which is in less than 3 days. We hope our responses and clarifications have fully addressed your concerns. If you have any additional questions, we would be happy to provide further explanations. Thank you!

Dear Reviewer pFaJ,

1. We still use the Depth-AnythingV2 model for generating the depth maps to construct the order maps in this case; we have only replaced the backbone of our main architecture with MAE ViT-B backbone to be consistent with prior SOTA methods that use the MAE ViT-B backbone. Please note that the experiments in Table B and C are solely to show that our method is robust to different backbones, while keeping the depth prediction model consistent.

2. We are currently running an experiment that reuses the Depth-AnythingV1 encoder as our backbone. This is a time-intensive process, and we will include the results in the revised version of our paper. However, we hope that our experiments in Table B and C have adequately confirmed that our performance improvements are not due to a more advanced pretrained backbone.

   Please also note that to show that our method is robust to different depth-prediction models while keeping the backbone consistent, we have conducted additional experiments with DepthAnything V1 [1] and ZoeDepth [2] as the depth prediction models as shown in Table E. In all the experiments in Table E, we have only changed the depth prediction model that generates the order maps, we have fixed the backbone of our model to our original setting. The results consistently show minor performance variations, with our method outperforming the current SOTA method, SegNext, in all cases. For further details, please refer to point 10 of our response to Reviewer rqn7 and point 1 of our response to Reviewer r4kn.

Table E: Performance comparison on DAVIS using depth maps from different depth prediction models.

Hopefully these explanations address your concerns. We are happy to answer any other questions that you have. Thank you very much for your time and effort!

[1] Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data, CVPR 2024
[2] ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth

Dear Reviewer pFaJ,

As suggested, we retrained our framework using the DepthAnything V1 pretrained backbone as the image encoder and the DepthAnything V1 decoder for depth prediction. It is important to note that DepthAnything V2 was fully replaced by DepthAnything V1 in both the image encoding and depth prediction processes. The results, shown in Table F, demonstrate that the performance improvement achieved by our proposed approach significantly exceeds the gains obtained from utilizing a more powerful backbone. This finding aligns with the analysis presented in our response 2(b) and Table C, which highlights the effectiveness and importance of our proposed method.

Table F: Comparison of performance improvement of order and object-aware attention with the different backbones on HQSeg44K dataset.

Dear Reviewer pFaJ,

As the discussion period will end today, we sincerely hope our responses have answered your concerns. If there are any other questions, we are willing to make further clarification. We appreciate your engagement and constructive suggestions to our work. Thanks!

Dear Reviewer jpbL,

We appreciate your detailed reviews and valuable opinion. We have tried to address your concerns below.

1. Difference between order-aware/object-aware attention and masked attention ... only in the source of the input mask, ... slightly innovative.

Besides the source of the input mask, the way of mask guidance is also different. Please see our response to all reviewers titled “Highlighting Our Technical Novelty” (especially points 1(b) and 2(a)).

2. Fair Comparison with methods trained on COCO

The current state-of-the-art methods SegNext and HQ-SAM use HQSeg44K as their training dataset. So, to ensure fair comparison with these methods, we trained our method on the HQSeg44K dataset.

For the fair comparison with other traditional methods solely trained on COCO, we train our model using COCO dataset only and compare with traditional interactive segmentation methods on HQSeg44K and DAVIS dataset. The results in Table A below show that our model still outperforms other methods by a large margin, which indicates the effectiveness of our model.

Table A: Performance comparison with methods trained on COCO.

3. Objects with variable depth

Thanks for this valuable suggestion. We have added qualitative results for objects with variable depth in Section A.9 of our revised paper. As demonstrated in Fig. 17, the target objects exhibit considerable depth variation, yet our model consistently delivers accurate and high-quality segmentations. This robustness stems from the proposed object-aware understanding module that ensures the model perceives the target object as a whole, enabling it to handle significant depth variation effectively. In addition, please note that the “object-awareness” also allows our model to negate the effects of erroneous depth prediction (Please see section A.9 in the supplementary and our response to Reviewer r4kn - 2a and 2b).

Dear Reviewer jpbL,

This is a kind reminder that the discussion period will close on November 26, which is in less than 3 days. We hope our responses and clarifications have fully addressed your concerns. If you have any additional questions, we would be happy to provide further explanations. Thank you!

Dear Reviewer jpbL,

As the discussion period will end today, we sincerely hope our responses have answered your concerns. If there are any other questions, we are willing to make further clarification. We appreciate your engagement and constructive suggestions to our work. Thanks!

Dear Reviewer QSAV,

Thank you for the invaluable suggestions and for pointing out some critical questions. Below, we have tried to address each of your questions.

1. (a) Relatively easy to understand method section

Thank you very much for your positive comment!

1. (b) The motivation and problems to be solved are not concise... want to solve many problems and propose many improvements.

We have carefully summarized our motivation and proposed solution below.

Motivation:

We find that current interactive segmentation methods (SegNext, HQ-SAM, SAM, SimpleClick, etc) often fail to accurately separate target objects from the background in challenging cases with occlusions, multiple objects interacting with one another, and for thin and intricate objects with a vibrant background (Figs. 1, 4, 5, 8, 9). These issues occur due to a limited understanding of “order”, which we define to be the “relative depth of objects from one another in the scene”.

Solution:

To address the aforementioned issue, we propose the following solution:

(a) We aim to incorporate the concept of “order” in our interactive segmentation model. This is performed by our order-aware attention module.

(b) We incorporate the concept of “objects” to ensure that our model can distinguish objects belonging to the same “order”, and to ensure that our model is robust to erroneous order map construction (order maps depend on the quality of depth information available). The concept of “objects” is incorporated in our object-aware attention module.

In addition, please see our response to all reviewers titled “Highlighting Our Technical Novelty”.

1. (c) ... hard to understand why the author proposes certain techniques

It would be very helpful if you could specify which technique needs to be better motivated. We will try our best to provide an intuition. In addition, please see our response to all reviewers titled “Highlighting Our Technical Novelty”.

2. (a) Results on extra dataset

We have used the same datasets for evaluation as the recent state-of-the-art interactive segmentation methods like SegNext, HR-SAM++, HR-SAM, and HQ-SAM. Moreover, compared to the four suggested datasets, our evaluation datasets are more challenging:  HQSeg44K contains a larger scale of data with much higher target annotation (44k images with over 1000 semantic classes); all images from the DAVIS dataset are real-world complex scenarios. Hence, these two datasets provide convincing evidence of our method’s efficacy.

Additionally, following the suggestion, we evaluate our method on the GrabCut dataset, as shown in Table A. Note that none of the SAM-based methods have been evaluated on this dataset.  We aim to conduct the experiments on the other three suggested datasets and update the results in the revised version of the paper.

Table A: Performance comparison on GrabCut dataset.

2. (b) “there is a lack of comparison with some recently published (CVPR 2024, etc.) ”

Could you please point to these specific works?

To the best of our knowledge, we have included the relevant state-of-the-art methods, including SegNext (CVPR 2024), HQ-SAM (NeurIPS 2023), SAM (ICCV 2023), InterFormer (ICCV 2023), and SimpleClick (ICCV 2023). However, we might have unintentionally missed some methods, owing to the overwhelmingly large number of papers in this domain. It would be great if you could please let us know what specific works to compare against.

Based on your suggestion, we included discussion on some additional recent works (MFP [1] and GraCo [2] from CVPR 2024). We include MFP in Table A, however, MFP did not evaluate on HQSeg44k and their performance on the DAVIS dataset (NoC90: 5.32, NoC95: 11.27) is much lower than ours (NoC90: 3.80, NoC95: 8.59). GraCo, uses additional part-object training and operates under a different setting (multi-granularity), so we have excluded it from our comparison.

[1] MFP: Making Full Use of Probability Maps for Interactive Image Segmentation, CVPR 2024
[2] GraCo: Granularity-Controllable Interactive Segmentation, CVPR, 2024

3. CVPR rather than ICLR

Our paper fits in the “applications to computer vision, audio, language, and other modalities” subject area of ICLR 2025. To our knowledge,  many methods on interactive segmentation [3, 4], and depth-guided segmentation [5] have been previously published in ICLR, which closely relate to our work.

[3] AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation, ICLR 2024
[4] Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching, ICLR 2024
[5] DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation, ICLR 2024

Thanks to the author's thoughtful response. This paper introduces depth information into the field of interactive segmentation. I recognize this contribution, so I will revise my score.

Thank you very much for increasing the score! We are glad that we could address your concerns!

Dear Reviewer r4kn,

Thank you for your thorough feedback and constructive comments. We appreciate the opportunity to clarify the aspects you have mentioned. Our responses can be found below.

1. (a) In Table 3, does the computing cost of the depth model also get included?

Yes, we reported the SAT Latency metric [1] in Table 3, which includes the total time involved in prediction (encoding the image, depth map generation, encoding the clicks, and decoding the final segmentations). We have now further clarified this in Section 4.4 of our revised version. Please note that despite incorporating depth, our model achieves significantly lower latency while maintaining high segmentation accuracy, owing to the superior time-efficiency of recent SOTA depth prediction models like DepthAnything V2.

[1] Rethinking Interactive Image Segmentation with Low Latency, High Quality, and Diverse Prompts, CVPR 2024

1. (b) Impact of using various depth prediction models to the segmentation performance

Table A below shows our results using different depth prediction models. Please note that our work outperforms SegNext (the previous SOTA) by a substantial margin, with all depth prediction models we use, highlighting the robustness of our carefully designed object-aware and order-aware attention modules. We also provide detailed discussion and visualizations (Fig.15, 16) in Section A.9 of the revised version of our paper.

Table A: Performance comparison on DAVIS using depth maps from different depth prediction models.

[2] Depth Anything V2, NeurIPS 2024
[3] Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data, CVPR 2024
[4] ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth

2. (a) The robustness of the segmentation model to the errors brought by depth prediction

In our previous answer, we have shown that our model is robust to using different depth prediction networks. This robustness is attributed to our proposed object-aware understanding module which ensures that our model comprehends the object as a whole, enabling it to handle depth prediction errors effectively. We present additional qualitative results in Section A.9 to highlight this. For instance, in Fig. 15, while ZoeDepth and DepthAnything V1 misinterpret the depth of the animal's tail, our model is robust to this error and can accurately segment the tail. In Fig. 16, despite a significant depth prediction error of all the depth models where the dancer’s hat blends into the background audience, our model correctly recovers the error and accurately segments the hat.

Also, please note that the key role of depth in our approach is to address complex scenarios where distinguishing the target from the background is challenging, as demonstrated in Fig. 1 and Fig. 4. In these scenarios, our model significantly outperforms existing methods. For cases where depth is less critical (Fig. 17), we show that our model remains robust and unaffected by potential negative effects of depth. This demonstrates that our model can effectively solve challenging cases while maintaining strong performance in standard scenarios, which we have discussed more detailedly in Section A.9 of the revised paper.

2. (b) “when segmenting neighboring objects with close depth distance, how much can the depth model contribute?”

It is true that order maps may not be able to separate the foreground object from all background objects. However, we would like to point out that: 1) order maps can still help to eliminate distractions from most background objects. 2) our pipeline also leverages appearance / objectness features, on top of order maps. We show this qualitatively in Fig. 16, in which the depth prediction model fails to differentiate the duck’s boundary from the surrounding water body. However, our model’s “object-awareness” allows it to successfully segment the duck.

3. Ablation on HQSeg44K dataset

Thanks for the suggestion! Table B below shows the ablation results on the HQSeg44K dataset. These results are consistent with the findings from the ablation study on the DAVIS dataset (Table 4 of the main paper), reaffirming that each proposed module plays a crucial role in enhancing overall performance.

Table B: Ablation experiments on HQSeg44K.

Dear Reviewer r4kn,

This is a kind reminder that the discussion period will close on November 26, which is in less than 3 days. We hope our responses and clarifications have fully addressed your concerns. If you have any additional questions, we would be happy to provide further explanations. Thank you!

Thank you very much for increasing the score! We are glad that we could address your concerns!

Dear Reviewer rqn7,

Thank you for your detailed review and insightful suggestions. Your feedback is invaluable for improving our work. Here, we address each of your concerns.

1. Page limit

We followed the official ICLR 2025 guidelines which state that “the optional ethics statement will not count toward the page limit”. Our main text follows the 10 page limit.

2. (a) “Technical contribution and novelty of this paper are incremental”

Please see our response to all reviewers titled “Highlighting Our Technical Novelty”.

2. (b) Unique benefit of order in interactive segmentation

For challenging cases in interactive segmentation, such as objects with occlusions, thin and intricate structures, etc, (refer to Fig. 1 and Fig. 4), users are often required to perform multiple interactions, i.e., redundant positive and negative clicks to capture the entire foreground and remove parts of the background. Our proposed design of using order maps to guide the attention to image features can help the model understand the relative depth between objects better. This enables the model to distinguish the target from background effectively with significantly fewer user interactions as shown in Table 1 and 2 of the original paper.

3. Ordering of order-aware understanding and object-aware understanding

Our choice of this sequence is intuitive. The object-aware understanding module is placed before the order-aware understanding module to ensure that the model first develops a clear notion of the target object and locates it accurately. Following this, the order-aware understanding module is used to refine the predictions by eliminating background regions and incorporating missed foreground regions with the help of the order maps.

We further study the significance of this sequence on the DAVIS dataset in the following Table A. The results show that having the order-aware understanding module first, followed by the object-aware understanding module slightly decreased performance than our current sequence (object-aware understanding module first, followed by the order-aware understanding module), indicating the effectiveness of our current sequence choice.

Table A: Performance comparison on DAVIS dataset with different sequence ordering of object and order-aware understanding modules.

4. Parallel structure for order-aware and object-aware understanding modules

We thank the reviewer for the suggestion. Incorporating this could be interesting, which would require significant effort on retraining and experiments. We plan to leave this for future work.

However please note that our current choice of placing these modules is straightforward and intuitive (please see the previous answer for the reasoning behind this) which helps us achieve SOTA performance.

5. Prior work also enhances foreground-background distinction

Thanks for the suggestion. We have now discussed this work in the detailed related work section A.3 in our revised version.

Please note that the suggested paper discovers foreground objects by training an additional network to predict foreground object features, and uses these object features as a prior input to the segmentation network. In contrast, we discover and refine foreground objects through our novel object-aware and order-aware attention modules, which is very different. On the DAVIS dataset, our method achieves an 88.05 1-mIoU, outperforming the suggested paper's 80.02, which indicates a stronger foreground-background distinction capability in our approach.

Dear Reviewer rqn7,

This is a kind reminder that the discussion period will close on November 26, which is in less than 3 days. We hope our responses and clarifications have fully addressed your concerns. If you have any additional questions, we would be happy to provide further explanations. Thank you!

References:

[1] Masked-attention mask transformer for universal image segmentation, CVPR 2022
[2] Putting the object back into video object segmentation, CVPR 2024
[3] HODOR: High-level Object Descriptors for Object Re-segmentation in Video Learned from Static Images, CVPR 2022