P2Seg: Pointly-supervised Segmentation via Mutual Distillation

WK4: Ambiguities

A4: (1) Regarding your observation about the weights of the loss components in $L_{S2I}$, we would like to clarify that the weights for $L_{offset}$ and $L_{segmentation}$, respectively, do not necessarily need to sum to 1. This is because these weights are not probabilities but rather scaling factors that balance the contributions of the two-loss components to the overall loss function. In our case, the weight for the segmentation loss is set to 1.0, signifying its primary importance in the segmentation task. The offset loss, with a weight of 0.01, plays a secondary role, fine-tuning the segmentation results rather than driving them. This choice of weights is based on empirical observations where we found this configuration to provide the best balance between the two types of loss, leading to optimal performance in our experiments.

Q2: Why BESTIE is retrained, for both PascalVOC-2012 and COCO-2017.

(2) Our team decided to retrain BESTIE due to the unavailability of its original code. Despite our best efforts to replicate the performance of BESTIE, we encountered challenges in fully reproducing its results. This deviation could be attributed to variations in implementation details, which are not available in the public domain. Furthermore, we reached out to the authors of BESTIE for assistance and access to the code, but unfortunately, we did not receive a positive response. In light of these circumstances, retraining BESTIE was a necessary step to ensure the integrity and reproducibility of our research findings. It also allowed us to adapt the model more closely to the specific requirements of our dataset and research objectives. Should you have access to the original BESTIE code or can facilitate a connection with the authors, we would be immensely grateful and more than willing to integrate it into our research.

(3) We observed that AttentionShift employs a strong pre-trained model, which might not provide a fair basis for direct comparison with our approach.

(4) In our study, we utilized Point2Mask in the context of instance segmentation, considering its capability in panoptic segmentation. This was based on the understanding that a model effective in panoptic segmentation should also perform well in instance segmentation tasks. The discrepancy in our reported mAP values stems from our experimental setup, where we adapted Point2Mask for instance segmentation. In addition, we consider BESTIE as the base of our paper's method, and Point2mask as a comparative experiment can better enrich our experiment.

WK5: Other comments

A5: Thanks. (1) We have made corrections in the manuscript： change “denotes the β times Hardmard power” to “denotes the β times Hadamard power”, “The Hardmard power of the affinity matrix provides additional information” to “The Hadamard power of the affinity matrix provides additional information”.

(2) We have already explained the 'instance ownership relationship' in Sec3.2: “Specifically, in the instance similarity matrix, if two pixels belong to the same instance, their value is set to 1, otherwise, it is set to 0. ”

(3) We have made corrections in the Pg. 5, eq. (5), we have added an explanation that “A is assumed to be generated by the network HRNet”

(4) We have made corrections in the Pg 6: "COCO (i.e. COCO 2017) includes 118k images." The experiment of the test set has been supplemented.

Table 1* Quantitative comparison of the state-of-the-art WSIS methods on MS COCO 2017 test-dev

(5) We have made corrections in Pg 6： "We assess the performance of instance segmentation using two measures. We measure the performance using the standard protocol mean Average Precision(mAP)."

Dear Reviewer,

Thank you for your insightful comments.

W1:Novelty

A1: (1) Methodological Distinction: Our mutual distillation method (MDM) fundamentally differs from BESTIE in its approach. While BESTIE utilizes a unidirectional process for semantic knowledge transfer, our MDM is a bidirectional process. This two-way interaction enables a more robust and comprehensive learning mechanism, which is critical for the complexity of point-level weak supervision in instance segmentation.

(2) Performance Improvements: As demonstrated in our visual results, the MDM approach shows superior performance compared to BESTIE, especially in handling the boundaries of adjacent objects of the same class. These improvements are significant as they address a common challenge in instance segmentation, demonstrating the practical impact of our method.

(3) Innovative Aspects: As noted by the first four reviewers, our approach is both innovative and interesting. The bidirectional nature of our MDM method represents a novel contribution to the field, offering a new perspective and methodology, for instance, segmentation tasks.

W2:Organization

A2: (1) Instance Affinity Matrix and Class Map in HRNet: HRNet serves as a backbone network in our experiments, functioning primarily for feature extraction. It can indeed be substituted with other networks such as ResNet. We will update our paper to include this clarification and cite the HRNet paper (Yuan et al., 2019).

Q1: How the semantic segmentation is obtained.

(2) Following the concept from BESTIE regarding the use of semantic segmentation maps, our semantic segmentation maps are derived from weak supervision, specifically employing the PMM model ("Pseudo-mask Matters in Weakly-Supervised Semantic Segmentation").

Q4: Provide details about the number of points used and the relations points-targets.

(3) In response to your query about the number of points used and their relation to the targets, our method indeed involves single-point participation, similar to the class center point setting used in the articles "BESTIE" and "Point-to-Box Network for Accurate Object Detection via Single Point Supervision". This approach is guided by our aim to maintain a balance between the accuracy of instance segmentation and the efficiency of the annotation process.

Q3: Why so many iterations are used, and what is included?

(4) I would like to clearly clarify that in our methodology, although the LossI2S and LossS2I are iteratively calculated, they are concurrently aggregated and backpropagated within the same iteration, which guarantees that the training process is synchronized, and there is no increase in the number of training iterations.

WK3:In particular

A3: (1) To clarify, our approach, similar to the BESTIE model, utilizes a class center point for point-level weak supervision in instance segmentation. We have added explanations in Sec4.3： In Table 5, the term “hard pixel ratio” refers to the proportion of challenging samples used in loss computation.

(2) We have conducted a comprehensive comparison of our method with the BESTIE method in terms of GFLOPs and FPS. The results of this comparison, as detailed in Table 2* as follows, reveal that our method only slightly increases the GFLOPs and FPS values compared to BESTIE. The increase in computation is mainly attributed to the additional label generator component of our method, which produces labels for training a fully supervised segmenter. However, during the inference stage, only the fully supervised segmenter is utilized, which means the inference time remains the same as that of BESTIE.

Table 2* Comparison of our method with the BESTIE method in terms of GFLOPs and FPS.

(3) The impact of semantic segmentation:

Writing and Presentation improvement

(1) We have made corrections in the manuscript: change “denotes the β times Hardmard power” to “denotes the β times Hadamard power”, “The Hardmard power of the affinity matrix provides additional information” to “The Hadamard power of the affinity matrix provides additional information”.

(2) We have already explained the 'instance ownership relationship' in Sec3.2: “Specifically, in the instance similarity matrix, if two pixels belong to the same instance, their value is set to 1, otherwise, it is set to 0. ”

(3) We have made corrections in the Pg. 5, eq. (5), we have added an explanation that “A is assumed to be generated by the network HRNet”

(4) We have made corrections in the Pg 6: "COCO (i.e. COCO 2017) includes 118k images." The experiments of the test set have been supplemented.

(5) We have made corrections in Pg 6: "We assess the performance of instance segmentation using two measures. We measure the performance using the standard protocol mean Average Precision(mAP)."

(6) We have updated our paper to cite the HRNet paper (Yuan et al., 2019).

• Sun, Ke; Xiao, Bin; Liu, Dong; Wang, Jingdong. "Deep high-resolution representation learning for human pose estimation." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5693-5703. 2019.

(7) We have added explanations in Sec4.3： "In Table 5, the term “hard pixel ratio” refers to the proportion of challenging samples used in loss computation."

(8) Dim-align in Fig.3: The term 'Dim-align' refers to 'dimension alignment,' a necessary step in our method. Prior to matrix multiplication, we rearrange the dimensions of our tensors for compatibility, a process accomplished using the transpose or permute functions in PyTorch.

(9) Instance Adaptive Grouping in Sec.3.1: Our network creates final instance segmentation labels based on a combination of predicted semantic segmentation results, instance similarity matrices, and point annotations. If conflicts arise between semantic predictions and point-based instance segmentation, we prioritize the point annotations. For points lacking final instance predictions, we assign a pseudo-box of size 16x16. This method ensures that our semantic segmentation results, a fusion of network predictions and instance similarity matrices, are more representative of instance characteristics, thereby improving instance segmentation outcomes.

(10) Green Arrow in Fig.3 (OffsetMap): The green arrow represents the OffsetMap, which, in conjunction with the Semantic Map, generates new instance segmentation results. This process is detailed in the right half of Fig. 5.

(11) We have cited these references：

• Kweon, Hyeokjun, Yoon, Sung-Hoon, & Yoon, Kuk-Jin. (2023). Weakly Supervised Semantic Segmentation via Adversarial Learning of Classifier and Reconstructor. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11329--11339).

• Cheng, Zesen, Qiao, Pengchong, Li, Kehan, Li, Siheng, Wei, Pengxu, Ji, Xiangyang, Yuan, Li, Liu, Chang, & Chen, Jie. (2023). Out-of-candidate rectification for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 23673--23684).

• Rong, Shenghai, Tu, Bohai, Wang, Zilei, & Li, Junjie. (2023). Boundary-Enhanced Co-Training for Weakly Supervised Semantic Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 19574--19584).

• Ru, Lixiang, Zheng, Heliang, Zhan, Yibing, & Du, Bo. (2023). Token contrast for weakly-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3093--3102).

(12) We have carefully modified the introduction to make it more readable.

We have revised the sentence: "Instance segmentation is a critical task in computer vision, where semantic segmentation estimation and instance discrimination are equally important."

Additionally, we have updated the phrase: “and it aims not only to locate objects accurately but also to estimate their boundaries to differentiate.”

We have also replaced the terms “semantic segmentation” and “instance segmentation” with “semantic information” and “instance information” in the second paragraph of the introduction to provide a clearer and more accurate description.

Writing and Presentation improvement

(1) Dim-align in Fig.3: The term 'Dim-align' refers to 'dimension alignment,' a necessary step in our method. Prior to matrix multiplication, we rearrange the dimensions of our tensors for compatibility, a process accomplished using the transpose or permute functions in PyTorch.

(2) Instance Adaptive Grouping in Sec.3.1: Our network creates final instance segmentation labels based on a combination of predicted semantic segmentation results, instance similarity matrices, and point annotations. If conflicts arise between semantic predictions and point-based instance segmentation, we prioritize the point annotations. For points lacking final instance predictions, we assign a pseudo-box of size 16x16. This method ensures that our semantic segmentation results, a fusion of network predictions and instance similarity matrices, are more representative of instance characteristics, thereby improving instance segmentation outcomes.

(3) Green Arrow in Fig.3 (OffsetMap): The green arrow represents the OffsetMap, which, in conjunction with the Semantic Map, generates new instance segmentation results. This process is detailed in the right half of Fig. 5.

(4) We have cited these references：

• Kweon, Hyeokjun, Yoon, Sung-Hoon, & Yoon, Kuk-Jin. (2023). Weakly Supervised Semantic Segmentation via Adversarial Learning of Classifier and Reconstructor. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11329--11339).

• Cheng, Zesen, Qiao, Pengchong, Li, Kehan, Li, Siheng, Wei, Pengxu, Ji, Xiangyang, Yuan, Li, Liu, Chang, & Chen, Jie. (2023). Out-of-candidate rectification for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 23673--23684).

• Rong, Shenghai, Tu, Bohai, Wang, Zilei, & Li, Junjie. (2023). Boundary-Enhanced Co-Training for Weakly Supervised Semantic Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 19574--19584).

• Ru, Lixiang, Zheng, Heliang, Zhan, Yibing, & Du, Bo. (2023). Token contrast for weakly-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3093--3102).

(5) We have carefully modified the introduction to make it more readable.

We have revised the sentence: "Instance segmentation is a critical task in computer vision, where semantic segmentation estimation and instance discrimination are equally important."

Additionally, we have updated the phrase: “and it aims not only to locate objects accurately but also to estimate their boundaries to differentiate.”

We have also replaced the terms “semantic segmentation” and “instance segmentation” with “semantic information” and “instance information” in the second paragraph of the introduction to provide a clearer and more accurate description.

(6) We have made corrections in the manuscript: change “denotes the β times Hardmard power” to “denotes the β times Hadamard power”, “The Hardmard power of the affinity matrix provides additional information” to “The Hadamard power of the affinity matrix provides additional information”.

(7) We have already explained the 'instance ownership relationship' in Sec3.2: “Specifically, in the instance similarity matrix, if two pixels belong to the same instance, their value is set to 1, otherwise, it is set to 0. ”

(8) We have made corrections in the Pg. 5, eq. (5), we have added an explanation that “A is assumed to be generated by the network HRNet”

(9) We have made corrections in the Pg 6: "COCO (i.e. COCO 2017) includes 118k images." The experiments of the test set have been supplemented.

(10) We have made corrections in Pg 6: "We assess the performance of instance segmentation using two measures. We measure the performance using the standard protocol mean Average Precision(mAP)."

(11) We have updated our paper to cite the HRNet paper (Yuan et al., 2019).

• Sun, Ke; Xiao, Bin; Liu, Dong; Wang, Jingdong. "Deep high-resolution representation learning for human pose estimation." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5693-5703. 2019.

(12) We have added explanations in Sec4.3： "In Table 5, the term “hard pixel ratio” refers to the proportion of challenging samples used in loss computation."

Dear Reviewer,

We appreciate your valuable suggestions and insights, we have addressed each point as follows:

WK1&Q1: Writing and presentation need improvement.

A1: Thanks. We have explained confusing descriptions in a dedicated section for replies and clarifications.

WK2: Some key technical details are missing. The required number of MDM stages is not specified or experimented.

A2: In our study, we intentionally set the number of stages to 1. We found that even with a single MDM stage, our method achieved significant and satisfactory results. Your constructive suggestions for multiple MDM stages may potentially enhance the performance of our model. We intend to make a further investigation of the performance and efficiency in the future.

WK3&Q2: What impact do the hyperparameter β and the number of MDM stages have on performance?

A3: We added a new table (Table 8) for the ablation study about the parameter β in Section 4.3 as follows:

The parameter β: As shown in Table 8, setting β to 1 enables our model to attain an optimal equilibrium, culminating in the most effective performance within our weakly-supervised framework. It is noteworthy that an increment in β corresponds to a decrement in performance by 0.5%.

WK4&Q3: Does the use of SAM destroy fair comparisons, and would other pseudo-label generators be useful? The used off-the-shelf segmentation map should be considered as extra data?

A4: (1) In weakly supervised learning, pseudo-labels are inferred from limited or incomplete information. For image segmentation, this might mean inferring pixel-level labels from image-level labels or bounding box labels.

The role of SAM is to use existing weak labels or partial information to generate these more detailed pseudo-labels. Weakly supervised learning involves not only training with incomplete labels but also the process of processing and enhancing these labels. Here, SAM acts as a 'label enhancer,' transforming weak labels into more detailed forms, making them more suitable for image segmentation tasks.

This approach differs from fully supervised learning, which relies on fully annotated training data. In weakly supervised learning, even if the quality of pseudo-labels may not be as high as fully supervised data, models can still learn from them and improve performance.

Using SAM to generate initial pseudo-labels and for training weakly supervised segmentation models represents an innovative strategy in weakly supervised learning. The key to this strategy is that it enhances the learning ability of the model by inferring more detailed labels from limited information while maintaining the fundamental principles of weakly supervised learning — not relying on fully annotated data.

(2) For extra data: In our study, 'no extra data' means that no data outside of what was originally provided in the dataset was used during the training and testing processes. The segmentation maps we used were generated from the original dataset, not from external sources. Therefore, we believe this conforms to the definition of 'no extra data'. These segmentation maps were obtained through preprocessing of the existing data, rather than being a separate, external dataset.

The segmentation maps obtained through the PMM weak supervision method are part of the model training, not an additional input independent of the main dataset. This approach is taken to enhance the performance of the model rather than to introduce external data. This improvement in our method can be confirmed in Table 1.

WK5&Q4: Will the proposed S2I method be sensitive to the position of the points?

A5: We added a new table (Table 9) for the ablation study about the sensitivity of our proposed method to the positioning of points in Section 4.3 as follows:

Drift-point: As shown in Table 9, we conducted an ablation study where we applied Gaussian random perturbations with standard deviations (σ) of 5, 10, and 15 to the coordinates of the center points of each object. Our results reveal that when the centroid points are subject to a Gaussian perturbation with σ = 5, the $mAP_{50}$ scores are consistent with those obtained when the centroids are precisely located. However, as the standard deviation increases to σ = 10 and σ = 15, we observe a decrement in mAP50 by 0.6% and 1.5%, respectively, compared to the precise centroid placement. Our findings indicate that while the performance of our method does experience a slight impact when the points undergo displacement, the overall effect remains within an acceptable range.

Thanks for the careful reply. It addressed my concerns. I would like to raise my score.

Writing and Presentation improvement

(1) We have cited these references：

• Kweon, Hyeokjun, Yoon, Sung-Hoon, & Yoon, Kuk-Jin. (2023). Weakly Supervised Semantic Segmentation via Adversarial Learning of Classifier and Reconstructor. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11329--11339).

• Cheng, Zesen, Qiao, Pengchong, Li, Kehan, Li, Siheng, Wei, Pengxu, Ji, Xiangyang, Yuan, Li, Liu, Chang, & Chen, Jie. (2023). Out-of-candidate rectification for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 23673--23684).

• Rong, Shenghai, Tu, Bohai, Wang, Zilei, & Li, Junjie. (2023). Boundary-Enhanced Co-Training for Weakly Supervised Semantic Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 19574--19584).

• Ru, Lixiang, Zheng, Heliang, Zhan, Yibing, & Du, Bo. (2023). Token contrast for weakly-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3093--3102).

(2) We have carefully modified the introduction to make it more readable.

We have revised the sentence: "Instance segmentation is a critical task in computer vision, where semantic segmentation estimation and instance discrimination are equally important."

Additionally, we have updated the phrase: “and it aims not only to locate objects accurately but also to estimate their boundaries to differentiate.”

We have also replaced the terms “semantic segmentation” and “instance segmentation” with “semantic information” and “instance information” in the second paragraph of the introduction to provide a clearer and more accurate description.

(3) Dim-align in Fig.3: The term 'Dim-align' refers to 'dimension alignment,' a necessary step in our method. Prior to matrix multiplication, we rearrange the dimensions of our tensors for compatibility, a process accomplished using the transpose or permute functions in PyTorch.

(4) Instance Adaptive Grouping in Sec.3.1: Our network creates final instance segmentation labels based on a combination of predicted semantic segmentation results, instance similarity matrices, and point annotations. If conflicts arise between semantic predictions and point-based instance segmentation, we prioritize the point annotations. For points lacking final instance predictions, we assign a pseudo-box of size 16x16. This method ensures that our semantic segmentation results, a fusion of network predictions and instance similarity matrices, are more representative of instance characteristics, thereby improving instance segmentation outcomes.

(5) Green Arrow in Fig.3 (OffsetMap): The green arrow represents the OffsetMap, which, in conjunction with the Semantic Map, generates new instance segmentation results. This process is detailed in the right half of Fig. 5.

(6) We have made corrections in the manuscript: change “denotes the β times Hardmard power” to “denotes the β times Hadamard power”, “The Hardmard power of the affinity matrix provides additional information” to “The Hadamard power of the affinity matrix provides additional information”.

(7) We have already explained the 'instance ownership relationship' in Sec3.2: “Specifically, in the instance similarity matrix, if two pixels belong to the same instance, their value is set to 1, otherwise, it is set to 0. ”

(8) We have made corrections in the Pg. 5, eq. (5), we have added an explanation that “A is assumed to be generated by the network HRNet”

(9) We have made corrections in the Pg 6: "COCO (i.e. COCO 2017) includes 118k images." The experiments of the test set have been supplemented.

(10) We have made corrections in Pg 6: "We assess the performance of instance segmentation using two measures. We measure the performance using the standard protocol mean Average Precision(mAP)."

(11) We have updated our paper to cite the HRNet paper (Yuan et al., 2019).

• Sun, Ke; Xiao, Bin; Liu, Dong; Wang, Jingdong. "Deep high-resolution representation learning for human pose estimation." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5693-5703. 2019.

(12) We have added explanations in Sec4.3： "In Table 5, the term “hard pixel ratio” refers to the proportion of challenging samples used in loss computation."

Dear Reviewer,

Thank you very much for your valuable comments and suggestions, which deeply improve our manuscript.

Q1: The author should provide comparisons about efficiency, like GFLOPS, FPS, inference time, and so on, with existing methods, to demonstrate that the improvement does not come from the extra computational budget. In addition, will multi-stage training derive into more training iteration numbers?

A1: (1) We have conducted a comprehensive comparison of our method with the BESTIE method in terms of GFLOPs and FPS. The results of this comparison, as detailed in Table 2* as follows, reveal that our method only slightly increases the GFLOPs and FPS values compared to BESTIE. The increase in computation is mainly attributed to the additional label generator component of our method, which produces labels for training a fully supervised segmenter. However, during the inference stage, only the fully supervised segmenter is utilized, which means the inference time remains the same as that of BESTIE.

Table 2* Comparison of our method with the BESTIE method in terms of GFLOPs and FPS.

(2) In terms of training iterations, I would like to clearly clarify that multi-stage training in our model does not lead to an increase in the number of training iterations. The iteration number is 1 because although the LossI2S and LossS2I are iteratively calculated, they are concurrently aggregated and backpropagated within the same iteration, which guarantees that the training process is synchronized, and there is no increase in the number of training iterations.

Q2: The ablation study on the COCO dataset. Will the improvement become less?

A2: In our research, we conducted a series of ablation studies on the COCO dataset to provide a comprehensive evaluation of our methodology. The results of these ablation experiments have been incorporated into Section 4.3 of our paper, specifically under the subsections 'Effect of S2I' and 'Effect of I2S':

As shown in Table 7, S2I obtains 3.2 mAP$_{50}$ performance improvement on PSIS tasks compared with the previous methods on COCO, showing the effectiveness of S2I statistically.

Effect of I2S. The I2S module improves instance discrimination in the same class by using the instance affinity matrix. As shown in Table 7, I2S raises mAP$_{50}$ from 17.4 to 17.6. This discrepancy in performance enhancement between the two datasets can likely be attributed to their inherent differences in features and complexity. The variations in performance improvements across datasets suggest that each module plays a crucial role, depending on the specific characteristics and challenges presented by the dataset.

Q3: What's about the performance of the method on the COCO dataset with ResNet?

A3: We updated the Table 2 in the manuscript. We have added the performance of our method on the COCO dataset with ResNet-101, which achieved an AP of 15.2.

We are grateful for your insights and are hopeful that our revised manuscript address your concerns.

Writing and Presentation improvement

(1) We have carefully modified the introduction to make it more readable.

We have revised the sentence: "Instance segmentation is a critical task in computer vision, where semantic segmentation estimation and instance discrimination are equally important."

Additionally, we have updated the phrase: “and it aims not only to locate objects accurately but also to estimate their boundaries to differentiate.”

We have also replaced the terms “semantic segmentation” and “instance segmentation” with “semantic information” and “instance information” in the second paragraph of the introduction to provide a clearer and more accurate description.

(2) We have cited these references：

• Kweon, Hyeokjun, Yoon, Sung-Hoon, & Yoon, Kuk-Jin. (2023). Weakly Supervised Semantic Segmentation via Adversarial Learning of Classifier and Reconstructor. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11329--11339).

• Cheng, Zesen, Qiao, Pengchong, Li, Kehan, Li, Siheng, Wei, Pengxu, Ji, Xiangyang, Yuan, Li, Liu, Chang, & Chen, Jie. (2023). Out-of-candidate rectification for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 23673--23684).

• Rong, Shenghai, Tu, Bohai, Wang, Zilei, & Li, Junjie. (2023). Boundary-Enhanced Co-Training for Weakly Supervised Semantic Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 19574--19584).

• Ru, Lixiang, Zheng, Heliang, Zhan, Yibing, & Du, Bo. (2023). Token contrast for weakly-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3093--3102).

(3) Dim-align in Fig.3: The term 'Dim-align' refers to 'dimension alignment,' a necessary step in our method. Prior to matrix multiplication, we rearrange the dimensions of our tensors for compatibility, a process accomplished using the transpose or permute functions in PyTorch.

(4) Instance Adaptive Grouping in Sec.3.1: Our network creates final instance segmentation labels based on a combination of predicted semantic segmentation results, instance similarity matrices, and point annotations. If conflicts arise between semantic predictions and point-based instance segmentation, we prioritize the point annotations. For points lacking final instance predictions, we assign a pseudo-box of size 16x16. This method ensures that our semantic segmentation results, a fusion of network predictions and instance similarity matrices, are more representative of instance characteristics, thereby improving instance segmentation outcomes.

(5) Green Arrow in Fig.3 (OffsetMap): The green arrow represents the OffsetMap, which, in conjunction with the Semantic Map, generates new instance segmentation results. This process is detailed in the right half of Fig. 5.

(6) We have made corrections in the manuscript: change “denotes the β times Hardmard power” to “denotes the β times Hadamard power”, “The Hardmard power of the affinity matrix provides additional information” to “The Hadamard power of the affinity matrix provides additional information”.

(7) We have already explained the 'instance ownership relationship' in Sec3.2: “Specifically, in the instance similarity matrix, if two pixels belong to the same instance, their value is set to 1, otherwise, it is set to 0. ”

(8) We have made corrections in the Pg. 5, eq. (5), we have added an explanation that “A is assumed to be generated by the network HRNet”

(9) We have made corrections in the Pg 6: "COCO (i.e. COCO 2017) includes 118k images." The experiments of the test set have been supplemented.

(10) We have made corrections in Pg 6: "We assess the performance of instance segmentation using two measures. We measure the performance using the standard protocol mean Average Precision(mAP)."

(11) We have updated our paper to cite the HRNet paper (Yuan et al., 2019).

• Sun, Ke; Xiao, Bin; Liu, Dong; Wang, Jingdong. "Deep high-resolution representation learning for human pose estimation." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5693-5703. 2019.

(12) We have added explanations in Sec4.3： "In Table 5, the term “hard pixel ratio” refers to the proportion of challenging samples used in loss computation."

Dear Reviewer,

Thank you very much for your valuable comments and suggestions, which deeply improve our manuscript.

Q1: It would greatly enhance the quality of this paper if the authors could provide a quantitative analysis of these cases.

A1: In Section 4.4 of the manuscript, we have added a quantitative analysis to complement the visual results presented in Figures 7 and 8:

As indicated in Table 10, this analysis is grounded in two key metrics: Missing Rate and Adjacent Rate. The Missing Rate quantitatively evaluates the instances where our method fails to segment objects as precisely as depicted in the ground truth. It is noteworthy that, in comparison to the BESTIE method, our approach demonstrates a significant reduction in the occurrence of object misses, with a Miss Rate that is lower by 3.9%. On the other hand, the Adjacent Rate metric assesses the proportion of successfully identified adjacent objects. Here, our method markedly enhances the capability to recognize neighboring objects, exhibiting a 30.4% higher Adjacency Rate than that achieved by BESTIE.

Q2: The writing of the introduction should be improved somewhat.

A2: Thanks. We have carefully modified the introduction to make it more readable.

We have revised the sentence from "Instance segmentation is a critical task in computer vision and is equally important in semantic segmentation estimation and instance discrimination." to "Instance segmentation is a critical task in computer vision, where semantic segmentation estimation and instance discrimination are equally important."

Additionally, we have updated the phrase “and it aims not only to locate objects accurately but also estimate their boundaries to differentiate” to “and it aims not only to locate objects accurately but also to estimate their boundaries to differentiate.”

In line with your advice, we have also replaced the terms “semantic segmentation” and “instance segmentation” with “semantic information” and “instance information” in the second paragraph of the introduction to provide a clearer and more accurate description.

We believe these revisions have improved the clarity and quality of our manuscript. Your insightful feedback has been instrumental in enhancing our paper, and we are grateful for your contribution to our work.

Dear Reviewer,

Thank you very much for your valuable and detailed feedback on our manuscript. We appreciate your insights and suggestions for improving our work.

Q1: Some important and recent WSSS methods are missing.

A1: Thanks, we cited the important and recent WSSS methods mentioned by you in our revised manuscript, and discussed them in the related work. Further, we also improved the writing and presentation of our paper.

Q2：The performance table is incomplete. First, please show the result of mAP25. Also, there have been several works using the transformer backbone, such as Point2Mask or AttentionShift. Please compare with them. In addition, where is BESTIE (with Res101)? Finally, please include the results of the COCO test set.

A2: (1) We updated Table 1 in the manuscript and added the result of mAP25 and the result of BESTIE (with Res101):

Table 1 Quantitative comparison of the state-of-the-art WSIS methods on VOC 2012 val-set(Part).

We denote the supervision sources as: $\mathcal{F}$ (full mask), $\mathcal{B}$ (box-level label), $\mathcal{I}$ (image-level label), $\mathcal{P}$ (point-level label), $\mathcal{S}$ prompting SAM with ViT-Base for object mask annotations and $\mathcal{C}$ (object count). The off-the-shelf proposal techniques are denoted as follows: $\mathcal{M}$ (segment proposal), $\mathcal{R}$ (region proposal), and $\mathcal{S_I}$ (salient instance segmentor).

(2) Comparison with Point2Mask: As you rightly pointed out, we have included a comparison with Point2Mask in our original submission (Table 1). We believe this comparison showcases the strengths and unique aspects of our proposed method in the context of current advancements. Regarding AttentionShift: We carefully considered including a comparison with AttentionShift. However, we observed that AttentionShift employs a strong pre-trained model, which might not provide a fair basis for direct comparison with our approach.

(3) BESTIE (with Res101) Results: We have also added the results for BESTIE with ResNet-101 in Table 1 in the manuscript. BESTIE achieved mAP50 of 52.3 on this setting.

(4) COCO Test Set Results: Regarding the COCO test set results, our model achieved an AP of 17.4. We give the performance comparison in the Table 1*:

Table1* Quantitative comparison of the state-of-the-art WSIS methods on MS COCO 2017 test-dev

Q3：Compare the proposed method with the SAM

A3: We updated Table 1 and Table 2 in the manuscript and compared our method with SAM. Parts of Table 2 are given as follows:

Table 2 (Part)

As shown in Table 1 in the manuscript, our experimental results demonstrate that our method outperforms SAM on the VOC dataset, indicating its effectiveness in handling diverse segmentation scenarios.

Furthermore, we updated Table 2 in the manuscript, while evaluating the COCO dataset, we observed that SAM shows slightly higher performance in terms of AP and AP50 metrics. However, our method significantly surpasses SAM in the AP75 metric.