Discriminatively Matched Part Tokens for Pointly Supervised Instance Segmentation

Response to Weakness

W1: The section of related work is inadequate in weakly supervised instance segmentation. Some weakly image-level supervised methods[1] and box-supervised methods[2][3] not listed.

R1:As suggested, we have integrated the previous researches, including IRNet, BoxInst and Box-LevelSet, into the section of related work for a comprehensive survey.

W2: The inference speed of whole method could be reported and it can better demonstrate the superiority of the proposed method.

R2: In the supplementary, we measured and reported the FPS metric of our DMPT model and others, which displays DMPT's superiority of inference efficiency. As suggested, we will move this section in the main paper to improve the presentation of our strengths.

W3: Some typos exist, like ''Combed with ..." in section 3.5 should be ''Combined with ..."

R3: Thinks for your advice, the typos have been revised and we think the fluency of our paper has been enhanced.

Answer to Questions

Q1: If some parts of an object are directly filtered as background due to low activation values in the self-attention map, how could this be further optimized in subsequent steps?

A1: While some object parts are occluded in some images, in statistical sense, all object parts of the same category will appear in the dataset. Specially, our part classifiers statistically learn the fine-grained semantics from the whole dataset. As long as most of the parts visible in the images, part classifiers can be optimized to represent and recognize the stable fine-grained semantics by using visible parts. In addition, our token-classifier matching mechanism allows the part classifier not matching any parts if its corresponding part is occluded. This characteristic ensures that the classifiers not only are able to represent the fine-grained pattern from visible parts, but also can keep steady optimized when object part absents in some images.

Q2: It would be beneficial to include comparisons with more weakly supervised methods as the performance reference for the readers.

A2: Thanks for your constructive and insightful comments. We have added more comparisons with weakly supervised methods in the revised version.

R1: Due to challenge of less supervision information, the location precision of object remains imperfect in pointly supervised instance segmentation, which limits accuracy of part tokens generation. Meanwhile, part deformation and occlusion result in difficulty for clustering methods to produce part tokens to represent anthropogenic defined object parts, like "head" or "leg". This does limit the ability of the part classifier to actually activate the object part uniformly.

Surprisingly, DMPT is still able to learn relative stable and consistent fine-grained semantics than the baseline method. We conduct statistical analysis of feature similarity on different aspects in Figure 1, updated in "rebuttal_visualization.pdf" of the new supplementary materials. In Figure 1(a), the similarity between part features and background of DMPT is lower than that of AttentionShift, which showcases that our part-based modeling mechanism can effectively suppress background noise, obtaining stable semantics. As we utilize part classifiers to uniformly represent fine-grained semantics, DMPT exhibits stronger discrimination among objects of different categories, In Figure 1(b). Particularly, we calculate the feature similarity of different parts of the same object, Figure 1(c). It can be seen that the part features of DMPT are more discriminative than AttentionShift, implying that we do learn consistent fine-grained semantics.

R2.1: In Equ.8 of main paper, the token-classifier matching allows the case when part tokens' number is less than that of part classifiers ($K \textless N$), due to part occlusion or other factors. We visualize the matching matrix of some cases in Figure 2 of the "rebuttal_visualization.pdf".

R2.2: No, the matching mechanism do not handle cases of over-segmentation, because it has been handled before token-classifier matching. First, we pre-define that each object has $N$ parts, based on which we initialize $N$ part classifiers and partition the object into $N$ cluster centers. Then, similar cluster centers are merged according to their cosine distances, avoiding over-segmentation on consecutive part semantics. After this procedure, we produce $K$ part clusters for each object. Note that $K$ might be different values for different objects. In order to uniformly represent the number of part clusters across all objects, we use $K$ to control the upper limit of the number of object parts after merging similar clusters. In addition, $K\leq N$ is an implicit setting in our paper.

Thanks for this constructive advice. For more clear presentation, we will add the details about how to handle the over-segmentation problem and relation of $K$ and $N$ in the paper.

R3: $N$ is a hyper-parameter to pre-define the number of classifiers optimized. $N$ is experimentally determined for achieving the balanced performance of all the classes within a dataset.

For special usage for other scenarios, $N$ is empirically set to a relative high value to ensure classifiers can sufficiently cover various fine-grained semantics. As our DMPT owns two characteristics, (i) token-classifier matching allows some classifiers not matching any part tokens during model training (R2.1), and (ii) the number ($K$) of part tokens of different objects dynamically adapts to object semantics (R2.2), part classifiers have the potential to be dynamically optimized with objects with various part semantics. However, mismatched number of tokens ($K=3$) and classifiers ($N=7$) could result in insufficient optimization on stable fine-grained semantics, Figure 6(c) of main paper.

R4: The setting you mentioned is right. However, getting one accurate pseudo-mask needs a selection upon three pseudo-masks when using one single point to prompt SAM. Note that producing three masks from one single point prompt is a design of SAM to address ambiguity of the prompt. In the setting of pointly supervised methods, there is no ground-truth mask as guidance to select the most accurate one. We only can select pseudo-masks leveraging location scores (IoU predictions) of these masks predicted by SAM. However, these scores are usually unreliable for obtaining the most accurate mask.

In order to address this issue, DMPT acts a prompter before prompting SAM by generating part points with the the single point, which we termed as DMPT-SAM. The generated part points contains more spatial information ($e.g.$, the deformation, scale and extent of object), which is less ambiguous than one single point. With these points as prompt, the location scores are more reliable to select the accurate pseudo-masks.

Quantitative results in Table 1 and qualitative visualizations in Figure 4(b) of main paper demonstrate that DMPT-SAM facilitates generating more precise pseudo-labels for pointly supervised instance segmentation.

R5: Thanks for your constructive advice and we will further simplify and refine our notations for higher presentation in the camera-ready version.

All my concerns have been thoroughly addressed, except for the query regarding SAM. I still believe that a comparison between DMPT-SAM and the direct application of SAM would enhance the quality of this paper.

In their response, the authors mentioned SAM's ambiguity, potentially compromising the quality of self-supervision. However, I posit that this issue may persist even when employing more prompts through the proposed part-based approaches. I still cannot understand why the authors do not address this concern in the most clear and powerful way: quantitative comparison. The experiments using SAM directly seem not that difficult to conduct, at least for me.

Nevertheless, I deeply appreciate the authors' extensive efforts for the rebuttal. It remains evident that DMPT (excluding SAM) outperforms existing methods, and the overall quality of this paper is pretty good even without DMPT-SAM. As a result, I am inclined to raise my rating.

W1: The paper shares the same idea of using self-attention maps with prior works (AttnShift), but the differences are not well elaborated.

R1: As our baseline method, AttentionShift generates multiple anchor points to represent object parts. The method roots on parameter-free clustering, like mean-shift, which however experiences difficulty to uniformly represent part semantics and suffers from background noises. As a significant difference, DMPT leverages instance classification loss to train part classifiers so that it can dynamically match part tokens. After the loss gradient back-propagation, the classifier parameters are updated to recognize and represent fine-grained features as well as suppress background noises.

Furthermore, the key-point shift procedure in AttentionShift does not consider part spatial deformation, which might cause key-points false shift to the background or another instances. Convincingly, DMPT leverages learnable parameters and deformation constraint to estimate the deformation of each part token, which facilities handling spatial deformation.

With above essential promotions, DMPT respectively achieves significant (2.0% mAP$_{50}$ and 1.6% AP) performance gains on PASCAL VOC 2012 and MS-COCO 2017.

W2: The limitation of the part deformation constraint is not considered. For example, will it work properly if the target is a snake with a long and thin shape?

R2: The proposed part deformation constraint has the potential to handle these extreme cases. Specially, thanks to the ability of global attention mechanism to effectively capture the overall deformation of the object, DMPT can dynamically decompose the object into various parts, acting as reference points for local spatial deformation, which makes the model focus attention on object parts. Based upon these reference points, the proposed part deformation constraint further enhances local features by coupling the deformation representation to counter part deformation. Therefore, incorporating our part deformation constraint with the global attention mechanism is crucial for effectively resolving substantial object deformations.

Undoubtedly, there are some limitations of the spatial deformation. On the one head, initializing deformation constrain $d(x_i,y_i)$, defined in Equ.3, as a square distance feature might be less rational, as the part of some objects like snake sometimes represents a slender shape, whose spatial feature should have different initialized weights upon horizontally and vertically. On the other head, the value of $d(x_i,y_i)$ should be dynamically reduced before coupling with part classification scores when meeting extreme cases, as spatial constrain might be less unreliable in this situation.

Thanks for your advice, which inspires us to further improve our method.

W1: The training process seems complex. How much computational cost is introduced by the newly introduced modules?

R1: We compare the efficiency of training and inference of our method upon the AttentionShift in the following Table.

The additional training time (0.36h) spends on the proposed part token allocation and token-classifier matching. Note that the batch size is 16 on 8 Tesla-A100 GPUs, training PASCAL VOC for 36 epochs. And we test the FPS with an image, whose the shortest side is 800 pixels, on a single Tesla-A100.

W2: It is not clear how much performance improvement is brought by the introduction of spatial constrains (eq. (5)). One more experiment is needed to verify this.

R2: As suggested, we have conducted experiment to display the performance of spatial constrains in Figure 6(b) of main paper and analyzed the effectiveness in paragraph "Token-Classifier Matching" of Sec 4.5 in the paper. For more clarity, we have revised part of paragraph as following:

''When $\alpha=0$, only part classification score determines whether a patch token should be allocated as the part token or not, which achieves 51.2% upon mAP50, 1.1% higher than that (50.1% in Fig.6(a)) allocating cluster centers as part tokens, defined in Equ. 2. After introducing part deformation constraint, $\alpha=5$ reports the best mAP$_{50}$ (52.1%), which outperforms the best method by 0.9% (52.1% vs 51.2%). ''