Boosting Weakly Supervised Referring Image Segmentation via Progressive Comprehension

Sincerely thanks for useful comments. To the weaknesses, our response is as follows:

[W1]: Conditional Referring Module (CRM) is implemented through multiple cross-attentions, which is a relatively common approach, lacking novelty.

Thanks for your comments. We would like to address the concerns about its novelty and highlight its importance in our approach:

• Firstly, the proposed CRM module is well-motivated. In our method, we observe that referring text descriptions typically contain detailed cues on how to localize the target object. By progressively integrating these target-related textual cues from the input description, we hope to enhance the target localization step-by-step. Therefore, we introduced the CRM module to model this progressive comprehension process. More importantly, the module is combined with the RaS loss to implement instance-level supervision, which has not been explored in previous approaches.
• The CRM module includes two cross-attentions, each playing a crucial role. The first, vision-to-text attention, aims to obtain vision-attended cue features by incorporating visual context information. The second, text-to-text attention, modulates the referring query using the target-related textual cue embeddings. Through the interaction between referring query and different textual cue embeddings, this module is designed to achieve a more discriminative referring embedding. In Table 4, we provide ablation experiments for the module, which also validate the effectiveness of our CRM design.

[W2]: the general idea of classification loss mentioned in the paper.

We sincerely apologize for the less detailed presentation about the Classification loss $\mathcal{L}\_{`cls`}$ in TRIS [25] due to space restriction. The loss $\mathcal{L}\_{`cls`}$ aims to establish a global alignment between the visual content and the referring texts by formulating a classification process. We provided a detailed explanation of the idea and implementation of the loss in the general response. We will include this in the revision.

[W3]: what does the ⊙ symbol represent in Equation 5.

Ans: The $\odot$ denotes the hadamard product. In Equation 5, we multiply the binary mask obtained from SAM and the response map element by element. Thanks for your comments. We will add the explanation in the updated version.

[W4]: this article only conducted ablation experiments on the RefCOCOg dataset, so the ablation experiments are insufficient.

In our main paper, we follow previous works, e.g., SAG[14] and TRIS[25], to conduct the ablation experiments on the representative RefCOCOg dataset. As suggested, we conducted more ablations on other datasets (i.e., refcoco, refcoco+ datasets) and shown the results in the following tables. The results on these datasets lead to the same conclusions as those on RefCOCOg, further validating the necessity and effectiveness of each component of our method.

• RefCOCO

• RefCOCO+

[W5]: The order of the tables is somewhat confusing, with Table 9 preceding Table 6.....

Thanks for pointing out the problem. We will reorder the two tables to make them more in line with logical sequence or reading habits in the updated version.

[W6]: Which datasets were the ablation experiments in Tables 7-9 conducted on? Please specify.

We conduct the ablation experiments on Refcocog (google split) dataset as described in lines 275-276.

[W1]: In table 1, only the results refined by SAM[16] are reported. Considering that FreeSOLO [40] is an unsupervised segmentation model, I am curious about the results refined by FreeSOLO.

[Ans]: Thanks for your advice. Here we present the mIoU after applying FreeSOLO refinement, as shown in the table below. The first row displays the results for TRIS[25], while the second row presents the results for our proposed method. Although FreeSOLO's extracted mask proposals are not as accurate as those from SAM, leading to reduced performance, our method still exhibits clear and substantial improvements across three established benchmarks.

[W2]: This work introduces multiple stages for progressive localization. It may increase the training and inference time of the model. The time comparison of model training and inference may be needed.

[Ans]: Thanks for your comment. Our method incorporates a multi-stage refinement process, which naturally leads to increased training and inference times. Below, we provide a comparative analysis of the time costs associated with various methods. Notably, the overall time expenditure of our method with three stages remains well within acceptable limits.

[Q1]: The mask proposals by SAM are often part of the whole instance. It will affect the refined results. Does this work do anything special to alleviate the problem? The author can give more details about this issue.

[Ans]: You are correct that SAM's mask proposals often represent only a portion of the complete instance. Consequently, even when our method achieves accurate localization, it may not necessarily translate to precise instance masks. To demonstrate this, the last row ("Oracle") of Table 1 presents the mIoU scores achieved by selecting the mask proposal that best aligns with the ground-truth mask. These results support our observation and suggest that integrating a more advanced instance detector, such as GroundSAM [A], could further enhance our method's mask prediction accuracy (mIoU).

[Q2]: Table 1 shows the results of the pseudo labels obtained at the weakly-supervised training stage. Can the authors give the results after pseudo labels based on supervised training?

[Ans]: Thanks for your question. We present additional results below. In our main paper, we focused on the quality of pseudo masks generated by our method (i.e., one stage). Here, we further investigate their effectiveness by training the LAVT [44] network using these pseudo masks as supervision. We report the resulting PointM (first row) and mIoU (second row) metrics on three benchmarks, along with the performance achieved when refining mask predictions with SAM (third row). These findings highlight the potential for further improvement in mask prediction through fully supervised training based on our generated pseudo masks. We will incorporate these results into the revised version of our paper.

References:

[A] Grounding dino: Marrying dino with grounded pre-training for open-set object detection ECCV2024

Sincerely thanks for your useful comments. To the weaknesses and questions, our response is as follows:

[W1]:

Thank you for you careful comment. We sincerely apologize for the less detailed presentation regarding the loss $\mathcal{L}_{`cls`}$ in TRIS [25] due to space restrictions. We provided a detailed explanation of the idea and implementation of this loss in the general response. We will include this loss explanation in the revision.

[W2]:

The idea of two loss functions is different and there are two essential differences between them:

• Motivation. The calibration loss in TRIS is used to suppress noisy background activations and thus help to re-calibrate the target response map. In contrast, in our method, we observe that, multiple referring texts corresponding to different instances in one image may locate the same (or we say overlapping) instances (described in lines 210-212), due to the lack of instance-level supervision in WRIS.

• Implementation. The calibration loss adopts the global CLIP score of image-text to implement a simple contrastive learning for revising the response map. Differently, we simultaneously infer the response maps of different referring texts from the same image, and obtain the instance-level localization results by choosing the mask proposal with max alignment score. IaD can achieves better localization accuracy with instance-level differentiation.

• To further verify the superiority of our loss, we conduct two-groups ablations on the RefCOCO (val) dataset. The first ablation used PCNet without IaD loss as the baseline, and the second used the TRIS without calibration loss. We then separately introduced these two loss functions for comparison. Both ablations demonstrate that the IaD loss not only refines the response map (mIoU metric) but also significantly improves the localization accuray (PointM metric).

  • baseline (CLS+CRM+RaS)

    Baseline	Calibration loss	IaD loss	PointM	mIoU
    ✓			56.1	28.7
    ✓	✓		56.9	29.9
    ✓		✓	60.0	31.3

  • baseline (CLS)

    Baseline	Calibration loss	IaD loss	PointM	mIoU
    ✓			50.3	24.6
    ✓	✓		51.4	26.4
    ✓		✓	54.7	26.3

[W3]:

We would address your concerns from the following four aspects:

• Mask generator does not contain any semantic information. Though it can be used as post-processing method to improve the completeness of the mask, here we aims to improve our localization accuracy by introducing suitable constraint based on the mask proposals.

• Our core idea is to improve the understanding of textual cue information by progressive process. The ablation experiments (Table 2) demonstrate that even without RaS and IaD loss, our approach improves localization to some extent. However, relying only on global classification loss is suboptimal due to its lack of modeling at the instance level. Therefore, we introduce RaS, which leverages mask proposals to model multi-stage refinement and enhance instance-level cross-modal alignment.

• In lines 200-204, we point out that a complete mask proposal can improve the mask quality to some extent, but it is not the core of our RaS loss. In the following, we conducted a quantitative analysis on RefCOCO val set.

  • The first line denotes the results of baseline (include classification loss and CRM module). In the second line, we utilize the mask proposal generated by SAM (refer to line-189) as pseudo mask to calculate the IoU loss between it and the response map. The third line denotes the results of our RaS loss.
  • While the guidance of the mask proposal can improve the accuracy of the response map to some extent, there is a significant gap with the RaS loss on the localization performance. Besides, as shown in Table 1, even with the less accurate mask proposal by FreeSOLO, our methods still achieve superior localization performance. These results show that the performance improvement of our method is not exactly due to the introduction of mask generator.

  Methods	PointM	mIoU
  Baseline (CLs +CRM)	53.5	26.4
  Baseline + IoU_loss	54.5	27.9
  Baseline + RaS	56.1	28.7

• In addition, the IaD loss only utilizes the mask proposal as a localization result to derive the loss formula, there is no exploitation of mask knowledge.

[Q1]:

I guess the question concerns how $Q_{n+1}$, not $Q_{n+t}$ (absent in Fig. 2), is used for calculating the Classification loss. As explained in the general response, $Q_{n+1}$ in stage $n$ contains one positive and $L$ negative referring embeddings per image. Given this known text-image correspondence, computing the cross-entropy loss is straightforward.

[Q2]:

Leveraging the refer_id and image_id annotations in datasets, which link texts to unique instances, we randomly sample texts referring to different instances from the original. Simultaneously the cues for each text in the batch is sampled (one cue is used for referring embedding modulation at each stage).

[Q3]:

About the residual design, we draw inspiration from the previous works (like TRIS[25] and DenseCLIP[A]). The design can integrate useful cross-modal information while preserving the original modal information from degradation.

[Q4]:

The order of the text inputs would not distinctly affect the performance of our method. In the table below, we conducted a quantitative comparison on the RefCOCO (val) to verify this.

References:

[A] Denseclip: Language-guided dense prediction with context-aware prompting CVPR2022

Thank the authors for the detailed response. The original formulation of the classification loss was quite confusing, but now it makes sense. Also, the comparison with the baseline + IoU loss convinces me of the contribution of the RaS loss. The authors also addressed my other concerns. It depends on further discussion with other reviewers, but I am leaning towards raising my rating. Please add the conducted comparison experiments with the calibration loss in the final draft.

Thanks for your response and your willingness to consider raising the score. We genuinely appreciate your time and the valuable feedback. We are pleased that our rebuttal has addressed your concerns. However, we notice that your current rating still suggests that our work is still not qualified to be accepted. Considering that the discussion period is ending soon, if there are any remaining issues or concerns, we would greatly appreciate the opportunity to discuss them further with you.

Sincerely thanks for useful comments. To the weaknesses and questions, our response is as follows:

[W1]:

Thank you for your advice. Although both DGA [A] and our method adopt multi-stage refinement, there are significant differences:

• The motivation is different. DGA focuses on the fully-supervised visual grounding task and aims to model visual reasoning on top of the relationships among the objects in the image. In contrast, our work addresses the weakly RIS task and aims to alleviate the localization ambiguity by progressively integrating fine-grained attribute cues.
• The implementation is different. In the absence of mask, we build the relation between the response map of different stages using the proposed RaS loss.
• We will discuss and include their differences in the revision.

[W2]:

Thank you for your advice. PPT [B], appearing on Arxiv near the NeurIPS deadline, is considered contemporaneous work. It introduces a parameter-efficient point prompting framework. The Table below presents a comparison on the val sets using mIoU. While our method utilizes down-sampled SAM mask proposals, we also tested with full-resolution masks for fairness, aligning with PPT. In both scenarios, our method demonstrates superior target localization compared to PPT. We will discuss and include their differences in the revision.

[W3 & W4]:

• Explanation about Sec. 3.4
  • Motivation. Considering that each image generally contains multiple instances, each paired with its reference description, the weak discriminative capability of the model in WRIS often leads to multiple texts for different instances in a single image activating the same region (i.e, response map overlap in Line-209). Thus, we propose the IaD loss in Sec. 3.4 to encourage referring texts for different instances (in the same image) to activate different regions.
  • Implementation. Since this loss function is applied individually to each image sample, for clarity, we omitted the batch concept and consider only one sample, and omitted the subscript a used in the Line 215-Line 227. Specifically, given one image-text pair $\\{\mathbf{I}, \mathbf{T}\\}$, we sample $N\_d$ (By default, $N\_d=1$) texts for current image sample, which means that we have two referring texts (i.e., $\mathbf{T}$ and $\mathbf{T}\_{d}$ ) for the image $\mathbf{I}$. Then, we can obtain response maps $\mathbf{R}$ and $\mathbf{R}\_{d}$ according to the description in Sec 3.2, and get the alignment scores $S$ and $S\_{d}$ according to Eq. (5). Afterwards, we get the instance indexes $`argmax`(S)$ and $`argmax`(S\_{d})$ as the proposal predictions by them. Considering $`argmax`(\cdot)$ is a non-differentiable operation during gradient backward, we adopt a differentiable implementation by Eq. (8). Finally, we can impose MSE loss on the indexes to encourage the two texts (in the same image) to activate different regions by Eq (9).
• We will revise the figure to make the subscripts clear in the revision.

[Q1]:

• Thanks for pointing out this issue. This is a typo, and the range shold be [0,2], as in some cases, it may occur that $\text{IoU}(𝑅_𝑛,𝑚_𝑓) < \text{IoU}(𝑅_𝑛,𝑚_𝑏)$.

• It is less feasible. The shrinking loss constraint the response map toward the compact orientation while maintaining located in the foreground area. $𝑚_𝑏$ include less foreground part and more background than the $m_f$. It would lead to more background activation when choosing the $𝑚_𝑏$ as the complement of the foreground.

[Q2]:

It is possible to have different instances for different stages. According to our statistical result, this issue occurs in a small number of cases (less than 10%) and has minimal impact on loss optimization. We will include this analysis and revise the Eq. (7) in the revision.

[Q3]:

LLM outputs for referring texts often vary in attribute cue count (2-5). To enable parallel training, we standardize them to 5 cues via repetition padding (Sec. 3.1). The Tab. 3's ablation study on stage numbers shows optimal performance with 3 stages, thus our framework's implementation utilizes three stages.

[Q4]:

In addition to the standard training batch ($B$ image-text pairs), we sample $N_d$ extra texts ($N_d=1$) per image. Leveraging the refer_id and image_id annotations in refcoco datasets, which link texts to unique instances, we randomly sample texts referring to different instances from the original.

[Q5]:

All results utilize a single peak point as the SAM prompt. As suggested, we compared different prompts on the validation set (see table below). Using the response map as the prompt proved sensitive to thresholds, impacting noise ratios and yielding inferior performance compared to the single peak point prompt.

References

[A] Dynamic graph attention for referring expression comprehension--ICCV 2019

[B] Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentation--CVPR. 2024

Thanks very much for your positive feedback that we have addressed your most concerns and your willingness to raise the score. Here we give further explanations about IaD loss and Q4.

• IaD loss. Thanks for your careful comments! In GroupViT, the Grouping Block groups similar semantic regions using the hard assignment strategy during the group tokens training and inferring process. In our IaD loss, we also adopt a similar hard assignment for deriving the loss function. The reason is that we aims to get the pseudo mask prediction by the accurate peak value point (i.e., the hard assignment results) instead of relying on whole score distribution $S(\cdot)$ (e.g., $S_{a}$, $S_{d}$ in Section 3.4). Thus utilizing the hard assignment to derive the IaD loss well matches our purpose, which helps rectify the ambiguous localization results. If we use the soft assignment (e.g., measuring KL divergence between $S_{a}$ and $S_{d}$), though the equivalent may be simpler, it not only does not match our purpose but also introduces more tricky components for optimization (e.g., extra distribution regularization is required). In order to verify the argument, we conduct a comparison on RefCOCOg(G) val dataset as follows. The KL_Loss even causes a slight decline, while the IaD loss brings a clear improvement on localization.

  Methods	PointM	mIoU
  $\mathcal{L}_{`CLs`}$	51.7	25.3
  $\mathcal{L}_{`CLs`}$ + KL_loss	51.2	24.8
  $\mathcal{L}_{`CLs`}$ + IaD_loss	53.1	26.6

• Q4. Yes, you are right. refer_id is important information for the data_loader sampler and adopted in TRIS and SAG. Here, we further consider and utilize the prior (i.e., different texts corresponding to different instances within the same image should activate different regions) for localization optimization.

If there are any additional clarifications needed, please do not hesitate to reach out.