Segment, Select, Correct: A Framework for Weakly-Supervised Referring Segmentation

We thank the reviewer for the time spent on the paper as well as for their comments.

The main idea behind our zero-shot approach is the decomposition of RIS into the segmentation of all objects belonging to the desired target (segment in Stage 1) followed by a selection step to choose the right mask to return (select in Stage 2). This simple idea realized by combining pre-trained models is shown to be highly effective in practice, beating other published zero-shot baselines by as much as 19% in some evaluation sets.

By leveraging the richness of all the segmentation masks from the segment stage, we are able to correct a weakly-supervised model trained on the outputs of our zero-shot approach and obtain new state-of-the-art performance in this setting, reducing the gap between fully and weakly-supervised methods from 33% (in previously published work) to as little as 14% in some cases.

On presentation and notation. Our goal by introducing the dots and colors to represent datasets and stages in our framework was to simplify the process of reading the paper. However, we understand how this could be overwhelming from the perspective of the readers, so in the updated manuscript we have replaced the original dataset (gray) by $\mathcal{D}$ (and defined it in Section 2.1), the full masks dataset (blue) by $\mathcal{S}_1$ to identify it is the output of Stage 1, and the selected masks dataset (green) by $\mathcal{S}_2$ to identify it is the output of Stage 2. These are defined in the beginning of Section 3. We hope these changes improve the readability of the paper.

On the contributions beyond combination. First, while it is true that stage 1 and 2 of our method are mostly combinations of pre-trained methods, these have not been proposed together in this way to solve the problem of Referring Image Segmentation in a zero-shot setting. This approach, while simple, achieves state-of-the-art in the benchmarked datasets, beating more complicated strategies like GL-CLIP from Yu et al., (2023) by as much as 19% in some cases. We believe this is one of our contributions.

Secondly, the main contribution of the paper is the design of a new objective function and training algorithm (constrained greedy matching) in Stage 3. As recognized by Reviewer g3AQ, this is a novel contribution from our paper, yielding significant gains and establishing new state-of-the-art results in the weakly-supervised setting.

On the unclear key details of the correction stage. The reviewer mentions that there are ‘some important key details about the correction stage that are not clear, which requires further explanation.’ We would appreciate it if the reviewer could clarify exactly what is unclear about this section.

If the clearness issue comes from the definition of index $c$ as per the question from Reviewer g3AQ, we clarify that $c$ is the index of each binary mask $m^{c}\_{j,k}$ in the set of masks for each reference in the image, i.e., it is an index for the set $\\{m^1_{j,k}, \dots, m^C_{j,k}\\}$ where $C$ is the total number of masks. To avoid confusion we will use the extended set notation throughout the paper, and refer to $c \in \\{1,\dots,C\\}$. We have clarified this at the end of Section 3.1 and at the beginning of Sections 3.2 and 3.3 in the revised manuscript. If there is anything else that is unclear please let us know.

We thank the reviewer for the time spent on the paper as well as for their comments.

(W1 & W2) On notation. We thank the reviewer for bringing these details to our attention. $c$ is the index of each binary mask $m^{c}\_{j,k}$ in the set of masks for each reference in the image, i.e., it is an index for the set $\\{m^1_{j,k}, \dots, m^C_{j,k}\\}$ where $C$ is the total number of masks. To avoid confusion we will use the extended set notation throughout the paper, and refer to $c \in \\{1,\dots,C\\}$.

This change should also clarify the notation from equations 2 and 3. Equation 2 corresponds to the optimization problem we are trying to solve, where the goal is to choose the assignment $\delta_{j,k,c} \in \\{0, 1\\}$ (i.e., either that as mask is chosen or not) for each of the possible ground-truth masks in $\\{m^1_{j,k}, \dots, m^C_{j,k}\\}$ such that the sum of the intersections over unions of the chosen masks is maximized (with the matching conditions described by the constraint $\Delta$ defined below Equation 2). Once this problem is solved through greedy input matching to obtain $\delta^*_{j,k,c}$, the actual loss we use is defined in Equation 3, penalizing the cross-entropy of the matched ground-truth mask and the network output $\hat{m}_{j,k}$.

We have clarified this at the end of Section 3.1 and at the beginning of Sections 3.2 and 3.3 in the revised manuscript. If there is anything else that is still unclear in those equations please let us know. Thank you once again for pointing this out.

(W3) On the intuition of Stage 3 from Figure 1. We agree with the reviewer that this step is crucial to the success of our framework. The idea of Figure 1 is to give an overview of the whole pipeline where, for conciseness, we avoid providing excessive details on each of the steps. To highlight the intuition and details of Stage 3, we use Figure 4, which we believe gives readers a better understanding of the process in detail. For the final version of the paper, we will attempt to introduce more details of this stage in Figure 1 directly without overloading it.

(Q1) Ablations on validation and test sets. The ablation studies in Section 4.2 (Stage 1) and 4.3 (Stage 2) are used as cross-validation to justify the design choices for those stages and, as such, including validation and test set results could be seen as overfitting to those datasets. Instead, we make the decisions on those stages using a small subset of 1,000 samples from the training set and evaluate them against the fully-supervised ground-truth available on that dataset alone, before evaluating the full method on the validation and test sets (Section 4.1).

(Q2) On correcting in Stage 2. This is a good point, and one that we explore in the last paragraph of Section 4.3 (copied here):

“A natural question at this point is whether we can correct the zero-shot choice earlier than the training time at Stage 3, and then simply train a model on those masks. To test this, we apply the constrained greedy matching algorithm by replacing $\ell_{\text{match}}$ from Eq. 2 with the CLIP similarity from Eq. 1 to the generation of masks for the first 1, 000 training set examples of RefCOCO, obtaining an oIoU of 36.34 and an mIoU of 38.11. This suggests that attempting to correct the zero-shot choice is not nearly as effective as our Stage 3.”

On top of this insight we also note that inference on our final S+S+C model takes only $0.2s$ compared to $1.78s$ per sample, so introducing a LAVT-based model is also beneficial in terms of inference running time.

We believe we have answered the reviewers’ questions and clarified the weaknesses. We hope this is useful to the reviewer in reconsidering the score attributed, and look forward to continuing the discussion.

Respectfully, we believe this review falls well below the high review expectations from a conference such as ICLR. None of the weaknesses stated is specific enough to be addressed. We would request further details from the reviewer to be able to rebut the points, as the comments provided are generic and unhelpful in improving the paper.

We thank the reviewer for the time spent on the paper as well as for their comments.

The main idea behind our zero-shot approach is the decomposition of RIS into the segmentation of all objects belonging to the desired target (segment in Stage 1) followed by a selection step to choose the right mask to return (select in Stage 2). This simple idea realized by combining pre-trained models is shown to be highly effective in practice, beating other published zero-shot baselines by as much as 19% in some evaluation sets.

By leveraging the richness of all the segmentation masks from the segment stage, we are able to correct a weakly-supervised model trained on the outputs of our zero-shot approach and obtain new state-of-the-art performance in this setting, reducing the gap between fully and weakly-supervised methods from 33% (in previously published work) to as little as 14% in some cases.

(W1) On multiple references of the same object. We do not believe the assumption of different references for objects in the same image is a strong one. While in the studied datasets the multiple references were the result of multiple human annotations, it is easy to imagine that a dataset can be automatically generated to have multiple references to the same object, given some of these references could even be synthetically generated using a causal language model (e.g., following Brown et al., (2020) or Touvron et al., (2023)). Given the quality of some of the references present in the commonly used for evaluation (RefCOCO, RefCOCO+ and RefCOCOg), it is likely that such a system might yield plausible references that are potentially better than human annotators. The key takeaway is that this assumption, while small, yields strong improvements at the level of performance as shown by our experiments.

(W1) On the dataset class projection. We start by noting we do not claim to tackle an open-world setting in this paper, as highlighted by the evaluation section which trains and tests on specific, widely used datasets. While testing these systems in an open-world setting is an interesting open question, it is outside the scope of our paper and method so we leave it for future work.

Our goal is to improve performance in the absence of ground-truth referred segmentation masks. Within this setting – which is a common one for many other applications – having the knowledge of a broad, mutually exclusive set of classes of interest is not a major barrier. It is an assumption often made in many applications such as autonomous driving. Note also that this set of classes can be more extensive than the references in the dataset, as is the case with RefCOCO/RefCOCO+/RefCOCOg which only reference object instances that are a subset of the full COCO class set.

(W2) Novelty. First, while it is true that stage 1 and 2 of our method are mostly combinations of pre-trained methods, these have not been proposed together in this way to solve the problem of Referring Image Segmentation in a zero-shot setting. On the particular point from the reviewer that “methods for obtaining object proposals, and matching those proposals with zero-shot prompting, are already well-explored techniques”, we highlight that our method differs from GL-CLIP from Yu et al. (2023) which generates mask proposals for all objects in the scene, as opposed to our approach which generates masks for all objects of the target noun phrase. As shown in the paper results and in the further baselines the reviewer proposed (see below), this key difference is novel and important for the success of our zero-shot method which achieves state-of-the-art in the task. We believe this is one of our main contributions.

Secondly, the main contribution of the paper is the design of a new objective function and training algorithm (constrained greedy matching) in Stage 3. As recognized by Reviewer g3AQ, this is a novel contribution from our paper, yielding significant gains and establishing new state-of-the-art results in the weakly-supervised setting.