Renovating Names in Open-Vocabulary Segmentation Benchmarks

We thank the reviewer for acknowledging our paper to be “on the right path” and that our presentation to be “easy to follow” and have “clear logic”. We address the rest of the comments below.

1. Noise types and rates.

Besides the noise types we visualized in Figure 1, there are indeed many other noise types in the real world, such as “being too specific”, “being too ambiguous”, “being culturally/socially inappropriate”, and many more. However, since no groundtruth names are known, it is infeasible to determine exactly how many types of name mistakes are there. It is thus also impossible to determine the “noise rates” for each noise type. However, in the future, one can use human annotators or even VQA models to annotate a dataset for such name mistakes. That would certainly be very valuable for our understanding how noise in names affect the models.

Nevertheless, we can still use human-verified RENOVATE names on the validation set to estimate the “noise rates” of different datasets. Specifically, we can define noise rate$=1- sim($original name, renovated name$)$ to assess how different the original names are from the RENOVATE names, and use the averaged score to assess the “noise rate” of a dataset. Under this definition, the noise degree of ADE20K and Cityscapes are 0.59 and 0.39, respectively. We can see that ADE20K exhibits a higher noise ratio compared to Cityscapes, potentially because ADE20K has more samples and classes and is likely to have more errors.

2. On the name selection process.

As written in L179-182, we perform name ranking and selection for each segment separately. Therefore, only one groundtruth mask is needed for each segment. Since we do per-segment renaming, there is no intra-class variance to deal with.

3-(a). Sensitivity to GPT4 prompts.

Our GPT prompts have the following components: (1) original name; (2) context names from image captioning; (3) suggestions on the two types of names (synonyms and short contexts) to focus, with examples; (4) instructions on how to make use of the original name, with examples.

In Table B.1, we performed an ablation study on the different choices of the context names and we can see that better context names result in significant gains in the renaming performance. We further ablate components (3) and (4) by removing them from the GPT-4 prompt and report the results in the following table. From our empirical studies, we find that GPT-4 is typically robust to the specific wording of the four components and that removing any key component (esp (2) and (3)) can significantly affect the results.

3-(b). Manual check for GPT-4 results

Our pipeline does not require additional manual checks at the GPT-4 results as they already incorporate knowledge from our “context names” and the original name.

4. Why only focus on the segmentation task?

We chose the segmentation task as this is one of the most representative tasks for 2D recognition. Please note our segmentation task covers semantic, instance, and panoptic segmentation. Our method is readily applicable to detection (by replacing the mask by the bounding box) and classification (by replacing the mask by the whole image).

Specifically, we conducted an additional study on an open-vocabulary object detection model YOLO-World [1] and by replacing the original names with RENOVATE names for fine-tuning, we improved the performance on ADE20K from an AP of 18.1% to 21.2%, which is equivalent to 17.1% relative improvement. We will add the results to the paper.

5. More related works

We thank the reviewer for pointing out more related works. Both works mentioned by the reviewer propose to decompose the class names into a set of attributes or prototypes and are indeed also addressing the problem that class names are typically not descriptive enough for open-vocabulary segmentation tasks.

However, note that our work addresses the problem of class names in a different way – our RENOVATE framework is able to deal with various kinds of problems with the original names and directly improve the name quality for the segmentation task. We will discuss these related works in our final camera-ready version.

[1] Cheng, et al. "Yolo-world: Real-time open-vocabulary object detection." CVPR 2024.

We thank the reviewer for the response and we are pleased to see that our rebuttal has addressed the reviewer's concerns.

If any remaining concerns hold the reviewer’s opinion on the current borderline recommendation, we would be happy to provide further clarification and discussion. Otherwise, we invite the reviewer to kindly raise the recommendation rating.

Update: We have noticed the recommendation score raising and would like to thank the reviewer again for recognizing the value of this work.

We thank the reviewer for acknowledging the novelty of our paper and our efforts in making our writing “engaging”. We address additional comments as the following:

1. On the renaming model and downstream segmentation task.

Thanks for the suggestion, we will work on making this part more streamlined and clear. Now to answer the specific questions:

First, we train the renaming model for all categories of a dataset.

Second, it is an interesting idea to use the renaming model for downstream segmentation. To do so, we need to use it without gt-mask attn biases and use names from all the classes as the text queries. Note that the “rand gt mask” in Table B.2 still uses gt masks – it only randomizes in the intermediate layers of the transformer decoder, but always uses gt masks at the initial layer. Using a renaming model without gt masks would compromise the performance significantly and make the model unable to distinguish different instances from the same class, as reflected in its segmentation performance on ADE20K: an mIoU of 33.2% (semantic segmentation) and AP of only 12.9% (instance segmentation). Similarly, OVSeg models cannot perform as well as our renaming models in the renaming task as they do not make full use of the available knowledge for renaming (see discussion in 2.).

While the renaming model may not be useful for downstream segmentation, it can be used for other tasks such as generating mask-name annotations for image datasets by matching caption nouns or tags with mask proposals, making it potentially useful to improve many large-scale datasets. See the PDF for an example.

2. The necessity of an architecture for renaming.

Compared to OVSeg, the renaming task has more knowledge of the inputs (gt masks and gt classes) and only focuses on finding optimal names within candidate names of each class for each visual segment. Thus, when designing renaming model architecture, it is a good idea to make use of these different task requirements to ensure high-quality renaming results.

In fact, our renaming architecture is nearly identical to Mask2Former except that we use text embeddings as input queries and gt masks (with randomization in intermediate layers) as attention biases for cross-attention computation. As shown in Table B.2, both modifications are essential to improve the renaming performance (e.g., from an mIoU of 27.1% to 37.1%). We further reported the renaming performance when using OVSeg architecture FC-CLIP under the same setup as in Table B.2:

We can see that our renaming architecture is indeed much superior at the renaming task.

3. Other architectures for training with the RENOVATE names.

We thank the suggestion for extending the current training experiments to include more architectures.

We further experiment with a very different architecture, YOLO-World[1], which does not rely on a CLIP visual backbone. To compare the training name quality on COCO, we fine-tune YOLO-World-M on the COCO object detection dataset and evaluate OV object detection on ADE20K. By replacing the original names with RENOVATE names, we improve the AP from 18.1% to 21.2%, which is equivalent to 17.1% relative improvement. This further demonstrates the general applicability of RENOVATE names to help model training.

4. On the evaluation with renovated names.

The reviewer raised an interesting misclassification case where the visually close misclassifications are of high interest and suggested that “one should seek to highlight key errors in the evaluation”. We totally agree with this.

We want to first point out that the “key errors” are highly domain-dependent. For example, in autonomous driving, identifying a “car” as a “truck” is much less dangerous than identifying it as a “road” for an autonomous driving car. In this case, the key error is not “car/truck” but “car/road” misclassification. In this domain, classification mistakes with high semantic differences (e.g., “car/road” mistake) are often thought of as more “wrong” as it is consistent with our intuition of a classification mistake.

In the X-ray case, we can use mistakes in “placement and small characteristic outline patterns” as the key errors instead of “visual similarity” and design the evaluation metric accordingly.

Finally, regardless of how to design the evaluation metric, renovated names make it possible to conduct fine-grained analysis on mistakes with different semantic distances. Without our renovated names, we can only see the coarse misclassifications without a fine-grained understanding of the model.

We will add this discussion of the evaluation metric in our revision.

5. Further applicability of the renaming pipeline with rich captions.

We want to first point out the fact that collecting a high-quality multi-caption dataset is very expensive — current caption data collection is mainly from web scraping which is of low quality, often misses many objects in the scene, and doesn’t consist of multiple/rich annotations. Curating these datasets already requires a lot of effort and costs[2,3].

Compared to this, it is very economically efficient to curate a dataset with relatively coarse annotations and use our RENOVATE to improve the quality of annotations. From this perspective, RENOVATE is indeed very useful and meaningful to improve current large-scale datasets with coarse web-scraped annotations. In the PDF, we showed an example where the renaming model is applied to generated masks and tags and produces good mask-name matches. This shows that RENOVATE has broad future applicability.

[1] Cheng, et al. "Yolo-world: Real-time open-vocabulary object detection." CVPR 2024.

[2] Gadre, et al. "Datacomp: In search of the next generation of multimodal datasets." NeurIPS 2024.

[3] Fang, et al. "Data filtering networks." arXiv preprint (2023).

We thank the reviewer for pointing out that we addressed an “important but underexplored” problem and that our approach is “technically sound”, “properly explained”, and “would have positive impact on future research”. We address the remaining comments below.

1-(a). Batch collation to train the renaming model

Our renaming model processes all images in a batch (16 images) in a single forward pass – all the text queries from the gt segments are processed in parallel. Specifically, each input query is to assess the alignment between one gt mask and one of its candidate names — the input query itself is initialized with the text embedding of the candidate name, and it is refined by the cross-attention layers that use the gt mask as attention biases.

Since the model can process multiple queries in a single pass (like Mask2Former), we can process all such mask-name pairs in one batch in a single pass. For images where the number of pairs is less than the number of queries, we simply pad with empty queries for collation.

We will make the batch collation more clear in the paper.

1-(b). Memory usage

Our model consumes similar memory to a standard Mask2Former model. For the memory consumption, the key factor is the number of queries. In Mask2Former, this number is 250. In our renaming model, it usually varies between 100 to 350, depending on the number of candidate names and segments in images in a batch.

2-(a). Name merging for evaluation in Table 3

For results in Table 3, when evaluating, we append the “renovated names” instead of “candidate names” to the other naming variants. After merging the names, the number of names per class is 6.44, 4.98, and 6.89 for COCO, ADE20K, and Cityscapes, respectively. This has barely any influence on inference time (0.3sec/img) since the inference cost is mostly in processing the image and query features instead of the final cosine similarity computation. One can even choose to use more test names (e.g. 10 names per class) depending on the use case and will not incur any noticeable extra cost of inference time.

2-(b). The usage prompt template in inference

For each prompt name, we first use ViLD templates to create 63 different sentences and extract their CLIP text embeddings. We then average the text embeddings after normalization and use this single embedding as the text embedding for the corresponding name. This is also the common practice in other OVSeg works [1,2].

2-(c). The need of renaming model for the evaluation

Since we need “renovated names” for evaluation, we need to train the renaming model to obtain them. Evaluating using candidate names would introduce unnecessary complications in the evaluation as they are quite noisy (as indicated by the inferior performance when training with candidate names in Table 3).

3-(a). Baselines for the renaming model

We thank the reviewer for the suggestion of using average-pooled CLIP features as a baseline. In prior works, this baseline is known to be inferior to tailored OVSeg models[1]. In [1], it’s shown that with gt masks as region proposals, average-pooled CLIP features only reach an mIoU of 20.1% on ADE20K, being significantly lower than OVSeg models (at least over 30%).

Our baseline for the renaming model is the FC-CLIP with gt attn biases (row 1 in Table B.2), which is much stronger than the average-pooled CLIP features. Our model significantly improves upon this baseline (from 27.0% mIoU to 37.1% mIoU) by incorporating our designs of “rand gt masks” and “text queries”.

3-(b). Necessity of training the mask prediction module.

Relatedly, since CLIP features are not good enough, we need to improve them. The pixel decoder in our renaming model is responsible for improving the CLIP features to a per-pixel level. The mask prediction module is irreplaceable since it’s the only source of supervision signals for these per-pixel features.

We also note that even if we use gt masks as attn biases, the mask prediction is still non-trivial. As we write in L160-164, the gt masks are only guiding the cross-attention to focus on the segment regions.

3-(c). Alternatives of the mask prediction module

As an alternative to our mask prediction module, one may design different proxy losses to train the renaming model. For example, a region-level contrastive loss that forces matched mask-name embedding pairs to be closer. As our focus of this paper is to demonstrate the importance of names and renaming, we leave such exploration to future work.

4. Table 3 OpenSeg results are lower than the original FCCLIP results.

This is because the performance of OV Seg model depends on the test name set. In Table 3, our test names merge all three name sets. This is for a fair comparison of models trained with different names. We will update the paper accordingly to make this clear.

5. Figure 5 setups.

The x-axis in Figure 5 shows the number of images available in the training set. The training images are randomly selected from the whole training set.

6. Negative sampling variants

Negative sampling is highly effective to train with a large vocabulary in NLP. Recently, some OVSeg models also have adopted a similar approach to ours [2].

While negative names from other classes are guaranteed “true negatives”, same-class renovated names can be complicated. For example, for one segment, “man” might be the most precise, but “person” is also not wrong. It is thus not ideal to penalize “person” in the same way as names from other classes (e.g., “car”) during training. To remedy this, one may add an extra step to first select “true negatives” among each class. This can indeed be an interesting future extension of our current sampling approach.

[1] Liang, et al. "Open-vocabulary semantic segmentation with mask-adapted clip." CVPR 2023.

[2] Cheng, et al. "Yolo-world: Real-time open-vocabulary object detection." CVPR 2024.

As the discussion period concludes, we would like to express our sincere gratitude to all the reviewers for their valuable feedback and suggestions. We are pleased to see that all the reviewers acknowledged our rebuttal effectively addressed their concerns and subsequently raised their recommendation ratings, recognizing the value of our paper. We will carefully consider all the feedback and incorporate it into our revised paper.