Scaling up Image Segmentation across Data and Tasks

• The expression of Fig.2 is unclear, and I cannot intuitively understand the roles of text prompts and encoders from Fig.2.
• Fig.6 is too small. I can hardly see the text above.
• The performance improvement drive from your proposed query meld mechanism or multiple data co-training is not very clearly compared. Is the performance improvement due to your use of more synthetic data
• The paper should validate the model's performance on more diverse datasets that are not part of the training set. This would demonstrate the true generalization capability of QueryMeldNet in real-world scenarios.
• The paper should provide more details on the synthetic data generation process, including the models used (e.g., SAM, OFA) and the quality of the generated data.

1. The paper lacks system-level performance comparison with previous important baselines. In Table 2, the authors do the ablation experiments about the query strategies. However, the authors do not provide a detailed comparison with previous baselines (as shown in Table 2). Even the comparisons of COCO instance and panoptic segmentation, two of the most important benchmarks in image segmentation, are not given. Also, some recent works [R1, R2] were not compared in the table. In fact, the compared works were all online before mid-2023. Nevertheless, the proposed method does not have a notable better performance on main tasks. This will drastically trivialize the “scaling up” since we hope to see notable results after “scaling up”

2. In Table 1, the authors compare the supported tasks with previous works. Based on the comparison, the proposed method supports extra foreground segmentation. However, the experiments do not show the effectiveness of “foreground segmentation.” In addition, I do not think supporting “foreground segmentation” is a significant contribution, although the authors highlight that.

3. The technical contribution of the module design is not well studied. The authors want to claim that the different types of queries will have different specialties so that a combination of them will have a better result. However, the ratio of these two types of queries has not been studied in detail. Also, given that in Mask2former (a typical learnable query method), if the query number is too much, the performance may drop. Setting the number of queries to 400 may not be a good choice for fair comparison. At the same time, since the performance gain is not very obvious, it is questionable whether similar conclusions can be drawn with large amounts of data. Also, there are some other papers, such as Oneformer, that tackle semantic/instance joint training issue. Comparing these papers under the same setting can also help better understand the effectiveness of the method. In conclusion, I think these two types of queries are not new methods, and their combination has not been fully studied in this paper.

4. The method and contribution of data scaling up seem not significant. Mask labeling object365 is not new, and [R3] provides a much larger object-level captioning dataset. Therefore, both the “scaling data” and the effect of that are not notable.

[R1] Hierarchical Open-vocabulary Universal Image Segmentation.

[R2] OMG-Seg: Is One Model Good Enough for All Segmentation?

[R3] Tokenize anything via prompting.

I think if the position of this paper is to push the boundaries of all existing open-world segmentor, it would be better to provide different scales of pre-trained models for the community to use, e.g., based on ViT-S, ViT-B, and ViT-L. Moreover, it will be very good to see better results under more advanced vision encoders than Swin Transformers.

1. While the results show that QueryMeldNet is effective, large parts of the operation of the method aren’t explained properly. As a result, it is not fully clear how the method works, and why it behaves the way it does. The paper would be significantly better if the method were explained more carefully and in more detail. Concretely:

2. In L243, the paper mentions that the learnable and conditional queries interact with each other via “dual-query cross-attention”. As such, it is a major component of the newly proposed ‘query meld’ strategy. However, the paper does not provide any details about this cross-attention, making it unclear how it works, and how these queries interact. Do both types of queries cross-attend to each other? If so, in what order? What exact operations are used? These details are necessary to be able to understand and reproduce the model.

3. Similarly, the paper does not explain how the conditional queries are generated. L222-L225 describes that these queries are derived from the transformer encoder, and that this encoder is trained to predict region proposals, but no further details are provided. How does the encoder generate these queries exactly? Does QueryMeldNet use an existing method to generate these, or a new design? Is this ‘transformer encoder’ the same encoder as the ‘segmentation encoder’ shown in Fig. 2, or is it a separate component? The answers to these questions should be available in the manuscript.

4. In general, it is not clear what the architectural design is of the individual components of QueryMeldNet as shown in Fig. 2. The Appendix mentions that the model uses a Swin image encoder and CLIP text encoder, but the architecture of the segmentation encoder and segmentation decoder are not described. To understand the model and for reproducibility, this information should be available.

5. L299 mentions that, unlike OpenSeeD, the proposed QueryMeldNet method does not require that it is defined if a certain class is a ‘thing’ or ‘stuff’ class, i.e., if it requires a mask per individual object or a mask for all objects of a particular class together, respectively. However, the paper does not describe how QueryMeldNet handles situations where the ‘table’ is a thing class in one training dataset but a stuff class in another (as mentioned in L301, this is happens for ADE20K and COCO). How does the model handle these conflicting supervision signals? This is currently not clear, but it should be explained in the paper.

6. Similarly, it is not clear how the model knows what task it should conduct. For instance, for the results in Tab. 2, how does the model know that it should conduct semantic segmentation for ADE20K (and predict a mask for all cars together), but panoptic segmentation for COCO (and predict a mask for each individual car)? Is there an additional input to the model that specifies which task it should do? The paper should clearly describe how the model handles this.

7. In Tab. 3, the proposed QueryMeldNet paper achieves the best reported results on all benchmarks, including the SeginW benchmark for ‘in-the-wild’ segmentation. This makes it seem like QueryMeldNet achieves state-of-the-art results on this dataset. However, Grounded-HQ-SAM [a] achieves a mean AP of 49.6 for this dataset, compared to 43.4 by QueryMeldNet, showing that QueryMeldNet is far from the state of the art. To fairly compare QueryMeldNet to prior works, the performance of Grounded-HQ-SAM should be reported in Tab. 3, or the paper should clearly explain why such a comparison is not necessary.

8. The presentation of some figures should be improved.

9. In Fig. 1 (right), the linear line that interpolates between the different scores of QueryMeldNet is not meaningful. The x-axis is not linear and each point on the x-axis changes more than one variable (the data volume and number of tasks), so the linear interpolation between these results is not appropriate. The plot would improve if this line were removed.

10. Fig. 1 (left) is not clear. What does the dotted rectangle represent? And what do the arrows from the datasets outside of this dotted rectangle mean? What is the difference between the datasets inside and outside the rectangle? This is not clear. The figure should be clarified, or there should be additional explanations in the caption.

11. In general, the plots in Fig. 1 (right) and Fig. 6 are so small that they are not properly readable when the paper is printed. Moreover, the text in the plots (‘QueryMeldNet’ and ‘X-Decoder’) overlaps with the y-axis ticks and labels. The readability of the figures would improve if the plots (or at least the font size) were bigger, and if the text didn’t overlap.

12. In Fig. 6 (right), QueryMeldNet is compared to OpenSeeD, and QueryMeldNet is shown to achieve a better performance. However, in this comparison, OpenSeeD and QueryMeldNet are trained on different datasets. Therefore, from this experiment, it is not clear what portion of the performance improvement is due to the quality of the datasets used for training, and what portion is due to the difference in model architecture or training strategy. To get insights in this manner, and improve the value of this comparison, an experiment could be conducted where QueryMeldNet is trained on the same 1.8M data samples as OpenSeeD, across the same 2 tasks.

13. In L133, the paper states that QueryMeldNet can be trained and evaluated on any segmentation task. However, there are several segmentation tasks for which it is not clear how QueryMeldNet would solve them, such as part-aware panoptic segmentation [b] which requires objects and their parts to be segmented jointly, or promptable segmentation like SAM [c] does. Could QueryMeldNet solve these tasks? If so, how? If not, then the proposed method would still be valuable, of course, but then the statement in L133 should be changed.

14. Tab. 1 shows that X-Decoder [d] cannot conduct foreground segmentation and object detection, while the newly proposed QueryMeldNet can. However, the paper does not explain what changes it makes (compared to X-Decoder) to enable foreground segmentation and object detection. Besides the additional use of conditional queries, are there any fundamental differences between X-Decoder and QueryMeldNet? Or is X-Decoder just not trained for foreground segmentation and object detection, while it could have been trained for that purpose without significant changes? If the paper explained this, the value of QueryMeldNet over X-Decoder would be much clearer.

15. In L252-L258, the paper describes that a benefit of the ‘query meld’ strategy is that the two types of queries can lead to a “hierarchical and interactive feature representation,” where learnable queries capture the overall structure of the scene and conditional queries refine these global queries by attending to specific parts of the image. However, the paper does not provide any evidence that this happens. Could the authors provide evidence to support this claim? If not, then it should be clear that this is something that the authors expect might happen in the model, and not a certainty.

16. There are several text-related mistakes, limiting the readability. Some examples:

17. The citations are not in the proper format. For instance, L221: “two-stage object detection framework Ren et al. (2015)” should be “two-stage object detection framework (Ren et al., 2015). To fix this, the authors should use the \citep{ref} command, instead of \citet{ref}. Sec. 4.1 of the ICLR 2025 template provides more detailed instructions.

18. L022-L023: the hyphens in “data-generating” and “tasks-drastically” should be ‘en dashes’ or ‘em dashes’.

19. L090: “and hierarchical and interactive feature representation” => “and a hierarchical and interactive feature representation” or “and hierarchical and interactive feature representations”

20. L267: “This thanks to” => “This is thanks to”

21. L314-L315: “Since all … Figure 2.” is not a proper sentence. It could be improved in various ways, e.g., by removing the word “Since”.

22. L269: “is a triplet depicts” => “is a triplet that depicts”

23. L316: “dealing open-vocabulary” => “dealing with open-vocabulary”

24. L318: “lead QueryMeldNet is scalable” could be improved by something like “make QueryMeldNet scalable”

25. L456: “each tasks” => “each task”

26. The notation in Eq. 1 is not clear. Are $**P**^c$, $**P**^b$ and $**P**^m$ tensors, as suggested by the fact that the text (L310) calls them embeddings? Or are they functions, as suggested by the fact that they seem to take $**x**_i$ as an input in Eq. 1? This is confusing, and the notation should be improved.

[a] Ke et al., "Segment Anything in High Quality," NeurIPS 2023.

[b] De Geus et al., “Part-aware Panoptic Segmentation,” CVPR 2021.

[c] Kirillov et al., “Segment Anything,” ICCV 2023.

[d] Zou et al., “Generalized Decoding for Pixel, Image, and Language,” CVPR 2023.

1. What is the specific structure of the “dual-query cross-attention”? Is it simply directly concatenating the condition queries and learnable queries together and then using them as the query of vanilla cross-attention? This part requires an explanation.
2. The author needs to explain the difference between the proposed query design and the query of OpenSeeD[1], considering both selecting queries from feature + learnable query.
3. Using synthetic data to enhance training is one of the main contributions. Will the code and synthetic dataset be open-sourced?
4. Minor writing error: In Figure 3(c), should the output of the decoder be circled with slashes instead of squares with slashes?

Some related works are missing:

[a] COCONut: Modernizing COCO Segmentation https://arxiv.org/pdf/2404.08639

[b] Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP https://arxiv.org/pdf/2308.02487

[c] OMG-Seg: Is One Model Good Enough For All Segmentation? https://arxiv.org/abs/2401.10229

[d] Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactively https://arxiv.org/pdf/2401.02955