Q4 Comparison with Other Methods: Most of the results are compared with pre-SAM methods. I wonder how this method compares with other SAM-based semantic segmentation/instance segmentation methods, such as Grounding SAM and SEEM, etc. It would be interesting to see a comparison with other SAM-based methods and discuss the advantage of such approach.

A4: Thanks for the question. We first compare the SOTA SAM-based work Frozen Seg [2] (in our Table I, CVPR2024) with SAM-CP because that work experiments on our same setting (COCO->ADE20k). The result indicates that SAM-CP achieves superior open-vocabulary panoptic segmentation outcomes, which verifies the efficacy of composable prompts. Moreover, it demonstrates that the performance is not solely reliant on the internal capacity of SAM:

Then, We compared our method with OVSAM [1] and prompt segment anything [3] on the COCO dataset to evaluate the instance segmentation ability. All other methods need a pre-trained detector on COCO and use the detection results as the prompt for SAM. But our localization only relies on the original SAM, without tuning on COCO. The results show our performance is the best:

The comparison between our method and SEEM on the COCO dataset is listed as follows. With the same amount of backbone, we achieved higher performance. In addition to comparing with the several methods you mentioned, we also added a comparison with Semantic SAM. Even though it has point prompts in inference, our method outperforms them by a lot, which fully demonstrates that our method is superior to other SAM-based methods:

By iteratively calling our composable prompts, SAM-CP achieves the instance and semantic segmentation based on the vision foundation model SAM without mask tuning. For SAM2-CP in the first table of this question, we substitute SAM with SAM2 [5] in SAM-CP. SAM2 is faster than SAM. We utilize the CP trained on SAM-CP and transfer the weights to SAM2-CP without re-train, with the performance being 27.9 PQ. This indicates that our method is modular and can be extended to different patch generation methods (whether stronger or faster). Although the current SAM-based method is more time-consuming than the non-SAM-based method, with the improvement of the segmentation accuracy and effectiveness of the SAM-based foundational model, the new bottom-up perception paradigm designed based on its high generalization ability is valuable, which is also one of our main motivations.

[1] Open-Vocabulary SAM. ECCV 2024

[2] Frozenseg: Harmonizing frozen foundation models for open-vocabulary segmentation. In CVPR 2024.

[3] Prompt segment anything,(https://github.com/RockeyCoss/Prompt-Segment-Anything)

[4] Segment Everything Everywhere All at Once. NeurIPS 2023

[5] SAM 2: Segment Anything in Images and Videos, arxiv.org/abs/2408.00714.

[6] Semantic-SAM: Segment and Recognize Anything at Any Granularity, arXiv:2307.04767

Q5 At line 150, the authors state that the two prompts can be iteratively called when finer-level segmentation is required. I’m not entirely sure I understand this; my impression is that granularity is determined by SAM’s results. Could the authors clarify this point?

A5 A good question! We will clarify that:

• For the initial segmentation: When the Prompt I is called and the category name is "person", SAM-CP can find out all the patches that belong to person and merge them as the semantic segmentation results. When the Prompt II is called under the category name "person", SAM-CP can distinguish each two pathes whether they belong to the same instance and then the instance segmentation result is obtained.

• For finer segmentation： In order to segment the part 'Arm', the person in the original image is cropped with the instance segmentation map (or the bounding box). Enlarge the sub-image to conduct extra SAM everything-mode segmentation, and more parts will be obtained. Then, the prompt-I and prompt-II are iteratively called to achieve the semantic and instance segmentation of 'Arm'.

Q6 How do the authors view this approach in comparison to SAM-based generalized segmentation? Most other approaches appear to be training-free and straightforward to use. It would be interesting to discuss the advantages that SAM-CP offers.

A6: A good question! Indeed, there are some training-free methods to implement open vocabulary now, but most of them use SAM for segmentation and CLIP for classification, which can only achieve semantic segmentation. They cannot achieve instance segmentation. Also, the current segmentation quality is also not good enough, so, tuning is also needed for better performance, especailly in vertical field. In our response to Question 4, a detailed comparison was drawn between the advantages of our method and those of other SAM-based approaches. It is worth highlighting that all of these renowned SAM-based methods necessitate training. Specifically, certain methods train prompt generators to furnish prompts to SAM, while others require fine-tuning SAM on extensive datasets. Arguably, these methods cannot be regarded as truly training-free. In sharp contrast, our method attains state-of-the-art (SOTA) performance without the need to fine-tune SAM, which stands as a prominent and distinctive advantage of our approach.

Moreover, the majority of these methods are founded on the combination and enhancement of prior techniques. Nevertheless, our method is fundamentally differentiated in its conceptual framework. We embrace a bottom-up strategy, whereby the global information of the target is discerned from local observations. This approach diverges substantially from the conventional up-bottom methods. The triumph of our method implies that this novel concept harbors even more substantial potential for development within the domain of segmentation.

Q1 The advantage of SAM lies not only in its ability to segment an entire image but also in its prompt-based segmentation, which introduces more accurate masks. I think this method uses SAM only as an initial “superpixel” generator, which somewhat overlooks SAM’s full potential.

A1 A good question! We concur that the strength of SAM is not just confined to its capacity to segment entire images but also extends to its prompt-based segmentation capabilities, which yield more precise masks. However, we do not underestimate SAM's comprehensive potential. Firstly, it is valuable in performing non-prompt-based segmentation, for example, to extract more efficient visual features， which uses grid points to extract all potential patches as many as possible for recognition. Secondly, when our approach offers two additional types of prompts which can be integrated with SAM's prompts for more flexible segmentation -- an interesting example comes from the scenario that an instance is partially occluded and split into patches; in this situation, our Type-II prompt can provide additional clues (whether these patches belong to the same instance) for the top-level model to make the final decision.

Q2 Generalization: The proposed method requires training on specific datasets to learn how to merge patches, which seems to be not very generalizable. This approach may not translate well from natural images to medical images or even from natural images with large object segments to tasks requiring finer, more detailed segmentation. Is this the case in this setting? Would like to hear author feedback on this point.

A2 A good question! To further show the advantages of our SAM-CP, we conduct experiments on more perspectives. Due to the time limitation, we experiment with part segmentation and out-of-domain generalizations：

For part segmentation, we experimented the part segmentation on Pascal Part datasets. Pascal Part has 4465 images with 93 part categories. The performance is listed as follows:

We also provide some part segmentation visualization in Figure 16 of the main PDF (rebuttal section, page 24). It's important to note that SAM's predictions are primarily based on low-level information such as color, rather than semantics. For instance, SAM might separate the cuffs of clothing from the skin of the arms, even though the cuffs are still part of the arms. Since our approach builds upon SAM's results, it inherits some of these limitations. In the future, further tuning of SAM for this type of data could lead to more refined part segmentation.

For out-of-domain generalization, we conducted experiments on camouflage object segmentation and provided some visualizations in Figure 17 of the main PDF (rebuttal section, page 24).

Q3 Splitting and Merging: The proposed method only focuses on merging, which to some degree simplifies the problem. I believe splitting is equally important as merging, though it seems it is not within the scope of this method.

A3 A good question! Merging patches with two prompts is the main idea of SAM-CP, but we have also taken the "splitting" into consideration by interactively calling two prompts. With an instance segmentation result, if a human wants to split it for fine-grained segmentation, they can crop the sub-image with the segmentation map and enlarge it for extra SAM everything-mode segmentation. Then, more details can be segmented, as shown in Figure 19 in the main PDF (rebuttal section, page 25). When the foundation models become stronger in the future, these two prompts can be called interactively for fine-grained segmentation.

Q4 Although SAM-CP's speed heavily relies on SAM, the paper lacks a clear analysis of the computational efficiency of the proposed non-SAM modules. It is crucial to examine whether these modules actually enhance efficiency. Although the speed is influenced by SAM's computational efficiency, can you provide a detailed analysis of the speed of non-SAM modules and compare it with other SAM-based methods? Alternatively, could replacing SAM with a lightweight variant like EfficientSAM provide sufficient performance and computational efficiency for comparison with non-SAM methods?

A4 A good question! We opted for two state-of-the-art (SOTA) methods, namely FrozenSeg (which is SAM-based and presented at CVPR 2024) and FCCLIP (a non-SAM-Based method from NeurIPS 2023), to compare their time performance within our open-vocabulary setting. All experiments were carried out using an RTX 4090. The outcomes are as follows:

FrozenSeg, being a SAM-based approach, also requires everything-mode segmentation and thus demands the same SAM time as ours. However, our non-SAM module consumes less time and exhibits better performance. FCCLIP (the performance here is a reproduced version by both FrozenSeg and us, which is lower than that in the original paper) is a non-SAM-based method, while SAM-CP demonstrates better segmentation performance. In SAM2-CP, we substitute SAM with SAM2[2] in SAM-CP. SAM2 is faster than SAM. We utilize the CP trained on SAM-CP and transfer the weights to SAM2-CP without re-train, with the performance being 27.9 PQ. This indicates that our method is modular and can be extended to different patch generation methods (whether stronger or faster). Although the current SAM-based method is more time-consuming than the non-SAM-based method, with the improvement of the segmentation accuracy and effectiveness of the SAM-based foundational model, the new bottom-up perception paradigm designed based on its high generalization ability is valuable, which is also one of our main motivations.

[1] SAM 2: Segment Anything in Images and Videos, arxiv.org/abs/2408.00714.

[2] Frozenseg: Harmonizing frozen foundation models for open-vocabulary segmentation. In CVPR 2024.

[3] Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional CLIP. In NeurIPS, 2023. [1] SAM 2: Segment Anything in Images and Videos, arxiv.org/abs/2408.00714. [2] Frozenseg: Harmonizing frozen foundation models for open-vocabulary segmentation. In CVPR 2024. [3] Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional CLIP. In NeurIPS, 2023.

Q5 Figure 1's many colors and mixed visual elements are hard to read. The description of patches needs greater specificity. For example, as vaguely mentioned in Line 139, it's not specified if the patches are segments from SAM's segment-everything mode or just features from SAM's backbone.

A5 Thanks for the kind reminder. We will clarify the colors and mixed visual elements for better understanding. We give a detailed description of "patches" in Line 139：the patches are segments from SAM's segment-everything mode.

Q6 Does SAM-CP offer any advantages in training time compared to other SAM-based methods?

A6 A good question! The training time is our advantage. For Table I, we only need 12 epoch to achieve our performance while other methods FCCLIP （non-SAM-based） and FrozenSeg (SAM-based) need 36 epoch. This shows that our approach is better at achieving convergence.

Q7 Since SAM's segmentation results may not always be precise, the Affinity matrix used by the Unified Affinity Decoder could potentially accumulate errors. How is this issue addressed?

A7 Thanks for the question. Indeed, our approach can inherit the inaccurate segmentation mask of SAM, but it will not accumulate errors (e.g., making errors larger). To reduce the inaccuracy, one can either use a stronger foundation model (e.g., SAM-HQ or SAM2) or apply a post-processing algorithm to refine the boundary. In the Table of Q4, the results of SAM2 show the potential， we hope with the improvement of the segmentation accuracy and effectiveness of the SAM-based foundational model, the new bottom-up perception paradigm designed based on its high generalization ability is valuable, which is also one of our main motivations.

Q1 Prompt I classifies each SAM-segmented patch, potentially leading to ambiguities. For example, without contextual information, it’s challenging to determine if a patch representing clothing on a person should be classified as part of the person or as an independent clothing item. I would like to know if SAM-CP has effective strategies for addressing the aforementioned issues of segmentation ambiguity?

A1 Good question! This is exactly why the Type-II prompt was designed in its current form. Note that the Type-II prompt is to judge "whether two patches belong to the same instance given a specified category label." In the aforementioned example, whether the two patches (clothing and arm) belong to the same instance depends on the category label -- the answer is yes when the label is "person" but no when the label is "clothing." So, SAM-CP is naturally designed to avoid the ambiguity.

In further detail, here is an illustration of how two types of prompts interact to solve the above case.

• For a specified category "person", when the Type-I prompt is called, both the clothing and arms (of one person) are assigned affinity to this category and classified as "person." Then, for the patches found, the Type-II prompt is called, and the clothing and the left and right arms are considered belonging to the same "person". The "person" instance thus occupies both patches and, if necessary, feeds into a category with finer granularity (see below).
• For a specified category "clothing", when the Type-I prompt is called, the clothing patch is assigned affinity to this categoty and classified as "clothing", while the left and right arms (of one person) are not. This can achieve the finer granularity classification for those patches belong to "Person".
• For a specified category "arm", when the Type-I prompt is called, both the left and right arms (of one person) are assigned affinity to this category and classified as "arm," while the clothing patch is not. Then, for the patches classified as 'arm', the Type-II prompt is called, and the left and right arms are considered not belonging to the same "arm". Thus, these two patches are separated into two individual instances of "arm."

Q2 The performance gains are not significant, such as mIoU for the COCO->ADE20K and COCO->Cityscapes settings in Table 1.

A2: Thanks for the question. SAM-CP applies a bottom-up mechanism to merge patches, which is inherently friendly to instance segmentation. On the other hand, we find that SAM itself is more robust to instances than semantic regions. In particular, for some large semantic regions (, e.g., sky, mountain, tree, , etc.), SAM can suffer from inaccurate boundaries because these regions are not easily prompted. We hope that with the improvement of the segmentation accuracy and effectiveness of the SAM-based foundational model, the new bottom-up perception paradigm designed based on its high generalization ability is valuable, which is also one of our main motivations.

Q3 Table 2's comparison may be unfair due to the use of SAM, which is trained on a significantly larger scale. This complicates attributing improvements directly to the proposed method.

A3: A good question! Really, SAM has undergone extensive training on a significantly larger scale dataset. However, without fine-tuning in closed domain datasets, the quality of SAM's mask is actually inferior to those trained in COCO. The following table compares the mean Intersection over Union (mIoU) and missing rates with respect to different IoUs for COCO (val2017) instance segmentation.

(Here, mIoU refers to the IoU between the highest-IoU proposal and the ground truth, and mIoU${>0.5}$ indicates that we calculate the mIoU only for instances where the best proposal has an IoU higher than 0.5. MR${x}$ represents the proportion of instances without a matched proposal exceeding an IoU of $x$. )

The table reveals that SAM patches have a higher missing rate compared to the fine-tuned Mask DINO on COCO. Our SAM-CP, which is based on SAM patches without any fine-tuning, thus faces limitations in close-domain performance. However, when SAM-CP merges patches, it achieves better mIoU and lower missing rates, especially MR$_{0.75}$ (39.3% -> 33.3%). This indicates that after applying composable prompts, SAM-CP delivers improved segmentation outcomes. This also shows that the current performance comes from composable prompts we designed rather than only depending on SAM.

Q5: SAM can generate fine-grained masks for part, e.g. head, hand, body. Does SAM-CP have the potential to work on open-vocabulary part segmentation? Quantitative experiments and comparisons to related works like HIPIE would enhance the applicability of this work.

A5: A good question! It's impressive to see that SAM-CP demonstrates superior open vocabulary capabilities compared to HIPIE on the ADE20K dataset, despite being trained on a smaller set of data. This comparison highlights the potential of SAM-CP's approach in handling diverse and novel categories that may not be present in the training data.

• Open Vocabulary Comparison on COCO->ADE20K Dataset: In this setting, SAM-CP was trained on the COCO dataset, while HIPIE was trained on a more extensive collection of datasets, including Object 365, COCO, RefCOCO, Pascal, etc., before being tested on the ADE20K dataset. Despite the smaller training set, SAM-CP outperforms HIPIE in open vocabulary panoptic segmentation on ADE20K. This suggests that SAM-CP may be more effective in generalizing to new scenes and objects. |Method|Backbone|PQ|mAP|mIoU| | - | - | - | - | - | |HIPIE|ViT-H|22.9|19.0|29.0| |SAM-CP(ours)|Swin-L|27.2|17.0|31.8|

• Part Segmentation on Pascal Part Dataset: Because time limitation of rebuttal, we only present the potential of part segmentation without further tuning. For part segmentation, SAM-CP was tested on the Pascal Part dataset, which consists of 4465 images across 93 part categories. The performance metrics are as follows: |Part segmentation |mAP|AP$_{50}$|AP$_{75}$|AP$^{s}$|AP$^{m}$|AP$^{l}$| |-|-|-|-|-|-|-| |SAM-CP|13.7|30.5|10.6|8.3|21.1|24.2|

We also provide some part segmentation visualization in Figure 16 of the main PDF (rebuttal section, page 24).

Q2: The advantages of the proposed framework are not well discussed or presented. The performances are only comparable to previous SOTA. This paper should look for more perspectives to highlight its advantages, e.g. efficiency, part segmentation (also see Question 1), few-shot capabilities, out-of-domain generalizations, etc.

A2: A good question! To further demonstrate the strengths of our SAM-CP, we have conducted experiments from multiple angles. Due to time constraints, we focused on part segmentation and out-of-domain generalization:

For part segmentation, we conducted experiments on the Pascal Part dataset, for instance, segmentation tasks. The Pascal Part dataset comprises 4465 images across 93 part categories. The performance metrics are as follows:

We also provide some part segmentation visualization in Figure 16 of the main PDF (rebuttal section, page 24). It's important to note that SAM's predictions are primarily based on low-level information, such as color, rather than semantics. For instance, SAM might separate the cuffs of clothing from the skin of the arms, even though the cuffs are still part of the arms. Since our approach builds upon SAM's results, it inherits some of these limitations. In the future, further tuning of SAM for this type of data could lead to more refined part segmentation.

For out-of-domain generalization, we conducted experiments on camouflage object segmentation and provided some visualizations in Figure 17 of the main PDF (rebuttal section, page 24).

Q3: It could be more convincing if this paper can compare to other SAM-based methods on the same settings and show its advantages. For example, [1] and [2] also work on open-vocabulary segmentation with SAM.

A3: Thanks for the question. We first compare the SOTA SAM-based work Frozen Seg (in our Table I, CVPR2024) with SAM-CP because that work experiments on our same setting(COCO->ADE20K). The result indicates that SAM-CP achieves superior open-vocabulary panoptic segmentation outcomes, which verifies the efficacy of composable prompts. Moreover, it demonstrates that the performance is not solely reliant on the internal capacity of SAM:

Then, We compared our method with OVSAM on the COCO dataset. Their open-vocabualry setting is differen ty from ours, because the time limition of rebuttal, we compare with them on closed domain at the same setting to show the model ability. We found that OVSAM did not perform as well as our method on both R50 base backbones:

The comparison between our method and SEEM on the COCO dataset is listed as follows. With the same amount of backbone, we achieved higher performance. In addition to comparing with the several methods you mentioned, we also added a comparison with Semantic SAM. Even though it has point prompts in inference, our method outperforms them by a lot, which fully demonstrates that our method is superior to other SAM-based methods:

[4] Open-Vocabulary SAM. ECCV 2024

[5] Segment Everything Everywhere All at Once. NeurIPS 2023

[6] Semantic-SAM: Segment and Recognize Anything at Any Granularity, arXiv:2307.04767

Q4: As already discussed and illustrated in the paper, SAM-CP fails at small objects since there are no prompts on small objects. The efficiency is limited by SAM.

A4: A good question! We inherit the default option of SAM that starts with placing a fixed grid of point prompts on the image. When the target is small, it is possible that none of the prompts fall on the target, and thus, it is missing. This issue can be alleviated by dynamically adding denser point prompts to the regions with small objects (no re-training is required, but this strategy will slow down the inference). We show an example in Figure 18 of the main PDF (rebuttal section, page 25), where small targets were found with this simple mechanism. Figure 19 of the main PDF (rebuttal section, page 25) shows us that SAM is not friendly to small targets only because they occupy a relatively small proportion in the current image and the preset grid points of SAM cannot cover them well. Once we enlarge the small targets, SAM can also automatically segment them well.

Q1: The experiments on open-vocabulary segmentations (Table 1) are not well supportive. SAM-CP combines SAM and CLIP but is only comparable to FC-CLIP that is built with CLIP only. Also, the experiments on closed-vocabulary segmentations (Table 2) do not best address the potential of combining SAM and CLIP since it is worse than CLIP-only X-Decoder and supervised SOTA MaskDINO.

A1: Thanks for the question! We respond this question with experiments on Open domain and close domain:

For open domain: In the following table, we present a comparative analysis of two state-of-the-art (SOTA) models, FCCLIP [3] (CLIP-based, presented at NeurIPS 2023) and FrozenSeg [2] (CLIP&SAM-based, presented at CVPR 2024), within the COCO->ADE20K open-vocabulary setting. The initial row reflects the performance metrics reported for FCCLIP in its original publication. The subsequent two rows display the reproduced performance of FCCLIP as reported by the FrozenSeg paper and our own reproduction, respectively. Our method significantly outperforms both FCCLIP and FrozenSeg in terms of results, thereby achieving the SOTA. In SAM2-CP, we substitute SAM with recently published SAM2 in SAM-CP without re-training. SAM2 [1] is faster than SAM. We utilize the CP trained on SAM-CP and transfer the weights to SAM2-CP without re-train, with the performance being 27.9PQ. This indicates that our method is modular and can be extended to different patch generation methods (whether stronger or faster). Although the current SAM-based method is more time-consuming than the non-SAM-based method, with the improvement of the segmentation accuracy and effectiveness of the SAM-based foundational model, the new bottom-up perception paradigm designed based on its high generalization ability is valuable, which is also one of our main motivations.

[1] SAM 2: Segment Anything in Images and Videos, arxiv.org/abs/2408.00714.

[2] Frozenseg: Harmonizing frozen foundation models for open-vocabulary segmentation. In CVPR 2024.

[3] Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional CLIP. In NeurIPS, 2023.

For closed domain: In the realm of open-vocabulary tasks, SAM-CP has achieved state-of-the-art (SOTA) performance. However, in the closed-vocabulary scenario, our SAM-CP exhibited relatively lower performance in closed-set benchmark testing. The following table, which presents statistically analyzed data from the COCO dataset, delves into the reasons behind this. Primarily, this is because we chose not to fine-tune the masks of the visual foundation model SAM, unlike other methods, such as Mask DINO, which were trained on COCO.

(The table compares the mean Intersection over Union (mIoU) and missing rates with respect to different IoUs for COCO (val2017) instance segmentation. Here, mIoU refers to the IoU between the highest-IoU proposal and the ground truth, and mIoU${>0.5}$ indicates that we calculate the mIoU only for instances where the best proposal has an IoU higher than 0.5. MR${x}$ represents the proportion of instances without a matched proposal exceeding an IoU of $x$. )

The table reveals that SAM patches have a higher missing rate compared to the fine-tuned Mask DINO on COCO. Our SAM-CP, which is based on SAM patches without any fine-tuning, thus faces limitations in close-domain performance. However, when SAM-CP merges patches, it achieves better mIoU and lower missing rates, especially MR$_{0.75}$ (39.3% -> 33.3%). This indicates that after applying composable prompts, SAM-CP delivers improved segmentation outcomes.

Q3: The figures are low-resolution and notably blurry. Replacing them with high-resolution versions would improve readability and presentation quality.

A3: Thanks for the kind reminder. We will change the figures to a higher-resolution version to make them clearer.

Q4: The performance of SAM-CP on closed-domain segmentation is not superior over other baselines.

A4: Thanks for the question. In the realm of open-vocabulary tasks, SAM-CP has achieved state-of-the-art (SOTA) performance. However, in the closed-vocabulary scenario, our SAM-CP exhibited relatively lower performance in closed-set benchmark testing. The following table, which presents statistically analyzed data from the COCO dataset, delves into the reasons behind this. Primarily, this is because we chose not to fine-tune the masks of the visual foundation model SAM, unlike other methods, such as Mask DINO, which were trained on COCO:

(The table compares the mean Intersection over Union (mIoU) and missing rates with respect to different IoUs for COCO (val2017) instance segmentation. Here, mIoU refers to the IoU between the highest-IoU proposal and the ground truth, and mIoU${>0.5}$ indicates that we calculate the mIoU only for instances where the best proposal has an IoU higher than 0.5. MR${x}$ represents the proportion of instances without a matched proposal exceeding an IoU of $x$.)

The table reveals that SAM patches have a higher missing rate compared to the fine-tuned Mask DINO on COCO. Our SAM-CP, which is based on SAM patches without any fine-tuning, thus faces limitations in close-domain performance. However, when SAM-CP merges patches, it achieves better mIoU and lower missing rates, especially MR$_{0.75}$ (39.3% -> 33.3%). This indicates that after applying composable prompts, SAM-CP delivers improved segmentation outcomes. We hope that with the improvement of the segmentation accuracy and effectiveness of the SAM-based foundational model, the new bottom-up perception paradigm designed based on its high generalization ability is valuable, which is also one of our main motivations.

Q5 The design of SAM-CP relies solely on SAM's final predictions, disregarding intermediate features from SAM that could potentially enhance the decoder's performance. It would be valuable to include discussions on this aspect and to investigate whether integrating SAM's intermediate features with the proposed decoder could yield improved results.

A5 A good question! We've observed a significant decrease in performance when attempting to directly integrate SAM's intermediate features with our proposed decoder. Due to the time constraints of the rebuttal period, we plan to delve deeper into this issue post-rebuttal. For now, let's provide some analysis:

As depicted in Figure 15 of the main PDF (rebuttal section, page 24), a t-SNE comparison between SAM and SAM-CP reveals that SAM's features are class-agnostic. SAM predominantly focuses on low-level details such as color and boundary, which are insufficient for the model to distinguish between different categories. Consequently, a straightforward integration of SAM's intermediate features with the proposed decoder is not effective. Our approach of utilizing CLIP for classification and SAM for segmentation embodies the concept of a Mixture of Experts (MoE). However, it's worth investigating whether SAM's features could assist in calculating affinities and facilitate patch merging, which could potentially enhance the overall performance of our model.

Q1: The running time might be a potential issue due to the two-level prompt design (especially if an iterative approach is used). A comparison of running time performance with the original SAM and other segmentation models would be informative.

A1: A good question! We opted for two state-of-the-art (SOTA) methods, namely FrozenSeg (SAM-based) and FCCLIP (a non-SAM-Based) , to compare their time performance within our open-vocabulary setting. All experiments were carried out using an RTX 4090. The outcomes are as follows:

FrozenSeg, being a SAM-based approach, also requires everything-mode segmentation and thus demands the same SAM time as ours. However, our non-SAM module consumes less time and exhibits better performance. FCCLIP (the performance here is a reproduced version by both FrozenSeg and us, which is lower than that in the original paper) is a non-SAM-based method, while SAM-CP demonstrates better segmentation performance. In SAM2-CP, we substitute SAM with SAM2[2] in SAM-CP. SAM2 is faster than SAM. We utilize the CP trained on SAM-CP and transfer the weights to SAM2-CP without re-train, with the performance being 27.9 PQ. This indicates that our method is modular and can be extended to different patch generation methods (whether stronger or faster). Although the current SAM-based method is more time-consuming than the non-SAM-based method, with the improvement of the segmentation accuracy and effectiveness of the SAM-based foundational model, the new bottom-up perception paradigm designed based on its high generalization ability is valuable, which is also one of our main motivations.

[1] SAM 2: Segment Anything in Images and Videos.

[2] Frozenseg: Harmonizing frozen foundation models for open-vocabulary segmentation. In CVPR 2024.

[3] Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional CLIP. In NeurIPS, 2023.

Q2: Experiments: The current baselines do not include models based on SAM. Therefore, it's unclear whether the performance improvements come from the composable prompts or the internal capacity of SAM. Adding baselines building on SAM that perform instance [1] and semantic [2] segmentation can be incorporated to further demonstrate the effectiveness of the proposed composable prompts.

A2: Thanks for the question. We first compare the SOTA SAM-based work Frozen Seg (in our Table I, CVPR2024) with SAM-CP because that work experiments on our same setting (COCO->ADE20K). The result indicates that SAM-CP achieves superior open-vocabulary panoptic segmentation outcomes, which verifies the efficacy of composable prompts. Moreover, it demonstrates that the performance is not solely reliant on the internal capacity of SAM.

Since Prompt Segment Anything and Semantic Segment Anything are merely engineering projects available on GitHub and lack specific papers for support, our comparison with them is solely based on the datasets and settings utilized in these projects: Instance Segmentation: In the Prompt Segment Anything project, the segmentation performance is derived from the pretrained-detector H-Deformable-DETR, with the detection bounding-box results employed as prompts for segmentation. It is evident that our method outperforms Prompt Segment Anything in the Res-50 scenario. However, in the case of the Swin-L backbone, due to the SAM patches not being tuned on COCO, SAM-CP might be marginally inferior. But this is also an advantage in that when there is no additional tuning, we can achieve a composite result just by combining the results of the visual foundational model. The instance segmentation performance details are presented as follows:

Semantic segmentation: In the Semantic Segment Anything project, only its performance on the ADE20k dataset has been reported. We adjusted our method to the same settings for comparison. It's worth mentioning that, as mentioned above, our method has no extra tuning, we would like to propose a new bottom-up perception paradigm of "foundation model+ composable prompts" .

We are sincerely grateful for your thoughtful and comprehensive review of our paper. We have attentively dealt with each of the concerns you raised and highly esteem your valuable constructive feedback, as it has been instrumental in enhancing our work. As the Author-Reviewer discussion period is nearing its conclusion, please feel free to share any additional concerns or suggestions—we would be more than happy to address them. Thank you.

This message is because there is only one day left for the Author-Reviewer discussion period, yet we have not received your response. After our rebuttal, the other two reviewers raised their scores, and all other reviewers leaned towards acceptance, which indicates that our response has resolved their issues. However, we have not received your reply. We understand that this may be due to your busy schedule, but we would like to know if we have addressed your concerns. We are also very willing to fully discuss with you to polish our paper. This is a friendly message. Thanks again.