Response to Reviewer Rvjz

Thank you so much for acknowledging the strength of our method. We have carefully considered your constructive and insightful comments and here are the answers to your concerns.

Q1. Unclear unique contributions.
Please refer to General Response-Q1.

Q2. Confusing claim that it is challenging to apply prompt learning solely to VLM for OVSS should be argued.
We would like to re-clarify the claim as:
We point out two reasons for the claim (in L.32-41): (1) Current prompt provides only image-level granularity, which restricts VLMs from performing tasks related to pixel-level visual localization. (2) A lack of an image-text relationship, which hinders the exploration of VLMs’ potential for open-vocabulary scene understanding. To demonstrate this, we compare seven current PEFT methods applied to the baseline (in L.618-638), and show the comparison in Tab.9 and 10. We show the part of Tab.9 as:

We would like to re-emphasize the conclusion as: it is challenging to apply prompt learning solely to VLM for OVSS, and our method shows significant improvement (at least 20% mIoU improvement in VOC).

Q3. Concerns between image-text relationship attention map and language-aware deep fusion in [1*].
We analyzed their differences from three perspectives as follows:
(1) Motivation: We aim to directly output pixel-level predictions using only the frozen VLM, while [1*] aim to proposal a grounded language-image pre-training method.
(2) Structure: We use a train-free method to construct our map, while [1*] uses a trainable cross-attention module to construct it; we only need to introduce our map for the image encoder, while [1*] also introduces it for the text encoder. Thus, our method is more concise and direct.
(3) Training cost: we do not need additional training sets, parameter-intensive adapter and large-scale pre-training, which are all required by [1*].
In summary, our method achieves OVSS with lower training cost and simpler structure.

[1*] Grounded Language-Image Pre-training.

Q4. Fig.1, how are these attention maps plotted?
We directly visualize the results of eq.2, ${{m}}^i = {p}^i \cdot (t \odot {g}^i)^\top$, using a heat map.

Q5. What should matrix product refer to? Element-wise or column-wise?
Let the image and text feature denote $p \in \mathbb{R}^{n\times d}$ and $t\in \mathbb{R}^{c\times d}$. The matrix product $O\in \mathbb{R}^{n\times c}$ between them $O_{ij} = \sum_{k=1}^d p_{ik}t^\top_{kj}$.

Q6. Typos. Line 67 and 69, RPN to RPM. There is potential confusion between RPM and RPN.
In fact, RPM and RPN refers to Relationship Prompt Module and Relationship Prompt Network. As illustrated in L.61-62, RPN consists of RPM and VLM.

Q7. Line 113-114, concerns about reducing parameter consumption.
We clarify the sentence of L.113-114 that our method only add 3M trainable parameters with the frozen VLM to achieve OVSS, so our method is low-cost during training. As illustrated in Tab.1, we conduct efficiency experiment to demonstrate our method indeed consume very small amount of additional trainable parameters to achieve OVSS. In addition, we add the efficiency for ADE20K and Context as:

Q8. L.113-125, refer to a figure here is better for illustration purpose.
Thanks for your suggestion.

Q9. Fig.2, what is M2oE, ITP and APG block? A bit confusing.
Actually, Fig.2 has nothing to do with these blocks. Are you referring to Fig.3? If yes, we have mentioned these blocks in the legend of Fig.3. RPM consists of M2oE, ITP and APG block. M2oE refers to multi-scale mixture-of-experts, which aims to propose multi-scale vision knowledge (L.134-150). ITP refers to image-to-pixel semantic attention, which aims to enable the image encoder to learn open-vocabulary semantics from text features (L.151-168). APG refers to adaptive prompt generation, which aims to construct the adaptive image-text relationship for each pixel (L.169-179).

Q10. Tab.4, what does the first line mean and what is the difference between the first and the second line?
RPM (the first line) denotes the combination of M2oE, ITP and APG. Therefore, RPM without M2oE (the second line) denotes the ablation about APG and ITP.

Q11. Concerns about Fig.7. Only legends of head 1 and 2 are given, while others are missing.
First, we give all heads in Fig.7 (MAD evaluation) instead of head 1 and 2, and use different color point to denote different heads; due to the number of heads is large, only the symbols for the two heads are given in the legend, and the remaining heads are indicated by ellipses (note the legend at the bottom right).
Second, MAD can be used to explore the range of attention of each attention head, similar to the receptive field in convolutional neural networks (CNNs) (e.g., Fig.7 in ViT[1*] is also represented to valuate MAD). It is a common metric. A higher point indicates a larger receptive field, and greater point spacing signifies richer feature diversity. Fig.7 show that with the guidance of relationship prompt, the deep features of VLM gradually have a wider MAD value range, which indicates fine-grained semantic properties.

[1*] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.

Q12. Tab.1, instead of params, this should be Learnable params, right?
Yes, params denotes learnable params.

Thanks for authors prompt and active replies. For Q2, I think presented results convince me a bit. However, in spite of performance gains presented in this paper and many turns of debating, I am still confused on unique contributions from this paper and remain doubts on its novelty. Authors seem to claim very high-level/vague contributions (could claimed by most multi-modal learning papers) without theoretical or empirical justifications and unique insights, which do not convince me that much. So far I am inclined to reject this paper and waiting other reviewers response for a second opinion.

Response to Reviewer Rcdd

Thank you so much for acknowledging the strength of our method. We have carefully considered your constructive and insightful comments and here are the answers to your concerns.

Q1. Lack of a discussion on region-text relationships, focusing instead on directly leveraging pixel-text relationships.
Thanks for your valuable suggestion. The discussion on region-text relationship is worthy.
First, we clarify the difference between pixel-text and region-text relationship: the former links image pixel with text description; the latter links image mask with text description. Since image mask contains more information, it can more intuitively represent things or stuff, while the image pixels are the opposite. Thus, image mask can be more easily associated with text description. For example, region-text relationship can be established by directly matching image mask and text description, while when we need to establish pixel-text relationship after featuring image mask and text description.
Second, we consider the impact of region-text on OVSS. Although constructing region-text relationship seems to be simpler, is it more effective for OVSS? As illustrated in MaskFormer[1*], pixel-level classification is not necessarily better than mask-level classification for semantic segmentation, which seems to show that region-text relationship enable the VLM to achieve OVSS better than pixel-text relationship. However, adopting region-text relationship means that VLM need to output mask proposals, which are usually output by a DERT-like[1*,2*] decoder. This shows that VLM need a DERT-like decoder, and goes against our intended goal of achieving OVSS using only VLM without any segmentation-specific networks.
Third, we consider the impact of region-text relationship on open-vocabulary segmentation. VLM with region-text relationship can output class-agnostic mask, thus making it feasible to achieve instance segmentation and object detection. This is a very interesting direction worth exploring, and it gives us some inspiration for future work.

[1*] Per-Pixel Classification is Not All You Need for Semantic Segmentation.
[2*] End-to-End Object Detection with Transformers.

Q2. There should be a discussion and experiments with other visual prompt tuning methods applied to the baseline.
Thanks for your valuable suggestion. In fact, we have discussed seven current PEFT methods (including other visual prompt tuning) applied to the baseline as illustrated in L.618-638, and show the comparison in Tab.9 and 10. We show the part of Tab.9 as:

We would like to re-emphasize the conclusion as: our method shows significant improvement (at least 20% mIoU improvement in VOC), due to that existing PEFT methods mainly focus on fine-tuning the image-supervised VLM to improve performance on classification. But, our method enables VLM to directly achieve OVSS.

Q3. Please provide a more detailed analysis on the correlation between the relational abilities of the model and its performance in open-vocabulary tasks (separating seen and unseen classes).
Thanks for your valuable suggestion. We provide our analysis as follows:
The relational capability of the model is reflected in guiding the model to make pixel-level predictions for unseen classes by establishing relationship attention between text class information and pixel-level visual information, making its performance close to that of seen classes. We visualize the relationship attention map in Fig.1. The first line denote the relationship attention maps for seen class across each layer; the last two lines for unseen classes. For seen class, the model has prior pixel-level semantic information, so the relationship attention map only needs to focus on a small number of pixels to guide the model to make class predictions for these pixels (e.g., the relationship attention map for airplane has fewer highlighted areas). For unseen class, the model lacks corresponding pixel-level semantic information, so the relationship attention map needs to focus on more pixels to provide the model with more sufficient pixel-level semantic information (e.g., the shallow relationship attention map for sheep highlights the complete semantic information). In addition, to demonstrate the guidance mechanism, we conduct related interpretability experiments (i.e., MAD evaluation), as illustrated in L.271-282. The MAD results show that with the guidance of relationship prompt, the deep features of VLM gradually have a wider MAD value range, which indicates fine-grained semantic properties. We also conduct system level comparison (separating seen and unseen classes) as shown in Tab.2 and 7.

Response to Reviewer 2wFX

Thank you so much for acknowledging the strength of our method. We have carefully considered your constructive and insightful comments and here are the answers to your concerns.

Q1. No efficiency comparison for ADE20K and Context dataset.
We have verified the efficiency on two benchmarks as illustrated in L.245-249. According to your comments, we add the efficiency for ADE20K and Context as:

Q2. The ablation study in Tab.4 do not cover ITP and APG.
We would like to re-clarify the ablation study in Tab.4. Tab.4 mainly shows the impact of M2oE, ITP, APG and LPM (we use RPM to denote the combination of M2oE, ITP and APG). Although we do not directly mark APG and ITP in Tab.4, RPM without M2oE denotes the ablation about APG and ITP. Therefore, we have conducted the component of APG and ITP. In addition, due to APG and ITP together form our relationship prompt (ablating one of them will cause bug), it is notable that APG and ITP can not be separated for ablation.

Q3. The performance is not SOTA compared to some previous works.
Thanks for your suggestions, please refer to General Response-Q2.

Response to Reviewer DSMk

Thank you so much for acknowledging the strength of our method. We have carefully considered your constructive and insightful comments and here are the answers to your concerns.

Q1. The claim that our methods achieved SOTA results lack persuasiveness.
Thanks for your suggestions, please refer to General Response-Q2.

Q2. More detailed explanation of the content in Fig.2.
Thanks for your suggestion. We will modify the description in L.63 as: Fig.2 shows the key difference between RPN and other OVSS methods. RPN makes VLM directly output pixel-level predictions by prompt learning, while other methods use VLM to assist a complex segmentation-specific networks where VLM transfers its rich knowledge to the mask proposal network by knowledge distillation or make the semantic decoder output segmentation masks by feature adaptation.

Q3. Some works that achieve OVSS without segmentation-specific networks diminish the novelty of our contribution.
Thanks for your suggestions, please refer to General Response-Q1.

Q4. Ablation studies about the hyper-parameters $\lambda_1, \lambda_2$ in eq.11.
$\lambda_1, \lambda_2$ control the weights between $\mathcal{L}\_{focal}$ and $\mathcal{L}_{dice}$. We explore three common combinations. The results is shown as:

The results show that $\lambda_1=100$ and $\lambda_2=1$ are the best.

Q5. Why using class embeddings from only text layer 12, not others?
To explore which text layer of class embeddings to use, we conduct studies as:

One-to-one denotes using the same-layer embeddings from image and text encoder, and only last denotes using the last class embeddings, i.e., layer 12. We find that the shallow text embeddings cannot effectively guide VLM to achieve OVSS. The reason is that the shallow embeddings cannot accurately represent the class information, thereby constructing wrong relationship prompt.

Q6. A color legend is needed in Fig.1
Thanks for your suggestion. We will add the color legend in the future version. The degree of attention from low to high is marked by colors from dark blue to red. The more attention, the darker red; the less attention, the darker blue.

Q7. Explanation about some symbols in Fig. 4 to 6 is needed.
We will add these explanation in caption as: $\odot$, $\otimes$ and $\oplus$ denotes element-wise product, matrix product and addition. The symbols Expand, Einsum and Mul denote expanding class dimension, element-wise product and matrix product.

Q8. Ablation studies about the scale and the number of experts used in M2oE.
Note that the scale and number of experts are same parameter, which control the size of feature maps. Due to we crop images into $512\times512$, the scale can be set to 1/8 of the input feature maps at most. Thus, the scale $s_i = \frac{1}{2^{i-1}}$, where $i \in [1,4]$, i.e., the maximum number of experts is 4. The ablation studies is shown as:

The results show that embeddings with more diverse scales can improve OVSS performance.

Q9. Is the Hadamard product in eq.2 significant? Would using a matrix product between $p^i$​ and $g^i$ instead affect the results?
We rewrite eq.2 as: ${{m}}^i = {p}^i \cdot (t \odot {g}^i)^\top$, where $p^i \in \mathbb{R}^{n\times d}$, $t\in \mathbb{R}^{c\times d}$ and $g^i \in \mathbb{R}^{1\times d}$. Please note the calculation process figure in our top-level pdf for more details.
Hadamard product have an important effect in eq.2. Eq.2 (including Hadamard product and matrix product) can construct pixel-text relationship attention map to guide image encoder to transform image-level semantic to pixel-level. Firstly, Hadamard product assigns class weights to images in a batch by fusing $t$ used to identify different classes and $g^i$ used to identify each image in a batch, thus attaining image-text relationship. Based on this relationship, matrix product gets the class distribution of a pixel, which is normalized to one. Because $p^i$ contains pixel-level visual information, eq.2 achieves pixel-text relationship. Fig.1 visualizes the guidance process from image-text to pixel-text relationship. To demonstrate the image-to-pixel guiding scheme, we also conduct related interpretability experiments (i.e., MAD evaluation, L.271-282). The MAD results show that with the guidance of relationship prompt, the deep features of VLM gradually have a wider MAD value range, which indicates fine-grained semantic properties.
Due to that we need relationship map $m^i \in \mathbb{R}^{n\times c}$, thus matrix product between $p^i$​ and $g^i$ cannot output map of this shape. In fact, we have explored matrix product between $p^i$​ and $t$ in L.291-311 and Tab.5. The results show that missing any of the product operations will have a significant impact on performance.

Q10. Why is pixel-level information used instead of patch-level information?
In fact, pixel-level information for feature maps used refers to patch-level information for original images. Due to that ViT usually divides the image into multiple patches for encoding, a pixel of the feature describes a patch of the image.

Q11. Does the proposed algorithm produce block artifacts when generating segmentation maps for high-resolution images?
No, for high-resolution images, we adopt slide inference mode, i.e., set stride to $341\times 341$, and crop size to $512\times 512$.