OPMapper: Enhancing Open-Vocabulary Semantic Segmentation with Multi-Guidance Information

Dear reviewer WxDe,

We sincerely thank you for your thoughtful comments, valuable insights, and the considerable time and effort you have dedicated to carefully reviewing our manuscript. The concerns and suggestions raised by you have provided us with an excellent opportunity to significantly enhance the quality and clarity of our paper. In the following, we address each concern point-by-point and present additional analyses and experiments to thoroughly resolve your concerns.

1. W1: More training cost analysis

We thank the reviewer for raising concerns regarding training efficiency. It is worth noting that training-based approaches typically require updating the backbone or relatively heavy fusion/segmentation modules, which constitute the most time- and memory-intensive components. In contrast, our method, when training OPMapper, only requires training two lightweight modules, CAI and SAA.

Both CAI and SAA are designed to be extremely efficient, such that the entire training can be completed with less than 2GB of GPU memory while processing 8 images per GPU. Therefore, the difference in GPU memory consumption between using KL divergence and pixel-wise cross-entropy is practically negligible in our current setting. Moreover, pixel-aware cross-entropy loss is also not well-suited for measuring the discrepancy between two distributions.

Under current settings, the training takes approximately 3 hours with ViT-B and 4 hours with ViT-L. Furthermore, once trained, these modules can be seamlessly integrated into other models without additional retraining.

To provide a more comprehensive comparison, we present the training costs of representative training-based methods in the table below. The results clearly indicate that, while enhancing generalizability, our method achieves a substantial reduction in training costs, including both training time and GPU memory consumption.

2. W2: In-Depth Analysis of the Motivation

We first clarify the motivation of this work: existing methods, when computing attention weights, primarily emphasize local compactness (q-q, k-k or q-q&k-k attention) while neglecting or down-weighting global connectivity. To address this limitation, we introduce the CAI module, which explicitly integrates both local compactness and global connectivity to directly output pixel-aware attention maps. In parallel, we employ the SAA module to guide CAI in preserving CLIP’s inherent cross-modal alignment, thereby enabling open-vocabulary segmentation.

To validate the drawbacks of other methods in regard to global connectivity and demonsrate that our method better accounts for global connectivity compared to existing approaches, we conducted the following experiment on 50 randomly selected images. Specifically, using the attention matrices from training-free methods such as MaskCLIP and ProxyCLIP, we measured the attention value between each token and same-category tokens at varying spatial distances. We defined tokens as “long-range” if their spatial distance exceeded the average distance from the given token to all same-category tokens. For each image, we aggregated the attention values of these long-range tokens and computed their total value, followed by averaging across the entire image to obtain the mean long-range attention value. A higher value indicates greater emphasis on global connectivity. The results are presented in the table below.

As shown in the table, our method achieves the highest mean long-range attention strength, reaching 0.373, while most other methods fall within the range of 0.08 to 0.15, with the maximum being only 0.289 (achieved when incorporating external models such as DINO to provide global connectivity). These results demonstrate that our method effectively captures more information from same-category but spatially distant tokens, thereby ensuring more comprehensive global connectivity. This further substantiates our argument that existing methods tend to emphasize local compactness while underrepresenting global connectivity, whereas our CAI module explicitly enhances global connectivity, thereby achieving a more effective construction of attention relationships.

In addition, we kindly refer the reviewer to Figure 2 in our supplementary material. There, we visualize both our designed attention weights and the v-v attention produced by DINO. To facilitate a clearer comparison, we applied brightness enhancement (so that the sum of each row is not equal to 1). We also sincerely apologize for the typo appearing in the Figure 2 and will correct this in our paper. From the visualization, it is observed that in DINO’s v-v attention, the brighter values predominantly appear near the diagonal of the attention matrix, indicating that local compactness is the dominant factor when merging information from other tokens, even though global connectivity is heavily emphasized during DINO’s training compared with other training-free methods.

3. W3: Distribution of $W_{qk}$ and $W_{p}$ in Eq 6

Thank you for highlighting this important point. We agree that clarification regarding the normalization of Eq. (6) is necessary. Specifically, the components $W_l$ and $W_g$ in Eq. (6) are indeed normalized distributions: $W_l$ is normalized via a standard softmax operation (line 173), ensuring each row sums to 1. Similarly, $W_g$ is constructed as a uniform distribution (line 179) among pixels of the same category, which can also be viewed as a special case of softmax normalization, guaranteeing each row sums exactly to 1 as well.

Since the weights in Eq. (6) (i.e., $\lambda$ and $1-\lambda$) sum to 1, the resulting combined attention matrix $W_{qk}$ is inherently normalized, ensuring each row also sums to 1. Regarding the predicted attention $W_p$ , it is also normalized via a softmax operation during training. However, we acknowledge that the explicit softmax symbol was omitted in the manuscript, potentially causing confusion. We will clearly include this explicit normalization step in future manuscript revisions to avoid any misunderstanding.

4. W4: The choice of $\lambda$

We have conducted additional experiments on the choice of $\lambda$ in Table 2 of the supplementary material to further validate the rationale behind our selection. We kindly invite the reviewer to refer to these results for more details. Moreover, following the reviewer’s suggestion, we also performed a quick experiment by treating $\lambda$ as a learnable scalar. The result shows that when $\lambda = 0.3372$, the average performance reaches 41.9, which is very close to the manually set value of 0.3. This further substantiates the effectiveness of our ablation study.

Regarding high-frequency regions (e.g., boundaries), since they violate the assumption of local smoothness, the corresponding local weight $\lambda$ is set to be smaller in these areas.

5. W5: The design of SAA

First, we would like to clarify that our method does NOT rely on providing a trained segmentation module to achieve the final segmentation. The ONLY component inserted into other methods is the CAI module.

Second, we address the question of why we designed the SAA module instead of simply using an MLP. During the training of CAI, the most direct supervision signal comes from our manually constructed “ideal” attention matrix, i.e., the q-k weight map, which serves as an explicit form of supervision. The purpose of SAA, however, is to provide an indirect, implicit supervisory signal. Specifically, the predicted q-k weights are combined with the original v from the frozen CLIP to form the input to SAA, which then generates a segmentation map for loss computation. In the process of transforming input into output, SAA iteratively performs image-to-text and text-to-image fusion, ensuring proper alignment of the two modalities in CLIP. This alignment is propagated back to CAI through the predicted q-k weights. Therefore, SAA functions more like a mechanism for computing the loss, enabling CAI outputs to not only achieve the transition from object-level to pixel-level representations but also preserve the original cross-modal alignment. A simple MLP cannot fulfill this design objective.

To further verify this, we conducted an experiment where we replaced SAA with MLPs of approximately the same parameter size, applying it to transform the q-k weights, while keeping the subsequent segmentation generation process identical. When using ViT-B, this model achieved an average mAP of 40.5 across five datasets (ADE, Cityscapes, Context59, Object, and VOC-20). In comparison, the CAI+SAA combination achieved an average mAP of 42.2, while CAI without SAA yielded 41.3. These results demonstrate that directly using an MLP is suboptimal, as it even suppresses part of the improvement brought by CAI itself (41.3 → 40.5).

6. W6: Scaling experiments

Following reviewer's advice, we conducted a model scaling experiment by inserting the ViT-L version of CAI into Cat-Seg. As shown, this integration led to improvements of +0.6, +1.5, +0.4, and +0.5 mIoU on the VOC, Context-59, Stuff-171, and ADE-150 benchmarks, respectively.

We hope that our response has satisfactorily addressed your concerns, and we thank you again for your thoughtful feedback.

Best regards,

Authors

2. The design and interpretability of the ideal attention map

Your question is indeed very meaningful. However, we would like to clarify that our attention map is not a self–self attention. Instead, it is computed from the query and key after transformation by CAI.

Next, I will address the inspiration behind our design. This point was also discussed in our Response 1 to Reviewer ohL3, and for completeness, I will reproduce that explanation here.

Our design is not a simple intuition but fundamentally grounded on two well-established theoretical principles widely recognized in semantic segmentation literature:

1. Local Compactness: Fundamentally, semantic segmentation can be conceptualized as a problem of spatially continuous local clustering, wherein adjacent pixels are not independent but instead exhibit strong categorical coherence due to the inherent spatial smoothness of natural images. This property underscores the expectation that meaningful segmentation should leverage local contextual regularities, thereby reinforcing the robustness and consistency of the learned representations. This is commonly agreed in classical segmentation literature (FCN [Long et al., CVPR’15], DeepLab [Chen et al., TPAMI’18]). In our paper, Equation 4 explicitly incorporates Euclidean distance into attention computation to reinforce local semantic smoothness.

2. Global Connectivity: High-quality semantic segmentation fundamentally relies on the capacity to capture long-range semantic dependencies, beyond local contextual cues. This capability ensures that spatially distant pixels of the same semantic category—such as occluded regions of an identical object—are jointly recognized and consistently segmented (AAF [Ke et al., ECCV'18]). Without modeling such non-local associations, segmentation results risk fragmenting semantically coherent regions, thereby undermining both accuracy and robustness. Vision transformers (ViT, Swin Transformer) have also demonstrated this principle in modeling semantic relations between distant tokens and our Equation (5) precisely formulates a semantic-aware global connectivity pattern that aligns perfectly with this principle.

To further address the reviewer’s concern with rigorous empirical evidence, We conducted the following experiment: we randomly selected 50 images containing open-set categories. For each image, we generated three attention maps using our proposed construction method, Cat-Seg (which was trained on other close-set categories but not trained on these open-set categories), and Mask2Former (which was trained on both close-set and open-set categories, and thus its attention map is treated as the "optimal" one), respectivlly. We then computed both the average cosine similarity and average Structural Similarity Index (SSIM) to measure the similarity between two attention map. As shown in the table:

Ours (prior attention matrix), average cosine similarity = 0.834 and average SSIM = 0.808, shows a consistently high similarity compared to Cat-Seg, rigorously demonstrating that our prior attention construction is much more similar to the fully-supervised attention structure.

Furthermore, as clearly shown in Supplementary Figure 2 and the Figure 3 of our paper, the attention patterns generated by CAI supervision remarkably resemble those produced by vision foundation models (e.g., DINO), providing strong qualitative validation, and the segmentation results remain sharp along boundaries and coherent across object parts compared with other methods. These analyses collectively provide rigorous justification beyond mere intuition.

We sincerely hope that our response will gain your recognition and foster further constructive feedback.

Best regards,

Authors

Dear reviewer WxDe:

We sincerely apologize for the delay, and we are very glad to discuss with you the details of attention matrix training. Below, we provide a point-by-point clarification.

1. Memory cost of KL loss.

The reason our method has a relatively small memory usage can be explained as follows:

1. The parameters of ViT-B are completely frozen and do not participate in any training.
2. There might be a slight misunderstanding: our KL divergence is not computed at the pixel level, but at the token level, since the attention map is token-based rather than pixel-based. Thus, the $n$ of O($n^2$) is 576.
3. Because each row of the attention map represents a complete distribution and is independent from the others, we are able to perform row-wise streaming computation.
4. Our CAI module is extremely lightweight, containing only 0.8M parameters.

Next, we provide a detailed example for calculation. Based on the ViT-B/16 model with an input resolution of $384 \times 384$, the final layer outputs $24 \times 24 + 1 = 577$ tokens. Since we exclude the CLS token when computing the attention map, the sizes of q and k are both 576, resulting in an attention map of size $576 \times 576$. The batch size equals 8 and we use fp32. Based on these values, the memory consumption during training is approximately:

If we switch to cross-entropy (CE) loss, the memory usage would be the same as the row-streaming implementation of KL loss (which means we can achieve memory usage comparable to CE loss), i.e., $20.8 / 576 = 0.0361$ MB. Within the overall memory consumption, KL loss accounts for only $20.8 / 1360 = 0.015$. Under this setting, the additional memory overhead introduced by KL loss compared to CE loss is negligible.

Therefore, although the complexity of KL divergence is indeed $O(n^2)$ as you pointed out, the value of $n$ remains sufficiently small (576), and the overall network is extremely lightweight, enabling us to train CAI with only 2GB of GPU memory. We kindly invite the reviewer to consult Figure 1 in our supplementary material for a clearer understanding of the CAI module.

Due to space limitations, we kindly refer the reviewer to another response for the second point.

Dear reviewer ReDz,

We sincerely thank you for your thoughtful comments, valuable insights, and the considerable time and effort you have dedicated to carefully reviewing our manuscript. In the following, We have summarized the related concerns into one unified concern, and address each unified concern point-by-point.

1. W1+Q3: limited performance improvement on training-free methods

First, we would like to clarify that our CAI module is extremely lightweight, and importantly, its training process and loss functions are entirely independent of the segmentation modules and categories, respectively.

Then, in our experiments, we observe clear improvements in average performance among 8 benchmarks across most models. The only exception is ProxyCLIP, where the improvement is about 1%. This is because ProxyCLIP already leverages more powerful external models (e.g., DINO, MAE) to optimize its attention weights. The parameter scale and training cost of these models are substantially larger than those of our CAI. Therefore, the relatively limited improvement (~1%) observed on ProxyCLIP is reasonable and consistent with expectations. Importantly, excluding ProxyCLIP, our method achieves gains ranging from 2% to 18.9% on other methods among 8 benchmarks, highlighting both the stability and the broad applicability of our approach.

2. W2+W3+Q1+Q2: In-Depth Analysis of the Motivation: Why Prior Methods Lack It and How Our Approach Validates It

We first clarify the motivation of this work: existing methods for computing attention weights mainly emphasize local compactness (q–q, k–k, or q–q & k–k) while neglecting or down-weighting global connectivity. To overcome this, we propose the CAI module, which explicitly combines local compactness and global connectivity to produce pixel-aware attention maps. In parallel, the SAA module guides CAI in preserving CLIP’s cross-modal alignment, thereby enabling open-vocabulary segmentation.

To validate the limitations of existing methods regarding global connectivity and to demonstrate that our approach better addresses this aspect, we conducted an experiment on 50 randomly selected images. Using the attention matrices from training-free methods such as MaskCLIP, we measured the attention between each token and same-category tokens at different spatial distances. Tokens were defined as “long-range” if their distance exceeded the average distance to all same-category tokens. For each image, we summed the attention values of these long-range tokens and then averaged across the image to obtain the mean long-range attention value. A higher value indicates greater emphasis on global connectivity. The results are shown in the table below.

As shown in the table, our method achieves the highest mean long-range attention strength (0.373), while most other methods fall between 0.08 and 0.15, with the maximum only 0.289 (when incorporating external models such as DINO). These results indicate that our method captures more information from same-category but distant tokens, ensuring stronger global connectivity. This supports our argument that existing methods emphasize local compactness while underrepresenting global connectivity, whereas our CAI module explicitly strengthens global connectivity to construct more effective attention relationships.

In addition, we kindly refer the reviewer to Figure 2 in our supplementary material, where we visualize both our designed attention weights and the v–v attention produced by DINO. For clearer comparison, we applied brightness enhancement (so the sum of each row is not equal to 1). We sincerely apologize for the typo in Figure 2 and will correct it in the paper. The visualization shows that in DINO’s v–v attention, brighter values mainly appear near the diagonal of the matrix, indicating that local compactness dominates when merging information from other tokens, even though DINO’s training emphasizes global connectivity more than other training-free methods.

3. W4: Difference with other methods that aim to improve attention maps

We thank the reviewer for raising this concern. The originality of OPMapper and its distinctions from other methods are reflected in the following aspects:

1. We propose an explicitly designed local–global composite attention weight construction that directly maps q–k to obtain pixel-aware attention weights. In contrast, other methods compute attention either by incorporating external modules or by relying on self–self attention that primarily enhances local compactness. These two categories of approaches only achieve pixel-aware features indirectly or approximately.

2. We introduce the cross-modal semantic alignment module (SAA), which leverages alternating text–image fusion as an implicit supervision signal to guide the learning of the mapping module (i.e., CAI), while preserving CLIP’s original vision–language alignment. Other methods, however, neglect maintaining this alignment and disregard the utilization of the CLIP text encoder.

3. Our method is modular, lightweight, and incurs no additional overhead during inference, making it adaptable to a wide range of approaches. In contrast, other methods lack such flexibility and generalizability, limiting their applicability across diverse settings.

4. Q4: Baseline and comparison with recent methods

We thank the reviewer for the valuable suggestion. The baselines we selected are widely cited, come with publicly available code, and are commonly used as SOTA benchmarks (e.g., ClearCLIP). We also chose CLIP as it is the foundational training-free model on which most subsequent methods build.

Beyond training-free methods, we integrated our approach into training-based SOTA models as well, where we similarly observed consistent performance improvements.

With respect to more recent methods, we identified two training-free approaches (LPOSS [CVPR’25], CASS [CVPR’25]) from CVPR 2025. Incorporating OPMapper into these methods, we achieved performance improvements, and we will include the corresponding experimental results in the revised manuscript.

5. Q5+Q6: The choice of using KL loss and the effectiveness of $W_{qk}$

KL divergence is inherently well-suited for measuring the difference between two probability distributions, which makes it particularly appropriate for normalized probability-based representations such as attention matrices. In our implementation, both attention matrices—$W_{qk}$ and $W_{p}$—are row-wise normalized via softmax, thereby meeting the requirements of a probability distribution. We kindly refer the reviewer to our Response 3 to Reviewer WxDe, where we provide a more detailed explanation of the distribution of $W_{qk}$ and $W_{p}$.

In contrast, using MSE loss would treat the attention weights as continuous values without accounting for their probabilistic nature. While MSE penalizes absolute deviations, it does not capture the asymmetric information divergence between two distributions. This limitation is especially critical in attention modeling, where preserving the relative probability structure across tokens is more important than minimizing element-wise differences. Thus, KL divergence provides a more principled objective by emphasizing the distributional discrepancy rather than raw magnitude differences.

To further substantiate the validity of $W_{qk}$, we respectfully refer the reviewer to our Response 1 to Reviewer ohL3, in which we provide a detailed and rigorous justification.

6. Q7: q-k weights vs. (q-q)+(k-k) weights

We respectfully note that the reason we adopt the approach of mapping object-aware query and key embeddings into pixel-aware representations to construct a more effective attention map is because self–self attention, by design, tends to allocate the majority of attention weights to the diagonal and adjacent tokens, thereby being dominated once again by local compactness. Given that we have designed a ideal attention map as the target distribution, directly computing q–k attention provides a more principled and learnable formulation than first computing q–q and k–k attentions and then aggregating them.

To further validate this intuition, we performed an additional experiment on CLIP, where CAI was used to transform q and k to generate q–q + k–k attention. The results demonstrate that our current approach achieves an average performance of 42.2, compared to only 40.8 for the q–q + k–k variant, thereby confirming the effectiveness of our design.

7. Q8: Dimensions of Equation 10

Thank you for pointing out this potentially confusing issue. We would like to clarify the misunderstanding. As shown in Eq. 10, $\mathbf{S}_{`CLS`-\mathbf{T}}$

is obtained by computing the cosine similarity between $`CLS`$ (with dimensions $1 \times C$) and $\mathbf{T}$ (with dimensions $C \times L$, where $L$ denotes the number of categories input to the CLIP text encoder). Similarly, $\mathbf{S}_{\mathbf{V-T}}$ is computed between $\mathbf{F_1}$ (with dimensions $\hat{H}\hat{W} \times C$) and $\mathbf{T}$. Therefore, their resulting dimensions are indeed $\mathbb{R}^L$ and $\mathbb{R}^{\hat{H}\hat{W}\times L}$, respectively. In the revised paper, we will clarify the dimensional relationships more explicitly to prevent any possible confusion.

We hope that our response has satisfactorily addressed your concerns, and we thank you again for your thoughtful feedback.

Best regards,

Authors

Thank you for your detailed responses. Most of my concerns have been resolved. I do, however, have a follow-up question.

You provided a clear explanation of the motivation behind your method and its advantages over self–self attention. However, I am curious how your method compares with CDAM [1].

CDAM also does not rely on simple self–self attention. Instead, it generates attention maps using class distributions derived from the cosine similarity between image and text features. Therefore, I believe CDAM already addresses the issues you mentioned in Response W4—namely, "neglecting to maintain cross-modal alignment and disregarding the utilization of the CLIP text encoder." Moreover, CDAM operates in a training-free manner without requiring additional parameters or fine-tuning.

In fact, CDAM appears to align with the same motivation you described: it leverages class distribution derived from prior models to generate new attention maps that likely preserve both local compactness and global connectivity. It is also designed to be plug-and-play compatible with CLIP-based methods.

Could you please clarify how your method is meaningfully different from CDAM, particularly with respect to the motivation, architecture, or empirical behavior?

[1] Kang et al., “Class Distribution-induced Attention Map for Open-vocabulary Semantic Segmentation,” ICLR 2025.

I truly appreciate your efforts, and all of my concerns have been fully addressed. I will be increasing the score accordingly. It would be great if the rebuttal contents could be incorporated into the revision. If there is not enough space in the main paper, including them in the supplementary material would also be acceptable.

Of coarse, we will implement the code to compute this value and follow up with the result within approximately one or two hours.

I sincerely appreciate your effort in answering my questions. I have one final question: could you provide the experimental results for the mean long-range attention value comparing OPMapper and CDAM?

Additional distinctions are summarized as follows:

1. Threshold sensitivity in complex scenes: CDAM relies on a manually selected entropy-based threshold to filter category distributions. As acknowledged by the authors, this threshold is difficult to determine in complex scenes. In contrast, OPMapper requires no threshold or heuristic hyperparameter and learns attention weights in a fully end-to-end manner.

2. Differences in global semantic modeling: CDAM enhances correlations among nearby same-category patches using JS divergence, but its handling of long-range same-category tokens still depends on thresholding. In scenarios with fragmented layouts or multiple instances, this may lead to suboptimal attention (as noted in CDAM’s limitations). OPMapper, by design, assigns high attention weights explicitly to distant same-category tokens, ensuring more consistent global connectivity.

3. Applicability scope: CDAM is currently only applicable to training-free methods. In contrast, OPMapper is compatible with both training-free and training-based segmentation frameworks, showing broader versatility.

4. Deviation from CLIP’s original inference mechanism: CDAM generates image features by applying the learned attention directly to the value (v) without softmax normalization, diverging from CLIP’s standard computation pipeline. OPMapper, however, solely remaps the q–k pair and retains all other aspects of CLIP’s architecture unchanged.

By reviewing CDAM’s code, we found that it only requires reusing the q and k from the final CLIP layer after CDAM’s weight computation, feeding them into OPMapper for mapping, and then computing an additional attention weight (referred to as the OPMapper weight). The final weight is obtained as:

The remaining steps can simply follow CDAM’s original pipeline. Therefore, our method can be seamlessly integrated with CDAM to provide further enhancements.

We integrated OPMapper into the official CDAM implementation and evaluated it on the predefined benchmarks provided in official repository. It is worth noting that CDAM’s attention weights do not satisfy the row-sum-to-one constraint, making their scale substantially larger than that of conventional attention. Under these circumstances, we refrain from applying any normalization to CDAM weights and simply add the OPMapper weights, which still yields clear performance gains. The detailed results are presented in the following table:

In summary, while both CDAM and OPMapper aim to produce attention weights tailored for dense prediction tasks, they are driven by fundamentally different motivations and designs. Crucially, the two methods are complementary rather than conflicting, and can be jointly employed to further boost baseline performance.

We sincerely hope that our response will gain your recognition and foster further constructive feedback.

Best regards,

Authors

Dear reviewer ReDz:

We sincerely apologize for the delay, and we are very glad to discuss with you the details of the comparision with CDAM. Over the past few days, we carefully studied the CDAM paper and its code, as you recommended for comparison. CDAM is an excellent work that ingeniously leverages the category distribution correlations in the image–text similarity map to construct attention better suited for precise localization, and it achieves strong results in experiments.

That said, our method and CDAM differ significantly in terms of motivation and innovations, model framework, adaptation process, and experimental characteristics. In the following table, we summarize the approaches of both methods across these dimensions and highlight the key differences.

Due to space limitations, we kindly refer the reviewer to another response for more details about other differences among two methods, how to adapt OPMapper into CDAM, and the performance.

Dear reviewer Jwvm,

We sincerely thank you for your thoughtful comments, valuable insights, and the considerable time and effort you have dedicated to carefully reviewing our manuscript. The concerns and suggestions raised by you have provided us with an excellent opportunity to significantly enhance the quality and clarity of our paper. In the following, we address each concern point-by-point and present additional analyses and experiments to thoroughly resolve your concerns.

1. W1: Dense prose / heavy notation without an illustration

We sincerely thank the reviewer for the insightful suggestion. Equations (3–8) are each associated with a specific meaning, and the three attention operations correspond to three distinct processes in our framework. In line with your suggestion, we will provide a more detailed explanation of these processes in the revised manuscript. In particular, we will add an illustrative figure to make the workflow more transparent and to ensure that readers can clearly grasp the rationale and implementation of our approach.

2. W2: Provide an overview of semantic segmentation for easily understanding

We sincerely thank the reviewer for the valuable suggestion. In the revised version, we will incorporate an introduction to the semantic segmentation task and cite relevant related works. This addition will help provide readers with a clearer understanding of the background and motivation for open-vocabulary semantic segmentation.

3. W3: Scaling experiments about training data and model size

We fully agree with the reviewer that the proposed scaling experiment is indeed necessary. In fact, we have already provided results in the Table 6 of our supplementary material showing CAI trained on different datasets, and we kindly invite the reviewer to consult the supplementary material for more details.

Regarding the adaptation to larger CLIP image encoders such as ViT-H/14, we believe it will bring performance gains, though the improvement margin is expected to be narrower compared to ViT-B. In our Response 4 to Reviewer ohL3, we explained in detail why the improvement from ViT-L is smaller than that from ViT-B, and the same reasoning applies to ViT-H. Nevertheless, we sincerely appreciate this suggestion, and we plan to adapt CAI to ViT-H in our subsequent work and update the manuscript with the corresponding experimental results.

Additionally, as noted in our Response 6 to Reviewer  WxDe, we have included an experiment where CAI is integrated into Cat-Seg with a ViT-L backbone, and the results further demonstrate that CAI continues to improve performance in this setting.

4. W4: Marginal gains on strong baselines

We thank you for highlighting this interesting point, which was also observed by both you and Reviewer ReDz. First, we would like to clarify that our CAI module is extremely lightweight, and importantly, its training process and loss functions are entirely independent of the segmentation modules and categories, respectively.

Then, in our experiments, we observe clear improvements in average performance among 8 benchmarks across most models. The only exception is ProxyCLIP, where the improvement is about 1%. This is because ProxyCLIP already leverages more powerful external models (e.g., DINO, MAE) to optimize its attention weights. It is consistent with the explanation we mentioned in our Response 4 to Reviewer ohL3. The parameter scale and training cost of these models are substantially larger than those of our CAI. Therefore, the relatively limited improvement (~1%) observed on ProxyCLIP is reasonable and consistent with expectations. Importantly, excluding ProxyCLIP, our method achieves gains ranging from 2% to 18.9% on other methods among 8 benchmarks, highlighting both the stability and the broad applicability of our approach.

5. Q1: Discussion on the diminishing returns of training based methods

We sincerely appreciate your valuable suggestion, which will further enhance the practicability of this work. We posit that the limited, and in some cases negative, performance improvements observed on training-based methods may stem from two primary factors. (1) Their training datasets often overlap with the benchmark categories, leading the attention to become increasingly tailored to the local–global composition favored by those methods. (2) Many of these approaches fine-tune certain backbone parameters, which in turn induces substantial changes in the distributions of q, k, and v. This limitation has also been explicitly acknowledged in our manuscript.

We think (i) quantify the distributional gap between the fine-tuned and base encoders, then correct it to better accommodate OPMapper, and (ii) explicitly tie OPMapper’s parameters to the statistics of the fine-tuned model are promising directions. In the revised manuscript, we will explicitly discuss this phenomenon and make efforts to address this issue in our future work.  

6. Q2: Inference details when appending CAI to ProxyCLIP

Yes, the DINO encoder is still kept at inference, otherwise, the ProxyCLIP essentially degenerates into an approximate version of ClearCLIP. The outputs of CAI and DINO will be combined as the attention weights for ProxyCLIP.

7. Q3: Results on ADE-847 and PC-459 benchmarks

We thank you for your valuable suggestion about extending experiments to more benchmarks. Typically, training-free methods do not provide results on ADE-847 and PC-459, as their performance on these large-scale open-vocabulary benchmarks is often extremely limited. Ensuring reliable performance in such large-category open-set tasks remains a challenging and valuable research direction. In our revised paper, we will make an effort to include experiments on these two benchmarks to further substantiate our approach.

We hope that our response has satisfactorily addressed your concerns, and we thank you again for your thoughtful feedback.

Best regards,

Authors

Dear reviewer ohL3,

We sincerely thank you for your thoughtful comments, valuable insights, and the considerable time and effort you have dedicated to carefully reviewing our manuscript. The concerns and suggestions raised by you have provided us with an excellent opportunity to significantly enhance the quality and clarity of our paper. In the following, we address each concern point-by-point and present additional analyses and experiments to thoroughly resolve your concerns.

1. W1: Deeper analysis of the ideal attention matrix

We appreciate the reviewer for pointing out this concern regarding the theoretical rigor behind our ideal attention matrix design. We would first like to clarify that the term 'ideal' used in our paper should be more appropriately understood as a prior attention matrix. Since it represents the target we expect the model to learn, we referred to it as 'ideal' in the current version. However, to avoid potential confusion with an objective ideal, we will henceforth refer to it as a prior attention matrix (e.g., $W_{qk}$) in this response and update the paper accordingly.

Then, we clarify that our prior attention matrix design is not a simple intuition but fundamentally grounded on two well-established theoretical principles widely recognized in semantic segmentation literature:

1. Local Compactness: Fundamentally, semantic segmentation can be conceptualized as a problem of spatially continuous local clustering, wherein adjacent pixels are not independent but instead exhibit strong categorical coherence due to the inherent spatial smoothness of natural images. This property underscores the expectation that meaningful segmentation should leverage local contextual regularities, thereby reinforcing the robustness and consistency of the learned representations. This is commonly agreed in classical segmentation literature (FCN [Long et al., CVPR’15], DeepLab [Chen et al., TPAMI’18]). In our paper, Equation 4 explicitly incorporates Euclidean distance into attention computation to reinforce local semantic smoothness.

2. Global Connectivity: High-quality semantic segmentation fundamentally relies on the capacity to capture long-range semantic dependencies, beyond local contextual cues. This capability ensures that spatially distant pixels of the same semantic category—such as occluded regions of an identical object—are jointly recognized and consistently segmented (AAF [Ke et al., ECCV'18]). Without modeling such non-local associations, segmentation results risk fragmenting semantically coherent regions, thereby undermining both accuracy and robustness. Vision transformers (ViT, Swin Transformer) have also demonstrated this principle in modeling semantic relations between distant tokens and our Equation (5) precisely formulates a semantic-aware global connectivity pattern that aligns perfectly with this principle.

To further address the reviewer’s concern with rigorous empirical evidence, We conducted the following experiment: we randomly selected 50 images containing open-set categories. For each image, we generated three attention maps using our proposed construction method, Cat-Seg (which was trained on other close-set categories but not trained on these open-set categories), and Mask2Former (which was trained on both close-set and open-set categories, and thus its attention map is treated as the "optimal" one), respectivlly. We then computed both the average cosine similarity and average Structural Similarity Index (SSIM) to measure the similarity between two attention map. As shown in the table:

Ours (prior attention matrix), average cosine similarity = 0.834 and average SSIM = 0.808, shows a consistently high similarity compared to Cat-Seg, rigorously demonstrating that our prior attention construction is much more similar to the fully-supervised attention structure.

Furthermore, as clearly shown in Supplementary Figure 2, the attention patterns generated by CAI supervision remarkably resemble those produced by vision foundation models (e.g., DINO), providing strong qualitative validation.

These analyses collectively provide rigorous justification beyond mere intuition. Therefore, we hope these can fully address the reviewer’s concern.

2. W2: More ablations for the choice of hyperparameters

To address concerns regarding hyperparameter sensitivity and robustness, We have already conducted experiments in Tables 2 and 3 of the supplementary material for the choice of $\lambda$ and weights of different losses. We paste the results from the supplementary material below, and we kindly encourage the reviewer to refer to the supplementary material for detailed settings and explanations. The performance observed therein substantiates the rationality of our chosen hyperparameters and reinforces the reliability of the conclusions drawn in this work.

The choice of $\lambda$ in Eq.(6)

The choice of $w$ for different loss in Eq.(16). Settings denotes for ($w_{KL}$, $w_{DICE}$, $w_{BCE}$)

3. W3: Confusion about Eq 9 and Eq 12

We respectfully clarify that Equation 9 and Equation 12 are not redundant. Equation 9 introduces the general formulation of text-guided enhancement, establishing the theoretical basis of our approach. Equation 12, on the other hand, specifies the implementation that integrates an attention mask which further filters out irrelative categories before merging text guidance, thereby operationalizing the general form under our proposed framework. To avoid any potential confusion, we will further clarify their relationship in our paper, ensuring that readers can clearly distinguish between the general formulation and its masked instantiation.

4. W4: Smaller improvements for ViT-L/14

We appreciate the reviewer for raising this insightful question. The relatively smaller improvement observed with ViT-L/14 can be reasonably attributed to its inherently stronger global modeling capability due to its larger number of parameters and deeper architecture. Specifically, larger Vision Transformers (e.g., ViT-L/14) inherently exhibit more robust global connectivity, as evidenced by previous studies which consistently demonstrate improved semantic consistency and stronger long-range attention modeling as transformer models scale up in size and depth.

Since our proposed OPMapper is explicitly designed to enhance both local compactness and global connectivity—particularly addressing the deficiency of global semantic connections in smaller vision transformers—the room for improvement naturally diminishes as the global modeling capability of the base encoder strengthens. This observation is in fact a positive indication of OPMapper’s core motivation: it significantly boosts performance precisely when global connectivity is inherently limited, such as in smaller transformer variants like ViT-B/16. Thus, the reduced improvement margin observed in ViT-L/14 experiments further validates and highlights OPMapper’s effectiveness in explicitly compensating for inadequate global connectivity.

We hope that our response has satisfactorily addressed your concerns, and we thank you again for your thoughtful feedback.

Best regards,

Authors