Refining CLIP's Spatial Awareness: A Visual-Centric Perspective

We appreciate your valuable comments and questions. We hope that our response can address your concerns.

Response to W1. & Q1 (presentation): Thanks for your helpful advice, we've strengthen the presentation quality of our paper in the revised version, please kindly refer to the highlighted contents.

---

Response to Q2 (why CLIPSelf and RegionText work worse than vanilla CLIP): Thank you for the insightful question. Multi-modal dense prediction tasks demand a dual capability: aligning dense representations with language and preserving visual-centric spatial awareness. RLA methods tend to prioritize the former, often at the cost of the latter. Given CLIP's inherently limited dense-level vision-language alignment, this trade-off can yield relatively good performance in multi-modal tasks.

However, from a visual-centric perspective, relying on language supervision poses significant challenges. The modality gap and the inherent limitations of language supervision often lead to the loss of critical local visual semantics, which is essential for traditional dense prediction tasks that rely on precise localization and recognition. As a result, these methods often perform worse than vanilla CLIP.

In this paper, we argue that spatial awareness is just as crucial as vision-language alignment—a perspective often overlooked in recent RLA literature. Importantly, we demonstrate that achieving robust region-language alignment does not require compromising a model's spatial awareness. By addressing this balance, our experimental results show improvements in both visual-centric and multi-modal prediction tasks. We hope these findings will inspire future designs of RLA methods and vision-language models.

Response to W3: Sorry for the typo, which should be “Fig.2. To …”. We’ve corrected this line in the revised version.

---

Response to Q1 (consistency of RoI): There are two settings for defining region proposals: (a) Random Proposals: Proposals are randomly sampled and consistently applied to both the student and teacher encoders; (b) RPN Proposals: Following CLIPSelf, the RoI regions are pre-generated using an additional RPN structure before fine-tuning. These approaches ensure that misalignment of the RoIs is not a concern.

---

Response to Q3 (segmentation related visualizations): Thank you for your valuable suggestion. In response, we have incorporated the off-the-shelf MaskCLIP [1] as the evaluation protocol and included visualized segmentation results in Fig. 7 and Fig. 17 of the revised manuscript. We believe this addition better demonstrates the generalizability of our approach and provides clearer clarification.

Reference

[1] Zhou et al. "Extract free dense labels from clip." ECCV 2022.

We appreciate your valuable comments and questions. We hope that our response can address your concerns.

Response to W1 (novelty of SCD): Our core contributions regarding SCD do not primarily lie in proposing a novel distillation mechanism. More importantly, we identify the necessity of integrating a visual-centric constraint with RLA to mitigate the neglected dense perception degradation issue widely encountered. As discussed in the Related Work section, the potential of correlation distillation in multi-modal scenarios remains largely untapped. Moreover, we also conduct an ablation study in Sec 4.5 to showcase the effects of our SCD’s technical modifications to make correlation distillation compatible with RLAs.

---

Response to W2 & Q2 (more analysis): We would like to answer W2 and Q2 together. Our claim is that multi-modal requires both vision-language alignment and robust spatial awareness. While RLAs primarily focus on the former, our approach emphasizes the latter. To support this, we have conducted several analyses highlighting the success of our designs from a visual-centric perspective. These include dense-level t-SNE visualizations (Fig.1), affinity visualizations (Fig.5), and unsupervised segmentation (Fig.5). Our analysis demonstrates that R-SCD significantly enhances the model's dense-level separability and localization capability (The activation highly matches the object contour). Therefore, when integrated with the RLAs, our method exhibits significant improvement including the dense classification task in Tab.1.

Also, according to your suggestion, we hope an additional analysis can further solve your concern (presented in Appendix C.3):

The Refiner is designed to mitigate the semantic contamination from irrelevant context, therefore enhancing the model's sensitiveness to local semantics. To validate this, we concatenate two independently sampled images $X_A$ and $X_B$ side by side, denoted as $X_{AB}$, which introduces context disturbance from $X_B$ to $X_A$. We forward $X_A$ to the image encoder to obtain regional feature map $Z_{A|AB}, Z_{B|AB}, Z_A, Z_B$. We derive the coupling ratio as:

$\text{CR} = E_i[\frac{cos(Z_{A|AB}[i], Z_{B|AB}[j])}{cos(Z_{A}[i], Z_{B}[j])}], j = \text{arg}\max_k cos(Z_{A|AB}[i], Z_{B|AB}[k]),$

where we identify the most similar token $j$ in $Z_{B|AB}$ to the token $i$ in $Z_{A|AB}$, and analyze whether this similarity arises from the coupling of irrelevant semantics introduced by the concatenation operation. Ideally, the CR value is expected to be close to 1, as $X_A$ and $X_B$ possess independent semantics. The measured CR values are reported as below:

The results indicate that both the original and CLIPSelf-finetuned CLIP models are significantly affected by semantic coupling. In contrast, our proposed Refiner effectively addresses this issue, demonstrating high consistency with its intended design goals of eliminating semantic contamination.

Similarly, for your question:

would the contents of embeded sampled images {X_i} affect the SPATIAL AWARENESS ?

Randomly introduced irrelevant background $X_i$ can degrade the spatial awareness of the foreground $X_t$ if semantic contamination is present. To evaluate this effect, we compare the unsupervised segmentation performance of the isolated $X_t$ and the concatenated image $X_t |X_i$ on the Cityscapes dataset, yielding the following results:

Overall, the above analysis highlights another key aspect of our model's effectiveness: the Refiner mitigates the dominance of dense representations by unintended contextual influences, making them more sensitive to local semantics, more distinguishable, and better suited for dense-level tasks. As a result, the R-SCD pipeline enhances CLIP's image encoder, making it more effective for dense prediction tasks in both open-vocabulary and visual-centric settings.

Response to W2 (applicability): We appreciate the reviewer’s insights and would like to address the concerns regarding R-SCD’s applicability.

The application of the SCD is restricted.

RLAs are important since the dense-level potential of VLMs has not been fully explored, especially in the era of LLM. SCD serves as a valuable visual-centric complement to RLAs, especially when integrated with the Refiner, offering a pathway to designing novel VLMs with enhanced spatial awareness. We believe this integration has the potential to enable a wide range of applications in future research. For instance, large VLMs such as Qwen-VL [1] and BLIP-2 [2] rely on the CLIP image encoder to capture fine-grained visual information for their LLM modules. In this context, R-SCD-based RLAs could serve as a potentially feasible method to address CLIP’s dense-level limitations, thereby further improving the performance of these advanced VLMs.

Regarding the concern about "equal weights of teacher and student models," this is not a necessary condition for R-SCD to function. Since R-SCD is a relational constraint, it does not require the student and teacher to share the same embedding space, parameters, or architecture. For instance, we demonstrate that using DINO V2 + Refiner to enhance CLIPSelf-based EVA-CLIP results in effective improvements:

Application of R-SC-V.

R-SC-V is specifically designed to enhance image-only dense prediction tasks (e.g., segmentation and detection) and is not intended for CLIPSelf-based methods. For example, applying R-SC-V to DINO V2 demonstrates its effectiveness. To further explore its potential, we integrated R-SC-V with MAE for unsupervised segmentation, yielding the following results:

These results highlight R-SC-V’s ability to enhance spatially-aware dense embeddings and its potential benefits to the SSL community. We will include additional results and insights in future work to further expand the understanding and applicability of our method.

---

Response to W3: (generalizability and scalability) We additionally conduct two experiments to address your concerns: (a) for generalizability, we perform R-SCD on more VLMs like DFN [3] and Meta-CLIP [4]. (b) for scalability, following CLIPSelf, we train R-SC-CLIPSelf on CC3M dataset for 1 epoch. We've added the results in our revised version. We adopt the off-the-shelf zero-shot segmentation as in MaskCLIP [5] as it's training-free and directly reflects both spatial awareness and vision-language alignment quality.

(a) generalizability: training on more VLMs:

(b) scalability: training on CC3M:

Reference

[1] Bai et al. "Qwen-vl: A frontier large vision-language model with versatile abilities." ArXiv 2023.

[2] Li et al. "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models." ICML 2023.

[3] Fang et al. "Data filtering networks." ICLR 2024.

[4] Xu et al. "Demystifying clip data." ICLR 2024.

[5] Zhou et al. "Extract free dense labels from clip." ECCV 2022.

We appreciate your valuable comments and questions. We hope that our response can address your concerns.

Response to W1: (Refiner vs CLIPSelf)

We appreciate your feedback and would like to clarify that even when $K=N$, $i.e.$ finetuning the entire image encoder as the Refiner, there are fundamental distinctions between the two training strategies:

(a) Motivation:

CLIPSelf: CLIPSelf observed that region patches lack consistent alignment with text representations. Their motivation is to address this gap by using the CLS token, which encapsulates global semantic information, to guide and refine region-level features, thereby improving region-level text alignment.

Ours: We highlight a different issue—CLIP's lack of spatial awareness, particularly the localization and recognition degradation of dense representations in dense tasks due to its focus on image-level alignment with text. This inherent noise in CLIP ViTs' spatial representation is identified as a core limitation.

(b) Strategy:

To provide a clearer comparison, we recall the respective objectives:

$L_{CLIPSelf} = L_{Align}(\text{RoIPool}(Z_i), CLS_i), \\ L_{Refiner} = \sum_j L_{Align}(Z_i[j], Z'_i[j]),$

where $Z_i$ is the i-th regional feature map.

CLIPSelf: It extracts region representations using RoI pooling and aligns them with the $[CLS]$ token as supervision. While this approach appears similar to our "global-to-local" dynamic, the supervision signal in CLIPSelf comes from the $[CLS]$ token, which is already aligned to the text domain. This alignment inherently discards crucial local spatial details, leading to significant visual-centric degradation—a core limitation of CLIPSelf that we aim to address.

Ours: In contrast, our method leverages point-level representations $Z[j]$ with finer granularity that better encodes visual-centric knowledge. By adopting the corresponding feature $Z'[j]$ from a local crop as the teacher, we enable a purely visual-centric fine-tuning strategy, preserving high-quality local semantics.

(c) Experimental Results:

As mentioned in the motivation, CLIPSelf only targets the alignment between text and regions, ignoring the spatial correlation among the patches. To substantiate this difference, we conduct an additional ablation where the Refiner is fine-tuned using CLIPSelf's objective. As shown below, the output quality of the Refiner-CLIPSelf significantly declines under the CLIPSelf-based objective. Additionally, our qualitative analysis (see Fig. 16; Fig. 15 in the original version) highlights that the local representations from the CLIPSelf-based model share high similarity with surrounding patches, even those without relevant semantics, being dominated by global semantics. This behavior demonstrates its inability to serve as effective visual-centric local supervision. We have also included further empirical studies in the revised Appendix C.3.

Ablation on K

In our original version, we conducted an ablation study on the depth of the Refiner ($K$), but deferred it to Appendix E (Tab. 12 in the revised version) due to space constraints. For your convenience, we provide the details below:

We investigate the impact of the depth of the Refiner on the performance of the distilled model. The depth will affect the distillation process from two aspects: i) the balance between the capacity of refining and preserving the original visual knowledge learned by the visual encoder, and ii) the computational efficiency of the training process. We conduct experiments with different depths of the Refiner. A deeper Refiner will increase the parameter size and the complexity of the fine-tuned model, but more difficult to preserve learned knowledge of the pre-trained model. The model with a 4-layer Refiner achieves the best performance, obtaining balance between the refining capacity and knowledge preservation.

---