AlignCLIP: Enhancing Stable Representations in Vision-Language Pretraining Models through Attention and Prediction Alignment

W1: One of the main concerns comes from Eq.(6) in the manuscript. The proposed AAL views the $G_{D-V}$ as the groundtruth attention distribution and aligns $A_c$ and with $G_{D-V}$. Given the fact the value vectors is problematic and $G_{D-V}$ is obtained based on the V vector at the D-V branch, I do not know why $G_{D-V}$ can be viewed as the groundtruth guidance.

Reply: Thank you for your valuable comments regarding the use of the D-V structure's attention map as a target distribution for attention learning. We would like to offer a multi-faceted response that supports the validity of our approach.

1. Empirical foundation from related work: The issue of attention misalignment in CLIP models has been empirically demonstrated in the work of CLIP Surgery [1]. It has been shown that the attention mechanism during training, particularly the interactions between Query (Q) and Key (K), does not always capture the semantic essence of the tokens, which is more accurately represented by the Values (V). Based on this finding, the CLIP Surgery paper [1] proposed a modification of the Q-K-V attention structure that replaces the Query and Key vectors with the V vector. Empirical studies in this paper demonstrate that the V-V-V structure can obtain ideal and reasonable predictive attention maps (please refer to the CLIP Surgery paper).

   In this paper, the purpose of proposing the D-V structure is to explain that not only the V-V-V structure proposed in the CLIP surgery paper can work, but also works as long as Q-K is replaced by an equivalent structure (such as K-K, Q-Q, or diagonal matrix), as shown in Figure 8 at the Appendix. Based on this finding, we propose the D-V attention structure that replaces the product of Key and Query in self-attention with a diagonal matrix. The advantage is that has a small computation complexity. we found that the GFLOPS for the V-V-V structure amounts to 0.46478, whereas for the D-V structure, it is significantly lower at just 0.0003. Our D-V structure prioritizes this semantic essence of V vectors, therefore providing a more aligned target for attention learning.

   [1] Yi Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li. CLIP Surgery for Better Explainability with Enhancement in Open-Vocabulary Tasks. arXiv, 2023.

2. Self-attention mechanism in transformers: In the Transformer model, Self-Attention is achieved through three vectors: Query (Q), Key (K), and Value (V). For each input element (image patch in ViT), Q, K, and V are computed. Standard Self-Attention weights the sum of Values (V) by the matching scores of Queries (Q) and Keys (K). In our work, the product of Key and Query in Self-Attention is replaced with a diagonal matrix, meaning that the new representation of each input element relies solely on its own information, rather than the interrelation with other elements. This change encourages the model to focus more on the salient targets rather than the background when generating attention maps. By shifting the focus from inter-token relationships to the intrinsic semantic information of each token, our approach ensures that the attention mechanism is more closely aligned with the inherent content of the input, thus serving as an ideal target for attention learning. We believe that this modification is a significant step towards more accurately modeling visual attention in line with human cognitive patterns. We thank the other reviewers for appreciating our contribution to attention alignment. Reviewer TgXT points out that "The proposed method that aligns the attention distribution with the semantic similarity of label and patch embeddings sounds interesting."

Revision: We have further elaborated on this argument on Page 23, Section of C.3 in the manuscript, providing additional empirical evidence and mechanism analysis to strengthen our claim.

W1: For the two technical contributions on attention alignment and label smoothing, I feel they are disjointed. The motivations are not well aligned as well.

Reply: Thank you for your feedback on the motivations behind our attention alignment and label smoothing techniques. Firstly, we appreciate the opportunity to clarify how these are indeed cohesively aligned:

1. Unified Motivation: Both techniques aim to address the misalignment in CLIP models by leveraging textual information, ensuring consistency between visual and textual modalities. The unifying motivation behind both Attention Alignment Loss (AAL) and Semantic Label Smoothing (SLS) is to enhance the adaptability and generalization of CLIP models in open-world settings, where both domain shifts and unseen classes are present. While AAL focuses on optimizing the attention mechanism to better focus on relevant features, SLS aims to refine the prediction mechanism to be more semantically accurate and less prone to misclassification. Together, they address different aspects of the model’s performance but work towards the same goal of improving stability and reliability in complex, real-world scenarios.

2. Common Goal of Stable Representations: Both attention alignment and semantic label smoothing contribute to the overarching goal of achieving stable representations in Vision-Language Pretraining (VLP) models. This stability is crucial for handling challenges like domain shifts and recognizing unseen classes in open-world environments. Therefore, their shared objective is to improve the representational stability and generalization of CLIP models, crucial for performance in open-world settings with domain and class variations.

3. Textual Information Utilization: Textual information is integral to both methods—fine-tuning attention layers and refining output distribution—demonstrating their conceptual harmony.

4. Stable Improvement from Both Modules: We have revised the bar charts in Figure 3 on Page 9 in the revised manuscript to demonstrate the stable improvement. The updated chart better highlights the incremental improvements provided by AAL and SLS. Despite appearing modest, these gains are statistically significant and essential for the robustness of our model. AAL improves accuracy by approximately 1.0% in the VLCS dataset and 2.8% in the Office-Home dataset. SLS alone enhances accuracy by 0.7% in VLCS. Importantly, their combined use results in a consistent and notable improvement (1.4%) in the Domain+Class scenario, underscoring our model's effectiveness in handling real-world domain shifts. The revised figure distinctly contrasts SLS with standard label smoothing. SLS shows improvements of 3.21%, 2.29%, and 0.89% in LabelMe, SUN, and VOC2007 datasets, respectively. These enhancements affirm the capability of SLS to utilize semantic class relationships effectively, proving particularly beneficial in complex scenarios marked by distribution shifts and the presence of open classes.

Revision: 1. We have enhanced the detailed analysis of how these modules interlink on Page 8, Section 4.4 in the revised manuscript, providing additional evidence of their synergistic relationship. We hope this addresses your concerns and better illustrates the aligned motivations of our contributions. 2. We have revised the bar charts in Figure 3 on Page 9 in the revised manuscript to include annotations that distinctly showcase the substantial contributions of AAL and SLS in scenarios involving domain shift and open classes.

W1: The technical details are not presented clearly, especially how the two branches synergize. Figure 2 is critical to illustrate the method but is quite confusing. After carefully reading the paper, I know that the branch of new blocks is only used for deriving the target distribution for the attention alignment. However, this is supposed to be delivered clearly in the illustration.

Reply: Thank you for your valuable feedback regarding the clarity of the presentation in Figure 2. We acknowledge the critical importance of conveying our methodology with precision and clarity.

Revision: In response to your comments, we have revised Figure 2 to more clearly depict the distinct but complementary functions of the two branches. We will also enhance the accompanying textual description to articulate how these branches synergize within our methodology. Specific revisions include: 1. Improved Annotations: Clearer labels and annotations in Figure 2 to directly indicate the role of each branch; 2. Descriptive Legends: A legend that explains the symbols and process flow, ensuring that the function of new blocks in deriving the target distribution is unmistakable; 3. Focused Narrative: A revised narrative in the figure caption and the corresponding text section to explicitly describe the interaction between the branches and their collective contribution to attention alignment. We hope these adjustments will fully address your concerns and greatly appreciate your guidance in improving our manuscript.

W2: The argument that attention is biased towards background lacks solid evidence. The background tokens always occupy a large proportion so this argument is kind of weak. Despite the qualitative examples, this argument is still not well clarified.

Reply: We thank you for your insightful critique regarding our assertion of attention bias towards the background in CLIP models. Our argument is substantiated by the following evidence:

1. Support from Related Work: Two seminal studies [1,2] have demonstrated that CLIP models exhibit a preference for background regions over foreground elements, a tendency that deviates from human perception. These papers provide substantial evidence that CLIP's attention mechanisms often prioritize background information, contradicting what is typically expected in human-centered image understanding.

2. Empirical Evidence from Our Studies: In our work, we present clear evidence through empirical analysis (see Figure 1, Supplementary Figures 7 and 8). These figures illustrate the tendency of CLIP models to disproportionately focus on background elements. Our analysis goes beyond mere observation by offering a methodological approach to recalibrate the attention mechanism, thereby refocusing it on salient objects rather than the background.

3. Insights from Causal Analysis: The training dataset of CLIP, consisting of billions of image-text pairs, often reflects images as effects of certain concepts (Y), akin to the mental visualization preceding its depiction. The accompanying text, however, encompasses a broader semantic range, capturing not just the effect (content) but also the cause (background context). Post-training, CLIP models are proficient in identifying these causal relationships that push the CLIP model pay attention on the background.

   [1] Yi Li, Hualiang Wang, Yiqun Duan, Hang Xu, and Xiaomeng Li. Exploring visual interpretability for contrastive language-image pre-training. arXiv, 2022b.

   [2] Yi Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li. Clip surgery for better explainability with enhancement in open-vocabulary tasks. arXiv, 2023.

Revision: in the revised manuscript, we have provided additional analyses on Page 22, Section C.2, detailing how our approach effectively shifts attention from the background to the relevant foreground objects, thereby improving the model's interpretability and performance. We believe these additions will address your concerns by providing a clearer, evidence-based rationale for our claims about attention bias and its rectification.

• D-V structure has computation advance. Our decision was primarily driven by the goal of achieving greater computational efficiency. To elaborate, we have performed a detailed GFLOPS computation for both structures in the context of a multi-head attention mechanism. For the V-V-V structure, the FLOPs per head are calculated as $2 \times N \times (d_v \times d_{model} + d_v \times d_{model})$, where $N$ represents the sequence length, $d_v$ is the dimension of the value, and $d_{model}$ is the model dimension. In contrast, for the D-V structure, the FLOPs per head simplifies to $N \times d_v$. Applying these calculations to a vision transformer model with a 12-head attention mechanism, we found that the GFLOPS for the V-V-V structure amounts to 0.46478, whereas for the D-V structure, it is significantly lower at just 0.0003. This stark contrast indicates that the D-V structure reduces the computational demand by approximately 1549 times compared to the V-V-V structure. Such a substantial reduction in GFLOPS directly translates into markedly faster processing times and reduced resource consumption. This efficiency makes our approach particularly advantageous for real-world applications, especially in contexts where computational resources are constrained or limited. We believe that these calculations demonstrate the superiority of the D-V structure.

  • Quantitative results for V-V-V and D-V structure. Regarding the quantitative performance comparison, we have conducted additional experiments to provide a direct comparison between the D-V and V-V-V structures. As shown in Table 1, we observed that the D-V structure marginally outperforms the V-V-V structure in specific tasks. We have included these results in Table xx in the revised manuscript.

    Table 1: The quantitative results for V-V-V and D-V structure on the VLCS dataset.

    Method	Caltech101	LabelMe	SUN09	VOC2007	Avg
    V-V-V	99.6±0.2	68.7±0.2	81.1±0.1	90.1±0.2	84.9
    D-V	100±0.0	68.9±0.3	80.9±0.1	90.1±0.2	85.1

3. Differences between TeS (Wang et al., 2023) in B and Semantic Label Smoothing (SLS): TeS is a complex instance-level label smoothing technique using a projection head for diverse reference distributions, whereas SLS is a simpler class-level method focusing on representation stability by aligning predicted distributions with semantic class relationships in open-world environments. While both TeS and SLS are approaches to label smoothing, they exhibit distinct methodologies and applications. TeS represents a more intricate instance-level label smoothing method, using a projection head to align vision representations with textual space. This allows TeS to generate diverse reference distributions for different samples, even within the same class. Conversely, SLS, a more straightforward class-level label smoothing technique, focuses on stabilizing representation in open-world scenarios. It does so by aligning the model's predicted category distribution with the semantic relationships among classes. The key differences between TeS and SLS are threefold:
   • Implementation and Complexity: SLS is more user-friendly in terms of implementation and comprehension. It utilizes a class similarity matrix derived from text embeddings of each class. The cosine similarity between these class text embeddings guides the regularization process during fine-tuning. TeS, on the other hand, involves projecting vision representations into text space using a projection head, followed by processing the text embedding through a text encoder (like CLIP or BERT), and then comparing this information with anchor points.
   • Computational Demand: SLS is less computationally demanding. It requires computing the class similarity matrix only once at the beginning of fine-tuning. In contrast, TeS demands recalculating the similarity matrix for each sample at every iteration, significantly increasing computational overhead.
   • Applicability and Effectiveness: SLS is generally more effective and straightforward for broad classification tasks, particularly in situations involving distribution shifts or new class introductions. TeS, alternatively, is designed to enhance the generalization capabilities of pre-trained models for specific tasks. It addresses issues like catastrophic forgetting post-fine-tuning and biases inherent in the pre-trained models.

Revision: We have added the discussions about the difference between our work and CLIP Surgery on Page 24, Section C.2 in the Appendix of the revised manuscript. We have discussed the central difference between our work and TeS (Wang et al., 2023) on Lines 9-12, Page 6, Section 3.3 in the revised manuscript. The main differences between our work and existing works are much clearer now.

W1: Limited novelty: Although it achieves SOTA results, I have some concerns regarding the technical contribution of the work. Specifically, I find it as a combination of existing techniques. a) ... b) ....

Reply: Thank you for the essential comments. These comments can significantly improve the quality of this paper and make it more logical and self-contained. First of all, we thank you for appreciating our work.

1. Thank you for pointing out that our work achieves SOTA results. On almost all 21 open-world datasets, our work achieves the best performance. This justifies that the two misalignment problems are objective and our target solutions are practical.

2. Thank you for pointing out that the D-V structure is interesting. The purpose of proposing the D-V structure is to explain that not only the V-V-V structure proposed in the CLIP surgery paper can work, but also works as long as Q-K is replaced by an equivalent structure (such as K-K, Q-Q, or diagonal matrix), as shown in Figure 8 at the Appendix.

The comprehensive responses to all your concerns are as follows.

1. About the combination of existing techniques: This is a potential confusion that our work did not combine existing techniques and has new contributions that never existed before.

   • Our work is the first to address the attention misalignment problem at the fine-tuning phase. In fact, CLIP Surgery (Li et al., 2023) is a new type of model interpretability method that cannot be used to adopt the CLIP model to downstream tasks. Please check the CLIP Surgery paper for detailed descriptions. Although our work is inspired by the idea of CLIP surgery to modify the Q-K-V attention structure, this modification is not the core of our contribution, and the core contribution in this part is the newly designed Attention Alignment Loss that utilizes the better similarity distribution derived from the D-V structure to realign the attention distribution across each multi-head attention layer.
   • Our work proposes a new regularization technique, semantic label smoothing, that advances the popular label smoothing technique. Semantic label smoothing can preserve prediction hierarchy based on class similarity derived from textual information. Aligning the model's predicted category distribution, semantic label smoothing can significantly improve the representation stability and enhance the open-class detection in open-world scenarios.

2. Differences between CLIP Surgery (Li et al., 2023) and our work: We appreciate the opportunity to clarify the aspects you have raised concerning the comparison between our proposed D-V structure and the V-V-V structure as mentioned in Li et al., 2023.

   • The research topics are different. The research topic of CLIP Surgery is model interpretability while our research topic is out-of-distribution generalization and out-of-distribution detection. Specifically, CLIP Surgery concerns neural network interpretability while our work addresses the challenge of creating stable representations in VLP models for adaptability in open-world environments, particularly dealing with domain shifts and unseen classes.

   • Our work has the V-V-V attention structure advance. As for the modified structure of attention, our work provides a new insight for the community. Specifically, the CLIP surgery paper only points out only the V-V-V structure is working, but our work points out that not only the V-V-V structure can work, but also works as long as Q-K is replaced by an equivalent structure (such as K-K, Q-Q, or diagonal matrix), as shown in Figure 8 at the Appendix. This insight opens opportunities for future studies to design better model interpretability methods and even to guide the design of vision-language-pertaining strategies to avoid the attention misalignment problem.

   • Our work has more methodology contributions. It is important to highlight that our contributions extend beyond the adoption of the D-V structure. While we acknowledge the foundational work done by Li et al., our approach diverges in addressing the misalignment problem rather than pointing out it only. These include the AlignCLIP which includes a new optimization objective (Attention Alignment Loss) and a regularization technique (Semantic Label Smoothing). Our work thus provides a novel perspective and contributes to the field by enhancing the stability and adaptability of representations in VLP models, ensuring better performance across varied datasets and out-of-distribution contexts.