Rethinking Misalignment in Vision-Language Model Adaptation from a Causal Perspective

We thank Reviewer 7jjJ for the valuable feedback and constructive suggestions. The mentioned issues are addressed as follows:

---

W1: There is no obvious evidence to show that the CDC will improve the prediction in certain cases, which category previous wrong and correct with CDC. It would be better if several compared failure cases to demonstrate that the CDC can solve the question at which level and which case still needs more advanced methods.

A1: Thank you for your suggestions. We provide several examples in the attached PDF of the global rebuttal to compare the prediction results of CDC with those of the baseline method MaPLe. We will include these analyses in the appendix to the final version of our paper, to enhance an intuitive understanding of our proposed CDC.

In the success cases in Figure 1, we observe that:

(1) Based on different prompt templates, the model captures different semantic information of the samples and generates diverse prediction results. This validates the effectiveness of our semantic decoupling strategy.

(2) By fusing the prediction results based on different templates (i.e., different semantic sets), CDC can obtain correct classification results. As shown in Figure 1 (e), the fused result is not a simple average of all predictions and may differ from all of them. This indicates that the fusion process comprehensively considers the credibility and consistency of each prediction result, enabling CDC to correctly classify samples that MaPLe fails to recognize.

For the failure cases in Figure 2, we discover that when the predictions based on each semantic set consistently lean towards the wrong category, CDC also struggles to correct the samples misclassified by MaPLe. Taking Figure 2 (a) as an example, intuitively, from different semantics such as the character's posture and the appearance of the yo-yo, the image is easily misidentified as playing the flute, making it difficult to classify. In the future, richer training data may enhance the model's ability to distinguish different objects and scenes, further improving CDC's performance.

---

W2: It is interesting how the misalignment between CLIP and downstream pattern will change as the model’s capabilities increase, such as ViG-Large, and whether the more powerful model can solve the misalignment problem. Hence, it will be better if some comparisons of models can be added.

A2: Refer to the global rebuttal for the details.

---

W3: As we all know, MLLMs already sweep through the multimodal community, it will be better if expand some discussion about the misalignment in this paradigm and current method whether easily general to that.

A3: Thank you for your suggestions. The misalignment in MLLM is consistent with that presented in our paper. Theoretically, our approach can be applied to MLLM.

MLLMs have demonstrated remarkable potential across a wide range of tasks. After aligning features from different modalities through pre-training, MLLMs often employ techniques such as instruction tuning to encourage the model to complete downstream tasks effectively.

Take Instruct-BLIP as an example. During the instruction tuning phase, Instruct-BLIP learns from specific datasets for certain tasks to build an instruction-aware Q-Former, thus assisting the model in extracting informative features tailored to the given instruction. The learned Q-Former is expected to generalize to unseen datasets and unseen tasks. However, for different tasks, the informative features for similar instructions can be diverse. Therefore, the learned Q-Former risks overfitting the training data. This is similar to prompt tuning analyzed in our paper, where prompts overfit base classes. Therefore, we argue that the discrepancies between training and testing in Instruct-BLIP introduce "data misalignment" in a general sense.

Our proposed CDC has the potential to address the data misalignment issue in Instruct-BLIP. To mitigate the interference of visual features that overfit training tasks, we can decouple the obtained features and estimate the importance of the semantics in the features. By performing a weighted fusion, we can assist the test tasks in extracting more task-relevant visual features, thereby improving overall task performance.

In conclusion, when aiming to enhance the zero-shot generalization performance of a model, the misalignment between training and testing data is a crucial issue that must be carefully considered. Our proposed SCM comprehensively models the misalignment problem in training and testing. CDC has the potential to be transferred to scenarios where misalignment exists, providing a valuable framework for addressing this common challenge in machine learning. In future work, we will investigate the specific implementation of CDC in MLLM to solve the data misalignment problem in it.

---

W4: How many parameters are tuned during downstream adapting? Compared to the pre-trained model, what is the ratio of tuned parameters?

A4: Thank you for your valuable comments. In Table 5, we provide the number of learnable parameters of our method. Compared to MaPLe, our proposed CDC increases the number of learnable parameters from 3.55M to 14.20M, which brings an average performance improvement of 1.70% in the Base-to-New setting.

In the implementation, we can reduce the model overhead by having all templates share the V-L functions, i.e., CDC* in Table 5. CDC* adds 0.027M parameters compared to MaPLe, bringing an average performance improvement of 1.14%. The analysis of the parameters highlights the efficiency of CDC.

Table 5. Comparison of prompting complexity among different methods.

We thank Reviewer JJqY for the valuable comments and constructive suggestions. The mentioned issues are addressed as follows:

---

W1: Figure 1(a) appears to illustrate task misalignment. Consider enhancing the caption of Figure 1 with more detailed explanations to clarify this concept.

A1: Thanks for your valuable suggestions. The example we present in Figure 1(a) is indeed to illustrate the task misalignment problem.

In the final version, we will modify the caption of Figure 1 to: "The motivating examples of the two-level misalignment. (a) Task Misalignment. Determined by the contrastive learning mechanism of CLIP, a given image is more similar to the entire textual description in the embedding space than to a single semantic element, which is inconsistent with the demands of the classification task. (b) Data Misalignment. The learned model intends to overfit on base classes. On the DTD dataset, as the number of training epochs increases, the accuracy of base classes rises, while the accuracy of new classes first rises and then drops."

---

W2: Regarding data misalignment, it would be beneficial to provide a more precise definition. Does it specifically refer to discrepancies in classes between training and testing processes? It's important to clarify that data misalignment encompasses both label misalignment and distribution misalignment. A brief explanation of each type would improve understanding.

A2: Thank you for your suggestion. In our paper, data misalignment encompasses label inconsistency and distribution inconsistency. In the final version, we will further clarify the concept of data misalignment in the appendix to avoid confusion.

Data misalignment refers to the inconsistency between the distribution of training data and testing data. This inconsistency can arise due to two main reasons:

(1) Label Inconsistency: The training and testing classes do not completely overlap. For instance, some classes present in the training data might not appear in the testing data and vice versa. We refer to the classes that appear in training as base classes and the classes that appear in testing as new classes.

(2) Distribution Inconsistency: Even if they share the same class names, the distributions of the classes in the training and testing data may differ, resulting in distribution inconsistency. In such cases, the testing classes are essentially also new classes.

Label inconsistency and distribution inconsistency together constitute the problem of data misalignment.

Furthermore, we have considered the effectiveness of CDC under both types of inconsistencies. The base-to-novel experimental setup mainly involves label inconsistency, while the cross-dataset and cross-domain setups address both label inconsistency and distribution inconsistency. Our proposed CDC has demonstrated effectiveness in all three experimental setups, highlighting its capability to address both types of inconsistencies.

---

W3: In Figure 3, the term "fuse" is used. It would be helpful to clarify the meaning and context of this term within the figure.

A3: Thank you for your constructive suggestions. In Figure 3, "fuse" refers to iteratively combining the evidence from all templates according to Equation (7) to obtain the final classification results. We will add an explanation of the "fuse" operation to the caption of Figure 3: "fuse" represents the process of iteratively combining all evidence according to Equation (7) to obtain the final results.

---

W4: How about accuracy if we directly use zero-shot test for CDC?

A4: Thanks for your valuable comment. The CDC method we proposed preserves the zero-shot ability of CLIP. In the cross-dataset transfer experiments, we use exactly zero-shot testing for the target datasets. To further verify the performance of our method under the zero-shot setting, in addition to the experiments in Table 2 of our manuscript, we have employed another architecture, i.e., ViT-L/14, to evaluate the zero-shot performance of CDC. The results of the additional experiments are presented in Table 4.

As shown in the table, compared to the baseline method MaPLe, our CDC method achieves an average performance gain of 0.43% under the zero-shot setting when using the ViT-B/16 structure, and an average zero-shot performance improvement of 1.05% when using the ViT-L/14 structure. These results further demonstrate the effectiveness of the CDC method in preserving the model's zero-shot ability.

Table 4. Comparison of CDC on the zero-shot classification to evaluate the out-of-distribution generalization of CDC in the cross-dataset setting.

We thank Reviewer Ptf4 for the valuable suggestions. The mentioned issues are addressed as follows:

---

W1: In the experiments section, the method is currently adapted solely to the CLIP model. This limitation may not fully demonstrate the model's universality. The authors can adapt the method to various vision-language models with different architectures to showcase broader applicability.

A1: Thank you for your valuable suggestions. We believe that most current VLMs suffer from the misalignment problems we analyzed in our paper. The SCM we developed effectively models the overfitting problems that VLMs could encounter when adapting to downstream tasks. The proposed CDC, guided by the SCM, can effectively alleviate the interference of task-irrelevant information in VLMs on downstream tasks, thereby enhancing performance.

To validate the universality of our proposed method, we conduct further experiments based on two additional models:

(1) CLIP based on ViT-L/14. ViT-B/16 is the most common setting in prompt tuning and is also employed in our paper. Compared with CLIP based on ViT-B/16, CLIP based on ViT-L/14 is a more powerful model, and experiments on ViT-L/14 can verify the effectiveness of CDC across VLMs of different capacities;

(2) ALIP based on ViT-B/32. ALIP introduces a bi-path model that integrates raw text supervision and synthetic caption supervision. We conduct experiments on ALIP to demonstrate the generalization of our CDC to VLMs with different pre-training objectives.

Refer to the global rebuttal for the experimental results and more analysis. The experimental results demonstrate that our proposed CDC can effectively improve the performance of various VLMs on downstream classification tasks.

---

W2: The experiments are exclusively conducted on image classification tasks. The authors can explore adapting vision-language models (VLMs) to a wider range of tasks, such as object detection, image captioning, or visual question answering, to further validate the model's versatility and performance across diverse applications.

A2: We appreciate your valuable suggestions. To validate whether our proposed method is beneficial for applying VLMs to a wider range of tasks, we explore the application of our method in one-shot object detection. As shown in Table 3, our approach achieves the state-of-the-art performance. The experimental results confirm the effectiveness of our method.

In one-shot object detection, each foreground object category has only one labeled instance available for training. Due to the limited amount of training data, it is often inconsistent with the true data distribution of the dataset. Therefore, there exists a serious misalignment of the training data with the testing data, which is in line with our definition of data misalignment. Our proposed SCM framework is highly effective in addressing this challenge, which is achieved by transferring our SCM (Figure 2 in the original manuscript) to the scenarios of one-shot object detection. Specifically, $D$ in our SCM can be interpreted as the knowledge contained within the base model of one-shot object detection, which encompasses both knowledge related to one-shot object detection (denoted as $G_r$) and knowledge unrelated (denoted as $G_i$). The process of fine-tuning the base model on a single instance can be seen as an attempt to extract $G_r$ while eliminating $G_i$. However, due to the scarcity of samples required for fine-tuning, $G_i$ is often not accurately identified, thus interfering with the modeling of the true causal relationships between the testing instances and their corresponding categories. $\hat{G_i}$ becomes a confounder.

Our proposed CDC effectively mitigates such confounders. Concretely, we generate feature vectors for the foreground object categories using the text encoder of CLIP and align the visual features from the pre-trained base model with the corresponding text features in CLIP through a projector. Additionally, we enhance the feature alignment via prompt tuning and CDC within the CLIP text encoder. As shown in Table 3, our proposed CDC has achieved state-of-the-art performance among works in recent years. Compared to the baseline method DeFRCN [R2], our approach yields an average performance improvements of 1.61% across three different data splits, demonstrating its effectiveness in extending VLM to other tasks.

In future work, we will continue to explore the potential of CDC for enhancing the application of VLMs, e.g., CLIP, across a wider range of tasks.

Table 3 One-shot experimental results of AP50 on the VOC dataset [R3] (%).

[R2] Qiao L, Zhao Y, Li Z, et al. Defrcn: Decoupled faster r-cnn for few-shot object detection[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 8681-8690.

[R3] M. Everingham, L. V. Gool, C. K. I. Williams, J. M. Winn, A. Zisserman, The pascal visual object classes (VOC) challenge, Int. J. Comput. Vis. 88 (2) (2010) 303–338.

[R4] P. Kaul, W. Xie, A. Zisserman, Label, verify, correct: A simple few shot object detection method, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 14237–14247.

[R5] X. Lu, W. Diao, Y. Mao, J. Li, P. Wang, X. Sun, K. Fu, Breaking immutable: Information-coupled prototype elaboration for few-shot object detection, in: B. Williams, Y. Chen, J. Neville (Eds.), AAAI 2023, AAAI Press, 2023, pp. 1844–1852.

[R6] J. Ma, Y. Niu, J. Xu, S. Huang, G. Han, S. Chang, Digeo: Discriminative geometry-aware learning for generalized few-shot object detection, in: CVPR 2023, IEEE, 2023, pp. 3208–3218.