Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models

Thank you for your detailed feedback, which we believe will improve our final manuscript. Please see our responses to the helpful points raised in your review below:

W1 Compared to existing methods, one dataset is missing. Why was CIFAR-100 removed from the forgetting benchmark? Is it because the method performs poorly on this dataset?

A1 We appreciate your concern regarding the exclusion of CIFAR-100 from the X-TAIL setting. We have added experiments of including CIFAR-100 dataset in the X-TAIL setting. Due to the space limitation, please refer to the Experiment 3 in the general response. Our methods surpass existing SOTA methods on CIFAR-100 across all metrics.

However, the primary purpose of X-TAIL is to evaluate a model's cross-domain discriminative capabilities while transferring continually to distinct domains. CIFAR-100 was excluded because it includes many classes that overlap with those in other domains; for instance, the 'forest' class in CIFAR-100 overlaps with 'broadleaf forest' in EuroSAT. During testing, if an image from EuroSAT labeled as 'broadleaf forest' is classified simply as 'forest,' this would be factually correct but considered incorrect in the evaluation metrics. To prevent the redundancy of learning overlapping classes and to maintain the integrity of the setting, CIFAR-100 was not included in X-TAIL. We will include a detailed explanation in our manuscript for clarification.

W2 Determining whether a sample is OOD relies on learned class labels, which inevitably use label (text) information. I am concerned that this could lead to label leakage.

A2 We would like to respectfully clarify that our method does NOT lead to label leakage. We determine whether a sample is OOD (i.e., belonging to unlearned domain) during the testing process by checking if the CLIP zero-shot predicted class belongs to the set of classes learned by the adapter. If the zero-shot result of CLIP falls within the recorded set, the test image is classified as ID; otherwise, it is classified as OOD. This process does not involve any exposure to the labels or domains of the test images, thus does NOT lead to label leakage.

We will emphasize this aspect more explicitly in the final version to ensure there is no confusion.

W3 Figure 2d's graph is consistent with ZSCL, only styled differently, which I find unnecessary. It may not highlight the novelty of the X-TAIL setting.

A3 We acknowledge that the metric presented in this figure is not a novelty of our work but rather an essential aspect of understanding the metrics involved in the X-TAIL setting. The styling similarity to ZSCL is intended solely for clarity and consistency in presentation, not as a highlight of novelty. Our intent is to facilitate reader understanding of the setting's metrics, as evidenced by the confusion expressed by Reviewer ThWm. We will add another citation to ZSCL in the caption of Figure 2 in the final version for further clarification.

Q1 Additionally, during the fine-tuning stage, is each task fine-tuned using only the text corresponding to that task to ensure no label leakage occurs during the training of each task?

A4 During the fine-tuning stage, we ensure that labels (texts) from future tasks are not accessible to current task, thereby preventing any label leakage problem.

Q2 Why is the setting for Order2 different from existing methods? I am concerned about the actual performance of this method.

A5 The order 2 of X-TAIL is randomly shuffled. In response to your concerns, we have conducted additional experiments as outlined in Experiment 4 in the general response, following the Order 2 sequence in ZSCL paper[1]. Our methods consistently surpass existing methods, similar to the results observed in the two orders discussed in the manuscript. This demonstrates the robustness of our methods to variations in task order.

Q3 How does Formula 1 compute one-hot with embedding, given that Y and Xe are of different dimensions? Could you clarify the content of the formula clearly?

A6 We apologize for the confusion regarding Formula 1. This was indeed a typo in our manuscript. $\mathbf{X}_e$ should be multiplied by a weight matrix $\mathbf{W}$ to map it to the same dimension as $\mathbf{Y}$. We will correct this formula according to your observation and the suggestion provided by Reviewer Tz45.

Limitation I am primarily concerned about the application scenarios and practical significance of the proposed setting, as well as the insufficient experimental validation.

A7 The X-TAIL setting primarily addresses a significant limitation in the existing continual learning scenarios for Vision-Language Models, specifically within the MTIL setting where specifying the domain information of test images is necessary. The contribution of our setting is also acknowledged by Reviewer Tz45. To further validate the effectiveness of our method, we have conducted additional experiments in the general response. If you have any questions about our setting, we would be glad to provide additional clarification during the discussion.

Based on these additional results and clarifications, we hope you could consider increasing your score in support of this work. If not, could you kindly let us know what additionally needs to be done in your assessment to make this work ready for publication?

Reference

[1] Zheng, Zangwei, et al. "Preventing zero-shot transfer degradation in continual learning of vision-language models." ICCV. 2023.

We appreciate the reviewer's positive feedback on our work, and further thank the reviewer for finding our work novel in adopting traditional machine learning techniques for continual learning and appealing in its approach to absolute memorization based on analytic techniques. Below, we address the reviewer's concerns in turn:

W1 Could you explain the main difference between primal and dual forms in the CIL problem? It could be quite difficult in this community to understand.

A1 In the proposed method, the primary distinction between the primal and dual forms lies in the differences between primal ridge regression and dual ridge regression: the former explicitly projects CLIP extracted features into a higher dimensional space, while the latter does so implicitly (refer to Appendix B). For tackling the CIL problem, we extended both forms of ridge regression to iterative solutions by Theorem 1 and Theorem 2, respectively. We summarize the difference of two methods as follows.

• Primal Ridge Regression:
  • Advantages:
    • Since the dimensionality $d$ is fixed, the computational complexity, mainly determined by matrix inversion, does not increase as data accumulates.
  • Disadvantages:
    • Requires explicit specification of an appropriate projection function, which can be challenging to design optimally.
    • A sufficiently large $d$ is necessary to ensure that the features are expressive enough for accurate classification, which can be computationally expensive.
• Dual Ridge Regression:
  • Advantages:
    • Leveraging the advantage of the kernel trick, the projection function can be implicitly defined based on the choice of the kernel function, avoiding direct computation and storage of high-dimensional features. It allows for the selection of appropriate kernel functions based on different tasks, enhancing the adaptability of the method.
    • The method is computationally efficient when the amount of data is small, especially in the few-shot case.
  • Disadvantages:
    • As the amount of data increases, the computational complexity of kernel matrix ($N \times c$) inversion becomes significant, potentially making the method computational expensive for very large incoming dataset.

W2 Experiemnts seems a little bit thin in the manucript. Perhaps try to move some of the experiments in the appendix to the main content.

A2 We agree with your suggestion to enrich the Experiment section of the main manuscript. We will move a portion of the experiments in the Appendix into the main content for the final version.

Based on these additional clarifications, we hope you can keep your support of this work.

I thank the authors for providing the rebuttal. After reading the rebuttal and other reviewers' comments, all my previous concerns have been adequately addressed. I will keep my positive rating.

Thank you for detailed feedback and valuable suggestions.

W1 Concerns of paper understanding.

A1 Thank you for bringing up your confusions. Based on your suggestion, we decide to make following revisions in the final version.

• We will update the introduction with clearer explanation of both primal and dual methods. We will include a detailed description of how the CLIP extracted features are projected into higher dimensional space and then classified with ridge regression.
• We agree that Algorithm 2 in the Appendix provides a clearer explanation of the process for identifying ID and OOD classes. We will move this algorithm to the main manuscript to improve paper's self-containment.

W2 Evaluation on full data MTIL.

A2 We have added experiments of comparing the proposed method with SOTA methods ZSCL[1] and Moe-Adapter[2] under the full data MTIL setting, reported Experiment 1 in the general response. Our methods consistently surpass existing methods under this setting. We will include the experiments in the final version.

W3&Limitation Concerns for large data.

A3 Thank you for pointing out the concerns regarding the computational complexity of our method. Our method requires only forward pass and matrix computations for parameter updates, without back-propagation or gradient updates. Since the backward pass typically accounts for approximately 70% of the entire foward-backward training time[3], and our method trains on every dataset over only one epoch, it is faster than back-propagation-based methods. Regarding the impact of the amount of data, we provide the following analysis:

Primal Ridge Regression: As detailed in Theorem 1 of our manuscript, the computational complexity for updating parameters mainly relies on the matrix inversion operation in Eqn. 6. Assuming $N$ is the amount of training data and $d$ is the dimension of features, the size of $\mathbf{Φ}^{(n)}$ is $N \times d$, and the size of $\mathbf{M}_p^{(n)}$ is $d \times d$. The large size of the $\mathbf{Φ}^{(n)}\mathbf{M}_p^{(n−1)}\mathbf{Φ}^{(n)T}$ matrix, which is affected by data amount, can be managed by using mini-batches. By breaking down the $\mathbf{Φ}^{(n)}$ into manageable batches and updating $\mathbf{M}_p^{(n)}$ iteratively, our primal method prevents computational complexity from escalating with large data amount.

Dual Ridge Regression: We acknowledge that the limitation of high computational complexity in large data scenarios is inherent to kernel-based methods (e.g., kernel SVM), including our dual method. As stated in Theorem 2, the complexity mainly lies in inverting the $\mathbf{K}^{(n)} + \lambda \mathbf{I}$ matrix, where the size of $\mathbf{K}^{(n)}$ is $N \times N$, determined by the amount of data involved in constructing the kernel, making it more suited for few-shot settings. However, existing techniques, such as Nyström approximation[4], can mitigate the computational complexity of kernel-based methods under large data scenario, enhancing the scalability of our method.

Furthermore, to justify the computational efficiency of our method, we added experiments on the comparison of computational times between our proposed RAIL methods and the SOTA methods ZSCL[1] and Moe-Adapter[2] in the X-TAIL setting, reported in Experiment 2 in the general response. Our methods are significantly faster than the SOTA methods.

W4 Typos.

A4 We thank the reviewer for pointing out the typo and we will fix it in the final version.

Q1 Concerns for finetune.

A5 We chose finetune as a baseline as it represents a classic method for transferring CLIP to downstream tasks. However, it is important to clarify that finetune is NOT the upper bound or the ideal situation for our method.

First, our method differs fundamentally from finetune in terms of optimization. We use ridge regression to achieve a closed-form solution for parameter updates, while finetune relies on back-propagation. The trainable parameters in finetune are those of CLIP encoders, while our method freezes the encoders and only trains additional adapter parameters instead. Notably, our method is proved to be an optimal solution to the optimization problem of joint training, as evidenced by Theorems 1 and 2, making our method equivalent to its own continual learning upper bound.

Second, in the few-shot setting, finetuning the whole encoders of CLIP could potentially lead to overfitting. By contrast, our method freezes the encoders and only trains the adapter via ridge regression, which can overcome overfitting with appropriate regularization parameter, mitigating this risk. Thus, it is reasonable for our method to outperform finetune.

Q2 How are existing methods evaluated?

A6 We replicated existing methods with open-source codes from ZSCL[1] and Moe-Adapter[2] for evaluation. For methods before the advent of CLIP, we adopted implementations from ZSCL's open-source repository. For instance, applying iCaRL to CLIP involves maintaining an exemplar set and updating representations, consistent with the original iCaRL paper. In the X-TAIL setting, we eliminated the specification of domain identity for all methods to ensure a fair comparison between baselines and our method.

Given these clarifications, would you consider raising your score for our paper?

Reference

[1] Zheng, Zangwei, et al. "Preventing zero-shot transfer degradation in continual learning of vision-language models." ICCV. 2023.

[2] Yu, Jiazuo, et al. "Boosting continual learning of vision-language models via mixture-of-experts adapters." CVPR. 2024.

[3] Huo, Z., Gu, B. and Huang, H. "Decoupled parallel backpropagation with convergence guarantee." ICML. PMLR, 2018.

[4] Chen, Y. and Yang, Y.. "Fast statistical leverage score approximation in kernel ridge regression." International Conference on Artificial Intelligence and Statistics. PMLR, 2021.

Thank you for your overall supportive review of our work. Moreover, we are glad that you found this paper is well-written with good soundness and contribution to the NeurIPS community. Please see our responses addressing your specific concerns below:

W1 There is little difference in performance between the primal and dual ridge regression methods on most datasets, as shown in Figure 6. Could you provide more analysis on this?

A1 Thank you for pointing out that the differences in performance between the primal and dual forms of ridge regression might appear minimal in Figure 6. This issue primarily stems from the scale of the axes used in the plot, which does not sufficiently emphasize small variations in performance metrics. We understand that this could potentially obscure meaningful distinctions in the comparative evaluation. To provide a clearer picture of the performance differences, we kindly refer you to Table 1, where the last performance of both methods are detailed. The results are also copied below. Here, you can observe that the dual method outperforms the primal method by 3.3% on average in the last performance.

W2 Could you compare the parameters of primal and dual ridge regression methods? What are the advantages and disadvantages of each?

A2 Parameter Differences:

• Primal Form: The parameter matrix W in primal ridge regression is determined by the dimensionality $d$ of the projected features, which is a $d \times c$ matrix where $c$ represents the number of classes. The dimension $d$ remains fixed, meaning that the complexity associated with the solution of parameter matrix W does not increase as the amount of data grows.
• Dual Form: The parameter matrix α in dual ridge regression is a $N \times c$ matrix, where $N$ is the number of data points involved in constructing the kernel matrix. As training progresses and more data is incorporated, i.e., $N$ increases, the computational demands of the method get impacted.

Advantages and Disadvantages:

• Primal Ridge Regression:
  • Advantages:
    • Since the dimensionality of features $d$ is fixed, the computational complexity, mainly determined by matrix inversion, does not increase as data accumulates.
  • Disadvantages:
    • Requires explicit specification of an appropriate projection function, which can be challenging to design optimally.
    • A sufficiently large $d$ is necessary to ensure that the features are expressive enough for accurate classification, which can be computationally expensive.
• Dual Ridge Regression:
  • Advantages:
    • Leveraging the advantage of the kernel trick, the projection function can be implicitly defined based on the choice of the kernel function, avoiding direct computation and storage of high-dimensional features. It allows for the selection of appropriate kernel functions based on different tasks, enhancing the adaptability of the method.
    • The method is computationally efficient when the amount of data is small, especially in the few-shot case.
  • Disadvantages:
    • As the amount of data increases, the computational complexity of kernel matrix ($N \times c$) inversion becomes significant, potentially making the method computational expensive for very large incoming dataset.

W3 Please provide more explanation of Figure 2. For example, what do the different colored blocks represent?

A3 Blocks in Figure 2 represents the classification performance of the model across all domains after learning each specific domain.

• The blue blocks in the upper-right matrix indicate the model's zero-shot performance on these domains before learning these domains. These blocks are utilized to evaluate the ability of continual learning methods to “transfer” zero-shot capabilities of the VLM.
• The gray and green blocks under the diagonal show the classification performance on these domains after the model has learned these domains. Specifically, the green blocks represent the model's “last” performance on these domains after learning all domains. These blocks evaluate the adaptability of continual learning methods to new domains and their ability to retain the newly acquired knowledge throughout the continual learning process.
• The orange blocks indicate the “average” performance across all time stamps for each domain.

We will update the explanations mentioned above into the caption of Figure 2 and the corresponding subsection to clarify the metrics and facilitate easier understanding. In light of these clarifications, would you consider increasing your score for our paper? Otherwise, could you let us know any additional changes you would like to see in order for this work to be accepted?