TaCA: Hot-Plugging Upgrades for Foundation Model with Task-agnostic Compatible Adapter

Q1: The authors should validate the flexibility of the proposed TaCA module. How about the performance when the TaCA is aligned with the other LLMs?

A1: The proposed TaCA demonstrates exceptional flexibility by seamlessly integrating with a range of downstream tasks, eliminating the need for additional fine-tuning costs. Its versatility has been validated across tasks such as video-text retrieval, video classification, visual question answering, zero-shot image classification, and zero-shot image-text retrieval. Table 3 further illustrates TaCA's compatibility with the BLIP-2 LLM, confirming its robust performance when paired with different language models.

Q2: Addressing the Lack of Qualitative Analysis and Visualizations

A2: We recognize the importance of qualitative analysis and visualization in our experimental assessment. The manuscript currently discusses qualitative aspects, and visualizations are provided through the inference framework and image generation samples in Appendix C. We will enhance our manuscript with a more comprehensive qualitative analysis and additional visualizations in line with your feedback.

Q1: TaCA forces large-scale visual model to align with relatively small-scale visual model using adapter. in Table 2, TaCA provides marginal improvements on downstream video classification tasks comparing to directly using large-scale visual model.

A1: The purpose of TaCA is to efficiently align the old model with a large-scale or powerful visual model, thereby enhancing downstream task performance. In Table 2 in the main paper, the de-emphasized lines represent the results achieved by fine-tuning all downstream heads, which are task-specific and inefficient. Conversely, the emphasized lines depict the performance obtained by directly deploying the TaCA model without making any modifications to the downstream modules.

Q2: Add experiments on image-related tasks like image-text retrieval, image segmentation, and few-shot image classification.

A2: We have broadened the scope of our experiments to include fundamental image classification and retrieval tasks. The results, detailed in Tables 1 and 2, demonstrate TaCA's consistent performance improvements across these basic image tasks.

Table1: Zero-shot image classification on ImageNet and CIFAR100 datasets. We report the accuracy.

Table2: Zero-shot image-text retrieval on MSCOCO and Flickr datasets. We report the Recall@1.

Q3: What is the training overhead in terms of time?

A3: The fine-tuning schedules and inference details for these paradigms are presented in Appendix C. In the video-text retrieval task, we fine-tune three distinct sets of downstream heads on the MSRVTT, MSVD, and DiDeMo datasets, resulting in the training of a total of 42M parameters. For the video classification task, fine-tuning the classification head requires nearly twice as many parameters as the TaCA model. However, TaCA requires training just once, with no additional fine-tuning costs, yielding a more parameter-efficient and training-free application in downstream tasks. This discussion will be incorporated into the final paper.

Table3: Comparison on Video-Text Retrieval datasets. The new visual fundation model (VFM) is ViT-L/14, and the old VFM is ViT-B/16.

Table4: Comparison on Video Classification datasets. The new visual fundation model (VFM) is ViT-L/14, and the old VFM is ViT-B/16.

Q1: About the motivation of hot-plugging adapters.

A1: (1) We acknowledge the confusion in our initial explanation. Our aim is to highlight not the superiority of adapter fine-tuning over task-specific head fine-tuning, but rather the inefficiencies and inflexibility of the latter. Hot-plugging adapters offer a task-agnostic, generalized, and efficient alternative, enhancing the performance of various downstream tasks without incurring training costs.

(2) As shown in Table 1, direct fine-tuning of downstream heads can outperform TaCA, but it remains task-specific. The fine-tuning schedules and inference details for these paradigms are presented in Appendix C. In the video-text retrieval task, we fine-tune three distinct sets of downstream heads on the MSRVTT, MSVD, and DiDeMo datasets, resulting in the training of a total of 42M parameters. For the video classification task, fine-tuning the classification head requires nearly twice as many parameters as the TaCA model. However, TaCA requires training just once, with no additional fine-tuning costs, yielding a more parameter-efficient and training-free application in downstream tasks. This discussion will be incorporated into the final paper.

Table1: Comparison on Video-Text Retrieval datasets. The new visual fundation model (VFM) is ViT-L/14, and the old VFM is ViT-B/16.

Table2: Comparison on Video Classification datasets. The new visual fundation model (VFM) is ViT-L/14, and the old VFM is ViT-B/16.

Q2: Expanding Experiments to Additional Tasks (e.g., image generation or some NLP tasks).

A2: We appreciate your suggestion. Our extension to other downstream tasks, such as image generation, is documented in Appendix C.4, page 17. Although the styles of generated images are subtly differentiated, our method is adept at maintaining detailed semantics. This represents the early phase of our research; future work will aim at refining these solutions.

Q3: Generalization to Non-CLIP Models and Classic Vision Tasks. Can the proposed method work on classic vision tasks, like detection or segmentation?

A3: We have validated TaCA's effectiveness on non-CLIP models, such as Dinov2 and BEiT, in Table 9 (page 9). These models show slight improvements and meet the compatibility criterion for various tasks.

We have also applied our method to classical vision tasks, such as zero-shot image classification and zero-shot image-text retrieval, with results presented in Tables 1 and 2 demonstrating consistent improvements. The potential application of TaCA to detection and segmentation will be explored in forthcoming studies.

Table1: Zero-shot image classification on ImageNet and CIFAR100 datasets. We report the accuracy.

Table2: Zero-shot image-text retrieval on MSCOCO and Flickr datasets. We report the Recall@1.

Q4: Does TaCA work well when replacing different foundation models? Can we use it to replace a DINO ViT with an ImageNet-22k Supervised ViT?

A4: TaCA's adaptability across diverse foundational backbones is discussed in Table 9 of the main paper. While it is possible to replace a DINO ViT with an ImageNet-22k Supervised ViT using TaCA, the performance may be suboptimal if the new foundation model's representational capabilities are inferior. The efficacy of downstream tasks is contingent upon the foundational models' improved representational quality.

Q5: Incorporating Knowledge Distillation in Related Work

A5: We have included a new subsection on knowledge distillation in the related works of our revised manuscript, which can be found highlighted on page 18.

Q1: The approach is incremental.

A1: We reclaim that our key contributions are:

(1) For the first time, we introduce "hot-plugging upgrades", which directly deploys the new foundation models without retraining cost. Specifically, we introduce the task-agnostic compatible adapter (TaCA), which enhances performance across a variety of downstream tasks.

(2) Our method not only demonstrates compelling practical value but also boasts remarkable industrial applications. Unlike the traditional approach of fine-tuning specific downstream heads, which often incurs prohibitively high training costs and exhibits suboptimal transferability, our approach shines. To illustrate, the process of retraining a Q-Former in BLIP-2 demands approximately 9 days utilizing 16 A100 GPUs. TaCA, in contrast, offers a versatile task-agnostic capability, enabling seamless adaptation to various downstream tasks without incurring any finetuning costs.

(3) The significance and novelty of our contributions have been duly acknowledged by Reviewer #TD34 and #N62r, further attesting to the groundbreaking nature of our work.

Q2: Add experiments on more basic image classification and retrieval datasets (e.g., MSCOCO and imagenet)

A2: We have broadened the scope of our experiments to include fundamental image classification and retrieval tasks. The results, detailed in Tables 1 and 2, demonstrate TaCA's consistent performance improvements across these basic image tasks.

Table1: Zero-shot image classification on ImageNet and CIFAR100 datasets. We report the accuracy.

Table2: Zero-shot image-text retrieval on MSCOCO and Flickr datasets. We report the Recall@1.

Q3: TaCA utilizes an adapter to align a large-scale visual model with a smaller-scale visual model. In Table 2, TaCA only shows marginal enhancements in downstream video classification tasks compared to directly employing a large-scale visual model

A3: We acknowledge the confusion in our initial explanation. The purpose of TaCA is to efficiently align the old model with a large-scale or powerful visual model, thereby enhancing downstream task performance. In Table 2, the de-emphasized lines represent the results achieved by fine-tuning all downstream heads, which are task-specific and inefficient. Conversely, the emphasized lines depict the performance obtained by directly deploying the TaCA model without making any modifications to the downstream modules.

Q4: Add experiments on image-related tasks like image-text retrieval, image segmentation, and few-shot image classification.

A4: For further details on the experiments encompassing image-text retrieval, image segmentation, and few-shot image classification, please refer to the response to Question 2.

Q5: What would happen if Old VFM and New VFM were different? Can knowledge of different architectures be distilled? For example, distilling knowledge from miniGPT or LLAMA (decoder only architecture) to CLIP?

A5: (1) Our research in Table 9 establishes that TaCA can generalize across various foundational models, allowing for knowledge distillation between different architectures. However, the performance may be less optimal if the new model is not as capable as the previous one (e.g., replacing ViT-B with ResNet-50). (2) While this paper confirms TaCA's efficiency with visual models, its applicability to language models, such as distilling knowledge from decoder-only architectures like miniGPT or LLAMA to CLIP, remains an area for future exploration.