We acknowledge the oversight in omitting specific details, including the adapter structure and design rationales, assuming a shared background with the readers. In response to your valuable feedback, we recognize the need to enhance accessibility for a broader audience. More implementation details will be meticulously incorporated into both the main paper and supplementary materials.

C4.1 Why do you set up N adapters for each domain? Why insert these adapters after the self-attention and feed-forward layers, respectively?

We regret any confusion. Given that adapter design is not the primary focus of our work, we adhere to recent literature [1,2,3] in incorporating multiple adapters (N) into the pretrained model for multimodal applications. These adapters are typically inserted after the self-attention and feed-forward layers in Transformer-based architectures. The primary objective of employing multiple adapters is to augment the model's capacity with minimal impact on computational cost and parameters. This design choice allows for the dynamic merging of expert models for each test instance with negligible overhead, as detailed in Section C3.3 of the rebuttal.

C4.2 What is the exact structure of the adapter?

Following previous works [1,2,3], the adapters in our approach consist of a downsampling and an upsampling linear layer, along with a residual connection. The linear layers are set with an 8x downsample scale to intermediate hidden size and the upsampler maps back to the original dimensionality [1].

[1]: Sung, Yi-Lin et al. "Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks." CVPR 2022.
[2]: Houlsby, Neil, et al. "Parameter-efficient transfer learning for NLP." ICML 2019.
[3]: Yang, Antoine, et al. "Zero-shot video question answering via frozen bidirectional language models." NeurIPS 2022.

C4.3 What exactly is meant by the pre-trained model f in Eqn. (1)?

We refer to the original pretrained model as “f”, i.e. the model used for zero-shot inference (FrozenBiLM in our case) except that we extract the hidden states from the 4th last layer as the output. We thank the reviewer’s suggestion and will add a detailed description in the supplementary materials.

C4.4 What does the symbol k in the baselines section on page 6 refer to?

We regret any confusion caused by the omission of the definition of the symbol 'k' in our manuscript. In our context, 'k' denotes that only the top-k models participate in the merging process (as described in Section 3.3). The rationale behind selecting only the top-k models is to filter out irrelevant adapters, while still capitalizing on the advantages of adapter merging. Additionally, maintaining a fixed 'k' ensures a constant computational cost, regardless of the number of domains. We appreciate the reviewer's understanding, and we will include this clarification in the revised manuscript.

Besides the implementation details, please take a look at our General Response above to see more about experimental setup and generalizability. We are happy to have more in-depth discussions with you if you have further concerns!

If our response addresses your concerns, please consider increasing the scores. Also, feel free to ask follow-up questions during the rebuttal period.

We appreciate your insightful review and constructive suggestions. In response, we have diligently addressed all raised concerns, and we believe our revisions effectively resolve them.

C3.1 The definition of domain regarding VideoQA is not clear.

Our definition of the domain diverges from the conventional definition in the continual learning community. Please refer to the “Definition of domains” of G1 in General Response for more detail.

We also report the results on 7 diverse VidQA domains (separated by video type) in “DIL on more diverse domains” of G2 in General Response.

C3.2 It would be better to experiment with more clarified domains instead of datasets.

We further validate our method DAM on 7 more diverse domains: social videos (MSVD), instructional videos (iVQA), movie videos (LSMDC), long activity videos (ActivityNet), indoor human videos (AGQA), traffic videos (TrafficQA) and virtual videos (Env-QA). Results show that DAM has even less forgetting on more clarified domain settings.

Please refer to "DIL on more diverse domains" of G2 in General Response for more details.

C3.3 The proposed DAM necessitates all the adapters for inference. It is necessary to analyze the efficiency.

The adapters introduced in each domain contribute merely 2.5% of the pretrained model's parameters (CLIP-L/14 + DeBerTa-V2-XLarge), totaling 30M parameters. With 10 domains, this results in only a 25% increase in parameters, a reasonable augmentation given DAM's robust performance.

In terms of computational cost, merging adapter parameters incurs just 0.09 GFLOPs (30M *(2k-1), k=2 in our case), notably lower than the 162 GFLOPs required by CLIP-L/14 for a single image processing. We appreciate the reviewer's suggestion and will integrate this analysis into the revision.

C3.4 The authors obtain the upper-bound results by individually finetuning target datasets. My concern is that this ‘upper-bound’ may not be the actual upper-bound for incremental learning because of data augmentation.

We conducted both individual finetuning (Ind-FT) and multi-task finetuning (MLT-FT), the latter involving joint training on all datasets. As indicated in the table, MLT-FT yields a notable 1.8% enhancement in MSVD-QA, while maintaining a comparable average accuracy to Ind-FT (52.5% vs. 52.6%).

In the realm of continual learning, multi-task finetuning is conventionally considered an upper bound. We apologize for the confusion and will replace the upper bound with MLT-FT in the revised manuscript.

C3.5 The gap between DAM's results and these upper bounds is too small to show the need for future efforts in a novel task setting. Also, why is the performance on ActivityNet not as good as others?

We thank the reviewer for pointing out the room for future work. There are still unaddressed issues that lead to a relatively large accuracy drop on some specific datasets, i.e. MSRVTT (-4.5%) and ActivityNet (-6.1%). Other future work includes experimenting with many more domains (or datasets) like 100, and extending our method to extremely large models (i.e. > 10B parameters).

Regarding the ActivityNet dataset, we analyze the performance drop might be due to the high similarity of its question type with the iVQA dataset, while the videos are quite different. This inconsistency may lead to confusion within our proposed model.

C3.6 Providing more analyses /comparisons on an OOD setting (aside from Fig.3(b)) would make the experiments more sound.

We further experiment on 7 diverse domains as described in G2 of the General Response. We continually train the proposed DAM on the first 3 domains and evaluate the trained model on the other 3 domains. From the table, DAM outperforms S-Prompts consistently in all the domains, attributed to the larger capacities of adapters and the robustness of dynamic merging.

C3.7 In Sec. 3.2, what specific model does ‘f’ refer to?

We refer to the original pretrained model as “f”, i.e. the model used for zero-shot inference (FrozenBiLM in our case) except that we extract the hidden states from the 4th last layer as the output. We thank the reviewer’s suggestion and will add a detailed description in the supplementary materials.

If our response addresses your concerns, please consider increasing the scores. Also, feel free to ask follow-up questions during the rebuttal period.

C2.7 It is worth discussing whether there are unique challenges related to continual learning in the domain of VideoQA.

As mentioned on Page 2 (below Fig. 1) in the main draft, in contrast to unimodal tasks like image classification, VidQA involves both video and text inputs, necessitating the model to jointly reason on both modalities. Within VidQA domains, question types may exhibit significant similarity (e.g., when, how, where), while the corresponding answers can vary considerably. The juxtaposition of a shared input space (video+question) and a diverse output space adds complexity to continual VidQA. Consequently, router-based continual learning methods may face challenges in accurately predicting domain identities, leading to inferior answer prediction accuracies.

Our proposed method, DAM, effectively mitigates the issue of inaccurate routers by dynamically merging domain adapters (refer to Sec. 4.4). We appreciate the suggestion and will incorporate this discussion in the revision.

If our response addresses your concerns, please consider increasing the scores. Also, feel free to ask follow-up questions during the rebuttal period.

We appreciate your insightful review and constructive suggestions. In response, we have diligently addressed all raised concerns, and we believe our revisions effectively resolve them.

C2.1 Limited Dataset Diversity: The article uses 6 datasets with relatively small differences between them, especially MSVD and MSR-VTT.

We thank the reviewer for the valuable suggestion. We believe the confusion is caused by the question of whether domains should be orthogonal.

We elaborate on our point in “Dataset Diversity” of G1 in the General Response above, please refer to that section for more detail. We also report the results with more diverse datasets in G2 in the General Response.

C2.2 The out-of-date issue proposed in Fig. 1 hasn’t been evaluated.

We further evaluate the CIL scenario to mimic the evaluation of the out-of-date issue. The old model may not be able to answer questions in new tasks as they never see the classes in the new tasks before, which is similar to the example in Fig. 1 that a VidModel trained in 2021 may struggle with questions about the 2023 movie “Barbie”. Our proposed DAM method archives significant improvements compared to the second-best approach on CIL settings.

We refer the reviewer to G2 (Generalization) in the General Response for more details.

C2.3 The use of internet-sourced videos in the datasets does not fully explore the potential challenges posed by more diverse datasets, such as those collected in virtual environments (Env-QA), traffic scenarios (TrafficQA), or indoor human activities (AGQA).

Thanks for the insightful suggestion. We further experiment with 7 diverse video datasets including all the datasets that the reviewer mentioned. Results show that our DAM method achieves even less forgetting on more diverse datasets.

Please refer to “DIL on more diverse domains” of G2 in General Response with more detail.

C2.4 Their simple design might not generalize well to more challenging datasets. The reviewer has doubts about this.

In G2 Generalization of the General Response, we showcase DAM’s generalization ability by experimenting on DIL with more diverse domains, CIL, and TIL scenarios. In Sec 4.5 and Tab. 4 in the main paper, we show DAM can be applied to the visual (image)-QA task using a different large model BLIP2.

All the experimental results support the conclusion that DAM can generalize well to challenging datasets, scenarios, and even more tasks and models. This is because we do not need complex design on top of pretrained large models as they already provide enough generalizability.

C2.5 The article does not provide a fair comparison with few-shot learning. A direct comparison between in-context learning using FrozenBiLM and the proposed approach could offer a more comprehensive evaluation.

We appreciate that the reviewer pointed out another interesting direction (in-context learning, ICL) to address the continual learning problem. We further experiment with the one-shot in-context learning using FrozenBiLM and report the results below. The proposed DAM outperforms one-shot FrozenBiLM by 29.3% in average accuracy. The inferior performance of one-shot ICL is because LLM with at least 6.7B parameters begin to have in-context learning ability on multimodal tasks (Koh et al. 2023).

We will add this comparison and mention that ICL with LLM could be a potential direction for continual learning in the revised manuscript.

Koh, Jing Yu et al. "Grounding Language Models to Images for Multimodal Inputs and Outputs." (2023).

C2.6 The article does not provide sufficient evidence of severe catastrophic forgetting in current large models.

As shown below, sequentially finetuning (Seq-FT) the large model (FrozenBiLM) without any continual learning technique leads to a 12.7% accuracy drop, which we believe is sufficient to show severe catastrophic forgetting. Thanks for the suggestion and we will add this baseline in Table 1 in the revision.

We appreciate your insightful review and constructive suggestions. In response, we have diligently addressed all raised concerns, and we believe our revisions effectively resolve them. Should any of your questions or concerns persist, please feel free to communicate with us.

C1.1 The loss function used during the training of adapters.

We didn’t apply additional loss functions to the adapters. Our loss function is the cross-entropy loss between the predicted tokens and ground-truth answer tokens, which is the same as the original VidQA model, i.e. FrozenBiLM in our paper.

C1.2 The contributions of the paper may be insufficient, i.e. related to combining domain-specific adapter learning and model merging techniques.

While we acknowledge drawing inspiration from both continual learning and model merging, our work introduces a novel technique, "Dynamic Adapter Merging (DAM)," previously unexplored in both domains. Unlike existing methods that yield a single or a set of fixed models, DAM innovatively generates a personalized expert model for each testing sample with minimal overhead, which is not a simple adaption or combination of existing model merging methods and continual learning methods.

Furthermore, our novelty extends to the in-depth analyses (as highlighted by [thbM] and [DkBm]) in Sec. 4.3 and Sec. 4.4, detailing how and when model merging can enhance the effectiveness of the router-based technique in the continual learning domain.

Finally, the empirical results of DAM could be inspiring to the model merging community. As shown in the table below, we compare DAM with the other merging approaches, including average merging and RegMean (Jin et al., 2022), while all the approaches merge the same set of domain models that are individually finetuned on each dataset. Unlike the other approaches, DAM determines the merging ratios for domain adapters based on the input instance, and this flexibility makes DAM outperform RegMean by 6.0% and average merging by 7.5% in average accuracy. The results show the potential of the proposed selective and dynamic merging strategy to inspire model-merging communities.

Thanks for your suggestion and we will highlight these contributions in our revised manuscript.

C1.3 The non-parametric router function is simply based on cosine similarity with no improvements.

As demonstrated in Table 2, Section 4.3 of the main paper, our straightforward cosine-similarity-based router surpasses all existing counterparts with more complex router design, including MLP-based learnable routers proposed by L2P and CODA-Prompts, as well as the KMeans-based router employed in S-Prompts. Notably, our simple non-parametric router exhibits a 2.7% higher accuracy in domain-identity prediction compared to the second-best router while having no training cost and faster inference speed. We also tried more complex router functions on the DAM framework. As shown in the table below, their performance is only comparable to our router. Thus, we keep our router simple but effective in the DAM approach.

C1.4 Generalizability: Can such ideas be extended to Class-Incremental Learning (CIL) and Task-Incremental Learning (TIL) scenarios?

We thank the reviewer for the valuable suggestion. We report both the CIL and TIL results in G2 in General Response. Results show that DAM is capable of outperforming the state-of-the-art continual learning approach under all settings.

Additionally, In Sec 4.5 and Tab. 4 in the main paper, we show DAM can be applied to the visual (image)-QA task using a different large model BLIP2 on 4 ImageQA datasets.

C1.5 Stability: How were the experimental results in the article obtained, multiple runs or a single run?

The results in our main table (Tab. 1) are obtained with 5 different random seeds. We report an average accuracy of 50.23 ± 0.12%. Furthermore, in Tab. 9 (Ablations on domain orders) of the Appendix (following the reference section in the main paper PDF), we randomly sampled five different domain orders, observing consistent average accuracies (50.56 ± 0.26%).

These findings affirm the stability of our method to variations in both random seeds and domain orders.

If our response addresses your concerns, please consider increasing the scores. Also, feel free to ask follow-up questions during the rebuttal period.