Learning without Forgetting for Vision-Language Models

Q4 There's no analysis of time and memory usage, except for a basic mention of memory requirements in supplementary materials.

A4 We thank the reviewer for the suggestion. Running time and memory budget are two important factors for real-world applications. Correspondingly, we have compared PROOF to other competitors to show the memory budget comparison in Supplementary Section D.2 in the initial submission. We can find that PROOF uses the same scale parameter as other competitors while having the best performance, showing the memory efficiency of our proposed method.

Additionally, according to the reviewer’s comments, we have added the running time (seconds) comparison of different methods on a single Tesla V100 GPU in Supplementary Section D.3 and report the results here.

As we can infer from the table, PROOF has the lowest running time compared to other competitors. It indicates that PROOF has the potential to run efficiently in real-world applications.

In summary, we show PROOF is efficient in terms of the memory budget or running time.

Q5 The paper's narration could also be improved.

A5 We thank the reviewer for the suggestion. In this revision, we have made Section 3 much shorter by deleting discussions of baseline methods and increased the length of Section 4 (especially Section 4.2) to fully introduce the cross-modal fusion module. Additionally, we have added the requested experimental results and discussions in Supplementary Section D.3, E.4, E.5. We hope the revision can improve the quality of the narration.

Q6 It looks like the supplementary materials are more informative and present more interesting results w.r.t. the main paper. Why did the authors exclude them from the main paper?

A6 We thank the reviewer for the suggestion. In the main paper, we use class-incremental learning as the most widely applicable continual learning setting and conduct experiments on various class-incremental learning datasets and protocols (main paper Section 5.2, Table 1, Figure 2). We also investigate each component’s influence with ablation studies (main paper Section 5.3, Figure 3). Finally, we also verify PROOF’s potential in other continual learning settings (e.g., continual cross-modal retrieval) and other applications (preserving zero-shot ability) in the main paper Section 5.4 and Figure 4. In the supplementary, we mainly supply additional results and implementation details that cannot be included in the main paper due to the page limit.

However, all essential parts of the supplementary material are mentioned and discussed in the main paper. For example, Supplementary Section B indicates PROOF can work competitively in continual cross-modal retrieval, which is discussed in the main paper Section 5.4 and Figure 4 (a). Supplementary Section D.10 discusses the results on a new TV series dataset, which is discussed in the main paper Section 5.4 and Figure 4 (b). Supplementary Section C discusses how to maintain the model’s zero-shot performance, and we have also included it in the main paper Section 5.4 and Figure 4 (c). Apart from these parts, other results in the supplementary act as the additional results of the main paper, e.g., the full results or detailed performance in Table 4. Hence, there is no critical information excluded from the main paper.

We sincerely appreciate your great efforts in reviewing and joining the discussions of this paper.

We first summarize the prior discussions. In our rebuttal, we have tried our best to supply experiments and revise the manuscript. As the reviewer comments, only concerns about the innovation and applicative scenarios are not clearly addressed. Hence, we thank the reviewer for being satisfied with our efforts on different class orders (Q3), memory usage and running time (Q4), narration (Q5), paper organization (Q6), definitions (Q7), and analysis of exemplars (Q8).

Afterward, we would like to further discuss with the reviewer on applicative scenarios. As the reviewer commented in the initial review, "It would be more interesting to see its performance on tasks like incremental captioning or retrieval." Correspondingly, in this paper, we design learning mechanisms to enable a pre-trained vision-language model to learn new tasks continually. To achieve this goal, we first design additive projections based on the frozen encoder so that pre-trained features can be optimized to trace task-specific features. Additionally, to promote cross-modal fusion and contextualized classification, we design a cross-modal fusion module for better-contextualized embeddings. We also utilize visual and textual prototypes as the context information and design a novel context prompt to serve as task anchors to resist forgetting. We present PROOF as a universal algorithm that can be applied to various continual learning scenarios and various vision-language models. Apart from the benchmark class-incremental learning setting against other state-of-the-art methods, we also conduct experiments on continual cross-modal retrieval tasks with other vision language models (BEiT-3) in the main paper Section 5.4, showing PROOF still outperforms other competitors by a substantial margin. Such experiments are already contained in the initial submission, which adequately fits the reviewer's comments that "it would be more interesting to see its performance on tasks like incremental captioning or retrieval."

Hence, the reviewer's comments make us confused on this aspect. On the one hand, the reviewer is expecting "its performance on tasks like incremental captioning or retrieval." On the other hand, we have shown that the initial submission already contained the required results, but the reviewer still says, "having concerns on applicative scenarios."

After tasking about the applicative scenarios, we move on to the "innovation" part. As the reviewer claimed, "the PROOF mechanism, while innovative, lacks depth. Projection networks are common in continual learning, and the new context definition isn't explored." Although the reviewer did not mention any related papers supporting "Projection networks are common in continual learning," we have shown in the previous rebuttal that related papers on projections and continual learning are working on the gradient aspect. These works aim to project gradient updating directions not to harm existing knowledge. By contrast, our paper works on feature projection, which projects the pre-trained features into parallel subspaces to build task-specific features. Hence, they differ substantially from the motivation to the operation, except for the similar name of "projection." As we commented in the prior rebuttal, we sincerely hope the reviewer can provide any other related papers to support the claim of "having concerns on the innovation," we will try our best to tell the difference.

Finally, we highlight again our contributions in this paper. Through expandable task-specific projections, we enable a pre-trained vision-language model for continual learning. We also designed a cross-modal fusion module to contextualize the embeddings and less forgetting. The ideas of learning task-specific projections, learning cross-modal fusion with context information, and joint adaptation with context prompts are innovative and have not been explored by any prior work. PROOF is a unified framework that can be applied to various continual learning scenarios and various vision-language models, showing strong performance in various applicative scenarios.

Once more, we appreciate the time and effort you've dedicated to our paper. We sincerely hope the reviewer can review our rebuttal to dispel the misunderstanding on the innovation and applicative scenarios.

We sincerely appreciate the time and effort you have invested in reviewing our paper, and your insightful feedback has been a critical factor in enhancing the overall quality of our work. Following your latest comments, we have revised the paper to highlight our innovation and application scenarios. Specifically,

• In Abstract, Section 1 and Section 3.1, we highlight the diverse application scenarios of PROOF, including class-incremental learning and continual cross-modal retrieval.
• In Section 2, we discuss related works on gradient projections and show the differences between PROOF and these works.
• In Section 5.4, we add more details and experimental results about the continual cross-modal retrieval task in order to highlight the various application scenarios of PROOF.
• In Supplementary Section E.6, we fully discuss the differences between our work and related topics in gradient projection. We show that these works differ from idea to practice and highlight our main contributions in enabling a vision-language model to continual learning tasks.

We have highlighted the revised part in our manuscript in blue color. Following your suggestions, we have made the innovation for our paper much clearer, added more discussions about related works, added more details and results of other continual learning scenarios, and tried our best to address all other concerns.

As the discussion period is approaching its conclusion, we are eagerly anticipating your thoughts on our revision. Your additional feedback at this stage would be invaluable to us.

Please let us know if there are any further clarifications or specific experiments you would recommend to strengthen our paper. We are committed to engaging in a productive dialogue to address any residual issues.

Moreover, if our revisions and detailed responses meet the concerns initially raised, we would be grateful if you could consider supporting the paper by increasing the score. Your acknowledgment of our efforts in improving the paper would be immensely appreciated. Thank you once again for your invaluable contributions to refining our research.

Best regards,

Authors

Q2/Q5 The extent to which contextual information is effective has not been extensively studied. The projection fusion method proposes to contextualize and merge embeddings and contextual information using self-attention. However, in the experiments, only the results of Projection & Fusion are compared with Projection & Fusion & Context Prompt, without explicitly evaluating the effectiveness of the concatenated context information in Q, K, V as [P_i(z), Context] in self-attention, or the effectiveness of the context prompt. In other words, the final context information is defined as Context = [P, W, C], but the specific contributions of W and C to the final results need further analysis. How does each component of the contextual information contribute to the final results?

A2 We thank the reviewer for the suggestion. In fact, the ablation study on the context components is extensively researched in Supplementary Section D.5 in the submitted version. We explain the general results here and refer the reviewer to supplementary materials for more details. Specifically, we conduct experiments by varying the compositional components in the Context, i.e., Context=[P], Context=[W], Context=[P, W], Context=[P, W, C]. For these variants, we keep the other settings the same and report the incremental performance.

From the results, we observe that using visual prototypes or textual prototypes alone yields similar performance, and the impact of adjustment is marginal. However, when visual and textual prototypes are jointly utilized as context information, the model can learn from cross-modality and achieve better performance. Lastly, the introduction of context prompts into the context further enhances the performance of PROOF, resulting in the best performance among all variations. These ablation studies verify that each component in the context helps in the final prediction.

Q3/Q4 The evaluation metric used may not provide a comprehensive measure of the extent of forgetting. To what extent the proposed method could alleviate forgetting?

A3 We thank the reviewer for the suggestion. As we responded in A1, we consider the forgetting measure in [1][2] to measure the task-wise forgetting degree. Results show that our proposed PROOF has the lowest forgetting among other competitors. We have included these additional results in Supplementary Section E.4.

[1] A Comprehensive Survey of Continual Learning: Theory, Method and Application. arXiv 2023.

[2] Online continual learning in image classification: An empirical survey. Neurocomputing 2022.

The authors have addressed most of my concerns. I would like to raise my score to 6.

Q2 Representation Drift - The post-attention module representations learned by the frozen projections may drift or shift slightly during new task training due to weight updates elsewhere. Small drifts can accumulate.

A2 We thank the reviewer for the suggestion. It must be noted that there are two main steps in our method, i.e., feature extraction and cross-modal fusion. The extraction process contains the pre-trained embedding and projection, which is designed to extract task-specific features. By contrast, the attention module is responsible for fusing cross-modal information so that features can be more discriminative. Hence, such fusion is irrelevant to the task, which should be generalizable across tasks. In other words, even if the features shift slightly, the attention process still works to fuse and highlight task-specific information.

On the other hand, the whole optimization flow is continuous, even if the feature representation is changing due to projections. In other words, the drift of former extracted features will result in the corresponding adaptation of the attention module. Since the model is trained with exemplars and current datasets (as defined in Eq. 7), optimizing the loss considers all essential features among all seen classes for decision. The representation drift of former stages will be compensated by the latter stages.

Finally, we must highlight an essential module in the attention process, i.e., the context prompt C. It serves as the task anchor for each task, which is specially learned for each new task. Since the cross-modal fusion process will contextualize the visual embedding with the context information, the context prompt can identify the specific task to highlight important features. As a result, the negative impact of representation drift can be alleviated in our method.

Thanks to the Authors for the extended explanations.

They are certainly helpful. Although they improve my view of the paper, I am confirming my initial rating.