CLIP model is an Efficient Online Continual Learner

Q1. Definition of asymmetric negative example and comparison with similar methods We appreciate your suggestion to compare our method with [1] and [2]. In our work, asymmetric negative examples are defined as text labels without corresponding image samples within a batch. As discussed in Section 3.3, these samples, while useful for retaining knowledge of old classes, adversely impact the learning of new classes.

While [1] and [2] address cross-modal contrastive learning, their methods assume symmetry between positive and negative examples, unlike the task-agnostic online continual learning setup we address. Additionally, our experiments indicate that logit adjustment techniques used in [1] and [2] do not yield significant improvements in our setting. This prompted us to adopt a simpler yet effective approach of excluding asymmetric negative samples. We will revise the manuscript to clarify these distinctions and highlight the unique challenges addressed by our method.

Q2. Choice of CLIP as the Vision-Language Model Thank you for your feedback regarding the choice of CLIP over models like Flava and BEiT3. While these models offer strong multimodal capabilities, they are larger and more complex than CLIP, making them less practical for the resource-constrained online continual learning scenarios we target. CLIP’s simplicity and proven effectiveness across various tasks make it a better baseline for fair benchmarking.

Moreover, our ablation studies confirm that the proposed method is not dependent on any specific parameter-efficient fine-tuning (PEFT) technique, demonstrating generalizability. This underscores that our method is not merely a "trick" tailored for CLIP. We will explicitly include this rationale in the revised manuscript to better justify our choice.

Q3. Complexity of MiD-Blurry Online CL Setting We recognize the need for realistic benchmarks for online continual learning. The MiD-Blurry setup addresses this by introducing dynamic and imbalanced class distributions, which better resemble real-world online CL scenarios than classical benchmarks. As described in Section 3.1, MiD-Blurry challenges models with continually shifting class distributions, adding complexity beyond that of Si-Blurry or standard CIL setups.

To further illustrate its relevance, our experiments in Table 2 demonstrate the model's adaptability across distinct fine-grained datasets like CUB200, StanfordCars, and FGVCAircraft, which have minimal semantic overlap. This setup tests the model’s ability to adapt to diverse, fine-grained tasks. We will enhance our discussion to emphasize these challenges.

Q4. Suitability of Current Online CL Benchmarks We appreciate your point about the limitations of existing benchmarks for online CL. To fairly evaluate our method's performance, we compared it to [3]'s task-agnostic online CL setup, including both the Si-Blurry and our proposed MiD-Blurry settings. In [3], the DualPrompt method showed competitive results, and we include it as a comparison in our work as well. However, we explicitly note that DualPrompt was not designed for task-agnostic online CL and uses a non-VLM architecture. We have clearly marked its results in Table 1 for transparency.

Q5. Clarification on Line 138 and Incremental Classifier Limitations We appreciate your suggestion to compare with more non-VLM continual learning methods. In addition to DualPrompt and MVP, we are working to reproduce [4] and include that comparison as well.

We understand that adding incremental classification heads is a valid approach, as you rightly pointed out, it often underperforms and requires careful initialization. In contrast, VLM-based methods, by matching image-text pairs, effectively mitigate these issues and can leverage semantic information in language. Furthermore, open-world classification is a good property of VLMs, which is why CLIP is our preferred choice—not the only choice.

While we did not emphasize efficiency in the manuscript, we would like to clarify that text encoding can be precomputed and used as a fixed classifier, which is computationally equivalent to non-VLM methods.

---

[1] Contrastive Audio-Language Learning for Music. ISMIR 2022

[2] CrossCLR Cross-modal Contrastive Learning for Multi-modal Video Representations. ICCV 2021

[3] Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inference. ICLR 2022

[4] Anytime Continual Learning for Open Vocabulary Classification. ECCV 2024

Q1. Clarification of Figure 1 and SIT Visualization We appreciate the feedback on Figure 1 and acknowledge the potential confusion regarding the visualization of SIT. The figure's purpose is to illustrate the conceptual distinction between traditional classification and CLIP-based approaches. Specifically:

• Upper Part: The upper panel demonstrates how a classification model computes logits for prediction.
• Lower Part: The lower panel shows CLIP extracting features for both images and classes separately to perform matching. For SIT, the dashed arrow indicates that asymmetric class labels are not used during training at step t, highlighting that negative samples from prior classes are excluded. We will revise Figure 1 to make these distinctions clearer, including relabeling arrows and adding explanatory annotations.

Q2. Representation in Figures 4 and 5: We acknowledge that Figures 4 and 5 can be challenging to interpret. In these figures, we reduced the features to the same lower-dimensional space using PCA to facilitate the comparison of differences between methods. In Figure 4, we can observe that Continual-CLIP is effective at distinguishing data from different domains (CIFAR-100 and Flowers102), but it struggles to differentiate data within the same domain. SIT, on the other hand, significantly improves the matching of seen data (CIFAR-100) while maintaining the ability to distinguish unseen data (Flowers102). However, AIT excessively pushes image features away from text features, which undermines the model's ability to differentiate unseen data.

We will work on improving the clarity of these visualizations by adjusting the color schemes, labeling, and providing additional context to guide readers through the figures.

Thanks for your response. My concern about the figure representation has been addressed. However, I don't think the authors have addressed or tried to address my concern about the setting of SIT. I understand why you want to remove asymmetric contrastive samples. As I said, SIT can give you good performance when you evaluate each dataset, which is expected. However, this training manner cannot help you distinguish the positive images and negative texts across different datasets, which is important when doing open-world classification with the CLIP backbone. I don't see how compare SIT with other loss function can disprove this point.

Due to the above reasons, I will keep my rating for now.

Q1. Comparison with similar methods Thank you for highlighting the importance of comparing with [1] and [2]. Both works utilize complementary memory systems and learnable branches to mitigate catastrophic forgetting while ensuring flexibility. These methods are particularly suited for open-vocabulary classification tasks. Our method focuses on online continual learning by incorporating lightweight learnable adapters into each transformer block of the pretrained model, aiming for better flexibility and efficiency. We are actively reproducing the results from [2] and will include comparisons in the revised manuscript to highlight the relative performance of our approach.

Q2. Training and inference efficiency Thank you for your feedback regarding training and inference efficiency. PEFT methods typically focus on parameter efficiency by introducing relatively small learnable adapters and fine-tuning them, thus avoiding modifications to the pretrained parameters. While PEFT methods are known for parameter efficiency, their runtime performance during training and inference is important. Below is a table comparing the training and inference efficiency of PEFT methods against full fine-tuning, conducted on a single NVIDIA 3090 GPU.

Q1 The Difference Between Task and Time Step We apologize for any confusion in our description. In offline continual learning (CL), a "task" typically corresponds to an independent dataset or subset, with distinct label spaces or domains for different tasks. However, in online CL, we define a "task" as a stable data distribution over a specific time period, whereas a "time step" refers to a single batch of data sampled from the task's distribution.

In task-agnostic CL settings such as Gaussian, Si-Blurry, and our proposed MiD-Blurry, task boundaries are not clearly delineated, creating a more challenging scenario. Figure 1 illustrates these distributions, where the x-axis represents time steps, the y-axis represents classes, and color intensity indicates sample density. In Gaussian and uniform distributions, classes can appear at any time step, while decay classes persist beyond their initial appearance, leading to blurred task boundaries. We provide a clearer and more detailed version of this figure in Section A.1 of the appendix.

Q2. Reason for removing asymmetric negative samples Thank you for raising this point. As detailed in Section 3.3, we decompose the loss into $L_S$ (symmetric) and $L_A$ (asymmetric) components. The $L_A$ term penalizes the proximity of current image features to the text representations of previously seen classes. While $L_A$ aids memory retention, as the number of seen classes increases, it disproportionately hinders learning for new classes.

In Figure 3, we present a gradient analysis of the loss function, where the magnitude of the gradient represents the degree to which a particular class influences the model parameters. In Figure 3 (a,d), we evaluate the overall impact of each class on model parameter updates, with the class index sorted based on the first appearance time. As shown, in AIT, older classes predominantly influence the model as asymmetric negative samples (ANS), which contribute to $L_A$. At the same time, the gradients produced by symmetric negative samples (SNS) are significantly smaller than those of ANS. This indicates that image features are more difficult to pull closer to their corresponding text features and are more easily pushed away by other asymmetric text features. This conclusion is consistent with the observations in Figure 4. Therefore, we conclude that ANS has a more negative impact on continual learning.

To validate this point, we conducted experiments under the same settings as in Table 2, comparing SIT with other loss functions. Among these, Focal Loss [2] and LDAM [3] are classic loss functions designed for imbalanced learning. We also considered introducing weights into the loss function as $L=L_S+ \lambda_A L_A$, with $C_{batch}$ representing the number of classes in the current batch and $C_{seen}$ representing the total number of seen classes. As shown in the table, while more advanced logit adjustment strategies show incremental improvements in continual learning settings, they add complexity compared to the simplicity and effectiveness of SIT. Thus, we believe our approach offers a favorable trade-off between performance and complexity.

Q3. Concerns about Potential Class Leakage in CLIP Pretraining We acknowledge the concern regarding potential leakage from pretrained class names in CLIP. Per the data overlap analysis in [4], while CLIP uses extensive pretraining data, its performance still falls short on specialized benchmarks, suggesting limited leakage effects. Moreover, in task-agnostic CL settings (e.g., Si-Blurry, MiD-Blurry), some overlap in classes is unavoidable, and leakage is acceptable within these settings. We also emphasize that CLIP's zero-shot capabilities remain suboptimal in fine-grained scenarios, reinforcing the need for methods like ours to address such gaps.

Q4. Zero-shot performance We recognize the observed trade-off in zero-shot accuracy post-online CL (Table 2). This trade-off reflects the inherent balance between improving performance on seen classes and maintaining generalization to unseen ones. We note that methods like MaPLe prioritize generalization but exhibit limited plasticity, achieving performance close to untrained baselines in continual learning. SIT, by contrast, achieves a more balanced performance across both seen and unseen classes, as evidenced by its competitive results in both domains.

Q5. Comparison with Non-VLM Methods in Open-World Settings We agree that incorporating non-VLM baselines in open-world settings would provide additional insights. While open-set recognition and online CL have distinct goals—learning universal representations vs. avoiding catastrophic forgetting—we acknowledge the relevance of a comparative analysis. Preliminary results for open-set recognition methods, as shown in the extended Table from Q1, indicate SIT’s strong performance. Additionally, we are replicating methods from [5] to enhance our comparative evaluation. These results will be included in the final manuscript.

Q6. Typo in Equation (3) Thank you for pointing out the typo. We confirm that the numerator in Equation (3) should indeed be $Z_S$ for the $L_A$ component. This will be corrected in the revised version.

---

[2] Focal Loss for Dense Object Detection. ICCV2017

[3] Learning imbalanced datasets with label-distribution-aware margin loss. NeurIPS 2019

[4] Learning transferable visual models from natural language supervision. ICML 2021.

[5] Anytime Continual Learning for Open Vocabulary Classification. ECCV 2025.

Q1. Reason for removing asymmetric negative samples

We appreciate the opportunity to clarify this. As described in Section 3.3, the loss is decomposed into $L_S$ (symmetric) and $L_A$ (asymmetric) components. The $L_A$ term captures the impact of past classes, which indirectly pushes image features away from the text representations of previously seen classes. While this aids memory retention, it can hinder the learning of new classes as the number of seen classes increases.

Our gradient analysis in Figure 3 reveals that gradients from asymmetric negative samples (ANS) dominate and create a disproportionate influence on parameter updates compared to symmetric negative samples (SNS). This makes image features harder to align with their corresponding text features, as confirmed by Figure 4. Removing ANS mitigates this negative impact, as shown by the improved results of SIT-CLIP over baselines in Table 2.

Additionally, while more advanced logit adjustment strategies (e.g., Focal Loss, LDAM, weighted $\mathcal{L}_{\text{A}}$) show incremental improvements in continual learning settings, they add complexity compared to the simplicity and effectiveness of SIT. Thus, we believe our approach offers a favorable trade-off between performance and complexity.

Q2. Generalization of the Proposed Loss Modification We conducted extensive experiments across varied online continual learning settings such as Si-Blurry (see Section A.4) and compared our approach with offline CIL baselines (Section A.5). These results demonstrate the generalizability of our method. Notably, despite the added challenges of online CL, our method achieves competitive performance relative to offline baselines, underscoring its robustness and applicability.