AnaCP: Toward Upper-Bound Continual Learning via Analytic Contrastive Projection

We acknowledge the similarity of our method to RanPAC; however, RanPAC directly uses features from the frozen PTM without adapting them to downstream task data. Our motivation was to address this limitation by introducing the contrastive projection, which enhances feature quality before classification.

• Question 1:
  Our method has a contrastive projection (CP) layer that improves the features by pulling together samples from the same class and repelling class means from one another. This projection is learned analytically based on the class means and the Gram matrix of the PTM features, and does not require any exemplars or replay data.

  However, the prediction is performed by an ELM classifier built on the output of the CP layer. In principle, we could also learn the ELM classifier analytically (similar to the CP layer), but this would require storing the class means and Gram matrix of the projected features. Unfortunately, the CP layer changes with each new task — each time a task is added, the projection is updated — so unlike the PTM, the output space of the CP layer is not fixed. Therefore, to compute these statistics in the new space, we rely on generated pseudo-replay data.

  Note that if we remove the ELM classifier and use an NCM classifier directly with the target prototypes, we no longer need the pseudo-replay data. The results of the NCM classifier are slightly worse than those of ELM, but still outperform the baselines, as shown in row 3 of Table 3.

• Question 2:
  Thank you for the suggestion. We have included a comparison without the random projections. The results below show final accuracy using the DINOv2 backbone:

  proj_dim	CIFAR-100	ImageNet-R	CUB	T-ImageNet	Cars
  no projection	90.35	83.29	89.34	85.30	87.29
  1000	90.64	84.01	89.56	85.48	88.23
  2000	91.35	85.15	89.97	86.24	89.93
  5000	92.15	86.60	90.48	87.71	90.65
  10000	92.75	86.86	90.60	88.11	90.73

  We observe that using no projection performs slightly worse than setting "proj_dim=1000". As the projection dimension increases, accuracy also improves. Note that the original feature dimension is "d=768".

• Question 3:
  Conceptually, our approach follows a similar idea as SimCLR and other contrastive learning methods: pulling same-class samples together and pushing different classes apart. The key difference lies in how these objectives are achieved. SimCLR and similar methods rely on gradient-based optimization to learn embeddings using a contrastive loss, which is not well-suited for continual learning due to the incremental nature of learning and catastrophic forgetting from backpropagation. Our motivation is to achieve the same contrastive effects analytically, so we can support incremental updates while avoiding forgetting (also see the forgetting results below).

• Question 4:
  We did not initially report forgetting metrics because our method, like most prototype-based methods, is inherently immune to catastrophic forgetting. This is because we operate on a frozen PTM, and all statistics (class means and Gram matrix) can be incrementally updated without overwriting previous information. Note that even if we had access to all data simultaneously, we would end up with the same class means and Gram matrix, indicating that our approach does not suffer from the traditional forgetting.

  Although this might not hold for the final ELM classifier, which is trained using pseudo-replay, we found its effect to be minimal in practice. This also explains why our method can achieve accuracy on par with joint training. To empirically support this, we have included additional results under the task-incremental learning (TIL) setting, where the task ID is known at inference time. We present the matrix A[t][i], whose elements represent the accuracy on task i after learning the first t tasks on CIFAR-100 with 10 task splits, using DINOv2 as the backbone.

  The matrix A[t][i] is shown below:

  0.987   -       -       -       -       -       -       -       -       -    
  0.987   0.981   -       -       -       -       -       -       -       -    
  0.989   0.978   0.980   -       -       -       -       -       -       -    
  0.988   0.981   0.979   0.984   -       -       -       -       -       -    
  0.988   0.975   0.981   0.986   0.984   -       -       -       -       -    
  0.987   0.977   0.978   0.982   0.980   0.985   -       -       -       -    
  0.988   0.973   0.978   0.982   0.980   0.984   0.958   -       -       -    
  0.987   0.976   0.976   0.979   0.979   0.983   0.955   0.988   -       -    
  0.985   0.973   0.977   0.979   0.979   0.983   0.957   0.984   0.951   -    
  0.986   0.973   0.977   0.978   0.981   0.983   0.955   0.984   0.947   0.979
  

  We chose the TIL setting for this analysis because it provides clearer insight into forgetting. In CIL, accuracy will naturally drop as more tasks are learned, not necessarily due to forgetting, but because the classification problem becomes increasingly more difficult (e.g., distinguishing between 100 classes is harder than between 10).

  In contrast, TIL keeps the classification difficulty fixed because the task ID is provided at inference time. For example, task 1 always involves classifying among the same 10 classes, regardless of how many tasks have been learned. If forgetting occurred, it would show as a drop in earlier task accuracy. However, as seen in the matrix, the first column (task 1 accuracy) remains nearly constant across all steps. The same applies to other columns, indicating our method retains prior knowledge effectively.

We appreciate your comments. If we have addressed your concerns, we would be grateful if you could consider updating your score.

• We appreciate the suggestion regarding the presentation of the mathematical formulation. We will improve the readability in the revised paper.

• We acknowledge that our method relies on PTMs for well-learned representations, which was also mentioned in the limitation section in the paper. However, this is a common characteristic of most modern CIL approaches using PTMs such as prototype-based methods, prompt learning, and even fine-tuning, which similarly require access to high-quality representations from PTMs. Additionally, we believe this dependence is not restrictive, as PTMs are increasingly accessible across different domains and modalities, and their quality is also getting better.

• The motivation behind Lemma 4.1 is to move the class means in directions that reduce their pairwise cosine similarity. Specifically, each $w_i$ in the subspace $SV_T$ is perturbed by $\alpha \delta_i e_i$, where $e$ vectors form an orthonormal basis. This means each $w_i$ is shifted in an orthogonal direction by a magnitude of $\alpha$. The sign of $\delta_i$ determines whether the shift is in the direction of $e_i$ or the opposite. This condition arises from the proof of Lemma 4.1, where we take the derivative of the function $f_i(\alpha)$, representing the sum of cosine similarities for class $i$. By choosing $\delta_i$ in this way, we ensure that the slope of $f_i(\alpha)$ at $\alpha = 0$ is negative, guaranteeing that a small positive $\alpha$ will reduce cosine similarity, i.e., shift the class means away from one another.

• We agree that drawing connections to neural collapse provides a valuable perspective, and we will address this in the revision. Indeed, our contrastive projection shares important similarities with this phenomenon. In neural collapse, all samples within a class collapse to their class mean, which aligns with our method, where we explicitly map all samples from a class to their respective class means. Neural collapse also leads to class means forming a simplex equiangular tight frame, i.e., every pair of class means is equally separated. However, unlike neural collapse, our goal is not to enforce equidistant class means. Instead, we seek to preserve the underlying geometry of the features while pushing the class means away from one another, inspired by contrastive learning. Also, neural collapse occurs in the terminal phase of training (TFT) when the loss approaches zero, which makes it difficult to achieve in continual learning due to incremental learning and catastrophic forgetting. Our method minimizes inter-class variation and enhances class separability analytically without fine-tuning the PTM.

• We will also carefully review the manuscript for grammar and language clarity.

• Comparison regarding RanPac:

As shown in Table 3, increasing the dimension $D$ generally leads to improved performance. We wanted to use the same dimension for all the methods that utilize this technique (KLDA also uses $D=5000$). Therefore, we set $D=5000$ for all methods to allow a fair comparison. Regarding the activation function, we did not observe significant differences between ReLU and GELU in our experiments. We opted for GELU as it seemed more intuitive: since the goal is to expand the feature dimension, ReLU would zero out all negative values, which contradicts the intention of fully utilizing the expanded dimension. Regarding fixed regularization, we aimed to demonstrate that the method performs consistently well across datasets using the same regularization value. Otherwise, it would raise the concern of what would happen if we wanted to learn two datasets that require completely different regularization values at the same time.

To further alleviate your concern, we re-ran both ACP and RanPAC with DINOv2 as the backbone, using $D=10000$ and ReLU for both methods. For ACP, we retained our fixed regularization. For RanPAC, we report results using both fixed regularization and its original cross-validation approach:

The results show that cross-validation yields slightly better results overall, although it hurts performance on dataset T-ImageNet. We also note that ACP could be extended to use cross-validation if desired. More importantly, ACP continues to outperform RanPAC consistently across all datasets.

• Motivation to regress a class prototype:

If we understand your question correctly, you are asking why we regress the target prototypes instead of directly returning the class means as the output of the contrastive projection (CP) layer. If that is indeed the question, this approach would not work, since, contrary to the suggestion, we do not have an explicit mapping from the samples to the class means during inference, as the labels are not available. Therefore, directly outputting the class mean is not feasible at test time. Also, the target prototypes we regress are not exactly the original class means; the class means are shifted to improve their separation through Lemma 4.1. We are effectively learning a mapping that, without requiring the label during inference, improves the features by pulling samples from the same class together and pushing them away from other classes. Then, the classifier is learned on top of the improved features, as depicted in Figure 1, which is better than learning it directly on top of the original PTM features.

• Whitening and de-whitening steps:

Our goal is to improve class separability, which we quantify by the sum of pairwise cosine similarities between class means. It is indeed possible to apply SVD directly to $C$ and modify its singular values to reduce this sum. However, there is a key limitation with cosine similarity that we aim to address: cosine similarity does not take into account the scale or variance of each feature dimension. For example, in a dimension with low variance, even a small separation is significant, whereas in high-variance dimensions, a much larger separation is needed to achieve the same effect.

To resolve this, we apply whitening transformation, which is a common data processing method. Whitening sets the variance of all feature dimensions to 1, allowing for a more meaningful measure of separation across directions. This process is closely related to Mahalanobis distance, which has been shown to outperform cosine similarity in related contexts (we refer the reviewer to FeCAM). By whitening the features, we essentially assess separation using Mahalanobis distance, and after increasing the separation, we de-whiten the features to restore the original variance. We will clarify this point further in the revision.

• Lemma 4.1 explanation:

The intuition behind Eq. 13 is that the diagonal shift is intended to perturb the class means in directions that reduce their pairwise cosine similarity. Specifically, each $w_i$ in the subspace $SV_T$ is perturbed by $\alpha \delta_i e_i$, where $\{e_i\}$ vectors form an orthonormal basis. This means each $w_i$ is shifted in an orthogonal direction by a magnitude of $\alpha$. The sign of $\delta_i$ determines whether the shift is in the direction of $e_i$ or the opposite. This condition arises from the proof of Lemma 4.1, where we take the derivative of the function $f_i(\alpha)$, representing the sum of cosine similarities for class $i$. By choosing $\delta_i$ in this way, we ensure that the slope of $f_i(\alpha)$ at $\alpha = 0$ is negative, guaranteeing that a small positive $\alpha$ will reduce cosine similarity.

Regarding the choice of $\alpha$, if it is too small, the shift becomes negligible; if it is too large, the reduction in cosine similarity may not hold. While it is again possible to optimize $\alpha$ via cross-validation on each dataset, our goal was to show that a single value works across all datasets. We empirically selected $\alpha = 1$ by validating on CIFAR100 training data, searching over the range $[0.1, 1, 2, 5, 10]$. This value was then used consistently across all datasets without tuning.

We agree that a more detailed explanation of Lemma 4.1 and its implications could enhance clarity, and we will include an expanded discussion in the revision.

• Suggested papers:

Thank you for the helpful suggestions. We will ensure that the recommended papers are cited and discussed in the revised manuscript, as well as incorporated into the experiments.

InfLoRA and HidePrompt are different approaches from ours. LoRanPac, which is more related to our work, projects the PTM features into a very high-dimensional space (e.g., 100,000) and solves the over-parameterized least-squares problem by truncating the singular value decomposition of the projected features. Their improvement over RanPAC seems to stem from the use of very high-dimensional projections. We note that this technique may also be used in our system to solve the extreme learning machine objective using 100,000 projection dimensions instead of our default of 5,000. Our contribution is entirely different: we propose an analytical contrastive layer that pulls samples of a class toward their mean and pushes class means away from one another, which improves class separation and consequently classification accuracy.

• Question 1:

As for Eq. 11, we clarify that $\Sigma^{-1/2}$ represents the whitening transformation, i.e., the inverse square root of the covariance matrix. For this to exist, $\Sigma$ must be positive-definite, which requires at least $n \geq d$ linearly independent data points. If this condition is not met (e.g., due to a small data size), we add a small regularization term $\epsilon I$ to the covariance to ensure it becomes invertible. In our implementation, we used $10^{-4} I$.

• Question 1:
  Intuitively, if class means are too close (i.e., having high cosine similarity), they become easily confusable, making the classification problem harder. Lemma 4.1 enables a transformation where class means are repelled from one another. We will consider including a more detailed geometric visualization in the revision. For now, we refer the reviewer to the t-SNE visualization in Figure 1 (right). The black dots represent class means. As shown, the class means of the "extracted features" are less separated, whereas after applying ACP, the data distributions and the class means in the "enhanced features" are more distinct, which helps classification.

• Question 2:
  The assumption behind our pseudo-replay mechanism is that each class distribution is characterized by its mean and a covariance matrix, an approach also used in prior work such as SLCA. However, maintaining a separate covariance matrix for each class increases the number of parameters. We instead use a shared covariance matrix across all classes, which drastically reduces the parameter cost while preserving the performance. This essentially implies that feature correlations are consistent across classes, i.e., if two feature dimensions are positively correlated for one class, they are likely to be so for others as well. In other words, the correlation belongs more to the feature extractor than to specific classes.

  We found that this works well across all datasets we tested (see Tables 1 and 2), and did not observe any failure cases. This shared covariance assumption is also used in the classic Mahalanobis distance and linear discriminant analysis (LDA) methods. It has been shown to perform well in various settings.

• Question 3:
  Our approach does not rely on any task information for classification or pseudo-replay data generation. In CIL, tasks are often used to organize the sequence of class arrivals. We follow this in our experiments, but our method treats each class independently; we can learn a single class at a time. To generate a pseudo-sample for a class, we sample a noise vector from the shared covariance and add it to the corresponding class mean, which is then labeled with that class.

• Question 4:
  We appreciate the suggestion to enhance Figure 1. In the revised version, we will update the figure to better illustrate the construction and application of each component.