Tensor Decomposition Based Memory-Efficient Incremental Learning

We sincerely appreciate the reviewer's insightful feedback and constructive suggestions. Below, we address the key concerns raised.

1. Why diversity can be ensured by the second-stage sampling and its contribution to performance

The paper proposes a novel exemplar selection strategy that enhances representativeness and diversity. Specifically, In the first stage, we use herding to select a small, equal number of representative original samples per class, prioritizing high-quality exemplars. In the second stage, we choose similarly an equal number of sample factors per class, focusing on sample quantity. The diversity of the selected exemplars is primarily ensured through the more stored samples and the uniform sampling strategy.

About the diversity's contribution to performance, as shown in Fig. 3, when $\epsilon$ remains within a small range, meaning a sufficient number of samples are retained in the second stage, this strategy consistently yields great performance improvements. However, as $\epsilon$ increases further(means fewer samples are stored and lower diversity ), the model’s final performance gradually gets close to the original. For MEMO, when $\epsilon$ exceeds 0.6, the performance gain on CIFAR remains below 1%.

2. Can you theoretically or experimentally demonstrate that actively selecting samples with low reconstruction error after tensor decomposition contributes to performance improvements?

Yes, we can. Here, we provide results (Average Accuracy) for the sample selection strategy prioritized by the minimum reconstruction error (M= 2k, 10-task); note that we only use reconstructed samples. It can be seen that AA gradually improves as the reconstruction error decreases.

3.Clarification of notation in the exemplar selection strategy

We will revise the description of the Exemplar Selection Strategy to distinguish samples selected in each stage.

We sincerely appreciate the reviewer's thoughtful feedback and constructive suggestions. Below, we address the concerns raised.

1. Lacks ablation experiments for the proposed method

In our manuscript, we have conducted experiments (Section 4.2, Fig. 3, and Tab. 6) to evaluate the impact of key components, e.g., decomposition rank $R$, proportion $\epsilon$, as shown in Fig.3 when $\epsilon = 0$, we only use reconstructed samples without the two-stage selection strategy. Tab. 6 demonstrated the effect of different compression methods on performance.

2.Specific experimental setups in Tab. 1 and Tab. 3

We acknowledge this oversight. For the experimental setups in Tab. 1 and Tab. 3, we set the memory budget to 2k and explained other settings in the section "Protocols."

3. Performance discrepancy on iCaRl and DER

For the performance discrepancy in DER, since they chose resnet18 as the backbone network for CIFAR(for resnet32, their results are close to ours), we used more lightweight resnet32 (that's what most do.). Thus, there is a performance gap; for ImageNet, we use the same backbone network with similar performance. For iCaRL, we have double-checked it without any changes.

4. Lacks comparison results with memory-efficient methods

In our manuscript, we have provided comparison results with some recent memory-efficient replay methods in Tab. 5 and Tab. 9. All the results demonstrate our method's superiority.

5.Exlaination of Herding Strategy

For exemplar selection, we clarify that herding is employed independently within each class, ensuring that selected samples best represent the class distribution. While herding traditionally minimizes the difference between selected samples and the overall dataset, its application per class in our method aligns with class-incremental learning settings. We will further clarify this in the manuscript.

We deeply appreciate your rigorous review and constructive feedback. All suggested revisions will be incorporated to strengthen the manuscript’s clarity, technical depth, and experimental validation. Thank you again for your time and consideration.

We sincerely appreciate the reviewer’s thoughtful evaluation and constructive feedback. Below, we address each concern raised and outline revisions to strengthen the manuscript.

1. Additional experiments are needed to validate the importance of incorporating reconstructed samples into training

In experiments, we observed that pure replay-based methods, such as iCaRL and FOSTER, achieved strong performance even without incorporating reconstructed samples during training. This is primarily because they operate on a fixed network structure and employ strategies like knowledge distillation to mitigate forgetting.

However, methods like DER and MEMO adopt a dynamical model structure. Their primary mechanism for combating forgetting is freezing previously trained sub-networks, although they do retain some old samples for replay. Nevertheless, their reliance on sub-network freezing is more significant.

To illustrate, when training on the first task, the corresponding sub-network lacks the ability to recognize reconstructed samples unless they are included in the training. When the second task arrives, although some reconstructed samples from previous tasks are preserved and added to training, relying solely on this limited number of old samples does not ensure enough generalization, and there may be some contradiction between the outputs of these two networks. Therefore, for methods like this, not incorporating reconstructed samples during training leads to a drastic decrease in performance.

Here, we report model performance under both scenarios in the table below(10-task,$M = 2k$, $R = 12$, $\epsilon = 0.4$). For consistency, we choose to include reconstructed samples in training across all cases.

2. Setting an appropriate reconstruction error threshold and a decomposed tensor rank can be challenging

We have provided the ablation experiments of Rank $R$ in Fig. 3 in the manuscript, and the results show that our method always produces positive results when $R \in [\frac{H}{3}, \frac{H}{2}]$.

As for the reconstruction error threshold $\tau$, which is primarily introduced to filter out samples with failed decompositions (rarely occur), which typically exhibit reconstruction errors exceeding 0.1. Indeed, for RGB images, our experimental results show that the reconstruction error is minor when $R \in [\frac{H}{3}, \frac{H}{2}]$, essentially no noisier samples are introduced. For instance, on CIFAR-100, when using a CP rank $R = 12$, the average relative squared error (RSE) is already below 0.05. Increasing $R$ to 16 further reduces the average RSE to 0.034. Under a memory budget of $M = 2k$ and $\epsilon = 0.1$, each class retains 2 raw images and approximately 50 sets of decomposition factors(only 10% of the total data). For a threshold $\tau$ of around 0.07, its effect on the final result is negligible.

3.Experimental results on ImageNet-R

Here we provide experimental results on ImageNet-R under 5, 10, and 20-task settings; we set $M = 2k$, $\epsilon = 0.4$, $R = 80$. It can be seen that our approach also responds effectively in this scenario.

4. Combining with recent replay-based methods

Here, we provide experimental results (10-task) on CIFAR-100 with our method integrating into MRFA[1], we set $M = 2k$,$\epsilon = 0.1$, $R = 16$, we report Average Accuracy (AA) and Last Accuracy (Last). It can be seen that our method also provides positive gains.

5. Lacks visualization of reconstructed images

We will provide figures showcasing the original image and its corresponding reconstructed image at different ranks in the revised version.

[1] Multi-layer Rehearsal Feature Augmentation for Class-Incremental Learning, ICML 2024

We appreciate the reviewer's constructive feedback and valuable suggestions. Below, we address the concerns raised.

1. Failing to explicitly report the actual memory costs

In our manuscript, we have provided parameter configurations of different datasets in Tab. 7, e.g., for CIFAR, $R = 12$, $\epsilon = 0.2$, when $M = 2k$, according to ep.3 compression rate $\eta \approx 0.26$, which means we saved 400 original samples and about 6153 sets of sample factors. We will elaborate on this in the next release.

2. Lacks comparative experiments under fixed memory of varying sizes

We have indeed provided comparative experiments under fixed memory of varying sizes (see Tab. 2 and Tab. 4 for two different fixed memories) in our manuscript. We can add more explanations for clarity.

3. Computational latency of TD for image compression/decompression

On a single GPU (NVIDIA 3090), decomposing a CIFAR-100 image takes approximately 17 ms, while reconstruction takes approximately 2 ms, since the decomposition can be done in parallel with the training process, this part of the delay is negligible. The main computational delay is caused by the inclusion of reconstructed samples in the training, which requires about 40% extra computation for DER and MEMO. Furthermore, as we pointed out in our response to reviewer qf47, the extra calculations for incorporating reconstructed samples during training are not necessary for iCaRL and FOSTER.

4. Lacks experiments on ImageNet-1k

While ImageNet-1k is a valuable benchmark, our current experiments on high-resolution datasets (e.g., ImageNet-100) and complex scenarios (e.g., 200-class Tiny-ImageNet in Table 8) already demonstrate the scalability and robustness of our method. Specifically:

1. ImageNet-100: As a standard high-resolution benchmark in CIL literature, our method achieves consistent improvements (e.g., +6.42% AA for DER in Table 3), validating its effectiveness under realistic settings.
2. Tiny-ImageNet: With 200 classes and 64×64 resolution, this dataset mimics the complexity of large-scale tasks. Our method boosts DER’s accuracy by 9.53% (Table 8), illustrating strong generalization.
3. Community Practices: For ImageNet-1k, prior works (e.g., iCaRL, DER) typically adopt a memory budget of 20k and 10-/20-task splits, which aligns with our experimental protocols. While computational constraints limited direct validation on ImageNet-1k, the consistent gains across varying resolutions and class numbers (ImageNet-100 to Tiny-ImageNet) suggest scalability to larger datasets.

We acknowledge the value of ImageNet-1k experiments and will provide results as time permits. For now, the results on ImageNet-R (provided in response to qf47) further corroborate our method’s adaptability to domain-shifted and large-scale scenarios.

5.Other questions for authors

We acknowledge our oversight and clarify it here. CP is the abbreviation of "CANDECOMP/PARAFAC."[1]. We will ensure terms are properly introduced when first mentioned and provide visualizations of reconstructed samples in the revised manuscript.

[1]Tensor decompositions and applications.

We sincerely thank the reviewer for the thoughtful feedback and constructive suggestions. Below, we provide a point-by-point response to the comments raised.

1. Inconsistent terminology for evaluation metrics

In the revised manuscript, we will align our metric naming with the community standards."Average Incremental Accuracy (AIA)" will be replaced with "Average Accuracy ", and "Average Accuracy " will be renamed to "Last Accuracy''

2. More comprehensive comparisons and analysis for efficient replay methods

We thank the reviewer for highlighting the importance of comparative analysis. The initial manuscript has provided the comparison results in Tab. 5 and Tab. 9, and all the results demonstrate our method's superiority. Here, we offer more comparison results (10-tasks) and analysis.

As our introduction and related work noted, pixel-level compression methods (e.g., MRDC[1], CIM[2]) directly compress images in the high-dimensional pixel space, often neglecting the low intrinsic dimensionality and local correlations inherent to natural images. This oversight leads to a significant loss of discriminative information.

In contrast, our Tensor Decomposition (TD)-based method explicitly leverages these properties by factorizing images into low-rank components. This not only achieves lower storage complexity (e.g., a compression ratio of 0.34 vs. CIM’s 0.56 in Tab. 6) but also preserves more discriminative information through high-fidelity reconstruction. As shown in Tab. 6, training on TD-compressed data achieves 69.9% accuracy on CIFAR-100, significantly closer to the upper bound (72.3%) than pixel-level methods like downsampling (44.1%) or CIM (66.9%). This demonstrates TD’s ability to retain essential information while drastically reducing memory costs.

The superiority of our method stems from two key factors:

1. Efficient compression: TD captures multi-dimensional correlations (spatial, channel-wise) in images, preserving more discriminative information while keeping a great compression ratio, avoiding the "brittle" compression of pixel-level methods.
2. Adaptability: Unlike methods reliant on fixed heuristics (e.g., CIM’s center cropping), TD flexibly adapts to varying resolutions and dataset complexities, as evidenced by consistent gains across CIFAR-100, Tiny-ImageNet, and ImageNet-100 (Tab. 1, 3, 8).

These revisions will solidify our method’s advantages over existing efficient replay techniques, and we hope this can address the reviewer’s concern for deeper comparative insights.

3.Combing with online continual learning

This is an excellent suggestion. While our current focus is on Class-Incremental Learning, we recognize the importance of online settings. In the revised manuscript, we will add a discussion about the applicability of our method to online continual learning and outline plans for future work in this direction.

4. Effect of varying ranks and $\epsilon$ on reconstruction quality

Regarding the reconstruction quality, which depends on the CP rank $R$, we provide some results on CIFAR-100(evaluated by mean relative squared error). In the table, ''comp'' means compression ratio, and ''accuracy'' represents the classification accuracy after finishing offline training on compressed data. About the effect of $\epsilon$, in the initial manuscript, Fig. 3 has shown its influence on final performance.

5. Lacks visualization of reconstructed images

We will provide figures showcasing the original images and their corresponding reconstructed images at different ranks in the next version.

[1]Memory replay with data compression for continual learning [2]Class-incremental exemplar compression for class-incremental learning