Firstly, we extend our gratitude for the time and attention devoted to reviewing our paper. Below, we have carefully addressed each of your concerns to the best of our knowledge to improve the overall contribution of the paper.

The necessity of utilizing frequency space for continuous learning for edge devices.

Thank you for this great question. Edge devices are constrained by limited memory and computing resources, underscoring the significance of training efficiency and peak memory usage in continual learning. Presently, continual learning primarily operates in the spatial domain, where the filters in a CNN are generally smooth due to the local pixel smoothness in natural images, resulting in spatial redundancies that impact the efficiency and memory usage of continual learning. Conversely, in the frequency domain, it is possible to train on images using distinct frequency domain features, significantly reducing redundancy in the learning process and improving the performance of continual learning methods.

The symbols in Figure 2 is unclear.

Sorry for the unclear expression. $\otimes$ represent this feature being masked. In the final version of the paper, we will improve the clarity of the figure.

The design of the CFFS needs to be more convincing.

In a previous study [1], it was found that continual learning has task-sensitive parameters for different tasks, and reducing the overlap of these parameters can significantly improve the performance of continual learning. Similarly, in the frequency domain, we observed that it also has task-sensitive parameters. As shown in Table 3, when we pruned 75% of the weights, the performance of the CLFD method improved significantly. Considering the presence of task-sensitive parameters in the classifier, which causes different tasks to be sensitive to specific frequency domain features, we propose CFFS to balance the reusability and interference of frequency domain features.

[1] Sarfraz, Fahad, Elahe Arani, and Bahram Zonooz. "Sparse coding in a dual memory system for lifelong learning." AAAI. 2023.

Grammatical errors need to be corrected.

We appreciate the reviewer's feedback regarding the grammatical errors. We will update the paper according to your suggestions in the final revision.

The method's design throughout is very ad hoc and heuristic.

We agree with the reviewer that the method's design is very heuristic. However, the design of our method is well-founded. Heterogeneous Dropout has already been demonstrated to be effective in continual learning within the spatial domain [1,2]. Building on this foundation, we propose Frequency Dropout (Eq. 4). Additionally, in this work, we emphasize that our primary focus is on enhancing the efficiency of continual learning training by leveraging the frequency domain, thereby promoting its application on edge devices. Our research is among the first in this area, and we hope it will inspire the continual learning research community to further explore the frequency domain.

[1] Sarfraz, Fahad, Elahe Arani, and Bahram Zonooz. "Sparse coding in a dual memory system for lifelong learning." AAAI. 2023.

[2] Vijayan, Preetha et al. “TriRE: A Multi-Mechanism Learning Paradigm for Continual Knowledge Retention and Promotion.” NeurIPS. 2023.

Other evaluation measures.

In Appendix Table A2, we provide the forgetting measure, while Figure 7 presents the task-specific ACCs, and Figure 8 illustrates the stability and plasticity of our framework. We will incorporate the key experimental results into the main text.

There are no experiments for larger Buffer numbers.

We agree with the reviewer that conducting experiments on larger buffer sizes is essential. We conduct re-experiments under the buffer settings of SparCL-ER [1] and provide a detailed discussion in the general response.

[1] Wang, Zifeng, et al. "Sparcl: Sparse continual learning on the edge." NeurIPS. 2022.

The experiments do not evaluate larger datasets.

We agree with the reviewer's suggestion that incorporating larger datasets can enhance the credibility of the results. Therefore, we introduced the Split Imagenet-R dataset as an alternative and provided a detailed discussion in the general response.

The proposed method will likely perform poorly when the number of tasks increases.

We sincerely thank the reviewers for raising this concern. On the Split Tiny ImageNet and Split ImageNet-R datasets, each comprising 10 tasks, our framework consistently demonstrates strong performance, further validating the effectiveness of our framework. We emphasize that although our accuracy remains high even with a large number of classes, the principal contribution of our work lies in introducing the concept of continual learning in the frequency domain, which significantly improves training efficiency and facilitates the application of continual learning on edge devices.

We once again thank the reviewer for providing detailed feedback. We have made an utmost effort to resolve all the concerns raised. Please let us know in case we have missed something.

We extend our sincere gratitude for your thorough review of our paper. Below, we have diligently addressed each of your concerns to the best of our understanding, aiming to enhance the paper's overall contribution.

It is better to include larger databases.

We agree with the reviewer that introducing larger databases can make the results more convincing. Therefore, we introduced the Split Imagenet-R dataset as an alternative and provided a detailed discussion in the general response.

Table-1 leaves some unclear items.

We appreciate the reviewer's feedback regarding the shortcomings in Table 1. The highest results are marked in bold, and shadowed lines indicate the results from our framework. We introduced the definitions of Class-IL and Task-IL on line 147. In the simplest testing configuration, we assume that the identity of each upcoming test instance is known, a scenario defined as Task Incremental Learning (Task-IL). If the class subset of each sample remains unidentified during CL inference, the situation escalates to a more complex Class Incremental Learning (Class-IL) setting. We will update Figure 1 in the final version and provide definitions for Class-IL and Task-IL in the experiment.

The classification performance of the proposed method seems have a large gap with the state-of-the-art methods.

The reason for the low classification performance is that we only use the polar buffer size. When we adjust the buffer size to be consistent with state-of-the-art methods, the classification performance significantly improves, and our method continues to enhance the performance of various rehearsal-based methods. We provide a detailed analysis in the general response.

The time-consuming of the processes of wavelet transform and CLFD module.

We appreciate the reviewers for identifying the lack of training time analysis. We calculate the training time for the wavelet transform, CLFD module, ER method and CLFD-ER method throughout the entire Split CIFAR-10 training process. The wavelet transform accounts for only 1.5% of the ER training time, while the CLFD module accounts for only 13.1% of the ER training time. It is worth noting that when we integrated the CLFD module with the ER method, the training time was reduced by 2.3$\times$. This demonstrates the efficiency of our framework.

Some discuss about the different frequency domains.

We discussed different frequency domains in sections 2.2 and 3.2. The discrete cosine transform and discrete Fourier transform both cause a complete loss of spatial information in images, making it difficult to utilize data augmentation techniques. Therefore, they are challenging to apply in continual learning.

Once again, we express our gratitude for your thoughtful evaluation and consideration. We have conscientiously endeavored to address each of the concerns you raised, and we are dedicated to ensuring that our paper makes a meaningful contribution to the conference proceedings.

I thank the authors for answering my comments.

I believe that the authors' responses can solve most of my concerns of this paper, and I would like to re-rate this paper to weak-accept.

We express our gratitude to the reviewer for dedicating time to thoroughly review our work. We value the positive feedback provided on our manuscript. Below, we address the weaknesses and queries raised:

More ablation studies about hyperparameters.

We appreciate the feedback from the reviewers and acknowledge the importance of analyzing the robustness of hyperparameter selection. However, our framework can integrate with various continual learning methods, ablation studies on hyperparameters require substantial computational resources and extensive experimentation. Given this, we have demonstrated the effectiveness of our framework by using default hyperparameters for all datasets, methods, and buffer sizes.

The ways to replay data.

We use the widely accepted continual learning repository Mammoth CL [1] for data replay. We adopt a reservoir sampling strategy [2] to save samples, storing all data in tensor format. During each batch’s training, we read an additional batch of samples from the memory buffer and train them together with the current batch.

[1] Buzzega, Pietro, et al. "Dark experience for general continual learning: a strong, simple baseline." NeurIPS. 2020.

[2] Vitter, Jeffrey S. "Random sampling with a reservoir." ACM Transactions on Mathematical Software (TOMS) 11.1 (1985): 37-57.

More details about CFFS.

We apologize for any confusion. In the final version of the paper, we will optimize the details of CFFS. For a given task, we select only 60% of frequency domain features for classification. Each frequency domain feature is monitored for its selection frequency during training. Essentially, for each class, each frequency domain feature is assigned a selection counter $\mathcal{F}$ that increments when the feature's value is among the top 60%. Before selecting frequency domain features, each feature is assigned a dropout probability. During the initial epoch of each task training, we apply frequency dropout. In the later epochs of training, we use semantic dropout and update its probability after each epoch (Eq. 5). After completing the first task training, for subsequent tasks, we calculate the similarity of the classes (Eq. 2) and update the frequency dropout probability (Eq. 4) at the end of the first epoch. The pseudocode is provided in the attached PDF.

We hope that the clarification provided has resolved your concerns and inquiries. Should you require further assistance or elaboration, we are willing to provide additional information to ensure a comprehensive understanding of our work.

We conduct supplementary ablation experiments to examine the influence of different feature selection proportions on model performance. Our analysis concentrated on two methods: CLFD-ER and CLFD-ER-ACE. Figure 8 in the appendix illustrates that ER exhibits the highest degree of plasticity, whereas ER-ACE demonstrates the greatest stability. We conduct tests on the S-CIFAR-10 dataset with a buffer size of 50. The accuracy results under the Task-IL setting are as follows:

And the accuracy results under the Class-IL setting are as follows:

We also conduct an ablation study on the other hyperparameters presented in Section E.2. Since our framework can be integrated with various continual learning methods, we do not use grid search; instead, we individually investigate the impact of each hyperparameter on continual learning performance. We focus on the CLFD-ER-ACE method for this study, as it performs well across all datasets and does not introduce any additional hyperparameters. We conduct tests on the S-CIFAR-10 dataset with a buffer size of 50. The accuracy results under the Class-IL setting are as follows:

Our choice of hyperparameters in the paper yields optimal performance, and our framework demonstrates robustness to these hyperparameter selections. We hope that these ablation studies will provide the reviewer a more thorough comprehension of our work. We extend our appreciation for the feedback provided. Should there be any aspects requiring further elucidation, we are prepared to offer further explanations.

We express our sincere gratitude to the reviewer for offering thoughtful feedback and providing a constructive evaluation of our work. The valuable input has significantly contributed to the improvement of our paper.

The paper does not thoroughly discuss its performance across a broader range of scenarios or more complex datasets.

We agree with the reviewer that conducting experiments on a broader dataset is essential. Accordingly, we have supplemented our experiments with the Split Imagenet-R dataset and provided a detailed discussion in the general response.

The reliance on buffer size might limit its utility in extremely constrained environments where memory is severely limited.

While the dependence on buffers does consume some storage resources, our method can achieve excellent performance even with smaller buffer sizes by preserving the frequency domain features of the image instead of the original image, as shown in Table 1. In contrast, many continual learning methods [1,2] that do not rely on buffers need extra parameters, such as teacher models, to prevent model forgetting, which use significantly more storage than rehearsal-based methods.

More importantly, for continual learning on resource-constrained devices, the limitation of memory resources is much more critical than that of storage resources. This necessitates continual learning methods to have lower peak memory usage. As shown in Table 1, our method can achieve up to a 3.0 $\times$ reduction in peak memory usage. This effectively promotes the application of continual learning methods on resource-constrained devices.

[1] Smith, James, et al. "Always be dreaming: A new approach for data-free class-incremental learning." ICCV. 2021.

[2] Li, Zhizhong, and Derek Hoiem. "Learning without forgetting." IEEE transactions on pattern analysis and machine intelligence 40.12 (2017): 2935-2947.

Freezing the FFE based on the first task might introduce scalability issues.

We agree with the reviewer that freezing the FFE does affect the plasticity of the model. However, not freezing the FFE leads to severe forgetting problems. To balance the plasticity and stability of the model, it is crucial to ensure that no cross-task learnable parameters are introduced. As shown in Figure 8 in the appendix, our method consistently improves the balance between plasticity and stability in continual learning methods.

The framework underperforms in the Task-IL setting.

We appreciate the reviewers for identifying the lack of the Task-IL accuracy analysis. This result can be explained from two aspects: (1) FFE encodes the frequency domain features of the input image, reducing the size of the input feature map to one-fourth of its original size. This leads to fewer learnable features and increases the difficulty of classifying classes within the task. (2) To minimize the overlap of frequency domain features between inputs with different semantics across tasks, CFFS selects only a subset of frequency domain features for classification. This selection also increases the difficulty of classifying classes within tasks. However, as the task difficulty increases, the performance of our framework becomes more pronounced in the Task-IL setting. For instance, as shown in Table 1 in the attached PDF, our framework significantly enhances Task-IL accuracy on the Split Tiny ImageNet dataset with buffer sizes of 200 and 500.

Considering that Task-IL is generally regarded as an easier CL scenario compared to Class-IL, this research, consistent with previous work [1], primarily focuses on the more intricate Class-IL setting (as mentioned in line 150). Furthermore, we emphasize that the novelty of this work lies not only in improving accuracy but also in significantly enhancing the training efficiency of continual learning. Additionally, we have validated our framework on edge devices, thereby promoting the application of continual learning in such environments.

[1] Wang, Zifeng, et al. "Sparcl: Sparse continual learning on the edge." NeurIPS. 2022.

The technical and typographical issues.

Thanks for pointing out these technical and typographical issues, we will revise them in the next release.

We once again thank the reviewer for providing detailed and insightful feedback. Please let us know if there are any open points that we may have overlooked.

We express our sincere appreciation for the thoughtful feedback offered by the reviewer and the constructive evaluation of our work. We have addressed all the technical details and typographical issues as per your suggestions. Additionally, following your recommendation, we conduct a detailed comparison and ablation study on the other hyperparameters presented in Section E.2. Since our framework can be integrated with various continual learning methods, we do not use grid search; instead, we individually investigate the impact of each hyperparameter on continual learning performance. We focus on the CLFD-ER-ACE method for this study, as it performs well across all datasets and does not introduce any additional hyperparameters. We conduct tests on the S-CIFAR-10 dataset with a buffer size of 50. The accuracy results under the Class-IL setting are as follows:

Our choice of hyperparameters in the paper yields optimal performance, and our framework demonstrates robustness to these hyperparameter selections. We hope that these ablation studies will provide the reviewer with a more comprehensive understanding of our work. We once again extend our gratitude for your thoughtful and highly constructive feedback.

We have completed the experiments with other methods on the Split ImageNet-R dataset under the 20-task setting. The results are as follows:

These results provide a more comprehensive demonstration of the scalability of our framework. We have included a comprehensive analysis of the framework's scalability in the final version of the paper.