Parameter-Efficient Fine-Tuning via Circular Convolution

Thanks for your constructive comments! We try our best to address your concerns about the novelty and empirical evaluations.

Weakness 1: doubts on plagiarism and novelty.

We appreciate our esteemed reviewer for pointing out this concern, and we have the following responses.

• We would like to express our sincere acknowledgment of the pioneering work by Ding et al. (2017), who was the first to apply block circulant matrices in the deep learning community. We apologize for the unintentional omission of this important citation in our methodology section and will incorporate proper acknowledgment in the revised camera-ready version.

• We wish to clarify that we do not claim to have developed new forward and backward algorithms for block circulant matrix-based learning. Our contribution lies in applying this existing approach within the context of PEFT. We appreciate the feedback regarding the perceived limitations of our paper's novelty and would like to offer further clarification. The fine-tuning scenario—where data scarcity necessitates regularization techniques—is fundamentally different from the standard training scenario. As discussed in lines 74–79, the evolution from CNNs to Transformers underscores the benefits of removing intrinsic biases (i.e., special patterns or algorithmic restrictions) when abundant data is available. This shift explains why model compression techniques, such as low-rank decomposition and circulant-matrix-based learning, have become less prominent in the era of large language models (LLMs), which are pretrained with a considerable amount of data. However, as we posit in lines 133–139, these methods remain invaluable within the PEFT community, where limited data is still a stumbling block and intrinsic biases are highly beneficial.

• In fact, the PEFT community often draws inspiration from traditional deep learning research, especially from the model compression field. For example, the influential LoRA method focuses on low-rank decomposition, a topic that has been widely studied (e.g., [1,2]). Additionally, the Fourier Transformation-based method our reviewer mentioned has also been explored in [3]. While these traditional approaches have not fully thrived in the era of modern LLMs, their foundational contributions continue to offer valuable insights and benefits for advancing areas like PEFT.

[1] Zhang, Xiangyu, et al. "Accelerating very deep convolutional networks for classification and detection." TPAMI, 2015.

[2] Denton, Emily L., et al. "Exploiting linear structure within convolutional networks for efficient evaluation." In NeurIPS, 2014.

[3] Koutnik, Jan, et al. "Evolving neural networks in compressed weight space." In GECCO, 2010.

Question 1: Comparison against FouirerFT.

Thank you for your suggestion. We have conducted comparisons between C3A and FourierFT on both RoBERTa-Base and ViT-Base models. To ensure fair comparisons (i.e., a similar parameter size), we set $b = 768$ and $n = 1,000$ for RoBERTa-Base. For ViT-Base, we chose $b = 768/12 = 64$ and $n = 10,000$. The resulted accuracies are reported as follows:

The parameter size is listed below:

From the result, we can observe that C3A generally outperforms FourierFT when constrained with similar parameter size.

Weakness 2 & Question 2, 3: Experiments and explanation on memory cost.

We thank the reviewer for raising this valuable concern. We would like to correct our space complexity analysis as follows. On the GPU, the cuFFT backend automatically parallelizes the FFT operation along non-operated axes, with the parallelization size depending on the available resources. Therefore, the auxiliary tensor size should be $nb$ instead of $b$, where $n$ is the parallelization size. Consequently, the theoretical memory lower bound becomes $2\sqrt{n d_1 d_2}$ when $b^* = \sqrt{\frac{d_1 d_2}{n}}$. Given that $d_1 = d_2 = 768$, if $n > 1$, a larger $b$ could naturally result in lower memory consumption than when $b = d_1 = d_2 = 768$.

As suggested, we have conducted an additional comparison of the actual memory cost between C3A and LoRA for Table 3 and Table 4. To ensure fairness, we follow the same experimental settings and set the sequence length to 256 for LLaMA families. The results are reported as follows:

We can observe that C3A continues to demonstrate lower memory consumption, consistent with the results observed in the GLUE benchmark.

Weakness 3 & Question 4: Experiments on initialization.

Thanks for your suggestion. We have incorporated an empirical comparison of different initialization strategies using LLaMA3-8B on GSM8K, SQL, and PIQA, which are the representative datasets of three instruction-tuning tasks. We have carefully tuned the hyperparameters to the best of our ability, and the results are presented below:

The observed variations among different initialization strategies are relatively insignificant, which is consistent with similar trends noted in the GLUE benchmark and image classification.

Question 5: Training details about precision.

Thank you for bringing up this important issue. All our experiments were conducted using half-precision (BF16). Specifically, both the activations $\mathbf{x}$ and the pretrained weights $\mathbf{W}$ were initialized in BF16. However, due to the current lack of BF16 support in cuFFT, we temporarily converted $\mathbf{x}$ to FP32 for the circular convolution operation, then converted it back to BF16 afterward, maintaining the foundation model in BF16 throughout.

Thank you for the feedback.

We have updated the manuscript to address your concerns regarding the plagiarism issue. The revised parts are highlighted in magenta for your convenience. We would greatly appreciate any further feedback you may have.

Thank you for the provided comments and helpful suggestions! We have incorporated additional experiments and discussions, and we hope they adequately address your concerns.

Weakness 1, 3 & Question 1: Ideal scenarios for C3A.

We thank the reviewer for highlighting this important issue. In practical applications of PEFT methods, there is an increasing demand for deployment in ever-changing scenarios and personalization. However, applying LoRA across multiple instantiations [1] presents challenges due to its large parameter size, particularly in terms of storage and transmission, especially for low-resource hardware and consumer-grade networks [2]. In contrast, we'd like to highlight that C3A achieves competitive performance while requiring not only fewer parameters but also reduced actual GPU memory. Therefore, we believe C3A offers a strong alternative for real-world deployment on resource-constrained devices.

[1] Sheng, Ying, et al. "S-LoRA: Serving Thousands of Concurrent LoRA Adapters." In MLSys, 2024.

[2] Borzunov, Alexander, et al. "Distributed Inference and Fine-tuning of Large Language Models Over The Internet." In NeurIPS, 2023.

Question 2: Ablation study on block size.

Thank you for your valuable suggestion. We perform extensive evaluations of C3A with varying block sizes across several representative datasets. Specifically, tasks on CoLA and STS-B are evaluated using RoBERTa-Base, while Cars and DTD utilize ViT-Base. For larger-scale models, we conduct evaluations on GSM8K, SQL, and PIQA using LLaMA3-8B. The results are presented below:

As observed, different tasks respond differently to changes in block size. Simple tasks like STS-B and PIQA perform well even with a very large block size, which corresponds to a lower parameter count. In contrast, more complex tasks such as Cars and GSM8K require a smaller block size to achieve optimal performance. Therefore, much like selecting the appropriate rank $r$ when using LoRA, we recommend searching for the optimal block size $b$ when approaching a new task.

Weakness 2 & Question 3: Practical support of FFT.

Thank you for the instructive comments. We acknowledge the fact that FFT is a rather complex method to implement. However, we have observed that leading GPU manufacturers, including NVIDIA [1], AMD [2], and Intel [3], have provided easy-to-use implementations of FFT, which might alleviate the concern about the practicality.

[1] https://docs.nvidia.com/cuda/cufft/index.html

[2] https://rocm.docs.amd.com/projects/rocFFT/en/latest/

[3] https://docs.openvino.ai/2022.3/openvino_docs_MO_DG_prepare_model_Supported_Frameworks_Layers.html

Dear reviewer VhRA,

Thank you once again for taking the time and effort to provide valuable feedback on our paper. We have carefully addressed your concerns by adding discussions and additional experiments. We would greatly appreciate the opportunity to discuss whether there are any remaining issues or concerns you would like us to address.

Thank you, Authors

We really appreciate the reviewer for listing the detailed strengths, which have clearly recognized the novelty and contributions of our paper. As requested by the reviewer, extensive additional experiments are conducted to further support our contributions.

Question 1: Convergence analysis of different initializations.

Thanks for raising this question. As suggested by the reviewer, we have collected data on the convergence time, which is represented by the percentage of training progress, for 4 initialization methods. The results are presented as follows:

The results observed indicate that different initialization strategies display similar convergence speeds, further supporting C3A's resilience to variations in initialization.

Question 2: Additional evaluations on larger-scale datasets or more complex tasks. (More LLM Datasets.)

Thanks for the suggestions. For comprehensiveness, we further assess the adaptability of C3A on a more complex and larger scale Commonsense170K dataset [1], a widely used commonsense reasoning benchmark that is framed as multiple-choice questions. Specifically, it integrates the training sets of eight distinct datasets, including BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-e, ARC-c and OBQA. The evaluations are then conducted on the test sets of the individual datasets.

We have conducted a comparison of C3A against LoRA and DoRA using the LLaMA3-8B model. For consistency, we set $b = 128$ for C3A and $r = 32$ for the rest, ensuring an identical parameter count as outlined in Table 3. We try our best to tune the models and the results are presented as follows:

As observed, C3A continues to demonstrate competitive performance while consuming only half the parameter size, highlighting the effectiveness of circular convolution in managing generalized and complex tasks.

[1] Hu, Zhiqiang, et al. "LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models." In EMNLP, 2023.

Question 3: Ablation study on block size.

Thank you for your valuable suggestion. We performed extensive evaluations of C3A with varying block sizes across several representative datasets. Specifically, tasks on CoLA and STS-B were evaluated using RoBERTa-Base, while Cars and DTD utilized ViT-Base. For larger-scale models, we conducted evaluations on GSM8K, SQL, and PIQA using LLaMA3-8B. The results are presented below:

As observed, different tasks respond differently to changes in block size. Simple tasks like STS-B and PIQA perform well even with a very large block size, which corresponds to a lower parameter count. In contrast, more complex tasks such as Cars and GSM8K require a smaller block size to achieve optimal performance. Therefore, much like selecting the appropriate rank $r$ when using LoRA, we recommend searching for the optimal block size $b$ when approaching a new task.

Question 4: Additional baselines.

Thank you for your suggestion. We have conducted comparisons between C3A and FourierFT [1] on both RoBERTa-Base and ViT-Base models. To ensure fair comparisons (i.e., a similar parameter size), we set $b = 768$ and $n = 1,000$ for RoBERTa-Base. For ViT-Base, we chose $b = 768/12 = 64$ and $n = 10,000$. The resulted accuracies are reported as follows:

The parameter size is listed below:

From the result, we observe that C3A generally outperforms FourierFT when constrained with similar parameter size.

[1] Gao, Ziqi, et al. "Parameter-Efficient Fine-Tuning with Discrete Fourier Transform." In ICML, 2024.

Question 5: Discussion on limitations.

We thank the reviewer for this valuable question. We acknowledge that a limitation of C3A is the absence of an easy-to-use serving framework, such as S-LoRA [1]. However, given its effectiveness and efficiency, we believe that developing a serving framework for C3A is a promising and worthwhile direction for future exploration.

[1] Sheng, Ying, et al. "S-LoRA: Serving Thousands of Concurrent LoRA Adapters." In MLsys, 2024.

Dear reviewer F24A,

Thank you once again for taking the time and effort to provide valuable feedback on our paper. We have carefully addressed your concerns by adding additional experiments and revisions. We would greatly appreciate the opportunity to discuss whether there are any remaining issues or concerns you would like us to address.

Thank you, Authors

Dear Reviewer F24A,

Thank you for your time and the valuable insights you’ve provided on our work. As we near the conclusion of the author-reviewer discussion period, we truly appreciate the opportunity to address your concerns.

We kindly request you to review our responses to your comments to ensure they adequately address the points you raised. If you have any additional questions or need further clarification, please don’t hesitate to reach out—we’d be happy to provide more details.

Best,

Authors

Dear reviewer F24A,

Thank you for your valuable contributions to the review process. As the deadline for the author-reviewer discussion period approaches, we sincerely appreciate the opportunity to address your concerns.

We understand that you have a multitude of responsibilities. To facilitate a swift evaluation of our responses, we have summarized the corresponding changes as follows:

• Convergence Analysis: We added the analysis of convergence across different initialization strategies (Q1).
• Empirical Evaluations: We expanded the evaluations to include larger-scale and more complex tasks (Q2) and comparison against more recent baseline (Q4).
• Ablation Study: We added an ablation study on the block size (Q3).

We kindly request your feedback on whether our responses adequately address your concerns. If you have any further questions or additional points for discussion, please let us know—we would be happy to provide further clarification or additional details.

Thank you for your time and consideration!

Best,

Authors

Dear Reviewer,

Could you kindly respond and indicate whether authors have addressed your concerns?

Thanks, AC