LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuning

Thanks for the time and effort in providing detailed feedback. We have carefully considered all comments and provide comprehensive responses below.

Inadequate empirical support for theoretical claims (Weakness 1).

Thanks for the detailed comments. However, we would like to clarify several misunderstandings:

• Clarification of Central Claim: The interpretation of our central claim is not accurate. Our central claim is that optimal frequency-domain-based reconstruction can be achieved through individual selection of frequency coefficients and their optimal locations, rather than through random selection of frequency components. The comparison between random frequency selection and LoRA is a secondary finding in our theoretical framework.

• Experimental Validation for the Theoretical Finding: Please note that the results we obtained in Theorem 1 show that the Fourier method with randomly selected frequency component locations yields a larger expected reconstruction loss compared to the optimal approximation based on low-rank decomposition. FourierFT is a very valuable and meaningful work. Our comparison is conducted solely from the perspective of reconstruction, which has direct experimental validation. Specifically, Figures 6 and 7 in Appendix G provide rigorous experimental validation of our theoretical claims across different ranks and dimensionalities of weight matrixes, where 'R' represents the reconstruction capability of low-rank decomposition and 'U' represents that of random frequency selection in Fourier decomposition. These experiments directly validate our theoretical findings.

• Apparent Contradiction with FourierFT's Results: Regarding the apparent contradiction with FourierFT's empirical results in its Appendix C.2, it is important to note that our analysis focuses on matrix reconstruction capability, which, while important, may not directly translate to downstream task performance in all scenarios. As we explicitly discuss in Section 5.5 (Performance under Different Parameter Budgets), our theoretical analysis concerns expected performance rather than performance in every specific case. Task-specific structures may indeed allow FourierFT to outperform LoRA in certain instances without contradicting our theoretical framework.

• Selective FourierFT and LoCA Evaluation: Please refer to the response of Question 4.

Omitted comparison with VeRA (Weakness 2).

Thanks for the comment. In the revised manuscript, we have included comparisons with VeRA. Specifically, we have added VeRA as a baseline on the GLUE benchmark (Table 1) and included a new section (Section 5.2) on Natural Language Generation, where we evaluate our method against VeRA on the E2E NLG Challenge dataset (Table 2). Please refer to the revised manuscript for details.

Parameter overhead due to optimizable locations (Weakness 3 & Question 2).

Thanks for the concern about parameter overhead. However, we would like to clarify several important points regarding LoCA's actual computational and memory efficiency. First, the optimization of location parameters only occurs during the initial training phase (typically the first 10% of iterations), similar to the dynamic parameter allocation strategy successfully employed in AdaLoRA. After this initial phase, we do not regard locations as trainable parameters and no gradient computation is required.

Regarding the parameter count, while LoCA does require storing additional location parameters, the actual storage overhead is minimal. For instance, storing 150,000 integer location parameters only adds approximately 0.57MB to disk storage - a negligible increase compared to the base model's size. More importantly, the parameter count does not directly mean runtime memory usage or computational efficiency. To see this, we have conducted comprehensive empirical evaluations comparing LoCA, LoRA, and FourierFT across different datasets, model scales, and parameter budgets. The results have been added to the revised paper (Table 10) and corresponding analysis section (Appendix J). As demonstrated in Table 10, LoCA actually shows comparable or better memory efficiency compared to FourierFT across different model scales and datasets. This is partly because FourierFT requires complex-domain computations to obtain real-valued network parameters, leading to additional memory overhead. Therefore, while LoCA may have a higher parameter count, its practical scalability and efficiency in terms of actual memory usage and computation time remain competitive.

Thank you for your follow-up questions. We appreciate the opportunity to provide further clarification on these important points.

Stable Convergence with Dynamic Frequency Selection: Regarding the effectiveness of finite-difference approximation in capturing frequency dynamics: The central difference approximation we use essentially computes the discrete derivative of the loss with respect to location changes in the frequency domain. This approach is theoretically justified because it provides an unbiased estimate of the true gradient in the discrete frequency space. Most importantly, it captures not just the immediate effect of moving a frequency component, but also the interaction effects with neighboring components through the chain rule in backpropagation.

Regarding the learning rate ratio between location and coefficient updates: We determine this based on the theoretical properties of the frequency domain. Coefficient updates directly modify the magnitude of contribution from each frequency component. Location updates, however, have a more fundamental effect on the representation. Therefore, we set a small location learning rate to ensure stable adaptation of the frequency basis.

About the smoothness assumption: We believe there may be a misunderstanding here. The smoothness of frequency components is an inherent mathematical property of the DCT basis functions themselves, not a property that depends on how many or which components we select. Each DCT basis function is continuous and differentiable by definition, and selecting a subset of these functions does not change their smooth nature. This is analogous to how selecting certain terms from a Fourier series still results in a smooth function - the smoothness is intrinsic to the basis functions, not to how many we use. Furthermore, our empirical results in Figure 2 demonstrate stable convergence during training, confirming that our selection of frequency components maintains the desired smoothness properties in practice.

Risk of Losing Task-Relevant Information: The relationship between magnitude and importance in our method is more nuanced than simply selecting high-magnitude components: While our theoretical analysis uses magnitude as a proxy for component importance to establish upper bounds on expressivity for these methods, the practical implementation uses gradient-based optimization to determine importance. This is consistent with all other PEFT methods. For instance, LoRA aims to approximate the full fine-tuning matrix using low-rank decomposition, but it is also actually updated through backpropagation. In other words, our frequency-domain component selection is directly controlled by task-relevant gradient signals. The task-relevant information will be preserved. When task-specific optimization demands stronger expressivity, our method can perform better than LoRA and FourierFT, since LoCA has a stronger expressivity by strategically selecting frequency components.

Regarding the adaptation to different tasks, the gradient-based optimization automatically adapts to task-specific patterns by strengthening relevant components through training. The alternating optimization strategy allows for exploration of different frequency combinations early in training before settling on task-relevant components.

Thanks for the thorough review and insightful comments. Below we address each concern in detail.

Instruction tuning on MathInstruct (Question 1 & Weakness 1).

Thanks for the valuable suggestion regarding the use of MathInstruct Dataset. Following this recommendation, we have initiated experiments using the MAmmoTH codebase [R1] to evaluate our method on this more mathematically-focused benchmark. As the dataset is relatively large and we are simultaneously conducting other experiments requested by Reviewer kgMQ, 3NDS, and 96ov, we may not be able to provide complete results for both LoCA and all baselines before the deadline. However, we are committed to sharing preliminary findings as soon as they become available.

While we acknowledge the reviewer's concern about MT-Bench's stability, we would like to respectfully note that MT-Bench and Vicuna have been widely accepted as standard benchmarks for evaluating instruction tuning in the PEFT community. Recent influential works in this field, including DoRA, VeRA, and FourierFT, have all employed these benchmarks for the instruction tuning experiment. To address the stability concern of GPT-4 evaluation, we have taken careful measures in our experimental design: all results reported in Table 3 were obtained through fresh runs using the same version of GPT-4 within the same time period, ensuring a fair comparison across different methods. This controlled setting helps mitigate the potential instability issues in evaluation. Moreover, we have provided detailed example outputs in Appendix K to offer qualitative insights into the performance differences between methods.

Nevertheless, we will continue our ongoing experiments with MathInstruct and update our results accordingly.

[R1] Yue, et al., MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning, ICLR, 2024

Analysis on computation and memory costs (Weakness 2).

Thanks for the comment. We have actually conducted comprehensive analyses of these aspects in our paper, particularly in Section 4.3, Appendices I and Appendix J.

Computational Efficiency. Our alternating optimization strategy ensures that coefficients and locations are not optimized simultaneously, preventing additional computational overhead compared to coefficient-only optimization. As demonstrated in Section 4.3, our central difference approximation for gradient estimation can be efficiently implemented. The gradient computation in Eq. (5) shows that $Z$ can be reused for all location updates (can be found in the code provided in the Supplementary Material), making the additional computation negligible. Our complexity analysis in Appendix I formally proves that the computational complexity remains in the same order as coefficient-only optimization.

Memory Usage and Runtime Performance. We have conducted extensive empirical comparisons among LoRA, FourierFT, and LoCA across various datasets, model scales, and parameter budgets (detailed in Appendix J, Table 10). Our empirical study reveals that while LoCA theoretically has different asymptotic complexity, its practical running time is comparable to FourierFT and only marginally slower than LoRA (which benefits from highly optimized GPU implementations of matrix operations). Regarding memory consumption, LoCA demonstrates consistently lower memory usage compared to FourierFT, though both methods require slightly more memory than LoRA.

Future Optimization Opportunities. Our current fast DCT implementation uses FFT, which introduces some computational overhead. We identify potential improvements through specialized fast DCT algorithms. Since our method operates on real-valued data, DCT is theoretically more efficient than FFT as it avoids unnecessary complex number operations.

Additional benchmarks for fine-tuning Roberta (Question 2).

Thanks for the comment. We would like to clarify our rationale for benchmark choices and explain how we have enhanced our evaluation scope in the revised manuscript.

For RoBERTa fine-tuning experiments, we utilized the GLUE benchmark, which has become the de facto standard for evaluating PEFT methods in recent literature. Our evaluation comprehensively covers 8 tasks from GLUE, which is more extensive than recent PEFT works such as VeRA and FourierFT that only evaluated on 6 tasks. These 8 tasks encompass diverse aspects of language understanding, providing a comprehensive assessment of model capabilities.

To further strengthen our evaluation, we have added a new section (Section 5.2: Natural Language Generation) in the revised manuscript, which evaluates different PEFT methods on the E2E NLG Challenge dataset using GPT-2 Medium and Large models. The E2E NLG Challenge is a widely-adopted benchmark in the PEFT community. This addition complements our GLUE evaluation by examining performance on generation tasks, offering a more complete picture of our method's effectiveness across different types of language tasks.

We believe this combination of comprehensive GLUE evaluation and additional NLG experiments provides robust validation of our method's effectiveness, while maintaining comparability with existing PEFT literature.

Comparison with other LoRA variants (Question 3).

Thanks for the comment. We have included comparisons with DoRA [R2] (Table 1) and VeRA [R3] (Table 1 and Table 2) as baseline methods. Specifically, DoRA was included in our original submission, while VeRA was added in response to reviewer STG6's suggestion. We acknowledge the importance of comprehensive comparisons and believe our current baseline methods provide a strong foundation for evaluating our proposed approach.

[R2] Liu et al., Dora: Weight-decomposed low-rank adaptation, ICML, 2024.

[R3] Kopiczko et al., Vera: Vector-based random matrix adaptation, ICLR, 2024.

Performance scaling on Figure 3 (Question 4).

Thanks for the comment. We have conducted additional experiments with $r=16$ for both LoRA (91.23%) and LoCA (91.34%) on QQP. While there is indeed a slight improvement, it is worth noting that the y-axis scale in Figure 3 spans a relatively small range, indicating that the performance gains are actually quite marginal. Moreover, due to QQP's large dataset size, experiments with other rank values are computationally intensive. The observed pattern suggests that the performance is approaching saturation, making $r=8$ a reasonable trade-off between computational efficiency and model performance for practical applications.

Thanks again for your suggestion. Due to computational resource and time constraints, we have currently conducted preliminary fine-tuning only on the LLaMA-7b base model, comparing FourierFT and LoCA. We conducted FT for 2 epochs, with the batch size set to 16 (gradient accumulation steps = 8). We apply PEFT modules on q_proj, k_proj, v_proj, up_proj, down_proj, with 50K frequency components for reparameterizing each matrix. Other hyperparameters are maintained the same as shown in Table 8 (e.g., learning rates and scaling values). Below are current results.

In-domain Results:

Out-of-domain Results:

While these initial results show comparable performance, we acknowledge that a more comprehensive evaluation with different hyperparameter settings and model scales would provide stronger validation. We will continue to explore these points.

Thanks for the valuable comments. We address each point raised in the review with detailed responses below.

Limited amount of content on related work (Weakness 1).

Thanks for the comment. We respectfully disagree with the assessment that our literature coverage is insufficient. Our paper adopts a strategic organization of related work that avoids redundancy while maintaining comprehensiveness. Specifically:

• The Introduction section provides a thorough overview of PEFT methods, systematically categorizing them into adapter-based methods, prompt-based approaches, partial fine-tuning, and low-rank adaptation methods. We also discuss various LoRA variants, establishing the necessary background and motivation for our work.

• The Related Work section takes a different angle, focusing on connecting PEFT with matrix compression techniques. This novel perspective allows us to frame PEFT methods through the lens of matrix compression, and draw parallels between low-rank decomposition and frequency-domain compression. Beside, it identifies and addresses a critical gap in PEFT literature regarding theoretical comparisons between these approaches.

• Detailed discussions of individual PEFT methods are now presented in Appendix C, where we provide details of baseline methods.

We believe this organization is more effective than duplicating method descriptions across multiple sections. Each section serves a distinct purpose: Introduction establishes context, Related Work provides unique insights through compression perspective, and Appendix C offers detailed method descriptions.

Experimental results on more datasets (Weakness 2).

Thanks for the suggestion. In response to this concern, we have expanded our experimental evaluation in the revised manuscript to include a new section (Section 5.2) that examines LoCA's performance on the natural language generation (NLG) task. Specifically, we conduct experiments on the E2E NLG Challenge benchmark using GPT-2 Medium and Large models. The results, presented in Table 2 in the revised manuscript, demonstrate that LoCA achieves superior performance compared to existing PEFT methods across multiple established metrics, particularly when for the GPT-2 large model. All experimental hyperparameters are reported in Table 7 of the revised manuscript.

Our current experimental framework now comprehensively evaluates LoCA across NLU, NLG, instruction tuning, and computer vision. This diverse evaluation encompasses varying model scales (from RoBERTa-base to LLaMA-13b, and from ViT-base to ViT-large) and tasks of different complexity (from basic classification to open-ended generation). We believe this comprehensive evaluation is sufficient to illustrate the applicability and robustness of LoCA.

Thanks for the careful reading and thoughtful comments. In the following, we address each of the concerns raised in detail.

Computational complexity of alternating optimization and central difference approximation (Weakness 1).

Thanks for the concern about the computational complexity of LoCA. We would like to clarify several points that demonstrate LoCA's computational efficiency.

Although LoCA employs alternating optimization, it sequentially optimizes coefficients and locations rather than simultaneously. This means the computational overhead per iteration remains constant compared to coefficient-only optimization. During each iteration, we only update one set of parameters while keeping the other fixed.

Regarding the central difference approximation, we have provided the expression in Section 4.3 and a formal complexity analysis in Appendix I. The gradient computation for locations can be efficiently implemented by reusing the DCT of the gradient matrix across all locations and coefficients (the code can be found in the Supplementary Material). This leads to computational complexity comparable to coefficient-only optimization. In practice, the training time per iteration remains stable throughout the training process.

To address similar concerns from Reviewers 3NDS and vduh, we have conducted comprehensive empirical studies comparing LoRA, FourierFT, and LoCA across different datasets, model scales, and parameter budgets. Our results (Table 10, Appendix J in the revised paper) demonstrate that the practical running time of LoCA is comparable to FourierFT and only marginally slower than LoRA (which benefits from highly optimized GPU implementations). Besides, LoCA consistently shows lower memory consumption than FourierFT, though both require slightly more memory than LoRA.

Furthermore, we would like to identify that our current fast DCT implementation uses FFT, which introduces some overhead. A specialized fast DCT implementation in Pytorch could improve efficiency. We leave this efficiency improvements in future implementations. Beside, DCT is theoretically more efficient than FFT for real-valued data, as FFT's complex number operations introduce unnecessary computations

We appreciate the comprehensive evaluation and insightful suggestions. Below we provide detailed responses to address each of the concerns raised.

Results on StanfordCars and FGVC datasets (Weakness 1. & Question 1).

Thanks for the comment. We would like to emphasize that our implementation is based on the official FourierFT codebase, and we have taken great care to ensure experimental rigor. We use the same ViT models (pretrained on ImageNet-21k) as specified in our implementation details, and all experiments were conducted under identical conditions for fair comparison across methods. All hyperparameters for both our method and baseline methods are thoroughly reported in Appendix D to ensure reproducibility and fair comparison.

Regarding the specific performance on Stanford Cars and FGVC datasets, we have consulted with other researchers in the PEFT community who have confirmed that their baseline results closely align with ours. To ensure complete transparency and reproducibility, we have made our implementation publicly available in the Supplementary Materials. This allows anyone to verify our results and experimental procedures.

Comparison with FourierFT under the same parameter budget (Question 2).

Thanks for this valuable suggestion. We have conducted additional experiments to compare FourierFT and LoCA under equal parameter budgets (239K and 480K) for ViT models. The results are now updated in Table 4 in the revised manuscript.

For ViT-base with 239K parameters, LoCA achieves better performance than FourierFT across multiple datasets (e.g., OxfordPets: 94.10% vs 93.44%, DTD: 80.15% vs 79.43%, FGVC: 54.86% vs 52.26%). Similarly, for ViT-large with 480K parameters, LoCA consistently outperforms FourierFT (e.g., StanfordCars: 83.47% vs 82.27%, FGVC: 63.02% vs 56.96%).

Training time and memory cost (Question 3).

Thanks for the comment. We have conducted comprehensive empirical evaluations comparing LoCA, LoRA, and FourierFT across different datasets, model scales, and parameter budgets. The results have been added to the revised paper (Table 10) and corresponding analysis section (Appendix J).

Our empirical study reveals that while LoCA theoretically has different asymptotic complexity, its practical running time is comparable to FourierFT and only marginally slower than LoRA (which benefits from highly optimized GPU implementations of matrix operations). Regarding memory consumption, LoCA demonstrates consistently lower memory usage compared to FourierFT, though both methods require slightly more memory than LoRA.

We also discuss potential optimization opportunities: our current fast DCT implementation relies on FFT, which introduces some computational overhead. A specialized fast DCT algorithm could potentially improve LoCA's efficiency further. Additionally, we note that DCT is theoretically more efficient than FFT for real-valued data (which is our case), as FFT's complex number operations introduce unnecessary computations, leading to slower training speed and more memory cost. These optimizations represent promising directions for future work.

Thanks for the thorough review and constructive feedback. We have carefully considered all the comments and address each point in detail below.

Unprecise terminology (Weakness 1).

Thanks for pointing out this. We have revised the manuscript to use hypothesis space instead of optimization space, as it better reflects the set of all possible functions that a model can learn. Similarly, we have replaced flexible optimization with enhanced expressivity to more precisely describe the model's capability to represent diverse solution spaces. These words align better with standard terminology in machine learning literature.

Asymptotic normality of M-estimator (Weakness 2).

Thanks for the comment. We can explain this from two aspects. From a strict theoretical perspective, a more general version of M-estimator is derived in [R1] (page 47, Lemma 5.10). This generalization applies when both $\Psi(\theta)=0$ and $\Psi_n(\theta)=0$ yield unique solutions $\theta^*$ and $\hat{\theta}_n$ respectively, and $\Psi$ exhibits local monotonicity at point $\theta^*$. Notably, this relaxes the traditional requirement of global monotonicity. Furthermore, the conventional full rank condition is replaced by the pointwise convergence of $\Psi_n(\theta)$ to $\Psi(\theta)$ - a property that is satisfied in our case through the WLLN.

However, requiring a unique solution is somewhat restrictive in neural networks. When there exists a set of global minimizers for the risk function, the consistency property still holds when we consider $\Theta^*$, defined as the solution set to the equation $\Psi(\theta)=0$. Detailed expression of this result can be found in Theorem 5.14 (page 48) of [R1]. To satisfy the assumptions required by this theorem, we can constrain the parameter matrix by clipping each element to lie within the interval $[-M,M]$, where $M$ is a defined parameter range.

To establish asymptotic normality, apart from the above consistency property ($\hat{\theta}_n \xrightarrow{p} \theta^*$)， we need to verify another two conditions: (1) the existence of first and second derivatives, and (2) the finiteness of their corresponding expectations, provided that consistency has been established. These conditions are satisfied under the boundedness assumption. Therefore, we can focus solely on verifying the conditions for consistency (Theorem 5.21 of [R1], page 52).

From an empirical perspective, the asymptotic normality requires enormous data points and the minimizers of a deep neural network are extremely complex. Therefore we do not try to verify the conditions in theoretical analysis as they are are only sufficient conditions, not necessary conditions. In this work, we regard the asymptotic normality of M-estimators as a commonly used assumption in statistics and machine learning. Then, we shift our focus to the actual data in the experiments and conduct visualization (Figure 1a, Section 2), statistical tests (Figure 1b, Section 2), and ESD analysis (Figure 1c, Section 2) to validate the reasonableness of our assumptions about the data distribution.

[R1] Asymptotic Statistics, A.W. van der Vaart, 2000.

Limited scope of theoretical analysis (Weakness 3).

Thanks. Although a unified theoretical analysis encompassing all low-rank methods may not be feasible, we can still conduct case-by-case analysis, as all low-rank-based methods have an inherent upper bound on reconstruction capability.

For a given $\Delta W\in\mathbb{R}^{n\times n}$, VeRA [R2] decomposes it to $\Lambda_bB\Lambda_dA$ where $B,A$ are draw i.i.d. from a certain distribution and frozen and shared over all training steps and layers, $\Lambda_b,\Lambda_d$ are learnable diagonal matrix. From a reconstruction perspective, the $i$-th element of $\Lambda_b$ is the OLS coefficient while setting the response as $i$-th row of $\Delta W$ and covariate as $i$-th row of $B\Lambda_dA$. This idea enables us to find $\Lambda_d$ that maximize the correlation between $i$-th row of $\Delta W$ and $i$-th row of $B\Lambda_dA$. However $A$ and $B$ are chosen randomly independent of $\Delta W$, the reconstruction error is approximately the error we learn from white noise.

We can conduct a detailed theoretical analysis of DoRA [R3], here we only give the outline. For a given $\Delta W$, DoRA first decomposes it as $\Delta W=A\Lambda$ where $\Lambda$ is diagonal and each column of $A$ has magnitude $1$. The $r$-rank approximation is $A_r\Lambda$, where $A_r=U_r\Lambda_rV_r^T$, and $U_r,V_r\in\mathbb{R}^{n\times r}$ and $\Lambda_r$ contains $r$ largest eigenvalues of $A$. If each element in $\Delta W$ follows i.i.d. standard normal, we can derive the independency of $A$ and $\Lambda$. Using total expectation, we have the following reconstruction loss

As each non-zero element in $\Lambda$ follows i.i.d. $\chi(n)$ distribution. Subsequent calculations only require computing the reconstruction loss based on the distribution of $A$. At this point, the reconstruction loss is consistent with the LoRA method, except that the distributions are different. This requires complex calculations, but since each column of $A$ is the direction of a random normal vector, the difference should not be significant. The loss corresponding to DoRA should therefore be approximately the same as that of LoRA.

We have supplemented this discussion in Appendix O in the revised manuscript.

[R2] Kopiczko et al., Vera: Vector-based random matrix adaptation, ICLR, 2024.

[R3] Liu et al., Dora: Weight-decomposed low-rank adaptation, ICML, 2024.

Theory-practice alignment (Weakness 4).

Thanks for the comment. We would like to discuss this point from three key aspects. First, our theoretical analysis primarily focuses on matrix reconstruction capability, which may not perfectly align with downstream task performance since the task performance is influenced by multiple factors, such as random seeds, hyperparameters, etc. This phenomenon is evident in some cases where PEFT outperform FF despite higher reconstruction loss. However, matrix reconstruction serves as a reasonable proxy for model performance without task-specific priors. Second, all our theoretical results are derived in expectation, which analyse average-case behavior. As noted in Section 5.5 (Performance under Different Parameter Budgets), specific task structures may indeed favor low-rank methods or FourierFT in certain instances. These exceptions do not invalidate the general theoretical framework, since LoCA does outperform FourierFT in average performance on GLUE and ViT experiments. Third, the theorem requests to identify the optimal learnable locations for reconstruction. However, the practical implementation relies on gradient approximation, which may not achieve global optimality for all locations. We have acknowledged this limitation and discuss its implications in Appendix M (Remark).

Comparison between FourierFT and LoCA with the same number of trainable parameters (Weakness 5 and Question 2).

Thanks for the comment. Following similar feedback from Reviewer 3NDS, we have conducted additional experiments comparing LoCA and FourierFT under identical parameter budgets (using both 3000 and 10,000 frequency components) for ViT models. The updated results are now presented in Table 4 of the revised manuscript.