Efficient Learning with Sine-Activated Low-Rank Matrices

We thank the reviewer for their review and greatly appreciate the time and effort they have put in reviewing our paper. Below we will answer each of the reviewers questions in detail. We have also made changes as requested by the reviewer and have uploaded an updated version of our paper.

1. In formula 3, why the coefficient w and g are non-learnable ? If learnable, will it cause gradient problem? For the same model, has it been considered to provide different w and g for different linear layers? Can the recommended values for these two be provided?

In order for these coefficients to be learnable we found that initializing them the right way was hugely important. If we initialized them at zero we found training was affected and if we initialized them too large, we witnessed exploding gradients. For $g$ we followed the approach taken in the paper by He et al. [1]. We set $g$ to be $\sqrt{n}$ where $n$ is the number of rows of the weight matrix (i.e. the fan_in in PyTorch). This was chosen for the following reason. During backpropagation the sin function has a frequency $\omega$ and when taking a gradient the frequency $\omega$ comes out the front scaling each gradient. So as to mitigate gradient explosion we found that taking $g = \sqrt{n}$ worked well. For the frequency unfortunately there is currently no principled way to choose $\omega$. Theorem 1 suggests that larger values of $w$ is better but it does not quantify how large. Therefore, for each experiment and each rank we treated $w$ as a hyperparameter and tuned it to what gave the best results. We have made a remark about this for the reader in the appendix, see remark 2 in the appendix.

[1] Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification by He et al. ICCV 2015.

2. Since other nonlinear functions are almost useless, why not directly give the sine function in Equation 3?

We thank the reviewer for pointing this out and agree with their comment. Our main reason for including a general $\phi$ is to motivate the procedure as a general construction. We have added a sentence in line 171 saying that our work shows that sin is the best choice.

3. Regarding the right diagram in Figure 3, I don't think we can see a trend of the matrix rank increasing with the increase of frequency w. Because the rank of a matrix is only related to the number of nonzero singular values. So when w is 200 and 400, it seems that the ranks of the two are almost equal.

We understand the reviewers point with this figure. The purpose of figure 3 is to show that by applying a sin function with different frequencies we can move the spectrum to the right. For example if you look at the right figure in figure 3 you can see that the case of $w = 100$ there are few singular values that are almost zero but in the case of $w = 200$ these have become much larger showing that the numerical rank can be increased.

4. In the setting of section 4.1, why do not implement low-rank adaptations on all linear modules of LLM?

For our implementation of low rank adapations on linear module we followed work in the literature. For RoBERTa the low rank adaptation is applied only on $W_q$ and $W_v$ as this is what was done in the original LoRA paper and we wanted to compare our method directly with LoRA. However, for LLaMa3-8B the low rank adaptation is applied on $W_q$, $W_k$,$W_v$, $W_{up}$ and $W_{down}$ as this is what was done in the recent DoRA paper [2] and we wanted to compare our method directly with DoRA. This gives two different situations of applying low rank adaptations and in both cases our method performs better. Results of our method on DoRA can be found in A.2.1 Table 7. As can be seen from that table sine DoRA outperforms DoRA at all rank levels.

[2] Liu et al. DoRA: Weight-Decomposed Low-Rank Adaptation. ICML 2024

5. Does the rank of the decomposed matrix necessarily be less than that of the original matrix regardless of how the frequency is adjusted?

If $\omega$ is very small, then $\sin(\omega x) \approx \omega x$ and thus applying a sin simply scales the matrix by $\omega$ which cannot change rank. It is only when $\omega$ is large that we see that the rank increases. If $\omega = 0$ then applying $sin$ with frequency $\omega$ to the matrix produces the zero matrix which has zero rank and in general will thus have rank less than $A$. We have included remark 2 in the appendix that explains this.

Dear reviewer EgEA, we thank you very much for your review. Before the discussion period ends, we would like to check if our response has addressed the reviewers concerns or if there are remaining issues we may clarify that would increase your confidence that this paper should be accepted?

We thank the reviewer for their review and greatly appreciate the time and effort they have put in reviewing our paper. Below we will answer each of the reviewers questions in detail. We have also made changes as requested by the reviewer and have uploaded an updated version of our paper.

1. Lack of Rationale Behind the Sine Function's Effectiveness:While the theoretical and experimental validations in Sections 3.2 and 4 establish the efficacy of the sine activation, the paper doesn't delve into why the sine function specifically enhances parameter-efficient training.......

Thank you very much for your question. Our main reason for using a sine function is that it has periodic behaviour around the origin that can be controlled by a frequency parameter. This allows us to show that for any matrix $\mathbf{A}$, that is not the zero matrix, there are frequencies $\omega > 0$ that allow us to lower bound the Frobenius norm of $\sin(\omega\mathbf{A})$ by a quantity that depends linearly on the frequency $\omega$ and upper bound the operator norm of $\sin(\omega\mathbf{A})$ by a quantity that depends sub-linearly on the frequency $\omega$. This is the content of lemmas 1 and 2 in appendix A.1 and are the two key reasons why we can increase the rank of $\mathbf{A}$ after applying a sine function with a large enough frequency. When choosing a non-linear function to increase rank we found that our theory implied the function should satisfy such properties. This is how we then came to the conclusion that a sine function would be a good choice. The proof of both lemmas uses the fact that sine is a periodic function whose frequency can be controlled by a positive parameter $\omega > 0$ which is what we found was critical. In general, other non-linearities like Sigmoid and ReLU do not satisfy such properties and this is why we found that they cannot be made to increase rank. We have added a remark in the appendix after the proof of lemma 2 that clearly explains this for the reader.

2. Hyperparameter "w" Tuning: The role of the frequency hyperparameter "w" isn't thoroughly addressed. Is there a need to fine-tune "w" for each task, or does it generally benefit from being larger? More importantly, how do we determine the optimal "w"? A systematic exploration or guidelines for selecting "w" across different tasks would be helpful, as the current approach seems to leave this critical aspect underexplored.

Yes we treat the frequency $w$ as a hyperparameter and then do a search for the best parameters. In general the bigger $w$ is we found the better, as this also follows from Thm. 1, however the theorem does not quantify exactly how large we should take $w$ to be. Due to this we found we had to run a search to tune for the best $w$ for each experiment. It is our feeling that a principled approach to choosing $w$ for a given application is an extremely interesting problem that we hope to take up in future work. We have carried out ablations on the frequency in the appendix. Please see A.2.2 for the case of ViT-small, A.2.4 for the case of NeRF and A.2.5 for the case of 3D shape modelling.

3. Applicability to CNN Architectures: Although the paper demonstrates strong results on various architectures like ViT, LLMs, NeRF, and 3D shape modeling, it leaves a gap concerning CNNs.....

We thank the reviewer for their suggestion. As per requested we ran our method on the ConvNeXt-Tiny architecture [1], a state of the art convolutional architecture, on the CIFAR100 dataset. The results are shown appendix A.2.3 Table 11. For the convenience of the reviewer we show the results in the table below. As can be seen, our sine low-rank method achieves better accuracy at all rank levels and at rank 20 outperform the original full rank ConvNeXt-Tiny architecture.

Performance of ConvNeXt-Tiny model trained on CIFAR100 datasets.
We use frequency [400, 300, 300] for rank [1, 5, 20], respectively.

Thank you for your author rebuttal.

The authors have provided a clear explanation of the rationale behind the effectiveness of the sine function (W1). While the need for further exploration of the hyperparameter $w$ remains, this can be addressed in future work. Additionally, the effectiveness of the proposed approach within the CNN architecture has been adequately demonstrated.

I maintain my stance that this study should be accepted.

3. It would be good to provide space and time complexity analysis of the proposed method.

In the appendix we have added quantitative results on number of parameters, memory and training time see appendix A.2.1. While a theoretical analysis of the space-time complexity of our method is useful we note that adding a sine does not increase any parameters and thus only adds overheads during backpropagation when a derivative of the sine function is computed and during forward propagation when the sine function is applied to the weight matrix. This will be dominated by other components of the optimization process.

4. There is a lack of quantitative results on a standard dataset for the 3D shape modeling task. It would be good to evaluate the method on other popular tasks such as text to image generation.

We carried out experiments in the application of binary occupancy fields for 3D shape reconstruction as we found that the original binary occupancy paper [2] had 3000 citations and was commonly used for 3D shape modeling. Therefore we decided to test our method on this novel task. The work [2] used the Stanford 3D Scanning Repository (https://graphics.stanford.edu/data/3Dscanrep/) as their data sets and we thus followed that approach. To show the reviewer that we are committed to showing that our methodology works for various instances, we have run further experiments on binary occupancy fields using two more instances from the Stanford 3D Scanning Repository. The complete quantitative results can be found in appendix A.2.5. For you convenience we have shown the results below.

We thank the reviewer for suggesting text to image tasks however this is a completely different field to explore for our method. To show that our method is applicable across a variety of fields we carried out tasks on fine tuning LLMs, pre-training vision transformers, neural radiance fields and 3d shape modeling. Furthermore, we have added a new section in the appendix, namely appendix A.2.3, on applying our method to a state of the art convolutional architecture. We believe text to image is an interesting area to consider and will consider it for future work.

[2]: Occupancy networks: Learning 3d reconstruction in function space, CVPR 2019.

5. How do we choose the frequency parameter (ω) in practice? Is there an easy way to adjust it?....

We chose the frequency parameter by treating it as a hyperparameter and tuning it. We find that the optimal frequency varies according to the target rank. We believe finding a principled way to choose the frequency is an important problem that we plan on taking up for future work.

6. For each application, the implementation details of the proposed method should be provided.

We thank the reviewer for this suggestion. We have added all implementation details for each experiment in the appendix.

7. Are there any implementation challenges, like increased computation time during training? Does this technique slow down the model during inference, especially in environments with limited resources?

One of the advantages of our method is that it is a plug-in method that works on all feedforward architectures within a variety of neural network models. This was the main reason we applied it to a variety of different architectures such as ViTs, fine tuning LLMs, NeRFs, Binary Occupancy Fields, to show it works very easily in all these settings. The main challenge we faced is that we had to tune the frequency hyperparameter of the sine function for each experiment. We have uploaded code for the reviewer where it can be seen that our method is a simple plug in method.

2. While the proposed method is applied across various domains, the paper lacks a comprehensive comparison with existing low-rank approximation techniques. There are many existing LoRA variants and improved versions, e.g. [a] [b]....

We thank the reviewer for this suggestion. We would like to remind the reviewer that ICLR reviewer instructions state that works are considered contemporary if they were published at a peer reviewed venue after 1 July 2024, and as such authors aren't required to directly compare to such works or works that have not been published at a peer reviewed venue (see https://iclr.cc/Conferences/2025/ReviewerGuide last question and answer). We point out to the reviewer that reference [a] has not been published at a peer reviewed venue and was recently uploaded to the ArXiv on 17th of October 2024. Furthermore, [b] was published at ACL on 24th of August 2024. However, we are committed to making our paper as relevant as possible to the wider community. Therefore, as suggested by the other reviewers, we have ran our method on the recent state-of-the-art work DoRA [1] which was published on 21 July 2024 in ICML. The results are shown in appendix A.2.1 Table 7.

For the convenience of the reviewer the results for our method on DORA is shown in the table below. As can be seen our sine DoRA outpeforms the standard DoRA at each rank level, and Sine DoRA matches the performance of Rank 32 Dora at only Rank 8.

Performance and parameter count of the LLaMA 3-8B model fine-tuned using the DoRA and sine DoRA methods across varying ranks

[1] Liu et al. DoRA: Weight-Decomposed Low-Rank Adaptation. ICML 2024

We thank the reviewer for their review and greatly appreciate the time and effort they have put in reviewing our paper. Below we will answer each of the reviewers questions in detail. We have also made changes as requested by the reviewer and have uploaded an updated version of our paper.

1. The paper talks about the benefits of using a sine function for increasing rank, but it could explain more about why this function was chosen over others and why it works....

Our main reason for using a sine function is that it has periodic behaviour around the origin that can be controlled by a frequency parameter. This allows us to show that for any matrix $\mathbf{A}$, that is not the zero matrix, there are frequencies $\omega > 0$ that allow us to lower bound the Frobenius norm of $\sin(\omega\mathbf{A})$ by a quantity that depends linearly on the frequency $\omega$ and upper bound the operator norm of $\sin(\omega\mathbf{A})$ by a quantity that depends sub-linearly on the frequency $\omega$. This is the content of lemmas 1 and 2 in appendix A.1 and are the two key reasons why we can increase the rank of $\mathbf{A}$ after applying a sine function with a large enough frequency. The proof of both lemmas uses the fact that sine is a periodic function whose frequency can be controlled by a positive parameter $\omega > 0$ which is what we found was critical. In general, other non-linearities such as sigmoid and ReLU do not satisfy such properties and this is why we found that they cannot be made to increase rank. We have added a remark in the appendix after the proof of lemma 2 that clearly explains this for the reader.

3. In line 182, and 765 is the A_{ij} \neq 0 maybe a A_{ij \neq 0} ? It looks as if the indexes should not be 0 and not as if the assumption of A \neq 0 is formalized.

The way we have written this in the paper is correct. $A_{ij}$ denotes the $(i,j)$ entry of the matrix $A$. In line 182 and 765 the statement $A_{ij} \neq 0$ means the $(i,j)$ entry is not zero. This is used because we want to take the minimum over all the matrix entries of $A$ that are not zero.

4. The operator norm in equation 5 has one | too much.

Thank you. However, the operator norm as written is correct. The notation may look confusing because we are taking the absolute value of the matrix $A$ and then doing a square root. So we first form $|A|$ which is the component-wise absolute value of $A$ and then we take the component-wise square root of $|A|$ forming $\sqrt{|A|}$ and then take the operator norm of $\sqrt{|A|}$ forming $|| \sqrt{|A|}||_2$. The extra $|$ you see is coming from the absolute value part of $A$. To make this easier on the eyes we have changed the operator norm in equation (5) to be larger so the reader can clearly see what our notation is doing. We hope this make things easier to read.

5. Why is the observation in line 786 important for 11? This should be true in any case.

Thank you. The line 786 is redundant. As you say it is true in any case. We have removed that line.

6. Line 799 Says Equation equation 13. Probably an issue with cref

Thank you! This has been corrected.

7. The sentence starting in 855 is somehow messed up.

Thank you! You are absolutely right. We have removed that sentence.

8. Using parameter counts instead of compression rate in Figure 1 for ViT Classification would improve comparison with other state-of-the-art approaches.

Thank you! We have changed figure 1 and figure 4 to show parameter counts instead of compression rates.

9. Why does an increased rank improve performance? This point may have been demonstrated by experiments but an intuition could be given. Adding certain noises to the matrix may also increase it's rank without increasing it's parameter count but will probably not lead to improved performance......

High-rank matrices offer greater degrees of freedom than low-rank matrices, enhancing their representational capacity. For complex datasets, we hypothesize that this increased flexibility allows the network to capture and learn key features more effectively. While we acknowledge the reviewer’s suggestion that adding noise can also increase rank, this approach risks propagating noise through the output, potentially reducing the model's effectiveness. Our empirical results across all tasks support our hypothesis, demonstrating that increasing rank using a sine activation provides an effective way to model complex signals with low parameters. This approach balances representational power and parameter efficiency. We have expanded on this in Appendix A.1, Remark 3.

10. In line 480 I am not sure it is good to speak of overfitting regarding to NeRFs because this sounds as if it's negative when it is the desired behavior.

Thank you for the suggestion. We have decided to remove ``overfitting" from that sentence.

11. Figure 10 seems to indicate the frequency parameter has an optimum. How hard is it to choose it? Are the best results retrieved from a grid search or is there any intuition about it's value beforehand?

We chose the frequency by treating it as a hyperparameter and tuning it. In general the best results are indeed found via a grid search. We do however believe that finding a principled way to choose the frequency is an important problem that we plan on taking up for future work.

We thank the reviewer for their review and greatly appreciate the time and effort they have put in reviewing our paper. Below we will answer each of the reviewers questions in detail. We have also made changes as requested by the reviewer and have uploaded an updated version of our paper.

1. Figure 4 in the text is referenced as table 4.

Thank you we found this error on line 405 and have changed it.

2. Why is DoRA cited only once as Lora technique but then there is no reference to it's improved performance?

We thank the reviewer for pointing out this inconsistency. We have added results for DoRA in appendix A.2.1 Table 7. For the convenience of the reviewer we have given our DoRA results below. As can be seen from the table our sine DoRA outperforms DoRA at each rank level and rank 8 sine DoRA matches the performance of rank 32 DoRA.

Performance and parameter count of the LLaMA 3-8B model fine-tuned using the DoRA and sine DoRA methods across varying ranks

I have read the answers of the authors and am particularly pleased with the investigation into Dora. Even though a discussion into the effect size could follow at the conference, I think the paper should be accepted and the method discussed in the larger audience of the ICLR. Due to the numerous improvements and investigations I improved my rating to accept.