FouRA: Fourier Low-Rank Adaptation

We appreciate reviewer 4zX8 for their detailed feedback and an in-depth review to helped us improve our work.

Training Time: We provide detailed analysis of training time per epoch in Table R.1. One training epoch takes 24.5s (for FouRA with inference adaptive masking) compared with 22s (for baseline LoRA), by keeping the rank fixed across two methods. We will add this analysis to the paper.

GPU Memory: Thanks for the suggestion. We report peak memory usage in Table R.1. We further analyze performance with varying training complexity (training time, memory usage) in Figure R.1. To vary time, we report HPS scores of FouRA v/s LoRA at intermediate epochs. To vary the memory, we use rank. We observe that FouRA consistently achieves better performance v/s compute operating points compared to LoRA.

About Fourier: Similar to $\Delta{W_{lora}}$, $\Delta{W_{foura}}$ is defined as the weight projection of the second term in Eq.5. The term $\mathbf{G}$ is the output of $\mathcal{G}$. We will clarify this in the text. We also want to clarify in Sec. 4.1 that $\Delta{W_{2}} = \mathcal{F}^{-1}BA\mathcal{F}$ and $\Delta{W_{1}} = BA$ are FouRA(no-mask) and LoRA trained weights, not potential fitting targets. The singular value spread in Figure 4 is of a low-rank approximation of both these trained matrices, following prior works [1, 2]. It can be inferred from [1, 2, 3] that the compactness in eigen-spread proves the capability of FouRA adapters over LoRA in generating lower errors when rank is reduced. It also shows that the frequency domain can learn richer information given a sparsity constraint. We also show in Appendix B.3. how FouRA learns representations which are more de-correlated from base weights.

Our analysis above was on trained weights of the diffusion model, and did not require a toy task. Per your suggestion, we have conducted an analysis on fitting the MNIST task, comparing the training loss of FouRA (without gating) and LoRA layers. Figure R.3 shows results for two ranks. Train params are equal for both adapters. As observed, Fourier domain training leads to lower errors compared to LoRA. We also find that with reduced rank, the gap between LoRA and FouRA widens.

Adaptive Rank: We provide intuitions for our proposed adaptive gated rank selection algorithm in Sec. 4.2 of the paper. Adding to this, we argue that input dependent rank selection is advantageous as it not only selects rank, but also specific vectors in the low rank subspace. The ideal vector directions in the low-rank subspace vary with inputs having different characteristics e.g., certain vectors will be sensitive to specific frequencies, and we argue that our proposed input-based selection algorithm finds optimal vectors as compared to a frozen dynamic gating function [4]. In a diffusion model for instance, at varying diffusion timesteps (corresponding to different levels of input noise), optimal vectors vary based on their sensitivity to the noise. We also analyze this intuition in Fig.5 by plotting effective rank across denoising unet, across timesteps. Observe that the learnt effective rank reduces as the diffusion process concludes, meaning lesser noisy inputs are sensitive to lesser vectors. Similarly, higher input resolution ideally requires a higher number of vectors (hence the higher effective rank at up.3 and down.0 blocks in the diffusion Unet). We provide ablation studies in R.2 (and Fig.9 of text) to empirically validate this motivation, showing that FouRA with adaptive masking outperforms FouRA with frozen masking. Finally, from your suggestions, we also plot the training curves for Fourier v/s Fourier+gating in Fig.R.4, justifying that speed of convergence is not affected by gating.

Merging: Sec. 3.5 motivates the use of FouRA adapters in merging two adapters as compared to LoRA. Please see Appendix B.3.1 and B.4, as they are important analysis conducted to demonstrate that FouRA learns representations which have a higher likelihood of being disentangled between two FouRA adapters, compared to LoRA. This property proves to be critical in adapter merging, as FouRA can generate images which successfully retain capabilities of both adapters during adapter merging. We also observe higher amplification of subspaces not emphasized by base frozen model in Table B.2. This is important as FouRA is a training-free approach to improve the merging capabilities of low-rank-adapters, providing great flexibility over contemporary works which propose joint training methods for this orthogonalization of subspaces. Please see the results in Figure 7 and Section 5.2 and 5.3 for adapter merging. These are all from a training-free merge, which is a simple arithmetic add. Additional analysis we conducted show that scaling up the number of trainable params in LoRA to match FouRA does not affect performance, and FouRA continues to outperform LoRA with a similar delta. Thanks for proposing it, we will include this study.

Generalizability: We agree that the results on GLUE tasks might not be sufficient. Hence, we have performed further analysis on eight commonsense reasoning benchmarks, using Llama3-8B as our backbone. These results in Table R.3. show that FouRA with r=16 and r=32 outperforms LoRA at r=32, suggesting FouRA as a generalizable PEFT method. Additionally, while we agree that FouRA was originally motivated for text-to-image models, we believe its unique aspects such as compactness in the Frequency domain and adaptive rank are generalizable across domains. We prefer reporting GLUE results to show generalizability across multiple tasks. Having said this, we are open to move the GLUE results to the Appendix and reorganize the paper to further explain the benefits of FouRA in text-to-image tasks.

Q1: We thank you for pointing this out. Indeed, the line 248 is a typo. We use DeBERTaV3-base as the backbone from [4]. Appendix C has implementation details and Appendix I has a code snippet.

We appreciate reviewer yqCK for their meticulous review and insightful feedback, helping us improve our work.

Memory Overhead/Scaling with batch size: Thank you for raising the point on memory. We provide details including memory overhead in Table R.1 of the rebuttal pdf. The reported numbers in Table R.1 are for a batch size of 8. Further, we report the scaling based on batch size in the following table:

We can observe that the FouRA GPU memory overhead during training time is negligible and only 0.3-0.4% over LoRA. We will include our analysis in the paper.

Title: Thanks for the suggestion. We agree with your observation and will reflect these in the title (if the platform allows it). Having said this, we also show that FouRA is a generic approach which works on non-diffusion models, e.g., it shows benefits over LoRA on Commonsense reasoning in Table R.3 of the pdf as well as on GLUE benchmarks in Table 3 of the paper. On the commonsense reasoning in Table R.3 for instance, FouRA-LLama3(8B) model achieves and average of 85.3% accuracy, compared to the 82.9% achieved by the LoRA-LLama3(8B) model.

Minor: We highly appreciate your meticulous review of our work and have corrected these mistakes.

Question on L888: As discussed in Appendix C, we adopt an entropy-based gating approach to train the soft gating module as in prior work [7] (see global response for references). The numbers in question are derived from their code [7] and are consistent across all datasets/models. We use them for all experiments with adaptive gating. They act as temperature terms to scale the sigmoid function. Our analysis shows that the model isn't sensitive to these terms as we threshold the sigmoid output.

We thank reviewer AmEw for their constructive feedback and acknowledgement of our motivation/novelty.

Inference time: Thanks for suggesting the inference time analysis. As requested, we show the inference latency along with other compute analysis in Table R.1 of the provided pdf file. We observe that FouRA with dynamic frozen masking has same inference time (14.9 steps/sec) as baseline LoRA (after merging adapter into weights), while achieving better visual generations i.e., HPS score of 30.3 (FouRA with dynamic frozen masking) vs 27.7 (LoRA). While FouRA with inference adaptive rank selection incurs more inference latency (11.1 steps/sec), it does achieve best visual quality i.e., HPS score of 30.6. We will include this trade-off analysis in the paper.

Ablation on individual components of FouRA: Thank you for bringing this up. As suggested, we show individual contributions from FouRA modules in Table R.2 of the pdf. We fix rank=64 and $\alpha$=0.8, and provide results on the paintings validation set. As evident from LPIPS-Diversity and HPS scores, the adaptive mask selection strategy performs better than the dynamic fixed mask selection strategy. For the case without frequency transform, Inference-Adaptive masking improves the HPS score from 28.2 to 28.7. When accompanied with Frequency transform, the HPS increases from 30.3 for frozen dynamic masking to 30.6 for inference-adaptive masking. These improvements are similar to those shown on the blue-fire validation set in Appendix E.1. We will add the ablation study in Table R.2 with the full breakdown in our main paper.

We appreciate reviewer R4Sn for their insightful feedback to help us improve our work.

Quantitative Results: Thank you for the suggestion. Based on your recommendation, we provide more quantitative analysis in the Rebuttal pdf. We have trained FouRA adapters over a LLaMA3-8B model and tested on eight publicly available commonsense reasoning tasks, following the split from [5] and implementation from [6] (see global response for references). Our method outperforms LoRA scores across all benchmarks in both rank=32 and rank=16 settings, summarized in Table R.3 of the rebuttal pdf.

Compute Analysis: We have conducted a more in-depth analysis of both training and inference time, along with the gpu memory in Table R.1 of the provided pdf. In summary, for each training epoch, FouRA needs 24.5s vs. 22.0s by LoRA, and GPU memory consumption of FouRA and LoRA are comparable. We have also provided measurements for inference time in the table. Additionally, we provide training complexity v/s performance curves for FouRA and LoRA in Figure R.1 of the rebuttal pdf. From these results, it is clear that FouRA can provide a higher operating point in the performance v/s computational tradeoff as compared to LoRA. We will include this analysis to our paper.

Minor: Thank you for the suggestion, we will enlarge the fonts in Fig. 3 of the main paper.

What causes data-copying? Thank you for bringing this up. We have provided experimental analysis to answer this question. Please refer to Figure R.2 of the pdf. We track the LPIPS-diversity as a measure of data-copying and HPS-v2 scores as a measure of adapter quality. We do notice lesser data copying artifacts in the initial phase of training. However, the adapter quality and strength are sub-par due to inadequate training (i.e. the style is not visible in the image). This is visible in HPS-v2 alignment scores. The images produced are similar to those from the base model, and hence lesser artifacts exist. As the training epochs increase, images start to represent the adapter style (represented by HPS scores). Once we reach this point, the number of data-copying artifacts increase significantly in LoRA, as tracked by the LPIPS-diversity. FouRA can achieve the adapter style while being able to produce a diverse range of images, as seen in Fig.1 of the main text. We also observe this trend when we visualize images from intermediate epochs. We will include these results in our appendix.