Improving LoRA in Privacy-preserving Federated Learning

We thank the reviewer for the time and support of our paper as well as the valuable suggestions. Please see our response below with respect to the specific comments.

Unclear how the approach performs under adversarial attacks, concept drift, and personalization.

The main objective of our paper is on the challenges when we consider the task of fine-tuning of LLMs, and impact of . Although these questions are important in the context of Fed Fine tuning LLMs. They are beyond the scope of this paper.

The paper only evaluates NLP tasks with text data. Unclear if the benefits of FFA-LoRA generalize to other data types like image, speech, etc.

It would be interesting to see the performance of such fine-tuning in the context of computer vision and speech. We running experiments on these tasks and will update the results here once they are ready. We note that fine-tuning LLMs with DP is computationally intensive, and we need more time to finish the experiments.

The theoretical analysis and intuitions provided are informal. No formal convergence or privacy proofs given.

Regarding the scaling factor $\alpha$ for FFA-LoRA and LoRA, we have updated our draft to include more formal statements, and provide an additional theorem to formally show that FFA-LoRA is in fact equivalent to LoRA with $\alpha \rightarrow \infty$.

Theorem 1 (Scaling Factor): For local updates with the same initial condition on $\mathbf{W}$, vanilla LoRA update with scaling factor $\alpha\_{LoRA}$ produces trajectory $\\{ W\_{\alpha\_{LoRA}}^k \\}\_{k\in [K]}$, and FFA-LoRA with scaling $\alpha\_{FFA}$ produces trajectory $\\{W\_{\alpha\_{FFA}}^k\\}\_{k\in [K]}$. Then we have $\lim_{\alpha_{LoRA} \rightarrow \infty} W_{\alpha_{LoRA}}^k = W_{\alpha_{FFA}}^k, \text{ for all } k, \alpha_{FFA}.$

Regarding the convergence study of our proposed method. We first provide the following statements for some additional context on the theoretical aspects behind PEFT on LLMS.

Theorem 2 (Smoothness conditions): Assume that the loss function given weight and dataset is denoted $F(\mathbf{W}, D)$. For a low-rank decomposition on model parameter $\mathbf{W}$ such that $\mathbf{W}(\mathbf{A}, \mathbf{B}) = \mathbf{W}\_0 + \mathbf{B} \mathbf{A}$ satisfying Equation (1). We have the following properties.

1. If $\mathbf{B}$ is trainable, $\mathbf{A}$ is fixed with $||\mathbf{A}|| \leq C$ and $F(\mathbf{W}, D)$ is Lipschitz smooth with respect to $\mathbf{W}$ with factor $L$. The loss function $F(\mathbf{W}(\mathbf{A}, \mathbf{B}))$ is Lipschitz smooth with respect to $\mathbf{B}$ with factor $LC^2$.
2. If both $\mathbf{A}$ and $\mathbf{B}$ are trainable and $F(\mathbf{W}, D)$ is Lipschitz smooth with respect to $\mathbf{W}$ with factor $L$, the loss function $F(\mathbf{W}(\mathbf{A}, \mathbf{B}))$ has no Lipschitz smoothness guarantees with respect to $$ \mathbf{A}, \mathbf{B} $$.

All the smoothness notions are defined with respect to the matrix Frobenius norm, which is denoted as $||\cdot||$.

These results show that if the objective function loss defined on full fine-tuning with $\mathbf{W}$ satisfies technical conditions such as Lipschitz smoothness, the same assumptions would also hold for FFA-LoRA, but not for Vanilla LoRA.

Suppose that the objective function $F$ satisfies the conditions presented in existing works to ensure convergence of full federated fine-tuning [1] or DP tuning [2] on $\mathbf{W}$, then we can get similar convergence results for FFA-LoRA using the theorem above.

Regarding differential privacy, we formally state our corollary for privacy guarantee. Our analysis below is based on Theorem 1 of [3] and the parallel composition and resistance to post-processing of DP.

Corollary 3 (Privacy Guarantee): Given Theorem 1 with moments accountant in [3], the parallel composition and resistance to post-processing of DP, the mechanism updating FFS-LoRA with locally ran DP-SGD and FedAvg can satisfy $(\epsilon, \delta)$-DP given $\forall i, q=\frac{|B_i|}{|N_i|}$, the number of total local updates $T$ of each client and $\sigma = O(\frac{q\sqrt{T\log(1/\delta)}}{\epsilon})$.

(The exact $\sigma$ is computed by the Pytorch's Opacus package [4] numerically given $q, T, \epsilon, \delta$).

We refer to our "Response to All" for detailed proof.

---

[1] Lian, Xiangru, Yijun Huang, Yuncheng Li, and Ji Liu. "Asynchronous parallel stochastic gradient for nonconvex optimization." Advances in neural information processing systems 28 (2015).

[2] Wang, Di, Changyou Chen, and Jinhui Xu. "Differentially private empirical risk minimization with non-convex loss functions." In International Conference on Machine Learning, 2019.

[3] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. 2016 ACM SIGSAC conference on computer and communications security

[4] https://github.com/pytorch/opacus

We thank the reviewer for their kind acknowledgment of our response and that our response addresses most of their concerns.

We here provide an additional dataset on the computer vision task. We use the pre-trained vision transformer (https://huggingface.co/google/vit-base-patch16-224-in21k), and consider the task of fine-tuning on the Food-101 dataset (as shown on https://huggingface.co/datasets/food101) for image classification.

For context, we provide performance reported on huggingface as baseline. A centralized, fine-tuned model has an accuracy of 0.8539.

We first report the results in our centralized experimental setting in the table below.In this case there is no significant performance discrepancy between the two methods, implying that FFA-LoRA and vanilla has similar performance without consideration of DP and FL. This also aligns with Theorem 1 in Author's response.

In terms of the federated case, we first report the iid setting. In this setting, under similar hyper-parameters, we provide the learning curve of both FFA-LoRA and LoRA (https://imgur.com/5xnXWSu). as well as a last k-iterated (k=50) averaged accuracy.

It can be seen that compared to LoRA, FFA-LoRA has both (a) better convergence and (b) less fluctuations in training. The findings align with our findings in language-related tasks, showing that the properties of LoRA being discussed in our paper are not limited to language tasks only.

The non-iid setting as well as the DP setting are more time-consuming, and we are still running more experiments. Due to the time limit, we will add them to the final version of our paper if the paper is accepted.

What's the variance in the experiment?

In many of the experiments (especially for RoBERTa), we find that the variance is typically low, and the performance is consistent, therefore we mainly consider the performance. We conducted additional experiments (20 reruns with different seeds) the QNLI experiment, under the same setting as Table 1, and provide the mean and variance of the results here. We note that some rare cases where the algorithm failed to converge has been removed from the statistics.

[1] Yu, Da, Huishuai Zhang, Wei Chen, Jian Yin, and Tie-Yan Liu. Large scale private learning via low-rank reparametrization. In International Conference on Machine Learning,PMLR, 2021.

The benefit of FFA-LoRA on differential privacy (DP) is not very well backed by empirical evaluation. The performance gap between the vanilla LoRA and the proposed FFA-LoRA remains the same across various privacy budgets , including . Such an empirical result suggests that the impact of DP noise is the same on both the vanilla LoRA and the proposed FFA-LoRA.

We note that the exact performance of the algorithms are In Table 1, although there are counter-examples, the performance gap between FFA and vanilla is still evident, with the largest gap occurring $\epsilon = 1$ when taking average over the 4 tasks. Therefore we can say that FFA is more suitable for higher noise. The individual performance of one task, $SST-2$ in this case, depends on many factors such as the difficulty of the task.

We further point out that our claim on noise amplification is more apparent in Table 4. Where for rank as low as 2 or 4, Lora can not converge to any meaningful results when $\epsilon = 1$.

Additionally, when noise is small (no significant noise amplifying), the impact of noise is not as big, as shown in Figure 1 of our paper. The simulation in Figure 1 also aligns with our observation in Table 1 and 4.

Additionally, existing literature [1] (Figure 4) has shown that the relationship between performance gap and $\epsilon$ is not always significant, and varies between tasks. This observed behavior could be due to robustness of DPSGD-style algorithms. Currently, the dynamics of DPSGD optimization in deep neural networks is still not fully understood.

I do not see why the proposed FFA-LoRA is free from tuning the hyper-parameter . In Section 4, the authors claim that "FFA-LoRA does not rely on , and is equivalent to LoRA with ". Such a claim, in fact, suggests that the is fixed in FFA-LoRA. Then, in Theorem 1, the theoretical result suggests that tuning is equivalent to tuning the learning rate . I'm not able to fully follow the discussion here.

We would like to first reiterate that for the proposed FFA-LoRA algorithm, as confirmed by Theorem 1 in our paper, $\alpha_{FFA}$ can be seen as fixed. However, this does not make FFA-LoRA equivalent to vanilla LoRA with some given $\alpha_{LoRA}$. As we have stated, FFA-LoRA with any $\alpha$ in fact has the same dynamic as LoRA with $\alpha_{LoRA} \rightarrow \infty$, we provide the details below:

In order to reduce confusion and provide additional theoretical robustness to our paper, we provide an additional theorem as a more formal statement on the relationship between FFA-LoRA and vanilla LoRA. We refer to our "Response to All" for a more detailed explanation.

Is the FFA-LoRA approach more sensitive to random initialization? Suppose a bad initialization sets is the model still trainable?

This is a really good question. In our experiments, all the $A_0$ matrices were randomly initialized under random Kaiming distribution ( a default initialization method is deep learning and is consistent with previous works in LoRA) and according to Theorem 1, the scaling on the $A_0$ is not important for FFA-LoRA. And we briefly discuss the impact of other types of initialization below.

We know that for a zero-initialized A matrix, neither LoRA nor FFA-LoRA are able to train any meaningful results. However, suppose that we have A to be full rank (which is also satisfied for any random initialization in general), there are a number of different initializations that we could utilize, such as orthogonal random initialization and using the top $r$ singular vectors of $W_0$ as matrix A.

We provide some initial results here (without DP), it can be seen from the plot that matrix $A$ with orthogonal initialization seems to perform slightly better than the existing approach. However, the performance gap is not significant enough for a definitive answer.

However, although this is a great question with many interesting potential directions, we mainly consider the result of freezing A after initialization, and the choices of initialization is beyond our initial scope of this work. Although all these methods use SGD as the back-bone optimization algorithm, the key ideas and procedures for training are very different from DP. Thus, each of these challenges may deserve an independent project to be investigated. We will mentions these problems as future directions in our revision.

Thanks for replying.

1. There still lacks evidence to support the claim "FFA is more suitable for higher noise". I expected the performance gap between FFA-LoRA and vanilla LoRA to increase as the noise magnitude grows, but I did not see such behavior in Tables 1 and 4. Could the authors provide additional numbers or explanations?

2. Could the authors summarize the goal of the theoretical analysis before diving into the detailed technical claims on $\alpha$ and the full rank property? Also, is there any definition of the equivalent relationship between LoRA and its variance?

We thank the reviewer for their additional comments.

There still lacks evidence to support the claim "FFA is more suitable for higher noise". I expected the performance gap between FFA-LoRA and vanilla LoRA to increase as the noise magnitude grows, but I did not see such behavior in Tables 1 and 4. Could the authors provide additional numbers or explanations?

As we have stated in the paper, the performance gap between FFA-LoRA and LoRA widens when we are faced with a smaller privacy budget, which was verified in Table 1 and 4. To better illustrate this observation, we provide the following figure on the relationship between injected noise and averaged algorithm accuracy across the 4 tasks. As shown in the figure (https://imgur.com/a/7FmlPdQ), as more noise is added, the performance gap widens.

Could the authors summarize the goal of the theoretical analysis before diving into the detailed technical claims on and the full rank property?

We plan to update the draft such that the theoretical analysis is coherent with our existing statements in the paper. We provide a framework here on how the introduced theorems benefit our arguments made in the paper.

• Our goal is to show that
  • $\alpha$ does not matter for FFA-LoRA, which is one less hyper parameter to worry about. This claim is backed by the theorem in our paper.
  • For LoRA, generally big $\alpha$ is good for performance but bad for stability. FFA-LoRA is equivalent to $\alpha = \infty$ for LoRA without being unstable. The equivalence is backed by theorem 1 in our response we provide in our response.
• Many reviewers have mentioned the need for theoretical analysis. We state that the existing literature on the convergence of federated learning can apply to FFA-LoRA, but not LoRA. This is backed by our Theorem on smoothness.
• The full rank property of $A$ was mentioned in our response to comment on how to construct different methods of initialization. By definition of linear algebra, a full rank $A$ is more expressive that a rank-deficient one, which is why we consider full rank initialization. It is by no means the only approach.

Also, is there any definition of the equivalent relationship between LoRA and its variance?

The equivalence between LoRA and FFA-LoRA happens when $\alpha \rightarrow \infty$ for LoRA, we provide the formal definition in our theorem on equivalence. Or, informally speaking, we have

$\lim_{\alpha_{LoRA} \rightarrow \infty} W_{\alpha_{LoRA}}^k = W_{\alpha_{FFA}}^k,$

where the matrices $W_{\alpha_{LoRA}}^k, W_{\alpha_{FFA}}^k$ are produced by $k$ local iterations of LoRA and FFA-LoRA. We want to show that as $\alpha$ increases, LoRA's update on A and B gets closer to the updates made by FFA-LoRA. Which is good since in many cases we want LoRA to have big $\alpha$.

Thanks for replying. Two major issues remain, so I have decided to keep my rating.

1. There are several noticeable counter-examples in Tables 1&4, and the new figure does not address them. For example, we reach the largest gap between LoRA and FFA-LoRA with $\epsilon=1$ on the QQP and QNLI datasets instead of $\epsilon \in \\{3, 6\\}$. LoRA also outperforms FFA-LoRA by a significant margin on the SST-2 datasets with $\epsilon=1$. These results might be outliers, but we need more investigation before concluding that FFA-LoRA is more robust to DP noise.

2. The equivalence relationship between FFA-LoRA and LoRA never holds in practice and the benefit of large $\alpha$ is questionable. Theorem 1, in the revision, suggests that a necessary condition for such an equivalence relationship is $\alpha_{LoRA} = \infty$. In practice, $\alpha_{LoRA} \neq \infty$. Indeed, in the provided reference (Kuang et al., 2023), the maximum $\alpha=50.0$ or $128$ if I read the tables in the appendix correctly. Also, tuning $\alpha$ does not seem to impact the accuracy much, as Figure 5(a) in Kuang et al., 2023 shows. Experiments with $\alpha = 0.5$ and $\alpha=50.0$ have almost the same average accuracy.

We are grateful for the reviewer's careful review and constructive comments. Based on your comments, we would like to make the following clarification.

The motivation is straightforward and intuitive, without theoretical insights.

We appreciate the reviewer's comment on our motivation being straightforward and intuitive. We provide some additional theoretical results regarding the relationship between FFA-LoRA and LoRA, a theorem regarding the convergence properties of FFA-LoRA, and lastly a guarantee on differential privacy. All of which will be added to our revised draft. We refer to our "Response to All" for a more detailed discussion.

Why rank 16 for MNLI is worse than rank 8 in Table 2?

We report all our experiments truthfully. It has been shown that even for vanilla LoRA, there is an optimal rank for a specific fine-tuning task, and the performance of LoRA could decrease if rank is too large, the exact optimal rank is task-dependent [1]. This observation can be potentially explained by a similar reason.

We thank the reviewer for the careful observation, and will add statements to address this in the revised version of the paper.

Another intuitive variant is to alternative optimize the two LoRA weights. How would this perform compare with the proposed method?

This is a great question. We have conducted experiments regarding alternate updates on the weights. The results show slower convergence and no significant performance gains compared to vanilla LoRA. Empirically, this means that alternating update is hard to tune: with a smaller learning rate similar to LoRA, it converges slower; with a larger learning rate similar to FFA-LoRA, it is not robust and often unable to converge. For the case of non-iid agents, this algorithm is unable to converge under 5 different random seeds and multiple learning rates $\eta \in \\{0.05, 0.1, 0.2\\}$. (For this experiment, the best LR for LoRA and FFA-LoRA is $\eta=0.05$ and $\eta = 0.5$, respectively.) This can be confirmed by both accuracy scores as well as observations of the learning curve during training. Additionally, with the consideration of DP, the number of gradient calculations is doubled for alternative update methods, which could result in a higher noise added to ensure privacy. In this case, the drawbacks overshadow the benefits. We will provide a brief discussion on this potential solution in our revision.

We also include one result from the alternate update case here demonstrating its performance in the Table below.

Why non-iid performance is similar to iid performance in Table 3?

As we stated in Section 5.1, for the heterogeneous setting, we split data based on their labels. Due to the heterogeneity presented to these methods is not strong enough, there is not a strong separation between iid and non-iid experiments for some tasks. Therefore, a performance gap exists between FFA-LoRA and vanilla LoRA, but it is not very significant. To demonstrate the impact of heterogeneity, we included an additional experiment with even stronger heterogeneity.

In the previous MNLI results we ran in the original paper, the non-iid case was formulated using a data label split of $[0.6, 0.2, 0.2], [0.2, 0.6, 0.2], [0.2, 0.2, 0.6]$, which we denote as mild non-iid here. We consider a stronger hetergeneity here, with data label split ratio of $[0.9, 0.05, 0.05], [0.05, 0.9, 0.05], [0.05, 0.05, 0.9]$. The performance degradation is clearly shown, with FFA-LoRA significantly better than LoRA.

---

[1] Zhang, Qingru, Chen, Minshuo, Bukharin, Alexander, He, Pengcheng, Cheng, Yu, Chen, Weizhu, Zhao, Tuo. Adaptive budget allocation for parameter-efficient fine-tuning. The Eleventh International Conference on Learning Representation

FedBert seems to mainly focus on pre-training instead of fine-tuning.

The reviewer is correct. In our paper, citation of FedBERT had been referred to as a method for fine-tuning, it is instead intended for pre-training, we will change our statements accordingly.

Please cite “Communication-Efficient Learning of Deep Networks from Decentralized Data” for federated learning and the FedAvg algorithm.

We will cite this paper in our revision, and we thank the reviewer for the useful suggestion.

[1] Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. arXiv preprint arXiv:2110.05679, 2021

[2] Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie-Yan Liu. Large scale private learning via low-rank reparametrization. In International Conference on Machine Learning, 2021

[3] Xuechen Li, Daogao Liu, Tatsunori B Hashimoto, Huseyin A Inan, Janardhan Kulkarni, Yin-Tat Lee, and Abhradeep Guha Thakurta. When does differentially private learning not suffer in high dimensions? Advances in Neural Information Processing Systems, 2022.

[4] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security

We thank the reviewer for their time and effort reviewing our paper. Please see our response below with respect to the specific comments.

However, the federated learning setting in experiments seems a bit unconventional with a very small number of clients (only 3 clients). This might be categorized as a cross-silo setting, but it would be good to clearly discuss the targeted application

Yes, our focus is indeed the cross-silo setting (similar to existing work in FL LLM, cite some). The major reason is the immense computational burden of LLM training. Also, the difficulties we discussed in this paper are well suited in the context of cross-silo setting, since cross-silo setting is interested in problems such as data-heterogeneity, communication efficiency and privacy, which we were able to address. In our revision, we will clarify with the exact setting.

While I appreciate the motivation of analyzing LoRA in section 3, none of the explanations seems to be particularly convincing.

We thank the reviewer for acknowledging the motivations of our analysis.

Firstly, to further consolidate our analysis, we have added an additional theorems as well as a corollary. We refer to our "Response to All" for these theoretical results. We hope the reviewer find our statements more convincing with the addition of the theoretical results.

Secondly, the disparity we pointed out in the paper is more significant when the weight difference is not-negligible, i.e. $A_1\not\approx A_2$. In terms of averaging gradients from examples, this problem is less of a concern since the learning rate $\eta$ is generally small, and the disparity is not severe. However, for federated learning with averaging models after $K>1$ local update steps, the disparity becomes more problematic.

fail to understand why $\alpha$ becomes an issue for LoRA in Discordance (3) as it is only a scalar and might potentially be absorbed in learning rate. As shown in table 5, tuning the learning rate helps.

Yes, as we have stated in Section 5.2 of the paper, by jointly tuning learning rate $\eta$ and scaling factor $\alpha$, the fine-tuning algorithm for LoRA is able to converge. However, in vanilla LoRA, as reported by previous works, tuning learning rate $\eta$ and scaling factor $\alpha$ have very different effects on the dynamic of the training process as well as the performance, and the exact relationship between them is unknown. In a way, changing both the learning rate and scaling factor is equivalent to tuning the initialization of the random matrices $A$ in the network. There are existing works showing that for many tasks (FS-LLM), a bigger scaling factor (as high as 256) is able to converge to higher accuracy, however the algorithm becomes more unstable as $\alpha$ increases, and a re-exploration on the optimal learning rate is needed for a different $\alpha$.

Additionally, in the context of Differential Privacy, a higher scaling factor for vanilla LoRA would also change the clipping and noises added to the trainable parameters, which introduces additional difficulties.

On the other hand, for FFA-LoRA, the effect of $\eta$ and $\alpha$ is the same, which means we can set-and-forget one and focus on the other, meaning that we have one less hyper-parameter to tune in experiments.

Minor: The empirical results on local DP seem to be very good with very little accuracy drop compared to non-DP results. It is possible that DP fine-tuning is not a particularly hard task compared to training from scratch, but could the authors share more details about the privacy accounting and important privacy parameters?

We would like to clarify that no privacy was considered in pre-training, and we only consider the notion of differential privacy in finetuning.

Additionally, the difficulty in DP fine-tuning is an entirely different question. In general, the difficulty of DP training is associated with the dimension of the problem, with higher dimensional data requiring a lower signal-to-noise ratio. Fine-tuning LLMs with privacy was considered to be very difficult due to the high-dimensional trainable parameters. However, as pointed out by some recent works on DP fine-tuning, it has been shown that DP fine-tuning empirically exhibits much better performance [1, 2]. This was explained because of the lower intrinsic dimensions within LLMs [3], but the definitive reason for this behavior is largely an open problem.

We have added more theory as well as calculations behind our approach to ensure differential privacy in our "Response to All" (Corollary 3), which we reiterate below based on Theorem 1 of [4] and the parallel composition and resistance to post-processing of DP.

Corollary 3 (Privacy Guarantee): Given Theorem 1 with moments accountant in [1], the parallel composition and resistance to post-processing of DP, the mechanism updating FFS-LoRA with locally ran DP-SGD and FedAvg can satisfy $(\epsilon, \delta)$-DP given $\forall i, q=\frac{|B_i|}{|N_i|}$, the number of total local updates $T$ of each client and $\sigma = O(\frac{q\sqrt{T\log(1/\delta)}}{\epsilon})$.

(The exact $\sigma$ is computed by the Pytorch's Opacus package [2] numerically given $q, T, \epsilon, \delta$).

Proof: Firstly, we consider the local datasets $\{D_i\}_{i\in [n]}$for the FL network to be disjoint chunks of the global dataset. The DP-SGD with FedAvg used in our paper to train LoRA or FFA-LoRA can be considered as

• (A) locally updating trainable parameters with DP-SGD,
• (B) averaging the trainable parameters from clients on the server, and
• (C) repeating the above two steps for some iterations.

The privacy loss of (A) can be composed by moment accountants used in [1]. The privacy loss of all clients performing local updates can be composed by the parallel composition property of DP. The averaging on the server in (B) is a post-processing operation that does not introduce privacy loss. Privacy loss of multiple FL rounds of (C) can again be composed with moment accountants used in [1]. Eventually, we can convert the moment accountants to $(\epsilon, \delta)$-DP as Theorem 1 in [1].

[1] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security

[2] https://github.com/pytorch/opacus

Theorem 2 (Smoothness conditions): Assume that the loss function given weight and dataset is denoted $F(\mathbf{W}, D)$. For a low-rank decomposition on model parameter $\mathbf{W}$ such that $\mathbf{W}(\mathbf{A}, \mathbf{B}) = \mathbf{W}\_0 + \mathbf{B} \mathbf{A}$ satisfying Equation (1). We have the following properties.

1. If $\mathbf{B}$ is trainable, $\mathbf{A}$ is fixed with $||\mathbf{A}|| \leq C$ and $F(\mathbf{W}, D)$ is Lipschitz smooth with respect to $\mathbf{W}$ with factor $L$. The loss function $F(\mathbf{W}(\mathbf{A}, \mathbf{B}))$ is Lipschitz smooth with respect to $\mathbf{B}$ with factor $LC^2$.
2. If both $\mathbf{A}$ and $\mathbf{B}$ are trainable and $F(\mathbf{W}, D)$ is Lipschitz smooth with respect to $\mathbf{W}$ with factor $L$, the loss function $F(\mathbf{W}(\mathbf{A}, \mathbf{B}))$ has no Lipschitz smoothness guarantees with respect to $$ \mathbf{A}, \mathbf{B} $$.

All the smoothness notions are defined with respect to the matrix Frobenius norm, which is denoted as $||\cdot||$.

Proof: First we show that, given $\mathbf{W}(\mathbf{A}, \mathbf{B}) = \mathbf{W}_0 + \mathbf{B} \mathbf{A}$, and the gradient on $\mathbf{W}$ is denoted as $\nabla_W F$, then we can write the gradients on matrix $\mathbf{B}$ as $\nabla_B F = \nabla_W F \mathbf{A}^T$, since

$

\langle \mathbf{B}_1 - \mathbf{B}_2, \nabla_B F\rangle & = \langle \mathbf{W}(\mathbf{A}, \mathbf{B}_1) - \mathbf{W}(\mathbf{A}, \mathbf{B}_2), \nabla_W F\rangle\\ &= \langle \mathbf{B}_1 \mathbf{A} - \mathbf{B}_2 \mathbf{A}, \nabla_W F\rangle\\ &= \langle \mathbf{B}_1 - \mathbf{B}_2 , \nabla_W F \mathbf{A}^T\rangle

$

Similarly, we have $\nabla_A F = \mathbf{B}^T \nabla_W F$. Using the gradients on $\mathbf{A}$ and $\mathbf{B}$, we provide the proof for all the properties.

1. For property 1, we know that for any given $\mathbf{B}_1, \mathbf{B}_2$,

$

& ||\nabla_{B} F(\mathbf{W}(\mathbf{A}, \mathbf{B}_1)) - \nabla_{B} F(\mathbf{W}(\mathbf{A}, \mathbf{B}_2))|| \\ =& ||\nabla_{W} F(\mathbf{W}(\mathbf{A}, \mathbf{B}_1)) \mathbf{A}^T - \nabla_{W} F(\mathbf{W}(\mathbf{A}, \mathbf{B}_2)) \mathbf{A}^T|| \\ \leq& L ||\mathbf{W}( \mathbf{A}, \mathbf{B}_1) - \mathbf{W}(\mathbf{A}, \mathbf{B}_2) || \times ||\mathbf{A}||\\ \leq& L || \mathbf{B}_1 - \mathbf{B}_2 || ||\mathbf{A}||^2 \\ \leq& L C^2||\mathbf{B}_1 - \mathbf{B}_2 ||

$

2. For the second property, for the ease of notation, we introduce the stacked variable $\mathbf{x} := [\mathbf{A}, \mathbf{B}]$. We construct a counter-example such that the function is not Lipschitz smooth with respect to $\mathbf{x}$.

We consider $\mathbf{W}, \mathbf{A}, \mathbf{B} \in \mathbb{R}^{d\times d}, F(\mathbf{W}) = \frac{1}{2}||\mathbf{W}||^2$ with $\mathbf{W}\_0 = 0$. Then we consider a sequence $\\{\mathbf{x}\_k\\}\_{k \in \mathbb{N}}$ such that $\mathbf{x}\_k = [\mathbf{A}\_k, \mathbf{B}\_k] = [k \mathbf{I}\_d, k \mathbf{I}\_d]$, then

$

& \lim_{k \rightarrow \infty} \frac{||\nabla_x \mathbf{W}(\mathbf{A}_k, \mathbf{B}_k) - \nabla_x \mathbf{W}(\mathbf{A}_0, \mathbf{B}_0)||}{||\mathbf{x_k} - \mathbf{x_0}||} \\ =& \lim_{k\rightarrow \infty} \frac{||\nabla_A \mathbf{W}(\mathbf{A}_k, \mathbf{B}_k) - \nabla_A \mathbf{W}(\mathbf{A}_0, \mathbf{B}_0)|| + ||\nabla_B \mathbf{W}(\mathbf{A}_k, \mathbf{B}_k) - \nabla_B \mathbf{W}(\mathbf{A}_0, \mathbf{B}_0)||}{||\mathbf{A}_k - \mathbf{A}_0|| + ||\mathbf{B}_k - \mathbf{B}_0||}\\ =& \lim_{k\rightarrow \infty} \frac{||k^3 \mathbf{I}_d|| + |k^3 \mathbf{I}_d||}{||k \mathbf{I}_d|| + ||k \mathbf{I}_d||}\\ =& \infty

$

For this example, we can see that although $F(\mathbf{W})$ is 1-Lipschitz smooth, the function is not smooth with respect to $\mathbf{x}$.

Theorem 1 (Scaling Factor): For local updates with the same initial condition on $\mathbf{W}$, vanilla LoRA update with scaling factor $\alpha\_{LoRA}$ produces trajectory over $k$ local iterations $\\{ W\_{\alpha\_{LoRA}}^k \\}\_{k\in [K]}$, and FFA-LoRA with scaling $\alpha\_{FFA}$ produces trajectory $\\{W\_{\alpha\_{FFA}}^k\\}\_{k\in [K]}$. Then we have $\lim_{\alpha_{LoRA} \rightarrow \infty} W_{\alpha_{LoRA}}^k = W_{\alpha_{FFA}}^k, \text{ for all } k, \alpha_{FFA}.$

Proof: The theorem starts with initial condition on $\mathbf{W}$, since $\mathbf{W} = \mathbf{W}\_0 + \alpha \mathbf{B}\_\alpha \mathbf{A}\_\alpha$ and that the initialization of $\mathbf{A}$ is non-zero, this condition implies that $\mathbf{A}\_\alpha = \mathbf{A}\_1 = \mathbf{A}$, and $\mathbf{B}\_\alpha = \frac{1}{\alpha}\mathbf{B}\_1$. Now we compare the update of the two algorithms given the same initial conditions.

From Theorem 1 we know that for FFA-LoRA, different $\alpha_{FFA}$ does not affect its dynamics, without loss of generality, we consider the case where $\alpha_{FFA} = 1$.

The FFA-LoRA update is as follows, the only update is on $B$:

$

    W^{k+1}\_{FFA} = W\_0 + 1 \times B^{k+1}A^k = W\_0 + (B^k - \eta \nabla B^k)A^k = W^k  - \eta \nabla B^k A^k

$

The rest of the proof is given by induction, as long as the limit holds for the $k+1$-th local iteration given that the $k$-th iteration holds. Without the loss of generality, we first consider when $\alpha_{LoRA} = 1$, then for iteration $k$, we denote the learning rate as $\eta_1$, denote the matrices and their gradient as $\mathbf{A}^k_{1}, \mathbf{B}^k_1$ and $\nabla \mathbf{A}^k_{1}, \nabla \mathbf{B}^k_{1}$, respectively. And by definition, we have the update that

$

    \mathbf{A}^{k+1}\_1 &\leftarrow \mathbf{A}^k\_{1}-\eta\_1 \nabla \mathbf{A}^k\_{1}\\\\
    \mathbf{B}^{k+1}\_{1} &\leftarrow \mathbf{B}^k\_{1}-\eta\_1 \nabla \mathbf{B}^k\_{1}

$

And the update of the original weight matrix $W$ becomes

$

    W^{k+1}\_{1} = W\_0 + \Delta W^{k+1}\_{1} &=  W_0 + (\mathbf{B}^k\_{1}-\eta_1 \nabla \mathbf{B}^k\_{1})(\mathbf{A}^k\_{1}-\eta\_1 \nabla \mathbf{A}^k\_{1})\\\\
    &= W^k\_1 - \eta\_1 \left(\nabla \mathbf{B}^k\_{1}\mathbf{A}^k\_{1} +  \mathbf{B}^k\_{1}\nabla\mathbf{A}^k\_{1} \right) + \eta^2 \nabla\mathbf{B}^k\_{1}\nabla\mathbf{A}^k\_{1}

$

Since LoRA do not satisfy the conditions provided in Theorem 1, changing $\alpha\_{LoRA}$ will affect its updates. When we choose a different $\alpha\_{LoRA} = \alpha$ and corresponding $\eta\_{\alpha} = \frac{\eta\_1}{\alpha^2}$, we can write the update of LoRA as

$

    \mathbf{A}^{k+1}\_\alpha &\leftarrow \mathbf{A}^k\_{\alpha}-\eta\_\alpha \nabla \mathbf{A}^k\_{\alpha} 
    = \mathbf{A}^k\_\alpha-\frac{\eta\_1}{\alpha}\nabla \mathbf{A}^k\_\alpha
    =\mathbf{A}^k\_1-\frac{\eta\_1}{\alpha}\nabla \mathbf{A}^k\_1\\\\
    \mathbf{B}^{k+1}\_\alpha &\leftarrow \mathbf{B}^k\_\alpha-\eta\_\alpha \nabla \mathbf{B}^k\_\alpha
    = \mathbf{B}^k\_\alpha-\frac{\eta\_1}{\alpha}\nabla \mathbf{B}^k\_\alpha
    = \frac{1}{\alpha}\mathbf{B}^k\_1-\frac{\eta\_1}{\alpha}\nabla \mathbf{B}^k\_1\\\\
    W^{k+1}\_\alpha &=  W\_0 + \alpha 
    (\frac{1}{\alpha}\mathbf{B}^k\_1-\frac{\eta\_1}{\alpha}\nabla \mathbf{B}^k\_1)
    (\mathbf{A}^k\_1-\frac{\eta\_1}{\alpha}\nabla \mathbf{A}^k\_1)\\\\
    &=  W_0 + \mathbf{B}^k\_1\mathbf{A}^k\_1 - \eta\_1 \nabla \mathbf{B}^k\_1 \mathbf{A}^k\_1 - \frac{\eta\_1}{\alpha} \mathbf{B}^k\_1 \nabla \mathbf{A}^k\_1 - \frac{\eta\_1^2}{\alpha} \nabla \mathbf{B}^k\_1 \nabla \mathbf{A}^k\_1

$

Therefore we have

$

    \lim\_{\alpha\_{LoRA}\rightarrow \infty} W\_{\alpha\_{LoRA}}^{k+1} &= 
    \lim\_{\alpha \rightarrow \infty} W\_0 + \mathbf{B}^k\_1\mathbf{A}^k\_1 - \eta_1 \nabla \mathbf{B}^k\_1 \mathbf{A}^k\_1 - \frac{\eta\_1}{\alpha} \mathbf{B}^k\_1 \nabla \mathbf{A}^k\_1 - \frac{\eta\_1^2}{\alpha} \nabla \mathbf{B}^k\_1 \nabla \mathbf{A}^k\_1\\\\
    &= W^k\_1 - \eta\_1 \nabla \mathbf{B}^k\_1 \mathbf{A}^k\_1\\\\
    &= W^{k+1}\_{FFA}

$

Which completes our proof.