LoRA Dropout as a Sparsity Regularizer for Overfitting Reduction

Hi authors, I wanted to kindly follow up and inquire if the rebuttal has been completed. I look forward to hearing back from you at your earliest convenience.

W4: theoretical contribution of the paper

We’d like to claim that there are intrinsic differences between our theory and [1] in multiple aspects.

1. The problem formulation in [1] does not actually fit LoRA tuning. [1] formulates LoRA tuning as a so-called “sparse” tuning algorithm. However, in practice, LoRA updates all linear weights with a low-rank dense residual module, which is actually NOT the case of [1]. The sparsity condition in [1] only holds when part of the linear matrices are tuned – yet still in a very rigid “sparsity” pattern, which is also not the mainstream practice currently.
2. Applying dropout mechanism to LoRA makes it a sparse tuning method, as we mentioned at the beginning of Section 2. However, the problem formulation in [1] is still NOT the case, because dropout does NOT have a fixed sparsity pattern as [1] proposed. In Eq.2-4 of [1], the sparsity mask M is fixed as long as the algorithm is determined, while dropout applies stochastic masks and does NOT fit this problem formulation. Hence, we refine the problem formulation with the expectation regularization term as shown in Eq.1-2.
3. As the problem formulation changes, the proofs are also produced in a different way due to the expectation term. Though proofs are still based on the PHS error bound theorem, we relax the conditions of the inequality for better generalization. First of all, we do not assume a sufficiently large sample size n as [1] did (See the condition of the inequality of $f_s(u)-f_s(v)$ in Lemma 1 [1], which actually may not lead to overfitting when data is sufficient). Moreover, taking suggestions from other reviewers, we no longer assume the Hessian matrix to be positive-semidefinite but instead constrain the regularization strength to be reasonably large, which is consistent with the practice of conducting hard dropout masking instead of soft regularization. To summarize, we believe that our theoretical results have different insights, more proper problem formulation, and more general applicable conditions than [1], which is of great significance to be presented in our paper.

[1] On the effectiveness of parameter-efficient fine-tuning, AAAI 2023.

W5: demanding more training time

We must admit that extra time overhead is a big limitation of our work, as we mentioned in the “Conclusions and Limitations” section. The additional training time makes it possible for our model to achieve better performance, and we are working to reduce the time overhead and achieve a better balance between performance and efficiency.

Q1: LoRA models can achieve good performances with a small rank

Yes, sometimes the LoRA-based models do perform well with a small rank, but it does not indicate that it has achieved optimal performances. Smaller ranks have a lower risk of overfitting, while higher ranks have more expressive power. Therefore, it is tricky to pick a proper rank. Hopefully, with the proposed LoRA Dropout method, the LoRA model could maintain a relatively high rank (high expressive power) and meanwhile have a low risk of overfitting.

Q2: The setting of rank and LoRA Dropout

We use rank=8 in the NLU tasks. And for the dropout rate, empirically we find values around 0.5 will all lead to a fine result for most cases, and the dropout rate is set to 0.5 in all our experiments. Furthermore, we do not use the original dropout in LoRA when LoRA Dropout is applied, and we have compared the performances between LoRA Dropout and original dropout.

Q3: Results of masking rows in dropout

Here we provide the results on NLU tasks that only rows were masked.

From the results, we could find that only masking rows does not lead to obvious improvements in performance. The reason for better performances of LoRA Dropout is that by simultaneously masking random columns and rows, LoRA Dropout could provide much more diverse sparsity patterns across the parameter space than only masking columns or rows.

As shown in Eq.(3) from Section 2.1, effectively applying the dropout mechanism in LoRA training requires sampling sufficiently diverse sparsity patterns (d from the Bernoulli distribution) for better estimation of the expectation term. An ideal way is generating 0-1 masks for each element in the delta parameters. However, this method is unrealistic since generating such sparsity masks will lead to great GPU memory overhead (i.e., as large as the original weight matrices).

In LoRA Dropout, by masking both rows and columns, generated sparsity patterns become more diverse and comprehensive than simply masking columns or rows, while GPU memory overhead is not significantly increased. This means our dropout strategy offers a better estimation of the sparsity regularizer in Eq.(3).

Q4: Why increasing the sparsity leads to better performance.

Sorry for the confusion. In the lines mention in Q4, we try to discuss the unnecessity of performing dropout on the rank dimensions, not the necessity of increasing the sparsity of parameters. Suppose we conduct dropout on the rank dimensions of matrices A and B, the production matrix $\Delta W$ will not be a sparse matrix, leading to an invalid sparsity regularizer.

Moreover, as we explained in Q3, the reason we conduct dropout on both columns and rows of $\Delta W$ is not to increase the sparsity but to get more diverse sparsity patterns. And we do agree that more sparsity does not always equate to better performance. As shown in Eq.(7) of our paper, only a proper sparsity ratio (dropout rate) will lead to a good balance between overfitting and underfitting.

We sincerely appreciate your comments and suggestions. We made every effort to address all the concerns and revised our paper following your suggestions. In the following, we quote your comments and then give our detailed response point-by-point.

W1: The relationship between "high-rank overfitting" and LoRA Dropout

The bold sentence is used to highlight the core problem of this paper——the overfitting issue. However, we do not mean by “overfitting only exists when the rank is high”, but instead, “it is hard to select a proper rank that could balance the expressiveness and overfitting risk”. Specifically, a relatively high rank of LoRA, which means more learnable parameters, indicates high expressive power, but as well makes the model prone to overfitting. Simply reducing the rank of LoRA (reducing number of learnable parameters) helps control overfitting but might lead to suboptimal performance. Thus, it is tricky to pick a proper rank that strikes a balance between overfitting and underfitting.

On the other hand, Dropout methods (including our LoRA Dropout method) can address the overfitting issue without cutting the budget of parameter, and our theoretical results also support that Dropout can alleviate overfitting by introducing sparsity regularization instead of directly reducing parameters. Therefore, with the LoRA Dropout method, the LoRA model could maintain a relatively high rank (high expressive power) and meanwhile achieve a low risk of overfitting.

W2: AdaLoRA and overfitting issue

In AdaLoRA, each LoRA matrix shares the same rank during initialization, and the ranks of LoRA matrices are automatically adjusted during training based on the importance scores. The AdaLoRA method shares the same insight of reducing the parameter budgets for controlling high-rank overfitting with the method that directly reduces LoRA ranks but in an automatic way. Here, we refer to the sentence from the original AdaLoRA paper, “Less importance ones (LoRA matrices) are pruned to have lower rank to prevent overfitting and save the computational budget.” However, the ability of AdaLoRA to alleviate overfitting is insufficient. The importance scores used in AdaLoRA are learned during training based on the training data, making the selected parameters less generalizable to unseen test data.

W3: motivation and logic of the introduction section

The logic of the introduction resembles the logic that you summarized in W3, and we are sorry if we didn’t make it clear in the original paper. We have updated the introduction in the revised paper, and we’d like to summarize the current logic of the introduction here:

LoRA models face the problem of overfitting. Simply reducing the rank of LoRA (reducing the number of learnable parameters) helps but might lead to suboptimal performance. AdaLoRA automatically prunes parameters to mitigate overfitting but is insufficient. Dropout can address the overfitting issue without cutting the budget of parameters, but the theoretical mechanism of its effectiveness on LoRA remains unclear. The motivation of our paper is to find out the theoretical evidence for Dropout to alleviate the overfitting issue on LoRA and to further propose a better dropout method on LoRA based on our theoretical framework.

Dear Reviewer uknh,

I hope this message finds you well.

I am writing to kindly remind you that the deadline for the discussion phase regarding our paper is drawing close. We greatly appreciate the time and effort you have already dedicated to reviewing our work, and we have made point-by-point responses to your constructive suggestions and questions. However, it appears that we have not yet received your response to our rebuttal, and we are wondering if we have addressed your concerns so far. Your insights and feedback are invaluable to us, and we believe they have significantly enhanced the quality of our manuscript.

After carefully addressing the comments and corresponding revisions, we kindly request your consideration of a more positive evaluation of our work if deemed appropriate. If you require any additional information or clarification from our end, please do not hesitate to reach out to us. We are more than willing to assist you in any way possible to ensure a thorough and constructive review process.

Thank you once again for your time and consideration. We are looking forward to hearing from you soon.

Warm Regards

We sincerely appreciate your detailed comments and constructive suggestions. We made every effort to address all the concerns. In the following, we quote your comments and then give our detailed response point-by-point.

W1/Q1: The reason of using DeBERTaV3 and LLaMA2-7B for evaluation

We choose the above two models following the common practice in previous works. DeBERTaV3 and LLaMA2-7B are two most commonly adopted pre-train models for evaluating the effectiveness of PEFT methods. DeBERTa (or its older BERT-based twin Roberta) is a classical model that has been adopted in the experiments of many PEFT papers[1,2,3,4], and LLaMA2-7B is used to evaluate the performance in larger and more advanced models, and is also widely adopted [2,3,4].

Still, we find your suggestion of conducting experiments on more recent models quite instructive. We are currently running experiments on LLaMA3-8B, and will update the results as soon as they come out.

[1] AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. ICLR 2023

[2] VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projections. NeurIPS 2024

[3] LoRA+: Efficient Low Rank Adaptation of Large Models. ICML 2024

[4] Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation. ICML 2024

W2/Q2: About the results in ablation studies.

In fact, the ablations are conducted on all 8 tasks of the GLUE benchmark. Here, we provide the results along with the standard deviations.

We can find the results are still consistent on other tasks besides MRPC and CoLA.

W3/W4/Q3: About the runtime of LoRA Dropout in larger models and its feasibility in practice

The runtime (both training and inference time) is related to the number of sampled dropout instances. Basically, the time is linear with $n_{sample}$, since the LoRA Dropout aggregates outputs from $n_{sample}$ dropout instances. Therefore, for larger models that require more time for a single forward pass, LoRA Dropout does further increase the training and inference time.

However, the computational overhead would not make LoRA Dropout unusable in practice. First of all, one can adjust the number of samples to conduct a trade-off between overfitting control and training/inference time cost on their demand.

Moreover, for training, the overfitting problem will practically become severe only when the data amount is limited. Under this scenario, the training time will not be a great problem (training time is short for a limited data scale) in comparison with an unacceptable overfitted performance.

For inference, the test time ensembling could be replaced with a re-scaling method at the cost of a slight decrease in performance. In the re-scaling method, dropout is not conducted during testing, and the outputs are re-scaled based on the dropout rate to adapt to the train-test difference. Without ensembling, the re-scaling method requires no additional cost of inference time. As for the performance, please refer to the results of Drop$_{train}$ from the table in the above response, which indicate the performance with the re-scaling method (conducting dropout during training and re-scaling during testing).

I noticed a revision of the paper that removes the nonsensical equation (1). This was already factored in my score.

The revision also makes some changes in the theorem, without really addressing my objections, which are that (1) the theorem does not explicitly discusses dropout, (2) the actual argument could have been made without the theorem, and (3) and it would have applied to the actual algorithm instead of the impractical one discussed in the theoretical section.

I realize that there is much pressure to have a theoretical section. However I also believe that a theoretical section does not help when it so loosely connected to the actual algorithms. On the other hand, the empirical results are convincing enough for me. This has not changed. Therefore I maintain my score.

We sincerely appreciate your suggestions and positive comments on our empirical results. We made every effort to address all the concerns and have uploaded a revised version of the paper following your suggestions. In the following, we quote your comments and then give our detailed responses point-by-point.

W1: Equation (1) is nonsensical.

Thank you for your suggestion, and we change the problem formulation to $\min_{\Delta \theta} \mathcal{L}(\theta^0 + \Delta \theta), \ \mathrm{s.t.} \ \mathbb{E}_{d \sim \mathrm{Bern}(p)} ||d \odot \Delta \theta||_2^2 \leq c$, where $c$ is a constant. However, incorporating c does not influence the duality form of this optimization problem, which is mainly discussed in our theoretical analyses, as $c$ can be eliminated as a constant. Hence, the correctness of our theoretical results is still ensured. We have made the corresponding modifications in our revised manuscript.

W2/Q1: Theorem 2.4 does not rely on the presence of dropout

In fact, Theorem 2.4 does have strong connections with dropout. The dropout rate $p$ appears in the rightmost term of Eq.(7) in theorem 2.4, and it is the only hyper-parameter that does not depend on the problem formulation and model architecture. When the dropout rate p increases, the value of the rightmost term of Eq.(7) decreases, indicating the generalization gap is reduced. By varying the dropout rate p from 1 to 0, we could derive the argument of lines 189 to 193 (lines 193 to 197 in the revised paper). Though the argument seems to be common in practice, our Theorem does offer solid theoretical evidence for it.

W3: Mismatch of dropout scheme in section 2 and section 3

We admit that the current practice of LoRA Dropout is not an exact match to the regularized problem in Eq.(3), yet the ideal solution is infeasible in practice and our solution is actually a more proper way to approximate the strict Bernoulli mask.

Specifically, in section 2, we discussed the regularized problem with dropout instances sampled from a Bernoulli distribution. An ideal way to achieve this scheme is generating 0-1 masks for each element in the updated parameters. However, this method is unrealistic since such dropout masks will be as large as model parameters, which is quite huge in current LLMs. Generating such dropout masks will lead to great GPU memory overhead.

Therefore, we hope to strike a trade-off between the theoretical scheme and practical feasibility. In section 3, we propose LoRA Dropout, trying to generate as many sparsity patterns (dropout patterns) as we can without significantly increasing GPU memory overhead. By masking both rows and columns, LoRA dropout provides more diverse sparsity patterns and offers a better estimation of the sparsity regularizer in Eq.(3) than the original dropout mechanism, where only rows are dropped out. Therefore, our practical contribution lies in a more memory-efficient and expressive approximation of the ideal dropout scheme.

Q2: What is the practical cost of the test time ensembling

The cost of test time ensembling is related to the number of sampled dropout instances. Basically, the time is linear with n_sample, since the results should be aggregated from n_sample models. One can adjust the number of samples to conduct a trade-off between performance and inference time cost on their demand.

Moreover, the test time ensembling could be replaced with a similar re-scaling method as in the original dropout method. In the re-scaling method, dropout is not conducted during testing and the outputs are re-scaled based on the dropout rate to adapt to the train-test difference. Compared with the test time ensembling method, this method requires no additional cost of time, but will lead to a decrease of performance. Here we provide the results using the re-scaling method compared with test time ensembling.

Q3: What is the rank used for LORA Dropout in Figure 4

The rank is set to 8 for the experiments shown in Figure 4. We have also conducted experiments of varying the dropout rate on different ranks like 4. Here, we provided the results on r=4 and r=8, respectively.

Thank you for your careful reading and further questions.

Firstly, we have also made the similar derivation of Eq.(1) in Proposition A.2 in the Appendix, and we are pleased to see your careful deduction reaches the same conclusion as we did. This also explains why $\lambda$ and $p$ always appear together in our theoretical results, as $\lambda p$ is the l2-regularization coefficient.

Secondly, as for what our theoretical results actually tackle, we'd like to claim that we are not studying the specific vanilla dropout scheme that randomly masks the neurons of hidden representations. Instead, we are studying a more general regularization strategy that drops out the parameter space, such as DropConnect [1] as well as vanilla dropout (which can be regarded as dropping the rows of parameter space). Therefore, our theoretical results apply to a more universal family of "dropout" regularization strategies. Furthermore, to obtain a better estimation of the Bernoulli mask of the entire parameter space as introduced in our theorems, we design the algorithm in Section 3, which enjoys a more diverse sparsity pattern than vanilla dropout without introducing large memory overheads.

Thank you again for your suggestions, and we sincerely hope our explanations could resolve your doubts.

[1] Wan, L., Zeiler, M., et al. Regularization of Neural Networks using DropConnect. ICML 2013

Here is how I interpret your last reply: The theorem indeed has very little to do with dropout because the so-called "dropout" parameter $p$ acts just as a multiplier on the L2 hyper-parameter $\lambda". And you knew it since you even wrote this important piece of information it in the appendix...

Even if we care about explicit or implicit L2 regularization instead of dropout, the theorem does not tell you much we did not know, since the $p\lambda$ controls the effective size of the set of weights (fixed pretraining weights and adjustable lora weights).

Therefore it is my opinion that the theoretical argument lowers the quality of the paper rather than enhances it.

Note that your actual dropout idea is valuable. But it would be much more scientific to say that you do not have a precise theoretical argument but have a good hand-waving explanation (the one in lines 189 to 193 of the original paper). And it might be all that matters since we still have to deal with all the dark corners of deep network training.

We sincerely appreciate your detailed comments and constructive suggestions. We made every effort to address all the concerns and have uploaded a revised version of the paper following your suggestions. In the following, we quote your comments and then give our detailed response point-by-point.

W1: Differences between theoretical analyses and LoRA Dropout practice

We admit that the current practice of LoRA Dropout is not an exact match to the regularized problem in Eq.(3), yet the ideal solution is infeasible in practice and our solution is actually a more proper way to approximate the strict Bernoulli mask. Specifically, in section 2, we discussed the regularized problem with dropout instances sampled from a Bernoulli distribution. An ideal way to achieve this scheme is generating 0-1 masks for each element in the updated parameters. However, this method is unrealistic since such dropout masks will be as large as model parameters, which is quite huge in current LLMs. Generating such dropout masks will lead to great GPU memory overhead.

Therefore, we hope to strike a trade-off between the theoretical scheme and practical feasibility. In section 3, we propose LoRA Dropout, trying to generate as many sparsity patterns (dropout patterns) as we can without significantly increasing GPU memory overhead. By masking both rows and columns, LoRA dropout provides more diverse sparsity patterns and offers a better estimation of the sparsity regularizer in Eq.(3) than the original dropout mechanism where only rows are dropped out. Therefore, our practical contribution lies in a more memory-efficient and expressive approximation of the ideal dropout scheme.

W2.1/Q1: The “Closeness” between $\theta _L (S^i)$ and $\theta _L (S)$ needs to be defined clearly

Thank you for pointing this out. We have re-examined our proof of Proposition 2.2 and made it clear that the “Closeness” between $\theta _L (S^i)$ and $\theta _L (S)$ should not be treated as an assumption for proving this Proposition, but an intermediate conclusion drawn from the proof. Please refer to Eq.(A.5) in the proof of Lemma A.1. This inequation shows that the closeness between $\theta _L (S^i)$ and $\theta _L (S)$ is automatically bounded by other arguments like Lipschitzness $\eta$ and dataset size $n$. Therefore, there’s no need to make extra definitions, and we have removed the “Closeness” assumption from the statement of Proposition 2.2.

W2.2/Q2: The discussion about how large $\eta$ is in practice is missing

Thank you for your suggestion. The Lipschitz continuity depicts that the model function $L(f(\theta))$ has a good continuity in the parameter space. For general neural networks including LLMs, we typically assume that the output (loss) space of the model is relatively smooth, which means that large variations in the output are unlikely to occur within the neighborhood around any particular set of parameters. Especially during fine-tuning, the parameters being optimized are initialized at a relatively stable point (the optimum of the pre-training task is often located at a geometrically flatter region), thus it is proper to assume a fine continuity in the parameter space.

Furthermore, we also conducted experiments to quantitatively present some empirical estimations on the value of \eta. We fine-tuned a LLaMA2-7B model and monitored the changes in parameters ($||\Delta \theta||$) and corresponding changes of losses (i.e., $\Delta L$) during the fine-tuning process. (as the direction of gradient descent indicates the largest loss variations at the current parameters, which is closer to the lower bound estimation of $\eta$) Based on the observed results, we could empirically estimate that the $\Delta L$ to $||\Delta \theta||$ ratio for a LlaMA2-7B model ranges from 0.01 to around 0.5. In other words, the lower bound of $\eta$ is around 0.5 empirically, which is a relatively small constant.

W3: Mismatch between the assumption in Proposition 3.1 and its proof

We are sorry for this mistake. In fact, the assumption in the proof in Appendix A.2 should be the same as the assumption in Proposition 3.1, i.e., the loss function is convex w.r.t . the final activation of the model. Such an assumption is realistic since most loss functions used for training, like cross-entropy and MSE, are convex w.r.t to their inputs. We have fixed this mistake in the proof from Appendix A.2 in our revised paper. Thank you again for pointing it out.

W4: Mistakes made in writing

Thank you for your suggestions and corrections. We followed your advice and improved our writing in the revised paper.

Thank you for your time and efforts in considering our rebuttal. We have made further adjustments to our manuscript according to your suggestions and presented detailed explanations point-by-point as below.

Questions about “closeness”:

First of all, we’d like to genuinely apologize for our mistakes in the previous modification. The assumption of closeness between $\theta_L(S^i)$ and $\theta_L(S)$ is indeed necessary as the error of second-order Taylor expansion could not be ignored if the closeness is not ensured. Hence, in this updated version, we recover the original assumption with a more specific constraint, i.e., $||\theta_L(S^i) - \theta_L(S)|| \leq \epsilon \rightarrow 0$. Under this assumption, the correctness of Lemma 1 is ensured as the error of Taylor expansion in Eq.(A.1) is $O(\epsilon^3)$ and can be eliminated for a small $\epsilon$. here we make further clarifications on: 1) the practical reasonableness of the closeness assumption, 2) how the theoretical results of PHS stability (Lemma 1) connect with the closeness assumption.

1. Practical reasonableness of the closeness assumption: In this paper, we are dealing with the fine-tuning scenario. Therefore, the algorithms optimizing $\theta_L(S^i)$ and $\theta_L(S)$ are both initialized with the same set of parameters $\theta_0$ (or zero-initialization of LoRA components $\Delta \theta$). Moreover, in LoRA fine-tuning, we also take a relatively small learning rate (e.g., 5e-5 on LLaMA2-7B), and the datasets only differ in one sample. All the factors above determine that the optima, $\theta(S^i)$ and $\theta(S)$, will be very close empirically. In conclusion, the way we translate the “closeness” assumption into practice is to select a small learning rate and a zero-initialization of $\Delta \theta$.

2. How the theoretical results of PHS stability (Lemma 1) connect with the closeness assumption: As shown in Eq.(A.5), we obtain another upper bound of the “closeness” except for $\epsilon$, and is further derived to obtain the final result of Lemma 1. Here, we clarify that, with a sufficiently large $\lambda$ (i.e., $\lambda \geq \frac{\eta}{n\epsilon} - \frac{1}{2}\Lambda_{\min}$), this bound is tighter than $\epsilon$. This means that under the assumption of the closeness of $||\theta_L(S^i) - \theta_L(S)||$ with a constant $\epsilon$, the optima could be closer than originally assumed if the regularization strength is sufficiently large. Meanwhile, this result also ensures the tightness of our final result in Lemma 1, i.e., the PHS stability is no larger than $\eta\epsilon$ (derived by plugging $\lambda$ conditions into (A.7)). The results also guide us to ensure the $\lambda$ is sufficiently large in practice, where we are actually applying hard dropout mechanisms and this could be regarded as a very large regularization strength.

In conclusion, the current version of our theoretical results ensures: a) correctness, b) tightness, c) practical reasonableness.

Questions about Proposition 3.1:

Here we’d like to clarify that we only assume the loss function (cross-entropy, MSE, …) is convex w.r.t. the model output $o$. This is a very natural assumption as it is true of mainstream loss functions. We also make our proofs more detailed in our updated version, where we proved for any distribution $O$ of outputs $o$, the Jensen inequality holds, then for any distribution $\mathcal{O}(x) = M(x; \mathcal{D}$), taking expectations of $x$ and $y$, the inequality still holds. We have further adjusted our proofs with more details in Appendix A.2.

Thank you once again for your careful reading and profound suggestions. We believe the clarity and quality of our theoretical results have been promoted in the new version of our manuscript.

Dear Reviewer jLmX,

I hope this message finds you well.

I am writing to kindly remind you that the deadline for the discussion phase regarding our paper is drawing close. We greatly appreciate the time and effort you have already dedicated to reviewing our work. Your insights and feedback are invaluable to us, and we believe they have significantly enhanced the quality of our manuscript.

We are wondering if we have addressed your concerns in our previous responses, and we are more than willing to provide any additional information or clarification as assistance.

Thank you once again for your time and consideration. We are looking forward to hearing from you soon.

Best Regards.

Thanks for all the constructive suggestions and comments concerning our paper. We made every effort to address all the concerns and have uploaded a revised version of the paper. In the following, we quote your comments and then give our detailed response point-by-point.

W1/Q1:Detailed Explanations on Lipschitz continuity condition on parameters and positive semi-definiteness of Hessian

Lipschitz continuity: This condition depicts that the model function $L(f(\theta))$ has a good continuity in the parameter space. For general neural networks including LLMs, We typically assume that the output (loss) space of the model is relatively smooth, which means that large variations in the output are unlikely to occur within the neighborhood around any particular set of parameters. Especially during fine-tuning, the parameters being optimized are initialized at a relatively stable point (the optimum of the pre-training task is often located at a geometrically flatter region), thus it is proper to assume a fine continuity in the parameter space.

We also quantitatively present some empirical estimations on $\eta$. We fine-tuned a LLaMA2-7B model and monitored the changes in parameters ($||\Delta \theta||$) and corresponding changes of losses (i.e., $\Delta L$) during the fine-tuning process. (as the direction of gradient descent indicates the largest loss variations at the current parameters, which is closer to the lower bound estimation of $\eta$) Based on the observed results, we could empirically estimate that the $\Delta L$ to $||\Delta \theta||$ ratio for a LlaMA2-7B model ranges from 0.01 to around 0.5. In other words, the lower bound of $\eta$ is around 0.5 empirically, which is a relatively small constant.

Positive semi-definiteness of Hessian: We admit that this assumption is too strong. Actually, we do not require such a strong assumption to support our proof. In detail, we only require $\Lambda + 2\lambda$ to be non-negative in the proof procedure (A.1) ($\Lambda$ denotes the eigenvalues of Hessian through unitary decomposition $Udiag{\Lambda}U^{-1}$). This means that we only need to constrain the regularization coefficient $\lambda$ to be sufficiently large (i.e., $\lambda \geq -\frac{1}{2}\Lambda_{\min}$), instead of hypothesizing an overly strong property of the optimized function. In fact, in our implementation, the regularization is accomplished in a “harder” way through dropout instead of the sparsity term, which complies with the setting of a large regularization strength $\lambda$. In the new version of our manuscript, we have modified the corresponding condition to the constraints on regularization strength. Thank you again for your suggestions.

W2/Q2: Details on the computational overhead and practicality

We have reported the quantitative stats of training time on GLUE in Table C.2 (Appendix C). The sampling of multiple dropout instances will indeed increase the training and testing time, and the increase is related to the number of samples.

However, the computational overhead would not make LoRA Dropout unusable in practice. Practically, the overfitting problem will become severe only when the data amount is limited. Under this scenario, the training time will not be a great problem (training time is short for a limited data scale) in comparison with an unacceptable overfitted performance. This also means one can adjust the number of samples to conduct a trade-off between overfitting control and training/inference time cost on their demand.

W3/Q3: More experiments with other LoRA variants

Thanks for the suggestions. Here, we provide the results on more LoRA variant models.

Results on LoRA+:

Results on LoRA-FA:

The results demonstrate the effectiveness of LoRA Dropout in different variants of LoRA.

Q4. Differences between LoRA Dropout and other dropout methods

The differences mainly lie in two aspects:

1. Model architecture. Other dropout methods, like DropKey, DropAttention, and HiddenCut, are designed specifically for the transformer architecture. LoRA Dropout is designed for the PEFT method LoRA, with no restriction to the model architecture.
2. Perturbation space. Dropout methods, like DropKey, DropAttention, and HiddenCut, conduct dropout in the input space, e.g., dropping random elements (tokens) from the sample’s self-attention matrix. Therefore, these dropout methods increase the diversity of samples and could be viewed as a kind of data augmentation method. LoRA Dropout conducts dropout in the parameter space. With theoretical support, LoRA dropout prevents overfitting by introducing proper sparsity regularization during training. In conclusion, the perturbation spaces of LoRA dropout and other mentioned methods are actually orthogonal. We regard studying the effect of combining both techniques as an interesting future work.