Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation

We sincerely thank Reviewer uABN for the insightful feedback, which helped improve both our analysis and presentation. Below, we respond to the main concerns point by point. Additional results are available via https://anonymous.4open.science/r/coto, with new content labeled as Tab. rX and Fig. rX.

Claims And Evidence

We agree with these concerns and will revise the claims for clarity.

• L19,r: Fig. r4 shows that models with the same initialization tend to converge to similar solutions, regardless of method or learning rate. We will also cite works supporting lazy training dynamics.

• L75-76,r: The term “hierarchical structure” was unclear. We will revise it to layer-wise importance differences, note that layer importance is modality-dependent, and specify “layer-wise LoRA-style PEFT methods” instead of the broader term PEFT.

• L416,l: Fig. 6-7 in the manuscript show pruning and marginal contributions at 25% training. Fig. r1 provides additional merging and pruning results over training steps.

• L430,r: As shown in [r1], the dropout objective is equivalent to adding a regularization term. Similarly, CoTo’s objective is a weighted sum of sub-objectives, where the weight ($w_6$ in Fig.2) of the standard objective gradually dominates as training progresses.

  [r1] On the Regularization Properties of Structured Dropout.

• L434,r: Fig. r5 has been updated with unified colorbars.

Experimental Designs

[E.1] Comparison to other stochastic techniques

Stochastic depth is designed for pretraining and not applicable to LoRA. [2] introduces a KL loss between dropout and non-dropout outputs during PEFT but is not open-sourced or directly applicable. Thus, we compare CoTo with Dropout [1] and a variant using our progressive strategy. As shown in Fig. r2 and Fig. r3, CoTo consistently outperforms naive dropout in general, merging and pruning tasks.

[E.2] Run LoRA with the same learning rate

• In Tab. 5 of ablation studies, we add LoRA-Pro results with 1e-4 and 2e-4 for fair comparison (55.8 ± 0.7, 40.9 ± 1.1). On classification tasks, CoTo also consistently outperforms LoRA on general, merging, and pruning performance across five learning rates (Fig. r6).
• The higher learning rate for CoTo offsets slower convergence from stochastic training. Full learning rate settings are provided in Appendix A.3.

[E.3] Remove claims or report quantitative measures

We will remove “significant” and move diffusion results to Appendix. Additional quantitative results (Tab. r2) and qualitative examples will be included.

[E.4] Why DTD for pruning analysis

DTD has relatively low zero-shot performance (~44%), making the LoRA's gain more visible. Results on all vision tasks are provided in Fig. r7 for completeness.

[E.5] Why use generative models instead of BERT-style models like DeBERTa-v3

• We follow LoRA-LEGO, which evaluates only generative LLMs, as they are more advanced and suitable for merging. BERT models for classification rely on task-specific poolers and classifiers, making merging less meaningful (e.g., not generalizable to OOD tasks). In contrast, generative models support instruction tuning across diverse tasks via a unified prompt format [r2], offering a more rigorous testbed.

  [r2] Finetuned Language Models are Zero-Shot Learners.

• As suggested, we have also added results on DeBERTa-v3 in Tab. r1, where CoTo still shows consistent improvements.

[E.6] Analysis of the magnitude of P

We optimize the alignment matrix P by minimizing the proposed $\Delta_{upper}$ using Adam (500 steps, lr=0.01) and analyze both the magnitude of P and the difference $\|\Delta W_f - \Delta W_m\|_2$ before and after applying P. Results are shown in Fig. r8. For LoRA, the magnitude of P is larger, and the difference (both before and after applying P) is smaller than in CoTo. This indicates that CoTo improves linear mode connectivity not by adapter-wise alignment but by addressing layer-wise misalignment.

Essential References

Thanks for highlighting these works. We will cite and discuss them in the revision.

Weaknesses

[W.1] Fig. 9 shows that CoTo requires longer training until convergence

The slower loss decrease in Fig. 9 is due to CoTo’s stochastic activation, which delays early convergence but improves generalization. Notably, CoTo with LoRA and DoRA reaches lower loss at 900 and 2k steps, respectively. Also, as each step in CoTo is faster (due to adapter skipping), total training time is reduced (Tab. 6). Any delay from stochasticity can be mitigated by adjusting the learning rate or activation schedule (Fig. r1).

[W.2] Intuition why Ensemble-CoTo performs worse

Ensemble-CoTo improves on DeBERTa (Tab. r1) but underperforms on LLaMA, possibly due to non-convergence. We will further investigate this and update the results if new findings emerge.

Comments

Thanks, we will revise accordingly.

Questions

Please see our response to E.6.

We sincerely thank Reviewer Xi9N for the thoughtful review and for highlighting both the strengths and limitations of our work. We especially appreciate the recognition of our experimental rigor and the inclination toward acceptance. Below, please find our responses to the main concerns, and let us know if any issues remain. Additional experiments conducted during the response period (see Fig. rX & Tab. rX) are available in this anonymous link: https://anonymous.4open.science/r/coto.

[Weakness] As the authors mentioned in related works, there are many works that tested applying structured dropout and stochastic depth. The novelty of the paper is limited, i.e., applying structured dropout to PEFT methods.

We fully understand the concern regarding novelty. Indeed, from a structural perspective, our method may appear similar to prior work on structured dropout and stochastic depth. We acknowledge this and appreciate the reviewer’s valuable observation.

• We welcome the opportunity to clarify CoTo's distinct contributions and novelty within the PEFT landscape. While there are superficial similarities, CoTo's core innovation goes beyond simply applying existing ideas. As discussed in Appendix C and summarized in Table 8, while CoTo, dropout, and stochastic depth all involve deactivation during training, they differ fundamentally in objective:

  • Dropout aims to prevent overfitting by randomly deactivating individual neurons or weights.

  • Stochastic depth is primarily designed for efficient pretraining of full-model by randomly skipping entire layers.

  • CoTo is designed as a training paradigm for the layer-wise PEFT method, aiming to balance the utilization of LoRA adapter layers. It prevents adapters from dominating and promotes better merging and pruning by employing structured, layer-wise deactivation coupled with a progressive activation strategy.

  In essence, CoTo synthesizes concepts from regularization (like dropout) and efficient training (like stochastic depth) but repositions and adapts them into a coherent strategy specifically targeting the unique dynamics and optimization challenges of layerwise PEFT.

• To empirically demonstrate this distinction, we provide additional experiments (Anonymous Figs. r2 and r3) on image classification comparing CoTo against direct applications of dropout to LoRA adapters:

  • Standard Dropout (fixed $p$).

  • Progressive Dropout (linearly decaying $p$).

  Fig. r2 shows LoRA adapter merging robustness under multiple seeds, and Fig. r3 evaluates structured pruning across adapter components. The results clearly show that naive applications of dropout strategies yield limited benefits and do not effectively improve merging or pruning robustness. CoTo consistently and significantly outperforms these variants in terms of accuracy, merging performance, and pruning robustness, highlighting that its effectiveness stems from its PEFT-specific design, not merely from applying dropout principles.

Finally, we note that CoTo is fully compatible with standard dropout, which is already included in our main experiments. We will revise the manuscript to better highlight these conceptual and empirical distinctions, helping readers better position CoTo within the broader landscape of PEFT techniques.

We sincerely thank Reviewer VYdq for the thoughtful and constructive comments. We address each concern in detail below and remain open to any further questions.

[W1] Notation is a bit unclear. What exactly does "a single LoRA layer" refer to? And what does "adapter" refer to? Does "one adapter" mean all LoRA modules added to these matrices in one layer?

We greatly appreciate the reviewer for bringing our attention to this notation, which indeed benefits from further clarification.

• Adapter: As stated in Lines 11–13 (right) of our manuscript, we use the term adapter to refer to the trainable parameters introduced by LoRA.
• A single LoRA layer = a single adapter denotes the collection of all LoRA modules inserted into a single Transformer layer.
  • The exact inserting locations of LoRA within each layer vary by task, following state-of-the-art practices, as detailed in the "Target Module" of Tab. 7.
  • Consequently, activation decisions are applied collectively to all LoRA modules within a layer, rather than to individual modules.

[W2] Explain in more detail how the adapter drop method works

Absolutely. Thank you again for the question. The adapter drop mechanism is a key component of our training strategy, described in Sec. 3.1 and illustrated in Fig. 1, and we appreciate the opportunity to clarify it further.

At each training step, we sample a Bernoulli variable $\delta_l\sim \text{B}(p)$ for each layer to determine whether the entire layer of adapters is activated. This sampling is implemented using the TrainerCallback function at each step. The activation probability $p$ increases linearly from 0 to 1 during the first three-quarters of training and remains fixed at 1 thereafter. When deactivated, all LoRA modules within a transformer layer are excluded from both forward and backward computation. As shown in Fig. 1, this strategy enables extensive layer-skipping during the early stages of training and gradually transitions to full activation of all adapters.

[W3] Table 1 performance improvement seems marginal

Thank you for this insightful question. While the improvements in Tab. 1 may appear modest in absolute terms, we note that:

• Tables 1 and 2 report results on standard benchmarks, where performance is already strong. CoTo still achieves consistent gains while maintaining lower cost (Sec. 5.3). Compared to prior methods, such as DoRA vs. LoRA and HiRA vs. DoRA, CoTo provides comparable or even greater improvements, setting new state-of-the-art results in several settings.
• In addition to generalization accuracy gains, CoTo significantly enhances the merging and pruning performance of LoRA adapters. This is another major benefit of our method.

CoTo improves generalization at both the sample level and the task level:

• Tab. 1–3 evaluate performance on the test sets of in-domain tasks, demonstrating sample-level generalization. In Tab. 3, we train on MetaMathQA and test on GSM8K, which differ in domain, reflecting robustness under domain shift.
• Tab. 4 includes explicitly held-out tasks in the merging experiments. Following the LoRA-LEGO protocol, we train LoRA adapters on a subset of tasks (in-domain) and evaluate merged adapters on both seen (ID) and unseen (OOD) tasks. CoTo consistently improves performance on these held-out tasks, demonstrating strong task-level generalization.

We will revise the manuscript to more clearly highlight these generalization settings and the role of held-out tasks in our experimental design.

[W4] Line 160-164, why does a decreasing learning rates facilitates the exploration of new local optima...

We acknowledge that the original statement "Coupled with decreasing learning rates, this strategy facilitates the exploration of new local optima while preserving properties established in earlier stages" was confusing and appreciate the opportunity to clarify.

• The "strategy" referenced here is the proposed CoTo strategy, which is responsible for facilitating the exploration of new local optima.
• What we intended to deliver is that "decreasing learning rates" coupled with the lazy training property of neural networks (cf. Line 158 - 160) is responsible for "preserving properties established in earlier stages", such as linear mode connectivity and dropout stability.
  • Specifically, as supported by prior work [r1, r2], using a higher learning rate during the early training phase promotes broad exploration of the loss landscape. Subsequently, gradual learning rate decay encourages convergence to flatter, more generalizable minima. In our empirical setting, we adopt a cosine decay learning rate schedule, which we believe helps maintain early-stage properties. We will revise the manuscript to better reflect this intent.

[r1] SGDR: Stochastic Gradient Descent with Warm Restarts.

[r2] A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performance.

We sincerely thank Reviewer Np5j for the valuable comments. Below, please find our responses to each concern, and let us know if any issues remain. All experiments during the response period (Fig. rX & Tab. rX) are accessible in this anonymous link https://anonymous.4open.science/r/coto.

[Q1] Direct gradient-based validation for layer-wise optimization balance

• Gradient-based validation

  To directly validate layer-wise optimization balance, we compute the average gradient magnitudes of layers from both the text and vision encoders on vision classification tasks. For comparability, gradients are normalized such that their sum across layers equals one. The results below indicate that CoTo leads to a more balanced gradient distribution across layers:

  	Lower Layers (0-4)	Middle Layers (5-8)	Higher Layers (9-12)
  LoRA (text)	27.59%	32.35%	40.06%
  LoRA-CoTo (text)	28.48%	33.53%	37.99%
  LoRA (vision)	24.06%	32.59%	43.35%
  LoRA-CoTo (vision)	26.74%	33.90%	39.35%

• Other validations already presented in our manuscript

  • Fig. 6 shows that CoTo-trained models consistently outperform LoRA, regardless of whether the lower, middle, or higher layer-adapters are pruned.
  • Fig. 7 presents Shapley value analyses, revealing that CoTo enhances the marginal contributions of lower and middle layers.

Together, these results support the effectiveness of CoTo in promoting balanced layer-wise optimization.

[Q2] The theoretical analysis may not fully generalize to transformers in attention layers.

For clarity and simplicity of notation, we formulate our method using fully connected networks, where $W_l$ represents the total parameters of layer $l$. However, since our analysis (e.g., Theorem 3.1) considers activation at the layer level (as described in Sec. 3.1), it does not rely on the internal structure of each layer. By the Law of Large Numbers, the CoTo objective can be reasonably approximated as a weighted sum of sub-objectives, making the analysis broadly applicable — including to transformer-based architectures.

[Q3] The experiments on CoTo's adapter merging are limited to LLaMA-2 and LLaMA-3. Existing works highlight the importance of adapter merging across different tasks/domains, but this paper's scope is too narrow to generalize.

We respectfully clarify below the broad scope of adapter merging evaluations, which we have deliberately designed to validate CoTo's high generalizability.

• Architectural scope includes LLaMA-2/LLaMA-3/CLIP/SDXL/DeBERTa V3
  • As recognized by your summary, we evaluate adapter merging on CLIP (Fig. 4), LLaMAs (Fig.4, Tab. 4), and SDXL (Fig. 5), covering widely adopted architectures in text, vision, and multimodal domains.
  • We also evaluate CoTo on the BERT-style architecture DeBERTa V3 (Tab. r1). Despite the challenges of merging BERT-style models -- requiring additional fine-tuning of task-specific poolers and classifiers -- CoTo still outperforms.
• We fully agree that adapter merging across different tasks/domains is critical. Aware of this, Sec. 4.2.2 ("LoRA Merging with Different Tasks") merges 7 adapters trained on 7 in-domain (ID) tasks, and evaluates the cross-task merged adapter on both ID and 2 out-of-domain (OOD) tasks. The consistent superiority with CoTo over baselines advocates our broad scope in cross-task adapter merging.
• Besides, our adapter-merging evaluations span diverse task types, including vision classification (Fig. 4), commonsense reasoning (Fig. 3), natural language understanding (Tab. 4), and customized generation via diffusion models (Fig. 5), further validating CoTo's generalizability across tasks/domains.

[Q4] Why use a linear increase in activation probability?

We select linear increase as it offers a favorable trade-off between simplicity and performance, preserving the core benefits of the proposed CoTo.

• Simplicity: Unlike other activation probability scheduling functions (e.g., sine and exponential), linear increase requires no additional hyperparameters given the first phase spanning 75% of the training duration (cf. Line 410 (left)).
• Performance: Following the reviewer's great suggestion, we compare linear, sine, and exponential activation probability functions. As evidenced in Fig. r1, linear increase strikes a delicate balance between generalization, merging, and pruning performance.