GoRA: Gradient-driven Adaptive Low Rank Adaptation

We sincerely appreciate the reviewer's insightful and encouraging comments regarding our work. We are grateful for their recognition of the theoretical foundations, extensive experimental validation, and clear presentation of our proposed method. Below we provide detailed responses to address the reviewer's specific concerns.

Q1. Does the memory usage ion Table 8 also include the gradient accumulation step? Why gradient accumulation step doesn't lead to extra memory usage?

We appreciate the reviewer's attention to the initialization cost of GoRA. We would like to provide further clarification about our initialization algorithm's memory efficiency and scalability.

Memory Management During Initialization:

Our initialization algorithm employs a carefully designed memory management scheme:

1. After computing gradients for a specific layer:
   • Gradients are immediately averaged across GPUs via all-reduce
   • Only the local rank-0 process stores the gradients in CPU memory
   • All other processes discard their gradients
2. Memory footprint breakdown:
   • GPU memory:
     • 16-bit model weights
     • 16-bit activations
     • 16-bit single-layer gradients
   • CPU memory:
     • One copy of 16-bit gradients for all target modules (e.g., 2-3GB for attention modules in an 8B model)

Comparison with Training Phase Memory Usage:

• Training phase GPU memory includes:
  • 16-bit model weights
  • 16-bit activations
  • 16-bit full-model gradients
  • 32-bit trainable parameter copies
  • 32-bit optimizer states
• The peak GPU memory usage during initialization remains consistently lower than during training, as demonstrated in Table 8 (which already accounts for initialization memory). Under the experimental conditions reported in this paper, the initialization process consumes 17.35GB of memory, while training requires 19.75GB.

Empirical Validation on Large Models:

We conduct an additional experiment on a 32B model under the following conditions:

1. Hardware Configuration:
   • 8 × NVIDIA A800 80GB GPUs
   • 105 × Intel Xeon Platinum 8336C CPUs @ 2.30GHz
   • 1.875TB RAM
2. Software Configuration:
   • Model: Qwen2.5-32B-Instruct
   • Dataset: MetaMathQA-395K
   • Batch size: 1 per GPU
   • Gradient accumulation steps: 8
   • Optimizations: Liger kernel + FlashAttention + Activation Checkpoint

Key Results:

1. Initialization Phase:
   • Completed in 18 steps (early stopped from planned 64 steps)
   • Total time: 2 minutes 44.65 seconds (~8.4s/step)
   • GPU memory: 64,821 MB
2. Training Phase:
   • Total time: ~11 hours
   • GPU memory: 66,381 MB

These results demonstrate that GoRA's initialization remains more memory-efficient than training even at 32B scale. For extremely large models (70B+), we can further optimize memory by:

• Offloading model weights to CPU during initialization
• Reducing GPU memory footprint to just single-layer weights

This design ensures GoRA's scalability while maintaining its theoretical advantages.

Q2. n-step gradient

We sincerely appreciate this insightful observation. The reviewer is correct—our original method computes gradients over n micro-batches but applies them in a single aggregated step (equivalent to large-batch training), rather than performing sequentialn-step updates. We apologize for the unclear terminology and have revised the manuscript to explicitly refer to this as one-step gradient accumulation to avoid confusion.

Introducing GoRA-pro: A Novel n-Step Gradient Framework

Motivated by the reviewer's question, we developed GoRA-pro, a memory-efficient algorithm that leverages truen-step gradients for rank allocation. Here's how it works:

1.Lightweight Pre-Training Phase:

• A layer-wise SGD optimizer performs n exploratory updates (η: learning rate): $p = p - \eta \cdot \nabla p$
• Gradients are discarded immediately after updates, minimizing memory (only 56,151 MB vs.72,971 MB in formal training).

2.Delta-Weight Calculation:

• Aftern steps (~1-2s/step), we compute the n-step gradient per layer:

  p.grad_stored += p.grad.detach().clone().cpu()  # Aggregate gradients
  p.iters += 1
  

• The final gradient is normalized:

  grad_stored = p.grad_stored / p.iters
  

3.Rank Allocation & Reset:

• Layer importance is estimated using then-step gradient and a CPU-stored reference model $W_{\text{cpu}}$.
• The original weights are restored, and low-rank adapters are initialized with the compressed gradients.

Advantages & Results

GoRA-pro bridges the gap between theoreticaln-step gradients and practical constraints:

-Accuracy: Simulates full fine-tuning dynamics more faithfully than single-step methods.

-Efficiency: No extra memory overhead (exploratory phase uses 23% less memory than formal training).

-Novelty: To our knowledge, this is the first work to usen-step gradients for low-rank adapter initialization.

We will include empirical comparisons (one-step vs.n-step) and a broader discussion of this direction in the revised manuscript. Thank you for highlighting this impactful opportunity!

Minor issues

Thank you for reminding! We will fix these issues in the revised version of our paper!

Limitations

We appreciate the reviewer's thoughtful comment regarding memory usage during the initial gradient accumulation step. We agree this is an important consideration for practical applications, particularly when scaling to larger models.

As we addressed in our response to Q1, our initialization method does not increase peak memory consumption during training. For clarity, we will:

1. Expand the discussion in Section 5.4 ("Computation and Memory Analysis") to explicitly address this point
2. Include a more detailed comparison of memory requirements between our initialization process and standard LoRA training process

The current brevity of this discussion was due to page constraints rather than oversight. We appreciate the opportunity to make this aspect clearer in our revised manuscript.

We sincerely thank the reviewer for their thoughtful and encouraging assessment of our work, particularly their recognition of GoRA’s originality, computational efficiency, and comprehensive validation. We greatly appreciate the time and care taken to highlight the strengths of our framework. Below, we address the reviewer’s concerns point by point.

Q1. Memory Claim

We sincerely appreciate the reviewer's careful examination of the memory issues. First, we apologize for the citation oversight in our paper - the correct reference should be Table 8. Regarding memory consumption:

1. Our method introduces no additional GPU peak memory:
   • Initialization peak memory: 16-bit model params + 16-bit activations + single-layer gradients
   • Training peak memory: 16-bit model params + 32-bit optimizer states + 32-bit param copies + 16-bit activations
   • Even with ZeRO-2 optimization, initialization memory remains significantly lower than training memory (as shown in Table 8)
   • Optional: Model can be offloaded to CPU during initialization for further GPU memory reduction
2. CPU memory clarification:

   • Storing 16-bit targeted modules’ gradients does incur additional CPU memory compared to LoRA (the gradients are only saved by the local-rank-0 process)

   • However:

     a) CPU memory is substantially more affordable than GPU memory

     b) Our experimental platform features 1.8TB CPU memory (per 8-GPU node)

     c) For 100B-parameter models (~200GB), this overhead is entirely manageable

We will include detailed memory analysis in the revised version and correct all relevant citations.

Q2. Initialization Phase for Larger Models/Datasets

We appreciate the reviewer's valuable feedback regarding the scalability of our initialization phase. In response, we conduct a additional experiment on larger-scale models with latest optimized algorithms (as detailed in our responses to Reviewer 4NWP's Q3-Q4). Below are the experimental results:

Experimental Setup:

1. Hardware:
   • 8 × NVIDIA A800 80GB GPUs
   • 105 × Intel(R) Xeon(R) Platinum 8336C @ 2.30GHz CPUs
   • 1.875TB RAM
2. Software:
   • Model: Qwen2.5-32B-Instruct
   • Dataset: MetaMathQA-395K
   • Batch size: 1 per GPU
   • Gradient accumulation steps: 8
   • Optimizations: Liger kernel + FlashAttention + Activation Checkpoint

Results:

1. Initialization Phase:
   • Completed in 18 steps (early stopped from 64 planned steps)
   • Total time: 2 minutes 44.65 seconds (~8.4s/step)
   • GPU memory usage: 64,821 MB
2. Training Phase:
   • Total time: ~11 hours
   • GPU memory usage: 66,381 MB

These results demonstrate that our optimized initialization remains highly efficient even for models at the 32B scale, with negligible overhead compared to full training. We will include this analysis in the revised manuscript.

Q3. How alpha is selected

Our setting for $\alpha$ is consistent with previous works[1,2,3] (i.e, $\alpha=16$)

[1] Wang S, Yu L, Li J. Lora-ga: Low-rank adaptation with gradient approximation[J]. Advances in Neural Information Processing Systems, 2024, 37: 54905-54931.

[2] Wang Z, Liang J, He R, et al. Lora-pro: Are low-rank adapters properly optimized? In: Proceedings of the International Conference on Learning Representations, 2025.

[3] Meng F, Wang Z, Zhang M. Pissa: Principal singular values and singular vectors adaptation of large language models[J]. Advances in Neural Information Processing Systems, 2024, 37: 121038-121072.

Q4. Dynamic Initialization of Matrix A

Thank you for your insightful suggestions about initialization strategy of Matrix A. We conduct follow experiments:

The results demonstrate that initialize A with gradient principle components yield comparable performance compared to standard GoRA while initialize B with weight principle components yield downgraded performance. We hypothesis that the noisy from the weight principle components can lead to sub-optimal gradient compression. We will further investigate and discuss these strategies in the revised version of our paper.

Q5. Typos

We sincerely appreciate the reviewer's careful attention to detail in identifying these typographical errors. We have thoroughly reviewed the manuscript and corrected all instances in our revised version.

Q1. Gradient Computing Cost

We appreciate the reviewer's important observation regarding memory requirements during gradient computation. To clarify our approach:

1. Memory-Efficient Initialization:
   • We employ layer-wise gradient computation with immediate all-reduce and CPU offloading
   • This eliminates the need to store gradients on GPU
2. Optimizer State Advantage:
   • Our method requires no optimizer states during initialization
3. Scalability:
   • The peak memory footprint matches single forward pass requirements
   • Enables handling of 13B+ models within standard GPU memory constraints (weights can also be offloaded)
4. Empirical Evidences:
   • For a 7b model trained on 8 * A800 with a batch_size=8 per GPU:
     • GPU memory (initialization): 56,139MB
     • GPU memory (training): 73,295MB
   • For a 32b model trained on 8 * A800 GPUs with a batch_size=1 per GPU:
     • GPU (initialization): 64,821 MB
     • GPU (training): 66,381 MB

Q2. Motivation of Our Non-zero Initialisation Method

Subtracting $A_0B_0$ from the pre-trained weights $W$ (as in PiSSA) is indeed feasible, this approach introduces critical practical limitations:

1. Reproducibility Challenges in Inference:
   • Methods like PiSSA rely on randomized SVD to compute $A_0B_0$, making it impossible to exactly recover the same $A_0B_0$ during inference.
   • Data-dependent methods (e.g., LoRA-GA) require training data to compute $A_0B_0$.
2. Storage Overhead:
   • Storing either (a) the modified $W - A_0B_0$ or (b) $A_0B_0$ itself would resolve the above issue but at the cost of significant additional storage.
3. Theoretical Limitation:

Considering there is a optimal downstream transfer $\Delta W^*$ for a downstream task to be tuned:

$\left\Vert AB - \Delta W^* \right\Vert_F^2 = \left\Vert W + (AB - A_0 B_0) - W^* \right\Vert_F^2 = \left\Vert AB - (A_0 B_0 + \Delta W^*) \right\Vert_F^2$

The minimal error of previous non-zero methods is given by $A_0B_0 + \Delta W^*$ which is a biased best-r approximation of $\Delta W^*$. This bias can lead to sub-optimal solution.

Q3. Optimal Solution for Decomposition

We appreciate the reviewer’s question regarding our choice of optimization objective. To clarify:

1. Optimality under L2:

   • The truncated SVD provides the optimal solution in L2 norm.
   • we use the pseudo-inverse because:
     • It naturally arises when solving the problem for optimal initialization given an initialized A. Solving this problem can achieve a compression result more aligned with the Eq. (4) in our paper.

2. Why Not L1?

   • While L1 might offer sparser solutions, it lacks a closed-form solution and requires iterative optimization.

3. Empirical Justification:

   • We tested a SVD-based initialization (similar to LoRA-GA):

   Method	GSM8K	HumanEval
   LoRA-GA	71.39±0.90	43.29±0.71
   GoRA(SVD)	72.68±1.06	44.71±0.93
   GoRA(pseudo-inverse)	72.91±0.76	48.98±2.14

Results demonstrate: Our method outperforms both SVD variants, justifies our design choice.

Q4. Dynamics change of rank

Thank you for your insightful question. We appreciate this opportunity to clarify our methodology and share additional empirical findings.

Current Limitations in Dynamic Rank Adjustment

we fully acknowledge that the ideal rank distribution may evolve during training. However, implementing dynamic rank adjustment faces two fundamental challenges in modern training frameworks:

1. Technical Constraints in Data-Parallel Training: Frameworks like DeepSpeed flatten and distribute gradients/optimizer states across workers, making it architecturally infeasible to modify parameter shapes mid-training.
2. Training Stability Concerns: Even if rank adjustment were possible, it would require:
   • Jagged learning rate scheduling
   • Complex mechanisms to maintain optimizer state consistency
   • Risk of destabilizing convergence

Given these constraints, our current work focuses on pretraining-adaptive rank allocation

Empirical Analysis of Layer Importance Dynamics

We conducted a detailed study tracking layer importance every 5 steps under:

Key Findings:

1. Consistent Hierarchy:
   • wv layers maintain significantly higher importance throughout training
   • wq/wk layers remain persistently less important
2. Temporal Stability:
   • Lower Layers: Slight importance decay after ~200 steps
   • Middle Layers: Moderate importance growth after ~200 steps
   • Upper Layers: Remarkably stable importance trajectories
   • Exceptions: Minority of layers show gradual monotonic changes (sum normalized importance ±<0.02)

This suggests that while minor importance shifts occur, the relative rank priorities remain largely stable during the fine-tuning process.

References

[1] Adalomo (ACL 2024)

[2] Full Parameter Fine-tuning for Large Language Models with Limited Resources. (ACL 2024)

Q5. Rerun the initialization strategy

We fully agree that periodically re-initializing the low-rank adapters to incorporate full fine-tuning training dynamics could further enhance performance. Below, we clarify the technical feasibility.

1. Technical Problem

re-compressing gradients into low-rank adapters is possible, though it requires tight integration with gradient collection pipelines (e.g., DeepSpeed). We are actively engineering this solution.

2. GoRA-pro: Integrating Fine-Tuning Dynamics

Motivated by the reviewer’s suggestion, we propose GoRA-pro, which refines initialization by simulating fine-tuning dynamics before formal training:

1. Pre-Training Optimization:
   • Use the AdaLomo to perform n-step updates on the full model parameters.
   • Store gradients at each step.
2. Reinitialization:
   • Reset the model to its original pre-trained weights.
   • Initialize low-rank adapters using the accumulated gradients.

Advantage over GoRA:

• GoRA-pro actively integrates fine-tuning dynamics into adapter initialization.
• no prior or concurrent LoRA work explores such a hybrid of full fine-tuning and low-rank initialization.

3. Implications and Future Work

Although currently GoRA-pro exhibits on-par performance compared to GoRA, we believe this direction could open new avenues for improving LoRA’s performance.

Q6. How far from the best compression

While it is challenging to directly compare the trained adapter matrices with the pre-training gradients, our initialization strategy ensures that GoRA begins training with a low initial loss, implying effective capture of gradient information.

To provide a rigorous and intuitive comparison, we conducted experiments on the Llama-3-8B-Base model trained on the MetaMathQA and Code-Feedback, evaluating the reconstruction error of initialized adapters against the full gradients across all adapted layers:

• Math:
  • Absolute Error: 0.0382
  • Relative Error: 0.1147
• code:
  • Absolute Error:  0.0322
  • Relative Error: 0.1152

This indicates that our compression retains ~90% of the gradient information. Notably, when combined with the n-step accumulation strategy (see Q5), the initial loss further drops sharply from 0.68 to 0.44, underscoring the efficacy of our compression.

Q7. MTBench Score

We sincerely appreciate the reviewer’s observation regarding performance variations on MTBench. We would like to address this concern through the following clarifications:

1. Nature of r_ref Parameter:

   • r_ref is fundamentally a configuration parameter rather than a tunable hyperparameter. Its primary role is to establish parity in total trainable parameters between GoRA and comparable methods (e.g., r_ref=32 in GoRA corresponds to r=32 in LoRA).
   • We explicitly recommend practitioners use the largest computationally feasible r_ref value within their resource constraints.

2. Understanding MT-Bench Variability:

   The observed performance fluctuations are consistent with well-documented phenomena in PEFT:

   • The non-monotonic relationship between rank size and model performance on MT-Bench has been previously established (e.g., in LoRA-Pro and LoRA-GA).
   • MT-Bench’s particular characteristics may contribute to this variability:
     • Limited scale
     • Possible distributional mismatch between training and evaluation data
     • Known stochasticity inherent in LLM-as-judge evaluation

3. Hyperparameter Management:

   While GoRA introduces four core parameters, we have additionally developed automated tuning mechanisms to enhance practical usability:

   • Our response to Reviewer 4NWP’s Q4 details these optimization procedures

Q8. Combine with QLoRA

We are pleased to confirm that GoRA can indeed be seamlessly integrated with QLoRA's quantization approach.

Implementation Details:

• We quantize the pre-trained weights to NF4 precision for storage
• During computation, we de-quantize these weights back to bf16 precision

Empirical Results:

Key Observations:

1. QGoRA consistently outperforms QLoRA
2. The improvement is particularly notable on GSM8K

Q9. Typo

Thank you for remind us this typo! We have fixed this typo in the revised version of our paper.

We sincerely thank the reviewer for their constructive feedback and for recognizing our efforts in addressing their concerns. We greatly appreciate their time and support in recommending acceptance.

We are particularly grateful for the reviewer's insightful observation regarding storing the random seed for randomized method such as PiSSA! In response to this point, we would like to clarify that the official PiSSA repository (search PiSSA in github) currently provides two approaches to ensure consistency, as mentioned in our original submission: (1) storing the modified pre-trained weights, or (2) retaining the initialized low-rank weights. This observation lead to our conclusion in our rebuttal.

We will add more discussions on these solutions (including storing the random seed) in our revised manuscript.

Once again, we are grateful for the reviewer’s valuable input, which has helped improve our work.

We sincerely appreciate the reviewers' recognition of GoRA's strengths, including its unified treatment of rank and initialization that avoids AdaLoRA-style masking overhead, its robust gains across multiple modalities (even surpassing full fine-tuning on GSM8K and HumanEval with rmax=128), and its comprehensive ablation studies. We also deeply value the reviewers' insightful concerns and valuable suggestions regarding computational efficiency (time/memory costs) and hyperparameter selection in GoRA. Below we provide detailed responses with experimental validations.

Experimental Setup

All experiments presented in this rebuttal were conducted on a computing node with:

• Hardware:
  • 8 × NVIDIA A800 80GB GPUs
  • 105 × Intel(R) Xeon(R) Platinum 8336C @ 2.30GHz cores
  • 1.875TB RAM
• Software Configuration:
  • model: Llama3.1-8b-Base
  • Liger kernel for RMSNorm, fused linear cross-entropy loss, and MLP operations
  • Batch Configuration:
    • Global batch size: 64
    • Per-GPU batch size: 8
    • GoRA HP $N$: 8 (64 reported in our paper, where the per-GPU batch size is 1)
    • Total number of samples for calculating gradients during GoRA initialization: 8 * 8 * 8 = 512
    • No gradient checkpointing or accumulation
  • Other configurations (including FlashAttention) match original paper settings

Q1: Compute Start-up Cost

We sincerely thank the reviewer for this insightful suggestion regarding initialization overhead. To address the computational cost of N full forward-backward passes, we have conducted additional exploration by implementing an adaptive convergence detection algorithm. This innovation automatically terminates gradient computation once layer-wise importance scores stabilize, significantly improving efficiency:

Impact:

• Full training time: 36min
• Time saved: 7.5% total reduction
• Accuracy maintained (±0.25 difference)

Q2: CPU Off-load Bottleneck

We thank the reviewer for this important observation. Our results indeed show the CPU offload overhead scales with GPU count:

Solution: Implemented optimizations (detailed in Q3) to address PCIe bottleneck.

Q3: Host RAM Usage

We greatly appreciate the reviewer's valuable suggestions concerning memory efficiency. In response, we have implemented additional architectural optimizations that specifically target these memory concerns, yielding significant improvements:

Optimizations Implemented:

1. Memory-Saving Techniques:

   • Layer-wise immediate reduction with inter-GPU all-reduce
   • Single-copy storage (rank-0 only) in CPU memory
   • Rank-0 importance computation with broadcast synchronization

2. Performance Improvements:

   • CPU memory: 8× → 1× (16-bit targeted modules' size)
   • PCIe bandwidth: Reduced via 8→1 offload + broadcast
   • Initialization time improvements:
     • 8-GPU: 3m46.98s → 34.52s (4-5s per step)
     • 8-GPU + early stop: → 20.32s

We thank the reviewer again for these important suggestions, we will add a section and re-write the algorithm in the revised version of our paper.

Q4: Hyperparameter Sensitivity

We greatly appreciate the reviewer's highlighting of HP sensitivity. To minimize manual tuning, we propose these automated strategies:

1. Rank bounds (r_min/r_max):
   • Auto-setting: r_max = 4×r_ref, r_min = 0.5×r_ref
   • Guarantees: Maintains comparable parameters to LoRA (Table 7)
   • Verification: Best performance across settings we tested (Table 5)
2. Initialization steps (N) (new!):

   • Dynamic convergence:

     a) Monitors inter-layer importance variation

     b) Early stops when <1% change for 2 steps

   • Outcome: Avg. steps reduced from 8→3 (2.8× faster)

3. Scaling factor (γ) (new!):

   • Auto-selection:

     a) Tests γ candidates ∈[1e-1,1e-5] during initialization

     b) Selects γ minimizing first-step training loss

   • Overhead: about 30s

   • best γ searched for CodeFeedback: 0.0549 (5e-2 reported in the paper)

   • best γ searched for MetaMathQA: 0.0858 (8e-2 reported in the paper)

Performance Impact:

Key Takeaway: Automated approaches maintain performance (<±0.5 GSM8K, <±3 HumanEval) while reducing tuning effort.

Q5: Pseudo-inverse Cost

Clarification: All pseudo-inverse computations are GPU-based with minimal overhead:

Below are initialization times for an 8B model across different ranks, measured using the following GPU-optimized implementation:

    def grad_compress_init(self, 
                      lr: float, 
                      scaling_by_lr: bool = False,):
        if not hasattr(self.weight, 'grad_stored'):
            return
        
        # Convert weight_a to float32 on correct device
        origin_weight_dtype = self.weight.dtype
        weight = self.weight.to(torch.float32)
        weight_dtype = weight.dtype
        weight_device = weight.device
        grad_stored = self.weight.grad_stored.to(weight_dtype).to(weight_device)
        self.weight_a = nn.Parameter(torch.empty((self.lora_rank, self.in_features), dtype=weight_dtype, device=weight_device))

        AT = self.weight_a.T
        AAT = torch.matmul(self.weight_a, AT)
            
        AAT_inv = torch.linalg.pinv(AAT + 1e-8 * torch.eye(self.lora_rank).to(weight_device)).to(weight_dtype)
        AAT_inv_AT = torch.matmul(AT, AAT_inv)

        # Compute weight_b using grad_stored (convert to float32 for computation)
        weight_b_data = torch.matmul(grad_stored, AAT_inv_AT)

        if scaling_by_lr:
            stable_gemma = (lr / math.sqrt(self.lora_rank/self.in_features)) * (self.scale_rank)
        weight_b_data *= (stable_gemma / self.scaling_alpha)

        self.weight_b = nn.Parameter(weight_b_data.contiguous())

Key Observations:

• Even at rank 128 (requiring ≈2M FLOPs per inverse), the total initialization adds <4s overhead.

• This represents <0.1% of typical training time, making the cost negligible in practice.

We will clarify these implementation details and include benchmarking results in the paper. The GPU-based design ensures scalability, and we find no practical bottleneck from pseudo-inverse computations.

Conclusion

We sincerely thank the reviewers for these valuable suggestions. We will:

1. Add a detailed section on computational optimizations
2. Rewrite the algorithm description
3. Clarify pseudo-inverse costs
4. Include all experimental results in the revised paper

We sincerely appreciate the time and effort Reviewer 4NWP has dedicated to evaluating our work. We are particularly grateful for the insightful suggestions regarding the communication/computation costs and HP sensitivity of GoRA.

These constructive comments have guided us in further optimizing GoRA’s algorithm, significantly reducing initialization cost and HP tuning cost overhead—especially in multi-GPU scenarios. We hope our revisions have adequately addressed the concerns raised, and we would be happy to provide additional clarification or discuss any further questions the reviewer may have.

Thank you once again for your valuable feedback.