Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?

Weakness1 & Question1: Theoretical explanation of similarity between low-rank and full-rank scaling factors.

Thanks for your advice. As advised, we provide a more rigorous theoretical explanation of similarity between low-rank and full-rank scaling factors. In this theorem, we prove that the expected value of the ratio between full-rank and low-rank scaling factors is a constant, which demonstrates their similarity.

Due to time constraints, we only present the final conclusion along with a brief proof. In the final version, we will further refine the proof details, complete the theoretical proof, and attempt to relax the conditions to derive more general conclusions.

Theorem. (Similarity between low-rank and full-rank scaling factors). $G_t \in \mathbb{R}^{m \times n}$ is the full-rank gradient where $m\le n$, $P_t \in \mathbb{R}^{r \times m}$ is the projection matrix, $R_t=P_tG_t \in \mathbb{R}^{r \times n}$ is the projected low-rank gradient, where $r$ is the rank, and $\psi$ is the gradient correction function of the optimizer (e.g., Adam). The expected ratio of full-rank scaling factor $s_g=\frac{||\psi(G_t)||}{||G_t||}$ to low-rank scaling factors $s_r=\frac{||\psi(R_t)||}{||R_t||}$ is a constant $c$:

$\mathbb{E}[\frac{s_g}{s_r}]=\mathbb{E}[\frac{||\psi(G_t)||}{||G_t||} \cdot \frac{||R_t||}{||\psi(R_t)||}]=\mathbb{E}[\frac{||R_t||}{||G_t||}]\cdot\mathbb{E}[\frac{||\psi(G_t)||}{||\psi(R_t)||}] = c.$

Proof:

As noted in GaRare [1]、FLoRA [2], the impact of projection matrix $P_t$ (sampled randomly from a normal distribution) on the final results is minimal. For the sake of simplifying the proof, we therefore assume $P_t \sim \mathcal{N}(0, \sigma^2)^{r \times m}$.

For the first part $\frac{||R_t||}{||G_t||}$, it is equal to:

$\frac{||R_t||}{||G_t||}=\sqrt{\frac{||P_tG_t||^2}{||G_t||^2}}=\sqrt{\frac{||G_t^{\top}P_t^{\top}P_tG_t||^2}{||G_t||^2}}.$

For $(P_t^{\top}P_t)[i,j]$ where $i\neq j$，since all entries of $P_t$ are mutually independent:

$\mathbb{E}[(P_t^{\top}P_t)[i,j]]=0.$

For the case where $i=j$, $(P_t^{\top}P_t)[i,i]$ follows a chi-squared distribution, with an expected value of:

$\mathbb{E}[(P_t^{\top}P_t)[i,i]]=\sum_{k=1}^r P_{k,i}^2=r\sigma^2.$

From this, we derive: $\mathbb{E}[P_t^{\top}P_t]=\sum_i \mathbb{E}[P_t^{\top}P_t[i,i]]=r\sigma^2\mathbb{I}_m.$

Thus: $\mathbb{E}[\frac{||R_t||}{||G_t||}]=\mathbb{E}[\sqrt{\frac{||G_t^{\top}P_t^{\top}P_tG_t||^2}{||G_t||^2}}]=\sqrt{\frac{r^2\sigma^4||G_t||^2}{||G_t||^2}}=r\sigma^2.$

For the second part $\frac{||\psi(G_t)||}{||\psi(R_t)||}$, it is equal to: $\frac{||\psi(G_t)||}{||\psi(R_t)||}=\frac{\sqrt{||\frac{M_t^G}{\sqrt{V_t^G}}||^2}}{\sqrt{||\frac{M_t^R}{\sqrt{V_t^R}}||^2}}=\frac{\sqrt{\sum^m_{i=1}\sum^n_{j=1}(\frac{M_t^G}{\sqrt{V_t^G}})^2[i,j]}}{\sqrt{\sum^r_{i=1}\sum^n_{j=1}(\frac{M_t^R}{\sqrt{V_t^R}})^2[i,j]}},$ where $M_t$ and $V_t$ are the first and second moments.

Due to the element-wise division between $M_t$ and $V_t$, direct decomposition becomes challenging. To address this, we adopt a conclusion from Adam-mini [3]: the adaptive learning rates associated with $V_t$ in the Adam optimizer contain significant redundancy. Specifically, the $V_t$ values within a parameter block can be replaced by a single, well-chosen $V_t$.

Based on this, we assume that different variances within a matrix can be replaced by the average of all $V_t$ values:

$\frac{||\psi(G_t)||}{||\psi(R_t)||}=\sqrt{\frac{\frac{\sum_{i=1}^{r} \sum_{j=1}^{n} V^R_t[i,j]}{rn}}{\frac{\sum_{i=1}^{m} \sum_{j=1}^{n} V^G_t[i,j]}{mn}}\cdot \frac{||M_t^G||^2}{||M_t^R||^2}}=\sqrt{\frac{m}{r}\cdot\frac{\sum_{i=1}^{r} \sum_{j=1}^{n} V^R_t[i,j]}{\sum_{i=1}^{m} \sum_{j=1}^{n} V^G_t[i,j]}\cdot \frac{||M_t^G||^2}{||M_t^R||^2}}.$

Assume that $beta_1$, $beta_2$ are exponential decay rates，for $\frac{||M_t^G||}{||M_t^R||}$，we have：

$\frac{||M_t^G||}{||M_t^R||}=||\frac{(1-\beta_1)\sum_{k=0}^{t-1}\beta_1^kG_{t-k}}{(1-\beta_1)\sum_{k=0}^{t-1}\beta_1^kR_{t-k}}||.$

For each term, result is similar to the derivation of the first part $\mathbb{E}[\frac{||R_t||}{||G_t||}]=r\sigma^2$. Thus:

$\mathbb{E}[\frac{||M_t^G||}{||M_t^R||}]=\frac{1}{r\sigma^2}.$

Moreover:

$\frac{\sum_{i=1}^{r} \sum_{j=1}^{n} V^R_t[i,j]}{\sum_{i=1}^{m} \sum_{j=1}^{n} V^G_t[i,j]}=\frac{\sum_{i=1}^{r} \sum_{j=1}^{n} (1-\beta_2)\sum_{k=0}^{t-1}\beta_2^kR_{t-k}[i,j]^2}{\sum_{i=1}^{m} \sum_{j=1}^{n} (1-\beta_2)\sum_{k=0}^{t-1}\beta_2^kG_{t-k}[i,j]^2}=\frac{(1-\beta_2)\sum_{k=0}^{t-1}\beta_2^k||R_{t-k}||^2}{(1-\beta_2)\sum_{k=0}^{t-1}\beta_2^k||G_{t-k}||^2}.$

By the same logic, for each term:

$\mathbb{E}[\frac{||R_{t-k}||}{||G_{t-k}||}]=r\sigma^2.$

From this, we derive: $\mathbb{E}[\frac{\sum_{i=1}^{r} \sum_{j=1}^{n} V^R_t[i,j]}{\sum_{i=1}^{m} \sum_{j=1}^{n} V^G_t[i,j]}]=r^2\sigma^4.$

Thus:

$\mathbb{E}[\frac{||\psi(G_t)||}{||\psi(R_t)||}]=\sqrt{\frac{m}{r}\cdot r^2\sigma^4 \cdot \frac{1}{r^2\sigma^4}}=\sqrt{\frac{m}{r}}.$

Given that $\mathbb{E}[\frac{||R_t||}{||G_t||}]=r\sigma^2$，we have：

$\mathbb{E}[\frac{s_g}{s_r}]=\mathbb{E}[\frac{||\psi(G_t)||}{||G_t||} \cdot \frac{||R_t||}{||\psi(R_t)||}]=\mathbb{E}[\frac{||R_t||}{||G_t||}]\cdot\mathbb{E}[\frac{||\psi(G_t)||}{||\psi(R_t)||}]=r\sigma^2\sqrt{\frac{m}{r}}.$

It follows that $\mathbb{E}[\frac{s_g}{s_r}]$ is a constant.

Weakness2: Including other memory-efficient optimizers baselines.

Thanks for your advice. As advised, we add comparisons with additional memory-efficient optimizers, including Adam-mini [3], SlimAdam [4].

Table R1: Validation perplexity ($\downarrow$) for pre-training LLaMA 60M on C4 dataset.

As shown in Table R1, Fira outperforms these baselines.

We will include more memory-efficient optimizers baselines in the final version.

Weakness3: Testing on additional architectures.

Thanks for your advice. As advised, we conduct additional experiments on more architectures (Gemma-7B, Mistral-7B) to improve the generality.

Table R2. Comparison results on MMLU tasks.

Results in Table R2 show that Fira remains competitive compared to baselines across different architectures.

Due to time constraints, further experiments (e.g., pretraining experiments on more architectures) will be included in the final version.

Question2: Memory overhead of the residual term $G_t - P_tR_t$.

Thanks for your comments. The residual term $G_t - P_tR_t$ is computed and used within the same update step and does not need to be stored for other steps. Thus, it introduces no additional memory overhead compared to GaLore.

Question3: Use findings in [1] to explain Fira’s superior performance.

Thanks for your advice. We agree that the findings in [1] help explain Fira’s superior performance over full-rank training in some settings, as shown in Figure 5. Specifically, [1] analyzes and compares the loss landscapes of low-rank and full-rank setting，and then demonstrates that low-rank constraints improve the optimization landscape by avoiding sharp minima, which aligns with Fira’s results.

Although GaLore's low-rank constraints enable a better optimization landscape, they discard significant gradient information outside the subspace, leading to suboptimal results. In contrast, Fira leverages low-rank constraints for a better optimization landscape while effectively capturing gradient information outside the subspace, resulting in superior performance.

We will explicitly cite [1] and further discuss with this work to clarify its impact on Fira's results in the final version.

Limitations:

1. The method has only been evaluated on LLaMA architectures, and its generalization to other popular LLM architectures (e.g., GPT, OPT, Mistral) remains unverified.

See response to Weakness3.

2. While the paper demonstrates strong results on LLaMA-7B, larger models have not been tested. It is unclear how Fira performs at even greater scales (e.g., 13B or 65B), where memory and stability challenges may differ.

Thanks for your comments. Testing Fira's performance at even greater scales (e.g., 13B or 65B) requires significant computational resources. Due to time constraints, we will conduct experiments at greater scales to test our Fira in the final version.

[1] On the Optimization Landscape of Low Rank Adaptation Methods for Large Language Models, ICLR'2025
[2] Flora: Low-rank adapters are secretly gradient compressors, ICML'2024
[3] Adam-mini: Use Fewer Learning Rates To Gain More, ICLR'2025
[4] When Can You Get Away with Low Memory Adam?

Thanks for your valuable feedback! All suggested improvements will be developed and carefully incorporated into the final version.

Weakness1: Real application evaluation is limited: pre-training uses only the C4 corpus and runs 10 K steps on the 7 B model, so long-horizon stability is untested.

• Datasets for pre-training.

Thanks for your advice. Following the experiment setup in GaLore [1], we used the C4 dataset for pre-training. Due to time constraints, we will include pre-training experiments with additional datasets in the final version to further validate our approach.

• LLaMA 7B pre-training.

Thanks for your advice. To clarify, pre-training LLaMA 7B for 150K steps requires significant computing resources. For example, GaLore utilized 8-node training in parallel with a total of 64 A100 GPUs to complete the LLaMA 7B 150K-step pre-training experiments. If we use 8 A100s GPUs to complete them, it may take more than 2 months.

At the same time, most of the concurrent work, e.g., Grass [2], FRUGAL [3], were unable to complete the pre-training of LLaMA 7B for 150K steps.

As advised, we will conduct experiments with longer steps for pre-training LLaMA 7B in our future work to test the long-horizon stability.

Weakness2: Memory footprint evaluation seems limited to the optimizer part.

Thanks for your comments. To clarify, in Tables 3 and 4 in our manuscript, our memory footprint evaluation is not limited to the optimizer part, but includes memory footprint evaluation of total parameters, gradients, and optimizer states.

Weakness3: Fira is heavily based on Galore’s approach.

Thanks for your comments. While both Fira and GaLore target memory-efficient LLM training, they differ fundamentally: GaLore is restricted in low-rank gradient subspaces, thus completely failing to utilize the gradient information outside the subspace. However, Fira consistently employ full-rank gradients to train full-rank weights, achieving nearly unaffected performance despite reduced rank (PPL($\downarrow$) 31.06→32.59) compared to GaLore's severe performance degradation (PPL($\downarrow$) 34.88→56.57).

Moreover, Fira is built upon a profound observation regarding the similarity of scaling factors between full-rank and low-rank training. It represents the first attempt of enabling consistent full-rank training under low-rank constraints. This breakthrough not only improves the memory-performance trade-off but also offers new insights for efficient LLM training.

Question1: Does it make sense to compare against system-level baselines such as ZeRO-offload in [23] or quantized optimizers?

Thanks for your comments. To clarify, our method Fira and system-level baselines (like ZeRO-offload [23] or quantized optimizers) achieve memory efficiency memory efficiency from distinct perspectives. Our method, Fira, achieves memory efficiency at the algorithm level, which is orthogonal to these system-level techniques. In fact, Fira can be easily combined with them to further boost memory efficiency. For this reason, direct comparisons between them may not be entirely fair.

Question2: Is it fair to say that the method reduces the optimizer memory but activation and parameter memory can still dominate the total memory footprint?

Thanks for your comments. To clarify, in LLM pre-training, especially when the number of parameters is very large, optimizer memory is often more critical than activation and parameter memory.

For example, as mentioned in the Introduction in our manuscript, pretraining a LLaMA 7B model with FP16 using Adam requires 14GB for parameters, 24GB for optimizer states (2x the parameters memory), and only 2GB for activations. In this case, optimizer memory is significantly larger than activation and parameters memory. This effect becomes even more pronounced as model size increases.

Question3: Downstream impact is somehow limited but it seems enough to demonstrate the applicability of the approach. What limitations do you forecast on vision or SFT or RLHF?

Thanks for your comments. As noted, we forecast the limitations when extending Fira to vision.

Compared to LLMs, vision involves diverse modalities (e.g., images, videos, point clouds) and tasks (e.g., generation, detection, tracking). These differences may lead to distinct loss landscapes between LLMs and vision models. Thus, Fira may not work directly for vision. We would need to re-evaluate the phenomena and conclusions observed in LLM training, and adjust the method based on vision-specific characteristics.

We will explore this point in our future work.

Question4: How similar is conceptually and its effect the norm-growth limiter to gradient-clipping?

Thanks for your comments.

Conceptually, the norm-growth limiter and gradient-clipping both address gradient instability but differ in focus. The norm-growth limiter adaptively controls the relative increase magnitude of the gradient norm, accounting for differences across different weight matrices. In contrast, gradient-clipping clip the gradients based on their absolute norm, setting an upper limit to avoid overly large gradients.

In terms of effect, as shown in our ablation study (Table 5 in our manuscript), the norm-growth limiter outperforms gradient-clipping.

[1] GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection, ICML'2024
[2] Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients, EMNLP'2024
[3] FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training, ICML'2025

Thanks for your valuable feedback! All suggested improvements will be developed and carefully incorporated into the final version.

Weakness1: Details about motivation for scaling.

Sorry for confusing you. Below, we provide clarified details on the comparison between scaling factors for the low-rank and full-rank gradients.

1. Are the scaling factors for the low-rank and full-rank gradients both computed within the same training run?

No, they are computed within different training runs. As shown in Table 1 and Table 10 in our manuscript, we conducted multiple experiments under both low-rank and full-rank settings, computing and collecting the scaling factors independently for each experiment. Then, we averaged these scaling factors to assess their similarity between low-rank and full-rank settings.

2. How is the similarity of scaling factors between low-rank and full-rank settings calculated?

For each training run, we compute column-wise scaling factors separately and combine them into a vector. After multiple experiments, based on two vectors computed under low-rank and full-rank settings, we can calculate their similarity using metrics like cosine similarity.

Weakness2: Fira vs full rank.

Thanks for your comments. Below, we address the concerns raised from two perspectives. We will include a more detailed analysis in the final version.

1. Why does column-wise Fira-only-scaling outperform element-wise Adam?

• Robustness to data noise

As noted in Adam-mini [1] and APOLLO [2], Adam's element-wise scaling contains significant redundancy. Using a shared scaling for a group of parameters, such as column-wise scaling in Fira-only-scaling, is sufficient to replace Adam's element-wise scaling. In this scenario, by aggregating historical gradient information across multiple parameters, column-wise scaling may be more robust than Adam's element-wise scaling when dealing with training data noise.

• SGD-like updates improve generalization

Unlike Adam's element-wise scaling, column-wise Fira-only-scaling preserves original gradient direction within each column, leading to SGD-like updates. In many deep learning tasks, SGD yields better generalization [3, 4]. However, this benefit rarely extends to LLMs, as SGD struggles to train them [5]. As [5] explains, this is because SGD uses a uniform learning rate across all weight blocks, overlooking their inherent differences. However, Fira-only-scaling can take these differences into account by employing adaptive column-wise scaling. This may allow Fira to extend the strong generalization of SGD-like updates to LLMs. A similar perspective is presented in Apollo [2, Section 5.5].

2. Why does Fira (column-wise + element-wise) outperform Fira-only-scaling?

• Low-rank element-wise scaling benefits main gradient components

Prior works [6] find that low-rank element-wise scaling may improve the optimization landscape by avoiding sharp minima. In this case, low-rank element-wise scaling may be more effective than corresponding column-wise scaling, and can be helpful for the main components of the gradient. This may explain the improved performance of combination strategy Fira.

• More precise parameter grouping

While column-wise scaling reduces redundancy in element-wise Adam, it may not be optimal for all cases. For example, parameters may have diverse training dynamics within a certain column, requiring more fine-grained parameter grouping. Here, element-wise scaling may better handle the more fine-grained training dynamics thus complementing column-wise scaling.

Weakness3: Related works and baselines.

Thanks for your advice. As advised, we will cite and discuss Adam-mini, AdaLomo, MicroAdam, SlimAdam, and FRUGAL in related works and add them as baselines in the experiments.

Related Work:

Adam-mini, which changes element-wise Adam to block-wise based on the Hessian structure of neural networks; AdaLomo, which enhances low-memory optimization (LOMO) with adaptive learning rates for better robustness to hyperparameters and convergence; MicroAdam, which compresses gradient information to reduce memory with theoretical convergence guarantees; and SlimAdam, which uses layer-wise SNR analysis to replace second-moment tensors with their means for memory efficiency.

FRUGAL operates almost same as GaLore-add: both directly add the gradients discarded in GaLore, rather than exploring scaling factors for more effective updates of these gradients (as Fira does). But FRUGAL additionally provide theoretical convergence guarantees when using SGDM for subspace updates and SGD for updates outside the subspace.

Baselines:

We conduct additional experiments comparing Fira with Adam-mini, SlimAdam, and FRUGAL. As shown in Table R1, Fira achieves competitive performance against these low-memory training methods.

Table R1: Validation perplexity ($\downarrow$) for pre-training LLaMA 60M on C4 dataset.

Due to time constraints, we will include all mentioned baselines in the final version.

Question1: Writing.

Sorry for confusing you.

1. Motivation for the proposed scaling technique

As advised, we add more details in response to Weakness1 to further explain the motivation for our proposed scaling technique.

2. Inconsistency of the comparison between matrix and column scaling

In our manuscript, matrix scaling computes and applies scaling factors at matrix level, while column scaling operates at individual columns. In Table 5 of our ablation study, we compare Fira (column scaling) and Fira-matrix (matrix scaling), showing that the finer-grained column scaling performs better. Similarly, in Appendix E, Table 11, confirms that Fira-only-scaling (excluding the Adam term) performs better with column scaling than matrix scaling.

Question2: Limited experimental results.

Thanks for your advice. As advised, we conduct additional experiments on more architectures (Gemma-7B, Mistral-7B), more datasets (MMLU), and more adaptive optimizers (Adagrad) to strengthen the generality of our claims. We will add more experiments in the final version.

1. Architectures and datasets

Table R2. Comparison results on MMLU tasks.

As shown in Table R2, Fira remains competitive with the baselines across different architectures and datasets.

2. Adaptive optimizer

Table R3: Validation perplexity ($\downarrow$) for pre-training LLaMA 60M on C4 dataset.

As shown in Table R3, we conduct experiments on addtional adaptive optimizer Adagrad. Fira's strong performance demonstrates its compatibility with other adaptive optimizer.

3. Including full-rank experiments for LLaMA 7B pretraining.

Due to high computational cost for pretraining LLaMA 7B, we will include it in the final version.

Question3: Queations about Fira's memory improvement and throughput decrease.

Thanks for your comments.

1.Memory improvement

Table 3's memory estimation only includes parameters, gradients, and optimizer states. However, Table 8's actual memory estimation includes additional parts, such as activations, memory occupied by systems (e.g., temporary computation buffer), etc. These extra components will reduce the magnitude of the memory improvement in Table 8. Actually, the memory reduction is basically consistent between two tables: Table 3 shows a 3.42GB reduction (10.4GB - 6.98GB), and Table 8 shows a 3.9GB reduction (58.5GB - 54.6GB).

2.Throughput decrease

Compared with GaLore, the decrease in Fira's throughput depends not only on time consumed by additional operations introduced by Fira, but also on time consumed by the shared operations. The longer time of shared operations, the smaller the decrease in Fira's throughput.

Compared to fine-tuning, pre-training's shared operations dominate due to a larger batch size (16 for fine-tuning and 512 for pre-training). This explains why the throughput decrease of Fira is relatively small in pre-training.

Question4: Apply norm growth limiter at the matrix or column level.

Thanks for your comments. Like scaling in Fira, norm growth limiter can also be applied at the matrix or column level. Below are the experimental results comparing matrix-level and column-level norm growth limiters.

Table R4: Validation perplexity ($\downarrow$) for pre-training LLaMA 60M on C4 dataset.

As shown in Table R4, applying the norm growth limiter at the matrix or column level yields nearly identical results, indicating no significant difference between the two strategies.

Limitations.

As advised, we further discuss the reasons behind Fira's benefits in response to Weakness2, and include additional experiments on additional architectures, datasets, and adaptive optimizer in response to Question2.

We will further revise the Limitations section in the final version to provide a more comprehensive discussion of Fira's constraints.

[1] Adam-mini: Use Fewer Learning Rates To Gain More, ICLR'2025
[2] APOLLO: SGD-like Memory, AdamW-level Performance, MLSys'2025
[3] Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning
[4] Improving Generalization Performance by Switching from Adam to SGD, NeurIPS'2020
[5] Why Transformers Need Adam: A Hessian Perspective, NeurIPS'2024
[6] On the Optimization Landscape of Low Rank Adaptation Methods for Large Language Models, ICLR'2025

Thanks for your valuable feedback! All suggested improvements will be developed and carefully incorporated into the final version.

Weakness1: Additional comparison with GaRare [1].

Thanks for your advice. As advised, we compare our method Fira with more recent training approaches, such as GaRare [1], and conduct experiments integrating the gradient random projection proposed in GaRare with our Fira.

Table R1: Validation perplexity ($\downarrow$) for pre-training LLaMA 60M on C4 dataset.

As shown in Table R1, Fira with gradient random projection proposed in GaRare outperforms GaRare thereby demonstrating the improvement in Fira remains effective under gradient random projection. Besides, Fira yields better performance than Fira + gradient random projection, indicating that the original SVD-based gradient projection may be more suitable than random projection under the Fira's framework.

In the final version, we will revise the related work section by adding citations and a discussion of GaRare. Additionally, we will include more experiments and analyses on it to further strengthen the study.

Weakness2 and Question1: Additional quantitative analysis on computational overhead.

• Precise FLOPs

Thanks for your advice. As advised, we additionally measure the precise FLOPs of GaLore and Fira.

Table R2: Precise FLOPs of GaLore and Fira.

• Measured runtime overhead

Thanks for your advice. As advised, we additionally measure the real computational time of optmizaion step.

Table R3: Optimizer step time (in seconds) across LLaMA-1B with a sequence length of 1024 on a single A100 GPU.

As shown in Table R2 and Table R3, compared to GaLore, Fira only slightly increases the computational overhead, which is acceptable.

Question2: Comparisons with other recent low-rank or gradient-efficient methods.

Thanks for your advice. As advised, we additionally elaborate on how Fira compares to other recent low-rank or gradient-efficient methods, e.g., GaRare [1] and FLoRA [2].

Table R4: Pretraining LLaMA models on the C4 dataset. Validation perplexity ($\downarrow$) is reported.

As shown in Table R4, Fira achieves better performance compared to these methods across different model sizes.

[1] On the Optimization Landscape of Low Rank Adaptation Methods for Large Language Models, ICLR'2025
[2] Flora: Low-rank adapters are secretly gradient compressors, ICML'2024