From Weight-Based to State-Based Fine-Tuning: Further Memory Reduction on LoRA with Parallel Control

Thank you for your thoughtful and positive comments. We would like to clarify on a few points to address the concerns raised.

1.Discussion about the difference between the current work and the pioneering work (Zhang et al., 2024b).

We agree with the reviewer that it is important to clearly highlight the differences between our approach and previous work. To address this, we would like to include the following discussion in the revised version.

In general, the differences between the current work and the pioneering work lie in two aspects.

(1) Different Aspects of Control Theory. The previous work primarily focuses on the controllability aspect of control theory, aiming to understand how system parameters can be adjusted to influence system behavior. In contrast, our work introduces the concept of state-based tuning, drawing inspiration from feedback control systems. Rather than focusing on the controllability of the system, our method emphasizes how adjusting the system's state (as opposed to directly manipulating model weights) can lead to improved performance. This distinction highlights that these two papers target different areas of control theory.

(2) Memory and Computation Efficiency. The most notable difference lies in the memory and computation efficiency. The prior approach enhances the LoRA method by introducing additional cross-patch attention mechanisms to modify the low-rank matrices. While this enhances the model's complexity, it also leads to increased memory consumption and training time due to the added operations and parameters. In contrast, our primary goal is to achieve memory efficiency by reducing the memory footprint, specifically by bypassing large intermediate states during model execution.

2. Control theory is like theoretical embellishment.

We notice the reviewer’s concern regarding the role of control theory in our work and would like to clarify its necessity and impact.

(1) Bridging PEFT and State-Based Control. While PEFT methods like LoRA primarily focus on tuning model weights, classical control theory is centered around adjusting system states, as seen in feedback control. These two perspectives may initially seem unrelated, but our key argument is that LoRA can be understood as a special case of state-based fine-tuning. This reframing is not just a theoretical exercise—it provides a new lens for understanding and designing PEFT algorithms. This allows our framework to establish a direct connection between PEFT and state-based control in deep NNs. This bridging alone is the key message that authors want to convey in the first part.

(2) Unlocking New Opportunities. Control theory offers a wealth of results that could inspire and inform the study of PEFT algorithms. By framing LoRA within a control-theoretic perspective, we naturally arrive at several open research questions. For example, an essential question arises: Which states should we tune? This corresponds to selecting the appropriate edges on the computational graph, analogous to choosing control variables in feedback systems. Many well-established results from control theory could potentially be applied here, but they have not been fully explored in the PEFT community.

Thus, the integration of control theory is not mere embellishment; it serves as a foundation that opens new avenues for both theoretical and practical advancements. We will ensure that this motivation is articulated more clearly in the revised version.

Distinguishing the Proposed Method from LoRA

The second part of our paper provides a shift where we reposition the low-rank matrices. We acknowledge that our approach is simple and appears structurally similar to LoRA, yet the underlying formulation corresponds to distinct control frameworks.

(1) Different Forms of Control Systems. From a control-theoretic standpoint, LoRA and DoRA can be viewed as non-affine control mechanisms:

$\dot{x}(t) = f(x(t), u(t))$,

where the control is deeply entangled with the model’s attention mechanisms. In contrast, our proposed parallel control formulation aligns more closely with affine control:

$\dot{x}(t) = f(x(t)) + x(t) u(t)$,

where the control component is decoupled from the function f. This distinction is crucial—affine and non-affine controls represent fundamentally different classes of control design, each with unique stability and efficiency properties. Our work leverages this difference to optimize memory consumption.

(2) Practicality of the Proposed Algorithm. Beyond above considerations, a key advantage of our method is its simplicity and ease of integration. By making minimal modifications to existing architectures, our approach remains practical and can be readily applied to various model designs.

3. More experimental results if using other units like a total block.

Please kindly refer to our response to Reviewer 1 (hdej).

Thank you for your valuable suggestion.

1. The idea of state-based fine-tuning seems to apply the LoRA from the QKV matrix to the FFN/ATTN block

We understand the reviewer’s concern that our contribution may seem limited to proposing an algorithm, and we would like to clarify on this.

First of all, our goal of proposing the State-FT method is to provide a new viewpoint to link the LoRA method with control approaches. While LoRA is typically framed as a low-rank fine-tuning method that modifies specific model weights, this contrasts sharply with classical control theory, which focuses on adjusting system states, as seen in feedback control. These two areas have largely developed independently. A central argument of our work is that LoRA can, in fact, be viewed as a special case of state-based fine-tuning. As noted by Reviewers hdej and uZCq, our framework aims to establish a direct connection between parameter-efficient fine-tuning and control formulations in deep neural networks. This is the key message we want to deliver when establishing the state-based FT.

Such a viewpoint also paves the way for new algorithmic design. We propose to treat fine-tuning neural networks as modifications to its computational graph (DAGs), allowing us to rethink tuning as a structural modification of the network itself. This means we can introduce or remove specific edges within the computation graph based on task-specific requirements, rather than being limited to conventional weight adjustments. For example, we can add edges that may not exist on the original network, to alter the computation flowmodifying the computation flow—something not captured by the original LoRA method. This viewpoint raises several open questions within the state-based tuning paradigm. For instance, which states should be adjusted within the network, and how can we determine the optimal control strategy? There are many well-established results from control theory, and we would expect them to be applied to finetuning research.

Finally, we introduce an algorithm that tunes entire blocks as a unified unit. Our design philosophy is straightforward: prioritize simplicity and effectiveness. By making the whole architecture simple and making minimal changes, we expect our approach remains broadly applicable across various network architectures. In the meanwhile, we would expect the algorithm to be effectively reduce the GPU memories, as demonstrated in the following experiments.

In summary, we want to emphasize that our goals would extend beyond one specific algorithm—we aim to bridge weight-based fine-tuning with the well-established principles of state-based control theory. This is the primary contribution of this paper. This perspective not only offers a fresh way to interpret LoRA but also raises several open questions, paving the way for future exploration. We will make these points clearer in the updated version.

2. The empirical results show some improvements in memory efficiency and training time, but the performance gains are marginal as 0.12%.

As our title suggests, the core objective of our method is to minimize memory consumption and computational overhead, making it an efficient solution. We demonstrate that our approach reduces memory usage from 18.010GB to 12.280GB on the ViT toy model, achieving a 31.8% reduction.

The performance evaluations we include serve a specific purpose: to demonstrate that these efficiency gains do not come at the expense of model quality. We hope the reviewer can understand that the 0.12% difference is not intended to suggest a significant improvement over the baseline. Rather, its goal is to convey another message—that our method enhances efficiency without sacrificing effectiveness.

3. Strong theoretical foundation to justify the state-based tuning framework.

Overall, we hope the reviewer can understand that this paper is not intended to be a theoretical paper. Instead, the analysis is mainly to support our claims. As alluded to earlier, the primary objective of our algorithm is to demonstrate that improvements in memory and computational efficiency do not come at the cost of model performance. Therefore, our analysis is centered on this key point.

For linear cases, Theorem 4.1 establishes that the parallel control method retains the same expressive power as LoRA, provided both methods share the same total rank. This guarantees that adopting parallel control does not lead to a loss in expressiveness. For nonlinear cases, Theorem 4.2 further demonstrates that parallel control can offer greater adaptability, particularly in scenarios where the original model exhibits degeneracies. These results reinforce the robustness of our approach across different settings.

If the reviewer has any additional concerns regarding necessary analyses, we would greatly appreciate the feedback and are willing to address them accordingly.

We sincerely thank the reviewer for your positive and insightful feedback.

1. Quantization Method like QLoRA/QA-LoRA. Can State-FT use QLoRA method independently?

Yes, the State-FT method can independently leverage the QLoRA/QA-LoRA approach. GPU memory usage mainly arises from three sources: model weights, forward states, and backward gradients. QLoRA and QA-LoRA reduce memory by quantizing model weights and using low-rank matrices to minimize backward gradients. In contrast, State-FT focuses on reducing forward states and backward gradients. These methods are complementary, addressing different sources of memory cost.

State-FT can be combined with the QLoRA method. This allows control to be operated on a quantization model, hence reducing memory consumption across all three categories. To validate this, we evaluate on the commonsense dataset with 8-bit LLaMa3-8B model.

As shown, the use of QLoRA further reduces the memory consumption of the double-control approach by an additional 4.38 GB. However, this reduction in memory comes with trade-offs: training time increases by 91.40% and accuracy drops by 0.4%.

We further investigated the effects of quantization with 4-bit methods on the RTE dataset.

(1) As shown above, 4-bit quantization techniques effectively reduce GPU memory consumption. (2) Like its full-precision counterpart, Q-Control offers slight gains in memory, training time, and accuracy by reducing forward state memory, though overall savings remain modest. (3) Similar to 8-bit, this reduction comes at a cost—Q-Control incurs a 3.48% accuracy drop and an 81.61% increase in training time on the RTE dataset.

In summary, the proposed state-FT can also benefit from methods like QLoRA. Quantization can effectively further reduce memory consumption for control and other PEFT methods, but comes with trade-offs, such as increased training time and slight performance degradation.

2. Sensitivity Study of Rank Settings.

We conduct a sensitivity analysis on the RTE dataset by varying the control rank to evaluate its impact on GPU memory usage, training time, and accuracy.

As the rank of the control parameters increases, GPU memory usage remains relatively stable, with only a slight rise. This is due to the use of low-rank matrices, which help reduce the memory consumption of backward gradients. For accuracy, we observe a steady improvement up to a rank of 32. However, at a rank of 64, performance slightly declines compared to rank 32, indicating diminishing gains beyond a certain threshold.

Next, we compare the performance of Control, LoRA, and DoRA across different ranks:

(1) At lower ranks (r=8), LoRA/DoRA achieve higher accuracy than the control method. (2) As the rank increases (r=16 and above), the control method begins to outperform both LoRA and DoRA. (3) Given that increasing rank to 16 or 32 has a minimal impact on training time and memory usage, we recommend using a relatively high rank to maximize performance. (4) This trend is also observed on CoLA and SST-2 datasets.

3. Large Models like 13B, and Different Domains

We expect State-FT to achieve similar memory reductions on larger models like 13B. Training these models may take several days or even weeks, exceeding the rebuttal period. We are actively working on these experiments.

Below, we present preliminary results from a multi-task evaluation with VL-BART:

In terms of efficiency, the control method achieves a 31.6% reduction in memory usage and a 65.3% reduction in computation time compared to DoRA.

4. Essential References Not Discussed

We notice [1] is a relevant paper and will include it in our reference list.

[1]AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. ICLR 2023.

We sincerely thank reviewer for the insightful and constructive feedback.

1. Sensitivity to the Choice of Controlled Blocks

Controlling either the MLP or attention layer yields comparable performance, as demonstrated by the ViT model results:

These results indicate that controlling either the MLP or attention layer individually leads to similar performance, with attention control slightly reducing accuracy and increasing GPU memory usage. In contrast, controlling the entire residual block results in a more notable 0.43% performance drop. This is likely because each ViT layer consists of two distinct residual blocks, and treating the entire block as a single unit reduces its effectiveness.

To further examine the effects of tuning different portions of the MLP, we selectively tune either the first six or the last six layers of the ViT model while doubling the rank to maintain the same parameter number.

These results highlight that selectively tuning a subset of the MLP layers leads to a more pronounced drop in accuracy. Specifically, tuning only the first six layers results in a 0.82% accuracy reduction. On the other hand, tuning only the last six layers leads to a more substantial memory reduction, as forward states for the first six layers no longer need to be stored. However, this comes at the cost of a significant 4.73% accuracy drop.

While these experiments are conducted on the ViT model, we observe a similar trend across other architectures. These ablation studies will be added to the appendix.

2. Combining two sets of control edges.

If two sets of edges share the same starting and ending nodes on a pretrained model, their combination can be conducted as existing LoRA merging techniques, such as ZipLoRA [1], K-LoRA [2] and SVD methods [3-4]. For example, given two sets of additional edges with weights $\Delta W_1$ and $\Delta W_2$, we can apply a similar approach as [1] and optimize $\ell_{\text{merge}}$ with $\Delta W = v_1 \circ \Delta W_1 + v_2 \circ \Delta W_2,$ where $v_1$ and $v_2$ are vectors, and $\circ$ denotes element-wise multiplication, such that the j-th column of $\Delta W$ is scaled by the j-th element of v. The purpose of this lightweight tuning is to mitigate potential conflicts between different sets of low-rank matrices.

If the two edges have different starting and ending nodes, they can be incorporated into the computation graph independently. However, two key considerations must be addressed. (1) Preserving the Directed Acyclic Graph (DAG): It is essential to ensure that the newly introduced edges do not violate the DAG structure. For example, if the edges include both $A \to B$ and $B \to A$, a cycle would be introduced, requiring structural trimming. (2) Resolving Conflicts: Different sets of edges may result in conflicting weight updates. To optimize performance, lightweight tuning remains a recommended step.

3. Compatibility with Gradient Checkpointing and Memory-Reduction Methods.

Yes, we use gradient checkpointing when training 7B/8B models to reduce memory. For other methods like quantization, please kindly refer to our response to Reviewer 2 (uZCq).

4. Overhead During Inference and Deployment on Standard Inference Platforms.

Extra control edges may lead to a slight increase in inference time. In the worst-case scenario, such as the double-control approach with two extra edges, our experiments show that the extra inference time is around $2.1\% \pm 0.3\%$ extra inference time for RoBERTa-base, and $4.6\% \pm 0.5\%$ for the LLaMA2-7B model.

The introduction of control edges primarily impacts the forward pass by modifying the computation graph. Despite this, our method remains compatible with standard inference platforms like HuggingFace Transformers and Accelerators. While certain platform-specific optimizations, such as quantization, may require some adaptation, we do not anticipate significant changes.

5. Pseudocode and How to Implement Skipping Intermediate State.

In general, steps to skip big intermediate states include:(1) Inherit the original layer and insert low-rank matrices; (2) Define a control function to compute the controlled state $x_c$; (3) Add $x_c$ to the original output.

We will include the pseudocode in the appendix to provide further clarity and would like to thank you once again for this valuable suggestion.

[1] ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs. ECCV 2024.

[2] K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAs. ArXiv 2025.

[3] Task Singular Vectors: Reducing Task Interference in Model Merging. Arxiv 2024.

[4] Model merging with SVD to tie the Knots. ICLR 2025.