Thank you for your detailed review of our work. Before responding to the weaknesses and questions you raised, please allow us to once again clarify our research background, scenario definition, and the experimental metrics used to measure results.

Background, Scenario Definition, and Key Experimental Metrics

First, please allow us to reiterate our research background, scenario definition, and the experimental metrics used to measure results.

Loquetier is a framework for unified fine-tuning and serving of multiple LoRAs. As mentioned in the Introduction, the research background is that LoRA and its related efficient fine-tuning techniques have demonstrated excellent performance across various specialized tasks and show significant potential in many tasks, such as personalization and user preference-related tasks, leading to their increasingly widespread adoption and promising future applications. However, the hardware resources available for LLM training, fine-tuning, and serving remain extremely limited. This leads to our application scenario: how to improve the efficiency of multi-LoRA fine-tuning and inference tasks under constrained hardware conditions.

Specifically, most existing system frameworks require a hard switch when handling different tasks, meaning they must first stop the current task before starting another. Such switches consume significant time and require saving fine-tuning states and migrating inference requests, which occupy substantial bandwidth and storage space. During downtime, the occupied computational resources cannot be utilized, leading to a significant decline in the system's hardware resource utilization efficiency.

Unifying fine-tuning and inference of multiple LoRAs upon one base model can address the aforementioned issues and improve the efficiency of multi-LoRA fine-tuning and inference tasks. Integrating the two distinct tasks is a practical approach, as the only difference between them lies in the gradient calculation and parameter updates during the backpropagation process. Regardless of the task type, sharing the same base model can significantly reduce memory requirements.

According to the description in the Experiments section, we measure the efficiency of these two tasks as follows, and list the meanings of all experimental metrics in Appendix B:

• The efficiency metric for the fine-tuning task is the throughput of fine-tuning and evaluation, i.e., the number of tokens fine-tuned and evaluated per unit of time, which is equivalent to the time required to fine-tune with the same data and configuration.

• The efficiency metric for the inference task is inference throughput, which is the number of tokens inferred per unit of time. Additionally, serving metrics targets are used to define the availability of the service system, so the secondary metric is the service level objective attainment (rate). Appendix C clarifies the service metric settings, including the maximum request wait time, maximum decoding latency per token, and average decoding latency. By calculating whether each request meets these strict serving metrics, our experimental results have sufficient practical significance.

Latency Impact Caused by Virtualized Module

Due to the objective limitations on the number of LoRAs that can be loaded on each GPU, in practical use, we can pre-create a sufficient number of virtual base models for the maximum number of LoRAs that can be accommodated, along with an equivalent number of empty LoRAs, and perform data copying when actually loading LoRAs. Even if these virtual modules need to be dynamically created, one or two pre-prepared virtual models can always be reserved. In this case, the latency introduced by Virtualized Module can be resolved during the framework startup phase, so this latency does not affect performance and efficiency in latency-critical environments.

Extension to other PEFT methods

As stated in the paper and the first section "Background, Scenario Definition, and Key Experimental Metrics" in our response, Loquetier is a framework for unified fine-tuning and inference task design in multi-LoRA scenarios, rather than overall optimization of various PEFT methods. We use a Virtualized Module mechanism to support the loading of other PEFT models. We also support optimization work for other PEFT methods. For example, if a study has designed an efficient operator for a certain method, we only need to introduce this operator into Loquetier to achieve optimization of this method.

Models and Datasets

We did not propose improvements to the effectiveness for fine-tuning and inference methods, but rather optimizations to the computational processes for multi-LoRA fine-tuning and inference tasks. So there is no need for validating the effectiveness of fine-tuning on vast downstream tasks with F-1, rouge-L, etc. The current dataset is sufficient to evaluate all the experimental metrics mentioned earlier.

In this paper, we made improvements at the kernel level. The industry typically evaluates computational quality through the following two parts of validation, and based on our validation results, Loquetier does not affect the quality of computational results.

1. Verifying whether kernel computational outputs align with standard results: We performed 10,000 kernel operations using random inputs, and the results indicate that no errors were generated in kernel computational results at float16 precision.
2. Verifying whether the loss converges normally during fine-tuning: Based on data generated during the experiment, the loss converges consistently with the PEFT baseline method.

Regarding the dataset used for inference tasks, we applied the same data input fairly to all test baselines. The data used reflects the highest throughput of our framework, and this result is not influenced by the dataset, verifying that the inference task has achieved its objectives. Additionally, we designed a unified task for fine-tuning and inference with dynamic input scenarios within hardware constraints (Section 4.2 - Unified Fine-tuning and Inference). Experimental results show that Loquetier can handle any dynamic input changes under hardware limitations.

Framework Design and Motivation

As mentioned in the paper and the first section "Background, Scenario Definition, and Key Experimental Metrics" in the response, there is a lack of work on multi-LoRA fine-tuning and inference in the context we describe. Therefore, we designed a unified framework for fine-tuning and inference to significantly improve the execution efficiency of these tasks and hardware utilization in scenarios with limited hardware resources, while maintaining good compatibility with other PEFT methods and related optimization work.

Loquetier does not modify user input or output, which is consistent with other related work. Specifically, each request only needs to provide its text content or token sequence, along with the specified LoRA (LoRA ID). The framework will return the generated text content or token sequence sequentially. For fine-tuning requests, we save the obtained model in the specified file path, and immediate fine-tuning results can be obtained by accessing the originally created LoRA model. The data flow is illustrated in Figure 1 (architecture diagram) of the paper. Due to NeurIPS restrictions on images and external links in replies, we cannot reproduce the image here; please refer to the submitted paper.

Scenario and Contribution

As noted in the paper and the first section "Background, Scenario Definition, and Key Experimental Metrics" in our response, our scenario involves fine-tuning and inference tasks using multiple LoRA models under limited hardware resource constraints, with the goal of improving the efficiency of these tasks in such scenarios.

Our memory optimization is achieved through Virtualized Module, a mechanism that enables different LoRA or PEFT models to share the same base model. By loading only one base model, we significantly reduce GPU memory usage in multi-LoRA scenarios. The specific virtualized model architecture is detailed in Figure 1 (architecture diagram) of the paper. Our computational optimization synchronizes computations by loading different data into a single kernel call for input data and various LoRA weights, thereby efficiently utilizing hardware resources to accelerate the computational process.

Our work includes over 1,000 lines of CUDA kernel and its related code, nearly 4,000 (3,893) lines of code supporting kernel design, fine-tuning, inference workflows, and model adaptation, as well as over 1,000 lines of code supporting PyTorch native training. We also made modifications to the kernel calls in FlashInfer to support Llama 3's RoPE and other settings.

Additional Supplement

Please allow us to update the performance of FlexLLM in multiple LoRA inference scenarios. We conducted a smaller-scale test, and the results showed that FlexLLM ran for several hours while repeatedly loading LoRA in a loop for a test that should not have lasted more than 5 minutes—it indeed did not get stuck in an infinite loop. We have not studied its underlying implementation in detail, so we will not speculate on the cause of this phenomenon.

Thank you for your detailed review of our work. Our responses to the weaknesses you raised are as follows:

Baselines

We have received valuable suggestions from the reviewers regarding the evaluation. We would like to provide additional explanations and the derived results to address the issues raised.

For S-LoRA, FlexLLM have compared with it. S-LoRA is only optimized for inference tasks and performs worse than FlexLLM in fine-tuning and inference tasks; FlexLLM's experiments include the Peft baseline for fine-tuning tasks, and our results in similar experimental settings are better than the equivalent fine-tuning performance of Peft in the S-LoRA+Peft baseline in FlexLLM's experiments, while the performance of the Peft is better than that of FlexLLM in several experimental environments (FlexLLM has not tested a direct comparison with Peft when only performing fine-tuning tasks; based on the result in their paper and the number of GPUs used, we can infer that Peft's fine-tuning efficiency is higher than that of FlexLLM under the same hardware resources). Therefore, based on the proxy evaluation method, we can also confirm that our performance is better than that of FlexLLM and S-LoRA. Additionally, S-LoRA significantly outperforms vLLM in its multi-LoRA inference experiments, and its experimental environment is comparable to ours. Furthermore, S-LoRA does not support newer models, including Llama 3. For these reasons, we did not design experiments comparing S-LoRA and vLLM.

As mentioned in the Introduction, there are very few training-inference integrated frameworks that support multi-LoRA. FlexLLM may be the only proper baseline. The other methods mentioned in the Related work are mostly general optimizations for base models or further improvements to the LoRA method itself, and they are not suitable for evaluation as test baselines; our work is also compatible with these methods.

For the performance of FlexLLM in multiple LoRA inference scenarios. We conducted a smaller-scale test, and the results showed that FlexLLM ran for several hours while repeatedly loading LoRA in a loop for a test that should not have lasted more than 5 minutes—it indeed did not get stuck in an infinite loop. We have not studied its underlying implementation in detail, so we will not speculate on the cause of this phenomenon.

Models

Since we are not an enterprise-level organization and are constrained by hardware resources, it is challenging to conduct experiments with larger models. We are willing to analyze the expected results of larger models on multiple GPUs from a theoretical perspective.

The LoRA method itself does not conflict with common parallel strategies, such as pipeline parallelism or tensor parallelism. Therefore, from a technical standpoint, the fine-tuning and inference of LoRA models can utilize these parallel strategies to load larger models on multiple GPUs. In terms of pipeline parallelism, data only needs to be transferred once between the two layers on different devices, so our work fully supports this parallel strategy. We also fully support parallel strategies like tensor parallelism that require data sharing between GPUs:

The forward and backward calculations of LoRA can be expressed by the following formulas:

$V = X @ A, Y = V @ B$

$B' = V^T @ Y', V' = Y' @ B^T, A' = X^T @ V', X' = V' @ A^T$

Data parallelism is typically split along the hidden_states dimension, so the weight matrix needs to be split and stored across multiple GPUs, with the requirement of an all-reduce or all-gather performed after each of the $V, Y, V',$ and $X'$ computations are complete. Loquetier's computation process can be split using all-reduce / all-gather, so we also support these parallelization strategies.

This means we can run larger models in a multi-GPU environment without incurring any additional overhead compared to existing methods.

One point we would like to highlight is that, based on the relevant research we have collected, there has been little targeted optimization for such parallel strategies when using LoRA, especially in fine-tuning tasks, as the LoRA computation itself already has no further room for parallel optimization. Furthermore, for strategies like tensor parallelism, as shown in the above formula, the communication overhead required for each LoRA linear layer to synchronize is $2 \times s \times (r + h) \times n$, where $n$ is the number of GPUs involved in parallelism. Considering the number of LoRA layers and the dimension of hidden_size in large models, the communication overhead is extremely high compared to the benefits, so these parallel strategies is typically not used.

Datasets

We did not propose improvements to the effectiveness for fine-tuning and inference methods, but rather optimizations to the computational processes for multi-LoRA fine-tuning and inference tasks. So there is no need for validating the effectiveness of fine-tuning on vast downstream tasks with F-1, rouge-L, etc. The current dataset is sufficient to evaluate all the experimental metrics mentioned earlier.

In this paper, we made improvements at the kernel level. The industry typically evaluates computational quality through the following two parts of validation, and based on our validation results, Loquetier does not affect the quality of computational results.

1. Verifying whether kernel computational outputs align with standard results: We performed 10,000 kernel operations using random inputs, and the results indicate that no errors were generated in kernel computational results at float16 precision.

2. Verifying whether the loss converges normally during fine-tuning: Based on data generated during the experiment, the loss converges consistently with the PEFT baseline method.

Thank you for your detailed review of our work. Before responding to the weaknesses and questions you raised. We clarify our research background, scenario definition, and metrics used to measure results.

Background, Scenario Definition, and Key Experimental Metrics

Loquetier is a framework for unified fine-tuning and serving of multiple LoRAs. As mentioned in the Introduction, the research background is that LoRA and its related efficient fine-tuning techniques have demonstrated excellent performance across various specialized tasks and show significant potential in many tasks, such as personalization and user preference-related tasks, leading to their increasingly widespread adoption and promising future applications. However, the hardware resources available for LLM training, fine-tuning, and serving remain extremely limited. This leads to our application scenario: how to improve the efficiency of multi-LoRA fine-tuning and inference tasks under constrained hardware conditions.

Specifically, most existing system frameworks require a hard switch when handling different tasks, meaning they must first stop the current task before starting another. Such switches consume significant time and require saving fine-tuning states and migrating inference requests, which occupy substantial bandwidth and storage space. During downtime, the occupied computational resources cannot be utilized, leading to a significant decline in the system's hardware resource utilization efficiency.

Unifying fine-tuning and inference of multiple LoRAs upon one base model can address the aforementioned issues and improve the efficiency of multi-LoRA fine-tuning and inference tasks. Integrating the two distinct tasks is a practical approach, as the only difference between them lies in the gradient calculation and parameter updates during the backpropagation process. Regardless of the task type, sharing the same base model can significantly reduce memory requirements.

According to the description in the Experiments section, we measure the efficiency of these two tasks as follows, and list the meanings of all experimental metrics in Appendix B:

The efficiency metric for the fine-tuning task is the throughput of fine-tuning and evaluation, i.e., the number of tokens fine-tuned and evaluated per unit of time, which is equivalent to the time required to fine-tune with the same data and configuration.

The efficiency metric for the inference task is inference throughput, which is the number of tokens inferred per unit of time. Additionally, serving metrics targets are used to define the availability of the service system, so the secondary metric is the service level objective attainment (rate). Appendix C clarifies the service metric settings, including the maximum request wait time, maximum decoding latency per token, and average decoding latency. By calculating whether each request meets these strict serving metrics, our experimental results have sufficient practical significance.

Motivation and Objectives

Our motivation and objectives are mentioned in the paper and the first section "Background, Scenario Definition, and Key Experimental Metrics" in response. Our work is not an extension of FlexLLM but rather an optimization of unified fine-tuning and inference work in the rapidly growing and highly promising LoRA domain. We also present experimental metrics for evaluating these tasks and compare our results with other works under the same conditions, demonstrating significant improvements achieved by our approach.

Regarding Table 1 in the paper, we naturally use symbols and footnotes to indicate whether the baseline can complete each scenario. We use $\checkmark$ to indicate that it can run in the corresponding scenario, $\triangle$ means it encountered some issues that caused the test to fail but can be achieved by degrading to a more basic method, and $\times$ refers that it cannot run in that scenario.

Regarding the high-priority requirements you mentioned, according to technical reports, official blogs, and other customer cases from leading internet companies and cloud platforms, such differentiated services are typically provided at the application layer, including real-world scenarios involving the scalable use of LoRA. Our work is a unified underlying framework for fine-tuning and inference tasks, designed to efficiently handle incoming requests to the best of its ability. Loquetier is compatible with various cluster management frameworks and business frameworks, so such requirements can be fulfilled through collaboration with these systems. Based on related work involving multi-LoRA or multi-model parallel inference, including FlexLLM which adopts Orca's research findings, no priority differentiation or other differentiated services have been implemented. As mentioned earlier, this is not the responsibility of the underlying framework.

Experimental Details Related to FlexLLM

FlexLLM is the first work in the field of unified fine-tuning and inference for multi-PEFT models. We fully acknowledge its pioneering nature and have never denied its contributions in our paper. FlexLLM still has limitations in the specific scenarios it claims to support, which is precisely why we should use it as a critical baseline for comparison. We were also surprised by FlexLLM's performance in multi-LoRA inference scenarios, but our experimental results indicate that it cannot support multi-LoRA inference in a single runtime instance.

Please allow us to update the performance of FlexLLM in multiple LoRA inference scenarios. We conducted a smaller-scale test, and the results showed that FlexLLM ran for several hours while repeatedly loading LoRA in a loop for a test designed for 5 minutes. We have not studied its underlying implementation in detail, so we will not speculate on the cause of this phenomenon.

We will conduct additional tests to fully demonstrate FlexLLM's performance in multi-LoRA inference scenarios. We will provide feedback on the experimental results as soon as these tests are completed.

Regarding the trade-off between resource efficiency and latency, we will elaborate further from the following two aspects:

As mentioned in the first section, we are more concerned with the performance of multi-LoRA tasks when a large number of requests arrive under limited hardware resources. As stated in your review comments, this differs from FlexLLM's objectives. We have outlined the experimental metrics in Appendix B of the paper and in the first section in our response, and these metrics comprehensively reflect whether we have achieved our stated objectives.

Additionally, we mentioned the issue of differentiated services in the “Motivation and Objectives” section. Since altering the request arrival rate of inference tasks also affects latency, we believe this conclusion also applies to differentiated services.

We have still managed to conduct inference latency evaluations in both single-LoRA and multi-LoRA scenarios to address your concerns. However, our background and scenario definitions are focused on the fine-tuning and inference of multiple LoRA models, so the comparison of single-LoRA inference scenarios is not our primary focus.

In a single LoRA scenario, FlexLLM achieves 60 tokens/s (tps), corresponding to a latency of 17ms, while Loquetier reaches 30tps and 33ms. In a multi-LoRA scenario (4 LoRA models with 1 infer request per LoRA), FlexLLM achieves 38 tps and 105ms, while Loquetier reaches 120 tps and 33ms. The Peft baseline lags significantly behind in both scenarios. To better address the scenarios we are discussing, Loquetier's design approach remains the optimal solution at present.

Baselines

We have received valuable suggestions from the reviewers regarding the evaluation. We would like to provide additional explanations and the derived results to address the issues raised.

For S-LoRA, FlexLLM have compared with it. S-LoRA only optimized for inference tasks and performs worse than FlexLLM in fine-tuning and inference tasks; FlexLLM's experiments include the Peft baseline for fine-tuning tasks, and our results in similar experimental settings are better than the equivalent fine-tuning performance of Peft in the S-LoRA+Peft baseline in FlexLLM's experiments, while the performance of the Peft is better than that of FlexLLM in several experimental environments (FlexLLM has not tested a direct comparison with Peft when only performing fine-tuning tasks; based on the result in their paper and the number of GPUs used, we can infer that Peft's fine-tuning efficiency is higher than that of FlexLLM under the same hardware resources). Therefore, based on the proxy evaluation method, we can also confirm that our performance is better than that of FlexLLM and S-LoRA. Additionally, S-LoRA significantly outperforms vLLM in its multi-LoRA inference experiments, and its experimental environment is comparable to ours. Furthermore, S-LoRA does not support newer models, including Llama 3. For these reasons, we did not design experiments comparing S-LoRA and vLLM.

As mentioned in the Introduction, there are very few training-inference integrated frameworks that support multi-LoRA. FlexLLM may be the only proper baseline. The other methods mentioned in the Related work are mostly general optimizations for base models or further improvements to the LoRA method itself, and they are not suitable for evaluation as test baselines; our work is also compatible with these methods.

Workload Design

As for the metadata you mentioned that FlexLLM created from third-party sources, this was added in the version of the paper submitted in May 2025. When we were working on Loquetier, we referred to the only version available at the time (the first version in Feb 2024). We'd rather compare it to that version, but we are open to further discussion on the points you raised. Please let us know if you would like to discuss this part further.

Thank you to the authors for the great effort during the rebuttal phase. It's encouraging to see empirical results supporting Unified Fine-tuning and Inference.

and the reason for the failure to reproduce was not due to us

To clarify, it was never my intention to blame the authors for the failures of previous approaches, nor did I suggest that the authors were responsible for fixing those failures. My original suggestion was to explore reasonable proxy evaluations. In retrospect, there may have been a misunderstanding—by proxy evaluations, I did not mean qualitative assessments or indirect inferences, but rather empirical results that could closely reflect FlexLLM’s performance in cases where direct reproduction was not possible. That said, it's great to see the authors were able to work through this issue.

The current empirical results are solid, and I am raising my score to 3. The last missing piece, in my view, is the BurstGPT data. While the authors designed their workload to follow known patterns, subtle differences may exist that are not captured by available statistical data. Additionally, the segmentation into morning, afternoon, and overnight periods feels overly coarse—I’m not yet convinced it adequately reflects fine-grained workload fluctuations.

I've tried to respond as quickly as possible to leave the authors enough time to incorporate these points before the rebuttal phase concludes.

I've tried to respond as quickly as possible to leave the authors enough time to incorporate these points before the rebuttal phase concludes.

Thank you for giving us as much time as possible to provide further responses.

Based on your suggestion, we conducted another dynamic workload experiment using the BurstGPT dataset. We are willing to include this experiment and the previous experimental results on FlexLLM in the paper. The specific data and results are as follows:

The BurstGPT dataset includes 60 days of request arrival times, input lengths, and output lengths. According to the description in the BurstGPT paper, their testing tool provides sampling slices with a duration of 20 minutes. The latest version of FlexLLM paper used a single 20-minute sampling duration for testing.

We aim to demonstrate Loquetier's adaptability to dynamic workload as comprehensively as possible. Therefore, we sampled a total of 6 time periods, each lasting 20 minutes, resulting in a total simulated duration of 2 hours. Specifically:

• 1 slot is a low-load time slot (average RPS < 1);
• 2 slots are medium-load time slots (average RPS between 1 and 2);
• 3 slots are high-load time slots (average RPS > 2).

Based on our statistics of the BurstGPT dataset, less than one-third of the time slots are high-load time slots, with the majority being low-load time slots. Since low-load time slots may be considered less challenging for the framework, we adopted the above settings.

The number of requests for the 6 time slots are as follows: 676, 2145, 1465, 2823, 2360, and 1856.

Since NeurIPS does not allow the upload of images or links, we regret that we are unable to show you the charts, as this is the most intuitive way to present the experimental results. We hope to describe the chart content as clearly as possible in writing. Our statistical table is as follows:

• tps denotes token per second.

In the first 20 minutes, the low load enabled the fine-tuning task to achieve a throughput of 2,500 tps, which is consistent with the performance observed in the initial phase of the dynamic workload experiment described in our paper.

Between 20 and 25 minutes, there was a sudden spike in high load, with the highest peak RPS reaching 11. At this point, the inference throughput rapidly increased to approximately 580 tps, while the fine-tuning task yielded to it, dropping to approximately 1,600 tps. After the 25th minute, the rate of incoming inference requests decreased, and the fine-tuning throughput returned to its previous level.

Starting at the 40-minute mark, moderate load caused the inference task's throughput to fluctuate around 280±100 tps, while fine-tuning throughput decreased to around 2000 tps.

Starting at the 60th minute, continuous high load for 40 minutes caused the inference throughput to exceed 600 tps at one point. Subsequently, the request arrival rate fluctuated significantly, and Loquetier's inference and fine-tuning throughput fluctuated accordingly (with an amplitude of approximately ±50 tps), indicating that our work can respond promptly to high-frequency load changes.

In the final 20 minutes of the medium-load scenario, the request arrival rate still fluctuated significantly, causing the throughput of both tasks to continue oscillating. The inference throughput gradually decreased as the load decreased, while the fine-tuning throughput gradually increased as computational resources were released.

The final result of SLO Attainment is 92.37%, with all requests that failed to meet service metrics occurring during workload spikes under high workload (RPS>5). Based on previous experiments and responses, this exceeded the hardware load limit. During other periods, Loquetier was able to meet our objectives.

Thank you for your detailed review of our work. Our responses to the weaknesses and questions you raised are as follows:

Baselines

We have received valuable suggestions from the reviewers regarding the evaluation. We would like to provide additional explanations and the derived results to address the issues raised.

For S-LoRA, FlexLLM have compared with it. S-LoRA is only optimized for inference tasks and performs worse than FlexLLM in fine-tuning and inference tasks; FlexLLM's experiments include the Peft baseline for fine-tuning tasks, and our results in similar experimental settings are better than the equivalent fine-tuning performance of Peft in the S-LoRA+Peft baseline in FlexLLM's experiments, while the performance of the Peft is better than that of FlexLLM in several experimental environments (FlexLLM has not tested a direct comparison with Peft when only performing fine-tuning tasks; based on the result in their paper and the number of GPUs used, we can infer that Peft's fine-tuning efficiency is higher than that of FlexLLM under the same hardware resources). Therefore, based on the proxy evaluation method, we can also confirm that our performance is better than that of FlexLLM and S-LoRA. Additionally, S-LoRA significantly outperforms vLLM in its multi-LoRA inference experiments, and its experimental environment is comparable to ours. Furthermore, S-LoRA does not support newer models, including Llama 3. For these reasons, we did not design experiments comparing S-LoRA and vLLM.

As mentioned in the Introduction, there are very few training-inference integrated frameworks that support multi-LoRA. FlexLLM may be the only proper baseline. The other methods mentioned in the Related work are mostly general optimizations for base models or further improvements to the LoRA method itself, and they are not suitable for evaluation as test baselines; our work is also compatible with these methods.

Further Attempts on FlexLLM testing

For the performance of FlexLLM in multiple LoRA inference scenarios. We conducted a smaller-scale test, and the results showed that FlexLLM ran for several hours while repeatedly loading LoRA in a loop for a test that should not have lasted more than 5 minutes—it indeed did not get stuck in an infinite loop. We have not studied its underlying implementation in detail, so we will not speculate on the cause of this phenomenon.

We will further attempt to rebuild FlexLLM from scratch based on more commits, which will take some time. We will provide feedback on the results after thorough testing. However, as mentioned in the paper, FlexLLM is missing a critical operation operator (the case for OP_CONV2D in FusedOp::peft_bwd_task), and we have not found any implementation records for this operator in historical commits. Therefore, we do not expect the built version to support LoRA fine-tuning functionality.

Comparison of Virtualized Module and Selections from Other Works

The implementation of PEFT directly modifies the base model, thereby rendering it non-reusable once processed. PEFT adopts a high-coupling design for LoRA, incorporating multiple parameters from different LoRAs within the same linear layer module. For different efficient fine-tuning methods, it designs the MixedModel module and related modules, enumerating and storing each supported efficient fine-tuning method for storage and invocation, causing different methods mixed together, resulting in extremely high coupling and poor maintainability.

For other frameworks, we categorize them into two types. The first type involves frameworks designed from scratch and built from the ground up. Since these frameworks do not need to consider compatibility issues, they generally do not retain the original structure of the base model, as the framework can handle structural modifications. The second category includes systems like S-LoRA and Punica, which merge multiple LoRA models for processing and directly modify the original model structure. This not only renders the original model unusable by other methods but also couples different LoRA models together, thereby reducing operability.