BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing

Comment: "Limited empirical improvements: The performance gain is marginal, making it unclear whether the complexity trade-off is justified."

Answer to C1: Thank you for your comment. Please refer to R2C4.

---

Comment: "No throughput/memory efficiency evaluation: The paper claims efficiency improvements, but no evidence of speedup or memory savings is presented in the main paper."

Answer to C2: Thank you for your comment. The goal of our method is to reduce parameter redundancy through effective parameter sharing. Our experimental results demonstrate that BSLoRA achieves better performance with fewer trainable parameters, indicating a significant improvement in parameter efficiency. In the appendix, we provide a detailed comparison of the memory usage between standard LoRA and BSLoRA under varying numbers of concurrently deployed downstream tasks. As shown in Figure 6, BSLoRA consumes substantially less GPU memory during inference, enabling more downstream tasks to be deployed simultaneously. This highlights the practical value of our method in multi-task inference scenarios, especially under memory-constrained conditions.

We also test the inference speed comparison on the dataset of wikitext2 and ptb, the results are shown in https://anonymous.4open.science/r/BSLoRA-1E72/ . Results indicate that both GT and KE can achieve inference acceleration.

---

Comment: "Not highly novel: The method mainly repackages known techniques (parameter sharing, Kronecker product, low-rank adaptation) rather than introducing fundamentally new ideas."

Answer to C3: Thank you for the thoughtful question. As pre-trained language models continue to scale up, even parameter-efficient fine-tuning methods like LoRA introduce a substantial number of trainable parameters. Moreover, in real-world inference settings, LoRA adapters are typically kept separate from the backbone model to support simultaneous deployment of multiple downstream tasks. This makes reducing the memory footprint of LoRA adapters a critical challenge. Existing methods such as ShareLoRA focus only on inter-layer parameter sharing, neglecting the intra-layer redundancy that also contributes significantly to parameter inefficiency. BSLoRA addresses this gap by introducing both intra-layer and inter-layer sharing mechanisms, aiming to further reduce redundancy and achieve comparable or even improved performance with fewer trainable parameters. To overcome the challenge of sharing parameters with incompatible shapes, we introduce three lightweight but effective shape transformation techniques (SS, GT, KE), enabling flexible parameter sharing across diverse model components. While each individual technique may not be novel, their systematic combination and targeted application to large language model fine-tuning constitutes a meaningful contribution. Looking ahead, we plan to explore even more advanced sharing strategies to further enhance parameter efficiency.

---

Comment: "Limited practical advantage: The main benefit only appears in large-scale multi-task settings (>200 tasks), which isn't a common real-world scenario."

Answer to C4: Reducing the size of LoRA adapters is practically important not only in large-scale multi-task settings (e.g., >200 tasks), but also increasingly so as model sizes continue to grow. For instance, applying LoRA with rank 64 to all linear layers in LLaMA 70B consumes approximately 1.4 GB of memory, and this rises to 4.5 GB for a 405B model. When deploying more than 10 tasks concurrently—common in real-world cloud services or multi-user systems—the cumulative memory usage of LoRA adapters alone can reach 45 GB, posing a significant burden for inference servers and edge devices alike. While smaller models may require hundreds of tasks to reach memory bottlenecks, larger models hit these limits even with far fewer concurrently deployed adapters. This makes parameter reduction a critical bottleneck for scaling both in terms of deployment cost and system responsiveness. BSLoRA addresses this challenge by reducing memory usage by over 50% compared to standard LoRA, enabling significantly more efficient multi-task deployment without sacrificing performance.

Comment: "... settings for Table 1 and Table 2 are not consistent. The ablation study evaluates the effectiveness of the three different modules by removing each component sequentially."

Answer of C1: Thank you for your observation. The settings in Tab 1 and Tab 2 were designed to evaluate the overall performance of BSLoRA in real-world scenarios. Therefore, the rank settings in Tab 1 and Tab 2 were selected through extensive experiments to achieve optimal performance while maintaining parameter efficiency. In contrast, we chose rank=8 in the ablation study to clearly isolate each component’s contribution. These different configurations address different experimental goals.

---

Comment: "Could you validate the effectiveness of BSLoRA by comparing it with ShareLoRA at different ranks, while keeping the number of trainable parameters similar?"

Answer to C2: Thank you for your suggestion. We conducted additional experiments comparing ShareLoRA at ranks 4, 8, 16, and 32 with BSLoRA configured to have approximately same trainable parameters. The results (https://anonymous.4open.science/r/BSLoRA-1E72/) show that at lower budgets, only the KE outperforms ShareLoRA. This suggests that under tight parameter budgets, SS and GT have limited capacity to extract useful features. However, as the rank increases, all three variants consistently outperform ShareLoRA, indicating that BSLoRA is more effective at leveraging redundant capacity when more parameters are available. These findings further support that there exists significant redundancy both within and across layers, and our sharing strategies can lead to more efficient parameter utilization, especially as the parameter budget increases.

---

Comment: "For the hyperparameter setting, how is r selected for different reshaping methods? Why not choose a consistent r across all methods?"

Answer to C3: Thank you for your insightful comment. SS, GT, and KE each have distinct structural characteristics, so they they naturally require different r configurations to achieve optimal performance. Using a single rank across all methods can be suboptimal, as each interacts with model layers differently. Instead, we determined empirical best r values through systematic experiments, balancing parameter efficiency and accuracy across benchmarks.

---

Comment: "... the performance of GT is comparable to the other two methods (i.e., SS and KE) in Table 1 and Table 2. How can this phenomenon be explained?"

Answer to C4: Your question is very interesting. One possible explanation is that GT provides an implicit regularization effect, preventing overfitting to specific layers and encouraging a more generalizable representation. Additionally, the learned input and output transformations may help refine the information flow, ensuring that key features are preserved while removing redundant ones.

Comment: '“... local component of LoRA performs best on HellaSwag, ARC-e, BoolQ, and SIQA, while the intra-layer shared component excels on PIQA and ARC-c.” Could you provide some insight into this phenomenon?'

Answer to C5: Thank you for your insight. The varying effectiveness likely arises from differing task requirements. Tasks like PIQA and ARC-c may "share similar patterns", so consistent features from shared parameters help. Meanwhile, tasks like HellaSwag and BoolQ may "demand fine-grained contextual understanding", benefiting more from local parameter usage. This underscores BSLoRA’s hybrid design in adapting to different task needs.

---

Comment: “The results in Figure 4(c) show that modifying the constant improves both expressiveness and information content.” However, this conclusion is not obvious from Figure 4(c). How was it derived?”

Answer to C6: Thank you for your comment. The current radar plot in Figure 4(c) does not clearly show the improvement. We inferred it by comparing average performance across methods. The results used in this analysis are shown in the link above. We will clarify this in the revised paper and improve either the figure or the accompanying explanation.

---

Comment: "In Table 10, different rank settings achieve similar performance, and the claim that 'Assigning a lower rank to the local component and a higher rank to the shared parameters yielded better performance, further illustrating the redundancy in the standard LoRA parameters' is not obvious."

Answer to C7: Thank you for your observation. Adjusting only the local component's rank merely brings minor performance improvement or worsen, indicating low-rank local component can achieve comparable performance. Besides, keeping a low-rank local component and adjusting shared components consistently results in comparable performance. Therefore, we can assign a lower rank to the local component and high ranks to shared components to achieve better parameter efficiency.

Comment: "Given that LoRA already reduced the trainable parameters dramatically compared to full training, the additional reduction from the proposed intra-layer and inter-layer sharing is relatively small."

Answer to C1: Thank you for your insightful comment. While it is true that LoRA significantly reduces trainable parameters compared to full fine-tuning, the memory footprint of LoRA adapters becomes increasingly non-negligible as the size of the pre-trained model and the number of downstream tasks grow. In real-world inference systems, LoRA adapters are typically not merged with the base model to support dynamic multi-task serving. For instance, applying LoRA with rank 64 on LLaMA 70B introduces over 1.4 GB of additional memory. As more tasks are deployed simultaneously, e.g., 100 tasks would take about 140G memory, this overhead quickly becomes a major bottleneck.

To address this, recent works such as Vera, ShareLoRA, and Tied-LoRA explore further parameter reduction. Building on these efforts, BSLoRA introduces a more fine-grained parameter sharing framework that combines local, intra-layer, and inter-layer components. To enable sharing across weights of different shapes, we also propose three shape transformation strategies (SS, GT, KE). This design not only reduces trainable parameters but also improves model performance, as demonstrated in our experiments. From our experiment's results, BALoRA can reduce over 50% trainable parameters on average and achieve an average 1.25% performance improvement. Importantly, BSLoRA can be seen as a flexible superset of LoRA, offering a tunable architecture that adapts to both performance and efficiency needs.

---

Comment: "Given the complexity of hyper-parameter tuning and shape transformation (SS, GT, KE), I am not fully convinced that the gain would outweigh the effort."

Answer to C2: We appreciate your concerns regarding the complexity of hyperparameter tuning and shape transformation. In practice, BSLoRA does not introduce additional tuning overhead compared to standard LoRA—the only tuning required is selecting appropriate rank values for the chosen sharing strategy. Importantly, all three proposed sharing strategies (SS, GT, KE) consistently reduce parameter count while maintaining or improving performance, making the selection process flexible and low-effort. In our experiments, we empirically chose rank settings that best fit each sharing method, but our ablation studies (Tables 7, 8, and 9) show that BSLoRA performs robustly across different configurations, demonstrating that extensive hyperparameter tuning is not necessary for effective deployment. By default, we adopt the Kronecker Extension (KE) strategy in our setup, as it offers the best trade-off between parameter efficiency and performance. Overall, BSLoRA is a simple yet effective solution that achieves higher efficiency without adding tuning complexity.

---

Comment: "Typo in the caption of Figure 1: (left) and (right) should be (top) and (bottom)"

Answer to C3: Thank you for pointing this out. We will correct the caption of Figure 1 from "(left)" and "(right)" to "(top)" and "(bottom)" in the revised version.

Comment: "Fairness of Comparison with Baselines: The comparison between BSLoRA and baselines such as ShareLoRA and Tied LoRA may be unfair. ... To evaluate the true effectiveness of BS LoRA’s design, how does applying LoRA to both attention and MLP modules compare to applying it to only one module type (e.g., attention-only or MLP-only)? ...."

Answer to C1: Thank you for the thoughtful suggestion. We agree that evaluating the contribution of parameter sharing across different module types is important for a fair comparison. To address this, we conducted an additional ablation study, comparing BSLoRA applied to only the attention modules, only the MLP modules, and both modules simultaneously.

The results show that applying BSLoRA to the attention modules alone yields better performance than applying it only to the MLP modules, indicating that redundancy in attention layers is more significant. Moreover, applying BSLoRA to both attention and MLP modules further improves performance over attention-only tuning. This suggests that redundancy exists not only across layers but also across different module types within the same layer, as discussed in Section 2.2. These findings support our core claim: combining intra-layer and inter-layer sharing strategies enables more effective reduction of parameter redundancy, while also contributing to better overall performance. The results are shown in https://anonymous.4open.science/r/BSLoRA-1E72/ .

---

Comment: "Rationale for Tied-LoRA Variant Selection"

Answer to C2: Thank you for raising this important question. After reviewing the original Tied-LoRA paper, we observed that both TL5 and TL6 perform well across various tasks. In our own experiments, we replicated both the TL5 and TL6 configurations. Based on the specific training setup and evaluation methods used in our study, we found that TL5 outperformed TL6 under these conditions. Consequently, we decided to present the results for TL5 in the main text of the paper. We will clarify this choice in the revised version of the paper and provide a detailed comparison between TL5 and TL6 in the appendix to ensure transparency and consistency in the reported findings. The results are shown in https://anonymous.4open.science/r/BSLoRA-1E72/ .

Inconsistent Optimal Methods Across Setups Tab 2 results show that no single config among SS, GT, KE consistently outperforms others across tasks, model sizes, or ranks.

Answer to C3 : Thank you for your thoughtful observation. Indeed, the optimal configuration of SS, GT, and KE varies across tasks, model sizes, and ranks. This variability arises due to the task-specific nature of the benchmarks and the differences in model layer architectures. Our approach aims to provide flexibility by offering multiple methods that can be tailored to the requirements of different tasks and model sizes.

For example, as shown in Table 2, the SS method performs best on Llama 1-7B, while the KE method performs best on Llama 3-8B. This flexibility allows users to select the method that fits their deployment context. Moreover, the performance variation across different methods is minimal, highlighting the robustness of BSLoRA across diverse settings. We believe this flexibility is one of BSLoRA’s strengths, enabling it to be adapted for various deployment scenarios without the need for extensive hyperparameter tuning for each task.

---

Comment: "The improvements compared with ShareLoRA and Tied LoRA are marginal."

Answer to C4: Thank you for your comment. While the performance gains of BSLoRA over other parameter-sharing methods such as ShareLoRA and Tied-LoRA may appear modest, our method introduces a more fine-grained parameter sharing strategy. As discussed in Section 2.2 of our paper, redundancy in LoRA parameters exists not only across layers but also within the same layer across different parameter modules. However, existing approaches focus only on inter-layer sharing for modules at the same relative position, overlooking the redundancy among intra-layer components.

BSLoRA addresses this limitation by jointly implementing intra-layer and inter-layer parameter sharing, enabling a more comprehensive reduction in trainable parameters. To overcome the challenge of sharing across weights with mismatched shapes, we further propose three shape transformation methods (SS, GT, KE), which allow the model to flexibly align and share information across diverse modules. This design increases the parameter efficiency and enhances the model’s ability to capture shared features across tasks. Our experimental results show that BSLoRA not only achieves a further reduction in parameter redundancy but also delivers consistent performance improvements, showing a better balance between efficiency and effectiveness than existing sharing-based methods.