LoRA-Gen: Specializing Language Model via Online LoRA Generation

Q6: Require a lot more explanation than the description on 216-220?

Ans: Following meta-learning, we seek to utilize a unified representation linked to task-specific information so as to improve the generalization capabilities across various tasks. Therefore, this representation is termed as meta-token, generated by the cloud-side large language model autoregressively. Specifically, For each given task description, we derive L tokens, one for each layer of the edge-side language model, with each meta-token directing expert routing at its respective layer, referring to lines 197 - 202 of our revision.

Q7: Confusion on Figure 1

Ans: It seems there exists some misunderstanding. All methods employ few-shot samples as the prefix input. Specifically, these samples are injected into the cloud-side large model in our method, while other approaches concatenate them directly with the user input for the edge-side model (Qwen-1.5B). Furthermore, we aim to illustrate our inference advantage, which is particularly relevant in application-driven scenarios.

Q8: Typo and Citation Error.

Ans: We sincerely apologize for the errors. The citation issues have been modified, including the correct LLaMA3 citation (please refer to lines 53, 321, and 475). The Typo error (``by generates") has been corrected in line 200. Thanks again.

Q1: Missing ablations/comparisons to the other context compression methods.

Ans: We focus on mitigating the efficiency constraints observed in previous LoRA-MoE-based methods and provide a new perspective that a large cloud-side model generates parameters for a smaller edge-side model, enabling improved specialization. While this approach shares similarities with compression methods, our primary focus is on enhancing generalization for unseen tasks through meta-learning strategies, alongside improving operational efficiency. As shown in Table 20, our comparisons with the methods mentioned by the reviewer demonstrate that our method achieves superior performance while maintaining competitive latency.

Table 20: Performance Comparison on unseen tasks with 5-shot samples among compression methods. † indicates that the method is without pretraining due to their weights not being publicly available. The latency is measured on an Nvidia V100 GPU.

Q2: Comparison of more efficiency metrics.

Ans: Please refer to the response for Reviewer PsM9's Q1, and we have already presented it in the A.5 of our Appendix.

Q3: Statistical significance testing.

Ans: Thanks for the great advice. The standard error is shown in A.3 of Appendix. The results of the bootstrap significance test are shown in Table below. Our approach seeks to explore a novel perspective on LoRA MoE, focusing on enhancing generalization capabilities and reasoning efficiency while maintaining comparable performance.

Table 21:The lower value of confidence interval in bootstrap testing.

Q4: Validation set results for all the experiments in section 4.4.

Ans: We follow the counterparts (MixLoRA and LoRAMoE) to construct the evaluation settings that utilize the same set of datasets. The official datasets SIQA, WinoG, and PIQA do not contain valid parts, so we utilize the results of the test set to calculate the average score. Additionally, we take the ablation experiments of the Auxiliary Loss Coefficient as an example. The results on both the validation and test sets, as shown in Table 22, exhibit a consistent performance trend.

Table 22: Validation set results.

Q5: Is this referring to the 10.1x compression ratio on the prompt?

Ans: The 10.1x metric indeed reflects our token compression rate. As the <EOS> character in the generation mode differs across models and is not manually standardized, compression rate serves as a consistent evaluation metric. We have modified the manuscript in line 101.

The term meta token still seems inadequately described in this response. Reviewer UJNj and Reviewer Kb9r also noted that this term confused them as one of the first items in their reviews, suggesting that this is a more general critique shared by multiple reviewers.

Concretely, is this statement from my original review accurate: "these tokens are not specialized for this parameter generation task but are just regular llama tokens given the prompt"?

At the current level of detail, that seems to be true based on your response "for each given task description, we derive L tokens" which are "generated by the cloud-side large language model autoregressively".

Regardless, there are still concrete details that aren't clear:

• Which output of the cloud-side large language model is used for the meta-token? Is it the final hidden state or the embedding of the discrete output token? If it's the embedding of the discrete output token, how is it sampled from the softmax distribution?
• Since these $L$ tokens are generated autoregressively, how are earlier tokens incorporated into the context? Is this standard Llama decoding?
• If the model is not trained to generate these output tokens, what is the rationale for assuming that each token corresponds to a layer? Without training, the generated tokens will represent a response to the task description prompt. If the model is trained to generate these output tokens, the method doesn't appear to be described.

We sincerely appreciate the reviewer's thoughtful and patient feedback. We completely understand your concerns and will further clarify the significant test and AutoCompressors comparison through two main points.

1. We realize that there is a technological mistake in the significance test of Table 21 (we mistakenly take the average accuracy result of the entire task as one sample, which means there are only 8 samples during the entire test, which is inconsistent with the test approach you provided [1]). To rectify this, we re-conduct the bootstrap test following the methodology outlined in [1]. Given that larger sample sizes enhance the reliability of bootstrap tests, our results on the Hellaswag dataset (10,046 test cases) are presented in Table 24.

[1] https://aclanthology.org/P18-1128/

Table 24: AutoCompressors with OPT-2.7b and LoRA-MoE-based methods with Gemma-2b.

2. Our application scenario emphasizes small-sized language models on edge-side devices. To this end, we conduct an additional comparison with the AutoCompressors method using the OPT-2.7B model, as shown in Table 25. Our method achieves a 1.5-point improvement in the average accuracy of unseen tasks over AutoCompressors. Additionally, the bootstrap test interval presented in Table 24 confirms a statistically significant enhancement compared to AutoCompressors. Furthermore, we measure the latency of AutoCompressors, with the results showing a 1.52x speedup that emphasizes the efficiency of our approach. The above outcomes align with the four contributions discussed in lines 75-93.

Table 25: Comparison with AutoCompressors.

We fully agree with the reviewer that AutoCompressors should be considered as a baseline. We have added the comparison results with the AutoCompressors method to the appendix in the revision shown in lines 739-744. We thank you for the constructive feedback again and would like to confirm if this addresses the concerns regarding the significance test and the AutoCompressors baseline. We look forward to your kind response. Thanks a lot.

Q1: Its positioning with respect to previous methods seems a bit unclear and could use some improvement.

Ans: Thanks for your valuable suggestions. We calculate the computing cost (FLOPs), GPU Memory and Latency across training and inference modes as shown in Table 12, and we have already presented it in the A.5 of our Appendix. MixLoRA† indicates the method without specific optimization. All metrics are measured on a Nvidia A100 GPU. FLOPs are measured using an input of 100 tokens and an instruction of 200 tokens, while memory and latency are evaluated in training mode with a batch size of 8 per GPU. By leveraging the unified representation from the aspect of meta-learning and reparameterization approach, we achieve minimal FLOPs and the shortest latency during the inference phase.

Table 12: Efficiency Comparison.

Q2: The section on limitations is very short and could be more detailed.

Ans: For offline scenarios involving fixed and extended system prompts, the cloud-side model can generate customized LoRA parameters in one inference, which are then supplied to the end-side model. We have revised it in the paper.

Q3: Minor formatting/typographical errors.

Ans: Sorry for the errors. We have revised them in Line 250 and Line 404 of the revision.

Q4: Have the authors tried using an even lower coefficient of auxiliary loss.

Ans: We have tried this hyperparameter tuning, detailed results are shown in Table 19. A coefficient of 0.001 is the best.

Table 19:Performance comparison under different loss coefficients.

Q1: Critical Implementation Details Missing.

Ans: Thanks for your advice, we provide more explanation here and have revised our manuscript.

1. Please refer to the response of Reviewers Kb9r's Q2. We also have added a more detailed explanation in line 197.

2. Sorry for missing information. Specifically, both the "direct method" and the meta token (indirect way) are derived using the causal token paradigm of LLM and subsequently mapped to the parameter space via a feedforward neural network. The key difference lies in their shapes: the $i$-th token of the former has a shape of $[1, 3\times 2\times d\times r]$, whereas the meta token has a shape of $[1, n]$. $d$, $r$, and $n$ indicate the hidden dimension of edge-side small LM, the low rank of the LoRA setting, and the number of LoRA experts, respectively. We have revised it in Line 464.

3. As the reviewer mentioned in the strengths section (real-world applications), we aim to extend our approach to industrial scenarios, which typically feature a fixed specialized system prompt and varying user inputs. In such cases, the cloud-side large model performs the generation of customized LoRA weights through a one-time system prompt inference and supplies these weights to the edge-side small model.

Q2: Questionable Latency Comparison and Task Agnostic Discussion.

Ans: First, we consider there are some misunderstandings. Methods such as MixLoRA and LoRA-MoE, are token-wise MoE strategies, which means they need to rout experts for each token, which will undoubtedly cause more latency. These approaches claim that the MoE strategy is able to mitigate knowledge forgetting and enhance generalization capability. We want to emphasize the reparameterize ability of our method where the generated weights can merge into SLM seamlessly without compromising performance. Then, we propose a fresh perspective in this paper: employing a large cloud-side LM to generate parameters for a smaller edge-side model, enabling better specialization. As the reviewer mentioned, this strategy is highly relevant in industrial contexts, which are typically task-specific. In such cases, users provide a consistent system prompt but submit a wide variety of questions, which general methods fail to manage. Moreover, we consider that the generated LoRA may reluctantly handle task-agnostic scenarios given a general system prompt.

Q3: Statistical Rigor Concerns.

Ans:

1. We prioritize edge-side scenarios, where computing resources are constrained, leading us to concentrate on the effectiveness of small-size models. Furthermore, we also evaluate the performance of an 8B-parameter model (Llama3-8b) without additional system prompt, as outlined in Table 17:

Table 17: Performance Comparison between our method and baseline based on Llama3-8B.

2. We conduct the evaluation following counterparts such as MixLoRA and LoRAMoE. The testing data size of all tasks is summarized in Table 18:

Table 18: Data size for each task evaluation.

3. Across all experiments, we randomly select few-shot examples to ensure robustness. Additionally, for varying few-shot counts shown in Figure 1 of the manuscript, we calculate the results multiple times, resulting in a standard deviation of 0.0146.

Q4: Writing Questions about manuscript.

Ans:

1. We sincerely appreciate your suggestion and have updated the manuscript accordingly. Please refer to line 298 for the correct group name.

2. There seems to be some misunderstanding here. Section 4.1 provides an overview of the meta-information of the data used, while Section 4.3 details how we allocate the dataset into seen and unseen parts to evaluate our capabilities in multi-task learning and generalization to unseen tasks. However, we acknowledge that there are indeed duplications in the citation part. We have addressed this issue and made the necessary modifications (shown in line 363). Thanks again for the reviewer's valuable suggestion.

3. Thanks for your advice, we have added this information in A.4 of Appendix

We greatly appreciate your additional feedback. Our responses to each point are provided below:

1. As noted in line 321, generating meta-tokens involves a training phase.

2. We perform the multi-task evaluation as shown in Table 2, which is consistent with the original objective of MixLoRA. Focusing on the seen-task section, we train and evaluate across these five joint tasks, achieving a 5.3x speedup (MixLoRA's 141.9ms vs ours 26.7ms on Qwen-1.5B) with an average accuracy of 57%, compared to MixLoRA's 56%. Although our method excels in specific-task scenarios, this result underlines its comparable performance in multi-task cases.

3. We fully understand your concern. Due to time constraints, we take Gemma-2B you mentioned as an example and conduct two runs. The statistical mean and variance are presented in Table 26. We will update the results in the revised version.

Table 26: Mean and variance results.

4. Baseline in Table 17 refers to the native LLaMA3-8b without finetuning.

Q1: Please explain “edge-side language models”.

Ans: Thanks for the great advice, we have added the explanation on line 33 of the revision. More specifically, ``edge-side language model" is a term in the industry domain, that usually indicates a powerful artificial intelligence system deployed on edge devices, such as mobile phones and embedded systems. It operates independently to deliver efficient, real-time intelligent services and is typically optimized to minimize computational and storage costs.[1]

[1] Qu, Guanqiao, et al. "Mobile edge intelligence for large language models: A contemporary survey." arXiv preprint arXiv:2407.18921 (2024).

Q2: More explanations about the meta tokens generation and utilization.

Ans: Referring to line 197 of the manuscript, Given a series of few-shot samples or task-specific system prompts as input of cloud-side LM, the LM appends $L$ special tokens called <$meta$> behind them and transfers the inherent knowledge into these tokens with causal masks in a single forward pass. We define these tokens as meta tokens $\\{T_i^{meta}\\}_ {i=1}^L$, where $L$ represents the number of layers of the subsequent edge-side small language model (SLM). We take the $i$-th layer of SLM in the edge-side device as an example to show how the $i$-th meta token indicates the generation of specialized LoRA weights. We first utilize a lightweight feedforward neural network (2 linear layers with Batch Normalization) to transform the last hidden state of the token to expert space and get the router $R^i$. Then we adopt a KeepTop-K strategy to obtain the gate $G^i$ of $n$ experts (in case of $\{K=2, n=3\}$, pseudo $G^i$ may be $[0.83, 0.17, 0 ]$ ). Finally, the reparameterized weight of $i$-th layer can be formulated as: $\bar{w}^i = w^i + \sum_{j=1}^{n}G ^iE_j$.

Q3: Add confidence intervals to your results in Table 2 and 3.

Ans: The standard error is illustrated in Table 10, and the results are also provided in A.3 of the Appendix.

Table 10: Standard error on language model benchmarks.

Q4: Citation issues.

Ans: Thanks for the valuable suggestions. We have addressed these issues in the revision.

Q5: Performance improvements are not consistent.

Ans: Our core contribution lies in providing a specialized edge-side model that combines strong generalization capabilities with context compression for unseen tasks, which balances effectiveness and efficiency across both seen and unseen tasks. Accordingly, we use the harmonic mean, arithmetic mean across various tasks, and latency to evaluate the overall advantages of our method compared to counterparts, rather than focusing on specific tasks. Apologies for any possible misunderstanding; the polished version is shown in line 365.