Q1: The training-free generalization claim requires broader validation across diverse tasks.

Ans: As shown in Table 2, Table 3 in the manuscript and Table 10 in the response to reviewer HcLz, we have mainly validated our method in the fields of mathematics, commonsense reasoning, science, daily life and tool usage. In the future, we will further expand to more fields to verify our method.

Q2: How cloud-to-edge knowledge injection works mechanistically.

Ans: As shown in Table 12, in the response to reviewer TR5Y, when we do not utilize the online LLM to select experts, the overall performance decreases, further highlighting the importance of the online large model in converting instructions into internal knowledge to guide edge-side models.

Q3: Hyperparameter sensitivity.

Ans: Our exploration of different hyperparameter settings, as shown in Table 5 of the manuscript, revealed that adding more experts to the LoRA Expert Pool does not inherently improve results. We speculate that this may be due to the mismatch between the number of experts and the scale of available training data, resulting in poorly tuned expert classifications, and the auxiliary loss coefficient is the result of a hyperparameter search.

Q4: No analysis of how varying cloud-side models (larger/smaller LMs) affect performance?

Ans: We have conducted an experiment to show that the LLM with stronger reasoning ability brings better results, as shown in Table 13 in the response to reviewer TR5Y.

Q5: The paper could benefit from a more detailed discussion on how LoRA-Gen compares to other non-LoRA methods, Prompt tuning?

Ans: We conduct an experiment to show performance comparison between LoRA-Gen and Prompt-tuning as indicated in Table 15.

Table 15. Performance Comparison.

Q6: Reliance on cloud infrastructure and the limitations of the method and potential areas for improvement.

Ans: Our method supports diverse application scenarios, such as tool calls, personalized virtual assistants, offline intelligent systems, IoT device control, and tasks necessitating long system prompts. However, the current paradigm needs to predefine a pair of cloud and edge-side LM. The model-agnostic framework leaves an open question for future work.

Q7: Clarify the computational overhead of cloud-side LoRA generation.

Ans: After specifying the task description or system prompt, the cloud model only needs to perform one inference to complete the generation of LoRA parameters. For specific FLOPs, Memory, and Latency in the training and inference stages, please refer to Table 4 in the supplementary material.

Q8: Fix typos: Page 6, "utilyze" → "utilize"; unify "LoRAMoE"/"LoRA-MoE".

Ans: Thanks for your valuable advice. We will refine this in our revision paper.

Q9: How does the computational cost of cloud-side LoRA generation compare to edge-side efficiency gains?

Ans: In fact, in the deployment and reasoning phase, the cloud model only needs the computational overhead of a single inference to complete the specific task. The main computational overhead comes from the training phase. As shown in Table 15 in response to reviewer TR5Y, the 13B model will bring some gains over the 7B model.

Q10: What task characteristics make LoRA-Gen most effective (e.g., prompt complexity, task diversity)?

Ans: Our method is more beneficial in scenarios that typically feature a fixed specialized system prompt and varying user inputs. In such cases, the cloud-side large model performs the generation of customized LoRA weights through a one-time system prompt inference and supplies these weights to the edge-side small model.

Q10: What are the primary failure modes of LoRA-Gen, and how might they be addressed?

Ans: The major limitation of the current paradigm needs to predefine a pair of cloud and edge-side LM. If we do not use the same model pair as in the training phase, it will lead to a primary failure mode. A possible solution is to train multiple models in cascade, which we will continue to explore in future work.

Q1: It's not clear to me that context is needed for trained (non-base models), especially on seen tasks.

Ans: Sincerely sorry for the confusion. We aim to utilize the online large language model and the online LoRA Expert Pool to convert user instructions, system prompts, or contextual content into tailored LoRA parameters, improving the performance of the small model. Therefore, during training, we will provide a few-shot instruction to the online LLM to guide the model in acquiring this functionality.

Q2: It'd be interesting to see how a small amount of training would help.

Ans: As shown in Table 12, a small amount of training can lead to a small increase of some tasks.

Table 12: Performance with Few-Shot Training.

Q3: Ablate different lora generators.

Ans: As shown in Table 13, we conduct an ablation study on the average accuracy and harmonic mean of the LLM-based LoRA generator and the general LoRA generator across 8 datasets, using TinyLLaMA-1.1B as the small LLM. The results demonstrate that the LLM-based LoRA generator outperforms the general LoRA generator. Additionally, as presented in Table 2 of the manuscript, the performance of LoRA-Gen surpasses that of several other LoRA-based MoE methods.

Table 13. Ablation of Different LoRA Generators.

Q4: How is training-free setting in GPT4Tools trained?

Ans: As mentioned in Section 1.4 of the supplementary material, we process the Alpaca dataset through GPT-4, resulting in a filtered and abstracted set of 37,658 training samples. Then we use this part of the data for pre-training to obtain the instruction-aware LoRA generator and LoRA expert pool. We directly utilize the checkpoint to perform generalization tests on the GPT4Tools test set, corresponding to the 4th row "without training setting".

Q5: How important is the LoRA generator base LLM?

Ans: We conduct an experiment to verify the impact of the LoRA generator. As shown in Table 14, LLaMA3-8B (with the strongest reasoning ability) does bring better results, and increasing the size of the LoRA generator will also improve to a certain extent.

Table 14. Ablation of Different LLM-based LoRA Generators.

Q6: Are there scaling properties?

Ans: In this work, we offer an initial verification of the advantages of this paradigm. Moving forward, we will expand the dataset size to explore more effective and resilient training paradigms.

Q1: Do not include reasoning-specific datasets such as mathematical reasoning.

Ans: Thanks for the great advice, we further evaluate our method on the mathematical reasoning dataset GSM8K. As shown in Table 10, LoRA-Gen brings gains. Due to time constraints, we will collect more types of tasks for evaluation in the revised version.

Table 10: Diverse Tasks Performance.

Q2: The paper lacks experiments on training-free unseen tasks.

Ans: In this work, we mainly explored the generalization capabilities of mainstream reasoning scenarios and agent scenarios. We set Hellaswag, Winogrande, and PIQA as generalization test datasets, which do not appear in the training period, to verify the training-free unseen capabilities of our method as shown in Table 2 of manuscript. At the same time, we further verified the generalization performance of our method by testing the unseen tools in the GPT4Tools dataset as shown in Table 3 of manuscript. We will further explore the performance of LoRA-Gen in a wider range of scenarios in our future work.

Q3: The paper does not provide a detailed discussion on the specific experts included in the LoRA expert pool.

Ans: We analyzed the weight distribution across different experts when processing various tasks, as shown in Table 11. For instance, expert 5 is frequently activated for agent tasks but rarely for SIQA tasks, demonstrating that the online LoRA Pool implicitly assigns knowledge to different experts.

Table 11: Expert Weight Distribution.