Mixture of LoRA Experts

We thank the Reviewer for the insightful suggestions. Regarding the questions raised:

---

Q1: Could you provide more results on different NLP datasets and tasks?

Q2: For NLP tasks, the improvement is marginal in many cases of cross-domain and in-domain settings. Hence, more experiments are needed to demonstrate the effectiveness of the proposed MoLE.

A1 & A2. We conducted extensive experiments across various tasks, including Translation, Struct to Text, Closed-Book QA, and multiple subtasks within the challenging Big-Bench Hard (BBH) datasets. Additionally, we introduced a new LoRA merging variant called PEMs $^{[1]}$, recently proposed in the field of NLP.

The corresponding experimental results are summarized in the following table. In summary, our MoLE surpasses state-of-the-art LoRA merging variants on four distinct datasets and tasks, showcasing robust performance. With a notable highlight on the BBH dataset, our MoLE achieves an average performance improvement of 3.8 over LoRAHub and outperforms PEMs by a substantial margin of 9.0.

In the domain of natural language generation tasks (Translation and Struct to Text), our MoLE consistently demonstrates superior average performance in the Translation task set, surpassing LoRAHub by 1.5 and PEMs by 2.7. Similarly, within the Struct to Text task set, our model achieves an average performance advantage of 2.1 over LoRAHub and 2.6 over PEMs. These results highlight the effectiveness of our model in generation task.

References

[1] Zhang, J., Chen, S., Liu, J., & He, J. (2023). Composing parameter-efficient modules with arithmetic operations. arXiv preprint arXiv:2306.14870.

---

Q3: The improvement is marginal for both VL and NLP tasks. From Table 1, we can find that MOLE cannot achieve the best text-alignment score for 2/4 cases and cannot achieve the best image-alignment score in the second row for 3/4 cases.

Q4: The samples are not enough to support the claims in VL tasks.

A3 & A4. As shown in following Table, there is usually a trade-off between text-alignment and image-alignment$^{[1][2]}$ on the multi-concepts fusion in V & L domain (The visualization of this trade-off can be seen in the Figure.12 in the supp). High image-alignment leads to a decrease in text-alignment where sample generations have less variance and are close to input target images. As a result, our MoLE may not perform better in both image and text sub alignment.

However, as depicted in Table 1 in the main text, it is worth noting that when evaluating the overall average across all configurations, our MoLE consistently outperforms both in average text alignment and image alignment. This underscores MoLE's capability to achieve superior alignment between text and images. Furthermore, we present additional results in the V&L domain following to further validate MoLE's effectiveness. In these results, it's evident that MoLE achieves the highest average scores in both image alignment and text alignment.

References

[1] Kumari, N., Zhang, B., Zhang, R., Shechtman, E., & Zhu, J. Y. (2023). Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1931-1941).

[2] Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., & Cohen-Or, D. (2022). An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618.

---

Q6: Experiments: How were the number of experts chosen for both the experiments?

A6.

• For V&L, we follow the common diffusion multi-concept composition setting$^{[1][2]}$, which usually compose 2~3 concepts into a new multi-concepts image.
• For NLP domain, We determined the number of LoRA candidates based on the dataset sizes of various subdomains within the FLAN dataset. Specifically, within the NLI-related datasets in FLAN, there are several sub-datasets, including ANLI (R1-R3), RTE, CB, WNLI, QNLI, MNLI, among others. Therefore, we selected a subset (5~6) of these as LoRA candidates, while the remainder served as the training set. Given the limited number of LoRA concepts, we conduct new experiments by extending the number of experts from 5 to 128, the following results are shown in A7.

Q7. In both image-generation and NLI experiments, the number of experts trained is between 3 and 5. How do you think the approach would scale to a larger number of experts?

A7.

NLP domain

We conducted experiments with an extended number of LoRA (8, 24, 48, 128) in the NLP domain, and the experimental results are as follows Table.

• Our MoLE achieved optimal performance in multiple settings with different numbers of LoRA. Particularly, when the number of LoRA is large, such as 48 and 128, our MoLE outperformed LoRAHub by 2.5 and 3.0, respectively. Through analysis, we found that LoRAHub's optimization algorithm tends to assign most LoRA weights to zero when there are many LoRA candidates. This limitation hinders LoRAHub from fully utilizing the characteristics of all LoRA. In contrast, our MoLE mitigates this issue.

• Besides, as the number of LoRA increases to a very large value (128), the performance of all methods deteriorates. This is an intriguing phenomenon, indicating that current methods, including our MoLE, do not perform well when dealing with a large number of LoRA. This is an area we plan to explore and address in the future.

The result is the average EM on the Big-Bench Hard (BBH) dataset

V&L domain

As illustrated in the table below, we systematically increased the number of fused LoRAs in V&L experiments. Typically, experiments that involve fusing multiple visual concepts set the fusion number to be in the range of 3 to 4. Fusing too many concepts can potentially lead to ambiguous model outputs; hence, we set the fusion number in the range of 3 to 6. Our findings consistently demonstrate that as the number of LoRAs increased, our MoLE consistently outperformed other models in terms of average text alignment and image alignment. This observation highlights the robustness of our MoLE to varying numbers of LoRAs and its enhanced fusion capability.

References

[1] Kumari, N., Zhang, B., Zhang, R., Shechtman, E., & Zhu, J. Y. (2023). Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1931-1941).

[2] Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., & Cohen-Or, D. (2022). An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618.

Q3: Training method: Section 3.5 says that only the gating parameters were trained, and the others were frozen. Does this mean the LoRA parameters were first individually trained and then kept frozen while the gating parameters were being trained? The current text implies that the LoRA parameters were never trained.

A3. Your understanding is correct. The purpose of our paper is to explore a better way to merge LoRA candidates. Here, "LoRA candidates" refer to LoRA models that have been independently trained on different datasets for distinct purposes. After obtaining these LoRA candidates, we then freeze the parameters of them and train the gating function to merge them.

---

Q4: Results: The results in Table 1 are unclear. What is the difference between LoRA expert 1 vs. expert 2 in the image alignment experiments? Is it the number of experts used? Why are these settings not shown for text alignment?

A4. We apologize for any ambiguity in our explanation. In the rows pertaining to image alignment, the expert indices (e.g., expert 1, expert 2) indicate which expert's training images were utilized to calculate the visual similarity with the final generated image. For example, expert 1 might correspond to 'Dog' while expert 2 corresponds to 'Wolf'. The 'text-alignment score' signifies the similarity between the final generated image and the text prompt used to create it.

To clarify further, we employ a specific prompt (e.g., t: "a photo of a [𝑉1] dog wearing [𝑉3] sunglasses, with a [𝑉2] cat beside") to generate the final image F, which includes elements such as [𝑉1] dog, [𝑉2] cat, and [𝑉3] sunglasses. We determine the similarity between F and t using CLIP, resulting in the text-alignment score. Additionally, we compute the similarity between F and the images corresponding to [𝑉1] dog, [𝑉2] cat, and [𝑉3] sunglasses (corresponding to LoRA expert 1, 2, and 3 as mentioned in the table) to calculate the image-alignment score.

We acknowledge the need for clarification, and we will ensure that this ambiguity is addressed comprehensively in the final version of our paper.

---

Q5: In Table 2, the labels under the “Model” column in the “In-Domain” setting must be typos. Instead of MoLE^{r}, MoLE^{c} etc., these should be MoLE^{ar1}, MoLE^{ar2} etc. Can you please confirm?

A5. Thank you for bringing this to our attention, and we appreciate your diligence in reviewing our work. We sincerely apologize for the typos in Table 2. In the 'In-Domain' setting, the labels under the 'Model' column should indeed be MoLE$^{ar1}$, MoLE$^{ar2}$, MoLE$^{ar3}$, and MoLE$^{q}$, respectively. We acknowledge this oversight, and rest assured, we will rectify these errors in the final version of our paper.

Thank you for your detailed, helpful feedback. We address your feedback point by point below.

---

Q1: The related work in Section 2.2 can be more clearly explained. Particularly, it is claimed that the "arithmetic operation-based fusion" suffers from "identity confusion among multiple LoRAs". This issue needs to be clarified. How does the proposed approach fix this issue?

The problem of "identity confusion among multiple LoRAs" in arithmetic operation-based fusion is rooted in our

• Observation 1 (in Section 3.1): "Fusing multiple LoRA candidates directly impacts the generative ability of the original model, while applying weight normalization before the fusion process maintains this generative capacity."

In the context of the V&L domain, directly merging multiple LoRAs can result in the model producing images with unclear meaning, as depicted in Figure 3 of the main text. Conversely, fusing LoRAs with weight normalization can lead to the model losing the distinctive characteristics of individual LoRAs, which is undesirable.

In the NLP domain, we trained a LoRA on a combination of ANLI-RA, ANLI-R2, and QNLI datasets and observed that using five or more LoRAs caused the model's output to become chaotic, repeating words like 'like, like, like...'. This demonstrates that directly fusing multiple LoRAs in NLP can negatively impact generative abilities. Additionally, when we applied weight normalization to combine LoRAs trained on five datasets (ANLI-R1, ANLI-R2, ANLI-R3, QNLI, and WNLI), we noticed a decrease in performance on four of the datasets, as shown in the table below:

How does the proposed approach fix this issue?

Our proposed approach is primarily based on

• Observation 2 (Section 3.1), where we found that different layers of LoRA represent different characteristics.

In the V&L domain, as illustrated in Figure 2 of the main text, different layers generate different features for target objects. In the NLP domain, as supported by our experiments, different layers of LoRA acquire distinct feature representations, as shown in the table below:

Our approach leverages this observation to achieve more dynamic fusion. We allocate higher weights to the characteristics needed within each layer and reduce weights for unnecessary features, effectively addressing the issue of identity confusion among multiple LoRAs. This allows us to achieve more tailored and effective fusion within limited parameter capacity.

For instance, in the V&L domain, as depicted in Figure 9 of the main text, our MoLE effectively preserves the characteristics of individual objects. In the NLP domain, as shown in Table 2 of the main text, our MoLE demonstrates comprehensive performance improvements compared to simple direct fusion. These experiments collectively demonstrate that our MoLE can dynamically select and weight LoRA features through hierarchical control, resulting in more effective and meaningful fusion.

---

Q2: Also, the details of the "reference tuning-based fusion" method are unclear. Is the approach from Gu et al., 2023 comparable to this work? If so, why is this approach not compared against them?

A2. The reference tuning-based fusion method predominantly centers on the V&L domain and necessitates the incorporation of supplementary information, such as deliberately designed object position masks. This methodology introduces a kind of prior knowledge that our model does not depend on. Consequently, conducting a direct comparison between our model and this method may not be entirely equitable, given the disparate assumptions and information inputs involved.

Thank you for your detailed, helpful feedback.

---

Q10. Each of the experts is trained on one NLI dataset, and the mixture is evaluated on a mixture of NLI datasets. These are all essentially the same task. What are the experts expected to learn differently in these tasks? It would make more sense to build task specific experts and generalize to new tasks.

Q11. Moreover, the base model used is Flan-T5 which was already trained on all the NLI datasets used in this paper. It would make more sense to adapt the model to other datasets, or use a different base model.

A10 & A11. To further validate the effectiveness of our MoLE, we conducted generalization experiments. Specifically, all LoRA candidates and LoRA merging variants, including MoLE and LoRAHub, were trained on NLI tasks such as ANLI-R1, ANLI-R2, ANLI-R3, QNLI, and WNLI, among others. Subsequently, we evaluated these methods on the challenging Big-Bench Hard (BBH) dataset.

As illustrated in following Table, our MoLE consistently outperforms other LoRA merging variants across nearly all sub-tasks. It achieves an average performance advantage of 2.4 over LoRAHub and 3.7 over PEMs, underscoring its robust and superior performance.

---

---

We want to express our sincere gratitude for your review. We apologize for the delayed response, as we have been dedicating an extended amount of time to conducting experiments, which has kept you waiting. Please let us know if any of your points were not addressed properly, or if you have any additional questions.

Q9: It would make more sense to adapt the model to other datasets, or use a different base model.?

A9. We conducted extensive experiments across various tasks, including Translation, Struct to Text, Closed-Book QA, and multiple subtasks within the challenging Big-Bench Hard (BBH) datasets. Additionally, for the sake of thorough comparison, we introduced a new LoRA merging variant called PEMs $^{[1]}$, recently proposed in the field of Natural Language Processing.

The corresponding experimental results are summarized in the following table. In summary, our MoLE surpasses state-of-the-art LoRA merging variants on four distinct datasets and tasks, showcasing robust performance. With a notable highlight on the BBH dataset, our MoLE achieves an average performance improvement of 3.8 over LoRAHub and outperforms PEMs by a substantial margin of 9.0.

In the domain of natural language generation tasks (Translation and Struct to Text), our MoLE consistently demonstrates superior average performance in the Translation task set, surpassing LoRAHub by 1.5 and PEMs by 2.7. Similarly, within the Struct to Text task set, our model achieves an average performance advantage of 2.1 over LoRAHub and 2.6 over PEMs. These results highlight the effectiveness of our model in generation task.

---

References

[1] Zhang, J., Chen, S., Liu, J., & He, J. (2023). Composing parameter-efficient modules with arithmetic operations. arXiv preprint arXiv:2306.14870.

---

Q8: In Table 1, results vary across settings (e.g.: robot+dog+clock vs. table+can+teapot) and the baselines are better in some settings. Do you have any insights into why MoLE works better in some settings and not all?

A8. As shown in following Table, there is usually a trade-off between text-alignment and image-alignment$^{[1][2]}$ on the multi-concepts fusion in V & L domain (The visualization of this trade-off can be seen in the Figure.12 in the supp). High image-alignment leads to a decrease in text-alignment where sample generations have less variance and are close to input target images. As a result, our MoLE may not perform better in both image and text sub alignment.

However, as depicted in Table 1 in the main text, it is worth noting that when evaluating the overall average across all configurations, our MoLE consistently outperforms both in average text alignment and image alignment. This underscores MoLE's capability to achieve superior alignment between text and images. Furthermore, we present additional results in the V&L domain following to further validate MoLE's effectiveness. In these results, it's evident that MoLE achieves the highest average scores in both image alignment and text alignment.

---

References

[1] Kumari, N., Zhang, B., Zhang, R., Shechtman, E., & Zhu, J. Y. (2023). Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1931-1941).

[2] Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., & Cohen-Or, D. (2022). An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618.

We are grateful to the Reviewer for the extensive review. We address your questions point by point below

---

Q1: From the motivation of the proposed MoLE for hierarchical control, do you think it's better suited for the text-to-image generation task than the NLI task?

A1. Our motivation was grounded in the consistent patterns we observed in both the NLP and VL domains. We apologize for the lack of experimental support for our motivations in the main text concerning the NLP domain. Our NLP domain support experiments are as follows:

• Observation1: Fusing multiple LoRA candidates directly impacts the generative ability of the original model, while applying weight normalization before the fusion process maintains this generative capacity.

We combined multiple LoRAs with the FLAN-T5 model. When using five or more LoRAs, the model's output became chaotic, repeating words like 'like like like...'. This shows that directly fusing multiple LoRAs in NLP can affect generative abilities. Additionally, we applied weight normalization to combine LoRAs trained on five datasets (ANLI-R1, ANLI-R2, ANLI-R3, QNLI, and WNLI). As shown in the following table, we observed a decrease in performance for the fused LoRA on four of the datasets. This indicates that while weight normalization preserves generative capacity, it also impacts the intrinsic properties of LoRA.

• Observation2: Different layers of LoRA learn distinct feature representations, collectively forming the overall identity of LoRA.

We trained a LoRA on a combination of ANLI-RA, ANLI-R2, and QNLI datasets, as shown in the table below (Visualized results can be found in Figure 11 of the PDF). We observed significant performance variations across different layers of this LoRA when evaluated on these sub-datasets. Specifically, the 0%~20% layers performed best on QNLI, the 40%~60% layers excelled on ANLI-R2, and the 80%~100% layers outperformed others on ANLI-R1. This reinforces the idea that in the NLP domain, different layers of LoRA acquire distinct feature representations.

Based on the two sets of experiments presented above, we can conclude that the motivations we proposed in Section 3.1 hold true in the NLP domain as well. We will include the supporting experiments in the NLP domain mentioned above in the final version to substantiate our observations

---

Q2: Combining MoE with LoRA seems straightforward.

A2. We thank the reviewer for valuable insights. The main motivation behind our work is not simply to integrate Mixture-of-Experts (MoE) with LoRA, but rather to build upon our two observations (Section 3.1).

• Observation1: Fusing multiple LoRA candidates directly impacts the generative ability of the original model, while applying weight normalization before the fusion process maintains this generative capacity.

• Observation2: Different layers of LoRA learn distinct feature representations, collectively forming the overall identity of LoRA.

Firstly, we recognize that LoRA represents distinct features across different layers. Leveraging this insight, we believe that introducing hierarchical control can enable more flexible and dynamic combinations of LoRA while preserving their individual characteristics—a common challenge faced by the research community. In this context, the MoE serves as a valuable tool to facilitate this hierarchical control, effectively harnessing this property to address the challenging problem of LoRA fusion. In summary, our primary contribution lies in pioneering the exploration of dissecting and layering LoRA for the first time and utilizing this observation to achieve more dynamic LoRA combinations. MoE is a tool we employ to realize this vision.

V&L domain

Given the absence of suitable LoRA fusion methods for comparison in the V&L domain, we introduced two state-of-the-art algorithms for multi-concept fusion that are not LoRA-based: Custom $^{[4]}$ and Textual Inversion $^{[5]}$. Both of these algorithms focus on fine-tuning all parameters to achieve better results.

As illustrated in following Table, our model significantly outperforms existing state-of-the-art LoRA fusion methods. For instance, in terms of text-alignment, our MoLE surpasses SVDiff by a margin of 0.031, and in image-alignment, it outperforms by 0.028. Moreover, compared with full-tuning methods (Custom and Textual Inversion), our MoLE also outperforms Textual Inversion in both image-alignment and text-alignment, and surpasses Custom in text-alignment. This further underscores the effectiveness of our MoLE in comparison to methods involving full parameter tuning.

---

Text-alignment Results

---

Average Image-alignment Results

References

[1] Huang, C., Liu, Q., Lin, B. Y., Pang, T., Du, C., & Lin, M. (2023). Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13269.

[2] Han, L., Li, Y., Zhang, H., Milanfar, P., Metaxas, D., & Yang, F. (2023). Svdiff: Compact parameter space for diffusion fine-tuning. arXiv preprint arXiv:2303.11305.

[3] Zhang, J., Chen, S., Liu, J., & He, J. (2023). Composing parameter-efficient modules with arithmetic operations. arXiv preprint arXiv:2306.14870.

[4] Kumari, N., Zhang, B., Zhang, R., Shechtman, E., & Zhu, J. Y. (2023). Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1931-1941).

[5] Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., & Cohen-Or, D. (2022). An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618.

Q3: Overall, for NLI tasks in NLP domain, the proposed MoLE shows similar average performance compared with LoRAhub. Q4: No other LoRA merging variants are compared in experiments.

In the NLP domain, we conducted comparisons between our MoLE model and the state-of-the-art LoRA merging variant, LoRAHub $^{[1]}$ (submitted to ICLR24, too). In the V&L domain, we compared our MoLE with the leading LoRA merging variant, SVDiff $^{[2]}$. Our results demonstrate that in both domains, our MoLE consistently achieves superior performance. To further validate the effectiveness of our MoLE, we have conducted additional experiments, including comparisons with recently-proposed LoRA merging variant baselines.

• NLP domain

We conducted extensive experiments across various tasks, including Translation, Struct to Text, Closed-Book QA, and multiple subtasks within the challenging Big-Bench Hard (BBH) datasets. Additionally, we introduced a new LoRA merging variant called PEMs $^{[3]}$, recently proposed in the field of NLP.

The experimental results are summarized in the following table. In summary, our MoLE surpasses state-of-the-art LoRA merging variants on four distinct datasets and tasks, showcasing robust performance. With a notable highlight on the BBH dataset, our MoLE achieves an average performance improvement of 3.8 over LoRAHub and outperforms PEMs by a substantial margin of 9.0.

In the domain of natural language generation tasks (Translation and Struct to Text), our MoLE consistently demonstrates superior average performance in the Translation task set, surpassing LoRAHub by 1.5 and PEMs by 2.7. Similarly, within the Struct to Text task set, our model achieves an average performance advantage of 2.1 over LoRAHub and 2.6 over PEMs. These results highlight the effectiveness of our model in generation task.

---

Q4: For the gating imbalance, instead of introducing a new loss, can we just increase the temperature? What's the difference in empirical performance?

A4. Thank you for the inspiration provided. We conducted relevant experiments on the NLI dataset, and after increasing the temperature (denoted as MoLE$^{T1}$, MoLE$^{T2}$, MoLE$^{T3}$ where T1 < T2 < T3), we observed a decrease in the model's performance with temperature increasing, with it becoming more similar to Simple Fusion in terms of performance. We believe that while increasing the temperature can alleviate the imbalance issue in the gating network, it also limits our MoLE from learning more dynamic LoRA combinations, leading to suboptimal results.

---

We want to express our sincere gratitude for your review. We apologize for the delayed response, as we have been dedicating an extended amount of time to conducting experiments, which has kept you waiting. Please let us know if any of your points were not addressed properly, or if you have any additional questions.

We thank the Reviewer for the useful suggestions. Regarding the questions 1 raised:

Q1: Authors motivate (section 3.1) the need for Mixture of LoRA for the vision domain but it is not clear if it is also required for the NLP domain as well or not (as also indicated by marginal improvement in results).

A1. To validate our motivations in the NLP domain, we conducted the following experiments to confirm that our two observations also hold true in the NLP domain:

• Observation1: Fusing multiple LoRA candidates directly impacts the generative ability of the original model, while applying weight normalization before the fusion process maintains this generative capacity.

We combined multiple LoRAs with the FLAN-T5 model. When using five or more LoRAs, the model's output became chaotic, repeating words like 'like like like...'. This shows that directly fusing multiple LoRAs in NLP can affect generative abilities. Additionally, we applied weight normalization to combine LoRAs trained on five datasets (ANLI-R1, ANLI-R2, ANLI-R3, QNLI, and WNLI). As shown in the following table, we observed a decrease in performance for the fused LoRA on four of the datasets. This indicates that while weight normalization preserves generative capacity, it also impacts the intrinsic properties of LoRA.

• Observation2: Different layers of LoRA learn distinct feature representations, collectively forming the overall identity of LoRA.

We trained a LoRA on a combination of ANLI-RA, ANLI-R2, and QNLI datasets, as shown in the table below (Visualized results can be found in Figure 11 of the PDF). We observed significant performance variations across different layers of this LoRA when evaluated on these sub-datasets. Specifically, the 0%~20% layers performed best on QNLI, the 40%~60% layers excelled on ANLI-R2, and the 80%~100% layers outperformed others on ANLI-R1. This reinforces the idea that in the NLP domain, different layers of LoRA acquire distinct feature representations.

Based on the two sets of experiments presented above, we can conclude that the motivations we proposed in Section 3.1 hold true in the NLP domain as well.

We thank the Reviewer for the insightful suggestions. Regarding the questions 2 raised:

---

Q2: Did authors conduct experiments on the task of machine translation or summarization or any other text generation task?

A2. We conducted extensive experiments across various tasks, including Translation, Struct to Text, Closed-Book QA, and multiple subtasks within the challenging Big-Bench Hard (BBH) datasets. Additionally, we introduced a new LoRA merging variant called PEMs $^{[1]}$, recently proposed in the field of NLP.

The corresponding experimental results are summarized in the following table. In summary, our MoLE surpasses state-of-the-art LoRA merging variants on four distinct datasets and tasks, showcasing robust performance. With a notable highlight on the BBH dataset, our MoLE achieves an average performance improvement of 3.8 over LoRAHub and outperforms PEMs by a substantial margin of 9.0.

In the domain of natural language generation tasks (Translation and Struct to Text), our MoLE consistently demonstrates superior average performance in the Translation task set, surpassing LoRAHub by 1.5 and PEMs by 2.7. Similarly, within the Struct to Text task set, our model achieves an average performance advantage of 2.1 over LoRAHub and 2.6 over PEMs. These results highlight the effectiveness of our model in generation task.

References

[1] Zhang, J., Chen, S., Liu, J., & He, J. (2023). Composing parameter-efficient modules with arithmetic operations. arXiv preprint arXiv:2306.14870.

---

We want to express our sincere gratitude for your review. We apologize for the delayed response, as we have been dedicating an extended amount of time to conducting experiments, which has kept you waiting. Please let us know if any of your points were not addressed properly, or if you have any additional questions.