What Limits Bidirectional Model's Generative Capabilities? A Uni-Bi-Directional Mixture-of-Expert Method For Bidirectional Fine-tuning

Weakness 1&2: Test on More Models: The authors have only tested the Qwen 1.5 series of models. The experiments on more models are yet to be conducted. Test on Recent Models: It is beneficial to conduct experiments on recent LLMs.

Thank you for your advice. We evaluate the effectiveness of our proposed method on the recently released Qwen2.5 series, and UBMoE still achieved performance comparable to that of causal language models.

Weakness 3: Experimental Details: Several important experimental details are missing or not properly explained.

We provide the experimental setup for Figure 2 in Appendix B. For the model, we use Qwen1.5-0.5B.

We set the value of λ (lambda) to 0.001.

We provide detailed data sources in Section 5.1. The data used to train the gating layer is sampled from the Tulu-v2-SFT mixture, with only 32,000 samples used for training. These details will be added to Section 5.1 in the revised manuscript.

Question 1: It seems that FFN+ATT, as well as FFN+EMB perform better on the model size of 0.5B in Figure 2. I would like to understand why this is the case.

For LLMs with fewer parameters, fine-tuning both the FFN and attention layers can yield modest performance gains due to the limited model capacity, as shown in our results in Table 2. However, as the model size increases, this performance gap narrows significantly.

Typos

Thank you for your thorough review. We will address this and carefully check our paper in a next revision.

Summary

Thank you for your careful reading and suggestions. We will improve our work according to your suggestion. We hope these additional analyses address your concerns and improve the manuscript's rigor.

Weakness 1: The authors claim that the inferior performance ... this conclusion more convincing.

As we replied to Reviewer bS72, Table 1 shows that there is a consistent trend between subsequent dependence and the general capability of the model. This indicates a correlation between subsequent dependence and the model’s general performance, suggesting a potential causal relationship.

Besides, we analyzed the relationship between generation performance and attention-based forward or subsequent dependence. The experimental results are shown in the table below. As the level of subsequent dependence increased, the generation performance of the model declined. In contrast, our method reduced the subsequent dependence introduced by incorporating bidirectional experts while improving generation performance. This indicates a correlation between the distribution of attention weights and the model's generation performance.

Weakness 2: The results in Tables 2, 3, 4 demonstrate ... current results are not enough to support the claims proposed by the authors.

Thank you for your advice. We added MNTP-LLM as a baseline, which is trained using the Wikipedia dataset through the MNTP method. As shown in the table below, the experimental results demonstrate that UBMoE-LLM still exhibits strong performance.

Weakness 3: However, I believe additional ablation studies would be helpful to analyze the role of the gate control layer.

Thank you for your helpful advice. As shown in the table below, we conducted experiments on the STS task. The gate control layer effectively enhances the ability to embed tasks.

Question 1: This paper focuses on bidirectional finetuning ... embeddings tasks compared to bidirectional pretraining tasks.

We appreciate the reviewer’s suggestion. However, it is well known that there is currently a lack of proper similar scale bidirectional LLMs to serve as a baseline. We are trying to train to the bidirectional model for testing.

Question 2: I note that UBMoE-LLM improves the generation performance on TruthfulQA. Is it possible to provide an additional explanation for the improvements?

We're glad you're paying attention to that. We believe this is mainly due to the introduction of bidirectional experts, which enhances the model’s resistance to hallucination.

Summary

Thank you for your valuable advice. We have addressed these suggestions through additional experiments and analyses. We hope these revisions strengthen the validity of our findings.

Weakness 1: Limited discussion of potential trade-offs in training complexity.

Thanks for the helpful comments. Our training consists of two main parts: bidirectional expert training and UBMoE-LLM model training. For bidirectional expert training, compared to previous bidirectional approaches, we only fine-tune the FNN layers, which significantly reduces training costs. As for UBMoE-LLM training, we train for only 1,000 steps, using just 32 samples per step. The training of UBMoE-LLM (7B) can be completed in just 1 hour on a single H100 GPU.

Weakness 2: Additional analysis on the interpretability of attention weights could enrich the findings.

Thanks for the helpful comments. We analyzed the relationship between generation performance and attention-based forward or subsequent dependence. The experimental results are shown in the table below. As the level of subsequent dependence increased, the generation performance of the model declined. In contrast, our method reduced the subsequent dependence introduced by incorporating bidirectional experts while improving generation performance. This indicates a correlation between the distribution of attention weights and the model's generation performance.

Weakness 3: Clarify the distinction between preceding and subsequent dependencies.

We provide a detailed explanation of the computation methods for preceding and subsequent dependencies in Section 3.2. As shown in formula 1, preceding dependence represents the average score of attention before token i in the attention layer. Subsequent dependence represents the average score of attention after token i.

Weakness 4: Include a broader comparison with other hybrid models.

Thanks for your advice and we added MNTP-LLM as a baseline, which is trained using the Wikipedia dataset through the MNTP method.

Question 1: How does UBMoE-LLM perform on tasks beyond text generation and embedding, such as reasoning tasks?

As shown in the table below, we evaluated the performance of UBMoE-LLM on a physics problem benchmark and a mathematics problem benchmark. The experimental results indicate that UBMoE-LLM still demonstrates performance comparable to that of a causal language model.

Question 2: Have you explored the impact of scaling UBMoE-LLM beyond 7B parameters?

As shown below, we present the experimental results on Qwen2.5-14B-instruct, where UBMoE-LLM maintains performance consistent with that of a causal language model in terms of generation quality.

Summary

Thank you for your advice. Your suggestions have brought great help to the improvement of our work. Our results show that the method efficiently combines both generative and embedding tasks efficiently, which supports our hypothesis of attention dependence. We hope these additional analyses address your concerns and improve the manuscript's rigor. We sincerely appreciate your constructive feedback and have carefully addressed all the points raised. We hope the revised version demonstrates significant improvements and better aligns with your expectations.

W1: The new model developed ... computational demands for generation tasks.

UBMoE activates only one expert for each token, adding computational overhead during inference solely for token routing. Compared to the computational cost of the causal language model itself, this additional overhead is minimal. As shown in the table below, our approach does not result in significant additional computational overhead.

W2: For tasks with clear distinctions such ... requirement to train a router).

A model with both generation and embedding capabilities can reduce deployment costs by eliminating the need to deploy multiple models. Meanwhile, the method proposed in this paper can also enhance the model’s resistance to hallucination to some extent by introducing bidirectional experts, which improve the model’s performance against linguistic priors. Some work has reflected the advantages of combining unidirectional and bidirectional models $1$

$2$ . Our results at Truthfulqa benchmark in Table 4 also show that the bidirectional module effectively reduces the LLM hallucination.

[1] LLM2Vec: Large Language ModelsAreSecretly Powerful Text Encoders [2] BatGPT: A Bidirectional Autoregessive Talker from Generative Pre-trained Transformer

W3: In Section 3.1, under "Preliminaries," ... based on this point is insufficient.

As shown in Table 1, there is a consistent trend between subsequent dependence and the general capability of the model. This indicates a correlation between subsequent dependence and the model’s general performance, suggesting a potential causal relationship.

Besides, we analyzed the relationship between generation performance and attention-based forward or subsequent dependence. As shown in the table below. As the level of subsequent dependence increased, the generation performance of the model declined. In contrast, our method reduces the subsequent dependence introduced by incorporating bidirectional experts while improving generation performance. This indicates a correlation between the distribution of attention weights and the model's generation performance.

Q1: What is the experimental setup for ... employ bidirectional attention or causal attention?

We provide detailed experimental settings in Section 5.1. For the evaluation of embedding capability in Table 2, we maintain the same settings as in the training phase and enable bidirectional attention.

Q2: In the results of Table 3, which layer of ... used during the evaluation?

Following the computation method described in Section 3.2, we use the average across all layers. During the calculation, we adopt causal attention to remain consistent with the generation task.

W4: The experiments seem to lack any comparison with previous state-of-the-art (SOTA) methods.

Thanks for your advice. We added MNTP-LLM as a baseline, which is trained using the Wikipedia dataset through the MNTP method. As shown in the table below, the experimental results demonstrate that UBMoE-LLM still exhibits strong performance.

Summary

Thank you for your helpful advice. We will improve our work based on your suggestions. It is worth noting that the concept of attention dependence proposed by us is highly correlated with the generative ability of the model in experimental verification, and has important significance for the study of unidirectional and bidirectional architecture of the model. We hope our response provides a clearer perspective on our work.