Are Bert Family Good Instruction Followers? A Study on Their Potential And Limitations

Thanks for your critical feedback, which is crucial in improving this work. We hope the following responses can answer some of your questions and mitigate your concerns.

To Question 1: "Why don't...".
Response: We agree that providing the results of XMLR without instruction-tuning can clarify whether the relative improvement comes from instruction tuning. However, it is difficult to collect the evaluation results of XMLR-w/o Instruction tuning owing to its poor performance. We provide several examples below to give an intuitive explanation (we italicize the content of the prompt). You can find more examples in Appendix A.6 in the revised paper.

We find that XML-R fails to complete these tasks without instruction tuning, i.e., it generates the special tokens [PAD] in XNLI and unrelated texts in machine translation.

To Question 2: ''I am a bit confused..."
Response: We are sorry for the confusion! Exactly, our implementation is what you have summarized "you just concatenate ...''. We also describe this in the left of Figure 1 (begin with "In practice").

We describe "it as if you train ...'' at the beginning of Section 3.2 to give a clear motivation for our concrete implementation.

1. A separate "decoder" is only needed if we fully follow the practice of the original encoder-decoder model. ---> 2. "This additional MLM can completely share parameters with the MLM encoder. Thus, we can adopt only one pre-training BERT model" ---> 3. The original mix-attention mechanism first acquires the last layer's source representation, and we must pass the model twice. ---> 4. We transform the original mix-attention mechanism into the dynamic mix-attention mechanism, leading to the current training format to improve training efficiency.

To Concern 1: "What's the core contribution..."
Response: We agree with you that this version merely pointed out and preliminarily analyzed some of the potential and limitations of existing Encoder-only LMs in completing various generation tasks in an instruction-following manner. There is still much work to do in the future (as another reviewer said "It can potentially lead to another series of research.'').

Though preliminary, this paper still provides the following observations to the community.

• It is a common impression that Decoder-only and Encoder-Decoder models are good at following instructions in generating the expected output for various tasks, including understanding, extraction, generation, etc. We demonstrate that Encoder-only also has the potential to follow instructions and complete various tasks in a generation manner.
• The BERT family plays a vital role in the development of pre-trained LMs and are still strong competitors in many scenarios, e.g., understanding and information extraction. It is might not surprising that the BERT family can complete simple classification and understanding tasks with zero-shot prompt learning, given proper task format and subtle label name search strategy. However, no existing works provide suggestions and observations on whether the BERT family can follow instructions and complete various tasks in a generation manner. This paper fills this blank.
• As discussed in Question 1, Encoder-only models cannot directly complete various tasks in a generation manner. We provide a basic method with details that allow for such an exploration.
• We present the key factors to make Encoder-only models competitive with the other two variants and disclose their limitations, which might promote further study.
• From the aspect of non-autoregressive generation, this is the first work that builds an instruction following models that can generalize across tasks and languages based on pre-trained LMs.

We also agree that "It would be truly surprising ...". But we also believe the future success of any possible alternatives of existing methods/models can start from "potential", .e.g., better than mT0 on XCOPA tasks, 3.1x faster than AR models.

Last, we have removed "surprisingly" from our paper.

To Concern 2:"The experiments..."
Response: We agree that "many confounders exist ...". The trends of close-source and soaring computation costs of LLMs make it difficult (sometimes impossible) to exactly control variables in experiments. To avoid any misunderstanding in our observations, we use"potential, risks, possible, etc" throughout the whole paper if necessary. On the other side, empirical observations and suggestions without fine-grained and strict control have also promoted the development of LLMs in the past year.

Response to the question: Thank you for acknowledging the value of this paper! We are not sure how much benefit the transformer decoder architecture brings. However, we do have some empirical observations about the effects of model architecture and the decoding paradigm.

• Though the community of decoder-only structures is much larger than other architectures (owing to their merits of sample-efficiency, structure simplicity, ease of monitoring during pre-training, etc.), other architectures like transformer Encoder-Decoder (e.g., FlanT5-UL2 [1], AlexaTM [2]), RWKV [3], Convolutional Seq2Seq [4] can also achieve very promising generation performance with the next-token prediction formulation.
• The next-token prediction (auto-regressive) decoding paradigm is one of the most effective strategies in various generation tasks, which is usually better than other paradigms, such as semi-autoregressive and non-autoregressive.

We firmly believe there exist alternative architectures. There is no theoretical guarantee to prove that the existing decoder-only structure is optimal or a good sub-optimal.

We think the most three import factors in self-supervised pre-training are data, model capacity, and objectives, where 1) data decides the patterns to be discovered and composed, 2) model capacity corresponds to the capability to memorize, manipulate, and generalize existing patterns, 3) the objectives correlate the learning efficiency for different patterns. In other words, the model capacity might be more important for LLMs than the structure.

BTW, we also believe many valuable techniques can be started from “competitive” (like RWKV) or “potential”.

To Comment 1: Thanks a lot for pointing out our mistakes. We have fixed the citation format issues in the latest paper version.

[1] Scaling Instruction-Finetuned Language Models
[2] AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model
[3] RWKV: Reinventing RNNs for the Transformer Era
[4] Convolutional Sequence to Sequence Learning

Thanks a lot for the very helpful comments in the first round of discussion! We hope the following responses can answer some of your questions.

To Comment 1: "Longer sequence generation tasks like dialogue, summarization, etc."
Response: Exactly! It is essential to discuss the performance of the introduced method on longer sequence generation tasks. Besides machine translation, we further provide evaluation results on 5 additional sequence generation tasks with 6 datasets (with longer output sequences than most previously reported tasks). We summarize these tasks and datasets below. More details of these tasks and datasets can be found in Appendix A.4 of our revised paper.

We present the experimental results below, along with preliminary analysis. We give the variance (this number denotes the performance range, i.e., the performance boundary) of 5 different prompts in the bracket. Since length prediction plays a key role in language generation tasks in non-autoregressive models, we also report the results of Instruct-XMLR decoded with ground truth length (w/ GL) to explore the effects of length prediction.

R-1/2: ROUGE-1/2, B-2: BLEU-2, D-2: Distinct-2

In general, we can find that:

• The results in most tasks are consistent with what we mentioned in the original paper, i.e., Instruct-XMLR-XL outperforms BLOOMZ-3B on 4 out of 6 tasks (with the left 2 tasks comparable) and underperforms mT0-3.7B without target length.
• Length prediction significantly impacts some specific tasks where there is no strong alignment relationship between source length and target length (e.g., COMMONGEN and PersonaChat). Since the BLEU score is sensitive to length, length prediction leads to a serious decline compared to decoding with ground truth length.
• Instruct-XMLR can generate more diverse texts for all text generation tasks due to the non-autoregressive generation manner, i.e., unlikely to be trapped by common n-gram patterns.

To Comment 2: "To analyze how stable the method is. Need to report the variance or significant test."
Response: We are sorry to miss it in the previous version. In our revised PDF, we supplement the difference of various prompts and the correlated analysis in Appendix A.3.

To Comment 3: "The proposed method cannot handle fewshot prompt learning."
Response: We have tried to make up for this flaw by including some few-shot instruction data for training rather than the only zero-shot instruction data we adopted in our paper. Unfortunately, this doesn't bring any profit. Part of the results are given below, where more details can be seen in Appendix A.5.
" w/zs denotes the model trained on zero-shot instruction data, w/fs denotes the model trained on mixed zero/few-shot instruction data, w/ICL4 denotes we adopt four examples in the demonstration for few-shot prompt learning. ''

• Results for seen languages.

  Model	XCOPA	XNLI	XWinograd
  Instruct-XMLR w/zs	0.646	0.518	0.582
  Instruct-XMLR w/zs w/ICL4	0.548	0.414	0.547
  Instruct-XMLR w/fs	0.642	0.489	0.565
  Instruct-XMLR w/fs w/ICL4	0.571	0.442	0.553

To Question: We are now experimenting with the biggest version of the BERT Family, the 11B XMLR-XXL model, with only around 1/2 data of the original Instruct-XMLR-XL model being processed till now. We present the intermediate results below to give intuitive analysis.

• Results for unseen languages.

  Model	XCOPA	XNLI	XWinograd
  Instruct-XMLR-XL	0.599	0.466	0.539
  Instruct-XMLR-XXL	0.593	0.523	0.562

We can find that:

• With only 1/2 instruction data, Instruct-XMLR-XXL already outperforms Instruct-XMLR-XL on XNLI and XWinograd, indicating the positive effects of larger model size.
• Compared with Instruct-XMLR-XL, Instruct-XMLR-XXL seems more effective for language generalization.