BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writing

Q3. Empirical results

A. WER on Rare Words 

We acknowledge that a well-aligned speech-text LLM model should ideally perform ASR tasks with a lower WER, and we recognize that a 20% WER might seem high. In our experiments, the WER remains approximately 20% even after removing stop words. This outcome can be attributed to our training approach, which emphasizes semantic alignment over lexical precision. For instance, phrases like "I love cats" and "I like cats" would generate the same continuation in our model, as the adapter isn’t explicitly trained to capture fine-grained lexical details.

In our continuing research, we have found that incorporating a modest amount (approximately 10%) of ASR data alongside our continuation writing data can notably enhance the WER, reducing it to about 4-5%, while simultaneously preserving the model's broad instruction-following capabilities. However, it is important to highlight that this paper primarily concentrates on developing methods to facilitate general instruction-following abilities. Consequently, we have intentionally chosen not to optimize performance for any specific task.

B. Impact of Different Prompts on WER

You are indeed correct in noting that different instructions can influence WER. As highlighted in footnote 4, employing the prompt "Please repeat the following words." results in lower WER scores—17.0 on LibriSpeech-dev and 17.4 on LibriSpeech-test. However, our current method aligns speech and text at a semantic level. Consequently, the adapter cannot capture detailed lexical information, making it difficult to address this issue merely through changes in prompts.

Q4. 'Continuation Writing' vs. 'Next Token Prediction'

The term 'next token prediction' applies broadly to a variety of LLM training scenarios, including those that involve the ASR task. In such cases, the model optimizes the prediction of the reference transcript through 'next token prediction,' utilizing both textual instructions and speech features. However, we have chosen to use the term 'continuation writing' in our work to specifically distinguish our training task from traditional 'ASR' or other similar tasks.

Q5. More detailed analysis on the multilingual space.

Thank you for the suggestion to quantitatively measure isomorphism.

We selected 1,000 quadruples (English text, English speech, French text, French speech) from the CVSS test set. We extracted text features at LLM input via word embedding and obtained speech features from the output of either the Whisper encoder or our modality adapter. Sentence embeddings were then computed through average pooling, allowing us to calculate the relational similarity.

As indicated in the table, the multilingual space of LLM inputs shows strong isomorphism, as does the space of Whisper outputs. Even when using only English ASR data for learning the transformation, the adapter’s output space retains the isomorphism characteristic of Whisper.

We appreciate your accurate assessment of the strengths of our paper and your insightful comments about its weaknesses. Before responding to your specific questions, we would like to highlight your observation regarding the use of our method in a general-purpose QA system. In Section 4.3, we demonstrated through cross-modal conversation demos that our model can process general-purpose speech instructions and generate accurate responses. A video demonstration showcasing this capability is available at https://anonymous4blsp.github.io/iclr24-3521.github.io/, as referenced in our paper.

Q1. Loss of Speaker and Paralinguistic Information

As discussed in Appendix D and referenced in the conclusion section, one of the limitations of our current method is the loss of speaker and paralinguistic information. Our focus is on aligning the semantic content of speech and text within the context of LLMs, a challenge that remains a focus of active research in the community [1,2,3]. Consequently, in the scope of our study, the statement "the LLM should behave the same, whether given the speech segment or its transcripts as input" holds true when considering semantic content alone.

However, our approach provides a pretrained end-to-end speech-text LLM with potential adaptability to capture speaker and paralinguistic information through fine-tuning on appropriate training data. Our analysis in Table 8 of Section 4.2 demonstrates that our model can be more effectively fine-tuned for downstream tasks (ST in this case) compared to models without pretraining or those pretrained with ASR. We plan to explore ways to capture speaker and paralinguistic information in future work.

$1$  Zhang, Dong, et al. "Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities." arXiv preprint arXiv:2305.11000 (2023).

$2$  Shu, Yu, et al. "Llasm: Large language and speech model." arXiv preprint arXiv:2308.15930 (2023).

$3$  Wang, Mingqiu, et al. "SLM: Bridge the thin gap between speech and text foundation models." arXiv preprint arXiv:2310.00230 (2023).

Q2. Representations of Modalities

A. Representation Collapse

The occurrence of mode collapse in speech features resulting from ASR training is not immediately evident. In computer vision, BLIP2 [4] has shown that training solely on image caption data can enable general instruction-following capabilities for zero-shot image-to-text generation tasks. Similarly, in the field of speech input, pretraining the model using the ASR task has been a common practice in prior work [1,2]. However, the benefits and limitations of this approach were not thoroughly studied.

We have explored different methods to utilize the ASR task for enabling instruction-following capabilities. As described in footnote 2, we used GPT4 to generate 100 diverse prompts for the ASR task, yet representation collapse still occurred. We also experimented with training on both English ASR and English-to-Chinese speech translation (ST) tasks using respective instructions. However, the resulting model still failed to demonstrate instruction-following capabilities for unseen tasks. Generally, training on specific downstream tasks tends to limit the model's abilities to those particular tasks, making it challenging to generalize to unseen instructions. This phenomenon has also been highlighted in a recently released work [5]. Additionally, we want to emphasize that labeled speech-text training data is quite limited outside of ASR and ST.

$4$  Li, Junnan, et al. "Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models." arXiv preprint arXiv:2301.12597 (2023).

$5$  Pan, Jing, et al. "COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learning." arXiv preprint arXiv:2311.02248 (2023).

B. Choice of Fixed-Length Subsampling

Thank you for highlighting this important aspect and suggesting the exploration of dynamic downsampling strategies. We recognize that different downsampling methods can impact the model's performance, and there are indeed various design options to consider. In this study, our primary focus was on investigating the instruction-following capability for speech input. For this purpose, we chose a simple yet efficient approach, employing a CNN followed by a bottleneck layer as the adapter structure.

We are currently exploring more sophisticated methods for the adapter, which could potentially enhance the model's performance further. However, this exploration is part of our ongoing research and will be detailed in a future publication.

Thank you for your feedback. It appears there may be some misunderstanding regarding the motivation, contributions, and experimental design of our work. We encourage you to refer to the "General Response" provided above and the point-by-point response below for clarifications. We sincerely hope that these clarifications will be helpful for your reassessment of our research.

Q1. The Goal of The Paper

While we believe that an end-to-end approach can eventually outperform a pipeline approach with expanded capabilities (e.g., using paralinguistic information), that is not the immediate goal of this paper. As explained in the "General Response," our objective is to extend the instruction-following capability of LLMs to speech input to enable zero-shot capabilities for unseen tasks in an end-to-end model.

In Section 4.1, we demonstrate this by using textual instructions to prompt the model for unseen speech-to-text generation tasks. For instance, our model, trained solely on continuation tasks, can be prompted during inference for ASR tasks as follows:

###[Human]: Please transcribe the following speech into text. [speech features]
###[Assistant]: [response]

Here, the ASR instruction can be substituted with instructions for other tasks, such as ST and SLU tasks evaluated in the paper, or tasks like summarization and question-answering. We emphasize that our goal in this paper is not to outperform the pipelined approach, but to demonstrate the model's ability to follow instructions across modalities in an end-to-end manner. The comparison with the pipeline approach is to both highlight the promising results of the end-to-end approach and the performance gap with the pipeline approach, encouraging future research to further reduce this gap and eventually surpass it. This aspect is discussed in the limitations of the current approach in the Appendix and referenced in the conclusion section.

In Section 4.2, we present analyses to highlight different aspects of our model. However, these should not be viewed as the main experiment. Table 8 illustrates that our method allows for effective fine-tuning on downstream tasks. Figure 3 demonstrates successful speech-text modality alignment, while Figure 4 investigates the impact of data scale on alignment effectiveness.

Q2. Performance Gap with Baseline Models

As explained in the response to Q1, the focus of our approach is on cross-modal instruction-following capabilities, rather than on the improvement of performance in specific tasks. Similar to the early days of end-to-end speech translation, our approach is not immediately expected to surpass the performance of ASR+LLM or Text+LLM models. Our primary contribution lies in proposing an end-to-end training methodology for speech-text LLMs, thereby demonstrating the model's valuable ability to follow instructions across modalities, despite the current performance gap. We hope our work will be beneficial for the research community in progressing towards the ultimate goal of developing more effective end-to-end approaches.

Q3. Minor Improvement in Table 8

As noted in response to Q1, our main focus in this paper is on the instruction-following capabilities for speech input, as presented in quantitative zero-shot evaluations in Section 4. The analyses in Section 4.2 are designed to highlight different aspects of our model. Specifically, Table 8 in Section 4.2 is intended to demonstrate that our pretrained model is effective not only in its instruction-following capabilities but also for fine-tuning on downstream tasks. The improvement over models without pretraining is substantial in both BLEU and COMET metrics, and is significantly better than ASR pretraining, with an average improvement of 0.5 BLEU and 1 COMET score across eight directions. However, we wish to emphasize that this aspect of the model is not the main focus of our paper.

Thank you for recognizing the value of our work and for your insightful comments. Below are our responses to your questions.

Q1. Choice of Speech Encoder & Performance Gap with Text-Based Model

We acknowledge that using Whisper-small as the speech encoder might not represent the strongest baseline. However, our primary focus is not on the model's performance on downstream tasks, but rather on demonstrating our training method's ability to transfer instruction-following capabilities from text-based LLMs to the speech modality. As you pointed out, we opted for Whisper-small due to computational considerations during the development of our approach. However, we did conduct some experiments (not included in the paper) with Whisper-large models and observed improved zero-shot performance on unseen tasks. These experiments, however, did not alter the paper's primary conclusions.

As highlighted in Appendix D and referenced in the conclusion section, the performance gap compared to the pipelined approach remains a limitation of our current approach. We are actively exploring various strategies to bridge this gap and to expand the model's capabilities. It is our hope that the methodology proposed in this paper will contribute to the research community's efforts toward achieving these goals.

Q2. Overlapping Speech Embeddings

You are correct in your observation. Models trained with ASR data tend to lose their ability to follow diverse instructions. When different instructions are used to prompt speech input, nearly identical features are produced, leading to the overlapping of variously colored triangles in Figure 1. We will make this point clear in the revision.

Q3. Choice of 8x Reduction Factor

We agree that a larger reduction factor could diminish lexical information, as demonstrated in [1]. However, a smaller reduction factor would result in longer speech features, excessively consuming context in the LLM. This could decrease training and inference efficiency and limit the number of rounds in multi-turn interactions. In our empirical studies, we observed that 1 second of speech corresponds to approximately 3-4 text tokens. Given that the output frequency of the Whisper encoder is 50Hz, we chose an 8x downsampling factor to balance training efficiency with the preservation of lexical information. We believe that choosing a larger or smaller reduction factor would not alter the main conclusions of the paper.

$1$  Fathullah, Yassir, et al. "Prompting large language models with speech recognition abilities." arXiv preprint arXiv:2307.11795 (2023).

Q4. Using a Different Speech Encoder?

We acknowledge that there are various options for the speech encoder, such as the Whisper model's encoder [1], USM [2], or other pre-trained speech models, which could impact the model's performance. However, our work focuses on the methodology of aligning a frozen speech encoder with a frozen LLM to achieve instruction-following capability for speech input on unseen tasks. As such, an in-depth investigation into different speech encoders is not the primary focus of this work, and we believe that changing the speech encoder would not fundamentally alter the conclusions of our paper.

We chose the encoder of the Whisper model due to its widespread recognition and to facilitate comparison with a strong ASR model in the pipelined approach. This choice mirrors our decision to use Llama2 as the LLM, despite the availability of many open-source LLM options. Our aim was to employ well-known and robust components to showcase the effectiveness of our methodology.

$1$  Shu, Yu, et al. "Llasm: Large language and speech model." arXiv preprint arXiv:2308.15930 (2023). 

$2$  Wang, Mingqiu, et al. "SLM: Bridge the thin gap between speech and text foundation models." arXiv preprint arXiv:2310.00230 (2023).

Thank you for your feedback. It appears there may be some misunderstanding regarding the motivation, contributions, and experimental design of our work. We encourage you to refer to the "General Response" provided above and the point-by-point response below for clarifications. We sincerely hope that these clarifications will be helpful for your reassessment of our research.

It's important to note that while we included studies on ASR task-based pretraining, our goal was to demonstrate the inadequacy of ASR task-based pretraining for our purpose, as models trained in this manner cannot follow general instructions at all. Therefore, in all experiments evaluating zero-shot instruction capabilities – which are central to our study – we do not compare our results with those of ASR task-based pretraining. The comparison with ASR task-based pretraining is made only in the fine-tuning experiment in Section 4.2. This is to illustrate that BLSP is a more effective pretraining method for fine-tuning on downstream tasks.

Q1. Use of ASR System as the Speech Encoder?

We believe there is a misunderstanding that we would like to clarify. The Whisper ASR system is comprised of an encoder for acoustic feature extraction and a decoder for text generation. In our work, we utilized only the pretrained Whisper encoder as the speech encoder for extracting acoustic features, while discarding the Whisper decoder. Our proposed model functions as an end-to-end system, where the speech encoder's role is to extract features from the input speech, rather than to transcribe the speech into text. Including a Whisper decoder, as used in a standard ASR system, would necessitate an argmax operation, which would impede the feasibility of training this model in an end-to-end manner. Additionally, this would introduce the limitations of a pipelined approach, as discussed in our "General Response."

The modality adapter plays a significant role in aligning speech features with the textual space, and it does not merely perform an identical transformation. Learning a mapping that effectively transfers the instruction-following capability from text to speech input is a nontrival task. As demonstrated in Section 2, a modality adapter trained solely on the ASR task lacks general instruction-following capabilities.

Q2. Choice of Adapter Structure

The modality adapter in our study serves two primary functions. First, it reduces the length of output from the Whisper encoder, which operates at a frequency of 50Hz. Second, it maps the speech feature space to the textual space of the LLM. We acknowledge that various structural options exist for the adapter. For length reduction, methods like CNN downsampling, frame stacking, or q-transformer are potential choices. Similarly, for cross-modality alignment, both MLP and transformer-based methods are feasible.

However, our study's focus was not on exhaustively exploring different adapter structures. Instead, our main contribution lies in proposing a novel training method to connect a frozen speech encoder with an LLM. Consequently, we opted for a simple yet effective approach, employing a CNN followed by a bottleneck layer for the adapter. This decision allowed us to concentrate on the core aspect of our research while maintaining effectiveness in achieving our objectives.

Q3. Performance of Fine-Tuned ASR Module in a Cascaded System

BLSP is designed as an end-to-end speech-text model. In the experiments outlined in Table 8, we fine-tuned the entire model using speech translation (ST) data via an end-to-end instruction-tuning approach:

###[Human]: Please translate the following speech into German. [speech features]
###[Assistant]: [translation]

During this process, the Whisper encoder was fine-tuned along with other model parameters, but the corresponding Whisper decoder was not used or fine-tuned. Consequently, it's not feasible to measure the ASR performance of the fine-tuned Whisper encoder as would be done in a traditional Whisper ASR system. Moreover, since the ST data comprises (speech, translation) pairs rather than (speech, transcription) pairs, it does not lend itself to optimizing a separate Whisper ASR system. Therefore, we are unable to report the performance of a cascaded system using a fine-tuned ASR module.

The purpose of Table 8 is to show that our proposed approach, in comparison to direct training on ASR data, offers better generalization and is more suitable for fine-tuning on downstream tasks. However, we emphasize that this is not the main contribution of our work. Our primary contribution, as detailed in Section 4.1, is the demonstration of the cross-modal instruction-following capability of our model and its ability to generalize to unseen tasks through text instructions.