SSR: Alignment-Aware Modality Connector for Speech Language Models

We deeply appreciate the time and effort you have dedicated to reviewing this paper. We value your insights, and in response to your points of concern, we offer the following explanations:

…some discussions on comparing the proposed method with cascade systems should be emphasized.

Thank you for raising this concern. We will add further motivation to clarify how our system differs from the cascaded system. The primary advantage of an end-to-end system is that the language model is exposed to encoded speech information, rather than just textual information. This allows an end-to-end system to leverage both semantic and paralinguistic information. We kindly refer you to our "general response to all reviewers" for additional experiments we conducted on the non-semantic aspects of our model. In this work, we focus on the semantic aspect of SpeechLM because there are still gaps between end-to-end systems and cascaded systems in terms of speech and text understanding. Our goal is to bridge these gaps with our modality fusion approach.

what is the unit of the transcription sequence y for alignement?

The unit of transcription sequence is subword level (from Llama3’s tokenizer). We kindly refer you to line 298-305 for details.

why are both similarity loss and MSE loss used? How to determine the weight for combining them?

The use of Cosine and MSE loss is an empirical choice. During training, we observed that the MSE loss initially decreases to about 0.1, after which the Cosine loss begins to decrease. We found that using Cosine loss in conjunction with MSE loss provides better guidance. This is because Cosine loss encourages speech representations to maintain similar relational structures to text embeddings, rather than focusing solely on the absolute differences in embedding vectors, as optimized by MSE. To determine the appropriate weight, we monitored the scale of both MSE and Cosine loss after some training and selected $\lambda =5$ to ensure that the Cosine loss is on the same scale as the MSE loss.

Why is the Char-CTC model used for evaulation since it didn't perform as good as UnityY2 in Section 4?

The findings of our stage fine-tuning should generalize to different aligners. We choose Char-CTC because it might be more useful in practice as it does not require transcripts to compute the alignment.

LLaMa2 was used for the cascade system while LLaMa3 was used in the proposed method. This seems unfair.

We acknowledge that using the Llama3 based cascaded system is more fair. However the difference between Llama2 and Llama3 on the speech understanding benchmarks (sWUGGY, sBLIMP, and Storycloze) will be marginal because these benchmarks’ text sentences are not hard for text-only models. Therefore, we simply re-use the results from the SpiritLM paper to save computational efforts as the primary comparison is made between our SSR-based system and prior SpeechLM.

Dear reviewer j1u2,

We sincerely appreciate your valuable suggestions for improving our paper and hope our response has effectively addressed your concerns. Should you have any additional suggestions or questions, please do not hesitate to let us know. We would greatly appreciate it if you could consider raising the rating. Thank you!

Best regards, Authors

We deeply appreciate the time and effort you have dedicated to reviewing this paper. We value your insights, and in response to your points of concern, we offer the following explanations:

However, this paper lacks an analysis of the proposed model’s understanding of paralinguistic information and focuses solely on performance from a content comprehension perspective.

Thanks for raising the concern over paralinguistic information and we have conducted some additional analysis to show that, even though our focus is on semantics for modality fusion, the paralinguistic information can be preserved through speech encoder and utilized by the SpeechLM. We kindly refer you to our “general response to all reviewers” for more details.

However, given that many prior SpeechLMs still lacks behind cascaded system in terms of speech/text understanding, we believe that better modality fusion approach that improves speech understanding and mitigates catastrophic forgetting over text modality is meaningful, even if only semantic (content comprehension) is considered.

It is challenging for the approach to capture paralinguistic information from speech.

We respectfully disagree with the assertion regarding the model’s ability to capture paralinguistic information. Firstly, since speech representation is integrated into language models, paralinguistic information is retained, as demonstrated by our additional analysis using the Expresso dataset. The key challenge lies in teaching models to utilize this information effectively and improving the encoding of paralinguistic information within the speech encoder. Our analysis shows that the model can capture some paralinguistic information through in-context learning, without requiring emotion-relevant fine-tuning or expressive tokens.

Secondly, considering how other works, such as Salmonn, enable emotion recognition, their connector pre-training step also focuses primarily on semantics, using ASR format data to pre-train the Q-former based connector. They then teach models to utilize paralinguistic information through instruction fine-tuning. Our proposal is a foundational SpeechLM that emphasizes content comprehension, and our method is complementary to any instruction fine-tuning (Salmonn, Qwen2-audio) or special expressive token approach (SpiritLM) aimed at utilizing paralinguistic information (e.g., for emotion recognition or expressive speech generation).

In my view, using a model that doesn’t rely on transcription for alignment would better align with the scenario proposed in the paper.

We agree with your point that an aligner is more practical if it does not rely on transcription, which is why we develop CTC and CIF-based aligners for the SSR-connector. We kindly refer you to Section 3.2 (lines 174-177), where we discuss our motivation for using a CTC-based aligner to support textless alignment computation. In all of our experiments, we discussed the CTC-based SpeechLM and demonstrated its superior performance over prior systems in both spoken language understanding and text understanding.

Could you include more baseline models, such as SALMONN and Qwen2-Audio?

Thanks for the suggestion and we acknowledge that SALMONN and Qwen2-Audio are indeed capable speech-text systems. To alleviate your concern, we benchmark SALMONN over Storycloze (S->T) and Speech MMLU and compare it with our system. We kindly refer you to our “general response to reviewers” for details for results.

However, we would like to emphasize that our focus in the paper is on how to improve the connector-based modality fusion for better foundational SpeechLM, instead of how to better instruction-finetune a SpeechLM for various downstream tasks as focused by SALMOON or Qwen2-Audio. Therefore, it is more reasonable for us to compare with prior systems on speech/text understanding benchmarks.

Thank you very much to the author for the efforts made in the rebuttal. However, I still have the following questions:

However, given that many prior SpeechLMs still lacks behind cascaded system in terms of speech/text understanding, we believe that better modality fusion approach that improves speech understanding and mitigates catastrophic forgetting over text modality is meaningful, even if only semantic (content comprehension) is considered.

I completely agree with the author that catastrophic forgetting and modality fusion are critical topics in SpeechLM. However, the proposed approach focuses solely on content and heavily relies on ASR data throughout the training process, which essentially rules out the possibility of using other tasks for training. Therefore, it’s not simply a matter of "even if only semantic (content comprehension) is considered," but rather that evaluation is restricted to the semantic level alone.

Firstly, since speech representation is integrated into language models, paralinguistic information is retained, as demonstrated by our additional analysis using the Expresso dataset.

Thank you to the author for conducting the additional experiment. Even if the author demonstrates in the experiment that SSR-Connector can capture emotion information (although I believe it is not as effective as Qwen-Audio and SALMONN in this aspect), its capability is still very limited and does not prove that SSR-Connector performs well in understanding elements beyond content. Additionally, other paralinguistic information, such as age and gender, is likely even more challenging to capture effectively.

Our proposal is a foundational SpeechLM that emphasizes content comprehension, and our method is complementary to any instruction fine-tuning (Salmonn, Qwen2-audio) or special expressive token approach (SpiritLM) aimed at utilizing paralinguistic information (e.g., for emotion recognition or expressive speech generation).

First, the foundational SpeechLM is not solely focused on content comprehension. The first phase of training for both Qwen-Audio and Qwen2-Audio includes various speech-related tasks. Second, your argument is not very convincing because it does not demonstrate whether this is indeed a good foundation model. Specifically, the model after the instruction fine-tuning phase is more important and practically valuable, but you have not provided experimental evidence that SSR-Connector can retain its speech/text understanding capabilities after instruction fine-tuning. Finally, you claim that your proposal is a foundational SpeechLM, but I cannot find the paper provides any explanation to state this point, which contradicts your response.

We appreciate your additional responses and questions. Before addressing them, we'd like to invite you to review our work from a different mindset. It appears that you have a preconceived notion of an ideal system (a capable SpeechLM that can perform multiple tasks, understand both semantic and paralinguistic information) based on prior work such as Salmonn or Qwen-audio. You then compare our work to this ideal system, highlighting aspects where we fall short (e.g., emotion recognition, multitask pre-training). However, our focus is entirely different. Since our submission is not a product/system demo, we propose a more suitable way to assess our paper:

• Review our problem definition: Is the speech-text modality fusion problem worth investigating?
• Review our method (SSR-connector): Does it effectively address the identified problem?
• Review our experiments/analysis: Do our empirical results demonstrate the efficacy of the proposed method, and do our analyses provide new insights into the problem?

To challenge our method, we suggest focusing on aspects where our approach may not adequately address the raised issue (modality fusion, catastrophic forgetting, etc.). Avoid introducing new problems (e.g., emotion recognition or multitask pre-training) that are not within our scope to solve. If you wish to hold us to a different standard by redefining/expanding the problem scope to paralinguistic information or instruction-finetuning stage, you must first discredit the importance of the problem we are solving (i.e., show that modality fusion or catastrophic forgetting is a trivial or already-solved problem for the scientific community).

Now we provide detailed response to your questions below:

However, the proposed approach focuses solely on content and heavily relies on ASR data throughout the training process, which essentially rules out the possibility of using other tasks for training.

Since ASR data has the largest quantity (due to the ease to synthesize them), it is natural to focus on this data format for pretraining (and previous capable systems like Moshi, Salmonn, SpiritLM, TWIST, etc., all focus on ASR data for pretraining). Additionally, our method works with any kind of speech-text, speech-only, or text-only input data, so please provide clarification on the “other tasks” you are referring to here.

its capability (on non-semantic aspect) is still very limited…other paralinguistic information, such as age and gender, is likely even more challenging to capture effectively.

we provide our few-shot experiments on Expresso to show that non-semantic information is retained and we also stress that our method is complementary to approaches that facilitate model understanding beyond semantics. Since the non-semantic aspect is not the focus of our study, it is beyond the scope for us to prove our method can outperform previous studies in this aspect.

...your argument is not very convincing because it does not demonstrate whether this is indeed a good foundation model.

Please refer to our evaluation on spoken language understanding and MMLU.

Specifically, the model after the instruction fine-tuning phase is more important and practically valuable...

The notion of “instruction fine-tuning phase is more important and practically valuable” is very problematic (a pre-trained LLM like Llama3 is not useful compared to llama3-instruct, does this mean pre-training method and details of Llama3 are not worth publishing if no instruction tuning result is reported?).

Please be aware that this is a paper submission, not launching a new product/speech-text system. Our paper aims to address and provide more insights on speech-text modality fusion in the context of a connector-based SpeechLM and we focus on the pre-training stage. Instruction-finetuning phase experiments are nice to have but should not be used as critism to reject the paper, similar to emotion/paralinguistic information.

Finally, you claim that your proposal is a foundational SpeechLM, but I cannot find the paper provides any explanation to state this point, which contradicts your response.

While we did not explicitly use the term "foundational SpeechLM" in our paper, our method focuses on training speech connectors for SpeechLM and did not use instruction-tuning dataset. Moreover, our evaluation suite on spoken language understanding and text understanding should make it clear that our model is a foundational model rather than an instruction-fine-tuned one. Our comparison is also made with foundational SpeechLM like SpiritLM. We will add additional clarification in our paper to make this point clear. Thanks for raising this concern!

We hope you could consider changing your rating criteria. Your points about paralinguistic/instruction-tuning are valid. However, there is no perfect work that can target all aspects of SpeechLM. We hope you could assess the quality of our work following our motivation and problem definition.

We deeply appreciate the time and effort you have dedicated to reviewing this paper. We value your insights, and in response to your points of concern, we offer the following explanations:

Speech often results in catastrophic forgetting...Have you quantified the severity of this issue?

Yes, to understand how modality fusion impacts the original LLM's performance on the text modality, we kindly refer you to our experimental results (MMLU performance) in Table 4, Table 6, and Figure 5. Here are some figures to provide a rough sense of catastrophic forgetting: For SpiritLM [1], the injection of speech tokens results in an MMLU drop from 65 to 53.5. In our method (see Section 5.1), vanilla fine-tuning decreases MMLU from 65 to 48.8 after 20k steps, whereas a multitask training setup maintains an MMLU of 63.1.

Other prior works [2,3] also mention, though not always quantified, catastrophic forgetting in their studies. For instance, Moshi [2] observed a drop in MMLU from 54.3 to 49.8 after incorporating speech information.

is there a risk of forgetting within the LLM itself? Specifically, I am referring to the LLM's performance before and after incorporating the speech modality.

After incorporating the speech modality, the LLM's performance on NLP tasks tends to degrade, as evidenced by the drop in MMLU. A key contribution of our work is to mitigate this degradation through our alignment-aware speech connector, which only results in a drop of 2 points compared to severe degradation from prior work like SpiritLM.

However, if we have misunderstood your question, we kindly ask for further clarification on what you mean by "forgetting within the LLM itself."

Could you provide the specific figure for the original Llama3 as listed in Table 6?

We kindly request more clarification on this question. Are you referring to the original (text-only) Llama3 model’s performance on MMLU in Table 6? If so, it is 65.3 (same as our stage1 distilled model because we did not modify Llama3 weight during stage 1 distillation)

Could you provide further explanation for the statement "lack the ability to support text or speech generation unless further instruction-finetuned"

Thanks for raising this question. Here we are referring to the connector-based methods [3,4,5], which uses ASR format data to fine-tune the connector. After the connector is trained, the SpeechLM can only perform the specific kind of task it is trained on (e.g., speech recognition or translation) but cannot serve as a foundational SpeechLM like SpiritLM or our model. We also provide further comparison with connector-based model like SALMONN [3] in our "general response to all reviewers"

...In that case, why not simply use an ASR with text (or sub-word units) output directly?

Thank you for the question. We agree that the cascaded ASR+LLM system is a straightforward and effective approach. However, from our perspective, modeling speech and text jointly within an end-to-end system is the future for SpeechLM. This approach allows the system to leverage information beyond text, which relates to your next question: Does the Speech Signal Representation (SSR) convey any audio/voice information beyond text?

Currently, we still observe a significant performance gap between previous SpeechLM models and the cascaded baseline, as shown in Table 4. This observation motivates us to develop a more effective modality fusion approach to bridge the gap between an end-to-end SpeechLM and the cascaded system.

Does the Speech Signal Representation (SSR) convey any audio/voice information beyond text? If so, have you conducted an analysis on this aspect?

Thank you for raising the concern regarding paralinguistic information. We have conducted additional experiments to demonstrate that our model retains information beyond text. We kindly refer you to the "general response to all reviewers" section for more details on these experiments. The results indicate that paralinguistic information is preserved in the representation, which opens up opportunities for future research to integrate our modality fusion method with other approaches to learn non-semantic information. Lastly, we would like to emphasize that the primary focus of our study is still on semantics. We believe that bridging the gap between the end-to-end SpeechLM and the cascaded system is meaningful in its own right.

References:

[1] Nguyen et al., (2024). Spirit LM: Interleaved Spoken and Written Language Model

[2] Défossez et al., (2024). Moshi: a speech-text foundation model for real-time dialogue

[3] Tang et al., (2024). Salmonn: Towards generic hearing abilities for large language models

[4] Wu et al., (2023). On decoder-only architecture for speech-to-text and large language model integration

[5] Yu et al., (2023). Connecting speech encoder and large language model for asr

Dear reviewer 98Pj,

We sincerely appreciate your valuable suggestions for improving our paper and hope our response has effectively addressed your concerns. Should you have any additional suggestions or questions, please do not hesitate to let us know. We would greatly appreciate it if you could consider raising the rating. Thank you!

Best regards, Authors

We deeply appreciate the time and effort you have dedicated to reviewing this paper. We value your insights, and in response to your points of concern, we offer the following explanations:

…it does not compare enough to Figure 1(a).

Thanks for raising the concern. For previous connector-based models (using compressor like Figure 1(a)), they either only focus on a special speech-to-text task like speech recognition or speech translation [1, 2], or necessitates instruction-tuning to support speech-text continuation [3]. If we compare with models like [1,2], their performance on spoken language understanding will be random as such models are only trained toward recognition/translation. For systems like SALMONN [3], to compare with their tasks, we need to perform instruction-tuning with our foundational SpeechLM, which is beyond the focus of this work.

However, we added experiments to benchmark SALMONN on the Storycloze and Speech–MMLU dataset, and we kindly refer to our “general response to all reviewers” for details. We would like to highlight that our modality fusion mechanism is used to train speech-text foundation model, therefore our evaluation is focused on understanding and text-ability preservation rather than downstream applications, which is therefore more aligned with our compared prior works like SpiritLM, TWIST, VoxtLM etc.

A single-stage or 2-stage baseline with a much simpler 1(a) design can motivate the more complex design in this paper

Thank you for the suggestion. We would like to clarify that the approach depicted in Figure 1(a) is not compatible with our two-stage training process because the compressed speech representation does not match the length of text tokens. Our ability to employ two-stage training stems from the fact that our compressed speech representation has the same shape as text tokens.

Regarding the single-stage baseline, our preliminary experiments without distillation (as mentioned in lines 213-215) resulted in severe catastrophic forgetting, with MMLU dropping below 30 and suboptimal speech understanding. We will add further clarification on these points to enhance the clarity of the paper.

Need to explain why unitY2 based method is much better than others

Thanks for the suggestion and we would like to refer you to section 4.3 for our aligner comparison. Basically unitY2 aligner is better because it produces better alignment (as verified by the word boundary error and word duration in Table 3) compared to CTC, which helps the stage 1 distillation to match speech to text representation.

The aligner in the proposed method is trained with paired ASR speech-text data, which is unfair to compare with SPIRITLM

Thank you for your comment. We agree that leveraging segmentation from pre-trained aligners provides a strong prior for our SpeechLM to excel in the ASR task. However, we respectfully disagree with the notion that comparing our model with SpiritLM is unfair. Since both our SpeechLM and SpiritLM are foundational models, we believe it is appropriate to evaluate them on the same tasks. Note that even if SpiritLM is explicitly fine-tuned with ASR data (as shown in Table 5 of SpiritLM paper[4]), its 10-shot ASR performance (WER=6.0) is still worse than our best system.

Missing description of speech-MMLU

Thank you for raising this concern. Currently due to page limitation, we provide a brief description of Speech-MMLU dataset from line 263-268, and put additional information in Appendix C. Please let us know if you need any further clarification/information about the Speech-MMLU dataset.

References:

[1] Wu et al., (2023). On decoder-only architecture for speech-to-text and large language model integration

[2] Yu et al., (2023). Connecting speech encoder and large language model for asr.

[3] Tang et al., (2024). Salmonn: Towards generic hearing abilities for large language models

[4] Nguyen et al., (2024). Spirit LM: Interleaved Spoken and Written Language Model

Dear Reviewer izXP,

Thank you for acknowledging our responses and increasing the rating from 6 to 8! If you have any additional questions or suggestions, please do not hesitate to let us know.

Best regards, Authors

Comparison with Connector-based Model (SALMONN)

Besides concern over paralinguistic information, another common suggestion (reviewers izXP and Bk9J) was to compare our model with connector-based models like SALMONN [2]. We initially did not pursue this comparison because previous connector-based models are typically limited to specific speech-to-text tasks, such as speech recognition or translation [4,5], or they require instruction-tuning [2,3]. In contrast, our approach focuses on developing a foundational SpeechLM with high speech and text understanding, aligning more closely with prior works like SpiritLM, TWIST, and VoxtLM.

However, to alleviate the reviewer's concern, we conducted experiments with SALMONN with our benchmarks: Storycloze (S->T) and Speech-MMLU, as SALMONN supports only the speech-to-text format. For Storycloze, we used the prompt: <Speech><SpeechHere></Speech> Based on the audio, continue the story. <Text Continuation>, and calculated the likelihood of good versus bad continuations for prediction.

For Speech-MMLU, the prompt we used was: <Speech><SpeechHere></Speech> Please listen to the audio and answer the question. Choose from the following options: {Choice String}. Where the choice string covers four options from MMLU and we compute the model's likelihood for each option for prediction.

The evaluation result is shown in the table below:

When comparing SALMONN with our method, we observe that SALMONN's performance is sub-optimal, particularly on the more challenging Speech-MMLU benchmark, where its performance is nearly random. It's important to note that SALMONN is fine-tuned on Vicuna-13B-v1.1 [6], which achieves an MMLU score of around 50. These results highlight the challenges of modality fusion and suggest that previous instruction-tuned SpeechLMs have experienced significant catastrophic forgetting.

We hope these additional experiments and analyses could address reviewers’ concern and provide further justification for the significance of our work.

References:

[1] Nguyen et al., (2023). EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis

[2] Tang et al., (2024). Salmonn: Towards generic hearing abilities for large language models

[3] Chu et al. (2024). Qwen2-audio technical report

[4] Wu et al., (2023). On decoder-only architecture for speech-to-text and large language model integration

[5] Yu et al., (2023). Connecting speech encoder and large language model for asr

[6] The Vicuna Team, (2023). Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality