Distilling an End-to-End Voice Assistant Without Instruction Training Data

This paper proposes the Distilled Voice Assistant (DiVA), a Speech Large Language Model (LLM) trained on ASR data alone to enhance instruction-following abilities through self-supervised distillation. The authors align DiVA’s responses with those of a text-only LLM to enable cross-modal transfer of instruction adherence. However, the approach does not present substantive methodological innovation, as it closely mirrors existing distillation and instruction-following techniques. Furthermore, the work critically lacks comparisons to similar method in this domain, failing to substantiate its claimed improvements over key baseline methods.

This paper proposes a method named DiVA for training a speechLLM without using any instruction data in the speech modality. To perform this, an LLM is adapted to enable speech input and output by first combining it with a Whisper based speech encoder, and then using two Novel loss term to train the new modules. The two losses are a cross-model alignment loss which aims to alight the speech and text modalities, and a (simplified) KL divergence loss on the distribution of text and speech outputs. The authors evaluate the paper on 3 groups of tasks, which are spoken question answering, speech classification and speech to speech translation. In all 3 task groups the method reaches state of the art results and surpassed the baseline methods.

Overall, this paper shows a significant contribution and I recommend to accept it.

This paper proposes a novel approach for training Speech LLMs in the absence of explicit instruction-following data. The authors introduce a method to transfer the instruction-following and conversational capabilities of text-based LLMs to speech-based models by leveraging only ASR data. Specifically, the proposed approach aligns input tokens using a cross-modal token alignment loss and output representations via embedding distillation loss, effectively bridging the modality gap between speech and text. Experimental results demonstrate that this method generalizes well to downstream tasks, including spoken QA and speech translation.

This paper proposes DiVA, a voice assistant model that is able to follow both spoken and written instructions. It is trained via a dual distillation and alignment loss and shows relatively strong results on various benchmarks including head to head comparison with Qwen 2 Audio. The authors propose a "q-former" style injection of audio into the model, which is initialized from a Whiper model, and a text model initialized with Llama weights and left frozen which consumes the audio input. This model is then trained for relatively small amount of steps on CommonVoice data to learn input alignment as well as output alignment (as compared to text labels).

We thank the reviewers for their comments! We are working hard to provide substantive responses backed with experimental results to address reviewer concerns wherever possible. We will continue to provide concrete responses to these throughout the discussion period! The first of these empirical results is in regards to the reviewer request for further baseline comparisons to other recent Speech LLMS. Where possible we have run additional evaluations to directly address this feedback!

First, we focus on specific points surrounding particular models that the reviewers have requested comparisons to. These models are BLSP, BLSP-KD, and AudioChatLlama:

1. We have run our evaluations on BLSP, which performs hard-distillation on the outputs using cross-entropy loss, showing that DiVA improves performance across all metrics (see Table below) compared with this model. | Task | DiVA | BLSP | | ----------- | ------ | ------ | | SDQA-Mean | 53.58% | 44.03% | | HeySquad | 55.17% | 46.78% | | IEMOCAPS | 50.60 | 42.88 | | MELD | 41.34 | 40.12 | | UrFunny | 50.24% | 49.92% | | MUSTARD | 52.61% | 48.41% | | COVOST-Mean | 13.80 | 5.05 |

2. Neither BLSP-KD (published on Arxiv on May 29, 2024) and AudioChatLlama (published in NAACL on June 15, 2024) release their model weights or the code used to produce these models. We have emailed the authors of these works to see if we can get access to perform these comparisons. The BLSP-KD author unfortunately said that they cannot release their model as it was produced during a company internship. We are still waiting on a reply front the AudioChat Llama author to reply and will respond further when they do.
   Without code or model weights it is difficult to compare directly to these recently released works, highlighting the importance of our entirely open-source pipeline for this work. However, we have done our best to provide empirical comparisons to these works based on available information in the next two points.

3. Our "Distillation Only" ablation closely mirrors AudioChatLlama's described methodology, which performs distillation on outputs only, revealing important limitations in generalization. Specifically, while this approach proves effective for Spoken Question-Answering (as both our works show), our broader evaluation reveals that it fails to generalize to tasks requiring text instruction following. This highlights the importance of our usage of both input and output distribution alignment.

4. Without access to the particular model, we instead compare the results from our model on the same COVOST languages they evaluate on in BLSP-KD to the results they report in their paper. We show this comparison below, where DiVA outperforms BLSP-KD on 5/7 languages used in the BLSP-KD paper.

As noted in our paper, Llama (and therefore DiVA) produces romanized, rather than native script, responses in Chinese and Japanese which leads to the reduced performance in those languages compared to BLSP-KD.

There are always many models which can be included as baselines, especially in quickly growing research areas. We feel our selection of SALMONN, Qwen Audio, and Qwen 2 Audio as primary baselines instead is supported by:

1. Their state-of-the-art performance on chat-style benchmarks and paralinguistic tasks as shown in the recent Qwen 2 technical report in which these are the top 3 performing models [1].
2. Open availability of model weights and inference code which makes evaluation of them possible without re-implementing and retraining the models.
3. Their widespread usage overall as the most commonly used reference point and/or foundation models for other Speech LLM works. This is also likely due to their open-access release which makes their usage possible in downstream research, similar to DiVA.

[1] Chu, Y., Xu, J., Yang, Q., Wei, H., Wei, X., Guo, Z., ... & Zhou, J. (2024). Qwen2-audio technical report. arXiv preprint arXiv:2407.10759.

In the second of our empirical results in response to reviewer comments, we'd like to address the concerns on our use of the Q-Former and clarify how our work improves over other usages of Q-formers.

We agree with Reviewer Rub3 that the Q-Former is indeed well-established in this domain, as acknowledged in our paper (lines 168-170). We also acknowledge Reviewer uKgu's point about potential limitations of the Q-Former! We noted ourselves that attention-based pooling typically increases data requirements compared to simpler heuristic pooling methods (lines 173-174).

Novel Contribution

Our contribution on the particular topic of the architecture of the adapter is simple, but empirically effective: initializing the Q-Former using the Whisper Decoder with static query tokens (discussed on lines 176-181). While this approach limits our architecture choices somewhat, we hypothesized that leveraging the pretraining of the Whisper Decoder would be more practically beneficial. To provide stronger evidence for this hypothesis, we conducted additional empirical experiments.

Experimental Setup

We performed three comparative training runs:

1. Our proposed approach
2. Control 1: Q-Former with random initialization
3. Control 2: MLP with random initialization

All other training parameters remained constant across runs including training data order, other component initializations, and hyperparameters.

Results and Analysis

Performance Across Tasks

We see that much lower training and evaluation loss are achieved by the approach we take in our paper v.s. either of the controls. We have attached the training and evaluation loss curves in anonymous links here.

Beyond lower loss, our empirical results demonstrate significant improvements across most tasks. Most notably, the Q-Former initialization is critical to achieving non-zero translation performance:

Qualitatively, the MLP connector seems to learn only to produce the first token correctly and then immediately generates EOS afterwards. This leads to reasonable classification performance, but poor performance in generative tasks. On the other hand, the randomly initialized Q-Former is able to generate coherent QA responses, but fails to adapt to varied text-based instructions leading to zero BLEU scores in translation.

A Note on Length Generalization

This limitation applies equally to MLP with average pooling since it produces tokens at a fixed-ratio to the input tokens. The Whisper Encoder (also used in Qwen 1 & 2, SALMONN, BLSP, and BLSP-KD) produces mixed 1500 tokens for all input lengths and has a maximum input length of 30 seconds. As noted in our response to Reviewer uKgu on the 448 token limit, in practice, this limitation can be mitigated both for DiVA and other models that use the Whisper Encoder by encoding multiple chunks of 30 seconds for length generalization.

Conclusion

We hope that these additional experimental results address the reviewer concerns by showing that:

1. The Whisper-initialized Q-Former generally improves over the suggested simpler alternative from Reviewer uKgu
2. Our contribution on this particular front is not the architecture itself, but instead a simple but previously unused initialization strategy to reduce the data demands of attention-based pooling. This adjustment is likely to benefit not just DiVA, but other approaches which use the Q-Former for SFT based training.

We would like to thank the reviewers for their time! We are glad to hear from the reviewers that there are limited concerns about technical soundness after the discussion period, despite remaining questions about novelty. The main open soundness concerns are from Reviewer WQdk, who unfortunately we did not have a chance to discuss further with after their initial review. We hope that similar to Reviewer uKgu "the additional experiments help address most of [their] concerns", given that they had shared concerns about the baselines compared to.

While we understand the reviewers concern that the novelty of our work is not a "big leap", we want to highlight the following practical merits of our work:

• The reviewers agree that the model is clearly empirically effective and outperforms similarly situated models. We feel the practical utility of a strong open-weights model which can be used for interpretability studies, further refinement, and self hosted application can be significant.
• Releasing the full code to reproduce such a model on the openly-accessible infrastructure we used to train DiVA meaningfully contributes to open and reproducible science and enables future works leveraging the underlying code for scalable training.
• The ablations studying the effects of our varied contributions help practitioners understand each separately, allowing them to utilize different useful components even if they do not use all of the methods we study.

As the discussion period is coming to an end, we want to close with a final expansion on these points.

DiVA has Strong Empirical Results For Downstream Usage:

• While individual components of our work may be seen to lack novelty, DiVA demonstrates a practical advance in speech-LLM integration. Following our additional baseline experiments, all of the reviewers who responded agreed that our model's performance is clearly stronger than other end-to-end speech models.
• Unlike some recent works the reviewers mentioned, our work releases model weights which can be finetuned further for specific tasks, styles, or languages. We contacted the authors of these works and either did not hear back or were told the weights will not be released due to corporate policy. Given these works seem unlikely to be released for further usage or study, releasing an open model with state-of-the-art capabilities is especially likely to enable new work from the broader community.
• Our open-source implementation supports reproducibility, which RRHC highlighted as a strength: "Authors use publicly available datasets and fine-tuned models". By releasing training code which uses computing infrastructure which is freely available to academics, our work could enable more practitioners to enter and advance this research topic.

Multiple Methods, One Effective Model

• While Reviewer Rub3 noted that "using q-former with pre-trained model weights is not really in and of itself a novel technique," they acknowledged it is "definitely a noteworthy and useful method as part of a greater work". Notably, this initialization method is one part of what leads to DiVA's success and complements our separate distillation components.
• Reviewer uKgu notes that "ablation analysis confirms the distinct contributions of cross-modal token alignment and embedding distillation". This provides more nuance to understand why DiVA is so empirically effective with regards to other distillation works.
• While Reviewer Rub3 notes that other works have explored $L_2$ or feature based losses, we feel these works are meaningfully different from ours. The most notable difference is that, specific to our case of adding a new modality to an otherwise frozen LLM, we show that feature distillation leads to an equivalent solution to KL Divergence distillation at reduced cost due to the shared components.

DiVA offers Comprehensive Evaluation and Analysis

• As demonstrated in our response tables, we conducted thorough comparisons across architectures, losses, and to all baselines which were requested and available. Across these, we have demonstrated that DiVA is effective and often exceeds the performance of models trained on far more data. We are glad to see that there are limited soundness concerns about which tasks we selected given DiVA achieves "strong results on [this] wide variety of tasks" [RRHC].
• As uKgu noted, this includes "a qualitative user study [which] shows DiVA's strong alignment with human preferences in conversational quality". This pairwise preference study for Speech LLMs is a first of its kind and we release the open-source code to enable future such studies by others. We hope that the contribution of this first pairwise preference study of Speech LLMs clearly highlights the benefits of this approach, since as we note on line 485, our results provide early indications that many current "benchmarks may not correlate with practical usage."

The paper introduces DiVA (Distilled Voice Assistant), a model for training speech-based large language models (Speech LLMs) without requiring explicit instruction-following data in the speech modality. The key components of the approach are: 1) integrating a Whisper-based speech encoder with a text model initialized from Llama weights, allowing the model to process both speech and text inputs. The text model remains frozen while the combined model is trained on a relatively small amount of data, including CommonVoice. 2) training the model using dual distillation and alignment loss techniques, with cross-modal alignment loss to align speech and text modalities and embedding distillation loss to align output representations of speech and text. With these components, DIVA demonstrates competitive performance across several task groups, including spoken question answering, speech classification, and speech-to-speech translation, and surpassing baseline methods, such as Qwen2-Audio.

Strength of this paper

• Competitive performance: DiVA surpasses previous SOTA models, like Qwen2-Audio, with significantly lower resource requirements, making it a more resource-efficient solution. Evaluations show competitive results across various tasks and benchmarks, with DiVA performing well in instruction-following and cross-modal alignment. These experiment results support the model’s efficacy.
• A cost-effective methodology of using ASR data alone to improve instruction-following behavior and reducing reliance on annotated instruction data. The paper also demonstrates that it is sufficient to teach an existing instruction-tuned model to understand audio using relatively lightweight techniques, effectively enabling the model to follow instructions via audio queries.
• The paper is clearly written, and it uses publicly available datasets and fine-tuned models, which increases the paper's accessibility and reproducibility.

Weakness of this paper

• Limited Novelty: the method lacks novelty, as many components (e.g., attention-based pooling like Q-former, L2 loss as a replacement for KLD) are well-known in existing literature. Combining these pieces to process audio input with Llama models, though interesting, has been explored in previous works like Nvidia's SpeechLLM and SALMONN, which also use similar strategies. The proposed distillation approach, which relies on self-supervised and cross-modal transfer methods, does not go beyond well-established techniques and lacks meaningful innovation, limiting its impact.
• Unclear Model Outputs and Evaluation: The paper does not clarify how speech (waveform) is generated from the model output, leaving a gap in understanding how the system produces audible speech. There is no direct evaluation of speech quality, such as MOS (Mean Opinion Score) experiments, which would be valuable to assess the quality of the generated speech. The speaker similarity and translation quality (ASR-BLEU) metrics are missing for the speech-to-speech translation task. These would help evaluate the overall quality of the translations and speaker consistency. Besides, The evaluation of the voice assistant claims is unclear. The paper mentions the creation of a voice assistant but does not explain or evaluate what this entails, raising questions about the paper's practical applications and performance in real-world tasks.
• Lack of Comprehensive Comparisons: The paper does not compare the method with important works, such as those using cross-entropy or other distillation methods (e.g., BLSP, AudioChatLlama), which would clarify the advantages and uniqueness of the proposed approach. A more detailed comparison with existing methods like Qwen2-Audio and the Llama-3 backbone is needed, as claims about DiVA outperforming state-of-the-art models may be misleading due to discrepancies in model architectures and backbones.
• Lastly, related work and references is not incomplete, e.g., omitting numerous relevant works in the Speech LLM and instruction-following domains. This makes it harder to understand the position of this work, and what's the trade-offs and how proposed work compared with existing approaches, to help readers better understand the method's limitations and future applicability.

Reject