An Exploration of Speech Conditioned Large Language Models (SLMs)

Lack of Novelty

The author proposes to mixing the synthetic spoken instruction following dataset in the training to improve the spoken instruction following performance. The resulting model, SiM, outperforms other counterparts in OpenHermes and Alpaca benchmark. While this indicates advancement in speech interaction of the chatbot application, the improvement seems to solely come from adding in-domain data instead of any novel methodology, which brings questions to the novelty of the proposed methods. To prove that other SLMs are indeed performing worse in that task, various forms text prompts should be examined and the author may report the best scores among them.

Limitation

The paper aims to explore the design space of SLM. However, the conclusion from this work is not generic enough. For example:

• The architecture is limited within the form of encoder-adaptor-LM, where the encoder embeds the speech input in the continuous space. The findings in this paper couldn't be applied to the SLMs based on discrete speech units.
• The target of "Alignment Training" is limited to ASR task. Whether the design choice selected by lowest WERs also leads to better performance is not examined in this paper.

Lack of details

I would suggest disclosing more training details for reproducibility including common hyper parameters, training infra, etc. Also, for the synthetic dataset, since it's not open-sourced, examples should be given.

1. While the paper aims to provide an effective framework for developing SLMs by thoroughly exploring the design space, its focus on only ASR and instruction-following tasks significantly narrows the research scope, limiting the general applicability of its findings.
2. The ASR component lacks novelty, as it has been extensively studied in the field. Prior work, such as "An Embarrassingly Simple Approach for LLM with Strong ASR Capacity" (Ma et al., 2024), has already benchmarked various combinations of large language models and speech encoders, establishing optimized LLM-based ASR systems. This limits the originality of the ASR contributions in this paper.
3. In investigating the Adaptor Architecture, the study does not control the audio token reduction rate (ATR), which varies across experiments. With an ATR greater than 1, downsampling audio tokens likely leads to information loss and potential performance degradation. This lack of control limits the validity of conclusions drawn from these experiments.
4. The experimental comparisons are somewhat limited, as they only involve Llama3 8B Instruct and Llama3 8B with a simple masking strategy and solely rely on the Whisper encoder. This narrow scope restricts the conclusions, as a broader range of comparisons with different models and encoders would yield more generalizable insights. Furthermore, the title of this paper seems inconsistent in the pdf and the system: "An Exploration of Speech Conditioned Large Language Models (SLMs)" vs "An Exploration of the Design Space of Speech-Conditioned Large Language Models (SLM)"

1. The paper claims to thoroughly explore SLM design space, but in fact only explore model architecture choices under the category of SALMONN or Qwen Audio like Speech-LLM, whose scope is very limited and may not be able to extend to other high level architecture designs. Besides the architecture used by the paper, there are more different Speech-LLM models like AudioPaLM, Moshi, AudioFlamingo, BESTOW, etc. The paper fails to analyze and compare the architecture differences and motivate the reason to use the proposed architecture.
2. Almost all of the components in the paper are not new, which limits its unique contributions: i) the architecture itself follows SALMONN or Qwen Audio ii) the study in adapter architecture and trainable module is very similar to previous works like https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10445874 iii) the way to build the synthetic spoken instruction following dataset is not new, e.g. https://arxiv.org/pdf/2310.13289 and https://arxiv.org/abs/2309.09843 and https://arxiv.org/abs/2406.12946
3. The comparison of different design choices is completely based on ASR. Whether these conclusions can hold on to other tasks is a big unsolved problem, which limits the value of these conclusions, considering one of the central value of Speech-LLM is its speech instruction following and speech understanding ability . Moreover, the authors should also consider translation tasks.
4. The way to evaluate speech instruction following ability is limited. The authors should consider public benchmarks like AIR-bench, Dynamic Superb or SD-Eval.
5. Speech instruction SLM and text instruction SLM are two different downstream applications or problems. Most of the baselines the paper tries to compare with is designed for text instruction SLM, which is unfair to compare on speech instruction testsets, especially considering the proposed work is mainly designed for speech instruction.

1. In Section 3.1 (Alignment Training), the experiments on model architecture and training strategy choices are evaluated solely on the LibriSpeech test-clean. Since LibriSpeech test-clean contains only clean audiobook data, the experiments seem more like an exploration of the best-fitting strategy for this specific domain. This issue is particularly evident in Section 3.1.5 (Choice of Training Data), where the authors attempt to explore data-mixing strategies between CommonVoice and LibriSpeech but rely only on evaluation results from LibriSpeech test-clean. Given the distinct domains of Common Voice and LibriSpeech, this approach may lead to significant biases in the final conclusions. I recommend adding more diverse test data, such as LibriSpeech test-other and CommonVoice, to enhance the robustness and reliability of the experimental conclusions in this section.
2. The final SiM model architecture largely aligns with mainstream MLLM architectures like LLaVA[1] and SLAM-ASR[2]. In Section 3.2.1, the authors find that using Synthetic Spoken Instruction Data (SiM) improves spoken instruction following compared to models such as Qwen2-Audio and SALMONN, which were not trained with spoken instruction data. However, this result seems self-evident. The use of synthetic spoken instruction data to enhance instruction-following ability has already been proposed in prior work, such as SpeechGPT[3], though SpeechGPT used Chain-of-Modality for this purpose, while SiM does not generate corresponding speech transcriptions. Overall, the paper’s main methods lack novelty and offer few technical contributions.

[1] Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2024). Visual instruction tuning. Advances in neural information processing systems, 36.

[2] Ma, Z., Yang, G., Yang, Y., Gao, Z., Wang, J., Du, Z., ... & Chen, X. (2024). An Embarrassingly Simple Approach for LLM with Strong ASR Capacity. arXiv preprint arXiv:2402.08846.

[3] Zhang, D., Li, S., Zhang, X., Zhan, J., Wang, P., Zhou, Y., & Qiu, X. (2023). Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprint arXiv:2305.11000.