SALMONN-omni: A Speech Understanding and Generation LLM in a Codec-free Full-duplex Framework

1, Lack of innovation. The proposed end-to-end system is a combination of different existing methods. 2, One of the core parts of this work is the LLM. However, there is no description of the LLM module of this work at all. From the current version of the paper, the readers cannot tell if the LLM is pre-trained by the authors or if it is a public checkpoint from open-source work. Describing the LLM module is essential. 3, Cross-attention in both LLM and TTS modules seems to be one of the key components of this work. However, this part is not clearly described. Using equation is suggested for precise description. 4, Insufficient experiments. i) The experiment part only reports the result of the proposed method without comparing it to other baseline method. Such baseline methods could be either traditional speech enhancement/dereverberation methods, or LLM-based methods. ii) There is no ablation study in terms of the performance of each submodule. One example is to compare the performance of Mamba to other autoregressive architecture for the speech encoding. Or dropping the “thinking” token to see how the performance is degraded. 5, The proposed work focuses on a streaming framework. However, there is no system latency result reported which is necessary, e.g., how long of the look ahead of speech signal is required to process. It would be even better if an ablation study of the performance against different latency settings. 6, Concepts are confusing. The paper mentioned that a codec-free method is proposed. However, in the TTS module of VoiceCraft, codec codes are generated. It seems the codec-free only occurs in the encoding and speech understanding stage, but not in the TTS stage. To have a consistent presentation of each part of the paper, you need to either drop the “codec-free” or make it more specific to codec-free in understanding part. 7, Some of the evaluation metrics are not clearly described. One example is how you measure the turn taking accuracy. Can you describe it using an equation? The other example is how you measure the success rate of the spoken QA task. Can you also describe it either using an equation or cite a reference paper? 8, Writing needs to be improved. i) There are many typos in the paper. One example, line 355-356, “both LoRA and VoiceCraft are fine-tuned” should be “both LLM and VoiceCraft are fine-tuned”. ii) More detailed explanation is needed for these Tables (Table 2 and 4) in terms of the evaluation metrics. For example, if the results are corresponding to speech recognition, it’s necessary to include WER in the table or list as a notation.

The paper highlights that the proposed model represents speech features in a continuous space without relying on codecs or discrete tokens; however, it remains unclear whether this design choice is optimal, as no comparative analysis has been provided. An evaluation using cascaded systems (eg: Qwen2 with VoiceCraft) could provide valuable insights into performance. There are few missing details on task formulation, model specifications (text LLM) and training details that can help improve paper clarity.

The primary weakness is a lack of clarity in the presentation. The model, architecture and loss functions are described conceptually more than mathematically, which in itself is not a bad thing, it's a bit hard to follow the wealth of methods and details described. Some abbreviations are not explained until the Conclusion; several aspects of the experimental evaluation are hard to follow.

Regarding the novelty of the work, I do think that, in the context of Speech LLMs, the use of cross-attention instead of self-attention is novel compared to the published studies I am aware of -- though the question of cross-attention vs self-attention in the context of speech LLMs is commonly discussed in the community. The submission presents arguments for the advantages of cross-attention, and demonstrates the practical viability of their architectural choices, but it does not compare head-to-head with the self-attentional version of their system to demonstrate experimentally some of those advantages. One could make the same comments about the use of continuous embeddings instead of discrete tokens. This, too, is novel (and interesting) compared to many published studies -- but the work AFIACT doesn't offer head-to-head comparisons of use of continuous embeddings vs discrete tokens.

The "understand then generate" strategy described in the submission is interesting, and is a key point of the study. However, the use of "thinking" tokens seems very similar to the use of empty/blank tokens in Seide et al. (2024), "Speech ReaLLM -- Real-time Streaming Speech Recognition with Multimodal LLMs by Teaching the Flow of Time", https://arxiv.org/abs/2406.09569, which I think should be cited.

Given that this is not the only full duplex Speech LLM system, the reader is left wondering which of the many architectural decisions are key differentiators, beyond the top-level ones (continuous embeddings vs discrete tokens, and cross-attention vs self-attention). Perhaps the work could be improved by offering a clearer overall contrast listing pros and cons w.r.t. e.g. the well-known Moshi model?

• The paper has substantial issues in its presentation. Many explanations are either insufficiently detailed or lack internal consistency, and the descriptions of the figures are also lacking in detail. As a result, I struggled to understand much of the technical content of the proposed approach.
• The lack of reproducibility is a serious concern. Part of this stems from the presentation issues mentioned above, as the paper does not provide enough detailed explanations of the techniques to allow for the full reproduction of their methods. Furthermore, Section 5.2 does not include adequate information on how the training data is generated, particularly regarding SE, dereverberation, and TSE.
• The experimental results do not show a comparison with the other models, and it is difficult to justify their numbers.
  • At least Librispeech results in Table 2 are a way behind compared with streaming-based approaches (6.0 vs. 2.8 [1] in test_clean and 10.5 vs. 7.3 [1] in test_other).).

[1] Moritz, Niko, Takaaki Hori, and Jonathan Le. "Streaming automatic speech recognition with the transformer model." ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE,