Roadmap towards Superhuman Speech Understanding using Large Language Models

1. The paper suffers overall from overclaims that use the terms ‘superhuman’, ‘AGI’, and ‘surpassing human performance’ too loosely without properly defining them or providing experimental results to support. At its core, this paper is a useful speech LLM benchmark that analyzes speech LLM performance on increasingly difficult tasks (from level 1 to level 5 difficulty). To avoid misconceptions, I recommend the authors reduce the hype-like formulation using loaded terms like AGI and focus on making scientific claims. Takeaway 4 (lines 340-345) is an overclaim as well.
2. The task categorization is not very novel e.g. an existing benchmark like Dynamic SUPERB already categorizes by levels 1-3, which are called content, paralinguistics, and semantics respectively in Dynamic-SUPERB; this is acknowledged in Appendix A of this paper as well. The level 4 and 5 categories (speech specialist and ‘AGI’) are new, but this benchmark does not add enough tasks in those categories (only 4 COVID-based tasks in L4 and 2 tasks in L5), so these new levels are not covered sufficiently. The authors say they will be adding more tasks, but they are not included in this submission.
3. A class of baselines that I think are missing are text-based baselines that try to infer emotion/stress from the transcript of the utterance, and/or with additional automatically-extracted metadata from the speech. These baselines would help understand how much of the benchmark is solvable with text alone and how much requires speech data.
4. The writing for the benchmark description repeats the definition of each level in different places, losing out on space. I recommend making Section 2 much tighter so that some important experimental details (currently in appendices) can be included in the main text.

• The paper does not address speech generation tasks (e.g., Text-to-Speech (TTS) or dialogue system responses) and lacks completeness as a roadmap or benchmark for the field.
• Certain terminology choices, such as "paralinguistics," may not align with conventional use within the speech community, leading to potential misinterpretation. For instance, "paralinguistics" typically encompasses emotional aspects of speech, as illustrated by the Computational Paralinguistics Challenge organized by ISCA Interspeech, which includes emotion recognition tasks. This divergence in terminology creates ambiguity and suggests the roadmap may lack refinement in this respect.
  • Due to these terminology issues, distinctions between levels—specifically L2 and L3 tasks—are unclear. Some tasks categorized at L2, such as pitch perception, seem more complex than certain L3 tasks, such as gender recognition.
• I am skeptical of the overall categorization scheme, as some tasks listed (e.g., acoustic scene classification and L4 tasks) are not strictly "speech" tasks but are more accurately described as audio processing tasks. Additionally, L4 tasks are likely to lack significant speech semantics, making them atypical for speech understanding. However, if L4 included tasks involving doctor-patient conversations with consultative dialogues, it could be more suitable for speech understanding.
• The paper overlooks relevant existing speech and audio benchmarks, such as the SUPERB and HEAR benchmarks, which could strengthen the study’s scope and positioning.
• The benchmark section itself feels incomplete. As noted in the footnote, tasks for Levels 4 and 5 have not been fully identified or outlined.
• Finally, the analysis sections appear somewhat disconnected from the roadmap (especially Sections 4.3 and 4.4), with limited cohesion between the analysis content and the proposed roadmap.

1. The current version of the SAGI benchmark is limited in its coverage, particularly for Levels 4 and 5, which involve advanced and generalized speech comprehension tasks.

2. The tasks proposed in the SAGI benchmark do not align well with the definitions for each level as outlined by authors. Specifically, the current level 5 tasks mainly focus on pronunciation scoring and native language identification, whereas the criteria for level 5 (lines 265-266) emphasize encouraging creativity and diverse thinking. I believe these tasks do not fully satisfy the intent.

3. I am concerned about the effectiveness of both the human baseline and GPT-4o evaluation. For the human baseline, I find it problematic that all the speech language models surpass humans on the “auto-speech recognition” task at level 1, which seems highly unrealistic and raises questions about the validity of this baseline. Additionally, the GPT-4o evaluation was conducted manually without using an API, making it especially sensitive to environmental sounds and the recording quality of the devices used, which casts doubt on the reproducibility of this baseline.

The primary concern with the paper is its over-claims and the lack of a well-defined conceptual foundation in its design. While the authors’ efforts to propose an abstraction for speech language modeling in the LLM era are commendable, the current formulation—particularly the high-level abstractions—lacks convincing theoretical grounding. Specific issues include:

• In Section 2.2, the paper outlines the philosophy of the proposed roadmap; however, the definitions of each level lack theoretical depth, serving more as categorizations than rigorously justified stages. Even if the defined levels are accepted, the sequence of the first three levels is debatable. For instance, Level 2, which addresses basic paralinguistic perception, could logically precede speech recognition, as suggested by previous studies. If the framework is to be positioned as a roadmap, further justification is essential to clarify the rationale behind the level definitions and their order with enough theoretical background support.

• The benchmark relies on a limited selection of open-source corpora, but the chosen tasks lack alignment with the intended focus. For example, at Level 4, the authors utilize the COVID-cough dataset, which is not typically associated with speech tasks (usually classified as general audio tasks given its limited language-related content). While the authors aim to exclude general audio/music-related tasks, the current framework inadvertently includes such elements, particularly when analyzing music or environmental sounds. More robust justification is needed to delineate the scope of the proposed method. At Level 5, computer-assisted language learning traditionally viewed as analogous to ASR, is positioned at an AGI level, which may be overly ambitious. As the paper discusses, speech AGI aims to surpass human performance in speech-based tasks, but the selected tasks fall short of representing this objective effectively.