Time-Accurate Speech Rich Transcription with Non-Fluencies

4. Could you clarify why your models consider codecs?

Our work involves two different 'codecs.'

(1) The first is related to Universal Articulatory-to-Acoustic Inversion (UAAI) codec model [1], which generates articulatory trajectories (X) from speech. Since our articulatory modeling requires speech's articulatory trajectory, which is typically unavailable in real speech data, we use offline UAAI to synthesize X. The UAAI work we referenced demonstrates that speech can be converted to articulatory trajectories which, combined with other speaker features, can resynthesize highly intelligible speech. This suggests that articulatory trajectories serve a similar role to traditional neural speech codecs like SoundStorm, and thus, it was called articulatory codec models.

(2) The second codec is SoundStorm [2], which we used in our experiments (Table 1). Our motivation was to test whether our proposed gestural scores are more scalable than neural speech codecs, and our current findings suggest they are.

[1] Cheol Jun Cho, Peter Wu, Tejas S Prabhune, Dhruv Agarwal, and Gopala K Anumanchipalli. Articulatory encodec: Vocal tract kinematics as a codec for speech. arXiv preprint arXiv:2406.12998,2024

[2] Zal´an Borsos et al. Soundstorm: Efficient parallel audio generation. arXiv preprint arXiv:2305.09636, 2023.

5. Regarding bias.

We believe one primary bias stems from the distribution gap between training and inference data, as the model typically tends to overfit to the training data distribution. A significant aspect of this bias is accent variation. Our best-performing model was trained on Liri-Co-Dys, which was simulated based on LibriTTS. This dataset primarily contains United States American accents and does not cover a wide range of other accents. This limitation raises two questions.

(1) Can our model, trained on US accents, generalize to accented data from datasets like VCTK, Common Voice [1], and GLOBE [2] for dysfluency detection?

To answer this, we evaluate GPT4-o speech API, SSDM, and our proposed SSDM 2.0 in zero-shot dysfluency detection across three accented corpora:

VCTK contains speech data from 109 native English speakers with various accents. We randomly selected approximately 20 speakers (~10 hours) for inference.

Common Voice comprises 3,347 hours of audio from 88,904 speakers, recorded at 48kHz. For our evaluation, we focused on the English portion, randomly selecting 100 speakers (~1 hour) representing 20 diverse accents.

GLOBE contains recordings from 23,519 speakers at 24kHz, totaling 535 hours. We randomly selected 10 hours of data covering 20 accents for inference.

Note that Common Voice exhibits higher noise levels compared to VCTK and GLOBE.

(2) Is accent dysfluency or not?

This question is crucial for evaluating accented speech. The definition of accent is broad and not unified, encompassing pronunciation variations, stress and intonation variations, and speaking rate (rhythm) variations. In SSDM 2.0, we consider phoneme replacement or phonetic error, which significantly overlaps with "pronunciation variations." Therefore, if SSDM 2.0 predicts a phonetic error that aligns with accent-related variations, this should be considered an accurate prediction rather than a true false positive. This means that using false positives (FP) as an evaluation metric can sometimes yield desired values. Since accented speech can be considered partially dysfluent when the detected dysfluency type is phonetic error, we cannot assume the ground truth FP is zero, and thus leave it blank.

In addition to FP, we introduce phonetic pronunciation error rate (PPER), calculated by dividing the number of utterances containing detected phonetic errors (counted as one regardless of multiple errors) by the total number of samples. Evaluating results using only FP and PPER presents challenges. Some predicted false positives or phonetic errors may precisely correspond to accents, making them not necessarily undesirable (i.e., lower values aren't always better). Given the lack of ground truth accent labels and the prohibitive cost of human evaluation, we employ a heuristic method: measuring the overlap between FP and PPER. The intuition is that closer FP and PPER values suggest predicted phonetic errors more likely match actual accents. We define Ratio as PPER divided by FP.

Results in Table-12 show that all models—GPT-4-o, SSDM, and SSDM 2.0—can predict both non-existent dysfluencies and dysfluencies corresponding to accents. Based on our heuristic evaluation methods, SSDM 2.0 appears to predict most accents accurately. However, determining the true false positive rate remains challenging and is left for future work.

Please refer to Appendix A.16 (highlighted in yellow) and Table 10 in our updated manuscript for detailed results.

[1] Ardila, Rosana, et al. "Common voice: A massively-multilingual speech corpus." arXiv preprint arXiv:1912.06670 (2019).

[2] Wang, Wenbin, et al. "GLOBE: A High-quality English Corpus with Global Accents for Zero-shot Speaker Adaptive Text-to-Speech." arXiv preprint arXiv:2406.14875 (2024).

• Another interesting direction, as suggested by Reviewer i1kS, is testing our models on fluent speech. This evaluation should yield lower false positives (FP) and better word error rate (WER). For ASR tasks, we use Whisper V2 as a baseline and GPT4-o real-time speech API for testing false positives. We first evaluate performance in a zero-shot setting on the LibriTTS corpus. We then fine-tune our model on LibriTTS with target dysfluency labels set to "None" (denoted as SSDM 2.0-Tuned), with results shown in Table 11 (updated). Results indicate that Whisper achieves the best ASR performance due to its scaling efforts. While GPT4-o speech real-time interface produces some false positives on fluent speech, SSDM shows higher FP and worse WER scores. SSDM 2.0 demonstrates better FP rates than GPT4-o but falls short of Whisper's ASR performance. After "fluent" fine-tuning, SSDM 2.0-Tuned achieves the best FP scores and ASR performance comparable to Whisper. For detailed results, please refer to Appendix A.11 and Table 9 (highlighted in red) in our updated manuscript. We welcome any additional feedback!

3. Articulation modeling.

This is also the concern from reviewer i1kS, so we unify the reply, and already added more introduction in Sec. 3.1 and Section 3 (before 3.1), highlighted via green in our updated manuscript. Feel free to check. And we detail them here too:

Current speech representation modeling typically uses explicit $D \times T$ matrices, where T represents time and D represents channel dimension, and these are learned densely in a data-driven manner. However, human speech is produced by a sparse set of articulators with sparse activation in time. If we define basic moving patterns of articulators as a dictionary and project articulatory data into this dictionary space, we obtain a sparse activation matrix. The dictionary is called gestures and the sparse activation matrix is called gestural scores [1]. This motivates us to ask: instead of densely learning elementwise speech representations, can we sparsely and implicitly learn such structural speech representations via human-prior rules?

In this work, we propose Neural Articulatory Flow, which involves a Semi-implicit Speech Encoder (Section 3.1) to predict only the indices of active regions in articulation, and an Articulatory Flow (Section 3.2) to distill speech intelligibility from pretrained acoustic speech embeddings (WavLM [2]). We also optionally introduce Interpretable Posterior Training to visualize how speech is physically produced in articulation and to derive articulatory feedback for pronunciation.

Given a speech waveform, we adopt UAAI (Universal Acoustic-to-Articulatory Inversion) [3] to obtain its articulatory trajectory $X\in \mathbb{R}^{D\times T}$ at 50Hz. UAAI consists of a WavLM [2] encoder followed by a linear layer, which takes speech waveform as input and predicts articulatory trajectories. The model was trained on the MNGU0 corpus [4], which contains 75 minutes of EMA (Electromagnetic midsagittal articulography) data collected during newspaper reading. In our setting, $D=12$ corresponds to the x and y coordinates of 6 articulators measured in the EMA recording process. We define articulatory trajectory kernels (gestures) $G^{T'\times D\times K}$, where $T'$ is the window size and $K$ is the number of kernels. By projecting $X$ onto the kernel space, we obtain gestural scores $H\in \mathbb{R}^{K\times T}$, which is high-level abstraction of $X$. $H$ denotes when the articulators are activated for how long. In this work, we directly predict $H$ from $X$ without gestures $G$ for simplicity, focusing only on active indices that indicate articulatory activation with a Count Encoder, a Index Encoder and a Value Encoder. We provide a visualization of gestural scores $H$ and its correlation with $X$ and Gestures $G$ in Appendix. A.5. “

Feel free to let us know if you want more content to appear in the main body.

[1] Catherine P Browman and Louis Goldstein. Articulatory phonology: An overview. Phonetica, 49(3-4):155–180, 1992.

[2] ​​Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518, 2022.

[3] Cheol Jun Cho, Peter Wu, Tejas S Prabhune, Dhruv Agarwal, and Gopala K Anumanchipalli. Articulatory encodec: Vocal tract kinematics as a codec for speech. arXiv preprint arXiv:2406.12998,2024

[4] Korin Richmond, Phil Hoole, and Simon King. Announcing the electromagnetic articulography (day1) subset of the mngu0 articulatory corpus. In Twelfth Annual Conference of the International Speech Communication Association, 2011.

2. Care should be made for establishing clear cases of dysfluency and conditions that cause it

This is a truly in-depth comment. Currently, we focus only on transcribing dysfluencies, which are common behavioral symptoms across all speech disorders. We leave diagnosis either for human effort as a next step or for our future work. However, we can discuss this further.

(1) In collaboration with clinical partners, we recognize that understanding the underlying causes of specific disorders remains an open problem. This investigation might require exploring brain data or conducting joint brain-speech analyses. Different disorders, or even different subjects within the same disorder, can have different causes. These causes can be both neurological (brain) or physiological (articulatory) conditions.

(2) Computational methods such as our dysfluency transcriber could help establish relevant biomarkers at the behavioral level, which is an easier step, rather than at the neurological or physiological levels, which are harder to evaluate. One approach would be to quantify different types of dysfluencies and analyze their patterns as potential biomarkers. For instance, nfvPPA might be associated with increased missing or insertion frequencies across specific age groups. We believe our framework has the potential to enhance such diagnostic capabilities.

Following your suggestion, we plan to include case studies in our future work. We would greatly appreciate your additional thoughts and suggestions on this direction.

We thank the reviewers for their in-depth reviews. We provide our rebuttal below, and we have updated our manuscript with the changes highlighted in yellow.

1. Organization issue. (Definition of dysfluency, nfvPPA, etc)

We appreciate the suggestion to make a more general case for dysfluent speech, and specifically nfvPPA in the main paper. To make it clearer and more understandable for all audiences, we have reorganized the introduction by adding discussions about motivations, dysfluency, and nfvPPA at the beginning. Here is our logic (which is also reflected in the updated paper with yellow highlights).

We first introduce dysfluency and its relevance, even in general purpose ASR or conversational analysis, and conclude that disordered speech represents an extreme but generalizable case that we primarily focus on in this work.

Current automatic speech recognition (ASR) [1] or speech language models (SLMs) [2] mostly transcribe speech to text while ignoring dysfluency (dysfluency, non-fluency, disfluency are interchangeable), which includes repetition, deletion, insertion, replacement of sounds, filler words, or hesitations [3]. Accurately transcribing dysfluencies has three major applications:

(i). In spontaneous or daily conversational speech, dysfluencies help understand speaker intent and conversational behavior;

(ii). For accented speakers and language learners, dysfluencies provide effective feedback for pronunciation improvement;

(iii). For people with speech disorders like Aphasia, dysfluency transcription could reduce speech language pathologists' workload and aid diagnosis.

Since the first two cases involve near-fluent speech, dysfluent speech transcription serves as a fundamental method in both typical and atypical speech recognition, extending beyond clinical applications and is a superset of the ASR problem. Looking at the interface, ASR models typically output only what the person intended to say, while its superset, dysfluency transcription, provides two outputs: what the person intended to say and what the person actually said. For example, Please call stella and P-Please c-call st-ah-lla. Actually, a significant portion of dysfluencies arise in disordered speech only. By accurately transcribing these, we can naturally extend this capability to less severe cases, such as conversational or accented speech. Thus, we prioritize disordered speech dysfluencies as a foundation, which can then be extended to a superset of ASR problem.

We then highlight the relevance of a common set of behavioral markers (dysfluencies) across various speech disorders, emphasizing the potential for developing a unified framework that addresses all speech disorders. Following this, we discuss our rationale for focusing on nonfluent primary progressive aphasia (nfvPPA) as our initial target. Finally, we provide an overview of nfvPPA, including its defining characteristics and clinical significance.

Speech disorders are accompanied by various symptoms and can be associated with conditions such as dyslexia, aphasia, apraxia, Parkinson's disease, stuttering, dysarthria, cerebral palsy, autism spectrum disorder (ASD), stroke, and other neurological conditions. Testing all these disorders is not practical due to data constraints. Different disorders also present varying levels of neurological conditions and dysfluency behaviors (severity). However, they all share the same set of dysfluencies as mentioned before. Thus, we aim to develop a general dysfluency transcription system that can be used by all disorders for follow-up diagnosis correspondingly.

In this work, we perform inference on real-world data including open-source stuttering data and nonfluent variant primary progressive aphasia (nfvPPA, [4]). nfvPPA uniquely presents with pronounced speech dysfluency, characterized by deficits in syntax and motor speech (apraxia of speech), making it an ideal candidate for testing the pipeline. In summary, while disorders may have different neurological or physiological causes, dysfluency transcription captures their shared behavioral symptoms (dysfluencies), and thus we only focus on how to accurate transcribe the dysfluencies in this work. “

Please feel free to review our updated manuscript highlighted in yellow and we would also greatly appreciate any suggestions from you on how to make these concepts (dysfluency, etc) more accessible to the broader research community.

[1] Radford, Alec, et al. "Robust speech recognition via large-scale weak supervision." ICML 2023.

[2] Haibin Wu, et al. Towards audio language modeling-an overview. arXiv preprint arXiv:2402.13236, 2024.

[3] Lian, Jiachen, et al. "Unconstrained dysfluency modeling for dysfluent speech transcription and detection." ASRU, 2023.

[4] Gorno-Tempini, Maria Luisa, et al. "Classification of primary progressive aphasia and its variants." Neurology 76.11 (2011): 1006-1014.

Thank you for your valuable new suggestions. Here is our reply

1. We appreciate the suggestion and believe 'lexicalized speech' (the words that were spoken) and 'uttered speech' (how the words were spoken) to be more accurate statements. Thus, we adopt them in the paper. Thanks for pointing them out!

2. We agree that disordered speech modeling is a stronger case than L2 language learning, so we have removed this along with conversational speech dysfluencies. We focus solely on pathological speech. Since nfvPPA is indeed associated with articulatory-based dysfluency, we make our statement as follows:

• Current automatic speech recognition (ASR) systems and speech language models (SLMs) typically transcribe speech into the words that were spoken (lexicalized speech) rather than how the words were spoken (uttered speech). For example, when someone says P-Please c-call st-ah-lla, these systems usually perform denoising and output please call stella. This approach is suitable for most spoken dialogue scenarios and services, and helps reduce confusion in communications. However, in pathological speech domain, uttered speech is required to accurately identify articulation and pronunciation problems for diagnostic purposes. The gap between lexicalized speech and uttered speech is referred to as dysfluency, which includes repetition, deletion, insertion, replacement of sounds, filler words, and hesitations. Accurate transcription of speech dysfluencies could substantially reduce the workload of speech language pathologists while facilitating diagnosis and serving as a powerful clinical assessment tool.

• Pathological speech disorders are typically caused by neurological or physiological factors and are associated with various dysfluencies, such as motor and phonological (articulatory) dysfluencies (e.g., nfvPPA, Parkinson’s disease, Broca’s aphasia) and higher-order (or semantic) dysfluencies (e.g., svPPA, Wernicke’s aphasia, ASD). Diagnosing these disorders is challenging due to case-by-case variability and differences in severity. However, they share a common set of dysfluencies at the behavioral level, making it possible to develop a general dysfluency transcription system that can accommodate all disorders and support follow-up diagnosis. Due to data constraints, however, it is not feasible to test such a system on all pathological speech data. Therefore, we focus on nfvPPA, leveraging the currently available data to develop a dysfluency transcription tool specifically targeting articulation-based dysfluencies.

We also emphasized articulatory dysfluencies in abstract.

In Section 7, we discussed that as future work, other articulatory-based disorders and semantic-based disorders will be explored.

3. In Section 3, we also mentioned that learning articulatory representations is a natural way to model articulatory dysfluencies.

4. Your suggestion regarding codec makes sense. The other insight is that codec is more acoustic-focused while dysfluency representations would be more semantic-focused. Using SSL units is sufficient, so we have removed the codec.

We really appreciate your high-quality review. Please check our updated manuscript again (highlighted in deeper yellow than before). Feel free to let us know if you have any other questions or suggestions

Dear reviewer

Thank you for your constructive feedback, particularly regarding the paper's presentation. And sorry for bothering you at this time. We apologize for any additional effort on your part. If we have addressed the identified concerns, we understand that according to recent new PC regulations, we are eligible to respectfully request a score raising. Beyond the writing problems only, if you have any additional concerns regarding the technical contributions (algorithms, models, etc.), we would be happy to clarify and discuss those as well.

Sincerely

4. Lack of Ablation Studies

Thank you for raising this point!

While we acknowledge the need for more ablation studies, we'd like to first clarify a potential misunderstanding. Table 1 focuses solely on evaluating the NAF (Neural Articulatory Flow) component, specifically assessing the intelligibility of our proposed gestural score representations. This evaluation is independent of other modules like FCSA and NICL, as NAF can be trained separately. In response to reviewer wcgM's request, we have added WavLM-Large to our ablation studies and updated Table 1 (highlighted in blue). The results show that WavLM-Large provides only marginal improvements over HuBERT and still performs significantly worse than our NAF in phoneme recognition.

We assume you might have concerns for Table 3 (But please correct us), which requires a more comprehensive exploration of ablations across different modules (NAF, FCSA, NICL) and loss objectives in Eq. 11. This aligns with reviewer wcgM's feedback, and we respond as follows:

SSDM 2.0 has introduced three major modules (NAF, FCSA, NICL) and multiple loss objectives for each module, making it necessary to explore the importance of each component. While Table 3 discussed the results when replacing individual SSDM modules, here we present additional ablation studies. For simplicity, we focus on Libri-Dys inference experiments. We first explain the notation. Starting with the baseline SSDM, single-module replacements are denoted as: SSDM+NAF, where we replace SSDM's gestural scores with our NAF gestural scores; SSDM+FCSA, where we replace SSDM's CSA with our FCSA; and SSDM+NICL, where we replace SSDM's vanilla language modeling with our NICL. We can also replace multiple modules simultaneously: SSDM+NAF+FCSA, SSDM+NAF+NICL, and SSDM+FCSA+NICL. Note that SSDM 2.0 is equivalent to SSDM+NAF+FCSA+NICL.

For loss objective ablations, we refer to the complete loss function in Eq. 11. The NAF module involves three losses: $\lambda_1\mathcal{L}\_{\text{KL}}$ + $\lambda_2\mathcal{L}\_{\text{FLOW}}$ + $\lambda_6\hat{\mathcal{L}}\_{\text{PIT}}$, where only $\hat{\mathcal{L}}\_{\text{PIT}}$ can be ablated as the first two are essential. FCSA includes two losses: $\lambda_3\mathcal{L}\_{\text{PRE}}$ + $\lambda_4\mathcal{L}\_{\text{POST}}$, each of which can be ablated. NICL has a single loss $\lambda_5\mathcal{L}\_{\text{CON}}$ that can be ablated. These ablations are denoted as $\text{SSDM+NAF-}\hat{\mathcal{L}}\_{\text{PIT}}$, $\text{SSDM+FCSA-}\mathcal{L}\_{\text{PRE}}$, $\text{SSDM+FCSA-}\mathcal{L}\_{\text{POST}}$, and $\text{SSDM+NICL-}\mathcal{L}\_{\text{CON}}$.

All ablation results are presented in Table 12 in the updated manuscript. In terms of both F1 score and Matching Score (MS), replacing any single module in SSDM leads to performance improvement. Replacing an additional module (two modules in total) further enhances performance. Regarding the loss function, the posterior interpretable training (PIT) loss appears to have minimal influence. An interesting observation with FCSA is that incorporating both losses, $\mathcal{L}\_{\text{PRE}}$ and $\mathcal{L}\_{\text{POST}}$, delivers strong performance. However, removing either one results in a performance drop, although this trend becomes less pronounced with more data available. Overall, each module and each loss in our proposed framework demonstrates its effectiveness. For scalability, when all components are integrated, scalability increases dramatically. However, using only one or two modules yields less effective scalability improvements.

Please feel free to check Appendix A.18 which was highlighted in Blue in the update manuscript.

• Lastly, in Section 6.7, we present results without the NICL module in Table 7 (highlighted by green). While these results show slightly lower performance compared to the full SSDM 2.0 model, they still significantly outperform SSDM, demonstrating the importance of the NICL module.

3. Lack of Real-World Data.

We acknowledge our limited use of real-world data. There are two key constraints that distinguish dysfluent speech from fluent speech:

(1) Dysfluent speech primarily occurs in stuttered/disordered speech. Acquiring such data involves ethical and privacy concerns that limit scalability. For instance, the nfvPPA data we currently use required several months of IRB approval, and even then remains limited in scale.

(2) Both training and inference require detailed dysfluency annotations. Currently available real-world datasets like UCLASS [1] and SEP28K [2] only provide utterance-level class labels, which are insufficient for our needs. Additionally, both UCLASS and SEP28K are relatively small-scale datasets.

Thus we did the following explorations:

(1) We relabeled UCLASS and SEP28K by adding time labels, where train/test split is consistent with YOLO-Stutter [3], and performed zero-shot inference on these stuttering datasets. SEP-28K is a large-scale dataset containing 28,177 clips from public podcasts. UCLASS contains recordings from 128 stuttering speakers.

We evaluated both dysfluency type and timing. As UCLASS and SEP-28K primarily contain repetition, prolongation, and block as dysfluency types, we focused our evaluation on these categories. For timing assessment, we used the Time F1 metric proposed in YOLO-Stutter. Results in Table 11 demonstrate that SSDM 2.0 achieves state-of-the-art performance across all settings. For detailed results, please refer to Appendix A.17 (highlighted in green) and Table 11 in our updated manuscript.

(2) Accent speech (same reply with Reviewer cH6N). Accent speech partially overlaps with dysfluent speech, particularly regarding phonetic pronunciation replacement. We evaluated GPT4-o speech API, SSDM, and our proposed SSDM 2.0 in zero-shot dysfluency detection across three accented corpora:

VCTK [4] contains speech data from 109 native English speakers with various accents. We randomly selected approximately 20 speakers (~10 hours) for inference.

Common Voice [5] comprises 3,347 hours of audio from 88,904 speakers, recorded at 48kHz. For our evaluation, we focused on the English portion, randomly selecting 100 speakers (~1 hour) representing 20 diverse accents.

GLOBE [6] contains recordings from 23,519 speakers at 24kHz, totaling 535 hours. We randomly selected 10 hours of data covering 20 accents for inference.

Note that Common Voice exhibits higher noise levels compared to VCTK and GLOBE.

In SSDM 2.0, we consider phoneme replacement or phonetic error, which significantly overlaps with "pronunciation variations." When SSDM 2.0 predicts a phonetic error that aligns with accent-related variations, this should be considered an accurate prediction rather than a true false positive. Consequently, using false positives (FP) as an evaluation metric can sometimes yield desired values. We introduce phonetic pronunciation error rate (PPER) in addition to FP. PPER is calculated by dividing the number of utterances containing detected phonetic errors (counted as one regardless of multiple errors) by the total number of samples. Given the lack of ground truth accent labels and the prohibitive cost of human evaluation, we employ a heuristic method: measuring the overlap between FP and PPER. The intuition is that closer FP and PPER values suggest predicted phonetic errors more likely match actual accents. We define Ratio as PPER divided by FP.

Results in Table-12 show that SSDM 2.0 appears to predict most accents accurately. However, determining the true false positive rate remains challenging and is left for future work.

Please refer to Appendix A.16 (highlighted in yellow) and Table 10 in our updated manuscript for detailed results.

(3) Dysfluent speech in real-world scenarios primarily occurs in stuttered or disordered speech. Regarding our clinical speech data, we are continuing data collection and will report on more subjects in the future. Additionally, we plan to collect "human simulated" stutter data, including from our own simulations, as part of our future work.

[1] P. Howell, et al, “The uclass archive of stuttered speech,” Journal of Speech Language and Hearing Research, vol. 52, p. 556, 2009.

[2] C. Lea, V. Mitra, et al, “Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter,” ICASSP, 2021

[3] Zhou, Xuanru, et al. "Yolo-stutter: End-to-end region-wise speech dysfluency detection." Interspeech, 2024

[4] Yamagishi, Junichi, et al. "CSTR VCTK Corpus: English multi-speaker corpus for CSTR voice cloning toolkit (version 0.92)."

[5] Ardila, Rosana, et al. "Common voice: A massively-multilingual speech corpus." arXiv preprint arXiv:1912.06670 (2019).

[6] Wang, Wenbin, et al. "GLOBE: A High-quality English Corpus with Global Accents for Zero-shot Speaker Adaptive Text-to-Speech." arXiv preprint arXiv:2406.14875 (2024).

We thank the reviewers for their critical reviews regarding the writing style, ablation studies, and other concerns. While we have made extensive revisions to improve clarity, we welcome further feedback!

1. The reviewers noted that the paper's writing style was characterized by overly formal language and verbose phrasing.

For the specific instances mentioned, we have rewritten the text to introduce gestures and our neural articulatory flow in a more accessible and easy-to-read manner. We have also added clarification for "self-contained" concepts such as "UAAI" and provided detailed information about its training data. These revisions can be found in Section 3 (the text immediately following the title but preceding Section 3.1) and Section 3.1, which are highlighted in green in our updated manuscript. To enhance reader comprehension of Figure 2, we have added a new Section 2 that introduces the SSDM 2.0 architecture (highlighted in red).

Additionally, as suggested by Reviewer i1kS, we have expanded our explanation of gestures and gestural scores in Section 3.1, marked in red for easy reference.

Furthermore, following Reviewer cH6N's recommendation, we have reorganized the introduction to include more comprehensive background information about dysfluency and disordered speech processing, helping readers better understand our motivation. These changes are highlighted in yellow in the Introduction section.

Please feel free to review our updated manuscript and let us know if you have additional suggestions for improving the writing style and clarity.

2. Regarding SSDM 2.0 Complexity.

SSDM 2.0 incorporates different components (NAF, FCSA, NICL) which may appear complex, or as one reviewer noted, "adds more layers of complexity to the model." We address these concerns as follows:

We evaluated five baselines for dysfluency transcription: SALMONN [1], LTU-AS [2], GPT-4 (speech), GPT-4o (speech), and SSDM [3]. As shown in Table 4, the first four speech language models (SLMs) perform significantly worse than SSDM and SSDM 2.0. Although models like SALMONN appear more concise, their weak performance makes them unsuitable for our task. Even after our fine-tuning SALMONN with our Libri-Co-Dys data (denoted as SALMONN-13B-FT in Table 4), improvements were minimal.

When a simpler model yields poor performance while a more complex model demonstrates substantial improvements, the latter would become the more practical choice. Regarding model parameters, we follow LTU-AS [2] by using only LLaMa-7B, which is significantly more efficient than SALMONN-13B. Therefore, SSDM serves as a more appropriate comparison point.

In SSDM, gestural scores are derived through complex processes including convolutional matrix factorization, sparse sampling, modeling, self-distillation, and phoneme classification. In contrast, our NAF employs only an encoder process and uses lightweight flow matching for self-distillation, making it substantially simpler than SSDM's gestural scores. Additionally, FCSA introduces no additional computational cost compared to SSDM's CSA.

In summary, while SSDM 2.0 would be more efficient than traditional SLMs like SALMONN, it is more appropriate to compare it with SSDM. Importantly, SSDM 2.0 achieves greater efficiency than SSDM, particularly in gestural score modeling.

Regarding complexity, we primarily focus on training convergence rate and representation sparsity. As summarized in Appendix A.14, our model achieves a 10X faster convergence rate compared to SSDM. Furthermore, as shown in Figure 3, our gestural scores (green dots) exhibit sparse representations (with sparsity around 0.9). This sparsity allows us to utilize sparse operations during backpropagation, which we believe contributes to the faster convergence rate. As per reviewer's request, we have moved these discussions from the main paper to the appendix for future exploration while removing them from the main paper. We welcome any additional feedback on these aspects!

[1] Tang, Changli, et al. "Salmonn: Towards generic hearing abilities for large language models." ICLR, 2024

[2] Gong, Yuan, et al. "Joint audio and speech understanding." 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2023.

[3] Lian, Jiachen, et al. "SSDM: Scalable Speech Dysfluency Modeling." arXiv preprint NeurIPs, 2024

Dear Reviewer:

We appreciate that the original weaknesses and questions have been addressed, and thank you for your new concerns. However, we respectfully disagree with these new concerns and would like to clarify:

1. This is not incremental work on SSDM. Our proposed models - Neural Articulatory Flow (NAF), Full Connectionist Subsequence Aligner (FCSA), and Non-fluency In-Context Learning (NICL) - are completely different from SSDM. If this is your primary concern for rejection, please specify which modules you consider incremental. We are happy to discuss more. This also seems to contradict your original review's "strengths" section where you praised our contributions.

2. Regarding "broad significance to the ICLR community": Reviewer cH6N raised a similar concern, suggesting we should focus on general "articulation-based dysfluency" rather than just nfvPPA. We believe this concern has been addressed as it relates to writing problems. Our SSDM 2.0 has significant applications in speech disorders, language screening, and language learning, as noted in your "strengths" section. Per ICLR's code of ethics (https://iclr.cc/public/CodeOfEthics), particularly "Respect the Work Required to Produce New Ideas and Artefacts" and "Be Fair and Take Action not to Discriminate," we disagree that our paper has limited ICLR audience appeal or would be discriminated. Moreover, SSDM was published at NeurIPS, which has a comparable audience size to ICLR.

While we're not suggesting any violation of ICLR's code of ethics from the reviewer, we don't believe these are valid grounds for rejection. We welcome further clarification of your concerns and would appreciate it a lot if you present all concerns at once to allow for prompt rebuttals.

3. Can the authors provide more comprehensive ablation studies to verify the necessity of each proposed module, including experiments showing the impact of each loss term in Equation 11 on model performance?

This is a great suggestion which is also a request from Reviewer f1SE and thus we use the same reply.

We conducted additional comprehensive exploration of ablations across different modules (NAF, FCSA, NICL) and loss objectives in Eq. 11. This aligns with reviewer wcgM's feedback, and we respond as follows:

SSDM 2.0 has introduced three major modules (NAF, FCSA, NICL) and multiple loss objectives for each module, making it necessary to explore the importance of each component. While Table 3 discussed the results when replacing individual SSDM modules, here we present additional ablation studies. For simplicity, we focus on Libri-Dys inference experiments. We first explain the notation. Starting with the baseline SSDM, single-module replacements are denoted as: SSDM+NAF, where we replace SSDM's gestural scores with our NAF gestural scores; SSDM+FCSA, where we replace SSDM's CSA with our FCSA; and SSDM+NICL, where we replace SSDM's vanilla language modeling with our NICL. We can also replace multiple modules simultaneously: SSDM+NAF+FCSA, SSDM+NAF+NICL, and SSDM+FCSA+NICL. Note that SSDM 2.0 is equivalent to SSDM+NAF+FCSA+NICL.

For loss objective ablations, we refer to the complete loss function in Eq. 11. The NAF module involves three losses: $\lambda_1\mathcal{L}\_{\text{KL}}$ + $\lambda_2\mathcal{L}\_{\text{FLOW}}$ + $\lambda_6\hat{\mathcal{L}}\_{\text{PIT}}$, where only $\hat{\mathcal{L}}\_{\text{PIT}}$ can be ablated as the first two are essential. FCSA includes two losses: $\lambda_3\mathcal{L}\_{\text{PRE}}$ + $\lambda_4\mathcal{L}\_{\text{POST}}$, each of which can be ablated. NICL has a single loss $\lambda_5\mathcal{L}\_{\text{CON}}$ that can be ablated. These ablations are denoted as $\text{SSDM+NAF-}\hat{\mathcal{L}}\_{\text{PIT}}$, $\text{SSDM+FCSA-}\mathcal{L}\_{\text{PRE}}$, $\text{SSDM+FCSA-}\mathcal{L}\_{\text{POST}}$, and $\text{SSDM+NICL-}\mathcal{L}\_{\text{CON}}$.

All results are presented in Table 12 in the updated manuscript. In terms of both F1 score and Matching Score (MS), replacing any single module in SSDM leads to performance improvement. Replacing an additional module (two modules in total) further enhances performance. Regarding the loss function, the posterior interpretable training (PIT) loss appears to have minimal influence. An interesting observation with FCSA is that incorporating both losses, $\mathcal{L}\_{\text{PRE}}$ and $\mathcal{L}\_{\text{POST}}$, delivers strong performance. However, removing either one results in a performance drop, although this trend becomes less pronounced with more data available. Overall, each module and each loss in our proposed framework demonstrates its effectiveness. For scalability, when all components are integrated, scalability increases dramatically. However, using only one or two modules yields less effective scalability improvements.

Please feel free to check Appendix A.18 which was highlighted in blue in the update manuscript.

We greatly appreciate the reviewers' time and their in-depth feedback on our paper.

1. HubERT Training and Table 1 ablations.

In Section 6.3, we would like to provide a more detailed explanation of our approach. We use HuBERT-Large representations as input and employ a two-linear probe ((D,D)+(D,40)) for frame-level phoneme classification, where D is the input dimension (768 for HuBERT-Large). The phoneme labels in our task are dysfluent labels. For example, the word "please" with its standard phoneme sequence "P P L L IY IY IY IY Z Z" could have dysfluent labels like "P P L L IY IY IY IY L L IY IY IY IY SIL SIL SIL SIL Z Z", indicating a repetition of "L IY" and a block.

Due to the dysfluent nature of the labels, we found that traditional speech representations (HuBERT, WavLM, or Codecs) exhibit poor scalability: increasing the training data size and model capacity does not significantly improve phoneme classification performance. We observed some improvement by adding a CTC branch specifically for dysfluent phoneme recognition (without duration). Our current architecture therefore consists of two branches: a two-linear layer probe for phoneme classification and a two-layer CTC for phoneme recognition. During inference, we only utilize the first branch.

Throughout this stage, we maintain either HuBERT or our NAF fixed. Our experiments with finetuning either HuBERT or NAF showed minimal performance differences.

• WavLM ablation

We have expanded our ablation studies to include WavLM-Large and updated Table 1 (highlighted in blue). The results demonstrate that WavLM-Large offers only marginal improvements over HuBERT and still performs substantially below our NAF model in phoneme recognition. We welcome any feedback on our updated manuscript.

2. Could the authors include more diverse baseline model comparisons, such as using a speech encoder like HuBERT or Whisper encoder with fine-tuning via LoRA, to better assess the impact of the new modules on performance improvement?

Thank you for raising this point. Using a Whisper encoder with LoRA finetuning is the approach implemented in SALMONN [1] for spoken language understanding. We evaluated SALMONN-13B in zero-shot settings and also fine-tuned it with our Libri-Co-Dys corpus (denoted as SALMONN-13B-FT in Section 6.5, Table 5). The results indicate poor performance, suggesting that this is not merely a data-driven problem. This further validates the effectiveness of our proposed modules.

[1] Tang, Changli, et al. "SALMONN: Towards Generic Hearing Abilities for Large Language Models." ICLR, 2024

Dear Reviewer,

Thank you for your detailed reviews, which have helped us significantly improve our manuscript. We would greatly appreciate your feedback on whether our revisions have adequately addressed your concerns. If so, according to recent PC regulations, we would like to sincerely request consideration for a score raising. If any concerns remain, we are happy to provide additional clarification or engage in further discussion.

Thank you again for your time and consideration!

Sincerely

6. Libri-Co-Dys (with multiple dysfluencies) is presumably harder than Libri-Dys (with a single dysfluency). However, the F1 scores using SSDM 2.0 on Libri-Dys appear to be no worse than that on Libri-Co-Dys (across 30, 60, 100 training data settings). Could the authors comment on this?

This is an interesting point. To clarify, you mean "F1 scores using SSDM 2.0 on Libri-Dys appear to be worse than those on Libri-Co-Dys" (versus your initial point about "no worse").

There are two categories.

(1) Phonetic transcription results. Looking at results from Table 1 and Table 2, for Libri-Dys, the scores are 92.0, 93.2, 95.0, and for Libri-Dys-Co, they are 92.7, 93.8, 96.0 respectively. Although Libri-Co-Dys contains more complex speech patterns, the results seem to be consistently better than those on the seemingly easier Libri-Dys data. We believe this is due to the nature of the labels. The labels we used for training and evaluation in Table 3 and Table 4 are real dysfluent phoneme labels. For example, assume the original text is "P L IY Z". After simulation, for Libri-Dys, the label might be "P L EY Z" indicating a replacement, while for Libri-Co-Dys, the label might be "P L P L EY Z" indicating both a stutter and a replacement. If we were to use the original text "P L IY Z" as labels, we would expect lower F1 scores on Libri-Co-Dys since it contains more disrupted speech. However, since we use the actual "dysfluent phoneme" transcriptions (what people actually said), the labels in both Libri-Co-DYs and Libri-Dys accurately reflect the speech content. Therefore, these are not actually "harder" samples. Data becomes "harder" only when labels don't match the speech (such as when using fluent text as labels).

(2) Dysfluency detection Looking at results from Table 3 and Table 4, for Libri-Dys, the scores are 80.0, 83.2, 86.2, and for Libri-Dys-Co, they are 81.4, 83.0, and 87.0 respectively. Results on Libri-Co-Dys appear to be better. For this task, the labels are dysfluency labels, formatted as {Phoneme: "IY", Start: 0.68, End: 0.79, Type: "replace"}, as shown in Appendix A.2.1. Since we have accurate dysfluent labels, Libri-Co-Dys samples are not necessarily harder. Samples typically become harder when labels are less accurate or match the speech less well. Consider an analogy from image recognition: if an image shows a dog in the foreground and a cat in the background, and the label is just "dog," this would be a harder sample than an image containing only a dog. However, if the label is "dog + cat" perfectly matching the image content, it becomes difficult to determine whether this is a hard or easy sample. But this is indeed an interesting point. For both cases, as for why Libri-Dys-Co performs slightly better than Libri-Dys, we believe this is because Libri-Co-Dys is larger than Libri-Dys (6k versus 4k hours). Thus, for each portion (30%, 60%, 100%), we have more data in Libri-Co-Dys. We welcome any comments on this analysis.

7. Is there any fraction of Libri-Co-Dys that does not contain dysfluencies? If not, it is useful to check how SSDM 2.0 performs on fluent instances. Please comment.

Thanks for bringing this up. Libri-Dys-Co does not contain fluent speech. Let's examine speech recognition from a unified perspective. Assume a person says "Pl-l-l-ease c-c-call stella"; a good ASR system should provide two outputs: "what the person actually said" (the pronunciation transcription) and "what the person intended to say" (which is what current ASR models like Whisper produce). Our system can accurately handle both tasks.

For the latter output, if we simply ask the system "what the person intended to say," it functions as a normal ASR system, and we report these results in Table 8. Note that under this setting, the targets/labels are the ground truth text "what people should have said." Since SSDM 2.0 already demonstrates good performance in this task (even better than Whisper), we speculate that it will also perform well on fluent speech in terms of ASR.

However, it is indeed important to test these two tasks on fluent speech. To do this, we run SSDM 2.0 on LibriTTS test-clean and test-other sets to conduct two experiments:

Dysfluency pronunciation transcription (we report false positives: how many pronunciation problems are detected when they don't actually exist)

Normal ASR where we prompt the system to generate what the person intended to say

We report both new results in Appendix A.15 in the updated manuscript. The basic conclusions are that for the dysfluency detection task, we found only a small portion of false positives, which indicates some room for improvement in future work. For the ASR task, performance is comparable but slightly worse than Whisper, which we attribute to not having tuned the model specifically for ASR tasks.

4. Can the authors further motivate their choice of using a count encoder + index generator to identify active regions? One could devise alternate schemes like using discretized binary gates (using Gumbel softmax) for every element in $X_i$ to denote whether or not it is part of an active region. It would be useful to know if the proposed design was motivated by some articulatory or speech production priors. There are two reasons why we do not use element-wise prediction (whether or not it is part of an active region).

First, as we illustrated before, based on human speech production priors, the human speech production system only cares about when each articulator is activated and for how long. This motivates us to develop a more compositional method to model the human speech production system, distinct from traditional element-wise dense speech representation learning methods.

Second, in the SSDM work, they already used such dense element-wise prediction, predicting the intensity of each element followed by post-processing sparsity methods. That pipeline is overly complex, and we aim to simplify it. Additionally, our region-wise prediction is much more efficient than element-wise prediction. As shown in the example in Appendix A.5, element-wise prediction requires $K\times T$ predictions, while our index-count pipeline needs only 3 count predictions, 6 index predictions, and some intensity predictions. Note that in SSDM, H is still a $K \times T$ matrix which remains dense, whereas in our method, H is a sparse matrix and we only store the activated elements (about only 5%-10% of elements) during computation.

5. From the proposed SSDM 2.0 architecture and the demo page, the LLM responses are descriptive and in natural language. Apart from objective measures of F1/WPER, did the authors consider evaluating these free-form responses containing pronunciation feedback via a human evaluation?

That is a good suggestion. Our primary goal is to predict the time and type of dysfluencies, which is already well-evaluated by automatic metrics including F1 score and matching score (taking time IoU into consideration), as visualized in Appendix A.3.2. In terms of prediction correctness, human scoring may not be necessary. However, we identify two cases where human evaluation could be valuable:

First, we use pre-designed prompts to extract the time and type of dysfluencies. To verify that these prompts don't fail in certain cases (i.e., fail to correctly extract information), we can use human efforts to manually evaluate the correctness. This essentially evaluates the effectiveness of prompts as shown in Appendix A.11. We could use a scoring system of 0, 0.5, and 1, where 1 indicates the prompt successfully extracts both type and time, and 0 indicates failure.

Second, regarding interface design, users desire more friendly and effective feedback. Even when dysfluencies are detected, the feedback (such as "strengthen the pronunciation of sound k") is still generated somewhat randomly. We could use mean opinion scores, commonly used in TTS evaluation, to assess how satisfied humans are with the general feedback. We plan to implement these evaluations in the future. Additionally, whether such labeled data can further benefit training (like RLHF) might also be an interesting topic to explore.

We greatly appreciate your detailed reviews. Here are our responses:

1. The writing could be improved overall, especially the main technical parts in Sections 3, 4.

We acknowledge that some descriptions were unclear. To address this, we have added Section 2 (highlighted in red) in the updated manuscript to provide an overview of SSDM 2.0.

SSDM 2.0 (shown in Fig. 2) processes dysfluent speech and a textual prompt as inputs, generating pronunciation transcriptions as output. To achieve this, we propose the Neural Articulatory Flow (NAF), which generates scalable speech representations referred to as gestural scores. Subsequently, the gestural scores are aligned with the reference text using a Full-stack Connectionist Subsequence Aligner, producing aligned speech embeddings for each token in the reference text. These aligned embeddings, combined with pre-defined prompts, are then input into a LLaMA module for instruction tuning, a process we term Non-fluency In-context Learning. The following subsections provide a detailed explanation of each module. Please review Section 2 in our updated manuscript. Note that we have moved the "Related Work" section to the appendix to save space. However, we welcome your feedback on these changes.

We also corrected the notations as pointed out. Please check our updated manuscript for content highlighted by red.

2. Explanation on Gestural scores H, Articulatory trajectory data X and number of gestures K.

X is articulatory data in $R^{D\times T}$, where T represents time and D is the channel dimension. In our setting, D equals 12, corresponding to x,y coordinates of 6 articulators. As raw data, X is at a relatively low level, making it difficult to capture high-level information. Therefore, we use convolutional matrix factorization (also called dictionary learning) to decompose X into two parts: the dictionary G in $R^{T'\times 12 \times K}$, which contains K kernels, where each kernel is a basic time-series pattern in $R^{T'\times 12}$, and the gestural scores H, which are the projections of X onto the kernel space G. H has shape $R^{K\times T}$, where K is the number of kernels (a hyper-parameter).

We visualized X, G, and H in Appendix A.5. Essentially, H represents when different kernels are activated. To illustrate with an example from Appendix A.5, consider articulatory data X where only the upper lip (purple lines), lower lip (gray lines), and tongue dorsum (red lines) move. In the simplest case, kernel 1 (G1), kernel 2 (G2), and kernel 3 (G3) represent the basic moving patterns of the upper lip, lower lip, and tongue dorsum, respectively. The gestural score H then shows when each kernel is activated and for how long. In the example, H shows that kernel 1 is activated for 3 time steps, kernel 2 for 6 time steps, and kernel 3 for 3 time steps. This figure presents a simplified example for illustration; in reality, the patterns can be more complex, with one kernel potentially involving multiple articulators. In summary, H provides a high-level abstraction of X that is explainable, sparse, and scalable (as demonstrated in our experiments). This is analogous to speech SSL, where SSL units provide another form of high-level abstraction (in terms of semantics) of raw audio waves.

Although we provide a detailed illustration in the Appendix, we have added further explanation in Section 3.1 (highlighted in red) to enhance clarity, following your suggestion.

3. How is H Rule-Generated and How is Speech Production Prior Incorporated?

Traditional speech representation is a $R^{D\times T}$ matrix, where values are densely learned through data-driven methods. Our intuition is that in human speech production (prior knowledge), only several articulators are moving, and the activation (gestural scores) of these articulators is really sparse (in terms of space and time). This motivates us to ask: Can we develop rules to generate gestural scores instead of using traditional data-driven speech representation learning methods? By rules, we mean we simply predict the index of each activation for articulators and inject sparsity into them (restricted in the count encoder). So instead of predicting a $R^{D \times T}$ matrix as speech representation, we predict where and when the articulators are activated, which we term as rule-based or semi-implicit representation learning. Actually, this is not a purely rule-based system; it is more like representation learning with injected human priors.

Also following Reviewer f1SE's suggestion, we have added introductory sections to Section 3 (before 3.1) and Section 3.1 in our updated manuscript, highlighted in green.

Dear Reviewer,

Thank you for your detailed reviews, which have helped us significantly improve our manuscript. We would greatly appreciate your feedback on whether our revisions have adequately addressed your concerns. If so, according to recent PC regulations, we would like to sincerely request consideration for a score raising. If any concerns remain, we are happy to provide additional clarification or engage in further discussion.

Thank you again for your time and consideration!

Sincerely