SSDM: Scalable Speech Dysfluency Modeling

We really appreciate the time and effort put into giving such in-depth comments.

• Yes. Our model was trained entirely on synthetic data. We chose high-quality open-source multi-speaker TTS models like VITS and StyleTTS2 to mitigate potential overfitting, though we lack concrete evidence of overfitting to specific TTS models. We did tests using VITS on VCTK data and they showed similar results, with slightly lower detection accuracy but still a significant gap between simulated and nfvPPA data. We may have overfitted to fluent TTS models, as no large-scale dysfluent TTS model currently exists.

  • Regarding the questions, we think those are great approaches to try. We all know that current multi-speaker TTS models are trained within a limited set of speakers. We do see efforts in scaling TTS, such as Voicebox, SeedTTS, etc., that were trained on a wider range of speakers. However, the scaling effort is still limited, and those pretrained models are not planned to be open-sourced at this moment. So for a stronger and broader scaling plan, using SSDM on speech in the wild to generate pseudo labels should be a powerful and effective way for further scaling planning. Thank you so much for this insightful suggestion.

  • We strongly agree that SSDM can be used for TTS output evaluation. We can think of two ways to evaluate it: First, for the normal fluent TTS task, we would expect fewer dysfluencies to be detected given ground truth text and speech as input, with our "pronunciation prompt" as additional input. Second, we can also check the CSA alignments output, and we would expect the CSA alignments to be as monotonic as possible.

  • We believe that the evaluation results also depend on the TTS method used for simulation. Popular open-source TTS methods such as StyleTTS2 or VITS are non-AR based methods, which might implicitly inject more monotonicity for speech-text alignments. This might have double-sided effects: (1) The simulated Libri-Dys might not be that natural because of the forced monotonic effect that was naturally encoded. (2) There might also be bias in that our SSDM will be naturally monotonically affected, so some non-monotonic outputs from AR methods might not be affected. These are our assumptions. We should also try AR-based methods to generate simulation data, which should train our SSDM to then evaluate TTS outputs from both AR and non-AR based methods. We think this is a pretty interesting research direction.

• Regarding our CSA alignments: The intuition is that vanilla CTC treats each speech frame - text pair $y^{i,j}$ equally $\forall i,j$, or CTC treats each alignment equally. However, LCS only cares about the "first element" for each sub-sequence and will not take the remaining elements into the loss objective, as shown in Fig. 3. So our purpose is to create some inductive bias to make $y^{i,j}$ not equally considered for different $i,j$.

  • The leftmost sub-figure in Figure 4 gives an algorithm example. Assume we have a true alignment where text $C_2$ is aligned to $y^{2,2}, y^{3,2}, y^{4,2}$. In this example, the emission probability for $y^{3,2}$ and $y^{4,2}$ is actually not important since they are not considered in the loss objective in the off-line LCS algorithm. So there will be two changes in comparison to vanilla CTC: The transition probability should ideally happen between $y^{5,3}$ and $y^{2,2}$ instead of $y^{4,2}$, so $y^{3,2}, y^{4,2}$ are skipped and are called transition skip. However, it is hard for us to really do such a skip operation in the coding. We performed another slack, where we can just make $y^{4,2} = y^{3,2} = y^{2,2}$ so that we still model transition probability between $y^{5,3}$ and $y^{4,2}$. This is what emission copy means.

  • Then let's look at the forward algorithm in the middle part of Figure 4. We have $\alpha^{i,j}$ primarily determined by $\alpha^{i-1,j}$ and $\alpha^{i-1,j-1}$. $\alpha^{i-1,j} -> \alpha^{i,j}$ denotes emission copy. Here we would assume $\alpha^{i-1,j-1}$ has already copied $\alpha^{i-2,j-1}, \alpha^{i-3,j-1},...$, and the transition between $\alpha^{i-1,j-1}$ and $\alpha^{i,j}$ will be able to represent any transition between $\alpha^{i-k,j-1}$ and $\alpha^{i,j}$. So in this method, LCS is recursively encoded. Then we still take into consideration the other text $\alpha^{i-1,j-k}$ with a shrink factor, as this implicitly captures some dysfluencies such as missing. The approximation algorithm is developed primarily based on heuristic analysis, and there is no strict proof of why this approximation is truly LCS-aware. However, our experiments may have already proven this assumption. We are still working on the theoretical aspects to make it more solid. We will make this clearer and add more captions for Figure 4 in the updated work.

• We are still investigating the LLM part. We suspect that LLaMA might be treating this task as a "phoneme transcription" task instead of understanding, since the alignments from CSA already capture highly dysfluency-aware information. LLaMA might just be performing transcription on that. However, we are conducting experiments to do a more in-depth check as you suggested, focusing on in-context learning and reasoning abilities. We also suspect that this may be "due to the granularity of tokens" since LLMs were trained at word or token level (usually corresponding to many frames for speech), while phonemes are at a finer-grained level. So perhaps SSDM would benefit more from a phoneme-level or other hierarchical LLaMA.

• In p.5, l.172, we double-checked and believe there are some symbol issues. We believe there is no "i" since "i" is defined in Eq. 9. The alignment function should be $\gamma: {C_1,C_2,...C_L} \rightarrow \mathcal{P}(\tau_1,...\tau_t')$, which means we map $C_i$ to a subset of the sequence $\tau_1,...\tau_t'$, where $i=1,2..,L$.

• All other typos you proposed are valid, and we will correct them!

Sorry for the brevity in the previous review.

My main concern on this submission is: it is an application work focused on a very specific task on a narrow domain, which doesn't seem to have sufficient interest for the general audience of NeurIPS.

According to this paper (Sec. 1 and Sec. 2.1), speech dysfluency modeling is a task that detects both the types and the timing of disfluency in speech recordings given the reference text. By this definition, it is a very specific and narrow domain task, with forced alignment as the core technique.

All the key related works on this task as referenced by this paper in Sec. 1 [1, 2, 32, 33, 34, 35, 36, 27, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50] and [82] in Sec 2.1, including the "current state-of-the-art solutions [1, 2]", were published in various domain specific venues, notably in ICASSP and Interspeech.

Because this work is quite task-specific, I personally don't have an expertise for a thorough assessment for its technical contribution on this task.

That said, in addition to the main concern, I do have a few technical questions:

1. The proposed method uses an LLM (Llama) as the last components, which outputs in free form text. How were the numeric-based evaluation (F1, MS, etc.) done with the free-form text response?

2. According to Figure 2, the proposed method predicts only a single timestamp as the timing for each disfluency instance. What does this timestamp represent, starting, ending, or something else? Either way, without being a time range, how does IoU (Intersection over Union)-based Matching Score calculated in the evaluations?

3. In Sec 6.5 (and Table 3), the paper claims the proposed method as "State-of-the-art Dysfluency Detection". However, it's not compared to either of the "current state-of-the-art solutions [1, 2]". Although there is a comparison to [2] in Table 2, but Table 2 only include results on simulated datasets, without using the real disfluency dataset (nfvPPA). Does such evaluation support the claim?

4. In Table 3, ChatGPT is used as a baseline. Isn't ChatGPT a text-based model? How does it work for detecting disfluency in speech?

Dear Reviewer,

We understand that confusion or misunderstanding may arise from the definition of this research area. We believe we can agree that in the speech research community, the majority of people are working on TTS, ASR, or spoken language models (SLMs) due to the breakthrough of generative AI and LLM techniques. Only a few people are working on some rare research areas like our domain: dysfluent speech modeling. Because of this, most of the works we have heard about recently are related to TTS, ASR, LLMs, SLMs, etc., especially for those who have just entered the speech area for research and are excited about it. We believe this is the fundamental reason you might think this is a narrow domain rather than a popular or general one. This is totally understandable and makes sense.

However, we would like to explain our motivation further. The fact that not many people are working on this topic certainly indicates it is a "narrow domain," at least at this moment. However, it does not mean it is not important. We also indicate in the abstract that the research of dysfluency modeling is still at its "LeNet moment." So at the time when LeNet was invented, maybe other people also thought it was a narrow domain and not important. That's why Hinton's neural network was not taken seriously until many years later.

Before elaborating on our motivation for this work, we might pose a general question: Why are so many people working on popular areas like TTS or SLMs in the speech domain? We think it's fundamentally driven by the needs of the world, or more specifically, the market size. Because there is a huge need for some techniques, money will follow, and many people are needed in this area. Usually, people working together will create much more impactful work, such as those highly influential speech works we cited in the related work section.

If this is true, we have also done early market investigation showing that the market size for dysfluent speech (speech therapy and spoken language learning) is also substantial. We already reported the numbers in the introduction. The exact figure for the speech therapy market value (6.93B) is even larger than the TTS market size (3.45B) and is comparable to ASR. This means that, in terms of needs and market size, dysfluent speech modeling is as important as other areas such as LLMs, TTS, SLMs, ASR, etc. Also, we reported at the beginning of the introduction that 1 in 10 kids globally suffer from dyslexia. What does this mean? It means if I have a child in the future, there's a 1/10 probability that my child will also suffer from dyslexia. Since there is not an effective AI framework, I would absolutely encourage my child to try this SSDM to get a diagnosis. This 1/10 probability applies to all people globally. In this case, maybe this is even a more important area than popular areas such as TTS or ASR. We're not trying to compare which area is more important, which would be meaningless. We just mention that this is not a narrow domain and instead will become an increasingly important one.

But you might further ask, since it's such an important area, why aren't more people working on it? We mentioned this in the introduction and related work sections, as well as in the demo on the first page. The current large-scale SLMs such as LTU-AS, SALMONN, ChatGPT have limited ability to tackle this problem. Although we mentioned that, technically, dysfluent modeling is a spoken language understanding problem that could be classified under either ASR or SLMs, the work including [1,2] and our method indicate that the efforts needed to bridge the gap between ASR/SLMs and dysfluent modeling are much greater. It is a super difficult problem. Because of this difficulty, we personally think dysfluent speech modeling should be separated as an independent research area. Even in some speech conferences such as Interspeech/ICASSP/SLT/ASRU, this area has not been defined. So when we were suggested to submit this work to Interspeech/ICASSP, there might still not be a suitable venue for us. Regarding the specific point that the reviewer has raised, we checked the official NeurIPS requirements again and found:

"The Thirty-Eighth Annual Conference on Neural Information Processing Systems (NeurIPS 2024) is an interdisciplinary conference that brings together researchers in machine learning, neuroscience, statistics, optimization, computer vision, natural language processing, life sciences, natural sciences, social sciences, and other adjacent fields. We invite submissions presenting new and original research on topics including but not limited to the following: Applications (e.g., vision, language, speech and audio, Creative AI)."

So we would think the NeurIPs still encourages the speech application work submission. But if this is the core concern and still not clear, we would also love to invite the program chairs or area chairs to look through our work again and to analyze our qualifications.

We have clarified our points regarding the reviewer's primary concerns. We understand that the confusion happens because there are indeed not too many people working in this area. However, we are making efforts to make this area more well-known in the future and to attract a wider audience. So feel free to discuss this further. If possible, we would also love to invite you to help announce this paper to a broader audience in the future to benefit more patients with speech disorders.

Dear reviewer:

Thank you for your attention to our research. We greatly respect your perspective on our research domain, which we acknowledge is somewhat narrow. However, we cannot construct a meaningful rebuttal without understanding your specific concerns. We would appreciate clarification on any technical flaws you've identified, such as issues with our experimental design or methodology. We are eager to address these concerns or any confusion you might have. We also welcome any suggestions you might offer. We are happy to provide a more detailed explanation of our methods and to re-explain our work and its importance using clear, accessible language. We will explain our research step-by-step, from background to technical details, to enhance your understanding.

• Approximately 1 in 10 people suffer from speech disorders such as aphasia or dyslexia.

• When people suffer from speech disorders, they typically seek treatment at hospitals. However, the cost is often prohibitive, usually exceeding $1,000 for a single diagnosis, which can take over 5 hours. Additionally, there is a shortage of doctors and speech-language pathologists (SLPs). Consequently, such costs are often unaffordable for low-income families. Even in countries where the government covers these fees, treatment is time-consuming, and many people lack access due to the scarcity of specialists.

• This raises the question: Can AI automate this process? AI has the potential to provide an efficient and more affordable solution. For AI to automate speech disorder diagnosis, it needs to perform deep transcription of speech, including dysfluencies such as insertions, omissions, replacements, stutters, prolongations, and phonetic errors. Accurate timestamping is also crucial. SLPs have informed us that these are the official diagnostic procedures.

• Here's an example of what an AI model should output: Given an impaired speech sample, we expect our system to produce: "Y-You [block] wi-[stutter]-sh [missing] know all [prolongation] about my grandfather". In contrast, state-of-the-art models like Whisper, ChatGPT, or Salmonn typically produce a perfect transcription: "You wish to know all about my grandfather", which is not useful for our purposes. This demonstrates that current large-scale Automatic Speech Recognition (ASR) models show limited performance on these tasks.

• To underscore the importance of this research: Recall that a single treatment session costs about \$1,000 and takes approximately 5 hours. Our AI solution could potentially save both patients and healthcare providers (or governments) \\$1,000 and 5 hours per session. Given that 1 in 10 people have disordered speech, the market size is substantial. To our knowledge, AI exploration in this area is limited. Therefore, our work has broad impact, including both clinical and business implications.

Now, let's discuss the technical aspects of our work. We will introduce these step-by-step, and please don't hesitate to ask for clarification.

Our pipeline has two inputs: dysfluent speech and either golden truth text or a prompt.

• We obtain speech representations from dysfluent speech.

  • 2.1. While we could use traditional speech representations such as HuBERT, we found that our gestural scores are more scalable for this task, as demonstrated in Table 1. Therefore, we employ gestural scores instead of traditional SSL units.
  • 2.2. To obtain gestural scores, we first need articulatory kinematics data. We use Acoustic to Articulatory Inversion (AAI) to derive articulatory data X from speech. We then developed a VAE framework to generate gestural scores H from X. Gestural scores are sparse and duration-aware, so we developed a duration predictor, intensity predictor, and sparse sampler. We're happy to discuss any of these steps in more detail if you have questions.

• Connectionist Subsequence Alignment (CSA): We discovered that the Longest Common Subsequence (LCS) algorithm naturally captures speech dysfluencies, as shown in Figure 3. For the ground truth phoneme "AH", it aligns to the speech "AH AH ER AH", indicating a repetition of "AH" and an insertion of "ER". Based on this discovery, we developed a stochastic and differentiable LCS, named CSA, based on the vanilla CTC algorithm to provide dysfluency-aware alignments.

• CSA takes gestural scores H and text as input and outputs an alignment. This alignment is then input into LLaMA with pre-designed instructions to achieve end-to-end learning.

• Due to the lack of large-scale data, we developed a large-scale, TTS-based simulation dataset called Libridys, comprising about 4,000 hours of speech. We trained our entire model on Libridys. We then performed evaluations on Libridys, VCTK++, and real disordered speech data (nfvPPA).

We look forward to your feedback and are happy to provide further explanations or clarifications as needed. Please let us what step might confuse you.

Thanks reviewer for the thoughtful summarization and comments.

• Regarding the TTS question proposed in Strengths: We summarize that we performed textual dysfluency editing and simply input that text into a pretrained StyleTTS2 model, which is one of the open-sourced state-of-the-art TTS works. Our contribution is primarily in textual engineering, and we did not tune the model itself. Therefore, we believe this part of the contribution, at least technically, might not be novel enough to be considered one of our main contributions. We did observe some unnatural simulations, such as robot-like noise in stutter simulation, which we believe requires a learning method to overcome. For example, we were considering developing a GAN objective and fine-tuning the StyleTTS2 framework to address these issues. However, we have not yet accomplished this. Once we achieve a new technical-level breakthrough in TTS, we would be happy to list that as a core contribution. Due to page limits and the existence of other technical contributions (VAE, CSA, etc.), we do not have much space to elaborate further here. At this moment, we believe the core contribution might still be the open-sourced data. We hope to develop an updated TTS framework in our future work, and any suggestions are welcome!

• We acknowledge that our entire pipeline is complex. We attempted to simplify the system design initially but did not achieve the desired performance boost that we now have. As of now, all modules we proposed or utilized play an essential role and cannot be replaced. However, we are continuing our efforts to make SSDM simpler, more transferable, and more deployable. The reason we include so many details in the appendix is that we feel some essential modules, like gestures, are not well-familiarized in the speech domain. We hope that in the future, as people begin to realize the existence and importance of articulatory speech modeling, this article will be naturally simplified.

• Regarding specific questions, we believe we have introduced both motivation and methods in detail. However, some of this information might be included in the appendix due to page limits. For example, regarding self-distillation, there are two motivations we wish to explore:

  • Articulatory kinematics data are sparsely sampled from human articulators. This definitely results in some information loss, including intelligibility loss, pitch, or speaker identity information loss. The advantage is that it is scalable, as proven in Table 1, and interpretable for pronunciation training, as shown in section 6.6. So, how can we take advantage of these benefits while also guaranteeing intelligibility? We have "borrowed" more intelligibility from WavLM via self-distillation. This motivation was briefly discussed in lines 79-84 and detailed in Appendix A.3.

  • It has been evidenced in [1] that gestural score H implicitly captures linguistic boundaries such as phoneme segmentation. We also drew intuition from DINO [2], which showed that self-distillation in visual representations can implicitly enable image object segmentation, a so-called "Emerging property". Following work in speech [3] shows that self-distillation can also implicitly enforce word segmentation and better intelligibility [4]. Based on this intuition, we use self-distillation to enforce linguistic boundaries for gestural scores and to improve the upper bound of intelligibility that WavLM already achieves. These points are discussed in detail in Appendix A.3, where we also list other papers in this area. We mentioned in the main paper on line 159 that you can refer to our Appendix. However, we still feel sorry that due to page limits, we can only list this information in the appendix.

• We admit that dPER was indeed not well explained. Although we cited its reference, we should have explained it in our paper. Thank you for the suggestion!

• Regarding the question, ChatGPT was initially proposed as a language model only. However, in 2023, it also included speech I/O and became available (only) in mobile apps. So we now refer to ChatGPT as a general assistant that can process multiple modalities. We used the iPhone ChatGPT app, the 4-o, and conducted the inference experiments in May this year. However, we also note that the 4-o version with speech I/O that is currently available in the app is not the true 4-o speech demo they presented recently.

[1] "Deep Neural Convolutive Matrix Factorization for Articulatory Representation Decomposition"

[2] "Emerging Properties in Self-Supervised Vision Transformers"

[3] "SD-HuBERT: Sentence-Level Self-Distillation Induces Syllabic Organization in HuBERT"

[4] "DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learning"

Dear Reviewer:

We are pleased to address your proposed weaknesses and questions.

For weaknesses:

1. We acknowledge that the numerous modules and Figure 2 may be challenging to grasp. The training process involves all modules in Figure 2. For inference, we follow this path: speech -> gestural scores -> CSA alignments + instruction -> LLaMA -> response, removing modules like self-distillation and gestural decoders. Solid lines represent formal I/O processes in neural networks, while dashed lines denote other cases, such as loss functions. In the general rebuttal, we also provide additional explanations about our entire pipeline for further review. We will clarify it further in the revised version.

2. Our results show a small improvement with LLaMA, which we are still investigating. One hypothesis is that LLaMA was trained at the token or word level, whereas SSDM operates at the phoneme level. A phoneme-level LLaMA might yield more significant improvements. We do have a traditional method (as reviewer mentioned, the category label output), which is exactly the H-UDM [2]. H-UDM outputs the type and time of dysfluencies without integrating a "smoothed" language model. Tables 1 and 2 compare H-UDM in terms of scalability and detection effectiveness.

3. We compared three baselines: LTU-AS, SALMONN, and ChatGPT, which represent state-of-the-art speech understanding models. LTU-AS and SALMONN were trained via instruction tuning with both speech and text inputs, like our approach. So we believe it is fair comparison. For ChatGPT, we used the GPT-4o version from the iPhone app with a speech I/O interface, though it's unclear if it still uses Whisper. This comparison is fair given the extensive training data for these models. For a better fair comparison, we also fine-tuned LTU-AS-13B-FT and SALMONN-13B-FT with both speech and text inputs, but the performance boost was limited.

For questions:

1. Previous research [1] explored gesture numbers of 20, 40, 60, and 80, finding that 40 gestures yielded the lowest PER on MNGU0 dataset. While this may not apply to our larger Libri-Dys dataset, we adopted 40 gestures as default, which also aligns with the number of CMU phonemes. Using [1]'s code, we tested higher gesture numbers (100, 200, 1000) on MNGU0. Although intelligibility didn't degrade, interpretability decreased, making it difficult to associate gestures with specific articulatory movements. This interpretability is crucial for providing articulatory-based pronunciation feedback (Sec. 6.6). Thus, we used 40 gestures in our formal experiments. We will explore numbers below 100 in a revision.

2. There are two types of articulatory representations: a. Articulatory kinematics data X from UAAI; b.Gestural scores H derived from X. Both serve as "representations" of speech and are considered "articulatory representations."

• Articulatory data X, sparsely sampled from six human articulators, primarily conveys speech content but may miss acoustic details like speaker pitch. Research [4] shows that articulatory kinematics can serve as a universal speech codec. When combined with speaker embedding and signals like pitch, it can fully recover intelligible speech, achieving decent WER, CER, and MOS scores for resynthesized speech (LibriSpeech, VCTK, and AISHELL) as shown in Tables 2 and 3 of [4]. Note that [4] also primarily considers articulatory data as speech content, playing the same role that HuBERT plays in [5] or wav2vec2 plays in [6].

• Gestural score H from articulatory data X: Due to slight information loss in articulatory data, there is also inevitable loss in the gestural score H, as we focus on reconstructing articulatory data X. Our goal is to leverage its scalability and interpretability. To address this, we use self-distillation, allowing H to distill more acoustic information from pretrained WavLM representations. We believe this makes our articulatory representations H nearly fully intelligible. Although we did not directly evaluate H's information retention, we used two implicit metrics:

  • In [3], the original reconstruction loss when recovering X from H is about 20% with 40 gestures, corresponding to a phoneme error rate of 11.02 (7.52 on melspec). This indicates a loss of intelligibility in the gestural score. With our self-distillation method, the reconstruction loss drops to 2.08%, showing nearly full intelligibility. We plan to include more analysis in the updated paper.
  • Table 1 reports phoneme classification accuracy (F1 score), showing that our gestural score H outperforms HuBERT representations and achieves full intelligibility from speech.

3. Yes. Gestural scores H differ from traditional speech representations due to physical constraints. [1] defined speech as temporal activations of sparse gestures (articulators), making gestural scores naturally sparse. We applied a sparse constraint for physical consistency. Without the sparse constraint (mask), we still achieved good detection results. However, two points are noteworthy:

• Scalability is affected: Without the mask, we tested that scalability becomes SF1=0.14 and SF2=0.17, matching HuBERT (Table 1).

• Interpretability is lost: Without the sparse constraint, gestural scores lose interpretability, making articulatory pronunciation feedback (Sec 6.6) impossible.

  In essence, while removing the mask doesn't significantly impact detection, it affects scalability and eliminates interpretability benefits.

[1] "Articulatory phonology: an overview"

[2] "Towards Hierarchical Spoken Language Dysfluency Modeling"

[3] "Deep Neural Convolutive Matrix Factorization for Articulatory Representation Decomposition"

[4] "Articulatory Encodec: Vocal Tract Kinematics as a Codec for Speech"

[5] "Speech Resynthesis from Discrete Disentangled Self-Supervised Representations"

[6] "Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations"

Dear Reviewer:

We again acknowledge that the writing is not as straightforward as it could be due to the complexity of the subject matter, and we are making efforts to improve its visualization (global rebuttal and future work). Apart from that, please let us know if your primary concerns/questions have been addressed. Feel free to discuss or ask any other questions you may have so that we can provide prompt feedback!