VoiceNoNG: High-Quality Speech Editing Model without Hallucinations

1.& 2. The overall contribution is minimal with no original idea presented in the paper. It appears that the authors have merely replaced the feature representation in VoiceBox model with DAC features.

We would like to first thank the reviewer for recognizing that our paper is well-written with a clear description and strong motivation. The goal of this paper is to address the problems present in current state-of-the-art speech editing models: VoiceCraft (struggles to generate speech accurately following the target transcription) and Voicebox (suffers from reduced speech quality when background audio is present). Identifying these issues is also one of the contributions of this paper.

Although we considered a more sophisticated framework to tackle these issues (e.g., applying a speech enhancement model to disentangle background audio from speech and performing the infilling separately), we found that our current simple framework can already effectively address these challenges (please listen to the demos in our demo page).

Because our proposed solution is relatively elegant, we decided to focus more on the motivation and experimental parts to provide insights to the community.

Although the model framework is simple, we believe this paper makes significant contributions to the related research community.

• 1, We identify the robustness problem of neural codec language models (e.g., VoiceCraft) in speech editing.

• 2, We conduct comprehensive experiments to highlight the pros and cons of VoiceCraft and Voicebox.

• 3, Our ablation study and Figure 3 demonstrate the importance of flow-matching for modeling pre-quantization features with the VQ module, which can improve the robustness of models against small prediction error by implicitly converting the regression problem into a token classification task similar to NCLM.

• 4, The proposed VoiceNoNG achieves state-of-the-art performance in both objective and subjective evaluations.

• 5, Considering the potential for neural codecs to become a new audio format standard (such as mp3 format), the assumption that all codec-generated speech is fake may soon be unrealistic. Therefore, we propose a new and challenging practical scenario for deepfake detection, contributing to the relevant community.

We kindly ask the reviewer to listen to the demos of our proposed method and compare them with other SOTAs (https://anonymous.4open.science/w/NoNG-8004/), and please don’t overlook the contribution of this paper due to its simple framework.

3. The statements in the paper are sometimes very vague such as "This diversity makes RealEdit more challenging compared to ..." on line 222. Given that this is a scientific paper, there needs to be an explicit description of what other datasets lack which RealEdit provides.

In our original paper, the full context is: “The edits span various types—insertions, deletions, and substitutions—across different lengths, from short (1-2 words) to long (7-12 words), with single or multiple spans. This diversity makes RealEdit more challenging compared to other speech synthesis evaluation datasets.” As stated in the context, the diverse edit types (insertions, deletions, and substitutions) and different edit lengths (short, long) are what other datasets do not contain. We will revise the paper to make this more clear.

4. Evaluation of edited speech by WER is not helpful in determining the quality/intelligibility of generation because ASR models have an implicit language model that corrects mispronunciations and even words based on context.

We partially disagree with this statement. While it is true that ASR systems may correct mispronunciations based on context, they still provide valuable information about the intelligibility of generated speech. Additionally, previous speech editing works, such as VoiceBox and VoiceCraft, DO report WER results of ASR as a metric of intelligibility. For instance, section 4 of VoiceBox states: “Correctness and intelligibility: This can be measured by the word error rate (WER) of the synthesized speech’s transcription with respect to the input text, which has been adopted in prior work [Wang et al., 2018]. Public automatic speech recognition (ASR) models are used for comparability.” Similarly, VoiceCraft reports WER results in their Tables 3 and 4.

The paper presents a rather simplistic introduction to the proposed method, with much of the first two chapters focusing on popular science explanations.

The goal of this paper is to address the problems present in current state-of-the-art speech editing models: VoiceCraft (struggles to generate speech accurately following the target transcription) and Voicebox (suffers from reduced speech quality when background audio is present). Identifying these issues is also one of the contributions of this paper.

Although we considered a more sophisticated framework to tackle these issues (e.g., applying a speech enhancement model to disentangle background audio from speech and performing the infilling separately), we found that our current simple framework can already effectively address these challenges (please listen to the demos in our demo page).

Because our proposed solution is relatively elegant, we decided to focus more on the motivation and experimental parts to provide insights to the community.

Although the model framework is simple, we believe this paper makes significant contributions to the related research community.

• 1, We identify the robustness problem of neural codec language models (e.g., VoiceCraft) in speech editing.

• 2, We conduct comprehensive experiments to highlight the pros and cons of VoiceCraft and Voicebox.

• 3, Our ablation study and Figure 3 demonstrate the importance of flow-matching for modeling pre-quantization features with the VQ module, which can improve the robustness of models against small prediction error by implicitly converting the regression problem into a token classification task similar to NCLM.

• 4, The proposed VoiceNoNG achieves state-of-the-art performance in both objective and subjective evaluations.

• 5, Considering the potential for neural codecs to become a new audio format standard (such as mp3 format), the assumption that all codec-generated speech is fake may soon be unrealistic. Therefore, we propose a new and challenging practical scenario for deepfake detection, contributing to the relevant community.

We kindly ask the reviewer to listen to the demos of our proposed method and compare them with other SOTAs (https://anonymous.4open.science/w/NoNG-8004/), and please don’t overlook the contribution of this paper due to its simple framework.

In the abstract, the authors state that the poor performance of the NCLM-based model is due to attention errors (hallucination phenomena), but they provide only a few examples to support this claim, lacking more extensive experiments on attention mechanisms.

Please see our reply to Q1.

The paper mentions that the model combines the advantages of VoiceCraft and VoiceBox, but it seems to only merge the Codec with VoiceBox.

We believe the discrete Codec token is crucial not only for NCLM but also for the diffusion model, particularly the VQ module in the Codec. The VQ module can implicitly convert the regression problem into a token classification task, enhancing robustness against prediction errors made by the editing model, as shown in Figure 3.

Additionally, the authors point out that the poor performance of VoiceBox is due to the HiFi-GAN being trained on clean speech; however, the experimental section lacks a comparison with a HiFi-GAN trained on noisy speech for VoiceBox.

Please see our reply to Q4.

The pre-quantization feature by predicting DAC is a variant of Latent Diffusion, which has been widely proven to be effective. So the solution is not novel....

The main technical contribution we aim to highlight is that applying a VQ module provides additional robustness, as demonstrated in Section 3.1.6. We argue that VQ transforms the regression problem into a token classification task (similar to NCLM). As a result, small prediction errors—provided they do not exceed the token decision boundary—are still mapped to the correct token after VQ. While simple, we believe this observation will be valuable to the broader research community.

The goal of this paper is to address the problems present in current state-of-the-art speech editing models: VoiceCraft (struggles to generate speech accurately following the target transcription) and Voicebox (suffers from reduced speech quality when background audio is present). Identifying these issues is also one of the contributions of this paper.

Although we considered a more sophisticated framework to tackle these issues (e.g., applying a speech enhancement model to disentangle background audio from speech and performing the infilling separately), we found that our current simple framework can already effectively address these challenges (please listen to the demos in our demo page).

Because our proposed solution is relatively elegant, we decided to focus more on the motivation and experimental parts to provide insights to the community.

Although the model framework is simple, we believe this paper makes significant contributions to the related research community.

• 1, We identify the robustness problem of neural codec language models (e.g., VoiceCraft) in speech editing.

• 2, We conduct comprehensive experiments to highlight the pros and cons of VoiceCraft and Voicebox.

• 3, Our ablation study and Figure 3 demonstrate the importance of flow-matching for modeling pre-quantization features with the VQ module, which can improve the robustness of models against small prediction error by implicitly converting the regression problem into a token classification task similar to NCLM.

• 4, The proposed VoiceNoNG achieves state-of-the-art performance in both objective and subjective evaluations.

• 5, Considering the potential for neural codecs to become a new audio format standard (such as mp3 format), the assumption that all codec-generated speech is fake may soon be unrealistic. Therefore, we propose a new and challenging practical scenario for deepfake detection, contributing to the relevant community.

We kindly ask the reviewer to listen to the demos of our proposed method and compare them with other SOTAs (https://anonymous.4open.science/w/NoNG-8004/), and please don’t overlook the contribution of this paper due to its simple framework.

The new method proposed in the article, in my opinion, should be experimented on both TTS and speech editing, and compared with the VoiceBox model.

Thank you for your suggestion! We will consider exploring the TTS setting as part of our future work, as the proposed VoiceNoNG can also function as a TTS model. However, we believe speech editing presents a more challenging task in terms of content coherence. For the edited segment to sound natural, the speaker characteristics and background audio (e.g., noise, music, etc.) must remain consistent with the surrounding context.

This article describes the VoiceCraft model extensively, just to show that using gigaspeech data sets can enhance the effect of speech editing with background sound. In my opinion....

For a fair comparison, we also report the Voicebox results trained on GigaSpeech, as shown in Tables 1 and 2. Despite being trained on GigaSpeech, Voicebox still exhibits worse WER and speech quality compared to the proposed VoiceNoNG. This highlights a key drawback of Voicebox: its model framework, which relies on mel-spectrograms and HiFi-GAN.

Directly replacing the HiFi-GAN vocoder with the DAC decoder is also NOT an optimal solution. Although Tables 1 and 2 show that using 'Post-quantization' with DAC results in better WER (4.73 vs. 4.97) and speech quality (18.93 dB vs. 16.90 dB) compared to Voicebox*(Giga,Mel), applying Pre-quantization with the VQ module and CE loss achieves the best performance (WER: 4.54, speech quality: 20.44 dB).

Rough writing, lack of model or flow chart. There is also a problem with the organization of the article...

We apologize for giving you this impression. In fact, we carefully polished the paper before submission. Given that our proposed solution is relatively simple, we chose to emphasize the motivation and experimental sections to provide valuable insights to the community. Regarding the organization of the paper, we will follow your suggestion and make revisions.

4.Previous Work on Codec Embeddings:

Yes, NaturalSpeech 2 has explored generating codec embeddings using diffusion models.

However, as described in our section 3.1.3: “As noted in Section 2, the VQ module offers additional robustness against prediction errors made by our model. In contrast, if the output is the post-quantization features (similar to NaturalSpeech 2), only the DAC decoder is required for waveform reconstruction.”

We aim to highlight to the community that applying a VQ module can provide extra robustness benefits, as verified in our section 3.1.6. Unlike NaturalSpeech 2, which directly models quantized vectors without applying a VQ during inference (see Figure 2 in their paper), we argue that VQ can convert the regression problem into a token classification (similar to NCLM). Hence, a small prediction error, as long as it is not larger than the token decision boundary, will still be mapped to the correct token after VQ. We believe these insights can help the community build more robust diffusion models.

5.Concerns About WER Evaluation:

No, we followed the Voicebox approach, where unmasked regions are directly copied from the original waveform. As mentioned in Section 3.2: 'Additionally, for the audio condition, besides the original VoiceNoNG setting where non-edited segments come from the original audio, we consider a more challenging setting where non-edited segments are also resynthesized from the codec. We refer to this condition as VoiceNoNG (resyn).' The scenario where the entire utterance is passed through the vocoder or codec decoder to generate the waveform is only considered in the study of detecting edited speech.

6.Reference missing:

Thanks for sharing these papers. We will cite and discuss these papers in section 1 (Introduction).

1. Unbalanced Structure:

Thank you for pointing this out! The goal of this paper is to address the problems present in current state-of-the-art speech editing models: VoiceCraft (struggles to generate speech accurately following the target transcription) and Voicebox (suffers from reduced speech quality when background audio is present). Identifying these issues is also one of the contributions of this paper.

Although we considered a more sophisticated framework to tackle these issues (e.g., applying a speech enhancement model to disentangle background audio from speech and performing the infilling separately), we found that our current simple framework can already effectively address these challenges (please listen to the demos in our demo page).

Because our proposed solution is relatively elegant, we decided to focus more on the motivation and experimental parts to provide insights to the community.

Although the model framework is simple, we believe this paper makes significant contributions to the related research community.

• 1, We identify the robustness problem of neural codec language models (e.g., VoiceCraft) in speech editing.

• 2, We conduct comprehensive experiments to highlight the pros and cons of VoiceCraft and Voicebox.

• 3, Our ablation study and Figure 3 demonstrate the importance of flow-matching for modeling pre-quantization features with the VQ module, which can improve the robustness of models against small prediction error by implicitly converting the regression problem into a token classification task similar to NCLM.

• 4, The proposed VoiceNoNG achieves state-of-the-art performance in both objective and subjective evaluations.

• 5, Considering the potential for neural codecs to become a new audio format standard (such as mp3 format), the assumption that all codec-generated speech is fake may soon be unrealistic. Therefore, we propose a new and challenging practical scenario for deepfake detection, contributing to the relevant community.

We kindly ask the reviewer to listen to the demos of our proposed method and compare them with other SOTAs (https://anonymous.4open.science/w/NoNG-8004/), and please don’t overlook the contribution of this paper due to its simple framework.

2. Questionable Claim About Voicebox Performance:

Several research papers [1-2] have indicated that “HiFi-GAN does not generalize well to non-speech audio such as sound or music” [1]. Reviewers can listen to some distorted examples of HiFi-GAN generated music at this link: https://bigvgan-demo.github.io/. In fact, our subjective listening test (Figure 6) shows that Voicebox has comparable quality to VoiceCraft and the proposed VoiceNoNG under clean conditions (LibriTTS). However, as shown in Figures 7 and 8, when there is background audio (YouTube and Spotify), the speech quality from Voicebox is worse than that of VoiceCraft and the proposed VoiceNoNG .

[1] Vyas, A., Shi, B., Le, M., Tjandra, A., Wu, Y. C., Guo, B., ... & Hsu, W. N. (2023). “Audiobox: Unified audio generation with natural language prompts.” arXiv preprint arXiv:2312.15821.

[2] S.-g. Lee, W. Ping, B. Ginsburg, B. Catanzaro, and S. Yoon. “Bigvgan: A universal neural vocoder with large-scale training.” arXiv preprint arXiv:2206.04658, 2022.

3. Prior Work Overlooked:

Thank you for pointing this out, we will cite and discuss NaturalSpeech 2’s method when introducing equation (1).