FINALLY: fast and universal speech enhancement with studio-like quality

Dear Reviewer,

Firstly, we would like to thank you for your invaluable work. Below, we address your concerns about the paper.

W1. Diffusion Models and GANs for Conditional Sampling

We generally agree that generative models, such as GANs and diffusion models, are not necessarily used to allow multiple generations from posterior distributions but rather due to their high-quality results. However, we think that this consideration does not downplay our contribution. We provide a theoretical interpretation for the case when samples are generated using an LS-GAN generator. Theoretical interpretations are important for in-depth analysis and might have a significant impact on how the field evolves in the future. From the practical side, we argue that learning the whole posterior distribution might not be necessary for the speech enhancement problem and therefore diffusion models might be solving an unnecessarily complex task. Our analysis reveals that GANs provide a natural remedy to regress for the most probable speech reconstruction directly, thus speech enhancement GANs solve a simpler task with fewer potential resources. Therefore, we believe that Section 2 holds a significant contribution to the field.

W2. Standard Datasets and Metrics for Speech Enhancement

We would like to clarify the reasons behind our choice of datasets and metrics.

We chose the VoxCeleb and UNIVERSE validation set because these data include several degradations at the same time, and the strongest baselines for universal speech enhancement, which are HiFi-GAN-2 and UNIVERSE, release results on this data. Other methods from the literature usually consider only one degradation (e.g., only noise or reverberation) or are significantly inferior to HiFi-GAN-2 and UNIVERSE. For instance, the popular speech enhancement VCTK-DEMAND dataset considers only additive noise as the distortion, and methods achieving good results on this dataset tend to poorly generalize to real data. Thus, comparison on this data is not particularly important from a practical point of view.

Our paper does not include similarity-based metrics such as PESQ and STOI for two reasons. First, since PESQ and STOI require ground truth reference audio, we are unable to compute them for real data such as VoxCeleb and LibriTTS, as there is no ground truth reference for such data. Second, there have been a number of works consistently reporting low correlation of reference-based metrics with human perceptual judgment [1, 2, 3]. In particular, the [1] study reports that no-reference metrics (including DNSMOS, which we reported in our work) correlate significantly better with human perception and therefore have higher relevance for objective comparison between methods. Furthermore, in our study, we report the MOS score, which directly reflects human judgments of restoration quality.

However, we agree that outlining conventional metrics such as PESQ and STOI on a popular VCTK-DEMAND benchmark could facilitate comparison with prior work. One can find this comparison in the table below. Note that we will add these results in the appendix of the camera-ready paper.

Table 1. Comparison with baselines on VCTK-DEMAND dataset.

As one can see, our model clearly outperforms all baselines in terms of the MOS metric and reference-free objective metrics, while reference-based metrics correlate poorly with human judgments. Note that our model has a slightly higher MOS than the ground truth likely due to the fact that our training data is of higher quality than the VCTK ground truth samples. Additionally, we would like to point out that one can find a comparison on real data with the mentioned DEMUCS model in Table 2 of the paper.

Q1. Qualitative Comparisons Against the Baselines

We agree that the work could benefit from more qualitative results. We will provide more qualitative comparison results on the paper's web page upon acceptance, as we cannot modify supplementary materials during the rebuttal period according to NeurIPS rules.

[1] Manocha, Pranay et al. "Audio Similarity is Unreliable as a Proxy for Audio Quality."

[2] Manjunath, T. "Limitations of perceptual evaluation of speech quality on VoIP systems."

[3] Andreev, Pavel et al.. (2022). "HiFi++: a Unified Framework for Bandwidth Extension and Speech Enhancement."

Dear Reviewer,

Thank you very much for your high assessment of our work. We truly appreciate your valuable comments. Below, we address your concerns.

W1. Comparison with HiFi++ & Q1. Advantage of incorporating the WavLM Encoder

The original HiFi++ model was proposed for speech denoising and bandwidth extension applications separately. Therefore, a significant practical difference between our work and HiFi++ is that our model was trained to support a wide range of degradations emerging in practice, and thus our model is able to generalize to real-world data. Additionally, our model is trained with novel LMOS and HF losses, the effectiveness of which is validated by ablation studies. Therefore, our training framework is substantially different from that of HiFi++, and the importance of this difference is validated by practical observations.

However, you are absolutely correct that the generator architecture of our model is mostly based on HiFi++. The main differences in this regard are the introduction of Upsample WaveUNet and the WavLM encoder. While the importance of Upsample WaveUNet is clearly validated by ablation studies, the effect of the WavLM encoder appears to be somewhat marginal in terms of MOS score (although we must pinpoint that objective metrics are considerably higher for the case with WavLM). To validate the importance of the WavLM encoder, we have conducted an additional ablation test on the UNIVERSE validation set, which contains more challenging cases than the Voxceleb data. The results are provided in the table below.

Table 1. Ablation of WavLM encoder on UNIVERSE validation.

Thus, we conclude that the introduction of the WavLM encoder is very important to achieve high quality on more challenging data. We will provide these additional results in the camera-ready paper.

W2. Description of the Model Size/Training Data Scale for the Baseline Methods

We agree that for better contextualization of our result, detailed information on the model size and training data of the baseline models is needed. Below, we provide a table, which will be included in the camera-ready version:

Table 2. Comparison of resources with baselines.

We note that, while our model has a larger number of parameters than the baselines, most of these parameters are used to process low-resolution features (e.g., the Transformer of WavLM operates on 320-times downsampled representations of the waveform, i.e., at 50 Hz). In contrast, models like HiFi-GAN-2 mostly operate at full waveform resolution (due to WaveNet). This allows our model to be more compute-efficient and thus have a much lower Real-Time Factor (RTF).

W3. Self-contained Description of Evaluation Dataset

Since we took the evaluation data from prior work, we did not describe it in much detail. However, we agree that our paper would benefit from such a description. We will include it in the camera-ready paper. Please find the evaluation dataset details below.

VoxCeleb Data: 50 audio clips selected from VoxCeleb1 [1] to cover the Speech Transmission Index (STI) range of 0.75-0.99 uniformly and balanced across male and female speakers.

UNIVERSE Data: 100 audio clips randomly generated by the UNIVERSE [2] authors from clean utterances sampled from VCTK and Harvard sentences, together with noises/backgrounds from DEMAND and FSDnoisy18k. The data contains various artificially simulated distortions including band limiting, reverberation, codec, and transmission artifacts. Please refer to [2] for further details.

W4. The Results for the 1st Stage (Pretraining Setup)

Thank you very much for noticing this issue. Indeed, the table with these results is missing in the Appendix. We provide it below and will include it in the Appendix of the camera-ready paper.

Table 3. Results on Voxceleb data after pretraining with different regression losses.

Q2. More Focused Evaluation

We agree that additional results concerning individual degradations would be helpful. We will include qualitative examples on the project web page upon paper publication.

[1] Nagrani, Arsha et al.. "Voxceleb: a large-scale speaker identification dataset."

[2] Serrà, Joan, et al. "Universal speech enhancement with score-based diffusion."

Dear Reviewer,

Thank you for your time and consideration. We would like to address your concerns about the paper.

W1. It still needs objective metrics such as PESQ and STOI for a solid evaluation

Our paper does not include similarity-based metrics such as PESQ and STOI for two reasons. First, since PESQ and STOI require ground truth reference audio, we are unable to compute these metrics for real data such as VoxCeleb and LibriTTS, as there is no ground truth reference for such data. Second, there have been numerous works consistently reporting low correlation of reference-based metrics with human perceptual judgment [1, 2, 3]. In particular, study [1] reports that no-reference metrics (including DNSMOS, reported in our work) correlate significantly better with human perception and therefore have higher relevance for objective comparison between methods. Furthermore, in our study, we report the MOS score, which directly reflects human judgments of restoration quality. Therefore, we believe that our evaluation could be considered solid.

However, we agree that outlining conventional metrics could facilitate comparison with prior work. Therefore, we have decided to measure these metrics on the popular denoising benchmark VCTK-DEMAND

Table 1. Comparison with baselines on VCTK-DEMAND dataset with conventional metrics.

As one can see, our model clearly outperforms all baselines in terms of the MOS metric and reference-free objective metrics, while reference-based metrics correlate poorly with human judgments. Note that our model has slightly higher MOS than the ground truth due to the fact that our training data is of higher quality than VCTK ground truth samples.

W2. The three-stage training process of FINALLY is complex, and the performance gains over HiFi-GAN-2 do not justify the increased cost

We agree that our training process is slightly more complex than that of HiFi-GAN-2. However, we would like to point out that our training pipeline, although intricate, is not considerably more complex compared to HiFi-GAN-2. HiFi-GAN-2 (similarly to our model) has three stages: 1) acoustic feature prediction network, 2) WaveNet training, and 3) adversarial training. Furthermore, our final model is more than 10 times faster (0.03 RTF compared to 0.5 RTF for HiFi-GAN-2). Therefore, we believe that the complexity of the training process is well justified by the dramatic increase in the efficiency of the resulting model.

W3. Reducing the model size by 90% for ablation studies is not convincing, as it can significantly affect the results.

We follow a well-established practice in deep learning literature of conducting ablation studies on a smaller scale in order to reduce the costs of training, as many leading papers in the field have done (e.g., [4]). While this procedure may influence the results, practical considerations in a resource-intensive field such as ours remain significant. Therefore, we use a smaller model for some parts of the ablation study.

Q1. Why DNSMOS and other metrics trend differently in Table 3.

Due to the imperfections of different objective metrics, they can trend differently as some of them pay more attention to certain artifacts than others. Please consider taking into account relevant papers [1, 2, 3] in order to understand the issues with objective metrics for speech quality in more detail.

Q2. Order of metrics in Tables 2 and 3.

Thank you very much for noticing this. We will rearrange the columns to have a consistent order in these tables to improve the clarity of presentation in the camera-ready version.

[1] Manocha, Pranay, et al.. "Audio Similarity is Unreliable as a Proxy for Audio Quality."

[2] Manjunath, T. "Limitations of perceptual evaluation of speech quality on VoIP systems."

[3] Andreev, Pavel, et al. "HiFi++: a Unified Framework for Bandwidth Extension and Speech Enhancement."

[4] Karras, Tero, et al. "Analyzing and improving the training dynamics of diffusion models."

Dear Reviewer,

We are very grateful for the high assessment of our work and your valuable suggestions. Below, we address your concerns.

W1. Work by Maiti and Mandel (2019)

Thank you very much for pointing out this work. We will add a discussion of it to the related work section in the camera-ready version.

W2. Minor comments.

• “Presumably it was the human listening test described in the appendix, but this is not clear at this point in the manuscript.” – You are correct. We will add a reference to the relevant appendix section.
• “Equation (2): please define phi.” – Phi denotes the WavLM-Conv feature mapping. We will add this clarification in the camera-ready version.
• “Since these are algorithms predicting human feedback, I don't think their outputs can be called human feedback itself.” – We can call these losses “predicted human feedback losses” instead of “human feedback losses.” We believe this clarification will indeed improve clarity, and therefore we will use this naming in the camera-ready version.

Ibrahim, hello! Thank you for your interest to our paper and your effort to implement it.

1. we have 454 M model. More precisely:

• spectralunet: 4.5M;

• hifi: 47M; standard v1 hifi ~14M and preblocks 32M , processing tensor before hifi(which were not mentioned in paper by our fault): [[[Conv1d(512, 512, kernel_size, dilation), conv1d(512,512, kernel_size)] for dilation in [1,3,5]] for kernel_size in [3,7,11]]

• waveunet: 10.3M;

• spectralmasknet: 16.7M;

• wavlm_large: 315M;

• post wavlm processing block: 50 M; (7 x Conv1d(ch_in=1536, ch_out=1536, kernel = 3, dilation=1), 1 x Conv1d ch_in=1536, ch_out=512, kernel=1, so it is actually) we have different numbers here since in paper we have not --specified depth of this block;

• upsampler 15M;

2. In both stages, we randomly sampled from from LibriTTS-R;

3. We used audio samples with length 4.096 seconds;

4. Batch size 32 was used for all stages;

5. As positional embedding, in SpectralUNet we add dimension with number = frequency_index / 513. So [B, F, T] -> [B,1,F,T];

6. The main reason for such choice was the time limit, so our models haven't fully converged;

7. According our experience, STFT loss works better than Melspec loss, and stage 2 and 3 help to remove artifacts typical for STFT loss. Could you specify what kind of artifacts do you observe?