FIRING-Net: A filtered feature recycling network for speech enhancement

Thank you for your insightful comments. We appreciate the opportunity to clarify the theoretical foundation behind how the model extracts target and non-target features, as well as the originality of the attention module based on energy segmentation.

Q1: The paper extensively discusses how the model uses operations such as generating masks, convolutions, and additions/subtractions to obtain the so-called target features and non-target features mentioned in the text. However, it lacks a theoretical explanation for why these operations result in target features or non-target features. Additionally, it is unclear whether the attention module based on energy segmentation is an original innovation or borrowed from other papers.

A1:The key operations for obtaining target and non-target features in our model involve techniques such as residual calculation, energy segmentation, and the use of interaction modules. Here, we provide a more detailed theoretical explanation for why these operations lead to the separation of target and non-target features:

1. Residual Calculation (for LM):

The extraction of non-target features relies on the concept of residuals, which is grounded in the back-projection principle. The core assumption is that the information filtered out by the network primarily corresponds to noise. By calculating the residuals between the input and output features, we isolate components that deviate from the target speech features, which are dominated by noise. The role of normalization layers after each convolution is to maintain the energy consistency between the input and the extracted residuals, ensuring that the non-target features are not distorted and are effectively separated as noise-like residuals. This operation is crucial for distinguishing non-target features from the desired target features.

2. Energy Distributions in Self-Attention Maps (for GM):

The second method uses energy distributions in self-attention maps to segment the feature space. This approach assumes that target features, which are speech-dominant, reside in high-energy regions of the feature map, while non-target features, primarily composed of noise, are located in low-energy regions. The energy-based segmentation allows the model to focus on regions of the feature space that contain noise-like elements, ensuring clean separation of target and non-target features. The uniform application of this segmentation approach across the same spectral representation prevents overlap between features and avoids the introduction of learnable parameters, preserving the assumption that non-target features are inherently noise-dominant.

Regarding your inquiry about the originality of the attention module based on energy segmentation, we would like to clarify that while the concept of energy-based segmentation shares some commonality with previous approaches [1], the specific implementation in our model is novel. The use of energy distributions in self-attention maps for segmenting feature space into regions of high and low energy is an adaptation tailored to our goal of separating target (speech) and non-target (noise) features. While energy-based segmentation techniques have been explored in other contexts, our approach integrates this concept within the framework of self-attention mechanisms, specifically designed to address the challenges of speech separation in noisy environments. This integration ensures an efficient separation of speech and noise without relying on additional learnable parameters, which distinguishes our method from prior work.

We have included a dedicated section in the revised Appendix A that analyzes the feasibility of the Recycling Process, offering a clearer theoretical explanation of how these operations contribute to the effective distinction between target and non-target features.

[1] Chen X, Li H, Li M, Pan J. Learning a sparse transformer network for effective image deraining. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2023 (pp. 5896-5905).

We would like to truly thank for your valuable time and dedicated efforts to provide us with these constructive feedbacks.

Q1: There are similar methods to this approach that also use cascaded strategies to enhance the model's ability to extract target speech, such as TaylorSENet [1]. However, these methods were not compared in the paper.

A1: TaylorSENet remains focused on modeling the target speech by iteratively refining the spectrogram generated at the initial stage by calculating residuals to enhance its quality. To compare the performance of FIRING-Net and TaylorSENet, we added experiments on the publicly available DNS-Challenge dataset. The comparative results are presented in Table 7 of our revised Appendix C.1.

Q2: For the experimental section, the model trained on the DNS Challenge training set should naturally be tested on the public DNS Challenge test set. However, the authors chose two other datasets, which is quite odd. I believe the authors should test the model on the DNS Challenge test set to better demonstrate the model's performance.

A2: We initially did not conduct experiments on the VoiceBank+DEMAND dataset but not the DNS Challenge testing set since the results of most of the state-of-the-arts approaches can be obtained on the former dataset. We have added the model comparison on the DNS Challenge test set in Table 7 of our revised Appendix C.1. The comparison plase see above Table.

Thank you so much for your valuable comments, which have been very helpful in improving our manuscript.

Q1: The main weakness is there seems lacking a clear explanation as why the proposed architecture, especially the Local Modules (LMs), can promote target speech and non-target noise features extraction in the way as claimed in the paper. To be more specific, for the proposed LM-E (and LM-D) in Figure 2, what is the actual mechanism or reason that leads to the model assigning speech to the target branch and noise to the non-target branch? Please provide more detailed explanations on how such network design promotes speech and noise separation to carry out the proposed feature recycling scheme. Could the authors explicitly associate the "Network for speech extraction" in Figure 1 to the corresponding module(s) or layer(s) in Figure 2? This will help understanding where the recycling scheme actually takes place. Similarly, association of the "Network for noise extraction" in Figure 1 to the place(s) in Figure 2 will be helpful.

A1: In LM, the extraction of non-target features through residuals is based on the principle of back-projection [1]. This method assumes that the information filtered out by the network primarily consists of dominant noise components, making it highly relevant for identifying non-target features. To ensure the effectiveness of this approach, normalization layers are applied after each convolutional module. These layers help preserve the energy range of the generated features, keeping them within the scale of the input features and preventing any significant deviation in overall output energy. This step is crucial, as uncontrolled variations in feature energy could distort the separation process, leading to undesired feature overlaps or loss of important information. By maintaining consistent energy distribution, the network can effectively isolate noise-like elements as residuals, which then serve as refined non-target features.

GM utilizes energy distribution patterns in self-attention maps to extract non-target features. The attention mechanism divides the feature space into regions based on energy levels. High-energy areas represent target features, which are speech-dominant, while low-energy regions mainly contain noise-like elements. By measuring cross-similarity between these energy-based segments, the method ensures a clear separation of target and non-target features, avoiding any overlap of information. A key aspect of this approach is the consistent application of energy segmentation and feature extraction across the same spectral representation, which prevents unintended mixing of features. Additionally, the method does not rely on extra learnable parameters, ensuring that the non-target features are inherently noise-dominant, in line with the assumption that noise resides in low-energy regions

We add the explanation of LM and GM for target and non-target feature extraction in Appendix A, while revising the content of Figure 2 and rewritten its caption to ensure that it aligns with the FRI strategy proposed in Figure 1.

[1] Haris M, Shakhnarovich G, Ukita N. Deep back-projection networks for super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition 2018 (pp. 1664-1673).

Q2: Another weakness of the paper is that the proposed SE model is tested on synthesized noisy speech data only. To better demonstrate the superiority of the proposed model, it will be nice if testing on real-world recorded noisy speech samples can be performed (e.g., using the realistic noisy samples from the CHiME-3 dataset: J. Barker et al., “The third ‘CHiME’ speech separation and recognition challenge: Analysis and outcomes,” Comput. Speech Lang, 2017). In this case, since there will be no ground-truth references to compute the objective metric scores used in the paper (i.e., PESQ, STOI, and SI-SNRi), the comparison with baselines may be done by using subjective listening testing or using automatic speech recognition (ASR) as evaluators.

A2: We have presented added a comparison in terms of the WER metrics for FIRING-Net and the baseline models, evaluated on the same ASR system after processing the CHiME-3 real recorded test set, in Figure 10 and Appendix C.3. The results are shown as follows:

Q3: In Figure 2, the LM-E (or LM-D) module has addition and subtraction operators. However, the subtraction operation may not lead to a different result from using addition if the feature value range is not constrained.

A3: Before the subtraction operation, feature normalization is consistently applied. In the revised manuscript, we have validated the impact of normalization in Appendix C.5. Additionally, we added ablation experiments in the same section to evaluate the effect of normalization and the choice between addition and subtraction operations for the LM module , further demonstrating the effectiveness of our approach. The results are shown as follows:

The result indicating that when the normalization operation is removed, i.e., the features are unconstrained, there is little performance difference between subtraction and addition. However, when normalization is applied, constraining the features, the performance of the subtraction operation is significantly superior to that of the addition operation.

I would like to thank the authors for providing detailed responses to my questions. My main concerns regarding LMs have been addressed by further clarification and added experimental results from the authors. The reviewer especially finds that the experiments on comparing addition and subtraction operations in the presence and absence of the normalization after the convolutional layer in LMs are very helpful (Figure 15 and Table 12). The revised manuscript has also improved the clarity and presentation of the main idea. Given the above, I increased my score from 5 to 6.

Suggestion: Although the ASR comparison in Appendix C.3 is helpful, I think it would be nice to provide further details regarding the chosen ASR model.

We sincerely appreciate you for considering that our work is novel and effective. We respond your comments as follows:

Q1: The paper's starting point is that "when noise and speech components are highly similar, it is difficult for the SE model to learn effective discrimination patterns." However, this aspect of the contribution is not prominently mentioned in the subsequent methodology analysis and experiments.

A1: When background noise contains speech from other speakers or exhibits high energy that masks certain frequency bands of the speech signal, the inherent characteristics of speech may fail to distinguish themselves from noise. This often results in high similarity between speech and noise components, leading to speech distortion and residual noise [1]. To address this issue, this paper introduces the Filter-Recycle-Interguide (FRI) strategy, which recycles the information filtered by the network module and facilitates mutual guidance with the module's output. The recycled filtered information and the output are primarily dominated by noise and speech features, respectively, enabling mutual guidance to enhance their discriminability. This approach effectively increases the separability of noise and speech components from both feature perspectives, mitigating the issues of distortion and residual noise. In the experimental section, we evaluate the proposed FIRING-Net under challenging noise conditions using the AVSpeech + Audioset test set. Specifically, Babble and Baby Crying and Laughter noises are selected to test its performance under speech-interfering background noise, while Traffic and Engine noises are chosen to assess scenarios where high-intensity masking occurs in certain frequency bands of the target speech. The results presented in Table 2 demonstrate the superior performance of FIRING-Net under these noise conditions, further validating its effectiveness in addressing the aforementioned problem. To better illustrate this point, we have added the visualization of spectrograms, as shown in Figures 8 and 9, Appendix B.3, showing the enhanced speech under -5 dB conditions for CMGAN, MP-SENet, and FIRING-Net in babble noise and factory noise environments, to provide a more detailed comparison of its performance with other models under conditions where noise and speech components are highly similar.

Q2: In the field of speech enhancement, there have been some related studies on refining and interacting features that were not referenced in the paper. Therefore, the selection of baseline methods for comparison in the experiments also lacks specificity. There is a lack of qualitative analysis of the method and experiments, which results in a lack of interpretability and fails to distinguish the proposed method from existing research and the baseline methods used in the comparative experiments.

A2: There exist some related studies on interacting features, such as PHASEN [2], SN-Net [1] and Inter-SubNet [3]. These approaches have been adopted as the baseline methods, and our method differs from these approaches in that: 1) PHASEN is a model that employs an interactive framework for jointly modeling magnitude and phase information; 2) SN-Net adopts a dual-branch architecture to directly model the interaction between speech and noise features; 3) Inter-SubNet leverages subband interaction to explore cross-band dependencies; whereas CMGAN and MP-SENet, despite being state-of-the-art speech enhancement models, do not incorporate interactive feature modeling. FIRING-Net introduces a novel interaction mechanism between a network module's output and the residuals between its input and output. Compared to PHASEN, the features involved in FIRING-Net's interactions exhibit stronger inherent correlations than the magnitude and phase features used in PHASEN. In contrast to SN-Net, FIRING-Net effectively addresses the challenge of modeling noise features, which SN-Net fails to resolve. Furthermore, compared to InterSub-Net, FIRING-Net achieves a more comprehensive interaction by integrating both local and global feature perspectives. To more clearly present the experimental conclusions, we have revised and integrated this content, along with more detailed experimental analyses, into Section 5.1 of the main text.

[1] Zheng C, Peng X, Zhang Y, Srinivasan S, Lu Y. Interactive speech and noise modeling for speech enhancement. In Proceedings of the AAAI Conference on Artificial Intelligence 2021 May 18 (Vol. 35, No. 16, pp. 14549-14557).

[2] Yin D, Luo C, Xiong Z, Zeng W. Phasen: A phase-and-harmonics-aware speech enhancement network. InProceedings of the AAAI Conference on Artificial Intelligence 2020 Apr 3 (Vol. 34, No. 05, pp. 9458-9465).

[3] Chen J, Rao W, Wang Z, Lin J, Wu Z, Wang Y, Shang S, Meng H. Inter-subnet: Speech enhancement with subband interaction. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2023 Jun 4 (pp. 1-5). IEEE.

We would like to thank you for your efforts in reviewing our manuscript, and providing many helpful comments and suggestions, which will all prove invaluable in the revision and improvement of our paper.

Q1: I am not completely sure that noise types used in training are not also seen in testing. What is clear is that SNR levels are same in training and testing cases. I also noticed that SNR scale is pretty extreme (from -5 to 5 db). How about using more moderate noise levels? Would the differences to competing models then disappear?

A1: We confirm that the noise types used in the testing set of the AVSpeech+AudioSet dataset are not present in the training set. During training, we follow the existing setting [1] to draw noise samples from Audioset, DEMAND, and Freesound, and use AVSpeech + Audioset [2] for testing. The two common noise types “Babble” and “Traffic” that we have used in training, and the other two types “Engine” and “Baby Cry and Laughter” have not been observed. From the results in Table 2, we can see that our model can well generalize to new noise types. Regarding the generalization to unseen SNR levels, we have conducted additional experiments on the WSJ0-SI 84 + NOISEX-92 dataset at an unseen SNR level of -10 dB in two challenge noise environments, i.e., “Babble” and “Factory”. The results have been reported in Table 8 of our revised Appendix C.2, which are summarized in the table below:

and demonstrate that FIRING-Net maintains competitive performance even in such extreme conditions, highlighting its robustness under challenging scenarios. Moreover, we have demonstrated our model’s performance under more complex noise conditions, in Figure 6. This figure illustrates the performance of FIRING-Net in scenarios involving multiple noise types, indicating a more challenging and realistic test of the model’s generalization capabilities.

Q2: All ideas seem to be gathered from previous publications (like CV papers). It would be good to emphasize the novel technical contributions in the Introduction.

A2: We acknowledge that some inspirations for FIRING-Net, such as back-projection and Top-K mechanisms, originate from existing works, particularly from computer vision studies. However, one contribution of our FIRING-Net lies in its integration and tailored application of these ideas to the unique challenges of speech enhancement. The main contribution is that we propose a Filtering and Recycling (FRI) strategy, which is novel in leveraging the interaction between retained (target) and filtered-out (non-target) features. This strategy is specifically designed to address the unpredictable overlap between noise and speech, ensuring mutual refinement that enhances distinguishability. Moreover, the two-stage coarse-to-fine architecture in both local module (LM) and global module (GM) is a unique contribution that enables precise separation of target and non-target features. FIRING-Net further innovates by adapting interaction modules (IMs) differently in the encoder and decoder to accommodate the distinct feature compositions of these stages, enabling a robust process for noise elimination and speech reconstruction. These innovations are specifically crafted for speech enhancement, going beyond the direct adaptation of existing methods, and represent significant technical contributions to the field. We emphasized these points more clearly in the revised Introduction to highlight the originality of our approach.

[1] Reddy CK, Dubey H, Gopal V, Cutler R, Braun S, Gamper H, Aichner R, Srinivasan S. ICASSP 2021 deep noise suppression challenge. InICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2021 Jun 6 (pp. 6623-6627). IEEE.

[2] Ephrat A, Mosseri I, Lang O, Dekel T, Wilson K, Hassidim A, Freeman WT, Rubinstein M. Looking to listen at the cocktail party. ACM Transactions on Graphics. 2018 Jul 30;37(4):1-1.