CM^2: Cross-Modal Contextual Modeling for Audio-Visual Speech Enhancement

a) Firstly, while both RTFS-Net and our method utilize pre-trained models to extract visual features, the ways we use them, the methods by which we obtain these features, and our objectives all vary substantially. RTFS-Net hopes to extract target speaker information from visual lip movements with the pre-trained lipreading network. Conversely, our approach integrates the multi-modal semantic information from both noisy audio and visual inputs by the self-supervised audio-visual speech model (AV-HuBERT), which is used to guide and enhance the subsequent construction of signal-level context. Utilizing AVHuBERT or similar models to encode visual information is common in many AVSE/AVSS applications. For instance, AV-Gen in 2023 (https://arxiv.org/abs/2306.01432) employs AVHuBERT to derive visual embeddings, where the visual features of each layer in AVHUBERT would be weighted to sum to inform an audio-based Score Modal for AVSE; SSL-AVSE in 2023 (https://arxiv.org/abs/2307.07748) leverages AVHuBERT as a front-end encoder for audio-visual feature encoding, where the audio-visual features in each layer are weighted to sum to feed a decoder for AVSE. These implementations are distinctly different from ours in both purpose and the subsequent manner to use the obtained features.

b) Secondly, as detailed in Section 3.2, our audio encoder consists of a validated structure comprising four convolutional blocks. This configuration was determined through our experiments and is independent of the audio encoder used in RTFS-Net.

c) Thirdly, concerning the modal fusion module, we highlight three key differences between our CM$^2$ module and RTFS-Net:

  i) Overall, our CCFM is not a typical modal fusion module. CCFM is a context fusion module, as detailed in Sections 1 and 3.4, differing significantly from RTFS-Net’s CAF Block in both target and specific design. CCFM focus on using semantic context to guide signal context modeling. Initially, it upsamples the multi-modal semantic context, and then combines it with the signal context at the channel level. This is followed by a fine-grained fusion process, implemented in an attention-like manner, to enhance integration. This approach is totally different from RTFS-Net’s method of Gated Fusion for audio-visual features, as evidenced by the distinct methodologies depicted in our Figure 3 compared to RTFS-Net Figure 2.

  ii) Moreover, we emphasize the impact of speaker’s visual appearance on the audio frequency domain. As mentioned in the previous point 2, most TF-domain AVSS / AVSE methods, including RTFS-Net, focus only on fusing audio-visual features with the same dimension. These methods typically upscale the visual features in temporal domain and then repeat them across the frequency dimension, with the target of just obtaining the same dimension as audio features in the TF-domain. In contrast, we intentionally introduce the learnable upsample to the audio-visual semantic context features in both time and frequency dimensions. Then, in addition to the fusion module on the time dimension, a frequency dimension fusion module is applied for deeper integration of the two types of context. Table 6 demonstrates the effectiveness of this design.
  
  iii) The channel operations in our CCFM's Channel Swapping Block (CSBlock) differ significantly from those in RTFS-Net. RTFS-Net's operations are concentrated on the real and imaginary components of the audio spectrum, assigning the first half channel dimensions of audio features to the real part and the second half to the imaginary, then employing a complex multiplication-like method for spectral output. However, as detailed in Section 3.4, our CSBlock operates under our premise that different channel information highlights different aspects, and redundant information in a channel of one modality might be complementary for the other modality. Hence, we swap half the channels between two modalities to facilitate preliminary fusion. This method is distinct from RTFS-Net.

d) In discussing Time-Frequency (TF) domain modeling, it is important to note that modeling time and frequency separately is a standard practice in SE and SS tasks, as seen in TF-Gridnet (2022-9-8 released, ICASSP 2023) and RTFS-Net(2023-9-29 released, ICLR 2024), and earlier in CMGAN (2022-3-28 released, Interspeech 2022, which is already cited in our paper) (https://arxiv.org/abs/2203.15149). Moreover, the TF domain modeling itself is not among our claimed contributions. Instead, we emphasize the significant role of the visual modality in restoring audio frequency domain characteristics, as detailed previously in point 2.

e) Our decoders are implemented based on CMGAN and are totally unrelated to RTFS-Net or TF-Gridnet.

f) We implemented a discriminator inspired by CMGAN and Metric GAN, transforming the non-differentiable metric PESQ into a training target. Neither RTFS-Net nor TF-Gridnet employs a similar approach.

For the second question, you mentioned that the operation in the CSBlock was first introduced in DPRNN (2019) and subsequently used in DPTNet (2020), TF-Gridnet (2022), and RTFS-Net (2023). We have conducted extensive research and identified a significant misunderstanding on the part of the reviewer. As detailed in Section 3.4 and previously clarified, our CSBlock is specifically responsible for the preliminary fusion of two different levels of the context. This contrasts sharply with the single-modality models DPRNN, DPTNet, and TF-Gridnet mentioned by the reviewer. This fusion is based on our channel redundancy premise among different modalities as explained previously, and we did not identified any similar ideas or operations in these methods. The only possible similarity to RTFS-Net seems to be the point that both methods involve operations on channel dimensions; however, as we have already clarified as above, their approach to channel operations is different from ours. Regarding the misunderstanding, we suspect that the reviewer may have confused our channel swapping operation with the segmentation operation introduced in DPRNN. This segmentation technique divides long input sequences into shorter chunks, which are then concatenated to form a 3-D tensor. Subsequently, two separate RNNs process these sequences along two dimensions of the tensor, a method depicted in DPRNN Figure 1. Similar approaches have been adopted in subsequent mentioned works like DPTNet, TF-Gridnet, and RTFS-Net. However, our CM$^2$ module does not utilize this segmentation method.

Finally, we appreciate your valuable feedback on our work. We hope the explanations provided could resolve any misunderstandings. Should you have any further inquiries or require additional clarification, please do not hesitate to contact us. We are fully willing to address any further concerns you might have and look forward to your response. Additionally, we kindly request you revising the wording of the original review to remove some hurtful wordings such as "plagiarism".

Dear Reviewer 6pDV,

Thank you for your recognition, suggestions, and evaluation of our work. Regarding the research integrity issues you raised, we must clarify that the contributions we claimed are entirely different from those in the mentioned works like RTFS-Net.

Firstly, you mentioned that our Figure 2 looks similar to RTFS-Net and there is a mount of overlap between the contributions of our paper and the contributions in RTFS-Net and TF-Gridnet. We believe this is a misunderstanding. In AVSE / AVSS, the pipeline of “audio / visual encoders → audio-visual fusion module → (Time-Frequency) enhancer/separator → decoder” is quite common, as seen in OVA in 2020 (https://ieeexplore.ieee.org/document/9053033), Muse in 2021 (https://arxiv.org/abs/2010.07775), and VisualVoice in 2021 (https://arxiv.org/abs/2101.03149). This commonality in the overall pipeline frameworks may cause superficial similarities between different works, especially when using relatively generalized diagrams for overview representation. However, this process itself is common, and also not a contribution we claimed. Considering the generic nature of this pipeline, it is improper to deem our work as potentially plagiaristic which is such a hurtful term. Moreover, the issues addressed by RTFS-Net and TF-Gridnet are entirely different from our paper (speech separation aims to separate different speakers, while speech enhancement aims to enhance target speech from noisy input). Besides this point:

1. The first contribution we claimed is that we propose a new perspective to solve AVSE. This perspective draws inspiration from cognitive studies on human auditory processing in noisy environments. Drawing inspiration from how humans use semantic insights to compensate for noise-affected segments, we align this approach with the objective of AVSE—to enhance the output speech signals. Based on this general idea, we extract semantic-level information at first, which then guides the subsequent modeling of speech at the signal level so as to lead to an enhanced, high-quality speech signal at the end. This general idea has not been previously proposed. RTFS-Net's main contribution is a lightweight model that performs AVSS in the time-frequency domain with minimal resource consumption, achieving impressive performance. After carefully reviewing the RTFS-Net，we acknowledge it as an important and influential work in the field of speech separation. We are also pleased to include RTFS-Net and other significant speech separation studies mentioned in your comment. However, we must clarify that our contributions are entirely distinct from these works.
2. Our second contribution specifically emphasizes the importance of visual information for modeling the frequency characteristics of audio information—an aspect previously neglected in works like RTFS-Net and other AVSE/AVSS approaches. Earlier time-frequency domain AVSE/AVSS methods, including RTFS-Net, focused solely on aligning audio-visual features in the time dimension. They typically repeated temporal visual features along the audio frequency dimension merely to make the visual feature’s dimension equal to audio feature’s. In contrast, we noticed the role of visual information for recovering audio signals and intentionally introduced simple learnable upsampling of the audio-visual semantic context along the frequency dimension. This simple learnable manner aims to maximize the influence of visual data on every frequency bins of the audio spectrum.
3. Finally, before we discuss the specific module comparisons, it is important to note that our development process was guided by a new perspective and particular objectives. Each module in our system is introduced with unique objectives, distinct from RTFS-Net or any other systems.