RTFS-Net: Recurrent Time-Frequency Modelling for Efficient Audio-Visual Speech Separation

We appreciate the effort you've invested in reviewing our manuscript and providing detailed feedback. Your insightful suggestions have been incredibly valuable, and we're confident that by addressing the points you've raised, we can enhance the quality of our paper. Below, we've addressed each of the concerns you've highlighted in your review:

Q1: As the authors mention, the dual-path RNN idea is not new to this problem. I understand that the authors chose SRU for their complexity-related considerations, but I also wonder if the audio processing module could benefit from its own self-attention mechanism, such as in the SepFormer model.

A1: We experimented with different sequence processing architectures in Appendix D, which has now been updated to include several transformer designs. The self-attention mechanism leads to a decrease of 1 dB SI-SNRi, while significantly increasing the memory requirements of the model. Other transformer approaches resulted in extremely high parameter counts, which contradicts our aim of constructing a lightweight AVSS model.

Q2: The spectral source separation module might be the weakest contribution, because complex masks have been extensively studied in the audio-only source separation literature.

A2: Some research has been conducted in the AOSS field using complex masks, but such studies are relatively scarce in the context of audio-visual speech separation. Additionally, to our knowledge, current state-of-the-art audio-only source separation methods, such as Sepformer, still employ real-valued masks. Indeed, TF-GridNet, the current AOSS SOTA method, does not use masks at all and directly computes the separated sources. We provided a study on all these different methods in Table 4. It is important to note that results in AOSS do not necessarily translate to the AVSS field.

Q3: I wish the paper provides more details on the TDANet block for video processing, which is relegated to the reference in the current version.

A3: We have included an explanation of the video preprocessing block (VP BLOCK) in Appendix A.

Q4: The authors chose to "add" $\mathbf{f}_1$ and $\mathbf{f}_2$ (eq 11) after the CAF processing. I think it's a little abrupt in the sense that there might be other choices that preserve the unique information that each vector learns, such as concatenation. Have the authors considered other ways to combine the two vectors?

A4: In Table 2 of our paper we compare the CAF block with pure concatenation (CTCNet Fusion (adapted)) of the audio and visual features. We observe that despite our CAF Block using only 3.6% ofthe parameters and 1.3% of the MACs, it outperformed pure concatenation by a large margin.

The purpose for the CAF Block is to produce a low-parameter and MAC alternative to CTCNet’s fusion approach. If we directly concatenated $\mathbf{f}_1$ and $\mathbf{f}_2$, and then downscaled to the correct channel dimension using a convolution, this would be the same parameter and MAC count as the pure concatenation discussed above and in Table 2 of our paper. However, in order to fully answer your question, we ran an experiment where we concatenated $\mathbf{f}_1$ and $\mathbf{f}_2$ and then downscaled using a group convolution. Using a group convolution does not avoid the memory increase that comes from the concatenation, but it does avoid increasing the parameter and MAC counts significantly. The results show that this is an ineffective operation, and that a simple summation should be preferred.

Thank you very much for your valuable feedback. We have revised the manuscript and added some experiments to demonstrate the superiority of our model. We would also like to make some comments below to address your specific questions:

Q1: The baseline methods listed in table 1 should include their references.

A1: Thank you for spotting that, we have updated the table.

Q2: RTFS-Net-12 is only about 10% more efficient that CTCNet. How much difference does that make in practical applications?

A2: From the 10% figure we assume that the efficiency you mention is RTFS-Net's inference time of 109.9ms vs CTCNet's inference time of 122.7ms. The inference time is the time taken to process 2 seconds of audio. Scaling up to 1000 hours of audio, it would take 54.95 hours for RTFS-Net-12 and 61.35 hours for CTCNet. Since the separation performance of RTFS-Net is higher, and the time taken is shorter, the logical solution would be to use RTFS-Net as it would save around 6 and a half hours of GPU processing, and the associated electricity cost. Further, the performance drop from switching to RTFS-Net-4 is pretty minimal. Using this model, 1000 hours of audio would only take 28.9 hours.

The total parameters and MACs are also important factors to be considered in practical applications. Compared to CTCNet, RTFS-Net-12 has reduced computational costs by three times, while the number of parameters utilized is only one-tenth.

Q3: Some of the comparison examples are not distinguishable to this reviewer. This makes me wonder how to interpret the relative SNR gains.

A3: You are correct. The purpose of the demo, and indeed this paper, is to show that SOTA performance can be replicated using a fraction of the parameters and a greatly reduced model complexity. For example, in Appendix C we showed that if we do not compress the frequency and time resolutions of the audio features (i.e. q=1), we can outperform the SOTA method by a significant margin using only R=4 repetitions. Using 16 or more layers with q=1 would greatly increase the SNR further. However, the large memory requirement and computational complexity associated with this approach defeats the purpose of our paper.

Q4: Since the best performance is achieved with R=12, why not explore a higher R?

A4: A higher value of R leads to increased computational demands, which contradicts our intention to propose a lightweight model. Nonetheless, we still present a result as shown below. The results indicate that a higher R achieves better separation quality, but it comes with a more significant computational cost. Note that because we share parameters, increasing R only increases the MACs and not the parameter count.

RTFS-Net with 16 and 20 layers on LRS2 dataset

RTFS-Net-20 outperforms CTCNet by over 1dB in both SDRi and SI-SNRi, while still utilizing only 91G MACs – just over half the MACs utilized by CTCNet. However, 20 RTFS Blocks requires a much larger GPU memory for training. As a result, we had to use NVIDIA 3090/4090s in order to obtain these results in a short period of time.

Q5: Have the authors considered conducting studies with human listeners? If the target application is ASR, would it be helpful to measure WER in a recognition task?

A5: We tested the Automatic Speech Recognition (ASR) results on the LRS2 test set. The results show that RTFS-Net generalizes well to downstream tasks, achieving almost identical performance to CTCNet – the previous SOTA method. For details, please refer to our response to the third question from Reviewer #1FXL.

We have shared a demo in the supplementary material that can be listened to at your leisure, and we have shared our results with other members of our lab. However, conducting a formal study with large quantities of human participants or professional stenographers lies beyond the scope of our current research and is not feasible within the limited timeframe given for responding to reviews and questions.

We would like to thank you for your very positive feedback. We are honored that you have found potential in our work. Please find below our answers to the main questions that you have raised:

Q1: Testing is limited to only 2 speaker mixtures. Performance with more speakers is uncertain.

A1: We tested our model on two-speaker mixtures because most existing methods are verified in this manner. Furthermore, it is important to note that our model incorporates visual information from a single speaker to isolate and extract the audio of this target speaker. The integration of visual cues plays a crucial role in enhancing the accuracy of our model, as well as other AVSS models. Hence, we are not limited by the number of speakers.

We randomly sampled speakers from the LRS2-2Mix test set to obtain a dataset with three speakers. The results indicate that the presence of multiple speakers somewhat impacts the model's performance. However, it is important to note that these models were not trained on the dataset with three speakers, only tested on the dataset with three speakers. In addition, RTFS-Net-12 outperformed CTCNet by a significant margin, scaling much better to the harder task.

Q2: Missing PESQ evaluation in results which most other target speech extraction papers provide.

A2: We have added the PESQ evaluation metrics to Table 1 in the updated paper. As with SI-SNRi and SDRi, the PESQ scores show that RTFS-Net obtains comparable accuracy to the previous SOTA method (CTCNet) across all three datasets, while utilizing fewer parameters and MACs.

Q3: Doesn't include latest SOTA model comparison: Dual-Path Cross-Modal Attention for Better Audio-Visual Speech Extraction, ICASSP 2023. https://arxiv.org/pdf/2207.04213.pdf. This gives superior performance for SI-SNRi and provides PESQ results as well. It does not provide MACs analysis.

A3: The model code for this ICASSP publication is not open source. Its results on the LRS3 dataset for the model “AV-TASNET” are different from ours, indicating that either their mixing method is different, or that their train/eval/test splits might be different. For similar reasons, we also did not include AV-Sepformer (https://arxiv.org/abs/2306.14170) in our paper. Both of these methods are Sepformer-like, meaning the parameter count for each model is at least 20 million. However, by comparing our paper with AV-Sepformer’s PESQ values, this method also does not outperform CTCNet or RTFS-Net (AV-Sepformer: 2.31, CTCNet: 3.00, RTFS-Net: 3.00).

We would also like to emphasize that the purpose of RTFS-Net is not to obtain SOTA performance. As can now be seen in the updated Appendix C, we can easily outperform CTCNet by directly applying TF-GridNet to the audio-visual setting with only 4 RTFS-Net block layers. The focus of RTFS-Net is to obtain comparable performance to the SOTA method using a very small model footprint. Our contribution is a fast, efficient and quick-to-train model.

We would like to thank you for the time taken to review our paper and for your extensive comments. Your suggestions are very helpful, and we believe we can significantly improve the paper by making the respective adjustments. We have compiled the issues that you have pointed out and answered them as follows:

Q1: Explain the benefits of shared parameters.

A1: The purpose of sharing parameters between the RTFS Block layers is to create an overall network architecture that uses recurrent connections to continuously refine the clarity of the model’s output through the R “time steps”, just like an RNN. Additionally, this approach also reduces the quantity of model parameters, creating a more lightweight design. A more detailed discussion can be found in Appendix B on the updated paper. From the results in Appendix B (copied below), we can see that sharing the parameters reduces the model size from 2 million parameters to 739,000 parameters, while also delivering a slight performance improvement.

Q2: Is there any overlapping speech in the train/test data?

A2: In our approach, we did not impose any restrictions on the extent of overlap in the speech samples within the training and test sets. As a result, the data in both sets encompasses a diverse range of overlaps, including heavily overlapped, moderately overlapped, and non-overlapping speech. This variety ensures a comprehensive evaluation of the model's performance across different degrees of speech overlap.

Q3:Downstream task.

A3: The speech separation task is a front-end task generally used to enhance speech recognition accuracy. To address your question, we tested the speech recognition accuracy on the LRS2-2Mix test dataset. We utilized the publicly available Google Speech-to-Text API to obtain the recognition results. Our focus was on measuring the Word Error Rate (WER), where a lower rate is indicative of superior performance. The experimental results demonstrate that our model can achieve competitive speech recognition accuracy, validating the performance of our model in downstream tasks. However, the task of separating background noise, also referred to as universal voice separation, was not investigated in this paper as it falls outside the scope of our current research. Our work, and all previous works that we compare with in our paper, are specifically designed for target speaker extraction – using the lip movements of a target speaker to extract that speaker’s audio signal. As there are no lip movements associated with background noise, implementing new modules, or modifying the architecture would be necessary in order to address the task of speech-to-speech dubbing. However, it represents an intriguing task for future research.