Continual Audio-Visual Sound Separation

Thank you for the valuable comments! We appreciate the reviewer highlighting the clear structure, fluent expression, and effective experimental results of our paper. We address the raised concerns below and are willing to answer any further questions.

Q1: Difference between the innovation of the Cross-modal Similarity Distillation Constraint (CrossSDC) and Dual-Audio-Visual Similarity Constraint (D-AVSC) in [1].

Thank you for your question! The Dual-Audio-Visual Similarity Constraint (D-AVSC) [1] only focuses on the feature similarities within data generated by the current training model, while our proposed CrossSDC further integrate features learned by previous model into the contrastive loss. In this way, our CrossSDC incorporates the cross-modal similarities knowledge acquired from previous tasks into the contrastive loss, which not only facilitates the learning of cross-modal semantic similarities in new tasks but also ensures the preservation of previously acquired knowledge in audio-visual sound separation.

Moreover, the experimental results also demonstrate the superiority of our method with CrossSDC (7.33/13.55/13.01 for SDR/SIR/SAR) compared to the AV-CIL with D-AVSC [1] (6.86/13.13/12.31 for SDR/SIR/SAR).

Q2: Generalization ability on other datasets.

Thanks for the suggestion! Besides the MUSIC-21 dataset, we conducted experiments on the AVE [2] dataset and have included the experimental results in Appendix. The results of Fine-tuning/LwF/PLOP/AV-CIL/ContAV-Sep(Ours) are 2.07/2.19/2.45/2.53/2.72 and 5.64/6.43/6.11/6.64/7.32 for SDR and SIR, respectively, which further demonstrates the effectiveness of our proposed method. For more details, please kindly see Section A.3 in Appendix.

Q3: Is the significant performance gap between ContAV-Sep and the fine-tuning method due to the small data scale and limited number of categories?

We would like to clarify that the significant performance gap between ContAV-Sep and fine-tuning is not due to the small data scale and limited number of categories. Experiments on a larger dataset, AVE [2], demonstrate a similar performance gap. On AVE, fine-tuning achieves an SDR of 2.07 and SIR of 5.64, while ContAV-Sep achieves 2.72 and 7.32, respectively. This persistent gap underscores that the difference is not solely due to the dataset. For further details, please refer to Section A.3 in the Appendix.

Q4: Whether this paper has been submitted twice?

We confirm that this paper is not a dual submission. The current version is the revised submission of our previously withdrawn paper. We apologize for the oversight in the teaser of the demo video, which contained a typo. This issue has been corrected in our demo.

[1] Audio-Visual Class-Incremental Learning. In ICCV 2023.

[2] Audio-Visual Event Localization in Unconstrained Videos. In ECCV 2018.

Thank you for your valuable comments and suggestions! We appreciate your recognition of the novelty in our work's problem formulation and approach, as well as its effectiveness demonstrated in our experiments. We address your questions below and welcome any further inquiries.

Q1: Replace the "mask" variable.

Thank you for your suggestion. We have replaced the "mask" variable with a variable m.

Q2: Weird fontsizes like the one in equation 9.

Thank you for your suggestion. We have fixed this fontsize issue.

Q3: Rounding up all performance numbers to a one decimal precision.

Nice suggestion. We have updated performance numbers to one decimal precision in our paper.

Q4: Larger amount of samples per class in memory. A graph would make the visualization better.

Thank you for your suggestion! We conducted experiments with the number of samples per class in memory with 10, 20, and 30. The results are shown below. Moreover, as per your advice, we have included a graph in our paper to better show the impact of different sample sizes per class in memory.

Q5: Missing references in continual learning for sound processing tasks.

Thanks for the suggestion! We have added relevant references on continual learning for sound processing tasks, including the two suggested works and others [1, 2, 3, 4], and discussed them in our revised paper:

[1] Ma, Haoxin, Jiangyan Yi, Jianhua Tao, Ye Bai, Zhengkun Tian, and Chenglong Wang. "Continual learning for fake audio detection." arXiv preprint arXiv:2104.07286 (2021).

[2] Bhatt, Ruchi, Pratibha Kumari, Dwarikanath Mahapatra, Abdulmotaleb El Saddik, and Mukesh Saini. "Characterizing Continual Learning Scenarios and Strategies for Audio Analysis." arXiv preprint arXiv:2407.00465 (2024).

[3] Wang, Yu, Nicholas J. Bryan, Mark Cartwright, Juan Pablo Bello, and Justin Salamon. "Few-shot continual learning for audio classification." In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 321-325. IEEE, 2021.

[4] Zhang, Xiaohui, Jiangyan Yi, Jianhua Tao, Chenglong Wang, and Chu Yuan Zhang. "Do you remember? overcoming catastrophic forgetting for fake audio detection." In International Conference on Machine Learning, pp. 41819-41831. PMLR, 2023.

Q6: Generalize the experiments to a larger dataset like AudioSet.

Nice suggestion! Following common practice, we used the MUSIC dataset as our primary benchmark and included an additional dataset, AVE, in the appendix to validate audio-visual sound separation performance. Extending our approach to very large datasets like AudioSet would indeed be interesting. However, due to limited computational resources and time constraints, conducting extensive experiments on AudioSet is currently challenging. We will investigate this in our future work.

Q7: Identify the memorization of individual user’s data inside the memory.

Thank you for your suggestion. We will address the potential memorization of individual users' data within our continual learning method in the limitations section. This discussion will include the importance of future work to empirically analyze and prevent data memorization, ensuring the development of robust and privacy-preserving models.

We appreciate the reviewer highlighting our proposed approach and writing. We address the raised questions below and are happy to answer further questions.

Q1: The performance improvement of ContAV-Sep (with iQuery) is not significant.

Thanks for the comment! We would like to clarify that the performance improvement of our ContAV-Sep (with iQuery) is non-trivial. Compared to the baseline continual learning methods LwF/EWC/PLOP/EWF/AV-CIL, our proposed ContAV-Sep shows improvements of 0.57/0.68/0.30/1.98/0.47 in SDR and 0.78/0.54/0.25/2.20/0.42 in SIR, respectively. Furthermore, as demonstrated in the demo included in our supplementary material, our ContAV-Sep (with iQuery) achieves better separation results and sound quality compared to the baselines, highlighting its effectiveness in mitigating catastrophic forgetting.

Q2: What does iSTFT stand for in Figure 2?

Thank you for your question. iSTFT stands for Inverse Short-Time Fourier Transform, a common technique used to convert spectrograms back into audio signals. We have added this explanation to the figure caption for clarity.

Thank you for your valuable comments and encouraging remarks! We address your questions below. If there are any additional questions, we are happy to address them and revise our paper.

Q1: Direct comparison between sound separation.

Thank you for your comment! Indeed, our paper focuses on developing continual learning methods to address catastrophic forgetting in audio-visual sound separation, rather than introducing new separation models. Our research demonstrates that existing audio-visual sound separation models suffer from catastrophic forgetting. We explored two separators, including the state-of-the-art iQuery model, and found that without advanced continual learning mechanisms, performance degrades significantly (e.g., SDR drops to 3.46, while our method achieves 7.33).

Therefore, we believe a fair comparison involves applying various continual learning methods to the same separator model, as directly comparing different models without continual learning would result in significantly lower separation performance. This approach allows us to isolate the impact of continual learning techniques on the task. In future work, we will also explore developing a more general and robust audio-visual sound separation model architecture that inherently mitigates catastrophic forgetting.

Q2: Figure 2 is too small.

Thanks for the suggestion! We have found more space for Figure 2 and enlarged it, by moving the dataset details into the Appendix.

Q3: More detailed contrast to existing work in related work.

Thank you for your suggestion! We have added more details of existing works in the related work, such as "...Park et al. [47] extend the knowledge distillation-based [2, 15] continual image classification method to the domain of video by proposing the time-channel distillation constraint. Douillard et al. [14] proposed to tackle the continual semantic segmentation task with multi-view feature distillation and pseudo-labeling...". Due to space constraints, we cannot include the entire section here.

Q4: All tasks have distinct classes. Is it a special case? Do you have an intuition or looked into variations where actually a smaller part of instances from a specific class is part of all tasks?

No, this is not a special case. In continual learning, models are typically trained on a sequence of new tasks with distinct classes. This setup is standard for evaluating a method's ability to tackle catastrophic forgetting.

In scenarios where a smaller subset of instances from a specific class appears across all tasks, the forgetting issue for that specific class would be minimal or nonexistent. However, our proposed methods could still be applied to mitigate catastrophic forgetting in other classes and leverage knowledge from previous tasks to enhance fine-tuning on the shared classes in new tasks.

Q5: Are there any compromises due to the hardware limitation?

Great question! We did not compromise the original data quality, such as image resolution or compression. Since the video encoder, object detector, and image encoder are frozen, we can pre-extract these features offline and use them to train the subsequent parts of the model, which allows us to remove the computational load of these three frozen components from the GPU, enabling the training process on a single RTX A5000 GPU with 24GB of memory.

Q6: What is the memory size? How does the memory and computation scale to new tasks as the memory keeps growing beyond 4 tasks?

Great question! The memory size is set to 1 sample per old class. By keeping only 1 sample for each old class in memory, it becomes easy to scale to new tasks.

Q7: Why choose 11kHz as the audio sampling rate?

We choose 11kHz as the audio sampling rate as it is a common setting for the MUSIC-21 dataset in existing audio-visual sound separation papers [1,2,3].

Q8: How does this scale to hard examples, where both instruments are the same? How much the approach is able to derive from movement and how much from appearance in the visual modality?

In our model architecture iQuery [1], it consists of a spatial-temporal video encoder to extract the motion information from the given video. This allows the model to differentiate between instruments based on their motion patterns, even when they are from the same category and have similar audio characteristics.

In hard cases where instruments have a similar appearance, the separation capability primarily relies on motion cues from the visual modality. However, when instruments have distinct appearances, the visual modality's appearance features can provide sufficient visual cues to effectively guide sound separation.

Q9: How does Figure 3 scale to n-tasks? One chosen class after following tasks.

Nice suggestion! We randomly select one class ("accordion") from the first task and report its performance after training for each task. The results are shown in the following table (please see the following Official Comment with the title of "Results table of question Q9: How does Figure 3 scale to n-tasks? One chosen class after following tasks"), in which we can see that our method has an overall better performance compared to baselines. We can also observe that with incremental step increases, the performance of each method on this class tends to improve. This is because the memory data can be paired with data from previously unseen new classes to acquire new knowledge for old classes in subsequent tasks, as discussed in our paper. However, the performance drop compared to the upper bound still exists, demonstrating that catastrophic forgetting still occurs.

[1] iQuery: Instruments as Queries for Audio-Visual Sound Separation. In CVPR 2023.

[2] The Sound of Pixels. In ECCV 2018.

[3] Co-Separating Sounds of Visual Objects. In ICCV 2019.

Thank you for your valuable comments and suggestions! We address the raised concerns below. If there are any additional questions, we are willing to address them and revise our paper.

Q1: The font size in Figure 2 is too small.

Thank you for your suggestion! We have enlarged the font size in Figure 2.

Q2: Same architecture as iQuery.

In our work, we address the proposed continual audio-visual sound separation problem, focusing on the challenge of training audio-visual sound separation models continuously while mitigating the catastrophic forgetting typically associated with continual learning. To benchmark and fairly compare with existing continual learning approaches, we do not introduce new separation architectures. Instead, we concentrate on developing new learning approaches based on current state-of-the-art separators, i.e., iQuery [1], to tackle the catastrophic forgetting problem in separation, aligning with similar research goals in the broader continual learning literature.

Furthermore, besides iQuery [1], in Table 1 of Section 4.2, we present experimental results based on another model architecture, Co-Separation [2]. Our method outperforms baseline methods in this context as well, demonstrating the generalizability of our proposed approach to different separation model architectures.

Q3: Cross-modal Similarity Distillation Constraint is just the combination of the contrastive loss implemented on the modalities and features extracted from different training step.

We would like to note that the proposed Cross-modal Similarity Distillation Constraint (CrossSDC) is a non-trivial contribution. CrossSDC is the first method to integrate previously learned features into the contrastive learning process of the current task. This enables maintaining cross-modal semantic similarity incrementally while distilling prior knowledge from older tasks directly into the current model. This simple yet effective unified training target/constraint effectively addresses catastrophic forgetting in our Continual Audio-Visual Sound Separation task. Extensive experiments with two different separation models validate its effectiveness.

Q4: Experiments conducted on other datasets can provide a more comprehensive evaluation.

Thank you for suggesting additional datasets such as Music, VGGSound, and AVE! Following common practice in recent audio-visual sound separation works, we use the Music-21 dataset as our main benchmark. Note that Music [3] is a smaller subset of Music-21. To further validate our method, we conducted experiments on the AVE dataset and have included the results in the Appendix. The results for Fine-tuning/LwF/PLOP/AV-CIL/ContAV-Sep (Ours) are 2.07/2.19/2.45/2.53/2.72 for SDR and 5.64/6.43/6.11/6.64/7.32 for SIR, respectively, which further demonstrate the effectiveness of our approach. For more details, please refer to Section A.3 in the Appendix. We will move these results to the main paper. Additionally, experiments on larger datasets such as VGGSound and AudioSet would be very interesting for future work, and we will discuss these in the main paper.

[1] iQuery: Instruments as Queries for Audio-Visual Sound Separation. In CVPR 2023.

[2] Co-Separating Sounds of Visual Objects. In ICCV 2019.

[3] The Sound of Pixels. In ECCV 2018.

Many thanks to the reviewers for their responses, I was surprised by how quickly the authors downloaded and processed the 100-class VGGSound dataset from YouTube. The 100 classes may contain roughly 90K videos. I was also surprised by the training speed of the authors. However, I am still not confident enough to recommend this paper for acceptance, mainly because of its low technical contribution. I will maintain my original score.

We’d like to provide more details about our VGGSound experiments to address the reviewer’s doubts. We randomly selected the following 100 categories:

['playing theremin', 'donkey, ass braying', 'playing electronic organ', 'zebra braying', 'people eating noodle', 'airplane flyby', 'playing double bass', 'cat growling', 'footsteps on snow', 'playing tennis', 'black capped chickadee calling', 'bouncing on trampoline', 'playing steelpan', 'waterfall burbling', 'subway, metr', 'people clapping', 'chipmunk chirping', 'chopping food', 'people shuffling', 'elk bugling', 'alarm clock ringing', 'people booing', 'canary calling', 'chopping wood', 'people humming', 'lathe spinning', 'playing tuning fork', 'playing violin, fiddle', 'singing choir', 'playing timbales', 'children shouting', 'chicken crowing', 'car passing by', 'driving motorcycle', 'bull bellowing', 'lawn mowing', 'playing bugle', 'mouse squeaking', 'child singing', 'playing tympani', 'hair dryer drying', 'basketball bounce', 'driving snowmobile', 'train whistling', 'thunder', 'dog bow-wow', 'ocean burbling', 'cuckoo bird calling', 'sheep bleating', 'splashing water', 'air conditioning noise', 'cattle mooing', 'eagle screaming', 'air horn', 'playing bass guitar', 'sloshing water', 'tap dancing', 'running electric fan', 'playing ukulele', 'playing guiro', 'playing shofar', 'people sniggering', 'people whispering', 'people finger snapping', 'car engine idling', 'bathroom ventilation fan running', 'police car (siren)', 'roller coaster running', 'playing french horn', 'swimming', 'lighting firecrackers', 'playing electric guitar', 'playing castanets', 'people babbling', 'arc welding', 'wood thrush calling', 'wind rustling leaves', 'playing darts', 'planing timber', 'crow cawing', 'shot football', 'writing on blackboard with chalk', 'people slapping', 'using sewing machines', 'raining', 'dog howling', 'playing cello', 'playing trumpet', 'fox barking', 'bowling impact', 'people crowd', 'pumping water', 'ice cracking', 'baby crying', 'playing bass drum', 'playing bongo', 'tornado roaring', 'playing steel guitar, slide guitar', 'playing squash', 'typing on typewriter'].

As you can see, our dataset is diverse, containing not only musical instruments but also human sounds, sports, traffic, animals, and various other categories. To the best of our knowledge, recent state-of-the-art methods, including the two suggested by the reviewer, have rarely been tested on such a diverse range of categories in the context of audio-visual sound separation.

The total number of videos in this subset is 61,195. For each category, we randomly selected 20 videos for validation, 20 videos for testing, and the remainder for training. This results in 57,195 videos for training, 2000 mixtures for validation, and 2000 mixtures for testing. We divided the 100 categories into 4 incremental tasks, each containing 25 categories. Both our method and the baseline methods were trained for 100 epochs per task.

We hope these details further demonstrate the breadth and rigor of our experiments on VGGSound, strengthening our contributions.