Learning to Stylize Soundscapes from In-the-Wild Videos

• My main concern about this paper is that I do not see any fair comparison with the models in the literature that also perform audio-style-transfer in a similar fashion. Specifically, it is surprising that Separate & Remix (Petermann et al., 2022) would perform so comparably with the model proposed here which is trained specifically for this task and is much more computationally heavy. I believe that the remixing strategy of the aforementioned baseline could be fine-tuned for each example in a more proper way. For instance, it is evident that in Figure 5, first row, the Input-audio is remixed at a very low signal-to-noise ratio (SNR) for the Separate & Remix column which would probably make it sound worse. Or for example in the same Figure 5, final row, Separate & Remix seems to not be embedding the full input-audio, which clearly does not make sense, since the separation model should be simply run on the conditional audio input, extract the background noise and then remix the background noise with the input speech. Thus, I consider that by fixing those issues, a proper baseline model could be used to compare against this method.

• Building upon my previous argument, the proposed method is composed of multiple elements and many more millions of trainable parameters and extra computational complexity compared to other baselines with much simpler models as for example Separate & Remix (Petermann et al., 2022). Thus, the models should either try to match the computational and memory complexity and perform the same experiments for the baseline models or at least use the Separate & Remix (Petermann et al., 2022) as an input condition to the model to at least show that they can further enhance the performance of the overall model.

• The authors throughout the text try to make certain that their method is general enough to be applied to any type of sound, but it seems to me that the authors have only tested their model with speech style transfer, thus, I think they should be more explicit in what exactly the application of this paper is. Moreover, the pre-trained vocoders are not self-supervised, and thus, in a similar sense, the authors should try to avoid using this characterization or at least try to be more explicit on which parts were self-supervised during training.

I would be more than happy to increase my score if all the above weaknesses are addressed by the authors.

1. The model proposed by authors is quite straightforward without enough technical novelty, most of them are existing components like the VAE and latent diffusion models for style transfer, and the HiFiGAN for vocoder.

2. The baselines which the method compares against are not strong, these baselines are not targeted towards soundscape style transfer. I think at least Chen et al.2022 should be compared as the model architecture is easy to implement.

3. I feel like the quality of style transferred results shown in the demo website are not good enough, and some examples after transferring lost too much clarity in speech, and some examples don’t match the environment acoustic property.

4. The appendix provides some details of user study but is not enough. What is the distinct number of the voters, their qualification level like historical approval rate, how much time on average did they spend for each vote etc?

• The proposed method’s pipeline is very much utilizing methods from previous works, making it challenging to discern the authors’ highlighted contributions throughout the manuscript.
• It’s unclear why the model needs to take both the enhanced signal $a_e$ and the original target signal $a_0$ as the input. Since the soundscape information should be captured from $a_c$ not $a_0$ (where the soundscape of $a_c=a_0$), the objective of the proposed system is to just predict the noise applied on $z_t$. With this approach, the model doesn’t need to encode any information from $a_c$ or $i_c$, where it should focus on capturing the soundscape information from them.
• Regarding objective evaluation: With the current approach, the model can be evaluated on reconstruction task as Equation (1). Yet, the paper only uses evaluation metrics related to generation task. The reviewer suggests evaluating on intrusive objective metrics to focus more on acoustic matching rather than the generative approach.

presentation feedbacks

• Table 1 is misleading. All the captioning-based models also use AudioLDM, which looks conflicting with the AudioLDM on the Table.
  • The choice of AuidoLDM and Separate & Remix as baseline system is puzzling, as these systems are not suited for the task of generating or matching reverberation applied on input speech signal. To compare with these methods, the proper way would be to remove all speech signals and compare only with the soundscape audio, and to perform acoustic matching with different baselines.
• Table 3 is misleading since the final version of model uses the fine-tuned model of CLAP and not fine-tuned model of CLIP. The reviewer suggests adding a column to distinguish FT-A and FT-V in the table for better clarity.
  • In Section 3.2 Conditional audio-visual representations, the manuscript mentions using a ResNet-18 backbone for audio | image representation. Maybe it’s better to remove this sentence since the final model is using CLAP & CLIP?

• The proposed task is rather constrained as such reference video/audio can be tricky to find in practice. Some discussions on the potential applications and use cases would be helpful to motivate why this task is important and worth investigating.
• The authors did not compare the proposed model against the high-relevant model proposed by Chen et al. (2022). While the authors showed that the proposed method can outperform a naive "Separate and Mix" baseline, it would be helpful to compare the proposed model against Chen et al. (2022).
• The dataset only contains 223 videos, which is small in terms of diversity. It remains unclear whether the model can generalize to out-of-distribution samples.
• While the authors claim that the system can be trained on in-the-wild videos, the dataset contains a specific type of high-quality egocentric videos. It remains unclear whether the proposed method can be easily scaled up or be trained on noisy in-the-wild videos.