FOLEYCRAFTER: BRING SILENT VIDEOS TO LIFE WITH LIFELIKE AND SYNCHRONIZED SOUNDS

We sincerely appreciate your review and addess the major concerns below.

W1. Limited innovations

We respectfully disagree with the assessment of limited innovation. FoleyCrafter introduces several technical breakthroughs:

1. Novel End-to-End Approach:

• Previous methods either use video-to-text-to-audio pipelines (Xing et al., 2024; Wang et al., 2024; Xie et al., 2024b) compromising video alignment, or train from scratch on noisy datasets (Iashin and Rahtu, 2021; Luo et al., 2023) sacrificing audio quality.
• FoleyCrafter is the first to generate high-quality audio directly from video while maintaining strong temporal alignment.

2. We attribute the breakthroughs of FoleyCrafter by the following model designs:

• Semantic Adapter: While sharing architectural similarities with IP-Adapter, we pioneer its application for cross-modality conditioning from video to audio which differs from image-to-image in IP-Adapter. As discussed in L263-269, we develop a specialized video encoder to capture visual cues for audio generation. Our novel random dropping strategy enables both effective visual guidance and text controllability, providing a new paradigm for video-guided audio synthesis.
• Temporal Adapter: Unlike traditional ControlNet which performs spatially aligned feature residual addition from image to image, we address the unique challenge of video-audio temporal alignment through two novel approaches:
  • Timestamp event mask-based method
  • Energy map-based control that eliminates the need for video labels, enabling training on larger-scale datasets.

These adapters make FoleyCrafter the first to employ plug-and-play modules to achieve high-quality, video-aligned audio generation directly from visual content.

3. These innovations deliver substantial improvements over the strongest baselines:

• 14.33% improvement in audio quality (FID scores).
• 10.6% Better temporal alignment performance across multiple metrics.
• Unique flexibility supporting both video-only and video-text inputs. Our work bridges the gap between high-quality audio synthesis and precise video alignment, establishing a new paradigm for neural foley sound generation.

W2. Inference latency and trainable parameters.

FoleyCrafter demonstrates superior inference time. We evaluate the 10-second audio generation time for different methods using the same video frame rate on the same device, and present the results below. For implementation, both the semantic adapter and temporal adapter can be parallelized with the UNet during the sampling process, resulting in fast inference times.

The total trainable parameters include two adapters for 415M and temporal estimator for 31M which is a lot smaller than Diff-foley with 859M and comparable with SpecVQGAN and SonicVisionLM with approximately 400M.

W3. Analysis of text prompts.

We would like to clarify that FoleyCrafter is primarily designed for video-to-audio generation, without requiring text input. While most existing video-to-audio models are limited to visual inputs, our semantic adapter uniquely enables optional text-based control. We demonstrate this additional capability qualitatively in Figure 6 of the main paper.

In response to the reviewer's concern, we conducted additional comparison of text-based video-to-audio generation results with those of other methods. As described in the main paper, here we also use wav2clip (Wu et al., 2022) to calculate the embedding similarity between audio embeddings, text embeddings and visual embeddings. We conduct the evaluation on AVSync15 (Zhang et al., 2024) and use the "The sound of [label]" as prompt.

As shown in the table below, when both text prompts and visual information are provided, FoleyCrafter achieves the best performance on the text clip score, indicating the best alignment with the prompt. These results demonstrate FoleyCrafter's flexible ability to condition generation on both text and video inputs according to users' intents.

Dear reviewer,

We have submitted the response to your comments. Please let us know if you have additional questions so that we can address them during the discussion period. We hope that you can consider the raising score after we address all the issues.

If you still have questions and concerns, please feel free to comment here. We will reply it as soon as possible.

Thank you!

I appreciate the author’s responses to my concerns. However, I would like to share some remaining concerns that have not yet been fully resolved.

[About Novelty]
The contributions of this work are primarily based on the effective application of existing architectures. While the authors have clarified that the Semantic Adapter and Temporal Adapter represent new applications of existing methods, the work lacks further theoretical analysis or novel architectural design that aligns with the quality of ICLR in the context of video-to-audio generation.

[About Inference latency and trainable parameters]
Could you clarify why Diff-Foley achieves lower inference latency despite having twice the number of parameters? Is it due to the proposed two separate adapters? Additionally, some experimental details are missing. For instance, do FoleyCrafter and baselines employ acceleration techniques for attention, such as flash-attention? What is the exact inference time for each submodule of FoleyCrafter? Addressing these points could help make the results more compelling.

[About Analysis of text prompts]
Thank you for providing the experimental results. The response from the authors has addressed my questions.

[About Potential Failure Cases]
Although Figure 6 shows the effectiveness of the proposed method, the paper does not adequately discuss potential failure cases in video to audio generation. It would be beneficial for the authors to address scenarios where the model might underperform, such as wrong temporal information predicted by the temporal estimator that lead to failures, difficulties with extremely long video inputs, or challenges with some special video contents. Understanding these limitations would provide a clearer understanding of the proposed model.

[About Details of Subjective Evaluations]
 The details of the subjective evaluations are missing, i.e., what the demographics are like for the raters in subjective evaluations, whether there are attention checkers, how the results are quality-checked etc. Besides, there is no information about the compensation, or criteria for hiring human subjects for the subjective evaluation.

Dear reviewer:

Thanks for your comments. Here are our replies for your remaining concerns.

[About Novelty]

Thank you for acknowledging our new applications of the Semantic Adapter and Temporal Adapter for video-to-audio generation. We argue that the potential of developing plug-and-play modules based on off-the-shelf T2A (text-to-audio) models for video-to-audio generation is highly underrated. High-quality text-audio paired datasets are significantly larger and more reliable compared to video-audio paired datasets, which are often low-quality and inconsistent. We believe that training plug-and-play modules for video-to-audio adaptation based on text-to-audio models is a promising direction to achieve high-quality results while maintaining training efficiency, especially compared to training video-to-audio models from scratch (e.g., SpecVQGAN and Diff-Foley).

To advance the development of such a plug-and-play mechanism:

1. We take the first step by designing a Semantic Adapter to encode video features for direct attention by audio generation models. This approach avoids relying on text as an intermediate bridge, as seen in prior works (e.g., Seeing and Hearing, SonicVisionLM, and V2A-Mapper), which often leads to suboptimal semantic alignment.

2. We conducted the first extensive study on both timestamp-based and energy-map-based audio signal estimation to address the spatial misalignment problem between visual and audio modalities when applying ControlNet.

We sincerely hope that these insights and explorations, supported by our extensive experiments and strong results, will inspire future research and drive progress in the video-to-audio community.

[About Inference latency and trainable parameters]

Thanks for pointing out this issue. We will add the following clarifications to the final version.

• Inference latency of Diff-Foley. While Diff-Foley has twice the number of trainable parameters (859M for the entire model trained from scratch), it achieves lower latency during inference since its total model size remains 859M. While FoleyCrafter has fewer trainable parameters (415M for adapters), it has about 1.2B (additional 890M for the frozen UNet) model parameters during inference, leading to slightly higher inference latency. It is important to note that inference latency can also be influenced by various design factors (e.g., network architecture, downsampling factors, feature extraction, etc.). Nonetheless, by comparing the overall inference latency and trainable parameters, we demonstrate that FoleyCrafter achieves competitive inference latency while offering the added advantage of high training efficiency, thanks to its plug-and-play framework.

• Comparison settings. We have checked and ensured that all the models (including FoleyCrafter) were tested using their native implementations, without employing acceleration techniques such as flash attention, to ensure a fair comparison.

• Exact inference time for each submodule. The exact inference time for each submodule of FoleyCrafter is as follows: Semantic Adapter (0.58 seconds for visual encoding), Temporal Adapter (0.12 seconds for audio signal estimation), UNet inference (1.59 seconds for complete diffusion sampling), and IO (0.84 seconds for reading video frames), resulting in a total inference time of 3.1 seconds. We recognize that FoleyCrafter has potential for further optimization in terms of inference latency, which we plan to address in future work.

We have provided additional details and clarifications in response to your remaining concerns, and we would like to know if there are any further questions or issues we can address. We are committed to engaging fully and will reply as promptly as possible.

Thank you once again for your insightful and constructive feedback, which we believe has been instrumental in strengthening our work. We sincerely hope the improvements we have made address your key concerns, highlight the contributions of our paper, and contribute positively to the development of this community. We kindly ask you to reconsider your evaluation in light of these changes, and we remain happy to further address any additional concerns you may have.

Dear Reviewer p9QH,

As the discussion deadline approaches, we would like to know whether we have addressed your remaining concerns.

We highly value your feedback and have made additional clarifications and necessary revisions in the manuscript, highlighted in $\color{red}{red}$. Your time and feedback are greatly appreciated, and we look forward to your response.

We sincerely appreciate your positive feedback regarding our 'unique and original thinking,' 'high research quality with extensive experiments,' 'well-structured' presentation, and 'innovations with substantial application potential.' We address the remaining concerns below.

W1. Novalty in leveraging T2A models for V2A generation.

FoleyCrafter is the first to employ plug-and-play modules to achieve high-quality, video-aligned audio generation directly from visual content. It investigates the off-the-shelf high-quality pre-trained audio generator for video-to-audio (V2A) generation. The pre-trained model exhibits a robust text-to-audio generative capability that we find advantageous for V2A tasks. By utilizing this powerful audio generator, FoleyCafter can produce more realistic and higher-quality audio compared to existing video-to-audio models.

However, generating video-aligned audio with such a well-trained audio generator still remains a challenge. To address this, we propose the semantic adapter and temporal adapter to enable visual relevant and synchronized audio generation. In summary, we investigated how to leverage the existing T2A model and introduced new modules to adapt it for V2A tasks.

W2. About the generation speed.

FoleyCrafter has a fast generation speed. We evaluate the inference time of existing works using the same video frame rate and on the same computing device, and report the results below.

For implementation, both the semantic adapter and temporal adapter can be parallelized with the UNet during the sampling process, resulting in fast inference times.

W3. Ablation of Semantic Adapter and Temporal Adapter.

We indeed include ablation experiments for semantic and temporal adapters in Table 3 and Table 4 of the main paper.

We report audio quality and audio-visual relevance with and without the semantic adapter in Table 3 L483-L484. The improvement in MKL and CLIP scores indicates that the semantic adapter effectively integrates detailed visual embeddings for audio generation. Additionally, the lower FID demonstrates that the semantic adapter enhances the utilization of the well-trained audio generator for high-quality video-to-audio generation.

For the temporal adapter, we assess the temporal alignment of audio samples generated with and without it in Table 4 L457-L460. The significant improvement (10.6%) of Onset AP, AV-Align and Energy MAE shows temporal adapter improved the synchronization of FoleyCrafter.

W4. Ablation of λ in Cross-Attention.

As described in L240-L269, the semantic adapter utilizes the parallel cross-attention with a variable parameter λ. Here we show the ablation results of different λ. The results indicate that as λ decreases, the MKL, CLIP score, and FID all decline, demonstrating that the semantic adapter is crucial for audio-visual alignment and high-quality generation.

Dear reviewer,

We have provided comprehensive responses to each concern. Please let us know if you have any additional questions that we can address during the discussion period. We hope that you can consider the raising score after we address all the issues.

If you still have questions and concerns, please feel free to comment here. We will reply it as soon as possible.

Thank you!

We sincerely appreciate your review and addess the major concerns below.

W1. Additional Evaluation in terms of FD and FAD.

We appreciate the reviewer's suggestion regarding the evaluation metrics. Following this feedback, we conducted comprehensive evaluations using both Fréchet Distance (FD) and Fréchet Audio Distance (FAD) as below. Our results demonstrate that FoleyCrafter consistently outperforms existing methods, achieving state-of-the-art performance across both metrics. Specifically, FoleyCrafter achieves an FD of 20.32 and FAD of 2.69, representing a 19.41% improvement over the strongest baseline.

Improvement of writing style.

We appreciate the reviewer's detailed suggestions on improving the presentation. We have standardized all table, figure, and citation references, and expanded Section 2 with more comprehensive related works. All changes can be found in our uploaded revision.