Thank you for your detailed review and constructive feedback. Below, we address your concerns and questions point by point:

Weaknesses:

1. Motivation and Validity of RFM for this Task:
   The use of Rectified Flow Matching (RFM) is motivated by recent advancements in the flow matching model on image, video and audio generation. As RFM encourage straight trajectory, it usually result in better sample quality with less sampling steps. Another important motivation for us to use RFM is the pretrained text-to-video model we used, OpenSora-v1.2, is trained with RFM. To maintain formulation consistency we choose also to use RFM.

2. Why No Audio-to-Video Adapter or Two-Way Adapter:
   Our goal is to adapt the pretrained OpenSora model (via multi-stage training) to simultaneously generate video and audio while keeping modifications to the video generation tower minimal. To achieve this, we introduced a unidirectional modality adaptor for video-to-audio fusion, ensuring the video tower remains largely unchanged.

3. Use of Anonymized Models in Evaluation:
   We understand the concern regarding anonymized model names in the initial submission. To address this, we have de-anonymized the baseline models in the revised manuscript as follows:

   • T2A-1: AudioLDM 1
   • T2A-2: AudioLDM 2
   • V2A-1: SpecVQGAN
   • V2A-2: DiffFoley
   • T2VGen: OpenSora

   These updates clarify the configurations and performance of the baseline models. Furthermore, these baselines were chosen based on their reproducibility and widespread use in text-to-audio or video-to-audio tasks.

4. Confusion Regarding SyncFlow-GH w/ and w/o Modality Adaptor:
   First of all please allow me to clarify the setting of "SyncFlow-GH w/o Modality Adaptor": In this configuration, features from the video generation tower are directly used as conditions for the audio generation tower, bypassing the modality adaptor. This setup evaluates the impact of the modality adaptor on the performance of audio generation and audio-visual alignment. SyncFlow-GH with the modality adaptor performs better in CLAP and KL scores because the adaptor effectively enhances the conditioning from video tower (which also includes text information) to audio tower. This design ensures that the audio generation process leverages the processed and potentially more optimal conditioning information.

5. Temporal Alignment Metrics:
   We acknowledge that ImageBind, while useful, is not optimal for evaluating temporal alignment. As suggested, metrics like Onset Accuracy and AV-align scores provide more robust evaluations of synchronization. Due to time constraints during the review period, we were unable to implement these metrics but plan to incorporate them in future evaluations.

6. Performance Comparisons on AIST++ or Landscape Datasets:
   This is an important direction for future work, and we will acknowledge this limitation in the updated manuscript.

---

Questions:

1. Fairness of Cascaded Model Comparisons:
   Most of the V2A baseline models used in our experiments, such as SpecVQGAN and Diff-Foley, are trained or fine-tuned on the VGGSound dataset. While part of the T2A baseline models we employed were not specifically fine-tuned on VGGSound, they still included VGGSound as part of their training data. We apologize for any confusion caused by the anonymization of these models in the initial submission, which may have obscured these details.

---

Thank you once again for your valuable feedback, which has helped us improve the clarity and quality of our work. We hope these clarifications address your concerns and provide a better understanding of SyncFlow’s contributions and design choices.

Thanks for the authors' response. Some concerns (anonymous models, ablation study) are solved, but I maintain the original rating.

1. Expanding an audio model with the same structure to extend the existing RFM-based video models is not considered a structural novelty. It also does not show the advantages of this structure compared to the performance of other structures.

2. I found that the main contribution of this paper is temporally synchronized video and audio generation, but the evaluation for temporal alignment is still limited. I also watched the demo site, but there was a lack of samples to evaluate this.

We appreciate your detailed feedback and constructive suggestions. Below, we address the identified weaknesses:

Weakness #1:

We have revised the introduction section to incorporate the suggested reference [1].

Weakness #2:

We use only the video captions as input to the model to prevent it from overly relying on audio caption information during content generation. This design choice encourages the model to infer audio content by leveraging video information.

Weakness #3:

We have employed evaluation metrics similar to those used in the TAVGBench [1]. We have updated the evaluation section to include a citation to the recommended paper, ensuring our methodology is more aligned with established practices.

[1] Mao, Yuxin, et al. "TAVGBench: Benchmarking text to audible-video generation." Proceedings of the 32nd ACM International Conference on Multimedia. 2024.

Thank you for your thoughtful review and constructive feedback. Below, we provide responses to address your comments and clarify the raised concerns:

Weakness #1

Our goal is to adapt the pretrained OpenSora model (via multi-stage training) to simultaneously generate video and audio while keeping modifications to the video generation tower minimal. To achieve this, we introduced a unidirectional modality adaptor for video-to-audio fusion, ensuring the video tower remains largely unchanged. The adaptor implicitly learns temporal alignment during training through the design of the model architecture, which integrates video features as conditions for audio generation. We acknowledge that better metrics for quantifying temporal alignment could further validate these improvements and consider this an important direction for future work.

Weakness #2:

The decision to use only video captions for our model is based on the following considerations:

Consistency with the Pretrained Backbone:

Our model builds on the pretrained text-to-video model OpenSora (or T2VGen in the previous anonymized version), with the goal of extending it for joint video and audio generation. OpenSora was originally trained exclusively with video captions, and maintaining this setup ensures that its video generation quality remains consistent with its pretrained performance. Adding audio captions could potentially disrupt the pretrained dynamics of the video backbone.

Encouraging Video-Dependent Audio Generation:

By excluding direct audio captions, we aim to encourage the audio generation process to rely more on visual information from the video. This approach helps avoid a scenario where the model depends heavily on audio captions for audio generation, potentially overlooking visual cues. Enhancing the reliance on video information can improve audio-visual alignment and lead to better synchronization between the two modalities.

Weakness #3:

We have updated the paper to include missing experimental details, such as:

• Learning rate and optimizer: We use the AdamW optimizer with an initial learning rate of $1 \times 10^{-4}$ for all training stages.
• Training approach: The Video VAE and Audio VAE are pre-trained independently and remain frozen during the SyncFlow training stages.
• Baseline models: The de-anonymized baseline models are as follows:
• T2A-1: AudioLDM 1
• T2A-2: AudioLDM 2
• V2A-1: SpecVQGAN
• V2A-2: DiffFoley
• T2VGen: OpenSora

Weakness #4

Please find our demo available here https://syncflow-core.github.io/syncflow-demo/

---

Responses to Specific Questions:

1. Comparison with MM-Diffusion and TAVGBench:
   While MM-Diffusion and TAVGBench are discussed in the related work, we did not directly compare them due to the lack of publicly available code or pretrained models for replicating their results. As MM-Diffusion does not open-source their model checkpoint pretrained on AudioSet, it is challenging to compare with MM-Diffusion on open-domain scenarios.

2. Use of video-derived text captions:
   Please refer to weakness #2

3. Temporal alignment evaluation beyond ImageBind:
   While we acknowledge that ImageBind is not an optimal metric for evaluating temporal synchronization, it effectively highlights clear differences between approaches. This suggests that there is still room for improvement, even in semantic alignment.

4. Ground truth ImageBind scores in Table 2:
   The suboptimal ImageBind score for the ground truth data likely arises from limitations in the pre-trained video captioning model used for evaluation. Captions may not fully represent the actual content, leading to a lower semantic alignment score. We have clarified this in the revised text.

Thanks to the authors for the responses, which clarified many of my concerns.

I have a follow-up question regarding the ablation studies:

• SyncFlow-GH w/o modality adaptor: Does this setting involve passing a null input to the cross-attention layer in the Audio Tower?
• Text-to-audio w/ SyncFlow Audio Tower: In this setting, does the cross-attention layer in the Audio Tower take the T5 embeddings as input?
• For these two settings, is the model entirely retrained, with changes limited to the cross-attention layer's input shape?

Additionally, I have some remaining concerns and suggestions:

1. While it is noted that Seeing & Hearing and V2A-Mapper do not provide code, I believe Seeing & Hearing can still be incorporated as a V2A baseline. Given that Seeing&Hearing employs keyframe extraction and captioning, it could address the limitation of VideoOFA relying on full-length videos, as mentioned in A.1 Limitations.
2. The paper provides an insufficient explanation of the baselines. A brief description of these baselines, either in the related work or methodology sections, would be helpful. Additionally, differences in the experimental setup for baselines should be clarified. For example, SpecVQGAN and Diff-Foley produce audio of different lengths, which necessitates explaining how this was managed.
3. As mentioned by authors in response to Question #1, fine-tuning on VGGSound tends to overfit the Video Tower due to the dataset's limited size. Therefore, the choice of training datasets (response to Question #5) should have included larger datasets. I recommend that the authors address this limitation.

My weakness #2, which I believe is a critical issue, remains unaddressed. I will check the other reviews and follow up soon.

Thank you for your valuable feedback and insights on our paper. We acknowledge that certain parts of our presentation may have caused misunderstandings, and we aim to address your concerns below:

Weakness #3: Experimental settings clarity

• Previously anonymised baseline models have been de-anonymized in the revised paper:
  • T2A-1: AudioLDM 1
  • T2A-2: AudioLDM 2
  • V2A-1: SpecVQGAN
  • V2A-2: DiffFoley
  • T2VGen: OpenSora
• The mention of contrastive loss was a typo in the text. We have clarified in the revised version that the final proposed system does not employ contrastive loss.
• Regarding "SyncFlow-GH w/o modality adaptor", this configuration directly uses features from the video generation tower as conditions for audio generation, bypassing the modality adaptor. This setup is designed to evaluate the significance of incorporating a modality adaptor before conditioning the audio generation tower.
• "Text-to-Audio w/ SyncFlow Audio Tower" conditions the audio generation solely on text, omitting information from the video tower, to analyze the importance of video input in enhancing audio generation.

---

Weakness #1: Lack of comparison with baselines

• We have compared our method with Diff-Foley (referred to as V2A-2 in the previous anonymized version) and SpecVQGAN (V2A-1). Results in Table 1 demonstrate that OpenSora+Diff-Foley does not perform comparably to our proposed model. The inclusion of SpecVQGAN further illustrates our approach's effectiveness.
• Regarding the Seeing & Hearing work, the T2AV implementation code is unavailable on their GitHub repository (https://github.com/yzxing87/Seeing-and-Hearing), which limits our ability to replicate their results for comparison.
• Similarly, for the V2A-Mapper baseline, the GitHub repository (https://github.com/heng-hw/V2A-Mapper) lacks inference code to reproduce their results. Therefore, direct comparisons in a way similar to how we employ Diff-Foley are infeasible in this case.

Weakness #2 and #4 are addressed in later comments due to space limitation

---

Responses to Specific Questions:

Question #1 Performance drop in Table 1 for SyncFlow-VGG-AV-FT:
The drop is likely due to overfitting the video tower during fine-tuning on VGGSound. Since the video tower (OpenSora) was pre-trained on large-scale video datasets, fine-tuning it on the smaller VGGSound dataset may restrict its generated data distribution. This degradation in video quality can negatively impact audio generation, as the audio tower is conditioned on video tower features.

Question #2 Text information in the audio tower (Table 4):
Incorporating text directly into the audio tower risks over-relying on text conditions while neglecting visual cues. Although this improves CLAP and KL scores, as shown in Table 4, it reduces semantic alignment (IB score) between video and audio, highlighting the importance of leveraging visual information.

Question #3 IB Gen-V exceeding ground truth (Table 2):
This may stem from imperfections in the video captioning model, which lowers IB scores for ground truth videos and text. Conversely, generated videos, conditioned directly on captions, align better with the captions. Another contributing factor could be resolution effects, where higher-resolution videos can potentially yield higher IB scores.

Question #4 Choice of training datasets:
As highlighted in the VGGSound dataset paper (Chen et al., 2020): "VGGSound ensures audio-visual correspondence and is collected under unconstrained conditions." With over 310 sound classes, VGGSound effectively approximates open-domain settings. We opted not to perform experiments on AudioSet due to concerns about the quality of audio-video alignment in its samples. Compared to AudioSet, VGGSound is more carefully curated for audio-visual correspondence while also focusing on open-domain scenarios, making it potentially more suitable for our task. Additionally, to evaluate performance in a narrower domain, we included the Greatest Hits dataset in our experiments. Chen, Honglie, et al. "Vggsound: A large-scale audio-visual dataset." ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020.

Question #5 Clarification on MM-Diffusion experiments:
MM-Diffusion provides open-domain audio-video generation samples in their appendix but does not report evaluation results for comparison. We have updated our paper to clarify this.

Question #6 Clarification on Figure 2 (Modality Adaptor):
The concatenation involves the modality adaptor output and flow matching step embedding $t$. The revised caption clarifies this aspect. We have added further clarification in the caption of Figure 2.

---

We hope this addresses your concerns and provides the necessary clarifications. Thank you again for your detailed review and constructive feedback.

(continued ...)

Weakness #2: Better evaluation metrics and why multi-stage learning

1. ImageBind as a Metric for Temporal Information:
   While the ImageBind feature is one-dimensional and therefore not ideal for capturing temporal information, it does not entirely lack such information. For example, the AVHScore proposed by Mao et al. 2024 also use Imagebind to assess audio-video alignment, which is the evaluation method we followed. While the evaluation may not achieve perfect accuracy, it can still highlight relative trends between models and demonstrate the improvements made by our method. This makes it a useful, although not perfect, metric for evaluating audio-visual synchronization.

2. Efficiency and Benefits of Multi-Stage Training:
   The efficiency of multi-stage training in our approach is evident for the following reasons:

   • Computational Efficiency: Training a single text-to-video generation model, such as OpenSora-v1.2, requires over 30 million video samples and approximately 35,000 GPU hours on H100 GPUs (as per the Open-Sora report). Attempting to train both the video and audio towers from scratch without leveraging multi-stage learning would require significantly more compute resources and time, making it impractical and inefficient.
   • Better Utilization of Task-Specific Data: Multi-stage training enables effective use of task-specific datasets. Text-to-video tasks typically benefit from a larger amount of high-quality data compared to audio-video data. By first training the video generation tower and subsequently training the audio generation tower, we ensure that the model takes full advantage of available data for each task, enhancing both efficiency and performance.

Weakness #4

We have created a demo website showcasing the results of our model: SyncFlow Demo Website.

We hope our responses address your feedback.

Thank you for your follow-up questions and for pointing out the areas that require further clarification. Below, we address your specific concerns regarding the ablation study settings (The PDF update may be delayed, but we will ensure all changes are reflected in the paper before the Nov 28 deadline):

• "SyncFlow-GH w/o Modality Adaptor":
  In this configuration, features from the video generation tower are directly used as conditions for the audio generation tower, bypassing the modality adaptor. This setup evaluates the impact of the modality adaptor on the performance of audio generation and audio-visual alignment. We apologize for any confusion in our earlier response and have added further clarification in the ablation study section of the revised manuscript.

• "Text-to-Audio w/ SyncFlow Audio Tower":
  In this setting, the cross-attention layer in the Audio Tower uses the T5 text embeddings (previously utilized as input to the video generation tower) as its condition. This evaluates the effect of using text-only conditioning on audio generation.

Both configurations are retrained from scratch, with the video tower initialized from OpenSora's pretrained weights. The key differences are:

1. Whether the modality adaptor processes video features before conditioning the Audio Tower.
2. Whether audio generation uses video features or relies solely on text embeddings."

---

For your remaining concerns and suggestions:

1. Incorporating Seeing & Hearing as a Baseline:
   We acknowledge that Seeing & Hearing could be a useful baseline for V2A generation. However, recent work (Zhang et al., 2024) indicates that Seeing & Hearing performs worse on V2A tasks compared to Diff-Foley, which influenced our decision not to include it as a baseline. More importantly, our experiments indicate that the fundamental limitation lies in the two-stage generation approach itself, rather than the choice of a specific V2A model. For example, even the best-performing open-source systems in a two-stage pipeline achieve an FAD of around 5.5, whereas our end-to-end system achieves a significantly lower FAD of 1.81. While incorporating the V2A component of Seeing & Hearing might provide additional insights, our current results sufficiently highlight the inherent drawbacks of two-stage systems. Nonetheless, we will consider including Seeing & Hearing as a baseline in future studies focusing on V2A models."

Zhang, Yiming, et al. "Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds." arXiv preprint arXiv:2407.01494 (2024).

2. Improved Baseline Descriptions:
   Thank you for your suggestion. We have revised the paper to provide additional explanations of the baselines, ensuring better context and understanding for readers.

3. Choice of Training Datasets:
   We agree that training on larger datasets, such as AudioSet (as previously suggested), could be beneficial. However, we also emphasize the importance of data quality. While AudioSet is larger in scale compared to VGGSound, the latter is more carefully curated for audio-visual correspondence. As noted in the VGGSound dataset paper (Chen et al., 2020): "VGGSound ensures audio-visual correspondence and is collected under unconstrained conditions."

---

We hope these answers address your concerns. Weaknesses #2 and #4 are addressed in the next part.