Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching

We highly appreciate your positive appraisal of our work and would like to discuss the issues you raised here.

[Computing alignment accuracy]

As stated in the Metrics part, Section 4.1, we adopt the alignment classifier provided in Diff-Foley[1] for calculating alignment accuracy. Specifically, we convert the generated audio to 128-bin mel-spectrogram and feed it to the classifier along with the CAVP features. We will emphasize these details in the revised version.

[Alignment learning ability of autoregressive models]

Thanks for pointing out this. We agree that recent large-scale autoregressive models like AudioLM and MusicLM have shown a strong ability to learn alignment between different modalities with self-attention and in-context learning. However, the good performance of these models often relies on large-scale transformer decoders and substantial amounts of training data. Early autoregressive baselines on VGGSound perform poorly in terms of temporal synchrony (see table 1 in our paper), this may be due to the limited model capacity and training data volume. We will modify our statements in the revised version of our paper.

[Other wording and scientific rigor issues]

Thank you for pointing out the issues in our writing, we will revise our wording in the revised version. For example, we may change line 18 to ... significantly enhanced the quality and diversity..., and change line 36 to ...but still has a gap in quality compared to state-of-the-art text-to-audio models and real-world audio.

[Selection of visual features]

In previous work [1], CLIP has been shown to be not effective enough in generating temporally aligned audio. In this work, we mainly use CAVP for a fair comparison with [1] and attempt to find a better visual representation for video-to-audio generation (taking MAViL as an example).

[Sampling rate and duration issues]

Following most previous audio generation models, we adopt 16kHz audio. This could be improved by using spectrogram, VAE, and vocoder with a higher sampling rate, which shouldn't be a big problem. On the other hand, most public-available on-the-shelf video-to-audio data, like VGGSound and AudioSet, are short video clips, restricting extending generation lengths. We may delve into these issues in future work.

[Plan for releasing code and weights]

Thanks for your interest. We plan to release our code and weights on GitHub several weeks later, no matter whether our paper gets accepted or not.

Once again, thank you for your effort in reviewing our work and your acknowledgment. We welcome further discussion with you.

[1] Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models. Advances in Neural Information Processing Systems, 36, 2024.

We are highly grateful for your positive appraisal of our work, and we'd like to discuss the issue you raised here.

[Generization experiments on landscape dataset]

Following your advice, we conduct experiments on the landscape dataset to investigate the generalization capability of our model. We compare our zero-shot performance with Diff-Foley. We also try to finetune our model on landscape for about 4k steps (about 268 epochs). The results are illustrated in the following table.

It can be seen that the zero-shot performance of Frieren significantly outperforms Diff-Foley in multiple metrics. On the other hand, we observe that finetuning improves in FD and FAD, with the differences being 4.49 and 0.70. However, it also leads to degradation of 0.41 and 2.00 in IS and KL, respectively. Due to the limited size of the landscape dataset, there may be a distribution gap between the training and testing splits. Finetuning could lead to a degree of overfitting on the training set, resulting in a decline in certain metrics.

[More empirical analysis on model performance]

First, we'd like to claim that both the transformer architecture and the rectified flow (RF) modeling method contribute to the model performance (please refer to our response to reviewer MSJt for details and additional results). Our rectified flow model brings improvement in IS, FAD, and Acc while enabling generation with fewer or even one step. Besides, adopting a better ODE solver can further improve the performance of RF and increase its performance gap with DDPM.

Second, we have conducted experiments with sequence-level concatenation for the conditional inputs (similar to cross-attention essentially, as the alignment is learned by attention). However, this model fails to generate meaningful audio, and the metrics are unacceptably bad as shown in the table below.

We also provide two pairs of results of sequence-level and channel-level concatenation in the PDF. It can be seen that the sequence-level model tends to generate flat, monotonous, and meaningless audio. However, the frequency bands where energy is concentrated are similar in the results of the two models, with the bright lines on the spectrograms being of similar height. We speculate that this indicates that the sequence-level model can extract semantic information from conditional inputs, but it fails to learn temporal alignment through attention. Adopting different positional embedding doesn't help. This seems a little weird, as cross-attention shows a fundamental alignment learning ability in the baseline diffusion model, but it just turns out not working in our architecture. Maybe it has something to do with model size and capacity. Anyway, these results illustrate the necessity of channel-level feature fusion.

Last, the design of our transformer block with 1D convolution derives from a diffusion-based text-to-audio generation model[1], and it is proven in the paper to perform better than 2D-convolution-based U-Net on audio generation, and has better potential to generalize to audio of longer and variable lengths. Due to the limited response time, we haven't made it to implement and train a 2D version of the model to examine the performance gap yet. We plan to add these results in the final revised version of our paper.

[Challenges in applying rectified flows for audio generation]

Compared to other tasks, we think that applying RF in video-to-audio generation faces the following challenges:

1. Our method shares some similarities with RF-based TTS models like VoiceFlow [2]. However, we find that compared to the strong and highly deterministic content condition (text) in TTS, V2A has a weaker condition and its performance relies more on guidance. Compared to RF TTS models, we found that using the CFG-corrected vector field as the regression target during the reflow stage is crucial for model performance, rather than using the same vector field as in previous RF models (see section 3.5).

2. As stated above, compared to text-conditioned generation like T2I and T2A, it turns out attention-based conditional mechanism fails to provide precise information in terms of semantic and temporal alignment for V2A. Hence we propose the channel-level feature fusion used with the feed-forward transformer architecture.

[CFG in reflow]

Yes. As shown in equations (8) and (9), we use the CFG-corrected vector field as the regression target during the reflow and distillation, where the CFG scale is the same as that used for generating reflow data. And we use the same CFG scale for sampling with the reflowed model.

Once again, thank you for your effort in reviewing our work and your acknowledgment. We hope our clarifications address your concerns, and we always welcome further discussion with you.

[1] Jiawei Huang, Yi Ren, Rongjie Huang, Dongchao Yang, Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, and Zhou Zhao. Make-an-audio 2: Temporal-enhanced text-to-audio generation. arXiv preprint arXiv:2305.18474, 2023.

[2] Yiwei Guo, Chenpeng Du, Ziyang Ma, Xie Chen, and Kai Yu. Voiceflow: Efficient text-to-speech with rectified flow matching. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 11121–11125. IEEE, 2024.

Thanks for your valuable comments. We'd like to make some clarification and discussion about the issues you raised.

[Difference with Diff-Foley in architecture]

We'd like to clarify that our model significantly differs from Diff-Foley in both architecture and alignment mechanisms. Diff-Foley adopts a U-Net denoiser with a cross-attention-based conditional mechanism. As stated in sections 1 and 4.2, cross-attention alone struggles to achieve precise audio-visual alignment, and therefore Diff-Foley relies on an additional classifier for guidance, which is complicated and unstable with fewer steps. In contrast, we adopt a transformer vector field estimator with channel-level cross-modal feature fusion, achieving higher synchrony and robustness with simpler architecture.

[Difference with other flow-matching-based audio models and novelty issues]

We agree that our model shares some similarities with previous flow-matching-based audio models. Nevertheless, we have delved deeper into certain aspects compared to previous speech models like VoiceFlow [1].

1. Compared to the strong and highly deterministic content condition (text) in TTS, V2A has a weaker condition and its performance relies more on guidance. We found that using the CFG-corrected vector field as the regression target during the reflow stage is crucial for model performance, rather than using the same vector field in both initial training and reflow as in previous rectified flow models (see section 3.5).

2. Upon reflow, we further investigate one-step distillation, which further improves the single-step performance and previous flow-matching-based audio models did not address it. We also investigate techniques like objective reweighting for further performance improvement.

[Effect of vocoder on qualitative results]

We agree that the vocoder significantly impacts audio fidelity and objective metrics. Our target is to build an integral system for video-to-audio generation with higher quality and generation efficiency, and therefore we replace the slow and low-quality Griffin-Lim with BigVGAN. For reference, we provide the results of Frieren with Griffin-Lim as the vocoder in the following table. The number of Griffin-Lim iterations is the same as Diff-Foley.

It can be seen that despite the performance drop, Frieren still surpasses Diff-Foley in KL, FAD, and ACC, with FAD showing a significant advantage while maintaining competitive FD and IS values.

Moreover, due to the limitations of objective metrics, we highly recommend you refer to our demo page (https://frieren-v2a.github.io/). It can be observed that in addition to audio fidelity, the samples from our model exhibit better semantic content and more precise temporal alignment compared to Diff-Foley, demonstrating the qualitative advantages of our rectified-flow-based model.

It's also worth mentioning that other than audio quality, Frieren achieves a generation speed $7.3\times$ that of Diff-Foley (see table 8 in the paper, taking only the spectrogram generation for consideration), demonstrating a significant advantage in terms of generation efficiency.

[Source of MAViL]

We adopt the MAViL implementation and checkpoints from a public-available audio-visual representation benchmark project (AV-SUPERB, https://github.com/roger-tseng/av-superb). We made a slight modification to the model input so that it takes 4FPS video rather than 2. Due to concerns about potentially violating anonymity policies by using links in our rebuttal, we must state that there is no overlap or connection between the authors of this project and our paper.

We hope our clarifications address your concerns and we are looking forward to your re-assessment of our work. We also welcome further discussion with you. Thank you again for your efforts.

[1] Yiwei Guo, Chenpeng Du, Ziyang Ma, Xie Chen, and Kai Yu. Voiceflow: Efficient text-to-speech with rectified flow matching. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 11121–11125. IEEE, 2024.

We'd like to offer an additional perspective to assess the impact of vocoders on model performance. We use BigVGAN, rather than Griffin-Lim, as the vocoder for both Diff-Foley and Frieren. The output from Diff-Foley is converted into an 80-bin mel-spectrogram and then fed into BigVGAN. The results are shown in the following table.

First, using BigVGAN for Diff-Foley improves its FD, KL, and KID, indicating the effectiveness of BigVGAN for Diff-Foley on improving audio quality. On this basis, Frieren outperforms Diff-Foley across all metrics, with a greater difference than when using Griffin-Lim for both. This further demonstrates that our model is significantly superior to Diff-Foley. In contrast, Griffin-Lim is too weak, forming a performance bottleneck that narrows the performance gap between Frieren and Diff-Foley.

Dear Reviewer,

As the end of the discussion period approaches, we are eager to get your feedback. We have tried our best to resolve your concerns and clarify misunderstandings. We would be grateful to hear your feedback regarding our answers to the reviews.

Best Regards, Authors

Thank you for your valuable comments on our work. We would like to discuss the issues you raised here.

[Effect of transformer architecture and rectified flow]

We believe that both the transformer architecture and the rectified flow (RF) modeling method contribute to the model performance. We will elucidate the role of RF from the following perspectives.

1. According to Table 1 in the paper, our RF model demonstrates an advantage in IS, KL, FAD, and Acc, with differences being 2.33, 0.13, 0.45, and 1.89, as well as higher MOS. These differences are actually quite significant. When the sampling steps increase to 40, these advantages remain consistent (see the table below). We believe these results serve to demonstrate the significant positive effect on model performance that RF has.

   Model	Step	FD↓	IS↑	KL↓	FAD↓	KID $\times10^{-3}$ ↓	ACC↑
   DDPM	40	11.63	10.28	2.87	1.72	2.18	95.26
   Frieren	40	11.87	12.63	2.74	1.31	2.39	97.19

2. Differences in the sampler may potentially reduce the performance gap between RF and DDPM. We adopt an advanced solver, DPM-Solver, for DDPM, in contrast to the simplest Euler solver for RF. Due to the differences in the models, it is difficult to eliminate this effect through an entirely consistent sampler. However, we can further unlock RF's potential by employing a more advanced sampler for RF. The following table shows the results of Frieren with the Dormand–Prince method (dopri5). We can see that the RF model holds an advantage in almost all metrics, especially in IS, FAD, and Acc, with only a slight disadvantage in KID. This further indicates the advantage of RF.

   Model	Sampler	Step	FD↓	IS↑	KL↓	FAD↓	KID $\times10^{-3}$ ↓	ACC↑
   DDPM	DPM-Solver	25	11.79	10.09	2.86	1.77	2.36	95.33
   Frieren	dopri5	25	11.63	12.76	2.75	1.37	2.39	96.87

3. Lastly, RF not only enhances the quality of generated audio but also reduces sampling steps through reflow and distillation, significantly improving the model's generation efficiency, which DDPM cannot achieve.

[Effect of reflow under different steps]

We briefly discussed this issue at the end of Section 4.2 and we'd like to provide a possible explanation with more details here. Theoretically, reflow should not alter the model’s marginal distribution. Yet in practice, the limited number of steps for reflow data generation (25 steps in our experiments) can affect the data quality, introducing errors into the regression targets during the reflow process. While reflow can straighten trajectories and improve generation quality with a few steps, such errors in the regression targets can degrade the model's generation quality with over 25 steps, leading to reductions in metrics such as FAD and IS. This might be mitigated by increasing the number of sampling steps during reflow data generation. Moreover, increasing the number of iterations of generating reflow data and conducting reflow can lead to the accumulation of more errors, and that's why we only generate data once for both reflow and distillation (as discussed in section 3.5).

[Design of transformer block]

Our design of transformer block derives from Make-an-Audio 2[1], which is proven to be efficient in audio generation, despite that it does not necessarily outperform standard ViT block significantly. In other words, it is not necessarily the best choice, but it is sure to be a good one.

[Model initialization]

Yes. We load the weight of the diffusion denoiser in Make-an-Audio 2[1], which is a text-to-audio ddpm model trained with more data. Despite that we did not observe significant improvement in metrics and the rate of convergence, it seems to slightly help the subjective perception quality. We will supplement these details in the revised version of our paper.

[Pooling on condition latent]

No. No pooling is conducted on the condition sequence, as we want to keep the temporal information for generating synchronized audio.

[Difference with RF-based speech models and other Insights]

We agree that our method has some similarities with RF-based speech models, such as VoiceFlow [2]. However, we believe our model delves more deeply into certain aspects.

1. Compared to the strong and highly deterministic content condition (text) in TTS, V2A has a weaker condition and its performance relies more on guidance. Compared to RF TTS models, we found that using the CFG-corrected vector field as the regression target during the reflow stage is crucial for model performance, rather than using the same vector field as in previous RF models (see section 3.5).

2. Upon reflow, we further investigate one-step distillation, which further improves the single-step performance and previous speech models did not address it.

We hope our clarifications address your concerns. If you find our response helpful, we would very appreciate it if you consider increasing your evaluation of our work. And we always welcome further discussion with you. Thank you again for your efforts.

[1] Jiawei Huang, Yi Ren, Rongjie Huang, Dongchao Yang, Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, and Zhou Zhao. Make-an-audio 2: Temporal-enhanced text-to-audio generation. arXiv preprint arXiv:2305.18474, 2023.

[2] Yiwei Guo, Chenpeng Du, Ziyang Ma, Xie Chen, and Kai Yu. Voiceflow: Efficient text-to-speech with rectified flow matching. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 11121–11125. IEEE, 2024.