MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation

We appreciate the constructive feedback from the reviewer. We would like to answer the questions and the comments below.

Duplicated statement about evaluation metrics and datasets in Sec 4.2 and 4.3 (Weakness 1)

Thank you for your suggestion. Based on your comment, we carefully checked our manuscript but could not find any duplication in the evaluation metrics and datasets described in Sections 4.2 and 4.3. Section 4.2 explains the details of the Landscape dataset, FVD, FAD, and IB-AV scores, which are used in both Sections 4.2 and 4.3. Then, Section 4.3 explains the details of the VGGSound dataset, IB-TV and IB-TA scores, which are used only in Section 4.3.

Comparison between "Seeing and Hearing" (Weakness 2)

Since they do not release the code for joint video and audio generation in their official repository [1], we cannot directly compare their method and ours. Therefore, here, we compare our method with Seeing and Hearing (SaH) from the perspective of theoretical difference and clarify the benefit of our method by using the reported numbers in their paper.

Our method differs from SaH by explicitly guiding base models' generation process toward a specific joint distribution. Specifically, we formulate our method to bridge the gap between an independent distribution $p(x)p(y)$ and a joint distribution $p(x, y)$. We assume that the training data is a sample of the target distribution $p(x, y)$ and derive the loss function to match the distribution learned by the base models with the target one. Therefore, the generation with our proposed method can be considered sampling from the target distribution $p(x, y)$. In contrast, SaH guides the generated samples to be closed in the latent space of a pre-trained foundation model, ImageBind. Although they successfully improve the audio and video alignment, the distribution of its generated samples is not explicitly defined. This difference makes our method preferable regarding two aspects: i) effectiveness as a guidance module for a joint generation model and ii) customization of base models for the specific dataset.

From the perspective of i), our guidance can be applied to the joint generation model to boost its performance. As shown in Table 2, our method successfully improves both FAD and FVD of MM-Diffusion. In contrast, SaH struggles to improve both metrics of MM-Diffusion (FAD improves from 7.7 to 6.4 while FVD degrades from 1141 to 1174 as they reported in Table 1 in their paper). Our method provides a similar benefit to the classifier or classifier-free guidance commonly used in conditional generation.

From the perspective of ii), our method successfully guides the same base models toward different joint distributions using several datasets (e.g., Landscape and VGGSound datasets). Tables 3 and 4 demonstrate this success as our method improves the FAD and FVD computed on each target dataset. Note that it is difficult to achieve this improvement by SaH because there is no guarantee that the distribution of generated samples by their method is closed to the target distribution. Our method allows us to customize base models for different datasets to achieve sampling from the specific target distribution.

[1] https://github.com/yzxing87/Seeing-and-Hearing?tab=readme-ov-file#todo

Reason why metric scores of MM-Diffusion differ from the original papers (Question 3)

Thank you for raising your concern about the mismatch in the metric scores. We generated pairs of audio and video by using the official implementation of MM-Diffusion and conducted evaluation on these samples. Although it is unclear that what causes the difference between our reproduced results and reported numbers in the original papers, this mismatch is also reported in one of the issues raised in their repository [1] and a previous work [2] (FVD=~1141 and FAD=~7.75 for Landscape in [2] vs. FVD =~ 229 and FAD =~ 9.39 in the MM-Diffusion paper). One clear difference is that we used implementations of FVD and FAD widely used in the community (see appendix A.3 for more details), whereas MM-diffusion uses their own implementation for computing these metrics. We consider that our evaluation is fair as we used the same implementation of evaluators to compare all methods.

[1] https://github.com/researchmm/MM-Diffusion/issues/17
[2] "Seeing and hearing: Open-domain visual-audio generation with domain visual-audio generation with diffusion latent aligners," CVPR 2024

Comparison between "Seeing and Hearing" (Question 4)

Since they do not release the code for joint video and audio generation in their official repository [1], we cannot directly compare their method and ours. Therefore, here, we compare our method with Seeing and Hearing (SaH) from the perspective of theoretical difference and clarify the benefit of our method by using the reported numbers in their paper.

Our method differs from SaH by explicitly guiding base models' generation process toward a specific joint distribution. Specifically, we formulate our method to bridge the gap between an independent distribution $p(x)p(y)$ and a joint distribution $p(x, y)$. We assume that the training data is a sample of the target distribution $p(x, y)$ and derive the loss function to match the distribution learned by the base models with the target one. Therefore, the generation with our proposed method can be considered sampling from the target distribution $p(x, y)$. In contrast, SaH guides the generated samples to be closed in the latent space of a pre-trained foundation model, ImageBind. Although they successfully improve the audio and video alignment, the distribution of its generated samples is not explicitly defined. This difference makes our method preferable regarding two aspects: i) effectiveness as a guidance module for a joint generation model and ii) customization of base models for the specific dataset.

From the perspective of i), our guidance can be applied to the joint generation model to boost its performance. As shown in Table 2, our method successfully improves both FAD and FVD of MM-Diffusion. In contrast, SaH struggles to improve both metrics of MM-Diffusion (FAD improves from 7.7 to 6.4 while FVD degrades from 1141 to 1174 as they reported in Table 1 in their paper). Our method provides a similar benefit to the classifier or classifier-free guidance commonly used in conditional generation.

From the perspective of ii), our method successfully guides the same base models toward different joint distributions using several datasets (e.g., Landscape and VGGSound datasets). Tables 3 and 4 demonstrate this success as our method improves the FAD and FVD computed on each target dataset. Note that it is difficult to achieve this improvement by SaH because there is no guarantee that the distribution of generated samples by their method is closed to the target distribution. Our method allows us to customize base models for different datasets to achieve sampling from the specific target distribution.

[1] https://github.com/yzxing87/Seeing-and-Hearing?tab=readme-ov-file#todo

Different (more powerful) base models (Weakness 4)

Compared to AudioLDM/AnimateDiff, Auffusion/VideoCrafter2 are stronger base models, and the results for these models (the bottom two rows in Tables 3 and 4) show the effectiveness of our proposed method even with more powerful base models. Even for those more powerful base models, our proposed method successfully improves both single-modal fidelity and cross-modal score, highlighting our method can be effective even in the challenging setting, in which the base models already achieve good generation results.

Other evaluation metrics for temporal alignment between generated audio and video (Question 1)

Thank you for your question. We computed the AV-Align score [1] to evaluate the temporal alignment between generated audio and video. The tables below show the results.

Our proposed method generates samples with better AV-align scores, indicating that our guidance successfully shifts the distribution of the generated samples towards the target joint distribution. While our proposed method is not explicitly designed to improve a specific type of alignment, it implicitly learns what the joint audio-video pairs should be from the dataset, resulting in better semantic and temporal alignment scores.

Note that this metric compares the onsets in audio and the peaks of optical flow in video, measuring how the object movement and its corresponding sound align in temporal perspective. Although our proposed method improves the AV-Align scores, it might be unsuitable for evaluating ambient sound, such as the landscape dataset. How to evaluate the temporal alignment of sounding videos should be an open issue to be explored.

[1] "Diverse and aligned audio-to-video generation via text-to-video model adaptation," AAAI 2024

Different discriminator structures or larger trainable parameters (Question 2)

Thank you for your question.

Larger trainable parameters

We tested two settings for larger trainable parameters: scaling channel size or number of layers. These results are shown in Tables 8 and 9 in the appendix. In short, both settings slightly improve the cross-modal alignment and fidelity.

Different discriminator structures

We additionally tested transformer-based structures for implementing our discriminator and compared them with our default discriminator structure.

In short, we observed performance improvement of all metrics except FVD, which is on par by incorporating a transformer encoder to fuse the extracted audio and video features. The Table below shows the notable results from Table 10 in the updated manuscript (the details about the network architecture are explained in appendix A.8).

From this experiment, we show that a more sophisticated discriminator design would improve the performance of joint generation by our proposed method, which would be an interesting direction for future work. We updated our manuscript to include the results and discussion of this ablation study of the discriminator structure in appendix A.8 and Table 10. Please see them for more details.

We appreciate the constructive feedback from the reviewer. We would like to answer the questions and the comments below.

Explanation about adjusting pre-trained modality-specific diffusion models towards a joint generator by training a discriminator (Weakness 1)

We thank the reviewer to point this out. We would like to clarify how our proposed method guides base models toward a joint generator in this response.

Our proposed discriminator distinguishes between samples from an independent distribution $p(x)p(y)$ and ones from a joint distribution $p(x, y)$. Intuitively, the discriminator has knowledge about the difference between non-aligned samples and well-aligned ones. Using the outputs from this discriminator, we can shift the base models' independent generation processes toward a joint generation process.

More specifically, we formulate our guidance to fill the gaps between the score function for a single-modal distribution and that for a joint distribution (Eqs. (7) and (8)). We show that these gaps can be estimated by a single discriminator trained for distinguishing between the samples generated independently by the base models and ones sampled from the real paired datasets (Eqs. (11) and (12)). We argue that the function of our discriminator is just shifting the score computed on a single-modal distribution toward that on a joint distribution, which is obtained by the dataset used for the training. In other words, we do not explicitly design our method to manipulate a specific type of alignment. The type of alignment mainly achieved in our model is determined by both the nature of the real dataset and the inductive bias of the joint discriminator. We train the discriminator on the ground truth audio-video pairs that are aligned both semantically and temporally. With its guidance, the generated samples can be considered sampled from the target joint distribution, resulting in higher semantic or temporal alignment.

Limited improvement of the metric scores for the OOD setting (Weakness 2)

As the reviewer pointed out, we observed slight performance degradation on only a few metrics: FVD on the VGGSound, IB-TV on the VGGSound and the Landscape datasets for AudioLDM/AnimateDiff.

Before discussing these degradations in detail, we argue that our primary focus is the cross-modal alignment score, which measures how well our proposed method integrates base models into a joint generation model. Our proposed method successfully improves cross-modal scores for all settings with a par or even better performance on the single-modal fidelity score except the ones described above. This illustrates that our proposed method effectively shifts the single-modal scores to the joint scores while keeping (or even improving) the performance of each modality.

Then, we would like to discuss the degradation in detail. One possibility causing this degradation is that, as we discussed in our manuscript, the captions added to the videos are biased to describe visual information, and the base audio models struggle to generate high-quality audio with these captions. To compensate for this bias, our model may focus more on aligning the score of audio to the target than the video. The IB-TA scores of AudioLDM / AnimateDiff (A) on both datasets are significantly worse than those of Auffusion / VideoCrafter2 (B), and performance improvement on IB-TA for (A) is more significant than that of (B) (15% and 8% improvements for (A) whereas 5% and 3% for (B) on the same datasets). This indicates that our discriminator focuses more on the audio part, resulting in a slight performance degradation in the IB-TV score. This bias would be mitigated by adopting a more sophisticated captioning strategy or discriminator's network architecture.

The OOD setting performs poorly in generating high-quality video with the audio track (Weakness 3)

Since we propose a method extending base models into a joint generation model, the quality of generated samples strongly depends on the performance of base models. Even though we adopted one of the state-of-the-art models publicly available (e.g., Auffusion and VideoCrafter2), the quality of these samples lags behind closed models, especially in the open domain generation (we used closed-domain datasets that include a less diverse landscape or dancing scene in the IND setting). However, our primary contribution is proposing a novel, computationally efficient way to integrate pre-trained single-modal models into a joint generation model. Our method can be applied to the new state-of-the-art models, as we verified its generalizability with various base models and datasets through experiments. Adopting new state-of-the-art models proposed in the future will improve performance under difficult settings, including OOD.

Thank you very much for your response. Most of my concerns have been mitigated. However, I still have some questions:

1. As mentioned in the response to Weakness1, the proposed method is not designed for alignment and is based on the ground truth audio-video pairs that are aligned both semantically and temporally. But, during the sampling stage, how does the proposed method ensure to generate the audio and video semantically and temporally?
2. For the comparison with MM-Diffusion, can you try the metric implementation from MM-Diffusion on your proposed method to evaluate the generated performance?

Thank you for the follow-up questions. We will answer all the questions.

1. For the comparison to MM-Diffusion, why does your obtained score have a big difference from the one from the original paper, in terms of FAD? It is understood the reproduced score is a little different from the original paper. Did you use the official checkpoint to sample the videos and then apply the evaluation implementation?

Yes, we used the official checkpoints and the official implementation (the generation script) to sample the videos, and then we applied the official evaluation implementation. Regarding FAD, the reported number in the original paper is 9.39 in the Landscape dataset, but our obtained score is 11.11. We are still determining what causes this difference. Note that, as we mentioned in the rebuttal response above, similar differences are reported by the issue on the official repository [1] or the reproduced results in the previous work [2].

[1] https://github.com/researchmm/MM-Diffusion/issues/17
[2] "Seeing and hearing: Open-domain visual-audio generation with domain visual-audio generation with diffusion latent aligners," CVPR 2024

2. As mentioned the paper mainly focuses on the cross-modal alignment by learning from the raw dataset. However, one main challenge of audio-video generation is to ensure a good temporal alignment between generated audio and video. I appreciate that the authors implement the AV-Align score on the baselines with or without the proposed method. It is better to show the alignment performance comparisons of your model with other existing models.

Thank you for your suggestion. We computed the AV-Align score for the existing models to comprehensively compare our method with existing models regarding both semantic and temporal cross-modal alignment. We updated our manuscript to include the explanation and results of this experiment (in appendix A.10 and Table 13).

During the evaluation of AV-Align, we noticed that our previous results were too high. We fixed this problem by adjusting the interval of onset detection to be compatible with TempoToken's configuration and computed AV-align scores for all models. Please refer to appendix A.10.1 in the updated manuscript for more details on the AV-Align computation. The table below compares our method with other existing models regarding cross-modal alignment.

For the AV-Align, "AnimateDiff + AudioLDM + ours" achieves the best score among all models. While it also achieves a better IB-AV score than existing models, the IB-AV score significantly lags behind GT's. Using more sophisticated base models (VideoCrafter2 + Auffusion + ours) significantly improves the IB-AV score, which has a comparable AV-Align score to the GT. Notably, our proposed method improves both AV-Align and IB-AV scores of both base models.

We also observed that using a more sophisticated network structure ("VideoCrafter2 + Auffusion + ours (Res / Frieren)") improves both scores. How to design the discriminator to improve the specific types of alignment would be a possible direction of future work. Additionally, how to create a dataset would be a possible direction to improve the alignment. For example, Diff-Foley proposes using time-shifted audio and video pairs to train audio-visual latent space, which is used for the condition of their generative model, to improve the temporal alignment. In our experiments, we only used generated samples from base models for the fake samples. Adding such fake pairs as simulated temporal misalignment for the fake samples would improve the alignment of generated samples, which we leave for future work.

We also included this discussion in the updated manuscript in appendix A.10.2.

3. Apart from the alignment, the other challenge is to ensure the quality of each generated modality. From the demo (especially for the OOD setting), the visual quality seems to be poor. Maybe the potential reason is that the quality of each generated modality is based on the used modality-specific pre-trained. Can I understand that the proposed method or trainable module does not contribute to the quality of each generated modality?

We respectfully disagree with your understanding. Our proposed method (or trainable module) has the capacity to contribute to the quality of each generated modality within certain limits. The quantitative results (Tables 3 and 4) show that the FVD improves in three out of four settings, and the FAD improves in all four settings. These improvements in the FVD and FAD indicate that the proposed method successfully aligns the distribution of the generated samples with that of the real dataset. Since the quality of the real dataset should be high (or at least higher than the generated samples by the base models), the generated samples with our method have higher quality than those of base models, indicating our method's contribution to the quality of each generated modality. Moreover, the additional subjective evaluations we mentioned in the rebuttal response above also support the audio and video quality improvement, at least against the generated samples from the base models.

On the other hand, we also notice the limitation of our proposed method. As you rightly pointed out, the quality of each generated modality strongly depends on the performance of the base models. Our proposed method guides the generated samples to have a higher density ratio between a joint distribution and independent distribution ($p(x, y) / p(x)p(y)$). Intuitively, our method allows us to pick the samples likely to be from a real joint dataset within (not far from) the range of base models' generation. We introduce a denoise loss to strengthen the guidance effect beyond the distribution gap in the OOD case. However, the base models still contribute a lot to the quality of the generated samples, especially when we try to guide the large base models by a lightweight module with a limited number of parameters. While a lightweight module struggles to improve the quality significantly, one of the benefits of the lightweight module is that we can easily apply our method to any new models released in the future. We expect the single modal generative models to continuously improve by introducing various sophisticated network designs, and combining new SoTA models with our proposed method (with less effort thanks to our model-agnostic formulation) will improve the quality of each generation modality.

Apart from updating base models, one possible way to improve the quality of each modality is the curation of the training dataset. In the OOD setting, we used the LandScape or VGGSound dataset. The LandScape dataset is of high quality but on a small scale, while the VGGSound is large scale but not of high quality enough. Since our method successfully improves the FAD and FVD, using high-quality videos for the training would further improve the quality of each generated modality.

We appreciate the constructive feedback from the reviewer. We would like to answer the questions and the comments below.

Computation cost for training joint discriminator (Weakness 1 & Question 1)

Thank you for the question. In short, our proposed method can be trained around 1.5 to 2 times faster than baselines. We updated the manuscript to add the results of additional experiments about the computational cost for training and inference of our proposed method in appendix A.9, Tables 11 and 12. We measured the computation time for the forward and backward of the base models and our discriminator, and the times for fine-tuning base models without any extension and training of our method were compared. The summarized results are shown below.

Note that we include forward passes through the frozen modules in "Ours training 1step". These results concretely demonstrate the computational efficiency of our proposed method. Please see the updated manuscript for more details. The complete numbers for each part of the computation are shown in Table 11, and a detailed discussion is provided in appendix A.9.

Text condition acts as a weak alignment signal (Weakness 2)

Thank you for your suggestion. We wonder that experiments without text conditions are not practical because the performance of the base model in unconditional generation is too low. We consider that there are three levels of experimental designs in terms of the strength of text condition: i) unconditional generation, ii) weak conditional generation (like label), and iii) conditional generation with captions. However, in the case of i), when we completely removed the text conditions for the base models, the generated samples from the base models were significantly degraded. As reported in many articles[1, 2], diffusion models struggle to generate decent outputs without classifier-free guidance. On the other hand, we tested case ii) in the OOD setting experiments on the VGGSound dataset described in Section 4.3. We used the original labels of the VGGSound dataset, which are simple and do not contain any detailed information specifically describing each video instance (e.g., "dog barking" or "playing hokey").

[1] High-Resolution Image Synthesis with Latent Diffusion Models, CVPR 2022 [2] Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding, NeurIPS 2022

Dear Reviewer F4ji,

With the discussion period ending in less than 8 hours, we would like to ensure we have satisfactorily addressed your concerns, as we believe we have. If there are any remaining questions or uncertainties, please feel free to leave a message before the rebuttal period ends. We would be happy to respond promptly.

Thank you for your time and valuable feedback.

Best regards,
The Authors

We appreciate the constructive feedback from the reviewer. We would like to answer the questions and the comments below.

Generalizability of our proposed method on larger datasets and application to the various base models (Weakness 1 & 2)

While the reviewer rightly pointed out the relatively small size of the AIST++ and Landscape datasets, we would like to emphasize that we also conducted experiments on the VGGSound dataset. The VGGSound dataset is commonly used as a large-scale audiovisual dataset among publicly available ones, which contains around 200 times more videos than the Landscape and AIST++. This dataset serves as a robust validation of our method's applicability to larger datasets.

It is also important to note that we tested our proposed method with not just one, but three base models, namely MMdiffusion, AudioLDM/AnimateDiff and Auffusion/VideoCrafter2. This demonstrates the versatility and applicability of our method across different base models. Please see Section 4.3 for the results of other two models; the results are shown in Tables 3 and 4 in our manuscript. The results of our experiments unequivocally demonstrate that our proposed method successfully enhances the cross-modal alignment score and video and audio fidelity for the various base models. This success is a clear testament to the generalizability and applicability of our method.

Subjective evaluation (Weakness 3)

Thank you for your suggestion. Our extensive quantitative evaluation of various base models and datasets illustrates the effectiveness of our proposed method, whose primary goal is to shift the score of the single modal distribution estimated by base models to be aligned with the score on the joint distribution. On the other hand, we agree that subjective evaluation would provide additional support for our proposed method's visual and aural improvements. We are planning to conduct a user study, but we are not sure whether we can collect enough feedback during this limited rebuttal period. If we successfully collect sufficient responses within the next week, we will provide additional subjective results for this rebuttal. Otherwise, we will continue to conduct a user study and include at least some results for the camera-ready version.

Computation cost (Question 1)

Thank you for the question. In short, our proposed method can be trained around 1.5 to 2 times faster than baselines. We updated the manuscript to add the results of additional experiments about the computational cost for training and inference of our proposed method in appendix A.9, Tables 11 and 12. We measured the computation time for the forward and backward of the base models and our discriminator, and the times for fine-tuning base models without any extension and training of our method were compared. The summarized results are shown below.

These results concretely demonstrate the computational efficiency of our proposed method. Please see the updated manuscript for more details. The complete numbers for each part of the computation are shown in Table 11, and a detailed discussion is provided in appendix A.9.

Longer video generation (Question 2)

As in typical diffusion-based generative models [1], our proposed method can generate longer video and audio by applying auto-regressive generation (i.e., the first several frames are fixed by the ones generated at the previous step, and the only remaining several frames are generated iteratively). We will conduct some experiments about this and hopefully provide results during this rebuttal period.

[1] "Video Diffusion Models," ICLRW 2022

Dear reviewers,

We express our sincere gratitude for your valuable comments, which we consider crucial for improving our work. This is a gentle reminder to reviewers to review our replies. We are eager to engage in an active discussion. Thank you again for your attention and consideration.