Weakness #1: Method Innovation

In addition to constructing latent spaces, another innovation of this paper is addressing the inherent differences between two modalities for consistent generation. Previous works on constructing latent space mostly focus on single-modal generation. In contrast to prior works, we have to consider a broader range of issues , such as unifying data representations and bridging the semantic gap between different modalities. In this work, we tackle these issues and achieve exciting cross-modal consistency. Our ablation studies also demonstrate the importance of establishing cross-modal correlations when constructing latent features for the multi-modal generation task.

Weakness #2: Open-domain Experiments

Thank you for your suggestion. Please refer to the official comment Q1.

Weakness #3: Long Video Generation

There is no need to violently increase the length of sounding video in our Multi-modal auto-encoder to synthesize long videos. Instead, we can synthesize long videos in an auto-regressive manner like prior works [1][2], which synthesize latents of subsequent video clips based on that of previous synthesized video clips. In practice, to extend our MM-LDM to long video generation, we utilize a condition encoder that takes the previous sounding video clip as input to guide the MM-LDM to synthesize subsequent sounding video clips. We have presented our new proposed approach and long sounding video samples in Appendix 1.6, where all audios can be displayed using Adobe Acrobat PDF Reader by clicking the figures. Please refer to our new PDF for details.

[1] Yu S, Sohn K, Kim S, et al. Video probabilistic diffusion models in projected latent space. CVPR 2023: 18456-18466. [2] He Y, Yang T, Zhang Y, et al. Latent video diffusion models for high-fidelity long video generation[J]. arXiv preprint arXiv:2211.13221, 2023.

Weakness #4: Human Evaluation

Thank you for your comments. Please refer to the official comment Q2.

Question #1: Audio-visual Continuation

Thank you for your comments! It is an interesting task and we incorporate it when extending our MM-LDM for long video generation. Please refer to Appendix 1.6 in our new PDF for details.

Question #2: Ablation Study on Loss Weights

Thank you for your question. Following prior work [1], we consistently utilized an adversarial learning weight of 0.1 in all experiments. In addition, we experimented with four configurations of loss weights, each obtained after a 20-epoch run (approximately 16K steps) on the AIST++ dataset. Across different settings, we assess metrics such as FVD, KVD, and FAD for samples at resolutions of 64 and 256. As reported in the following table, the performance of configuration #3 outperforms that of other configurations at both resolutions. Thus, we select configuration #3 as the final configuration used in the paper. [1] Sun M, Wang W, Zhu X, et al. MOSO: Decomposing MOtion, Scene and Object for Video. CVPR 2023.

Weakness #1: Comparison with Single-modal VAEs

Thank you for your suggestion. As indicated in the ablation studies "Our base autoencoder independently decodes signals for each modality based on the respective spatial information from perceptual latents.", we have evaluated the performance of our multi-modal auto-encoder that removes all cross-modal correlations, which brings a significant performance drop. The only difference between the multi-modal auto-encoder used in the first ablation experiment with separate single-modal auto-encoders is that the signal decoder of two modalities are shared in our multi-modal auto-encoder due to computation limits. For further exploration, we separately train two single-modal auto-encoders under the same settings and report their results in the following table. It can be seen that the multi-modal auto-encoder outperforms two single-modal auto-encoders in all evaluation metrics at both $64^2$ and $256^2$, demonstrating the necessity of cross-modal correlations.

Weakness #2: Requiring Human Evaluation

Synthesized videos on both AIST++ and Landscape datasets have been posted in the web (https://anonymouss765.github.io/MM-LDM). We have present the human evaluation results in the official comment Q2, please refer to it for details.

Weakness #3: Requiring Open-domain experiments

Please refer to the official comments Q1.

Question #1: MM-Diffusion Cross-modal Generation Performance

Thank you for bringing this issue to our attention. Based on the checkpoints released by MM-Diffusion, we re-measured its performance under V2A and A2V settings on the Landscape and AIST++ datasets. The results are reported in the following table, and we have modified the correlated part in our paper. The cross-modal conditional generation results of MM-Diffusion are better than their single-modal generation results. Nonetheless, our MM-LDM still outperforms MM-Diffusion by a large margin, demonstrating the necessity of establishing perceptual latent spaces for high-resolution sounding video generation.

Question #2:

Thank you for your comment. We have corrected this typo in our paper.

Question #3: The Necessity of Weight Initialization

In this study, we reformulate the task of signal decoding as conditional generation, i.e. synthesizing video frames and audio images condition on their 2D perceptual latents. As depicted in Fig. 3(b), we employ the diffusion UNet in image diffusion methods to be our generative backbone. Based on this design, the objective of a diffusion model is integrated into the auto-encoder training loss, as in Equ. (3), where L_{MSE} is the noise mean square error loss used to train the diffusion model through \episilon-prediction. Initializing the generative model with a pre-trained image diffusion model significantly reduces the overall training times, as the initialized weights provide a robust and well-trained starting point of the signal decoder. To be more convincing, we conduct a simple comparison between auto-encoders with and without the weight initialization on the AIST++ dataset with a total of 16K steps, and report the results in the following table. It can be seen that the auto-encoder with weight initialization obtains significantly better decoding performance, demonstrating the necessity of weight initialization.

Weakness #1: Reclaim the conditional generation approach

Thank you for your valuable suggestion. We have modified our presentation of this approach in our newly uploaded PDF.

Weakness #2: Check Visual Samples of MM-Diffusion

Thank you for bringing this issue to our attention! After carefully checking, we identified a flow in the released MM-Diffusion script. In particular, their two-stage synthesis script inconsistently saves mp4 videos from after super-resolution and png video frames from before super-resolution. This inconsistency impacts the visual quality of the results displayed in Fig. 4, since we directly paste those video frames when drawing Fig. 4. We sincerely apologize for our oversight, and we have rectified this figure in the current paper!

Weakness #1

We are sorry for making you confused and mis-understanding our paper. We find the key divergence lies in how image diffusion models act as signal decoders. In particular, an image diffusion model can perform the signal decoding task (i.e. decoding signals from features) through conditional generation (i.e. synthesizing signals under strong conditions). A similar approach is also adopted in a contemporary work GLOBER [2], as specified in the "Video Decoding" paragraph: "We view the synthesis of video frames as a conditional diffusion-based generation task". Notably, our MM-LDM focuses on the audio-video multi-modal generation and proposes multiple strategies to overcome the inherent differences between the two modalities, which are different from GLOBER. We also provide detailed explanations for each question, please refer to Questions 1, 2, and 4 below.

[2] Sun M, Wang W, Qin Z, et al. GLOBER: Coherent Non-autoregressive Video Generation via GLOBal Guided Video DecodER. NeurIPS 2023.

Weakness #2

Thanks for your comment. As explained in Equ. (5), the terms $n_a^t$ and $n_v^t$ represent noise features sampled to corrupt the real audio and video signals during the forward diffusion process. Notably, these noise features serve as our training labels rather than outputs of the model. Conversely, $n_a(\*; \theta)$ and $n_v(\*; \theta)$ signify the predicted audio and video noise features generated by our multi-modal latent diffusion model, as specified in Equ. (7), where $\theta$ represents the modal parameters. We did not put the sign \theta as the subscript of n since the subscript of n is occupied by modality indicators such as a and v. Besides this, our notations align with previous works like Equ. (2) in LDM[3]. According to your comments, we will adopt $\tilde{n}^a_{\theta}(\*)$ and $\tilde{n}^v_{\theta}(\*)$ to denote the predicted audio and video noise features to eliminate confusion.

[3] Rombach R, Blattmann A, Lorenz D, et al. High-resolution image synthesis with latent diffusion models. CVPR 2022.

Weakness #3

Due to limited space, we have put more details about the training details and model settings for the multi-modal latent diffusion models (i.e. the Video Generator) in Appendix 1.4. This is indicated in the concluding sentence of the "Implementation Details" paragraph in Section 4.1. Please refer to the appendix for details.

Question #1

Yes, the diffusion unet used in the image diffusion model is the main component of our signal decoder. The image diffusion model can perform the signal decoding task through conditional generation. To be specific, we reformulate the task of signal decoding (i.e. reconstruct the video frames and audio images based on their 2D perceptual latents) into synthesizing video frames and audio images condition on their 2D perceptual latents. To reduce stochasticity and obtain fidelity, the perceptual latent is mapped into multi-scale sub-features to provide strong guidance, as depicted in Figure 3(b). Moreover, the incorporation of other conditions such as modality and class embeddings also truncates the sampling space of the diffusion model and provides valuable instructions.

Question #2

Figure 3(b) depicts the denoising process of signal decoders, which synthesize video and audio signals based on multiple conditions, and T denotes the total diffusion steps used in this process. As specified in question #1, our signal decoder reformulates the signal decoding task into conditional generation, and utilizes a diffusion unet as the generative backbone.

Question #3

During each training step, we obtain a batch of sounding video clips with the batch size being B. Based on this batch of clips, we can obtain a batch of audio features $s_a^i, i=\{1,2,...,B\}$ and a batch of video features $s_v^i, i=\{1,2,...,B\}$. Obviously, $s_a^i$ and $s_v^i$ belong to the same clip, and $s_a^i$ and $s_v^j$, $i \neq j$, belong to different clips. As demonstrated in Equ. (1), we construct positive samples with paired audio-video features that belong to the same clip, and negative samples with unpaired audio-video features that belong to different clips. During implementation, audio-video features with the same batch indexes, i.e. matched pairs, are our positive samples.

Question #4

As specified in Question #1, our signal decoder synthesizes video frames and audio images based on multiple conditions using a diffusion unet, where \epsilon-prediction loss is a common loss used to train diffusion models. As specified in Equ. (3), the \epsilon-prediction loss (i.e. L_MSE) is the primary component of the autoencoder training loss.