Aligning What Matters: Masked Latent Adaptation for Text-to-Audio-Video Generation

We thank Reviewer aSVy for the thoughtful feedback and valuable suggestions. Below, we provide our detailed responses to the questions and comments raised.

---

W1: More qualitative comaprisons with TAVDiffusion

We will include additional visualized results generated by our model and TAVDiffusion in the updated manuscript.

---

W2: Presentation of Section 2 and Mask

We appreciate the reviewer’s feedback on the presentation clarity and agree that clearer exposition would improve accessibility.

Section 2 is intended to provide theoretical grounding for our method, particularly motivating the selective alignment mechanism via a causal latent structure. While this section is not directly operationalized in our implementation, it supports the theoretical analysis in Section 4, where we prove that our masked objective recovers the minimal set of shared latent factors. That said, we acknowledge that this connection could be made more explicit, and we will revise the manuscript to better link Section 2 with the training and alignment mechanism described in Section 3.

Regarding the guidance mask and dual adapters: we agree that Figure 3 currently emphasizes the inference-time generation strategy and underrepresents the full training architecture. To address this, we will update the figure to:

• Explicitly show the two adapter networks $q_V$​ and $q_A$​,
• Depict the dual masking modules $M_V$​ and $M_A​$,
• And annotate that these components are trained during Stage I.

We will also revise the figure caption and method description to clarify the distinct roles of these components and improve overall coherence across sections.

---

W3 & Q1:

The adapter modules $q_V​$ and $q_A$ are implemented as lightweight multilayer perceptrons (MLPs). Their function is to perform learnable reparameterization of the frozen encoder outputs into a shared latent space optimized for cross-modal alignment. Specifically, they help suppress modality-specific noise and enhance semantic factors that are predictive of the other modality.

To analyze the behavior of the video adapter, we conducted both suggested experiments:

• Modified individual dimensions of the adapted video latent and observed the corresponding changes in the generated audio. Qualitative results show that certain dimensions are highly sensitive, affecting acoustic features, indicating that the adapter learns structured and interpretable features.

• We mannually create a set of text prompts designed to generate different videos but similar audio content (e.g., “A brown panda eating bamboo” vs. “An black and white panda eating bamboo”; "A dog barking in the snow" vs. "A dog barking under the sun" ,.etc.), and report the mean cosine similarty (MCS) between feature after adaptation. Our analysis revealed that the masked aligned latents produced by the adapter are more similar (compared to direct aligned latents), suggesting that the masking effectively filters out visual identity and retains audio-relevant semantic actions like “eating bamboo.”

  Alignment	MCS
  w/ Mask	0.88
  wo/ Mask	0.74

Due to rebuttal rules, we are unable to attach the figures here. However, we will include these analyses and corresponding figures in the revised manuscript and appendix. We appreciate the reviewer’s thoughtful feedback, which helped us better characterize and validate the role of the adapter modules.

---

Q2:

Thank you for raising this important point. Our full objective (Eq. 8) includes three components: (1) cross-modal reconstruction loss, (2) masked alignment loss between adapted latents $\tilde{z}_v$ and $\tilde{z}_a​$, and (3) sparsity regularization over the learned masks. Each term serves a distinct role:

• Reconstruction loss ensures that the masked latent retains sufficient predictive information about the other modality
• Masked alignment loss promotes geometric consistency across modalities in the adapted latent space.
• Sparsity loss encourages the masking network to retain only essential dimensions, reducing overfitting to modality-specific noise.

In our experiments, we set the alignment loss weight $\alpha = 0.1$ and sparsity regularization weight $\lambda = 5.0$, based on a small-scale grid search. Empirically, we found that: Reducing $\alpha$ too much weakens the latent coupling, leading to semantic drift between modalities (e.g., misaligned motion and sound). Excessively high $\lambda$ over-prunes the mask. Disabling sparsity results in overly dense masks, which fail to isolate cross-modal signals and slightly reduce performance in AVHScore and CAVP similarity.

We will include a more detailed ablation table in the appendix showing how varying these weights affects both quantitative scores and qualitative outputs. Thank you for pointing out the need to clarify this aspect of our design.

We thank Reviewer p3WX for the thoughtful feedback and valuable suggestions. Below, we provide our detailed responses to the questions and comments raised.

---

W1.1: Details of reconstruction loss

The cross-modal reconstruction losses $\ell_A$ and $\ell_V​$ are computed by reconstructing each modality from the masked latent of the other modality, using the corresponding decoder. Specifically: $\ell_A(x_a, g_A(M_V \odot \tilde{z}_v))$ is the loss between the ground-truth audio spectrogram $x_a​$ and the audio reconstructed from the masked video latent via the audio decoder. We compute this using mean squared error (MSE) on log-Mel spectrograms, which are widely used for audio reconstruction tasks.

$\ell_V(x_v, g_V(M_A \odot \tilde{z}_a))$ is the loss between the ground-truth video frames $x_v$​ and the video reconstructed from the masked audio latent via the video decoder. This is computed as frame-averaged MSE over the decoded video frames.

These reconstruction losses serve a dual purpose: (1) they ensure that the masked latent retains sufficient predictive information about the other modality, and (2) when combined with the sparsity constraint on the masks, they encourage the model to preserve only the minimal set of cross-modally relevant dimensions. We will clarify these implementation details in the revised manuscript for improved transparency.

---

W1.2: Typo on line 198

Thank you for catching this typo. The reference to “Assumptions 4–5” on line 198 is a typo. It should be to "Assumption 4". We will correct this in the revised manuscript.

---

W2: Align notations in Figure 3

We appreciate this helpful suggestion. We will update the Figure 3 in the revised manuscript to annotat key variables such as $z_v$, $z_a$, $M_V$​, and $M_A​$ directly to improve clarity between the text and visual illustration.

---

W3: Derivation or proof for the claims made in Section 4.

Thank you for the suggestion. As noted in the main text, the full derivation and formal proofs supporting the theoretical claims in Section 4 are included in Appendix Section B of the submission (placed in the supplementary material folder). This includes all relevant assumptions, formal definitions (e.g., minimum sufficient latents), and complete proofs of our key results such as:

• Cross-entropy reduction to conditional entropy
• Sufficientness of reconstruction
• Sparsity-induced minimality
• Global block-alignment and recovery

These results establish that our masked reconstruction objective provably identifies the minimal shared latent subspace under standard identifiability assumptions. We will make this connection more explicit in the main paper by adding cross-references to the relevant sections for improved clarity.

---

W4: Audio-video alignment for baseline models

To ensure fairness and consistency, we adopted the standard evaluation setting introduced in TAVDiffusion [1], which includes both alignment-aware models (e.g., TAVDiffusion [1], CoDi [2]) and baseline systems (e.g., Two-Streams, CasC-Diff) for benchmarking. These methods represent the current landscape of T2AV models, covering both joint and independent generation strategies.

Importantly, our framework was evaluated under the same conditions:

• Using a shared audio decoder for independent baselines (Two-Stream, CasC-Diff),
• Applying the same evaluation metrics,
• Generating outputs from identical text prompts and datasets,
• And conducting inference in a cascaded setting where applicable.

We acknowledge that some cascaded models do not perform explicit audio-video alignment. Their inclusion is intentional: they serve as important baselines to highlight the effect of alignment mechanisms. In contrast, methods like TAVDiff and CoDi, which incorporate alignment strategies, offer direct comparisons to our alignment-aware approach.

To further strengthen the evaluation, we have conducted additional experiments incorporating recent alignment-aware models such as JavisDiT [3] and will include these results in the revised manuscript. We will also clearly annotate which models perform alignment to improve transparency in the baseline comparison.

Thus, while some baselines do not include alignment by design, we argue that their inclusion, alongside alignment-aware methods, offers a comprehensive and fair evaluation of our contributions.

---

Q1: Video-first generation strategy

We thank the reviewer for raising this important point. For practical consideration, visual content provides a more stable and structured prior for generating temporally aligned outputs. For example, the act of a drum being struck or a guitar being strummed is visually discrete and temporally grounded, providing strong cues for the expected acoustic outcome. Compared to audio, which may contain overlapping or ambiguous signals, visual representations often offer clearer event boundaries and object-specific dynamics. This structural clarity makes video a more reliable anchor modality when synthesizing aligned audio content. In addition, from a training perspective, finetuning the attention layers of a diffusion-based audio model to allow video-conditioning is substantially more efficient than modifying a large-scale video diffusion model, which further allows rapid adaptation to pretrained single-modal latent generative models.

---

Q2: Temporal consistency

Yes, all $\tilde{z}$ vectors are d-dimensional latent representations. However, to preserve temporal structure, we do not operate on the entire video or audio as a single global representation. As mentioned in the paper, we split both video and audio into aligned time segments (Using the sampling strategy as in [3]), and compute one $\tilde{z} \in \mathbb{R}^d$ per segment. This segment-level representation allows our framework to maintain local temporal consistency within each segment and encourages alignment across modalities in a temporally grounded way. Additionally, during inference, each video segment is independently mapped to an aligned latent that conditions audio generation for that segment, ensuring temporal synchronization between the two modalities across time.

---

Q3: Audio Adapter

$q_A​$ serves the same functional role as $q_V$: it is a learnable adapter module that reparameterizes the output of the frozen audio encoder into a common latent space suitable for alignment. Both $q_V$​ and $q_A$ are lightweight projection networks trained to isolate the cross-modal semantic content. To avoid confusions, we will update the figure in the revised manuscript to explicitly include $q_A$​ alongside $q_V$​, and annotate both as adapter modules to improve consistency and clarity.

---

Q4: Mask Blocker

Yes, the “Mask Blocker” in Figure 3 corresponds to the learnable masking networks $M_V$​ and $M_A$​ introduced in Section 3. These networks produce soft masks that identify the dimensions of the latent representations that are most relevant for cross-modal alignment. As for the inputs: the masking functions are conditioned on both the video latent $\tilde{z}_v$​ and the audio latent $\tilde{z}_a$​, rather than the raw video or spectrogram. This design choice is intentional and discussed in Section 3: conditioning on both modalities enables the model to disambiguate context-dependent relationships, such as when the same video content could be paired with different types of audio (e.g., narration vs. ambient sound). Using both latent representations allows the mask network to more accurately isolate semantically aligned features across modalities. We will revise the figure and caption to reflect this more clearly, specifically, by relabeling the “Mask Blocker” as $M_V$/$M_A$​ and clarifying that it operates on latent embeddings rather than raw inputs.

---

Q5: Joint optimization objective

Thank you for this important question. In our framework, the decoders $g_A$​ and $g_V$ are not updated during Stage I training. They remain frozen and are used solely to compute the reconstruction losses that guide the learning of the adapters $(q_A, q_V​)$ and masking networks $(M_A, M_V​)$. The notation in the loss function was meant to indicate dependency on the decoder outputs, but we agree it may misleadingly suggest that $g_A​$ and $g_V​$ are trainable. We chose to freeze these decoders in order to (1) preserve the pre-learned generative priors of the pretrained diffusion models, and (2) maintain modularity and efficiency by avoiding full end-to-end retraining. We will clarify this point in the revised manuscript by explicitly stating that $g_A​$ and $g_V$​ are fixed during alignment training, and we will revise the loss notation to avoid ambiguity.

---

Q6: Projection network

The projection network $P_\theta$ used in Stage II corresponds to the adapter network $q_V$​ that was trained during Stage I. Specifically, after generating a video using the pretrained video diffusion model, we pass its latent representation $z^t_v$​ through the learned adapter $q_V$​ to obtain the aligned latent $\tilde{z_v​}$, which is then used to condition the audio generation process. We refer to this trained adapter as $P_\theta$ in the inference phase to emphasize its role as a fixed projection function, but it is indeed the same module as $q_V​$ learned during Stage I.

---

Rerfernece:

[1] Mao, Y., Shen, X., Zhang, J., Qin, Z., Zhou, J., Xiang, M., Zhong, Y. and Dai, Y., 2024, October. Tavgbench: Benchmarking text to audible-video generation. In Proceedings of the 32nd ACM International Conference on Multimedia (pp. 6607-6616).

[2] Tang, Z., Yang, Z., Zhu, C., Zeng, M. and Bansal, M., 2023. Any-to-any generation via composable diffusion. Advances in Neural Information Processing Systems, 36, pp.16083-16099.

[3] Luo, S., Yan, C., Hu, C. and Zhao, H., 2023. Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models. Advances in Neural Information Processing Systems, 36, pp.48855-48876.

Dear Reviewer p3WX,

We would like to thank you again for your valuable feedback.

We hope all your concerns have been fully resolved following our responses. If there are any additional issues or clarifications needed, please let us know, and we'll promptly address them before the discussion period concludes.

If our responses have satisfactorily resolved your concerns, we kindly ask if you would consider updating your score accordingly.

Thank you for your time and consideration.

Best regards,

The Authors

We thank Reviewer wB3p for the thoughtful feedback and valuable suggestions. Below, we provide our detailed responses to the questions and comments raised.

---

Q1: Fix superscript "past"

Thank you for pointing this out. We have changed the notation in Equation 1 to use t−1 instead of the superscript “past.

---

Q2: Clarification of Eq. 2

Thank you for the thoughtful question. Equation 2 does not describe the VAE decoding step directly. Rather, it presents a conceptual generative model that reflects the causal structure between latent variables $(Z_V, Z_A, Z_C​)$ and the observed data $(X_V, X_A​)$. Specifically, it formalizes the assumption that the observed data is generated jointly from both modality-specific and shared latent factors.

In this formulation, $Z_V$​, $Z_A$​, and $Z_C$​ represent temporally evolving latent variables, not just their values at $t = 0$. That is, they refer to the full latent trajectories across time, and not a single time-step. The dependence of $X_A$​ on both $Z_A​$ and $Z_C$​ reflects the design choice that audio observations are generated from both audio-specific content $Z_A​$ and cross-modal (shared) content $Z_C$, for example, a barking sound depends on both the identity of the sound (modality-specific) and the event semantics (e.g., the presence of a dog, which is shared).

This abstraction justifies the use of our selective alignment approach, where only the shared component $Z_C​$ should be aligned across modalities. We will clarify this interpretation in the manuscript to avoid conflating it with the VAE decoding implementation.

---

Q3: Term "video diffusion encoder"

Yes, the term “video diffusion encoder” refers to the video encoder component used in latent video diffusion models (typically a 3D VAE) to map raw video frames into a lower-dimensional latent space. We will revise the manuscript to explicitly clarify this in order to avoid confusion.

---

Q4: Indices of dimensions selected by the soft masks

The soft masks in our model output continuous values in the range $[0,1]^d$, acting as attention-like weights over the latent dimensions. While these do not directly define a hard selection, we apply a thresholding operation to obtain binary masks that do. As described in Section 3 - cross-modal reconstruction, we compute sets which contain the indices of latent dimensions where the soft mask values equal 1 (after thresholding). These binary indices then define the selected subset of latent coordinates used in the masked latent representations. We will clarify this distinction and the terminology in the final version to avoid confusion.

---

Q5: VAE decoding process

To clarify, the masking is applied during the training stage only, where it serves to supervise the adapter network to learn which dimensions of the latent space are essential for cross-modal prediction. The actual decoding is not performed using a physically masked latent. Instead, the full latent is used during inference. The masking guides the learning of a projection that emphasizes cross-modal relevance, but does not constrain the decoder input to be literally sparse or zero-padded. This design ensures compatibility with convolutional VAE decoders, which typically expect full latent tensors. We will revise the manuscript to make this clearer.

---

Q6: Mask Blocker Input

Thank you for pointing this out. The masking network operates on the latent representations, not the observed inputs. The label “Mask Blocker” in Figure 3 was intended to refer to the masking network, but we acknowledge that the terminology and input arrows could cause confusion. We will update the figure and caption to accurately reflect that the mask network takes in the adapted latent embeddings (not raw inputs), aligning with the formal description in Section 3.

---

Q7: Relation between Eq. 7 and Eq. 8

Thank you for highlighting this. Equation 7 presents the abstract formulation of our training objective, where $Q_{g_A}​​$ and $Q_{g_V}$ denote the generative distributions parameterized by the respective decoders for audio and video. This notation is used to describe the conceptual framework underpinning our theoretical analysis in Section 4 and Section B in Appendix.

In contrast, Equation 8 appears in Section 3.1, which describes the practical implementation of our method. It introduces concrete instantiations of the reconstruction losses and explicitly includes components like the alignment loss and sparsity penalties. Although both equations describe the same overall objective, the change in notation reflects the shift from an abstract formalism (used for theoretical grounding) to an implementation-level formulation.

We believe that providing theoretical grounding is an important strength of our framework, and this structured exposition, moving from abstract principle to practical implementation, was chosen deliberately. That said, we recognize the potential for confusion and will revise the manuscript with clearer section headings and textual cues to explicitly link the two formulations.

---

Q8: Clarification of unidirectional and bidirectional - (text+video)-to-audio and text-to-audio-video.

The bidirectional alignment formulation described in the main method section serves two purposes: (1) it defines a symmetric training objective that allows the model to identify shared latent structure by cross-reconstructing both modalities, and (2) it provides theoretical grounding for the mask learning mechanism, as developed in Section 4.

We wish to clarify that the unidirectional generation process does not effect mutual or bidirecitional influence between the two modalities, we employ a masked latent alignment mechanism during training. This mechanism encourages the model to learn a shared cross-modal latent space by optimizing for both audio-to-video and video-to-audio reconstruction. As a result, although generation proceeds in a single direction at inference time, the model is trained to encode the mutual semantic structure of both modalities. This training design ensures that relevant information from both audio and video is captured in the shared latent space, thereby preserving semantic alignment without requiring bidirectional generation during deployment.

For practical consideration, visual content provides a more stable and structured prior for generating temporally aligned outputs. For example, the act of a drum being struck or a guitar being strummed is visually discrete and temporally grounded, providing strong cues for the expected acoustic outcome. Compared to audio, which may contain overlapping or ambiguous signals, visual representations often offer clearer event boundaries and object-specific dynamics. This structural clarity makes video a more reliable anchor modality when synthesizing aligned audio content. In addition, from a training perspective, finetuning the attention layers of a diffusion-based audio model to allow video-conditioning is substantially more efficient than modifying a large-scale video diffusion model, which further allows rapid adaptation to pretrained single-modal latent generative models.

That said, we have conducted an extended experiment on VGGSound+ to explore the bidirectional generation setup. We reverse the pipeline for video generation (audio generation pipeline is unchanged): raw matching audio is first generated using a text-conditioned AudioLDM [1] model, then encoded into a latent representation via the audio VAE. This latent is projected into the shared space using audio-to-video adapter. We then fine-tuned the CogVideo [2] architecture to support dual conditioning from both text and the projected audio latent. This allows the video generator to incorporate semantic signals from both modalities during generation.

Our findings show that this bidirectional setup results in improved cross-modal alignment, as measured by both AVHScore and CAVPSim. However, we also observe a slight degradation in video quality. Nonetheless, this result highlights a promising direction for future work, and we are actively exploring ways to refine this dual-conditioning scheme to achieve better trade-offs between alignment and generative fidelity.

---

Q9: Fine-tuning Details

In our framework, the video diffusion model is fine-tuned specifically for the audio-visual (AV) generation task, rather than as a standard text-to-video generator.

Concretely, we introduce a cross-modal latent embedding $z_c$, learned via an audio-to-video adapter trained in Stage 1, which captures audio-visual alignment. During Stage 2, we fine-tune the video diffusion model with LoRA adapters applied to its cross-attention layers, enabling it to interpret this additional conditioning signal. The model is trained using the standard noise prediction loss from the diffusion objective. However, unlike regular text-to-video finetuning, our conditioning includes both text embeddings and the projected audio-derived latent $z_c$. This dual-conditioning setup allows the video generation process to be influenced by both modalities, making the fine-tuning AV-specific.

---

References:

[1] Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W. and Plumbley, M.D., 2023, July. AudioLDM: Text-to-Audio Generation with Latent Diffusion Models. In International Conference on Machine Learning (pp. 21450-21474). PMLR. [2] Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang, Y., Hong, W., Zhang, X., Feng, G. and Yin, D., CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer. In The Thirteenth International Conference on Learning Representations.

2. Providing more implementation details in the paper

We sincerely appreciate this valuable suggestion. The reviewer is absolutely right that including more practical details about the fine-tuning process would further improve the clarity of our work.

While our theoretical analysis of masking was intended to motivate the selective alignment strategy, we recognize that readers may find the applied components, particularly the fine-tuning setup, more actionable for building upon our approach.

Following your comments, in the revised manuscript we will:

• Explicitly state in Section 3.1 that the video and audio diffusion models are fine-tuned using LoRA, and specify which layers are targeted.

• Move some implementation-specific details from the Appendix to the main text, particularly those related to LoRA-based tuning and adapter interaction.

• Clarify the distinction between Stage - alignment training and Stage - fine-tuning for generation so that the reader understands how each stage contributes to final output quality.

• Optionally condense part of the theoretical justification in the main text with a summary and pointer to the Appendix, making room for more practical insights in the method section.

We are grateful for this actionable feedback and believe these revisions will make our paper more accessible and useful to the community.

---

3. Change Two-stage into Three-stage for clarity

We thank the reviewer for this accurate and helpful observation. You are absolutely right. Our current terminology conflates the fine-tuning of diffusion models with inference, which can obscure the actual structure of our method. Following your comments, we will describe the process is consists of three distinct stages:

• Masked Latent Adaptation (Stage I): Learning cross-modal alignment via adapter and masking modules.

• Model Fine-tuning (Stage II): Fine-tuning the pretrained video and audio diffusion models using aligned latents to improve temporal and semantic consistency.

• Multimodal Inference (Stage III): Generating video from text, projecting it via the trained adapter, and generating audio conditioned on both the text and aligned video latent.

We appreciate the reviewer for highlighting this conceptual oversight, and we agree that revising this presentation will significantly improve clarity. Therefore, we will make the following revisions:

• Revise Figure 3 and its caption to reflect a three-stage framework. We will clearly separate:

  • Stage I: Masked Latent Adaptation

  • Stage II: LoRA-based Fine-tuning

  • Stage III: Cascaded Inference

• Update Section 3 heading structure to introduce each of the three stages explicitly, renaming “Stage II: Cascaded Diffusion Inference” to “Stage II: Diffusion Model Fine-tuning” and introducing “Stage III: Inference” as a distinct subsection.

• Add implementation details in Stage II describing the LoRA fine-tuning setup, including which model layers are adapted and why LoRA is chosen for parameter efficiency and modularity.

• Update the Abstract and Introduction to briefly summarize the framework as a three-stage pipeline rather than a two-stage one.

• Clarify evaluation timelines and training scope in the Experiments section, indicating which stages involve training and which are strictly for inference.

These changes will ensure that the conceptual flow of our method is transparent and aligned with the actual implementation. We thank the reviewer again for this important clarification.

---

We sincerely thank reviewer wB3p for your thoughtful comments and constructive engagement in the discussion. We hope that our responses and planned manuscript improvements have fully clarified the raised concerns and provided deeper insight into the design, implementation, and evaluation of our framework.

Could you please let us know if you have any remaining questions, we are more than happy to continue the dialogue.

Dear Reviewer wB3p,

Thank you for engaging in thoughtful discussion with us. We are glad to have addressed many of your concerns and remain fully open to further clarification or dialogue to resolve any remaining issues.

Below are our responses to your remaining questions:

---

1. Further clarification on cross-modal reconstruction losses computation.

Thank you for careful reading and raising this important question. As we have established, the core objective of Stage I is training adapters to learn a minimal and interpretable subspace of each modality's latent representation that is predictive of the other modality. This is achieved by using masking networks to identify and retain only those latent dimensions that are cross-modally informative.

We denote:

• $\tilde{z}_v = q_V(\hat{z}_v)$, the adapted video latent.

• $\tilde{z}_a = q_A(\hat{z}_a)$, the adapted audio latent.

• $M_V, M_A \in [0, 1]^d$, the learnable masks applied to these latents.

Then in Equation 5, we define the masked latents with clearer notations:

• $\tilde{z}_v^{\text{masked}} = M_V(\tilde{z}_v, \tilde{z}_a) \odot \tilde{z}_v$,

• $\tilde{z}_a^{\text{masked}} = M_A(\tilde{z}_v, \tilde{z}_a) \odot \tilde{z}_a$.

where $\odot$ donates element-wise multiplication.

In Equation 6, these masked latents are fed into the decoders:

• $\hat{x}_a = g_A(\tilde{z}_v^{\text{masked}})$, to reconstruct the audio from the video,

• $\hat{x}_v = g_V(\tilde{z}_a^{\text{masked}})$, to reconstruct the video from the audio.

The reviewer is correct that masked latents are indeed passed into the decoder to compute the reconstruction losses. We apologies for any confusion arose from our earlier phrasing, which suggested otherwise. To further clarify:

• During training, the masked latents $\tilde{z}_v^{\text{masked}}$ and $\tilde{z}_a^{\text{masked}}$ are directly passed into the decoders $g_A$ and $g_V$, respectively.

• The decoders are pretrained and frozen, they expect full-length vectors, so we multiply the latent by a soft or binary mask, but do not zero out or truncate the tensor structure itself.

• The masking operation $\odot$ retains the original dimensionality, but suppresses unselected dimensions via multiplication with mask values in $[0, 1]$ (or ${0,1}$ in the binary case after thresholding). This allows decoding to proceed without violating the input expectations of the pretrained VAEs.

Then, as shown in Equation 8, the reconstruction terms in the total loss are:

• $\ell_A(x_a, g_A(M_V \odot \tilde{z}_v))$, computed as MSE on log-Mel spectrograms,

• $\ell_V(x_v, g_V(M_A \odot \tilde{z}_a))$, computed as frame-averaged pixel-wise MSE over video frames in the segment.

These losses are computed between the original ground-truth modality and the output of the decoder fed with the masked latent from the other modality.

To summarise:

• The masked latent is used as input to the decoder during training.

• Masking is applied via element-wise multiplication but retains full tensor shape, ensuring compatibility with convolutional decoders.

• Reconstruction losses are computed using decoder outputs from masked latents.

To address the reviewer’s concern, we will revise our presentation to make the following changes:

• Revise Section 3.1 to explicitly state that masked latents are directly passed into decoders during training.

• Clarify the meaning of “masked latent” in Equation 5 and in the surrounding text: i.e., masking preserves dimensionality via element-wise multiplication.

• Clarify confusions around our earlier response (“decoding is not performed on physically masked latent”) to more accurately reflect the use of masked-but-full-shape latents.

• Add explicit notes in the Implementation Details (Appendix C) describing the structure of the masked latents and how compatibility with convolutional VAEs is maintained.

• Clarify the decoder loss terms in Eq. 8, including what modality is being reconstructed, what is being used to condition the decoder, and what loss function is applied.

We hope this clears up the ambiguity, and we thank the reviewer again for the opportunity to clarify this key component of our method.

Dear Reviewer wB3p,

Thank you again for your valuable feedback.

We hope you've had a chance to review our responses. If any questions or concerns remain, please let us know, and we'll promptly address them before the discussion period concludes.

If our responses have satisfactorily resolved your concerns, we kindly ask if you would consider updating your score accordingly.

Thank you for your time and consideration.

The Authors

We thank Reviewer iA4m for the thoughtful feedback and valuable suggestions, which have greatly helped us improve the clarity and depth of our work. Following are the responses to the questions and comments.

---

Q1: Current one-way design and potential bidirectional extension.

We thank the reviewer for raising this important point regarding the choice of a one-way relationship between video and audio in our framework.

We wish to first clarify that the unidirectional generation process does not effect mutual influence between the two modalities, we employ a masked latent alignment mechanism during training. This mechanism encourages the model to learn a shared cross-modal latent space by optimizing for both audio-to-video and video-to-audio reconstruction. As a result, although generation proceeds in a single direction at inference time, the model is trained to encode the mutual semantic structure of both modalities. This training design ensures that relevant information from both audio and video is captured in the shared latent space, thereby preserving semantic alignment without requiring bidirectional generation during deployment.

For practical consideration, visual content provides a more stable and structured prior for generating temporally aligned outputs. For example, the act of a drum being struck or a guitar being strummed is visually discrete and temporally grounded, providing strong cues for the expected acoustic outcome. Compared to audio, which may contain overlapping or ambiguous signals, visual representations often offer clearer event boundaries and object-specific dynamics. This structural clarity makes video a more reliable anchor modality when synthesizing aligned audio content. In addition, from a training perspective, finetuning the attention layers of a diffusion-based audio model to allow video-conditioning is substantially more efficient than modifying a large-scale video diffusion model, which further allows rapid adaptation to pretrained single-modal latent generative models.

That said, we appreciate the reviewer’s suggestion and have conducted an extended experiment on VGGSound+ to explore the bidirectional generation setup. In this setting, we reverse the pipeline for video generation (audio generation pipeline is unchanged): raw matching audio is first generated using a text-conditioned AudioLDM [1] model, then encoded into a latent representation via the audio VAE. This latent is projected into the shared space using a lightweight audio-to-video adapter. We then fine-tuned the CogVideo [2] architecture by Lora-tuning its cross-attention layers to support dual conditioning from both text and the projected audio latent. This allows the video generator to incorporate semantic signals from both modalities during generation.

Our findings show that this bidirectional setup results in improved cross-modal alignment, as measured by both AVHScore and CAVPSim. However, we also observe a slight degradation in video quality, likely due to the noisier and less temporally structured nature of audio signals being used as primary conditioning input. Nonetheless, this result highlights a promising direction for future work, and we are actively exploring ways to refine this dual-conditioning scheme to achieve better trade-offs between alignment and generative fidelity.

---

Q2: Additional Baselines for T2AV and V2A Comparisons

Thank you for this important and detailed feedback.

Regarding the use of CogVideoX [1] for our framework versus AnimateDiff [3] in some baselines: our primary goal was to demonstrate that our proposed alignment method improves audio-visual consistency regardless of the underlying video backbone. We adopted CogVideoX [1] to ensure high-fidelity video generation in our main method, and used AnimateDiff as baselines to remain consistent with prior works such as TAVDiffusion [4]. However, we acknowledge that this discrepancy introduces a potential source of variance. To address this, we have conducted control experiments with corresponding T2V+T2A baselines to isolate the impact of framework design choices. Specifically, we construct the Two-Streams and CasC-Diff baselines using the same fine-tuned CogVideoX as the video backbone.

The results demonstrate that the two-stream approach suffers from poor audio-visual alignment, while the vanilla cascade approach exhibits degradation in the quality of the generated audio conditioned on the generated video.

Regarding the selected baselines, TAVDiffusion [4] and SeeingHearing [5] were both released in 2024 for T2AV and A2V generation. However, we recognize the importance of comparing against the most recent models. As suggested, we will include our concurrent work, JavisDiT [6], under a zero-shot setting on both the VGGSound+ and AudioCaps datasets. In addition, we compare the video-to-audio generative quality with Diff-Foley [7].

In summary, to address your concerns regarding evaluation, we:

• Add direct comparisons with JavisDiT in the T2AV setting, the results will be reported once ready.
• Incorporate the recent V2A method, Diff-Foley.
• Include the corresponding T2V+T2A results.
• Clarify that our improvements primarily stem from enhanced cross-modal alignment rather than from superior unimodal generation models.

---

Q3: Ablation Study on Direct Alignment and Implementation Details

Table 3 in our manuscript reports the effect of bypassing Masked Latent Adaptation and directly training the adapter only. Our experimental results demonstrate that omitting the masking strategy significantly degrades audio generation quality and negatively impacts both the AVH score and CAVP similarity metrics. These findings underline the essential role of the masking strategy in effectively aligning the multimodal latents, thus justifying our proposed two-stage design. We will ensure this analysis and the corresponding insights are explicitly discussed in the revised manuscript.

Regarding implementation details, our framework is built around a two-stage design. In the first stage, we train lightweight adapters to align latent representations between modalities. During this alignment stage, all components of the diffusion models, including the video and audio generators as well as their associated VAE encoders and decoders are kept frozen. The adapters are trained using the alignment objective introduced in Eq. 8 (Depending on the ablation, the input of mask varies. For direct alignment, the masks are set to all 1 to allow direct aligning, the sparsity losses become constant), to project one modality’s latent into the latent space of the other.

Once the adapters are trained, we proceed to the second stage, where we enable conditional generation based on the aligned representations. To make the diffusion model responsive to the new conditioning signals that include both text and projected audio/video features, we apply LoRA-based fine-tuning only to the cross-attention layers of the target diffusion model. All other parameters remain frozen during this stage as well. This selective fine-tuning ensures efficient adaptation while preserving the pretrained capabilities of the base models.

In the specific case of the ablation studies shown in Table 3, each row corresponds to a different alignment strategy applied to the adapter. For all these comparisons, the downstream diffusion models are held fixed after the initial LoRA-based tuning. That is, we evaluate each adapter variant in a zero-shot fashion without further training the diffusion model. This setup isolates the impact of the alignment strategy and allows for a clean comparison of adapter effectiveness.

---

Q4: Presentation of Formula 3

Thank you for pointing this out regarding Formula 3. We will adopt a distinct notation for this term in the revised manuscript to clearly distinguish its transformed representation.

---

References:

[1] Liu, H., et al. AudioLDM: Text-to-Audio Generation with Latent Diffusion Models. In International Conference on Machine Learning.

[2] Yang, Z., et al. CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer. In The Thirteenth International Conference on Learning Representations.

[3] Guo, Y., et al. AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning. In The Twelfth International Conference on Learning Representations.

[4] Mao, Y., et al. Tavgbench: Benchmarking text to audible-video generation. In Proceedings of the 32nd ACM International Conference on Multimedia.

[5] Xing, Y., et al. "Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[6] Liu, K., et al. "Javisdit: Joint audio-video diffusion transformer with hierarchical spatio-temporal prior synchronization." arXiv preprint arXiv:2503.23377 (2025).

[7] Luo, S., et al. "Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models." Advances in Neural Information Processing Systems 36 (2023).

Dear Reviewer iA4m,

We would like to thank you again for your valuable feedback.

We hope all your concerns have been fully resolved following our responses. If there are any additional issues or clarifications needed, please let us know, and we'll promptly address them before the discussion period concludes.

If our responses have satisfactorily resolved your concerns, we kindly ask if you would consider updating your score accordingly.

Thank you for your time and consideration.

Best regards, The Authors

Thanks for the responses. Most of my concerns have been addressed.

Overall, I think this article is acceptable. Consequently, I decide to raise my score. Note that the related issues and discusses are need to be clarified in the final version.

Besides, I hope the author will public the source code of this work to facilitate the development of the T2AV community.