Towards General Modality Translation with Contrastive and Predictive Latent Diffusion Bridge

We sincerely thank the reviewer for their thoughtful feedback and for acknowledging the significance of our problem, our initial efforts, and the clear writing of the paper. We have addressed each question and concern raised below and are happy to provide any additional clarification if needed. We apologize for abbreviating some of the reviewer comment references due to space constraints in the rebuttal.

The key motivation of this paper is to extend DDBM to handle general modality pairs with differing dimensions. To achieve this, it is natural to extend it into latent space using an encoder-decoder architecture. However, a prior work DPBridge [21] has implemented a similar idea and shows the best performance on multiple tasks, even compared to these state-of-the-art methods in each domain. Hence, what differentiates the proposed method from this DPBridge [21]?

Thank you for the opportunity to clarify the distinctions between our framework and DPBridge and improve our work's clarity. We will incorporate the below discussion in the new revision's related work.

It is important to note that DPBridge is not a general multimodal translation framework under our problem definition in Section 4.1. It was specifically designed for image-to-image translation and is based on the Stable Diffusion (SD) model, which is restricted to the image domain. As such, it cannot be applied to the broader range of tasks we address, including 3D voxel prediction and our newly introduced audio-to-image and image-to-audio translation tasks (see our first response to Reviewer 8Cc5 for more details). Consequently, a direct comparison is not feasible. Beyond this, our method introduces several fundamental differences. First, we propose a composition of loss functions, combining InfoNCE (contrastive), predictive, and standard diffusion bridge losses, and we conduct a thorough ablation study to evaluate their individual and joint effects (see Table 6 in the appendix). DPBridge, in contrast, does not explore alignment-based losses. Second, DPBridge employs a pre-trained Stable Diffusion architecture as its backbone, which limits its generality. In contrast, we design and validate a modality-agnostic architecture specifically tailored to support diverse modality pairs without sacrificing performance. Additionally, while DPBridge relies solely on fine-tuning SD (with all other components frozen), our framework trains all components from scratch, and we are the first to explore various training strategies for building a general-purpose multimodal bridge model, extending the scope of our work beyond architectural and loss design. Finally, it is worth noting that SD v2.1, which DPBridge relies on, was pre-trained on hundreds of millions of images. As shown in their ablation study (Table 3 in their paper), their method performance drops significantly without the use of this foundational model, falling well below state-of-the-art levels.

The second contribution is to implement contrastive loss to enhance semantic coherence across domains...

Thank you for raising this important point and helping us improve the paper quality. We do include such an extensive evaluation in our work, as shown in Table 6. However, we appreciate you bringing to our attention that it was not clearly referenced from the main paper. To address this, we have revised the final submission to move this table into the main paper, taking advantage of the additional space now permitted and in accordance with your suggestion. In this evaluation, we present a quantitative comparison of the four loss combinations discussed in the paper. The results show that incorporating the InfoNCE contrastive loss into the training protocol significantly enhances model performance.

In Line 236, the authors claimed "Inspired by ... we design a denoiser that is better ...". However, the results of DiT on all tasks are worse than UNet in Table 2. What's the benefit of the new architecture?

We apologize for the lack of clarity in our original explanation and thank you for helping us improve the presentation of our contribution. We would like to emphasize that the denoiser architecture used in our framework is not a DiT. Instead, we propose an improved architecture tailored for general modality translation tasks, drawing inspiration from natural language translation. Specifically, we employ an encoder-decoder Transformer architecture as our denoising network—unlike DiT, which is based on a decoder-only Transformer design. In addition to this architectural shift, we integrate several components specifically designed to enhance performance in the multimodal setting such as spatial embedding and mask token. As demonstrated in our ablation study (Table 2), this architecture significantly outperforms the commonly used U-Net backbone. Moreover, Table 2 also shows that our denoiser surpasses a standard DiT in modality translation tasks. We have revised Section 4.5 of the paper to include this clarification, in light of this valuable feedback.

In Line 244, "unlike the original UNet..., our model consumes this memory using the cross-attention...". Such an operation has been employed in DPBridge [21].

In the DPBridge framework, the authors do not use the cross-attention feature of the UNet, which is reserved for text-prompt conditioning in stable diffusion. As the authors of DPBridge discuss in their paper, "As DPBridge is an image-conditioned generative framework, we employ null-text embedding for all cross-attention layers to eliminate the impact of text prompts," suggesting that the application of any cross-attention layer in their framework has been nullified. The target modality information is concatenated with the source modality at the input of the UNet, along the channels dimension. Contrastingly, since our model uses a transformer encoder and decoder, we use cross-attention to condition our modality translation.

In Line 210, "Unlike cycle-consistency losses..., our formulation does not require convertibility or backward mapping...". The cycle-consistency losses are designed for unpaired data. However, the proposed architecture is trained with paired data.

We would like to clarify that our intention was simply to convey that our method does not require a full cycle starting from the diffusion bridge and returning back. This was mentioned as a minor comment to help provide additional intuition for the reader, and was not intended as a comparison to unpaired data methods. We understand how this phrasing may have caused confusion. To address the reviewer’s concern and avoid any further misunderstanding, we will remove this line in the revised version of the paper.

To explore the new architecture, it is reasonable to ...

We appreciate the opportunity to clarify our position on this point. While we agree that comparisons to state-of-the-art methods can be informative, we would like to emphasize that such comparisons are not entirely fair to our framework. Our model is designed to be general and capable of handling a wide range of multimodal translation tasks. In contrast, state-of-the-art methods in the benchmarks we evaluate on are highly specialized, incorporating strong inductive biases tailored to the specific data domain through architectural choices, loss functions, and training procedures.

Nonetheless, we agree that including such comparisons can help convey the broader context and performance trade-offs. Therefore, in response to your suggestion, we have added the results of a task-specific state-of-the-art method [1] to the quantitative evaluation table for the Multiview-to-3D Shape Generation task (highlighted in gray). Additionally, for the Zero-Shot Low-to-High Resolution Generation task, we already include a comparison with DiWa. Finally, we have also added comparisons to relevant baselines for the new image-to-audio and audio-to-image tasks introduced in this work. Thank you again for raising this point and helping us improve the completeness of our benchmarking.

[1] Dan Wang, et al - Multi-view 3D Reconstruction with Transformer

For visual results in the main paper and supp., the fair comparison to prior works is missing.

We include a visual comparison in Figures 7 and 8 of the appendix. However, we acknowledge that there was no clear reference to this experiment in the main paper. To improve the clarity and reading flow, we have added a reference to this experiment in the revised version of the main text. Thank you for pointing this out.

This work extends DDBM ...

Please see our first comment.

The second contribution is the contrastive loss ...

Thank you for this question. Yes, we use the exactly same hyperparameters for both the basic and ours. Regarding the second point, we include ablations in the appendix; please see our second comment for full explanation.

In figure 1, what's the meaning of the different colors? The denoised output still has noise. How to train the encoder-decoder and diffusion denoiser together in eq.8?

Each color represents a category in the dataset—for example, a car or an airplane. We compute both the reconstruction loss and the predictive loss using the reconstructions of the original latent representation and the denoised latent representation. This setup enables standard backpropagation-based training using a single optimizer, as all components are implemented within a unified PyTorch module.

Results: Similar to the DPBridge, the paper would be ...

As previously addressed, we welcomed the suggestion and have added comparisons with state-of-the-art methods. Unlike DPBridge, which is specifically designed for the tasks it addresses and leverages a powerful pre-trained foundational model for fine-tuning, our framework is intended to be much more general, and thus a direct comparison may be unfair. As discussed earlier, this introduces a natural trade-off between performance and generality.

We thank the reviewer for the thoughtful feedback and for recognizing the novelty of our framework design and the extensiveness of our experiments. Below, we address the questions and concerns raised. We are happy to provide any additional clarification if needed.

The proposed approach is evaluated primarily on vision-based modality translation tasks, it would be interesting to see how well this approach works outside of vision (e.g., text to image).

To directly address this concern, which was also raised by another reviewer, diversify our benchmarking, and strengthen the robustness of our method results, we introduce two new tasks to our model: Image-to-Audio and Audio-to-Image, following the benchmark proposed in [1]. In these tasks, a multimodal translation model receives one modality as input and generates a meaningful representation, which is then used for downstream classification.

We evaluate our method against the SiT variant, which showed the overall second-best performance on other benchmarks, as well as against the current state-of-the-art, which is a task-specific method tailored for audio and image representations. The results are shown in the table below. For implementation and evaluation details, we refer the reader to [1], and we have also added full experimental details, including encoder and decoder architectures, in the appendix of the revised version.

As shown in the table—where higher values indicate better classification accuracy—our method outperforms SiT. However, a performance gap remains when compared to the specialized method in [1], which is expected given its very task-specific nature. Overall, the results below demonstrate the generality of our framework, its superiority among general multimodal translation models and its possible adaptation to more diversified domains.

Table: Performance comparison on cross-modal face-voice tasks.

[1] Seeing Voices and Hearing Faces: Cross-modal biometric matching - A. Nagrani, S. Albanie, A. Zisserman

The bridge model requires that the source and domain share a robust latent space. Ablations with lower quality encoder/decoders for each modality would elucidate the effectiveness of the proposed losses.

Thank you for raising this ablation perspective, which helps improve the understanding of our method. To address this point, we conducted an experiment within the Super-Resolution benchmark evaluating the impact of encoder architecture. Specifically, we examined three variants: (1) a pre-trained autoencoder [1], (2) the same encoder architecture without pre-training, and (3) a simpler encoder consisting of a 5-layer convolutional neural network. We report the results in the table below and will include this experiment in the final revision. Briefly, we observe that both (1) and (2) achieve comparable PSNR scores, suggesting that pre-training provides only marginal benefit in this case. However, variant (3) shows a noticeable drop in performance, indicating that architectural capacity and representation quality play a crucial role in the final outcome. This experiment underscores the importance of encoder quality in our framework.

Table: Comparison of PSNR across different encoder architectures. Values are mean $\pm$ std.

[1] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models

The image super-resolution experiments focus exclusively on faces. It would be interesting to evaluate how well the proposed pipeline generalizes to other domains such as natural images.

We incorporated the DIV2K dataset—a large, diverse collection of high-quality RGB images, into our framework and extracted visual results. It is encouraging to observe that our model successfully performs the super-resolution task, generating high-quality outputs on this dataset. For quantitative evaluation, our method outperforms DiWa, achieving a PSNR score of 28.9 compared to DiWa’s 28.2. These results will be included in the final revision of the paper.

The iterative training protocol significantly improves performance, however, it would be useful to provide more specific details. Is the training time for the iterative procedure significantly longer compared to two-step or end-to-end?

Thank you for raising this important point. For both the End-to-End and Iterative strategies, we used the same total number of training iterations (1,000,000). In practice, we observed that the Iterative strategy is faster in terms of wall-clock time, as it involves fewer gradient computations per iteration. Regarding the Two-Step strategy, we first trained the autoencoders for 100,000 iterations and then trained the diffusion bridge for an additional 1,000,000 iterations. The overall runtime of this variant is comparable to that of the Iterative approach. We will include this discussion and the corresponding runtime comparisons in the revised version of the paper. Thank you again for bringing this to our attention and helping us improve the clarity of our work.

It would be great if the authors could provide more intuition for the MASK token and some analysis for why its inclusion drastically improves performance on some tasks (+2.7 points on CelebA-HQ PSNR).

The MASK token, also known as the context summarization token, is a token added to a lot of common transformer-based architectures that participates in all the attention mechanisms, even through the causal ones. Intuitively, as the name suggests, this token learns to effectively combine all the information from the tokens in the context window and generates a summary embedding token. This token is commonly used when your application needs to use a summary of all the tokens in the context window, such as when using a ViT for classification applications, they pad with a [CLS] token that generates a summary embedding that can be fed into any downstream classifier model. In our method, we use this token to summarize the semantics stored in one of the modalities to effectively transfer them into the other modality using our technique. We have incorporated the above discussion in the final revision to help clarify the motivation.

In the iterative process, are all three losses applied for both reconstruction and bridge alignment steps?

That's a great question. During training, we compute all three losses, but only propagate gradients through the architectural components that are currently being trained. Specifically, when training the encoders and decoders, the bridge parameters remain frozen; conversely, when training the bridge, the encoder and decoder parameters are not updated.

It would be great if the specific DDBM noising method is provided in the background section or appendix for readers who are unfamiliar.*

We will be happy to make this information easier to find in the final revision and elaborate more on the topic in the appendix. The specific DDBM noising and denoising processes are described in Section 3, including the forward SDE in Eq. (1), the reverse-time formulation in Eq. (2), and the corresponding training loss in Eq. (3). We also clarify that we adopt the variance exploding (VE) noise schedule from [68] (lines 118–120). However, we will try to include more information that will help the reader to gain a better understanding of the DDBM framework.

We thank the reviewer for the constructive feedback and for acknowledging the novelty of our framework. Below, we respond to the questions and concerns raised. We are happy to provide any additional clarification if needed.

[Major] The paper targets to address generic transition tasks between different modalities, and the proposed method, including the training scheme and model architecture, is claimed to be designed for this purpose. However, the benchmark tasks are relatively simple, and especially, don't involve much different modalities but are mostly limited to pixel data like images/masks and voxels, which are still very similar. There is no clue that distinct modalities like language, audio, action, or other unusual sensor data formats are involved. Therefore the capability of the proposed framework is doubted.

To directly address this concern, which was also raised by another reviewer, diversify our benchmarking, and strengthen the robustness of our method results, we introduce two new tasks to our model: Image-to-Audio and Audio-to-Image, following the benchmark proposed in [1]. In these tasks, a multimodal translation model receives one modality as input and generates a meaningful representation, which is then used for downstream classification.

We evaluate our method against the SiT variant, which showed the overall second-best performance on other benchmarks, as well as against the current state-of-the-art, which is a task-specific method tailored for audio and image representations. The results are shown in the table below. For implementation and evaluation details, we refer the reader to [1], and we have also added full experimental details, including encoder and decoder architectures, in the appendix of the revised version.

As shown in the table, where higher values indicate better classification accuracy, our method outperforms SiT. However, a performance gap remains when compared to the specialized method in [1], which is expected given its very task-specific nature. Overall, the results below demonstrate the generality of our framework, its superiority among general multimodal translation models and its possible adaptation to more diversified domains.

Table: Performance comparison on cross-modal face-voice tasks.

[1] Seeing Voices and Hearing Faces: Cross-modal biometric matching - A. Nagrani, S. Albanie, A. Zisserman

The paper writing could be improved. Specifically, there are too many generally descriptive words (e.g. what a component is designed for) of the proposed method, but without actually illustrating what it really is (e.g. how a component is designed), since the abstract, introduction and until Ln 206. Concrete definitions of the proposed method should be summarized and emphasized even before the method section, with various gra

While we provide formal definitions of our method starting in Section 4 and Appendix A.2 (e.g., latent bridge in Eq. 4, full objective in Eq. 10), we agree that an earlier, more concrete summary would improve clarity. We have changed the paper structure to include a brief structured overview of the method (summarizing key components and pointing to Figure 2) earlier in the paper to better guide the reader through the technical core and directly explain components. We are more than happy to include any further clarification and restructuring if needed.

The detailed model architecture related to Sec. 4.5 needs to be presented in the main paper. Many descriptive words could be more concise and concentrated, and the analyses (Ln191-203) and even the background (Sec. 3) could be moved to the Appendix if necessary.

Thank you for pointing out this point, which will help improve the clarity of our paper. Initially, due to space constraints, we omitted the architecture figure related to Section 4.5. However, in light of your comment—and given that the revised submission allows for an additional page—we have restructured the paper to include this figure, which we agree significantly enhances clarity.

Is the bridge model in the predictive loss also being trained under the bridge loss, or is it frozen when is backpropagating?*

It depends on the training strategy. In both the Two-Step and End-to-End strategies, the model is always actively trained throughout the training process (With a small exception in the frist step of the Two-Step where only the encoders/decoers are trained). In the Iterative strategy, the model remains frozen if and only if the encoders are being trained, which occurs in an alternating fashion.

In terms of transitions between different modalities using diffusion models, how does the proposed method relate to and differ from CrossFlow [1] or FlowTok [2], which both directly map between images and texts?*

Thank you for raising this point and giving us the opportunity to clarify how the proposed method relates to and differs from our work. We have added the following discussion to the related work in the final revision to better differentiate our model:

The first fundamental difference between our method and the two referenced works lies in their task specificity. Both of the mentioned methods focus exclusively on a particular latent transition: from text to image. They incorporate task-specific components into the architecture and loss design to facilitate this transition. In contrast, our work aims to develop a general-purpose solution for latent bridging. We investigate several general aspects, including training strategies, which we study in depth.

Another major distinction is in the bridge architecture. While both prior works rely on off-the-shelf DiT architectures, our approach introduces a custom-designed architecture specifically tailored to the more general latent bridging setting. We demonstrate both its efficiency and effectiveness in this context.

Finally, from a theoretical standpoint, our method builds on Denoising Diffusion Bridge Models (DDBMs), a fundamentally different framework from the Flow Matching approach used in the referenced works. These frameworks differ in both their loss formulations and sampling procedures. Although Flow Matching is theoretically valid for bridging any two distributions, similar to DDBMs, the specific implementations in the referenced works are designed for Gaussian-to-data transitions. Indeed, both methods incorporate a Gaussian loss term on one side of the bridge, which is not required in our model.

Interestingly, recent theoretical work such as [3] proposes extensions to Flow Matching that could enable latent bridging without requiring one side to be Gaussian. This presents a compelling direction for future research. However, the current versions of those methods are constrained by this Gaussian constraint, whereas our DDBM-based approach operates without such a limitation.

[3] Stochastic interpolants with data-dependent couplings - Michael S Albergo*, Mark Goldstein*, Nicholas M Boffi, Rajesh Ranganath, Eric Vanden-Eijnden

Thank you once again for engaging in this discussion and providing valuable feedback. We will take your comments into careful consideration and incorporate them into our future work. We appreciate you pointing these aspects out.

We thank the reviewer for the constructive feedback and for recognizing the importance of our motivation, the novelty of our approach, and the extensiveness of our evaluation. Below, we address the concerns and questions raised. We'd be happy to provide further clarification on any additional points, if needed.

Although the results look very promising, the paper does not sufficiently address or analyze the inference time complexity of their bridge model approach compared to other multi-modal shared latent space methodologies that define a separate encoder decoder models. Secondly Sampling complexity of DDBMs are generally higher to achieve accurate interpolations, would be curious to see how is the sample quality affected

Thank you for raising these two points, which help clarify an important aspect of our work.

Regarding the first comment, we've added a Complexity Analysis in Section A.3.2 of the appendix, where we compare and contrast the inference-time complexity of our approach with other methods. We also include a memory analysis. The results demonstrate that our model offers a clear advantage in terms of both model size and runtime efficiency. To make this section more accessible to readers, we've added a reference to it in the main text. We appreciate you pointing this out.

To address the second point, we conducted experiments on the ShapeNet dataset using different sampling step counts, starting from 40 and reducing down to 10. We include the results in the table below. We also experimented with more than 40 steps, but this didn't lead to noticeable improvements. Interestingly, reducing the number of steps to 30 appears to be beneficial; however, below 30 steps, the sample quality begins to degrade significantly, as expected.

Table: Results across different step counts. Values are mean $\pm$ std.

Thank you again for highlighting this issue. We will include these findings in the final revision.

DDPMs inherently support compositional representations especially would be beneficial when performing multi-modal generations. The paper did not address ability to preserve and transfer semantic relationships, compositional representations between modalities and retain usefull properties for image tasks.

Thank you for raising this important point and helping us improve the quality of our paper. It's worth emphasizing that we intentionally designed our benchmark to focus on modality translation between modalities with different dimensionalities. However, we fully agree that including benchmarking tasks with aligned modality dimensions can further enhance the robustness and generalizability of our method. To address this, we leveraged the edges-to-bags task from the DDBM benchmark, which serves as an appropriate evaluation for your suggestion. We tested our framework on this task, analyzing two key aspects: sample quality (FID) and inference performance (runtime in seconds). The results are presented in the table below and will be included in the final revision of the paper.

Briefly, the findings highlight a clear trade-off: while our method remains competitive in terms of quality, it offers more than 2x faster inference speed compared to DDBM.

Table: Comparison of FID and inference time (in seconds).

Inference time is measured on a batch size of 256. We used [1] autoencoder as the encoder-decoder backbone for our method.

[1] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models

In general the training framework involves iteratively learning the auto-encoder/latent spaces jointly with the bridge essentially learning the marginal distributions making it a harder optimization problem.

We agree that jointly learning both the encoder/decoder (which define the latent space marginals) and the diffusion bridge (which assumes fixed marginals during training) introduces a nontrivial optimization challenge. To address this, we introduced and evaluated the iterative training strategy (see Appendix A.3.4 and Table 7) that alternates between optimizing the autoencoders and the bridge module. This approach reduces interference between components and improves the final performance. For example, our iterative setup achieves a 1-NNA of 0.508 and IoU of 0.664 on ShapeNet, outperforming two-step and fully end-to-end alternatives. Its important to note, that while the iterative setup is highly stable, we do not observe major stability issues in other setups as well.

While our experiments focus on jointly trained encoders, we note that the use of pre-trained modality-specific foundation models—such as CLIP or DINO for vision, and wav2vec or Whisper for audio—could further mitigate this training burden. These models may stabilize the latent marginals early in training and offer performance gains, particularly in low-data regimes. We consider this a promising direction for future work.

The paper demonstrates strong capabilities in multi-modal translation. To further showcase the versatility and robustness of the proposed modality agnostic transformer, suggest exploring its application to outpainting and/or image editing tasks in image-to-image translation? This might provide some insights on spatial relationships and compositional representations?

Please refer to the discussion in the 2nd response.

Sampling complexity of DDBMs are generally higher to achieve accurate interpolations can the authors elaborate on how the complexity is addressed ?

The DDBM sampler employs a sampling strategy similar to that used in the Elucidated Diffusion Model (EDM). We observe that even complex bridging tasks, such as the one considered in our work, can be accomplished in no more than 40 sampling steps—comparable to the number used by EDM for datasets like FFHQ or AFHQv2, which typically require around 40 steps depending on the implementation. Furthermore, the experiment discussed in the first comment provides additional insight, suggesting that further optimization of the number of sampling steps is possible. We'd be happy to provide any further clarification if needed.

While it is impressive that the approach does not require any conditioning, It would be good to see how does the objective affect the training complexity. Does the process of learning the 'bridge' between two distributions, and the modality-agnostic denoiser in a shared latent space, introduce a higher computational cost (e.g., longer training times, increased GPU memory usage) per iteration or per epoch compared to standard multi-modal EDMs. In additional how does the training convergence compared to standard diffusion pipelines.

Regarding the first point, please refer to our first response; we'd be happy to address any more aspects. As for the second point, we observe that the convergence behavior is very similar. The training time of our model closely matches that of the EDM variants, including both the U-Net and DiT implementations.