More Than Generation: Unifying Generation and Depth Estimation via Text-to-Image Diffusion Models

1. To weakness 1: “The comparison with baselines is not entirely fair, … such as JointNet or UniCon, are built on different models like Stable Diffusion.”

   Reply: Thanks. In the table below, we provide experiments on the Stable Diffusion version of MERGE (MERGE-SD). The experimental results demonstrate that MERGE-SD consistently outperforms JointNet and UniCon across multiple depth estimation benchmarks. We fully validate the advantages of our MERGE on multiple pre-trained Text-to-Image models, including Stable Diffusion, PixArt, and FLUX.

   Method	NYUv2-A.Rel ↓	NYUv2-δ1 ↑	Scannet-A.Rel ↓	Scannet-δ1 ↑
   JointNet	13.7	81.9	14.7	79.5
   UniCon	7.9	93.9	9.2	91.9
   MERGE-SD (Ours)	6.1	95.6	7.6	93.2

2. To weakness 2&3: “… converter introduces an incremental innovation by combining reused T2I model blocks with LoRA-like principles, the justification in Figure 4 appears model-specific and does not clearly generalize to architectures such as Stable Diffusion or FLUX. …”

   Reply: Thanks. Regarding your opinion that our converter design represents "incremental innovation," we would like to clarify that it fundamentally differs from methods like LoRA in motivation and architecture. 1) The core motivation is fundamentally different. The motivation of LoRA stems from prior work [1], which revealed that pre-trained models possess an extremely low-dimensional parameter space. Fine-tuning within this space achieves the same effect across the entire parameter space. Essentially, LoRA provides an efficient implementation of full fine-tuning. Our intuition behind designing the converter is based on the fact that diffusion-based image generation and generative geometry estimation both belong to the latent diffusion process. The transition from image generation to geometry estimation may only lack a "task-oriented" converter, which guides the latent diffusion of image generation towards geometry estimation. 2) The architectural implementation has significant distinctions. LoRA introduces low-rank matrices into existing layers to achieve efficient fine-tuning. In contrast, our approach injects a highly structured converter similar to Text-to-Image (T2I) blocks. The most direct advantage of our highly structured converter is that it enables converters to be initialized with the weights of pre-trained T2I models for parameter initialization, which LoRA does not support. The experimental results in the table below demonstrate that this initialization consistently outperforms random parameter initialization.

   Init.Type	NYUv2-A.Rel ↓	NYUv2-δ1 ↑
   Random	7.9	93.5
   Pre-trained	7.5	94.2

   The motivation for our Group Reuse Mechanism (GRE) is inspired by observing the common characteristics of large Transformer models, such as Fig. 1 in [2], Fig. 3 in [3], and Fig. 4 in [4]. Specifically, adjacent layers often have a high degree of feature similarity. Fig. 4 in our main text uses PixArt as a visual example of this common phenomenon, rather than as comprehensive evidence.

3. To weakness 4: “… lacks quantitative or visual evaluations of image quality for the image generation tasks. …”

   Reply: Good suggestion! The design of our method is to ensure the complete preservation of image generation capabilities. Specifically, all additionally introduced converters are skipped when performing the "text-to-image" task. It means that the inference process of image generation remains fully consistent with the original, unmodified, pretrained T2I model. Following your suggestion, we provide the quantitative evaluation results for image generation below. In addition, we present some images generated by our MERGE model in Fig. 1 of the main text. We sincerely thank your suggestion and commit to explicitly including the quantitative evaluation results of image generation in the final version of the paper to strengthen our core claims.

   Method	COCO FID-30K ↓	GenEval ↑
   SDXL	6.63	0.55
   SD3-medium	11.92	0.62
   JointNet	14.15	-
   OneDiffusion	-	0.65
   PixArt-α	7.32	0.48
   MERGE-PixArt (Ours)	7.32	0.48
   FLUX.1 dev	10.15	0.67
   MERGE-FLUX (Ours)	10.15	0.67

4. To weakness 5: “… the number of converters appears to violate scaling laws, yet no in-depth analysis is provided to explain this phenomenon. … discussion on the trade-off between depth estimation and image generation capabilities. … no ablation study exploring configurations with fewer than 110M learnable parameters.”

   Reply: Thanks. Since converters are inserted into the pre-trained T2I model in a sequential manner, increasing the number of converters sharply increases the network depth. For example, when the number of converters is 3, the network depth expands 4x, which can lead to optimization challenges[5][6] and ultimately result in a decline in performance.

   It is an excellent question regarding the "trade-off." Thanks to our plug-and-play design, our image generation capability remains constant and does not vary with changes in depth estimation performance.

   We present the ablation study with fewer than 110M parameters in Table 5, as shown in the table below. In this table, we explore settings with 56M (7 groups) and 32M (4 groups) learnable parameters by adjusting the number of groups in the group reuse mechanism (GRE). The experimental results show that our method remains effective under a lower parameter budget.

   Groups	GRE	#Param.	NYUv2-A.Rel ↓	NYUv2-δ1 ↑
   28	×	224M	7.0	94.7
   14	×	110M	15.6	78.8
   14	✔	110M	7.5	94.2
   7	✔	56M	7.8	93.5
   4	✔	32M	9.3	91.0

5. To question 1: “… feature similarity shown in Figure 4 and the motivation for reusing converters is not clearly established. … redundancy in the DiT model’s internal representations? …”

   Reply: Thanks. Fig. 4 of the main text reveals the high similarity between the output features of adjacent layers in the DiT model. Our idea is that if T2I block i and i+1 outputs are similar, the feature transformation for handling depth estimation tasks should also be similar. Therefore, sharing a converter may largely capture this common transformation.

   Regarding the question of “redundancy”, we argue that it might highlight the widespread phenomenon of redundancy in the DiT model’s internal representations. We also note that a recent study[7] demonstrates that a significant portion of computation in global self-attention mechanisms is wasted on calculating near-zero attention scores between distant tokens, resulting in only a limited number of effectively updated features. Moreover, as Transformer models become deeper, feature representations in the deeper layers become overly similar[6], leading to a loss of diversity and expressive power, making additional layers computationally redundant. These findings indicate that the DiT model indeed contains redundancy.

6. To question 2: “…lack generalizability across architectures beyond PixArt… Stable Diffusion or FLUX, …”

   Reply: Good suggestion! We would like to clarify that we discuss MERGE based on PixArt and FLUX in the paper. Additionally, we present the performance of MERGE based on Stable Diffusion in the table below. Our three variants achieve outstanding performance on multiple depth estimation benchmarks. These results directly demonstrate the strong generalizability of our method, which is not limited to the PixArt architecture alone.

   Method	NYUv2-A.Rel ↓	NYUv2-δ1 ↑	Scannet-A.Rel ↓	Scannet-δ1 ↑	DIODE-A.Rel ↓	DIODE-δ1 ↑
   JointNet	13.7	81.9	14.7	79.5	-	-
   UniCon	7.9	93.9	9.2	91.9	-	-
   OneDiffusion	6.8	95.2	-	-	29.4	75.2
   MERGE-PixArt (Ours)	7.5	94.2	9.9	89.8	32.5	74.9
   MERGE-Stable-Diffusion (Ours)	6.1	95.6	7.6	93.2	30.8	93.4
   MERGE-Flux (Ours)	5.9	95.4	7.1	94.0	31.4	76.3

[1] Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning, ACL 2021.

[2] Do Vision Transformers See Like Convolutional Neural Networks?, NeurIPS 2021.

[3] Skip-Attention: Improving Vision Transformers by Paying Less Attention, ICLR 2024.

[4] Layerlink: Bridging remote sensing object detection and large vision models with efficient fine-tuning, PR 2025.

[5] Going Deeper with Image Transformers, ICCV 2021.

[6] Swin Transformer V2: Scaling Up Capacity and Resolution, CVPR 2022.

[7] Swin DiT: Diffusion Transformer using Pseudo Shifted Windows, Arxiv 2025.

1. To weakness 1: “… OneDiffusion uses 40K images from Hypersim dataset, without using Virtual KITTI used in this work. … the performance improvement comes from different training sets or the proposed method.”

   Reply: Good question! To address the concern that the performance improvement might stem from different training sets, we train a MERGE-L model using only Hypersim as training data, without Virtual KITTI. The experimental results are shown in the table below. Compared to OneDiffusion, the MERGE-L model trained solely on Hypersim still achieves comparable or even better results on the depth estimation benchmarks. This result demonstrates that the performance gain is primarily attributable to our proposed MERGE method rather than a specific data combination. In addition, we would like to mention that the depth map training data for OneDiffusion includes the Hypersim dataset and approximately 500K high-quality data, while we use 74K depth map training data.

   Method	Depth Map Training Data	NYUv2-A.Rel ↓	NYUv2-δ1 ↑	DIODE-A.Rel ↓	DIODE-δ1 ↑
   OneDiffusion	Hypersim+500K depth data	6.8	95.2	29.4	75.2
   MERGE-L (Ours)	Hypersim	6.2	95.2	31.0	76.5

2. To weakness 2: “… what kinds of scenarios is the proposed method particularly effective in?”

   Reply: Good question! Our current experimental setup is an out-of-distribution evaluation, where training is performed on synthetic data and evaluation is conducted on real-world datasets. Experimental results indicate that, compared to other unified models, our method demonstrates superior performance in zero-shot scenarios, especially in indoor scenarios. Additionally, Appendix B provides qualitative comparison results under challenging objects (Reflective Objects, Free-form Voids, and Filiform Objects). These results show that MERGE exhibits unique advantages in these challenging objects. We will highlight these findings in the main text of the final manuscript.

3. To weakness 3: “… lacks the comparison to traditional methods like Depth Anything[1] and Depth Anything V2 [2]. This may mislead the readers on the results of the proposed methods.”

   Reply: Good suggestion! To provide readers with a more complete overview of the current state of development in the depth estimation field, we commit to including the results of traditional methods like the Depth Anything series[1][2] in Table 1 in the final version.

[1] Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data, CVPR 2024.

[2] Depth Anything V2. A More Capable Foundation Model for Monocular Depth Estimation, NeurIPS 2024.

This is a highly valuable question that allows us to clarify the differences and advantages of our MERGE, compared to traditional depth estimation discriminative methods.

1. Question: How do they perform under different scenarios and datasets?

Reply: In terms of quantitative performance, state-of-the-art traditional discriminative methods (e.g., DepthAnything v2) still lead in monocular depth estimation, as shown in the table below, particularly on challenging benchmarks like DIODE that include diverse outdoor scenes. However, it is important to note that the performance gap narrows significantly on indoor datasets, for example, our MERGE achieves highly competitive performance to DepthAnything v2 on the NYUv2 benchmark (A.Rel: 4.4 vs. 5.9, δ1: 97.9 vs. 95.4).

2. Question: What is the benefit of using diffusion-based models for depth estimation?

Reply: The benefits are primarily twofold: 1) Efficient training enabled by strong priors. Generative depth estimation methods demonstrate impressive data efficiency by leveraging the powerful priors from pre-trained T2I models. For example, our MERGE achieves its competitive performance on NYUv2 while using only about 1% of the depth training data required by DepthAnything v2. 2) The same paradigm enables a unified framework. The emergence of generative depth estimation methods[1][2] makes it possible to unify the previously distinct tasks of geometric estimation and image generation. Building on this idea, we explore a novel unified paradigm for image generation and geometric estimation within a simple framework, which is not achievable with traditional discriminative depth estimation methods.

3. Question: Do diffusion-model priors bring some new knowledge for depth estimation, which traditional discriminative approaches may not have effectively learned?

Reply: The most important “new knowledge” brought by the diffusion model prior is its deep semantic and structural understanding of the visual world. These models can generate realistic and previously unseen content, which indicates that they have internalized common sense about objects, scenes, and physical laws. This “knowledge” may help the model produce physically plausible and reasonable depth structures for depth estimation tasks, especially when dealing with challenging materials such as textureless or blurry regions and reflective or transparent surfaces[3]. This profound world understanding, demonstrated by the model’s generative capabilities, is something traditional discriminative methods may not be able to learn effectively.

[1] Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation, CVPR 2024.

[2] GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image, ECCV 2024.

[3] Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction, ICLR 2025.

Thanks to the authors for the response. I have also read the comments from other reviewers. While the proposed method is based on diffusion models, I believe comparisons with discriminative approaches like DepthAnythingv2 are necessary to not only understand its relative performance but also to show the benefit of using diffusion models. It is recommended to include these discussions in the final paper.

While I acknowledge the contribution of this work, categorizing it as a unified model may be overstated. For example, prior unified models, such as OneDiffusion support multiple tasks with a single model, including segmentation, pose estimation, ID customization, and so on. By contrast, this work retains the original diffusion model's image generation ability without further improvement. After adding the converter, the proposed method can only support one task: depth estimation. Considering this, I prefer to categorize this work as a generative method, similar to Lotus. However, the proposed method underperforms prior generative approaches like Lotus. Considering these points, I will maintain my score as borderline accept, without raising it to a 5.

We sincerely thank your careful review of our work. We will follow your suggestion and include a comparison and discussion with discriminative approaches such as DepthAnythingV2 in the final paper.

Regarding your comment about “... may be overstated,” we would like to clarify that our approach is not limited to the depth estimation task. We have also extended it to normal estimation in the supplementary material. Furthermore, our method can be easily transferred to other tasks such as semantic segmentation and style transfer. We will continue to explore these tasks using our method in the future.

Regarding Lotus, we would like to clarify that Lotus focuses on fully leveraging the priors of pre-trained T2I models to achieve better depth estimation. In contrast, our method explores how to extend the capabilities of generative models without compromising their image generation abilities. We hope our approach can provide more insights to the community.

Once again, thank you for your time and valuable suggestions during the review process.

1. To weakness 1: “… motivation for preserving the pre-trained T2I model's image generation capability. … the lower performance compared to Marigold, … raises questions about the trade-off. Clarifying why a model capable of both image generation and depth estimation … .”

   Reply: Thanks. Recent works, such as Show-o[1], UniCon[2], and OneDiffusion[3], reveal a promising research direction that merges image generation capabilities with understanding or perception capabilities in a single model, which are originally independent. Thus, our method aims to explore a new paradigm that merges the previously independent image generation and geometric perception (e.g., depth estimation) capabilities into a unified model, rather than pushing the performance border of a single task. So, we would like to clarify that a fair and meaningful comparison should be made with unified-based methods like UniCon (ICLR 25) and OneDiffusion (CVPR 25) rather than with specialized models (e.g., Marigold). Notably, even compared to Marigold, our method shows only a minor performance gap (e.g., only 0.4 A.Rel gap on the NYUv2 benchmark). More importantly, our method preserves the original image generation capability of the pre-trained model, which Marigold does not. We believe that trading a minor difference in single-task performance to preserve the model's original image generation capability is a valuable trade-off for developing unified models for image generation and perception in the community.

2. To weakness 2: “Following on from the point above, if a key benefit of preserving the T2I prior is enhanced generalization capability, demonstrating its robustness on OOD data would further validate the importance of maintaining the generative prior.”

   Reply: Good suggestion! We apologize for not clearly explaining the experiment setting. Actually, our current training on synthetic datasets and evaluation on three real-world datasets (NYUv2, ScanNet, and DIODE) aligns with the out-of-distribution (OOD) test setting. Our MERGE outperforms other unified models, such as JointNet (ICLR 24), UniCon (ICLR 25), and OneDiffusion (CVPR 25), on these real-world benchmarks, proving that preserving and leveraging the robust visual priors from Text-to-Image models is crucial for enhancing a model's zero-shot generalization capabilities. Thanks for your suggestion. We will explicitly clarify our experimental setup in the final version.

3. To weakness 3: “The comparison in Table 3 is relevant to the discussion on motivating T2I prior preservation. Providing more details on this would strengthen the paper's rationale.”

   Reply: Thanks. Previous work [4] suggests that the catastrophic forgetting during the fine-tuning of Text-to-Image (T2I) models often leads to vague predictions in highly detailed areas. Thus, a naive idea is that preserving the T2I prior might alleviate this issue. So, we provide a direct and fair comparison between our method and Marigold under an identical T2I model. The results show that our MERGE not only achieves highly competitive and even better performance compared to the Marigold-P with fewer learnable parameters but also preserves the original generation capability, an advantage not supported by the full fine-tuning approach.

4. To question 2: “Would it be feasible to extend this framework to enable co-generation of an image and its corresponding depth map simultaneously from a text prompt?”

   Reply: This is a very insightful question, and the answer is yes! In Fig. 1 of the main text, we present examples of generating an image and its corresponding depth estimation annotation using a two-stage method (text-to-image -> image-to-depth). A possible one-stage solution to simultaneously generate the image and its corresponding depth map involves branching the features for image generation and depth estimation. After passing through a shared Text-to-Image block, one branch converts the features for the depth estimation task via our converters, while the other branch skips the converters to complete the original image denoising process. This allows for the simultaneous output of both the image and its corresponding depth map. Follow your suggestion, we will refine this solution in our future work.

[1] Show-o: One Single Transformer to Unify Multimodal Understanding and Generation, ICLR 2025.

[2] A Simple Approach to Unifying Diffusion-based Conditional Generation, ICLR 2025.

[3] One Diffusion to Generate Them All, CVPR 2025.

[4] Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction, ICLR 2025.

1. To weakness 1: “… not surprising to levarage the generation power to do the depth estimation … comparison we can see the depth estimation expert models (Marigold, GeoWizard and etc.) … . So for real world applications …”

   Reply: Good question! In general, image generation and geometric estimation (e.g., depth estimation) are considered two totally different tasks. The emergence of recent works in the generative field, such as Show-o[1], UniCon[2], and OneDiffusion[3], indicates that the unified of previously separate tasks, including understanding, generation, and perception, is a promising research direction. So, our method aims to explore a new paradigm for constructing a unified model for image generation and geometric perception in a simple framework, instead of simply pushing the performance border of a single task.

   Although methods like Marigold and GeoWizard achieve strong performance through full fine-tuning, their image generation capability is inevitably compromised. Thus, a relatively fair comparison would be with UniCon[2] (ICLR 25) and OneDiffusion[3] (CVPR 25), which also possess image generation and geometric perception capabilities. As shown in the table, even when compared to Marigold, our A.Rel metric shows a difference of less than 1 across three depth estimation benchmarks. Moreover, in the evaluation of normal estimation, our MERGE achieves an overall better performance. More importantly, our MERGE retains the capability for image generation, which Marigold does not.

   Considering the early exploration of merging geometric perception capabilities while preserving the original image generation capabilities of pre-trained models, we believe our work is valuable to researchers in this field.

   Method\Depth	NYUv2-A.Rel ↓	NYUv2-δ1 ↑	Scannet-A.Rel ↓	Scannet-δ1 ↑	DIODE-A.Rel ↓	DIODE-δ1 ↑
   Marigold	5.5	96.4	6.4	95.1	30.8	77.3
   GeoWizard	5.2	96.6	6.1	95.3	29.7	79.2
   MERGE (Ours)	5.9	95.4	7.1	94.0	31.4	76.3

   Method\Normal	NYUv2-m.↓	NYUv2-11.25◦↑	Scannet-m.↓	Scannet-11.25◦↑	iBims-1-m.↓	iBims-1-11.25◦↑
   Marigold	20.9	50.5	21.3	45.6	18.5	64.7
   GeoWizard	18.9	50.7	17.4	53.8	19.3	63.0
   MERGE (Ours)	20.1	54.3	18.3	60.2	18.2	66.2

2. To weakness 1: “… the value of the proposal in real world senarios is not clear unless the work shows the possibilities to extend the work to other image generation tasks … .”

   Reply: Good suggestion! We would like to clarify that our current experiments align with the evaluation of out-of-distribution (OOD), which involves training on synthetic datasets and testing on real-world datasets. The experiment results represent impressive performance, demonstrating the potential of our method in the real world. Additionally, following your suggestion, we extend our method to the normal generation task. Please refer to Appendix A.2 for specific details. Our method achieves highly competitive or even better performance than current state-of-the-art full fine-tuning models. Critically, these models lose their image generation capabilities after fine-tuning, whereas our MERGE method preserves the original text-to-image generation ability.

   In addition, image generation tasks like depth-to-image controllable generation are similar to generative depth estimation, as both belong to per-pixel-level conditional generation tasks. Therefore, we also explore extensions to these types of image generation tasks. However, due to the constraints of the rebuttal format, we cannot show the visualization results (these tasks typically lack quantitative evaluation). We commit to presenting these visualization in the final version.

3. To weakness 2: “Minor typo… .”

   Reply: typos: Plug-and-play -> play-and-plug

[1] Show-o: One Single Transformer to Unify Multimodal Understanding and Generation, ICLR 2025.

[2] A Simple Approach to Unifying Diffusion-based Conditional Generation, ICLR 2025.

[3] One Diffusion to Generate Them All, CVPR 2025.

Dear Reviewer RUJQ,

We sincerely thank your time and effort in reviewing our paper. We hope our explanations have addressed your concerns. As the discussion phase is nearing its end, please let us know if you have any further questions, and we would be happy to answer them. We look forward to receiving your feedback.

Best regards,

Paper 2750 Authors