A Simple Approach to Unifying Diffusion-based Conditional Generation

We greatly appreciate your thoughtful review and your recognition of our work's strengths. In the following, we address some concerns and provide additional clarifications to enhance our contributions.

“No open sourcing of the code/models is mentioned.”

Question 1: ”Do you plan to release the source code to ease reproducibility of the approach?”

We will open source all code (training and inference) and weights of models used in the paper upon acceptance.

“There is no human study comparing the effective quality of the approach compared to the others. There are nice qualitative examples in Fig 3, but a human study would be more convincing to demonstrate these examples are representative.”

Question 2: “Could you do a human study?”

Thanks for suggesting a human study. We conduct a user study on 4 models:

Our methods demonstrate comparable results to specialized conditional generation methods. We do not include Appearance model because a suitable baseline for comparison is currently unavailable.

“One of the most concrete applications of generative image models would be serving downstream tasks, such as classification or depth estimation as demonstrated in this work. However, the estimation performance is far from SOTA performance.”

We provide a qualitative comparison of our method against Marigold (SOTA diffusion-based depth estimation method) in Fig. 11. Though not achieving SOTA on depth estimation, our method generally performs better than Marigold on diverse test cases, especially for images with ambiguous structure (Fig. 11 Line 2,5) or non-realistic style (Fig. 11 Line 1,7,8).

Meanwhile, existing benchmarks often lack scene diversity, as noted in Depth Anything V2. Therefore, we further evaluate UniCon-Depth on the relative depth estimation benchmark DA-2K, introduced in Depth Anything V2, which comprises diverse high-resolution test images.

On DA-2K, Our UniCon-Depth model is superior to other models except for Depth Anything V2 (which serves as our annotator), demonstrating that our model better generalizes to diverse scenes.

There is a confusion in the architecture of Unets, at least a sentence to rephrase line 242...

Thank you for pointing this out. We understand the confusion caused by the imprecise phrasing. In the revised version, we will rephrase the sentence to differentiate between the original U-Net architecture and the U-Nets commonly adapted in diffusion models.

In figure 3, third column, the fact that both the loose control and Unicon approaches lead o a snowman on the beach would mean their latent space is really similar as both approaches build on Stable diffusion weights. Also the controlNet result (first column) is also very similar to the Unicon one. It makes question the generalization ability of the approach.

We agree that UniCon, LooseControl, and ControlNet exhibit similarities in latent space, as all are built upon the same base model, Stable Diffusion. In Fig. 3, the resemblance in the ControlNet result is primarily due to the strict conditioning on depth and the highly specific text prompt (“golden retriever in a field of dandelions”), which significantly constrains image appearance. We provide more qualititive comparison results in Fig. 12 to demonstrate the generalization ability of our approach.

Why is there no quantitative comparison with the Loose control results?

It is because of the lack of established metrics for evaluating loose control performance.

• Since there is no ground truth (GT) image, it is not possible to compute metrics like FID
• Condition fidelity cannot be reliably measured against the boxified depth condition used in LooseControl.

Moreover, the LooseControl paper itself does not present quantitative results, which suggests the inherent difficulty in defining standardized metrics for this approach.

How is the weight w_r set up in (7)

For guidance, we adopt an optimizer (e.g. AdamW) to compute the gradient and determine the weighting factor w_r in Eq. (7) instead of manually setting a fixed weight factor, as suggested by Readout-Guidance. We explain the details in Appendix (Line 754).

Thank you for your comments on our paper. We have submitted a detailed response and a revised paper on OpenReview. With the discussion phase ending on November 26, we would like to confirm whether we have effectively addressed all concerns, particularly whether the additional qualitative results (Fig. 12, Fig. 11) and human study provide a more comprehensive understanding of UniCon’s performance compared to other specialized methods.

We also hope you might consider raising the scores based on these updates and the ongoing discussion.

Thank you

Thank you for your detailed review and for recognizing the flexibility and generality of our proposed method in conditional generation. Below, we address some concerns and provide further clarifications to strengthen our contributions.

The method, as presented, is limited to only accepting conditioning in the form of 2D images/feature maps. As it has now become common to support those combined with feature vector conditioning, as done in, e.g., Uni-ControlNet [Zhao et al., NeurIPS 2023], Composer [Huang et al., ICML 2023], or CTRLorALTer [Stracke et al., ECCV 2024].

Thanks for pointing out these related works. To clarify, there is a fundamental conceptual difference between UniCon and the mentioned works (e.g., Uni-ControlNet). Uni-ControlNet focuses on unifying multiple control signals within a single model to achieve controllable generation. In contrast, UniCon aims to unify multiple generation tasks within a single model, for a specific image condition.

Specifically, the mentioned works primarily address unifying various conditioning signals for controllable image generation. However, they cannot handle estimation, joint generation, or rough conditioning. Our method, on the other hand, learns a joint image distribution, emphasizing symmetry between the image and its conditioning. While it does not support conditions represented as feature vectors, it facilitates a broader range of generative behaviors beyond controllable generation.

The paper would benefit from additional qualitative examples in the appendix, as is commonplace in similar works. Regarding the weaknesses, I'd recommend the authors incorporate more related work into their quantitative comparisons and related work. Especially Uni-ControlNet seems important to compare against, as it is very closely related (except for the joint generation part). For additional qualitative comparisons, I recommend following the example set by IP-Adapter, which provides a great overview of the qualitative performance of a wide range of methods.

Thanks for suggesting more quantitive results. In Fig. 12, we provide a quantitive comparison of UniCon against ControlNet, Uni-ControlNet on conditional image generation, following IP-Adapter example.

We also provide a qualitative comparison of our method against Marigold on depth estimation in Fig. 11.

Question: “The ablation study only focuses on RGB generation in a single setting. Have you also ablated effects in other sampling settings.”

Currently, we have only tested the ablation models on depth-to-image generation. Evaluating the models on additional sampling settings, such as depth estimation, would certainly enhance the reliability and comprehensiveness of the ablation study. We plan to include results on depth estimation once sufficient computational resources become available to test all ablated models.

Thank you for the response.

Regarding your first response, I first want to apologize for the seemingly cut-off comment in the initial review. I don't know how that happened. Given your response, at least the point I was trying to make still came across. For reference, the point I was trying to make was that (jointly) conditioning on non-image-like conditions has become an important aspect that is not covered by the presented work. This still holds for the updated work. However, I agree with the authors that implementing what we would typically implement as a conditional distribution as a joint distribution instead is an orthogonal task.

Regarding the second, I appreciate the authors providing additional qualitative results. Regarding the depth examples, while they do demonstrate the impressive strengths of the presented method in out-of-distribution situations that the authors also mentioned in other comments, I would appreciate the authors also including some more normal photo examples without extreme reflections or depth of field in a potential camera-ready version to make the comparison more representative of many real-world use cases. I also share xQEt's concerns about this performance potentially being more of a consequence of teacher and dataset choice than the underlying approach being superior for depth estimation, which is reflected in the model quantitatively underperforming on datasets typically considered in-distribution while overperforming on samples typically considered out-of-distribution.

Regarding the third, I would appreciate it if the authors added that.

While I still think that the paper would benefit greatly from further improvements, the author's response to my main point of criticism and the additional clarifications and results provided in response to other reviews, I will consider raising my score to reflect the improvements on the paper.

Thank you for your comments on our paper. We have submitted a detailed response and a revised paper on OpenReview. With the discussion phase ending on November 26, we would like to confirm whether we have effectively addressed all concerns, particularly whether the additional qualitative comparison on conditional generation (Fig. 12) and depth estimation (Fig. 11) provides a good overview of the performance of UniCon against other methods. We also hope you might consider raising the scores based on these updates and the ongoing discussion.

Thank you

Thanks for your reply!

This figure shows depth estimation samples on normal photos. We will include both normal photos and out-of-distribution samples in our final version. We are working on training an UniCon-Depth model using ZoeDepth as the annotator to address the shared concern of you and xQEt and will report the result when it is done.

We include the depth estimation results in ablation study. The accuracy on DA-2K dataset is reported:

The depth estimation performance is generally consistent with the conditional generation performance. Ablations that removes depth loss (annotated as -Depth loss) do not support depth estimation, thus not included.

Thanks again for helping us improve our work!

We greatly appreciate your thorough review and your recognition of the strengths of our work. In the following, we address your concerns and provide further insights to strengthen our work.

Weakness: Condiitonal generation using the proposed method requires performing multiple denoising paths, which makes the inference compationally intensive compared to direct conditioning, especially for multiple conditions.

Question 1: It would be great if authors could provide an analysis on the computational cost of inference using their method.

Thank you for highlighting the concern regarding inference cost. Before presenting our analysis, we want to clarify our model implementation. Though illustrated as two branches (image and condition), our approach uses one diffusion model and processes two inputs in a single feed-forward pass. It is realized by selectively applying our condition LoRA to the condition sample. Detailed implementation is provided in Appendix A, and we will further clarify the architecture in the revised version.

Below, we provide an analysis of inference speed.

Table A: Inference Speed Comparison (Iterations per Second)

Table A compares UniCon with ControlNet for controllable image generation. UniCon incurs additional inference overhead due to the separate computation of LoRA layers (condition LoRA and joint cross-attention LoRA) instead of fusing them into the pretrained weights. Therefore, separate LoRA layer computation incurs heavy overhead. This overhead can potentially be mitigated by saving multiple attention weights (with and without LoRA) directly in the model.

Table B: Latency Comparison for Depth Estimation (on A800 GPU)

Table B compares UniCon with other depth estimation methods. Diffusion-based methods (e.g., UniCon and Marigold) generally exhibit higher latency than traditional approaches. However, UniCon can leverage techniques like Latent Consistency Models (LCM), as demonstrated by Marigold, to significantly improve inference speed.

In summary, while UniCon currently has higher inference costs, its computational efficiency can be enhanced through optimizations like LoRA weight fusion and LCM integration. We plan to explore these directions in future work.

Question 2: "In Section 4.2, line 233, authors mention "When the conditional image differs from a natural image, we add a LoRA to the y-branch." How is the branch trained for conditioning on natural images, considering that the base image generation U-Net is fozen and no LoRA is used?"

For models including a natural image condition (e.g., ID, Appearance), we do not apply LoRA to the conditional branch and only train the joint cross-attention modules to learn the correlation between two images (e.g. two images from one person, two frames from one video). Note that in this case, two branches are symmetric and the model will perform the same as the base model if we remove all joint cross-attention modules.

Thank you for your comments on our paper. We have submitted a detailed response and a revised paper on OpenReview. With the discussion phase ending on November 26, we would like to confirm whether we have effectively addressed all concerns. We also hope you might consider raising the scores based on these updates and the ongoing discussion.

Thank you

Thank you for your constructive feedback and for highlighting the strengths of our work. Below, we address some concerns and provide additional details to further clarify and enhance our contributions.

Weakness 1: Missing Comparisons/References

Question 2: "How does your method compare with ZoeDepth, Depth Anything, and Depth Anything v2?"

We show our model's performance on zero-shot affine-invariant depth estimation compared to ZoeDepth and Depth Anything V2.

Although our UniCon model still lags behind state-of-the-art (SOTA) depth estimators on NYUv2 and ScanNet, existing benchmarks often lack scene diversity, as noted in Depth Anything V2. To address this limitation, we further evaluate UniCon-Depth on the relative depth estimation benchmark DA-2K, introduced in Depth Anything V2, which comprises diverse high-resolution test images.

Table A:

*: results from Depth Anything V2

On DA-2K, Our UniCon-Depth model is superior to other models except for Depth Anything V2 (which is used as our annotator), demonstrating that our model better generalizes to diverse scenes. We also add a qualitative depth estimation comparison against Marigold in Fig. 11. Note that our method is trained on 16k images without GT depth annotation.

"In addition, it will be helpful to discuss "DAG: Depth-Aware Guidance with Denoising Diffusion Probabilistic Models" [Kim et al.], which appears to be a relevant line of work for conditional generation with depth information.”

DAG guides a diffusion model to generate geometrically plausible images using depth prior. We will add a discussion about DAG in Sec. 2.

Weakness 2: Technical Ambiguities

Though illustrated as two branches (image and condition), our approach uses one diffusion model and processes two inputs in a single feed-forward pass. It is realized by selectively applying our condition LoRA to the condition input. We explain our detailed implementation at Appendix.A. We will clarify our model architecture in our revised version.

We provide an analysis of inference speed for controllable image generation.

The table reports denoising iterations per second for each method. Our method produces more inference overhead than ControlNet. Currently, we do not fuse LoRA weights into the pretrained weights, for both condition LoRA and LoRAs in joint cross-attention modules. Therefore, separate LoRA layer computation incurs heavy overhead. The problem can be potentially solved by saving multiple attention weights (w./w.o. LoRA) in the model. We plan to improve the inference efficiency in future work.

Question 1: What is the inference cost of depth prediction compared to other traditional and recent approaches?

We show the inference latency of depth estimators in the following table:

Diffusion-based estimation methods (UniCon, Marigold) generally require more inference costs than others. Similar to Marigold, UniCon can improve the inference speed by incorporating Latent Consisteny Model (LCM).

Weakness 3: Joint Distribution Modeling

Question 3: "What are the specific advantages of joint distribution modeling over other approaches discussed in the paper, particularly in terms of inference, accuracy, and other metrics?"

Thanks for pointing this out! The advantages of joint distribution modeling are central to our work. Below, we outline its benefits compared to three specialized methods. We provide qualitative results on depth estimation (Fig. 11) and conditional image generation (Fig. 12).

• Vs Marigold for depth estimation. As shown in quantitive results on DA-2K (Table A) and qualitative results (Fig. 11), our method generally performs better than Marigold on diverse test cases, especially for images with ambiguous structure (Fig. 11 Line 2,5) or non-realistic style (Fig. 11 Line 1,7,8). Our joint modeling helps maintain SD's strong prior on natural images and helps our model to generalize to OOD test cases never seen during training. Marigold finetunes the whole model to enable conditional generation p(depth|image), losing the prior on diverse natural images.
• Vs ControlNet for controllable generation. ControlNet sometimes generates over-saturated images (SoftEdge 2,4 in Fig. 12) or images not aligned with the condition (Pose 4 in Fig. 12), while it rarely happens to our method. We attribute it to that joint modeling stimulates our model to fully capture the bidirectional correlation between image and condition, while conditional modeling might learn to rely on minor clues in the condition image. Note that UniCon outperforms ControlNet in condition fidelity (Table 1).
• Vs LooseControl for rough depth to image. Our joint modeling enables UniCon to take noisy conditions as input. Therefore, our models can accept various rough depth conditions, including the boxified depth condition used by LooseControl. In comparison, LooseControl requires finetuning on the new depth format to achieve this.

Thank you for your comments on our paper. We have submitted a detailed response and a revised paper on OpenReview. With the discussion phase ending on November 26, we would like to confirm whether we have effectively addressed all concerns, particularly:

• Whether our evaluation on DA-2K and the qualitative results sufficiently demonstrate UniCon's advantages in depth estimation.
• Whether our analysis against three specialized methods (Marigold, ControlNet, LooseControl) demonstrates the benefits of joint modeling.

We also hope you might consider raising the scores based on these updates and the ongoing discussion.

Thank you

Thank you for your thoughtful and detailed review. We appreciate your recognition of our method's lightweight design, strong performance, and the novelty of our independent timestep scheduling. In the following, we address some concerns and provide clarifications to further strengthen our work.

Weakness 1: Although there are five image conditional models trained using the proposed framework, it seems that only three (Depth, SoftEdge, Pose) types are compared quantitatively. This asymmetry in the quantitative/qualitative demonstration makes the manuscript incomplete.

Weakness 2: Moreover, there are also other metrics that compares the conditional generation. ... I do not want to recommend showing all those metrics that I have mentioned (these are for examples), but I believe it is required to have a complete table of quantitative metrics at least for models presented.

Thanks for your detailed suggestion on our experiments and metrics! We have now included additional metrics and experiments to address this. (User Study, CLIP Score, $\delta_1$).

We add CLIP Scre and $\delta_1$ to Table 1 for depth conditonal generation:

In terms of Pose metrics used in FineControlNet, we found that CLIP Identity Observance requires instance description that is absent from our validation dataset, and Pose Control Accuracy is similar to PCK we used.

We conduct a user study on 4 models:

For the Appearance model, evaluating it independently is challenging due to the following:

• Applying it in isolation often produces outputs that closely resemble the input image. We must adjust the condition weight or text prompt for specific use cases. Its effectiveness is most apparent when combined with other models. Please check Figs. 1, 4, 7, 10 for its use cases.
• A suitable baseline for comparison is currently unavailable.

Weakness 3: It would be better to have a quantitative comparison in generated images from synchronous and asynchronous noise level scheduling in the inference time. The demonstration in the manuscript shows that the method works, but it does not show that this asynchronous generation leads to diversity and quality comparable to synchronous one.

Below is a comparison of depth conditional generation under synchronous and asynchronous noise level scheduling:

For synchronous setting, we denoise image and depth together while replacing depth input with noisy GT depth at each step. The results demonstrate that asynchronous noise level scheduling offers clear advantages in depth conditional generation.

Does the presented method applicable to applicable/robust to inpainting/editing tasks? Supporting this feature will make the proposed method far more persuasive.

Thank you for raising this point. We add Fig. 9 to showcase our model's performance on inpainting or/and editing to image or/and condition. Fig. 5, 10 also show some related results.

Empirically, we observe that our model is robust for editing tasks. However, its inpainting performance is less stable, which is a limitation of the base model (Stable Diffusion). A straightforward solution is to apply our method to an inpainting-specific diffusion model, such as SD-Inpainting.

This is a very minor concern, but do the authors want to extend this model beyond SD1.5, which seems to be deprecated in the generative AI community?

Yes, we plan to train models on more advanced models, such as SD-XL, as computational resources allow.

Thank you for your comments on our paper. We have submitted a response and a revised paper on OpenReview. With the discussion phase ending on November 26, we would like to confirm whether we have effectively addressed all concerns, particularly whether the additional experiments and metrics adequately resolve the limitations you highlighted regarding a comprehensive manuscript. We also hope you might consider raising the scores based on these updates and the ongoing discussion.

Thank you

Thanks for your reply! Currently, we include all additional results (tables and figures) in the Appendix.E. We will add some key results to the Results section and rearrange the contents in a potential camera-ready version.

We acknowledge that the current version is superfluous regarding the Appearance model due to its lack of quantitative evaluation. So, we decided to lower the emphasis on it and treat it as an auxiliary model that can help transfer image appearance when using other models (Fig. 1, 7). We have removed Lines 405-410 and Appearance model samples in Fig. 4.

Following, we provide our understanding of the appearance model for your information. Unlike AnimateDiff, which is trained on consecutive frames, our Appearance model is trained on random frames from the same video. Its goal is to generate frames that "seem to come from the same video" as the input image, ensuring consistent appearance without temporal awareness.

This figure illustrates how we can use the model to obtain the output images with different levels of similarity. While applying the Appearance model independently is limited in scope, combining it with other control signals (e.g., depth) offers meaningful outputs, such as here. Video models like AnimateDiff cannot generate such discrete frames, so comparing the Appearance model against them is hard. While outside the scope of this paper, the Appearance model could be extended for video editing/stylization using a dedicated pipeline ( which is explored in a separate under-review paper).

Thank you again for helping us improve our work!