Diffusion in Diffusion: Cyclic One-Way Diffusion for Text-Vision-Conditioned Generation

We sincerely thank you for providing your valuable suggestions and pointing out our oversights! Please see our response below:

• Q1: The approach lacks a theoretical foundation, so it is not very intuitive to express why it can work.
• A1 :
  • Our method mainly builds upon the observed "diffusion phenomenon" of the pre-trained model as demonstrated in Fig. 2. This phenomenon is inspired by the nature that the diffusion model follows a stochastic Markovian chain between the data distribution and Gaussian distribution, characterized by a gradual entropy increase through Gaussian perturbations. In the reverse diffusion process, models focus on different levels of information at different phases, essentially embodying an extremely-noisy-to-semantic-formation-to-refine process.
  • The two main proposed manners for utilizing the above properties are "One-Way" control and the "Cyclic" construction. By constantly replacing in the Semantic Formation Phase, We exert influence from the given visual condition to the rest of the generated image while preserving the content of the visual condition, thus achieving a one-way conditioning. However, the pretrained model can easily fail to generate a pleasant result with a roughly spliced starting point. So we design a cyclic "disturb and reconstruct" process to recursively exploit the former generation progress under the visual condition, thus achieving better generation results.
  • Hope this additional explanation can help solve your concerns.

• Q2: The results of comparison methods are a bit too bad. More information about the setting should be given.
• A2:
  • We follow the corresponding official codes to conduct all the comparison methods. For ControlNet, we directly use the official webui of Stable Diffusion. We have included codes of other methods in the updated supplementary materials.
  • The unsatisfying performance of compared customization methods (DreamBooth, TI) on semantically complex conditions like human face has also been reported in prior works [a, b, c]. One major drawback of these methods is that they are not training-free and thus tend to easily overfit the limited finetune data and incorrectly entangle object identity and spatial information. Instead, our work is learning-free, leveraging the property of diffusion denoising itself for the customized generation.
  • The unsatisfying performance of SD-inpainting and ControlNet comes from the compulsively preserved visual condition, which may easily generate results with significant fragmentation (as shown in Fig. 1 of the main paper). Providing a small area as visual condition is a big challenge for these two methods. Meanwhile, these methods lack flexibility of fitting in multiple conditions, thus cannot be used for attribute editing or style transfer applications where the users use text prompts to slightly revise the visual conditions.

[a] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning. arXiv preprint arXiv:2304.03411, 2023.

[b] Yuheng Li, Haotian Liu, Yangming Wen, and Yong Jae Lee. Generate anything anywhere in any scene. arXiv preprint arXiv:2306.17154, 2023.

[c] Ziyi Dong, Pengxu Wei, and Liang Lin. Dreamartist: Towards controllable one-shot text-to-image generation via contrastive prompt-tuning. arXiv preprint arXiv:2211.11337, 2022.

Thank you for your professional feedback, which played a crucial role in refining our manuscript. Please see our response below:

• Q1: Most of these examples are based on putting an object in bounding box to a large image by adding backgrounds. This setting has applications, however, whether this approach can be applied to whole image generation/editing instead of pasting object on a larger image.
• A1: Thanks for the valuable suggestion! We conduct experiments on the whole image generation/editing, achieving superior results. We have added those samples in the new appendix (Fig. 11 of the appendix).

• Q2: In the main body of this paper, the authors should present failure cases if it has, and analyze the reasons.
• A2:
  • Thanks for the valuable suggestion. We have added the failure cases and the analysis in 13 of the revised paper.
  • We found the proposed method usually failed in two cases. First, there is a strong orientation mismatch between the visual condition face and the text prompt (e.g., back view text prompt but with front view face visual condition). Second, the model may fail when there is a clear text instruction of face coverings but the visual condition is a unobstructed face. In this case, the model may generate the covering at other random positions in the image.

• Q3: The inner diffusion cycles for "disturb", "reconstruct" may introduce additional computational overhead. More details on the computational balance on the number of cycles and its effect on the results should be given.
• A3:
  • The inner diffusion cycles for "disturb"and "reconstruct" are only conducted in part of the denoising path (20% of the whole process). The "disturb" step does not include additional computation overhead, since we just change the noise level from 500-th (low noise level) back to 700-th (high noise level). In this way, the denoising process can be viewed as returning back to the starting point of 700-th.
  • Since there is redundancy in the cyclical process, we can apply skip steps strategy of DDIM to speed up the "reconstruct" progress by 100x. As a result, our computation cost is only 20% higher than the original Stable Diffusion inpainting model (6s versus 5s).
  • To better address this concern, we have conducted more ablation studies to investigate the computation cost and performance of different cycle numbers and starting/ending points. The quantitive results are shown in the table below. As we can see, a cycle number of 10 is sufficient and achieves a good performance/speed balance for the inner diffusion process.

• Q4: There are typos in the paper, e.g., t around eq.(1). Please check the whole paper.
• A4: Thanks for pointing it out! We have checked the whole paper and polished the writing again. The updated version has been uploaded in OpenReview.

We greatly appreciate your insightful and detailed feedback. The comments and suggestions made us think more deeply about our method and were very helpful in improving the quality of our work. Please see our point-to-point response below:

• Q1: The proposed method repeatedly replaces part of the diffusion latent variable $x_t$ with the corresponding visual condition, which strongly maintains the visual condition in the generated image. However, this method may have an intrinsic drawback: the visual condition may be too strong and conflict with other conditions. in which case, the generated image may be unrealistic.
• A1:
  • In fact, the steps of "repeatedly replace" operation of the visual condition only occupy 20% of the generation process, while the text condition works throughout the whole process (from the layout to the detail). Our approach demonstrates a great ability to deal with different levels of conflicts between the visual and the textual conditions in a unified pipeline, as shown in Fig. 1 and Fig. 4 of our original submitted manuscript.
  • In the meanwhile, we note the reviewer's concern on the visual-textual conflict is more of a problem for general customization tasks, but not specific to the methodological design of COW. Essentially, it is about how we would want a generative model to comprise between controversial given conditions. For instance, existing methods such as DreamBooth may also produce unrealistic images given very different conditioning prompts.
  • In experiments we confirm COW is still able to handle different levels of conflicts. In Fig. 7 of the new appendix, we show a gradual change in different conflict levels of textual and visual conditions. COW could even fit in the case where the visual condition has a very strong conflict with other conditions (e.g., cross-domain transformation from human to lion in Fig. 8). This further validates our model has the ability to fit in both conditions at the same time.

• Q2: There are no quantitative analyses of the number of cycles, or positions of the start and end points. These experiments are important for us to understand the effectiveness of the proposed method.

• A2:

  • We further conduct an ablation study on different hyper-parameters of COW. The table below provides quantitative analysis on the ablation of different settings of cycle numbers, the position of the starting and the ending points. We also included the results in Table 4 of the revised Appendix.
  • Here we use three quantitative metrics to analyze different models. CLIP-T is the CLIP-text cosine-similarity, which is the similarity of generated image and text condition calculated by the prestrained CLIP (ViT-B/32) model. To better evaluate the visual quality of the generated samples, we also perform a human evaluation on those ablation models with 10 volunteers. Each volunteer carefully went through 216 images generated by the six model variants, and rated the best model variant as 5 and the worst as 0. Here we report the averaged ratings for each model. Since different starting and ending points would involve different inversion steps, the time cost is slightly difference according to the diffusion inversion cost.
  • The results show the model performance is sensitive to the starting and ending points, which confirms our motivation that we should repeatedly go through the Semantic Formation phase to have a controllable and directional generation.

  Cycle	Position     |	Clip-T$\uparrow$	Human Rating$\uparrow$	Time Cost$\downarrow$
  10	700$\Rightarrow$500  |	0.31	4.6	5.21
  10	900$\Rightarrow$700  |	0.30	1.5	5.52
  10	800$\Rightarrow$400  |	0.30	3.2	7.96
  10	500$\Rightarrow$300  |	0.27	0.8	4.85
  5	700$\Rightarrow$500  |	0.29	2.5	3.32
  20	700$\Rightarrow$500  |	0.30	3.6	8.83

Thank you for appreciating our contributions and providing valuable feedback! Please see our response below:

Q1: What is the level of changes that may occur in the region on the seed image following the cyclic one-way diffusion? According to the results presented some changes occur sometimes in the region of the seed image from the generated image, but not in other cases presented.

A1: The level of changes that may occur within the seed image depends on the discrepancy between textual and visual conditions. In most cases, the seed images can be well preserved given no explicit conflict between the visual-text conditioning pairs. At the same time, appropriate levels of text-guided changes can also be well reflected in the case of attribute editting, style transfer, and cross-domain transformation (as shown in Fig. 7 of the new supplementary file).