FashionR2R: Texture-preserving Rendered-to-Real Image Translation with Diffusion Models

We appreciate the reviewer for acknowledging the importance of the problem, the contribution of our dataset, the sound method and our presentation, the effective evaluation, and additional information in code and supplementary. The suggestions and questions also inspire us to improve our work. Below we address them in detail:

Some of the contributions of this work are very tied to a specific application. It is unclear how the proposed contributions, particularly the TAC could be leveraged for other applications. Are there any applications for which the proposed modules could be used, that are not specific to garment generation?

In general response, we further elaborate the scientific contributions of our work. In regard to other applications, our method can apply to more general rendered-to-real translation problems. We choose fashion images as they are the combination of highly articulated, complicated, structured, and meaningful content, which is also grounded in more direct commercial applications. In 3D games and animation, our method also have some potential if the problem of computational cost is mitigated in the future, as the reviewer mentioned and detailed in our discussion below. Besides, our method could also apply in other synthetic-to-real image generation applications, such as collecting data for model training in autonomous driving or robotics.

The proposed TAC module enables training-free control during the diffusion generation process. This feature allows for its application to a broader range of tasks, including text-to-image and image-to-image generation, such as personalized generation and image editing. As the number of model parameter becomes bigger and bigger, the training-free control can be utilized in an even wider variety of scenarios.

The analysis of the related work is somewhat lacking. Please see "Questions" for a few suggestions.

We thank the reviewer for suggesting these related works and will add discussion in our paper.

The proposed method is very computationally costly and makes it unsuitable for many applications, including interactive virtual try-on. A more powerful computer graphics engine (eg Unreal Engine 5) could achieve better and faster than the baselines utilized by this paper, and make the method in this paper less relevant.

We conduct some analysis on computational cost comparing to other SOTAs, in the rebuttal to Reviewer SbYx. Our method cannot handle real-time applications for now, but has potential for improvement with future integration with SD Turbo or SD Lightning. Note that comparing to other diffusion-base methods such as VCT, our method takes much less memory and time during testing as we do not need to perform additional optimization for each testing image.

As for rendering baselines, we build the 3D projects with Style3D Studio and use its integrated rendering tool based on rasterization. Using UE5 could potentially improve the rendering quality but will not diminish the effectiveness of our method. To verify this, we use more advanced rendering techniques via ray tracing (based on V-ray) to obtain rendered images, and our method consistently demonstrates its advantages in realism. Two visual examples are shown in Figure 2 of the PDF. We do not apply massive evaluation due to lack of time.

The paper is sometimes lacking in clarity, being overly convoluted in its writing. The title "Make Fashion Real" is confusing.

We thank the reviewer for bringing up these issues and will refine our writing. We consider to change "Make Fashion Real" into "FashionR2R" for better reference to the main focus of our paper.

We thank the reviewer for the recognition and encouragement on our novel pipeline, significantly more real qualitative results, effective quantitative results, and generally well-written paper. We would also like to thank the reviewer for the rigorous analysis and suggestions to help us improve our paper. Below we address the concerns and questions in detail:

Scientific contribution, insights, and interests for the community

We believe this is a misunderstanding and maybe due to somewhat lack of elaboration in our paper. In general response, we further explain our scientific contributions. We do not intend to compose an engineering pipeline to somehow manipulate diffusion generation, but propose to study a fundamental research problem, which might not be so obviously achieved and could be interesting to the NeurIPS community.

Quantitative results from the ablations

We conduct ablation study on two datasets in a drop-one-out manner. The quantitative results are in the table below and two visual examples are shown in Figure 2 of the attached PDF.

Without source DKI (embedding), the model tends to recover the input rendered image with DDIM inversion noise.

Without target DKI (finetuning), the rendering effect slightly decreases but the output is still not real enough due to lack of concentrated knowledge on real human and clothing. Lacking either component in DKI will make the output closer to the rendered input (higher SSIM and lower LPIPS), but much less real (higher KID).

Without TAC, the semantic structure such as face identity and clothing design can significantly deviate from the input. Although the output is clearly real (lower KID), it is almost completely different from the intended human and clothing (lower SSIM and higher LPIPS).

Other reconstruction metric (e.g. LPIPS, FID), standard deviations

As discussed in our paper and agreed upon by Reviewer SbYx, defining and quantifying "realism" still remains an open question. SSIM and KID roughly reflect the objective in enhancing photorealism while maintaining consistency. We follow the reviewer's suggestion and add LPIPS to evaluate texture preservation regarding rendered images, which is closer to human perception. The results are shown in the table below. We follow [2] and adopt KID rather than FID, for KID's unbiased nature and less sensitivity to sample sizes. Note that a good SSIM or LPIPS score does not guarantee better photorealism if the KID score is worse.

We' d also like to thank the reviewer for pointing out the missing information in statistical significance and add the standard deviations of the results in the table. The relatively low stds demonstrate the robustness of our method.

Additionally, as suggested by Reviewer SbYx, we conduct user studies to augment the quantitative evaluation. Our results in "real human perception" turn out be overwhelmingly better than other methods. Please refer to related rebuttal for more details.

Preserving appearance or enforcing realism.

The results we present in the paper do emphasize more on preserving appearance, under the assumption that the human model identity and clothing design should remain largely consistent to the input. However, the realism enhancement of our method could be more 'aggressive' given more tolerance to modifications to the input content. In Figure 6, we show an example of the trade-off in using different hyper-parameters.

The description of TAC is somewhat unclear.

We will modify the description of TAC. The CG-domain self-attention features are derived from the reverse sampling process starting from the noisy latent, which is obtained by performing DDIM inversion on the input image latent. In contrast, the R-domain self-attention features differ due to the incorporation of negative domain guidance and the self-attention injection.

Figure, table captions and typos

We thank the reviewer for pointing out these issues and will carefully modify our paper.

[2] Richter, Stephan R et al. "Enhancing photorealism enhancement." IEEE Transactions on Pattern Analysis and Machine Intelligence 45.2 (2022)

We thank the reviewer for the follow-up.

The standard deviations here show the variance over test inputs for a fixed model. For each method, the standard deviations of SSIM/LPIPS are calculated with this formula $\sigma = \sqrt{\sum{(x_i - \mu)^{2}}/{N}}$, where $x_i$ is the value in SSIM/LPIPS for the test input $i$, $N$ is the number of test input samples (7500), and $\mu$ is the corresponding mean of this metric over $N$ test inputs. For KID, we adopt the public API torch-fidelity [1]. We do not conduct calculation over training random seeds due to lack of time and have not noticed obvious instability over training seeds in experiments.

Additionally, we want to kindly remind the reviewer that defining and quantifying "realism" is still an open question, which means the metrics do not accurately align with the objective. A better score in a stand-alone metric, SSIM/LPIPS for preservation to rendered inputs, or KID for realism enhancement, does not always lead to advantages in reaching the goal of the task, due to the trade-off between these two goals. Given the roughly aligned metrics, the further statistical analysis might be a bit less informative than in other well-defined problems. Indeed, we have to admit that we could not emphasize or claim statistical significance in this work but only roughly demonstrate the stability and generalization ability, which is probably why the relevant information is missing in many generation/translation works. We want to thank the reviewer for noting this issue and will include discussion in our paper.

In the statistical comparisons, our SSIM is indeed very close to UNSB, but we do have a much lower LPIPS (0.1206 VS 0.2287 for FaceSynthetics, 0.0671 VS 0.1299 for SynFashion) and a much lower KID (73.831 VS 76.389 for FaceSynthetics, 54.720 VS 59.496 for SynFashion). As the reviewer suggested, LPIPS is better correlated with human perception than SSIM. The user studies in response to Reviewer SbYx also show significant statistical preference for our results over UNSB.

[1] High-fidelity performance metrics for generative models in PyTorch. (Zenodo,2020), github.com/toshas/torch-fidelity, Version: 0.3.0, DOI: 10.5281/zenodo.4957738

Thank you once again for taking the time to review our response. As the discussion stage is coming to an end, we would like to ask if all the issues have been satisfactorily addressed. If there are any remaining concerns, we would be more than happy to discuss them further. If you find that your concerns have been resolved, we would greatly appreciate it if you could consider raising your score. Thanks!

If the reviewer is interested in further analysis between our work and UNSB, we believe one main advantage of our method is that, we propose to control certain attributes concerning color and geometric information through self-attention injection with our TAC. This leverages the prior in the diffusion UNet structure and somehow "disentangles" the gaps between rendered image and real image domain to some extent. Note that the gaps between these two domains include not only fine-grained material properties and illuminaion, which are directly related to the "realism" we want to "enhance", but also other variables such as human poses and clothing styles that we want to maintain and preserve. This makes it difficult for general unpaired image-to-image translation methods to handle if there are no specific designs to deal with it, especially for images with higher resolution.

We appreciate the reviewer's positive comment and acknowledgement on the TAC design of our method, the value of our SynFashion dataset, and the experimental results. The suggestions on user studies and further discussion are very helpful. Below we address them in detail:

The paper acknowledges that defining and quantifying "realism" remains an open question. More robust metrics or user studies could provide better insights into the perceived realism of the generated images.

Defining and quantifying "realism" is indeed an open question, given the nuanced change between the input rendered image and its realistic counterpart, which is unknown and non-unique. We compute SSIM and LPIPS (as suggested by Reviewer cE2m, please refer to the related rebuttal) to roughly evalute texture preservation to rendered images, and use KID to measure the distance between the outputs and real photos in distribution. However, these metrics still cannot accurately reflect the objective of translation in photorealism.

We thank the reviewer for suggesting user studies to provide better insights quantitatively, regarding the perceived realism, image quality, and consistency to input rendered images. We follow [1] StyleDiffusion (ICCV 2023) in style-transfer and compare our method to previous works in pairs. Specifically, we sample 100 image pairs from each dataset for a user to choose, where each pair consists of one image generated from our method and a corresponding image generated with anthor random method side by side in random order. The users are asked to choose (1) which result is more realistic and (2) which result has overall better image quality (3) which result has better consistency with the rendered image.

We obtain approximate 2000 votes for each question from 20 users and show the percentage of votes that existing methods are prefered to ours in the following Table. The lower numbers indicate our method is more prefered than the competitors. Our method achieves overwhelming preference in overal realism and image quality, as well as obvious advantage in consistency.

Can you discuss the trade-offs involved in maintaining texture details versus achieving overall visual coherence in the generated images?

Maintaining centain parts of an image during generation, such as texture details, can lead to potential visual incoherence. One example is inpainting with diffusion models. However, in our work, our proposed TAC tackle this issue resorting to attention injection from a dual diffusion branch. Emperically, we discover the shallow layers of diffusion UNet preserves texture details while deep layers are more related to semantic information, and the injection of self-attention from shallow layers can blend smoothly with other features in the generation branch. In Figure 6, we show the effect of different ratio of TAC control steps during diffusion generation. We can see that the coherence in the generated image is not an obvious issue in our framework.

The two-stage process involving Domain Knowledge Injection (DKI) and Realistic Image Generation (RIG) can be computationally intensive. Training and deploying the framework may require substantial computational resources, which could limit its applicability in real-time or resource-constrained environments.

Computational cost and resource requirements are indeed relevant issues of our method, and probably similar to many diffusion-based methods. We thank the reviewer for mentioning this and will add it into our discussion on limitation and future work. Besides, we conduct some analysis on computational cost comparing to other SOTAs. We test the inference time and resource consumption for a 512x512 image on an RTX 3090, as shown in the table below. Note that comparing to VCT, which is also based on diffusion, our method takes much less memory and time during testing as we do not need to perform additional optimization for each testing image.

[1] Wang, Zhizhong, Lei Zhao, and Wei Xing. "Stylediffusion: Controllable disentangled style transfer via diffusion models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Thank you for taking the time to review our rebuttal. We appreciate your positive re-evaluation and look forward to integrating your insights into the final version of our paper.

We thank the reviewer for recognizing our contributions in the proposed dataset and "impressive" performance of our framework, as well as the valuable feedback and suggestions. Below we address the concerns and questions in detail:

There is no updating on diffusion structure but relies on pre-training and post-processing.

The main contribution of our proposed method is in regard to novel domain knowledge tranfering with established T2I diffusion-based models, combining with specific content control through attention injection. The TAC module is an exploration on diffusion structure and a slight modification of the structure to achieve fine-grained texture preservation. Our training and inference process is standard but convenient, without needing additional optimization or hand-crafted post-processing during testing.

The approach is more engineering application than academic paper.

As discussed in general response, our paper studies an important scientific problem, brings out a new perspective, proposes a novel pipeline, establishes a comprehensive benchmark, asks some essential questions, has potential in introducing more resources, and would work as an inspiration for future work.

The framework seems to lose or even change the background.

Changing image content is a well-known side-effect of image-to-image translation with generative models, especially for diffusion models.

In our paper, we propose a TAC module to mitigate this problem through attention injection from specific layers of the diffusion UNet. In particular, the high-level semantic structure such as face identity and clothing structure, as well as low-level texture patterns are largely preserved.

The background can consist semantic layout structure and texture details, which would lie in similar distribution in the image space and thus be preserved by our method, as shown in Figure 4.

In Figure 6, the background changes significantly when increasing denoising strength and decreasing ratio of TAC control steps, which is to show an effective range of the hyper-parameters and the trade-off between content preservation and realism enhancement.

The structure and grammar of the paper could be further improved for better clarity and readability.

We appreciate the suggestion and will modify our paper to improve its clarity and readability.

There is a problem regarding the diffusion models that where generated images exhibit significant randomness given the same segment mask. How does the author’s work address this problem?

Given the same input, the randomness can decrease when the denoising strength is lower and less noise is added to the input. However, to leverage the generative power for image translation, the denoising strength cannot be too low, as shown in Figure 4. In our work, we propose a TAC module to help preserve fine-grained structural image content, which can decrease its related randomness while better exploiting the diffusion model's generative power in other layers. Empirically, we found a combination of denoising strength 0.3 and TAC ratio 0.9 achieves the best balance.

There is a confusion in model training.

In Introduction Line 41, we mention that the T2I diffusion model is trained on real photos with derived captions from BLIP. In Method Line 166, we note that the text embedding is also an input in model training. We will further clarify this in our paper writing and Figure 1.

Does this framework perform differently on various patterns of texture? Such as a sentence?

Our framework can robustly handle various patterns including textual texture. The second row in Figure 3, the last row in Figure 8, and two additional examples in Figure 1 of the attached PDF are some visual examples.

Add some in-depth discussion on potential negative societal impacts and provide relative solutions.

We thank the reviewer for suggesting this and will add more discussion on potential negative societal impacts, including potential violation of portrait rights, racial bias in generation, NSFW content, etc. Relative solutions can include but are not limited to using authorized, diverse and balanced training data and training detection models to prevent inappropriate content generation.

Thank you for taking time to read the response and being positive about our submission. The reviewers updated their comments and we engaged in discussion. If there are any further questions or additional suggestions, please feel free to share them.