Mitigating Occlusions in Virtual Try-On via A Simple-Yet-Effective Mask-Free Framework

We appreciate all your valuable suggestions and constructive comments.

W1: The methodological descriptions and accompanying notations in the paper are a bit hard to follow. In particular, the explanation of the background pre-replacement strategy could be improved by supplementing the mathematical formulation with a clear, step-by-step visual illustration. Such a figure would enhance clarity and accessibility, as the current presentation may hinder comprehension.

R1: We appreciate your valuable suggestions, they are very helpful in improving our paper. In the revised version of this paper, we will follow your comments to add the visual illustrations to supplement the mathematical formulation of each step in our designed background pre-replacement strategy, so as to enhance the clarity and accessibility of such operation.

W2: The qualitative results are primarily demonstrated using images from fashion datasets or under controlled studio photography settings. It would be more meaningful to evaluate the proposed method on in-the-wild data to better assess its robustness and generalization to real-world scenarios. And Q3: Does the proposed method generalize well to in-the-wild images, particularly for subjects with greater variation in body shape, such as individuals with heavier or more muscular physiques?

R2: Due to the background augmentations used in the background pre-replacement operation, our method should technically be able to generalize to real-world in-the-wild images. In the first and last two rows of Figure 10 and the last two rows in Figure 11, we present some qualitative results with more practical backgrounds rather than plain colors. And moreover, the right block of Figure 8 also displays some results of heavier persons with sideways poses. Nevertheless, as suggested, we will add more results under in-the-wild scenarios in our revised version to further validate the generalization ability of our method.

Q1: Do g and g_un both refer to upper cloth？

RQ1: Actually, $g$ and $g_{un}$ indicate different garments from the same cloth category (e.g., lower body, upper body, or dresses). Specifically, $g$ denotes the garment that matches the one worn by the person in image $p$, forming a paired sample, whereas $g_{un}$ is a garment randomly sampled from the dataset.

Q2: It appears that the identity of the subject changes noticeably after applying the target clothing. What might be the underlying cause of this issue? Have the authors considered any strategies to preserve identity more effectively during the try-on process?

RQ2: This is mainly due to the limitations of latent diffusion models. Encoding and decoding human images with a pre-trained KL-regularized autoencoder (e.g., VQGAN) in latent diffusion models will inevitably discard a certain amount of face details (especially around the eyes) [1]. Most existing virtual try-on methods often mitigate such information loss by performing the following post-processing step, which adopts an inpainting mask to paste the human identity area from the original image onto the same region of the generated result.

$p_{result} = m_{agn} \odot p + (1 - m_{agn}) \odot p_{un}$ However, since the inpainting mask used above may also be imprecise and introduce some new occlusions, we abandon this post-processing step in our proposed framework. Prompted by your comment, we will incorporate optimization of the latent diffusion model generation quality [1] in our future work to achieve further improvements.

 

[1] Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models, CVPR 2025.

We appreciate all your valuable suggestions and constructive comments.

W1: The technical contribution is incremental. While the proposed approach for identifying a more reliable training objective is valuable, it appears to be more of an engineering effort, as the optimization target remains training or fine-tuning a generative model. Moreover, the problems of Inherent Occlusion and Acquired Occlusion have already been defined in [16], which further diminishes the novelty of this work.

RW1: We cannot agree with your comment that “The technical contribution is incremental” based on the reasoning that “training or fine-tuning a generative model appears to be more of an engineering effort”. As we know, “pre-training and fine-tuning” has already been a widely used paradigm to generalize the powerful foundation model that per-trained on large-scale datasets to the downstream tasks, such as virtual try-on. Since most downstream tasks have their own characteristics, designing a novel and effective fine-tuning strategy to address the problems arising in the generalization process is definitely a strong technical contribution. Moreover, although the problems of Inherent Occlusion and Acquired Occlusion have already been defined in [16], our work provides a more insightful analysis on the causes of these two types of occlusions (see Section 3). And after that, we designed a targeted solution based on the analysis, which is simple-yet-effective for addressing the occlusion problems in (mask-free) virtual try-on. Therefore, we believe that our proposed method has sufficient novelty.

W2: Although cloth/body occlusions are greatly improved based on the figures and supplementary materials, the generation quality of human faces and limbs is significantly worse than that of the baseline methods. The generated faces are often distorted and blurred. Is there a specific explanation for this?

RW2: This is mainly due to the limitations of latent diffusion models. Encoding and decoding human images with a pre-trained KL-regularized autoencoder (e.g., VQGAN) in latent diffusion models will inevitably discard a certain amount of face details (especially around the eyes) [1]. Most existing virtual try-on methods often mitigate such information loss by performing the following post-processing step, which adopts an inpainting mask to paste the human identity area from the original image onto the same region of the generated result.

$p_{result} = m_{agn} \odot p + (1 - m_{agn}) \odot p_{un}$ However, since the inpainting mask used above may also be imprecise and introduce some new occlusions, we abandon this post-processing step in our proposed framework. Prompted by your comment, we will incorporate optimization of the latent diffusion model generation quality [1] in our future work to achieve further improvements.

 

[1] Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models, CVPR 2025.

Q1: It seems that the mask-free approach serves as an additional image refinement step built upon the inpainted results from mask-based methods. Similarly, the proposed approach appears to be an extra refinement step based on a pre-trained mask-free model. Is this understanding correct? If so, direct comparisons with mask-based methods may not be strictly fair, given that those methods involve little to no refinement.

RQ1: This is a misunderstanding. The mask-free and mask-based strategies are two different paradigms to address the unpaired data problem, and the greatest difference between them is that the mask-free methods DO NOT require to generate inpainting masks in the inference phase. To eliminating such requirement, the mask-free methods can synthesize pseudo reference images from a pre-trained generative model (SD1.5 used in our implementation) or a mask-based method. So the mask-free approach is not an additional image refinement step for mask-based methods. In addition, our proposed method is built by fine-tuning the SD1.5 diffusion model from scratch not on another pre-trained mask-free model, so it does not serve as an extra refinement step. As shown in Figure 2, similar to other mask-free methods, our approach first synthesizes and selects the pseudo reference images and their corresponding occlusion-augmented person images. During this process, our designed background pre-replacement and covering-and-eliminating operations are employed to deal with inherent and acquired occlusions, respectively. After that, we fine-tune the SD1.5 diffusion model, which is a widely-used foundation model for virtual try-on tasks, by using the selected paired samples. According to the above explanation, since both mask-free and mask-based methods start from the same pre-trained diffusion model, and mask-free methods (including our approach) even do not utilize the inpainting mask information during the inference phase, we believe that the comparisons between our method and mask-based methods are sufficiently fair.

Q2: Why did the authors use SD1.5 instead of SDXL as the base generative model? BooW-VTON has shown that SDXL produces better results, so it is unclear why SDXL was not used.

RQ2: To make fair comparison with existing virtual try-on benchmarks, we adopt their commonly used SD1.5 backbone. And moreover, due to the limited computational resources of our laboratory, it is infeasible to conduct experiments using SDXL. Based on the above reasons, we use SD1.5 instead of SDXL as our base generative model. Nevertheless, the qualitative results in Figures 3 and 4 and the quantitative results in Table 2 demonstrate that, even with the relatively weaker SD1.5 backbone, our method outperforms SDXL-based counterparts such as BooW-VTON and IDM-VTON on several metrics. This provides sufficient evidence for the effectiveness of our approach.

Q3: In Section 4.2, Equation 10, the gating parameter $\gamma$ is determined solely by the sizes of the neck and arms. What is the rationale behind using only these two body parts?

RQ3: We have explained the reason for using these two body parts in Lines 198–202 of our paper. The gating parameter $\gamma$ is used to indicate whether the garment $g_{un}$ fully covers $g$ or not. For the upper-body clothing, the body areas that are most likely to expose skin are the arms and neck. Therefore, we use these two body parts to generate $\gamma$ so as to determine the coverage of different garments.

Q4: The authors claim that their method generalizes to in-the-wild scenes. However, according to Section 5.2, the model was trained on the VITON-HD dataset and tested on the DressCode dataset, which is not strictly in-the-wild evaluation. I suggest including additional experiments on real-world images of humans and clothing sourced from the internet.

RQ4: Considering the background augmentations used in our background pre-replacement operation, the proposed method should technically be able to generalize to in-the-wild images. In the first and last two rows of Figure 10 and the last two rows in Figure 11, we present some qualitative results with more practical backgrounds rather than plain colors. And moreover, the right block of Figure 8 also displays some results of heavier persons with sideways poses. Nevertheless, as suggested, we will add more results under real-world in-the-wild scenarios in our revised version to further validate the generalization ability of our method.

We appreciate all your valuable suggestions and constructive comments.

Q1: From the visualizations in Fig.3 and in supplementary materials, I noticed that there exists common distortion in human models' face area, especifically around the eyes. In addition, the generated image resolution of the proposed method seems to be lower than comparison baselines. Do authors notice these issues, and what do you think might be the cause?

R1: Yes, we have noticed these issues. We discuss the causes of them as follows:

(1) The reason of human face distortion is because that encoding and decoding human images with a pre-trained KL-regularized autoencoder (e.g., VQGAN) in latent diffusion models will inevitably discard a certain amount of face details (especially around the eyes) [1]. Most existing virtual try-on methods often mitigate such information loss by performing the following post-processing step, which adopts an inpainting mask to paste the human identity area from the original image onto the same region of the generated result. $p_{result} = m_{agn} \odot p + (1 - m_{agn}) \odot p_{un}$ However, since the inpainting mask used above may also be imprecise and introduce some new occlusions, we abandon this post-processing step in our proposed framework. Prompted by your comment, we will incorporate optimization of the latent diffusion model generation quality [1] in our future work to achieve further improvements.

(2) The lower image resolution is mainly caused by the fewer inference steps we adopt. In order to speed up the visualization process of the qualitative results of our method, we only run 20 denoising steps during inference while other comparison methods employ 50 denoising steps. If more inference steps are used, our method can generate better results with higher resolution.

 

[1] Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models, CVPR 2025.

Q2: From the illustration of Background Pre-Replacement process in Figure 13, I notice that there still exists inherent occlusion issue, where the mask for right arm is not accurate. This leads to another question regarding the background pre-replacement process. It essentially depends on the quality of the segmentation mask, so how can it eliminate the inherent occlusion issue brought out by inaccurate masking?

R2: This may be a misunderstanding. Our background pre-replacement operation is designed to address the problem of inaccurate segmentation masks. Take Figure 13 as an example, if without the background pre-replacement operation, the clothing in the unmasked areas of the right arm will be incorrectly preserved in the pseudo reference image. In the correct situation, such clothing information should be generated by the model, but now they can be simply maintained like the human body or background because they appear in both pseudo reference image and ground-truth image. Therefore, training with these samples can establish incorrect associations between the target clothing information and the human body or background pixels. In contrast, by using our background pre-replacement operation, we replace the inaccurate unmasked clothing areas in both pseudo reference image and ground-truth image with the same pure background or random scene image. By doing so, those areas will only contain the background information that should be maintained during the generation process, instead of target clothing pixels (as illustrated in Figure 2). Therefore, the clothing information in other accurately masked areas that need to be generated can be more effectively differentiated from the background, thus leading to better generation results.

We appreciate all your valuable suggestions and constructive comments.

W1: Quality of the results: In Figure 5 there are still seams/mismatches in the "GANs + Ours" column. And Q2: It seems that in Figure 5 (GANs), there are some artifacts in the results. Can the authors explain why? Is there a significant difference in the results for GANs as compared to diffusion models?

R1: As shown in Figure 5, there are some artifacts observed in the experimental results of both "GANs" and "GANs+Ours". But when we apply our method to a diffusion-based backbone (SD 1.5 used in this work), we can find that these artifacts are eliminated (see our results in Figure 3 and 4). So we believe that such problem lies in the relatively weaker generative capabilities of the GANs backbone compared to diffusion models. The adversarial training process of GANs is usually unstable and may introduce those undesirable artifacts into the final results. In addition, the quantitative comparisons in Table 2 and 3 also demontrate that the diffusion-based methods consistently achieve the best and second-best results across all evaluation metrics, outperforming the GANs-based benchmarks. Therefore, there is a significant difference in results between diffusion models and GANs.

W2: Data: only standard fashion datasets were used, limiting the variety of clothing, body shapes, and poses.

R2: To ensure a fair comparison with existing methods, we employ the same datasets as them for our performance evaluation. The DressCode dataset, in particular, comprises 53,792 garment images spanning three categories—upper body, lower body, and dresses—and an equal number of human images featuring diverse skin tones, body shapes, and poses. Therefore, achieving state-of-the-art results in both quantitative and qualitative experiments on this dataset, especially in handling occlusions, is strong evidence of our method’s effectiveness and its advantages over existing approaches.

W3: There are no examples of results with more complicated backgrounds even though the model technically should be able to handle such examples due to the background augmentations that are used. Also see Line 305: “The results in Fig. 6 show that our method can better handle person images with diverse background styles” -> Fig. 6 contains examples with mostly white backgrounds only. And Q3: Can the authors introduce some qualitative results with more complex backgrounds?

R3: In the first and last two rows of Figure 10 and the last two rows in Figure 11, we present some qualitative results with more complex backgrounds rather than plain colors. We can see that our method can handle these background styles well. Moreover as suggested, in the revised version of this paper, we will show more results on these backgrounds and introduce other results under in-the-wild scenario in Figure 6 to support our claim.

W4: The paper does not contain many examples of failure. And Q1: I think that showing and discussing limitations even further in this work would be useful for the community. The paper already identifies and suggests potential solutions to two interesting limitations. Can the authors include more qualitative results that depict the failures of the proposed method (showing a larger variety of clothing, body shapes, and poses)?

R4: We have already discussed the limitations and future work in Section F of our Appendix and have displayed a failure case in Figure 12. Nevertheless, as suggested, we will add more examples of failure in our revised version to better illustrate the current limitations of the proposed method and guide our future work.

W5: There are some typos, for example: Line 165 aim -> aims?

R5: Thanks for pointing out this. We have carefully checked our paper and will correct all such typos (including L165: aim -> aims) in the revised version.