Enhancing Virtual Try-On with Synthetic Pairs and Error-Aware Noise Scheduling

1. The paper does not offer significant technical contributions, primarily combining existing methods, such as combining GAN outputs with I2SB for post-processing.

The proposed EARSB is a training-based refinement method that learns to refine the artifacts of an initial image. The major contributions of our work are two folds: a) from the data perspective, we propose a synthetic data augmentation strategy to improve virtual try-on models; b) from the model perspective, we introduce spatially adaptive noise schedule in the Schr"odinger Bridge, which improves the low-quality region of an initially generated image based on a spatially-varying noise schedule that targets known artifacts.

2. According to Table 3, combining existing methods with synthetic data sometimes yields better performance than the proposed method.

The superior performance of Stable-VTON+H2G-UH in Table 3 is partially attributed to the powerful generation ability of Stable Diffusion backbone, which is pretrained on millions of images. We updated Table 1 by using the SD-based model CatVTON to generate the initial image in EARSB(SD)+H2G-UH/FH. See the comments posted for reviewer RZB7 for the updated Table 1. It is shown that EARSB(SD)+H2G-UH/FH gains over the SD-based methods, including Stable-VTON+H2G-UH.

3. Minor formatting issues. For example, some equations (e.g., Eq 2 and Eq 8) are missing proper punctuation.

Thanks for pointing out the typos. We've corrected them in the revised draft.

4. Inconsistency in terminology, with both "error-map" and "error map" used throughout the paper.

We use "error-map" as a compound modifier before a noun (e.g., error-map-reweighted noise schedule) and "error map" as a noun phrase.

5. I feel the training process of EARSB is not clearly explained. How are data pairs prepared for this stage? What initialization is used for the U-Net, and what is the training loss?

The training of EARSB is conducted in three steps: 1) Preprocess the images in the training set and feed the preprocessed inputs to existing try-on models to get the initial images $x_1$; 2) Obtain the error maps $M$ on the initial images $x_1$ using our WSC. 3) Use $M$ to reweight the noise schedule in I$^2$SB and train the noise prediction UNet in EARSB following Eq. (9). We have clarified the above procedure in the revised draft.

We use Xavior initialization for the UNet. Following prior diffusion models, the training loss is a L2 loss between the predicted noise and the true noise. More details are included in the appendix.

6. In I2SB, training assumes that the posterior, given a boundary pair, follows a Gaussian distribution. Does WSC affect the Gaussian properties?

WSC does not affect the Gaussian properties because the predicted error map acts as a multiplier to the Gaussian noise in I2SB, which only affects its variance.

We thank the reviewer for the informative feedback and address the concerns below:

1. IDM-VTON, OOTDiffusion, and CATVTON should perhaps be included in the comparison experiments.

IDM-VTON from ECCV 2024 was published at the time of the submission. OOTDiffusion is an unpublished work and CATVTON is a concurrent work. To compare with the most recent model that shows better performance, we use CatVTON, which is a concurrent work, as one of our baselines. The experiment section and Table 1 have been updated accordingly as below:

In the updated table, EARSB(SD)+H2G-UH/FH uses diffusion model CatVTON to generate the initial image. In summary, EARSB and EARSB+H2G-UH/FH improve the GAN baselines while EARSB(SD)+H2G-UH/FH gains over the SD-based methods. Overall, our refinement model improves the metrics of existing methods on both datasets, with further improvements from incorporating synthetic training pairs.

2. The results of LaDI-VTON on VITON-HD and Stable-VITON on DressCode-Upper are missing in Table 1.

Thanks for catching this! The results of LaDI-VTON on VITON-HD are now included in Table 1 as the above. As for Stable-VITON, the authors have not released their model checkpoint on DressCode-Upper. We require all diffusion models to be evaluated using the same number of sampling steps for fair comparison. Therefore, their results on DressCode-Upper will remain blank in Table 1.

3. Although there are some visualized examples of the synthetic garment images, I am somewhat concerned about the quality of the generated images. Could you provide some quantitative result to support this?

Yes, we compute the FID, KID, SSIM and LPIPS of the generated garment on the paired test set of VITON-HD and DressCode-Upper, respectively. Results are reported under 1024x1024 image resolution:

4. I am curious whether EARSB could have a similar effect on diffusion-generated images as well.

In the updated Table 1, we use the diffusion model CatVTON to generate the initial image in EARSB(SD)+H2G-UH/FH. It is shown that EARSB(SD)+H2G-UH/FH gains over the SD-based methods, which is a consistent conclusion we draw from using GAN-generated image as the initial image. Note that while EARSB(SD)+H2G-UH/FH shows better performance, it also brings extra computational costs for obtaining the initial image.

5. Could you analyze the impact of the annotation ratio on model performance?

The reason for annotating a small portion of the generated images is to get reasonable detection results of the artifacts while minimizing manual labeling efforts. It is expected to get higher detection accuracy with higher labeling costs and a higher annotation ratio. 5% might not be the optimal ratio to balance the detection accuracy and the labeling efforts, but it is cost-effective in making the classifier outperform self-supervised and unsupervised training methods, as illustrated in Figure 7. The refinement results in Table 1 also demonstrate that it is effective in improving existing methods. In future work, following the data scaling law, we will explore the relationship between the labeling ratio, the time spent on annotating the data, and the final model performance.

6. I would like to confirm which dataset the human-to-garment model was trained on. Was it VITON-HD or DressCode?

For H2G-UH which contains mostly upper-body human images, we train the human-to-garment model on VITON-HD and for H2G-FH which contains mostly full-body human images, the human-to-garment model is trained on DressCode-Upper.

We thank the reviewer for the informative feedback and we address the concerns as follows:

1. Novelty

The major contributions of our work are two folds: a) from the data perspective, we propose a synthetic data augmentation strategy to improve virtual try-on models; b) from the model perspective, the major difference between the proposed EARSB and prior diffusion models is the introduced spatially adaptive noise schedule, which improves the low-quality region of an initially generated image based on a spatially-varying noise schedule that targets known artifacts. To clarify the second contribution, we have highlighted the spatially adaptive noise schedule in the introduction section of the revised draft.

2. Relevance

Mentioning that our weakly-supervised classifier can be adapted to other tasks is a great suggestion. We added the following description in the first paragraph of Section 3.4 : "This refinement goal applies to the general setting where we want to refine a flawed image output. In the scope of this paper, we focus on the virtual try-on task. "

3. Details about the manual annotation

The reason for annotating a small portion of the generated images is to get reasonable detection results of the artifacts while minimizing manual labeling efforts. It is expected to get higher detection accuracy with higher labeling costs and a higher annotation ratio. 5% might not be the optimal ratio to balance the detection accuracy and the labeling efforts, but it is cost-effective in making the classifier outperform self-supervised and unsupervised training methods, as illustrated in Figure 7. The refinement results in Table 1 also demonstrate that it is effective in help improving existing try-on methods. In future work, following the data scaling law, we will explore the relationship between the labeling ratio, the time spent on annotating the data, and the final model performance.

4. Target clothing shape

The shape of the upper clothing when worn by an individual is primarily decided by two factors: a) the physical interactions between the upper clothing and the lower clothing; b) the size match between the garment and the individual. For a, n example is that the tank top in ll328-333 becomes shorter on the human because it is tucked into the pants. For b, an example is that the same outfit could look loose or tight on different people depending on their body shape. These are difficult to solve in the 2D domain without knowing the garment or body measurement data. Some prior work addressed this problem in a human-interactive setting [1,2]. In this paper, we focus on non-interactive virtual try-on and only aim to transfer the texture of the garment to the person in a reasonable shape. Note that our EARSB is a refinement-based model that targets known artifacts. Therefore, it is expected to follow the shape of the clothing in the initial image unless it is misaligned with the pose or has other artifacts around the boundaries. In future work, we might explore incorporating human interactions into the model to control the dressing style and clothing length.

[1] Chen, C. Y., Chen, Y. C., Shuai, H. H., Cheng, W. H. Size does matter: Size-aware virtual try-on via clothing-oriented transformation try-on network. In ICCV 2023.

[2] Pan X, Tewari A, Leimkühler T, Liu L, Meka A, Theobalt C. Drag your gan: Interactive point-based manipulation on the generative image manifold. In ACM SIGGRAPH 2023.

5. What is the motivation behind using a GAN based base network as opposed to a diffusion based base network?

This is because generating the initial image using a GAN model is cheaper as the diffusion-based model has a notoriously low sampling speed. To test EARSB's performance without considering the sampling cost, we use the diffusion model CatVTON to generate the initial image and refine it with our EARSB. We have updated Table 1 with the new results, denoted as EARSB(SD) +H2G-UH/FH. See the comments posted for reviewer RZB7 for the updated Table 1. EARSB and EARSB+H2G-UH/FH improve the GAN baselines while EARSB(SD)+H2G-UH/FH gains over the SD-based methods. Overall, our refinement model improves the metrics of existing methods on both datasets, with further improvements from incorporating synthetic training pairs.

6. How many total number of images are 5% of generated images?

We use three GAN models to generate the initial image in training, which makes the total number of annotations 30.0511647=1747 on VITON-HD and 30.0513564=2034 on DressCode-Upper.

We thank the reviewer for the informative feedback and we address the concerns as follows:

1. There is no way to prove whether EARSB can be applied to other methods and bring about similar performance improvements.

The proposed synthetic data augmentation strategy has been proven to be beneficial when added to other try-on models. At the same time, the introduced EARSB model architecture is specific for the virtual try-on task and might need major architectural changes to be adapted to other vision tasks. We mainly focus on virtual try-on under the scope of this paper and therefore did not test EARSB on other applications. But, the refinement-based idea in EARSB still applies to the general setting where we want to refine a flawed image output, as suggested by reviewer d9nt. In future work, we could expand the scope and apply EARSB to other image editing applications.

2. More visualizations are needed to show how good regions are assigned less noise and bad regions are assigned more noise. Simply relying on Figure 4 does not adequately convey the precise meaning of this innovation.

More visualized examples of the error map can be found in Figure 10 of the Appendix. The error map reweights the noise schedule such that good regions are assigned less noise and bad regions are assigned more noise. More precisely, we multiply the noise distribution $\epsilon$ with the error map $M$ as in Eq. (4). As a result, a higher volume of noise is assigned to the low-quality regions so the model can focus on refining these regions. The intuition behind this is that good regions need less modification learned from the diffusion process while bad regions need more. We have replaced Figures 3 and 4 with another image example to better clarify the benefit of the noise reweighting.

3. The Diffusion-VTON series of methods lacks comparisons with some necessary approaches, such as quantitative and qualitative comparisons with IDM-VTON, DCI-VTON, and other methods.

IDM-VTON from ECCV 2024 was published at the time of the submission and DCI-VTON is an earlier method published in ACM MM 2023. To compare with the most recent model that shows better performance, we use CatVTON, which is a concurrent work, as one of our baselines. The experiment section and Table 1 have been updated accordingly. See the comments posted for reviewer RZB7 for the updated Table 1. In the updated table, EARSB(SD)+H2G-UH/FH uses diffusion model CatVTON to generate the initial image. In summary, EARSB and EARSB+H2G-UH/FH improve the GAN baselines while EARSB(SD)+H2G-UH/FH gains over the SD-based methods. Overall, our refinement model improves the metrics of existing methods on both datasets, with further improvements from incorporating synthetic training pairs.

4. The explanation of the EARSB model in the paper is not sufficient, especially regarding the details of how the model identifies and corrects local errors. Can the optimization of local details be achieved solely through weakly supervised masks?

We have revised the draft in Section 3.2 to clarify our model EARSB. Our model identifies the local errors in the error map predicted by the weakly-supervised classifier and refines these errors via the Schr"odinger Bridge sampling with spatially adaptive noise schedule reweighted by the error map. Technically, the local refinement can be solely achieved by inpainting the erroneous regions in the weakly supervised masks. However, learning the error-to-ground-truth mapping in our EARSB will provide more explicit guidance on how to map specific errors for refinement.

We updated Table 3(c) with the above inpainting method, denoted as Inpaint below. Inpaint's degraded performance demonstrates the importance of learning the error-to-ground-truth refinement in our model.

5. The paper mentions that SD-based methods require large-scale pretraining, while EARSB does not. How does EARSB ensure the effectiveness and generalization ability of VTON in scenarios beyond the dataset?

From the perspective of a fair comparison, directly comparing EARSB and SD-based models is less fair because SD-based methods are trained on large-scale data. However, since EARSB is a refinement-based model, we can use images generated by SD-based methods as the initial image and refine their clothing textures in EARSB. This is an implicit approach to ensure the generalization ability of EARSB. We've updated Table 1 to include this approach, denoted by EARSB(SD)+H2G-UH/FH. See the comments posted for reviewer RZB7 for the updated Table 1. In the updated table, EARSB(SD)+H2G-UH/FH uses diffusion model CatVTON to generate the initial image, showing performance gains over the SD-based methods.