Incorporating Visual Correspondence into Diffusion Model for Virtual Try-On

(continued from Part 1/2)

Q3: Semantic point matching for loose garments.

A3: Yes, we agree that for the cases of loose garments like dresses or skirts, some semantic points of the garment may fail to exactly fall on the human body. However, these points still convey certain local details of the garment (garment features), and the visual correspondence prior in latent space is also robust to warping results (see discussion in Q1). This facilitates VTON of loose garments in local detail preservation. Both Figure 7 in the appendix and the attached Fig. G demonstrate the effectiveness of our SPM-Diff for loose clothes like dresses or skirts.

Q4: Details of garment/geometry feature encoders.

A4: To be clear, the garment/geometry feature encoder shares similar model structure as Garment/Main-UNet. As introduced in L363-365, we jointly pre-train these two encoders in a basic dual-branch network with additional depth/normal map as inputs but without our proposed SPM mechanism, i.e., the ablated run Base+Geo map in Table 4 of the main paper. Then, the pre-trained garment/geometry feature encoder is utilized to extract garment/geometry point features, which are further injected into the self-attention layers of Main-UNet in our SPM-Diff. We will add the details in revision.

Reference:

[1] Tang, Luming and Jia, Menglin and Wang, Qianqian and Phoo, Cheng Perng and Hariharan, Bharath. Emergent correspondence from image diffusion. In Neurips, 2023.

[2] Xu, Yuhao and Gu, Tao and Chen, Weifeng and Chen, Chengcai. Ootdiffusion: Outfitting fusion based latent diffusion for controllable virtual try-on. In arXiv preprint arXiv:2403.01779, 2024.

[3] Chong, Zheng and Dong, Xiao and Li, Haoxiang and Zhang, Shiyue and Zhang, Wenqing and Zhang, Xujie and Zhao, Hanqing and Liang, Xiaodan. CatVTON: Concatenation Is All You Need for Virtual Try-On with Diffusion Models. In arXiv preprint arXiv:2407.15886, 2024.

[4] Xie, Zhenyu and Huang, Zaiyu and Dong, Xin and Zhao, Fuwei and Dong, Haoye and Zhang, Xijin and Zhu, Feida and Liang, Xiaodan. Gp-vton: Towards general purpose virtual try-on via collaborative local-flow global-parsing learning. In CVPR, 2023.

We sincerely appreciate the kind comments from Reviewer THne. We also thank Reviewer THne for recognizing the merits in our paper (e.g., the idea is interesting and reasonable), and your kind patience in engaging in this rebuttal. Let us address some key concerns here. We hope the information here may further supplement our main submission and justify the soundness of SPM-Diff. We will include the primary results and discussion here in the main paper.

Q1: Contribution.

A1: Appreciate the comment. Existing warping-based methods (e.g., the mentioned GP-VTON) commonly adopt warping model to warp the input garment according to the input person image, and then directly leverage the warped garment image as hard/strong condition to generate VTON results. This way heavily relies on the accuracy of garment warping, thereby easily resulting in unsatisfactory VTON results given the inaccurate warped garments under challenging human poses (see the results of GP-VTON in Fig. E). On the contray, our proposed SPM-Diff adopts a soft way to exploit the visual correspondence prior learnt via the warping model as a soft condition to boost VTON. Technically, our SPM-Diff injects the local semantic point features of the input garment into the corresponding positions on the human body according to a flow map estimated by the warping model. This way nicely sidesteps the inputs of holistic warped garment image with amplified pixel-level projection errors, and only emphasizes the visual correspondence of the most informative semantic points between in-shop garment and output synthetic person image. Note that such visual correspondence of local semantic points are exploited in latent space (corresponding to each local region), instead of precise pixel-level location. Thus when encountering mismatched points with slight displacements within local region, SPM-Diff still leads to similar geometry/garment features as matched points, making the visual correspondence more noise-resistant (i.e., more robust to warping results). As shown in Fig. E, given warping results with severe distortion, our SPM-Diff still manages to achieve promising results, which basically validate the effectiveness of our proposal. We will add the results and discussion in revision.

Q2: Evaluation of the point-matching method.

A2: Thanks for this point. As suggested, we examine the point matching capability of the pre-trained warping module adopted in our SPM-Diff, in comparison with a vanilla stable diffusion model. Note that here we use a training-free approach (DIFT [1]) as baseline that exploits the stable diffusion model to match the corresponding points between two images. Specifically, we construct the point matching test set by manually annotating semantic points on a subset of garment images and their corresponding points on the person image. Next, given the target semantic points on the garment image, our adopted warping module and DIFT perform point matching on the person image. Finally, we report the mean square error (MSE) between the matched points via each run and the annotated ground truth. As shown in the table below, our adopted warping module demonstrates stronger point matching capability than the vanilla stable diffusion model. In addition, we visualize the point matching results of each run in Fig.F, and our adopted warping module yields more accurate point-matching results than vanilla stable diffusion model. We will add these results and discussions in revised version.

(To be continued in Part 2/2).

We sincerely appreciate the kind comments from Reviewer SQRJ. We also thank Reviewer SQRJ for recognizing the merits in our paper (e.g., precise and accurate garment reproduction), and your kind patience in engaging in this rebuttal. Let us address some key concerns here. We hope the information here may further supplement our main submission and justify the soundness of SPM-Diff. We will include the primary results and discussion here in the main paper.

Q1: Sensitivity of semantic point count.

A1: Thanks for this point. Yes, we agree that significantly increasing semantic point count (e.g., from $K = 25$ to $K = 50$ or $75$) will result in redundancy and interpolation/projection error, leading to degraded results as discussed in the main paper. However, when taking an in-depth look at the sensitivity of semantic point count around the optimal value $K = 25$, the semantic point count is not sensitive. To validate this, we experimented by varying semantic point count $K$ in the range of $[20, 40]$ within an interval of 5 on VITON-HD dataset. As shown in the table below, the performance under each metric only fluctuates within the range of 0.1, which basically validates that the performance is not sensitive to the change of semantic point count within this range. We will include the results and discussion in the revised version.

Q2: Accurate 3D conditions.

A2: It is worth noting that the 3D cues (i.e., depth/normal map) adopted in our training and inference are the estimated results by using pre-trained 3D human reconstruction model (OSX [2]), rather than manually annotated accurate 3D conditions. Considering this comment, we conducted experiments by evaluating SPM-Diff with varied estimated 3D cues achieved from different 3D human reconstruction models (e.g., PyMAF-X [1], OSX [2], 4DHumans [3]) on VITON-HD dataset. The results are summarized in the table below, and similar performances are attained across different pre-trained 3D human reconstruction models.

Moreover, to further evaluate the generalization of our SPM-Diff under challenging scenario, we take in-the-wild person images with complex backgrounds as inputs (in contrast to the clean backgrounds in VITON-HD dataset). As shown in Fig. C, our SPM-Diff manages to achieve promising results.

(To be continued in Part 2/2).

(continued from Part 1/2)

Q3: User-centered metrics such as perceived realism.

A3: Appreciate this comment. As suggested, in addition to widely adopted metric FID for image realism evaluation, we include a new user-centered metric to access image realism in user study. In this way, we have three criteria in user study: 1) preservation of the fine-grained appearance details, 2) preservation of the garment contour across diverse human body shapes, 3) the perceived image realism. We show the additional comparison results of image realism evaluation in the attached Fig. D, which again demonstrates the superiority of our SPM-Diff against all baselines in terms of image realism. These results and discussion will be added in revised version.

Q4: Contribution; warp-based methods.

A4: We appreciate the suggested reference WarpDiffusion [5], and we are happy to discuss the differences between our SPM-Diff and these existing warp-based methods [4, 5]. In particular, existing warp-based methods [4, 5] commonly adopt warping model to warp the input garment according to the input person image, and then directly leverage the warped garment image as hard/strong condition to generate VTON results. This way heavily relies on the accuracy of garment warping, thereby easily resulting in unsatisfactory VTON results given the inaccurate warped garments under challenging human poses (see the results of GP-VTON in Fig. E). On the contrary, our proposed SPM-Diff adopts a soft way to exploit the visual correspondence prior learnt via the warping model as a soft condition to boost VTON. Technically, our SPM-Diff injects the local semantic point features of the input garment into the corresponding positions on the human body according to a flow map estimated by the warping model. This way nicely sidesteps the inputs of holistic warped garment image with amplified pixel-level projection errors, and only emphasizes the visual correspondence of the most informative semantic points between in-shop garment and output synthetic person image. Note that such visual correspondence of local semantic points are exploited in latent space (corresponding to each local region), instead of precise pixel-level location. Thus when encountering mismatched points with slight displacements within local region, SPM-Diff still leads to similar geometry/garment features as matched points, making the visual correspondence more noise-resistant (i.e., more robust to warping results). As shown in Fig. E, given warping results with severe distoration, our SPM-Diff still manages to achieve promising results, which basically validate the effectiveness of our proposal. We will add the results and discussion in revision.

References:

[1] Zhang, Hongwen and Tian, Yating and Zhang, Yuxiang and Li, Mengcheng and An, Liang and Sun, Zhenan and Liu, Yebin. Pymaf-x: Towards well-aligned full-body model regression from monocular images. In IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.

[2] Lin, Jing and Zeng, Ailing and Wang, Haoqian and Zhang, Lei and Li, Yu. One-stage 3d whole-body mesh recovery with component aware transformer. In CVPR, 2023.

[3] Goel, Shubham and Pavlakos, Georgios and Rajasegaran, Jathushan and Kanazawa, Angjoo and Malik, Jitendra. Humans in 4D: Reconstructing and tracking humans with transformers. In ICCV, 2023.

[4] Xie, Zhenyu and Huang, Zaiyu and Dong, Xin and Zhao, Fuwei and Dong, Haoye and Zhang, Xijin and Zhu, Feida and Liang, Xiaodan. Gp-vton: Towards general purpose virtual try-on via collaborative local-flow global-parsing learning. In CVPR, 2023.

[5] Li, Xiu and Kampffmeyer, Michael and Dong, Xin and Xie, Zhenyu and Zhu, Feida and Dong, Haoye and Liang, Xiaodan and others. WarpDiffusion: Efficient Diffusion Model for High-Fidelity Virtual Try-on. In arXiv preprint arXiv:2312.03667.

We sincerely appreciate the kind comments from Reviewer 37Fc. We also thank Reviewer 37Fc for recognizing the merits in our paper (e.g., the model's robustness and superior performance), and your kind patience in engaging in this rebuttal. Let us address some key concerns here. We hope the information here may further supplement our main submission and justify the soundness of SPM-Diff. We will include the primary results and discussion here in the main paper.

Q1: Details of local flow warping process.

A1: To be clear, the local flow warping module is employed to estimate the dense displacement field $M_{G \rightarrow H} \in \mathbb{R}^{H \times W \times 2}$ from in-shop garment $I_g$ to the target human (i.e., the condition triplet $C$ comprising human pose, densepose pose, and preserved region parsing). Each element $M_{G \rightarrow H}^{(i,j)}$ represents the relative displacement (i.e., 2D offset vector) for each point at coordinate $(i, j)$ on in-shop garment $I_g$ relative to the person image $I_p$. Here we employ flow warping module like GP-VTON to estimate the dense displacement field in between. Formally, given in-shop garment $I_g$ and the condition triplet $C$ of target human, we leverage two feature pyramid networks, garment feature extraction $E_g(\cdot)$ and person feature extraction $E_c(\cdot)$ to extract multi-scale features, denoted as $E_g(I_g) = \{g_1, g_2, \cdots , g_N\}$ and $E_c(C) = \{c_1, c_2, \cdots, c_N\}$. These pyramid features are then fed into $N$ fusion blocks to perform local flow warping. Note that these fusion blocks exploit local-flow global-parsing blocks to estimate $N$ multi-scale local flows $f_i$, yielding the final estimated displacement field $M_{G \rightarrow H}$. Considering this comment, we will include an additional figure to depict the technical details of local flow warping process (Fig. A) in the revised version.

Q2: Matched points in Figure 5.

A2: To clarify, in Figure 5, all green point pairs/mappings basically reflect the correct correspondences between the generated image and the input garment. Note that here we follow the typical point mapping method in SuperPoint[1] to perform HPatches homography estimation (i.e., the estimation of correst matches). Specifically, we perform nearest neighbor matching between all interest points and their descriptors detected in the in-shop garment image and those in the synthesized person image. Then we employ OpenCV's implementation of findHomography() with RANSAC to compute the correctly matched points in the image pairs, which are further marked in green. In this way, the denser the green point mappings covered over two images, the better the perservation of interest points (i.e., garment details) for synthesized person image.

Q3: Add figure of feature injecting process.

A3: Appreciate this comment. As suggested, we include an additional Fig. B to illustrate the feature injection process. Technically, given the geometry features $F_{GA}^l$ and garment features $F_{G}^l$ of semantic points, we first augment intermediate hidden states of Garm-UNet/Main-UNet (i.e., $K^l$/$K_g^l$) with $F_{GA}^l$/$F_{G}^l$. Then the augmented intermediate features of Garm-UNet/Main-UNet are concatenated, followed with $l$-th self-attention layer in Main-UNet:

$$Attn(Q^l,K^l_{*},V^l,F_{GA}^l,F_{G}^l) = Softmax(\frac{(Q^l + F_{GA}^l) \cdot [K^l + F_{GA}^l, K^l_g + F_{G}^l]^T}{\sqrt{d}}V^l).$$

In this way, the hidden states of Main-UNet are strengthened with additional geometry & garment features of semantic points and intermediate features of Garm-UNet through feature injection.

Reference: [1] DeTone, Daniel and Malisiewicz, Tomasz and Rabinovich, Andrew. Superpoint: Self-supervised interest point detection and description. In CVPR Workshop, 2018.

Dear Reviewer 37Fc,

It seems we have not received your further feedback on our updated response on technical contribution. Thus here we provide a more detailed clarification on technical contribution of our proposed semantic point mapping, especially compared to existing warping-based methods. Though warping module is exploited to estimate the dense displacement between garment and target person, the success of our work is more than that. Our contributions are from three aspects:

i. The input condition of diffusion model is new. Unlike existing warping-based models directly leverage holistic warped garment image as inputs to trigger diffusion process, our work seeks the most informative semantic points as a new condition to enable VTON task. Such semantic point inputs go beyond traditional holistic warped inputs by emphasizing unique regional fine-grained texture details, thereby encouraging better preservation of fine-grained details in VTON.

ii. The injection of additional condition into diffusion model is novel. Existing warping-based methods commonly take the warped garment image as hard/strong condition to guide VTON. This way heavily relies on the accuracy of garment warping, thereby easily resulting in unsatisfactory VTON results given the inaccurate warped garments under challenging human poses (see the results of GP-VTON in Fig. E). On the contray, our proposed SPM-Diff adopts a soft way to exploit the visual correspondence prior derived from semantic points as a soft condition to boost VTON. Note that such visual correspondence of local semantic points are exploited in latent space (corresponding to each local region), instead of precise alignment of pixel-level location in typical warping-based models. Thus when encountering mismatched points with slight displacements within local region, SPM-Diff still leads to similar geometry/garment features as matched points, making the visual correspondence more noise-resistant (i.e., more robust to warping results). As shown in Fig. E, given warping results with severe distortion, our SPM-Diff still manages to achieve promising results.

iii. To amplify our proposed semantic point matching along the whole diffusion process, we devise a point-focused diffusion objective that emphasizes on the reconstruction of semantic points over target persons (pursuing better preservation of fine-grained texture details), rather than typical reconstruction of holistic target person image.

We therefore kindly invite Reviewer 37Fc to reconsider assessment on our SPM-Diff's essential technical contributions, depending on the above discussions.