VTON-VLLM: Aligning Virtual Try-On Models with Human Preferences

We sincerely thank Reviewer ZRwi for recognizing the merits in our paper (e.g., well-motivated approach, strong practical contribution that can be widely adapted to existing VTON models, convincing experiments and ablation studies), and your kind patience in engaging in this rebuttal. Below, we provide a point-by-point response. We hope the information here may further supplement our main submission and address your concerns.

A1: Performances across fine-grained categories.

Q1: Appreciate the comment. The reviewer pointed out that it is crucial to evaluate the model generalization ability of VTON techniques across diverse demographics. As suggested, we reported the performances across different genders, body types and skin tones. In the evaluation, gender, body type, and skin tone are roughly categorized into several classes. The quantitative results on VITON-Bench are summarized as:

(1) Gender: Female, Male

(2) Skin Tone: Black, White, Yellow, Brown

(3) Body Type: Slim, Average, Overweight

For the quantitative and qualitative results on lower-body clothing and dresses, please refer to Table 1 & 2 and Figure 2 in the supplementary material for more details.

Q2: Some duplicate presentation of motivations in paper.

A2: Appreciate this point. For our initial purpose, the discussion in Section 3.1 serves to recap key challenges and motivations for the later introduction of the proposed approach. As suggested, we will combine Sec 3.1 into the introduction section in the revised version.

Q3：Face mask in figures.

A3: Sorry for the confusion. To clarify: the human images in Fig. 1(a) and Fig. 3 are collected from real-world fashion data and depicted as real individuals, and face masks were applied to these images solely for privacy preservation. In contrast, the images in Fig. 1(b) and Fig. 2 are synthesized by generative models and therefore do not require masking. Moreover, it is worth noting that no face-masking operations were involved during model training or inference. We will add this clarification in the revised version.

We sincerely thank Reviewer hywb for recognizing the merits in our paper (e.g., well-motivated approach, valuable resources such as new benchmark and human-centric metrics for future VTON research, significant improvement over the baselines on VITON-Bench), and your kind patience in engaging in this rebuttal. Below, we provide a point-by-point response. We hope the information here may further supplement our main submission and address your concerns.

Q1: Technical contribution.

A1: Though the latest techniques (SFT, RLHF, TTS) established in LLM/VLM literature are adapted to our proposed well-integrated pipeline for VTON, the success of our work is more than that. Our contributions are from three aspects:

• (1) The adaptation of VLLM to VTON synthesis is not trivial. In our experiments, we found that the efficacy of the pre-trained base VLLM in predicting human judgments on VTON images is unsatisfactory. That's why we devise a new VTON-VLLM (acting as ''fashion expert'') particularly for VTON task. Specifically, we collected high-quality human feedbacks on 40K synthesized VTON images across six fine-grained sub-aspects, which is utilized to fine-tune the base VLLM and align it with human judgments for VTON. Moreover, in contrast to general image quality estimators (e.g., HPS, AES, CLIPScore, ImageReward) that provide only scalar scores, our VTON-VLLM can generate detailed and human-aligned assessments of VTON outputs, which is further utilized to steer the VTON refinement process.
• (2) The VTON refinement model guided by fine-grained instruction is new. Existing VTON methods mostly adopt one-stage inference, which often exhibit limited detail preservation and generation fidelity. In contrast to existing works, our proposed approach novelly combines these VTON models and a plug-and-play VTON refinement model in a two-stage framework, which significantly boosts VTON by transforming the low-quality results to high-quality and preference-aligned ones. Moreover, different from general-purpose image refinement models (e.g., SDXL-Refiner) using generic prompts, our VTON refinement model is guided by the reference garment and fine-grained instructions generated by the trained VTON-VLLM, which provides the visual details of the garment and explicitly identifies local regions in the synthesized image that fail to meet user expectations, respectively.
• (3) The human-centric metrics for VTON are new. Traditional metrics, such as LPIPS and FID, often fail to capture fine-grained inconsistencies in detail preservation and demonstrate poor correlation with human judgments in VTON. To address these limitations, we propose a human-centric evaluation paradigm based on the trained VTON-VLLM, which demonstrates high agreement with human preferences. These new metrics further function as reward signals to guide the VTON refinement model in test-time scaling (TTS), which converts the initial VTON outputs into human-preferred outcomes through iterative refinement process.

Q2: Ablation study on VRM-Instruct and TTS using more baseline VTON models.

A2: Thanks for this point. In the current version, we actually have conducted ablation study on the effect of these two components using a base VTON model (i.e., CAT-VTON) in Table 3 of the main paper. As suggested, we further evaluated the individual contributions of each component using more VTON models presented in Table 1 and 2 by sequentially adding our proposed VRM-Instruct and TTS to them. Please note that TTS is triggered during the iterative refinement process with our VRM-Instruct model as the execution model, and thus cannot be solely integrated into the base VTON models by excluding VRM-Instruct.

As shown in the revised Table 1 and 2 below, by incorporating the fine-grained instructions from VTON-VLLM to identify imperfect regions in the synthesized garment and guide the refinement process towards human preferences, base VTON model+VRM-Instruct achieves notable improvements compared to the base models. Furthermore, we incorporate TTS into base VTON model+VRM-Instruct to enable iterative enhancement with human-preference-aware reward provided by VTON-VLLM in the full run (base VTON model+VRM-Instruct+TTS), surpassing all the other settings. We will add the discussion in revision.

This table is the revised version of Table 1 in the main paper.

This table is the revised version of Table 2 in the main paper.

Q3: Practical deployment without TTS.

A3: According to the ablation study in Table 3 of the main paper and the additional results provided in the above answer A2, substantial performance gains can be attained through single-pass refinement using only VRM-Instruct. Specifically, compared with the base model CAT-VTON in the improved Table 2 from A2, CAT-VTON+VRM-Instruct boosts $\mathbf{SSIM}$ and $\mathbf{IQ_p}$ from 0.751 to 0.772 and from 4.238 to 4.579, respectively, by refining the VTON results with human-preference-aligned instructions via VRM-Instruct. Employing TTS can further improve the generation quality. Compared with CAT-VTON+VRM-Instruct, the best scores in $\mathbf{SSIM}$ (0.788) and $\mathbf{IQ_p}$ (4.685) are achieved in the full run CAT-VTON+VRM-Instruct+TTS.

We sincerely thank Reviewer TTMh for recognizing the merits in our paper (e.g., consistent improvements on top of SoTA VTON models, high preference agreement rate with humans), and your kind patience in engaging in this rebuttal. Below, we provide a point-by-point response. We hope the information here may further supplement our main submission and address your concerns.

Q1: Importance of training VLLM on human feedback dataset: comparisons between Base-VLLM and our VTON-VLLM.

A1: Appreciate the comment. As suggested, we provide quantitative comparisons between two settings: (1) Base-VLLM: The VTON refinement model is trained on and guided by the instructions generated by the Base-VLLM without human feedback fine-tuning, (2) VTON-VLLM: The VTON refinement model is trained on and guided by the instructions generated by our VTON-VLLM, which is fine-tuned using human feedbacks. Both models are trained and evaluated on VITON-HD dataset. This table lists the quantitative results:

It can be observed that directly leveraging the Base-VLLM to prompt our proposed VTON refinement model can, to some extent, improve the VTON results compared with the base VTON model. Further empolying our VTON-VLLM to align the refinement with human preferences consistently outperform those with the Base-VLLM across all the metrics, which validates the effectiveness of training VTON-VLLM with human feedbacks on VTON images. We will add this discussion in the revised version.

Q2: Refinement iterations $N$ and computation overhead.

A2: Thanks for the comment. We have conducted ablation study on ''Effect of Iteration Count $N$ in Test-Time Scaling'' in the supplementary material, and the best results were achieved when $N = 3$ (please refer to lines 17-19 and Table 3 in the supplementary material for more details). Therefore, the iterative refinement proceeds until the output image meets a predefined quality threshold (e.g., high scores across all the sub-aspects defined in the paper) or a maximum of 3 iterations is reached. The computation time for the base VTON model and a single pass of refinement is about 7 seconds and 13.6 seconds (0.35 seconds for instruction generation with VTON-VLLM and 13.25 seconds for image generation with VTON refinement model), respectively. The total computation time for the refinement process with $N$ iterations is approximately $N$ times that of a single pass. The refinement process can be further accelerated by over 50% through the use of caching techniques and model distillation in practical applications. Also, users can manually terminate the refinement process if the current output meets their requirements, even after a single refinement step, through an interactive UI.

We sincerely thank Reviewer 5M4J for recognizing the merits in our paper (e.g., latest techs and consistent improvements in VTON, well-structured paper), and your kind patience in engaging in this rebuttal. Below, we provide a point-by-point response. We hope the information here may further supplement our main submission and address your concerns.

Q1: Effectiveness of the proposed method on different datasets.

A1: Thanks. To be clear, the mentioned newly curated VITON-Bench (Table 2 in the main paper) is adopted to evaluate our proposal under challenging VTON scenarios. In addition to VITON-Bench, we actually have conducted experiments on two widely adopted benchmarks (i.e., VITON-HD and DressCode), which consistently demonstrates the effectiveness of our approach. Specifically, (1) Table 1 in the main paper reports quantitative comparisons across methods on the VITON-HD dataset, (2) Table 1 and 2 in the supplementary material summarize the results on the DressCode dataset. To support reproducibility and advance the research in VTON, we will release all the training data, source codes and model weights.

Q2: Instruction following in VTON refinement model.

A2: Appreciate this comment. Technically, to enforce the VTON refinement model to effectively refine images from any VTON model by adhering to instructions, we supervise the training of VTON refinement model with well-constructed training triplets $(I_g, I_h^{LQ}, I_h^{HQ})$, which consists of a reference garment $I_g$, a low-quality person image $I_h^{LQ}$ and a corresponding high-quality version $I_h^{HQ}$. For each triplet, our VTON-VLLM generates a fine-grained instruction $t_r$ to steer the refinement. Please refer to lines 144--166 in the main paper for more training details.

As suggested, we additionally validated the capability of instruction following in our trained VTON refinement model. Specifically, we randomly sampled 2,000 ground-truth garment-person pairs from the test datasets. For each pair, we first generated a low-quality try-on image and a corresponding refinement instruction using a base VTON model and our VTON-VLLM, respectively. These low-quality images were then refined by our VTON refinement model (VRM-Instruct) based on the provided instructions. Finally, we employed both our VTON-VLLM and GPT-4o as automated evaluators to assess whether the executed image refinement aligns with the given instruction. The success rate of instruction following evaluated by VTON-VLLM and GPT-4o is 98.79% and 98.32%, respectively. We will add the discussion in revision.