Reconstruction of Manipulated Garment with Guided Deformation Prior

Thank you for your valuable reviews.

To provide a context, we would first like to briefly describe our pipeline. Given the point cloud, we first map each point to the UV space using the UV mapper. This yields a sparse UV map and a sparse panel mask. We then use Eq. (10) to fit the optimal latent code $**z**^*$ for the ISP model from the sparse panel mask. Note that with $**z**^*$, we can recover a complete panel mask and a rest-state garment mesh that defines the vertices and faces of the garment. Then, we leverage the diffusion model to recover the complete UV map, utilizing the sparse UV map and the complete panel mask as the guidance in the reverse diffusion process.

This being said, below is the actual response to your comments.

1. Garment-specific design.

Our approach is not specific to a particular garment. However, due to the challenging nature of the task where only a partial garment is observed, we consider category-level garment reconstruction as in much prior art such as GarmentNets [27] and GarmentTracking [2]. Since a folded shirt and a pair of folded trousers can exhibit similar shapes as shown in Fig. 2 of the attached PDF file, we need to use different models for them.

2. The recovered garments are smooth.

The qualitative results in our paper are rendered without smoothing. Both the ground truth meshes and our reconstructed meshes are visualized in their raw form. The fact that our reconstructions seem smoother than the ground truth meshes can be attributed to the tendency of neural networks to learn low-frequency functions [Rahaman2019], which yields smooth reconstructed UV maps. Furthermore, our guided denoising process finds the expected reconstructions $\hat{x}$ given observations $y$ and the starting noise $x_T$, where $\hat{x} = E(x|y,x_T) = \sum_x xP\lbrace x|y, x_T\rbrace$. In areas where the data is missing, it provides only weak guidance and the reconstructions are naturally smooth. In future work, we will explore methods to enhance our diffusion model to capture finer details.

3. Are the recovered garments meshes able to be used for further animations?

Yes, our recovered meshes can be used for animation and simulation directly. In Fig. 3 of the attached PDF file, we show the simulated results for the recovered shirts using Blender, where we drop them onto a horizontal bar.

4. How to connect the edges in the recovered mesh from point clouds?

We do not compute edges for the point cloud to generate the garment mesh. Instead, we use Eq. (10) to fit the optimal latent code $**z**^*$ for ISP model from the sparse panel mask. Using $**z**^*$ alongside the ISP meshing process, we reconstruct a garment mesh in rest state as illustrated in the bottom-right of Fig. 2 of the main paper. This mesh defines the vertices and faces of the garment. To reconstruct the garment in the observed deformed state, we update the vertex positions by $**V**=\mathcal{M}[u,v]$, where $\mathcal{M}$ is the recovered UV map and $(u,v)$ is the corresponding UV coordinate of $**V**$.

5. What is the averaged number of points for different garments? Is the model able to deal with large number of points?

The averaged numbers of points provided by VR-Folding dataset [2] are 30K for Shirt/Pants/Skirt and 27K for Top. However, instead of using all points, we randomly sample 4000 of them as the input. Therefore, we are able to handle large number of points.

6. Garments worn by the human body.

Using ISP model to recover on-body garments has been shown to work in [1,40]. Consequently, we focus on the more challenging task of recovering garments not being worn. Our method differs from [1, 40] by utilizing a diffusion model as the deformation prior and leveraging UV mapping along with the proposed fitting method (Sec. 3.3 and 3.4) to recover complete garment meshes from partial point clouds. With appropriate training data, our method can also handle garments worn on the human body. However, due to time constraints, we are unable to present results for this specific scenario.

References

N. Rahaman, A. Baratin, D. Arpit, F. Draxler, M. Lin, F. Hamprecht, Y. Bengio, and A.Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, 2019.

Thank you for your appreciation of our work. We address your questions and comments as follows:

1. Handling noisy or highly sparse point clouds.

To evaluate performance under noisy conditions, we add per-point Gaussian noise to the input data, varying the standard deviation. As shown in Fig. 1 (b) of the attached PDF file, the results on the Folding Pants subset indicate that reconstruction error increases with noise levels; however, the errors remain relatively low across different noise levels. Additionally, the evaluation of real-world data in Sec. 4.4 demonstrates the robustness of our method, even when the input point cloud, generated using Nerf, is noisy and inaccurate.

Regarding sparsity, the captured points are generally dense in visible areas. Instead of using all available points, we randomly sample 4000 points from them as the input. We also evaluate the influence of point quantity on reconstruction quality by analyzing errors with varying input point numbers on the Folding Pants subset. The results, shown in Fig. 1 (a) of the attached PDF file, reveal that while a reduction in points leads to increased error, we maintain a relatively low error margin even with only 2000 points.

2. Generalization across a wide variety of garment types and materials without retraining is not fully explored.

Due to the challenging nature of our task where only a portion of the garment is observed, we consider category-level garment reconstruction as in prior art GarmentNets [27] and GarmentTracking [2]. However, we acknowledge the reviewer's point that investigating the generalization across types and materials is an important direction for future work.

We thank you for your acknowledgement of our contribution in manipulated garment reconstruction. Below are our responses to your comments and questions.

1. Comprehensibility.

Thank you for pointing this out. At the end of each stage and to enhance comprehensibility, we will revise our paper and refer the reader to the intermediate results in our main framework figure (Fig. 2 of the main paper).

2. How many points do the input point cloud contain?

We use 4000 points randomly sampled from the captured point clouds as the input. To evaluate the influence of point quantity, we analyze the reconstruction errors by varying the number of points used as input on the subset of Folding Pants. The results are reported in Fig. 1 (a) of the attached PDF file. A reduction in points correlates with increased error. However, even with 2000 points, we maintain a relatively low error margin. We will include this experiment in our final version of the paper.

3. How accurate do the points have to be to produce a high-quality reconstruction?

To evaluate the influence of input point noise, we add per-point Gaussian noise to the input with varying standard deviation. Fig. 1 (b) of the attached PDF file shows the results on the subset of Folding Pants. It illustrates that as the noise level rises, so does the reconstruction error; nonetheless, the errors remain relatively low across different noise levels. We will include this experiment in our final version of the paper. Additionally, the evaluation of real-world data in Sec. 4.4 of the main paper also demonstrates the robustness of our method, where the input point cloud is generated using Nerf which is noisy and inaccurate.

4. Quantitative evaluation on the number of intersections or more animated reconstructions.

In Table 1 of the attached PDF file, we evaluate the intersections of our reconstructions and compare them with those of GarmentTracking [2] using the ground-truth initialization. We compute the average ratio of faces with intersection as the evaluation metric. Notably, our results exhibit fewer intersections compared to GarmentTracking on Pants, Top and Skirt. We will revise our paper to include this evaluation and more reconstruction results.