Robust-PIFu: Robust Pixel-aligned Implicit Function for 3D Human Digitalization from a Single Image

Weakness 1a:

We list a few of the important differences between our Disentangling Diffuser and the Stable Diffusion’s inpainting model here:

1. Stable Diffusion’s inpainting model does inpainting and thus requires a mask that indicates exactly which part/region of the input image needs to be infilled. In contrast, our Disentangling Diffuser does not do inpainting and is only given a Human Subject Selection Map, which is a map that indicates which human subject in the input image is to be retrieved. This map can be any of three arbitrary but fixed patterns, and it certainly does not indicate the exact location of the human subject that we wish to retrieve. For example, if the given Human Subject Selection Map is the first pattern, then the leftmost human subject in the input image will be retrieved, regardless of whether the leftmost human subject is physically located at the leftmost or rightmost end of the input image.

2. Stable Diffusion’s inpainting model is used for the inpainting task, while our Disentangling Diffuser is used for the task of external occlusion removal. These two tasks are very different and have very different inputs and outputs. In inpainting, you are given a mask that indicates which region of the input image needs to be filled, and the task is to fill that region up using context from the non-masked parts of the input image. But our task of external occlusion removal is more complicated and involves many different sub-tasks. Firstly, an input image can have more than 1 human subject, so our Disentangling Diffuser needs to understand how many human subjects are in the input image. After that, the diffuser needs to find out which of the human subjects is the target human subject. In addition, not only does the non-target human subjects need to be removed, our Disentangling Diffuser also needs to find out whether any body parts of the target human subject is being occluded by the non-target human subjects. This means that our Disentangling Diffuser needs to understand which body part belongs to which human subject. The occluded body parts of the target human subject need to be filled in autonomously since we do not provide any hint of where or what body parts are being occluded. Extraneous objects, such as chair, table etc., need to be autonomously removed as well. Again, there will be no hint of what these objects are and where these objects will be. If these extraneous objects are occluding the target human subject’s body parts, then the occluded body parts need to be generatively predicted as well. In addition, if you refer to the 3rd and 4th rows of Fig. 11 in our manuscript, you will see that our Disentangling Diffuser has the additional responsibility of correcting/optimizing camera position. Moreover, regions on the input image that are occluded by darkness (the image has inadequate lightings) need to be generatively predicted by our Disentangling Diffuser as well. These various subtasks are certainly not completed in the inpainting task.

3. Our Disentangling Diffuser uses CLIP Image Embedder to encode the input image and then apply Cross Attention to incorporate the input image as a conditional prior. The Stable Diffusion’s inpainting model neither uses a CLIP Image Embedder nor Cross Attention for their conditional prior.

4. Our Disentangling Diffuser is adapted to the domain of human images, while the latter works in the domain of natural images. For our Disentangling Diffuser, we trained a specialized autoencoder to learn latent encodings of images that contain at least one and at most three human subjects. Majority of the images have external and/or internal occlusions, and this means the human subject is usually only partially visible. In contrast, the Stable Diffusion’s inpainting model is trained on natural images. Consequently, the Stable Diffusion’s inpainting model has difficulties dealing with images that contain humans (Please see the 3rd and 4th row in Fig. 21 of the Stable Diffusion’s Arxiv paper at https://arxiv.org/pdf/2112.10752)

Thank you for the revisions and your responses to my questions.

After reviewing the additional explanations and the revised paper appendix, I have reconsidered my evaluation and adjusted the score. However, there are still areas for improvement in the revised version.

1. The explanation of the background knowledge for the method presented in the paper is insufficient, and it is written in a way that may be difficult for readers who are not familiar with the related methodologies to understand.

2. There is a lot of verbal explanation regarding the method, and it has not been expressed in mathematical formulas, which reduces clarity. Additionally, such formalization would have helped simplify the paper, but since it was not done, the paper became longer and important content could not be included in the main paper.

3. The literature review is lacking, particularly in terms of mentioning and comparing recent methods. Papers such as [1], [2], and [3] are recent works related to this paper and were published before the ICLR submission deadline. Some of these methods also utilizes diffusion models.

4. The rationale for dividing the human subject selection map in "DISENTANGLING DIFFUSER" into three sections is unclear, and results for various poses and human locations are not provided. There are likely limitations, but there is no analysis of failure cases, nor any mention of potential improvements.

Regardless of the outcome of this submission, I hope that these points will be taken into consideration for further revisions.

[1] Chen, Mingjin, et al. "Ultraman: Single Image 3D Human Reconstruction with Ultra Speed and Detail."

[2] Lee, Jungeun, et al. "PIDiffu: Pixel-aligned Diffusion Model for High-Fidelity Clothed Human Reconstruction."

[3] Ho, I., Jie Song, and Otmar Hilliges. "Sith: Single-view textured human reconstruction with image-conditioned diffusion."

Question 8:

For the ablation on P (previously L481, now at L500 L506 in the revised manuscript), when P is not used, the pixel-aligned implicit model is still given the disentangled image (from D), N_s, and N_c, but the N_c consists of only two normal maps (front and back). Also, the two normal maps are not generated by P, but are generated by a separate normal predictor that is commonly used in existing works (i.e. PIFuHD, ICON, and ECON). The reason for still including the two normal maps is because existing works typically use these two normal maps in their models. If we exclude these two normal maps, we are over-exaggerating the impact of our P.

For ablations on the implicit model (previously L492, now at L506 L512 in the revised manuscript), when the Multi-dimensional Layered Normal Grid is not used, the values (processed or not) from the Multi-dimensional Layered Normal Grid are not given as inputs to the MLP of our pixel-aligned implicit model.

When the concatenation of N_s + N_c is not used, we do not feed N_s and N_c as inputs to the stacked hourglass network of our pixel-aligned implicit model. Instead, we use front and back normal maps that are predicted by a separate normal predictor (not P) that is commonly used in existing works like PIFuHD, ICON, and ECON. The front and back normal maps are concatenated with the disentangled image (from D) before being fed as inputs to the stacked hourglass network. Again, we include these two normal maps in order to ensure fairness.

Hence, when both Multi-dimensional Layered Normal Grid and concatenation of N_s + N_c are not used, the only inputs to the pixel-aligned implicit model will be the disentangled image and a pair of normal maps (front and back).

Question 9: Thank you for your question. Fig. 8 is an ablation on D instead of P. When D is not used, the best we can do is to use an off-the-shelf instance segmentation model to segment out the pixels that belong to each human subject (same as what is done by ECON). Then, we separately feed in the image of each segmented human subject into our P. As each of these images will be missing a number of pixels of a human subject, P will not be able to generate normal values at those missing pixels.

Question 10: Thank you for your feedback. Please refer to the last row of Fig. 26 in our revised manuscript.

Weakness 1:

Indeed, as we wrote in L46-53, the work by Wang et al. assumed that the groundtruth SMPL-X mesh is given. This can be confirmed by reading the work by Wang et al. (the 1st paragraph of their “3. Method” Section). However, in practical scenarios, the SMPL-X mesh must instead be predicted from the input image. If you refer to the Figure 2 in their work, you will observe that the input image has occlusions (i.e. the left arm and both legs are missing). Given such an occluded input image, it is very unlikely that the SMPL-X mesh that is predicted from the input image can be reasonably accurate. Thus, the assumption that a perfect/groundtruth SMPL-X mesh is given is a very strong assumption that is unlikely to be practical. For ECON, they too do not address the occlusions before trying to predict the SMPL-X mesh, thus their results have the same problems too (see either Fig. 1d, the 4th-5th rows of Fig. 10, or the 4th row of Fig. 20 in our manuscript).

Our Robust-PIFu’s approach is different because we do not attempt to predict the SMPL-X mesh from an occluded input image. Instead, we remove the occlusions from the input image first. As shown in Fig. 3 of our manuscript or 1st paragraph of our Sect. 3, we use Disentangling Diffuser (D) to remove such occlusions from the input image (i.e. the occluded regions are generatively filled up). Then, once such occlusions are removed, we can predict the SMPL-X mesh, which is now a straightforward task. The difference is observed if you compare our results and ECON’s results in 4th-5th rows of Fig. 10 or 4th row of Fig. 20 in our manuscript.

Weakness 2: Besides Fig. 10, we also provided Fig. 20 to show both single and multi-humans with at different degrees of occlusion. In addition, we also revised our manuscript to add in Fig. 26. Thus, if you require more results on this, please also refer to our Fig. 26 of our revised manuscript.

Weakness 3:

We took into account a number of reasons:

Firstly, we examined the 3D meshes that are given in various human datasets, such as BUFF, THuman2.0, and MultiHuman, before deciding on the design choice of using the specific 8 normal maps. Among the meshes that we examined, we find it is rare that more than 8 normal maps will be required. Moreover, if we try to increase the number of normal maps by increasing the number of layers, then there will not be enough 3D meshes that require more so many layers of normal maps, and thus we will not have adequate training examples to train our Penetrating Diffuser to produce reasonable outputs for the additional layers.

Secondly, increasing the number of normal maps will require additional sampling runs of the latent diffusion model, and this translates to a significant increase in inference time. On the other hand, reducing the number of normal maps will translate to less coverage and defeat the purpose of having our Penetrating Diffuser. Hence, we have to strike a balance between inference time and coverage.

Thirdly, in order to construct a detailed 360 degree view of a clothed human avatar, we find that, at the minimum, only 4 view directions (front, back, right, left) are required. Adding more view directions (and also adding more normal maps) will lead to overlapping and replicated information. On the other hand, reducing the view directions to less than 4, like what is done in ECON (only 2 view directions - front and back), will result in poorly constructed right and left views (see 1st row of Fig. 2d and 1st row of Fig. 7d in our manuscript).

Question 1: Thank you for your feedback. We added a caption for Fig. 3 in our revised manuscript.

Question 2: Thank you for your feedback. We fixed the caption of Fig. 4 in our revised manuscript.

Weakness 1:

Thank you for your feedback. We will really appreciate it if you could specify which part was unclear to you. Our paper is structured as such:

1. Firstly, in Section 1, we introduce and explain the problem that we want to resolve. Terms and jargon are defined here. We explain the motivations behind our work and list our work’s contributions.

2. Next, in Section 2, we introduce related works and explain relevant concepts.

3. Then, in Section 3, we first introduce our proposed model and its various components from a high-level point of view. Then, in Section 3.1 - 3.4, we explain each of the components with more specific details.

4. After that, in Section 4, we first describe datasets that we used and the benchmark models that we compared against. Following that, we show our proposed model’s qualitative and quantitative results. This is followed by ablation studies of the different components that form our proposed model.

5. Finally, in Section 5, we describe our manuscript’s limitations before ending the manuscript with a conclusion.

Weakness 2: Please see our response to your "Question 1" below.

Weakness 3: Besides the last two paragraphs in Section 3.1 of our manuscript, we also provided more information on the training details of Disentangling Diffuser in Section “A.14 Implementation Details” in our Appendix.

Question 1: We understand that you want to see how our model will perform on real images that are different from what were already shown in our manuscript. Specifically, you asked for real images where the head of the human subject is occluded. Thus, we revised our manuscript and added Fig. 26 in the Appendix. Please refer to Fig. 26 of our revised manuscript. As shown in the figure, our model can provide reasonable outputs for these challenging scenarios as well.

Question 2: The last row of Fig. 26 in our revised manuscript shows how our Disentangling Diffuser will handle these situations. In addition, we also wrote more information on these situations in Section “A.3 How D deals with Vertically Occluded Humans” of our Appendix. In short, as long as the training images are prepared as required, the disentangling strategy of Disentangling Diffuser will still be effective. We could train our Disentangling Diffuser with images of vertically occluded humans and ask it to target the topmost human subject whenever pattern 1 (Human Subject Selection Map) is given as input, and the Disentangling Diffuser will learn to target the topmost human subject when the human subjects in the input image are perfectly aligned in a vertical line.

Weakness 1: The necessity of eight normal maps is illustrated in Fig. 23 of our manuscript under Section A.4 in the Appendix, but we agree that additional explanation will make it clearer. Thus, we revised Section A.4 of our manuscript and expanded the 2nd paragraph in Section A.4 to provide additional explanation.

Weakness 2: This design choice that you suggested was originally considered by us. Indeed, the initial design for our Penetrating Diffuser is to generate all eight maps (i.e. both front-back and left-right orientations) all at once. We realize early that this does not work well for a latent diffusion model. The reason is that there is no spatial relationship (i.e. the pixels do not align) between the front-back normal maps and the left-right normal maps. This causes a problem when we have to use an autoencoder to learn to encode the eight normal maps into a single latent encoding because the learning is exceptionally challenging. Despite many attempts, we are unable to obtain a latent encoding that achieves low reconstruction error. Whenever we decode the latent encoding back into the eight normal maps, we observe that the reconstructed front-back normal maps contain pixels that belong to the left-right normal maps instead, and the left-right normal maps contain pixels that belong to the front-back normal maps. It is clear to us that, during the encoding process, the front-back normal maps and left-right normal maps have negatively interfered with each other. Thus, we decided to have the Penetrating Diffuser to generate the front-back normal maps and left-right normal maps in two separate runs instead.

Weakness 3: Thank you for your feedback, we revised the fourth paragraph of Section “A.14 Implementation Details” in our Annex to clarify the structure of the MLP in our pixel-aligned implicit model. In short, our MLP follows the structure of the original PIFu except that its input dimension is increased in order to allow for the additional features from the eight normal maps to be used as inputs. Yes, these additional features are concatenated directly as inputs to the network.

Question 1: Please see our response to your “Weakness 1” and “Weakness 2” above.

Question 2: Please see our response to your “Weakness 3” above.