OccFusion: Rendering Occluded Humans with Generative Diffusion Priors

We thank the reviewer for your time and the helpful comments! We address your concerns below.

Rendering quality seems blurry

With only 10 mins of training time, OccFusion surpasses state-of-the-art occluded human rendering methods by a significant margin qualitatively and quantitatively, as illustrated in Table 1 and Fig. 1 and 5 of the main paper. With that said, with a longer training time, OccFusion is able to render in higher quality with much less blur. We added supplementary experiments to validate this statement – please see qualitative and quantitative results in Fig. 1 in the rebuttal PDF. We believe that our proposed 10-minute version of OccFusion is able to achieve the best balance of rendering quality and efficiency.

Is $M$ accurate?

As mentioned in the implementation details section of our Appendix, we derive $M$ from SAM (ICCV 2023), which is among the state-of-the-art in image segmentation, making it a very reliable way to calculate $M$. We would like to note that numerous other human rendering works [1, 2, 3] also use SAM as their go-to method for obtaining pseudo ground truth human masks during preprocessing.

[1] Kocabas, Muhammed, et al. "Hugs: Human gaussian splats." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.

[2] Pang, Haokai, et al. "Ash: Animatable gaussian splats for efficient and photoreal human rendering." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[3] Hu, Liangxiao, et al. "Gaussianavatar: Towards realistic human avatar modeling from a single video via animatable 3d gaussians." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Binary masks do not seem consistent — Is $\hat{M}$ accurate?

When dealing with in-the-wild data, the ground truth silhouette of occluded regions is always unknown. Since the silhouette of the human across different frames is supposed to be inconsistent due to human (and sometimes garment) movement, we will never know with certainty what the changes in the ground truth silhouettes would be across a video sequence. However, we find that by using a pre-trained Stable Diffusion prior that has been trained on a great amount of similar data with enforced consistency, we can ensure that the inpainted $\hat{M}$ falls into a reasonable estimation given the context of the pose and visible regions. As Fig. 4 of the main paper demonstrates, the shape of inpainted binary masks is much more reliable and consistent than the appearance of the corresponding RGB inpainted images. In our pipeline, we generate only one mask for every frame and find that $\hat{M}$ does well in supervising the training of later stages.

Robustness of Optimization Stage to wrong $M$ and $\hat{M}$

As we covered in the past two responses, the fact that we utilize pre-trained SAM to get $M$ and pose-conditioned pre-trained Stable Diffusion to inpaint $\hat{M}$ makes getting a “wrong” mask extremely unlikely. However, to test the robustness of our method to inpainted masks, we add experiments on ZJU-MoCap supervised on the complete SAM masks obtained from the unoccluded humans in the optimization stage. Please see the qualitative and quantitative results in Fig. 2 in the rebuttal PDF. We find that using the inpainted $\hat{M}$ can lead to good rendering quality that is comparable to using masks derived from the unoccluded images, validating that our model is robust to variances that exist in the masks.

Is pose simplification necessary for image + pose + mask -> image?

The purpose of Figure 3 is to show that Stable Diffusion fails when conditioned on challenging 2D poses. Since we use the same model for the image inpainting process (with the masks being used to calculate the necessary regions for inpainting), the pose simplification step we proposed is still necessary for the Stable Diffusion model to accurately condition on the pose of the human. So, we apply the pose simplification step whenever self-occluded joints are present to discourage problematic generations.

Fig. 5 shows that the leftmost column is an input. 3DGS is optimized to replicate images of a single person in the training set, and there is no input to the 3DGS.

This is correct. The leftmost columns are not really inputs but reference images at the corresponding time stamp from the training point of view. We apologize for this error and will correct the figure to clarify this point.

Fig 5 Second Column Second Row does not have a visible human in reference

In Figure 5, we validate renderings from novel views. The reference images on the right show the views from the new camera perspective that is provided by the dataset. For an in-the-wild dataset like OcMotion, sometimes the camera cannot capture the complete human due to occlusions. In the case of the example on row 2 column 2, the human is completely occluded from the reference camera view. We will make this point more clear in the final version.

We thank the reviewer for the positive assessment of our work and the helpful comments! We address your concerns below.

Lack of discussion about NeRF in the Wild, Ha-NeRF, Gaussian in the Wild and NeRF On-the-go

Thanks for the suggestion. We will add the following sentences to section 2.2 of our Related work section in our final version:

NeRF-W [1] and other works [2,3,4] are able to account for photometric variation and transient occluders in complex in-the-wild scenes, allowing them to render consistent representations from unconstrained image collections. 
However, these works are not designed to handle dynamic objects like humans.

[1] Martin-Brualla, Ricardo, et al. "Nerf in the wild: Neural radiance fields for unconstrained photo collections." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

[2] Chen, Xingyu, et al. "Hallucinated neural radiance fields in the wild." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[3] Ren, Weining, et al. "NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[4] Zhang, Dongbin, et al. "Gaussian in the Wild: 3D Gaussian Splatting for Unconstrained Image Collections." arXiv preprint arXiv:2403.15704 (2024).

More visual results

We provide more experiment results in the PDF of the general rebuttal, and we will include more extensive video results in the final version.

Diversity of dataset

This is a great observation. A big restriction of current human rendering datasets (ZJU-MoCap, OcMotion) is that they lack diversity. ZJU-MoCap consists of humans rotating in place in a brightly lit motion capture environment. While OcMotion is more representative of a real world scene, its diversity is still lacking, with all the sequences being collected in the same indoor room. In addition, since both datasets are collected from Chinese universities, the subjects are all East Asian men. We believe that a promising future step for this field is to collect more diverse data to test the generalizability of human rendering methods like ours. We will add a discussion of dataset diversity to our Limitations section in the final version.

We thank the reviewer for your time and the helpful comments! We address your concerns below.

Incompleteness of Figure 6

Thanks for the suggestion. The reason why we do not include results for all experiments for both subjects in Figure 6 is to showcase the benefits of our proposed components on the canonical and deformed spaces. We aim to show improvements in rendered appearance in the deformed space with the subject on the left, and aim to show how human completeness is improved and artifacts are reduced in the canonical space with the subject on the right. We include more visual experiment results in the rebuttal PDF.

Inpainting in deformed space

In the Optimization Stage, we apply Score Distillation Sampling (SDS) to both the deformed and canonical spaces (see Sec. 4.2 and Eq. 5 of the main paper). The space is controlled by a random variable with a 75% probability of applying SDS on the deformed space and 25% on the canonical space. We add additional experiments to show that our proposed random mix of deformed/canonical space SDS yields the best results compared to applying SDS solely on the deformed space or the canonical space (please see Fig. 4 in the rebuttal PDF).

In the Refinement Stage, we operate solely in the deformed space – we render posed human renderings from the Optimization Stage and juxtapose them with occluded partial observations from the input video in order to perform our in-context inpainting.

We thank the reviewer for your time and the helpful comments! We address your concerns below.

Discussion of off-the-shelf models

Thanks for the suggestion. We will include the following edited text in the Introduction of the final version:

In this work, we introduce OccFusion, an efficient yet high quality method for rendering occluded humans. To gain improved training and rendering speed, OccFusion represents the human as a set of 3D Gaussians. Like almost all other human rendering methods, OccFusion assumes accurate priors such as human segmentation masks and poses are provided for each frame, which can be obtained with state-of-the-art off-the-shelf estimators such as SAM and HMR 2.0. However, to ensure complete and high-quality renderings under occlusion, OccFusion proposes to utilize generative diffusion priors, more specifically pose-conditioned Stable Diffusion 1.5 with ControlNet plugins, to aid in the reconstruction process.

Instructiveness of Figure 2

Thanks for your suggestions. We will carefully revise Figure 2 to incorporate your comments.

Effect of length of video on training time

The training time of the video does indeed depend on the length of the video. As mentioned in section 5.1 of the paper, we train our model on 100 frames for ZJU-MoCap and 50 frames for OcMotion. We also show in Figure 1 that our model outperforms some baselines that require much more (>500) frames and much longer training time. With longer training time and/or more samples from the videos, OccFusion is able to provide even better results (see Fig. 1 in the rebuttal PDF).

Does training longer improve rendering results?

Yes. With only 10 mins of training time, OccFusion already surpasses all counterparts by a significant margin qualitatively and quantitatively. It is also true that with longer training time, OccFusion is able to render higher quality. We add experiments to validate this statement, please see the qualitative and quantitative results in Fig. 1 in the rebuttal PDF. We believe that our proposed 10 minute version of OccFusion is able to achieve the best balance of rendering quality and efficiency.

OccGauHuman has splats that are “ripped apart”

Although the number of 3D Gaussians does not change during training, as densification and pruning of splats are disabled, the mean, scale, and opacity of each 3D Gaussian is optimized and changed during training, resulting in the “ripped apart” appearance when rendered in 2D. This is because 3D Gaussians are crowded in the visible regions. This ripped apart appearance only occurs in occluded areas, demonstrating that OccGauHuman alone is unable to reconstruct unobserved areas due to its lack of generative capabilities.

Training time of OccGauHuman

The OccGauHuman model we use to compare to GauHuman and OccFusion in Table 1 is trained for 10 minutes. We will be sure to clarify this in the final version.

It is also worth noting that GauHuman and OccGauHuman do not benefit significantly from training for longer. We show in Fig. 1 of the main paper that increasing the training time from 2 to 10 minutes causes only a small boost in PSNR on ZJU-MoCap for OccGauHuman and actually causes a degradation in performance for GauHuman. Additional results in Fig. 1 of the rebuttal PDF also validate this.

OccGauHuman is not a contribution

Thanks for the suggestion. We will rephrase Sec 3.3 of our paper to clarify that we do not consider OccGauHuman a contribution but rather a better baseline.

Gaussians are a broad term

Thanks for the suggestion. We will follow current Gaussian splatting literature [1,2] and replace all mentions of “Gaussians” in our paper with the term “3D Gaussians”.

[1] Kocabas, Muhammed, et al. "Hugs: Human gaussian splats." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.

[2] Hu, Shoukang, Tao Hu, and Ziwei Liu. "Gauhuman: Articulated gaussian splatting from monocular human videos." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

We thank the reviewer for your time and the helpful comments! We address your concerns below.

Proof that applying SDS on RGB images causes appearance inconsistency

In Fig. 3 of the rebuttal PDF, we include additional experiments comparing the rendering results of applying SDS on RGB vs. on human occupancy maps (as proposed). It is clear that applying SDS on RGB leads to defective renderings in the Optimization stage, which ultimately cause inferior results in the Refinement stage. We attribute these defective renderings to the variances introduced from diffusion models, making it difficult for them to reconstruct a consistent human appearance across different frames and training steps. However, applying SDS on human occupancy maps does not suffer from these variances too much and allows for the enforcing of human completeness, as proposed.

Refinement stage could introduce cross-frame inconsistency

It is true that diffusion-based inpainting models suffer from inconsistencies. However, $\{\mathbf{\hat{I}}\}$, the renderings from the optimization stage, are consistent since they are recovered directly from the input video. As we use these consistent images as context for our inpainting, the resulting inpainted reference images are not dramatically inconsistent. It is also worth noting that any small inconsistencies present in the inpainted references are smoothed out by more weight on perceptual loss terms and the usage of L1 loss during optimization of the Refinement stage (lines 227-230 of the main paper).

Supplementary videos lack method names

Sorry for the inconvenience. We included the naming details in section “D” of the supplementary material, but we can see how it can be confusing. We will be sure to label the videos with method names in the final version.

Blur present in visible areas

With only 10 mins of training time, OccFusion surpasses state-of-the-art occluded human rendering methods by a significant margin qualitatively and quantitatively. In addition, with longer training time, OccFusion is able to render in higher quality with much less blur. We add experiments to validate this statement – please see qualitative and quantitative results in Fig. 1 in the rebuttal PDF. We believe that our proposed 10 minute version of OccFusion is able to achieve the best balance of rendering quality and efficiency.