ComPC: Completing a 3D Point Cloud with 2D Diffusion Priors

Q1: I can't see how your work can generate a surface like SDS-complete?

A1: Thank you for liking our work! In this work, we focus on the task of point cloud completion, as commonly defined in prior studies such as [1, 2], which involves recovering complete 3D point clouds from partial inputs. Generating 3D meshes is not the primary goal of this task, so we base our comparisons on point clouds.

However, our method has the potential to generate 3D meshes thanks to the introduction of the Grid Pulling module. As described in Sec. 3.3, Grid Pulling re-samples uniform points from the non-uniform point cloud $P_{surf}$ by fitting a signed distance function (SDF) $g(\cdot)$ to the overall shape of $P_{surf}$. Using Marching Cubes, as outlined in [3], meshes can be extracted from $g(\cdot)$. Qualitative results, presented in Sec. A.7 of the appendix, show that our method can produce higher-quality completed meshes compared to SDS-complete.

[1] PCN: Point Completion Network [2] GRNet: Gridding residual network for dense point cloud completion. [3] Neural-pull: Learning signed distance functions from point clouds by learning to pull space onto surfaces.

Q4: About the source codes to test a custom dataset.

A4: Thank you for your interest to our work! Due to the limited time of the rebuttal period, we do not have time to clean up our codes or make demos. But we promise we will make our codes open sourced once we have done clean up the codes.

Q5: Moreover, it is recommended to split Figure 2 into two-three separate pictures with clarified steps description.

A5: Thank you for your suggestion. We have carefully considered this suggestion, but splitting Figure 2 into three separate figures would greatly increase the paper length, potentially exceeding the 10-page limit. Instead, we have revised Figure 2 to improve clarity as following:

1. A simplified description of the entire pipeline has been added at the bottom, along with arrows to indicate the sequence of modules.
2. The caption has been adjusted to clearly correspond to the different modules in Figure 2.

Please feel free to propose any further suggestions.

Q1: Details about side methods like Zero 1-to-3, SDS guidance, PFNet are referenced in this section while there are very few particular discussions that are taken from these works. For example, what is the fractal approach discussion taken from PFNet?

A1: Sorry for the confusing parts. Zero 1-to-3 is a diffusion model trained to predict a 2D image from any specified camera pose relative to a given 2D input image, ensuring consistency with plausible 3D shapes. Score Distillation Sampling (SDS) guidance is typically used to align rendered 2D images from 3D representations with the distribution learned by a pre-trained diffusion model. By applying SDS guidance from Zero 1-to-3, we constrain the images rendered from our 3D Gaussians to align more closely with realistic 3D shapes. The Fractal approach, inspired by PFNet, focuses on completing only the missing parts of an incomplete input model while preserving existing regions, instead of directly generating fully completed point clouds like others. Following Fractal approach, we optimize $G_m$ to reconstruct the missing parts while freezing $G_{in}$, initialized from the original partial input, to retain existing shapes. We have added some details to the above mentioned sides methods in the Related Works, Sec.2.

Q2: What is the point of applying differentiable quantization to set the gaussians' opacity? It seems that there is a quite simple solution.

A2: Yes, another alternative is to constrain the opacity with loss to force it towards 0 or 1. If we set the optimized opacity attribute as $G_m^o$, such a loss can be described as $L_{opa} = w \cdot min(G_m^o, 1-G_m^o),$ where $w$ denotes the corresponding weight term. $G_m^o$ is between $0 \sim 1$. Below, we present a comparisons between $L_{opa}$ with different $w$ values, and our quantization method mentioned in Sec.3.2 of the original paper. The evaluation is conducted on the synthetic objects following Sec. 4.1.

We can see that our quantization has better promotions for CD metric, while $L_{opa}$ brings more improvements on EMD. However, we can also observe that $w$ will obviously affect the performances of $L_{opa}$, which means we may need to adjust it repeatedly to get the best setting. In contrast, our quantization method does not introduce hyper-parameters. Only $\delta$ in Eq.2 is always set as a constant 0.01 because smaller value will make Gaussian primitives fail to be optimized. In this sense, we think that our quantization method may be more general to use. These two methods can also be selected according to the specific application scenarios.

Q3: What is the reason for the density difference? In the section 4.4. of ablation study we can see that point cloud extraction causes a kind of separate clusters. What is the reason?

A3: Simply speaking, the separate clusters with different densities observed in the extracted results of Sec. 4.4 arise from the differing densities of $G_{in}$and $G_m$, as shown in Fig. 2.

Initially, the optimized 3D Gaussians are distributed below, on, and above the actual surfaces. To address this, we introduce Gaussian Surface Extraction, which identifies and retains only the Gaussians located precisely on the surface.

As described in Algorithm 1, we extract surface points by preserving only the first 3D Gaussian observed in each pixel with an opacity value greater than 0.5. When initializing $G_m$with the same number of partial points as in $P_{in}$, many Gaussians in $G_m$ that are not on the surface are removed during this process. Since $G_{in}$ is initialized directly from $P_{in}$, its points remain perfectly preserved, as they are all precisely on the surface.

As a result, the extracted surface points $P_{surf}$, which are composed of points from $G_m$ and $G_{in}$, exhibit naturally sparser point densities in regions contributed by $G_m$ compared to those from $G_{in}$. While increasing the number of initialized 3D Gaussians in $G_m$ can enhance point density in specific regions, it also significantly increases computational costs during optimization. In this context, our Grid Pulling method offers a more computationally efficient solution for achieving uniform results.

Q3: Comparisons with recent works [1,2].

[1] Diffcomplete: Diffusion-based generative 3d shape completion. [2] Shapeformer: Transformer-based shape completion via sparse representation.

A3: Thank you for your feedback. Unfortunately, we did not find the pre-trained models for DiffComplete [1]. Due to the limited time of the rebuttal period, it is also difficult to re-train it for further comparisons. However, we have included comparisons with ShapeFormer [2] as detailed below. Here is the result on synthetic objects:

Here is the result on Redwood dataset:

Our method demonstrates superior performance, while ShapeFormer [2], like other fully-supervised methods, may be constrained by its training dataset. We have included the corresponding references of [1] and [2] in Sec. 2.2 and updated Table 1 and Table 2 in the original paper accordingly.

Q1: Evaluation on multi-modal metrics such as TMD, UHD, MMD.

A1: To evaluate the multi-modal metrics of our diffusion-based method, we follow [1] to compute TMD, UHD, and MMD. We perform four repeated optimizations of both our method and SDS-complete on the in-domain categories of Redwood, as described in Sec. 4.2, to measure these metrics. The results are presented below.

We can see that our method still performs better on these metrics, confirming it outperforms existing diffusion-based test-time method SDS-Complete in the comprehensive performances of multiple completions.

[1] Diffusion-sdf: Conditional generative modeling of signed distance functions

Q2: Evaluation on multiple incompleteness levels;

A2: Thank you for your suggestion. Here, we use synthetic objects from Sec. 4.1 to construct evaluation sets with varying levels of incompleteness. Specifically, we initialize the first virtual camera at a pose of $elevation = 0^\circ$, $azimuth = -140^\circ$, and $fov \approx 80^\circ$. Additional virtual cameras are placed along the azimuth at 15° intervals. By merging 1, 3, and 7 consecutive depth maps, we generate partial point clouds with different levels of incompleteness. These data are used for comparison experiments. The quantitative results are presented below.

The mentioned quantitative results have been added to Table 10 in Sec. A.6 of the appendix, along with qualitative comparisons in Fig. 15. As shown, using more depth maps to construct the partial input enhances the completed results by providing finer details. Notably, our method consistently outperforms other approaches in this setting.

Q1: The weakness about introducing extra normal maps.

A1: As described in the Gaussian Attributes Setting in Sec. 3.1, all normal vectors are directly estimated using Open3D from the original partial input point clouds. Open3D computes the normal vector for each point based purely on statistical calculations involving its neighboring points. From an overall input perspective, this process does not introduce additional signals compared to other completion baselines, where only the incomplete point cloud serves as input.

Q2: The method introduce the input point cloud across all stages of the pipeline. It would be beneficial to examine how the method performs under varying occlusion levels and perturbation intensities.

A2: We incorporate the input partial points at different stages to ensure the completed results remain consistent with the partial input. To evaluate the performance of our method under varying occlusion levels and perturbations, we conduct the following experiments using the synthetic point clouds described in Sec. 4.1:

1. Partial Point Clouds with Varying Incompleteness Levels: We construct input point clouds with different levels of incompleteness by fusing depth maps from varying numbers of virtual cameras. Specifically, we initialize the first virtual camera at a pose of $elevation = 0^\circ$, $azimuth = -140^\circ$, and $fov \approx 80^\circ$. Additional virtual cameras are placed along the azimuth at 15° intervals. By merging 1, 3, and 7 consecutive depth maps, we generate partial point clouds with different levels of incompleteness. The quantitative results are presented below.

As shown in the table, more complete partial inputs, constructed from additional depth maps, result in improved completion outcomes. The qualitative comparisons are also provided in Fig. 15 of Sec. A.6 in the appendix. Notably, our method consistently outperforms other approaches under these settings.

2. Noised Partial Point Clouds with Different Perturbation Intensities: We generate noised partial point clouds by adding Gaussian noise with varying standard deviations (Std) to the original partial point clouds. Our method is evaluated on noised inputs with $std=\{0, 0.001, 0.005, 0.01\}$. The quantitative results are presented below.

While the performance of our method decreases as noise increases, it consistently outperforms existing approaches. Qualitative results in Fig. 14 of Sec. A.5 in the appendix demonstrate that even with a noise standard deviation of 0.01—causing noticeable blurring in the input partial points—our method can still recover the overall contour effectively. This confirms that our method exhibits a degree of robustness to noise.

Moreover, our primary contribution in this work is the development of a practical framework for leveraging 2D diffusion priors in 3D point cloud completion. Comparisons on real scans from the Redwood dataset further validate the effectiveness of our method in handling real-world data. The robustness against noise is not the core of this work, we can further explore it in our future research.

Q3: Could the method leverage multiple reference viewpoints during the ZFC step?

A3: It may be a little difficult. For the task of point cloud completion, all we can get is a partial point cloud. To complete it, we select a reference viewpoint—the viewpoint from which the partial point cloud appears the most complete. As illustrated in $I_{in}$ of Fig. 2, this viewpoint makes the partial points look like a complete point cloud at most, allowing us to extract comprehensive geometric priors from the diffusion model using SDS guidance. For real scanned point clouds, the reference viewpoint should actually be close to the sensor or camera's perspective. In this framework, the reference viewpoint is unique to our method. Adding additional viewpoints may degrade performance, as they could introduce more incomplete regions from the partial input into the reference image, ultimately disrupting the SDS guidance.

Q4: Could this method benefit from adopting 3D scene representations such as VolSDF [1], as used in SDS-complete, which implicitly encodes the SDF of the underlying surface?

A4: In this work, we adopt 3D Gaussian Splatting as the 3D representation for introducing 2D diffusion priors due to its strong coupling with point clouds. By defining color-related attributes, point clouds can be transformed into 3D Gaussians for differentiable rendering, while the optimization of 3D Gaussians directly manipulates the shape of the point clouds. Our experiments demonstrate that Gaussian Splatting-based frameworks are effective for point cloud completion.

To explore the usage of continuous 3D representations, such as VolSDF, one potential approach is to co-optimize VolSDF through the centers of 3D Gaussians and the 2D diffusion priors during Zero-shot Fractal Completion (ZFC). The final point cloud could then be sampled directly from VolSDF, eliminating the need for Point Cloud Extraction (PCE). This could potentially improve completion performance by avoiding precision loss associated with PCE for 3D Gaussians. However, incorporating another 3D representation would likely result in significantly increased computational costs. Due to the limited time of the rebuttal period, we cannot pursue this attempt at this moment, but it remains a promising avenue for future work.

Q5: Clarity about the citations and other contents.

A5: Thank you for your valuable feedback. We sincerely apologize for the incorrect reference in our original submission. We have thoroughly revised the paper to address all the issues mentioned.

Thank you for your insightful comments! Here, we will present response to your concerns point-by-point:

Q1: Clarity about the using of normal map;

A1: On the one hand, as stated in our previous response, all the information required for normal estimation is derived directly from the original point clouds. Although existing baselines based on supervised learning do not explicitly incorporate the estimated normal maps as input, their networks have learned to implicitly extract features from the input point clouds during training. The input of our framework is just the same partial point cloud as them, where the normals can actually be regarded as our manually extracted features. From this perspective, we think that normals should not be considered as additional input.

On the other hand, while the normal map is primarily used for colorizing the point clouds, its role is important but not irreplaceable.

In Table 3 of the original paper, we conducted an ablation study on the Redwood dataset, which shows that colorization using xyz coordinates achieves slightly lower but comparable performance to using the normal map. In this section, we provide a comparison on synthetic data, as shown in the table below, between these two colorization strategies.

The results indicate that our framework with coordinate-based colorization still outperforms other baseline methods, though slightly inferior to normal map-based colorization.

This confirms that the use of normal maps is not the sole reason for our method's superior performance over other methods. Instead, normal maps just act as a colorization technique that enhances completion performance by better describing surface shapes in the rendered reference image. More effective colorization techniques could be explored in future work to further enhance the completion performance.

Q2: Clarity about the robustness.

A2: We acknowledge that robustness to noise may be a potential limitation of our approach. We will discuss about it more thoroughly in the limitation section of our paper. However, this limitation does not diminish the value of our method.

Firstly, while other baselines may appear to maintain consistent performance under different noise, their results—illustrated in Fig. 14 of the revised paper—are clearly inferior from the outset. In contrast, our method produces significantly more applicable completion results.

Secondly, our experiments on real scans from RedWood dataset in Sec.4.2 shows that our method can perform superior in practical applications. It has confirmed that our method has robustness to real world data.

Thirdly, as mentioned in our previous response, robustness to noise is not the primary focus of this work. Instead, our core contribution lies in designing a practical framework for 3D point cloud completion that does not rely on training with specific datasets, additional prompts, but solely on 2D diffusion priors.

The robustness of this framework could potentially be enhanced by integrating additional denoising modules, such as advanced outlier removal methods for noisy point clouds. However, such extensions fall outside the scope of this work. We can consider about them in our future research.

Q3: Comparison on more incomplete data;

A3: In this work, we follow the setup introduced by PCN [1], where incomplete point clouds are derived from single-frame depth maps. Most existing point cloud completion methods, such as [2, 3], do not consider about more incomplete cases because a single-frame depth map is typically the smallest unit for practical sensor-based collection. Therefore, as shown in our previous responses and Fig. 15, we design varying levels of incompleteness by fusing different numbers of depth maps, reflecting a more practical scenario.

To further evaluate the robustness of our method, we also present comparisons on highly incomplete point clouds by removing points from the partial inputs, as detailed in the table below. Specifically, 30%, 50%, and 70% of points are randomly erased from the original partial point clouds.

The results demonstrate that the performance of our method remains relatively stable across different levels of incompleteness. A possible reason is that removing points does not significantly alter the overall surface shapes, and thus does not greatly affect the colorized reference image, unlike direct noise perturbations.

References:

1. PCN: Point Completion Network
2. TopNet: Structural Point Cloud Decoder
3. DiffComplete: Diffusion-Based Generative 3D Shape Completion

Please feel free to present any further comments or questions.

Dear Reviewer tTgt,

Thank you again for your insightful comments! Since the deadline of the reviewer-author discussion period is approaching today, we wonder if you have any further questions about our work, so that we can response to them timely.

In the previous comment, we presented responses to the proposed problems:

(1) About the normal map: Does our method outperform other baselines due to the advantage provided by the additional input normal map?

(2) About the robustness: although our method maintains better performance under varying noise levels, its performance decreases faster than other methods.

(3) About more incomplete data: how would the performance change with more incomplete input?

We would quite appreciate it if you could provide any further feedback to help us further improve our work.

Sincerely,

Authors