Unsupervised Point Cloud Completion through Unbalanced Optimal Transport

We sincerely thank the reviewer for carefully reading our manuscript and providing valuable feedback. We hope our responses to be helpful in addressing the reviewer's concerns. We highlighted the corresponding revisions in the manuscript in Brown.

$$

---

W1, Q. Optimal Transport Map is utilized to map the incomplete set to complete set. However, this assume the set of incomplete point cloud and the set of complete set should be complete set. For instance, this work utilized the complete point cloud from ShapeNet and partial point cloud from ScanNet. The original dataset only align the ShapeNet model with the real scans, however, the assumption in Optimal Transport is hard to be satisfied.

A. We are not entirely certain about the specific "assumption" the reviewer is referring to. If our response does not address the reviewer’s concern, we kindly ask for clarification, and we would be happy to provide a follow-up response.

First, if the assumption is about the target data setting, we evaluated our model on the dataset from USSPA [1]. As the reviewer commented, their benchmark is constructed by combining point clouds from two sources: partial point clouds from ScanNet (real scan) and complete point clouds from ShapeNet (synthetic). The motivation for this setup is that an unpaired approach is especially advantageous if we can train the completion model using the synthetic point cloud data, because it is infeasible to obtain the complete point cloud for real scans. In this respect, our results showed that the UOT map approach is effective for this setting. Specifically, learning the UOT map between the point cloud distributions from two sources provides meaningful completion. Moreover, we would like to emphasize that we demonstrated the appropriateness of this cost-minimizing pairing via the UOT map in Sec. 3.1 (Fig. 1 and Table 1).

Furthermore, we would like to highlight that we conducted additional experiments on the PCN dataset, where the complete and incomplete point clouds are from the same sources. In this additional experiment, our model outperforms other baseline methods (Table 9).

• Point cloud completion comparison on the PCN Dataset, assessed by L1 Chamfer Distance $cd^{l 1} \times 10^2$ ($\downarrow$).

Second, if the assumption is about the theoretical perspective, the Optimal Transport Map is a general framework for connecting two probability distributions: the source and target distributions. The given cost function determines how each source sample and target sample are connected by this Optimal Transport Map (Eq. 1). Therefore, there are no specific assumptions on the source and target distributions, except for some technical assumptions in Lines 73 - 81.

[1] Ma, Changfeng, et al. "Symmetric shape-preserving autoencoder for unsupervised real scene point cloud completion." CVPR 2023.

$$

---

W2 Experiments are only conducted in data from Scan2CAD. More experiments should be conducted on MatterPort3D or KITTI like "ACL-SPC: Adaptive Closed-Loop system for Self-Supervised Point Cloud Completion" and ModelNet or 3D-FUTURE like "CloudMix: Dual Mixup Consistency for Unpaired Point Cloud Completion".

A. Thank you for suggesting the relevant references. Since we could not find the code implementation of CloudMix, we conducted additional experiments on the KITTI dataset and compared our model with ACL-SPC and other unsupervised methods. We included the qualitative completion samples in Fig 11 and 12 in the Appendix of our revised version manuscript. Fig 12 illustrates that our model exhibits better completions compared to other methods, particularly by providing more global completions.

$$

---

We sincerely thank the reviewer for carefully reading our manuscript and providing valuable feedback. Moreover, we appreciate the reviewer for acknowledging that "UOT-UPC is the first UOT map model for unpaired point cloud completion" and that "UOT-UPC effectively handles class imbalance problem". We hope our responses to be helpful in addressing the reviewer's concerns. We highlighted the corresponding revisions in the manuscript in Red.

$$

---

W1. Limited Cost Function Exploration: The paper's claim of identifying the "optimal" cost function can be challenged. While the authors compare four different cost functions (l2, L2-Chamfer distance (cdl2), one-directional L2-Chamfer distance (cdl2fwd), and InfoCD), this is a relatively small selection of potential options. Other cost functions might exist that could yield even better performance. The study primarily relies on the ShapeNet dataset for evaluating these cost functions.

A. Thank you for the thoughtful comment. Following the reviewer's suggestion, we revised the statement from "identifying the optimal cost function" to "comparing various cost functions and identifying the most suitable cost function" in the revised version of our manuscript. Furthermore, we conducted an additional ablation study on the cost function using the PCN dataset to further support that InfoCD is the most effective cost function among the four options. The experimental results are presented below:

• Ablation study on the cost function $c(\cdot, \cdot)$ ($cd^{l 1} \times 10^2$ ($\downarrow$)) on the PCN dataset.

The results are consistent with the original ablation study (Table 6) in our manuscript. InfoCD achieved the best performance, while the L2 distance yielded the worst results. We incorporated this additional experiment into our manuscript, which further supports our cost evaluation analysis (Table 10).

Moreover, we would like to emphasize that we compared four widely used cost functions for point cloud data [1, 2], except for the Earth Mover's distance [1], which is too costly to apply during every training iteration. As the reviewer suggested, there may be other alternatives, such as introducing an encoder. However, investigating parametric cost functions is beyond the scope of this work. Additionally, while our cost evaluation was conducted on the ShapeNet dataset in our original manuscript, we expanded our experiments to include 10 categories in a single-category setting, as well as additional multi-category experiments.

$$

[1] Fan, Haoqiang, Hao Su, and Leonidas J. Guibas. "A point set generation network for 3d object reconstruction from a single image.", CVPR 2017.

[2] Lin, Fangzhou, et al. "InfoCD: a contrastive chamfer distance loss for point cloud completion.", NeurIPS 2023.

$$

---

W2. Dataset Dependence: The experiments primarily focus on a single dataset proposed in Ma et al. 2023, despite the availability of other datasets in the field. This raises concerns about the generalizability of UOT-UPC's performance to other datasets. Evaluating the model on a wider variety of datasets would strengthen the paper's conclusions and provide a more comprehensive understanding of UOT-UPC's capabilities and limitations.

A. We appreciate the reviewer for the valuable comment. Following the reviewer's advice, we conducted additional experiments on the PCN dataset. Our UOT-UPC model also outperforms previous unpaired approaches on this dataset. These additional results are incorporated into the revised version of our manuscript (Table 9).

• Point cloud completion comparison on the PCN Dataset, assessed by L1 Chamfer Distance $cd^{l 1} \times 10^2$ ($\downarrow$).

$$

---

Q. Besides, the source mixture trick seems interesting but not well-explained. Can you give more explanation and/or more insights about this trick?

A. The source mixture trick refers to setting the source distribution as a mixture of $50 \\%$ incomplete and $50 \\%$ complete point clouds (Lines 328-335). As discussed in Lines 511-521 (Sec 5.3), we introduced this trick to assist our transport map (generator) in generating the target distribution better. For the mode of the complete point cloud in the source distribution, the optimal transport map ideally learns the identity mapping. Our intuition is that learning the identity mapping for complete point clouds is easier than learning the completion mapping for incomplete point clouds. Note that without the source mixture trick, the transport map is solely trained to perform the completion mapping. In contrast, with the source mixture trick, the transport map is trained to handle both the identity mapping for input complete point clouds and the completion mapping for input incomplete point clouds. We hypothesize that this approach enables the generator to more effectively learn to produce complete point samples by leveraging the simplicity of the identity mapping task. The ablation study (Table 7) empirically supported our hypothesis. Introducing the source mixture trick consistently improved both metrics, Chamfer distance and F-scores. The improvement was more prominent in the fidelity of the generated point clouds (F-scores).

Q1. Compared with the diffusion based mehtods, what are the advantages of the proposed mehtod?

A. First, our model provides efficient sample generation with only 1 NFE (Number of Function Evaluations). In contrast, the diffusion models require a larger number of NFEs (usually $\geq 100$), i.e., neural network inference. This makes our UOT-UPC significantly faster for sample generation. Moreover, our UOT-UPC leverages the Unbalanced Optimal Transport (UOT) theory. The UOT theory provides a theoretical justification for the robustness to class imbalance of our model. We validated this robustness through experiments (Table 5).

$$

---

Q2. Compared with USSPA, the proposed method has comparable (lower) performance in some categories. What is the reason?

A. In Table 3, our model outperforms USSPA in 7 out of 10 categories, achieves comparable results in 2 categories, and shows relatively inferior performance in the cabinet category (10.77 vs. 11.84). However, we would like to emphasize that our model outperformed USSPA on Average in Table 3 and also on PCN datasets, as discussed in the response to W2.

Regarding this cabinet category, note that we evaluated our model in all experiments using the same hyperparameter, i.e., across the multi-category setting and all categories in the single-category setting. However, the optimal cost intensity parameter $\tau$ in $c = \tau \cdot \text{InfoCD}$ depends on the data distributions. In the qualitative samples from the cabinet category (Fig 7), we observed that the point cloud completion from our model was more similar to the input partial point cloud, compared to USSPA. In this respect, we suspect that we applied a larger $\tau$ than the optimal value for the cabinet distribution.

$$

---

Q3. Is it possible to complete unseen categories?

A. We believe that this would be not feasible with our UOT-UPC model, because our model is based on the UOT map between the seen incomplete and complete point cloud data. We think this kind of generalization could be more suited to a Foundation Model trained on a large, diverse dataset. We also find the idea of exploring alternative approaches for point cloud completion on unseen categories interesting and look forward to future research in this direction.

$$

---

Q4. Do the authors think the CD or F_score are the best metrics for evaluating 3D shape completion?

A. Thank you for your insightful question. While the Chamfer Distance (CD) and F-score are commonly used metrics for evaluating 3D point cloud completion, we believe they each have their strengths and limitations. The CD is effective for measuring the closeness of point clouds, as it evaluates how well the generated points match the target points by finding the closest point between them. However, it might not capture certain structural or topological details. For example, when a large portion of points are concentrated together, the CD might not evaluate whether other small portions of details are recovered. The F-score, on the other hand, provides a balance between precision and recall, which can be particularly useful when evaluating how well the model captures both the presence of points and the overall shape. For our work, we chose these metrics because they are widely used in previous works. However, we agree with the reviewer that there is room for development in developing better metrics for 3D shape completion.

We sincerely thank the reviewer for carefully reading our manuscript and providing valuable feedback. We hope our responses to be helpful in addressing the reviewer's concerns. We highlighted the corresponding revisions in the manuscript in Blue.

$$

---

W1. The authors should test more categories to explore the effects of class imbalance.

A. Following the reviewer's advice, we conducted additional experiments on the class pairs of (lamp, trash bin) and (lamp, bed) under various class imbalance ratios $r$. As in Table 5, we evaluated both models using L1-Chamfer distance ($cd^{l1} \times 10^{2}$ ($\downarrow$)). Note that in our original manuscript, we conducted experiments on (TV, Table). The experimental results are presented below:

• lamp : trash bin

$$

• lamp : bed

As in the original results, our UOT-UPC consistently outperforms USSPA under class imbalance. We incorporated these additional results into the revised version of our manuscript (Table 8). We appreciate the reviewer for the valuable suggestion, which helped strengthen our claim.

$$

---

W2. The authors should conduct more datasets for demonstrating the effectiveness of the proposed mehtods, such as PCN datasets.

A. We appreciate the reviewer for providing constructive suggestions. Following the reviewer's suggestion, we conducted additional experiments on the PCN dataset. Our UOT-UPC model also outperforms previous unpaired approaches on this dataset. These additional results are incorporated into the revised version of our manuscript (Table 9).

• Point cloud completion comparison on the PCN Dataset, assessed by L1 Chamfer Distance $cd^{l 1} \times 10^2$ ($\downarrow$).

$$

---

W3. Some important SOTA methods for 3D shape completion are missing. The authors should compare and discuss them with the proposed method. [1] ASFM-Net: Asymmetrical Siamese Feature Matching Network for Point Completion [2] 3D Shape Generation and Completion Through Point-Voxel Diffusion

A. Thank you for suggesting SOTA methods for 3D shape completion. We additionally cited ASFM-Net and PVD in Line 36. However, we respectfully believe that these two works are supervised approaches, and therefore not directly comparable to our unsupervised model. Specifically, ASFM-Net requires paired partial and complete data to train an autoencoder in its first stage of training. PVD utilizes the shape information of partial point clouds as a condition for the diffusion model, which also relies on paired data. As our model is designed for unsupervised point cloud completion, these methods are not appropriate for direct comparison.

$$

---

W4. The computational cost should be analyzed and compared with the other methods.

A. We compared the training time with the most relevant previous work, USSPA. The point cloud completion cost is the same because we share the same backbone architecture. The training costs are comparable. Specifically, the training costs, for the same lamp category and the same 480 epoch, are as follows:

• Train time comparison on lamp

$$

---

W5. The authors are encouraged to provide code for reimplementation.

A. In the original submission, we included the source code of our model in the supplementary material.

$$

---