Point Cloud Dataset Distillation

Thank you for your thoughtful review and suggestions!

W1: Visualization Comparisons.

A1: In the revised manuscript, Section 5.6, we provide visualizations of synthetic datasets generated by DD3D and GM on ModelNet40 and ShapeNet. From these comparisons, we observe the following:

In GM, most points condense into clusters, while some isolated points are left as noise. Moreover, in ShapeNet, the shapes of point clouds are squeezed, making the global shapes difficult to recognize. On the other hand, DD3D generates synthetic point clouds with better quality. First, all points are combined into a whole without isolated points. Second, the global shapes are consistent with the real data without squeezing. This may be attributed to the point-wise generator that learns the relationship between points through implicit neural representations.

These visualizations underscore the strengths of DD3D and reveal the limitations of baseline methods. Additional visualizations are provided in Appendix F to further validate DD3D’s effectiveness.

W2: Applicability to Scene-Level Tasks.

A2: For indoor scenes, such as S3DIS [1], it is feasible to down-sample the real datasets for distillation and generate dense point clouds for validation. For extremely large scenes, such as Waymo [2], down-sampling will affect the quality of small objects, which is still a challenge for DD3D. We appreciate this suggestion and recognize the importance of extending DD3D for scene-level tasks in future work.

[1] 3D Semantic Parsing of Large-Scale Indoor Spaces. CVPR 2016.

[2] Scalability in Perception for Autonomous Driving: Waymo Open Dataset. CVPR 2020.

Q1: Theoretical Assumptions.

A1: In practice, the classifier is indeed a linear layer $W: \mathbb{R}^{d \rightarrow c}$, but the loss function is cross-entropy (CE) rather than MSE. To reduce their gaps, we use KL-divergence to represent CE and MSE:

• CE: $-\sum_i y_i \log x_i = KL(x||y) - C$.

• MSE: $||x-y||^2=KL(x||y), \ \text{s.t.} \ x \sim \mathcal{N}(x, I), y \sim \mathcal{N}(y, I)$.

Their difference is that CE requires a normalized discrete distribution, while MSE needs a continuous distribution sampled from the Gaussian distribution. To align CE and MSE, a simple way is to add some Gaussian noise $z \sim \mathcal{N}(0, I)$ to the representation, thus transforming it into a Gaussian distribution.

Q2: Toy example on aligned and misaligned orientations.

A2: Thanks for the great advice. In the revised manuscript, Appendix F, we use Grad-CAM [3] to visualize the importance of data for PointNet trained on aligned and misaligned datasets. The results show that for models trained on aligned datasets, the importance score is more concentrated, validating our theoretical analysis that rotation-invariant features better preserve principal components. This provides an intuitive understanding of the benefits of alignment.

[3] Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. IJCV, 2019.

Q3: Typo

A3: Thank you for pointing out the typographical issue on line 122. We will correct this in the revised manuscript.

We appreciate your detailed review and the recognition of our contributions.

W1: Limitation in capturing detailed geometric features.

A1: DD primarily focuses on coarse-grained (low-frequency) features that encode principal information. Incorporating high-frequency information to model fine-grained details is indeed challenging.

In our revision Appendix F, we include visualizations of DD3D with varying frequencies controlled by the hyperparameter $t$. A larger $t$ corresponds to incorporating higher-frequency components. These visualizations demonstrate that increasing the frequency enhances DD3D’s ability to capture fine-grained geometric features.

W2: Computational overhead.

A2: The generator introduces additional computations but also significantly reduces the memory overhead, which is more important for large-scale datasets [1].

To verify this, we first evaluate the time overhead of point cloud generation and gradient matching in ModelNet40. Specifically, we use line_profiler to evaluate the time percentages of these two components. The results below indicate that the generation process has minimal impact on computational overhead:

Additionally, we compare the memory usage and accuracy for DD and DD3D across varying CPC values:

Notably, under CPC=10, the memory cost of DD is 10x higher than DD3D, but its performance is still not as good as DD3D. In this way, when applied to large-scale datasets, DD3D can handle larger CPC, while others will face out-of-memory issues due to the limited GPU memory. Therefore, DD3D is more suitable for large-scale datasets.

[1] Dataset condensation via efficient synthetic-data parameterization. ICML 2022.

W3: Limitation for applications requiring high accuracy.

A3: We acknowledge the importance of achieving high accuracy for certain applications. Increasing CPC consistently improves performance, as shown in additional experiments conducted on ScanObjectNN and ModelNet40:

These results show that DD3D achieves near-real-dataset performance when CPC is sufficiently large, making it suitable for high-accuracy applications.

Dear Reviewer V3Ng:

Thank you for your insightful comments, which have been instrumental in refining our work. We greatly value your comments and have addressed your concerns in our revised manuscript.

As the system only allows us to update the revision within the next 24 hours, would you kindly take a moment to review our responses? Your feedback and rating are extremely important to us. We would also appreciate any additional suggestions you might have.

Sincerely,

Authors of Submission4069

We are grateful for your constructive advice and the opportunity to address your concerns.

W1: Generalization of the proposed method for detection tasks.

A1: We recognize that adapting DD3D for the detection task poses unique challenges due to the continuous nature of the label space (e.g., bounding box locations) compared to the discrete labels in classification and segmentation tasks. Given these constraints, pre-defining labels for synthetic datasets in the detection task is non-trivial. As such, we believe the detection task is beyond the current scope of this paper. We plan to explore methods for addressing this challenge in future work.

In addition, we have conducted classification experiments on MVPNet, which is a much larger dataset with 87k samples. The results validate the effectiveness of DD3D over baselines in large-scale datasets.

W2: Why adopt data distillation?

A2: DD significantly reduces dataset redundancy and accelerates neural network training by creating compact, task-specific synthetic datasets. While some information loss is inevitable, DD can achieve performance close to real datasets by increasing the synthetic dataset budget (e.g., CPC=100). The following results highlight this capability:

Furthermore, DD has proven useful in areas like continual learning [1] and privacy preservation [2], demonstrating its broader utility.

[1] An Efficient Dataset Condensation Plugin and Its Application to Continual Learning. NeurIPS 2023.

[2] Privacy for Free: How does Dataset Condensation Help Privacy? ICML 2022.

W3: More comparisons with state-of-the-art rotation-invariant models.

A3: We have discussed some rotation-invariant methods in Lines 197-201. While SOTA rotation-invariant models outperform DD3D on rotated datasets, our goal is different: enabling rotation-variant models (e.g., PointNet) to distill datasets with diverse orientations. Rotation-invariant models rely on specialized architectures, which are challenging to transfer to rotation-variant frameworks. In contrast, DD3D’s rotator is a general, modular approach that can be implemented as a wrapper for any model, offering greater flexibility.

W4: A deeper exploration of these limitations and potential mitigation strategies.

A4: We have revised the limitations section to provide a more thorough discussion of challenges and possible solutions:

• Application to Continuous Label Tasks: Existing methods are not directly applicable to tasks with continuous labels (e.g., detection). A potential solution is discretizing continuous labels, such as treating points within bounding boxes as a separate class.

• Encoding Fine-Grained Details: Current methods focus primarily on coarse-grained information (e.g., shapes), resulting in limitations for fine-grained tasks. Enhancing high-frequency information in gradient matching could address this limitation. We further explore this idea in Appendix F.

Dear Reviewer wYF4:

Thank you for your thoughtful review. Based on your feedback, we have made several improvements in the revision. If there are any remaining concerns or areas where further clarification would be helpful, we would be happy to address them promptly. We deeply value your input and look forward to continuing the discussion.

Best regards,

Authors of Submission4069

We sincerely appreciate your thoughtful feedback and insightful questions.

W1: The proposed method requires different distillation algorithms for different tasks.

A1: We acknowledge your concern regarding the need for task-specific distillation algorithms. We would like to clarify that DD3D is a general method for various point cloud tasks. While it requires task-specific gradients during distillation, the overall algorithm remains unchanged.

For instance, DD3D uses the same distillation pipeline in both classification and segmentation tasks. However, the segmentation task is inherently more complex as it requires both global and local information. To address this, we combine gradients from part-level (local) and shape-level (global) supervision to improve performance.

To validate this approach, we conducted an ablation study to assess the impact of local, global, and combined supervision in segmentation tasks. The results are summarized below:

Additionally, we include visualizations of point clouds generated by DD3D with different objectives in the revised manuscript, Appendix F. These visualizations highlight that global matching alone fails to preserve part-level details, while local matching struggles with spatial relationships between parts. Only the combination of local and global objectives generates high-quality point clouds.

W2: Concerns about the practicality of the proposed method.

A2: When the budget of synthetic datasets is limited, e.g., CPC=1, distillation-based methods significantly outperform the coreset-based methods. Moreover, on large-scale datasets such as MVPNet, DD3D demonstrates an average improvement of 15% over coreset methods, showcasing its practical utility.

Moreover, increasing CPC narrows the performance gap between synthetic and real datasets. We conducted additional experiments with CPC=100 on ScanObjectNN and ModelNet40, as shown below:

W3: Quality of the synthetic point clouds.

A3: We agree that the visual quality of synthetic point clouds is an important aspect. It is important to differentiate the goals of DD and traditional generation tasks:

Traditional generation task aims to approximate the real data distribution, i.e., $p(X_{\mathcal{T}}) \approx p(X_{\mathcal{S}})$ In contrast, DD focuses on creating compact, task-specific synthetic datasets, where $p(X_{\mathcal{T}}) \neq p(X_{\mathcal{S}})$, prioritizing informativeness over realism.

This distinction explains why synthetic data from DD often exhibits deformations. The same phenomenon can be observed in the distillation of image datasets. Despite these deformations, DD3D preserves the critical global shapes and avoids isolated nodes, as demonstrated in Section 5.6 of the revised manuscript.

Q1: Related work.

A1: We appreciate the reference to the related GM-based method. We have included a discussion of this work in the revised manuscript to provide a more comprehensive review.