Point Cloud Dataset Distillation

We are grateful for your constructive advice and the opportunity to address your concerns.

 

Other Strengths And Weaknesses

Q1: The task model is also trained on the original dataset. How does DD significantly reduce the computational cost?

A1: We would like to clarify that the task model is not trained on the original datasets. Generaaly, gradient matching employs a bi-level optimization framework:

• Inner-loop: Optimize task model using synthetic data.
• Outer-loop: Fix the task model and match the gradients between a batch of real and synthetic data.

After distillation, the synthetic data contains the gradient information of real data, thereby converging the models with fewer iterations, i.e., batches.

Q2: Do we need to further train another task model on the distilled dataset?

A2: Yes, another task model needs to be trained from scratch. Although the task model is optimized in the inner loop, its performance is far from ideal, as the synthetic data is not fully trained.

One benefit is that well-trained synthetic data can be used to train different task models. For example, we can use PointNet during distillation and train more complex models, such as DGCNN or Point Transformer, after distillation. A cross-architecture performance is reported in Table 3 of the main text.

Q3: For other objects, how to divide the parts? The division strategy impacts the performance and how?

A3: The parts of objects are defined by the labels of the real datasets. For example, the ShapeNetCore dataset contains 16 object classes and 50 part classes in total. There is a segmentation dictionary to store the correspondence between objects and parts, like {'Airplane': [0, 1, 2, 3], 'Bag': [4, 5], ...}.

In summary, the division strategy is fixed and depends on real data, which will not have different impacts on the performance of DD3D.

Q4: Performance of DD3D

A4: We understand your concerns about the performance of DD. We propose two strategies to mitigate the performance gap between models trained on real and synthetic data

1. Improving the number of CPC. We add the performance of DD3D with CPC=100 in ModelNet40 and ScanObject, and find that it can achieve comparable performance with the full dataset. | ModelNet40 | CPC=50 | CPC=100 | | ScanObject | CPC=50 | CPC=100 | | :--: | :--: | :--: | :--: | :--: | :--: | :--: | | GM | 81.74 | 84.17 | | GM | 57.52 | 62.82 | | DD3D | 83.91 | 86.68 | | DD3D | 61.96 | 65.51 | | Full | 88.05 | 88.05 | | Full | 66.96 | 66.96 |

2. Combined with knowledge distillation (KD). Recent DD methods suggest that leveraging KD can significantly improve the performance of DD. We can also adopt this strategy to achieve nearly lossless performance. | ModelNet40 | CPC=50 | CPC=100 | | ScanObject | CPC=50 | CPC=100 | | :--: | :--: | :--: | :--: | :--: | :--: | :--: | |DD3D | 83.91 | 86.68 | |DD3D | 61.96 | 65.51 | |DD3D+KD | 86.55 | 88.75 | |DD3D+KD | 65.06 | 66.84 | | Full | 88.05 | 88.05 | | Full | 66.96 | 66.96 |

Q5: Comparison with state-of-the-art point cloud analysis models.

A5: The performance of DD3D is behind the SOTA point cloud analysis models, as we only adopt the vanilla 3-layer PointNet as the distillation network for DD3D, which makes a good trade-off between effectiveness and efficiency.

Notably, in the image distillation task, DD methods often use a 3-layer ConvNet as the distillation network [1]. We report the performance of vision DD (Tiny-ImageNet) and point cloud DD (ModelNet40) below.

We can observe that the performance of the 3-layer ConvNet is far from the ResNet-18 model. But it still promotes the development of vision DD. On the other hand, the performance of the 3-layer PointNet is close to the SOTA models, like PointNext. We believe that our research will also benefit the 3D DD field, and future works will extend DD to the SOTA point cloud analysis models.

[1] Dataset Condensation with Gradient Matching. ICLR 2021.

Thanks for the detailed and helpful comments. We reply to the comments in detail. Hope to address your concerns.

 

Methods And Evaluation Criteria

Q1: Potential impact of incorporating more recent rotation-invariant or rotation-equivariant models.

A1: Thanks for the great advice. We have tried several rotation-invariant or equivariant models for DD. However, there are two issues.

1. The architectures of advanced rotation-invariant models are quite different, making it hard to transfer to rotation-variant models.
2. These models often involve complex components. A recent survey on DD [1] shows that more complex models do not necessarily lead to better performance. See Q2 for a performance comparison.

On the other hand, the rotator of DD3D can alleviate the above two issues as it is lightweight and model-agnostic.

[1] Dataset distillation: A comprehensive review. TPAMI 2024.

Q2: How such integration might influence the performance or generalizability of the proposed approach?

A2: We compare DD3D with some SOTA rotation-invariant methods, including RISurConv, TetraSphere (suggested by Reviewer vcTR), and SGMNet (suggested by Reviewer gVCR). The results are shown below.

We have the following observations:

1. Both RISurConv and TetraSphere perform well in the full dataset training. However, they are not suitable for the DD task, as they all need to build a k-NN graph to learn rotation-invariant representations. During distillation, the synthetic point clouds are dynamically optimized, resulting in different nearest neighbors. As a result, we need to re-calculate the k-NN graph in each iteration, which significantly increases the time and space overhead.
2. On the other hand, the rotator of DD3D and SGMNet are model-agnostic, which can be easily combined with different point cloud models. Here we use the 3-layer PointNet as the backbone, and their results are better than RISurConv and TetraSphere. This indicates that the complex models do not always lead to better distillation performance. In contrast, simple models may be more suitable for DD.

Q3: Performance drop compared to using the full dataset and limitation for high-accuracy applications.

A3: We fully understand your concerns about the accuracy. For applications requiring high accuracy, we recommend two ways to improve the accuracy of DD3D.

1. Improving the number of CPC. We report the performance of DD3D with CPC=100 in ModelNet40 and ScanObject, and find that it can achieve comparable performance with the full dataset.

2. Combined with knowledge distillation (KD). Recent DD methods suggest that leveraging KD can significantly improve the performance of DD. We can also adopt this strategy to achieve nearly lossless performance.

Q4: Potential degradation when scaling to larger or more complex tasks.

A4: We conduct semantic segmentation experiments on the scene-level S3DIS dataset, which contains more than 80M point clouds for training. Generally, the scene-level task is more challenging than the object-level task, as it includes more data and noise. We can observe that DD3D consistently outperforms GM by a large margin, demonstrating its effectiveness in large-scale scene-level tasks.

 

Experimental Designs Or Analyses

Q5: Compare the performance and feature characteristics of models trained on both aligned and unaligned orientations.

A5: Table 4 in the main text shows the performance of models trained on both aligned and unaligned orientations. We can see that the performance of GM and DM drops quickly without the help of the rotator of DD3D, demonstrating the effectiveness of DD3D in distillating 3D datasets.

 

Other Comments Or Suggestions

Q6: The authors should revise several typos and grammatical errors in the paper.

A6: We will carefully polish our paper and revise typos.

Thank you for your thoughtful review and suggestions.

 

Other Strengths And Weaknesses

Q1: Comparison between dataset distillation, knowledge distillation, and semi-supervised learning.

A1: Dataset distillation (DD) and knowledge distillation (KD) are two orthogonal directions in efficient deep learning. DD aims to compress data, and KD is used to compress models. A recent DD method [1] suggests that incorporating KD into DD can alleviate the performance gap between the real and synthetic data. To verify this, we report the performance of DD3D and DD3D+KD in the ModelNet40 and ScanObject datasets below.

We can see that with the help of KD, the performance of synthetic datasets is close to the real datasets, indicating the potential of combining DD and KD. Moreover, semi-supervised learning is also a potential application of DD. Leveraging the knowledge of a few labeled samples to compress the massive unlabeled data is also a promising direction.

[1] Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective. NeurIPS 2023.

Q2: Performance improvements over other methods not specifically designed for 3D tasks seem relatively modest.

A2: The advantages of DD3D over other methods not specifically designed for 3D tasks, e.g., GM, DM, and TM, concentrate on addressing the rotation and resolution issues.

1. For the randomly rotated datasets, DD3D outperforms other methods by a large margin. We report the results in Table 4 of the main text below. We can see that the performance of GM and DM drops quickly without the help of the rotator of DD3D, demonstrating the effectiveness of DD3D in distillating 3D datasets. | ModelNet40 | Random | GM | DM | DD3D | | :-- | :--: | :--: | :--: | :--: | | PointNet | 14.75 | 9.47 | 10.16 | 17.91 | | PointNet + PCA | 60.77 | 53.55 | 55.57 | 62.72 | | PointNet + Rotator | 70.13 | 68.92 | 69.31 | 71.27 |

2. For the large-scale datasets, DD3D enables “low-resolution training and high-resolution generation”, which significantly reduces the space overhead during distillation. We report the space overhead of GM and DD3D below, from which we can observe that the memory cost of GM is 10x higher than DD3D, but its performance is still not as good as DD3D. Therefore, DD3D can handle large-scale datasets and larger CPC. | CPC (1/5/10) | Memory (MB) | Performance | | :-- | :--: | :--: | | GM | 120 / 1200 / 6000 | 53.38 / 72.11 / 75.45 | | DD3D | 140 / 230 / 630 | 53.82 / 73.54 / 76.31 |

 

Other Comments Or Suggestions

Q3: It seems that the link in the paper is not anonymous.

A3: We kindly clarify that this link is not related to our paper (our code is in the Supplementary Material) but to a related work on point cloud DD. This link does not violate the double-blind reviewing policy.

We sincerely appreciate your thoughtful feedback and insightful questions.

 

Other Strengths And Weaknesses

Q1: Comparing DD3D to a scenario where a rotation-invariant model is used could help delineate the benefits of the proposed approach. Any discussion in this direction is welcomed.

A1: As suggested, we compare the performance between DD3D, SGMNet [1], and Li et al [2] in the randomly rotated ModelNet40 dataset. Specifically, DD3D and SGMNet are model-agnostic methods, and we use our 3-layer PointNet as the backbone. The results are shown below.

We have the following observations:

1. DD3D has a similar performance to SGMNet, demonstrating the effectiveness of the proposed rotator. Generally, DD3D leverages the fact that eigenvectors are rotation-equivalent to alleviate the influence of random rotations, i.e., $(PR)(R^\top U)=PU$. SGMNet uses the outer product to eliminate rotations, i.e., $(PR)(PR)^\top=PP^\top$. Compared to SGMNet, DD3D only needs to maintain a $3 \times 3$ matrix $U$ rather than an $N \times N$ matrix $XX^\top$, which is more efficient.
2. While Li et al’s work has the highest accuracy in the full dataset, it does not perform well in the distillation task. This observation is consistent with a recent work in DD [3] that networks with complex components may affect the performance of DD.
3. Li et al’s work is out-of-memory in the CPC=50 setting, due to the quadratic complexity of fastest sampling. Each time the synthetic point cloud is optimized, the fastest sampling needs to be recalculated, which significantly increases the time and memory overhead of the distillation process.

Based on the above observation, we have the following discussion:

1. Distillation network is important. The SOTA point cloud models often need to build a k-NN graph or apply fastest sampling on the point clouds, which is not suitable for DD as it costs too much time and space.
2. Model-agnostic v.s. Model-specific. Li et al’s work is model-specific and achieves the highest accuracy. However, it is infeasible to transfer it to some simple models, like PointNet. On the other hand, SGMNet is model-agnostic and can be used for different models, which is more suitable for DD.

Q2: The framework adds extra components (the rotator and generator) to the distillation pipeline, which increases the method’s complexity.

A2: Compared to the gradient matching process, the rotator and generator only slightly increase the computational complexity. To verify this conclusion, we use line_profiler to record the time used in rotator, generator, and gradient matching, respectively, and report their percentages. The results below indicate that the rotator and generator only slightly impact the computational overhead:

Q3: The authors could elaborate on the point-wise generator’s architecture and training process.

A3: Thanks for the great advice. We will add more description of the rotator and generator of DD3D in the revision. We briefly introduce the training process of the generator below.

• The point-wise generator $\mathbb{R} \rightarrow \mathbb{R}^3$ aims to map each sampled noise into a point. Therefore, we can adjust the amount of input noise to control the resolution of synthetic point clouds.
• During training, we sample fewer points from the real data and calculate their gradient as supervision to update the generator. During inference, we can sample more noise to ensure geometric details.

Q4: A deeper understanding of its training procedure or potential failure cases (e.g., ambiguous alignments for symmetric shapes) would be useful.

A5: We will add more visualization in the revision to illustrate how the rotations affect the training process of synthetic datasets.

We appreciate your detailed review and the recognition of our contributions.

 

Experimental Designs Or Analyses

Q1: The approach demonstrates relatively limited performance on fine-grained tasks.

A1: We propose two strategies to further improve the performance of DD3D.

1. Improving the number of CPC. We report the performance of DD3D with CPC=100, and find that it can achieve comparable performance with the full dataset.

2. Combined with knowledge distillation (KD). Recent DD methods suggest that leveraging KD can significantly improve the performance of DD. We can also adopt this strategy to achieve lossless performance.

Q2: This highlights potential limitations in its ability to capture detailed geometric features.

A2: The goal of DD is to make the synthetic data informative, making the synthetic samples deviate from the distribution of real data. As a result, the synthetic dataset may have some artifacts and overlook some detailed geometries. The same results can also be observed in vision dataset distillation.

Notably, Figure 4 (lines 385-395) illustrates that synthetic data generated by GM has much noise and isolated points, making the data unrealistic. On the other hand, synthetic data generated by DD3D is coherent and captures the global geometric shapes.

Q3: The proposed method focuses on shape-level tasks and does not easily extend to large-scale scene-level applications.

A3: We conduct a semantic segmentation experiment on the S3DIS dataset, which contains more than 80M points in the training data and has more noise than shape-level datasets.

We use Area 5 as the test set and other areas as the training set, and report the overall accuracy (OA) and instance mean IoU of different methods. We can observe that DD3D consistently outperforms GM by a large margin, demonstrating its effectiveness in large-scale scene-level tasks.

Q4: Including comprehensive benchmarks against state-of-the-art rotation-invariant methods.

A4: We compare DD3D with some SOTA rotation-invariant methods, including RISurConv, TetraSphere, and SGMNet (suggested by Reviewer gVCR). The results are shown below.

We have the following observations:

1. Both RISurConv and TetraSphere perform well in the full dataset training. However, they are not suitable for the DD task, as they all need to build a k-NN graph to learn rotation-invariant representations. During distillation, the synthetic point clouds are dynamically optimized, resulting in different nearest neighbors. As a result, we need to re-calculate the k-NN graph in each iteration, which significantly increases the time and space overhead.
2. DD3D and SGMNet are model-agnostic, which can be easily combined with different point cloud models. Here we use the 3-layer PointNet. Their results are better than RISurConv and TetraSphere, indicating that complex models do not always lead to better distillation performance. In contrast, simple models may be more suitable for DD.

 

Supplementary Material

Q5: The Appendix B is incomplete with only Rotator, there is no Generator. Also, the Algorithm 2 is different with the provided source code. In addition, the provided source code is incomplete too, without instructions to reproduce.

A5: We will revise our code to make it more complete and easier to read. Specifically,

• Add a PyTorch-style pseudo algorithm of the Generator.
• The implementation of Algorithm 2 is rooted in CINR.py/InvariantWrapper. We also provide a simpler vision of the rotator, which removes the hidden dimension of SIREN to reduce the model complexity.
• Add a Readme file to illustrate the reproduction process.

 

Essential References Not Discussed

Q6: There are several references regarding rotation-equivariant and invariant features that are worth discussing.

A6: Thanks for pointing out these important related works. We will cite and discuss them in the revision.