DC3DO: Diffusion Classifier for 3D Objects

We thank the reviewer for their comments on our submission:

W1) As shown in our images and described in the paper, we render meshes rather than point clouds, consistent with MVCNN methodologies. This ensures a fair comparison with the diffusion classifier. While feature aggregation could be enhanced, we aimed to maintain consistency with existing methods. Distribution shift remains a challenge, but our results demonstrate that DC3DO effectively handles such shifts within the scope of the study.

W2) We are aware of PointNet as a seminal work in point cloud classification. Unlike PointNet, our model integrates diffusion classifiers with mesh-based multi-view approaches, which offers different advantages, particularly in generalization capabilities. While PointNet has established benchmarks, it does not utilize diffusion processes, and there remains significant room for improvement in both models as highlighted in recent literature. Generalization across diverse datasets is an ongoing research question that our work aims to address. Due to computational constraints and the availability of pretrained models, we focused on limited number of classes.

W3) Our OOD experiments evaluate domain generalization within the same category (chairs) across different datasets, aligning with methodologies in prior studies[1]. This approach isolates dataset shift effects. Expanding OOD evaluations to include multiple categories is planned for future work.

W4) We have revised the paper to remove the term "accurate" where it inaccurately described our model's performance. Descriptors now precisely reflect the observed metrics.

W5) We have removed the redundant paragraph discussing the obvious relationship between image size, number of views, and processing time to enhance manuscript conciseness.

W6) We recognize the confusion caused by the focus of the ablation study. The ablation studies specifically analyze MVDC to understand the performance tradeoffs related to multi-view input resolution and processing time. This is consistent with the introduction's positioning of MVDC as a baseline, as our primary goal is to compare MVDC against DC3DO in the overall evaluation. While no ablation study is conducted directly for DC3DO, the analysis of MVDC provides complementary insights into multi-view configurations that are relevant for understanding diffusion-based approaches.

W7) We agree that the 20 seconds is a lot for single shape classification at the moment, but also NeRF[2] approaches took days, hours at the beginning, and have not been reduced to seconds. We also performed the classification on a single A10 GPU. Future work will focus on optimizing the inference pipeline and leveraging more advanced hardware to reduce processing times.

[1] S. Shen, Z. Zhu, L. Fan, H. Zhang, and X. Wu, “DiffCLIP: Leveraging Stable Diffusion for Language Grounded 3D Classification,” arXiv preprint, 2024. [Online]. Available: https://arxiv.org/abs/2305.15957.

[2] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis,” arXiv preprint, 2020. [Online]. Available: https://arxiv.org/abs/2003.08934.

Thanks for the detailed response. It partially solves my concern. I have raised my score to 3. However, the paper is still far from the ICLR standards. For example,

W1 & W2 & W3) I don't see why using PointNet is not a fair comparison. One can always sample points given the mesh. Since the authors mentioned [1], [1] also compared with PointNet, and [1] did the evaluation on the full dataset rather than just two classes.

W4 & W5 & W6) Thanks for revising the paper to be more accurate. However, my concern of the ablation study remains. While the analysis of MVDC can provide complementary insights, readers lose direct insights of the proposed method, DC3DO.

W7) The comment on NeRF is not fair. First, NeRF gives significant better novel view synthesis results than almost all previous methods in a wide range of scenes, while this paper only provides results on cars and chairs. Furthermore, traditionally in novel view synthesis, researchers focus on a single scene, and NeRF follows that, which, however, is not the case in point cloud classification.

We thank the reviewer for their comments on our submission:

W1) We acknowledge the limitation of evaluating only two categories. This is due to the availability of pretrained weights for LION, which are only provided for these two categories. Retraining these models for additional categories is computationally prohibitive given current GPU costs, making a broader evaluation infeasible.

W2) At the submission time, considering concurrent work, LION is still the current state of the art for diffusion on 3D without CLIP, 2D Priors, and the current available model to test, unfortunately it comes with pre-trained models only with 2 classes, therefore this is the motivation of these baselines and datasets for a fair comparison. We believe that this paper's findings are in line with concurrent work like DiffCLIP[1]. Using PointNet, DGCNN[2] and other classifiers without diffusion would defeat the purpose and focus of the paper that just aim to show that the diffusion in 3D can be applied in classification tasks. As mentioned we used LION, that has PointNet as an encoder, specifically PVCNN, we compare it with MVDC to demonstrate how pure diffusion-based methods perform on multi-view 3D data. Concurrent works like DiffCLIP further support this direction and are cited to contextualize our contributions.

W3) It could be possible to say the same for the Diffusion Classifier, it is just diffusion and image classifier, however we think that the translation in the 3D space has several novelty and important future research directions. While DC3DO builds upon the integration of LION and diffusion classifiers, its novelty lies in the incorporation of hierarchical latent spaces and advanced feature aggregation techniques specifically tailored for multi-view 3D classification. These strategic enhancements improve generalization capabilities and classification performance beyond a naive combination, distinguishing our approach from previous methods.

W4) We acknowledge the misuse of “zero-shot classification” in our manuscript. Our experiments focus on domain generalization, evaluating the model’s ability to generalize within the same category (chairs) across different datasets. We have revised the manuscript to accurately describe this as domain generalization instead of zero-shot classification, ensuring terminological precision.

Q1) As addressed in W1, our initial focus on two categories was due to the availability of pretrained models. LION only released the weights of these two categories, therefore that was the only possible test. Retraining such models is highly prohibitive for GPU costs.

Q2) See W2.

[1] S. Shen, Z. Zhu, L. Fan, H. Zhang, and X. Wu, “DiffCLIP: Leveraging Stable Diffusion for Language Grounded 3D Classification,” arXiv preprint, 2024. [Online]. Available: https://arxiv.org/abs/2305.15957.

[2] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon, “Dynamic Graph CNN for Learning on Point Clouds,” arXiv preprint, 2019. [Online]. Available: https://arxiv.org/abs/1801.07829.

We thank the reviewer for their comments on our submission:

W1) We acknowledge the limitation of testing on only two categories and a single dataset due to reliance on pre-trained LION model weights, which are currently only available for these two categories. This aligns with complementary and concurrent works.[1]

W2) While MVDC does not extend LION to a multi-view framework, we extended image classifier to a multi view. In fact, our primary contribution lies in DC3DO that extends LION as a classifier , which integrates hierarchical latent spaces with a diffusion classifier specifically tailored for 3D object classification. DC3DO introduces novel feature aggregation techniques and optimization strategies that enhance classification accuracy and generalization beyond a simple multi-view extension. These structural and strategic enhancements differentiate our approach from existing methods.

W3) We recognize that the multi-view diffusion process introduces significant computational overhead. We agree that the 20 seconds is a lot for single shape classification at the moment, but also NeRF[2] approaches took days, hours at the beginning, and have not been reduced to seconds. We also performed the classification on a single A10 GPU. Additionally, we argue that for applications where accuracy and generalization are critical, the performance gains justify the additional computational cost. We believe future advances will similarly reduce inference time for diffusion-based models.

W4) We acknowledge that the robustness claim could have been better supported with additional experiments. While we did not explicitly test the model under adverse conditions such as noise or varying lighting, our results demonstrate that DC3DO achieves consistent improvements in accuracy and generalization over the baseline across standard evaluation scenarios. Future work will aim to explore robustness more comprehensively under varied conditions.

[1] S. Shen, Z. Zhu, L. Fan, H. Zhang, and X. Wu, “DiffCLIP: Leveraging Stable Diffusion for Language Grounded 3D Classification,” arXiv preprint, 2024. [Online]. Available: https://arxiv.org/abs/2305.15957.

[2] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis,” arXiv preprint, 2020. [Online]. Available: https://arxiv.org/abs/2003.08934.