OpenDlign: Open-World Point Cloud Understanding with Depth-Aligned Images

We thank the reviewer for the interesting questions.

Q1: Would inconsistent textures in multi-view images of a single object impact performance?

Answer: We believe that inconsistent textures in multi-view images improve model performance in 3D representation learning, rather than hinder it. Here are our supporting arguments and experiment evidence:

Supporting Arguments:

1. Effective Knowledge Transferring: Inconsistent textures between multi-view generated images allow depth maps to align with diverse sets of RGB images. This diversity enhances the transfer of rich 2D knowledge embedded in CLIP to 3D representation learning.

2. Invariant Feature Learning: Generating geometry-consistent but texture-inconsistent multi-view images can be beneficial for learning 3D representations that emphasize object shape, size, and contour, which remain robust to texture variations. This approach makes sense because 3D data primarily provide geometric features rather than color or texture.

3. Independent View Pairing: Similar to methods like CLIP2Point, ULIP, and OpenShape, our OpenDlign pairs a single-view depth map with a single-view RGB image for contrastive learning, instead of using all multi-view images of a single object. This means each view is treated independently, eliminating the need to maintain texture consistency across views.

4. Conventional Augmentation in Contrastive Learning: Texture features are not considered robust not only in 3D data but also in 2D images, as evidenced by the widely used color jittering for data augmentation in 2D contrastive learning. Improving texture diversity is a conventional strategy to enhance the robustness of contrastive learning models.

Experiment Evidence:

Table 1 demonstrates that using generated images with inconsistent textures in (+dlign) settings systematically outperforms texture-consistent CAD-rendered images used by other methods without (+dlign). Furthermore, the ablation study in Table 5 shows that OpenDlign achieves a 3.8% and 6.8% accuracy improvement on the ModelNet and ScanObjectNN datasets, respectively, when using texture-inconsistent images compared to depth-rendCAD with consistent texture.

Q2: Using diffusion-generated images for 3D understanding is not novel.

Answer: We acknowledge the growing interest in using diffusion-generated images for 2D understanding. However, after carefully considering the feedback and reviewing the referenced work, we respectfully disagree with the assertion that our paper's novelty is similar to the referenced work. Below, we outline our reasons:

1. CLIP is designed for 2D representation learning, not 3D understanding. The referenced work [1] also focuses on enhancing CLIP's 2D image understanding using diffusion-generated images. In contrast, our approach transfers CLIP's 2D knowledge into 3D understanding, expanding its application to a new task. Currently, the use of diffusion-generated images in 3D understanding is largely unexplored, as shown by the papers citing [1].

2. The data scarcity challenge in 3D vision is much more severe than in 2D vision. State-of-the-art CLIP models for 2D understanding use billions of real-world image-text pairs, highlighting the substantial volume of pretraining data. The authors [1] recognize the need to explore if diffusion-generated images benefit larger CLIP models pre-trained on extensive datasets. However, the 3D datasets available for training 3D open-world models (i.e., '3D CLIP') are synthetic and contain less than a million samples. Our work mitigates this critical gap, emphasizing its broader impact and significance.

3. The contributions of this study are multifold, including contour-aware depth-map projection, a multimodal alignment framework, depth-aware prompt design, and a continual learning method to mitigate catastrophic forgetting. These innovations consistently result in significant performance improvements, as demonstrated in Table 4. The use of depth-aligned images is one of the key contributions, potentially benefiting all existing 3D open-world methods.

4. In [1], the generated images need only to be consistent with the text prompt. Our work, however, requires images to be consistent with both the object's pose and the text prompt, necessitating a more sophisticated generation process.

Furthermore, we reviewed recent work utilizing diffusion models for various 3D understanding tasks. These studies [2,3] primarily use diffusion features to replace CLIP features for specific downstream applications, such as semantic segmentation [2,3], with very few explicitly generating images. In contrast, our work, similar to PointCLIP and OpenShape, focuses on learning robust 3D representations. These representations are versatile and can enhance various downstream tasks, including point cloud classification and 3D object detection, without being limited to specific applications. 3D classification is just one of the downstream tasks used to evaluate representation robustness.

Q3: 3D understanding uses scene-level points in most downstream applications, please try on 3D scene-level datasets.

Answer: Open-world 3D representation learning is a new field, mainly focusing on object-level understanding due to limited 3D scene data. Our setup fully follows existing methods, and we included additional experiments showing OpenDlign's performance on open-vocabulary 3D detection using the scene-level ScanNet dataset, as shown in Table 1 in the rebuttal pdf. As noted in Appendix A.1, we plan to extend our work to scene-level representation learning because CAD-rendered images for multimodal alignment are not always available for 3D scenes.

[1] Is synthetic data from generative models ready for image recognition? (ICLR 2023)

[2] Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models. (Arxiv)

[3] Open-vocabulary panoptic segmentation with text-to-image diffusion models.(CVPR 2023)

We appreciate the reviewer's constructive questions and feedback.

Q1: The motivation of the study is not focused on training a good encoder specifically for 3D understanding from depth maps.

Answer: Our primary motivation is to learn robust point cloud representations for open-vocabulary problems, similar to the goal of PointCLIP [1], as highlighted in its title, "Point Cloud Understanding by CLIP". We train a depth encoder mainly to indirectly enhance representation learning from point clouds. While the reviewer's suggestions are beneficial for building a strong depth encoder specifically for depth map understanding, their benefits may be limited when the focus is on point cloud understanding.

Additionally, we want to clarify any confusion caused by our paper title. By "depth-aligned images," we refer to images generated using depth map information, not the problem of "depth-image alignment." Our approach aims to align point cloud projected depth maps and images, thereby implicitly enhancing point cloud-image alignment robustness. Recent works like ULIP and OpenShape use the term "3D understanding" broadly without explicitly mentioning "point cloud," but their core objective, like ours, is to learn robust point cloud representations.

Q2: Why not use mesh to directly render depth maps instead of projecting depth maps from point clouds?

Answer: Generating depth maps from point clouds is preferred over using 3D meshes for several reasons:

1. Real-Time Applicability: The contour-aware projection method in OpenDlign generates depth maps directly from point clouds, making it more efficient than 3D rendering from meshes. Creating rendered depth maps involves a computationally expensive multi-step process: first, reconstructing the surface from the point cloud to create a 3D mesh, and then rendering the 3D mesh to obtain depth maps. This process introduces noise and makes rendered depth maps impractical for many real-time applications [1].

2. Consistency with Related Works: The projection of multi-view depth maps from point clouds is a well-established approach. Studies like PointCLIP, PointCLIP v2, and CLIP2Point also focus on proposing better point cloud-to-depth map projection methods. The motivation of these studies, similar to OpenDlign, is to enhance computational efficiency towards real-time point cloud understanding.

3. Compatibility with Mainstream Methods: Mainstream 3D understanding methods, such as PointNet and Point Transformer, operate directly on point clouds. OpenDlign's use of point clouds ensures compatibility and efficient processing with these methods.

In summary, OpenDlign focuses on point cloud representation learning, not depth map representation learning. If our goal were 3D understanding from depth maps, we could use various methods to obtain them. However, since our focus is on point cloud-based 3D understanding, we constrain our depth maps to be projections from point clouds. We hope these explanations clarify our motivation and approach.

Q3: Why not use real images and predict the depth maps using existing depth estimators, such as DepthAnything?

Answer: Thank you for your insightful question. Using real images and pseudo-depth pairs can indeed increase data diversity. However, a key challenge is the domain discrepancy between depth maps predicted by estimators like DepthAnything and those projected from point clouds. This discrepancy arises because DepthAnything's depth maps are influenced by depth estimation errors, while point cloud-projected maps have noise from depth densification. Training with synthetic mono-depth from DepthAnything might limit the model's ability to generalize to point-cloud projected depth maps due to this domain shift.

To address this, a potential approach is to pre-train the model with a large dataset of image-pseudo depth pairs and then fine-tune it on projected depth maps. Using multiple depth estimators, such as DepthAnything, MiDaS, and ZoeDepth, to generate pseudo depth can further enhance data diversity and improve robustness. It sounds like a promising direction for our future work.

Q4: The definition of inverse depth is incorrect.

Answer: Thank you for pointing that out. We indeed use normalized inverse depth as conditional control for image generation, as evidenced by the correctly generated images shown in Figure 2 in the rebuttal PDF. We will correct this in the revised manuscript.

[1] PointCLIP: Point Cloud Understanding by CLIP (CVPR 2022)

We appreciate the reviewer's insightful feedback. We’re glad to hear that you found our work innovative, effective and reproducible.

W1: Limited Dataset for Depth-Aligned Images

Answer: Due to limited resources, our generated dataset is currently limited to ShapeNet, as discussed in the limitations section. However, post-submission, we generated more images from the ensemble dataset to enhance our model's generalization capabilities. The table for Reviewer ky3c shows that existing SOTA methods' performance keeps improving on all benchmark datasets as the dataset grows from 52K to 90K depth-aligned images.

W2 & Q2: Computational Resources and Feasibility for Generating Depth-Aligned Images on Large Datasets.

Answer: We understand the reviewer's concern about the computational constraints. Below we detail the computational resources necessary for generating depth-aligned images, discuss the feasibility of scaling our method to larger datasets, and confirm data accessibility:

1. Memory Efficiency: Multi-view depth-aligned images can be generated on any GPU with 24GB of memory using the ControlNet model, ensuring memory-efficiency.

2. Time Efficiency: Initially, generating images took two weeks using eight Quadro RTX 6000 GPUs (released in 2018). With newer RTX 4090 GPUs, which are 2.78 times faster, this process can now be completed in less than a week. The RTX 4090 is a consumer GPU, making it more accessible to researchers.

3. Comparison with Rendering: Previous methods that generate CAD-rendered images using rendering tools (e.g., Blender) also demand considerable computational power and time. Therefore, both CAD-rendered images and depth-aligned image generation face similar computational constraints.

4. Data Accessibility: We will open-source the existing depth-aligned image dataset for ShapeNet, allowing direct access for the research community to avoid redundant generation.

5. Faster ControlNet: Recent work, like DMD2 [1], could serve as a faster alternative to ControlNet. DMD2 achieves high-quality conditional image generation more efficiently with much fewer inference steps compared to traditional methods like ControlNet. Preliminary results indicate that DMD2's four-step inference process for depth-conditioned image generation is 2.59 times faster than ControlNet, as shown below:

W3 & Q3: Potential Biases in ControlNet.

Answer: Firstly, we require further experiments to quantify the effect of bias on the 3D representation robustness. One potential solution to mitigate bias is using a VLM (e.g., GPT-4V) or an image captioning model to evaluate generated image attributes like color, texture, and style. If the generated images have too many consistent attributes (like the same color), the diffusion model can be prompted to regenerate the images using different seeds. We will explore this direction in future work.

Q1: Application Study on Open-Vocabulary 3D object detection.

Answer: In section 4.3, we have demonstrated that OpenDlign outperforms PointCLIP and PointCLIP v2 by a large margin on the open-vocabulary 3D object detection task. Additionally, we followed the setting in [1] to compare our method with OV-3DET and CoDA on scannet dataset. Specifically, we use pseudo-box pre-trained 3DETR to locate objects and OpenDlign (ViT-H-14) to classify objects.

The results below show that OpenDlign performs comparably to the SOTA CoDA method. This is already promising because our method is not specifically designed for OV 3D detection. For detailed results across all categories, please see Table 1 in the rebuttal PDF.

Q4: Plans for Extending to Other 3D Datasets?

Answer: Yes, we plan to first apply our method to the entire ensemble dataset for a more robust 3D shape understanding. Our next goal is to adapt our approach to 3D scene understanding, inspired by studies like 3D-Vista [2], which have successfully used CLIP for learning 3D scene representations by aligning point clouds, images, and text. We envision our method serving as a strong baseline for this task.

Q5: Scalability of OpenDlign on Larger CLIPs.

Answer: Table 1 in the main paper shows the zero-shot performance of OpenDlign with four CLIP backbones (B-32, B-16, L-14, H14). The short answer is better CLIP in 2D understanding leads to better OpenDlign in 3D understanding. We will elaborate on this relationship in the revised paper. The table below shows the model size and pre-training datasets, and 2D zero-shot classification averaged over 38 datasets of different CLIP backbones (https://github.com/mlfoundations/open_clip/blob/main/docs/openclip_results.csv). ViT-H-14 is one of the most powerful CLIP models so far. We can compare this result with the results in Table 1 to validate OpenDlign's scalability.

[1] Open-vocabulary point-cloud object detection without 3d annotation. (CVPR 2023)

[2] 3d-vista: Pre-trained transformer for 3d vision and text alignment. (ICCV 2023)

We greatly appreciate the reviewer's insightful feedback and valuable recommendations.

Weakness 1: Unclear performance gain on ensemble dataset.

Answer: We understand the concern regarding the scalability of our methods on large datasets. Here are some explanations and additional experimental results to address this issue:

1. Low Proportion of Depth-Aligned Images: The performance on the ensemble dataset is not significant because only 6% of the images are depth-aligned (from ShapeNet), while the rest are CAD-rendered. It is impressive that even using a small portion of depth-aligned images improves the existing SOTA model's performance in most settings.

2. Small Benchmark Datasets: Training on the ensemble dataset typically results in larger performance gains on larger datasets like OmniObject3D compared to smaller ones like ScanObjectNN and ModelNet40. This is because the ensemble dataset offers a broader range of categories, which is more advantageous for larger datasets with more labels, as shown in Table 1 L.223. Specifically, ModelNet40, ScanObjectNN, and OmniObject3D have 40, 15, and 216 categories respectively.

3. Domain Shift in ScanObjectNN: The performance drop on ScanObjectNN is due to its domain shift with other 3D datasets. ScanObjectNN, being the only real-world point-cloud dataset in our experiments, features more sparse and noisy point clouds compared to those sampled from 3D meshes (e.g., ModelNet40 and OmniObject3D). This visual difference is demonstrated in Figure 1 of the rebuttal PDF. This domain shift causes previous methods to struggle with improving performance on ScanObjectNN as well. For example, OpenShape-PointBERT increases Top-1 accuracy by 21% on OmniObject3D but only by 0.9% on ScanObjectNN, as shown in Table 1.

4. Saturation of ModelNet40 Performance: ModelNet40 is a relatively easy benchmark, and existing methods have already achieved high performance on this dataset. The 1.2% improvement from TAMM-PointBERT to TAMM-PointBERT (+dlign) on ModelNet40 Top-1 accuracy is already more significant than the 0.6% improvement of TAMM-PointBERT over OpenShape-PointBERT on the same metric.

To further validate the benefits of depth-aligned images on a larger dataset, we conducted an additional ablation study. We increased the proportion of depth-aligned images in the training dataset from 6% to 10%, resulting in a total of 90K multi-view depth-aligned images for training TAMM-PointBERT and OpenShape-SparseConv. The results in the table below can be compared to Table 1 (L.223), where all scores improve, demonstrating that these methods achieve positive gains on all metrics when using more depth-aligned images for training.

However, we acknowledge that generating depth-aligned images for the entire ensemble dataset is time-consuming and computationally expensive, which will be our focus in future work.

Weakness 2: What is the name of the diffusion model? How do you ensure multi-view consistency?

Answer: The diffusion model used in our work is the ControlNet v1.1 [1] depth model (L.151), which allows us to use depth information to control the pose and shape of generated images.

There is no multiview inconsistency in terms of geometry because we generate depth-aligned images view by view, each with its respective depth map. ControlNet's conditional control helps maintain the same shape and pose as the input depth map, and a text prompt ensures the generated images have consistent object identity. Since these depth maps come from the same 3D point cloud, they result in consistent depth-aligned images across different views. Figure 2 in the rebuttal PDF provides visual examples to further demonstrate this consistency. Multi-view consistency remains an open problem for diffusion models when generating multi-view images from a single-view input [2,3], rather than from multi-view inputs.

While the generated depth-aligned images are geometry-consistent, they are not texture-consistent across views. Each view may have different textures, which we believe positively impacts generalizable representation learning, as discussed in the main paper.

Weakness 3: Why not generate multiple sets of multi-view images for a 3D asset to enhance training diversity?

Answer: Due to computational constraints, we only generated one image per depth map (view) for each 3D asset in our experiments. We agree that generating multiple sets of multi-view images using different random seeds for a 3D asset can further enhance training diversity. This will be considered in our future work.

[1] ControlNet 1.1: https://github.com/lllyasviel/ControlNet-v1-1-nightly

[2] Shi, Ruoxi, et al. "Zero123++: a single image to consistent multi-view diffusion base model."

[3] Liu, Yuan, et al. "Syncdreamer: Generating multiview-consistent images from a single-view image."

Thank you for your feedback. We're pleased to know that our rebuttal has addressed your most concerns. If this resolves the issues, we would greatly appreciate it if you could reconsider your rating. Otherwise, we are always happy to answer further questions that you may have.