One for All: Multi-Domain Joint Training for Point Cloud Based 3D Object Detection

Q1: In lines 60-62, the authors claim that 3D sparse convolution is better than point-based feature extractors due to its robustness to domain gaps. Could the authors elaborate on this in detail?

A1:

Table 3-1: Comparison with point-based feature extractor

Point-wise feature extractors typically exploit metric space distances to learn local features. They extract features from point clouds by leveraging geometric characteristics through operations such as sampling and grouping. These operations heavily rely on the geometric information and metric space distances inherent in point clouds. Consequently, when dealing with point clouds from different domains, especially between indoor and outdoor scenes, the vast differences in scene size, object dimensions, and other aspects can severely disrupt the ability of point-wise feature extractors to learn generalizable features. In contrast, 3D sparse convolution operates directly on the voxelized feature space, making it robust to the difference in point clouds. This makes it better suited for the requirements of multi-domain joint training.

We also conduct the comparative experiments and list the results in the above Tab. 3-1. As can be seen, 3D sparse convolution performs significantly better than the point-wise feature extractor. This further demonstrates the effectiveness of 3D sparse convolution when it comes to the large domain gap in point clouds, compared to point-wise feature extractors.

Q2: Regarding the domain router, how is the domain label n obtained? Does data from the same dataset share the same domain label?

A2: We will number the n datasets from 0 to n−1, assigning these numbers as the dataset classification labels. Data from the same dataset will share the same domain label, which will then be used for classification within the domain router.

Q3: In Table 1, different datasets have different views. I wonder if any design could be implemented to tackle this difference for better generalization ability.

A3: Both the model architecture and method designs support the better generalization ability for view difference. On the one hand, our OneDet3D adopts a fully sparse architecture. This architecture operates directly on points without requiring dense features with the fixed sizes, making it more flexible and robust in feature extraction when dealing with datasets from different views. On the other hand, in domain-aware partitioning, scatter partitioning is used to partition the normalization layers of point clouds from different domains. This approach prevents data-level interference between point clouds from different domains, thereby effectively addressing the issue of different views.

Q4: In Equation 1, experiments regarding the hyperparameters $\alpha$ and $c$ are missing. The values of these hyperparameters are not mentioned.

A4:

$\alpha$ is set to 0.25, and $\xi$ is set to 2.0. $c$ is not a hyperparameter and is just the binary target class label.

To illustrate its reasonability, we also conduct the ablation study about these two hyperparameters on the SUN RGB-D dataset, as in the below Tab. 3-2 and Tab. 3-3. As can be seen, our model is relatively robust to the choice of hyperparameters and we utilize the optimal ones.

Table 3-2: Hyperparameter analysis on $\alpha$

Table 3-3: Hyperparameter analysis on $\xi$

Q5: In Table 2, the best performance achieved by existing methods is not highlighted in bold.

A5: Thank you for your suggestion. We will highlight the best performance achieved by existing methods in Table 2 in bold.

Q6: Table 4 underlines the second-best performance, while Table 5 does not. It would be beneficial to maintain consistency.

A6: Thanks for your suggestion. We will underline the second-best performance for both in the final version to maintain consistency.

Q1: The performance in Table 2 for the nuScenes dataset is strange.

A1: The reason is that we train and evaluate only on the car category of the nuScenes dataset. Since only the car class is involved in training, we do not need to use the CBGS sampler for class balance optimization during training, which significantly increases the number of training iterations. These factors cause our results to not be aligned with the numbers reported by other methods. For example, for the UVTR method, the original paper reports the AP of 60.9% for all 10 classes, while its AP for the car category is 84.8%. Without using the CBGS sampler, the car AP becomes 80.6%, which is the result we report in our paper. We will illustrate this in the final version.

Q2: Are any resampling strategies adopted? How to mix them up? The training scheme and implementation details should be added, such as training schedule, data augmentation, and so on.

A2: When mixing different datasets, we first perform dataset-wise uniform sampling across different datasets. Then, we sample individual point clouds from each dataset for training. Detailed explanations of the training scheme and implementation details are provided in the appendix (L460). The augmentations mainly include global translation, rotation, scaling, and ground-truth sampling augmentation. We train the model with the AdamW optimizer for 20 epochs. The initial learning rate is set to 0.0001 and is updated in a cyclic manner.

Q3: How to deal with the different channel sizes?

A3: During multi-dataset training, the attribute channel size is set to the least common multiple of the attribute dimensions from the different datasets, which is 6-dim. The attributes of the point clouds from different datasets are repeated accordingly to match this unified channel size.

Q4: There are many basic grammar issues.

A4: Thanks for your suggestions. We will correct them in the final version.

Q5: There is no comparison or discussion with previous work with fully sparse structures such as [1 - 4]. What are the differences between the proposed architecture and theirs? Do the issues they addressed still exist?

A5:

Table 2-1: Comparison with previous work with fully sparse structures

We compare our method with some existing fully sparse architectures, as shown above. These methods typically aim to achieve a more elegant and efficient network design. However, these methods are generally designed for outdoor scenes. Consequently, the specific designs of their detection heads, such as vote-based methods or BEV detection, are influenced by the structure and content of point clouds and thus are only applicable to outdoor scenes. Additionally, these methods lack designs to address multi-dataset interference, resulting in performance degradation across all datasets during multi-dataset joint training.

In contrast, the anchor-free detection head of our OneDet3D is more versatile for both indoor and outdoor scenes. Furthermore, domain-aware partitioning and language-guided classification can alleviate multi-dataset interference. Therefore, our approach provides a more universal solution for 3D detection.

Q6: Sec. 3.3 is hard to follow and should be rewritten with more details and sufficient explanation.

A6:

• We use the prompt "a photo of {name}" to extract language embeddings of the category names from different datasets using CLIP. These language embeddings are then used as parameters of the fully connected layer to perform the final classification, and are kept frozen during training. Our network does not need to predict such embeddings.
• Since the language embeddings from CLIP are dense features, to utilize the fully connected layer with such embeddings for classification, it is necessary to convert the sparse point features to dense features.
• Dense features refer to commonly used non-sparse features. They are expressed in the form of multi-dimensional matrix and stored as tensors within the network for computation.
• Since the language embeddings from CLIP are kept frozen during training, the parameters in the fully connected layer for classification are frozen. Backpropagation is thus unfeasible in the fully connected layer. This makes the network training relatively difficult.
• We have two classification branches. One branch is the frozen fully connected layer, utilizing CLIP embeddings. The other is a sparse convolution layer. This is trainable and is utilized for class-agnostic classification. “Both branches” in L215 thus denote these two branches.

Q7: Lack of comparison with other similar methods such as [5,6].

A7:

Table 2-2: Comparison with Uni3D

• Uni3D: We compare with Uni3D in the above Tab. 2-2. It can be seen that Uni3D can only handle outdoor point clouds, while OneDet3D provides a universal solution for all point clouds.
• ST3D: ST3D is designed for the unsupervised domain adaptation (UDA) task. Firstly, ST3D can only be applied to outdoor scenes and cannot be used for indoor scenes. Secondly, due to its focus on UDA, ST3D requires unlabeled target domain point clouds during training to perform cross-dataset experiments at test time. The trained model can only be used in the corresponding target domain, making this paradigm relatively inflexible. In contrast, our OneDet3D is more universal. It can be used in both indoor and outdoor scenes. Moreover, once trained, it can be directly applied to various scenes without requiring point clouds from the corresponding domain during training. Therefore, our model is more general and flexible.

Q1: It is recommended to make a clearer comparison and have additional discussions with closely related works

A1:

• Our main contribution is that we propose a universal 3D detector that can directly generalize across various indoor and outdoor point clouds, once trained. Existing works only support either indoor or outdoor point clouds and cannot achieve cross-dataset, especially cross-scene, generalization. We are the first to implement a universal solution for all types of point clouds, which is our core novelty.
• For this, we tackle multi-dataset joint training involving both indoor and outdoor point clouds. There exist significant differences in structure and content between indoor and outdoor point clouds, making this task highly challenging. In contrast, [R1] and [R4] deal with multi-dataset training with RGB images, [R2] and [R3] focus on outdoor-only multi-dataset training, where the discrepancies between different datasets are far less than those between indoor and outdoor point clouds. OneDet3D demonstrates that despite these substantial differences, 3D detection can still be addressed with a universal solution. This is a crucial advancement for generalization in the 3D domain.
• Our designs also differ significantly from existing methods. [R1] and [R2] merely perform knowledge aggregation across different domains. In contrast, we design a domain router to direct information flow, better preventing data-level interference. [R3] uses dataset-aware channel-wise calculation for mean and variance in BN, while we decouple the scaling and shifting parameters. Compared with [R4], we also address structural optimization, using sparse convolution for class-agnostic and FC layers for class-specific classification. In a word, our methods are specifically designed to address the requirements of the challenging problem, making them fundamentally different from existing methods. These designs form a comprehensive system, spanning model architecture, method design, and training, making it more than just a simple combination.
• We compare with Uni3D in the below Tab. 1-1. Uni3D can only handle outdoor point clouds, while OneDet3D provides a universal solution for all point clouds. We will include a discussion of these methods and reposition the related work section to make the discussions more comprehensive.

Table 1-1: Comparison with Uni3D

Q2: Supplementing the results with more recent single-dataset training approaches could further improve the comprehensiveness

A2:

Table 1-2: Comparison with more recent methods

These recent methods target at specific 3D scenes. They may outperform OneDet3D in those particular datasets, but AP tends to drop when the scene changes, especially when switching from outdoor to indoor. After multi-dataset training, due to the dataset-aware interference, AP on all datasets degrade severely. In such multi-dataset scenarios, OneDet3D still achieves the best. Even compared with these recent methods, OneDet3D is still the first universal 3D detector that can generalize across various point clouds.

Q3: The manuscript could benefit from having more graphical illustrations or algorithm flows.

A3: We include a flowchart in Fig. 1 of the rebuttal document. We will incorporate this as an algorithm to provide a clearer explanation.

Q4: How does OneDet3D handle a new point cloud with unknown statistics during inference?

A4:

Table 1-3: Cross-dataset performance on ScanNet

The most dataset-specific statistics, i.e., the mean and variance, are calculated based on the current batch of point clouds. For point clouds from the new domain, the mean and variance are computed according to the current data. We primarily partition the scaling and shifting parameters. For new domain point clouds, the domain router will calculate the domain probability to assess their similarity to existing domains. As is shown in Equ. 3 of our paper, the domain probability weights the outputs from different dataset-specific statistics, addressing the inference issue for new domains.

We further train our model on SUN RGB-D and KITTI, and test it on ScanNet. The results show that our method remains effective for new point clouds with unknown statistics, because the weighted manner allows to select dataset-specific statistics most similar to the test data. This further validates the reasonability of our design.

Q5: How does OneDet3D handle the class mapping differences using the language-guided classification?

A5: Point clouds from different domains use their own CLIP embeddings for classification. They are independent in class-specific classification. As a result, point clouds from different domains can be classified independently, alleviating class mapping differences. Experiments show that despite the different definitions of "car" in KITTI and nuScenes, the performance improves after multi-domain joint training, demonstrating the effectiveness of language-guided classification.

Q6: The manuscript lacks a detailed analysis of the computational cost

A6:

Table 1-4: Efficiency comparison

As can be seen, due to the fully sparse architecture, our network is highly efficient.

Dear Reviewer PVw6:

We thank you for the precious review time and valuable comments. We have provided corresponding responses and results, which we believe have covered your concerns. We hope to further discuss with you whether or not your concerns have been addressed. Please let us know if you still have any unclear parts of our work.

Thanks for your thorough reviews, which are very helpful to improving the quality of our paper. We apologize for any inconvenience caused, but as the deadline for discussion (Aug 13 11:59 pm AoE) draws near, we would like to provide an update on our progress.

If you need further clarification or have additional questions, please don't hesitate to contact us. Again, we sincerely thank you for your time and effort in reviewing our paper.

Thanks