Investigating Domain Gaps for Indoor 3D Object Detection

We thank you for your recognition that our paper explores a potential topic, conduct comprehensive experiments and proposes two large-scale synthetic datasets. We would like to response to your concerns in the following aspects:

1. The comparison of existing synthetic dataset 3D-Front.

We have already discussed the 3D Front dataset in the related work section of the main text and the supplementary material. In Section A.3 in the supplementary material, we analyze that our proposed SimRoom and SimHouse datasets which are generated by ProcTHOR framework have following advantages: (1) 3D-Front dataset is created by professional indoor designers with fixed layouts, thus with high cost and could neither be extended nor easily modified. Our proposed datasets is extensible and could scale up to any size with very low cost. (2) 3D-Front dataset contains limited number of categories and instances. Some object categories in our benchmark which are common among indoor scenes such as TV, toilet, and garbage can are not included or annotated in 3D-Front. In ProcTHOR framework we could add 3D models of any category and could easily configure their numbers and layouts. We train the detector with 3D-Front dataset as source domain on 8 categories (our proposed benchmark include 15 categories). As shown in Table 4 in the supplementary material, the synthetic to real adaptation (3D-Front to ScanNet) remains challenging. Additionally, domain gap exists between manually designed layout and automatically generated layout (3D-Front to SimHouse).

2. The comparison with more existing approaches.

As suggested by reviewers, we also tested OHDA[1], a method that had not been officially published at the time of our submission. This method combines synthetic-to-real data augmentation, the mean teacher framework, and adversarial training techniques. The performance of these methods and detailed analysis can be seen in Page 10, lines 517-528, and Table 3 of the revised version of our paper.

3. The practical meaning of exploring synthetic-to-real domain adaptation.

Real 3D point cloud data faces challenges such as difficulty in collection, annotation, and significant data differences, which have limited the scale of available datasets. Using synthetic data allows for the generation of large-scale datasets at a lower cost and faster speed, enabling the training of scalable 3D perception models. However, there is inevitably a domain gap between synthetic and real data, which affects the effectiveness of synthetic data in supporting real-world models. Therefore, synthetic-to-real adaptation has been a focal point in various domain adaptation studies. This paper focuses on investigating how to bridge different domain gaps, including the synthetic-to-real gap. Reviewer EsE4 describe our motivation as "well-articulated, effectively highlighting the necessity and significance of the research". Cross-domain evaluation gaps show that this is indeed a challenging topic, and we further study more detailed domain gaps, such as object distribution. We believe that future research in this area will enhance the domain adaptation capability of indoor point cloud object detectors, enabling the use of various types of data to train scalable models.

We thank you for your recognition that our paper analyzes the domain gaps of multiple indoor 3D scene datasets, introduces two large-scale synthetic datasets and proposes meaningful domain adaptation benchmarks. We would like to response to your concerns in the following aspects:

1. Detailed domain gap factors in synthetic to real setting should be discussed.

Thanks for your valuable suggestion. Synthetic to real setting contains fine-grained domain gap factors such as texture material and object semantics. With the scalability and flexibility of ProcTHOR framework, we integrated objects from the ScanNet dataset to generate object placement layouts, with the objects being in an intermediate state derived from real-world data. As shown in the Table 4 in the revised paper, incorporating real objects leads to a noticeable performance improvement compared to fully simulated data, highlighting the semantic domain gap between datasets. However, compared to the target domain Oracle, the performance gap remains significant, suggesting that differences in object placement layouts are a more critical domain gap factor.

2. The practical meaning of proposing synthetic dataset and exploring synthetic-to-real domain adaptation.

Real 3D point cloud data faces challenges such as difficulty in collection, annotation, and significant data differences, which have limited the scale of available datasets. Using synthetic data allows for the generation of large-scale datasets at a lower cost and faster speed, enabling the training of scalable 3D perception models. However, there is inevitably a domain gap between synthetic and real data, which affects the effectiveness of synthetic data in supporting real-world models. Therefore, synthetic-to-real adaptation has been a focal point in various domain adaptation studies. This paper focuses on investigating how to bridge different domain gaps, including the synthetic-to-real gap. Reviewer EsE4 describe our motivation as "well-articulated, effectively highlighting the necessity and significance of the research". Cross-domain evaluation gaps show that this is indeed a challenging topic, and we further study more detailed domain gaps, such as object distribution. We believe that future research in this area will enhance the domain adaptation capability of indoor point cloud object detectors, enabling the use of various types of data to train scalable models.

3. The technical contribution for domain adaptation is limited.

As proposed in Section 4.3 of the main paper, we introduce multiple unsupervised domain adaptation techniques to improve performance on the target domain. When selecting domain adaptation methods, we considered the following approaches: (1) Mean teacher framework, which is commonly used in 2D domain adaptation and LiDAR point cloud domain adaptation; (2) VSS, proposed for domain-adaptive 3D semantic segmentation; (3) PPFA, proposed for semi-supervised learning on point clouds; (4) RV, designed for domain-adaptive point cloud classification. Furthermore, based on suggestions from other reviewers, we also tested OHDA[1], a method that had not been officially published at the time of our submission.

Although we do not claim these approaches as our technical contributions, we re-implemented all of them, making adjustments for the indoor 3D object detection task and reporting the improved results for a comprehensive evaluation.

[1] Yunsong Wang, Na Zhao, and Gim Hee Lee. Syn-to-Real Unsupervised Domain Adaptation for Indoor 3D Object Detection. arXiv preprint arXiv:2406.11311 (2024).

4. Suggesting to conduct experiments on real to synthetic setting.

The original intention of designing the synthetic-to-real domain adaptation scenario is to use large-scale and diverse synthetic data to assist in understanding real-world scenes, addressing the issue of insufficient real-world data. However, when the synthetic world is treated as the target domain, there is already a sufficiently diverse and ample amount of in-distribution data available for model training. Therefore, we did not include the real-to-synthetic setting in our benchmark.

Thank you for your suggestion. We conducted experiments in the real-to-synthetic setting, and the results are shown in the table above. The results indicate that models trained with small-scale, low-diversity, and out-of-distribution real data and labels have a significant performance gap compared to models trained with in-distribution synthetic data. Since synthetic data and labels are easily accessible, directly using synthetic data for training would be the optimal choice for improving model performance in the synthetic world.

5. How many unique designs of objects and layouts are included in our proposed synthetic dataset?

We used the ProcTHOR framework to generate synthetic data. This framework supports all 3D object assets provided by the AI2-THOR system, with a total of 3,578 unique 3D assets (excluding room structural components such as walls, doors, windows, floors, and ceilings) across 108 categories. Furthermore, the framework supports random selection of object materials, colors, and scene lighting during the generation process. Therefore, when considering the color attributes of each point in the point cloud, objects with the same shape can appear as different point clouds in different generated scenes. As shown in Table 1 of the main text, our proposed dataset includes bounding box annotations for millions of objects.

For scene layout generation, the ProcTHOR framework uses random seed values to ensure that each scene has a different layout. As a result, our proposed SimRoom and SimHouse datasets have 7,202 and 7,306 unique layouts, respectively.

Additionally, due to the flexibility of the generation framework, we are able to replace 3D object assets from any source during the generation process, and we can also introduce other layouts to simulate the placement of objects. We mentioned the experiment of replacing ScanNet objects in response of question 1.

6. Suggesting to resemble real-world environments when creating the scenes

We mentioned the experiment of replacing ScanNet objects in response of question 1.

We thank you for your recognition that our paper has a well-articulated motivation, introduces large-scale datasets, analyzes critical factors for domain gaps of indoor scenes, and has a rigorous logical flow and fluent expression. We would like to response to your concerns in the following aspects:

1. The efficiency analysis of dataset construction and annotation.

We propose using the ProcTHOR framework for low-cost, flexible, and scalable generation of 3D indoor scenes. Previous indoor 3D datasets, whether designer-built synthetic datasets or real room scan datasets, rarely mention the time and financial costs involved in their creation. The 3D-Front [1] dataset from the simulated environment only discloses that it is derived from 5,000 indoor rooms designed by expert designers in the company’s historical database, and that the dataset's organization and refinement process also relied heavily on the work of designers. The real-world ScanNet [2] dataset mentioned that they use a commodity RGB-D sensor attached to handheld devices such as an iPhone or iPad to scan thousands of rooms. It has annotations by more than 500 crowd workers on the Mechanical Turk platform. While the time and financial cost in these datasets are not explicitly stated, it is clear that the costs are considerable.

Our proposed dataset is fully automated using the ProcTHOR framework, where the generation process determines the precise absolute positions of objects within the scene, eliminating the need for additional efforts in annotation. Based on our measurements from running the generation framework locally on a computer with an AMD Ryzen 7 5700g CPU and 32GB of RAM, the time to generate a single house scene configuration is approximately 13.7 seconds. Using Unity to render it into a .obj format mesh file takes about 64.3 seconds, and sampling point clouds from the mesh file takes around 1.5 seconds. This generation efficiency far exceeds that of other synthetic datasets that rely on human involvement or real-world datasets.

After the review results are announced, we will open-source all our code, including the generation code for SimRoom and SimHouse (allowing users to generate simulation datasets of any scale) as well as all baseline domain adaptation methods we implemented on the benchmarks. At this time, we provide a part of our proposed datasets here. Through our work, we hope to inspire more future research on scene-level domain adaptation for indoor point cloud data.

[1] Fu, Huan, et al. "3d-front: 3d furnished rooms with layouts and semantics." ICCV 2021.

[2] Dai, Angela, et al. "Scannet: Richly-annotated 3d reconstructions of indoor scenes." CVPR 2017.

2. The comparison with more existing approaches.

As suggested by reviewers, we also tested OHDA[1], a method that had not been officially published at the time of our submission. This method combines synthetic-to-real data augmentation, the mean teacher framework, and adversarial training techniques. The performance of these methods and detailed analysis can be seen in Page 10, lines 517-528, and Table 3 of the revised version of our paper.

As for OV-3DET [2], thank you for your suggestion, and we have reviewed several open vocabulary methods for indoor 3D object detection and instance segmentation, many of which report cross-domain testing performance. We evaluate [2] on part of our proposed benchmarks, and report the results in the following table.

However, we believe these methods are outside the scope of comparison in this paper for the following two reasons:

1. OV-3DET uses a backbone network different from that of fully supervised object detection, and due to the lack of annotations, its baseline performance lags far behind that of fully supervised methods (target oracle 37.52 vs 20.79, 44.09 vs 27.92). We believe that a major bottleneck in addressing the cross-domain performance gap for unsupervised or open vocabulary problems lies in the cross-domain generalization ability of the 3D detector itself. Therefore, this paper focuses on cross-domain evaluation of the basic 3D object detector, without directly comparing semi-supervised, unsupervised, or open vocabulary methods.

2. [2] relies on 2D images paired with scene point clouds and the camera intrinsic and extrinsic parameters used to capture them, utilizing additional 2D information as auxiliary data to enable open vocabulary detection. In the benchmark we propose, the 3D point cloud datasets constructed from RGB-D images or videos contain this information, while point clouds directly sampled from synthetic 3D meshes lack inherent camera positions and parameters. Therefore, in the table above, we report only the results for the scan2sun and sun2scan benchmarks. This paper focuses on cross-domain evaluation of pure 3D point clouds, and the methods implemented in our main paper do not rely on additional 2D information. Using 2D information to assist cross-domain scene understanding will be part of our future work.

[1] Yunsong Wang. Syn-to-Real Unsupervised Domain Adaptation for Indoor 3D Object Detection, BMVC 2024

[2] Yuheng Lu. Open-Vocabulary Point-Cloud Object Detection without 3D Annotation, CVPR 2023

We thank you for your recognition that our paper proposes large-scale indoor 3D object detection datasets, and conducts extensive experiments to analysis the domain gaps for indoor point clouds. We would like to response to your concerns in the following aspects:

1. The lack of discussion on the unique values / differences between domain adaptive indoor 3D object detection and domain adaptive LiDAR-based outdoor 3D object detection.

In the related work section of the main text (Page 6, lines 186-200), we discuss domain adaptation methods for LiDAR point cloud detection and highlight the unique challenges of this task in indoor point clouds. We would like to further explain the unique differences in the following aspects:

• Domain gap factors: Outdoor point cloud datasets are collected from LiDAR in autonomous driving street scenes, featuring relatively similar landscapes and uniform point distribution patterns. Research on their domain adaptation mainly focuses on differences in car characteristics, LiDAR scanning beams, and weather conditions. [1] first studies domain gaps for outdoor point cloud detection, conducting experiments on large-scale autonomous driving datasets such as nuScenes, Waymo, and KITTI, and concluding that the primary adaptation hurdle to overcome is the differences in car sizes across geographic areas. [2] proposes a method for adaptation between different LiDAR scan beams. [3] proposes bridging the weather domain gap for LiDAR detectors. Subsequent work has been conducted on the above benchmark, designing general methods to address the aforementioned domain gaps. Our paper is the first comprehensive domain adaptation benchmark for indoor point cloud detection, investigating domain gaps such as point scanning quality (with or without significant point loss), synthetic vs. real data (both layout and object semantic gaps), and room layout configuration (single rooms vs. houses).

• Object categories: Outdoor domain adaptation focuses only on cars ([1, 2, 3] evaluate results only for the car category), while indoor object detection requires identifying and locating many more object categories (15 in our benchmarks) with rich semantic information.

• Object location distribution: The distance between cars or other objects is relatively large in outdoor road scenes. As a result, BEV methods can be applied for detector architecture, and point completion-based methods could be used [4] for domain adaptation. However, indoor scenes feature closely placed objects, with even overlapping bounding boxes.

Overall, outdoor LiDAR point cloud detection and indoor point cloud detection belong to two distinct domains, using different detector architectures and evaluation standards. Moreover, the scale of indoor point cloud datasets is much smaller than that of autonomous driving data, which further increases the difficulty.

[1] Wang, Yan, et al. "Train in germany, test in the usa: Making 3d object detectors generalize." CVPR, 2020.

[2] Wei, Yi, et al. "Lidar distillation: Bridging the beam-induced domain gap for 3d object detection." ECCV, 2022.

[3] Hahner, Martin, et al. "Lidar snowfall simulation for robust 3d object detection." CVPR, 2022.

[4] Xu, Qiangeng, et al. "Spg: Unsupervised domain adaptation for 3d object detection via semantic point generation." ICCV 2021.

2. The usage of ProcTHOR framework.

The focus of this paper is to study the domain gap factors in indoor 3D point clouds. We use the ProcTHOR framework to generate large-scale simulated indoor scene datasets, without conducting innovative research on the generation process itself. We adopted this framework for its scalability and flexibility. With these features, we are able to control the variables of various domain gap factors and investigate domain adaptation tasks in 3D point cloud detection. Leveraging the flexibility of the generative data framework, we integrated objects from the ScanNet dataset to generate object placement layouts, with the objects being in an intermediate state derived from real-world data. As shown in the Table 4 in the revised paper, incorporating real objects leads to a noticeable performance improvement compared to fully simulated data, highlighting the semantic domain gap between datasets. However, compared to the target domain Oracle, the performance gap remains significant, suggesting that differences in object placement layouts are a more critical domain gap factor.

3. The technical contribution for domain adaptation is limited.

We evaluate existing domain adaptation approaches on our proposed benchmarks. Although we do not claim these approaches as our technical contributions, we re-implemented all of them, making adjustments for the indoor 3D object detection task and reporting the improved results for a comprehensive evaluation.

We thank you for your recognition that our paper proposes a novel and meaningful benchmark and provides synthetic datasets with more diverse and scalable data. We would like to respond to your concerns in the following aspects:

1. Suggesting to evaluate more complex domain adaptation techniques such as adversarial training and self-supervised learning.

As proposed in Section 4.3 of the main paper, we introduce multiple unsupervised domain adaptation techniques to improve performance on the target domain. When selecting domain adaptation methods, we considered the following approaches: (1) Mean teacher framework, which is commonly used in 2D domain adaptation and LiDAR point cloud domain adaptation; (2) VSS, proposed for domain-adaptive 3D semantic segmentation; (3) PPFA, proposed for semi-supervised learning on point clouds; (4) RV, designed for domain-adaptive point cloud classification.

Specifically, PPFA utilizes adversarial training as part of its approach, and RV incorporates self-supervised learning when refining pseudo labels. Furthermore, based on suggestions from other reviewers, we also tested OHDA[1], a method that had not been officially published at the time of our submission. This method combines synthetic-to-real data augmentation, the mean teacher framework, and adversarial training techniques. The performance of these methods and detailed analysis can be seen in Page 10, lines 517-528, and Table 3 of the revised version of our paper. Note that although we do not claim these approaches as our technical contributions, we re-implemented all of them, making adjustments for the indoor 3D object detection task and reporting the improved results for a comprehensive evaluation. We will release the details of these re-implemented methods in our open-source code after the review process.

[1] Yunsong Wang, Na Zhao, and Gim Hee Lee. Syn-to-Real Unsupervised Domain Adaptation for Indoor 3D Object Detection. arXiv preprint arXiv:2406.11311 (2024).

2. Deteiled explanation is needed on how the synthetic framework mimics real-world point clouds, as well as how data quality and annotation consistency are ensured.

As mentioned in Section 3.2 of the main paper, we use the ProcTHOR[1] framework to generate synthetic 3D house configurations and sample point clouds from the 3D meshes.

ProcTHOR mimics real-world house configurations by utilizing a combination of realistic 3D models and algorithms that generate indoor environments with physical and semantic properties similar to those found in real homes. It simulates realistic object interactions and physical properties, such as object occlusion, lighting, and shadows, to make the generated environments more realistic and interactive. Though specially designed for mimicking the real world, the domain gap in synthetic data still exists. We adopted this framework for its scalability and flexibility. With these features, we are able to control the variables of various domain gap factors and investigate domain adaptation tasks in 3D point cloud detection.

The generation process involves first sampling the layout of the room's floor and walls, followed by a progressive sampling process to place large objects, wall objects, and small surface objects at specific locations within the room. The generated results are combinations of objects with determined relative positions in the simulator. As a result, the generated rooms inherently define the exact position of each object without the need for additional manual or automated labeling. Compared to real-world scene point clouds, which rely on manual annotation, this method is more cost-effective and provides inherently accurate ground truth for object detection.

[2] Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural generation. Advances in Neural Information Processing Systems, 35:5982–5994, 2022.

3. Future plans of open-source project.

After the review results are announced, we will open-source all our code, including the generation code for SimRoom and SimHouse (allowing users to generate simulation datasets of any scale) as well as all baseline domain adaptation methods we implemented on the benchmarks. At this time, we provide a part of our proposed datasets here. Through our work, we hope to inspire more future research on scene-level domain adaptation for indoor point cloud data.