ZOPP: A Framework of Zero-shot Offboard Panoptic Perception for Autonomous Driving

Response 4.1 Data labeling issue. (Weakness 1 and Question 1)

Previous literatures focus on generating high-quality 3D detection results as auto labels with offboard perception fashion. However, they still need high-quality human labels of the AD dataset as a prerequisite for training the whole pipeline. This is precisely the issue of data auto-labeling we aim to address.

Therefore, we propose a unified framework with a compact structure to support various perception tasks of AD scenes in an offboard manner, without any requirements of human labels from AD dataset. To be specific, each proposed module and stage of our method do not rely on any human labels to generate the corresponding perception results. Naturally, we can leverage ZOPP as a cold-start paradigm for existing auto-labeling methods.

Response 4.2 Computational resources and hardware specifications (Weakness 2)

We have introduced the corresponding computational resources in Sec. C (Implementation Details) of Appendix, please kindly refer to it.

The entire pipeline of our proposed method is a lightweight and compact framework, which does not rely heavily on many computational resources. Only the point completion module and the resconstruction module need to train the networks. We utilize 1 NVIDIA A100 to train the point completion network, and 4 NVIDIA A100 to accelerate the reconstruction with multi-processing settings.

Response 4.3 Writing structure and incoherent presentation (Weakness 3)

We will improve the presentation and polish the structure in the revised version.

Response 4.4 Insufficient evidence (Weakness 4)

Could you please kindly specify which aspect of the content "insufficient evidence" refers to?

Response 4.5 Training and testing procedures (Weakness 5)

We have presented the detailed training and resting procedures, hyper parameters, and settings for each module in Sec. C (Implementation Details) of Appendix, please kindly refer to it.

Response 4.6 Instance management across multi-view cameras (Question 2)

In section 3.1 of the main contents, we have proposed and introduced a multi-view mask tracking module to manage the object instances across multiple views. Specifically, we designed a simple yet effective similarity cost involving the computation of appearance and location similarities to facilitate object association. The appearance similarity is compared across different objects by computing the cosine distance of the visual features. The location similarity is derived by concatenating the images of all viewpoints in a panoramic order, followed by normalizing the pixel distances along the horizontal axis for each object. Therefore, objects with large similarity scores would be associated together with the same instance ID. We will then manage all the object instances with their corresponding unique IDs in the following stages of our approach.

Response 4.7 Removing background 3D points at the upper edges of foreground objects (Question 3)

Firstly, since LiDARs are always equipped higher than multi-view cameras on autonomous vehicles [1,2,3], the parallax occlusion issue will thereby arise at regions around the upper edges of foreground objects, rather than the center or bottom parts of foreground objects.

Secondly, we have designed an experiential threshold of distance $\theta$ to determine whether the projected background pints should be filtered out. Regardless of the disparity differences among these projected background points, as long as they exceed the threshold, we will filter them out.

Response 4.8 Specific kernel design of parallax occlusion filtering (Question 3)

The design of the kernel is influenced by the configuration of the sensors, specifically determined by the beam numbers and rotating frequency of the LiDAR. They respectively determine the vertical and horizontal densities/resolutions of the point cloud projected onto 2D image plane. Therefore, if the LiDAR has a large number of beams and a high rotation frequency, we need to use a smaller kernel size to handle the dense projection points and improve the accuracy of the filtering operation. Conversely, if the density of projection points is lower, we can increase the kernel size and step size along the vertical and horizontal directions, thereby enhancing the algorithm's operational efficiency without compromising filtering accuracy.

Response 4.9 Mathematical formulation of projection (Question 4)

The process of projecting all 3D point clouds onto a 2D image plane follows the same projection formula, as discussed in Sec. 3.2.1. There is no separate projection formula for background points; rather, they appear in the region of foreground objects in the 2D image due to the parallax occlusion problem mentioned above. This issue is precisely what our proposed method aims to address.

Response 4.10 Sub-pixel level results (Question 4)

Currently, we do not support sub-pixel level calculations. All 3D point cloud projections onto the 2D image are rounded to the nearest integer pixel value coordinates, to obtain pixel-level semantic and instance ID information. Additionally, the semantic and instance mask results generated in the previous stage of our method are also at the pixel level.

Response 4.11 Likelihood of applying in real-world applications (Question 5)

In our experiments, we leverage WOD as the primary benchmark to assess the effectiveness of our method. WOD is one of the large-scale autonomous driving dataset collected from real-world commercial vehicles. In addition to evaluating the main results across various perception tasks, we also perform auto labeling applications. As shown in Sec. D.4 of Appendix, we generate 3D boxes on 5% of training set as auto labels to train the onboard detection model. This experiment demonstrates that our method could generate comparable auto labels and serve as an efficient cold-start approach for existing perception models.

We are grateful to you for recognizing our efforts in addressing your concerns during the reviewing process.

Response 3.1 Lack of novelty (Weakness 1)

Perception and understanding play a vital role in current data-driven autonomous driving. Previous literatures focus on alleviating the burdens if human labor and the cost of labeling. And we found several challenges in this field:

• Only 3D object detection task is supported to generate auto labels in an offboard manner.
• Still require huge amounts of data with high-quality human-labels.
• Lack the capabilities of open-set and zero-short settings.

Therefore, we tackle these challenges by proposing a unified framework for various perception tasks in an offboard manner without the requirements of human labels in AD scenes. Although our method incorporates several existing foundation models, previous research has not explored these aspects to address the practical needs of auto labeling in AD. To the best of our knowledge, we are the first to propose such kind of work.

Response 3.2 Confusing presentation (Weakness 2)

We will revise the writing carefully in the next version.

We sincerely appreciate your efforts during the review and responses. We are wondering whether our response has addressed your concerns and do you have any more suggestions or questions. Moreover, if you find our response satisfactory, we kindly invite you to consider the possibility of improving your rating.

We are thankful to you for raising such important concerns and questions about our work, we highly appreciate your efforts during review process.

Response 2.1 Performance of [0, 30]m on Waymo dataset (Weakness 1)

We follow the experiment setting in prior zero-shot 3D detection work [1] to ensure consistency with other methods, to report the performance of [0, 30]m on Waymo open dataset in Table 2 of the main paper. Addtionally, results for full distances are presented in Table 1 of the main paper, Table 6 and Table 8 of Appendix. Please kindly refer to them.

Furthermore, we evaluate the performance of several methods, PV-RCNN, Voxel-RCNN, DetZero and SAM3D [1], across different distance ranges and levels of occlusion. The distance ranges include the following segments: [0, 30]m, [30, 50]m, [50, +inf)m. The occlusion levels are categorized into three grades based on the extent to which the object is obscured in the image perspective.

From the table, as the distance increases, the performance of all methods are decreased. Specially, VoxelRCNN decreases (L1 AP) with a ratio of 17.74% and 40.63% on [30, 50]m and [50, +inf)m compared [0, 30]m, PVRCNN decreases with 18.74% and 41.48%, our method decreases with 16.94% and 29.35%. The reason is that our method could utilize the entire temporal information in the point cloud sequence by our mask tracking module to generate the unique object ID. Therefore, we can overcome the influence of distance compared to other onboard methods, especially at farther ranges.

Reference: [1] Dingyuan Zhang et al. Sam3d: Zero-shot 3d object detection via segment anything model. arxiv preprint, 2023.

Response 2.2 Performance of occlusions. (Weakness 1)

We report the performance of the overall and the occlusion part on Waymo open dataset to compare with other methods. The occlusion levels are defined based on whether the objects are obscured in the image perspective, which are provided by WOD.

As we can see, VoxelRCNN and PVRCNN decrease with a ratio of 21.35% and 21.32%, SAM3D decreases with a ratio of 31.30%, our method decreases with a ratio of 11.02%. Our method could overcome the influence of occlusion by leveraging the temporal contexts with our mask tracking module.

Response 2.3 Failure pattern analysis (Weakness 1)

We have briefly summarized some representative challenging scenarios in Sec. 6 (Limitations and Broader Impacts) of the main contents of our submitted paper. Firstly, our method would fail to effectively recognize similar object categories (e.g., construction vehicle, truck, trailer) and some uncommon object categories (e.g., tree trunk, lane marker) with the foundation models (Grounding-DINO). Since this is the first stage of our entire method, it will result in subsequent stages lacking the output of corresponding perception results, such as 3D segmentation and occupancy prediction. Secondly, neural rendering methods may encounter numerous challenges in street-view scenes, constrained by practice factors (adverse weather conditions, sensor imaging issues), such as camera overexposure. In these scenarios, where it is impossible to generate geometrically plausible 3D reconstructions, our occupancy decoding will fail.

Please kindly refer to Fig. 2 of our global response PDF file to see the visualization.

Response 2.4 Lack of novelty (Weakness 2)

Please kindly refer to Response 3.2 for Reviewer Swqu.

We sincerely appreciate your valuable and helpful suggestions. We are wondering whether you have any more suggestions or questions after our response. Specifically, do you have any questions about the performance comparison based on distances and occlusions? Moreover, if you find our response satisfactory, we kindly invite you to consider the possibility of improving your rating.

We sincerely appreciate your positive acknowledgment of our work. We are pleased to provide the supplemental responses.

Response 1.1 Performance gap between our method and fully supervised methods (Weakness 1)

Yes. Indeed, our zero-shot method still exhibits a notable gap compared to fully supervised training methods when applied to datasets with abundant annotations. However, in scenarios where annotated data is scarce, our approach leverages autonomously generated perceptual outputs and can recognize objects whose classes were never labeled before. For instance, it can output 3D detection boxes for traffic signs and traffic lights, which are visualized in Fig. 1 of our global response PDF file.

Moreover, our research demonstrates the great potential of integrating foundation models into the field of autonomous driving. This integration is poised to advance traditional tasks substantially. Looking forward, as foundation models continue to enhance their performance, we believe they can be seamlessly integrated into our framework. Through continuous performance iteration and optimization, we anticipate further enhancing the effectiveness of our approach.

Response 1.2 Quantitative results of point completion module (Question 1)

We have supplemented the quantification results for the point cloud completion module. On our experimental set, there are around 4106 object samples with complete point clouds (obtained by merging all the point clouds of each object across the entire scene sequence). We then sampled the partial clouds and used them as input to generate 4096 points as a completion. The L1 Chamfer distance performance is summarized below. It shows the great effectiveness of our point completion module. As an additional reference of its effectiveness, please kindly refer to Table 8 of Appendix in our submission to see the performance differences in 3D box interpretation before and after point cloud completion.