Object-level Data Augmentation for Visual 3D Object Detection in Autonomous Driving

We are grateful for the thorough review you provided for our manuscript, as well as for your positive feedback on our research efforts. We address the ambiguities in our approach and incorporate additional experiments to respond to your queries.

Q1. How to guarantee the edited object be located in reasonable regions?

We design a set of detailed rules for batch object editing, which could ensure that the edited objects appear in reasonable positions:

• Adding objects:
  • In driving scenarios, objects are mostly located on the ground plane. Therefore, we tend to place the added object on the ground. To achieve this, we approximate the ground plane function by analyzing scene data, including point cloud and object bounding boxes.
  • We randomly select a point within a certain range on the ground plane, place the object at that position, and rotate the object by a random angle parallel to the ground plane.
  • Subsequently, we perform collision detection between the added object's bounding box and other objects or the background scene. If there is a physical overlap with other objects, we will attempt another placement.
• Moving objects:
  • When moving static objects, we randomly choose a direction parallel to the ground plane and move the object a certain distance.
  • When moving dynamic objects, because of tricky issues such as velocity and direction, we employ a simple strategy: For each edit, we randomly select a number x. When handling a certain moving object at time t, we move it to its position at time t-x.
  • After an object is moved, we conduct collision calculations. If there is a physical overlap, the object undergoes re-editing.
• Deleting objects:
  • We randomly select an object in the scene for deletion. Most of the time the deletion operation does not result in the violation of real-world physical laws.

Finally, we perform manual inspections of the edited scenes to eliminate implausible elements.

This part will be included in the final version of the paper.

Q2. How to control the appearance style of the edited object? Is there a way to generate an unseen appearance from the training data set?

The proposed augmentation method is based on the reconstruction of scenes and objects. Consequently, their appearance style is constrained to those present in the training dataset. The proposed method cannot generate object appearances with an unseen style.

Recently, we have come across some excellent works about synthesizing new appearance styles of objects. It is an interesting and potential idea to train a unified generator for the objects in the same class, which could learn their common features and control the generated appearance style. In future work, we will explore this idea.

Q3. Comparison of the training results within the original image and the synthesis data.

We design an experiment to discuss this issue. Considering the limited resources, we use 20% of the nuScenes data (a total of 140 scenes) as a mini-dataset for training. Additionally, we select 28 scenes from the mini-dataset and process them using different synthesis methods, which are then added to the mini-dataset. The synthesis methods we employed are as follows:

• Rerender: We rerender the original images by employing the same reconstruct & render method of the proposed method, without any camera setting and scene layout changes.
• Original: We make no modifications to the original images and add them directly into the mini dataset. This method serves as a baseline.

We employ these different mini datasets to train PETRv2, and the detailed training settings align with those outlined in Section 4.3 of the original paper. The detailed performance results are as follows.

[*] Note: We randomly sample a certain number of generated images, compare them to the original images, and calculate this PSNR value.

While there is indeed a gap in the quality between synthetic and original data, experimental results indicate that these differences do not significantly impact the model's detection performance. This further indicates that the proposed data generation method is applicable to downstream tasks.

We sincerely appreciate your feedback and suggestions. We are greatly encouraged that you found our method is 'well-motivated' and 'effective'. We conduct extensive studies and added experimental evaluations to comprehensively address your concerns.

Q1. Evaluation on a smaller dataset.

Thank you for your suggestions. We conduct validation on a smaller dataset.

• We have successfully implemented the scene reconstruction method on the KITTI dataset. Visualization can be found at anonymous URL https://anonymous.4open.science/r/kittivisualization/kittivis.png

• Unfortunately, due to inherent limitations in the KITTI dataset, further editing of the reconstructed scenes is not feasible.

  The proposed augmentation method requires the dataset to provide two types of labels: the driving trajectory of the ego and the bounding boxes of objects. Regrettably, sequences in the KITTI dataset often do not concurrently possess both of these labels.

  Specifically, in the KITTI Visual Odometry dataset, a subset of sequences from the original KITTI raw dataset is annotated with vehicle trajectories, while another subset in the KITTI detection dataset is annotated with object bounding boxes. Notably, there is no overlap between the sets of sequences annotated for these two types of labels.

Therefore, to facilitate validation on a smaller dataset, we adopt an alternative approach:

• We randomly sampled 20% of the nuScenes training set data to construct a small-scale dataset (including 140 scenes).
• Within this small-scale dataset, we further selected 20% of the data for augmentation (including 28 scenes) using the proposed augmentation method.
• Lastly, we employed the PETRv2 model (training settings consistent with those outlined in Section 4.3 of the paper) to train on both the augmented and non-augmented datasets, subsequently conducting tests on the validation set. Following is the performance data:

It can be observed that our method exhibits consistent performance in the small-scale dataset.

Q2. Data generation cost.

Please note that our data synthesis process is offline. We generate the augmented data and store them before the detection model training. During the training process, we directly use the stored synthetic data. Therefore, the proposed augmentation process does not affect the training efficiency of the detection model.

We also provide the detailed computation costs of generating the new training samples as below:

• In the reconstruction stage：

  • Time cost: Our descriptors and render are trained on 4 RTX 3080Ti (12G) GPUS, which takes about 5 hours for a scene with about 240 images (with a resolution of 1600x900).
  • Memory cost: For a single scene, trained point-cloud descriptors take about 50MB~200MB of storage, depending on the size of the point cloud. The trained UNet takes 116MB as the renderer.

• In the rendering stage, the trained renderer takes about 20 seconds to render a single image (with a resolution of 1600x900) from the point map on a single RTX 3080Ti (12G) GPU.

Q3. Results in nuScenes test set.

We conduct testing on the test set, and submit the results to the nuScenes detection challenge, obtaining the following performance metrics:

The results of the test set further demonstrate that our method has brought significant performance improvements to the baseline.

*Note： The best device we can acquire is RTX 3080Ti (12G), which is hard to support large image resolution and strong backbones. Therefore, we use a training setting consistent with the paper's Section 4.3: a lightweight backbone and lower-resolution inputs. Additionally, we only utilize the training set for training and augmentation. As a result, the data in the table differs significantly from models at the forefront of the challenge leaderboard, which employ heavy backbones, high-resolution image inputs, and train on the entire dataset (train+val).

Q4. Discussion and comparison with ROI-10D.

RoI-10D focuses on monocular 3D detection, which also provides certain methods to synthesize object-level edited images. In comparison to our method, the differences are outlined as follows:

• ROI-10D relies on CAD vehicle models to extract the vehicle meshes. Consequently, the range of augmented object categories and shapes is restricted. Our method, however, allows for the editing of all target objects present in the dataset, from small traffic cones to huge trailers.
• ROI-10D employs a single-frame image as input to reconstruct vehicles into 3D textured meshes. In contrast, our method utilizes the images and point cloud sequence across multiple frames, enabling the learning of more precise and detailed object shapes and texture information.
• ROI-10D's object editing methods are limited. For example, it cannot handle the removal of existing objects in the scene. Our method supports more flexible editing operations, including free rotation, movement, and deletion.

Regarding quantitative performance comparison, RoI-10D does not provide open-source code. Given the limited time window for the rebuttal phase, we regret that we are unable to reproduce it and give a quantitative assessment. We will complete the relevant experiments and present the results in the final version of the paper.

We hope these supplementary experiments and discussions can address your concerns. Please let us know if there are any additional clarifications or experiments that we can offer. We would love to discuss more if any concern still remains.

We express our gratitude for your valuable feedback and suggestions, which serve as a source of great encouragement to us. While there are still some concerns and questions, we are doing our best to address them.

Q1. Comparison with other methods.

Firstly, LiDAR-based augmentation methods are primarily employed for LiDAR 3D perception. Augmentation in the LiDAR point cloud is hard to synchronize to the image level. Our approach utilizes LiDAR as an intermediary to achieve augmentation of the image. In essence, our method serves as a bridge between LiDAR-based and image augmentation.

Secondly, Tong et al. (2023) mentioned does not necessarily rely on LiDAR data. In this method, LiDAR serves as a supplementary depth information source for NeRF procession. As Tong et al. (2023) do not public their source code and disclose detailed editing settings, within the limited rebuttal time windows, we apologize that we cannot conduct a quantitative comparison of augmentation algorithm performance. We will complete the relevant experiments and present the results in the final version of the paper.

Q2. Data size comparison of original and augmented data.

The original nuScenes training dataset comprised 700 scenes. We randomly selected 20% of these scenes for augmentation. In the final training dataset, the ratio between original data and generated data was maintained at 5:1.

Q3. Augmented performance of different classes of objects.

We provide the detailed detection performance data of each object in nuScenes of PETRv2 in the following table.

We can observe significant improvements in detection data for small objects (e.g., traffic_cone and barrier with 2.40 and 1.40 mAP increase respectively), medium-sized objects (e.g., car with a 1.10 mAP increase), and larger objects (e.g., truck and construction_vehicle with 1.28 and 0.76 mAP increase respectively).

It is worth noting that although the mAP increases for most object categories, there is a slight decline in the performance of motorcycles, bicycles, and trailers. Upon further analysis, we attribute this decline to the poor reconstruction performance of point cloud-based methods for objects with holes, such as the wheels of bicycles. Additionally, we identify challenges in the reconstruction of trailers, given their large and complex structures with diverse goods carried. These lead to insufficient quality in the generated data, causing a slight decline in performance data. In subsequent work, we will address and follow up on resolving these related issues.

Finally, we express sincere appreciation for your constructive suggestions, which have contributed to the comprehensiveness of our work.