Just Add $100 More: Augmenting Pseudo-LiDAR Point Cloud for Resolving Class-imbalance Problem

Thank you for your time and comments! Please see our response below.

---

[S1] Additional Paragraph for Related Work This is a great suggestion. We will add more paragraphs to the Related Work Section, summarizing the literature on data augmentation techniques.

---

[W1] Additional Comparison against Other Approaches This is a good question. We chose Real-Aug as our main competitor because it is the latest work and shows promising scores in the public nuScenes leaderboard. However, we fully agree with the reviewer that additional comparison with other augmentation techniques is needed. Thus, as shown in Table VII below, we conduct experiments to analyze the data augmentation effect with other recent approaches, including LiDAR-Aug [F], CA-Aug [G], and 3D-VField [H]. Our experiment confirms that all approaches help improve detection accuracy against the baseline. Notably, Real-Aug and (our proposed) PGT-Aug generally outperform the other approaches. We further provide qualitative analysis in Figure III in our rebuttal pdf file.

Table VII. Performance comparison with other data augmentation approaches.

Experiment Details We compare PGT-Aug with other 3D data augmentation methods on the public KITTI [8] Car benchmark. We create pseudo LiDAR point clouds for our method by randomly sampling ten cars from the CO3D dataset. The instances used are as follows: 106_12650_23736, 106_12662_23043, 157_17286_33548, 185_19982_37678, 194_20899_41094, 216_22796_47484, 216_22827_48422, 421_58405_112551, 206_21799_45886, and 421_58407_112553.

[F] Context-Aware Data Augmentation for LiDAR 3D Object Detection (ICIP 2023)
[G] LiDAR-Aug: A General Rendering-based Augmentation Framework for 3D Object Detection (CVPR 2021)
[H] 3D-VField: Adversarial augmentation of point clouds for domain generalization in 3d object detection (CVPR 2022)

---

[Q1] More Ablation Studies We conduct the following additional experiments: 1) without luminance generation (constant intensity), 2) instance generation according to the change in the number of images, and 3) without object level data alignment (random sampling, without rigid motion model). On the second row of Table VIII, the removal of luminance (intensity) generation in pseudo LiDAR leads to performance drop in downstream detection tasks. Also, as the percentage of multiview images used during 3D reconstruction decreases, the overall detection performance decreases accordingly. Finally, we conduct experiments on general alignment by replacing object-alignment with random sampling points from dense RGB point clouds and removing motion models. Both experiments show suboptimal performance compared to PGT-Aug.

Table VIII. Ablations in intensity, the number of images, data alignment

Thank you for your time and comments! Please see our response below.

---

[W1, Q1] Comparison between other methods that aim for class imbalance problems As the reviewer pointed out, there are other comparative methods dealing with class imbalance using loss-based methods [D, E] without adding data. We attach the comparison results with [D, E] in Table V. To match their baselines, we conducted experiments on PointPillar model. Additionally, we re-implemented Class-Balanced Loss [D] with beta 0.999 and resampled the number of objects. While we find out that loss-balancing methods such as Dynamic Weight Average [E] and CB Loss [D] are effective in enhancing minority-class performance, PGT-Aug, a data-augmentation based method, brings the largest performance gain against other approaches. Also, [E] experimentally proves that its GT sampling-based data augmentation was more effective than loss-based method in improving detection performance.

Table V. Comparison between other methods that aim for class imbalance problems

 | 81.0  | 72.3 | 50.2 | 50.1 | 49.0  | 63.4 | 10.7 | 34.3 | 32.9 | 6.9  | 44.6 |

| PGT-Aug | 83.0 | 71.8 | 54.8 | 51.1 | 54.9 | 69.7 | 20.2 | 39.5 | 49.6 | 14.5 | 50.9 | | PGT Aug + CBLoss | 82.7 | 74.7 | 56.5 | 55.9 | 54.4 | 68.7 | 20.9 | 34.1 | 53.5 | 20.5 | 52.2 |

[D] Class-Balanced Loss Based on Effective Number of Samples (CVPR 2019)
[E] Resolving Class Imbalance for LiDAR-based Object Detector by Dynamic Weight Average and Contextual Ground Truth Sampling (WACV 2023)

---

[W2, Q1] The motivation of this paper In 3D object detection, there have been many studies aimed at addressing class imbalance problems by modifying model structures (CBGS) or adjusting loss function (DWA). The most widely used method is the over-sampling known as GT-Aug, which involves inserting objects from other frames into the current frame. However, this method is limited to inserting objects from a predefined pool (training set), which restricts learning intra-class diversity, and generating or collecting 3D data for various objects has been extremely challenging and expensive. Recent advancements in differentiable 3D reconstruction techniques, such as NeRF and Gaussian Splatting, have made it possible to reconstruct dense 3D points at a lower cost. By leveraging these 3D reconstruction techniques, we proposed a novel and practical pipeline that overcomes the limitations of traditional insertion methods, particularly in terms of cost and diversity.

---

[W3] The detail of the framework Due to page limits, we have elaborated the details of our Intensity Domain Alignment network in line 559 of supplementary material. In summary, we adopt CycleGAN framework to learn unpaired translation between RGB and intensity. We designed generators and discriminators with PointNeXt encoders. It receives nuScenes long-tail class samples as real data, and we create fake data by translating, rotating, resizing, and projecting long-tail class RGB samples to the same (x,y,z,l,w,h, theta) of real data. Our generator is trained to generate fake intensity values from RGB features, while discriminator is trained to discriminate between real and fake intensity.

---

[W4] The effect of PGT-Aug on the majority of classes Our primary objective is to generate and insert the minority classes instead of the majority classes to relieve the class imbalance issue. Thus, we anticipated that the performance of the minority would improve while the performance of the majority classes would either remain stable or improve slightly due to the synergy effect of addressing the imbalance. To verify the effectiveness of the pipeline to majority classes, we conducted experiments on KITTI dataset as shown in Table VI. We reconstructed 10 cars of 3D RGB point clouds from CO3D dataset [I], (See Figure III in Rebuttal-Supp.) and created about 16,000 pseudo LiDARs of Car class. We apply pseudo LiDARs along with GT LiDARs during augmentation, and our PGT-Aug performance largely outperforms GT-Aug, Real-Aug, and other augmentation methods in car detection benchmark. Due to writing limit, please refer to Experiment Details (jNRg).

Table VI. The effect of PGT-Aug on the majority class

[F] Context-Aware Data Augmentation for LiDAR 3D Object Detection (ICIP 2023)
[G] LiDAR-Aug: A General Rendering-based Augmentation Framework for 3D Object Detection (CVPR 2021)
[H] 3D-VField: Adversarial augmentation of point clouds for domain generalization in 3d object detection (CVPR 2022)
[I] Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction (ICCV 2021)

---

[Q2] Memory usage and inference time If Object-level Domain Alignment is performed during training of detection, it may take additional time compared to Real-Aug. However, in 3D object detection methods, objects to be inserted were stored before training, and objects were loaded and inserted during training. Loading these objects from disk to memory in order to insert them into the scene takes time, and the memory and time complexity for the batch is O(n). Therefore, the time cost is the same as Real-Aug. Even though real-time performance was not a major consideration, we will add this discussion to the paper according to the valuable comments.

Thank you for your time and comments! Please see our response below.

---

[W1] Code Documentations Our apologies. We will revise the current documentation and make them easy-to-follow for researchers to use and understand our code easily. Plus, we will add a Jupyter Notebook-based tutorial to guide (i) how to reproduce our model, (ii) how to use our code, and (iii) how to visualize figures (shown in the paper).

---

[Q1] Data Collection from Videos with Dynamic Objects This is a good question. Our method relies on SfM to optimize camera poses for 3D reconstruction. However, such SfM would not triangulate well with images containing dynamic objects. This would be why we reconstruct the 3D shape of stationary objects, followed by using a rigid body motion model to represent dynamic objects. However, this must be an interesting direction worth exploring in our future work.

---

[Q2, Q3] View Consistent Representation This is a great question. We agree that the zero-degree spherical harmonic would have a similar effect to ours. However, we do not use it because (1) we empirically observed that our approach is more robust at capturing the original object's color and areas of dark shades (see our Figure I in rebuttal supp). (2) We wanted our model more generally applicable to various generative models, which may and may not use spherical harmonics. Additionally, (3) in terms of FID score, we observe our approach is generally better. As shown in Table IV, we compare ours with a variant model (with plenoxel's SH coefficient set to 0) to see the quality of generated pseudo LiDAR point clouds in terms of FID score. Our experiment further confirms our approach generates point clouds more similar to the actual LiDAR points. We will add discussion on this.

Table IV. FID score evaluation between SH coefficient 0 and ours

---

[W2, Q4] Differences to previous simulator works or generative models By leveraging current 3D reconstruction methods, our main goal is creating a novel and practical pipeline that overcomes the limitations of existing 3D object detections such as class imbalance problems, lack of annotations, in terms of cost and diversity. In other words, our pipeline is not limited to a specific generative model and can be applied to any model. The reasons we used explicit 3D representation models (3DGS or Plenoxel) in the paper instead of the models mentioned by the reviewer are as follows. According to [B] (see section Limitations and Failure Cases in supplementary material), Text-to-3D Generation (Shape-GPT, mesh-GPT, etc.) models tend to collapse modes when the target image distribution is overly concentrated in a single peak, resulting in abnormal 3D objects that are strongly related to specific views (Janus problem). Also, as shown in Figure II of Rebuttal-Supp., recent work of multi-view 3D reconstruction pipeline, DUSt3R [C] often fails to recover the details of miniature-scale objects. However, in order to place the restored objects in various positions, it is necessary to restore the entire shape of the object and create a bounding box. Therefore, we chose models that can reliably reconstruct the entire shape and perform robustly with relatively small objects.

[B] Taming Mode Collapse in Score Distillation for Text-to-3D Generation (CVPR 2024)
[C] DUSt3R: Geometric 3D Vision Made Easy (CVPR 2024)

---

[Q5] NuScenes Setting All detectors were tested under the same conditions, meaning no further information about our generated objects was used during the evaluation process, i.e., our model only uses the generated objects during training.

Thank you for your time and comments! Please see our response below.

---

[W1] Data Size Our dataset is created to provide pseudo-LiDAR point clouds of minority-class objects, which can be augmented into typical driving datasets (e.g., nuScenes, KITTI, and Lyft) to compensate for the class imbalance problem. Thus, the volume of our dataset should be smaller than that of these datasets but (we believe) sufficient to compensate for the class imbalance problem, as we reported in our experiments. Moreover, we would emphasize that (i) we provide an automated pipeline to generate such pseudo-LiDAR point clouds, enabling the community to use it to produce continuously growing datasets with various objects. (ii) We can generate view-dependent pseudo-LiDAR point clouds that can flexibly be placed anywhere in the scene, significantly improving data efficiency. We will clarify this in the final version of this paper.

---

[W2-1] Efficiency and Real-time Applicability of Pre-trained Models This is a good question. We would emphasize that (1) pseudo LiDAR point clouds are generated and stored offline, and (2) these generated point clouds are loaded (from memory) and augmented during the training of 3D object detectors. Thus, the need to run pre-trained models (e.g., unpaired domain transfer model) in real-time would be less significant. Further, in the following Table I and II, we analyze the processing time of each model and memory usage for generating pseudo LiDAR point clouds of different classes, i.e., construction vehicles, trucks, trailers, etc. This confirms point clouds can efficiently be generated through our pipeline, taking less than 300ms in total. We will discuss this in detail in the final version of this paper.

Table I. Average Processing Time (per instance, in msec)

Table II. Average Memory Usage (MB)

---

[W2-2] Ablation Studies with Pre-trained Models To demonstrate the impact of pre-trained models (e.g., unpaired domain transfer model and rigid body motion model), we further conduct ablation studies as follows: (1) We compare ours with a variant model without luminance generation (i.e., using constant intensity to see the impact of the unpaired domain transfer model). (2) We compare ours with a variant model without our rigid body motion model. In Table III below, we observe both experiments confirm the impacts of using these pre-trained models, showing a degradation without these models. We will discuss this more thoroughly.

Table III. Ablations in luminance generation and motion model

---

[W3] Data/content permission issue This is a good comment. We also take this copyright issue seriously. First of all, we will not download and re-release those video sources. Instead, we will release a list of YouTube links. Further, as the reviewer suggests, we have contacted the copyright holders to obtain permission to share their video links publicly. We will make sure to resolve this copyright issue upon publication.

---

[W4] Minor Comment We agree with the reviewer. We will move "and r is 0.1" to L190.

---

[Q1] Map information nuScenes dataset provides the BEV map, annotating commonly-observed map features such as road segments, lanes, crosswalks, walkways, stop lines, and parking lots. For all scenes, we generate an ego-vehicle-centered map in a top-down coordinate system, and over 34k maps are generated. Note that a range of 102.4m x 102.4m is set for generating a map where ego-vehicle is centered (effective forward sensing range of ego-vehicle is 51.2m). KITTI and Lyft datasets do not provide such map information (as mentioned in L291). Thus, we utilize a LiDAR-based ground segmentation method called Patchworks++ [A] to estimate ground. We will further clarify this in the final version of this paper.

[A] Patchwork++: Fast and robust ground segmentation method for 3D LiDAR scans (IROS 2022)

I appreciate the rebuttal and additional experiments. I have read the comments from other reviewers as well as the corresponding rebuttal. The rebuttal has largely addressed my concerns. In particular, I am very satisfied with the experiments and discussion related to [W1] and [W2-1].

Good work! Therefore, I have raised my rating to weak accept. I really look forward to seeing the revised version in the near future.