We thank the valuable comments and insightful suggestions, and we hope our detailed responses below can address your concerns.

Weakness 1: Performance Improvement

We would clarify that the proposed MonoMAE achieves clear performance improvement over the state-of-the-art. As shown in Table 1 of the submitted manuscript, MonoMAE outperforms the SOTA method MonoCD [1] by clear margins, especially for objects of Moderate and Hard categories. The scale of the improvement is also competitive as compared with several recently published works as listed in Table 1.

The experiments in Table 6 aim to validate the generalization ability of the proposed MonoMAE. We can observe that MonoMAE achieves the best or the second-best performance consistently across all metrics. Besides, the MAE metric used in Table 6 is hard to obtain significant increases. For example, MonoUNI [2] obtains slight performance gains and even performance drops on the nuScenes dataset.

[1] Yan, Longfei, et al. MonoCD: Monocular 3D Object Detection with Complementary Depths. CVPR, 2024.

[2] Jinrang, Jia, et al. MonoUNI: A unified vehicle and infrastructure-side monocular 3d object detection network with sufficient depth clues. NeurIPS, 2023.

Weakness 2 & Question 1: Experiments on Waymo

Thanks for your suggestion. We extend the experiments by examining the generalization over a new task KITTI3D->Waymo as shown in the table below, using $AP(IoU=0.5)$ as the metric. We can observe that our method has generalization ability on the Waymo dataset, and even outperforms some methods trained on Waymo.

* denotes this method is trained on Waymo.

[1] Ma, Xinzhu, et al. Rethinking pseudo-lidar representation. ECCV, 2020.

[2] Brazil, Garrick, et al. M3d-rpn: Monocular 3d region proposal network for object detection. ICCV, 2019.

Question 2: Performance Drop at Far Distances on the nuScenes dataset

The performance drop beyond 40 meters could be due to:

1. Limited visual information (in image resolution, etc.) is captured for objects beyond 40 meters;
2. Adverse conditions in lighting and bad weather could have a larger impact on the quality of distant objects;
3. The annotation accuracy could degrade for distant objects due to small object sizes, leading to degraded network training.

As shown in Table 6 of the submitted manuscript, existing SOTA methods all have performance drop in the nuScenes dataset at distances beyond 40 meters. The proposed MonoMAE achieves the second-best performance at a far distance.

Question 3: The Role of the Completion Network

The Completion Network aims to learn to complete occluded query objects for detection performance for various occluded objects. It is a lightweight network that introduces 2ms additional inference time only but can achieve effective completion in the feature space. During training, it learns to reconstruct and complete the masked object queries (simulating real object occlusions). During inference, it reconstructs the occluded queries as completed ones which clearly improves the performance of monocular 3D detection as shown in Table 2.

We managed to show the impact of the completion network by plotting the losses with and without using the completion network in Figure 7 of the submitted Appendix. The graph shows that the training loss with the Completion Network (the orange line) drops while the training loss without the Completion Network (the blue line) remains at a high level, showing that the Completion Network helps acquire occlusion-tolerant representations by learning to reconstruct the masked queries.

Question 4: Sensitivity of Our Method to the Mask Ratio

As defined in Equation (3) of the submission, the mask ratio $r$ of each query is determined by $r = 1.0 - \frac{d_{i}}{D_{max}}$. $r$ is the applied mask ratio for each query. $d_{i}$ is the predicted depth of the $i$-th query. $D_{max}$ is the maximum depth in the dataset that could manually adjusted to affect the mask ratio $r$.

The table below shows the performance of our method when changing the maximum depth $D_{max}$. In each cell of the table, the performance is listed as $AP_{3D}(IoU=0.7)$ / $AP_{BEV}(IoU=0.7)$.

We can observe that the best performance is achieved when the maximum depth is set at the original $D_{max}$. When the maximum depth is smaller than $D_{max}$, the performance drops. This is because if the depth of an object is too large, the query of this object will not be masked, leading to masking failures and hindering the training of the completion network.

When we set the maximum depth larger than $D_{max}$, the performance also drops. This is because the mask ratios are relatively high with the large maximum depth for all objects according to Equation (3). A high mask ratio will pose challenges for the Completion Network by reducing the available information since objects at far normally have limited pixels, hindering the reconstruction of occluded object regions.

Nevertheless, we can observe that setting the maximum depth between $0.75$ and $2$ $*D_{max}$ achieves similar 3D detection, indicating the robustness of our method with respect to this parameter.

We thank the valuable comments and insightful suggestions, and we hope our detailed responses below can address your concerns.

Weakness 1: Occlusion Levels: Definition and Usage

We appreciate the reviewers' insightful comments regarding the use of occlusion levels. Our method only uses the binary "occluded" and "non-occluded" labels. During training (Figure 2 of the submitted manuscript), the binary occlusion labels are used to train the Non-Occluded Query Grouping module for classifying the non-occluded queries that are masked and completed. During inference (Figure 5 of the submitted manuscript), the Non-Occluded Query Grouping module is used to classify the queries.

To this end, we only need to obtain the binary occlusion labels indicating whether the objects are occluded or not, without using the fine-grained occlusion labels which could make the task complicated. In this paper, we transform the ground truths of occlusion degrees $o$ provided in the KITTI 3D dataset, where $o \in$ {0, 1, 2, 3}, into binary ground truths. The transformation simplifies $o=0$ as non-occlusion and $o \in$ {1, 2, 3} as occlusion, leading to the binary ground truths of occlusion conditions $o^{gt} \in$ {0, 1} with 0 indicating non-occlusion and 1 indicating occlusion.

Weakness 2 & Question 1: Accuracy and Impact of the Occlusion Classification Network

Thank you for your suggestion. The accuracy of the occlusion classification network has been provided in Section F.3 of the submitted Appendix. The accuracy of the occlusion classification network is 96.46%, indicating that most object queries can be correctly classified as occluded or non-occluded.

Moreover, to validate the influence of the classification accuracy on the overall 3D object detection network, we add experiments using a trained network with fixed weights. During each inference, the occlusion classification accuracy is manually adjusted to 50% and 70% with the help of ground-truth occlusion labels. The experimental results are shown in the table below, where the original accuracy (96.46%) of the occlusion classification network is also exhibited. The metrics $AP_{3D} (IoU=0.7)$ and $AP_{BEV}(IoU=0.7)$ with Easy, Moderate and Hard categories are used. In each cell of the table, the performance is listed as $AP_{3D}(IoU=0.7)$ / $AP_{BEV}(IoU=0.7)$.

From this table, we can observe that the higher occlusion classification accuracy can contribute to better 3D detection performance, validating the importance of classification accuracy on the overall 3D object detection network.

We will add these results and analyses in the revised version.

Weakness 3 & Question 2: Performance of the Query Completion Network

Thank you for your suggestion! The performance of the Completion Network is measured through the similarity between the non-occluded queries before masking and the queries completed by the Completion Network. This similarity is defined by the Smooth L1 loss, as defined by Equation (6) of the submitted manuscript.

With the help of the Smooth L1 loss, we visualize the training losses with and without using the Completion Network in Figure 7 of the submitted Appendix. As Figure 7 shows, the training loss using the Completion Network (the orange line) drops while the training loss without using the Completion Network (the blue line) remains at a high level, demonstrating the effectiveness of the Completion Network in acquiring occlusion-tolerant representations effectively by learning to reconstruct the masked queries.

The visualization of masked and reconstructed queries is not provided since the queries are one-dimensional vectors and cannot be visualized in a meaningful approach.

Moreover, Table 2 in the submitted manuscript can further validate the effectiveness of the Completion Network quantitatively. Comparing Rows 2 and 6, and Rows 4 and 7, using Completion Network can improve the 3D detection performance effectively, showing that the masked queries are properly reconstructed.

We thank the valuable comments and insightful suggestions, and we hope our detailed responses below can address your concerns.

Weakness 1: Feature-Level Masking vs. Image-Level Masking

We would clarify that the strategy of masking and completion aims to generate pairs of non-occluded and occluded (via masking) data for learning occlusion-tolerant object representations. The strategy could be implemented in either image space or feature space. We chose to mask and reconstruct query features, as masking and reconstructing images is complicated and computationally intensive due to noisy image data and super-rich occlusion patterns. As shown in Table 3 of the submitted manuscript, masking and completing query features perform clearly better than masking and completing images. Nevertheless, we understand that the simulated occlusion features are different from that of natural occlusions (as briefly discussed In the Limitations section), and shared that this issue could be mitigated by introducing generative networks to learn the distribution of natural occlusions.

Weakness 2: Depth-Aware Masking v.s. Random Masking

We would clarify that the proposed depth-aware masking is implemented in the feature space. We examined how different masking strategies affect the monocular 3D detection by testing three masking strategies: 1) random masking in the image space; 2) random masking in the feature space; and 3) depth-aware masking in the feature space, as shown in Rows 1-3 in Table 3 of the submitted manuscript. We can observe that the depth-aware masking performs clearly the best. As discussed in Section A.1 of the submitted Appendix, the depth-aware masking modulates the mask ratio by lowering it for distant objects. This effectively retains more visual information for distant objects which are usually small and have limited pixels and visual information. In addition, it also facilitates the ensuing completion task as completing heavily masked small objects is challenging and liable to failures.

Weakness 3: Non-Occluded Query Grouping

Thanks for your valuable suggestion! Below is the detailed performance of the Non-Occluded Query Grouping classification module with your suggested metrics.

To evaluate the performance when applying completion on all queries, both occluded and non-occluded, at the test time, we conduct experiments and show the results in the table below. The metrics $AP_{3D} (IoU=0.7)$ and $AP_{BEV}(IoU=0.7)$ with Easy, Moderate and Hard categories are used. In each cell of the table, the performance is listed as $AP_{3D}(IoU=0.7)$ / $AP_{BEV}(IoU=0.7)$.

The results indicate a performance drop when applying completion to all queries, compared to completing the occluded queries classified by the occlusion classification network. This is because the Completion Network is designed to reconstruct the occluded queries to learn occlusion-tolerant visual representations. Consequently, applying completion to non-occluded queries introduces confusion, degrading overall 3D detection performance.

We thank the valuable comments and insightful suggestions, and we hope our detailed responses below can address your concerns.

Weakness 1: Depth-Aware Masking for Occlusion Simulation

We acknowledge that the proposed depth-aware masking may not perfectly replicate natural occlusion patterns, which could affect the model's reconstruction accuracy potentially.

To fill the gap between synthetic and natural occlusions, we have considered alternative methods, such as directly overlaying images of objects (e.g., using a car image to occlude another car). However, this approach poses significant challenges, particularly in obtaining accurate training labels (3D boxes, orientations) for the overlaying objects, making it impractical for our current implementation.

As shown in Section D.1 of the submitted Appendix, one possible solution is introducing Generative Adversarial Networks (GANs) that learn distributions from extensive real-world data for generating occlusion patterns that are very similar to natural occlusions. Specifically, a Generator could be employed to generate occluded queries based on non-occluded queries, while a Discriminator could be used to discriminate between these generated occluded queries and natural occluded queries. Through adversarial training, the Generator could produce occlusions that are very similar to natural occlusions.

Weakness 2: Lacking Extensive Cross-Dataset Validation

We would clarify that most existing monocular 3d detection methods [1, 2, 3] validate their generalization ability on the task KITTI3D->nuScenes only. We followed these prior studies to facilitate benchmarking. As suggested, we extend the experiments by examining the generalization over a new task KITTI3D->Waymo as shown in the table below, using $AP(IoU=0.5)$ as the metric. We can observe that our method has generalization ability on the Waymo dataset, and even outperforms some methods trained on Waymo.

* denotes this method is trained on Waymo.

[1] Shi, Xuepeng, et al. Geometry-based distance decomposition for monocular 3d object detection. ICCV, 2021.

[2] Kumar, Abhinav, et al. Deviant: Depth equivariant network for monocular 3d object detection. ECCV, 2022.

[3] Jinrang, Jia, et al. MonoUNI: A unified vehicle and infrastructure-side monocular 3d object detection network with sufficient depth clues. NeurIPS, 2023.

[4] Ma, Xinzhu, et al. Rethinking pseudo-lidar representation. ECCV, 2020.

[5] Brazil, Garrick, et al. M3d-rpn: Monocular 3d region proposal network for object detection. ICCV, 2019.

Question 1 & Limitation 2: Performance on Small Size Objects (Pedestrian and Cyclist)

The proposed MonoMAE can handle challenging cases with competitive detection performance. We conducted new experiments on the suggested pedestrians and cyclists that often have smaller scales and are more challenging to detect. The two tables below show the experimental results with metric $AP_{3D}(IoU=0.7)$.

Performance for pedestrian detection.

Performance for cyclist detection.

We can observe from the above two tables that the proposed MonoMAE performs well for the pedestrian and cyclist categories for monocular 3D object detection.

Question 2: Generative Approaches for Simulating Natural Occlusion Patterns

As briefly shared in Section D.1 of the submitted Appendix, the proposed MonoMAE could be improved by employing generative networks such as GANs to learn distributions of real-world data. The trained model will then generate occlusion patterns that are more similar to natural occlusions than our proposed feature masking, leading to better monocular 3D object detection. Take GAN as an example. The GAN generator will learn to generate occluded queries (with many non-occluded queries as reference), while the GAN discriminator will learn to discriminate between the generated occluded queries and naturally occluded queries. Through adversarial learning, the trained generator could generate more realistic occlusions, which further leads to better occlusion completion and monocular 3D object detection.

Limitation 1: Computational Efficiency

We would clarify that we have provided the inference time of the proposed MonoMAE and several state-of-the-art monocular 3D detection methods in Table 5 of the submitted manuscript. According to the inference time, MonoMAE can achieve above 26.3 FPS (frame per second) which demonstrates its great potential for real-time tasks. For resource usage, all our experiments were conducted on a computer with Intel Xeon Gold 6134 CPUs, CentOS operating system with 128GB RAM, and a single NVIDIA V100 GPU with 32 GB memory. We will highlight the above information in the updated manuscript.