Learning 3D Perception from Others' Predictions

We thank your valuable feedback to improve our manuscript further. Below, we respond to most, if not all, of your comments at this stage.

Answer to Weakness 1: Reference agents with camera-based detector.

Thank you for pointing this out! We are indeed glad to hear our study has great potential to lead to several promising research directions. We want to humbly highlight that this is the first paper to explore such a scenario to our knowledge, and we hope that our findings and approach have set the stepping stone to future research in this direction, including reference agents with camera-based detectors.

While it will be interesting to study the case where reference agents are equipped with camera-based detectors, we find it not to be easy and straightforward to directly conduct the study. Firstly, the dataset that we have used, V2V4Real, doesn’t contain camera frames. Moreover, as the reviewer mentioned, a lack of depth information in camera-based detectors could potentially lead to unsatisfactory performance for reference agents, and the ego car’s detector could result in suboptimal performance if we leverage predictions from such relatively less accurate predictions. However, we also want to note that reference agents do not necessarily need to be general experts, but could be local experts (Sec 3.1. Feasibility and practicality; Lines 174 - 179) in the geo-fenced area, and it could lessen the level of difficulty for reference agents to be well-trained. Still, we believe this challenge could be an interesting research question, and we leave it for future study.

Finally, we humbly note that no matter what sensors and detectors (camera-based or LiDAR-based) are in use, if we aim at 3D perception, they will generate 3D bounding boxes (as pseudo labels). We believe that those pseudo labels will likely show inherent localization, synchronization, and calibration errors, and the fundamentals of findings and solutions in our study would still hold.

Answer to Weakness 2: More baseline training methods.

Thank you for your comment. Based on your feedback, we are currently conducting experiments regarding the existing semi-supervised method, but it’s a bit delayed due to resource limits. We will share our observations when we have the results.

Again, we appreciate your valuable opinions, which will significantly improve our study. Please also kindly give us a chance to clarify any remaining unclear points.

I appreciate the response from authors. I was waiting for the semi-supervised learning results, so I did not reply until the last day.

For the first concern, I can understand that it is the constraint of the dataset. However, I think my second concern is more important. Semi-supervised learning should be a crucial baseline for this paper.

Besides, I partially agree with Reviewer R3jy in the application of off-board autolabelling. From my perspective, offboard auto-labeling models, which can also generate pseudo-labels, should be more accessible than those reference detector results on the street from different organizations. People can use these offboard models to annotate data and then train other online models. I understand this paper works on a different scenario and acknowledge the novelty of this scenario, so I do not require the authors to beat those offboard autolabling baselines (I believe it is difficult). However, it to some extent still hurts the practical usefulness of this proposed method.

We thank your valuable feedback to improve our manuscript further. Below, we respond to most, if not all, of your comments at this stage.

Answer to Weakness 1: About agents with different modalities.

Thank you for pointing this out! We are indeed glad the reviewer found our study to have great potential to encourage several promising research directions. We would humbly like to highlight that this is the first paper to explore such a scenario to our knowledge, and we hope that our findings and approach have set the stepping stone to future research in this direction.

We would also like to note that while it would be interesting to study such extreme cases (e.g., drastically different object detectors), it would not be easy and straightforward at the same time. Firstly, the dataset that we have used, V2V4Real, doesn’t contain camera frames. Moreover, if we consider camera-based reference agents and LiDAR-based ego car, camera-based detectors would typically perform worse than LiDAR-based detectors because of a lack of depth information. If we leverage predictions from such relatively less accurate predictions, the ego car’s detector could result in suboptimal performance. While we have provided a discussion about lessening the level of difficulty for reference agents to be well-trained (e.g., leveraging the repetition or background cues; Lines 174 - 179), we still believe this challenge could be one of the interesting research questions to explore. The opposite case where LiDAR-based reference agents and camera-based ego car would also be promising, and we leave them for future study.

Finally, we humbly believe that no matter what sensors and detectors (camera-based or LiDAR-based) are in use, if we aim at 3D perception, they will generate 3D bounding boxes (as pseudo labels). We believe that those pseudo labels will likely show inherent localization, synchronization, and calibration errors, and the fundamentals of findings and solutions in our study would still hold.

Answer to Weakness 2 and Question: About fully unsupervised setting.

We have already investigated a fully unsupervised setting by training our ranker with only synthetic data (Table 3 (a) in the main paper). The result shows the potential for entirely replacing labeling effort if we are able to access simulation data (Lines 464 - 470). Still, at the same time, we are currently conducting experiments regarding the existing semi-supervised method in response to the feedback, but it’s a bit delayed due to resource limits. We will share our observations when we have the results.

Again, we appreciate your valuable opinions, which will significantly improve our study. Please also kindly give us a chance to clarify any remaining unclear points.

We thank your valuable feedback to improve our manuscript further. Below, we respond to most, if not all, of your comments at this stage.

Answer to Weakness 1: Clarification of our setting.

We apologize for any unclear writing that could potentially lead to a misunderstanding of the scenario we focus on. Our overall goal is to develop a 3D perception model that can be deployed in an online setting. A conventional development process can typically be decomposed into four stages.

• Stage 1: data collection (online)
• Stage 2: data annotation, often by humans (offline)
• Stage 3: model training and validation (offline)
• Stage 4: model deployment and evaluation (online)

Our paper exactly follows the four stages, except that in Stage 1, we assume that some nearby agents (e.g., robotaxi, roadside unit) share their predictions as pseudo labels (e.g., bounding boxes). Our main contribution is in how to leverage these pseudo labels to reduce human annotation in Stage 2 while maintaining the trained model quality in Stage 3. We will rewrite our problem setup to clarify this (Lines 157 - 161).

Here, the stage where the ego car collects the nearby agents’ predictions is the first stage, which is online. We note that there is no training or inference regarding the ego car’s detector during this stage. After we collect the pseudo labels from the nearby agents, we refine the noisy pseudo labels with our proposed method and train the ego car’s detector, which is offline (Stages 2 and 3). We humbly note that reducing the computational cost during these stages (e.g., resource-efficient training or inference) is not the main goal of this study. For our final model evaluation (Stage 4), which is online, the detector’s computational cost is exactly the same as the standard detector.

Q: It is possible that the method can learn knowledge from other vehicles in real-time, which would expand the scope of practical applications?

We are glad that the reviewer found our scenario has the potential to raise several interesting topics. However, we respectfully want to emphasize that this is the first study to explore a new research direction, and we didn’t consider such a setting in this study. However, we humbly believe learning from the nearby agents in an online manner would be a very promising and practical direction.

Answer to Weakness 2 and Question 2: Box Ranker vs. existing IoU scoring methods.

Thank you for pointing out. We agree that our box ranker’s goal is similar to IoU scoring methods adopted in the existing detectors, such as Voxel-RCNN and PV-RCNN. We respectfully think that the key difference is in its motivation and usage. In our study, we are motivated to build a ranker assuming we already have pseudo labels for objects but with a certain amount of localization errors. Therefore, the objective of our ranker is to find the best candidate among a bunch of candidate boxes near the initial noisy pseudo label so that the selected box can serve as better pseudo labels. In our study, we wanted to demonstrate that this objective can be easily achieved with only a few frames (or even without any frame if we are able to use simulation data; Table 3 (a) in the main paper) and without sophisticated architectural design. While we agree that our ranker could be replaced with the existing methods, we kept it to a very simple regressor that can already be satisfactorily used to demonstrate the effectiveness in refining the noisy pseudo labels. To this end, we choose to use a simple pointnet-based model. We will add the discussion to our manuscript.

Again, we appreciate your valuable opinions on improving our study. Please also kindly give us a chance to clarify any remaining unclear points.

Thank you for the detailed feedback. After carefully reading it, we respectfully think there might be a misunderstanding, and we apologize if it has been caused by our paper organization. Below, we respond to most, if not all, of your comments at this stage. We apologize in advance if the response is long.

Answer to Weakness 1: Clarification of our setting.

We appreciate that you brought up the offboard setting and two references. After carefully reading them, we respectfully think that our paper does not belong to an offboard setting, and we explain it as follows.

Our overall goal is to develop a 3D perception model that can be deployed in an online setting. A conventional development process can be decomposed into four stages typically.

• Stage 1: data collection (online)
• Stage 2: data annotation, often by humans (offline)
• Stage 3: model training and validation (offline)
• Stage 4: model deployment and evaluation - on a disjoint set of data (online)

Our paper exactly follows the four stages, except that in Stage 1, we assume that some nearby agents (e.g., robotaxi, roadside unit) share the predictions from their viewpoints as pseudo labels (e.g., bounding boxes). Our main contribution is in how to leverage these pseudo labels to reduce human annotation in Stage 2 while maintaining the trained model’s quality in Stage 3. We will rewrite our problem setup to clarify this (Lines 157 - 161).

Onboard vs. offboard.

Despite such a difference, our Stage 4 strictly follows an online (onboard) setting. That is, one cannot rely on future frames (like the next several seconds) to enhance the current prediction. Our main result in Table 2 of the main paper exactly follows this setting.

In our humble opinion, this is different from the offboard evaluation. If we understand the two papers correctly, their focus is on improving Stage 2. That is, suppose a portion of the training data is human-annotated, can we effectively use it to auto-label the remaining data? Since this Stage is offline, one can leverage future frames to improve the labeling of the current frame. For example, one can use data from -2:2 seconds to predict the labels at 0 seconds. Their main (offboard) evaluation is on how well the auto-labeling matches human annotation.

Potential cause of confusion.

We think Table 1 of the main paper might create the wrong impression that we aim for an offboard setting. We put Table 1 in front of Table 2 mainly to highlight the challenge of leveraging nearby agents’ pseudo labels (i.e., initial boxes), but Table 1 is not our main result. We will clarify this in the final version.

Comparison to offboard setting.

That said, our paper and most of the label-efficient and domain adaptation papers (please see our related work) share a similar goal with the offboard setting: reducing human annotation effort (in Stage 2) for training the online perception model (in Stage 3).

In the two offboard papers, they aim to use a human-labeled training data portion to auto-label the remaining portion. In our setting, we leverage the pseudo labels offered by nearby agents (Stage 1). We humbly think that these two directions are not competitors, but complementary to each other: using different initial signals to achieve a common goal. One can potentially combine them for better performance.

We hope these explanations have resolved your concern. We will be more than happy to discuss further.

Currently, many label-efficient methods use low annotation cost to correct pseudo-labels or to mine pseudo-labels. The authors should also discuss these methods:

[1] Zhang D, Liang D, Zou Z, et al. A simple vision transformer for weakly semi-supervised 3d object detection[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 8373-8383.

[2] Liu C, Qian X, Huang B, et al. Multimodal transformer for automatic 3d annotation and object detection[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 657-673.

[3] Xia Q, Deng J, Wen C, et al. Coin: Contrastive instance feature mining for outdoor 3d object detection with very limited annotations[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 6254-6263.

[4] Yang Y, Fan L, Zhang Z. Mixsup: Mixed-grained supervision for label-efficient lidar-based 3d object detection[J]. arXiv preprint arXiv:2401.16305, 2024.

[5] Xia Q, Ye W, Wu H, et al. HINTED: Hard Instance Enhanced Detector with Mixed-Density Feature Fusion for Sparsely-Supervised 3D Object Detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 15321-15330.

[6] Liu C, Gao C, Liu F, et al. Ss3d: Sparsely-supervised 3d object detection from point cloud[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 8428-8437.

In contrast to unsupervised methods, these approaches exhibit greater comparative value.

Thank you so much for reading our long rebuttal and providing us with a timely response. If we understand correctly, our rebuttal has addressed many of your concerns that resulted from a different understanding of our paper. In the following, we would like to provide our response to each of your new concerns.

“The offboard methods and the method in this paper are both based on a well-trained pre-trained model, …, I still believe that the offboard methods are better baselines.”

“ … so compared to the unsupervised baseline selected in Table 2, … A more equitable experimental comparison is necessary …”

Thank you for your further comment, which allows us to better address your concern.

We agree that on the surface, both the offboard papers and our paper assume the existence of a pre-trained model. However, the accessibility to the model is drastically different, creating two distinct settings that are not directly comparable.

More specifically, suppose the overall goal is to develop an accurate onboard detector for the ego car. In the offboard papers, the pre-trained offboard model is directly accessible. One can use it to label the unlabeled data collected by the ego car ($X_E$ in Line 159). The resulting pseudo-labeled data can then be used to train the final onboard model.

However, in our paper, we do not have direct access to the pre-trained model, as it is deployed on the nearby agent (thus denoted by $f_R$ in Line 158), not the ego car. As such, we cannot use it to label the unlabeled data collected by the ego car. What we can access are the nearby agent’s predictions on the data it collects ($X_R$ in Line 159), and we attempt to use them as pseudo labels of $X_E$ to train the ego car’s onboard model. (Please see Lines 46 - 51 and Lines 176 - 191 for the feasibility and practicality of our setting.)

We note that the predictions on the nearby agent’s data ($f_R(X_R)$ in Line 161) are quite different from the predictions on the ego car’s data ($f_R(X_E)$), even if we represent them in the global coordinate. This is evidenced in Table 1, where the 5th-row “sharing detector” means one can access the pre-trained model to label the ego car’s data directly; the 1st-row means one can only access $f_R(X_R)$. The huge gap in pseudo label quality between the two rows (22.7% on Recall and 38.8% on Precision) highlights the core problem we aim to address, which results from mislocalization and viewpoint mismatch (please see Lines 205 - 215).

We also want to note that the unsupervised baseline in Table 2 can also receive $f_R(X_R)$ like ours. In other words, they have the equitable, same “limited” access to the pre-trained detector deployed on the nearby agent.

In sum, we appreciate the reviewer’s comment. However, we respectfully think the settings in the offboard papers and ours are quite different and not directly comparable. As such, we respectfully disagree that the offboard papers are better baselines: our setting does not allow direct access to the pre-trained model, and nearby agents cannot predict labels in an offboard fashion (as they need to make online predictions to drive safely). In our humble opinion, it makes little sense to claim which setting is “better,” as they stand for different scenarios where one can obtain supervision. In particular, our paper investigates a new problem — can we leverage nearby agents’ predictions (on the data they perceive) to train an accurate model for the ego agent?

Thank you for the valuable feedback and we apologize for any unclear explanation that has led to misunderstanding. Below, we respond to most, if not all, of your comments at this stage.

Answer to weakness 1: Practicality and applicability of the problem set.

Thank you for your detailed comment. In our current version, we have made some effort to justify the practicality and applicability. Please kindly refer to Lines 46 - 51 and Lines 174 - 191. We hope these text portions address your concerns to a certain extent. In the following, we provide specific responses.

Q. First, the approach relies heavily on the assumption that neighboring cars with accurate detectors are always available when needed, but this may not always be feasible.

We note that we rely on nearby agents’ predictions only during the data collection phase. We provide a decomposition of the model development process in response to Reviewer R3jy (Answer to Weakness 1. Stage 1 - stage 4), in which the data collection is in Stage 1. In other words, as long as nearby cars were present during part of the data collection, we can already leverage their predictions to train the model; the other portion of the data can be leveraged in a self-training fashion (cf. Step 2 in Section 3.5).

We also want to note that nearby cars are not the only sources of pseudo-label providers. Static agents like roadside units can also offer pseudo labels. These agents can be more easily deployed to cover a wide range of drivable areas, and their detectors can be more easily trained (please refer to Lines 174 - 179).

Last but not least, collaborative autonomous driving (CAD) has indeed become a promising framework to investigate in the self-driving, smart city, and transportation community. It is reasonable to expect that nearby agents’ predictions will be easier to obtain over time. As a research paper, we respectfully think that building a stepping stone for future scenarios should be encouraged.

Q. … attempting to rely on other cars’ detection sharing seems unsafe … This mechanism will also need to resolve multiple other dependencies and complexities such as identifying trustworthy vehicles (e.g., not malicious, not inaccurate), managing potential error accumulation (causing problems when passing to other cars), etc.

Building upon our response above, we would like to emphasize that collaborative autonomous driving (CAD) is a multi-faceted, cross-disciplinary topic. It is not a pure computer vision problem; instead, many papers within the scope study problems involving communication, infrastructure, security, and privacy ([a, b, c, d, e, f]). It is indeed encouraging as it allows the vision community to focus on developing collaborative perception methods to leverage data with mild synchronization and mislocalization errors [g, h, i, j as well references in Lines 142 - 144] while leaving some other concerns (e.g., communication safety and trustworthiness) to the research communities with expertise. In our paper, we follow most of the existing collaborative perception papers to assume the transmitted data does not include malicious data. We will include this additional discussion.

Additional experiment on # data and performance.

In response to the reviewer’s suggestion, we have conducted an additional experiment to investigate how much the number of training data collected by following nearby agents could benefit the detector’s final performance. In doing so, we train detectors with four different numbers of training clips, including 5, 10, 15, and 20, and report the overall AP at IoU of 0.5 and 0.7.

The above table shows that the performance consistently improves as the ego car collects more data (pseudo labels) from nearby agents. We humbly believe this again highlights the effectiveness of our newly explored scenario of learning from nearby agents’ predictions. We thank the reviewer for suggesting this helpful experiment and will add the result to our manuscript.

Again, please kindly give us a chance to clarify any remaining points that are unclear.

We sincerely thank the reviewer for taking time to review our rebuttal and for being willing to increase the score. We also thank you for highlighting our study as a valuable exploration. We humbly believe that our study of this new learning scenario has the potential to open a new direction and provide meaningful insights to the community. Please let us know if you have any additional concerns or suggestions – we would be more than happy to address them and engage in further discussion!

Dear Reviewer iYjB,

We would like to give you some updates:

• First, due to some computational issues, we need to delay a little bit about semi-supervised learning experiments, but we will surely try our best to provide the result by the end of the discussion period.
• Second, we appreciate your positive opinion of our paper. In light of our further clarification and experimental result, we want to kindly ask if there is an additional response you would like us to provide by December 3rd so that you may consider further increasing your score.

Discussion on inherent diversity of approaches for self-driving

We thank the reviewers for mentioning offboard methods and other semi-supervised (or label-efficient) algorithms. All these methods (including ours) have a common goal—how to learn perception without extensive human labeling. However, instead of sticking with a fixed setting, they explore different settings/scenarios, leveraging different and orthogonal information for the goal. For instance, label-efficient methods investigated how to use limited labels; offboard methods investigated how to use future-frame information (in an offboard setting); and ours investigated a novel scenario using nearby agents’ predictions as supervision to learn perception. (Some other methods like [b] use data from repeated routes, if accessible.) In our humble opinion, these methods are NOT competitors BUT complementary if the corresponding information is available, as demonstrated in our response above. We have strengthened the discussion in this aspect in the related work and will add more in the final version.

Overall, we think what makes autonomous driving an interesting research area is not only its potential impact on society but also the diversity of data cues to leverage. We humbly think exploring different data cues is a valuable direction. Individually, they have their appropriate application scenarios. Collectively, they may complement each other to further boost the perception accuracy and robustness. We will include this in the discussion in the final version.

[b] You et al., “Learning to detect mobile objects from lidar scans without labels,” in CVPR 2022.