RCDN: Towards Robust Camera-Insensitivity Collaborative Perception via Dynamic Feature-based 3D Neural Modeling

Thank you for your appreciation of our work. We will take your suggestions into account when preparing the final version. Below please find responses to specific questions.

"W.1: Can the authors showcase the motivation of using Nerf for the static and dynamic fields, are there any dominant advantages of nerf, compared to other 3d reconstruction methods in this method?"

A.1 The proposed geometry BEV feature avoids the issue of per-scene “network memorization” inherent in NeRF by employing generic feature representations. The decoupling of dynamic and static neural rendering adapts to real-world collaborative autonomous scenarios. We selected NeRF for its photorealistic rendering capabilities. However, research such as Mip-NeRF 360 (CVPR 2022) and Zip-NeRF (ICCV 2023) demonstrates that other reconstruction methods, such as Structure from Motion (SfM) and Multi-View Stereo (MVS), are less effective at novel view synthesis compared to NeRF-based methods. Meanwhile, to make NeRF a better adaption to the collaborative perception setting, we optimize the collaborative neural field by decoupling it into static and dynamic parts for better performance, which improves the segmentation performance of dynamic vehicles by 12.27% compared to the non-decoupled one.

"W.2: It is necessary to give more rigorous mathematic analysis of equations in this paper. The authors are required to introduce the details of each networks, including the training parameters, learning rate, weight values in eq. 12."

A.2 Thank you for pointing this out. The static and dynamic neural components of our model follow the architecture of Instant-NGP (SIGGRAPH 2022). The architecture for the collaborative perception component is detailed on L566-583 of the supplementary material. The initial learning rate is 5e-4 with the exponential learning rate decay strategy. The weight values in eq. 12 is set to 1.0, 1.0, 0.1, and 1.0, respectively. Further details about the loss function are provided on L632-636 of the supplementary material. We will also include these parameters in the main text for improved clarity.

"L.1: The current work focuses on addressing the camera-insensitivity problem in collaborative perception. It is evident that accurate reconstruction can compensate for the negative impact of noisy camera features on collaborative perception."

A.1 Thanks for your deep understanding of our proposed RCDN. To the best of our knowledge, We are the first to introduce the NeRF into the collaborative perception field to handle robust settings. Our proposed RCDN has shown significant improvements in our experiments, particularly in scenarios under multi-source noises.

Thank you for your insightful comments. We have carefully addressed all the questions raised. Please find our responses below.

"W.1: What is difference between sfw, sbw in equation (7)?"

A.1 Apologies for any confusion regarding the terms s_fw and s_bw. The term s_fw stands for forward scene flow, while s_bw refers to backward scene flow. Specifically, the forward scene flow (s_fw) estimates the flow from time t to t+1, whereas the backward scene flow (s_bw) estimates the flow from time t to t-1. By leveraging both s_fw and s_bw, we can achieve a more consistent representation of scene movements.

"W.2: The experimental section is a bit too short. I feel its not finished yet. However, there is limited space."

A.2 Thanks for your understanding. Due to the limited space available, we aimed to present the key experimental results and analyses as concisely as possible. However, we greatly appreciate your suggestion and agree that providing additional details could enhance the clarity and comprehensiveness of our work. In response to your comment, we will make the following improvements to the experimental section in the final version: i) include a more detailed description of the experimental setup and procedures ii) add additional experimental results and figures (such as detection results) to better support our conclusions.

"Q.1: How many message exchange tasks could be used overall (Figure 2.)"

A.1 The proposed RCDN does not add extra message exchange times, and as for communication cost, similar to DiscoNet (NIPS 2021), we only utilize the RGB labels during the training stage, meaning we leave the communication burden to the training stage and do not introduce extra information during the inference.

"Q.2: Will baseline code be published in combination with the dataset? When?"

A.2 We will publish the corresponding codes and dataset as soon as we get accepted.

"L.1: The most relevant limitation is the missibg real-time applicability."

A.1 We provide the corresponding latency times in the supplementary materials: the proposed static module takes approximately 4.47 ms, the dynamic module takes about 3.94 ms, and each rendering process takes about 21 ms. The training time ranges from 20 to 30 minutes. For more details, please refer to lines L556-583. Currently, RCDN requires further code optimization to meet real-time application requirements.

Thank you for your detailed review and valuable comments. Please note our top-level comment with additional experimental and theoretical results. Below we address specific questions.

"W.1: Have authors evaluated the method on different down tasks other than segmentation?"

W.1 Our proposed RCDN is general to different downstream tasks and is not limited to just BEV segmentation. We focus on BEV segmentation due to its crucial role in autonomous driving, with direct applications to other tasks such as layout mapping, action prediction, route planning, and collision avoidance. Additionally, we have validated RCDN for detection tasks. In our experiments, we replaced the original segmentation header with a detection header (see Figure 2 and Table 3 in the attached PDF). Table 3 shows that for CoBEVT, using RCDN improves the metrics of AP@0.50 and AP@0.70 by 19.05% and 24.99%, respectively.

"W.2: Technical contribution"

A.2 Robust perception is a significant challenge in single-agent systems; however, few studies have addressed this issue in the context of collaborative perception. Instead of adding additional modal sensors, such as LiDAR, as done in single-agent systems, our approach leverages the unique multi-view properties to mitigate the impact of noisy views. Notably, NeRF (Neural Radiance Fields) achieves photorealistic rendering by optimizing 2D multi-view images. To the best of our knowledge, we are the first to apply NeRF to the field of collaborative perception to handle robust settings. Meanwhile, to make NeRF a better adaption with the collaborative perception setting, we propose the geometry BEV features, which can improve the PSNR by about 9.30% (see more details in Appendix.B.6) and avoid NeRF network memorization. Additionally, we optimize the collaborative neural field by decoupling it into static and dynamic parts for better performance, which improves the segmentation performance of dynamic vehicles by 12.27% compared to the non-decouple approach.

"Q.1: How real is the synthetic OPV2V-N dataset? In other words, how can features learned in OPV2V-N dataset be translated to real-world usage?"

A.1 The OPV2V-N dataset is recorded by the co-simulation with SUMO (traffic manager) under the realistic platform CARLA simulator. As for the "How can features learned in OPV2V-N dataset be translated to real-world usage?", this pertains to the broader issue of sim-to-real transfer. Whether the RCDN pre-trained on OPV2V-N can be effectively applied to other datasets or real-world scenarios depends on the domain gap between the BEV (Bird’s Eye View) feature space learned from OPV2V-N and that of other datasets or real-world environments. Recent research, such as the DUSA approach (ACM-MM 2024), addresses sim-to-real adaptation for collaborative perception. DUSA proposes a unified unsupervised BEV feature adaptation module. Since our RCDN operates as a post-processing step built upon BEV features, it is theoretically possible for RCDN to leverage DUSA to bridge the BEV feature space gap between OPV2V-N and real-world scenarios.

"Q.2: Moreover, are there any real-world quantitative results on model trained on synthetic data?"

A.2 Please refer to our top-level comment and the attached PDF for further details. In summary, we can employ sim2real BEV feature space adaptation techniques, such as DUSA, to validate models trained on synthetic data in real-world scenarios. However, the current real-world collaborative perception datasets can not meet the RCDN's camera setting demands, and we will keep track of real-world dataset development, implementing the RCDN in real-world settings if the camera setting meets the demands.

"Q.3: How does one verify that the volumetric features are geometrically correct? (how accurate is the Geometric BEV features?)"

A.3 Our proposed module only inputs RGB images without introducing additional features such as depth images. Specifically, we supervise the geometry of volumetric features by using multi-view consistency and corresponding downstream tasks. From the experimental results, the visualization results (please kindly refer to Figure 5 in the manuscript Page 9.) demonstrate the high multi-view consistency of the rendered results. The generated BEV features successfully improve the performance of perception tasks in 3D space (i.e., BEV segmentation/detection), which further indicates the geometrical correctness of our BEV feature.

"Q.4: How is BEV segmentation evaluation differ on non-flat surfaces like hills or bridges?"

A.4 Thank you for your questions. Currently, the OPV2V-N segmentation labels are available only in 2D image format. As a result, non-flat surfaces such as hills or bridges are projected onto these 2D segmentation images. Consequently, from the perspective of static part segmentation, our BEV decoder classifies static elements into two categories: road and line.

Thank you for your detailed review and thoughtful comments. Please note our top-level comment with additional experimental and theoretical results. Below we address specific questions.

"W.1: Evaluation is only performed with OPV2V-N dataset which may result in overfitting. More evaluation with different dataset is required. "

A.1 This point was also raised by other reviewers, and we have addressed it in the general response (above). Please refer to our top-level comment and the attached PDF for detailed information. In summary, i) validated our approach on a newly collected V2XSet-N-mini dataset, demonstrating that the proposed RCDN stabilizes performance under noisy camera conditions.; ii) utilized the OPV2V-N pre-trained RCDN module for direct inference on the V2X-Sim 2.0 dataset, showing that RCDN effectively stabilizes the perception results.

"W.2: The way proposed algorithm is combined with prior method is unclear. The reviewer guessed that the MCP module can be replaced with prior methods, but it is not stated"

A.2 The MCP module stands for the Multi-agent Collaborative Perception module. Existing state-of-the-art (SoTA) MCP modules share a common pipeline: an encoder-fusion-decoder architecture. To ensure fairness in collaborative perception experiments, different MCP modules use the same encoder-decoder architecture but differ in the fusion process. The fusion process is responsible for the bird-eye view (BEV) feature aggregation. Therefore, the MCP module can be replaced by simply switching between different BEV feature aggregation processes.

"W.3: Many abbreviations are not explained sufficiently and terminologies the author defined are ambiguous and may be incorrect. MCP is short for the multi-agents collaborative perception process but the paper did not explain MCP module in details with no reference"

A.3 Thank you for pointing out the misuse of abbreviations. We will ensure to double-check and eliminate any ambiguous abbreviations. For the multi-agent collaborative perception (MCP) module, we employ mainstream collaborative methods based on intermediate BEV features for our experiments. The general setting of the MCP module is described in L570-578 supplements. To avoid any ambiguity, we will include additional technical details about the different MCP baseline methods, such as their respective pipelines and feature fusion processes, in the final supplements.

"W.4: BEV, no full name, no reference""There are more unclear sentences."

A.4 We will reintroduce the concept of bird’s-eye view (BEV) and include relevant references, such as BEVFormer (ECCV 2022). Additionally, we will address the misuse of the terms s_fw and s_bw, which refer to scene flow for forward and backward directions, respectively. We will also revise the paper to clarify any unclear sentences.

"W.5: “Camera-insensitivity” can be understood terminologies related to camera sensor sensitivity (how much the camera sensor accept photon…). ""Robust Camera-Insensitivity: Robust == Camera-sensitivity? The latter one may be redundant Line 6. introduce a new robust camera-insensitivity problem: cam be replaced “introduce BEV segmentation when the camera capture are unreliable (or noisy)?” Should be more concrete without ambiguous words"

A.5 Thank you for your careful and valuable feedback. Our intent in using the term “robust” was to emphasize the concept, but we will revise the description to avoid redundancy based on your suggestions. We will replace the phrase “introduce a new robust camera-insensitivity problem” with “introduce BEV segmentation when the camera capture is unreliable (or noisy).”

"W.6: Line19 “Ported to” mean?"

A.6 Apologies for any confusion regarding the term "Ported to". The "Ported to" means that it is fairly easy to apply to different existing feature backbones, as it is the post-processing step built on top of BEV features.

We appreciate your time in reviewing our work and your feedback on the paper's value and clarity! Please note our top-level comment with additional experimental and theoretical results. Below we address specific questions.

"Q.1: Have you tested RCDN on any datasets other than OPV2V-N? How does it perform on real-world data?"

A.1 This point was also raised by other reviewers, and we have addressed it in our general response (above). Please refer to our top-level comment and the attached PDF for detailed information. In summary, i) we validated our approach on the newly collected V2XSet-N-mini dataset, demonstrating that the proposed RCDN stabilizes performance under noisy camera conditions; ii) we also deployed the OPV2V-N pre-trained RCDN module for direct inference on the V2X-Sim 2.0 dataset, where RCDN continued to effectively stabilize perception results; iii) we will keep track of real-world dataset development, implementing the RCDN in real-world settings when the camera settings meet our requirements.

"W.1: The performance of RCDN is primarily validated on the OPV2V-N dataset, which may limit the generalizability of the results to other datasets or real-world scenarios."

A.1 Since the proposed RCDN operates as a post-processing step on top of BEV features, its direct applicability to other datasets or real-world scenarios depends on the domain gap between the BEV feature space learned by OPV2V-N and those of other datasets or the real world. i) for other datasets, we performed open-set validation by directly applying the OPV2V-N pre-trained RCDN module to the V2X-Sim 2.0 dataset. The corresponding results, shown in Figure 1 of the attached PDF, indicate that RCDN effectively stabilizes perception results. This is expected since both V2X-Sim 2.0 and OPV2V-N are recorded using the CARLA simulator, resulting in a minimal domain gap. ii) for real-world scenarios, the recently published DUSA (ACM-MM 2024) identified a significant domain gap between simulated and real-world environments. DUSA proposed an unsupervised BEV feature adaptation module to bridge this gap. Therefore, in theory, the proposed RCDN could leverage DUSA to mitigate the BEV feature space gap between OPV2V-N and real-world datasets.

"Q.2: What are the computational requirements of RCDN, and how feasible is it for real-time applications in autonomous systems?"

A.2 We provide the corresponding latency times in the supplementary materials: the proposed static module takes approximately 4.47 ms, the dynamic module takes about 3.94 ms, and each rendering process takes about 21 ms. The training time ranges from 20 to 30 minutes. For more details, please refer to lines L556-583. Currently, RCDN requires further code optimization to meet real-time application requirements.

"W.3: It would be nice if the authors provide failure cases"

A.3 Based on the assumption of the multi-view setting, the multi-view RCDN may fail when agents are at the edge of the communication range, resulting in minimal view overlap. This limitation motivated our effort to collect the OPV2V-N dataset for our experiments.

"Q.4: How well does the method scale with an increasing number of agents and cameras? Are there any performance bottlenecks?"

A.4 Regarding the increasing number of agents and cameras, we validated the impact of adding more cameras using the OPV2V-N dataset (corresponding scenario types are T section and midblock respectively) with the CoBEVT baseline. From Table 2 in the attached PDF, we observe the following: i) With a single overlapping camera view, the proposed method significantly improves baseline performance, and ii) While theoretically, more cameras can provide a larger overlap range, the addition of multiple cameras (depending on their positions) may introduce redundant viewing angles, resulting in less significant performance improvements.