Availability-aware Sensor Fusion via Unified Canonical Space

We appreciate your thorough review and constructive feedback. We address your concerns below:

Q1/W1: Results on the nuScenes dataset

The primary reason for selecting K-Radar is that this dataset contains extreme weather conditions such as heavy snow and sleet, where sensor surfaces become completely occluded (missing/failure), resulting in total loss of camera and LiDAR measurements. These extreme situations are essential for validating ASF's core contribution of maintaining robust performance through dynamic attention redistribution when individual sensors completely fail. This unique characteristic cannot be found in nuScenes or other datasets.

Thank you for this important suggestion. While K-Radar is unique in providing extreme adverse weather conditions (complete camera/LiDAR failure), we agree that demonstrating generalizability is valuable. We appreciate your suggestion and are currently experimenting with ASF on the nuScenes dataset. Since the nuScenes dataset differs significantly from K-Radar in sensor configuration and ASF requires careful hyperparameter tuning (as shown in Table 4), the adaptation process is ongoing. We will publicly release all experimental code and trained models on the project page once the optimization is complete.

Q2/W2: Clarification on uncertainty handling

Thank you for pointing out this important distinction. We apologize for the confusion. To clarify, ASF addresses sensor availability rather than uncertainty quantification (e.g., aleatoric uncertainty). We will revise the words "uncertain" and "certain" used in line 64 of the introduction and line 217 of subsection 3.3 to "degraded" and "available". Thank you for clearly pointing out this terminology confusion.

Q3/W3: Figure 3 improvements

We appreciate your constructive feedback. To improve Figure 3's clarity as you suggested, we plan three enhancements: (1) overlaying ground truth bounding boxes onto prediction results to facilitate direct comparison, (2) inserting clear visual separators between different weather conditions, and (3) revising captions to better emphasize key findings.

Furthermore, acknowledging that Figure 3 currently contains dense information, we will add a minimum of 2 supplementary scenes in the Appendix formatted as 4×2 layouts. This expanded format enables higher-resolution display of individual subfigures, improving detail visibility and addressing your clarity concerns. We will also make interactive visualizations (inference_results.gif from the appendix zip file) available on the project page, allowing readers to observe sensor attention dynamics more clearly.

Q3/W4: Abbreviation reduction

Thank you for your suggestion on readability. We will eliminate unnecessary abbreviations such as "autonomous driving (AD)" and "feature map (FM)", keeping only essential technical abbreviations like ASF, UCP, CASAP, and BEV. This will improve the flow and readability of the paper.

Thank you again for your constructive feedback that helps improve our work's clarity and impact. We will carefully incorporate all suggested revisions to make our paper better.

Thank you very much for your continued engagement with our work and for considering our rebuttal responses. We sincerely appreciate your willingness to reconsider based on additional evidence.

As mentioned in our rebuttal, we selected K-Radar as it uniquely provides extreme adverse weather conditions (heavy snow, sleet) where sensor surfaces become completely occluded, resulting in total loss of camera and LiDAR measurements - critical scenarios for validating ASF's core contribution of maintaining robust performance when individual sensors completely fail.

However, following your suggestion about nuScenes dataset experiments, we have been actively working on this evaluation as stated in our rebuttal. While we are making our best efforts to expedite the process, we would like to clarify the technical challenges involved:

1. Sensor Configuration Differences: nuScenes has significantly different sensor configurations compared to K-Radar. Specifically for LiDAR, nuScenes uses 32-channel LiDAR with 10 sweep accumulation, while K-Radar uses 64-channel LiDAR with a single frame. Additionally, nuScenes employs 6 cameras while K-Radar uses a single front camera (or stereo camera). These fundamental differences require additional time for architectural adaptation and validation of each sensor's feature extractor (i.e., sensor-specific encoders).

2. Hyperparameter Optimization: As demonstrated in Table 4, ASF is sensitive to hyperparameters, requiring careful tuning across multiple configurations (patch size, channel dimensions, number of heads, etc.) for optimal performance on a new dataset, necessitating multiple training and validation iterations.

3. Computational Resource Constraints: While many SOTA nuScenes fusion networks including UniTR (Wang et al., ICCV 2023) and Cross Modal Transformer (Yan et al., ICCV 2023) utilized 8 A100 GPUs with 80GB VRAM each (640GB total VRAM), we have been working with a single RTX 3090 GPU (24GB total VRAM) for our ASF experiments. Recognizing the importance of nuScenes evaluation for generalization as you suggested, we are in the process of acquiring and setting up additional GPUs. Meanwhile, despite VRAM limitations preventing SOTA-comparable training settings, we have progressed with dataset download and code adaptation for nuScenes, and are exploring technical solutions including gradient accumulation for batch size expansion.

Given these constraints and the review timeline, while we are working diligently to provide preliminary results, it may not be feasible to provide nuScenes results within the next few days. Therefore, we committed to releasing nuScenes experimental results, code, and trained models on our project page once optimization is complete. However, we can confirm the following:

• Upon completion of ASF optimization for nuScenes, we will release results and code on the project page
• Manuscript revisions: (1) Clarification of terminology (from "uncertain" and "certain" to "degraded" and "available") and removal of unnecessary abbreviations, (2) Improved Figure 3 visualization
• Appendix updates with additional experiments: (1) Quantitative analysis of sensor contributions across different object distances, revealing how each modality's importance varies with range, (2) Statistical analysis of attention weight distributions under various weather conditions, and (3) Additional qualitative results visualization

We sincerely appreciate your thorough and insightful review, as well as your constructive feedback.

I greatly appreciate the authors' transparency regarding this matter and understand the line of thinking. Furthermore, I appreciate the efforts to expedite the experiments. While I continue to maintain that the submission would have been more convincing with evaluation on nuScenes (or at least more than just one dataset), I believe the paper--as currently is--still warrants an acceptance, and I look forward to seeing the results on nuScenes in the next revision/project page. Thank you for your continued discussion.

We appreciate your thoughtful review. We address your questions and concerns below:

Q1/W1: How would stereo cameras impact the performance, as it would provide geometric information from the images?

Thank you for this excellent question. Stereo cameras can indeed enhance performance by providing depth information. We have implemented Chen et al.'s DSGN++ (IEEE T-PAMI 2022) on K-Radar and conducted experiments. We achieved improved performance compared to monocular cameras: 26.5% AP3D for Sedan class and 26.3% AP3D for Bus or Truck class. This demonstrates the value of geometric information provided by stereo vision. These results validate that stereo vision substantially addresses the camera's geometric limitations you mentioned.

However, K-Radar's ZED 2i camera has a short 12cm baseline compared to KITTI's 54cm, limiting depth estimation accuracy for distant objects. Nevertheless, we plan to integrate this stereo-based approach as ASF's camera-specific encoder. All experimental code and weights will be released on the project page upon completion.

Q2: Would any other representation than BEV be possible and what impact would it have on performance?

Thank you for raising this important point. Several representations beyond BEV are indeed possible. (1) Voxel representation can directly represent 3D space preserving height information, but significantly increases memory and computational costs. (2) Polar coordinate representation suits 4D Radar but is less intuitive for downstream tasks (e.g., planning, segmentation). (3) Perspective view fusion benefits camera viewpoint but struggles with occlusion handling.

We chose BEV because it: (1) enables unified 2D representation for all sensor features, (2) directly supports autonomous driving path planning, and (3) provides balance between computational efficiency and performance.

Q3: How much of the performance variation can be attributed to the feature extractor of each modality?

This is a very insightful question. While various feature extractors (termed sensor-specific encoders in our paper) have been developed for each modality, recent sensor fusion research typically fixes well-established extractors to isolate fusion module contributions, e.g., Lang et al.'s PointPillars (CVPR 2019) and Yan et al.'s SECOND (Sensors 2018) for LiDAR-specific encoders. As mentioned in Appendix A.1, we intentionally froze pre-trained encoders to clearly isolate ASF's contribution.

As you correctly note, better sensor-specific encoders would improve overall performance. As mentioned in Q1, we are experimenting with replacing the current monocular BEVDepth with the higher-performing stereo-based DSGN++ for ASF integration. Completed experimental code and weights will be uploaded to the project page to demonstrate how improved sensor-specific encoders impact performance.

Thank you again for your constructive feedback. We will carefully consider all points raised to improve our research, particularly regarding camera geometric information utilization.

We sincerely appreciate your time and effort in reviewing this paper and your recognition of our research.

We appreciate your thoughtful and positive review. We address your suggestions below:

Q1: How do you plan to address the limitations related to camera features integration? Particularly in difficult weather conditions.

Thank you for this excellent question. Camera vulnerability in adverse weather stems from its fundamental reliance on visible light. Our strategy is to address this inherent limitation in two ways:

First, ASF automatically assigns higher weights to other sensors (particularly 4D Radar) when camera performance degrades. As demonstrated in inference_result_2.gif (in the appendix zip file), when cameras fail due to heavy snow, ASF increases reliance on 4D Radar (blue) to maintain robust performance.

Second, we are exploring stereo-based approaches to enhance camera utilization. We have implemented Chen et al.'s DSGN++ (IEEE T-PAMI 2022) on K-Radar, achieving improved performance (26.5% and 26.3% AP3D for Sedan and Bus or Truck, respectively) compared to the monocular camera-based approach. We plan to integrate this as ASF's camera-specific encoder. However, it's worth noting that K-Radar's ZED 2i camera has a 12cm baseline compared to KITTI's 54cm, limiting improvement potential.

Q2/W2: The method is evaluated on K-Radar. This is enough, but it would be interesting to see evaluations on other datasets or on real-world data.

Thank you for this important suggestion. While K-Radar is unique in providing extreme adverse weather conditions (complete camera/LiDAR failure), we agree that demonstrating generalizability is valuable. We are currently conducting ASF experiments on the nuScenes dataset. Upon completing hyperparameter optimization, we will release results and code on the project page.

Q3/W1: I would recommend to improve a bit the figures. Figure 3 would especially benefit from a simpler organization.

Thank you for this constructive feedback. To address your concern about Figure 3's clarity, we will implement several improvements: (1) overlay GT bounding boxes on prediction results for easier comparison, (2) add clear separation lines between weather conditions, and (3) enhance captions to highlight key findings.

Additionally, recognizing that the current Figure 3 may be too dense with information, we will include 2 additional scenes in the Appendix using a 4×2 format. This larger format allows each subfigure to be displayed at a higher resolution, making details more visible and addressing your concern about clarity. We will also provide interactive visualizations (inference_results.gif in the appendix zip file) on the project page, enabling readers to dynamically observe sensor attention changes with better visual clarity.

Thank you again for your positive evaluation and constructive suggestions. We will incorporate these improvements to enhance our contribution.

We greatly appreciate your time and effort in reviewing this paper and your recognition of our work.

Thank you sincerely for your thoughtful review. We address your concerns below:

Q1: How does CASAP actually infer sensor reliability?

Thank you for this excellent question. CASAP empirically learns sensor reliability through cross-attention mechanisms trained on multi-sensor data. Features projected into the unified canonical space via UCP are optimized during training such that the reference query $Q_{ref}$​ (Eq. 5) achieves high correlation with features from reliable sensors. During inference, when a sensor is degraded or missing, its features exhibit low correlation with $Q_{ref}$​ and consequently receive lower attention weights. This is validated in Fig. 3, where 4D Radar assumes a dominant role when camera/LiDAR fail under adverse weather conditions. To facilitate understanding of this mechanism, we will release dynamic sensor attention map videos (inference_result_2.gif in the appendix zip file) on the project page.

Q2: Can the method generalize to datasets without 4D Radar?

Thank you for raising this important point. ASF's architecture is inherently sensor-agnostic by design. We selected K-Radar as it uniquely provides multi-modal data under extreme adverse weather conditions where cameras and LiDAR experience complete failure due to heavy snow occlusion. Following your valuable suggestion, we are currently conducting experiments with ASF on nuScenes. Given the differences in sensor configurations and training settings between nuScenes and K-Radar, as well as ASF's sensitivity to hyperparameters (as shown in Table 4), the optimization process requires careful hyperparameter tuning across multiple experimental configurations. Upon completion, we will release the training code and model weights for both benchmarks on the project page.

Q3: Camera contribution analysis?

Thank you for this insightful observation. While cameras are inherently vulnerable under adverse weather conditions due to their reliance on visible light, they nonetheless provide crucial contributions for long-range object detection. This is clearly demonstrated in inference_result_1.gif (in the appendix zip file), where red regions (indicating camera attention) dominate in distant object areas (upper-right portion of the video). This phenomenon aligns with findings from Wu et al.'s Virtual Sparse Convolution (CVPR 2023), which achieved 1st place on the KITTI 3D object detection benchmark, where their Figure 2 illustrates that virtual points (derived from camera features) are instrumental in achieving substantial AP improvements for distant objects. We are currently conducting quantitative analyses of distance-based sensor contributions, with results to be included in the revised Appendix in the camera-ready version.

W1: Camera performance in adverse weather

Your observation is indeed accurate. The K-Radar dataset comprises over 60% adverse weather frames, where heavy snow and sleet conditions result in complete occlusion of the sensor surfaces, leading to total loss of camera measurements (as demonstrated in inference_result_2.gif). These extreme conditions yield near-zero AP for camera-only detection, hence our notation of "-" (not available) in Table 3. This reflects the fundamental physical limitations of optical sensors rather than any methodological deficiency. ASF's primary contribution lies in gracefully handling such sensor failures through dynamic attention redistribution, maintaining robust performance despite individual sensor degradation. Comprehensive camera-only performance logs across all weather conditions will be made available on the project page, following the format provided in the supplementary materials (logs_10_seeds).

W2: Diagnostic analysis

We appreciate this constructive feedback. In addition to the t-SNE visualizations (Figs. 1, 5) and attention maps (Fig. 3) currently presented, we are conducting comprehensive diagnostic analyses to provide deeper interpretative insights: (1) quantitative analysis of sensor contributions across different object distances, revealing how each modality's importance varies with range, and (2) statistical analysis of attention weight distributions under various weather conditions, quantifying the dynamic adaptation of sensor reliability. These additional analyses will be incorporated into the revised Appendix in the camera-ready version, offering a more thorough understanding of ASF's adaptive behavior under diverse operational conditions.

Thank you once again for your thorough review. We believe these addressed concerns and planned enhancements will substantially improve the paper's impact and reproducibility.