$**W1: **$

Thank you for the constructive suggestion. In ROE, the occupancy information encoding leverages the depth distribution predicted by the depth head to represent the probabilistic distribution of an object along the camera ray. Indeed, replacing the depth distribution with a one-hot vector helps further analyze the contribution of depth modeling to overall performance. We conduct this ablation study and find that replacing the depth distribution with a one-hot encoding (i.e., using only the peak bin) leads to a drop in AP70 on the OPV2V dataset from 74.25 to 73.12. This performance degradation suggests that modeling the full depth distribution provides richer information about the object’s potential 3D location along the ray, which enhances the network’s ability to accurately localize instances via cross-view validation.

$**W2: **$

On the OPV2V dataset, a single agent communication cost per frame for RayFusion, IFTR, and late fusion are 276.6 KB, 18.8 MB, and 2.7 KB, respectively. Other intermediate fusion methods incur a communication cost of approximately 64.0 MB per frame. We will include the communication cost of all methods in Table 1 of the camera-ready version. Thank you for the valuable suggestion.

$**W3: **$

Thank you for the careful suggestion. We will thoroughly proofread the paper in the camera-ready version to eliminate any grammatical or typographical errors.

$**W1: **$

In Table 1, we adopt BEVFormer as the single-agent detector for baseline methods, following IFTR, since most existing collaborative perception approaches are BEV-based. In contrast, RayFusion uses Sparse4D as the single-agent detector. The rationale is twofold:

1. Resource constraints: BEV-based methods often demand high computational resources in camera-based collaborative perception tasks. For instance, training IFTR requires more than 40 GB of GPU memory on the OPV2V dataset when using the standard IFTR configuration. Due to limited training resources, we use the sparse detector Sparse4D as the single-agent detector to implement RayFusion, which only requires 12GB of GPU memory to complete training. Moreover, it offers significantly lower inference latency. As shown in Table 4, RayFusion improves AP70 by 8.21 (+12.43%) on OPV2V while reducing inference time from 298 ms to 114 ms.

2. Higher performance potential under sparse paradigms: In BEV-based methods, the information flow for instance-level perception typically follows: image features → BEV features → instance features. In contrast, sparse detectors directly derive instance-level information from image features: image features → instance features. From an information-theoretic perspective, our instance-level fusion strategy is better suited for sparse detectors, as it incurs less information loss.

To further ensure that the performance gain does not stem solely from the stronger single-agent detector, we include an additional baseline in Table 1 named CoSparse4D, which uses Sparse4D as the single-agent detector without our proposed fusion strategy. RayFusion consistently outperforms this baseline across all datasets, supporting the effectiveness of our method. Furthermore, extensive ablation and robustness experiments also validate the contribution of our proposed methods. We hope that the aforementioned rationale for adopting Sparse4D as the single-agent detector, together with the CoSparse4D baseline using the identical single-agent setup and our extensive ablation experiments, may help address your concerns.

$**W2: **$

Thank you for your valuable feedback. We acknowledge that the real-world DAIR-V2X dataset has limitations in terms of the number of collaborators and camera views. However, it remains the most widely used real-world dataset for camera-based collaborative perception [r1, r2], which makes it a standard benchmark for fair comparison with prior works.

To further evaluate the generalization capability of our approach, we have attempted to benchmark RayFusion on the V2X-Real dataset. However, due to the time-consuming data download process and inherent issues in the dataset — for example, some scenes are missing camera images (e.g., train/2023-03-17-15-53-02_1_0), and there are inconsistencies in camera parameter keys across frames (e.g., cam1_left in 000142.yaml vs. cam1 in 000001.yaml of the same sequence train/2023-04-04-15-58-18_30_0/2/). These issues caused initial training failures. We have since excluded the problematic samples and restarted training. Results from this evaluation can be included in the camera-ready version for reference.

$**W3: **$

We sincerely appreciate your valuable suggestion. However, camera-based models following the back-projection paradigm (e.g., BEVFormer and Sparse4D) typically fail to converge in the absence of both a pre-trained image backbone and LiDAR point cloud supervision [r3]. Moreover, applying LiDAR supervision during training incurs minimal additional cost, as point cloud data is still required during annotation even when LiDAR is not used for supervision. In addition, jointly collecting LiDAR and image data for depth supervision has become a standard practice and the community primarily emphasizes camera-only settings during the inference stage [r2, r4, r5]. To eliminate ambiguity, we will revise the statement in the camera-ready version to specify that the inference phase operates under a camera-only configuration.

$**W4: **$

We adopt a perception range of $51.2\mathrm{m}$ in each direction, following the setup in IFTR, to ensure a fair comparison. To more convincingly demonstrate the advantages of RayFusion, we conducte additional experiments with an extended range of $102.4\mathrm{m}$ in each direction. Under this setting, RayFusion achieves an AP70 score of 48.10, outperforming CoSparse4D (41.73) and IFTR (38.41).

[r1] IFTR: An Instance-Level Fusion Transformer for Visual Collaborative Perception

[r2] Collaboration Helps Camera Overtake LiDAR in 3D Detection

[r3] Sparse4D v2: Recurrent Temporal Fusion with Sparse Model

[r4] Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection

[r5] Temporal Enhanced Training of Multi-view 3D Object Detector via Historical Object Prediction

$**W1: **$

Your observation is absolutely correct — the depth prediction head is supervised using LiDAR point clouds, which indeed requires additional sensor data during training. However, the use of LiDAR supervision leads to significant performance gains [r1, r2]. In fact, camera-based models following the back-projection paradigm typically fail to converge in the absence of both a pre-trained image backbone and LiDAR point cloud supervision [r3].

$**W2: **$

As shown in Table 2, ROE improves AP70 by 2.21 (+3.07%) on OPV2V and 1.34 (+13.14%) on DAIR-V2X. For an already strong perception model, such performance gains are considered substantial. That said, we acknowledge that there is still room for further improvement in reducing depth ambiguity. For example, leveraging multi-frame information and depth hypothesis sampling has shown promise in addressing depth uncertainty [r4]. We plan to explore more effective strategies along these lines in future work.

$**Q1: **$

In camera-based perception systems, nearly all methods rely on complex projection operations based on intrinsic and extrinsic camera parameters, and inaccuracies in these projection matrices can significantly affect model performance. To provide a reference, we analyze the impact of errors in the camera extrinsic matrices on perception accuracy. Specifically, we simulate extrinsic perturbations using Gaussian noise with a mean of 0 and a variance of 0.02. Under these conditions, experimental results show that RayFusion’s AP70 drops from 74.25 to 24.11 (–67.53%), while IFTR’s AP70 drops from 66.04 to 20.89 (–68.37%).

$**Q2: **$

Thank you for pointing out the need for a more detailed explanation of how LiDAR point clouds are used to supervise dense depth prediction. We adopt a predefined depth range $d \in [2\mathrm{m}, 52\mathrm{m}]$, uniformly divided into 80 bins. Our implementation follows similar principles to BEVDepth and FB-BEV. The process is as follows:

1. We project the LiDAR point cloud into the pixel coordinate system using the camera intrinsics and extrinsics, resulting in a dense depth map $\bar{\mathcal{D}}_j^k$ and a depth mask $\bar{\mathcal{M}}_j^k$ for the $k$-th camera view of agent $j$. The mask $\bar{\mathcal{M}}_j^k$ is set to 1 at pixels where valid LiDAR points are projected within the predefined depth range, and 0 elsewhere.

2. Based on the predefined bins and a downsampling ratio, we convert $\bar{\mathcal{D}}_j^k$ into per-pixel depth classification targets $\tilde{\mathcal{D}}_j^k$ and get a updated depth mask $\tilde{\mathcal{M}}_j^k$.

3. The depth head predicts a per-pixel depth probability distribution $\mathcal{D}_j^k$, which is supervised using cross-entropy loss only on the valid pixels where the updated depth mask is equal to 1. The final depth supervision loss is given by: $\mathcal{L}\_{\text{depth}} = \text{CrossEntropy}(\mathcal{D}\_j^k, \tilde{\mathcal{D}}\_j^k, \tilde{\mathcal{M}}\_j^k).$

Finally, the overall training loss is defined as: $\mathcal{L} = \lambda\_{\text{depth}} \cdot \mathcal{L}\_{\text{depth}} + \mathcal{L}\_{\text{obj}}$, where $\mathcal{L}\_{\text{depth}}$ and $\mathcal{L}\_{\text{obj}}$ denote the depth supervision and object detection losses, respectively, and $\lambda\_{\text{depth}}$ is a hyperparameter set to 0.5 by default.

[r1] BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection

[r2] FB-BEV: BEV Representation from Forward-Backward View Transformations

[r3] Sparse4D v2: Recurrent Temporal Fusion with Sparse Model

[r4] Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection

$**W1: **$

Thank you for the insightful comment. We agree that crowded scenes present additional challenges for ray-based cross-view validation due to increased instance overlap and dense object distributions. However, the proposed ROE in our RayFusion framework is specifically designed to capture the probabilistic distribution of instance presence along each viewing ray, which helps mitigate such challenges. Concretely:

1. For a real instance $obj_1$ in 3D space, false positive predictions generated by different agents typically lie along their respective observation rays that intersect at $obj_1$. This property allows effective filtering of multiple false positives related to $obj_1$.

2. Suppose agent $i$ observes another real instance $obj_2$ along the observation ray of $obj_1$. However, the observation rays of $obj_2$ from other agents do not overlap with agent $i$’s observation ray of $obj_1$. Typically, there are two distinct intersection points corresponding to $obj_1$ and $obj_2$, so the true object $obj_2$ will not be mistakenly filtered out.

To support this, we conduct additional targeted evaluations on crowded scenes from the OPV2V dataset:

1. We select 467 test frames in which more than 15 objects appear within the detection range. The AP70 scores with and without ROE are 74.32 and 71.24, respectively.

2. We further select 50 congested frames with over 20 objects within the detection range. In this setting, the AP70 scores with and without ROE are 66.98 and 61.36, respectively.

These results demonstrate that RayFusion can effectively distinguish true positives from spurious detections even in highly crowded scenarios, confirming the robustness of the proposed occupancy-based encoding.

$**W2: **$

RayFusion performs spatial-temporal alignment on instance information from multiple agents, aiming to obtain spatial-temporal aligned and accurate instances. However, due to the inherent ambiguity of monocular depth estimation, a single agent may suffer from false positives and radial localization errors. To address this, we propose ROE, which leverages cross-view validation to effectively mitigate depth ambiguity, reduce false positive predictions, and enhance the ego’s perception performance. In addition, localization noise in real-world scenarios can significantly degrade the performance of collaborative systems. To tackle this issue, we introduce MIFA, which employs multi-window attention to capture both local and global spatial features in parallel. As shown in Figure 3(a), RayFusion exhibits strong robustness to localization noise.

$**W3, Q1: **$

Thank you for the valuable suggestion. A comprehensive evaluation of collaborative system robustness is crucial for deployment in real-world scenarios. Benefiting from instance features enriched with image semantics, MIFA uses multi-window attention to compensate for localization noise. In Figure 3, we analyze RayFusion’s detection performance under localization noise and communication delays, demonstrating its strong robustness against these factors. However, if agent $i$ predicts an instance $obj_3$ that cannot be seen by all other collaborators or the ego vehicle (e.g., due to severe occlusion), the model theoretically lacks the ability to identify or filter this instance. When $obj_3$ is visible to other collaborators or the ego vehicle but no corresponding prediction exists, we expect the model to learn to filter out such false predictions. Here, we further analyze the impact of corrupted poses and false positive predictions on perception accuracy:

1. Corrupted poses: We simulate corrupted poses with localization noise drawn from a Gaussian distribution with mean 0 and variance 10 m, and heading noise with mean 0 and variance 10°. Under these conditions, RayFusion’s AP70 drops from 74.25 to 33.44 (–54.96%), while IFTR’s AP70 drops from 66.04 to 8.45 (–87.20%).

2. False positives: We simulate false positive instances by replacing the anchor boxes of the top $N_{\text{fp}}$ high-confidence predictions from collaborators with those from low-confidence predictions. These modified instances are then merged into the collaborative instances. As $N_{\text{fp}}$ increases from 0 to 10, 50, and 100, RayFusion’s performance drops from 74.25 to 68.81, 67.68, and 66.42, respectively. This demonstrates that erroneous false positives from collaborators can indeed negatively impact model performance. In future work, we plan to further enhance robustness by integrating observations from multiple agents to suppress such unreliable predictions.