Point Cluster: A Compact Message Unit for Communication-Efficient Collaborative Perception

Thank you for your valuable comments and kind words to our work. Below we address specific questions.

Q1: Ablation studies on PCP, PCA, pose correction and latency compensation.

• About PCP. Without the PCP module, which achieves message packaging for other agents, this method degrades to single-agent perception for the ego agent without messages from other agents. We evaluate this baseline on the validation set of V2XSet, achieving AP@0.7 of 67.00, which is far behind our full CPPC that achieves AP@0.7 of 89.99. These results demonstrate the limitations of single-agent perception and validate the necessity of collaboration for achieving more complete scene perception.
• About PCA. Without the PCA module, which is designed to aggregate point cluster features from multiple agents, we directly merge boxes in messages from multiple agents like late collaboration. This setting is evaluated on the validation set of V2XSet, achieving AP@0.7 of 77.88, which is far behind our full CPPC that achieves AP@0.7 of 89.99. This validates that our PCA module can effectively utilize rich object information.
• About pose correction. We evaluate our CPPC under a positional error of 0.6m and a heading error of 0.6$^\circ$ without the pose correction module, achieving an AP@0.7 of 34.22 on the V2XSet validation set. This result is significantly lower than the AP@0.7 of 87.37 achieved with the pose correction module, highlighting its critical role in enhancing performance.
• About latency compensation. We evaluate our CPPC under time latency 500ms without the latency compensation module, achieving an AP@0.7 of 62.08 on the test set of DAIR-V2X-C. This result is below than the AP@0.7 of 64.61 achieved with the latency compensation module.

Q2: Comparison with BEV-based methods during aggregation.

• In Appendix C.2, we theoretically analyze that the computational complexity of point cluster aggregation is approximately linearly related to the number of potential targets in the scene, i.e., $\mathcal{O}(DN_\text{object})$, where $D$ is a constant. However, the computational complexity for BEV-based message aggregation is $\Omega(HW)$, which is related to the square of the perception range. Since $N_\text{object} \ll HW$ in most real scenes, our point cluster-based method is more efficient for long-range aggregation. We evaluate the inference time cost of our PCA module, i.e., 9.69 ms, and aggregation module in V2X-ViT, i.e., 41.42 ms.

Q3: Spatial cluster matching when pose errors exist.

• Details of spatial cluster matching. We calculate the Euclidean distance between the centers of point clusters detected by different agents. If the distance between two point cluster centers is less than a specified threshold $\epsilon_\text{pose}$, they are considered to belong to the same potential object.
• Cluster matching under pose errors. When pose errors are present, aligning the coordinate space of other agents with the ego agent may result in misaligned point clusters. To address this issue, we first obtain an initial matching result through spatial cluster matching. Subsequently, we optimize Eq. (7) to estimate the accurate poses of the surrounding agents, thereby correcting pose errors to achieve more reasonable matching based on the refined poses. The experiments depicted in Fig. 4 (c) and (d) demonstrate that the proposed strategy performs effectively under various pose error scenarios.
• Extension work. We appreciate your thoughtful suggestion and have carefully considered this aspect of the method. Indeed, in scenarios with significant pose errors—though rare—it becomes challenging for a manually set threshold to address all cases. To enhance the robustness of spatial cluster matching, we extend our current approach to a maximum common subgraph (MCS)-based spatial matching [1]. In this method, each cluster mapping within the MCS represents point clusters corresponding to the same potential object across the cluster graphs of different agents.

[1] A Partitioning Algorithm for Maximum Common Subgraph Problems, IJCAI, 2017.

Thank you for your valuable comments and kind words to our work. Below we address specific questions.

Q1: Thresholds may be hard to choose in complex or unseen scenarios.

• This is a great question. We ablate the thresholds involved in our CPPC in Tables 5 and 6. The results show minimal performance fluctuations over a wide range, demonstrating that our method is robust to threshold variations. Furthermore, after determining these thresholds through ablation experiments on the validation set of the V2XSet dataset, we directly trained and tested on the OPV2V dataset under the same threshold settings, achieving state-of-the-art performances. This validates the generalization ability of our method.

Q2: Comparision with previous point-cloud-based methods.

• CPPC vs. Cooper. Cooper is a foundational early collaboration method. Through cooperative sensing, Cooper can merge and align the shared data that is collected from nearby vehicles, which may provide data scopes coming from different positions and angles. Cooper relies on three types of human-designed ROI-based rules to select point clouds to meet actual bandwidth constraints, which may not generalize to complex real scenerios like DAIR-V2X-C. Differently, our CPPC directly transmits sparse point clusters to the ego agent, which achieves the state-of-the-art performance-bandwidth balance.
• CPPC vs. F-Cooper. Both CPPC and F-Cooper are intermediate collaboration methods. Constrained by its voxel-based representation and RPN network, F-Cooper must expand the feature map size during the information aggregation stage to cover the perception range of different agents. This leads to unnecessary computations when collaborative agents are far apart. In contrast, the computational complexity of point cluster aggregation in CPPC depends solely on the number of potential objects, effectively avoiding this issue. We evaluate F-Cooper on the V2XSet and DAIR-V2X-C datasets, achieving an AP@0.7 of 87.06 on the V2XSet test set and 60.31 on the DAIR-V2X-C test set. In comparison, CPPC outperforms F-Cooper with absolute improvements of 2.49 and 9.08 in AP@0.7, respectively.
• Novelty of implementation techniques. The core contribution of our paper is the introduction of a novel point cluster-based communication paradigm for collaborative perception. Building on the concept of point clusters, we develop a comprehensive system, CPPC, which achieves a state-of-the-art trade-off between performance and bandwidth, alongside robustness to various noisies. To implement our approach, we enhance existing technologies (e.g., FPS) due to their wide applicability within the point cloud processing community. Crucially, it is the design of the point cluster that enables the integration of these technologies into the domain of collaborative perception—an advancement that was not feasible with previous methods. We will explore novel implementation techniques which can further improve our CPPC in the future.

Q3: Ablation of the weights of semantic and spatial distances.

• We evaluate different weights, $\lambda_s$ and $\lambda_d$, for semantic and density scores, respectively, with a sample ratio of 1/16 on the test set of DAIR-V2X-C. For simplicity, we set $\lambda_d=1-\lambda_s$. Best performances are achieved when $\lambda_d=0.4, \lambda_s=0.6$. | Method | 0.0 | 0.2 | 0.4 | 0.6 | 0.8 | 1.0 | |------------|-------|-------|-------|-------|-------|-------| | AP@0.5 | 76.14 | 76.28 | 76.41 | 76.47 | 76.32 | 76.16 | | AP@0.7 | 69.01 | 69.07 | 69.22 | 69.03 | 68.84 | 68.62 |

Q4: Crowded environment performances.

• This is a great question. We analyze the target count in DAIR-V2X-C, as crowded environments typically involve a large number of agents. The dataset includes 42 scenarios with 30 or more targets. Our CPPC achieves an AP@0.5 of 70.83 and an AP@0.7 of 63.45, both surpassing the overall performance of existing methods, demonstrating its robustness in crowded environments.

Thank you for your valuable comments and kind words to our work. Below we address specific questions.

Q1: Criteria for determining which points should be selected.

• As shown in Algorithm 1 (Appendix A.1), our proposed SD-FPS strategy prioritizes points in point clusters with clear semantic features or key structural information. Concretely, we measure the ambiguous of each point by its confidence score $s_f$ in segmentation head of PCE, where a higher $s_f$ indicates richer semantic information for distinguishing objects. As for structural measures, the distribution density score $s_d$ is inversely proportional to its density estimation: $\frac{1}{\mathcal{N}(p)}\sum_{q\in\mathcal{N}(p)}K(p,q)$, where $\mathcal{N}(p)$ is a point set in the nearby area of $p$ , and $K(\cdot,\cdot)$ is a gaussian kernel function to measure the similarity between position of two points. Thus, a lower $s_d$ indicates removing it will not significantly affect the object shape.

Q2: Performance degradation when background information is included.

• We include all foreground and background points by setting the foreground threshold in PCE to zero. Experimental results on the V2XSet validation set indicate an AP@0.5 of 91.37 and an AP@0.7 of 89.23, reflecting absolute decreases of 0.64 and 0.76, respectively. The primary cause of this performance degradation is the introduction of background points, which can distort the shape description of potential objects during clustering. However, the negative impact is relatively minor, as the PCP phase retains only point clusters with positive proposals (line 229 in paper). Notably, while the inclusion of background points has limited impact on performance, it substantially increases communication bandwidth requirements.

Q3: Performance analysis in early or late collaboration by only bringing foreground information.

• Early collaboration is expected to achieve the highest performance when bandwidth constraints are not a factor, as it avoids information loss during communication. We agree that transmitting only foreground point clouds is an effective strategy for early collaboration methods to maintain accuracy while reducing bandwidth usage. To validate this, based on the early collaboration method that directly aggregates all raw point clouds, we introduce our trained foreground point segmentation network into the fusion pipeline. By applying an appropriate threshold to filter foreground points, we reduce the communication volume of the early collaboration method to a similar magnitude as other methods, enabling a fair comparison.

• On the DAIR-V2X-C test set, the early collaboration method with foreground filtering achieves 75.01 AP@0.5 and 65.71 AP@0.7, significantly outperforming Where2comm and V2XViT under the similar communication volume. Although the early collaboration with foreground filtering demonstrates a certain level of competitiveness, our approach still exhibited significantly superior accuracy, achieving performance improvements of 1.88 in AP@0.5 and 3.68 in AP@0.7. This improvement is attributed to the introduction of the point cluster feature and the more reasonable keypoint selection strategy employed in SD-FPS. In contrast, later collaboration methods use bounding boxes as message units, which primarily contain only foreground information.

Q4: Sharply rises in Figure 4 (a).

• This is a good question. When the communication volume is limited to approximately 14, the performance of our CPPC drops sharply. We believe that under such a stringent communication bandwidth constraint, the sampled points in the information packaging stage become overly sparse. Consequently, the remaining key points fail to adequately represent the shape characteristics of potential objects, making it challenging to generate high-quality detection boxes.
• When the communication volume is 14, we can calculate the spatial confidence mask rate in the state-of-the-art method, i.e., Where2comm, according to eq (10) in appendix C.1. The number of allowed collaborative message units is calculated: $N=\frac{2^{Comm}\times8}{16\times C}=\frac{2^{14}\times8}{16\times256}=32$, indicating mask rate is 0.9948 for BEV feature maps with shape $48\times 128$. This means that only around 0.5% BEV features are allowed to be transmitted, which are almost equivalent to no collaboration, with 55.54 AP@0.7 on the test set of DAIR-V2X-C.
• Overall, even under such strict communication volume constraints, our CPPC achieves 64.32 AP@0.7, outperforming Where2comm by an absolute margin of 8.78 in AP@0.7. This result validates the superiority of our point cluster-based communication mechanism.

Q5: It is unclear which method the accuracy improvement results are compared with.

• On each dataset, we compare our approach with the current best-performing methods: OPV2V on V2XSet, CoAlign on OPV2V, and V2X-ViT on DAIR-V2X-C.

Q6: Technique of late collaboration method.

• Late collaboration approaches adopt bounding boxes as message units. We implement this late collaboration baseline by directly using $\ddot{B}^s$ for point clusters matching the same potential objects, i.e., in $M_\text{share}$, and $B$ for objects exclusively observed by a single agent, i.e., in $M_\text{unique}$, as the final outputs, respectively. Thus, the late collaboration can not benefit from completing object semantic and structure information from mutli resources in PCA and use PCD (Appendix A.2) to refine them.

Q7: Solution for delay-induced errors that vary by object.

• This is a great question. In this paper, we evaluate the robustness to time latency on the test set of DAIR-V2X-C, which includes 10 km of city roads, 10 km of highway, 28 intersections, and 38 km² of driving regions, encompassing diverse weather and lighting conditions from real-world scenarios. Therefore, we believe this evaluation is sufficiently comprehensive to validate the effectiveness of our latency compensation module.
• However, we acknowledge that our existing solution has limitations. For instance, when objects move at high speeds, the artificially defined upper and lower bounds for temporal cluster matching may struggle to handle such significant displacements. We appreciate you highlighting this issue and propose addressing it by introducing point cluster flow prediction to mining temporal associations in cluster feature space in future works.

Thank you for your valuable comments and kind words to our work. Below we address specific questions.

Q1: Relationship with FSD.

• Single-agent perception vs. multi-agent collaborative perception. The goal of single-agent perception is to enhance perceptual capabilities within the scope of the ego view. In contrast, multi-agent collaborative perception seeks to address the occlusion limitation inherent to single-agent systems by facilitating complementary information exchange between surrounding agents. A key challenge in multi-agent collaborative perception is achieving superior scene-level perception performance under constrained communication costs. The development of multi-agent approaches must address challenges such as packaging observation information in a bandwidth-efficient manner, effectively aggregating multi-source data, and ensuring robustness against issues like pose errors and time latency—challenges absent in single-agent approaches.
• FSD vs. CPPC. We agree that both our CPPC and FSD both utilize sparse point cloud representations for 3D object detection instead of dense BEV representations. However, the contributions differ fundamentally:
  • For single-agent perception, FSD innovatively proposes a fully sparse pipeline, addressing the issue of center feature missing and eliminating the need for time-consuming neighborhood queries in purely point-based methods. It achieves state-of-the-art performance in 3D object detection while being much faster than previous detectors, especially in long-range settings. Although FSD has greatly promoted the development of point cloud-based sparse detectors, it did not consider the communication-related issues faced by multi-agent collaborative perception.
  • For multi-agent collaborative perception, CPPC introduces a novel communication paradigm based on our proposed message unit, i.e., point cluster, which effectively addresses key challenges in existing BEV-based intermediate collaboration approaches. These challenges include object feature degradation during message packing, inefficient message aggregation for long-range collaboration, and the communication of implicit structural representations. First, by prioritizing semantically and structurally rich points in point clusters through our proposed SD-FPS strategy, CPPC achieves the state-of-the-art performance-bandwidth balance. Second, the computational complexity of point cluster aggregation scales efficiently with the number of potential objects in the scene, rather than increasing quadratically with the perception range, making it highly suitable for long-range collaboration. Furthermore, extensive experiments demonstrate that CPPC maintains robustness to a wide range of pose errors and time latency without additional fine-tuning, thanks to the explicitly preserved coordinate information within the point clusters.