Drones Help Drones: A Collaborative Framework for Multi-Drone Object Trajectory Prediction and Beyond

[Weakness 1]: Grammar mistakes

We have carefully reviewed and corrected the grammar mistakes, typos, and formatting errors.

[Weakness 2]: Transmission volumes. Why the results of DHD in Table 2 bold?

Based on the V2V communication protocol [1], data broadcasting can achieve a bandwidth of 27 Mbps at a range of 300 meters. Considering the capture frequency of 2 Hz, the bandwidth requirement should be less than 13.5 Mbps to avoid severe delay. The original BEV feature map is 64 x 200 x 200 in size, corresponding to 9.77 MB. The result-level fusion utilizes the output segmentation and offset maps, which are 2 x 200 x 200 in size, corresponding to 0.305 MB. We calculate the transmission expenses for each method, as shown in Table VI of the attached PDF.

All partially connected methods of intermediate collaboration meet the above bandwidth requirements. Notably, although V2X-ViT and V2VNet achieve better performance, their high transmission expenses make them less feasible for practical applications. By the way, our proposed DHD has shown the best performance within the partially connected approaches for intermediate collaboration, so we highlight it in bold.

[1] Arena F, Pau G. An overview of vehicular communications[J]. Future internet, 2019, 11(2): 27.

[Weakness 3]: Exploring the number of drones varies from 1 to 4?

To explore how the number of collaborating drones affects performance, we conduct relevant experiments, as shown in the overall rebuttal.

For the short-range area of 50mx50m, three drones are sufficient to predict trajectories, with performance comparable to that of four drones. However, for the long-range area of 100mx100m, predictive performance improves as the number of drones increases. This difference arises because the drones in our dataset are positioned near the intersection to monitor traffic flow, making it easier to cover areas close to the intersection, which fall within the short-range area. In contrast, much of the information in the long-range area extends along specific road branches, which are only partially captured by the drones at the intersection. Therefore, having more drones results in more comprehensive coverage of the long-range area. Conversely, the reduction in the number of drones does indeed reflect the influence of potential drone failures on performance.

[Weakness 4]: Explore how the extent of extrinsics bias affects collaborative prediction?

We have conducted a series of experiments by introducing Gaussian noise to the extrinsic parameters of drone cameras, as shown in Fig. 2 of the attached PDF document and described in Answer 1 of the overall rebuttal.

[Question 1]: More details about Air-Co-Pred? Why are totally four drones?

The dataset is gathered using four drones, each with a field of view set to 70 degrees by default. The drones are positioned at an altitude of 50 meters, monitoring traffic flow at urban intersections. The speed distribution of the observed objects ranges from 0 to 10 m/s, with low-speed states primarily concentrated around 1.5 m/s and high-speed states around 8 m/s. In the CARLA simulations, speed is influenced by several factors, including the default safe speed settings and the necessity of complying with traffic regulations and preventing collisions.

Furthermore, the four drones are arranged in a square formation to achieve a square-shaped coverage area for comprehensive aerial observations. This configuration allows for overlapping fields of view to mitigate occlusions, and the unique observations from each drone enrich the out-of-sight perception capabilities of the entire system.

[Question 2]: The detailed derivation process.

According to the pinhole camera model, the transformation between the world coordinate system and the pixel coordinate system can be described by :

Next, we consider ${M} = {R}^{-1}{K}^{-1} $ and $ {N} = {R}^{-1}(-{T}) $ for further simplification and derivation:

By setting $ y = -H $, the depth upper-bound can be derived as:

[Question 3]: Clarify the differences with related works.

Indeed, these two works leverage the camera parameters of UAVs to enhance cross-view association. Specifically, they project 2D detection results from each perspective into a unified 3D space with camera parameters and associate targets based on feature similarity and spatial distance. In contrast, our proposed GBG module uses UAV camera parameters along with the available flight altitude to derive the depth upper bound of each pixel. This geometric prior helps to constrain depth estimation and calculate the viewing angle of each pixel, facilitating the subsequent BEV generation. By acknowledging these works and clarifying our distinct approach, we highlight the unique contributions and applications of the GBG module in multi-drone collaboration.

[Question 4]: Additional details or codes.

We have already provided the core code in the initial supplementary materials. Additionally, to ensure better understanding and reproducibility, we plan to release the Air-Co-Pred dataset, the source codes of the complete project, and the model checkpoints, which include our methods and previous baselines. This will contribute to further reproduction and improvement of the system.

[Weakness 1]: Questions about the framework's performance in more complex, real-world scenarios with sensor noise, communication delays, environmental variability, and other common difficulties in real-world experiments.

To address the reviewer’s concerns about the gap between real-world scenarios and simulated data, we have incorporated some of the mentioned challenges into our experimental setup to assess their impact on the performance of multi-drone collaborative prediction. Specifically, we examine the effects of sensor noise, which primarily affects the acquisition of camera extrinsics, as well as environmental factors such as drone vibrations and rough terrain, which can influence the drone's relative height to the ground plane. To evaluate the impact of these factors on performance, we conduct complementary experiments by introducing Gaussian noise into these aspects. The statistical results are shown in the attached PDF, and a detailed analysis is provided in Answer 1 of the overall rebuttal.

As to communication delays, we propose integrating a delay assessment module into the collaborative interaction stage of DHD. This module would compensate for lost or delayed frames by leveraging historical collaborative features. Moreover, the compensation process can be streamlined by prioritizing critical areas using the information volume assessment component in the SISW module. At the downstream decoder, missing predictions can also be filled by blending predictions from adjacent time frames and extrapolating trends accordingly.

[Weakness 2]: How does the performance scale with the number of collaborative drones? Is there an optimal or maximum number of drones for effective collaboration?

To explore how the number of collaborating drones affects performance, we conducted relevant experiments, as shown in the table below:

For the short-range area of 50m x 50m, three drones are sufficient to predict trajectories, with performance comparable to that of four drones. However, for the long-range area of 100m x 100m, predictive performance improves as the number of drones increases.

This difference can be attributed to the positional layout of the drones in the dataset, Air-Co-Pred. The drones are situated near intersections to monitor traffic flow, facilitating comprehensive coverage of the short-range areas close to these intersections. In contrast, much of the information in the long-range areas extends along specific road branches, which may only be captured by a single drone. Therefore, increasing the number of drones results in more comprehensive coverage of the long-range areas.

[W1 & Q1]: Differences with existing methodologies.

Our DHD builds on existing solutions, but we make significant improvements to address unique challenges in multi-drone collaborative prediction. For instance, long-range aerial observations can lead to substantial depth estimation errors, affecting BEV representations. Additionally, objects in view are typically small and sparse, with critical information making up only a minor portion. However, previous methods, primarily focusing on collaborative detection, have not adequately considered the selection and utilization of sparse information for joint prediction. Our DHD framework is specifically designed to overcome these challenges, distinguishing our approach from others.

Regarding the drone-specific BEV generation module, GBG, while it is based on [13], we introduce innovations of geometric priors to refine depth estimation, as acknowledged by Reviewer CzWU rEnT vjyF. Moreover, the improvements we've made in the LSS method are also applicable to other depth-based BEV generation techniques. As for the similarity with [14], our SISW module focuses on areas with significant feature variations for subsequent collaborative feature fusion, which are likely to contain objects or critical environmental information essential for prediction tasks. This solution is not considered by previous collaborative strategies, including [14]. In terms of implementation, the sliding window mechanism for information volume assessment is a major contribution of our SISW module, as noted by Reviewer CzWU nEkT. Additionally, for the selective sparse interaction component, we optimize the selection mechanism by replacing traditional threshold-based filtering with score ranking. This modification allows us to precisely control transmission volume based on predefined ratios, rather than relying on manually-tuned empirical thresholds.

For the feature fusion module, while not our primary innovation, our approach differs significantly from [14]. Instead of their attention-based fusion, we use a multi-layer convolutional method to calculate contribution weights for each collaborative feature guided by local features. Theoretically, attention mechanisms are better suited for global feature associations, while the convolutional paradigm is widely recognized for its strength in preserving local features. Since our instance-level prediction task benefits more from strong local feature representations, we opt for this fusion method and provide additional experiments to demonstrate performance differences.

[Weakness2]: A comparison with other datasets.

Please refer to Answer 3 of the overall rebuttal. Additionally, we provide a detailed comparison of key attributes in the table below:

Cam. Params. & Coord. Info. refers to camera parameters and coordinate information, which are critical for BEV generation.

[W3]: Additional datasets. (e.g., DAIR-V2X, V2XSet and OPV2V datasets).

The benchmarks you mentioned are primarily designed for collaboration in autonomous driving scenarios, where the viewing angles of vehicles are horizontal. This contrasts sharply with the aerial oblique perspectives in our multi-drone collaboration research, making our drone-specific BEV generation module, GBG, less applicable to these benchmarks. Besides, in the initial manuscript, we provided experimental results on the publicly available CoPerception-UAVs dataset in Table 4 of the initial manuscript, to demonstrate the generalization capabilities of our framework for collaborative detection.

We understand your concern about the reliance on simulated multi-drone datasets. In fact, we are currently developing a real-world multi-drone collaboration dataset to further validate and extend our work in future studies.

[W4]: Reasons for separating V2X-ViT and V2VNet from others.

Benefiting from the fully connected paradigm, V2X-ViT and V2VNet achieve complete feature-level interaction among collaborators and employ more complex techniques for collaborative feature fusion. As a result, it is reasonable for them to demonstrate superior performance compared to the partially connected paradigm.

From the perspective of communication overhead, it is logical to distinguish V2X-ViT and V2VNet from other communication-efficient methods for a fair comparison. Notably, our proposed SISW outperforms current partially connected methods and is comparable to the fully connected paradigm.

[W5]:

Thank you for your suggestion regarding the content layout.

[W1 & L1]: Consider variations in terrain, drone stability, and other environmental factors.

Please refer to Answer 1 in the overall rebuttal.

[W2]: Analysis of the computational efficiency and scalability of the overall framework

As seen in Table VI of the PDF, the differences in trainable parameters (which reflect model complexity) and model parameter sizes (which impact memory usage) among the compared methods are not significant, except for Who2com and When2com. Notably, the FLOPs for V2VNet and Late Collaboration are significantly higher than those of other baselines. This is due to the iterative processes involving multiple for-loops for information exchange, which generate multiple parallel paths during computation. Each branch performs independent calculations before merging the results, significantly increasing computational complexity. The same principle applies to the difference in FLOPs between our DHD framework and Where2comm.

For the investigation of stability, we explore the number of collaborative drones in Answer 2 of the overall rebuttal.

[W3]: Additional datasets for validation.

Please refer to Answer 3 in the overall rebuttal.

[W4]: SISW might oversimplify the information selection process. Can it handle highly dynamic or cluttered environments?

Our proposed SISW module is specifically designed with the characteristics of aerial perspectives, where targets are often small and sparse, and surrounding environmental textures can guide trend prediction. Specifically, SISW employs a sliding window mechanism to assess and select critical information. This design is simple yet efficient, outperforming previous sparse interaction baselines.

Although the SISW module is not primarily designed for highly dynamic or cluttered environments, our dataset, Air-Co-Pred, inherently includes such scenarios, thereby enhancing the module's adaptability and robustness. Specifically, our dataset contains substantial moving vehicles, which aligns with the highly dynamic scenarios mentioned. As illustrated in Figure 6 of the initial manuscript, the low proportion of targets within the field of view indicates a high presence of clutter. Furthermore, our adopted BEV representations help filter out much of the clutter in 2D vision during spatial projection by truncating the z-axis.

Therefore, the SISW module proposed in this paper can be considered applicable to dynamic and cluttered environments to a certain extent.

[W5]: A lack of BEV baselines.

Thank you for your feedback. In our work, we focus on optimizing depth estimation from aerial perspectives within the BEV generation process. The improvements we've made in the GBG module are indeed applicable to other depth-based BEV generation methods. We recognize the importance of comparing our method against a broader range of baselines.

To provide additional comparisons, we include methods like BEVDet4D, which considers temporal BEV representations, and BEVerse, which applies multi-task learning to improve BEV generation. As shown in Table V of the PDF, while multi-task learning offers minimal performance gains, incorporating temporal information does enhance BEV generation. However, our DHD framework with the GBG module, which leverages geometric priors, still outperforms these methods.

[W6]: Calculate the average displacement.

As illustrated in Table II of the PDF, a lower ADE does not necessarily indicate better prediction performance. For example, "No Collaboration", which relies on single-view observations, misses a significant number of objects. However, its calculated ADE is relatively low because this metric only considers matched trajectories. Similarly, methods that performed well in our initial experiments, such as Early Collaboration, V2VNet, V2XViT, and our proposed DHD, exhibit higher ADE compared to Where2comm and UMC. This is partly due to Where2comm and UMC having a higher missing ratio, which excludes some difficult trajectories from the ADE calculation. Therefore, when using ADE as a metric, it's important to also consider the ratio of missing trajectories. This is why we adopt evaluation metrics such as IoU and VPQ, as suggested in the research of our adopted predictor ''PowerBEV''. These metrics provide a more comprehensive assessment by considering missed detections, false positives, and the deviation between predicted and ground truth positions.

[W7]: Typo errors.

We have noted the incorrect usage of quotation marks and will correct it in the revised version.

[W8 & Q1 & L2 & L3]: Additional component ablation study about SISW module.

Regarding your comment that "SISW has shown little improvement, and the performance declines when the transition rate is 0.25," the SISW module is specifically designed to preserve critical information for the prediction task during sparse collaborative interactions. This concept is also positively acknowledged in Strength 2 of Reviewer nhVA's feedback. Given that our method transmits only a subset of features, it is understandable that there may be a slight performance decline compared to the complete feature transmission.

We have enriched the ablation studies as you suggested in Table IV of the PDF. Notably, when the SISW module is absent, we resort to randomly selecting 25% of spatial features for transmission to achieve spatial sparsity. The additional experiments show that when the transmission ratio is close to 1, our DHD framework, which includes both the GBG and SISW modules, provides a performance gain compared to the variant with only GBG, particularly in the VPQ metric. This improvement is attributed to the collaborative feature aggregation component in SISW, which enhances the weight of features that are beneficial for downstream prediction tasks.

The enhanced ablation experiments, which consider both module-level and transmission-level factors, clearly demonstrate the effectiveness of both the GBG and SISW modules.

Dear Reviewer nEkT. We hope that our rebuttals have clarified your concerns. if there are any specific analyses or complementary experiment that could clear your doubts, we would be happy to try and provide them. We sincerely thank you again for your time and feedback.

[W1]:

We envision a scenario where vehicles are centrally coordinated. In this setting, collaborative drones predict abnormal trajectories that may lead to accidents. Upon identifying the risks, the drones use wireless communication to alert nearby vehicles, prompting them to take evasive actions to prevent accidents. The system can also integrate with LLMs to provide solutions. Besides, it can be used to forecast traffic congestion, offering lane recommendations.

[W2]:

As to the concern about obtaining UAV coordinate systems in real-world scenarios, these parameters are considered readily available data sources in [1].

Regarding the pixel-level weight fusion in SISW, the reviewer's concern may stem from the computational overhead. Our approach focuses on fine-grained feature fusion, which is indeed more complex than simple addition or concatenation. As shown in TABLE III of the PDF, we supplement the results with common fusion methods. As expected, they perform worse than ours.

[1] Zhao M, Chen J, Song S, et al. Proposition of UAV multi-angle nap-of-the-object image acquisition framework based on a quality evaluation system for a 3D real scene model of a high-steep rock slope[J].

[W3]:

Below is a description of other CARLA-based datasets.

For Autonomous Driving:

• OPV2V: Focuses V2V collaborative perception to enhance safety and efficiency in autonomous driving.
• OPV2V-Occupancy: An extension of OPV2V, it introduces an occupancy grid map to represent the distribution of surrounding obstacles.
• V2X-Sim: Simulates both V2V and V2I modes, covering a range of complex scenarios. However, it does not simulate noise environments and includes only a single type of road.
• V2XSet: Simulates real-world noise in V2X collaboration.
• V2XSet-w: An extended version of V2XSet, it includes more adverse weather conditions.
• DeepAccident: Generates various traffic accident scenes and records relevant sensor data. This dataset is intended for traffic accident analysis and prevention.

For UAV Research:

• CoPerception-UAVs: Refer to Answer 3 in the overall rebuttal.

It is evident that the key difference between our Air-Co-Pred and other CARLA-based datasets is our aim to solve fundamentally different problems from the outset. Furthermore, in terms of dataset characteristics, unlike the horizontal perspective of autonomous driving scenes, our multi-drone dataset focuses on aerial observations with oblique views over wide ranges. Additionally, compared to CoPerceptionUAVs, our dataset supports additional temporal tasks such as prediction and tracking.

[W4]:

• CoBEVFlow: It addresses the challenge of asynchrony in collaborative perception by focusing on robustness to temporal misalignments. Although our current work does not account for asynchrony, the proposed solutions in their study provide valuable insights.
• HEAL: Lu et al. propose a framework for heterogeneous collaborative perception, focusing on collaboration among diverse data sources and perception models. Our work could benefit from their approach when dealing with heterogeneous members in collaboration.
• SimAug: This research explores learning robust representations for trajectory prediction using rich simulated data from CARLA. Specifically, they identify hard examples from multi-view information and combine them with the original view through adversarial learning to better adapt to real-world environments.

[Q1]:

Refer to Answer 2 of the overall rebuttal.

[Q2]:

• Why CARLA? CARLA offers the flexibility to create complex urban traffic scenarios, providing rich dynamic objects along with automated annotations, including the camera intrinsics, extrinsics, and coordinate information needed for BEV generation. It can generate extensive data for algorithm validation with minimal time and manpower investment.
• Is it feasible for multi-drone trajectory prediction? To the best of our knowledge, real-world trajectory prediction with multiple drones is still in an early stage. The most relevant multi-drone collaborative datasets, such as VisDrone-MDOT and VisDrone-MDMT, are primarily collected for tracking. However, their temporal continuity and ID information make them adaptable for trajectory prediction, indicating the feasibility of using multiple drones for joint prediction.
• Why 50m? At this altitude, four drones near intersections can collaboratively cover an area of around 100mx100m. This height also results in significant occlusions, such as vehicles being obscured by buildings or trees, which we aim to address through multi-drone collaboration.

[Q3]:

We set the interaction frequency to match the sampling frequency of 2 Hz. However, we can explore reducing this by leveraging recent multi-drone collaboration predictions, such as instance segmentation and offsets, to extrapolate trends and strategically fuse them with current single-view predictions.

[Q4]:

Refer to Answer 1 of the overall rebuttal.

[W2.1 & Q5]:

Refer to TABLE IV of the PDF.

[Q6 & Q2.4]:

Before DHD can be deployed in real-world applications, several steps are necessary:

• Robustness: Can handle sensor noise and communication delays, which are common in real-world scenarios.
• Scalability and Flexibility: Adapt to the varying number of drones and remain resilient to drone failures or disconnections.
• Environmental Adaptability:: Account for weather conditions, lighting variations, and complex terrains.
• Real-Time Performance: Optimize for lightweight deployment on edge devices with high FPS.

[Q7]:

To the best of our knowledge, we are the first to employ an end-to-end approach that combines multi-drone joint observations for trajectory prediction. Additionally, we provide a simulated multi-drone dataset to support this work, which can be used for various tasks and serves as a benchmark for future research in multi-drone collaboration.