Q1: Demonstrate the generalization capability to 2D datasets on more baselines.

A1: We evaluated our prediction strategy on four additional baselines using two widely adopted 2D datasets: ETH&UCY and SDD. All models were trained and tested on the same machine for fair comparison.

These results demonstrate that our approach consistently enhances the performance of multiple baselines, further validating its generalization capability to 2D trajectory prediction.

---

Q2: Report the accuracy for each prediction step.

A2: The prediction accuracy at each step for the baseline LBEBM and our method is presented in the table below. Additionally, a visualized line chart of these results is provided; Please refer to Figure-2 at the URL https://anonymous.4open.science/r/ICML_ID4436/README.md.

The results and their line chart indicate that our method achieves progressively more significant improvements over the baseline as the prediction horizon extends. This highlights our approach’s ability to mitigate the error accumulation problem in trajectory prediction, further validating its capability to reduce prediction complexity.

---

Q3: Discuss more recent work, for example, [1]-[4].

A3: We will incorporate a discussion of the recent works [1]-[4] in our manuscript.

---

Q4: It will be better to report some failure cases.

A4: We visualized a representative failure case in Figure-3 at https://anonymous.4open.science/r/ICML_ID4436/README.md, showing that our method struggles with trajectories featuring multiple sharp bends in short time frames. A more advanced interaction modeling could improve such case. However, as our primary focus is reducing the prediction complexity of 3D trajectories, we adopt a simple interaction modeling strategy, leading to suboptimal performance in this case. In the future, we will explore specialized 3D interaction modeling for complex motion patterns.

---

Q5: Maybe we can try GRU instead of LSTM.

A5: We evaluated the impact of replacing LSTM with GRU in our method, and the results are presented below:

The results indicate that GRU performs slightly worse than LSTM in our method. However, considering GRU’s higher computational efficiency, we recommend using the GRU-based version for low latency or resource-limited scenarios.

---

Q6: The content of Section 4.1 does not precisely match with Figure 4.

A6: Sorry for the typos in Section 4.1; we will refine them to ensure alignment with Figure 4.

---

Q7: Report the profiled results (latency, power, etc.).

A7: We compared our method’s profiled results with several top-notch methods. Specifically, we tested all models on an NVIDIA 2080 Ti GPU using an input size of 70×8x3, where 70 represents agents' number predicted simultaneously—exceeding the agent count in most real-world applications.

The results show that our method achieves the best performance with a relatively good model efficiency, making it suitable for deployment on embedded robotic systems. Additionally, with an inference speed exceeding 12 FPS, our method meets the real-time decision-making requirements of robotics.

---

Q8: Change the iteration and layer numbers on different scenes or datasets?

A8: It is advisable to adjust these hyperparameters for scenes with significant variations, as different environments impact agent movement to varying degrees. For instance, underwater environments introduce more excellent resistance and turbulence than aerial environments, necessitating different hyperparameters for optimal performance.

---

Reference

[1] Feng L, et al. Unitraj: A unified framework for scalable vehicle trajectory prediction. ECCV. 2024.

[2] Xu Y, et al. Adapting to length shift: Flexilength network for trajectory prediction. CVPR. 2024.

[3] Heravi M Y, et al. Deep learning-based activity-aware 3D human motion trajectory prediction in construction. ESWA. 2024.

[4] Shi Z, et al. Design of a UAV Trajectory Prediction System Based on Multi-Flight Modes. Drones, 2024.

Q1: I look forward to seeing future validation on the trajectory dataset of unmanned aerial vehicles.

A1: As outlined in our conclusion, future work will involve collecting a large-scale 3D trajectory dataset from unmanned aerial vehicles (UAVs) to validate our proposed methodology further. Specifically, we plan to capture trajectories across nine distinct environments, including large-scale indoor mall scenarios, urban road airspace scenarios, urban low-altitude logistics corridors, dense vegetation agricultural fields, large industrial park settings, cross-sea bridge inspection sites, post-disaster debris zones, open water airspace environments, and wilderness forest environments. For each environment, we aim to collect six hours of UAV trajectory data, allocating three hours for training, two for validation, and one for testing.

At the current stage, we have collected two environments—large-scale indoor mall scenarios and urban road airspace scenarios—though annotations remain incomplete. In the indoor mall scenario, three or four UAVs simulate coordinated formation movement to mimic goods pickup and delivery tasks. In the urban road airspace scenario, four or six UAVs navigate complex intersections to simulate 3D logistics transmission environments. To evaluate our method on a more general 3D dataset, we randomly selected one hour of annotated data from these two scenarios (30 minutes for training, 20 for validation, and 10 for testing) and benchmarked our approach against other state-of-the-art methods.

The results demonstrate that our method outperforms all investigated approaches, validating its effectiveness in diverse 3D scenarios. Besides, our method performs best at urban road airspace scenarios, demonstrating the method's ability to predict complex non-formation trajectories. Moreover, our approach performs better in UAV-based 3D environments than UUV-based underwater scenarios, further confirming the greater complexity of underwater trajectory prediction due to higher resistance and turbulence. We have also visualized several predicted trajectories in Figure-4 and Figure-5 at the anonymous URL https://anonymous.4open.science/r/ICML_ID4436/README.md, further illustrating the ability of our method in general 3D trajectory prediction.

---

Q2: Evaluating the computational cost and runtime performance.

A2: We compared our method’s computational cost, parameter, and runtime performance with several state-of-the-art methods. Specifically, we tested all models on an NVIDIA 2080 Ti GPU using an input size of 70×8x3, where 70 represents the number of agents predicted simultaneously—exceeding the agent count in most real-world applications. The results are presented below:

The results demonstrate that our method achieves the best performance with a relatively good model efficiency, making it suitable for deployment on embedded robotic systems. Additionally, with an inference speed exceeding 12 FPS, our method meets the real-time decision-making requirements of robotics applications.

Q1: The dataset is simulation data and has not been validated in real-world scenarios.

A1: While our dataset is based on predefined formation trajectories, real-world disturbances naturally cause deviations from planned paths and formation shifts. These deviations reflect real challenges in trajectory prediction. Moreover, most real-world robotic trajectories are pre-programmed rather than spontaneously generated, meaning our dataset aligns with actual robotic motion patterns, making it representative of real-world scenarios.

---

Q2: Validating the proposed method against top-notch methods on other 3D datasets.

A2: It is hard to find publicly available 3D trajectory dataset. We plan to collect a more general 3D trajectory of unmanned aerial vehicles (UAVs) to validate our method. Specifically, we will capture trajectories across nine environments from the civilian field. Currently, we have collected two environments: large-scale indoor mall scenarios and urban road airspace scenarios. We randomly selected one hour of annotated data from these two scenarios and benchmarked our method against top-notch methods. For more experimental details, please see Q1 of Reviewer cMBX.

The results show that our method outperforms all others, validating its effectiveness in diverse 3D scenarios. The visualized results in Figure-4 and Figure-5 at URL https://anonymous.4open.science/r/ICML_ID4436/README.md further illustrate its general 3D prediction capability.

---

Q3: Many 3D trajectories are military datasets with limited application scenarios.

A3: 3D trajectory prediction has broad civilian applications. Beyond UAV navigation and obstacle avoidance, it supports urban air mobility, precision agriculture, disaster response, and industrial inspections, ensuring safe and efficient operations. These applications extend its impact far beyond military use.

---

Q4: The claim that reduces the total number of parameters is misleading.

A4: Yes, our decoupling strategy simplifies the optimization process but does not reduce optimized parameters. We will clarify this in our paper.

---

Q5: The effects of modifying the data enhancement strategies of compared methods.

A5: To ensure a fair augmentation, we conducted a comparative study of 2D augmentations (rotation, flipping, and translation) versus their 3D counterparts across several top-notch methods.

The results show that 3D strategies provide slight improvements on others but significantly enhance our method, suggesting that 3D strategies are particularly beneficial for our model. For fairness, we use standard 2D strategies with 3D inputs for all methods.

---

Q6: Discussion of the SC and SA modules separately.

A6: We conducted separate ablation studies on the SC and SA modules.

The results indicate that removing the SC or SA module degrades the model's performance, highlighting their contributions.

---

Q7: Generalization to 2D Datasets with more baselines.

A7: We evaluated our prediction strategy on four additional baselines using 2D datasets: ETH&UCY and SDD. To ensure a fair comparison, all models were trained and tested on the same machine.

These results show that our approach consistently enhances the performance of multiple baselines, further validating its generalization capability to 2D trajectory prediction.

---

Q8: Refine the implementation principles of the SA and SC modules in Figure 4.

A8: We added the implementation details of the SA and SC modules in Figure 4. Please refer to Figure-1 at URL https://anonymous.4open.science/r/ICML_ID4436/README.md.

---

Q9: How does your method perform in real-world scenarios?

A9: As discussed in Q2, our method has been evaluated in urban road airspace scenarios, where groups of UAVs navigate complex intersections to simulate 3D logistics transmission. These scenarios involve complex interactions. Our method performs best in this environment, demonstrating its scalability to real-world complexities.

Q1: The paper tries to mitigate the computational complexity but has not provided ablations or method comparisons on inference cost.

A1: At first, we clarify that the complexity our paper aims to mitigate is prediction complexity, which is crucial for optimizing 3D trajectory prediction. However, this is distinct from computational complexity. It is our fault for misunderstanding. To avoid any confusion, we will explicitly define the term of prediction complexity in our paper. Theoretical analysis in the Appendix and ablation studies confirm that our method effectively reduces the prediction complexity of 3D trajectories.

Computational complexity is indeed an important aspect to evaluate the proposed method; therefore, below we compared inference costs across several state-of-the-art methods using an NVIDIA 2080 Ti GPU. To ensure fair comparisons, all tested methods were modified to accept inputs of shape 70×8×3.

These results show that our method performs best while maintaining relatively high model efficiency, making it suitable for deployment on embedded robotic systems.

---

Q2: Extending 3DMoTraj dataset to general 3D scenarios is difficult due to underwater dynamics.

A2: From the methodological perspective, our motivation is to reduce the prediction complexity of 3D trajectory in a manner that ensures robustness across diverse scenarios. The 3DMoTraj dataset is only one of the validation scenes for our method.

To further establish its generalizability, we are constructing a large-scale 3D trajectory dataset for unmanned aerial vehicles (UAVs). Preliminary results from this dataset confirm that our method extends effectively to general 3D trajectory prediction. Please refer to Q4 below for more details.

---

Q3: Further discussions on trajectory refinement works, such as MTR, QCNet, SmartRefine, etc.

A3: We modified the trajectory refinement components of MTR [1], QCNet [2], and SmartRefine [3] by replacing them with the refinement module in our method.

The results show that our SCA-LSTM performs best in refining initial predictions. This outcome is expected, as SCA-LSTM explicitly models the inter-axis correlations of predicted trajectories, which are intentionally ignored in the decoupled prediction stage to simplify optimization. In contrast, the refinement modules of MTR, QCNet, and SmartRefine focus on modeling interactions between predictions, global intentions, and local maps for refinement.

We will further elaborate on the differences between our method and other 2D refinement-based methods in our paper.

---

Q4: Provide a discussion on 3D trajectory dataset collected from unmanned aerial vehicles.

A4: As outlined in our conclusion, future work will involve collecting a large-scale 3D trajectory dataset from unmanned aerial vehicles (UAVs) to validate our method further. Specifically, we plan to capture trajectories across nine distinct environments. For each environment, we aim to collect six hours of trajectories, allocating three hours for training, two for validation, and one for testing.

At the current stage, we have collected two environments—large-scale indoor mall scenarios and urban road airspace scenarios—though annotations remain incomplete. We randomly selected one hour of annotated data from these two scenarios (30 minutes for training, 20 for validation, and 10 for testing) and benchmarked our approach against other state-of-the-art methods.

The results demonstrate that our method outperforms all the competitors, validating its effectiveness in diverse 3D scenarios. We have also visualized several predicted trajectories in Figure-4 and Figure-5 at the anonymous URL https://anonymous.4open.science/r/ICML_ID4436/README.md, further illustrating the ability of our method in general 3D trajectory prediction.

---

Reference

[1] Shi S, et al. Motion transformer with global intention localization and local movement refinement. NeurIPS, 2022.

[2] Zhou Z, et al. Query-centric trajectory prediction. CVPR. 2023.

[3] Zhou Y, et al. Smartrefine: A scenario-adaptive refinement framework for efficient motion prediction. CVPR. 2024.