Sports-Traj: A Unified Trajectory Generation Model for Multi-Agent Movement in Sports

[Q3] Soccer Metrics and Visualizations

[A3] According to the dataset[3], a soccer field measures 105 meters in length and 68 meters in width, mapped to a pixel resolution of 3840 x 2160. However, the image does not perfectly align with the soccer field dimensions, as noted in the dataset, and only a rough calculation can be made with some margin of error.

· 94.59 pixels converted to the horizontal direction (length): approximately 2.59 meters

· 94.59 pixels converted to the vertical direction (width): approximately 2.98 meters

We have included two visualization examples from the Basketball dataset, along with analysis, in the Appendix of our updated submission. The results show that the trajectories generated by our method are more accurate and smoother compared to the baselines, further validating the effectiveness of our proposed approach. We will include more qualitative results in the final version.

[1] Social-Gan: Socially Acceptable Trajectories with Generative Adversarial Networks. CVPR 2018.
[2] Trajectron++: Dynamically-Feasible Trajectory Forecasting with Heterogeneous Data. ECCV 2020.
[3] SoccerTrack: A Dataset and Tracking Algorithm for Soccer with Fish-eye and Drone Videos. CVPR 2022.

We sincerely appreciate your valuable suggestions and insightful comments. We hope our response effectively addresses your concerns.

Dear Reviewer uin8,

We sincerely appreciate your recognition of our contributions and your constructive suggestions to improve our manuscript. Below, we provide detailed responses to address your concerns.

[Weakness] Real-world Scenarios

[A0] That's a really insightful point, and we agree that addressing problems with erroneous data is a highly valuable direction, especially in the sports domain. While our current work focuses on a relatively simpler problem, we believe it still holds significant value for sports analysis. We have some initial thoughts on addressing the erroneous data scenario:

1. We could develop a probabilistic model to represent the erroneous data and incorporate approaches to quantify the uncertainty in the locations of the agents.
2. Beyond generating trajectories, we could also output the probabilistic distributions and uncertainty measures for these generated locations, which could provide deeper insights for sports analysis.

We will include these points in our final version to better highlight the potential and importance of this problem.

[Q1] Typo

[A1] Thank you for your careful review. That's a typo, it should be "w/o BTS" instead. We have corrected it in the updated submission on Line 474, highlighted in blue. We will thoroughly review and polish the entire draft twice before the final version.

[Q2] Sampling K Trajectories

[A2] That’s a very insightful question. We follow pioneering trajectory prediction works[1][2] in setting K=20 for multiple trajectory generation to account for inherent multimodality, where multiple plausible paths can exist. Based on your suggestion, we evaluated our trained models with different values of K (K=10, K=20, K=30) and reported the mean and standard deviation (std) of all metrics across three datasets. UniTraj* indicates that the results are presented as mean±std.

The results for all five metrics remain very similar across the three datasets with different values of K. In addition, both minADE$_{K}$ and mean ADE are very close, with consistently low standard deviations.

For the baselines, we implemented method MAT, Naomi, INAM, and SSSD for the deterministic generation, we implemented GC-VRNN with $K=20$ by sampling $Z$ from their prior distribution and reporting the minimum metrics.

We will include all these results and further discuss this point in the Experiment Section. Additionally, we will provide supplementary implementation details for the baselines in the final version

Thank you for sharing your initial thoughts on addressing the issue of erroneous data in real-world scenarios. The proposed directions, such as developing a probabilistic model for representing errors and incorporating uncertainty measures into trajectory generation, are intriguing and could significantly enhance the applicability of your work. Please provide more concrete details on these ideas, as they would strengthen the paper and demonstrate the potential of your approach to handle real-world challenges effectively.

Dear Reviewer ZPa2,

We sincerely appreciate your recognition of our contributions and your constructive suggestions to improve our manuscript. Below, we provide detailed responses to address your concerns.

[Q1] LSTM/CNN replace MLP

[A1] If we understand correctly, your suggestion is to replace the MLP decoder with LSTM or CNN, as mentioned in the limitation section. Following your suggestion, we implemented more powerful networks, including LSTM and CNN decoders, to replace the MLP decoder. The results are as follows:

For the w/ LSTM variant, we replace the MLP decoder with a single-layer LSTM with a hidden dimension of 128. For the w/ CNN variant, we replace the MLP decoder with a three-layer Conv2D network, where the first two layers use 64 filters with a kernel size of (3, 1), and the last layer uses 64 filters with a kernel size of (1, 1).

Our results show that using more powerful decoders improves performance across all three datasets. Among the tested decoders, LSTM outperforms CNN, as it better captures temporal dependencies between different time steps. However, since the primary contribution of our work is introducing the new trajectory generation setting, we did not devote too much to optimizing the module designs. We hope that our work inspires the community to develop more advanced network architectures for this task. We'll add those discussions in the final version.

[Q2] Equations

[A2] Thank you for your careful review. We revise Eq. 7 as follows: $\overset{\leftrightarrow}{F}_{bts}= 1/\exp(\varphi_s(\overset{\leftrightarrow}{S};\mathbf{W}_s))$ We will double-check and further polish the draft before the final version.

[Q3] Params and FLOPs

[A3] Thank you for your insightful comments. Below are the results of the number of parameters and GFLOPs on the Basketball dataset, compared with advanced baselines.

Our method has a total of 1.77M model parameters and 0.33 GFLOPs. Among all the advanced baselines, our model has the smallest number of parameters and the second lowest FLOPs, slightly higher than the baseline Naomi. This is because Naomi's backbone is an RNN, which requires fewer computations compared to our Transformer and Mamba structure. These results validate the efficiency of our method. We'll add those results in the final version.

We sincerely appreciate your valuable suggestions and insightful comments. We hope our response effectively addresses your concerns.

Dear Reviewer ZPa2,

Thank you for helping us improve our paper so far. We are very glad that we've addressed your concern. May we know if you have further questions? If not, would you consider increasing the score as a recognition of our rebuttal effort so far? Thank you so much!

Best regards,

Authors of Submission 4628

Dear Reviewer HqF9,

We sincerely appreciate your recognition of our contributions and your constructive suggestions to improve our manuscript. We provide the following detailed responses to address your concerns.

[Q1] Motivation

[A1] That’s a good question. The three tasks, trajectory prediction, imputation, and spatial-temporal recovery, are related but different, with differences in input formats, objectives, and methodologies. For example, trajectory prediction focuses on forecasting future values, imputation deals with filling in missing data, and spatial-temporal recovery simultaneously reconstructs both spatial and temporal patterns. In real-world applications such as sports analytics, these scenarios often occur together, creating a strong motivation for an all-in-one approach. A unified framework would seamlessly address these overlapping demands, reducing the need for separate pipelines and improving overall efficiency. While some related methods [1][2] attempt to tackle prediction and imputation simultaneously, their masking strategies fall short in handling diverse missing patterns. Empirical results further demonstrate the superiority of our proposed method.

We will enhance the introduction section in our final version to further strengthen and clarify the motivation.

[Q2] Challenges

[A2] The challenges of developing an all-in-one method include the following:

1. Different missing data patterns: Trajectory prediction requires extrapolation of future values, imputation often deals with random missing data, and spatial-temporal recovery involves complex dependencies in both space and time.

2. Balancing context and forecasting: A unified model must balance local context modeling for imputation and spatial-temporal recovery with long-term forecasting for trajectory prediction, which is inherently challenging.

3. Dynamic interaction modeling: Player interactions in sports vary significantly across datasets, requiring the model to generalize across diverse scenarios.

These challenges highlight why such an all-in-one approach has not been extensively explored and underscore the contributions of our work in addressing these issues. We will incorporate these discussions into the final version to emphasize the challenges.

[Q3] Comparison

[A3] Thank you for your valuable suggestions. We conducted experiments to separately compare our method with relevant approaches for each task.

For the prediction task, we evaluated our method on the pedestrian datasets ETH-UCY and SDD. The results are shown as follows:

Our method outperforms FlowChain [5] and achieves results comparable to, though slightly worse than, the state-of-the-art baselines MemoNet[3] and EqMotion[4]. However, these methods are challenging to adapt to other tasks, such as trajectory imputation, because their designs are specifically tailored for prediction tasks and are not well-suited for broader applications. Additionally, our focus is on modeling players within the sports domain, which differs from the pedestrian scenarios addressed by these methods.

For the imputation and recovery tasks, we conducted experiments on the recently open-sourced time-series imputation dataset Traffic-Guangzhou, comparing our method with the latest imputation approaches, CSBI[6] and BayOTIDE[7]. The results are shown as follows:

Our method outperforms CSBI[6] in both RMSE and MAE and achieves results comparable to, though slightly worse than, BayOTIDE[7]. One reason for this is that time-series datasets, unlike multi-agent datasets, lack the dense and structured interactions present among sports players. Additionally, these baselines are challenging to adapt to the trajectory prediction task.

In our submission, the baselines we compared, such as INAM[8] and GC-VRNN[1], were originally proposed for joint imputation and prediction tasks, and the empirical results demonstrate the superiority of our method. We will include these experiments and provide additional discussions in our final version to further showcase the effectiveness and generalizability of our proposed approach.

[1] Uncovering the Missing Pattern: Unified Framework Towards Trajectory Imputation and Prediction. CVPR 2023.
[2] Multiple-Level Point Embedding for Solving Human Trajectory Imputation with Prediction. TSAS 2023.
[3] EqMotion: Equivariant Multi-Agent Motion Prediction with Invariant Interaction Reasoning. CVPR 2023.
[4] Remember Intentions: Retrospective-Memory-based Trajectory Prediction. CVPR 2022.
[5] Fast Inference and Update of Probabilistic Density Estimation on Trajectory Prediction. ICCV 2023.
[6] Provably Convergent Schrödinger Bridge with Applications to Probabilistic Time Series Imputation. ICML 2023.
[7] BayOTIDE: Bayesian Online Multivariate Time series Imputation with functional decomposition.ICML 2024.
[8] Imitative Non-Autoregressive Modeling for Trajectory Forecasting and Imputation. CVPR 2020.

We sincerely appreciate your valuable suggestions and insightful comments. We hope our response effectively addresses your concerns.

[Q3] Prediction References

[A3] Thank you for providing those excellent papers relevant to our work. We have compared our results with them in our response [A1] and will also cite and discuss them in the related work section.

Specifically, in[1], a prediction model named EqMotion is proposed, which integrates equivariant geometric and invariant pattern feature learning with an invariant interaction reasoning module, achieving state-of-the-art performance across various tasks. EqMotion is lightweight and effective for diverse motion prediction scenarios.

MemoNet[2] is a trajectory prediction framework inspired by retrospective memory in neuropsychology. It utilizes memory banks to store representative past-future pairs and a trainable addresser to recall relevant instances, enabling more accurate and interpretable predictions. MemoNet achieves state-of-the-art results on multiple datasets, significantly improving prediction accuracy and diversity.

FlowChain[3] is a normalizing flow-based model designed for fast and accurate trajectory prediction and density estimation. By leveraging conditional continuously-indexed flows (CIFs), it can achieve promising performance.

We will include additional references on trajectory prediction in the related work section in our final version.

[1] EqMotion: Equivariant Multi-Agent Motion Prediction with Invariant Interaction Reasoning. CVPR 2023.
[2] Remember Intentions: Retrospective-Memory-based Trajectory Prediction. CVPR 2022.
[3] Fast Inference and Update of Probabilistic Density Estimation on Trajectory Prediction. ICCV 2023.

We sincerely appreciate your valuable comments and insightful suggestions. We hope our response effectively addresses your concerns.

[Q2] Ball and Category

[A2] That’s an insightful point. Following your suggestion, we conducted three experiments:
(1) keeping the ball while removing the offensive/defensive player categories, (2) removing the ball while keeping and concatenating the offensive/defensive player categories,
(3) removing both the ball and the offensive/defensive player categories.
The results are shown in the following tables.

We observe that removing the category information leads to a performance drop, as this information plays an important role in sports. Interestingly, better performance is achieved when removing the ball. A potential reason is that the ball's trajectory is often unstable and influenced by external forces, introducing randomness and outliers that may disrupt the model's ability to learn overall movement patterns. The gap in dynamic characteristics between the ball and player trajectories impacts learning.

Overall, our proposed method achieves the best performance when both the ball and category information are removed. We will include these results for a fair comparison and provide additional analysis in the final version.

Dear Reviewer YWxk,

Thank you so much for your valuable suggestions and detailed comments. We provide the following detailed responses to address your concerns.

[Q1] Generalizability

[A1] Thank you for your constructive comment. To assess the generalizability of our method, we followed your suggestions and conducted experiments on the ETH/UCY and SDD datasets for the trajectory prediction task. The results of stochastic predictions with K=20 are shown below:

Our method demonstrates better performance than FlowChain[3] and achieves results comparable to, though slightly worse than, the state-of-the-art baselines MemoNet[2] and EqMotion[3].

One reason is that our modules are specifically designed for sports datasets, which feature more structured interactions among players, whereas pedestrian movement patterns tend to be more casual and random. Despite this difference, our results show that the proposed modules effectively capture spatial-temporal features from observed trajectories, further validating the generalizability of our approach.

Additionally, our method is applicable to other trajectory-relevant tasks. In the final version, we will include more baseline comparisons for trajectory prediction to further emphasize the generalizability of our method.