Mixed Gaussian assumption: Our motivation is the difficulty of transforming a single-modal and symmetric original distribution to a complex target distribution by normalizing flow. We seek a solution to relieve the difficulty. We choose the problem setting to better provide a fair quantitative evaluation protocol. In the area of trajectory forecasting, the problem setups are different. Most works do not consider the map information (at least on the benchmarks of ETH/UCY and SDD) while some others consider the map information. We agree that map information and other factors can be important in practice, but we focus on the simple convention without considering them because (1) we follow the convention with the most baseline works to compare with; (2) our innovation is mainly about enhancing the expressiveness of complicated target distribution of normalizing flow. We would like to choose the setting with the minimum necessary factors to avoid irrelevant inductive bias, such as map information encoder design in different baseline works. Adding other conditional information will steer our focus to a different problem setting, though worth studying for practical purposes, which may confuse the contribution and intention of our work.

APD/FPD not reasonable: Without other information, such as the obstacle in the scene, we can't judge if a trajectory is reasonable or not. Our focus in this work is to design a method with a better diversity of the generation results by normalizing flow. The fundamental is to enhance the expressiveness when transforming to a complicated target distribution. I believe that can't say that the long-standing task of motion generation/forecasting is meaningless if no scene information is considered as it is a good platform to study the generative models' capacity as a probabilistic time series prediction task. Further adding other information can make the solutions from this task setting better fit with practical situations.

We agree that a single APD/FPD is not a perfect metric, it should be used along with the ADE/FDE. For example, equally cover all directions as a solution can reach high APD/FPD, but would probably cause low ADE/FDE which are the main metrics in the area of trajectory forecasting for many years.

The APD/FPD metrics are more inspiring through the new ablation provided in the pdf file (Table 1&2). It highlights that mixed Gaussian prior can significantly increase the diversity compared to the single Gaussian prior, while also boosting the accuracy measured by ADE/FDE even though the two sets of metrics are usually considered a hard trade-off.

Missing recent works: We appreciate the mentioned recent related works. As we emphasize in the appendix, it is chaotic for benchmarking in the area of trajectory forecasting that different methods use different data pre-processing and evaluation conventions but compare together. We made intensive efforts to present the different conventions and choose the most widely adopted one for our benchmarking. Please refer to Appendix A in the supplementary for details. We did not include some mentioned related works in the main paper due to this reason:

1. Ynet[1] and Yue, et al[2] use a different data convention from the widely adopted Social-Gan data convention. We explain this in Appendix A (Lines 575-577). So we can't include them in the benchmark.
2. [3] only provides NBA dataset processing details. In their GitHub code, they only provide the training and evaluation on the NBA dataset. It is not able to know the details of evaluation/training on the ETH/UCY dataset.

There are some recent works that we should have included, such as [4,5]. We will add these works to the comparison.

We agree that it would be more convincible to compare more related works following their separate data pre-processing and evaluation conventions. However, this requires us to train and evaluate our model in at least seven different evaluation protocols. We choose the benchmarking convention following Social-GAN which includes most previous arts as we believe it has the best representativeness. Moreover, many works such as [3] provide no details, code, or processed data for us to know their data processing and evaluation protocol. Therefore, this is impossible for us to compare with these works fairly. We may benchmark under other data conventions if enough baselines can be gathered to provide a more complete sense of our method's performance.

[1] "From goals, waypoints & paths to long term human trajectory forecasting"

[2] "Human trajectory prediction via neural social physics` share the same data with Ynet"

[3] "Leapfrog diffusion model for stochastic trajectory prediction"

[4] "Sparse instance conditioned multimodal trajectory prediction"

[5] "LG-Traj: LLM Guided Pedestrian Trajectory Prediction"

Not an appropriate task: Our goal is to design a generative model with good controllability, diversity, and invertibility and it is capable of modeling a complicated distribution with imbalanced training data. We understand the concern that diversity is not a high priority for practical trajectory forecasting cases (again, this goes back to the different opinions about whether the problem formulation is proper or worth studying). But we believe that it is a proper task for us to demonstrate our method because (1) the best-of-M evaluation protocol in this area provides the focus on both accuracy and diversity therefore capable of estimating whether our method can generate diverse results with faithful probability; (2) the task of trajectory prediction is a long-standing task in the computer vision community, thus provides a rich line of baselines to compare with. However, we agree that some other tasks, such as motion simulation and animation in a virtual environment, can also be good. They would be interesting future works to migrate our method into the graphics and animation communities.

Dear Reviewer pmqe,

Please let us know if our responses have addressed your concerns or if you have any questions or concerns regarding our submission. We are glad to further discuss them. Thanks!

Dear Reviewer pmqe,

We understand your concern that our task setting can't be fit to certain user scenarios directly, such as auto-driving trajectory planning. We would like to elaborate more about it to clarify potential misunderstandings.

Our problem setting is not to directly solve the auto-driving problem and we didn't claim it. We follow a long-standing computer vision problem setting for trajectory prediction/forecasting. It has been widely recognized as a main task for studying probabilistic generation diversity. A line of works published on top-tier venues, such as [1,2,3,4,5,6,7,8] (there are many more unlisted), follow this task setting. They all study on the same dataset as we use in this submission (ETH/UCY or SDD). They are proposed to study the balance between (1) more aligned/accurate max-likelihood prediction and (2) more diverse probabilistic generation. Both goals are expected to be achieved from a single model in a balanced fashion. Focusing on a single goal only causes defective solutions, either overfit deterministic methods (if only caring about accuracy/alignment) or meaningless random generation (if only caring about diversity).

The related works mentioned by Reviewer pmqe also emphasize that accurate and diverse trajectory generation is a main contribution of their method. For example:

1. [6] claims its main contribution as achieves precise and diverse predictions with fast inference speed.
2. [7] claims its contribution as exploits the proposed epistemic & aleatoric structure for diverse trajectory predictions across long prediction horizons.
3. [8] claims its main contributions including a balanced solution, i.e., a novel sparse instance, ...to guide the multimodal trajectory prediction...for the multimodal trajectory prediction.

If you have any concerns not addressed, please let use know. The authors will be pleased for follow-up discussions.

Reference:

[1] "AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting", ICCV'2021

[2] "Remember Intentions: Retrospective-Memory-based Trajectory Prediction", CVPR'2022

[3] "Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion", CVPR'2022

[4] "Progressive Pretext Task Learning for Human Trajectory Prediction", ECCV'2024

[5] "Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction", CVPR'2024

[6] "Leapfrog Diffusion Model for Stochastic Trajectory Prediction", CVPR'2023

[7] "From goals, waypoints & paths to long term human trajectory forecasting", ICCV'2021

[8] "Sparse instance conditioned multimodal trajectory prediction", ICCV'2023

Theoretical support behind our claim: Great suggestion! Our claim about the difficulty of training normalizing flow to transferring a naive and symmetric distribution, e.g., simple Gaussian, to a complex distribution, is mainly from our empirical observations. But there are some insights behind this observation:

1. VAE or Diffusion learns a conditional distribution to sample outcomes, whose representation is low-dimensional. For example, only $\mu$ and $\sigma$ as the output of VAE. On the other hand, normalizing flow is trained to learn a Jacobian between distributions, requiring a high-dimensional generation capacity.
2. Given the same input noise, VAE or Diffusion will generate different outcomes in multiple rollouts, only required to be constrained in an underlying distribution. However, normalizing flow has to do a deterministic transformation from the input noise to a certain sample in the target distribution.
3. Normalizing flow is typically trained with a single ground truth. With the naive Gaussian prior, if the target distribution is complex and multi-modal, we need to transform the single mode in the original distribution to more than one target mode. This causes a contradiction between training gradients.
4. Normalizing flow is designed to be inversible. To learn a Jacobian between a single-modal original distribution and a complex/multi-modal target distribution, the one-to-many mapping also causes difficulty when training inversely.

There are many discussions about the difficulty of training normalizing flow in practice to represent a complex target distribution, such as[1,2,3]. In [3], normalizing flow is considered inefficient and difficult to train due to invertibility constraints on transformations and as a consequence input and output spaces with the same dimension. This matches our discussion in the insight #1 above.

We have to admit that normalizing flows is usually not as good as GAN or diffusion for high-dimensional data generation, such as images. However, its unique property of invertibility is irreplaceable for controllable and explicable generation as we desire in this work.

[1] "Representational aspects of depth and conditioning in normalizing flows"

[2] https://distill.pub/2019/gan-open-problems/#tradeoff

[3] "Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models"

Distribution shift: The problem of distribution shift exists in all supervised learning arts. We believe the common and shared intentions across data are important to adapt to test scenes. And the history encoder allows MGF to adjust generated trajectories based on the input historical trajectories. For example, if there is a group of people located in front of the agent, trajectories of “go straight” sub-Gaussian will be transferred into left/right/other trajectories to avoid collisions. As long as the shared intentions such as "move to avoid collision" hold, our work can generate reasonable trajectories in new situations.

The influence of the value of M to APD/FPD: We add the ablation study about the value of $M$ in Table 3 in the submitted pdf file with $M$ increasing from 10 to 80. According to the results, APD/FPD is not sensitive to the value of $M$.

Comparison to other diversity metrics: Here we compare the mentioned diversity metrics with ours:

1. AMD (Average Mahalanobis Distance) and AMV(Average Maximum Eigenvalue) are proposed in [4]. AMD computes the Mahalanobis distance to measure how close the generated samples are to the ground truth, similar to ADE. AMV quantifies the overall spread of the predictions for diversity measuring. It requires the confidence of predictions, which limits its application in many cases where no confidence or probability is available for predictions.
2. The metrics of ED and EI are proposed in [5]. ED measures accuracy and EI shares the same formulation with APD. We note that [5] is accepted to ICML'2024 and was not publicly available yet at the due date of NeurIPS submission.

[4] "Social-implicit: Rethinking trajectory prediction evaluation and the effectiveness of implicit maximum likelihood estimation"

[5] "Evaluation of Trajectory Distribution Predictions with Energy Score."

Edit the direction of the cluster: The cluster center is represented as a vector in shape $12 \times 2$, which is the mean value of the corresponding sub-Gaussian distribution, sharing the same dimensionality as the output trajectory. We can directly manipulate the value of this vector to represent different motion intentions.

Empirical evidence of the data imbalance: Existing trajectory forecasting datasets contain imbalanced data where most trajectories are going straightforward. In Figure 4 in the paper, the weights of each cluster (represented by colors) are: Blue: 0.28, Brown: 0.09, Red: 0.12, Green: 0.1, Yellow: 0.16, Cyan:0.19, Purple: 0.05. This suggests very imbalanced data distributions for different motion intentions. Moreover, across all five subsets in the ETH/UCY datasets and one set in the SDD dataset, similar data patterns are observed: approximately 30% of the trajectories exhibit a 'stay' behavior, while 50-60% correspond to 'go straight' trajectories with varying speeds. Conversely, only 10-20% of the trajectories involve 'turn left/right' maneuvers, with 'U-turn' and other corner cases being extremely rare occurrences.

Difficulty of back-propagation: After sampling a noise point x from a mixture of Gaussian D, it is non-trivial to back-propagate the gradient from applied to x to the learnable parameters of D. 6 has addressed this issue using approximations. It requires substantially more computation. Gumbel-Softmax trick deals with the problem of “sampling discrete data from a categorical distribution”, it is not a viable solution for our case.

Thanks for your response. The literature on normalizing flow is also helpful for me. You may consider adding descriptions of editing the cluster directions and the empirical evidence of data imbalance to the revised paper. For distribution shift, according to your description, I tend to think that clustering based on relative positions other than absolute positions will help learn transferable moving patterns. I would like to point out that:

1. From my understanding, AMD and AMV are solely based on sampled trajectories. Confidence outputs from probabilistic methods are generally not a prerequisite. EI and ED are recently proposed metrics. You do not have to compare your metrics with them according to the submission guidelines of NeurIPS. Still, I believe properly evaluating the performance of all methods on more metrics can help better understand their differences, which can be added to the revision or left for future work.

2. As I understand, selecting a component from a Gaussian mixture model is a process of sampling from a categorical distribution, where the gumbel-softmax trick can play a role in allowing back-propagation. I am not sure if it will be practical and effective and can be left for future work.

Still, I have one minor concern.

The influence of the value of M to APD/FPD: The results in Table 3 of the attached pdf show that APD/FPD is stable for FlowChain, yet a noticeable decrease can be observed for MGF on both ETH/UCY and SDD datasets as $M$ increases. Since APD/FDP are computed by taking the average over sample distances, it is expected that they will be stable for a sufficiently large $M$ (I may be wrong). This expectation and the results for FlowChain do not match the results for MGF even for $M=80$. Can you provide more explanation?

Writing issues: We appreciate the detailed suggestions about writing, we could consider them and improve the writing details. L332 should be ADE and FDE.

Normalizing flow overfitting: We observe that normalizing flow tends to overfit to the single mode value (ground truth annotation) when using the naive Gaussian prior. This is because of the internal conflict between a probabilistic model and a deterministic and single ground truth supervision under the unimodal simple Gaussian prior. On the other hand, normalizing flow directly transfers a noise to a single outcome point, further causing the model to collapse to the max-likelihood value. As a comparison, VAE predicts a distribution where we can sample outcomes instead of a single outcome, relieving the overfitting issue with a reasonable value of distribution variance. Related discussion can be found in many previous literature, such as [1,2].

[1] "Noise Regularization for Conditional Density Estimation"

[2] "Why Normalizing Flows Fail to Detect Out-of-Distribution Data"

mixing coefficient: In our implementation, we decide the coefficients by steps. We use k-means clustering to partition the trajectory data into clusters. The mixing coefficient of each sub-Gaussian corresponds to the weight of the corresponding cluster.

Prediction clustering to improve accuracy: We add ablation experiments in Table 2 in the pdf file. The prediction clustering improves ADE/FDE with both single Gaussian and mixed Gaussian prior. Adding mixed Gaussian prior further boosts the performance.

Moreover, we provide the results of ground truth alignment (ADE/FDE) to demonstrate that our method can output a single "likely" trajectory in a batch at a similar level as the best arts. Its flexibility and advantage of enhancing generation diversity is our main goal.

APD in Dlow: Yes, we agree that a similar APD has been defined by Dlow[3]. We missed it as it was proposed on a different task and benchmarks. We will revise this paper and acknowledge where the metric was first proposed.

[3] "DLow: Diversifying Latent Flows for Diverse Human Motion Prediction"

Ablation studies on diversity: Great suggestion! We add the ablation study to measure the impact of modules on diversity in Table 1 in the pdf file. There are two interesting observations:

1. The prediction clustering can significantly improve the diversity only with mixed Gaussian prior. With the naive Gaussian prior, it makes no contribution to the diversity. This is reasonable as the single Gaussian prior tends to generate trajectories densely close to the most likelihood and prediction clustering can't cluster them into well-separated clusters for different motion intentions.
2. The learnable var. slightly hurts the diversity. This is because the learnable var. usually converges to a smaller value than the fixed situation ($\sigma=1$). For instance, the average value is ~0.4-0.6 on ETH/UCY. This is encouraged by the supervision from the ground truth (most likelihood) to a desired steeper Gaussian, thus hurting the diversity. It is the trade-off between (1) aligning with the single likelihood provided by ground truth and (2) generating diverse outcomes that hide behind this observation.

Adding map information: We agree that adding map information can be potentially more useful in realistic use cases. However it would lead to a different convention of benchmarking and the different processing of map information would introduce significant noise when comparing methods, especially considering many baselines have no open-sourced code or checkpoints. To focus on the main assumption and innovation in this work (a mixed Gaussian prior can improve the diversity of the normalizing flow-based trajectory forecasting), we decided not to use map information and follow the most widely adopted problem set up in this area. Adding map information would be an interesting future work but significant efforts are required to fairly compare different arts.

The reviewer acknowledges the contribution of mixture of Gaussians to boost the controllability of trajectory generation; its effectiveness of mining out different motion patterns out of the training data. The paper is well written and many intuitions and formulations are clearly explained. However, considering improving diversity as one of the main contribution of the paper, the experiments are showing mix signals and the diversity metric is only evaluated on one dataset for extensive benchmark against other SOTA models. The diversity metric is also not as original as the authors claimed. Plus, current submission lacks qualitative analysis showcasing the behavior of the different model component and model behavior under different type of inputs (for example, observation with slow speed vs. fast speed). Furthermore, the improvement of diversity alone might not be meaningful without considering the environmental and social factors in the prediction model, as social interaction and environmental constraint also dictates the behavior of human movement. Solely promote diversity without considering these constraints might not be a sound argument. Hence, the reviewer will keep the current score in terms of assessment.

The reviewer thanks the author for the response and the follow-up ablation studies. The author is able to address some of the original concerns and offers more insights on some of the questions.

Writing Issue: The reviewer thanks the author for considering updating the manuscript and clarifying the typo.

Normalizing Flow Overfitting: The reviewer thanks the authors' explanation and provide reference.

Mixing coefficient: Thank you for the clarification, however, could you elaborate a bit more on the weight and how is it defined in the k-mean clustering? Is it based the number instance belongs to that cluster? Or other definition? Because general k-mean clustering does not usually involve weights.

APD originality issue: The reviewer appreciates the author acknowledging this metric has been published before, however, these two tasks, although different in nature, are quite similar technically in their settings of the output. Please do make sure to revise the paper and give credit to the original literature, despite original work was on human motion prediction, motion trajectory and human trajectory could be both evaluated by APD metric, as the authors also chose to do so here, therefore it is not reasonable for the author to claim that metric as novel contribution here without mentioning the original work that it was first proposed.

Prediction clustering + Ablation:

• Indeed, using mixture of gaussian boosts ADE/FDE, compared to using stardard gaussian. Yet, the boost to ADE/FDE value with the addition of prediction clustering is significantly larger, the reviewer acknowledge ADE/FDE only reflect the best-of-m mode of the prediction, and does not reflect diversity or other quality of all model output. However, compare with switching to mixture gaussian as prior, ADE/FDE improvement by adding intention/prediction clustering is much more significant.
• The reviewer really appreciates the author providing new abalation study on the effect of model component on diversity. Indeed, the prediction clustering offers significant boost to APD/FPD to the model with mixture of gaussion prior (row 4 and row 5 in table 1) and less or no boost to even negative effect on APD/FPD (row1/row6 or row2/row7). However, if comparing row2 and row3, where the different is only between standard gaussian and mixture of gaussian as prior, the improvement on APD/FPD is less significant, especially on SDD dataset. Furthermore, in the main paper, the authors only offer APD/FPD benchmark against other model on ETH-UCY not SDD.
• The learnable var. indeed slightly hurts diversity, however the difference is less significant compare with the point above and the reviewer thanks the author for the analysis.
• Overall, although there is sign the mixture of gaussian as prior shows certain improvement of accuracy and diversity under specific circumstances, such improvement is less significant compared with the improvement brought by the prediction clustering. As a modeling choice, prediction clustering as a post-processing step is reasonable, however, since the author’s claim on mixture of gaussian improving sampling diversity is one of the main foci of the paper, more experiments to support the diversity claim on mixture of Gaussian are needed. One of such experiments would be to benchmark against other SOTA model on the SDD dataset, since the ablation on SDD especially shows mix signal and certain SOTA model such as AgentFormer is quite close in the diversity metric on the provided ETH-UCY diversity benchmark. The other suggested experiment would be showing more qualitative analysis to showcase through visualization of prediction outputs the effect of mixture Gaussian prior and prediction clustering. The reviewer understand such addition would be a significant amount of work, and might not be able to be added to the current submission.

Experiments on SDD: In the original paper, we raised SDD in the paper to encourage further more intensive research on the challenging dataset. However, we had to put the dataset as secondary because of some practical difficulties:

1. SDD is a more challenging dataset than ETH/UCY with more complicated motion patterns and intentions. Also, SDD is a more recent benchmark. Many existing methods do not study SDD and just report results on ETH/UCY.
2. Among the baselines we included in the benchmark on SDD, only PECNet[1] provides the checkpoint to reproduce their reported results.
3. Otherwise, only MID[2] and MemoNet[3] open-source their training code (no pre-trained checkpoint is provided). Moreover, we unfortunately could not reproduce MID's result as reported in its paper by the released code. We could reproduce MemoNet's results on SDD close to its reported result.

We encourage open-sourced research and transparent experiment settings on all datasets in this area. We would expect a more established benchmark on SDD with baseline methods. We wish to provide the diversity comparison for methods on SDD in the future version.

[1] "It is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction"

[2] "Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion"

[3] "Remember Intentions: Retrospective-Memory-based Trajectory Prediction"

Problem setups: The task of trajectory prediction/forecasting has been studied for a long time in the community of computer vision. We followed the common problem setting of this long-standing task. According to the additional information on map/surroundings and social interaction:

1. Most works in this area do not take the map information into consideration. Moreover, adding extra map/environment information makes the task a totally different task, asking for a different line of benchmarking. Under such problem setups, more noise is included, especially the map information encoder, which is designed in different fashions in related works. This will confuse and blur our main focus in this work when seeking transparent and fair experimental evidence.
2. In our design, our method can actually capture the social relations of different pedestrians (by feeding the historical trajectory of all agents into History Encoder). Also, we did compare the methods modeling social interaction explicitly. Our method's performance is much better than many works that model social interaction explicitly, such as Social-GAN.

more qualitative results: We did provide the visualization of prediction results in Figure 4 (in the paper) and Figure 6 (in the supplementary). Figure 6 is a good example to help understand the behavior and diversity difference between naive Gaussian prior and mixed Gaussian prior. We also provide illustration videos in the supplementary to understand the behaviors of different sub-Gaussians in our built model. We understand that Reviewer qRNj desires more quantitative visualization to assist in understanding the ablation study results. However, the qualitative difference is not significant for most pairs of comparison in the ablation study per our check. Moreover, we can't provide images or videos here per the rebuttal policy. We will consider the suggestion for the next version. Thanks.