Neuralized Markov Random Field for Interaction-Aware Stochastic Human Trajectory Prediction

SOTA performance

There are multiple papers claiming the SOTA performances with their proposed methods when applying the task settings in Social-GAN (CVPR’18), however, the comparison can be unfair since they directly choose the best model based on the test set instead of following the original train-val-test splits. This issue was mentioned in some GitHub repos (i.e., MID (CVPR’22) issue 5) and papers (i.e., NPSN (CVPR’22)). Therefore, we include the baselines that clearly adhere to the train-val-test splits, for example, AgentFormer (ICCV’21), SingularTrajectory (CVPR’24), or re-produce results with the correct splits setting.

For the baseline mentioned, View Vertically (ECCV’22) or SocialCircle (CVPR’24) also have the same unfair comparison issue, which can be checked from the plist files for the dataset in the released codes:

• V2-net has {'test': ['eth'], 'train': ['hotel', 'zara1', 'zara2', 'univ', 'zara3', 'univ3', 'unive'], 'val': ['eth']};
• SocialCicle has {'anntype': 'coordinate', 'dataset': 'ETH-UCY', 'dimension': 2, 'scale': 1.0, 'scale_vis': 1.0, 'test': ['eth'], 'train': ['hotel', 'univ', 'zara1', 'zara2', 'univ3', 'unive', 'zara3'], 'type': 'meter', 'val': ['eth']}.

This means when reporting results on the eth subset, they directly choose the best model based on the test set, instead of following the original leave-on-out subset method: train-val on 4 subsets and test on the remaining subset (from Social-LSTM (CVPR’16) and Social-GAN (CVPR’18)).

To have a fair comparison, we also correct the splits used in the most recent SOTA method SocialCicle (CVPR’24), and attach the results here:

It also supports our proposed method is SOTA under correct train-val-test splits. For the mentioned SICNet (ICCV’23), it is hard to validate whether the correct splits have been used since no official codes have been released, but our model still surpasses its reported performance on some subsets in ETH/UCY and on SDD (8.44/13.65 compared to ours 7.20/11.29).

In addition, we also show our model outperforms other baselines on the NBA and JRDB datasets, where data splits also strictly follow the prior works. Therefore, we finally claim the state-of-the-art prediction performances across multiple datasets.

Novelty of the method

Our novelty lies in an explicit modeling on full dynamics of joint space state transitions and crowd interactions with the MRF by a probabilistic distribution, and a neuralized realization for tractable learning and inference. We decompose the full dynamics into the observation model and the dynamics model, imposing a structure prior, unlike other methods that implicitly capture pedestrian motion and interaction mostly in an end-to-end paradigm.

The Markov property implemented in prior works is for agent-wise self-motion modeling to show that a pedestrian’s future state (the state can be expressed in either coordinates or abstract action) primarily depends only on its current state, so our proposed method differs starting from the problem formulation, considering the Markovian assumption for segment-wise evolution and in-segment interaction with MRF modeling.

Also, dynamic graph construction itself is a widely applied idea to model interactions as we mentioned in Related Works (c.f. Section 2 Interaction Modeling). However, instead of applying graph networks or cross attentions directly based on distance features and learning the pedestrian’s interaction implicitly in an end-to-end paradigm, we propose explicit modeling of full dynamics involves spatial and temporal evolution of the system for all agents and their interactions via decomposed observation model (c.f. Bayesian update) and dynamics model (c.f. self-evolution + interaction), thus benefits from the structured system modeling and neuralized implementation, finally achieving better performances.

Motivation and why MRF could achieve better performances

To deal with noisy observations or other uncertainties, the trajectory prediction problem can be formulated as solving a probabilistic distribution of the joint space configuration, in addition to a deterministic function map (c.f. L162-163). Since human self-intention will change and they are constantly interacting with each other, the pedestrian-interaction system dynamics is hard to obtain, and many prior works apply an end-to-end paradigm to learn the dynamics implicitly via different network designs. The main motivation for our work is the combination of structured modeling and neural networks, which is inspired by the performance improvements introduced by combining structured modeling and neural networks from other applications, i.e., semantic segmentation (ICCV’15) and stereo matching with MRF (CVPR’24). Therefore, we propose a novel formulation for the trajectory prediction task, explicitly modeling the full dynamics of joint space state transitions and crowd interactions.

We begin with the formulation of the human trajectory prediction task (c.f. L51-72 and Section 3.1) based on the Markovian motion assumption. Then, to capture the dependencies of pedestrians that are constantly interacting, we further factorize the spatial space distribution and propose an MRF-based evolution term (c.f. L72-81), finally realizing tractable learning and inference (Section 3.3) with a new proposed network architecture.

MRF is a very common tool to represent dependencies in the format of a factored probability model specified by undirected graphs, thus is well-suited for relationship modeling for interacting targets. Our MRF modeling takes advantage of the inductive bias in the downstream task, i.e., Markovian assumptions between segments and spatial relations in the graph construction. This structured modeling introduces important spatial prior for interactive agents, and reduces the difficulty for optimization, thus generating plausible samples even in the initialization (c.f. Figure 7 in the appendix where we compare our initializations with LED (CVPR’23), the SOTA model on the NBA dataset). Besides, with neuralized implementation, our MRF modeling overcomes the computational intractability, and learns the distribution from the data, instead of relying on hand-crafted probability functions.

More visualized results

Thank you for the suggestions and we have updated more visualizations in the appendix of the manuscript on ETH/UCY datasets.

Input modality, joint configuration space, and parameterization

As introduced in L158-161, $O_t$ is the observations at time $t$ which is a collection of all agents. In the default setting of the human trajectory prediction task (c.f. classical Social-LSTM (CVPR'16), Social-GAN (CVPR'18) methods), $O_t$ is a set of xy-coordinates, which means each datapoint represents a person’s world coordinate. Since many recent approaches (i.e., Social-Transmotion (ICLR'24), etc.) also extend the observations with 2D detection bounding boxes, 2D encoded image features, 2D human skeleton key points, etc., we use a general notation $O_t$ instead of restricting it with one specific modality. In our method, we adhere to the default xy-coordinates input modality, and it is also shown in Table 4. The pre-processing of observation sequence $O_{1:t}$ also follows the procedures applied by many previous works (i.e., Trajectron++ (ECCV'20), LED (CVPR'23)): within a single frame, coordinates are standardized with the mean position of all agents, and within an agent, coordinates sequences are aligned with the current time $t$ to ensure the relative position at $t$ is (0, 0). Usually, absolute positions and relative positions are concatenated together and input into the neural network.

Configuration space is usually used to describe the state of a whole system as a single point in a high-dimensional space or to describe assignments of a collection of points to positions in a topological space. We apply this word to describe the pedestrian-interaction system state at a time step (c.f. $X_t$), which includes all agents' spatial coordinates and geometric relationships. In each position we mentioned joint configuration space, there is an equation followed: L160 $X_{t+1:t+T} \triangleq \{X_{t+1}, X_{t+2}, ..., X_{t+T}\}$, L161 $X_t \triangleq \{X_{i,t}\}^M_{i=1}$, and L165 $S=\{S_1, …, S_K | S_k \triangleq X_{t+(k-1)\tau+1:t+k\tau}\}$. And $M$ individual states simply mean the states (in trajectory prediction task, spatial arrangements) of all $M$ agents within a time step.

Thank you for the suggestions and we added a reference to the figure for a better explanation of parameterization. The parameterizations in L181-183 are enumerated for different probabilistic function components in Equation 1, thus we include the details of them in Section 3.3 Network Architecture which is the neuralized implementation of each decomposed function, instead of expanding them right after Equation 1.

Two-stage approach

L472 standard normal distribution

First, I think there could be some misunderstandings about the two stages here. In Table 4 and 5, “Stage 1 Only” refers to CVAE Training as shown in Figure 3. And Stage 2 is Sampler Training. “Bayesian Update” and “MRF-based Evolution” are two sub-modules corresponding to two probabilistic function components “Bayesian update” and “self-evolution + interaction” in Equation 1, which is the problem formulation of our method, so they will be both included in Stage 1 and Stage 2.

The difference between the two stages is the sampler. That’s why we mentioned in L472 “replacing the sampling from a standard normal distribution with purposive sampling,” since the standard CVAE structure generates samples from the standard normal distribution $\mathcal{N}(0, I)$ (sampler in Stage 1), and in Stage 2 we further trained a sampler to realize purposive sampling with other encoders and decoders frozen. Stage 1 can be treated as a prototype validation as it is directly corresponding to the formulation in Equation 1. This procedure enables more stochasticity in trajectories. In Stage 2 we apply purposive sampling after the encoders and decoders are trained to make samples generated are dependent on the past, thus allowing more information embedded and that’s the reason for the case after applying Stage 2 the performance will be better.

Besides, in Equation 1 the probabilistic distribution is factorized into “Bayesian update” and “self-evolution + interaction” terms, which can be seen as the observation model and system dynamics model, and the iterative product is from Markovian assumption. While in many prior works, these two models are usually not distinguished and learned implicitly in an end-to-end manner, we are inspired by the improvements introduced by combining structured modeling and neural networks from other applications, i.e., semantic segmentation (ICCV’15) and stereo matching (CVPR’24), then propose a novel formulation for the trajectory prediction task.

Frequencies and Markovian assumptions

Here using “certain frequencies” we would like to express that the Markovian assumption might not hold under the original video/sensor frequency, so we use the “stride” to divide the sequences into segments, and assume the Markov property between segments.

Each segment $S_k$ contains all agents and their reciprocal interactions, thus we include the collaboration between each other. Note that the Markovian assumption we presented in L166-168 is for segments.

Short-term predictions and Markovian approximations

We agree that the long-term reasoning of humans is not Markovian because of external instructions, self-intentions, etc. (like the example mentioned, the player has the long-term goal to perform a maneuver to reach a target location).

Nevertheless, we assume the short-term motions of each agent are primarily influenced by the most recent actions of nearby agents, making Markovian modeling suitable between segments. For example, if a player’s maneuver is blocked by opponents, it is predictable that the player will respond to this (i.e., detour, sudden turns, etc. that can be observed in the data), regardless of their longer-term goal, which admittedly depends on their history.

In the human trajectory prediction task, we usually predict for the future 4-5 seconds based on 3-4 seconds observations in multi-agents interactive scenarios, thus more fitting for this assumption. In detail, under our assumption, we first update the future 1-2s trajectories using the observation. Then, trajectories for the further future are predicted based on the Markov property. We do utilize the history, and the Markovian assumption applies during the further future, representing short-term motions. And we do not handle the hidden intention estimation explicitly in our method.

Experiments for accessing history

Besides, we also attach the experiment results for the NBA dataset when recursively aggregating the “motion history” for the prediction as suggested (i.e., incorporating prediction as expected history $transform(S_k) \rightarrow O_{t+(k−1)\tau+1:t+k\tau}$):

There are no obvious improvements when accessing more “motion history”; contrarily, the inference speed will decrease. For other datasets we used, the phenomenon is the same (i.e., on the ETH set, we have the original metrics 0.257/0.374 compared to 0.264/0.381 with full history). It supports our assumption experimentally, and the Markov property can enable more time efficiency.

Thank you for pointing out that the original statement is not very rigorous, and we will rephrase this assumption as “Human movements are Markovian up to a certain time under local interaction in a short time horizon” in our manuscript.

The robustness experiment is not clear.

In Table 6, we randomly select 4 frames from all 9 observed frames to add Gaussian noise. And thank you for the suggestions, we attached the robustness tests where all 9 frames are disturbed by Gaussian noise:

Usually, a scale of 0.1 is treated as severe noise, and the results in the table show that our model is still robust under noise applied to all frames.

In table 1, it will be interesting to add the previous model Y-net

We have tried to include Y-net but it is hard to replicate and validate the results since the codes released on GitHub have several problems. First, the configs for Y-net on the ETH/UCY dataset are not provided (c.f. GitHub issue 63) and the codes have some bugs with the dataloader as well (c.f. GitHub issue 67). And there are no pre-trained models for this dataset. Second, based on the data folder released, it also applies an unfair comparison where the best model was also chosen directly based on the test set, which is different from the standard train-val-test split in Social-GAN (CVPR’18) and is mentioned by the authors of Social-STGCNN(CVPR’20) and NPSN (CVPR’22). We highlight this issue in the caption of Table 1 and finally show the performances of SOTA models that either adhere to the leave-one-out strategy or can be reproduced by re-training with the correct splits.

It seems there is a gap between the proposed method with the previous works on JRDB dataset

We double-checked and confirmed the task settings and data splits are the same as in previous works. As mentioned in our paper L301-302, we follow the same prediction task settings (i.e. predict for the next 12 time-steps given the past 9 time-steps at 2.5 fps) and data splits with the previous work Social-Transmotion (ICLR’24) to evaluate our model and LED (CVPR’23), the SOTA model on NBA dataset. In detail, gates-ailab for validation, indoor scenario series packard-poster-session and outdoor scenario series tressider for testing, and the other scenarios for training. Besides, the JRDB dataset provides toolkits for tracking tasks on GitHub, and we simply apply official codes to process their original detection labels to generate trajectories within each scenario. For baselines before Social-Transmotion, we follow the results in the Social-Transmotion paper.

In addition, since we have baselines Social-GAN (CVPR’18), Trajectron++ (ECCV’20), and LED (CVPR’23) on both the NBA dataset (c.f. Table 3) and the JRDB dataset, the differences between our proposed model and these baselines follow a similar trend. This phenomenon also supports our implementation.

Important citations about the dataset used in this paper are missing. Figure 1 was never referred to in the text

Thank you for pointing these out; we have updated them in our manuscript.