TrajCLIP: Pedestrian trajectory prediction method using contrastive learning and idempotent networks

Q1: This is not the first work to use constructive learning.

We thanks reviewers for providing a new perspective. The mentioned two works, long-tail analysis[1] and FEND[2], both utilize contrastive learning, but our work differs in the problems it addresses with contrastive learning. These two mostly use contrastive learning to capture features from historical trajectories. They aim to get better representations of historical trajectory encodings in latent space, which will improve their ability to predict trajectories for the long-tail part of the dataset.}

For example, long-tail analysis uses a Kalman filter to directly predict output trajectory errors as a basis, distinguishing difficult and easy samples. It then conducts contrastive learning on difficult (long-tail part) historical trajectories to enhance the feature representation capabilities of the long-tail part; FEND uses VAE to extract features from the entire trajectory and performs offline clustering on the features to obtain labels; during training, it uses offline labels for contrastive learning on historical trajectories to guide the feature generation of historical trajectories (making features extracted from more similar historical trajectories closer), enhancing the prediction effect of the long-tail part.

However, our work, not only uses contrastive learning for better encoding of historical trajectories but also acts on future trajectories. While ensuring historical and future motion consistency, it achieves a more unified latent space representation of future and historical trajectories, thereby enhancing the model's predictive generalization capability.

Q2: Missing citation and comparison for SingularTrajectory.

SingularTrajectory[3] introduces a diffusion model for multimodal pedestrian trajectory prediction, utilizing singular space rather than trajectory coordinates to convey motion information and capture the dynamics of trajectories. It also employs an adaptive anchor-based strategy for dynamically adjusting trajectory anchors. We acknowledge the merits of this technique, but our approach surpasses this work in terms of performance and adaptability. The summary is as follows:

(1) In the performance comparison on the ETH-UCY dataset as shown in Table 3, our model achieves a 14% lower Average Displacement Error (ADE) than the SingularTrajectory model (0.18 for our model versus 0.21 for SingularTrajectory), with the difference in Final Displacement Error (FDE) being negligible.

(2) For transfer learning and few-shot learning tasks, given the differences in task settings and data processing between our method and SingularTrajectory, we conducted experiments following the experimental framework described in their paper. The results of these experiments are presented in Tables. 2 and 3. Observations indicate that our model attains comparable ADE performance to SingularTrajectory in transfer learning tasks and slightly outperforms in FDE. In few-shot learning tasks, our model demonstrates superior performance in both ADE and FDE.

(3) Regarding the task of online adaptability, the architecture of SingularTrajectory, which incorporates complex diffusion models and scene semantic segmentation techniques, results in slower inference speed, making online learning impractical. In contrast, our model consistently outperforms in this task.

Q3:More detailed analyses for ablation experimental phenomena about CLIP and generative framework (IGN, CVAE, diffusion).

We further explain the analysis of the ablation study. The CVAE framework involves an encoder obtaining a feature and then randomly sampling to get the mean and variance, which are used as inputs for the decoder. The diffusion method starts with a random initialization of an input and then gradually denoises. This randomness makes alignment work less effective (that is, even if aligned, the presence of random sampling still makes the generated results uncontrollable). As shown in Table 4 in main content, the ablation experiments of CLIP and the generation framework were conducted. The comparative experiments (c)(d) and (e)(f) illustrate that the two random methods of CVAE and diffusion methods make the improvement effect of alignment work not significant. On the ETH_UCY dataset, the addition of CLIP only improved ADE/FDE by 0.01/0.03 and 0.01/0.01. However, the network structure of IGN is an MLP, which is an affine transformation and does not have this issue; hence, as in our method, alignment is significantly effective on IGN. When CLIP is removed, as shown in experiments (b)(c)(e), since the two feature spaces cannot be connected through IGN without alignment, the performance drops by 0.06/0.17 compared to when CLIP is added.

Q4:Is the affine transformation mentioned in Fig.1 same as the trainable matrix W in CLIP?

In our approach, CLIP is only utilized to facilitate the alignment of feature spaces during the pre-training phase. The affine transformation of the idempotent generative network operates independently of the parameters of CLIP. However, using CLIP's trainable matrix W as an initialization for the network's parameters could enhance the generative framework. Since CLIP's trainable matrix W can project the feature space into a unified fusion space, it is possible to employ this matrix W as an initial parameter within the idempotent generative framework and subsequently further train the idempotent generative network based on it. This is a perspective worth exploring, and we will conduct further research on this attempt in the subsequent studies.

[1] Makansi, Osama, et al. "On exposing the challenging long tail in future prediction of traffic actors." ICCV. 2021.

[2] Wang, Yuning, et al. "Fend: A future enhanced distribution-aware contrastive learning framework for long-tail trajectory prediction." CVPR. 2023.

[3] Bae, Inhwan, et al. "SingularTrajectory: Universal Trajectory Predictor Using Diffusion Model." CVPR. 2024.

Q1:I have some doubts about the intuition of using CLIP between history and future trajectories.

Regarding further clarification on the unified encoding of historical and future trajectories using CLIP. In our method, the trajectory encoders are designed to capture motion characteristics rather than static spatial data, enabling synchronization of feature spaces between the future and historical trajectory encoders. Considering that inputs for pedestrian trajectory prediction consist of sequential relative coordinates, the encoders focus on the agent's trajectory, surrounding dynamics, and their interactions. The decoder then infers future locations from these features. For datasets like ETH_UCY, where 20 frames equal 5 seconds, spatial consistency is maintained by analyzing the same two frames. The use of STIF and SAIF is for modeling trajectory and interaction features, respectively, aligning them in the feature space to prevent redundancy in trajectory predictions that could arise from spatial alignment.

Q2: There is no speed or computational cost comparison. Since the method involves multiple stages, a comparison of training/inference costs is needed.

For training process, our batch size was set to 64, epochs to 100, and the learning rate was 0.01, which was half-formed every 25 epochs. We used the Adam optimizer. Our model was trained on an RTX 3090, with the encoder and contrastive learning pre-training requiring approximately 8 GPU hours, and the full framework training taking about 9.6 GPU hours.

For inference process, as shown in Table 1, our medium-sized model meets the requirements for a real-time prediction task in terms of both inference speed and model size, as it can predict trajectories in 0.0615 seconds. Additionally, our lightweight model is only 3.45MB in size, and its computational complexity is relatively low compared to its model size, making it deployable on most hardware platforms.

Q3:The proposed method does not consider visual scene information.

Thank you for the new perspective provided by the reviewer. Our focus lies in how to align the trajectory features and interaction features of agents in the historical and future spaces within trajectory prediction, and we are concerned with whether this method can be generalized to various subtasks at a low cost. Due to the high real-time requirements of autonomous driving, we chose not to introduce image information. Admittedly, using the features of visual information from map scenes to improve the performance of trajectory prediction is indeed another angle worth paying attention to. We will consider how to efficiently integrate visual information to further enhance the predictive effect of the model in our subsequent work.

Q4: Minor comments: Section 2.2 Generalization Framework -> Generative Framework. They are different things. The text in Figure 2 is too small to read. Line 263 “We conducted comparative ”, Line 270 “we compare our model”. Better to use a consistent present tense when describing your method.

We will address these issues (e.g., typos, grammar mistakes) in the camera-ready version.

Q5:There is no error analysis. When does the method fail and why?

We express our gratitude for the feedback. Due to spatial limitations, a summary of the limitations of our methodology will be presented in the camera-ready version. Our approach primarily focusses on the modelling of trajectory prediction tasks, in line with common practices that omit scene imagery as input. Therefore, empirical validation of the model's performance in practical scenarios is essential. As illustrated in Figure 1 in attached PDF, our method falls short in ensuring collision avoidance in complex, high-density environments, necessitating further research.

Q1:The motivation for relying on an idempotent generative network over other generative models, e.g. GANs, is not well explained.

Further explanation on the starting point for choosing the idempotent generation framework. Our motivation is that, for trajectory prediction task, the trajectory features of an agent should be consistently aligned in the feature space of both historical and future trajectories. However, GANs and CVAEs (SocialGAN) both have separate encoding-decoding structures, and diffusion inference samples the initial space for generation, which leads to a certain degree of inconsistency in the feature space. The idempotent generation framework is an affine transformation in the feature space, which can effectively bridge the gap in the feature space caused by the generation framework. In addition, the idempotent neural framework has the characteristic of global projection, which can more effectively improve generalization ability, allowing our method to perform well on different tasks without fine-tuning. Our extensive quantitative and qualitative experiments have fully demonstrated this point. Please refer to Reviewer AYPL Q3 for more experiential analysis.

Q2: article NSP-SFM[1] performs better. The comparisons should include all state-of-the-art methods.

Further explanation on the comparison with the SOTA methods, we will compare and analyze with the methods mentioned by the reviewers in the final version. (1) We acknowledge that the model in the article NSP-SFM[1] has better predictive quantitative performance than TrajCLIP. However, the NSP-SFM method first relies on a CNN-based network to extract goal information from the map, which is not feasible for many complex scene prediction tasks as the goal information is not available. Secondly, the method's estimation capability for multimodal trajectories is not achieved by a generative model and does not consider the diversity of pedestrian intentions during the prediction process; moreover, the network has a large number of parameters, with its model parameter count being more than 50 times that of other trajectory prediction networks. In contrast, TrajCLIP's predictions do not need to rely on goal information and can be predicted independently, without being limited by the prediction anchor points, and with a smaller number of parameters. We are more focused on comparing with similar methods to demonstrate the model's performance and capabilities in the context of the predictive task's focus points.

We compared four recent trajectory prediction state-of-the-art method from CVPR 2024 in Table 4. our method achieves the best ADE/FDE results on the ETH-UCY dataset.

Q3:For the Online Adaptation task, prediction times are another crucial factor to consider alongside performance metrics. Is there any reason not to report them?

We have included implementation details in Table 1 in attached PDF. Our tiny model meets the requirements for a real-time prediction, as it can predict trajectories in 0.0615 seconds. Additionally, our lightweight model is only 3.45MB in size, and its computational complexity is relatively low compared to its model size, making it deployable on most hardware platforms.

Q4:Could the authors provide details of the hyper-parameters used during training? Implementation details are missing.

For training process, our batch size was set to 64, epochs to 100, and the learning rate was 0.01, which was half-formed every 25 epochs. We used the Adam optimizer. Our model was trained on an RTX 3090, with the encoder and contrastive learning pre-training requiring approximately 8 GPU hours, and the full framework training taking about 9.6 GPU hours.

Q5: There are significant discrepancies between the results you report in Table 1 for Trajectron++ and the results reported in other publications, such as Trajectron++ in "Human Trajectory Prediction via Neural Social Physics". Could the authors provide more details on the experimental setup?

Explanation of the reproduction details for Trajectron++. When reproducing Trajectron++[3], there was a data leakage issue in the original released code implementation, leading to better metrics. The original author also acknowledged this on GitHub issue #26. "The function used for differentiating is derivative of and calls np.gradient. np.gradients approximates in different ways for boundary points and non-boundary points. For an example of calculating velocities, np.gradients will give $v _x[t] = (x[t+1]-x[t-1])/(2*dt)$ at time t." This method inadvertently includes future trajectory information in the last frame of historical data, which significantly affects the model's predictive performance. The author of Trajectron++ suggests using the correct differentiation method, such as np.ediff1d, to preprocess velocity and acceleration inputs to avoid data leakage caused by using information beyond the last observed time step. (Due to conference policy restrictions, we cannot provide a link.) After resolving this data issue, we reproduced the experiment using the method suggested by the author, which resulted in differences in the cited values when compared with other papers. In addition, the recent CVPR 2024 paper: SingularTrajectory [2] and many other works also adopt the same practice when citing the Trajectron++ work, and the metrics are consistent.

[1] Yue, Jiangbei, Dinesh Manocha, and He Wang. "Human trajectory prediction via neural social physics." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.

[2] Bae, Inhwan, et al. "SingularTrajectory: Universal Trajectory Predictor Using Diffusion Model." CVPR. 2024.

[3] Salzmann, Tim, et al. "Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVIII 16. Springer International Publishing, 2020.

Q1: Add a section discussing the computational requirements and runtime performance of TrajCLIP.

We appreciate you bringing up the model performance experiment. As the table illustrates, we have contrasted our approach with alternative techniques in terms of model size, computational complexity, and inference speed. Our medium-sized model meets the requirements for a real-time prediction task in terms of both inference speed and model size, as it can predict trajectories in 0.0615 seconds. Additionally, our lightweight model is only 3.45MB in size, and its computational complexity is relatively low compared to its model size, making it deployable on most hardware platforms.

Comparison of our method with other existing methods in terms of model size, computational complexity, and inference speed. Inference speed refers to the time required to input an 8-frame trajectory and predict the next 12 frames.

Q2:More analysis of the limitations of the proposed method and potential failure cases would strengthen the paper.

We express our gratitude for the feedback. Due to spatial limitations, a summary of the limitations of our methodology will be presented in the camera-ready version. Our approach primarily focusses on the modelling of trajectory prediction tasks, in line with common practices that omit scene imagery as input. Therefore, empirical validation of the model's performance in practical scenarios is essential. As illustrated in Figure 1 in attached PDF, our method falls short in ensuring collision avoidance in complex, high-density environments, necessitating further research.

Q3: Provide more detailed explanations of the idempotent neural network implementation and training process.

We will provide a detailed summary of the model's experimental details and training process in the camera-ready version and will also release the source code. The experimental details of the idempotent neural network can be summarised as follows: First, we froze the pre-trained historical trajectory encoder and trained the manifold predictor, that is, the idempotent neural network, as well as the manifold decoder. The manifold predictor was trained using reconstruction loss, idempotent loss, and tightness loss, while the manifold decoder was trained using L2 loss. Training the idempotent and tightening losses requires gradient clipping. For clarity, we provide the training code as follows. Moreover, an RTX 3090 was used to train our model. Our training parameters included a 64-batch size, 100 epochs, and a 0.01 learning rate that was half-formed every 25 epochs. The Adam optimizer was employed.

def ign_train(f, f_copy, Z_H):
    # f, f_copy : MLP_θ
    predict_Z_F = f(Z_H)
      
    f_z = Z_F.detach()  
    ff_z = f(f_z) 
    f_fz = f_copy(fz) 

    # calculate losses
    loss_rec = (predict_Z_F - Z_F).pow(2)  
    loss_idem = (f_fz - fz).pow(2)  
    loss_tight = -(ff_z - f_z).pow(2)  

    # optimize for losses
    loss = loss_rec + loss_idem + loss_tight * 0.1
    opt.zero_grad()
    loss.backward()
    opt.step()

[1] Shocher, Assaf, et al. "Idempotent Generative Network." The Twelfth International Conference on Learning Representations.

The author's response has resolved some of my questions, but I still have questions about the Computational Complexity and Inference Speed of the paper. Compared to the previous state-of-the-art papers, this paper has significantly increased the number of parameters, but the performance improvement is limited. So I will maintain my "Borderline accept" score for this paper.