TSC-Net: Prediction of Pedestrian Trajectories by Trajectory-Scene-Cell Classification

Weakness 1 and 5: The referenced trajectory prediction papers focus heavily on older works. It would be beneficial to include more recent papers. There are other methods with better performance on the ETH-UCY dataset, making it difficult to claim state-of-the-art performance based on average metrics.

Response: In the experiment, we compared several papers, including FlowChain, TUTR, SICNet, LMTraj-SUP, E-V$^2$-Net_SC, which are published in 2023, 2024.

We already included 'View Vertically' in our manuscript (V$^2$-Net Wong et al. (2022) in Table 2). Compared with 'View Vertically', we significantly outperform it on SDD dataset. And we didn't claim SOTA performance on ETH-UCY dataset in our manuscript.

Weakness 2: If the authors claim that the ETH-UCY dataset only contains binary-level semantics, why not experiment with vehicle trajectory datasets instead?

Response: We believe vehicle trajectory prediction is not the same task as pedestrian trajectory prediction, as vehicle prediction involves more prior knowledge, such as traffic rules. Moreover, all the compared baseline methods in our manuscript focus only on the pedestrian scenario, where pedestrian trajectory prediction can sometimes be even more challenging. For example, pedestrian trajectories may include sudden changes in direction and speed, while vehicle trajectories are generally smoother.

Weakness 3: The CVAE-based trajectory decoder for goal prediction does not seem new. The sequential prediction of goals and trajectories has been widely explored.

Response: We did not claim that CVAE and two-step sequential prediction are the novelties of this work. Our main novelty and contribution lie in proposing a novel joint representation, the TSC feature, which maps trajectory and scene into a unified feature space.

Weakness 4: The ablation study sounds insufficient. It is hard to verify the effectiveness of the proposed TSC. For example, it is necessary to observe how prediction performance changes when certain features $z_{i,t}^{(sc)}$ or $a_{i,t,j,k}^{(step)}$ are excluded.

Response: We conducted an extensive ablation study in Section 4.2, such as excluding different feature components, changing the goal grid size, altering the sampling strategy, and modifying the prediction manner. Excluding $z_{i,t}^{(sc)}$ means revoming the scene, which is part of the ablation study on excluding different feature components. Excluding $a_{i,t,j,k}^{(step)}$ is not possible, as it serves as the query in the attention operation, which is one of our key designs in the prediction pipeline. However, the ablation on changing the prediction manner provides a comparison to an alternative approach where $a_{i,t,j,k}^{(step)}$ is used as the key and value input.

Questions 1: The ETH-UCY dataset also includes RGB images of the scene. Instead of using binary label semantics, why not use the RGB images? It’s unclear why RGB information was used for the SDD dataset but not for ETH-UCY.

Response: Following previous methods such as Y-Net, we use semantic scene maps for all the datasets instead of RGB maps. The only difference is that the semantic map in the SDD dataset has five categories, while ETH-UCY has only two categories (traversable and non-traversable, and traversable usually dominates the scene).

Questions 2: In equations 7 and 9, is the network $f$ the same as the $f$ in equation 3? If they are different, the notation should be distinguished. If they are the same, how can they be identical when the inputs in equations 3 and 7 are different?

Response: Yes, all the coordinate feature encoding $f$ is the same. Equation 3 computes the encoding of trajectory coordinates, while Equation 7 compute the encoding of different cell centers coordiante in the goal grid (Equation 9 encodes the cell centers coordiante in each step grid). The outputs of Equation 3 and Equation 7 are different and play different roles in the prediction pipeline.

Thank you for taking the time to respond to the authors' rebuttal. In this response we will expalin why our method cannot be applied on Argoverse dataset and provide more ablation study experiment results.

I believe conducting experiments on vehicle trajectory datasets is essential. Richer scene semantics in vehicle datasets, such as lanes, traffic signals, and crosswalks, are inherently tied to traffic rules and provide a better platform to evaluate how well your method can utilize scene information.

We agree that vehicle trajectory datasets like Argoverse provide richer scene semantics. However, scene maps in pedestrian trajectory prediction only indicate 2D images where different channels represent semantic categories. Our method is designed to only accept trajectory data and 2D scene maps as input. Differently, scenes in vehicle trajectory prediction are usually structured information, such as lane and traffic signal data, which cannot be processed by our model. For instance, the scenes in the Argoverse dataset are represented by lane centerlines, traffic direction, and intersection annotations rather than 2D images. As a result, our method cannot be directly applied to this dataset.

Nonetheless, we appreciate the reviewer highlighting the vehicle trajectory prediction task. Given the similar logic between the two tasks, exploring how to adapt our method for vehicle datasets would be an interesting research direction in the future.

By removing $a_{i,t,j,k}^{(step)}$ and observing the impact on performance, you can provide stronger evidence that it plays a critical role in the overall success of your approach.

Regarding the ablation study on $a_{i,t,j,k}^{(step)}$, we would like to point out that our framework is based on many attention operations, for example, by sending cells in a grid as query and historical trajectory as key and value, the cells summarize its own information with trajectory information so that it can predict whether next step is in the cell or not. Excluding $a_{i,t,j,k}^{(step)}$ means removing the query of an attention function, where the attention cannot get any output.

To evaluate the performance without $a_{i,t,j,k}^{(step)}$, attention functions in the trajectory completion step are removed, where the last step feature is directly sent to the prediction layer to get current step coordinates. Moreover, as $a_{i,j,k}^{(goal)}$ plays a similar role in the goal CVAE, we also modify the Goal CVAE framework in a similar way as excluding $a_{i,t,j,k}^{(step)}$ and evaluate the performance without both $a_{i,j,k}^{(goal)}$ and $a_{i,t,j,k}^{(step)}$. Following our exisiting ablation study, SDD dataset with short-term setting is evaluated. The results are shown in the table below.

From the table it can be seen that excluding $a_{i,t,j,k}^{(step)}$ results in a larger ADE. Although in this experiment goal CVAE network has not been changed, FDE is still larger than the full model. This is because the goal CVAE and trajectory completion networks are trained in end-to-end manner. If the training loss in trajectory completion network is much larger, the model will put most effort to optimize trajectory completion network and affect the goal CVAE optimization. After excluding both $a_{i,j,k}^{(goal)}$ and $a_{i,t,j,k}^{(step)}$, ADE and FDE is significantly larger than our full model.

Weakness 1: The layout design of Figure 2 is unappealing and does not effectively highlight the core innovations of this research.

Response: Thanks for the valuable suggestion. The revised Figure 2 is included in the re-submitted manuscript. In the new figure, 3 steps (encoding history, predicting goals, predicting next steps) are explicitly illustrated. A more detailed description is also added to the caption of the figure.

Weakness 2: what does $q$ in the Formula 7 and Formula 8 represent for and what is the meaning of $i$, $j$ and $k$?

Response: In formula 8, $q$ is the center coordinate for each cell of the grid. In these formulas, $i$ is the index of pedestrian, which is from 1 to $N$; $j$ and $k$ are the row and column indexes of the cell in the whole grid. Then coordinate $q$ is computed by the row and column index and the size of each cell. Then in the next step, $q$ is sent to coordinate feature encoding module (Formula 7).

Questions 1: In Formula 11, what is the prior distribution of $\epsilon$, and does $p(\epsilon)$ represent the standardized normal distribution?

Response: Yes. Formula 11 is a standard CVAE loss function, where the distribution of random variable $\epsilon\sim p(\epsilon)$ is assumed to be a standardized normal distribution.

Questions 2: How is $Z_T$ used for guidance ,which is mentioned in section 3.4.2?

Response: As shown in Figure 2, $Z_T$ is sent to step encoder in every step prediction procedure to provide the knowledge of where the goal is. (Figure 2 will be updated to make it clearer.)

Weakness 1 and Questions 1: Whether the TSC feature genuinely works better than the traditional CNN and heatmap approaches. And there is no ablation study provided to demonstrate the performance of replacing the TSC feature with traditional CNN and heatmap approaches. Explain the underperformance of all four approaches in ETH-UCY and demonstrate TSC feature genuinely works better than the traditional CNN and heatmap approaches.

Response: First, our TSC feature is not a plug-and-play module, nor is it a feature that can be straightforwardly replaced by a CNN feature or a heatmap. As explained in line 315 in our manuscript, TSC is a combination of coordinate encoding and scene encoding, which achieves the alignment of two different feature types. The prediction framework discussed in Section 3.4 is derived from our TSC feature, making it infeasible to replace the TSC feature with a CNN feature or a heatmap.

The CNN feature usually encodes the global scene as a vector, where the correspondence between coordinates and the scene around specific coordinates is lost. Heatmaps retain all spatial correspondences but cannot model pedestrian-pedestrian relations due to their computational complexity. Our TSC feature is capable of modeling both pedestrian-pedestrian relations and maintaining spatial correspondence to some extent.

Our method focuses on modeling the relations between the scene and pedestrians as well as among pedestrians. The SDD dataset, with its complex scene semantic maps, highlights the strengths of our method. As described in Table 2, our method outperforms all other methods. Without scene maps, predicting trajectories in this grid manner may not be as natural as coordinate auto-regression, yet our method still achieves comparable results.

Weakness 2: Different sample numbers in Y-Net and NSP-SFM.

Response: Y-net (NSP-SFM follows the same goal prediction framework) uses the best-of-20 metric. However, the 20 samples are obtained from clustering 10,000 samples from the heatmap (described as the Test-Time Sampling Trick in their supplementary material), which naturally covers larger possible areas of the scene.

Moreover, a comparison under the same setting is not practicable. Sampling 10,000 times in a heatmap is straightforward in Y-Net, whereas in our framework (and most of other methods with generation framework), sampling 10,000 times would requires going through the CVAE inference framework 10,000 times. This process is too time-consuming and cannot be completed within a limited time.

Weakness 3 and Questions 2: A more detailed statistical table should be included for irregular speed as author claims.

Response: Thanks for the valuable suggestion. Here, we define a velocity difference between history and future $V_{diff}$. Velocity is the absolute value of the Euclidean distance between two neighboring frames $v_t=||p_{t+1}-p_{t}||_2$. The history mean velocity is defined as the average velocity of all historical frames. Similarly, the future mean velocity is defined as the average velocity of all future frames. The velocity difference is then the absolute difference between the history and future velocities, where

$V_{diff}=|\frac{\sum_{t=1}^{\tau}v_t}{\tau}-\frac{\sum_{t=\tau+1}^{T}v_t}{T-\tau}|$.

We compute the ADE/FDE with different velocity differences. For example in SDD dataset with long-term prediction setting, for the 10% of samples with the largest velocity difference, our performance is 85.41/101.86, while Y-Net is 100.10/162.50. A more detailed comparison with the velocity difference distribution is included in the revised supplemental materials due to the page limitation of main paper.

Weakness 4 and Questions 3: Performance on more complex datasets, such as NBA[1].

Response: We evaluate our method on NBA dataset with 10-frame prediction setting, where the resutls are shown as follow.

Our method does not perform [1] but still achieves comparable performance when comparing to other methods in Table 2 in [1]. One possible reason is that trajectories in the NBA dataset are highly influenced by player-player interactions, as players need to react to each other’s movements, while the scene has little influence. Our method is designed to address feature alignment between the scene and trajectory, and its capability could be limited when scenes are excluded. The resutls are added to the supplimenal pdf file.

Weakness 1 & 2: The proposed model has shown tendencies to bias its trajectory predictions towards the centers of scene cells. The discussion of limitations is relatively brief. If the authors can provide some vivid examples, then it should be easier for readers to better understand.

Response: Thanks for the suggestion about the limitation discussion. As we discussed in line 525 in the paper, all the grids are centered on each pedestrian. It means that if the pedestrian has no movement, the goal prediction will be located at the center point of the central cell in the grid (with zero offset). With a non-negligible number of pedestrians without movement in the training set, this could bias the offset prediction.

Moreover, we compute the trajectory length distribution on SDD dataset with long-term prediction setting, where about 18% of the trajectory whose length is shorter than 1/10 of the cell edge size. To solve this problem, it requires further analysis and may requires an additional step to do the movement/non-movement classification in the future.

Weakness 3: It is still not clear if the proposed method is able to sufficiently address scenarios with higher pedestrian densities or complex environments especially in urban settings with numerous dynamic agents.

Response: A high-density scenario could be a task with different challenging points, as the relation modeling among different pedestrians has an O(N^2) complexity. With limited computational resources, the range of relation modeling needs to be carefully determined instead of including every pedestrian in consideration. Furthermore, interaction modeling may requires a specific movement simulation module to explicitly model the social interaction rules. We will carefully consider the possible solutions towards high-density scenario in our future works.