NoisyTraj: Robust Trajectory Prediction with Noisy Observations

• The relationship between the three proposed improvements, particularly the ranking loss, lacks a clear intrinsic connection. While the paper states that the ranking loss enhances performance, it does not provide a detailed explanation of why this is the case. It would be beneficial for the authors to include ablation studies on the hyperparameters of the loss function and clarify why adding ranking loss leads to better results compared to designing a prediction loss that only accounts for denoised prediction results.
• The experimental setup is insufficient. The two baselines used are overly simplistic, limiting the ability to assess the model’s effectiveness fully. Including the prediction results from the historical trajectory data before adding noise in all experiments would strengthen the comparisons.
• The paper lacks sufficient detail on the model’s training process. Specifically, whether the training dataset was also subjected to noise is unclear. Additionally, the limitation that the model training must be tailored to a specific trajectory prediction backbone, and that freezing the trajectory prediction backbone parameters may degrade performance, should be addressed in the main text.

• The recent paper (https://arxiv.org/abs/2312.15906) addresses transfer learning, but focuses specifically on making robust predictions in the presence of noisy trajectories within datasets. Since it deals with actual noise arising from errors in sensors, detection, and tracking processes, a comparison with this paper seems necessary.
• The assumption that $Y_{fut}$ is clean sounds somewhat unrealistic. Even for trajectories obtained via Lidar sensors, the mentioned errors from sensors, detection, and tracking processes are inevitable, although they might be less than those from camera-driven trajectories. It would be more convincing to show experimental results in settings where such two real-world different noise levels exist, for example, use clear trajectory as real Lidar-driven trajectories and noisy trajectory as real camera-driven trajectories.
• Typo: line 178, change X^ to X^$_{obs}$.

The task is new and challenging, there is not much real data to use for validation. This reviewer appreciate the author's efforts but maybe using additive gaussian noise is not the best way to simulate the realistic sensor input. It may cause model overfitting of certain types of the noise.

• Limited set of applications: The work tackles a simplified problem, in which the future ground truth data is assumed to be noise-free, while only the input observations are noisy. In general trajectory prediction tasks, e.g. in autonomous driving, this setup does not hold true. The authors give only one example where this method could be helpful: Use camera-only as inputs and camera + LiDAR as ground truth. However, even in this example camera + LiDAR are not noise-free as assumed in the paper but might have a lower / different noise than the input data. It would be of significant value if such a setup, where the ground truth data is not noise-free would be evaluated to confirm whether NoisyTraj still works well or whether it needs to be adapted. Furthermore, this very restrictive assumption should be mentioned at the beginning of the paper (if I'm not mistaken, it's only first mentioned in Section 3).
• Evaluation with higher noise values: /sigma=0.2 and 0.4 are evaluated but both noise levels are quite similar. How does NoisyTraj perform with larger noises, e.g. /sigma=2? From the results in Table 2, it also seems that the gap between NoisyTraj and the baselines is not widened besides the 2x noise increase. More data and analysis would be helpful here.
• Generalizability: The results in Table 5 actually show that the gap between the methods narrows when different train / test noises are used (while NoisyTraj still outperforms the baselines). Given that the baselines are rather weak (no good baselines seem to exist), it's unclear whether NoisyTraj truly generalizes well. Synthetic noise: The paper would be stronger if validation also covered real noise, e.g. from autonomous driving tasks. Real noisy observations with noise-free ground truth data is generally difficult to obtain though. One work that I'm aware of that analyzed the noise of a detection + tracking pipeline is https://arxiv.org/pdf/2004.01288, but they did not publish the dataset.

1. The major weakness of this paper is that the problem formulation does not align with real-world scenarios. The paper assumes that observed trajectories are corrupted by noise, and the authors add specific noise (e.g., Gaussian) to clear observations to simulate noisy observations. However, in practical applications, trajectory observation noise and errors are much more complex. In detection, not only do bounding boxes have noise, but there are also numerous false positives and false negatives. In subsequent tracking, bounding boxes belonging to the same subject need to be associated across different timestamps. Incorrect associations may link one subject's box at time T to another subject's box at T+1. Therefore, errors in the association process can result in tracked trajectories that significantly deviate from ground truth trajectories. The paper's assumption that noisy observations only stem from imprecise observed positions is fundamentally different from actual noisy/erroneous observations obtained from tracking, making it far from practical applications.

2. Another major weakness is the omission of several important prior works that have already explored trajectory prediction with noisy observations:

[a] Yu, Rui, and Zihan Zhou. "Towards robust human trajectory prediction in raw videos." 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2021.

[b] Weng, Xinshuo, Boris Ivanovic, and Marco Pavone. "MTP: Multi-hypothesis tracking and prediction for reduced error propagation." 2022 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2022.

[c] Weng, Xinshuo, et al. "Whose track is it anyway? improving robustness to tracking errors with affinity-based trajectory prediction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[d] Zhang, Pu, et al. "Towards trajectory forecasting from detection." IEEE Transactions on Pattern Analysis and Machine Intelligence 45.10 (2023): 12550-12561.

Different from NoiseTraj in this paper that focuses solely on position-based noise, these prior works consider the comprehensive impact of tracking-related issues on prediction, For instance, paper [a] provides an analysis of how various tracking problems (including noisy tracks, missed detections, spurious tracks, and ID switches) affect trajectory prediction performance. NoiseTraj could serve as a trajectory smoothing component, similar to how the Holt-Winters method is employed in [a].

3. Due to the strong coupling between tracking and prediction, addressing prediction upon noisy observations requires considering both tracking and prediction. However, this paper only focuses on prediction without considering tracking. Artificially adding noise to clear observations creates an artificial problem setting. Moreover, the evaluation metrics cannot simply adopt ADE and FDE from prediction upon clean observations. Since observed trajectories might be incorrect (rather than just having inaccurate positions), evaluation metrics need to consider both tracking and prediction. For example, ADE-over-recall curves are used in [a]&[e].

[e] Weng, Xinshuo, et al. "Inverting the pose forecasting pipeline with SPF2: Sequential pointcloud forecasting for sequential pose forecasting." Conference on robot learning. PMLR, 2021.

4. Figure 3 demonstrates that artificially noisy trajectories do not match the patterns of real-world noisy tracklets. The offset in the noisy (blue) trajectory exceeds several person-distances. In practical tracking associations, these points would never be associated with the same trajectory. For reference on normal noise patterns, see Fig. 5 in [a] (IROS'21).

5. In Section 4.2, the noise added to the SDD dataset is measured in meters, while the evaluation is in pixels. These experimental details need clarification.

6. Expression issues: a) Line 360: "blue" arrow should refer to the "green" arrow in Figure 2? b) Line 451: "Kalman" should refer to "Wavelet"?