LaKD: Length-agnostic Knowledge Distillation for Trajectory Prediction with Any Length Observations

We sincerely appreciate your time and efforts on evaluating our work. Here are my responses to your comments:

Comment 1: The performance improvement compared to the naive Random Masking baseline is not very significant. I doubt whether it's worth the complexity to use this method in practice.

Thanks for your comment. Firstly, during testing, our model disables bi-directional self-distillation, and remains fully consistent with the backbone structure without requiring any additional inference time or resources. Additionally, our method enables existing models to effectively handle observed trajectories of varying lengths. Secondly, as shown in Table 1, LaKD outperforms Random Masking with relative improvements ranging from 3% to 11%. Additionally, Figures 4, 5, 6, and 7 in the appendix demonstrate that LaKD yields better results than Random Masking across various lengths of observed trajectories. For example, on the Argoverse1 dataset using HiVT as the backbone, LaKD achieves relative improvements of 21.2%/19.2%/10.0% at K=1 and 15.3%/18.8%/28.9% at K=6 over Random Masking in terms of minADE/minFDE/MR with 2-frame observed trajectories.

Thank you for your insightful review. The following are our responses to the points you have raised.

Comment 1: I'm a bit confused about why the overall framework of bidirectional self-distillation works from an information theory / flow perspective.

Sorry for any confusion caused by our unclear presentation. Traditional knowledge distillation involves transferring knowledge from a deeper teacher model to a shallower student model given the same input, which is a form of model capability transfer. However, the bidirectional self-distillation in our LaKD first randomly masks the same trajectory to obtain trajectories of different lengths. Then, knowledge is transferred from the features of good trajectories to those of bad trajectories extracted by the same model, which is a form of data-level information transfer. We expect that this approach will enable trajectories of varying lengths to exhibit strong features for future trajectory prediction.

Question 1: Could the authors explain further what do they mean by “Knowledge Collision”?

As mentioned above, since our framework uses a single shared encoder to extract feature representations for trajectories of different lengths and then performs data-level information transfer through this encoder, the feature representation of good trajectories can be disrupted when knowledge is transferred to bad trajectories through updates to the shared encoder. We refer to this problem as "Knowledge Collision". We will add more details in the final version.

Comment 2: For the qualitative analysis in Fig. 3, I could hardly understand why this shows that the proposed LRKD method is better than the other methods.

Thanks for your comment. Although it seems that there is no significant difference among the three predictions in the above scenario of Fig. 3, their results are 3.5891/6.0794/1, 1.4298/3.0295/1 and 0.6915/1.0974/0 in terms of minADE/minFDE/MR, respectively, indicating that our LaKD provides more accurate predictions. To more intuitively demonstrate the effectiveness of our LaKD method, we have included additional cases in the Figure 1 of the attached pdf to show that our method succeeds where others fail, as shown in the "global" response.

Regarding the behavior of the ground truth in the below scenario of Fig. 3, we infer that the vehicle stops at the road ramp. We think this can be inferred by the features of the vehicle, e.g., speed, acceleration, etc.

Question 2: May I ask how could a trivial degenerate solution, or a mode collapse, be avoided when conducting bi-directional self-distillation?

We apologize for the confusion caused by our unclear presentation. In our method, we design a dynamic soft-masking strategy to effectively conduct bi-directional self-distillation. During the training process, we set different soft-masking weights based on the importance of the units. This protects the more important units from significant damage that could affect the model's performance on trajectories of different lengths. However, these important units are still slowly updating, and their importance does not necessarily increase, as shown in Equation 7. Additionally, our dynamic soft-masking mechanism recalculates the importance of units whenever a new batch of data is encountered, avoiding excessive protection of any single unit. Furthermore, when the model's performance is poor, highly important units also need to be trained. Therefore, we lower their soft-masking weights at the early stages of training to prevent mode collapse, as shown in Equation 8. We will add more details in the final version.

Question 3: A question about the overall framework is that when we distill the representation of a long history into a short history, the short history lacks information about the longer past.

Thank you for your insightful question. During the training process, we can perform our LaKD to distill knowledge from the longer trajectory A to the shorter trajectory C (Note that A and C belong to the same trajectory), allowing C to capture the intrinsic information of A and achieve similar prediction performance. By this means, we expect the shared encoder to extract feature representations for trajectories of different lengths as precisely as possible. During testing, we disable bi-directional self-distillation and only use the backbone module. When a new trajectory B (i.e., B and A belong to different trajectories) comes, we directly extract its features using the shared encoder for future trajectory prediction, which can capture the distinguishing information of B.

Question 4: For Ablation Table 3, may I ask why the performance degrades when M is larger than 3? Intuitively, the performance should improve with a larger mask number?

Thanks for your comment. Since our method involves randomly masking historical trajectories M times in each training iteration and continues for a sufficient number of epochs, observation trajectories of all different lengths are seen during training, regardless of the value of M. Consequently, the model's performance does not significantly fluctuant as M changes, indicating that the model is not sensitive to M. This makes M easy to set in real-world scenarios.

Regarding the performance degradation when M is larger than 3, this occurs because, despite our use of a dynamic soft-masking mechanism to prevent knowledge collision, the feature representation of good trajectories may be compromised when knowledge is transferred to bad trajectories through updates to the shared encoder. This compromise becomes more severe as M increases, resulting in reduced performance. We will add these in the final version.

We truly appreciate the reviewer of the constructive feedback. In light of these insightful comments, we would like to address them by providing the following clarifications.

Question 1: The qualitative results do not clearly demonstrate the scenarios where the authors have been motivated. As shown in Figure 3, there is no difference between (b) and (c), which makes me doubt about the contribution of this work. Not just closer to the ground-truth, it is strongly suggested where the proposed method works while others fail.

Thanks for raising this concern which helps us to clarify the contribution of our work. Due to the limited time during the submission phase, we simply visualized the results to demonstrate that our method can more accurately predict future trajectories. Although it seems that there is no significant difference among the three predictions in the above scenario of Fig. 3, their results are 3.5891/6.0794/1, 1.4298/3.0295/1 and 0.6915/1.0974/0 in terms of minADE/minFDE/MR, respectively, indicating that our LaKD provides more accurate predictions. Following your suggestion, we have added more results to demonstrate that our method works while others fail, as shown in the Figure 1 of the attached pdf in the "global" response. Thanks again for the insightful comments, and we will add these results in the final version.

Question 2: Different backbone models other than HiVT and QCNet would be needed to claim its generality.

Thanks for the comment. We chose to use HiVT and QCNet as our backbones in our paper because they are the most advanced models in the field of trajectory prediction and have recently ranked highly on the Argoverse dataset leaderboard. Based on your helpful suggestion, we conducted experiments using two other typical trajectory prediction models, TNT [1] and Vectornet [2], as our backbones to show the generality of our method. The results are as follows:

The experimental results show that our method still achieves the best performance when using TNT and Vectornet as our backbones. We will add these results in the final version.

Limitations: The paper does not provide any failure cases or insights into the limitations.

Thanks for your comment. Due to the limitations of the paper's length, we have analyzed our algorithm's limitations in the appendix as :

In this work, we aim to distill knowledge from 'good' trajectory to 'bad' trajectory for improving the prediction performance from observations of any lengths. However, how to determine a 'good' or 'bad' trajectory is an open problem. Currently, we adopt a heuristic strategy by utilizing the distance between the predicted trajectory and the ground-truth trajectory. More complex strategies, such as reinforcement learning, are worth further exploration and investigation.

[1] Zhao H, et al. "Tnt: Target-driven trajectory prediction" PMLR 2021.

[2] Gao J, et al. "Vectornet: Encoding hd maps and agent dynamics from vectorized representation" CVPR 2020.