Interactive Adjustment for Human Trajectory Prediction with Individual Feedback

We deeply appreciate the reviewer for the positive feedback and the thoughtful questions, and are glad to provide answers below. We have also revised the paper according to the suggestions. We have uploaded the revised version of our submission and highlighted changes in the main paper. The original version (P21-36) is attached below the Technical Appendices of the revised version as reference.

Q1: Theoretical justification.

We are glad to provide a theoretical justification as follows. We consider the base prediction model $\mathcal{M}$ learns a distribution $P _\mathcal{M}$ and makes predictions for a specific novel agent $\mathcal{A}$ at time step $T$ according to $P _\mathcal{M}$ given the observation of agent $\mathcal{A}$. Since $\mathcal{M}$ is optimized on a full set of training samples involving numerous individuals other than $\mathcal{A}$, there could be a gap between its learned distribution $P _\mathcal{M}$ and the real future trajectory distribution $P _\mathcal{R}$ of the agent $\mathcal{A}$. Therefore, we introduce individual feedback $\mathbf{F}$ of the agent $\mathcal{A}$, and our approach IAN as a function $f$, to produce $P _\mathcal{A} = f(P _\mathcal{M}, \mathbf{F})$ as a better approximation to the real distribution $P _\mathcal{R}$.

Q2: Overfitting the base model prediction.

We thank the reviewer for raising this thoughtful question. In this paper we use individual feedback to adjust base model's predictions so that they are more consistent to the target agent's specific patterns. During such process, the base model's parameters are not updated and therefore has no risk of overfitting.

Q3: Errors in base model prediction.

We would like to highlight that in IAN, the proposals are generated based on both base model predictions and individual feedback $\mathbf{F}$. It is our intention that the adjuster uses the individual feedback to generate proposals aiming at correcting potential errors of the base model predictions and therefore improve the performance.

We deeply appreciate the reviewer's suggestion on incorporating risk assessment measures. By assuming that risk assessment means "evaluating and managing uncertainties and potential hazards that could arise from predicted movements of agents", we suggest that one possible aspect of risk assessment is to check the predictions against collision with other agents.

Q4: More Explanations on IAN's architecture.

We thank the reviewer for the suggestion and kindly remind that such discussions are already provided in Sec.B of the Technical Appendices in the original version.

Q5: Description of figures.

We appreciate the reviewer for raising the concern and have added more descriptions of the process in the captions.

Q6: Limitation and future works.

Thank you for the suggestion! We have added discussion on limitations, failure cases and future works in Sec.E.5 of the Technical Appendices in the revised version.

Dear Reviewer Bpfi,

As the discussion period has come to an end, we would like to express our gratitude for the time you devoted to reviewing our paper. We deeply appreciate your thought-provoking questions, which have been very helpful in improving our submission. We also take great pride in having effectively answered them.

Best regards,

Authors of Submission 7210

We thank the reviewer for the positive feedback as well as suggestions for improvements, and gladly provide responses below. We have uploaded the revised version of our submission and highlighted changes in the main paper. The original version (P21-36) is attached below the Technical Appendices of the revised version as reference.

W1: Unclear writings.

We thank the reviewer for the suggestion and have added the algorithm parts in Sec.B of the Technical Appendices in the revised version for clearer understanding.

W2: Additional evaluation.

Thank you for the suggestion and we have added MissRate and Brier-ADE/FDE into the evaluation metrics. The results are shown in Tab.5 and Tab.6 in the Technical Appendices of the revised version. Here we only evaluate Brier-ADE/FDE using TUTR as base prediction model since only TUTR (among all base prediction models used in our paper) is capable of outputting confidence with its predictions.

Q1: Connecting the feedback.

As stated in L265-267 and L677-678 of our original version, we connect individual feedback, i.e. $\mathbf{F}$, into the networks by using it as the parameters of the adjuster. Such approach is naturally differentiable in both training and testing stages.

We thank the reviewer for the comments and especially the detailed suggestions to improve our presentation. In the responses below we address to the reviewer's concerns in detail and welcome any further discussions. We have uploaded the revised version of our submission and highlighted changes in the main paper. The original version (P21-36) is attached below the Technical Appendices of the revised version as reference.

W1 & Q1: Access to ground truths of previous predictions.

We believe that as long as an agent appears continuously in the scene for a duration longer than $\tau_{obs} + \tau_{pred}$, it is feasible in all practical scenarios for the ground-truth trajectories of the agent's previous predictions to be available for computing the agent's individual feedback. Here, $\tau_{obs}$ denotes the observation horizon and $\tau_{pred}$ denotes the prediction horizon.

We justify this point by clarifying that, according to the formulation of individual feedback in L156-183 of the original version, the ground-truth trajectories of previous predictions have already become the observation trajectories at the current timestep.

• Specifically, assuming an agent appears at time step 1 and $T$ denotes the current timestep, the ground-truth trajectories of "previous predictions made from time step $\tau_{obs}$ to $T-\tau_{pred}$" are actually the observation trajectory of the agent from time step $\tau_{obs} + 1$ to $T$.

• For example, the ground-truth trajectory of previous prediction made at time step $T-\tau_{pred}$ is the observation trajectory from time step $T-\tau_{pred}+1$ to $T$.

Therefore, the ground truths of previous predictions are naturally available through the progression of time in practical scenarios.

W2 & Q2: Differences from online learning approaches.

The essential difference between IAN and online learning approaches is that IAN does not update its parameters at test stage, as opposed to the very essential feature of online learning or test-time training approaches. This point fundamentally distinguishes IAN and online learning approaches, making them inherently different.

We have provided discussions about our differences with online learning/test-time training approaches and why the online learning approaches are not suitable for agent-specific adjustments in Section D.2 of the original version, and have further extended the discussions to adaptive works in Section E.3 of the Technical Appendices in the revised version.

W3: Analysis of loss function.

We thank the reviewer for the reminder and have added loss analysis in Section D.6 of the revised version.

W4: Moving Table 3 to the main paper.

We thank the reviewer for this suggestion! We have added a reference to Table 3 at the beginning of Section 3 to help address the notation issues, since we find it challenging to move Table 3 to the main paper due to the page limitations.

W5: Presentation.

We greatly appreciate the detailed suggestions provided in the comments and have improved these points in our revised version.

Q3: The inference time of IAN.

The average inference time of IAN is 0.02 seconds (L525). Assuming that the speed of a human is 1.5 m/s on average, he/she can travel only about 3 cm in such a short time (0.02s). Therefore, we suggest that the additional inference time is negligible.

Q4-1: Experiments on component isolation.

Currently our IAN features three main components, namely the feedback generator, the adjuster and the confidence network. Among these components, both the adjuster and the confidence network are essential to the pipeline and removing them would make IAN inoperable. Therefore, only the feedback generator can be removed from the pipeline as an ablation study, which we have provided in Table 2 (column "IAN w/o feedback").

Q4-2: Confidence Intervals.

We thank the reviewer for the suggestion, and have added confidence intervals of the main experiments in Table 4 of the Technical Appendices in the revised version.

Q5: For agents no feedback data is available.

If individual feedback is unavailable for an agent (meaning that the agent appears continuously in the scene for a duration shorter than $\tau_{obs} + \tau_{pred}$), IAN will not be applied to the agent. In such cases, the outputs of the prediction model will be used directly as the final outputs.

We deeply appreciate the reviewer's effort on reviewing our submission. Yet we respectfully suggest that there may have some misconceptions regarding our paper. We explain in detail below and are happy to answer any further questions. We have uploaded the revised version of our submission and highlighted changes in the main paper. The original version (P21-36) is attached below the Technical Appendices of the revised version as reference.

W1 & Q1: IAN v.s. prior works on adaptive trajectory prediction

We respectfully refute the comments since we have found some erroneous statements in the comments of this weakness and question, which may stem from some misunderstandings by the reviewer regarding our paper. Therefore, we believe it is unjustified to reject our submission based on this point.

We will address the comments systematically in the following order and rectify the inaccuracies within.

1. The comments say "this work is not the first to study feedback in trajectory predictions", however, we actually write "we are the first to adopt such agent-specific information into trajectory prediction tasks" in L112 of the original version.

   Here, such agent-specific information refers to "individual feedback, which reveals the prediction model’s performance in predicting a specific agent (an agent means a pedestrian or athlete)" in L108 of the original version. Please refer to the paragraph beginning from L155 of the original version for the mathematical formulation of individual feedback.

   Having conducted a thorough and comprehensive literature review, we are confident that our claim of being the first to study individual feedback (as distinct from other types of feedback or adaptation) is not overstated.

2. The statement "prior adaptive prediction work [1,2] that investigates this same idea" and "[1,2] focus on both temporal adaptation, equivalent to individual feedback, and geographic adaptation" is inaccurate.

   As suggested in the first point, we refute this statement by stating that our proposed individual feedback is very different from the temporal adaptation used in [1,2].

   By clarifying basic concepts that "a training/test domain" as "a specific scene with multiple moving agents" to avoid confusion, we explain how [1,2] and ours operate

   • [1,2]: The temporal adaptation actually uses the past trajectories of all the agents in the test scene as a whole to train or fine-tune the prediction model and update its parameters. Then, the prediction model trained or fine-tuned with the past trajectories of all the agents in the test scene is applied to predict the future trajectories of all the agents.

   • Ours: We collect individual feedback of each specific (single) agent and use IAN to adjust the output of the prediction model (not fine-tune the prediction model and update its parameter) for each agent respectively, i.e. individual feedback of a specific agent is only used to adjust the predictions on itself.

   Therefore, there are two significant differences between ours and [1,2]

   • Individual feedback is agent-specific, which differs from the scene-specific adaptation in [1,2].

   • [1,2] use data from the test domain to train or fine-tune the prediction model. In comparison, we only use the data from the train domain to train IAN as a separate model and do not use any data from the test domain for training. We also discussed about why our approach is different with test-time fine-tuning and why test-time fine-tuning is not suitable for agent-specific adjustments in Sec.D.2 (L806) of the original version.

Finally, we draw the following conclusions and answer the questions in the comments based on the responses above.

1. We suggest that our agent-specific individual feedback is a novel idea, different from previous work that studies scene-specific adaptations. Further, we do not use data from the test domain to train the IAN, which is different from prior adaptive works such as [1,2] that use data from the test domain to train the prediction model.

2. Why is IAN better or more preferable to use by practitioners?

   When we adopt approaches like [1,2], which are originally designed for scene-specific adaptation, for agent-specific adjustments, we need to train many separate prediction models with their own parameters for every agent in the scene. This is extremely costly for online deployment on autonomous vehicles and robots.

   In comparison, we propose IAN as an innovative solution to agent-specific adjustment, which brings little computational overhead.

3. How does this approach compare to prior works on adaptive trajectory prediction?

   As mentioned above, prior works like [1,2] are originally designed for scene-specific adaptation and not suitable for agent-specific adjustment, we suggest that this comparison is unnecessary and of limited significance.

Dear Reviewer,

Could you please look at authors feedback and see if your concerns (esp. the novelty / first work / comparison to prior literature) are addressed?

Thanks, AC

I thank the authors for their thorough response. To dive deeper into the two significant differences between the proposed approach and [1,2] (focusing in on the comparison to [2]):

1. Individual feedback is agent-specific, which differs from the scene-specific adaptation in [1,2].

Agent-specific feedback is a special case of the update equation (3) in the correction step of [2] ($w_{t+1|t+1} = w_{t+1|t} + K_{t+1}e_{t+1}$ is agent-specific when $K_{t+1}$ is diagonal)

2. [1,2] use data from the test domain to train or fine-tune the prediction model. In comparison, we only use the data from the train domain to train IAN as a separate model and do not use any data from the test domain for training. We also discussed about why our approach is different with test-time fine-tuning and why test-time fine-tuning is not suitable for agent-specific adjustments in Sec.D.2 (L806) of the original version.

The original writing (and author feedback) states that "Such trait indicates that when actually deployed, a distinct model for each of the individuals present in the scene needs to be stored and updated, which can be extremely costly for online deployment on embodied agents such as autonomous vehicles and robots." However, one of the core points of [2] is that, rather than updating the weights of the entire neural network (which indeed would be extremely costly), one only needs to update the last layer of the neural network (which entails a single prediction and correction step as detailed in Equations (2) and (3) of [2], which are akin to the prediction and correction step of a Kalman filter, which are extremely efficient to execute and are already executed, for example, as part of onboard robotic state estimation algorithms). To state this directly: there is no backpropagation happening onboard in [2]!

For these reasons, I am maintaining my original review score.