Motion Forecasting in Continuous Driving

We thank the reviewer for the detailed review as well as the suggestions for improvement. Our response to the reviewer’s comments is below:

Apologies for these unclear explanations.

• Data reorganization: As shown in Fig. 2(b), the dashed part refers to historical time steps (50 historical steps in AV2), and we select several split points on this part. We consider an original trajectory (the gray arrow) as a sequence, and our processing approach generates several shorter trajectories (sub-sequences). Besides, the split points are manually specified in our method and discussed in ablation Tab.4(b).

• Mode: All correct! Multiple trajectories are predicted for each agent to enable comprehensive motion possibilities, in which case $mode$ is commonly used to represent each prediction. We will make it clear. Besides, the dimension of $Y_{mo}$ is $[B, N_a, N_{mode}, 2]$.

• Model output: The output is several predicted trajectories with corresponding probabilities, as the reviewer understood. While providing the types of loss functions in Line 177, we indeed overlooked the concrete input items. We will clarify this.

We thank the reviewer for the detailed review as well as the suggestions for improvement. Our response to the reviewer’s comments is below:

Q1: Drastic improvement on $ADE_1/FDE_1$.

By comparing QCNet and other methods (such as ProphNet), it can be seen this phenomenon is common. This is due to different calculation of metrics: the metrics@1 is the error for the one selected trajectory according to the score, while the metrics@6 is the minimum error among 6 predicted trajectories. Hence, there is more room for improvement for the metrics@1.

Regarding the comparison between QCNet and ours, the improvement in $FDE_{6}$ is still salient. Besides, we provide the complete comparisons below. Since we can only integrate our restructured data, please refer to Tab.3 row 1 and 2 for comparison.

Q2: The comparisons are not fair.

We do NOT use more past information compared to existing methods, but just process past information in a different way with the same datasets. Taking existing methods on Argoverse 2 as an example, they directly process each independent scenes with 50 historical frames, while we split each scene into several continuous small scenes with less past information (30 historical frames), then processing them in a streaming fashion. Hence, existing methods and RealMotion all use the same past information, making a fair comparison.

Q3: Loss functions.

Apologies for unclear expression. We will add the formulation for these losses. Especially, $L_{\rm refine}$ is utilized to supervise the refined trajectories $Y_{\rm mo}$ mentioned in Eq(4), which can be formulated as $L_{\rm refine} = {\rm smoothL1}(Y_{\rm mo}, Y_{\rm gt})$, where $Y_{\rm gt}$ refers to the ground truth.

Q4: Fig.2.

Fig.2 depicts how we process existing datasets with independent scenes into a continuous format to simulate real-world situations. We will remove some redundant elements to make it clearer.

Q5: The architecture of RealMotion

To ensure the generalizability of our proposed modules, as most current methods we adopt a standard forecasting framework, which consists of an agent encoder and a map encoder to respectively encode agent and map features, a Transformer encoder to implement interaction, and a decoder to generate trajectories and corresponding probabilities. The figure is shown in PDF file Fig.1.

Thanks for the quick response.

Exactly. This is more than just simply considering longer historical frames like previous methods, otherwise, our method would not exceed previous alternatives. Our formulation allows to capitalize the sequential data more effectively in real world scenarios, not only emphasizing a longer historical scene context but also leveraging historical predictions, which cannot be achieved by simply increasing historical frames. We will further clarify in the revised version.

Thank you for your answers and your clarifications. After going through the paper in light of the provided answers, it seems I indeed misunderstood an important part of the submission.

To make sure I'm getting it right this time, would you agree your working hypothesis could be sumarized as follows?

• Given a large history (of 50 frames), it is better to have a model that operate on a smaller sequence (30 frames) + an embedding of the past that you can obtain using the same model on earlier frames, than directly on the full (50 frames) history like QCNet does.

Dear Reviewer GAo1

Thanks again for the valuable comments. We believe our responses addressed all the questions/concerns. It would be great if the reviewer can kindly check our responses and provide feedback with further questions/concerns (if any). We would be more than happy to address them. Thank you!

Best wishes,

Authors

Indeed, after going again through the submission with the new information from the rebuttal and the reviews, I find my concerns has been adressed properly in the rebuttal. Moreover, I believe the reasons for my initial wrong impression have also been mostly adressed thanks to the discussions between the authors and Reviewer fgWb. Since the authors have commited to make the necessary change, I'm happy to increase my rating.

We thank the reviewer for the detailed review as well as the suggestions for improvement. Our response to the reviewer’s comments is below:

Q1: Missing related work.

Thanks. We will add them.

Q2: Comparison with QCNet.

To ensure efficiency, our method adopts an agent-centric design different from QCNet. As mentioned with QCNet only the query-centric design is good at continuous motion forecasting and GRU is employed for fusing continuous predictions in the current scene. Hence, it should still be considered an independent method.

We have tried to integrate our stream modules with QCNet, but unfortunately, it cannot be trained in our environment due to huge memory overhead. We hence just provided the results using our reorganized data in Appendix B.3.

Q3: Continuity metrics.

Following Pang et al., we evaluate the $fluctuation$ for RealMotion and baseline using our reorganized data as shown below. It can be observed that our method certainly improves the $fluctuation$ metric, with the Trajectory Stream playing a particularly important role.

We also provide some qualitative results in PDF file Fig. 2, which demonstrates that RealMotion can better maintain temporal consistency of trajectories.

Thanks. We will further highlight the crucial parts.

We thank the reviewer for the detailed review as well as the suggestions for improvement. Our response to the reviewer’s comments is below:

Q1: Some of the framing to be a bit misleading.

Apologies for clarity issue in abstract and introduction. We intent to stress the significance of streaming design in real-world settings. We will improve the clarity.

As for Fig. 1, we discuss in real-world setting where scenes should be continuous, going beyond the datasets. In this case, current methods can only process independent scenes with limited history as set up in current datasets, while we advocate to utilize more abundant historical information.

Q2: Method description.

Apologies for unclear expression.

• 83: Indeed, the $length$ refers to time steps instead of meters. We will clarify this part.

• 114: It is not for past steps but for other agents. In addition to predicting the trajectories of agents of interest for evaluation, we also predict a single trajectory of other surrounding agents (not involved in evaluation) for auxiliary supervision. That is typical and beneficial for learning better motion pattern.

• Equal length: We assume this refers to Line 83 -- that means the newly generated sub-trajectories have equal total steps, but the number of steps in the history and future can differ.

• refine loss: $L_{\rm refine}$ is utilized to supervise the refined trajectories $Y_{\rm mo}$ mentioned in Eq(4), which can be formulated as $L_{\rm refine} = {\rm smoothL1}(Y_{\rm mo}, Y_{\rm gt})$, where $Y_{\rm gt}$ refers to the ground truth trajectories.

• RealMotion: In ablation, RealMotion without stream modules can be considered as a one-shot forecasting models similar to current methods. This variant cannot fuse temporal information, but utilizing our reorganized data as a data augmentation method is still possible. In Appendix B.3, it should refer to only using our reorganized data, which we will clarify.

Q3: Comparison with SEPT.

We consider that SEPT is a pretrained self-supervised method orthogonal to ours. For example, it utilizes all sets (including test set) to address the disparity between train set and test set, leading to more data used than current methods and making the direct comparison unfair. A property way of using SEPT as model pretraining is likely to improve general methods like ours, and we could include them for further improvements.

Q4: More ablations

• Actually, the current chunk has attended to several preceding chunks. For the trajectory stream, we explicitly maintain a memory bank for more history predictions from a longer history. For the scene stream, since previous chunk involves earlier context, the current can also utilize a longer context to some extent through implicit propagation. Besides, We have tried to propagate two frames of scene context as shown below:

  	Latency	Memory	$minFDE_6$	$minADE_6$
  2 frames	23ms	1.8G	1.28	0.64
  1 frames	20ms	1.4G	1.31	0.66

  This strategy can indeed bring slight improvements, but we finally did not adopt more frames due to considerations of simplicity, efficiency and memory overhead, especially in complex scenarios such as intersections.

• It is an excellent insight! To make the data more aligned with the forecasting task, we did not process future trajectories (such as Waymo mentioned below) in this paper, but it is indeed worth integrating and exploring distinct motion tasks (simulation, prediction and planning) and datasets. We intend to further extend our model in future work.

Authors have answered my technical and clarification questions in a satisfactory manner (Q2 - Q4). Please try to address them in a future manuscript version, some of these details are key to the method (e.g. trajectory stream attending to multiple past frames and L_refine definition etc) but are not sufficiently well described.

Wrt my Q1 I am not particularly reassured yet. Please explain what your new pitch is in 2 sentences. Also, while in theory your method can benefit from longer historical information, in practice you have not tried it, so it is not proven. What seems proven to me is that presenting the information in a streaming fashion may be superior. Fig 1 is confusing because it shows that for standard methods, each frame attending only its own timestep independently. This is not correct since it attends X frames of previous data but it is not shown (yet the past frames are shown in yellow for RealMotion).

Thanks for the recognition.

We cannot provide the evaluation of longer history due to there is no temporal relationships between different scenes in existing benchmarks. Once such longer history data becomes available, it is definitely insightful to test our approach then. That being said, we have now tested the longest possible history with the existing benchmark sampled from the real-world driving data.

Regarding Fig.1, please note each blue circle represents a specific scene rather than a single frame, which consists of historical trajectories and HP map within a certain range. In this case, current methods independently process every individual scene, while RealMotion is designed to model the temporal relationships across successive scenes. We will further clarify to minimize similar misunderstanding.

please explain what your updated intro contribution statement relative to existing methods is, in 2 sentences" to see how it would address my overclaiming concerns.

Thanks. Existing methods tackle motion forecasting task in a scene-independent manner, which could limit the model performance in real-world settings where motion is typically continuous while ego-car drives on. Upon this observation, we propose temporal relationship modeling across scenes and introduce an effective approach, RealMotion, to realize this idea in a two stream framework.

I suggest at the very least you augment the figure caption to clarify that each blue scene circle contains X seconds of past history, or some such.

Thanks for the suggestion and we will revise as suggested.