Direct Post-Training Preference Alignment for Multi-Agent Motion Generation Model Using Implicit Feedback from Pre-training Demonstrations

The authors have addressed most of my concerns during the rebuttal and I am happy to raise my score.

Q3. What are limitations of AFD? How many human direct annotations do you need vs how many demonstrations of how many cars do you need? How does alignment improves in terms of the labels provided in both cases? (scaling concern in multiagent setting)?

A3. In general, both pre-training and post-training preference alignment benefit from an increase in demonstration data, provided that the model capacity is properly scaled. In our experiment, we used all the Waymo open motion dataset for pre-training. Since our work focuses on the post-training preference alignment, we only demonstrated the effects of scaling preference data.

While our work does not directly investigate sample efficiency of pre-training and scaling of expert demonstrations, we recognize their potential impact on alignment and are excited to explore these implications in future work, particularly the relationship between pre-training sample efficiency and post-training preference alignment.

Q4. If demonstrations are multi-modal, will your method of comparing sampled based of OT-distance metric introduce conflicting gradients and leading to mode collapse?

A4. Thank you for this insightful question! As noted in LLM research, preference alignment methods (e.g., RLHF or contrastive preference update) inherently could lead to mode collapse and reduce diversity [1]. In such cases, current preference alignment algorithms may disregard minority preferences, leading to an overemphasis on majority preferences and a potential loss in output diversity.

In our experiments, we observed that the fine-tuned model generates significantly fewer unsafe modes, reflecting the alignment’s effectiveness in suppressing undesirable behaviors. To assess the impact on mode diversity, we calculated the L2 distance between each pair of trajectory modes among 32 generated modes (averaged across agents and traffic scenarios). A higher L2 distance indicates greater geometric diversity in generated motions. As shown in the table below, we did not observe a significant drop in mode diversity after alignment.

Nevertheless, we are excited about further investigating mode collapse and exploring mitigation strategies in future work. For example, explicitly modeling and optimizing for multi-modal rankings could help ensure that minority preferences are better captured, maintaining diversity while aligning with human preferences.

[1] Xiao, Jiancong, et al. "On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization." arXiv preprint arXiv:2405.16455 (2024).

Q5. How does your OT-distance metric factor in collision / progress / comfort features?

A5. We use the following features [collision status, distance to road boundary, minimum clearance to other road users, control effort, speed] to construct the feature vector $\phi$ when solving the coupling matrix (2).

Q6. Do you have qualitative figures that help readers understand why L2 distance drop at the end in Fig 3? Why does L2 distance metric will lead to missed turn in Fig5?

A6. L2 distance drop at the end in Fig 3. We would like to clarify that the L2 distance in Fig. 3 is used as a controlled variable. Specifically, we select sampled traffic simulations from the reference model and analyze the relationship between their group-averaged ADE to the expert demonstrations and their realism. The purpose of this analysis is to demonstrate how L2 distance correlates with model realism.

The sharp drop in the blue line of Fig. 3 means that when we try to select sampled traffic simulations with even smaller ADEs, this does not help improve the realism.

Why does the L2 distance metric lead to missed turn in Fig5? When using ADE to rank the generations, we compute the average ADE across all agents in the traffic simulation. In Fig. 5, while the vehicle highlighted near the red circle in the DPA-ADEF figure missed its turn, the overall ADE of the traffic simulation is significantly better than that of the reference model (reference Model ADE: 5.93, DPA-ADEF: 3.17) and slightly better than DPA-OMF (ADE: 3.36).

This example demonstrates that optimizing ADE does not necessarily lead to an improvement in realism. ADE primarily captures geometric proximity to expert trajectories but does not account for task-critical aspects.

Dear reviewer 3Bbi,

We are glad that our responses could address your questions, and we appreciate your reconsideration on the score of our paper! Thank you again for taking the time to review our paper and providing the insightful comments!

Best regards,

Authors

Q2. Some figures are confusing. Fig.1: What are the orange lines on the left above the motion token pred. model? A-hat is not explained. Fig.3: I don't know how to read this diagram. What's the takeaway? Fig.6: I'm completely lost as to what I'm supposed to do with these.

A2. We would like to thank the reviewer for raising these questions and we apologize for the confusion caused by insufficient explanations. We have added the following detailed explanations in the revised paper.

Orange lines in Fig. 1. In Fig. 1, the orange elements represent components associated with our proposed approach, DPA-OMF. Specifically, the gray dotted lines above the motion token prediction model indicate the reference model’s action distributions at each prediction step. The orange lines illustrate how these probabilities are updated after fine-tuning to align with human preferences.

hat notations. The hat notations represent the sampled actions during inference. Specifically, during inference, at each prediction step, actions from the previous step are sampled from the predicted distributions. These sampled actions are then used as inputs to the model to predict the conditional probability distribution for the current step's action tokens.

Fig. 3. Both ADE and our preference score try to measure if a motion generation is close to the expert demonstration. The key difference is that ADE calculates the L2 difference between motion geometries, while our preference score is derived from IRL and measures alignment between occupancy measures. The purpose of Fig. 3 is to show that our preference score correlates more strongly with the realism of generated motions, making it a valid metric for constructing preference rankings.

We demonstrate this in a post-selection analysis, where we select sampled traffic simulations from the reference model and analyze the relationship between their group-averaged distance (ADE or preference distance) to the expert demonstration and the realism of these samples (i.e., we control the distance to the expert demonstration, and measure the realism of the selected sampled traffic simulations).

In Fig. 3, as ADE decreases, the realism of model generations initially improves. However, beyond a certain point, further reductions in ADE have diminishing returns in terms of realism. In contrast, as we reduce the preference distance, we observe a stronger and more sustained correlation with realism, allowing the preference score to push realism from 0.725 to 0.76.

The takeaway is that our preference metric better captures model realism (alignment with expert preferences) than ADE, supporting its use as a basis for constructing preference rankings in our approach.

Left figure of Fig. 6. The left side of Fig. 6 illustrates how our approach, DPA-OMF, improves the reference model’s performance as the size of the preference dataset increases. A key advantage of DPA-OMF is its ability to leverage expert demonstrations and model generations to automatically construct preference rankings without requiring additional human annotations, making it highly scalable. However, due to limitations in training infrastructure, we were unable to scale the preference data to the desired level to fully maximize performance. Nevertheless, the observed scaling trends demonstrate that the performance of our approach improves with larger preference datasets. This suggests that, with enhanced training resources, our approach could achieve even better results.

Right figure of Fig. 6. The right side of Fig. 6 illustrates the phenomenon of preference over-optimization, a topic studied in most of the LLM preference alignment works. Since our work shares many connections with LLM alignment, we conducted a similar investigation in the context of multi-agent motion generation. Preference over-optimization tries to understand how much improvements we can gain as we allow the optimized model to deviate further from the reference model. The key takeaways from this plot are twofold: 1) aligning the model with a small preference dataset can actually hurt the performance of the reference model, rather than improving it. 2) if the optimized policy is allowed to deviate from the reference a lot (by applying a smaller weight to the reference model deviation cost during alignment), eventually the performance will degrade, which is consistent with findings in LLMs. However, increasing the amount of preference data can help mitigate this degradation. These two takeaways emphasize the critical role of large-scale preference data in improving imitative token prediction models, and further motivate our approach, which leverages expert demonstrations to construct scalable preference datasets.

Q3. The writing could use a proofreading pass. There are some minor spelling issues throughout.

A3. We would like to thank the reviewer for this comment. We have fixed the typos and grammar mistakes in the revised paper.

Dear reviewer mv7M,

We are glad that our responses could address your questions, and we appreciate your reconsideration on the score of our paper! Thank you again for taking the time to review our paper and providing the insightful comments!

Best regards, Authors