Ready-to-React: Online Reaction Policy for Two-Character Interaction Generation

Narrow background.

We strongly disagree with the statement that "the background of this work is too narrow and I do not think it is important to study." As noted by R2-T1pE, the task holds significant importance in two-person interaction motion generation. Additionally, R1-odxr and R3-qtLW emphasized the value and contribution of our proposed dataset to the community. Furthermore, we demonstrate that our method can be effectively generalized to other datasets (e.g., Inter-X) in the general response and $\textcolor{green}{\text{Appendix H}}$.

Lack of novelty.

Our primary focus is on two-character animation generation, and the core contribution lies at a conceptual level in addressing a task that better aligns with human interactive processes. This provides a fresh perspective and technical insight into motion generation for interactive scenarios.

As the reviewer R3-qtLW noted, the method is "simple yet effective and versatile", which aligns with our goal of real-time generation. We deliberately opted for a simpler architecture to achieve this objective.

Furthermore, this work explores the use of diffusion in real-time interaction tasks, a direction that has not been explored before, offering new insights into the field. The results demonstrate significant improvements over baseline methods, highlighting the effectiveness of our approach and its potential to inspire future research.

Generalize to other datasets.

Please refer to the general response: additional results on the Inter-X dataset and $\textcolor{green}{\text{Appendix H}}$.

Clarify how the diffusion head helps motion diversity.

The term "ensure" in the introduction (L72) is intended to convey "preserve", rather than "help". Previous works (e.g., MDM [1]) have demonstrated that conditional diffusion can enhance diversity, providing evidence for this aspect.

Pre-programmed baseline.

We speculate that you are discussing motion matching methods commonly used in games. Motion matching (motion matching example) relies on a set of manually designed features to match motions from a predefined library. It is primarily used to generate motions based on existing user inputs. In contrast, our method is capable of directly generating interactive boxing motions for two characters without requiring any user input. In games, automated boxing agents often rely on extensive rules to generate commands, which are then used to drive motion matching and produce specific character motions. We have not found any open-sourced baselines specifically designed for this task. If you could recommend one, we would be happy to compare our method with it.

---

[1] Tevet, Guy, et al. "MDM: Human Motion Diffusion Model". In ICLR. 2023.

Generalize to other datasets.

Please refer to the general response: additional results on the Inter-X dataset and $\textcolor{green}{\text{Appendix H}}$.

The dataset will be released as stated at L48 in the main paper.

The design choice of single-layer MLP in diffusion.

As stated in Appendix C (L795–L797), we explored various network architectures during the design phase. However, we ultimately found that a single-layer MLP not only offers faster inference but also achieves a lower FID. Recently, we surprisingly found a blog that has a similar conclusion to us. They found that an auto-encoder with 2 layers performs better than 8 layers.

We also add an ablation study towards the diffusion network layers. Below are the explanations of the ablated versions and results in the two-character setting. We also updated the whole table and details in $\textcolor{green}{\text{Appendix G.3}}$.

• 1-MLP(Ours): our single layer MLP.
• N-MLP: using N layers of MLP.
• N-ResNet: using N layer of ResNet. The ResNet is adopted from Stable Diffusion. We make some modifications: (1) we use linear norm, (2) we use linear layer instead of convolution.

From the table, we can conclude that the 1-MLP design can achieve better performance. We considered several possible reasons to explain this phenomenon. First, more complex networks may overfit the training dataset faster, potentially leading to poorer performance on the test dataset. Second, since the diffusion denoising process involves 1000 steps, simpler transformations at each step might already suffice to achieve the desired results.

Clarify some details in the method.

• Reason of excluding the y-axis of r_off and r_dir in agent motion representation. Since we are dealing with an auto-regressive problem, we place significant emphasis on addressing error accumulation. If we define the root as the position and rotation of the pelvis joint, it may result in vertical movement along the y-axis (gravity direction). However, boxing takes place on flat ground without elevation changes. Therefore, defining the root as the projection of the pelvis onto the ground helps mitigate this issue. Additionally, we align the x and z axes of the root coordinate system parallel to the ground, transforming root motion into purely 2D planar movement. This ensures that the pitch orientation of the root does not accumulate errors over time.

• Reason of excluding the opponent's r_off and r_dir as input is that r_off is determined by the opponent’s pelvis position, and r_dir is determined by the opponent's shoulder and hip joints. Therefore, including this information does not significantly impact the agent’s ability to understand the opponent’s movements. We also experimented with adding the opponent’s r_off and r_dir as conditions, and it showed no noticeable effect.

• Reason of encoding the agent by VQ-VAE while encoding the opponent by MLP. There are two main reasons. The first is to preserve the relative positional information of the opponent. Encoding the opponent using the same VQ-VAE trained for a single character would result in the loss of global information relative to the agent, which is crucial for accurate interaction modeling. To incorporate this information effectively, we were inspired by recent VLM models (e.g., LLaVA), which use fully connected layers to inject additional conditions into transformers. This approach ensures the opponent’s information is integrated efficiently into the model.

• The process of the two-character motion generation is conducted in an auto-regressive manner. A pseudo-algorithm is provided in Appendix B. This process does not require two separate trained networks; both agents share the same network.

Training and inference protocol of Interformer.

Interformer generates motions auto-regressively by predicting the next raw pose of the agent at each step. As a result, its training process does not require modification. We simply applied our inference protocol during the inference stage.

The transformer notations in Figure 2 have been updated to avoid ambiguity in the $\textcolor{green}{\text{revised main paper}}$.

The notation of "d" has been clarified to distinguish between the down-sampling factor and the initial number of frames in the $\textcolor{green}{\text{revised main paper}}$.

Generalize to other datasets.

We follow your suggestion to test our method on the Inter-X dataset. Please refer to the general response: additional results on the Inter-X dataset and $\textcolor{green}{\text{Appendix H}}$.

Include "boxing" in the title.

We acknowledge that the experiments in the main paper were conducted solely on boxing. However, our method was not specifically designed for boxing. Regarding the design based on root orientation, we believe it is a general consideration applicable to all two-person tasks.

We have reported the performance of our method on the Inter-X dataset. If you and other reviewers still feel the title should explicitly reflect this focus, we are open to revising it accordingly.

Suboptimal reaction movements.

We acknowledge that there is still room for improvement in the reactive movements. To further analyze the impact of contact on the generated results, we report the agent’s movement relative to the opponent’s movement following reviewer R1-odxr’s suggestions. Details and results are shown in $\textcolor{green}{\text{Appendix J}}$.

Foot sliding metrics are reported as the "FS" metric in L313–L315 and all tables in the main paper.

Penetration metrics.

Our skeleton representation is based on positions and rotations exported from motion capture software. In the main paper, we visualize the predicted rotations by mapping them onto the "Xbot" model for demonstration purposes. Note that the Xbot skeleton differs from the one we use in our method, which makes penetration evaluations based on Xbot less reliable. As a result, penetration cannot be directly evaluated at the mesh level.

To evaluate penetration between skeletons, we approximate the bones using triangular prisms with a distance from the centroid to a vertex = 5 cm and calculate penetration frame by frame between the two body meshes. Since boxing typically involves instantaneous contact, the proportion of frames with penetration is expected to be very low. Therefore, we report the total number of frames with penetration across the entire test set and the mean penetration volume in $cm^3$. The two-character setting results are shown below and the whole table and more details are in $\textcolor{green}{\text{Appendix I}}$.

• # Penetration represents the number of penetration frames.
• Mean volume represents the mean penetration volume ($cm^3$) over penetration frames.

From the table, we observe that our method achieves a lower mean penetration volume compared to the baselines, demonstrating its effectiveness in avoiding penetration.

The impact of sparse contact on motion generation.

To analyze the impact of sparse contact on motion generation, we first detect contact and then assess whether the agent moves backward or forward in response. Specifically, we identify contact by calculating the distance between one character’s skeleton mesh and the other’s hand joints. If the distance is less than 5 cm, we consider that the hand has successfully made contact with the other. This could include the opponent's hand hitting any part of the agent or vice versa. For these identified contact frames, we examine whether the agent’s root moves backward when the opponent's root moves forward and compute the proportion of such occurrences. For each frame, we switch the roles of "agent" and "opponent". The two-character results are shown below and the whole table is in $\textcolor{green}{\text{Appendix J}}$. A higher alignment with the ground truth indicates better performance.

• # OF represents the number of frames that the opponent moves forward.
• # OF_AB represents the number of frames that the agent moves backward when the opponent moves forward.
• Ratio represents # OF_AB / # OF.

From the table, we observe that our method achieves performance closer to the ground truth, demonstrating its ability to produce more realistic reactions.

Lacks explicit contact modeling.

Many motion generation works, such as MDM [1] and ReGenNet [2] and all of our baselines, focus on skeleton-level generation without directly addressing contact or penetration because dealing with dense contact often requires dedicated research efforts. These aspects are typically considered when addressing more intricate interaction scenarios. For studies explicitly addressing penetration or interaction within environments, works like [3] and [4] provide excellent references.

We agree that contact modeling is crucial, and we will consider incorporating explicit contact modeling in our future work to enable applications in real robots. Thank you for the valuable suggestions.

Ethics concerns have been updated in $\textcolor{green}{\text{Appendix A.3}}$.

---

[1] Tevet, Guy, et al. "MDM: Human Motion Diffusion Model". In ICLR. 2023.

[2] Xu, Liang, et al. "ReGenNet: Towards Human Action-Reaction Synthesis." In CVPR. 2024.

[3] Starke, Sebastian, et al. "Neural state machine for character-scene interactions." TOG. 2019.

[4] Huang, Zeyu, et al. "Spatial and Surface Correspondence Field for Interaction Transfer." In SIGGRAPH. 2024.