Think Then React: Towards Unconstrained Action-to-Reaction Motion Generation

We deeply appreciate your positive evaluation and encouraging feedback on our work. Below, we address your questions and suggestions in detail.

• Q1. Qualitative results and comparisons

Good remark! We have provided qualitative comparisons and failure cases on an anonymous webpage: Think-Then-React.github.io. We have included some of them in our revised paper (Line 432-449).

• Q2. Pose tokenizer in main paper and appendix

We introduce the architecture of our pose tokenizer in main paper Section 3.2.1 and elaborate implementation details of the pose tokenizer in Appendix A.1, rather than space tokenizer. We notice that the citations between these sections are inconsistent, as both of the related works use a similar VQ-VAE architecture. We have now corrected these inconsistencies between the main paper and the appendix (Line 652).

To further clarify, the space tokenizer is responsible for encoding spatial features, such as the (x, z) position and human body orientation (e.g., <x0, z0, r0> denotes a person facing the positive z-direction at the center point). This maintains absolute spatial information for both individuals. The pose tokenizer auto-encodes normalized human motions into token sequences (e.g., <p32><p48><p96> could describe "one person reaching out with their right hand"), which can then be used by LLMs for understanding and predicting reactions.

If there are any further inconsistencies or points that need clarification, please let us know, and we would be happy to address them.

• Q3. Pose-to-Pose training detail

The phrase "and vice versa" refers to the fact that we perform both operations: 1) Predicting a_second and b_first from a_first and b_second, and 2) Predicting a_first and b_second from a_second and b_first. We have clarified this in Section 3.3.1 (Line 289-291).

• Q4. Exclusion of y-axis from space representation

Most motions typically begin with the y-coordinate set to 0 (as they generally don’t start in mid-air), so we exclude the y-axis from the absolute pose for simplicity and to avoid introducing redundant information. However, subsequent pose tokens still capture vertical motion dynamics, such as jumping or climbing, through the changes in the encoded token sequence. We have added relevant clarification in Section 3.2 (Lines 217-218).

• Q5. Notation and phrasing issues

We have revised inconsistent notations and improved clarity in Sections 3 and 4, including corrections for "k," "M," and "L" and unifying timestep notation with t. Additionally, we rephrased unclear passages in Sections 3.3.1 and 4.

• Q6: Definition between pose and motion

To clarify the terminology, we use following definitions: 1) Pose refers to a human body posture or movement within a brief time frame (i.e., timestep), such as "taking one step forward." A pose can be represented as a single token. 2) Motion is a sequence of poses, starting with an initial spatial state represented by space features. For example, a motion might be "a person walks three steps."

• Q6. Qualitative comparisons and failure cases

Good remark! We have provided qualitative comparisons and failure cases on an anonymous webpage: Think-Then-React.github.io. We have included some of them in our revised paper (Line 432-449).

[1] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. CVPR 2022

[2] Han Liang, Wenqian Zhang, Wenxuan Li, Jingyi Yu, and Lan Xu. Intergen: Diffusion-based multi-human motion generation under complex interactions. IJCV 2024

We sincerely appreciate your valuable comments and recognition of our work. Below, we provide detailed responses to your questions.

• Q1. Effectiveness of unified tokenizer and settings of MotionGPT

Thank you for your insightful suggestion. To directly assess the effectiveness of our unified tokenizer, we have evaluated a plain tokenizer (i.e., a non-decoupled motion tokenizer that directly encodes multi-person motion) on motion reconstruction and action-to-reaction generation (i.e., use the tokens encoded by the plain tokenizer for LLM training and inference). The results in the table below indicate that the plain tokenizer fails to accurately reconstruct multi-person motion, leading to ambiguous motion-text understanding during training and inference. This hinders LLM's ability to effectively perform the thinking-then-reacting process, resulting in deteriorated reaction generation performance. We consider including this analysis in the final revision.

Regarding MotionGPT, we extend the motion tokenizer of MotionGPT to encode multi-person motion while keeping other settings unchanged. However, TTR w/o ALL PT is not equivalent to MotionGPT w/ Unified Representation. This is because we propose a thinking-then-reacting mechanism that fully leverages the inference capabilities of LLMs to mitigate accumulated errors in action-to-reaction generation. We added adaptation details of MotionGPT for multi-person scenarios to Section 4.1 (Line 365).

• Q2. A pair of persons or the reacteee facing the positive z-axis?

You are right. Only the reactee is aligned to face the positive z-axis when we reposition the reactee at the origin of the absolute coordinate system. This adjustment is done while preserving the relative spatial relationship between the two persons. Then the absolute spatial features of each individual, including their x and z positions and body orientation r, are extracted as space tokens. As shown in Figure 2(a), the reactee’s absolute spatial feature is represented as (x0,z0,r0), meaning they are positioned at the origin (x0,z0) and oriented to face the positive z-axis (i.e., r0). The other person's spatial feature is (x2,z5,r6), indicating their position is away from the origin and their orientation differs from the reactee's.

• Q3. Meaning of (x, z)

Yes, the (x, z) values indeed correspond to the 2D coordinates of the pelvis. These values represent the position of the pelvis in the horizontal plane, which serves as the reference point for defining spatial positions and relationships. We have explicitly stated this in the revised version (Line 216).

• Q4. Inconsistent terms

Thank you for pointing out the inconsistency. We have carefully reviewed and revised the terminology in Section 3.1 to ensure consistency throughout the text (Line 227). We have also clarified and aligned other inconsistencies in our revision and highlighted them.

• Q5. Why is the Inter-X evaluation model not adopted

The main reason is the difference between motion representations. To ensure computational efficiency, previous works [1,2] mainly adopt a standardized motion representation comprising skeleton positions, velocities, and orientations, amounting to approximately 262 dimensions. We follow this widely used motion representation. In contrast, Inter-X’s evaluation model utilizes a denser motion representation, SMPL-X, a 3D human body model that includes facial expressions and is based on skinning and blend shapes, with more than thousands of dimensions. Due to these differences in motion representations, we cannot directly adopt the Inter-X classification model.

Previous works [1,2] based on the same representations lack interaction labels, which is necessary for calculating accuracy. That's why we re-train an evaluation model.

Regarding the lower accuracy, there are two main reasons: 1) Inter-X employs a GNN-based evaluation model specifically optimized for classification tasks. In contrast, our evaluation model should balance between classification accuracy and ranking metrics (e.g., R-Precision). To achieve this, we adopt a CLIP-like training strategy, similar to [2]. 2) Our focus is on evaluating the quality of generated reactions. To this end, we mask a large proportion of the input action sequence to enhance the model’s discriminative power. While this improves reaction quality assessment, it comes at the expense of input detail and classification accuracy.

• On Q1: About our unified tokenizer

Without a unified motion representation enabled by our decoupled tokenizer, the performance of TTR deteriorates significantly due to much lower-quality training data, which is critical for LLMs. When using the plain tokenizer, we observe a more unstable training process. Worse still, TTR with the plain tokenizer often fails to predict a meaningful sentence during the Thinking stage, disabling our Think-Then-React framework. This results in the poorest performance among our ablation studies (comparable only to MotionGPT).

Regarding the details of the plain VQ-VAE, your understanding is correct. The plain tokenizer shares the same architecture and hyperparameters as the decoupled tokenizer.

We would like to clarify the details of the VQ-VAE with an example on our data processing pipeline, and we have also added Figure 9 at Line 774 of Appendix to illustrate this process in detail: First, we normalize the reaction location and orientation (to the original point and facing z-positive axis), e.g., from (x=1, z=2, r=45°) to (x=0, z=0, r=0). Simultaneously, the action’s location and orientation are adjusted to maintain their relative position and orientation, e.g., from (x=3, z=4, r=225°) to (x=2, z=2, r=180°). Then, for reaction, we directly encode it with VQ-VAE into pose tokens, which is already normalized, e.g., <p2><p5><p6>.

For action:

• With our decoupled tokenizer: We normalize it to (x=0, z=0, r=0) and then encode it with VQ-VAE to obtain pose tokens, e.g., <p1><p2><p8>. The final action sequence could be represented as <x2><z2><r180><p1><p2><p8>.

• With the plain tokenizer: We retain the unnormalized space features (x=2, z=2, r=180°), maintaining action-reaction relative position and orientation, and then encode it with VQ-VAE to obtain pose tokens, e.g., <p10><p21><p33>. The final action sequence could be represented as <p10><p21><p33> without space tokens, as the spatial features are implicitly embedded within these tokens.

Although the unnormalized action features theoretically combine spatial and motion information, encoding such features directly with VQ-VAE often leads to inefficient use of the codebook with limited size. For instance, the same motion sequence at different positions or orientations would be encoded as different token sequences, whereas our decoupled tokenizer only modifies the prefix of the sequence.

We also evaluated the perplexity (PPL) of the two codebooks (both codebook size is 256), and found that our decoupled tokenizer converges at 139.9, enabling higher precision in diverse motion representation, while the plain tokenizer achieves only 94.2, indicating underutilization or even collapse of the codebook.

• On Q5: Balancing semantic-reaction and action-reaction correspondence

Thank you for your insightful question. In brief, our findings suggest that the 1 FPS setting benefits both types of correspondence, although there may be more efficient solutions to explore.

The results in Figure 5 demonstrate the effectiveness of the 1 FPS setting in improving semantic-reaction correspondence. To further investigate its impact on action-reaction correspondence, we conducted an additional experiment: we introduced random spatial or temporal shifts to the ground-truth reaction, disrupting the action-reaction correspondence, and evaluated the sensitivity of our metrics (FID and ranking scores) to these shifts. We compared our 1 FPS setting against the default setting, which uses the entire action-reaction sequence as input. The results, expressed as "ours/default," are presented in the table below:

These findings indicate that, even without explicitly optimizing for action-reaction correspondence during training, our 1 FPS setting demonstrates greater discriminative power and is more sensitive to shifts that degrade action-reaction correspondence. We mainly attribute this to: 1) Action dominates overall semantics of interaction. As a result, feeding the entire action as input might lead to shortcut learning by the evaluation model, ignoring both semantic-reaction and action-reaction correspondence. 2) Even with a high masking ratio applied to the input action, the remaining sequences retain enough semantic and spatial-temporal details to guide accurate evaluation. Like videos, motion sequences are relatively low in information density compared to other modalities like texts.

We sincerely appreciate your recognition to our work! We will include detailed analysis in the appendix of the final version. Your support and increased rating mean a great deal to us!

We are grateful for your thoughtful suggestions and recognition of our contributions. Please find our detailed responses to your concerns below.

• Q1. Mixture of two datasets

Baseline methods do not use mixture datasets, as they typically cannot leverage single-person datasets like HumanML3D without a unified representation, which encodes both single- and multi-person scenarios. The reason we use mixture dataset is because we aim to enlarge pre-training data scale for better motion understanding and generation capabilities. For fairness, we also evaluate our method without using single-person data pre-training. The performance differences are minimal (FID changes from 1.942 to 2.007, as shown at w/o SP Data in Table 1 and Section 4.3). To better understand the impact, we analyze the overlap between datasets in Section 4.4 and present the findings in Figure 3. The analysis shows that the overlap between single- and multi-person data is minimal, meaning the inclusion of single-person data provides only a slight performance boost.

• Q2. Human evaluation

Thank you for this valuable suggestion! We conducted a user study using 100 samples from the Inter-X test set to assess user preferences between our TTR model and the SOTA ReGenNet. The results are summarized in Appendix Figure 7 of our revision (Line 722). Our findings show that TTR significantly outperforms ReGenNet in terms of user preferences, particularly when the action duration is longer. This further demonstrates the effectiveness of our proposed thinking process, which helps alleviate accumulated errors. We consider including this section in our final revision.

Additionally, we have updated our anonymous webpage: Think-Then-React.github.io, where we have included some video cases on user study. If you have further concerns, please don't hesitate to let us know. Your suggestions are valuable to us!

We’d also like to share some positive updates: after thorough discussions, the other three reviewers have provided positive feedback on our responses and revisions, with two of them deciding to increase their ratings. We hope this information may be useful as you continue to evaluate our work.

Thank you again for your time and guidance! We greatly appreciate your help in improving our submission.

Thank you for your insightful comments on our work. We have addressed your questions in detail below.

• Q1. Distinctions among key terminologies

To improve clarity, we more formally define the terminologies as follows: 1) Pose refers to a human body posture or movement within a brief time frame (i.e., timestep), such as "taking one step forward." A pose can be represented as a single token. 2) Motion is a sequence of poses, starting with an initial spatial state represented by space features. For example, a motion might be "a person walks three steps." 3) Action denotes the input motion initiated by a person, while reaction refers to the output motion in response by another person. We have added a paragraph in Section 3.1 to explicitly define these terms (Line 206-209).

We have also refined notations and phrasing throughout the paper particular on Section 3 to enhance clarity and readability.

• Q2. Training process

Nice suggestion! The training process is indeed complex as you pointed out. As illustrated in Figure 8 (Appendix Line 722), different tasks converge at different speeds and loss scales: for example, text and space generation tasks tend to converge in much faster speed and smaller loss than motion generation tasks. Thus, to balance different training tasks, we take validation losses after each training epoch as training task sampling weights, to dynamically switch the training source and tasks. For instance, the loss scale of Pose-to-Space (P2S) task is much smaller than Space-to-Pose (S2P), and the P2S task has a much faster converging speed, thus we apply a higher sampling weight on S2P task and a lower sampling rate on P2S. Relevant discussion is highlighted in Section 3.3.1 in our revision (Line 229).

If we have misunderstood your question or the specific meaning of "switching the training sources," please let us know so we can clarify further.

• Q3. Visualization of results

Good remark! We have provided qualitative comparisons and failure cases on an anonymous webpage: Think-Then-React.github.io. We have included some of them in our revised paper (Line 432-449).

• Q4. The necessity of the thinking process

The necessity of the thinking processes is validated through both experiments and studies for human cognitive process. As shown in Table 1, both the baseline method ReGenNet and our ablation method TTR w/o Thinking perform poorly due to instability and accumulated errors when directly generating reactions from actions.

In the real world, humans typically observe others’ actions, infer their intentions, and then react accordingly. Inspired by this, we designed our method to incorporate a similar reasoning process. Specifically, leveraging the well-annotated Inter-X dataset, which provides fine-grained descriptions for both actors and reactors, we train our model to progress from action captioning to reasoning. This enables the model to utilize the inference capabilities of LLMs in a Chain-of-Thought-like manner for reaction generation. What's more, the reacting process takes both input action and the predicted prompts from the thinking process, which introduces more semantics while preserving spatial-temporal details of the input action, mitigating potential information loss.