Sitcom-Crafter: A Plot-Driven Human Motion Generation System in 3D Scenes

We sincerely appreciate the reviewer’s thoughtful feedback and the effort invested in reviewing our paper. Below, we address the reviewer’s comment in detail.

---

Reviewer’s comment: Random generation of the object is insufficient to learn richer context of human behavior. The real-world human interactions with the scene and other people are complex and entangled. The proposed method achieves obstacle avoidance but still limits the applicability to the real-world plots.

We agree with the reviewer’s assessment. The proposed scene-aware human-human interaction generation module primarily focuses on ensuring the physical alignment of human-human interactions. Previous methods often generate physically unconstrained motions, resulting in frequent collisions or penetrations with 3D scenes (as demonstrated in our experiments). Our approach addresses this by introducing the random synthesis of surrounding scene objects, which significantly mitigates such issues.

However, as the reviewer correctly points out, the random generation of objects supplements 3D environmental information without increasing the diversity or complexity of the interaction motions themselves. Consequently, while our scene-aware module effectively improves obstacle collision avoidance, it contributes only marginally to learning richer human behavioral contexts.

To generate diverse and realistic human interaction motions for general real-world plots, we believe that exhaustive data collection remains a critical and effective approach—though it is both time- and resource-intensive. Such data collection could involve manual motion capture or automated extraction of motion data from images and videos. In the long term, we envision the emergence of scaling laws in the human interaction generation domain, akin to those observed in models like Stable Diffusion and ChatGPT, where large-scale datasets significantly enhance the richness and applicability of generated results.

Once again, we thank the reviewer for this insightful comment. We have supplemented and discussed this limitation in Appendix I of our revised manuscript, with all modifications clearly highlighted.

Reviewer’s comment: Not easy to reproduce. This framework is hard to scale in practice. I would prefer to regarding this work as a toy model to demonstrate the possibility.

We agree that our system is more complex than plug-and-play designs, requiring additional effort for setup and comprehension. To aid reproduction, we have disclosed as many implementation details as possible in both the main paper and the appendix (as noted by the reviewer). In addition, the supplementary materials include:

• Comprehensive documentation of each module's design (in Appendix).
• A detailed README file accompanying the supplementary code.
• The complete source code for the system.

These resources are intended to facilitate reproduction and subsequent research based on our work. We are also committed to further modularizing and optimizing the framework to enhance its usability.

Regarding the comment on being “hard to scale in practice”, we interpret this as a recognition that our work has room to improve in meeting real-world demands, such as those required for commercial applications (please let us know if we have misunderstood). We acknowledge that there is still a gap between the current system's output and the high standards of industry. Addressing this gap will be a key focus of our future efforts, which we plan to pursue in collaboration with the broader research community.

As the reviewer noted, this work can be viewed as an initial exploration and experimental validation of the feasibility of a comprehensive human motion generation system. By presenting our current progress, we aim to help the research community evaluate the state of the field, identify achievable milestones, and guide future research toward designs that contribute meaningfully to the ultimate goal of a fully unified system.

We sincerely thank the reviewer for recognizing the contributions of our work. Below, we address each of the reviewer’s comments in detail.

---

Reviewer’s comment: Though as a pioneering effort, this work is more like a systematical engineering effort, instead of a scientific paper. A little worried about if it fits well in this conference.

We acknowledge that our work involves significant engineering efforts. However, we would like to emphasize that it is research-driven and motivated by practical needs. As outlined in the introductory paragraph of the paper, professionals in animation and gaming industries require systems capable of generating diverse motion types guided by long plots. Current research typically focuses on single-motion-type generation, leaving the development of a unified system largely unexplored. Our research seeks to address this gap by investigating the extent of progress achievable with current human motion generation techniques, which is both beneficial and significant for advancing the field—a point also recognized by the reviewer. While research-motivated, the nature of a system necessitates engineering efforts to achieve the practical goal, which may contribute to its engineering-oriented appearance.

Regarding its relevance to ICLR, we are confident in the paper's fit. First, ICLR welcomes participants from diverse backgrounds, including academic researchers, industrial researchers, and engineers. Our work resonates with this audience by contributing engineering solutions that benefit industry while also inspiring academic researchers to explore future directions toward unified systems. Second, the ICLR call for papers explicitly includes applications in fields such as gaming and robotics. Our work directly supports practical workflows in animation and gaming, while human motion generation is also closely related to humanoid robotics. Thus, we believe our work aligns well with ICLR’s scope.

---

Reviewer’s comment: Unclear how motions are generated. Recommend to illustrate more clearly how the interactions are synthesized.

Thanks for this constructive suggestion. In our revised manuscript, we have added a new subsection, B.1 Detailed Workflow Within Generation Modules, in the Appendix (highlighted for visibility) to clarify how motions are synthesized within the generation modules. This section includes a detailed example to illustrate the logic and a diagram to visually present the workflow. We hope this addition makes the process clearer.

In brief, the motion generation workflow operates based on the interpreted command lists. Each command is sequentially assigned to the appropriate generation module. The generation of the current motion state is conditioned on the frames produced in the previous step, ensuring smooth chaining of motion segments. This approach enables the system to generate long-duration motions aligned with the guidance provided by extended plot text.

---

Reviewer’s comment: The overall performance cannot be guaranteed. The overall performance is relied on the functionality of each module.

Each module indeed plays a role in the system's final performance. However, each module addresses a distinct aspect of the system’s functionality. For instance, the human-human interaction module primarily influences the quality of generated interactions, while the motion synchronization module ensures smooth transitions between different motion types. As a result, the contributions of individual modules are largely decoupled, allowing their improvements to directly enhance specific aspects of the overall system.

From this perspective, the performance of our system is not only guaranteed but also benefits from its modular design. This modular architecture provides both flexibility and extensibility, enabling the seamless integration of novel methods or refinements into individual modules while ensuring the system remains compatible and fully functional.

Furthermore, the modular framework lays a solid foundation for advancing research in comprehensive motion generation. By offering a flexible and integrative platform, we aim to enable other researchers to easily test and incorporate their approaches, thereby accelerating progress in tackling the challenges of this complex and ambitious domain.

We appreciate the effort of the reviewer in reviewing our paper. After reading the comments, we feel that there may be some misunderstandings with regard to the contributions of our work. Before addressing the reviewer’s concerns individually, we would therefore like to recap the key points of our paper to provide proper context:

---

• What is the goal of our work?

We aim to develop a comprehensive human motion generation system capable of synthesizing diverse types of human motions in 3D scenes, guided by long plot sequences — a challenge that has not yet been thoroughly studied.

(Relevant sections in our paper: This objective is clearly highlighted in the title, the second sentence of the Abstract, and the first contribution listed in the final paragraph of the Introduction.)

• Why is this work important?

The motivation stems from three main reasons:

1. Practical and industrial relevance. Such a system addresses a pressing need in real-world applications, particularly for professionals such as animators and game designers. These practitioners often work with long plot-driven sequences involving varied types of human motion. A comprehensive system that supports diverse motion types holds immense practical value.

2. Academic and research significance. Existing works on human motion generation typically focus on a single type of motion, such as locomotion or human-scene interaction. Researching a unified, comprehensive system fills an important gap in the field, advancing knowledge and providing valuable insights to the research community. Moreover, the open-source release of our proposed system lowers the barrier for researchers to engage with this challenging topic, fostering further exploration and innovation in unified motion generation.

3. Technical challenge. Building such a system is challenging. It is far from the intuitive image of simply stacking building blocks to make it work. It requires addressing complex challenges like physics-based collisions, motion misalignment, generation conditions, and representation inconsistencies. These complexities make this topic both significant and deeply challenging, warranting thorough investigation.

(Relevant sections in our paper: These motivations are briefly outlined at the start of the Abstract, thoroughly discussed in the opening paragraph of the Introduction, and further emphasized in the final subsection of the Related Works.)

• How do we achieve our goal?

To build a stable, comprehensive system, we need not only to develop an innovative scene-aware human-human interaction module but also to design seven additional distinct modules to address the aforementioned challenges. Each module addresses a specific issue encountered during system development and is crucial for ensuring system integrity and performance. Key features include: A novel scene-aware human-human interaction module for better physics alignment; Practical designs for motion synchronization and collision resolution; User-friendly plot comprehension and motion distribution mechanisms; Visual improvements via hand pose retrieval and motion retargeting. The ultimate goal of the system can only be realized by integrating these carefully designed components.

(Relevant sections in our paper: The motivations and mechanisms behind our designs are briefly described in paragraphs 2–5 of the Introduction, with further details in Section 3 and Appendices B and C.)

• How significant and beneficial is our work to the field?

Based on the answers to the previous questions, our research on a comprehensive human motion generation framework is: 1) Practical and industrially relevant – addressing real-world needs; 2) Academically significant – advancing the state of research in human motion generation; 3) Technically challenging – tackling complex issues inherent to system construction. Our contributions to the community include: 1) The development of a comprehensive motion generation system; 2) A novel scene-aware human-human interaction module; 3) The identification of key challenges in building such systems and the provision of effective solutions throughout the process.

(Relevant sections in our paper: These contributions are summarized in the final paragraph of the Introduction.)

Now, we respond to the reviewer’s concerns one by one.

---

Reviewer’s comment: Lack of novelty in the core methodology / The core method is not very clear / The contribution of the proposed method to the field seems minimal / Unclear research question / This paper studies an overly broad task and lack of sufficient novelty in the main approach / Need a more defined research question and a method to address it effectively.

We believe the answers we provided to the four high-level questions earlier help clarify the reviewer’s misinterpretations of our work.

To succinctly address concerns about the novelty and significance of our work: A unified human motion generation system is both highly significant and impactful for the research and industrial fields. To the best of our knowledge, no prior work has tackled this problem comprehensively. By exploring the challenges and proposing solutions, we believe our work qualifies as both “novel” and “significant”.

Additionally, while simple and focused approaches have undeniable benefits, we believe that addressing only single motion types, as seen in existing works, has significant limitations. These approaches are insufficient for progressing toward the more complex and realistic scenarios that our work aims to tackle. Developing a unified system capable of handling such scenarios naturally requires a more intricate solution. We believe that focusing on this highly complex and uncharted problem is essential to drive meaningful progress in the field. The complexity of our solution should not detract from its perceived innovation but rather underscore its importance in advancing toward a comprehensive understanding of human motion generation.

Reviewer’s comment: The system is too complex.

We acknowledge that our system may appear complex due to the numerous components and designs it includes. However, as we have outlined in the responses to Why we do it and How we do it, this level of complexity is essential. Each design addresses a specific challenge, and together, they form a complete solution to the many difficulties involved in building a comprehensive human motion generation system.

Reviewer’s comment: Limited improvement over existing methods in each module.

First, as discussed earlier in response to What is the goal of our work, the primary goal of our work is to explore and establish a comprehensive system. Our focus is on designing innovative modules, e.g., the scene-aware human-human interaction module, to address the current technological gaps while ensuring the system as a whole functions cohesively and effectively, rather than merely incrementally improving existing modules. This goal has been successfully demonstrated in our work.

Second, our system does incorporate several novel designs that improve upon existing methods. For example, in the human-human interaction module, we propose a new and practical method to achieve better physics alignment. This approach is detailed in Section 3.2 and validated through experiments and visual results presented in Section 4. Moreover, since a comprehensive motion generation system of this kind has not been explored in previous work, the designs in our five augmentation modules, which link the generation modules to create an integrated system, are entirely novel compared to prior methods that focus solely on single motion types.

Reviewer’s comment: Generated motions often appear unnatural and lack clear interaction.

We acknowledge that the generated motions currently exhibit limitations and fall short of daily-use quality, such as commercial-grade outputs. Bridging this gap remains a long-term goal, not only for our team but also for the broader research community. This limitation has been acknowledged in Appendix I, and we see it as a crucial direction for ongoing improvement and future research.

Nevertheless, our experimental results demonstrate that our approach has already outperformed current state-of-the-art methods, highlighting the progress we have made in advancing human motion generation.

---

We would like to once again express our gratitude for taking the time to review our work. We hope our responses have addressed your concerns and will assist you in reevaluating the contributions of our work to the community. Should you have any further questions or concerns, we would be more than happy to engage in further discussion.

We sincerely appreciate the reviewer’s continued efforts in reviewing our response. Below, we address the reviewer’s follow-up concerns:

---

Reviewer’s comment: Question on whether it fits this conference's scope

We confidently assert that our work aligns well with ICLR’s scope.

Referencing the ICLR About Us section on the official homepage and the Subject Areas described in the Call for Papers, ICLR explicitly welcomes research on application areas such as gaming and robotics and also calls for submissions addressing implementation issues related to software platforms. Additionally, the conference targets a diverse audience that includes both academic and industrial researchers, as well as engineers.

Our work, which has been praised by the other reviewers (Reviewer 58jH and Reviewer CFvD) for its significant contributions and high relevance to the research community, fits squarely within this scope. It aligns with ICLR’s focus on gaming, animation, and humanoid robotics applications. Furthermore, our work addresses key topics relevant to ICLR’s audience:

• Research Community: By providing insights into the current state of human motion generation, highlighting challenges, and inspiring future research directions to advance towards unified systems.

• Industry Professionals: By detailing implementation issues and proposing solutions applicable to workflows and product designs.

These aspects strongly support the relevance of our work to ICLR’s scope.

To further substantiate this point, we reviewed papers published at ICLR in previous years. Many works that involve constructing comprehensive systems composed of multiple components or modules fall within ICLR’s scope. Examples from ICLR 2024 include:

• MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework (Oral)
• BooookScore: A Systematic Exploration of Book-Length Summarization in the Era of LLMs (Oral)
• DIFFTACTILE: A Physics-based Differentiable Tactile Simulator for Contact-rich Robotic Manipulation (Poster)
• Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks (Poster)
• COLLIE: Systematic Construction of Constrained Text Generation Tasks (Poster)

These examples further validate that our work, which similarly integrates multiple modules to address a complex problem, is well within the conference’s scope.

Thank you for the response.

For enhanced conditioning, incorporating the scene information as condition is widely used. The difference is this paper proposes to generate pseudo ground truth to give this condition which I agree with. For conditioning the last frame, we can observe a similar idea in "Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenes in CVPR 2021" and diffusion-based Motion In-betweening works, such as "Flexible Motion In-betweening with Diffusion Models" in SIGGRAPH 2024.

For new motion representation, there is a line of works that use marker-based representation, for example, "We are More than Our Joints: Predicting how 3D Bodies Move in CVPR 2021". and for body regressor for parametric reconstruction, it is really common in both human pose estimation and motion synthesis studies. For example "Convolutional Mesh Regression for Single-Image Human Shape Reconstruction in CVPR 2019", besides the "We Are More Than Our Joints".

For canonicalization, there is a quite relevant work "Generating Continual Human Motion in Diverse 3D Scenes in 3DV 2024" using goal-centric canonical coordinate frame for human motion in 3D scene.

Using a post-process to deal with the collision problem is also very common, e.g. " Populating 3D scenes by learning human-scene interaction in CVPR 2021". Replacing the human-scene to human-human is not a big contribution.

"Our five augmentation modules are entirely novel", to be honest, I don't find novel ideas in "Motion Synchronization", "Hand Pose Retrieval", "Motion Collision Revision" or "Motion Retargeting". Aren't they either existing solutions or very straightforward?

I understand authors have made modifications to all the things mentioned above. However, given these observations, I recommend a thorough review of related works to better position and differentiate this paper within the existing literature.

While I recognize the amount of effort to tackle this complex task, I can not identify aspects of the system that stand out as particularly novel or compelling enough to rate acceptance. I keep my rate.