Puppeteer: Rig and Animate Your 3D Models

We are grateful for the reviewer’s insightful comments. Please find below our detailed responses to your comments and concerns.

W1: The animation stage is optimization-based and not learnable, limiting the inference speed.

While per-scene optimization does limit inference speed (as discussed in our limitations), it offers two key advantages: (1) Unlike large-scale feed-forward methods such as L4GM [1], which require training on 128 80G A100 GPUs, our approach needs only a single A100 GPU while avoiding geometric distortions and achieving better generalization to unseen cases since we do not rely on training data distribution; (2) Compared to other per-scene optimization methods like AKD [2], which requires 25 hours per object, our method achieves significantly faster optimization (20 minutes for objects with up to 10k vertices) while producing stable, jittering-free animations.

[1] Ren et al., L4GM: Large 4D Gaussian Reconstruction Model, 2024.

[2] Li et al., Articulated Kinematics Distillation from Video Diffusion Models, 2025.

W2 & Q1: The method relies on video generation models to provide motion guidance. However, if the video is unrealistic or noisy, animation fidelity may degrade. And these models may struggle to generate complex or large-scale motions (e.g., dancing). Can the authors comment on how well their method handles such cases, and whether this limits the animation quality?

Thanks for raising this critical concern about video quality dependency. Current text-to-video models (e.g., Kling AI, RunwayML, Jimeng AI) have improved significantly and can generate complex motions with reasonable success rates, as demonstrated on the project page in supplemental material. However, we acknowledge that video quality does impact animation fidelity. For instance, motion blur or temporal inconsistencies in generated videos can degrade joint/vertex tracking accuracy and make optimization more challenging. To mitigate these issues, since we use off-the-shelf generation models, we can generate multiple video candidates and select the highest-quality one based on visual clarity and motion consistency to guide the animation optimization. While this approach helps reduce the impact of poor-quality videos, video generation quality still represents a limitation for highly complex motion scenarios. We will include comprehensive discussions of these robustness issues and mitigation strategies in the final version.

W3: Recent works like UniRig and MagicPose4D seem relevant and may provide useful points of comparison or inspiration.

Thanks for highlighting these relevant works. UniRig is a concurrent work on automatic rigging. We provide a comparison with UniRig on skeleton generation in the table below (Arti2.0 = Articulation-XL2.0, MR = ModelsResource, Pose = Diverse-pose). Our method demonstrates superior performance across all three benchmarks. Note that we do not compare skinning approaches since UniRig uses bone-vertex skinning while our method employs joint-vertex skinning. MagicPose4D addresses skeleton-driven motion transfer, targeting a different problem scope from ours. We discuss it in the related work section of the appendix.

Q2: Animation quality seems to depend on several factors, such as rigging accuracy, the viewpoint of the first frame, and whether important parts (e.g., limbs) are always visible in the video. Are there common failure cases when one of these steps goes wrong? Could the authors show or discuss a few examples where the method fails?

Thanks for this important question about robustness. Several key factors can affect animation quality: (1) Rigging: When skeleton generation produces insufficient joint density, fine-scale deformations suffer. For example, in our turtle case ("More Animation Results" section of supplementary project page), the reference video shows smooth forelimb motion, but our animation appears less smooth due to sparse joint placement in these regions. This limitation stems from our skeleton generation producing few joints in areas requiring fine-scale deformations. Animation-driven adaptive joint refinement could potentially address this issue in future work. (2) Viewpoint dependency and occlusion: Suboptimal camera angles can cause depth ambiguities and tracking failures. Since we have access to the input 3D mesh, we can select optimal viewpoints that maximize joint visibility and minimize occlusion. However, single-view optimization inherently suffers from occlusion issues when critical joints or limbs are hidden throughout the video sequence. Training a feed-forward model with multi-view priors could potentially alleviate these limitations by providing better geometric understanding of occluded regions. We will include comprehensive discussions with specific examples and potential mitigation strategies in the final version.

After the rebuttal, most of my concerns are resolved, thus I will keep my positive score.

We appreciate the valuable comments from the reviewer. Please find below our detailed responses to your comments and concerns.

W1 & Q1: Missing citation to MagicArticulate in introduction, unclear data scarcity claim.

Thanks for highlighting these important clarification issues. We will add the proper citation to MagicArticulate in the introduction when discussing the Articulation-XL dataset. Regarding the data scarcity claim: while MagicArticulate introduced a substantial 33K dataset, it primarily consists of objects in rest poses, limiting generalization to diverse pose inputs. Our Articulation-XL2.0 addresses this limitation by adding a diverse-pose subset with 11.4K samples containing varied articulation states. We will clarify this distinction in the final version by specifically mentioning both data scarcity and pose diversity as key challenges addressed by our work.

W2 & Q2: Joint vs. bone inconsistency: You argue for joint-based tokenization (compactness) but then state bone-based coordinates perform better.

These approaches are used in different modules with distinct objectives: (1) We use joint-based tokenization in skeleton generation because it represents skeletons with shorter token sequences, enabling faster training and inference while achieving comparable or better performance than bone-based tokenization for the autoregressive generation task; (2) We use bone-based coordinates with positional encoding in skinning weight prediction because bone coordinates better capture the relationships between adjacent joints and provide more informative spatial features for the attention mechanism, which is helpful for accurate skinning weight prediction.

W3 & Q4: Animation pipeline relies on off-the-shelf text-to-video models for motion guidance. How robust is your optimization to imperfect video quality? Failure case analysis for video quality issues is missing.

Thanks for raising this critical concern about video quality dependency. Current text-to-video models (e.g., Kling AI, RunwayML, Jimeng AI) have improved significantly and can generate complex motions with reasonable success rates, as demonstrated on our project page in supplemental material. However, we acknowledge that video quality does impact animation fidelity. For instance, motion blur or temporal inconsistencies in generated videos can degrade joint/vertex tracking accuracy and make optimization more challenging. To mitigate these issues, since we use off-the-shelf generation models, we can generate multiple video candidates and select the highest-quality one based on visual clarity and motion consistency to guide the animation optimization. While this approach helps reduce the impact of poor-quality videos, video generation quality still represents a limitation for highly complex motion scenarios. We will include comprehensive discussions of these robustness issues, failure case analysis, and mitigation strategies in the final version.

W4 & Q3: Comparison with MagicArticulate: Clarify what components are reused or redesigned, and consider providing a comparison table showing the key differences and improvements.

Thanks for your suggestion. Please find the comparison table between MagicArticulate and our method below. For the dataset, we built upon Articulation-XL, expanding it from 33K to 59.4K samples with a diverse-pose subset that enhances generalization to varied pose inputs. For skeleton generation, we use the same autoregressive transformer with point cloud conditioning but propose more efficient joint-based tokenization with hierarchical ordering and randomization to improve model performance. For skinning weight prediction, we propose a completely different attention-based architecture that achieves better generalization and faster inference than MagicArticulate's functional diffusion approach. Most importantly, MagicArticulate does not support automatic animation, while our method provides video-guided animation capability.

W5: Animation evaluation lacks perceptual animation metrics or user studies.

Thanks for this valuable suggestion. We conducted user studies with 21 participants to evaluate animation quality across three methods: L4GM, MotionDreamer, and our approach. Participants compared 8 animation examples across three evaluation criteria: (1) Video-Animation Alignment: Which animation result shows better alignment with the input video? (2) Motion Quality: Which one has a more natural and realistic motion? (3) 3D Geometry Preservation: Which method better maintains the original 3D object geometry without introducing distortions or artifacts? Results are shown in the table below. Our method outperforms both L4GM and MotionDreamer across all three evaluation dimensions. Note that Video-Animation Alignment is not evaluated for MotionDreamer since it uses text-driven motion generation rather than video guidance. Regarding perceptual metrics, standard measures (LPIPS, CLIP similarity, FVD, etc.) require ground truth animations from multiple viewpoints for comparison, which are unavailable in our setting. Developing appropriate perceptual metrics and benchmarks for video-guided 3D animation remains an open challenge that we would like to address in future work.

W6 & Q5: No runtime or compute cost is reported for the animation phase. How long does the optimization take for typical sequences, and how does it scale with mesh complexity or video length? Is this approach practical for interactive applications?

Thanks for this important question. We report runtime and computational costs in Appendix Section A.1 (lines 59-62). Our animation optimization takes approximately 20 minutes for objects with up to 10K vertices on a single NVIDIA A100 GPU, processing 5-second videos (approximately 50 frames at 10 FPS) generated by Kling AI or Jimeng AI. Runtime scales with both mesh complexity and frame count: (1) Mesh complexity: Models with more vertices will require additional Pytorch3D rendering time. For example, the bat case ($\sim$ 70K vertices) in the supplementary project page requires 90 minutes, while the turtle case ($\sim$ 15K vertices) takes 35 minutes. (2) Frame count: For a typical case taking 20 minutes at 50 frames (10 FPS, 5 seconds), increasing to 20 FPS (100 frames) extends optimization time to 41 minutes, while reducing to 4 FPS (20 frames) decreases it to 8 minutes, demonstrating approximately linear scaling with frame count. We will include these discussions in the final version.

As acknowledged in our limitations (Appendix Section E), this per-object optimization approach is not suitable for real-time applications. Future work will focus on developing an end-to-end learning framework to eliminate per-scene optimization and enable real-time animation generation.

Dear Reviewer h6uP,

The authors have provided detailed responses to your questions. What is your view after seeing this additional information? It would be good if you could actively engage in discussions with the authors during the discussion phase ASAP, which ends on EoA (Aug 6).

Best, AC

Thank you for your time and careful review of our work. Below we address your questions and weaknesses mentioned:

W & Q2 & Q3: More explanation about Articulation-XL2.0 dataset information and curation: 1) What specific data types were added and how did their inclusion impact model performance? 2) What kind of new poses are used in the "diverse-pose subset" is not mentioned or visualized.

(1) Articulation-XL2.0 expands the original Articulation-XL dataset with: (a) multi-geometric data from Objaverse-XL, where each mesh consists of multiple parts, increasing dataset scale (from 33K to 48K) and improving performance across all three benchmarks; (b) a diverse-pose subset that enhances model generalization to diverse pose inputs. To demonstrate the impact of these dataset enhancements, we provide an ablation comparison across three training configurations: original Articulation-XL, Articulation-XL2.0 without diverse poses, and full Articulation-XL2.0 with diverse poses (marked with *). Results are shown in the table below (Arti2.0 = Articulation-XL2.0, MR = ModelsResource, Pose = Diverse-pose). (2) The diverse-pose subset contains two components (detailed in Section 3.1 and Appendix Section B): (a) data with varied poses extracted from objects with animation in Objaverse-XL, where we specifically selected frames exhibiting maximum deviation from rest pose configurations to capture extreme articulations; (b) animal data with diverse poses generated using SMALR. Examples are visualized in Figure S2 in the Appendix.

For dataset curation, we follow the original MagicArticulate pipeline with VLM-based filtering and further enhance quality by eliminating meshes with unskinned vertices and conducting manual validation. For the diverse-pose subset, we apply dual quality filtering using both high-quality rigging data (from our curated list) and high-quality animation data (from Diffusion4D's list), then extract individual frames from these animations to construct the diverse-pose subset with varied articulation states. More details are provided in Section 3.1 and Section B in the Appendix.

Q1: Where does ground truth token sequence T come from? All ground truth data are preprocessed from Articulation-XL2.0?

The ground truth token sequence T in training consists of three components: (1) Shape tokens: generated by encoding sampled point clouds with normals from dataset meshes using the pre-trained shape encoder; (2) Skeleton tokens: obtained by applying joint-based tokenization to skeletons from the dataset; and (3) Positional indicators: learnable parameters that specify the next joint group in the auto-regressive sequence. All components except the positional indicators are directly preprocessed from the dataset Articulation-XL2.0.

Q4: Joint-based tokenization discretizes joint coordinates into a grid. Is there spatial information loss due to this process?

Our joint-based tokenization normalizes coordinates to $[-0.5, 0.5]$ and discretizes them into a $128^3$ grid, yielding a quantization resolution of $\sim0.008$ units per dimension. While this introduces small quantization error, it balances tokenization efficiency with spatial precision for skeleton generation. Our experiments show this quantization level preserves sufficient detail for anatomically plausible skeletons. We also experimented that higher resolutions ($256^3$) provide marginal improvements while substantially increasing computational cost.

Q5: How is the graph distance in Topology-aware Joint Attention calculated for very complex skeletons?

The graph distance calculation handles complex skeletons efficiently using Breadth-First Search (BFS) from each joint. Our implementation constructs an adjacency list from the bone connections, then performs BFS from each of the $J$ joints to compute shortest path distances to all other joints. This produces a $J\times J$ distance matrix. The computational complexity is $O(J^2)$ since we run BFS from each of the $J$ joints. We precompute this distance matrix once per skeleton and use it in Topology-aware Joint Attention.

Q6: Can current video models handle complex motions well, or maybe the video is the weak part for final animation quality?

Thanks for raising this important point. Current video generation models (e.g., Kling AI, RunwayML, Jimeng AI) have improved significantly and can generate complex motions with high success rates. Some examples are provided on the project page in supplemental material. However, video quality does impact our final animation quality. For instance, motion blur or temporal inconsistencies in generated videos can degrade joint/vertex tracking accuracy and make optimization more challenging. We will include this discussion in the final version.

Q7: The paper said that RigNet struggles with diverse object categories and fails to converge well. Is RigNet's design bad? Or just the dataset too hard for RigNet?

RigNet's performance issues stem from both factors: (1) Architectural limitations: RigNet's GNN-based design struggles to scale on large, diverse datasets like Articulation-XL2.0, which contains significantly more object categories than its original training data; (2) Dataset complexity: Our dataset's varied object orientations and geometric diversity exceed RigNet's original assumptions. This combination explains RigNet's convergence difficulties in our experiments.

Dear Reviewer WsQJ,

The authors have provided detailed responses to your questions. What is your view after seeing this additional information? It would be good if you could actively engage in discussions with the authors during the discussion phase ASAP, which ends on EoA (Aug 6).

Best, AC

Thanks for your constructive feedback. We have organized your comments from the weaknesses, questions, and limitations sections and provided our responses below.

Q1: Vague descriptions in Articulation-XL2.0 dataset section, such as "randomized valid poses".

Thanks for pointing out this vagueness. "Randomized valid poses" refers to poses generated from SMALR where we apply random rotation angles to animal joints while constraining the angles within anatomically valid ranges. This ensures diverse articulation states while maintaining biological plausibility. We will clarify this with specific angle ranges in the final version.

Q2: Core methodology appears largely composed of existing techniques; the deeper insights primarily emerge through ablation studies and limited visualizations.

While some components leverage established foundations, our work introduces substantial novel contributions across multiple dimensions: (1) Dataset enhancement: We expanded Articulation-XL from 33K to 59.4K samples with a diverse-pose subset that enhances generalization to varied pose inputs, addressing a critical limitation of pose diversity in existing datasets. (2) Technical innovations: (a) Joint-based tokenization with hierarchical ordering for skeleton generation, improving efficiency and performance; (b) Novel attention-based architecture with topology-aware joint attention mechanism for skinning weight prediction, achieving better generalization and faster inference; (c) First integration of video-guided optimization for automatic animation. (3) System-level contribution: While existing methods address rigging or animation separately, our unified framework enables the first pipeline from shape input to final animation - a significant advance in practical applicability.

We acknowledge the need for deeper analysis. Beyond ablation studies and qualitative examples in the main paper, we also provide quantitative results on three benchmarks (Tables 1, 2), inference time analysis (Tables S2, S5), generalization studies (Figure S4), more qualitative results in the appendix (Figure S5, Figure S7), and video results on the project page in supplemental material. However, we agree that a more comprehensive analysis would strengthen the work. In the final version, we will include failure case analysis, quantitative animation results, and a comprehensive comparison with MagicArticulate.

Q3: Insufficient discussion of the optimization-based animation; lacks discussions with related methods in 4DGS fields that employ deformation networks to predict motion parameters; joint-level fitting approach may explain limitations in fine-scale deformations.

Thanks for the suggestion. We acknowledge that animation deserves more detailed discussion. Our differentiable optimization directly optimizes joint transformations by minimizing rendering, tracking, motion regularization losses, without requiring neural network training. We chose direct optimization over neural approaches because: (1) it provides more precise control over joint-specific transformations; (2) it avoids the need for extensive motion training data; (3) it allows flexible adaptation to different video inputs without model retraining.

Regarding 4DGS methods: While both our animation optimization and 4DGS methods aim to deform vertices/Gaussians, there are fundamental differences: (1) We leverage learned skeletal structure for physically-plausible joint-based deformation, whereas 4DGS uses direct Gaussian deformation without anatomical constraints; (2) Our approach focuses on controllable object animation with video guidance, while 4DGS targets novel view synthesis of dynamic scenes; (3) Most importantly, rigging enables multiple control modalities. Beyond video-guided animation, rigged models can integrate directly into standard 3D animation software. This rigging-animation pipeline aligns seamlessly with current animation workflows in games and animated films.

The limitations in fine-scale deformation are indeed caused by our joint-level optimization approach. Fine-scale deformations require higher joint density in specific regions, but our current rigging pipeline generates skeletons with limited joint resolution. Animation-driven feedback to adaptively refine joint placement based on motion complexity may address this limitation in future work. We will include these discussions in the final version.

Q4: For animation, the authors primarily describe the optimization, but the actual animation process is not clearly presented, nor are there references to prior work that would help clarify it.

After optimization, our animation process follows standard skeletal animation principles [1]: (1) Forward Kinematics (FK): We compute global joint transformations from optimized local transformations using hierarchical forward kinematics, traversing the skeleton from root to leaves; (2) Linear Blend Skinning (LBS) [2]: Deform mesh vertices using weighted combinations of joint transformations, where each vertex is influenced by multiple joints according to skinning weights. This produces the final mesh animation sequence. We will add these implementation details and references to clarify the animation process in the final version.

[1] Parent, Computer Animation: Algorithms and Techniques, 2012.

[2] Lewis et al., Pose Space Deformation: A Unified Approach to Shape Interpolation and Skeleton-Driven Deformation, 2000.

Q5: The optimization framework uses seven different losses but lacks discussion on their necessity and balance.

Thanks for this important question. Our seven loss terms serve distinct objectives: (1) Rendering losses: RGB loss ensures photometric consistency; Mask loss maintains object boundaries; Optical flow loss enforces temporal motion consistency; Depth loss alleviates 3D ambiguities in monocular optimization. (2) Tracking losses align 3D skeleton joints and mesh vertices with keypoints tracked across video frames, ensuring optimized animation accurately follows the motion in the video. (3) The regularization term prevents temporal jittering by penalizing large transformation changes between consecutive frames. To balance these losses, we weight each term to ensure comparable magnitudes. In practice, regularization losses are down-weighted by 3–4 orders of magnitude relative to rendering and tracking losses to prevent over-smoothing. We will provide detailed discussions in the final version.

Q6: Limited examples of long sequences, complex motions, and humanoid morphologies.

Regarding sequence length, we primarily generate 5-second videos ($\sim$50 frames at 10 FPS) using Kling AI and Jimeng AI, which is longer than most existing 4D generation works. We will extend this to longer sequences (10 seconds) to better demonstrate long-range motion stability.

For complex motions, they typically involve rapid movements with large joint rotations between frames or extensive rotational changes throughout the video sequence. These challenges could potentially be alleviated by adaptive frame sampling based on optical flow magnitude, sampling more densely during rapid motion. For example, in the seahorse case on our supplementary project page, the tail exhibits large-amplitude oscillatory motion. While our method successfully captures the overall motion pattern, it struggles with precise alignment between the video and the tail's fine-scale movements. We will further discuss these in our limitations and future work.

In our paper figures and supplementary project page, we provide results across different categories: 4 humanoid examples, 4 quadrupedal animals, 3 marine creatures, and 2 flying objects to demonstrate generalization across morphologies. Since we cannot add visualizations during rebuttal, we will include more humanoid examples in the final version, which is more important for practical applications.

Q7: Lack of explanation for why spatial ordering performs better in Articulation-XL2.0.

To clarify, we actually use hierarchical ordering rather than spatial ordering in our method. We provide an ablation comparison between both ordering strategies in Section 5.5 and Table 3. While they achieve close performance in spatial alignment metrics, spatial ordering often yields disconnected skeletons because it may generate child joints before their parents, resulting in invalid parent references. We also provide additional discussion in Section D.1 (lines 127-129) and visualization in Figure S6 of the appendix.

Q8: Limitations regarding fine-scale deformations are acknowledged but not well illustrated.

Regarding fine-scale deformation limitations, an example can be seen in the turtle case in the More Animation Results section of our supplementary project page. While the reference video shows very soft motion of the turtle's forelimbs, our animation results appear less smooth due to insufficient joint density in these regions. This limitation stems from our skeleton generation producing few joints in areas requiring fine-scale deformations. Animation-driven joint refinement could improve smoothness but remains future work. We will include more detailed analysis and examples in the final version.

Q9: Missing comparison of optimization costs with baseline methods.

Thanks for your suggestion. We compare against two animation baselines in the paper: (1) L4GM (feed-forward method): While L4GM was trained on 128 80G A100 GPUs, our optimization approach needs only a single A100 GPU ($\sim$20 minutes for 10K vertex meshes) while avoiding geometric distortions inherent in L4GM; (2) MotionDreamer (per-object optimization): MotionDreamer takes approximately 10 minutes for meshes with less than 8K vertices but produces only 20-frame animations, while our method generates 50-frame animations in 20 minutes with superior quality and stability (refer to comparison videos in the supplementary project page). We will provide a comprehensive computational cost analysis in the final version.

Dear Reviewer kTdi,

The authors have provided detailed responses to your questions. What is your view after seeing this additional information? It would be good if you could actively engage in discussions with the authors during the discussion phase ASAP, which ends on EoA (Aug 6).

Best, AC