HEIR: Learning Graph-Based Motion Hierarchies

Response

We thank Reviewer s1zH for highlighting the relevance of our work and for raising valuable points regarding its distinction from rigging methods. In the following, we clarify the differences in objectives and supervision between our method and existing rigging techniques, and outline how the two can be complementary. We address the concerns on error accumulation through a depth-based analysis and provide new ablation results showing how training data quantity affects the accuracy of hierarchy reconstruction. These additions will be reflected in the revised version.

[W1, Q1: Comparison with rigging techniques]

We thank the reviewer for directing us to rigging techniques, which is an established field very relevant to our method. We will add discussion and corresponding references to the paper to adequately situate the work in the landscape of existing approaches. Below we summarize the key distinctions:

1. As the reviewer mentioned, compared to these methods, we generalize to arbitrary scenes and are not limited to mesh-representation or point-cloud. To further demonstrate this, we tested the model on an additional application involving arbitrary affine transformations. With minor architecture modifications, we were able to successfully recover motion hierarchies in a planetary system. The model identified parent-child relationships corresponding to orbital structure, correctly disentangling planets from moons. We will include visualizations and further details in the camera-ready version.

2. Although RigNet and UniRig generate a skeleton that has a very similar structure to our hierarchy graph, RigNet’s BoneNet and UniRig’s Skeleton-Tree GPT are both trained on thousands of meshes with ground-truth joints and skinning weights. Our hierarchy graph is directly learned from the motion from the scene via a self-supervised reconstruction loss; no pre-rigged data is required.

3. BANMo/DreaMo learn where to place deformable handles (‘neural bones’ in the paper) so the object can be rendered consistently across video frames whereas our method learns how those parts influence each other over time. This fundamental difference in objective makes their skeleton (only available in DreaMo) adjacency-based, and doesn’t yield the parent-child relationships we target.

4. Our goal is to provide a motion-centric hierarchy, which is, in fact, complementary to many rigging techniques. For example, a supervised rigger could seed an initial topology, while our optimization refines parent-child directions using dynamics. It is especially useful for long-range dependencies that K-NN initialization may miss. We will add these discussions to the revised version of the paper.

[W2, Q2: Additional error accumulation analysis]

• We thank the reviewer for highlighting this point and have conducted an explicit depth analysis. As we are unable to add figures here, we describe the results, and will provide an additional figure for the camera-ready version. We measured the normalized absolute velocity error $(|v_{abs}^{pred}(i) - v_{abs}^{gt}(i)| / |v_{abs}^{gt}(i)|)$ for each node on the excavator scene and aggregated the mean ± std values at each successive hierarchy depth.
• The mean value of the error stays bounded within the range of 1-3%, which indicates that our error doesn’t explode along deeper branches. This is because each hierarchy level is trained under the same end-to-end reconstruction loss, and the local motion term (observed motion = parent-inherited motion + local motion) re-anchors every child to the observed velocity field, correcting any errors induced by its parent.

[W3, Q3: Additional training data quantity analysis]

• We appreciate the reviewer for bringing up how the number of available timepoints influence the quality of results and have run an ablation in which we train on only a fraction f of the timesteps (T = 200 total) and measure hierarchy-reconstruction accuracy over 100 iterations.
• Accuracy climbs steadily, reaching roughly 45% accuracy when half the frames are available, and begins to plateau once approx. 70% of the timeframes are included. We will add the plot and a brief discussion in the supplementary material.

Dear Reviewer s1zH,

As the discussion period winds down, we would greatly appreciate it if you could let us know whether our rebuttal has addressed your concerns or if any additional clarification would help. If you feel your points have been satisfactorily resolved, we would be grateful if you could consider increasing your score.

Best regards,

Authors 13083

We thank Reviewer 3bfi for their assessment and detailed suggestions. In response, we have extended our benchmarks to include full D-NeRF comparisons with SC-GS and clarified performance gains with averaged metrics. In the following, we commit to improved figures that better illustrate deformation effects, and also expand the discussion of current limitations, including long-range dependencies and deformation artifacts, and outline straightforward extensions to address these in future work.

[W1, Q1: Additional benchmark scenes]

• We have extended our evaluation to the full D-NeRF dataset. The proposed method outperforms SC-GS on average over the full dataset across all metrics. Specifically, the average metrics of our method versus SC-GS are PSNR(↑) 20.35/19.30, SSIM(↑) 0.9438/0.9366, CLIP-I(↑) 0.9635/0.9621, LPIPS(↓) 0.0453/0.0659 and FID(↓) 68.65/107.03. These metrics will be added to the revised Table 2.

[W2, Q2: Improvements to Figure 3]

• We thank the reviewer for this suggestion. For better illustration, we will overlay the scene before and after deformation and use arrows to indicate the user-defined position and direction of the motion for Figure 3. We are not allowed to attach the revised figure in the rebuttal but it will be added to the camera-ready version.

[W3, Q3: Illustration of hierarchy levels]

• We thank the reviewer for this suggestion. In the revised version, we will include a figure to show different hierarchical levels of Gaussians as what we have done in the toy example.

[Q4: Clarification on long-range dependencies]

• We agree with the reviewer that our current implementation which builds the initial edges in the graph using k-NN may miss dependencies between widely separated parts. We made this choice to keep message-passing efficient, and it empirically yields strong results on the scenes we target.
• The method itself, however, is not limited to local edges: replacing k-NN with a sparsely sampled global attention layer, dilated‐radius neighbours, or a small set of randomly initialized long-range edges would allow information to propagate across distant regions without altering the loss or training procedure. We will add a discussion of this and note that exploring these global-connectivity variants is a straightforward avenue for future work.

[Q5: Clarification on unwanted deformation artifacts]

• We thank the reviewer for pointing this out. We do not impose hard rigidity constraints here to allow non-rigid, natural deformations, especially for the characters. We would like to note that as the first method to capture structured motions without manually designed hierarchies, we already achieved less stretching or compression artifacts than SC-GS. Moreover, the learned parent-child graph now provides an explicit motion structure, which allows us to selectively add local rigidity. For example, edges whose nodes maintain near-constant relative pose over time could be strictly rigidity-regularized during deformation. This would be an interesting future work for us and we will add this discussion in the revised conclusion section.

We thank Reviewer HD6y for their feedback and constructive questions. In the following, we clarify the relevance of our problem setting by highlighting its generality beyond standard 3D scene applications, supported by an additional planetary orbit reconstruction application. While the model is optimized per scene, this follows the established paradigm in NeRF-based pipelines and enables generalization to moderate unseen deformations. We address the reviewer’s questions by providing new ablations on the influence of numbers of training timepoints, and by detailing the computational and modeling advantages of using a neural network for hierarchy prediction over direct optimization.

[W1: Unclear problem relevance, additional training data quantity analysis]

• Our setting targets general hierarchical motion modelling: we learn a directed parent-child graph that decomposes every observed velocity into a motion inherited from its parent plus a local residual. It operates on generic motion elements (e.g., Gaussian splats, tracked points) and makes no assumptions about meshes or object class. As shown by our toy example, the approach applies equally to other fields, be it planetary orbits or rigid body motions. Existing rigging or dynamic-NeRF editors focus on character skeleton or 3D scenes, and are not able to tackle problems outside their domain. Thus, our method is able to cover a wider range of practical motion scenarios and provides relevance for motion analysis across diverse domains.

• We have tested our model on an additional application of planetary movements by expanding the scope to arbitrary affine transformations, with only minor modifications to the model. In this setting, the method successfully identifies relations in planetary orbits, i.e., which bodies orbit around which, allowing us to disentangle planets and moons. As we are unable to include new figures here, we will include them in the camera-ready version, alongside details on our synthetic dataset and the training setup.

• Regarding the reviewer’s concern that the model is optimized per scene, we note that this is by design, and akin to the standard test-time learning phase in NeRF and dynamic-GS pipelines. The method is fully self-supervised, optimized once and then ready to be reused for any subsequent deformations or queries.

• We acknowledge that the method depends on the observability of motion as we stated in the limitation section. Although the hierarchy is learned from the motions observed, it generalizes to moderate, previously unseen deformations. However, extreme edits outside the training distribution may yield less predictable semantics, similar to other data-driven methods.

• We appreciate the reviewer for bringing up how the number of available timepoints influence the quality of results. We have run an ablation in which we train on only a fraction of the timesteps (T = 200 total) and measure hierarchy-reconstruction accuracy over 100 iterations. Accuracy climbs steadily, reaching roughly 45% accuracy when half the frames are available, and begins to plateau once approx. 70% of the timeframes are included. We will add the plot and a brief discussion in the supplementary material.

[Q1: Additional training data quantity analysis]

• Please refer to the last paragraph in W1.

[Q2: Clarification on advantage of using neural networks]

• The motion hierarchy we learned is a discrete tree over N elements, and the number of possible trees grows super-exponentially with N. Direct optimization therefore becomes a hard combinatorial search: even with greedy agglomerative clustering one must evaluate all combinations of edge scores repeatedly and still risks getting trapped in local optima.

• If formulated as a minimum cost arborescence problem, we could find an optimal hierarchy with $O(N^2 \log(N))$ complexity using the Chu-Liu/Edmonds algorithm. However, this approach is not differentiable (if needed for other downstream applications) and is not fully parallelizable. Our GNN approach has a better time complexity of $O(N^2)$, and furthermore turns this discrete problem into a continuous, differentiable relaxation: it produces soft parent probabilities that are refined jointly with the motion-decomposition loss. Gradient descent then updates all edges in parallel without the need to enumerate all possible trees.

We thank Reviewer r1v9 for the thoughtful and constructive feedback. In the following, we clarify that the reported metrics differ from standard benchmarks due to the nature of our editing task, and include standard reconstruction scores to facilitate direct comparison. We also clarify the interpretability of our learned motion hierarchy, now supported with visualization together with expanded explanation and an additional application. Finally, we address concerns around occlusion handling and graph scalability by detailing how our method observes the full 3D scenes and leverages a coarse-to-fine motion representation. We will also correct the typo the reviewer kindly pointed out.

[W1: Low PSNR/SSIM values]

• We thank the reviewer for pointing this out. Dynamic-scene reconstruction methods aim to reconstruct the views or 3D model of a scene within the observable timeframe, whereas our goal is to learn an editable, interpretable motion hierarchy, given an already reconstructed dynamic scene. Accordingly, the PSNR and SSIM numbers in Table 2 are not reconstruction scores against ground-truth views (as in the standard D-NeRF benchmark). Instead, they are computed between the manually deformed scene and the original non-deformed one (explained in line 293-294). Because our edits deliberately introduce pixel-level warps, full-reference metrics that assume perfect spatial alignment inevitably drop; we keep them as a measurement to confirm that no gross photometric artifacts appear.

• To assess quality in a way that is robust to geometric displacement, we report LPIPS and CLIP-I because both compare deep feature embeddings that remain stable under moderate warps. They reflect perceptual and semantic fidelity, respectively. For additional reference-free evaluation we now also provide FID (Frechet Inception Distance) scores of 68.65 (our method) v.s. 107.03 (SC-GS). Note that this number is relatively high compared to values reported in generative models as the D-NeRF subset contains few images.

• For completeness, we report the reconstruction component of SC-GS that we use as input for our method, which achieves PSNR 40.72, SSIM 0.998 and LPIPS 0.021 before performing any scene deformation with our test-time method.

[W2: Lack of interpretability]

• We claim that the hierarchy graph we learned is interpretable because each node corresponds to an observed element (Gaussian or point), and the parent-child links correspond to semantically meaningful motion groups. In our toy example, the learned hierarchy recovers this: nodes with similar inherited motion are clustered under common parents. The edge weights reflect the likelihood of parent-child relationships, making the resulting graph an interpretable representation of the motion composition. In the dynamic 3D GS scene, we see the resulting groups of the hierarchy tree coincide with intuitive semantic parts like the shovel of the excavator and the limbs of the person. In the camera-ready version, we will add one illustrative figure visualizing the learned hierarchy with nodes overlaid on the excavator scene, showing the learned shovel-body relative motion. We will also expand the accompanying text to make this interpretability evidence explicit.

• Separately, we demonstrate the interpretability with an additional application, where we recover motion hierarchies in a synthetic orbital system by correctly identifying which bodies orbit around which. The learned graph reflects the expected structure of the solar system, with moons linked as children to their respective planets, and planets to a star, mirroring simplified orbital dependencies. We will include a visualization of the recovered hierarchy alongside details of the model adaptations required in the camera-ready version.

[L1: Reliance on observability of motion]

• Our method is a test-time learning approach that, by design, does not rely on priors. We agree with the reviewer that we cannot infer static objects in the training data as we stated in the paper limitation section. However, we are able to tackle occluded parts since we first reconstruct the full 3D scene from multiple views, like all the NeRF and GS methods. It gives us a full understanding of all the existing motion components in the scene, even if the object is occluded in some of the views.

[L2, Q1: Clarification on the limit of the number of points]

• We agree with the reviewer that we are limited by the graph size so we cannot deal with every Gaussian splat in large-scale scenes. In our experiments, we work with ~5K GSs per scene and use weight-based skinning to apply deformation on the remaining GSs representing the scene. We emphasize that the hierarchical structures in motion tend to be on a more macroscopic level, which means we are not sacrificing quality with these interpolations.

Dear Reviewer r1v9,

As the discussion period winds down, we would greatly appreciate it if you could let us know whether our rebuttal has addressed your concerns or if any additional clarification would help. If you feel your points have been satisfactorily resolved, we would be grateful if you could consider increasing your score.

Best regards,

Authors 13083

Dear Reviewer r1v9,

We wanted to kindly follow up on our earlier message. As the discussion period is nearly over, we would greatly appreciate it if you could share your thoughts on whether our rebuttal addressed your concerns or if further clarification is needed.

Best regards,

Authors 13083