TreeSplat: Mergeable Tree for Deformable Gaussian Splatting

Thank you for your insightful comments! Below, we provide a point-by-point response to address your concerns. We welcome further discussion to improve the clarity and effectiveness of our work.

Q1-1: Correlation between Gaussians from same parent.

We thank the reviewer for this insightful observation. It is indeed possible for two Gaussians cloned from the same parent to gradually diverge in motion trajectory during optimization. Our model addresses this through the decay-weighted motion aggregation defined in Eq. (6), where each Gaussian inherits motion from its ancestors scaled by a learnable decay factor $\beta$. Empirically, we observe that the learned $\beta$ values after training span a wide range: from as low as $3.64 \times 10^{-9}$ to 1.0, with a mean of 0.15. When $\beta_i$ approaches zero, the influence from parent motion nodes becomes negligible, allowing the Gaussian $G_i$ to rely primarily on its own leaf motion. This mechanism provides flexibility for Gaussians that were initially correlated (due to cloning) to learn independently when required, effectively address this limitation.

Q1-2: Lack mechanisms to split the tree into two or more different trees as well.

We appreciate the reviewer’s thoughtful question. While our method does not include an explicit tree-splitting mechanism, the decay-weighted motion aggregation introduced in Eq. (6) implicitly enables flexible motion decoupling. Specifically, since the influence of ancestor nodes is attenuated exponentially by powers of the learned decay factor $\beta$, each Gaussian can adaptively learn to minimize its reliance on earlier parts of the tree. When the accumulated decay becomes negligible (i.e., $\beta^k \approx 0$ ), the Gaussian effectively becomes disconnected from its ancestor nodes. This in practice achieves the effect of tree splitting without requiring explicit structural modifications.

Q2: Comparison with similar works of fixed tree structure.

We thank the reviewer for highlighting the need for a more detailed comparison with prior hierarchical motion modeling works. HiMoR [48] adopts a fixed 3-layer tree structure that is manually defined and not adaptable during training. This reflects a design constraint rather than a design choice, HiMoR’s architecture only supports pre-defined hierarchies. In contrast, our method builds a dynamic motion tree structure in a data-driven manner.

Direct empirical comparison with these two methods is challenging, as HiMoR [48] is tailored for monocular dynamic scene reconstruction and Hicom [47] is developed for streamable dynamic scenes. Both are evaluated on domain-specific benchmarks that differ from ours. To enable a fair comparison, we re-implemented their respective hierarchical motion strategies within our framework. As shown in the table below, our learned adaptive tree consistently outperforms these fixed or heuristic hierarchies across two representative scenes, demonstrating the effectiveness of our approach.

We will clarify these differences in the related work section and include the corresponding empirical results in the revised paper.

Q3: Confusion about training efficiency.

We apologize for the confusion. Our pipeline uses the same number of iterations (30k) as existing dynamic Gaussian Splatting works. Given that the method involves a large number of Gaussians with varying node levels, we devoted substantial engineering efforts to implement both the forward and backward passes of Eq.~(6) as optimized CUDA kernels (more details can be found in Appendix B.1). This ensures high computational efficiency during training. As a result, the additional overhead introduced by the hierarchical tree structure is minimal: compared to our direct baseline DynMF, the average training time on the Neural 3D Video dataset only increases from 31 minutes to 34 minutes. We will clarify this by explicitly mentioning these speedup details at the last paragraph of Section 4.2 in the revision.

Q4: Overhead of motion tree compared to baselines.

We appreciate the reviewer’s suggestion. As mentioned in the response of Q3, the extra training time overhead of our method is minimal. However, we acknowledge that the GPU memory overhead is moderately increased due to additional data structures required to maintain the hierarchical motion trees during training. Specifically, our method incurs an average of 200MB additional memory on DNeRF scenes and 850MB on Neural 3D Video scenes. We will include GPU memory usage during training in Tables 1 and 2 in the revised version. This will provide a more complete view of the method’s resource characteristics.

Thank you for your insightful comments! Below, we provide a point-by-point response to address your concerns. We welcome further discussion to improve the clarity and effectiveness of our work.

W1: Motion blur in video results.

We acknowledge this is one of our limitations. Such blurred artifacts are commonly observed in dynamic Gaussian Splatting without incoporating the temporal opacity (as described in Sec 5.3). Through ablation experiments, we found that enabling temporal opacity helps reduce such motion blur artifacts, particularly on fast-moving parts like the head of the dog. However, it introduces a notable increase in the number of Gaussians and slightly compromise overall rendering quality. We believe this reflects a trade-off between spatial sharpness and temporal consistency.

W2: Side-by-side video comparison.

We appreciate the reviewer’s interest in comparison with video results. While videos are not able to be included in the rebuttal, we kindly invite the reviewer to refer to Fig.2, which presents a side-by-side visual comparison with three representative baselines: 4DGaussian, DynMF (our direct baseline), and 4DGS. Visual improvements are zoomed in and cropped for clarity. Our method consistently produces sharper details and fewer artifacts, supporting the improvements we claim.

W3 & Q3: Clarification on unused leaf nodes.

Thank you for pointing this out. The term unused leaf node refers to those leaf motion nodes whose associated Gaussians have been pruned during training. We will clarify this term in the revision.

W4: Tree structure and computation overhead analysis.

Thank you for the insightful comment. In practice, our implementation is parallelized at the Gaussian level. Specifically, our custom CUDA kernel launches a thread for each Gaussian, which independently traverses its associated motion tree path to compute the aggregated motion. Therefore, the actual computational overhead scales with the number of Gaussians, and the computation time (latency) is influenced by the maximum depth of the tree path each Gaussian must traverse. For the tree structure, we analyze the distribution of tree depths across scenes. As shown in the table presented in response to reviewer QfP4, the tree structures follow a long-tailed distribution, with over 97% of motion trees exhibiting depths within 10. This indicates that the majority of Gaussians are connected to shallow trees, ensuring efficient computation while still capturing hierarchical motion.

W5: Compare to heuristic motion hierarchy.

Thank you for the reviewer’s suggestion to include comparisons with heuristic motion hierarchy construction. We implemented a baseline on the Neural 3D Video dataset by generating heuristic hierarchies using semantic segmentation. Specifically, the SfM points from initialization are projected onto the first frame of Camera 0. We apply the Segment Anything Model (SAM) to segment this frame and assign each projected point a semantic label based on its 2D location. Points projected outside the image are labeled as invalid (−1). For each valid semantic label, we manually construct a two-layer motion tree, assigning a shared parent motion node and a leaf node to each Gaussian within the same segment. Experimental results on two representative scenes (Cut Roasted Beef and Sear Steak) show that this heuristic tree structure leads to a drop in reconstruction quality.

Such heuristic motion hierarchy faces difficulty in reliably inferring motion dependencies from sparse SfM points in dynamic scenes, where inaccurate correlations could introduce significant noise and hinder reconstruction. In contrast, our tree is adaptively constructed in conjunction with Gaussian densification, guided solely by reconstruction loss.

W6: Clarification on tree depth vs. motion coherence.

We appreciate the reviewer’s insightful concern. As shown in the table presented in response to reviewer QfP4, most Gaussians are connected to shallow trees, indicating local motion coherence rather than independence. Moreover, our ablation experiments of tree depth (from line 315 to 325) in Sec. 5.3 demonstrate that moderate tree depth leads to the best reconstruction performance, deeper trees do not necessarily yield better results. This suggests that the benefits of TreeSplat come from modeling local hierarchies, rather than from increasing depth arbitrarily.

W7: Comparison with D-3DGS [21] and SC-GS[32].

We apologize for this omission. This was due to differences in the experimental protocols: all dynamic Gaussian Splatting baselines in Table 1 were trained with 30k iterations for comparison, whereas D-3DGS and SC-GS were trained with 40k and 80k iterations respectively. Including them directly may introduce inconsistency. Given the reviewer’s helpful suggestion, we will include both methods in revision, with a clear table footnote clarifying this discrepancy to ensure informative comparison.

W8: Potential confusion of the term Merging.

We appreciate the reviewer’s comment regarding this potential confusion. Tree merging refers to merge coefficients rather than merging the tree structure. It is the process of consolidating the hierarchical motion coefficients along a Gaussian's ancestral path into a single coefficient vector prior to rendering. We agree that the current term may be confusing and will adopt clearer terminology such as motion coefficient consolidation in the revision to better reflect the actual process.

Q1: Impact of Initial Point Cloud Size on Performance.

We thank the reviewer for this insightful question. We have empirically validated the impact of initial point cloud size on performance. On the DNeRF dataset, the initialization points are randomly sampled from a cubic volume. Most of these points are pruned before tree construction (typically by the 500th iteration), resulting in minimal influence on the final motion tree structure and negligible effect on runtime.

On the Neural 3D Video dataset, initialization is based on SfM point clouds, sampled using farthest point sampling. We varied the number of initialization points from the baseline 300k to 600k and 30k. Increasing from 300k to 600k leads to no observable training time increase, and the resulting number of Gaussians remains in the same order of magnitude. However, reducing the initialization to 30k increases the average training time from 34 minutes to 41 minutes. This is because a sparser initialization requires the model to densify more Gaussians during training, which in turn leads to a larger number of deep motion trees.

As discussed in our response to W4, the computation time (latency) is influenced by the maximum depth of the tree path each Gaussian must traverse. With a larger number of deep trees, each CUDA block during kernel launch is more likely to encounter threads with deep tree paths, leading to an observable increase in training time.

Q2: Time cost of tree merging.

Thank you for the question. The tree merging process is highly efficient with our customized CUDA kernel. It takes 0.84 ms per scene on DNeRF and 1.22 ms per scene on Neural 3D Video dataset when evaluated on NVIDIA RTX4500 Ada GPU. This is negligible compared to the training time and has no impact on rendering quality. We will clarify this point in the revised version.

Q4: Internal structure of motion trees.

We appreciate the reviewer’s interest in the internal structure of the motion trees. To investigate this, we visualized Gaussian trajectories on the Neural 3D Video dataset by coloring all Gaussians bound to the same motion tree with same color. The results reveal that coherent motion trees consistently emerge around moving objects. For instance, in the flame_salmon scene, a motion tree clearly corresponds to the motion of a hand, with its associated Gaussians localized around the hand. In contrast, the static background is typically composed of a large number of isolated Gaussians without parent node (i.e., node level = 0). This behavior arises naturally from our tree growth mechanism, which is coupled with Gaussian densification and driven by gradients from reconstruction loss. Since moving regions introduce higher pixel inconsistencies across time, their corresponding gradients tend to be larger, promoting more active tree growth and hierarchical modeling in dynamic regions.

Q5: SSIM and LPIPS on N3V Dataset.

We sincerely apologize for omitting them in the manuscript. In the following table, we provide the per-scene SSIM and LPIPS results. These results will be included in the revised version.

Q6: Typical decay factor.

We thank the reviewer for this question. For the learned decay factor $\beta$, we computed statistics across all Gaussians associated with leaf motion nodes on Neural 3D Video dataset. Initialized at 0.5, the learned $\beta$ values after training range from a minimum of $3.64 \times 10^{-9}$ to a maximum of 1.0, with a mean value of 0.15. This indicates that while some Gaussians retain strong influence from ancestors, many exhibit rapid decay, enabling flexible control of motion inheritance depth.

Thank you for your insightful and positive comments! Below, we provide a point-by-point response to address your concerns. We welcome further discussion to improve the clarity and effectiveness of our work.

W1 & Q3: Analysis of constructed motion trees.

We appreciate the reviewer’s suggestion to provide more insight into the learned motion forest structure. To address this, we have conducted a quantitative analysis of the distribution of tree depths across representative scenes from the DNeRF and Neural 3D Video datasets. In the following table, tree depths are divided into six bins. For each interval, the number of trees (#Tree) and the average number of Gaussians per tree (#Gauss) are reported across four scenes. This analysis reveals a long-tailed distribution: over 97% of motion trees have depths within 10, while only a few reach deeper hierarchies. This supports the claim that most Gaussians are captured through relatively shallow hierarchical motion patterns, with deeper structures emerging only where necessary.

Additionally, we visualized Gaussian trajectories on the Neural 3D Video dataset by coloring all Gaussians bound to the same motion tree with same color. The results reveal that coherent motion trees consistently emerge around moving objects. For instance, in the flame_salmon scene, a motion tree clearly corresponds to the motion of a hand, with its associated Gaussians localized around the hand. In contrast, the static background is typically composed of a large number of isolated Gaussians without parent node (i.e., node level = 0). This behavior arises naturally from our tree growth mechanism, which is coupled with Gaussian densification and driven purely by gradients from reconstruction loss. Since moving regions introduce higher pixel inconsistencies across time, their corresponding gradients tend to be larger, promoting more active tree growth and hierarchical modeling in dynamic regions.

While figures are not able to be included figures in the rebuttal, we will incorporate both the quantitative statistics and qualitative visualizations in the revision to better illustrate the structure and interpretability of the learnt motion trees.

W2: Exploration of Failure Cases.

We thank the reviewer for pointing out this limitation. Indeed, our current formulation may struggle to fully capture topological changes or complex deformations such as cloth tearing or liquid splashing. These cases involve discontinuous motion patterns that challenge the assumption of temporally coherent, hierarchical motion. While our framework is effective for a wide range of objects and scenes with structured dynamics (as demonstrated in the benchmark datasets), we acknowledge that future extensions could explore more complex tree topologies, or integrate additional cues such as segmentation or optical flow to better handle such extreme non-rigid phenomena. We will clarify this limitation in the revision.

Q1: Discussion on potential for user interaction.

We appreciate the reviewer’s insightful suggestion. We believe the contribution of TreeSplat is complementary to methods like SC-GS. SC-GS models the motion of static Gaussians via external control points and an implicit MLP, where the trajectory of each Gaussian is interpolated from nearby control points using a linear blend skinning [1]. It also incorporates an adaptive densification strategy for these control points. Points with high gradients is cloned to better capture complex motion.

Given this setup, TreeSplat can be naturally integrated into SC-GS by modeling hierarchical motion relationships among these control points. In such a framework, manipulating a high-level node (e.g., a learned "shoulder" node) could propagate intuitive transformations to its descendants (e.g., "arm" Gaussians), enabling hierarchical editing or animation. Although not the primary focus of this work, we believe this integration offers a promising direction for future research in interactive 3D scene control. This discussion of the potential for user control will be included in the revision to better contextualize our contribution.

[1] "Embedded deformation for shape manipulation." ACM SIGGRAPH 2007

Q2: Robustness to Topological Changes & Role of Tree Growth.

We appreciate the reviewer’s thoughtful question. Our current framework indeed models each Gaussian’s motion via a motion tree with a one-to-one binding between Gaussian primitives and leaf motion nodes. The tree growth, including Leaf Expansion and Depth Promotion mechanisms, are primarily designed to progressively refine and structure the motion hierarchy, capturing multi-scale, correlated motions across the scene. However, these mechanisms do not explicitly detect or adapt to topological changes such as objects breaking apart. In scenarios involving significant discontinuities or topological shifts, the fixed hierarchical relationship may not perfectly capture the underlying motion patterns. While such failure modes are rare in our benchmark datasets, we acknowledge this as a limitation and will clarify it in the revision.

Thank you for your insightful comments! Below, we provide a point-by-point response to address your concerns. We welcome further discussion to improve the clarity and effectiveness of our work.

W1-1: When to apply Leaf Expansion and Depth Promotion ?

It is great that reviewer is asking! Leaf Expansion and Depth Promotion are applied in an alternating manner at each Gaussian densification step during training. Following the original 3D Gaussian Splatting setup, Gaussian densification is performed every 100 iterations starting from the 500th iterations. At each densification step, we alternate the two operations according to a fixed schedule: for every 5 densification steps, we apply Leaf Expansion in the first 4 steps, followed by Depth Promotion in the 5th step. This simple strategy empirically balances the width and depth of the motion tree and leads to effective hierarchical motion modeling. Details about when to apply these two operations are elaborated from line 261 to line 264 in the manuscript, we will make it clearer in revision.

W1-2: Other factors in tree construction like densification threshold.

Thanks! In our method, we do not introduce additional hyperparameters for tree growth. It is triggered automatically along the Gaussian densification procedure. To further clarify, we will revise the paper to explicitly mention this.

W2: Threshold ablation of Leaf expansion and Depth promotion operations.

We appreciate the reviewer’s interest in the details of tree growth! As stated earlier, there isn't a single threshold for growth that we can ablate. But indeed, we do control the growth process by tuning the schedule of different operations. In Section 5.3, we conduct an ablation (Effect of Tree Depth) by varying this interval of Depth Promotion relative to Leaf Expansion. The best performance is achieved when Depth Promotion is applied after every 4 steps of Leaf Expansion, which balances tree width and depth. We will clarify this coupling and highlight the ablation results more explicitly in the revision.

W3: How to make the connected Gaussian primitives constructed in a motion tree if only dependent on the "Leaf Expansion" and "Depth Promotion" operations?

Thank you for the question! The tree grows in conjunction with Gaussian densification, where new Gaussians are generated through either cloning or splitting from existing ones. These newly generated Gaussians are spatially adjacent to their source Gaussians, and our Leaf Expansion and Depth Promotion operations explicitly link them under in the tree as parent-child. This ensures that Gaussians with spatial and temporal proximity are hierarchically connected within the same motion tree. Over time, motion trees naturally group structurally related Gaussians, thereby enabling collaborative motion modeling across connected primitives.

W4: In the initialization stage, each Gaussian primitive is bound to a single motion tree. And maybe it's not reasonable because there are some SFM points are connected.

Thank you for the thoughtful observation. We agree that Gaussians reconstructed from SfM may exhibit spatial or structural connections. In our framework, each Gaussian is bound to a single motion tree throughout training. This design choice stems from the difficulty of reliably inferring motion dependencies from sparse SfM points, where inaccurate correlations could introduce significant noise and hinder reconstruction. To empirically validate this, we conducted an ablation on the DNeRF dataset: we partitioned the initialization volume ($[-1.2, 1.2]^3$) into a $16 \times 16 \times 16$ voxel grid, and assigned each Gaussian to a tree based on its voxel location. This voxel-based pre-grouping introduces hardcoded spatial correlations. As a result, the average PSNR dropped from 37.11 dB (default) to 35.89 dB. This demonstrates that enforcing spatial proximity as motion correlation can be detrimental. Our method instead learns motion tree structure progressively in conjunction with Gaussian densification, which proves to be more robust and effective in practice.

W5: How to avoid the situation that the dependent Gaussian primitives are connected in a motion tree?

Thank you for raising this concern. Actually, our design intentionally embraces spatial dependency among Gaussians, rather than avoiding it. The core motivation of our motion tree is to capture and exploit such correlations through hierarchical structure. During densification, new Gaussians are created in local neighborhoods, and their motion branches are extended from existing parent nodes, naturally preserving spatial continuity. Thus, Gaussians that are spatially related are encouraged to share motion components through shared ancestry in the tree. This promotes coherent motion modeling rather than viewing such dependency as a conflict.

Thanks for authors' response. After reading the response of authors and other reviewers' comments, I think the response solved most of my concerns and I decide to keep my original positive score. And I hope the author can add the necessary citation and comparison with [1]. The implementation details should also be further refined in the revised manuscript.

[1] 4d gaussian splatting with scale-aware residual field and adaptive optimization for real-time rendering of temporally complex dynamic scenes, ACMMM2024 Best Paper Candidate.