LayeredGS: Efficient Dynamic Scene Rendering and Point Tracking with Multi-Layer Deformable Gaussian Splatting

● This paper primarily compares only three papers—Dynamic3DGS, SC-GS, and RealTime4DGS—while there are other online dynamic scene reconstruction methods that should be considered, such as 3DGSstream, Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos, and Streaming Radiance Fields for 3D Video Synthesis, etc..

● The dataset used for validation in the paper features relatively simple scenes and clear motion. The paper could consider conducting further validation experiments on the Nvidia Dynamic Scene Dataset, N3DV, and UCSD datasets.

● In the contribution summary, the paper mentioned “We demonstrate one novel application, Self-supervised Articulated Object Segmentation, showcasing the utility of deformation information for the first time.”. I don’t think this is the first time the deformation utility demonstrated, there are lots of works doing scene decoupling based on learned motion, like Dˆ2neRF, and NeuralDiff. Also, there are some works that focus on the “self-supervised articulated object segmentation” mentioned in the paper, like Watch-It-Move and MovingParts, but these works are not discussed in the paper.

• The contribution of the paper can be summarized in one sentence: using a fixed yet inflexible method to enforce the as-rigid-as-possible regularization, which has already been widely used in other dynamic 3DGS projects including SCGS and HiFi4G. Previous methods have abundant mechanisms to grow, prune, and adjust anchor points, while the proposed method does not present such flexibility for Gaussian clustering. This dramatically undermines the generalizability of the method.

• The clustering criterion is simply the Gaussians' location, which in most cases could not provide enough guarantee of similar deformation behavior, let alone the same affine deformation behavior. This is also why previous works adopt rigidity as a regularization term instead of a hard constraint.

• The multilayer design seems unnecessary. Equation (8) itself is simply an affine transformation (assuming there is no scaling in the scene, which widely holds in your evaluated dataset). And the nested multi-layer affine is still an affine. Since the benefits of reducing the optimization space have already been achieved by the finest layer of clustering, the motivation for multiple layer structure is questionable.

• The evaluation methodology raises several concerns:

  • Throughout the comparison, the authors claimed that "for fairness" they allow other methods to train for merely 100 iterations per new frame. Obviously, most other methods do not achieve model convergence, and the comparison in rendering quality in this case is meaningless. A better comparison setup would be the average iterations it takes for those models to converge as well as the final metrics after convergence. Time to converge could also be a very informative metric as long as you are running the experiment under the same hardware. All these projects are open-sourced, so you are not expected to worry about implementation variance.
  • The customized "accelerated dataset" is confusing, especially why reducing the number of frames in Particle-NeRF is necessary in the evaluation setup. The vanilla Dynamic3DGS could achieve a 39 PSNR on the Particle-NeRF dataset; however, in your setup, it only achieves less than 30 PSNR. While I understand your intention to magnify the advantages of your method, at least show the results of the original dataset so that people can judge your work fairly.
  • The exclusion of DNeRF and HyperNeRF datasets citing "online case" limitations appears selective, particularly given the existence of other online monocular reconstruction methods like DynOMo(https://arxiv.org/pdf/2409.02104).
  • The omission of N3D and Immersive datasets due to "small movement scale" seems unjustified. If these datasets are indeed simpler cases, demonstrating superior performance would strengthen the method's validation.

1. The introduction of Dynamic Novel-View Synthesis in the related works section is redundant. To be specific, it's better to focus on the difference between your method with other concurrent methods. Static novel view synthesis is not necessary since your paper is focus on dynamic scene reconstruction. Though this paper highlighted the difference between their method and SCGS in lines 153 to 158, the high-level idea of those two methods is the same, which are k-means cluster. The main difference is this paper learns clusters for each cluster, which is called multi-layer clusters in the paper.

2. It is not clear what exactly the loss function LayeredGS applied. Moreover, it is not clear how to formulate each Gaussian or cluster across time. As introduced in line 277, LayeredGS optimized $R_j$, $t_j$, $c_j$, and $s_j$. Do you need to optimize $R_j$, $t_j$, $c_j$, and $s_j$ for each frame? Besides, since the movement of each Gaussian is with respect to its corresponding cluster, how do you formulate the deformation of each cluster exactly? Generally speaking, this part is so confusing for me. Could you pls further provide:

• A clear formulation of the loss function used in LayeredGS.
• A detailed explanation of how Gaussians and clusters are formulated across time.
• Clarification on whether $R_j$, $t_j$, $c_j$, and $s_j$ are optimized for each frame.
• A precise formulation of how the deformation of each cluster is calculated.
• The detailed loss function and the optimization procedure.

3. Since 3DGStream and SpacetimeGS [1] are developed for multi-view 4DGS rendering, it is more convincing to compare LayeredGS with those two methods on the Neural 3D Video dataset and meet-room dataset, even though SpacetimeGS is an offline method. You may consider including comparisons with 3DGStream and SpacetimeGS on the suggested datasets, or explain why such comparisons were not included with convincing reasons.

4. The comparisons with other methods are not fair. First, 4DGS and SCGS are mainly designed for monocular dynamic scene rendering. Though 4DGS showed the performance on the Neural 3D Video dataset, the performance is still lower than 3DGStream or SpacetimeGS since 3DGStream and SpacetimeGS are designed for multi-view deformable scene reconstruction. Second, as mentioned in lines 407 to 417, the total iteration is set to 100 multiplied by the total number of frames for other compared methods. I'd say the total iteration is insufficient for those compared methods to reach their optimal performance. You can get better performance for compared methods by increasing the training period. Could you further:

• Provide a clearer justification for including monocular methods in their comparison, or focus more on multi-view online or offline methods.
• Run additional experiments with increased training iterations for the compared methods to ensure they reach optimal performance.
• Include a discussion of the limitations of their current comparison approach and how it might affect the interpretation of the results.

[1] Li, Zhan, et al. "Spacetime gaussian feature splatting for real-time dynamic view synthesis." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

The weakness of this paper lays in the following aspects. My final decision will consider the feedback of the authors during the discussion section and I would like to raise the score if some of the major concerns are solved.

1. Compared with the data-sets adopted in the DynamicGS, the comparative experiments are small and limited. The evaluation is primarily conducted on the Panoptic and FastParticle datasets. While these datasets are challenging, the paper would benefit from testing on a broader range of datasets, including more real-world scenarios, to demonstrate the generalizability of the method.

2. The paper lacks comprehensive ablation studies to quantify the contributions of different components of the proposed method. For instance, evaluating the impact of the number of layers in the clustering structure or the choice of clustering algorithms would provide deeper insights.

3. Some aspects of the multi-layer clustering and deformation optimization are not fully detailed. Providing pseudo-code or algorithmic summaries in the main text could enhance reproducibility. More explanations on how clusters are merged across layers and how this affects the deformation modeling would improve clarity.