D-MiSo: Editing Dynamic 3D Scenes using Multi-Gaussians Soup

We thank the reviewer for the remarks and excellent comments and appreciate the time taken to review our manuscript. We have responded to these remarks below.

W1 Multi-Gaussian representations are similar to SC-GS's control points ...

Our solution differs in architecture and training strategy from SC-GS, which uses control points while we employ two hierarchies of Gaussian components. More significantly, we have an extremely large possibility of editing dynamic scenes. All other Reviewers agree that D-Miso introduces significant novelty, as underscored by Reviewer 2 (“Experimental results show that this method not only matches the rendering quality of SC-GS but also enables the editing of more extreme large motions”). The SC-GS authors use control points, which influence a large part of the object. Therefore, when we modify some object element (e.g., the hand of the object, see Fig.4 in our manuscript), the full object changes (in the case of the example from Fig. 4, the head is moving). Thanks to using two levels of Gaussian components, we can modify objects easily without artefacts (see Fig. 4 in the main paper). Consequently, our new GS representation is better suited for editing dynamic scenes than SC-GS, and it is the paper's main contribution.

W2A Most demos are provided in synthetic datasets, which is not convincing. Results on real-world datasets such as [9, 13] are recommended to be included.

The SC-GS and other baseline models, which modify dynamic scenes, use only synthetic examples. In contrast, our paper includes examples of full scenes (please see Fig. 3), where we modified real-world dynamic scenes. Furthermore, in Appendix C and Fig. 14, we will also provide additional real-scenes modifications. We also add movies (GIFs) from real scene modification to supplementary materials. We agree that such experiments are valuable, and we will move more examples from the appendix to the main paper.

W2B The "Related Works" section should be reorganized. More discussions on multi-view dynamic scenes and flow-based radiance fields should be included.

Early works on dynamic scenes face difficulties when dealing with monocular settings and uncontrolled or lengthy scenarios. To enhance scene motion modeling, some works in NeRFs utilize flow-based techniques [10, g1, g2]. Method [10] extends the static scenario by including time in the domain and explicitly modeling 3D motion as dense scene flow fields. For a given 3D point and time, the model predicts both reflectance and opacity and forward and backward 3D scene flow. In [g1], the model learns scene appearance, density, and motion jointly. NeRFlow internally represents scene appearance and density with a neural radiance field and scene dynamics with a flow field. Finally, [g2] focuses on predicting explicit 3D correspondence between neighboring frames to achieve information aggregation. The correspondence prediction is achieved through the estimation of consistent depth and scene flow information across frames with the use of dedicated networks.

Following suggestions from another Reviewer (hxyh; W1), we plan to include additional works that operate in dynamic scene reconstruction: [b1, b2]. These are early Gaussian splatting works that significantly contributed to the dynamic scene reconstruction field, and we plan to include them in the revised version of the manuscript. These methods, however, require a multi-view setup similar to those already included [21].

[g1] Yilu et al. 2021, “Neural radiance flow for 4D view synthesis and video processing,” in Proceedings of the IEEE/CVF ICCV

[g2] Zhou et al. 2024, "DynPoint: Dynamic Neural Point For View Synthesis," in NeurIPS 36

W3 “If your method is based on [3], a brief review should be included in the main text as a preliminary to improve the presentation.”

Our model is based upon the Gaussian parameterization framework outlined in [3], with a detailed elaboration provided in Appendix A. To enhance readability, this section will be moved to the main text.

W4 More details such as storage costs and training time (for each stage) should be included. Additionally, would the rendering speed be affected by the large deformation network?”

Due to the rebuttal length, ablation studies are included in the global rebuttal.

W6 “What is n_core as mentioned in Eq.1?”

We have introduced the notation N_Core​ (Eq. 1) and N_Sub​ (Eq. 3) to clarify our references to Gaussian distributions for Core-Gaussians versus Sub-Gaussians.

Q1 “More comparison and discussions with SC-GS should be included since SC-GS can also support editing and share similar ideas with control points/Gaussians.”

We provide numerical comparisons with SC-GS in the main paper body (Tab 1, Tab 2, Appendix: Table 5 and Table 6 in our draft paper). Fig. 4 presents a visual comparison with the SC-GS model, highlighting the placement of key points in SC-GS. The SC-GS model lacks key points necessary for hand rotation, as demonstrated in our method (third image, second column). Additionally, overextending the SC-GS model results in "tearing" the human's arm, whereas our method handles scaling more effectively. Additionally, we will add more visual comparisons to illustrate our approach's robustness clearly.

Q2 “As I understand, deformation-based methods find it challenging to synthesize correct novel views with large motion. Could the authors provide comprehensive insight into this issue?“

Models based on deformable networks, including our model and SC-GS, create representations of key components (such as Core-Gaussians in our case and landmarks points in SC-GS) by positioning them optimally for ease of adjustment at each time instant. However, when there is significant movement in the training set, the network responsible for component localization may face challenges. Such a problem is the same in all models which use deformable networks.

We thank the Reviewer for the feedback and for pointing out improvements to our paper. We respond to these concerns in the points below.

W1: Lack of training time comparisons.

Here, we showcase a time comparison between SC-GS and D-Miso models, with the former operating noticeably faster than the latter. This can be attributed to two causes. First, unlike SC-GS, D-Miso benefits significantly from a larger batch size during training. Second, D-Miso incorporates two layers of Gaussian components (Core-Gaussians and Sub-Gaussians). On the other hand, SC-GS uses landmark points, making it faster in training but more constrained in editing and processing full scenes. Furthermore, our model obtains better results on real scenes, see Tab. 3 and Tab. 4.

W2: Lack of comparison against gs2mesh-based methods and cage-based methods.

- GS2Mesh based methods: We have already included the following mesh-based Gaussian splatting works, which operate in static setups [3, 26, 27]. To address the Reviewer’s remark, we will also include a more recent work that combines meshes with Gaussians and operates in dynamic scenarios [e1]. This framework reconstructs a high-fidelity and time-consistent mesh from a single monocular video. Building on this representation, DG-Mesh recovers high-quality meshes from the Gaussian points and can track mesh vertices over time, enabling applications such as texture editing on dynamic objects. However, the method relies on a Poisson Solver and differentiable Marching Cubes to recover the deformed surface, significantly complicating and slowing down the pipeline. Moreover, it does not explore geometry modification capabilities, which constitute a significant aspect of our work.

- Cage based methods: We thank the reviewer for suggestions regarding cage-based methods. In response, we will include two recent works that utilize the cage concept. Cage-based methods like [e2, e3] are crucial for the efficient and intuitive manipulation of 3D scenes. By providing a direct deformation approach, these methods simplify a traditionally complex task, making it more accessible. However, they require complex cage-building steps, which complicates the pipeline. These works in their current form do not address training on dynamic scenes so direct comparison to our method is not feasible. Additionally, cage-based methods may lack the flexibility and precision of manual, vertex-level deformation techniques, potentially missing fine details. In contrast, our work emphasizes small and precise element movement, addressing this limitation.

[e1] Liu et al. 2024, “Dynamic Gaussians Mesh: Consistent Mesh Reconstruction from Monocular Videos,” arXiv preprint arXiv:2404.12379. Retrieved from https://arxiv.org/abs/2404.12379

[e2] Huang et al. 2024, “GSDeformer: Direct Cage-based Deformation for 3D Gaussian Splatting,” arXiv preprint arXiv:2405.15491. Retrieved from https://arxiv.org/abs/2405.15491

[e3] Jiang et al. 2024, “VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality,” arXiv preprint arXiv:2401.16663

Q1: Lack of discussion with respect to the human-NeRF (Neural Body/HumanNeRF/UV-Volumes) and human-GS methods (Animatable Gaussians/D3GA/GoMAvatar).

Several works combine human faces/avatars and meshes, and those based on Gaussian representations are particularly relevant to this discussion. Examples include [f1-f5] - all of them can model dynamic avatars, most based on video data. However, these specialized works rely on human pose/face priors (e.g., FLAME fitting [f6]). We believe it would be valuable to explore the potential of adapting D-MiSo concepts in this context, building on existing works in the field. We envision Stage 1 focusing on effectively integrating such models in motion while the anchoring concept could be adapted to efficiently capture the small details of moving avatars. However, human avatars are a broad topic, with several notable and advanced works emerging each month. A thorough and accurate analysis of related works is required to fully investigate the best utilization and adaptation of our method in this context. We include the following papers in the Related Work and Future Work Sections.

[f1] Li et al. 2024, “Animatable Gaussians: Learning Pose-dependent Gaussian Maps for High-fidelity Human Avatar Modeling”, in Proceedings of the IEEE/CVF CVPR

[f2] Zielonka et al. 2023, “Drivable 3D Gaussian Avatars”, arXiv preprint arXiv:2311.08581.

[f3] Wen et al. 2024, “Gomavatar: Efficient Animatable Human Modeling from Monocular Video Using Gaussians-on-Mesh,” in Proceedings of the IEEE/CVF CVPR

[f4] Qian et al. 2024, "GaussianAvatars: Photorealistic head avatars with rigged 3d Gaussians," in Proceedings of the IEEE/CVF CVPR

[f5] Xiang et al. 2023, "Flashavatar: High-fidelity digital avatar rendering at 300fps." arXiv preprint arXiv:2312.02214

[f6] Li et al. 2017, “Learning a model of facial shape and expression from 4D scans”, in ACM Transactions on Graphics (Proc. SIGGRAPH Asia), 36(6)

We thank the Reviewer for the vote of confidence in our manuscript and for the in-depth feedback and suggestions, which we are happy to incorporate and feel will improve the manuscript. We respond to all the questions and concerns raised in the answers below.

W1: I think the following papers should also be cited, because they have made significant We contributions to the early dynamic scenes reconstruction based on Gaussian splatting …

Thank you for suggesting this; we will cite these papers in our manuscript. Method [b1] proposes approximating the spatiotemporal 4D volume of a dynamic scene by optimizing a collection of 4D primitives with explicit geometry and appearance modeling. This method uses 4D Gaussians parameterized by anisotropic ellipses and view-dependent, time-evolved appearances represented by 4D spherical harmonics coefficients. In [b2], a novel scene representation called Spacetime Gaussian Feature Splatting has been introduced. This approach extends 3D Gaussians with temporal opacity and parametric motion/rotation, enabling the capture of static, dynamic, and transient scene content. Additionally, the method incorporates splatted feature rendering to model view- and time-dependent appearances while maintaining a compact representation size. The method is notable for its high rendering quality and speed while also being storage-efficient. We agree that both these works have significantly contributed to the reconstruction of dynamic scenes and align well with the topic under discussion. Both of them will be included in our manuscript.

[b1] Li et al., 2024, “Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

[b2] Yang et al., 2024. “Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting”, Proceedings of the International Conference on Representation Learning

W2: L136, The concept of anchor Gaussian originally comes from Scaffold-GS [1], not from Spec-Gaussian. It would be even better if Scaffold-GS could be cited.

Thank you for this excellent comment. In [c1], the authors introduce the concept of anchor points to tackle the problem of overfitted models caused by redundant Gaussians. Scaffold-GS addresses this issue by distributing local 3D Gaussians according to anchor points and predicting attributes based on viewing direction and distance. This approach reduces redundancy, enhances scene coverage, and maintains high-quality rendering with improved robustness to view changes. Therefore, we will provide an extended description of [c1] in Related Works.

[c1] Lu et al., “Scaffold-GS: Structured 3D Gaussians for view-adaptive rendering,” Proceedings of the IEEE/CVF, CVPR 2024

W3: It would be even better if Deformable-GS [2] were included in the comparison of quantitative metrics (such as Tabs. 1-2), as it is the first deformation-based dynamic Gaussian splatting method and serves as the baseline for SC-GS.

The model Deformable-GS [d1] is indeed a prototype of the SC-GS model. Deformable-GS is one of the first works targeting deformable/dynamic scenes. It introduces an MLP-based deformation network to adapt Gaussian parameters to a given timestep. Below are the PSNR results of our comparison with SC-GS and Deformable-GS, which we will include in our article.

[d1] Yang et al., 2023, “Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction”, arXiv preprint arXiv:2309.13101.

W4 I think the ablation study is not thorough enough. Although the ablation on batch size is appreciated, I am unclear about the roles of other components of the method. Corresponding ablation studies are needed to make the paper more robust. For example, I am very interested in understanding the impact of the Sub-Rot Network mentioned in L218 on the rendering metrics.

The Sub-Rot Network, despite being a simple single-layer network, plays a crucial role in the 2nd stage. We appreciate it being highlighted for further analysis, as this allows us to demonstrate its significance. Below is a numerical comparison of PSNR for models with and without the Sub-Rot Network. Furthermore, we would like to emphasize that due to its shallow architecture, incorporating the Sub-Rot Network does not significantly impact training time. We decided to present the results on the jumpingjack dataset distinguishing between batch = {4, 8}. The experiments we performed using the RTX4090 GPU.

Since the Sub-Rot Network used is not deep and has only a single layer, it does not visibly affect the training time. Additionally, Reviewer 1 (u5NP) pointed us to consider the role of a number of Sub-Gaussian/Core-Gaussian. We agree that this is an interesting analysis, and we will incorporate it in the revised manuscripts.

Q1: A small tip: In Tab. 1, the authors used the metrics from the SC-GS paper. However, SC-GS did not use a consistent background; for example, the Bounce (and maybe Trex) scene used a white background. The authors clearly used a black background consistently. Although the reported metrics in the table show a slight decrease compared to SC-GS, I greatly appreciate the authors' honesty.

Thank you for raising this point. We believe it is important to use a unified framework for evaluation. Therefore, following most papers, we introduced results obtained on a black background.

We thank the Reviewer for the constructive remarks regarding our manuscript, to which we responded next to the posted questions below. We will revise the manuscript in accordance with the raised concerns.

W1 “I find the paper's writing style difficult to follow … I would appreciate a revision in writing to improve clarity.”

Thank you for this comment. We appreciate the complex nature of our manuscript, and we will edit all the relevant parts for clarity. In the paper, we introduced a Multi-Gaussian component, which is comprised of a Core-Gaussian "parent" and Sub-Gaussian "child," each parameterized by a triangle. Accordingly, the Core-Triangle corresponds to the Core-Gaussian, and the Sub-Triangle corresponds to the Sub-Gaussian. The terms "Triangle" or "Gaussian" are used contextually:

• When discussing rendering or properties such as color and transparency, we use "Gaussian."

• When referring to parameterized Gaussians and their modifications (e.g., location, scale, rotation), we use "Triangle."

Fig. 3 illustrates these differences. We will explain all these objects and clarify them in the Introduction.

Q1: Is there a strategy for selecting hyperparameters such as the number of Gaussian and Sub-Gaussian components?

Our model operates on several hyperparameters, primarily derived from the basic Gaussian Splatting (GS) framework. Given the introduction of additional stages and the multi-Gaussian component, one of the new hyperparameters is the number of Core-Gaussians and Sub-Gaussians. We are pleased to see this novel aspect being acknowledged. The number of Core-Gaussians is determined automatically in stage 1, utilizing the pruning mechanism implemented in GS. The Table below illustrates the training time corresponding to these parameters using the jumpingjacks dataset as an example. During the experiments, we used the RTX4090 GPU and densification until 5000 iterations (1st stage). We can see the speed of the 1st stage, which is used for Core-Gaussians preparation. Moreover, too few Core-Gaussians can cause a drop in quality (PSNR metric). The sub-rot network is shallow enough (it is a single layer in all of our experiments) that the increased cost of training time is not apparent. Therefore, the experiments suggest the number of Core-Gaussians is sufficient after the first phase. In all cases, a larger number of Sub-Gaussians improved the quality of renders.

Q2: It is unclear why the proposed method would perform worse than SC-GS on metrics like PSNR/SSIM for NeRF-DS and D-NeRF. Does this suggest that increased editability comes at the cost of model performance, or is it simply a matter of hyperparameter tuning?

Our main contribution is the ability to edit a dynamic scene better and more efficiently. As pointed out, there is generally a trade-off between editing and the quality of the reconstruction. However, in our model, such effects are minimal. Consequently, we obtained markedly worse results on synthetic datasets (Tab 1 in our draft paper) and better results achieved on full scenes (Tab. 3 and Tab. 4 in our draft paper). We tentatively believe this is caused by the suboptimal algorithm used to choose core points in SC-GS. SC-GS uses a heuristic approach based on distances to select core points, which might be problematic on a large scale. Our solution uses classical GS optimization procedures to establish the Gaussian number and size. Therefore, D-Miso covers 3D scenes better.

L1: The authors have discussed the limitations of their method, but I was unable to locate a statement regarding the broader impacts of the proposed method in the Limitations section, as claimed by the author.

Our model significantly improves rendering quality and advances 3D scene reconstruction and rendering, impacting multiple domains by enabling more realistic and efficient 3D modeling and animation. This technology could enhance VR/AR experiences [a1], robotics [a2], and medical imaging [a3]. It could also be used for interactive education [a3], scientific visualization, and a plethora of other commercial applications like product design and real estate [a4]. We will edit our manuscript to include these considerations and potential application fields of our method.

[a1] Linning et al. 2023, “VR-NeRF: High-Fidelity Virtualized Walkable Spaces”, SIGGRAPH Asia 2023 Conference Papers

[a2] Lisus et al., 2023, "Towards Open World NeRF-Based SLAM," 2023 20th Conference on Robots and Vision (CRV), Montreal, Canada, pp. 37-44, doi: 10.1109/CRV60082.2023.00013

[a3] Huang et al. 2013, “Exploring Learner Acceptance of the Use of Virtual Reality in Medical Education: A Case Study of Desktop and Projection-Based Display Systems,” Interactive Learning Environments 24 (1): 3–19.

[a4] Li et al. 2022, “Deep Learning of Cross-Modal Tasks for Conceptual Design of Engineered Products: A Review”, ASME 2022 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference