HiCoM: Hierarchical Coherent Motion for Dynamic Streamable Scenes with 3D Gaussian Splatting

Thank you for positive assessment of adding noise to 3D Gaussian representation learning, the effectiveness of our hierarchical coherent motion, and the simple deformation data structure to save memory. We now address the main concerns you have raised as follows.

1. The performance of HiCoM in reconstructing scenes with large motions.

As per your suggestion, we conducted experiments on the PanopticSports dataset. We set SH to 1 for fair comparison, and present results in the following table. Our HiCoM significantly surpasses the competitor 3DGStream in terms of PSNR, but slightly lower than Dynamic3DGS [3]. This is primarily because Dynamic3DGS utilizes segmentation masks as additional supervision, substantially boosts performance. According to Dynamic3DGS paper, removing segmentation masks leads to a significant performance drop (over 5dB), resulting in lower performance compared to our method. In the attached PDF, we present rendering results of all six scenes from one of the test viewpoints. The 3DGStream struggles with large movements; people and objects in the scene generally remain near their initial positions and gradually become invisible. We believe this is due to the limitations of its grid-based hash encoding and MLP, which cannot effectively handle high-frequency temporal motion information. In contrast, our HiCoM captures large movements of people and objects in the scene well, demonstrates sufficient flexibility, but there is still considerable room for improvement in reconstruction quality, especially for smaller moving objects.

2. Comparison with Particle-NeRF.

After carefully reviewing Particle-NeRF paper, we found it to be an excellent piece of work. It uses discrete particles for scene representation instead of the uniform grids used in previous NeRF work (InstantNGP). By optimizing particle positions and features through gradients, it achieves a similar effect to 3DGS, with the main difference being in the rendering pipeline. Particle-NeRF models motion through particle movement, focusing more on periodic, regular, and micro-level object motions. In contrast, our method targets general, scene-level dynamic changes. As we know, some Gaussian Splatting works [1, 2] also specialize in periodic and regular object motions. We believe Particle-NeRF might not be as effective in modeling the dynamic changes that our method addresses.

Despite this, we conducted experiments on the Particle-NeRF dataset, including five scenes, each with 40 viewpoints. We used 30 for training and the remaining 10 for testing. Please note that the Particle-NeRF paper mentions using only 20 training viewpoints and 10 testing viewpoints. As shown in the table below, our method achieves good reconstruction quality. Unfortunately, up to the time of submitting this response, we were unable to successfully run their open-source code, preventing us from conducting a more detailed comparison on the real-world dynamic scenes used in our paper.

3. Add more comparison figures and visualization videos.

Due to space limitations in the main manuscript, we have provided visual comparisons of all other scenes with the 3DGStream method in the supplementary materials. We have created videos to support our visualizations and are very willing to provide them. However, due to the review guidelines, we are unable to include external links here. We will consider including the comparison figures from the ablation studies in the revised manuscript and attach relevant visualization videos.

4. The introduction section is too long.

We appreciate the reviewer's feedback on the length and clarity of the introduction section. Our introduction in the manuscript spans from line 23 to line 95, totaling 72 lines, which is in line with the 70 lines recommended by the reviewer. Nevertheless, we will further refine this section in the final version to ensure it is as concise and precise as possible.

5. Integrate the 3D Gaussians into the initial 3DGS.

In our online reconstruction framework, we add some new Gaussian primitives to benefit the reconstruction and rendering of the current frame, but does not require computing their positions at $t=0$. Our primary objective is dynamic scene reconstruction and rendering, rather than object tracking.

We sincerely appreciate your detailed and constructive feedback. Your insights have been invaluable in refining our work, and we are committed to addressing your concerns to improve the overall quality of our paper. Thank you once again for your time and effort in reviewing our manuscript. We look forward to any further comments you may have.

[1] Yutao Feng,Xiang Feng,Yintong Shang, Ying Jiang, Chang Yu, Zeshun Zong,Tianjia Shao,Hongzhi Wu, Kun Zhou, Chenfanfu Jiang, et al. Gaussian splashing: Dynamic fluid synthesis with gaussian splatting. arXiv preprint arXiv:2401.15318, 2024.

[2] Licheng Zhong, Hong-Xing Yu, Jiajun Wu, and Yunzhu Li. Reconstruction and simulation of elastic objects with spring-mass 3d gaussians. In Proceedings of the European Conference on Computer Vision (ECCV), 2024.

[3] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In International Conference on 3D Vision (3DV), 2024.

Dear Reviewer,

Thank you for your comments. We would like to acknowledge that our method does indeed perform slightly worse than Dynamic3DGS on the two datasets you mentioned, where the appearance shows less variation and no dynamic addition or removal of content compared to the initial frame.

Beyond using masks, Dynamic3DGS includes the motion parameters for each Gaussian primitive at every time step, whereas our method allows Gaussians within a local region to share same motion parameters. While this may lead to a slight compromise in detail compared to Dynamic3DGS, it offers significant advantages in storage and training efficiency for online dynamic scene reconstruction.

We would like to further address your specific concerns as follows.

1. Why optimizing with masks doesn't improve the final PSNR of our HiCoM?

Masks represent the relative position of the foreground as a whole to the background, and they can be helpful in learning the overall motion of foreground. Our method already learns the overall foreground motion well without masks, demonstrated by the consistency between the rendered and the ground truth masks (will be included in the revised manuscript ). Therefore, using masks does not enhance the overall foreground motion learning. In contrast, Dynamic3DGS struggles with overall foreground motion without masks, highlights our method's strength in capturing large global motion.

While we effectively position the whole foreground, Dynamic3DGS’s ability to optimize each Gaussian’s position at different time steps leads to better internal detail and higher PSNR. However, we aim for our online dynamic reconstruction method to still perform well without masks, as in many cases, masks may not always be readily available. We believe that applying finer division to the foreground region in our framework could further improve the quality. While this is beyond the scope of the current work, it remains a promising direction for future research.

2. Performance comparison on the ParticleNeRF dataset.

The experimental results we presented show that our method performs better than ParticleNeRF, indicating that our approach is indeed capable of handling the uniform and periodic motions present in this synthetic dataset. The reviewer's expectation that our method should also surpass Dynamic3DGS may stem from a misunderstanding. As we previously explained, Dynamic3DGS benefits from its ability to optimize each Gaussian's position, which gives it an advantage, especially in these simpler synthetic scenarios.

We hope this clarification helps to address your concerns, and we sincerely appreciate your valuable feedback. We will make sure to thoroughly incorporate these clarifications into the revised version of our paper. Unfortunately, according to the review guidelines, we are unable to provide external links at this stage, but we would be happy to present the video results to you in a compliant manner at an appropriate time.

Thank you once again for your consideration.

Best regards,

The Authors

Dear authors, thanks for your reply. My concerns still exist.

1. As the authors addressed, if HiCoM can optimized with masks why it doesn't improve the final PSNR? What's the reason? I don't think it would demonstrate HiCoM can achieve better foreground rendering quality compared with Dynamic 3DGS.

2. As the author said, Particle-NeRF shows its advantages in uniform motions, but I think it is the subset of real-world dynamics. I think a dynamic representation can handle the complex motions in real-world scenes, it can also handle uniform and periodic motion. Dynamic 3D Gaussians also demonstrate that it would achieve good results in both particleNeRF dataset and its used Sports dataset.

Therefore, I think HiCoM should beat dynamic3DGS and particleNeRF in both synthetic animation datasets and panopticSports datasets, if evaluation metrics cannot demonstrate that, a video or figure is also convincible.

We greatly appreciate your positive feedback on the clarity and straightforward nature of our method. We are particularly encouraged by your recognition of the effectiveness of our perturbation smoothing strategy, as well as your appreciation of the simplicity. Your acknowledgment of the thoroughness of our ablation study is also highly valued. We have carefully reviewed your questions and concerns and address them in detail below.

1. The difference between our continual refinement and 3DGStream.

Our continual refinement may appear similar to the second stage of 3DGStream in form, but it fundamentally differs in approach. 3DGStream continuously applies learned motion to the Gaussian representation derived from the first frame, with newly added Gaussian primitives only used for rendering the current frame and not passed to subsequent frames. In contrast, our method merges new Gaussian primitives into the initial Gaussian representation, then removes the same number of less important Gaussian primitives based on importance metrics such as the opacity we used in this work or other indicators. If an object does not exist in the first frame, 3DGStream adds Gaussian primitives in every subsequent frame to fit the object, while our method can pass these Gaussian primitives to the next frame, eliminating the need to refit the object. We believe our design is superior and will clearly state this difference in the revised manuscript to reduce any potential confusion for readers.

2. The uniformly dividing the scene seems redudant.

Thank you for highlighting this issue. However, our method demonstrates that uniformly dividing the scene is simple and effective. We agree that adaptively dividing the scene based on the granularity of motion could further improve the motion representation: regions with minimal motion could be coarsely divided, while areas with complex motion could be finely partitioned. Based on your suggestion, we will explore more effective scene division methods in future work.

3. Incorporate the temporal dimension into motion.

Our online reconstruction framework focuses on representing and learning changes between adjacent frames in general real-world scenes, which keeps our method relatively simple but has proven effective in experiments. We agree that embedding the motion in space alone may have limited modeling ability, especially for complex movements such as intricate transformations within a small region. We acknowledge that some existing works specifically address such periodic motion patterns [1, 2]. Future work could enhance our framework by incorporating temporal information from observed frames to construct more comprehensive motion patterns.

4. Comparison with offline reconstruction methods.

We included data reported by state-of-the-art offline dynamic scene reconstruction methods in the supplementary materials. Specifically, Table 8 in the Appendix shows that our method is comparable in reconstruction quality across nine scenes from two datasets. We will highlight these comparisons more prominently and incorporate additional recent works in our revised manuscript.

5. The cost of initialization from SfM.

The cost associated with SfM does not significantly affect the online reconstruction workflow. In dynamic scene reconstruction, the viewpoints are typically sparse, meaning that SfM needs to be applied to only a limited number of views from the initial frame. Generating the initial point cloud usually takes only a few minutes. In practical scenarios, camera positions for video capture are often pre-arranged, allowing the majority of the SfM work to be completed before the reconstruction process begins.

6. The influence of the initial frame learning.

Our online framework reconstructs the scene’s fundamental geometry and appearance through initial frame learning, and assumes that most objects in the dynamic scene will not undergo significant appearance changes. Subsequent learning focuses on representing and learning the motion of objects from the initial frame, adding a few new Gaussian primitives to handle previously unseen content and to improve motion learning. In contrast, if the initial scene's geometry and appearance are poorly reconstructed, it increases the burden on subsequent learning, leading to slower convergence and degraded reconstruction quality, which can accumulate over time.

We are grateful for your constructive feedback, which has been instrumental in enhancing the quality of our manuscript. If our clarifications have addressed your concerns, we kindly ask you to reconsider your evaluation of our manuscript. Should you have any further questions or require additional clarifications, we are happy to provide further details. Thank you once again for your time and consideration.

[1] YutaoFeng,XiangFeng,YintongShang,YingJiang,ChangYu,ZeshunZong,TianjiaShao,HongzhiWu, Kun Zhou, Chenfanfu Jiang, et al. Gaussian splashing: Dynamic fluid synthesis with gaussian splatting. arXiv preprint arXiv:2401.15318, 2024.

[2] Licheng Zhong, Hong-Xing Yu, Jiajun Wu, and Yunzhu Li. Reconstruction and simulation of elastic objects with spring-mass 3d gaussians. In Proceedings of the European Conference on Computer Vision (ECCV), 2024.

Dear Reviewer QKxo,

I hope you’re doing well.

We understand you may have a busy schedule and might not have had the chance to review our rebuttal yet. We hope that our responses have adequately addressed your concerns. If you have any further questions or need additional information, please feel free to let us know.

Thank you again for your time and effort in reviewing our submission.

Best regards,

The Authors of Submission 809

Thanks for authors' reply and effort. I have read the rebuttal and discussion with other reviewers. Most of my concerns are addressed. However, for the comparison with offline reconstruction methods, I hope the author can clearly present the comparison of methods in a table. Table 8 of the supplementary materials mentioned by the author only shows the results of offline methods, which made me spend more effort to compare the results of this method again. It is acceptable that offline methods perform better than online methods because they invest more time and even storage to optimize.

Besides, I have one more concern. I found the reported result for 3DGStream is much lower than its original paper, especially for the meet room dataset (For N3DV, the 3DGStream reports 31.67 PSNR while the authors' reimplemented verision reports 30.15 PSNR. For meet room, the 3DGStream reports 30.79 PSNR while the author's reimplemented verision reports 25.96 PSNR). I really appreciate authors' effort for reimplementation and I understand that the open-source code may not have been systematically maintained. However, such performance differences might make it difficult to conduct a fair comparison in the future works, as two works placed different results on the same dataset. Has the author tried contacting the authors of 3DGStream? I am not sure if more discussion is needed at this point. It is just my concern.

Dear Reviewer,

Thank you for your detailed comments. We greatly appreciate your insights.

We agree with your suggestion and will follow your advice to present the results of all methods in a single table, making it easier for readers to compare and analyze.

Regarding the discrepancy in the results of 3DGStream, we consulted with the 3DGStream authors before they released the code. They confirmed that they did not use the original camera poses provided by the dataset and applied distortion correction to the views. During this work, we followed the data processing practices used by 4D-GS (CVPR 2024, in Table 8) and several previous works using this dataset, including the StreamRF (the paper that originally proposed the Meet Room dataset) in our Table 1, all of which utilize the original camera poses. Additionally, we noted in the Appendix that the processed views in 3DGStream contain approximately 50,000 fewer pixels than the original views. However, from our observations of the N3DV dataset, we did not notice significant distortion. Therefore, to ensure a fair comparison, we chose to remain consistent with the practices used by previous methods, using the original view images.

After submitting our paper, we also conducted experiments using the undistorted dataset following the open-source code provided by the 3DGStream authors. On the Meet Room dataset, the average PSNR was 28.32 dB, which is significantly lower than the 30.79 dB reported in their paper. Meanwhile, our HiCoM achieved a PSNR of 31.02 dB on the undistorted dataset. Following your suggestion, we will include these results in the paper to provide a more comprehensive comparison.

Thank you once again for your thoughtful feedback. If you have any further questions or concerns, please feel free to reach out, and we will do our best to address them. We will also carefully incorporate the discussions with you and other reviewers into the revised manuscript.

Best regards,

The Authors

Dear Authors/Reviewers,

I am Jiakai Sun, the first author of the 3DGStream paper. I am delighted to see that this paper on OpenReview recognizes our work as the previous state-of-the-art in this field and has made efforts to re-implementation it.

I would like to clarify a point: the results reported in our original paper are specifically based on the "discussion" scene in the MeetRoom dataset. This is consistent with the settings for StreamRF, as confirmed by the authors of StreamRF (https://github.com/AlgoHunt/StreamRF/issues/13). However, the authors of HiCoM appear to have misunderstood this and assumed that our reported results represent the average across all three scenes. This misunderstanding led to the claim in their response:

On the MeetRoom dataset, the average PSNR was 28.32 dB, which is significantly lower than the 30.79 dB reported in their paper.".

However, as can be seen in the Table7 in the OpenReview version of HiCoM , the result is 30.06 dB when they use the open-source code provided by the us, which is much closer to the metric we reported in our paper.

Since OpenReview serves as an open platform for paper reviews, I believe it is important to leave this clarification here to address potential confusion for future researchers who may encounter this discussion.

Thank you.

Dear AC,

Thank you for your prompt response. We have reached an agreement with the authors of HiCoM on how to update the instructions in our open-source repository's README, as well as enhance the descriptions in their preprint, potential revisions, and open-source code, to provide clearer context for future researchers.

Regarding the scores on MeetRoom, after confirming with the authors of StreamRF (prior to the CVPR2024 submission), we only reported the results for the discussion scene in our paper to ensure a fair comparison. The authors of HiCoM will also conduct a secondary confirmation with the authors of StreamRF. We believe that the metrics reported by the authors of HiCoM in Table 7 using our open-source code on the undistorted images (discussion: 30.06, trimming: 28.82, VRHeadset: 29.26) are credible and can be adopted by future work. If StreamRF confirms the use of an average across multiple scenes, we will update our draft to provide both the average scores and scores for each individual scene.

My comment here is to clarify and prevent potential misunderstandings about our work arising from this statement in OpenReview:

On the MeetRoom dataset, the average PSNR was 28.32 dB, which is significantly lower than the 30.79 dB reported in their paper.

I believe that our discussion here, along with planned updates to our (i.e., both 3DGStream and HiCoM) open-source repositories and papers will resolve this issue and provide clearer background information for future researchers.

Thank you again for your attention to this matter.

Best regards,

Jiakai Sun

Thank you for recognizing the strengths of our work, particularly the perturbation smoothing strategy, the exceptional performance on N3DV and Meet Room datasets, and the quality of our writing. In the following, we address each of your comments and questions in detail.

1. The impact of motion levels on performance.

We need to clarify the meaning of motion levels in Table 5. Our method uses three motion levels by default (the 3$^{rd}$ row), which we refer to as "fine," "medium," and "coarse", respectively. When motion levels is set to 1 (the 1$^{st}$ row), only the "fine" level is used. When set to 2 (the 2$^{nd}$ row), both "fine" and "medium" levels are applied. As the table shows, using only the "fine" level already achieves good performance. Adding the "medium" and "coarse" levels provides additional performance gains, e.g., the PSNR of Coffee Martini and Discussion scenes respectively boosts 0.23 dB and 0.59 dB, but the improvements diminish as the number of levels increases. This indicates that our HiCoM is effective and a few motion levels are sufficient to achieve good results. We will include this clarification in the revised manuscript and add experimental results for single-level motion (present in the table below) to provide a more comprehensive view.

2. The dynamic foreground and static background fall into the same voxel.

Your observation is indeed insightful and highlights an important consideration in complex scenes. However, Gaussian primitives are not points but ellipsoids, representing small regions in space. We observed that Gaussian primitives for moving objects are generally smaller and densely packed, whereas those for background regions are usually larger and sparsely distributed. As long as the finest motion level's coverage area is smaller than the radius of background Gaussian primitives, this issue can be substantially mitigated. Additionally, after the motion learning, we will add some new Gaussian primitives, which can further help in correcting inadequately reconstructed regions. Future work could investigate methods to distinguish dynamic objects from the static background, such as filtering based on the size of Gaussian primitive.

3. The absence of error bars.

We performed each experiment three times using the default random seed, but result variations were minimal. Given the number of metrics we reported, there was limited space in the tables. Additionally, some of the data were directly referenced from other papers that did not include standard deviations, so we omitted them for consistency. Considering your suggestion, we will include all avaliable standard deviations for PSNR metrics (as shown in the table below) in Tables 6 and 7 in our revised manuscript.

4. The necessity of online reconstruction.

Most methods for dynamic scene reconstruction work in an offline manner, but these methods may suffer from low storage efficiency, high resource demands, and insufficient real-time response for long-duration or real-time interactive dynamic scenes. For example, for a dynamic scene lasting 2 hours, methods using neural networks to model scene motion may require very large neural networks, substantial GPU memory, and may have slow convergence rates. The goal of online reconstruction is to mitigate these issues while achieving competitive reconstruction quality. Online reconstruction does not need to wait for all video capture to complete, and the cost for pose estimation is negligible. Real-time high-fidelity dynamic scene reconstruction would be valuable for modern applications such as AR/VR, free-viewpoint video streaming, and online meeting.

5. The perturbation noise present in Equation 6.

We would like to clarify that Equation 6 also includes a coefficient $\lambda_{noise}$, which adjusts the noise intensity. This coefficient allows us to control the perturbation magnitude, ensuring it is appropriate for the specific context of the reconstruction.

Thank you once again for your valuable feedback and insightful comments. We hope that our responses have addressed your concerns and clarified the points raised in your review. If you have any further questions or require additional clarifications, please feel free to leave a comment. We remain committed to providing any further information you may need.

Dear reviewer oHau,

We received diverse reviews for this submission, and you were initially negative. It would be extremely helpful if you could review the rebuttal and participate in the discussion over the next two days. Thank you very much!

• Your AC