ReCon-GS: Continuum-Preserved Guassian Streaming for Fast and Compact Reconstruction of Dynamic Scenes

We sincerely appreciate your recognition of our Adaptive Hierarchical Motion Representation framework design and particularly value your acknowledgment of the comprehensive experimental validation demonstrating superior performance over existing streaming frameworks. We will now systematically address each of your questions and concerns.

1. The Visualization of hierarchical Anchor Gaussian & The effectiveness of Anchor Guassian.

While NeurIPS rebuttal guidelines prevent us from providing visualizations of hierarchical anchor Gaussian, we will include this visualization in the supplemental appendix of our revised version to demonstrate the efficacy of Anchor Gaussians. Moreover, we have included comprehensive quantitative evidence to prove the effectiveness of Anchor Gaussians in Table #1. The data demonstrates that compared to HiCoM[1] (the current open-source SOTA in streaming frameworks), ReCon-GS generates a comparable number of initial Gaussians in the first frame but reduces subsequent incremental Gaussian generation by approximately 60%. This significant reduction indicates that ReCon-GS achieves high-quality scene reconstruction while requiring only minimal Gaussian increments (approximately 300 per frame). These metrics strongly validate the efficacy of our Adaptive Hierarchical Motion Representation in enabling efficient motion modeling.

Table #1: Statistics of 3DGS Quantities at Different Stages. (Averaged Over 3 Training Results)

2. More Quantitative Results Compared with Recent SOTA Offline Methods.

Based on your suggestions, we have compared the performance of ReCon-GS with STG[2] and SplineGS[3]. After reviewing their papers, we have also noticed the inconsistency in the benchmarks in the performance comparison between STG and SplineGS: the performance of STG reported by SplineGS (32.57dB) is higher than that in the STG paper (32.05dB). Since SplineGS is not currently open-sourced, we will directly use the performance reported in the paper to ensure fairness.

Table #2 demonstrates ReCon-GS's state-of-the-art (SOTA) performance in rendering quality, storage efficiency, training time, and FPS even when compared to recent offline methods. This superiority stems primarily from the Adaptive Hierarchical Motion Representation's efficient motion modeling capability. Offline methods use global MLPs to represent motion across space and time. This creates compact motion encoding, but storage limits hurt their generalization ability. Swift4D (ICLR25)[4] addresses this problem through static-dynamic 3D gaussian decoupling, allocating more storage to dynamic Gaussian motion representation. However, camera-related issues may introduce noticeable artifacts in static regions[5], compromising static scene reconstruction. Moreover, as training datasets trend toward higher frame rates, streaming frameworks have shifted optimization objectives from temporally global to temporally local deformation field refinement, emerging as a more effective paradigm. Recently, the fundamental challenge in streaming frameworks remains temporal error accumulation. ReCon-GS overcomes this via its novel Dynamic Reconfiguration Strategy, which preserves motion representation fidelity through periodic anchor Gaussian redistribution. This constitutes the underlying reason for ReCon-GS's performance superiority over current SOTA methods.

Table #2: Compare with recent offline baselines.

3. Some tables lack proper units for storage and training time.

We appreciate your suggestion. Providing more detailed units will make our manuscript clearer for readers, and we will add unit information in the tables in the subsequent revised version.

4. After replacing the explicit motion representation with MLP, can the performance be further improved?

This is an excellent advice. Indeed, many current online and offline methods use MLP for implicit representation of inter-frame motion. However, for the online framework, implicit representation will bring additional issues:

• The storage of MLP parameters. Without changing the number of anchor Gaussians, the introduction of MLP will consume more storage compared to the explicit motion representation of ReCon-GS.
• Extremely limited quality improvement. When ReCon-GS needs to sparsify anchor Gaussians due to storage constraints, the MLP-based motion representation method may lead to a huge decline in rendering quality.

Thank you again for your valuable feedback. We hope our responses meet your expectations. If there are any further questions or concerns, please do not hesitate to let us know.

Reference:

[1]Qiankun Gao, Jiarui Meng, Chengxiang Wen, Jie Chen, and Jian Zhang. "Hicom: Hierarchical coherent motion for dynamic streamable scenes with 3d gaussian splatting." In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2024.

[2]Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. "Spacetime gaussian feature splatting for real-time dynamic view synthesis." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

[3]Jihwan Yoon, Sangbeom Han, Jaeseok Oh, and Minsik Lee. "SplineGS: Learning Smooth Trajectories in Gaussian Splatting for Dynamic Scene Reconstruction." In The Thirteenth International Conference on Learning Representations (ICLR), 2025.

[4]Jiahao Wu, Rui Peng, Zhiyan Wang, Lu Xiao, Luyang Tang, Jinbo Yan, Kaiqiang Xiong, and Ronggang Wang. Swift4D: Adaptive divide-and-conquer Gaussian Splatting for compact and efficient reconstruction of dynamic scene. In International Conference on Learning Representations (ICLR), 2025.

[5]Youngsik Yun, Jeongmin Bae, Hyunseung Son, Seoha Kim, Hahyun Lee, Gun Bang, and Youngjung Uh. "Compensating Spatiotemporally Inconsistent Observations for Online Dynamic 3D Gaussian Splatting." In Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers (SIGGRAPH), 2025.

We sincerely appreciate your recognition of ReCon-GS, particularly regarding how its adaptive hierarchical motion representation framework and Dynamic Reconfiguration Strategy effectively achieve efficient and compact motion representation while resolving error accumulation caused by anchor Gaussian drift. Moving forward, we will systematically address each of your concerns and questions.

1. Regarding storage requirements, rendering quality, and training time compared to offline methods.

We acknowledge the critical importance of these factors in determining ReCon-GS's practical applicability. It should be noted that through its Storage-aware fidelity optimization objective, ReCon-GS enables storage regulation by adjusting the density of anchor Gaussians in its Adaptive Hierarchical Motion Representation framework while maintaining efficient motion modeling. As evidenced in Table #1, ReCon-GS achieves tunable quality-storage trade-offs ranging from 0.14MB/Frame to 0.63MB/Frame across varying anchor Gaussian densities. Crucially, at a 1/64 Anchor Guassian density, ReCon-GS maintains an average storage cost of 0.19MB/Frame – not only lower than offline methods (4DGS[1], STG[2], Swift4D[3]) but simultaneously achieving state-of-the-art (SOTA) rendering quality among all approaches. For training time, online frameworks inherently face disadvantages compared to offline paradigms due to computational constraints, representing an area for our future improvement. However, in real-time 4D reconstruction scenarios, ReCon-GS's online framework consequently offers greater practical utility than offline alternatives.

Table #1: Quantitative Comparison Across Varying Anchor Gaussian Density Levels. Storage values represent per-frame averages inclusive of the initial frame.

2. How often the dynamic hierarchy reconfiguration is triggered during training?

ReCon-GS employs a fixed 4-frame interval for reconfiguration. This design stems from the observation that reconstruction quality inherently fluctuates in streaming frameworks. Consequently, quality-based dynamic adaptation proves ineffective for this issue since quality variations within 3-5 frame intervals do not reliably indicate diminished motion representation capacity of anchor Gaussians. The 4-frame reconfiguration interval represents the optimal configuration determined through extensive experimentation.

3. How Anchor Gaussians quantitatively govern or parameterize general Gaussian deformation?

We adopt a simple yet effective paradigm where each General Gaussian directly inherits motion parameters from its associated Anchor Gaussian. Following Dynamic Reconfiguration Strategy, which redistributes Anchor Gaussians, General Gaussians are reassigned to new anchors via L1-distance-based association. After reassignment, General Gaussians subsequently inherit motion transformations from their newly assigned Anchor Gaussians during deformation field training (Stage 1).

4. Removing the Hierarchical Motion Representation leads to lower storage.

In Table #2, Our storage analysis reveals that removing hierarchical motion representation significantly reduces motion field storage requirements. However, this comes at the cost of diminished motion representation capacity, necessitating increased incremental 3DG generation to compensate for detail loss. Crucially, by implementing spherical harmonics at degree=1 for 3D Gaussians, the per-Gaussian storage footprint remains minimal, preventing substantial memory overhead despite increased Gaussian counts.

Table #2: The ablation study on storage.

5. The sensitivity to initial conditions when the first frame poorly captures the scene.

Your suggestion is profoundly insightful. Streaming frameworks indeed face catastrophic error accumulation when encountering poor quality of initial frame reconstruction. To demonstrate the superiority of ReCon-GS's motion representation framework, we conducted experiments under varying initial frame training iterations.

Table #3 reports ReCon-GS performance under varying initial frame training iterations. At 2,500 iterations, suboptimal initial reconstruction quality and sparse scene representation result in the steepest performance degradation slope. When trained for 5,000 iterations, quality degradation is substantially mitigated. Extending to 10,000 iterations yields converged initial frame training, further optimizing quality decay while achieving state-of-the-art (SOTA) performance. Responding to your insight, we will integrate this analysis in the future revised version to demonstrate the critical impact of initial frame quality on streaming frameworks while further validating ReCon-GS's superior robustness against error accumulation.

Table 3: The ablation study on Training Steps of Frame 0.

6. Is there a explicit motion relation between anchor gaussians across different level?

No, We deliberately avoid imposing hierarchical coarse-to-fine constraints on Anchor Gaussians because non-tree-structured organization enhances motion representation efficacy. This design accounts for subtle quasi-rigid motions occurring at rigid body junctions, which would be inadequately represented under tree-structured partitioning. Enforcing tree-structured partitioning would consequently compromise motion encoding efficiency.

We are grateful for your constructive feedback, which has been instrumental in enhancing the quality of our manuscript. Thank you once again for your time and consideration.

Reference:

[1]Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

[2]Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. "Spacetime gaussian feature splatting for real-time dynamic view synthesis." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

[3]Jiahao Wu, Rui Peng, Zhiyan Wang, Lu Xiao, Luyang Tang, Jinbo Yan, Kaiqiang Xiong, and Ronggang Wang. Swift4D: Adaptive divide-and-conquer Gaussian Splatting for compact and efficient reconstruction of dynamic scene. In International Conference on Learning Representations (ICLR), 2025.

[4]Qiankun Gao, Jiarui Meng, Chengxiang Wen, Jie Chen, and Jian Zhang. "Hicom: Hierarchical coherent motion for dynamic streamable scenes with 3d gaussian splatting." In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2024.

Dear Reviewer pBHc,

The authors have provided addition data in response to your questions. What is your view after seeing this additional information? It would be good if you could actively engage in discussions with the authors during the discussion phase ASAP, which ends on EoA (Aug 6).

Best, AC

We appreciate your recognition of the strengths of our work, particularly the achievement of state-of-the-art (SOTA) performance using a straightforward loss function during training, as well as your positive feedback on the quality of our writing. We will systematically address each of your concerns and questions.

1. The proposed method shares similarity with Scaffold-GS and Octree-GS.

ReCon-GS demonstrates fundamental distinctions from Scaffold-GS[1] and Octree-GS[2]. First, Scaffold-GS and Octree-GS target static 3D reconstruction, while ReCon-GS addresses dynamic 4D reconstruction, resulting in divergent frameworks and optimization objectives. Moreover, Scaffold-GS embeds Gaussian attributes via uniformly sampled voxel-grid anchors for compact scene representation. Octree-GS achieves multi-resolution compression through Level-Of-Detail (LOD) hierarchies. Both prioritize spatial compactness. In contrast, ReCon-GS employs adaptively sampled Anchor Gaussians that capture local motion characteristics. Its hierarchical structure compactly encodes inter-frame motion rather than spatial features. Consequently, ReCon-GS’s Adaptive Hierarchical Motion Representation fundamentally differs from these static approaches.

2. Include perceptual metrics in other experiments.

Your concern is indeed well-founded, as LPIPS provides a more comprehensive assessment of ReCon-GS's capabilities. As presented in Table #1, LPIPS(VGG) metrics comparing ReCon-GS against HiCoM[3] and 3DGStream[4] across both datasets are now included. We observe that ReCon-GS achieves state-of-the-art (SoTA) performance on this perceptual metric among streaming frameworks. Considering your suggestion, we will include the LPIPS metric for the MeetRoom and PanopticSports datasets in our revised manuscript.

Table #1: The Perceptual Quality Comparison on Meet Room and PanopticSports datasets (Averaged Over 3 Training Results)

3. Does the reported training time include the first-frame reconstruction?

We sincerely apologize for not explicitly specifying the methodology in the table caption. The time metrics reported in our tables include the first-frame training duration. Our first-frame training time typically ranges between 300 and 400 seconds, resulting in a negligible impact on the overall reconstruction process. To more clearly demonstrate our method's advantage in low training cost, Table #2 presents comparisons on the N3DV dataset, showing that ReCon-GS requires significantly less time than both HiCoM and 3DGStream for both first-frame training and average time per frames.

Table #2: Training Time Comparison on N3DV (Averaged Over 3 Training Results)

4. How many Gaussian primitives are generated in total?

Tables #3 and Table 6 in the manuscript demonstrate that ReCon-GS generates approximately 244k Gaussians in the first frame, which is comparable to HiCoM. However, due to ReCon-GS's implementation of spherical harmonics (SH) at degree=1, its actual first-frame storage occupies only 20% of HiCoM's requirement. During subsequent frame reconstruction, ReCon-GS produces approximately 6.4k incremental Gaussians, merely 40% of HiCoM's corresponding count. This evidences ReCon-GS's superior motion representation capability, enabling high-quality scene reconstruction through densification while requiring significantly fewer Gaussians.

Table #3: Statistics of 3DGS Quantities at Different Stages (Averaged Over 3 Training Results)

Thank you once again for your valuable feedback and insightful comments. We hope that our responses have addressed your concerns and clarified the points raised in your review.

Reference:

[1]Tao Lu, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai. "Scaffold-gs: Structured 3d gaussians for view-adaptive rendering." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.

[2]Kerui Ren, Lihan Jiang, Tao Lu, Mulin Yu, Linning Xu, Zhangkai Ni, and Bo Dai. "Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D Gaussians." IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.

[3]Qiankun Gao, Jiarui Meng, Chengxiang Wen, Jie Chen, and Jian Zhang. "Hicom: Hierarchical coherent motion for dynamic streamable scenes with 3d gaussian splatting." In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2024.

[4]Jiakai Sun, Han Jiao, Guangyuan Li, Zhanjie Zhang, Lei Zhao, and Wei Xing. 3dgstream: On-the-fly training of 3d gaussians for efficient streaming of photo-realistic free-viewpoint videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

We sincerely appreciate your recognition of ReCon-GS's performance. We will now systematically address the questions you have raised.

1. The meaning of "Continuum" in the title.

The term "Continuum" highlight ReCon-GS's characteristics. Through our Adaptive Hierarchical Motion Representation and Dynamic Reconfiguration Strategy, the framework effectively prevents progressive geometric distortion in streaming reconstruction – a capability conclusively demonstrated in Figure 7. This preservation of geometric consistency across frames fundamentally realizes a true continuum of scene representation, justifying the terminology.

2. How essential is Intra-Hierarchical Deformation Inheritance to achieving performance gains during Re-Hierarchization, considering deformation fields undergo per-frame retraining in Stage 1?

Given that inter-frame motion typically exhibits inertia, ReCon-GS initializes the motion state of the current frame’s Anchor Gaussians using that of the previous frame’s Anchor Gaussians. However, sustained displacements reduce Anchor Gaussians' capacity to represent quasi-rigid motion. This necessitates reassigning anchors through Re-Hierarchization. After reassignment, directly inheriting motion from old to new Anchor Gaussians would compromise temporal motion consistency. To address this, as detailed in Section 4.3, we proposes Intra-Hierarchical Deformation Inheritance. This mechanism enables new anchors to precisely inherit local motion inertia by uniformly acquiring deformation traits from three legacy anchors, which enables local motion inertia inheritance for new anchors while preserving temporal consistency.

To thoroughly validate the efficacy of this mechanism, we conducted an ablation study. As shown in Table #1, removing Intra-hierarchical Deformation Inheritance leads to a performance deterioration, accompanied by a pronounced increase in the rate of performance decline. This result confirms the critical role of this mechanism in maintaining motion prediction accuracy and temporal consistency. To enhance readers' understanding of ReCon-GS's validity, we will add this ablation experiment in the revised version based on your feedback.

Table #1: Ablation on Intra-hierarchical Deformation Inheritance.

3. Densification is per-frame or per-reconfiguration?

Densification operates on a per-frame scheme. Although this incurs a modest training cost, the frame-by-frame densification promptly compensates for limitations in ReCon-GS's quasi-rigid motion representation regarding local detail reconstruction.

4. Quantitative Result Comparison on other Datasets.

Based on your suggestion, we conducted supplementary quantitative evaluation on the Technicolor dataset[1].

As demonstrated in Table #2, our method achieves state-of-the-art (SOTA) performance across rendering quality, storage efficiency, and rendering speed metrics. Notably, compared to streaming approaches, ReCon-GS delivers approximately 0.5 dB higher rendering quality while reducing storage by over 40% and significantly accelerating rendering speed. Against offline methods, ReCon-GS attains SOTA rendering quality while achieving more than 25% storage reduction.

Table #2: Quantitative Comparison on Technicolor Light Field dataset.

5. How our method achieves more compact deformation data representation than HiCoM?

Unlike HiCoM, our method doesn't require indexing every deformation field parameter for preservation. ReCon-GS effectively achieve this through a geometrically ordered storage scheme: Our anchor Gaussians, sampled via grid-based Farthest Point Sampling (FPS), undergo motion data serialization relative to the world coordinate, where parameters are stored sequentially from the farthest anchor point from the origin inward along a predetermined spatial axis. This coordinate-based ordering inherently eliminates redundant index storage while ensuring consistent data retrieval, making ReCon-GS significantly more compact than HiCoM.

Thank you once again for your time and effort in reviewing our manuscript. We look forward to any further comments you may have.

Reference:

[1]Neus Sabater, Guillaume Boisson, Benoit Vandame, et al. Dataset and pipeline for multi-view light-field video. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition Workshops (CVPR workshops), 2017.

[2]Benjamin Attal, Jia-Bin Huang, Christian Richardt, Michael Zollhoefer, Johannes Kopf, Matthew O’Toole, and Changil Kim. HyperReel: High-fidelity 6-DoF video with ray-conditioned sampling. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition (CVPR), 2023.

[3]Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. "Spacetime gaussian feature splatting for real-time dynamic view synthesis." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

[4]Junoh Lee, ChangYeon Won, Hyunjun Jung, Inhwan Bae, and Hae-Gon Jeon. "Fully explicit dynamic gaussian splatting." In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2024.

[5]Jeongmin Bae, Seoha Kim, Youngsik Yun, Hahyun Lee, Gun Bang, and Youngjung Uh. "Per-gaussian embedding-based deformation for deformable 3d gaussian splatting." In Proceedings of the European Conference on Computer Vision (ECCV), 2024.

Dear Reviewer 43Ni,

The authors have provided addition data in response to your questions. What is your view after seeing this additional information? It would be good if you could actively engage in discussions with the authors during the discussion phase ASAP, which ends on EoA (Aug 6).

Best, AC