S4D: Streaming 4D Real-World Reconstruction with Gaussians and 3D Control Points

1. The motion relys on the estimation of optical flow. I am not sure how robust this method is to errors in optical flow estimation.

2. As authors metioned, the method is strongly relied on the initialized reconstruction and does not densify Gaussians in the subsequent frames. It can not handle the newly emerging objects or occlusion regions.

3. The lack of comparison to the existing methods. SC-GS[1] and SuperPoint-GS[2] share the similar idea with this paper, which use the sparse control points to model the motion. I think the comparison and discussion to these methods are neccesary.

[1] Huang Y H, Sun Y T, Yang Z, et al. Sc-gs: Sparse-controlled gaussian splatting for editable dynamic scenes[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 4220-4230.

[2] Wan D, Lu R, Zeng G. Superpoint Gaussian Splatting for Real-Time High-Fidelity Dynamic Scene Reconstruction[C]//Forty-first International Conference on Machine Learning.

Overall, the paper's writing can be further improved, with more concise sentences and better organization. The paper focuses on achieving streamable GS renders in contrast to the setups in the 4DGS line of work, but this point is not clearly stated in the introduction or abstract. The paper currently has an overall presentation of a technical report. It can be further improved if the technical merits, like the motion decoupling using local rays, are highlighted.

Although the system achieves better performance compared to Dynamic-GS, it uses more priors like semantic segmentation masks and optical flow, leading to a longer processing time in practice if accounting for those from the priors. Since processing and convergence time are crucial factors for streaming applications, the paper would be improved if the processing time of these priors were reported and accounted for in the time calculation. Further, the reliance on the priors can be further studied, eg. the behavior of the system if the SAM masks are producing inaccurate segmentations.

• Because the system processes the frame one by one, later information or observations cannot be fused into previous reconstruciton, which may lead to two possible situations, either the system depends on super good initialization from a multiview camera setup, or the earlier reconstruction will have larger error, and such error will accumulate as the online reconstruction goes to later frames. There is no evidence how the proposed method deals with such accumulated error when streaming to very long video, and how robust it is if one frame in the middle fail.
• Occlusion: the key challenge of 4D reconstruction is not what is observed (visible optical flow), instead, is what is temporally unobserved, it’s not clear that how the proposed control point construction method will behave when the object is largely occluded, this is not verified in the experiment because the dataset used in the experiments are very easy and widely studied ones.
• The experiments comparison is a little limited to only GS based methods, and there is no clear argument why the visual results are improved because of any novel design proposed in the paper.
• The writing and presentation is a little wired, in the method part, there is a very long overview where I mistake as the main method paragraphs at the beginning, also the English writing sounds a little over-polished by LLM.
• It’s also not clear that how robust the Multiview VOS is and how it affects the performance of the system, because if not monitored by human, the segmentation mask may have small noise or miss segmentation in real world videos. Also, it’s unclear how to make consistent segmentation level/granularity
• The related works is does not fully cover the most recent advances of 4D reconstruction, many recent papers are not included.

I watched the video first, and then I glimpsed through the experimental results. I was very impressed and eager to read the methodology of the paper. While the main diagram in Fig. 1 was clear, the description of the individual components was very confusing.

W1. The main innovation of the method (and based on the ablation having a large effect) is the control points. However, at no point in the paper, and not even in 3.3, is there any definition of how a control point is defined. There is a description of what is done with the control points but not what is a control point and what is not a control point.

W2. Dynamic reconstruction from multiple static views needs constraints for regularization: either a reduction to a canonical space or a prior for trajectories in terms of a neural prior or a trajectory basis. In this paper, it is not clear whether such a regularization comes from the segmentation and the object level manipulation or from the residual compensation.

W3: Line 211 and Appendix 1. It is a well-known fact that when the FOV is small the projection can be approximated with a scaled orthographic projection. The authors' calculation has to be set in the right textbook context. The alignment of the optical axis with the central ray of the neighborhood is a well-known fact in the approximation of small object projection with scaled orthograph (see Hartley and Zisserman, for example)

W4: The paper does not have a flow on the order of computation of things. If one tries to go backward from (10) , we have scenes $\mathbb G_t$ that are propagated with a function $\mathcal F_{\mathbb C_t}$ that is nowhere defined. We guess that this is somehow related to (t,q) in (6). Then, there is no formula for computing (t_i,q_i) for control points except an angular velocity formula in (5) or linear velocity in (3).

W5: The computation of linear and angular velocity in a frame centered at the target ray where scaled orthography is assumed can be computed with by transferring the classic optical flow formula for scaled orthography to a different frame. However, it is not clear how only the []:2:2 submatrix of rotation reflects the rotation of the optical axis to the target ray. If for example the target ray is along the x-axis, then the rotation is around the y-axis [ cos & 0 & sin \ 0 & 1 & 0 \ -sin &0 & cos] and the submatrix of the form [ cos ; 0 \ 0 ; 1] cannot capture the full rotation. You might be correct for the scaled orthography case but please explain.

W6: L_1 and L_{D-SSIM} should be defined. You either say "We use the the loss of the original Gaussian splitting paper..." or you define your symbols.

W7: Several dynamic Gaussian CVPR papers have not been cited: CVPR24: Control4D: Efficient 4D Portrait Editing with Text CVPR24: Neural Parametric Gaussians for Monocular Non-Rigid Object Reconstruction CVPR24: CoGS: Controllable Gaussian Splatting CVPR24: Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis

W8: There is a "double" citation in the paper Youtian Lin, Zuozhuo Dai, Siyu Zhu, and Yao Yao. Gaussian-flow: 4d reconstruction with dynamic 3d gaussian particle. arXiv preprint arXiv:2312.03431, 2023b.

Youtian Lin, Zuozhuo Dai, Siyu Zhu, and Yao Yao. Gaussian-flow: 4d reconstruction with dynamic 3d gaussian particle. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21136–21145, 2024.

W9: 3.5 is highly confusing, in particular, because of the introduction of calligraphic and mathbb fonts. If put in a classic context this is nothing else than error state propagation equation in the Kalman filter with the resisual being the control input.

W10: L. 53: ". Recognizing that optical flow is the 2D projection of scene flow," This is wrong. Optical flow (technically the motion field not optical flow) is not the projection of scene flow. $({\\dot x},{\\dot y}) \\neq \\frac{1}{Z} (({\\dot X},{\\dot Y},{\\dot Z})$. Please see a classic textbook like Berthold Horn.

W11: L. 53: Optical flow is a well studied field in classic articles from the 1980's and it is textbook material. It does not start with Ranjan and Black in 2017.