Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting

We thank the reviewer for the positive and detailed review as well as the suggestions for improvement. Our response to the reviewer’s comments is below:

Q1: The comparison with the concurrent work.

Very reasonable suggestion. We have updated the comparisons with the 4DGS [1] in the revised paper. For the Dynamic 3D Gaussian [2], we have discussed its differences in the related work section. In the discussion period, we additionally test its performance in the cut roasted beef scene using its officially released code. We present the comparison in the table below:

Note that the way described in the paper of Dynamic 3D Gaussian for obtaining the foreground/background mask cannot be simply applied to this scene, and its performance highly depends on these masks significantly. So we extract masks using an advanced video object segmentation model Xmem [3] with the masks for the first frame acquired via SAM [4]. In addition, we enable optimization of seg colors in the initial timestep and detach other parameters before rendering segmentation masks. Due to the above customization is not contained in the original Dynamic 3D Gaussian, we did not append this result in our revised paper.

[1] Wu, Guanjun, et al. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint, 2023.

[2] Luiten, Jonathon, et al. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. 3DV, 2023.

[3] Cheng, Ho Kei, and Alexander G. Schwing. Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model. ECCV, 2022.

[4] Kirillov, Alexander, et al. Segment anything. ICCV, 2023.

Q2: The application on the urban scenes.

Great thought. While this is beyond the scope of this work, we are delighted to share our explorations in this area. We test 4D Gaussians with minimal modification on selected dynamic sequences in the widely used Waymo Open Dataset. The modifications are summarized in the revised appendix, where we also provide some visualization of reconstruction and novel view synthesis results. The rendered videos can be found in the supplementary material. These results reveal that the proposed method can uniformly model both dynamic and static regions and achieve competitive reconstruction speed and quality without relying on any manually labeled dynamic mask. This indicates good potential of the proposed method.

Q3: Novelty concern.

In a sense, all work that extends static reconstruction methods to dynamic scenes can be viewed as the inclusion of time in addition to spatial dimension; how to achieve this is however non-trivial. Often such extensions will introduce new modules and optimizations, making the whole pipeline much more complex and less flexible. In this work, we implement a different extension by introducing an easily optimized and rendered parameterization of 4D Gaussian, and adapt the original rendering formulation in 3DGS. We believe that this extension is more flexible and inherits the advantages and spirits of the explicit modeling in 3D Gaussian Splatting, while it’s also fully interpretable and friendly to composition and editing. The 4DSH is designed as an integral part for appearance evolution over time. Indeed, the performance gain it brings is not impressive despite its obvious usefulness. Per our observation, this is due to limited cases of such time-dependent appearance involved within the evaluation video sequences. We will attempt to explore more extensive test data in the future work.

Q4: Modeling concern.

The issue raised by the reviewer 8jaG is common to splatting-based methods: we can draw an analogy in the 3D case, where many real-world 3D entities are composed of line segments with constant transparency, yet the 3D Gaussian also fades out beyond its $\mu_{xyz}$. However, this issue doesn’t significantly impede its expressive capability. The bullet times experiments in our supplementary material also suggest that this issue does not affect the temporal smoothness of 4D Gaussian. Furthermore, the unnormalized Gaussian function allows the 4D Gaussian to always have larger influence and does not fade out abruptly after $\mu_t$. For the issue of physically-grounded, it is better to clarify that we are not aiming to model physical reality, and the 4D Gaussian should not be seen as corresponding to some physical entity. Actually, most of the successful methods for the task of novel view synthesis rather than simulation are not completely physically-grounded, although they may draw inspiration from some aspect of physical reality. Nevertheless, we believe that solving this issue properly might result in a new more advanced representation paradigm in the future research.

We thank the reviewer for the positive and detailed review as well as the suggestions for improvement. Our response to the reviewer’s comments is below:

Q1: The number of 4D Gaussians.

The reviewer’s insight on the locality is very discerning. Actually, unlike spatial scaling which is set to the mean of the distance to the closet three points, we initialized a very large scaling in the temporal dimension as mentioned in implementation details, which allows most background Gaussians to cover a long time period (around the scene’s duration), ensuring that the total number of the fitted 4D Gaussians does not significantly exceeds that of 3D Gaussians.

As for the memory or storage consumption, we believe that there is no free lunch in this aspect. For any representation, with the increase of the video length, the storage requirements are unlikely to remain constant forever, because we can't store more information without any extra cost. Nevertheless, despite the possible storage consumption issues, the rendering speed of the proposed 4D Gaussian tends to be fixed as the video length increases. From this point of view, this characteristic of locality enables our proposed method to be more friendly to long videos than that of encoding information into implicit neural networks.

Q2: The visualization of time

Good suggestion. Since the temporal characteristics of the 4D Gaussian can be summarized by its variance and mean, we provide visualizations of them in Figure 8 of the revised appendix. Interestingly, these visualizations naturally form a mask of dynamic and static regions. This phenomenon not only shows the potential and the versatility of our approach, but also demonstrates that a big proportion of Gaussians are used for background modeling, with a long span over time. To demonstrate this more directly, we plot the marginal distribution of some representative Gaussians in Figure 9.

Q3: LPIPS

In the original submitted version, we only reported PSNR and SSIM following kplanes. Now we have updated LPIPS in the revised paper.

Q4: Long-term tracking

This is a great point and worth for deep investigation including both benchmark upgrade and algorithm advance. We consider this issue to be one of the most important aspects for extending our model. Considering that maintaining global correspondence may compromise the fitting capability and flexibility in complex dynamic scenes, we leave it for future work.

We thank the reviewer for the positive feedback and constructive suggestions. Our response to the reviewer’s concerns is below:

Q1: The number of 4D Gaussians.

This is a great question regarding the scalability of our 4D Gaussian model. We further clarify the following facets:

1. The total number of 4D Gaussians is not essentially larger than that of 3D Gaussians, as each 4D Gaussian has a certain time span and there exists a high proportion of background Gaussians active throughout the whole time duration of the scene. For the typical scenes in the DyNeRF dataset, the number of 4D Gaussian fitted on the videos with 300 frames is on the same order (millions) as the number of points in the 3D Gaussian fitted only on the first frame.

2. Besides, since we filter the Gaussians according to the marginal of time with a negligible time cost before the frustum culling, the number of Gaussians actually participated in the rendering of each frame is nearly constant, and thus exhibiting stable rendering speed with the increase of the video length and the total number of Gaussians. The statistic of Gaussian’s number is provided in Figure 10 of the revised appendix.

Q2: The potential conflict in the jointly optimization of the 4D means and the rotation.

Great question. Yes, the 4D means are also optimized during training, but they are fixed during inference and their displacements are derived from the 4D rotations only. So there is no ambiguity during inference. Regarding the potential interference caused by the joint optimization of the means and rotation, we show in the following that their interaction will not introduce ambiguity of displacement. Consider a simplified linear example for illustration purposes: $y(t; x_0, v) = x_0 + vt$, where $x_0$ and $v$ are the trainable parameters. Given sufficient observations {$(y_i, t_i)$}$_{i=1}^{N}$, it can converge to the unique correct solution, despite that $x_0$ and $v$ also interact with each other during optimization. However, our model is slightly different. Specifically, our model should rather be abstracted as the following system: $y(t; x_0, v, t_0) = x_0 + vt_0 + vt$. Fortunately, if we do a simple substitution $w = x_0 + vt_0$, the above equation can be reformulated as $y(t; w, v) = w + vt$, allows for the unambiguous determination of $v$, ensuring that there is no ambiguity in the learning of displacement.

Q3: Discussion on the advantage compared to the naïve baseline.

We ascribe this performance gain to the motion modeling capabilities stemming from the proposed 4D rotation. As shown in Figure 4 of the original paper, the displacement derived from 4D rotation can roughly capture the underlying dynamics of the scene, which enables more efficient information sharing across frames and leads to more seamless transitions between them. We also provide more visualizations and discussions of temporal slices for two versions of the 4D Gaussian in Section E and Figure 11 of the revised appendix.

Q4: Discussion on the 4D SH.

Thanks. The possible ambiguity could be substantially mitigated by delaying the training of time-dependent coefficients. Actually, rather than inducing ambiguity, 4DSH’s benefits in motion learning are more critical. Because unlike the static scenes, there exists temporal evolution in the appearance of static objects in dynamic scenes. Without a particular design as our 4DSH to tackle this variable, such a phenomenon can only be fitted by undesired motion that shouldn’t exist. Moreover, our experience indicates that 4DSH brings a positive impact on performance in most cases.