Fully Explicit Dynamic Gaussian Splatting

Table A. Quantitative results of the experiment on handling color changes without dynamic points.

$~$

$~$

Table B. Quantitative results on the Birthday scene in the Technicolor dataset.

$~$

$~$

Table C. Experimental results on the conversion rates to dynamic points.

$~$

$~$

Table D. Experimental results on different keyframe intervals and skipped frames.

[W1] The main motivation of separating dynamic points is to allow FESGS to handle temporal changes with either color or displacement variations, so stationary objects but temporally changing colors are also trained as dynamic points.

We carry out an additional experiment for the case where points with color changes are regarded as static points. We train 3DGS by masking only the moving parts of N3V, and measure the performance gain (To measure overall gain, we replace dynamic parts with ours) on the remaining parts. Please refer to Table A for the detailed results. As shown in the evaluation, if we do not consider the points with color changes as dynamic points, it is hard to obtain satisfactory results.

In the case of large movements, we conduct an experiment with longer frames on the birthday scene in Figure D of the uploaded PDF file. The rendering result shows that FEDGS detects the movements of objects as well as the color changes. The quantitative result in detail for the large movements will be covered in the next answer.

[W2] We conduct an experiment with different frames of the Technicolor dataset to see this issue. The selected interval is from the timestamp when the person is completely invisible to the timestamp when the bag is placed on the desk. The result is reported in Table B and Figure D of the uploaded PDF file. In this case, the initialized point clouds only use the COLMAP of the first timestamp (no prior point cloud for the person). This result shows that FEDGS can learn about appearing objects on the scene. This is because the densification of 3DGS works well due to the explicit representation that allows splitting from neighboring Gaussians.

[Q1] This is because 2% of the static points with the biggest movement are related to the dynamic part. We note that the 2% is determined empirically, and we observe that extracting more points causes too many dynamic points and extracting fewer points result in not enough dynamic points. The performance change according to the ratio is shown in Table C.

[Q2] The keyframe is only associated with dynamic points. And, the parameters optimized in each keyframe are about the position and rotation values, and they are optimized by RAdam. Using Equation 7, the CHIP interpolates the position values stored in the four neighboring keyframes for the dynamic points at timestamp $t$. The interpolated position is represented as a $R^3$ vector whose gradient can be obtained by the differential rasterizer of 3DGS. After obtaining the gradient, by Equation 6, the gradient of each keyframe's position can also be obtained by partial difference (we use PyTorch's autograd function) and used to update the position values stored in each keyframe. It is worth noting that thanks to CHIP, FEDGS can also learn the velocity change and momentum value of the dynamic point. Similarly, the keyframe rotation value can also be optimized by Equation 8 and the differential rasterizer in 3DGS. The difference with the keyframe position is that it uses two neighboring keyframes. [Q3] This means that for every timestamp, only the image from the earliest timestamp is used. For the N3V dataset, for example, the COLMAP result is provided for the first frame (about 20 images) out of a total of 300 frames. Note that 4DGS models or STGs use all frames (about 6000 images). This prevents COLMAP from taking an excessive computational time to optimize.

[Q4] Irregular motion can be handled by CHIP, given a small enough keyframe interval. CHIP uses the tangent and position of two neighboring keyframes. As shown in Equation. 7, the difference of positions between two adjacent keyframes is used for tangent. The position values are explicitly stored in each keyframe. With the two values, CHIP approximates dynamic motion as a polynomial. If the keyframe is too sparse, it becomes difficult to approximate the motion with CHIP (i.e higher degree polynomials are required), but if it is dense enough, most of the motion can be approximated. This is true even if the motion is velocity-variant or non-linear. This can be seen in the results in Table D. The experiment is performed on cook spinach scene in the N3V dataset. This scene includes a motion that changes direction, accelerates and decelerates. We also include experiments that skip frames to simulate larger movements (the more frames skipped, the larger the movement). These results show that if the keyframe of FEDGS is dense enough, it can be handled (even if the motion is not regular). In general, setting a keyframe interval of 10 is reasonable considering overall performance and storage size.

Table J. Experimental results on different keyframe intervals and skipped frames.

$~$

$~$

Table K. Quantitative comparisons on the Train scene in the Technicolor dataset.

Table F. Statistical analysis (Mean ± STD) of ablation studies measured by PSNR.

$~$

$~$

Table G. Statistical analysis (Mean ± STD) of ablation studies measured by SSIM.

$~$

$~$

Table H. Statistical analysis (Mean ± STD) of ablation studies measured by LPIPS.

$~$

$~$

Table I. Statistical analysis (Mean ± STD) on the N3V dataset.

Table A. Experimental results on different keyframe intervals and skipped frames.

$~$

$~$

Table B. Quantitative evaluation on the Technicolor dataset using PSNR.

$~$

$~$

Table C. Quantitative evaluation on the Technicolor dataset using SSIM.

†: Use structural similarity function from scikit-image library

$~$

$~$

Table D. Quantitative evaluation on the Technicolor dataset using LPIPS.

$~$

$~$

Table E. Quantitative results on the extremely long duration video sequences (Flame Salmon) in the N3V dataset.

[Q3] Thanks for introducing a relevant paper. We have looked at this paper and will state the differences between it and this paper.

SWinGS disentangles static and dynamic scenes using MLP. For dynamic regions, the model slices the timestamp using a sliding window strategy and trains a deformation MLP of the canonical space by window. SWinGS has similarities with FEDGS in that it divides static and dynamic parts and slices timestamps, but the differences between them are as follows:

• Detecting Dynamic part: SWinGS uses MLP to classify static and dynamic in a binary manner. It performs classification by thresholding L1 error and uses spatial location as input. FEDGS, on the other hand, determines dynamic points based on the magnitude by which the static gaussian moves. This can distinguish dynamic points even if two points are in the same location.
• Dynamic representation: SWinGS uses an implicit function defined for each window to find the deformation in the canonical space of the dynamic grid. On the other hand, we store the dynamic motion of each point independently in an explicit way. Therefore, we do not need the coordinates of the canonical space. We only need the position information of the neighboring keyframes.
• Keyframe selection: SWinGS uses the magnitude of the optical flow to measure the magnitude of the motion and divides the keyframes based on this. FEDGS, on the other hand, uses a fixed keyframe to take advantage of indexing, as described earlier. This interval is a hyperparameter.
• Optimization: SWinGS adds a loss of pixel difference to achieve temporal consistency. In FEDGS, point backtracking for dynamic point pruning, conversion of static points to dynamic points, and progressive learning scheme are added.

[Q1] The sensitivity of motions depends on keyframe intervals. However, instead of directly measuring motions, we decide to determine the keyframe intervals based on rendering quality. As explained in W1-2, the keyframe interval is based on both storage usage and rendering quality. Here, we assume that sufficiently small keyframe interval (in our cases, 10) can handle most of complex motion. We report the experimental evidence in Table J. To simulate and measure the motion speed, we intentionally skip frames in videos.

[Q2] Each answer is as follows:

1. As described in W1-1, we adopt the densification algorithm of the original 3DGS model, and this densification is applied to both static and dynamic points. In addition, static points are periodically converted to dynamic points so that even if there is a new object, it can be treated as a dynamic point by splitting the Gaussian around it. In our experiments, the initial Gaussians start at about 7000, but by the end of training, the scene usually includes more than 200,000 Gaussians. Although densification can handle new objects as long as there are Gaussians around the dynamic object that can be split (e.g., static Gaussians can be converted to dynamic after splitting, or dynamic Gaussians can be split and optimized), unfortunately, when there is a sudden appearance of an object as described in Section 6, it is difficult to learn because there are no Gaussians around it to split. This is the same problem as the original 3DGS, which does not learn well unless there is a Gaussian to split around it (e.g., random initialization). Also, for objects that appear repeatedly, there is no way to determine if the disappearing object and the reappearing object are the same because its temporal consistency is required to determine this. Our model represents these as separate objects when an object is temporally discontinuous. In our experiment, the rotating decoration that is occluded and reappears, our model learns that the dynamic points disappear and new dynamic points appear in Figure C of the uploaded PDF file. In this figure, you can see that a number of point clouds changes before and after the object flips. This shows that FEDGS creates different Gaussians when the object reappears. Nevertheless, you can see that the rendering results look good. We further carry out an experiment with the presence of occlusion. We have added comparison evaluation in Table K. Figure A of the uploaded PDF file shows the comparison results when occlusion is present. We select 100 frames of the train scene in the Technicolor dataset and compare FEDGS with the other models. We use the first frame as input and do not use the point cloud information of other frames. The results show that FEDGS renders dynamic objects well, even when they disappear and reappear. 4DGS learns dynamic parts well, but struggles to render static parts, while STG struggles to render dynamic parts. 4D Gaussians fails to render dynamic objects.
2. $\mathcal{D}$ represents the images of all timestamps in all training sets.The original 3DGS does not have it, and we add this because the opacity-based pruning method of the original 3DGS does not completely remove floaters caused by dynamic objects. This is because some dynamic objects are trained to move to an invisible area, instead of vanishing. This method measures how much each Gaussian contributes to the error in the rendered image, much like accumulating a gradient in densification. More specifically, using this operation on a rendered image, we can get the sum of the error (L1 or SSIM error) and the sum of the alpha values of all the Gaussians in the image. Dividing this cumulative error by the sum of the alpha values can normalize it. We threshold this error to remove Gaussians that cause large errors.
3. The reason, why we assume that static points have linear motion, is to distinguish dynamic points, as explained in W1-3. This does not aim to model the motion of static parts exactly. We agree with your idea of modeling the rotation of the static points, but in our method, the magnitude of the position change is enough to distinguish dynamic points. Different from the original 3DGS model learning positional information of static points, all static points in FEDGS are approximated to have linear movement. And we measure a magnitude of the motion between frames to reassign static point to dynamic point if the magnitude is more than a predefined threshold value. Interestingly, although static points are represented as linear motions over time, they can explain all temporal changes, including color, rotation, and opacity. This is because the static points are considered as moving points during optimization if one of the temporal changes exists. Therefore, instead of using a more complex model to represent static points, we use a simple yet effective way that can further reduce the computational complexity.

Answer to [Q3] is followed by the next comment.

[W1] Each answer is as follows. The revised version will reflect this description.

1. FEDGS follows the densification algorithm of the original 3DGS. This algorithm accumulates the gradient magnitudes of visible Gaussians (that is, the gradient in the x,y direction of the image space) for a camera during training, selects Gaussians whose gradient magnitudes are greater than a threshold and applies splitting. Here, temporal opacity does not directly involve the densification, but increases or decreases the gradient in 2D direction with respect to visibility over time, allowing the dynamic Gaussian to consider only the gradient in the timestamp in which it is visible.

2. The reason why we fix the keyframe interval is because of optimization issues. While a varying keyframe has advantages in terms of storage (e.g., allocate wider intervals for slow motion and narrower intervals for fast motion), it imposes a high cost on indexing, since the keyframe index of each Gaussian must be searched for every timestamp. On the other hand, if we use fixed timestamps, this cost can be greatly reduced by simple indexing (we can chunk keyframe data as a vector). We choose the fixed step keyframe because we believe that the benefit of reducing the computational cost outweighs the storage wasted by using fixed timestamps, and it is trained to be a reasonable size for the actual size of the storage. The keyframe interval is empirically chosen in consideration of rendering quality and efficiency. Shorter keyframes can handle more complex motion but are also easy to overfit and require more storage. On the other hand, large keyframe intervals often are inadequate to handle complex motion. We add the test result according to a variety of intervals on the N3V dataset in Table A.

3. The reason, why we assume that static Gaussians have linear motions, is to distinguish dynamic points from the other static points. We observe that there is a high correlation between their spatial movements and dynamicity when we train a scene having dynamic motions. We find that when we apply only a linear transformation (i.e., Equation 5) to the 3DGS model, the points with the largest transformation closely coincide with the dynamic part (e.g., selecting 2% of the N3V dataset in order of size matches points corresponding to person). Of course, we could adopt more sophisticated assumptions for the movement, but the linear motion assumption is enough to account for it. We note that the linear motion is normalized based on a distance between each Gaussian and a camera. This aims to impose higher weights on an object close to the camera than a distant one if they have the same distance.

[W2] Thanks for pointing this out. We have added Table B, Table C and Table D with scene breakdown results of PSNR, SSIM and LPIPS respectively for the Technicolor dataset. The reason why there are no results for the other methods is that there are implementation issues because the Technicolor dataset is not officially supported. Nevertheless, we add some baselines to reflect this as best we can.

[W3] We test our model on a 1000-frame scene from the N3V dataset as recommended. We have added an quantitive evaluation in Table E. The rendered images can be found in Figure E of the uploaded PDF file This result shows reasonable performance and memory usage.

[W4] The answers for each item are shown below. We will clarify them in the revised version of this paper.

1. We report the average results of all scenes in the ablation study N3V dataset.
2. In w/o dynamic points, the linear motion transformation in Equation 5 is applied to the static points.
3. As described in W4-1, our ablation reports average over all scenes in N3V dataset. We have added Table F, Table G and Table H with scene breakdown ablation results of PSNR, SSIM and LPIPS respectively for the N3V dataset. We ran the experiment 5 times for each ablation to get multi-run statistics of the ablation result, whose result is reported below. Compared to other cases, ours shows the highest mean PSNR and lowest STD values. Please check it, and please let us know it if you have more questions.

[W5] We report multi-run experiments on all scenes in the N3V dataset which reports only “ours” results of Table F, Table G and Table H. We run our method for 10 times per scene and report mean and STD for each. It shows that our method has lower STD values than other cases, which suggests that each component of our FEDGS contributes to stable performance. Please refer the Table I for detailed results.

The answers to [Q1, 2] are followed by the next comment.

[W1] We thank you for your careful comment about the static points. In practice, we observe that there are temporal color and luminance changes even if their position does not change. In this work, our method is designed to handle all the temporal changes as dynamic points. This includes not only the movement of objects, but also changes over time, such as appearances, disappearances, and color changes. In some cases, such as Hats and tables in our supplementary video, there will be temporal shadows or changes in luminance at some timestamp. In these cases, dynamic points account for the light or rotation changes as well, thanks to our spherical harmonics. To validate this, we conduct an experiment with learning from a stationary object to be completely static in Figure B of the uploaded PDF file. We train 3DGS on the coffee martini scene with a mask on dynamic objects which are a person and a glass. We report the results of the test set when 3DGS is used alone and when 3DGS replaces the dynamic parts with our model. The numerical results are shown in Table A. As you can see, learning completely static points, even those that do not change position, leads to poor rendering quality because they cannot handle the other temporal information such as color changes or shadows. Therefore, it is beneficial to treat them as dynamic points.

[W2, Q1] The reason why we have an advantage in the sparse COLMAP condition is that the explicit representation makes the densification algorithm of 3DGS work better. Since the sparse COLMAP condition is trained with fewer points, the empty space needs to be filled with new Gaussians, which are split from existing Gaussian points. In 3DGS, the splitting algorithm works on points that threshold the gradient of the x-/y-direction of the image space. This gradient must be large enough for the Gaussian to be split. However, when using an implicit function, the deformation of the implicit function is optimized first, rather than the gradient of the Gaussian. This interferes with the Gaussians splitting, so implicit models require a more accurate 3D prior.

We perform several experiments in this regard. The following experiments show results for scenes that are difficult to train. In these experiments, the ability to reconstruct dynamic scenes from sparse COLMAPs implies that these scenes can also be handled by FEDGS.

First, we select the 100 frames including the occlusion of dynamic objects in the train scene of the Technicolor dataset and compare FEDGS with others. We use the point cloud prior of the first frame to give no information about the reappearing object after the object is occluded. The results are reported in Table B and the example is displayed in Figure A of the uploaded PDF. In these results, the explicit-based models STG, 4DGS, and Ours perform significantly better. In the case of STG, the dynamic part is not learned well as it goes to the later and later frames, and 4DGS may show that the dynamic part looks good, but it has difficulty in handling the static part which has negative impact on the overall performance. In particular, 4DGaussians, which is an implicit model, fails to disentangle the static and dynamic parts, resulting in missing rendering of the dynamic part. Our model, on the other hand, shows good performance and ability to learn both static and dynamic parts well.

Next, we carry out an experiment to see if FEDGS can learn about newly appearing objects. To handle new objects well, we need to split the dynamic part well. To test this, we select 120 frames from the Technicolor Birthday scene in which a person appears from nowhere. We use the point cloud prior to a frame where the person is invisible. The numerical results are shown in Table C and the rendered images are shown in Figure D of the uploaded PDF file. This result shows that the FEDGS model is beneficial for splitting Gaussians in dynamic scenes and is able to handle newly appearing dynamic objects.

Finally, we conduct an experiment with longer frames (1000 frames, 20,000 images in total) of the flame salmon scene from the N3V dataset. The result is reported in Table D and the rendered image in Figure E of the uploaded PDF file. This result shows that our model can learn well with reasonable storage, even for extremely long videos.

[W3] This is because 4DGaussians can be trained with fewer Gaussians on the N3V dataset than on complex scenes such as the Technicolor dataset. However, 4DGaussians suffers from a huge FPS drop when the minimum number of Gaussians needed is large (i.e. complex scene configurations). To prove it, we conduct an additional experiment on a more complicated scene, Technicolor, and the result is in Table E.

Table A. Quantitative results of the experiment on handling color changes without dynamic points.

$~$

$~$

Table B. Quantitative comparison on the Train scene in the Technicolor dataset.

$~$

$~$

Table C. Quantitative results on the Birthday scene in the Technicolor dataset.

$~$

$~$

Table D. Quantitative results on the extremely long duration video sequences (Flame Salmon) in the N3V dataset.

$~$

$~$

Table E. Quantitative evaluation on the Technicolor dataset.

Table E. Experimental results on different keyframe intervals and skipped frames.

Table A. Quantitative evaluation on the N3V Dataset using PSNR.

$~$

$~$

Table B. Quantitative evaluation on the N3V dataset using SSIM$_{1}$.

$~$

$~$

Table C. Quantitative evaluation on the N3V dataset using SSIM$_{2}$.

$~$

$~$

Table D. Quantitative evaluation on the N3V dataset using LPIPS.

[W1, Q1] Thank you for letting us know the relevant papers and we believe that additional comparison with them makes this paper solid. We have thoroughly reviewed these models and will include them in the revised version. You can find the updated results in Table A and we report available values. For 3DGStream [1], we have directly referred to the results from their paper. Similar to 4K4D [2] and Im4D [3], the results have been taken from the STG paper [4].

While the works showcase impressive results and offer valuable insights that we deeply respect. However, our approach does have certain advantages over theirs. First, 3DStream [1] does not support random access to time, which means you cannot directly access the desired scene at any arbitrary timestamp. Moreover, 3DStream requires a trained 3DGS model for initialization. Second, both 4K4D [2] and Im4D [3] require background scenes for training. Unfortunately, the N3V dataset does not provide background scenes, our reproduction of Im4D 30.34dB for PSNR.

[W2] Thanks for pointing this out. We attach SSIM and LPIPS in Table B, Table C and Table D. Our results include the models mentioned in W1. We will add it to the revised version. Note that there are two implementation for SSIM metric. SSIM$\_{1}$ is structural similarity function from scikit-image library and SSIM$_{2}$ is implementation borrowed from 3DGS codebase.

[W3] Other NeRF baselines may have comparable PSNR, but they have drawbacks in rendering/training time or model size. It can also be seen that there is a larger gain in the sparse initial 3D from COLMAP, compared to the other Gaussian Splatting baselines.

[Q2] While final rendering quality depends on the keyframe selection, they are also related to efficiency. Shorter keyframe intervals improve rendering quality because they allow more complex motions to be approximated, but if the keyframe interval is small enough, further reducing the interval is easy to overfit and requires a larger storage size. Therefore, we need to select an appropriately large enough keyframe interval. We report the numerical evaluation of FEDGS with respect to the keyframe interval in Table E.

We note that the empirical setting can be varied according to properties of datasets. We promise that the evaluation result will be included in the revised version of this paper.

[4] Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.