Gaussian Splatting Lucas-Kanade

We thank the reviewer for the positive comments on the paper presentation, writing, and appreciate the thorough constructive feedback! We address the concerns in detail in the following and look forward to engaging in more in-depth discussions.

The claim that velocity fields can be derived from the forward warp field using modified Lucas-Kanade and displacement/rotation fields is seen as inaccurate; camera pose should also influence flow derivations.

The velocity field and scene flow field are both derived in a 3D space. The velocities of the Gaussians express their traveling speed in space, regardless of the camera viewpoints or their motions. So the velocity field derived should not be concerned with camera motion influences. Once the scene motion is projected to a certain camera plane, the flattened optical flow on the image plane would be comprised of both scene motions and camera motions, leading to the inclusion of the depth term to constrain the regularization of the scene motions. To this end, as the reviewer points out, it would likely be beneficial to further incorporate the variations in camera pose. We currently only consider the projected scene motion in this work due to scope limitations, but have been considering further disentangling the camera motion's contribution alongside the uncertainties arising from scene and camera motions in the follow-up work.

The method’s time integration is not well explained (e.g., time step Δt) in Section 4.2.

The time integration is performed with Monte Carlo integration, with a sampling size of 10 points over each domain. Monte Carlo integration is preferred over other deterministic approaches for efficiency and balanced performance. We can include this detail in the revision if it improves understanding.

Experimental limitations are noted, and additional baseline comparisons (e.g., with PhotoSLAM) are suggested.

We have thoroughly compared our approach with relevant papers on dynamic Gaussian scene reconstruction, as well as the ones employing flow supervision. The comparisons on DyCheck, Dynamic Scenes, Plenoptic Videos, and HyperNeRF datasets are reported in Tables 2, 1, 4, and 5, ordered by an increasing Effective Motion Factor in the datasets. We have carefully reviewed the suggested PhotoSLAM paper for its feasibility for comparison, and recognize its merits as a SLAM algorithm with Gaussians. Unfortunately, the paper is proposed for Gaussian SLAM in static scenes, whereas our objective does not cover localizing cameras and rather focuses on rendering dynamic scenes. For the planned extension of this current work, where the camera motion contributions would be more thoroughly studied, PhotoSLAM and this line of work would be incorporated for comparison.

Could the authors provide details on the experimental setup, particularly the time step Δt in Eq. (12)? Does the method maintain the small motions assumption as Δt increases?

Each time step is taken as the time difference between two frames. We use the inherent ordering of frames in each dataset without augmenting the frame selection paradigm to show the approach’s flexibility. We do add a randomly sampled Gaussian noise to the interval into (Δt+ϵ) to simulate perturbations for improved robustness. The small motion assumption would hold only when the trajectory is locally 2-Lipschitz, whereas increasing the Δt would break this condition since the trajectories would become piecewise and discontinuous in time.

The paper claims sensitivity to SfM initialization, yet 3DGS performs well with random point clouds. Can this claim be further clarified?

We acknowledge that using random initialization for 3DGS can yield reasonable performance on simple, static scenes. However, in more complex and dynamic scenarios, an accurate SfM initialization proves to be advantageous. A recent study (https://theialab.github.io/nerf-3dgs/) evaluated various initialization methods for static 3DGS, providing further evidence that robust initialization often results in improved performance.

A breakdown of time complexity and memory requirements during training would help evaluate efficiency and scalability.

We profiled the time consumption and memory requirements on the “Playground” video from the Dynamic Scenes dataset. Further optimizations to inversion implementation such as using Least Squares to approximate inverse instead of employing an exact solution could further reduce the profiled complexities.

Hi, seems like there is a certain level of miscommunication here. When we project the scene flow to a 2D camera plane to obtain optical flow $\hat{O}_{t\rightarrow t+1}$, the rendered optical flow does capture both the camera motion and the scene motion. Since the optical flow derivation and disentanglement of optical flow into camera + motion components are not the focus of our paper, we follow the standard experimental setup in Neural Scene Flow Fields (https://arxiv.org/pdf/2011.13084) only to show the effectiveness of the analytical flow field. With a pixel $p_t$ at time $t$, the pixel's forward location is computed by performing perspective projection of the point's 3D location with its corresponding scene flow displacement into the viewpoint at time $t+1$. This way, it yields a projected scene with both camera and scene motion warped forward in time.

If the camera motion was not considered in this setup, as suggested here, then the NVS result would have been blurry due to a lack of camera motion regularization, and the same consequence would have happened with many prior NeRF/GS papers following the same experimental setup. While an additional explicit regularization term for camera motion might be beneficial, this is not the main consideration of our paper.

We thank the reviewer for the detailed comments and for recognizing the efforts to technical novelty. We understand that there needs clarification on the network choice and our criteria for selecting benchmark comparisons. We hereby provide clarifications and will adhere to the reviewer’s suggestions in the revision.

The evaluation dataset is limited. Testing on benchmarks like DNeRF, HyperNeRF, and NeRF-DS would clarify performance relative to Deformable GS.

Our method is specifically designed for real-world scenarios characterized by minimal camera movement and the presence of highly dynamic objects. As such, testing on multi-view synthetic datasets found in D-NeRF or the highly specular dynamic scenes featured in NeRF-DS falls outside of the intended scope of our work. Instead, we focus our evaluation on datasets that are more closely aligned with our target use cases. As for evaluating our method on the HyperNeRF datasets, we currently show these quantitative results in Table 5.

Despite claims of analytical tractability, the method’s reliance on an MLP undermines this advantage.

For clarification, our method proposes a regularization of network estimated parameters, which is indifferent to different network choices. In fact, network Jacobian regularization is common in other areas of machine learning for improving prediction stabilities with perturbed inputs. Previous methods have shown that regularizing the network Jacobian, whether with domain-specific regularizers like our proposed method or through generic Frobenius norm, can enlarge the decision margin of the model and lead to more robust predictions. To prove that the method also generalizes well for other network choices, we implemented the method on the 4DGS framework. The experiment reports a PSNR value of 23.6 on the Playground scene, with a +2.2 increase from the vanilla 4DGS.

The supplementary video reveals visible unnatural blurring.

The included kid-running and train videos in supplementary were captured with cameras with very minimal translational motions, and we show comparisons with vanilla Deformable GS for comparison without our regularization. For videos like these with minimal translational motions, the rendering qualities are limited by mainly two sources of errors: accuracy of camera poses, and accuracy of trajectories. The blurring can be further improved with further optimized cameras, in addition to our motion regularizations. However, we focus on promoting scene motion integrity in our scope and suggest referring to other great works on camera pose optimization for reference (https://3d-aigc.github.io/GGRt/, https://oasisyang.github.io/colmap-free-3dgs/, https://huajianup.github.io/research/Photo-SLAM/).

Can you specify the network architecture of the warp field?

The network architecture of the warp field is identical to the one found in Deformable Gaussian Splatting.

We sincerely appreciate the provided valuable feedback to enhance our work’s quality! We have thoroughly considered the suggested experiments, and the suggestions on the writing. We will address these aspects appropriately and offer the following responses to the raised concerns, as well as open-source a template implementation upon acceptance to foster further studies on this end. The ambiguities in the writing will be addressed in the next revision.

What is the method’s robustness under noisy conditions, and can noise effects be further minimized? The noises in the final render with dynamic Gaussian scenes are multifaceted problems that usually arise from multiple factors: perceptual noises like motion blur and lighting changes, insufficient camera motion parallax, and insufficient coverage of transient objects. We tested the method with a randomly initialized point cloud on the Dynamic Scenes dataset and reported a PSNR performance of 25.12. With the motion constraint in our approach, we have shown to improve the motion integrity of transient objects but more than such constraint on motion would be needed to tackle noises that arise from other factors. Further investigations on reducing the noise level can potentially leverage the motion uncertainties that can be derived from our Jacobian formulation.

What are the computational advantages over purely learned models? We focus on geometric integrity over computational improvements. Given the time complexity bottleneck of computing the inverse of Jacobian, the approach can be further optimized by 1. employing a more efficient inversion method and 2. selectively sampling the Gaussians for regularization to reduce the complexity. Our method's memory and time consumption have been profiled as follows:

How is the threshold for motion masking determined, and do these parameters generalize well? The optical flow values are normalized to be ubiquitous to scene scales. The threshold of motion masking is then determined by taking statistics of flow guidances, and set at the 80 percentile of all motions, roughly at 0.1 for the “Playground” scene. Motion masking is mainly utilized for mining the Gaussians with larger motions, avoiding over-smoothing of the overall scene. We empirically ablated the choice of threshold and reported the performance differences in the following table, which will be also included in the supplementary material in the next revision.

We greatly appreciate the valuable suggestions and your time for the review. Your concerns help us to further improve the coherence of notations. We address the concerns in detail in the following.

While the reviewer appreciates the analytical elegance, they feel the contribution is not sufficiently differentiated.

While it may appear that our paper builds heavily on existing frameworks like [Flow Supervision for Deformable NeRF] and [GaussianFlow], we believe our work embodies a fundamental yet impactful insight. Indeed, simplicity might sometimes be mistaken for a lack of novelty. As the reviewer points out, our primary contribution lies in the reformulation of forward warp field deformation in Gaussian Splatting from the backward counterpart in NeRF, and while this may seem minor, it has significant implications for performance and generalization. With Gaussians representing explicit scene geometries, and their heavier dependency on accurate scene-to-camera projection, the deformation learned from networks can be piece-wise and even incorrectly deform supposedly static regions to compensate for photometric integrity, as shown in Figure 3 of the paper. Our approach, although simple, effectively enforces smoothness on the forward warp field, disambiguates the motions, and pushes the extent of perceptual improvements from precise motions.

How does Gaussian Flow address classic optical flow issues, such as:

• The aperture problem in the Gaussian context?

Classic optical flow issues, such as the aperture problem, would likely be inherited in the Gaussian context as part of the supervision. While no existing works have directly investigated the aperture problem in the context of 3DGS, many works have discussed in the flow estimation line of work (https://openaccess.thecvf.com/content/CVPR2022/papers/Schrodi_Towards_Understanding_Adversarial_Robustness_of_Optical_Flow_Networks_CVPR_2022_paper.pdf). The optical flow map used for comparison would inherit the ambiguities arise from repetitive patterns and cause potential instabilities in the regularizations, despite the Gaussian rasterized flow map reflecting the projected motions. To explore this further, we performed experiments on the Playground scene in which different methods for flow supervision were used. We evaluated [FlowNet] and [MemFlow], achieving PSNR scores of 26.38 and 27.04. As a reference, our original setup achieved a PSNR of 26.34. [FlowNet] produced less accurate flow maps, failing to capture scene movement effectively. Conversely, [MemFlow], the current state-of-the-art optical flow estimator, achieved the best results with sharp details and accurate flow.

• Rank deficiency in Gaussian covariance?

The covariance matrices of the Gaussians express the geometry of Gaussians. Rank deficiencies can occur with needle-like or disk-shaped Gaussians, with effective ranks lower than 2. These structures, despite their potential to lead to instabilities in the inversion, are needed to represent thin and elongated geometries for certain scenes. Here with our approach, we do not set constraints on the effective rank needed for utilizing our approach for showcasing the generality. That being said, there are many great approaches for promoting higher-rank Gaussians that can be explored if needed by such particular application (https://arxiv.org/html/2406.11672v1, https://arxiv.org/html/2403.06908v1).

• Ambiguities in scene structure?

This is a valid concern, and we are actively working on a follow-up project to address ambiguities and uncertainties in scene structure, with the goal of minimizing them. Our core assumption is that deformation networks exhibit distinct patterns in regions that are easier to reconstruct compared to those that are more challenging. To explore this, we are analyzing the Jacobian of these deformations with respect to time, enabling us to quantify uncertainty directly from these patterns. In the revised Appendix Fig. A6, we present preliminary results on a controlled synthetic scene, where the red and green balls have randomized positions in the training dataset, while the blue ball accurately corresponds to the correct time step. Brighter colors indicate higher amounts of uncertainty.

How would variations in spherical harmonic coefficients affect the results?

The inclusion of spherical harmonics did not affect the results much. For the tested Playground scene, the difference only ranged within ~+0.1 PSNR.

What effects arise from local deformations like scale or shear?

The local deformations like changes to scale and shear would induce subtle deformations that cannot be fully captured with trajectory regularizations. To prove this point, we included the rotation and scale terms of Gaussians, which lead to a subtle increase of ~+0.2 PSNR on the tested Playground scene.

To reviewers GbSS, GTCV, and uwP8. We have posted responses to address the concerns you have mentioned. If there are any additional concerns, we would be happy to discuss them further. Thanks!