FreeGaussian: Guidance-free Controllable 3D Gaussian Splats with Flow Derivatives

Q3. The explanation of "3D Spherical Vector".

The introduction of “3D Spherical Vectors” has significant implications. Previous methods[1][2] perform scene control by mapping the object motion state to a 1D control signal and perform control over the object by varying this 1D state vector. However, modeling such a 1D signal requires strong prior knowledge like the object state at each time step[1][2] or need to approximate the motion trajectory using PCA[9]. Without the control signal supervision during training, these models collapse, failing to reconstruct and losing scene control capabilities. We overcome this limitation by representing the Gaussian states with 3D spherical vectors, which can be directly obtained from dynamic Gaussian tracking trajectory. As shown in Fig.1 of the manuscript, during the training stage, we represent the Gaussian dynamics state using cluster trajectory coordinates and jointly train the network to model Gaussian dynamics. In the control stage, we manually input interactive 3D vector*,* retrieving the Gaussian dynamics from the network. This technique eliminates the requirement of control signals and curve fitting while increasing control flexibility.

Q4. Why can controllable editing of trajectories be achieved without any constraints during the control phase?

In our implementation, control over the object is achieved by querying the dynamic Gaussian trajectories. We store the motion trajectories of each Gaussian ellipsoid in advance and search for the nearest point on the original trajectory given the manually input vector in the control stage. This technique allows FreeGaussian to achieve "unconstrained" control over the object, as the control vector will always match the nearest point on the original trajectory.

Q5. Why can a "manually input interactive 3D vector" be used in the control phase? If only the original trajectories are trained during the training phase.

There exists a certain mapping relationship between the 3D spherical vector and the motion trajectory of the object's Gaussian ellipsoids, as formulated in Eq.(6) of the manuscript. During the training phase, we track the motion trajectory of the 3D Gaussians and encode the trajectory $\mathbf{v}_c^i$ , which is fed into a lightweight model $\Theta$ to predict Gaussian dynamics $(\Delta\mathbf{X}_i, \Delta\mathbf{\Sigma}_i)$. In the control stage, a manually input 3D spherical vector, serving as a trajectory $\mathbf{v}_c^\prime$, is fed into the network to obtain the dynamics of 3D gaussian ellipsoids. Finally, we query on the original motion trajectory, which is consistent with what is described in Q4.

Q6. The proposed method seems to show only a slight improvement compared to LiveScene[2].

It is essential to emphasize that our method is completely annotation-free, relying solely on an existing optical flow estimator to generate 2D optical flow as pseudo-GT, along with image RGB as real GT. In contrast, LiveScene depends on masks and dense control variables during training to perceive and control objects within the scene.

By eliminating the need for dense annotations and still achieving better performance, our method demonstrates significant effectiveness and superiority. Furthermore, it offers a viable solution for unsupervised, large-scale scenario reconstruction and control, highlighting that FreeGaussian is both innovative and promising. This advancement opens new avenues for research and application where annotated data is scarce or unavailable.

Q8. How does the predicted Gaussian flow identify dynamic 3D Gaussians through back-projection?

The back projection is implemented by Gaussian rasterization, as detailed in [3]. This process projects 3D Gaussians on the screen and converting a 3D Gaussian into 2D representations with perspective projection. By leveraging the GSplats[8], we acquire the 2D projection of 3DGS. Through the mask of dynamic gaussian flow within the flow map, we establish a correspondence between 2D pixel and 3D Gaussians, subsequently linking the dynamic flow to specific spatial elements in the 3D space.

Q9. Is the DBSCAN clustering distance-based?

Yes, we employ the Euclidean distance between the center of 3D Gaussians as the distance metric in the DBSCAN algorithm. as mentioned in Line 304. More detailed implementation can be found in Sec 4 Implementation details of the manuscript.

We appreciate the reviewer's suggestions. We hope that our revisions, details explanations, and comparisons with MotionGS can increase the likelihood of a higher score.

Reference

[1] CoNeRF: Controllable Neural Radiance Fields

[2] LiveScene: Language Embedding Interactive Radiance Fields for Physical Scene Rendering and Control

[3] 3D Gaussian Splatting for Real-Time Radiance Field Rendering

[4] Subspace methods for recovering rigid motion I: Algorithm and implementation

[5] Fast Rolling Shutter Correction in the Wild

[6] Linear differential algorithm for motion recovery: A geometric approach

[7] FILM: Frame Interpolation for Large Motion

[8] gsplat: An Open-Source Library for Gaussian Splatting

[9] CoGS: Controllable Gaussian Splatting

[10] MotionGS: Exploring Explicit Motion Guidance for Deformable 3D Gaussian Splatting

Q7 The benefits of using “translation velocity v and rotational velocity ω” and the difference between it and MotionGS[10].

Since MotionGS was not publicly available before the ICLR paper submission deadline of Oct 01 '24, direct comparison is unfair. We had not seen the MotionGS before the review period and only cited GaussianFlow. Nevertheless, we're glad to discuss the difference between MotionGS and our algorithm's in flow derivation and the benefits of ours. We will cite the article accordingly.

Detailed dynamic Gaussian flow analysis can be found at Sec 6 of the supplementary. FreeGaussian is formulated under instantaneous motion assumption. The camera and 3D Gaussians hold separate velocities (both rotation and translation) in consecutive frames 0 and t. In contrast to MotionGS, which solely considers rendering-level depth and computes optical flow through back-projection, we model the complete motion process at the Gaussian ellipsoid level and rigorously derive the optical flow relationship among optical flow $\mathbf{u}$ , camera flow $\mathbf{u}^\text{Cam}$, and dynamic gaussian flow $\mathbf{u}^\text{GS}$ under instantaneous motion in Eqs.9-16. According to Possion's equation[4], the rotation and translation velocities $\mathbf{\omega}$ and $\mathcal{v}$ can be defined with $\mathbf{R}^{\top}(t)\dot{\mathbf{R}}(t) = [\boldsymbol{\omega}]_{\times}$, $\mathbf{R}^{\top}(t)\dot{\mathbf{T}}(t) = \boldsymbol{v}$, and $\mathbf{R}^{\top}(t)\dot{\mathbf{T}^\text{GS}}(t) = \boldsymbol{v}^\text{GS}$. Specifically, the relative rotation is modeled using the small rotation approximation $\mathbf{R} = \text{Exp}(\mathbf{\omega}) \approx \mathbf{I} + [\mathbf{\omega}] _ {\times}$, where *$[\mathbf{\omega}] _ {\times}$*is the skew-symmetric matrix of rotation velocity $\mathbf{\omega}$. Note that our derivation is more rigorous, and in this task, since the camera pose is available, our algorithm degenerates to computing rotation and translation velocities using the poses of adjacent frames. Please note the residual term $\Delta$ in Eq.(9); we prove that simply using back-projection in the degenerate case will cause errors, with an error magnitude of $\Delta$, which is a significant difference from MotionGS in the degenerate case.

Based on our formalization, we find that considering the complete Gaussian motion and 2D optical flow changes from 0 to t at a deeper level of the Gaussian ellipsoid hierarchy leads to higher accuracy, as evidenced by the residual term $\Delta$ shown in Eq.(9) of the manuscript and ablation in Fig.6 of the supplementary. Furthermore, a more important advantage is that our rigorous formula is more general. MotionGS uses discrete camera poses to compute optical flow through backprojection, whereas we model camera motion using differential rotation and translation velocities, allowing us to compute optical flow at any time between 0 and t using Eq.(9), whereas MotionGS is limited to 0 or t. This difference is similar to that between discrete and continuous functions and has important applications in various computer vision tasks, which model the accurate time interval between frames, such as video frame interpolation, rolling shutter correction, and SLAM[5][6][7]. MotionGS is a degenerate solution of the method we proposed.

We believe the score of this paper should not be influenced by MotionGS, especially when derivation has significant differences in terms of formalization, generality, and algorithm motivation.

Q1. Fix expression, typo and citations in the paper.

Thanks. We have carefully revised the manuscript according to your comments. “self-supervised optimization” was originally used to highlight FreeGaussian's annotation-free nature, thus we have updated the terminology to “annotation-free optimization” as suggested. Sec 3.2 of the manuscript analytically decoupled dynamic Gaussian flow from optical flow and camera motion via differential analysis with alpha composition. We include the detailed derivations in the supplementary materials to preserve the brevity and clarity of the manuscript. We kindly invite the reviewer to refer to Sec 6 of the supplementary for further information.

Q2. The relevance between the constraint on Gaussian flow and the control of motion.

The constraint on Gaussian flow enables the separation of object motion from camera motion by decomposing optical flow into dynamic Gaussian flow and camera flow, which is essential for identifying controllable regions and achieving independent control over object motion in dynamic scenes.

We reiterate our motivation and the contributions, as well as the relationships between them. We are committed to reconstructing controllable scenes with multiple objects and achieving independent control over different objects in dynamic scenes. For all the controllable scene datasets like [2], there are complex camera motions and multiple object motions. To achieve independent control over objects, we must satisfy two fundamental requirements: firstly, identify all controllable regions in the scene, and secondly, separate the objects in the scene. To identify interactable regions, we utilize an optical flow estimator to detect changing regions in the scene. Given that the optical flow is a composite of object motions and camera motion, we must employ a complex flow decomposition process to separate it into two distinct components: dynamic Gaussian flow and camera flow. Here, the dynamic Gaussian flow is generated by the object motion itself, which is essential for motion control. After determining the controllable regions in the scene, we establish a correspondence between the controllable regions and their corresponding 3D gaussian ellipsoids through back projection. Finally, by modeling the trajectories of the gaussian ellipsoids corresponding to the controllable regions, we achieve control over object motion.

Thanks for your time and effort in reviewing our work. We want to kindly follow up on the review process and greatly appreciate your feedback. Look forward to your response.

Q3. Failure cases illustration.

As previously discussed in Q1, noise introduced by camera pose affects the model's performance, and we indeed showcased corresponding failure cases, as seen in row 5 in Fig.7 of the supplementary. The figure shows that although the mechanical dog has a larger residual shadow, our reconstruction result is still relatively better compared to other methods, despite being a failure case. In addition to camera pose, rapid object motion is another factor causing the model to fail in reconstructing the scene. To illustrate this, we create an interactable scene in the OmniGibson simulator[1] and manipulate a chest with extremely fast speed. Fig.14(b) shows the rendering results of our model, which shows that rapid motion leads to a failure of our first-stage model, yielding a blurry boundary. Despite successfully decoupling moving objects, our initial 4D pre-trained model failed to capture the motion trajectories of 3D gaussian ellipsoids, thereby hindering the completion of subsequent control tasks.

Q4. How much of a lighting variation would affect the method and to what extent.

Thanks for this question. Since our method relies on off-the-shelf flow estimator, the accuracy of optical flow estimation will significantly impact our model's performance. Existing flow estimators are susceptible to environmental and lighting conditions, and therefore, we investigate the effects of overly strong and weak lighting on optical flow and dynamic Gaussian flow. Fig.18 of the supplementary shows that due to uneven lighting, the flow estimator overestimates the environmental flow, resulting in corresponding high-brightness regions in non-moving areas. The inaccuracy of flow estimation affects the clustering results and ultimately influences the final control process.

We appreciate the reviewer's attention to the robustness of our model and hope that the additional experiments we have provided can demonstrate the robustness and superiority of our model, and we look forward to a higher score.

Reference:

[1] BEHAVIOR-1K: A Benchmark for Embodied {AI} with 1,000 Everyday Activities and Realistic Simulation

Q1. Qualitative and quantitative results on datasets with noisy camera poses.

When the camera pose is relatively noisy, the model's performance is indeed affected. In our implementation, we employed the pose optimizer during the training, which mitigates the issues caused by noisy poses. Specifically, the poses are optimized by the Adam, with an initial learning rate of 1e-4 and an exponential decay scheduler.

In our experiments, we evaluate on two real-world datasets, namely CoNeRF Controllable and InterReal. The poses of these datasets were obtained through COLMAP and inherently possess certain levels of noise. Tab.1 and Tab.3 of the manuscript demonstrate the superiority of our method over other methods in terms of rendering metrics such as PSNR, highlighting the robustness of our method in the presence of noisy poses. Fig.7 and Fig.13 of the supplementary show the visualization comparisons for these two datasets, from which it can be observed that our method obtain more detailed and accurate representations of the objects. In contrast, other methods failed to capture the object's shape and resulted in significant artifacts.

To illustrate the robustness of the proposed method, we also conduct a new experiment on #seq001 of OmniSim with manual noise, which is generated using a Gaussian distribution with a mean of 0 and a variance of σ. The table below shows the impact of σ ranging from 1e-4 to 1e-2 on the rendering metrics. It shows that FreeGaussian still demonstrates strong rendering and control capabilities at lower noise levels. As the noise increases, the model's performance gradually deteriorates, but still maintains at a certain level. Fig.17 of the supplementary also provides a good illustration of this process. When σ is 1e-2, the controllable parts like the door become blurry.

Q2. Could undesired clustering produced by DBSCAN affect the following process?

Yes. In the case of extremely close objects, the clustering effect may not be ideal. Please refer to Fig.5 of the manuscript, the microwave and top cabinet in the scene are very close to each other, resulting in unclear clustering boundaries between these two objects. Nevertheless, clustering centers still fall within the dynamic range of the corresponding objects, and FreeGaussian achieves satisfying rendering results in the control stage. The robustness of FreeGaussian comes from its polling operation, which weights all Gaussian trajectories within a control region, rather than treating each as an independent control variable. The rendering metrics in Tab.5 of the manuscript support its effectiveness.

Thank you for the explanations. Following up on the issue with DBSCAN instabilities since using DBSCAN for initializing the controls is one of the main contributions yet the current explanation does not thoroughly cover the pros and cons of this design choice. As many prior works on point cloud clustering have pointed out, the hyperparameter tuning for the DBSCAN is a major issue since it depends on factors like scene scale, local densities, and local manifold geometry. How is the hyperparameter tuning being handled in your approach? Were the hyperparameters manually tuned for the experiments or was there any tuning method employed and tailored for Gaussians? Besides DBSCAN there are also many other clustering methods, as explored in other GS control/editing frameworks like Di-MiSo - can you explain why DBSCAN is selected over alternatives and how it might be more suitable over alternatives for real/synthetic scenes?

Thanks for your time and effort in reviewing our work. We want to kindly follow up on the review process and greatly appreciate your feedback. Look forward to your response.

Q5. The evaluation dataset and the qualitative results on real dataset.

In our experiments, we employed three datasets, namely CoNeRF consisting of Synthetic and Controllable subsets, OmniSim and InterReal, where CoNeRF Controllable and InterReal are real-world datasets, and CoNeRF Synthetic and OmniSim are synthetic datasets. Please refer to [2] and [3] for detailed information about the datasets.

For CoNeRF Controllable and InterReal datasets, Tab.1 and Tab.3 of the manuscript provide the corresponding quantitative metrics. Additionally, Fig.7 of the supplementary illustrates the visualization results on the InterReal dataset, demonstrating that our model can still reconstruct complex controllable scenes in real-world scenarios.

The following table reports the PSNR metric between our method and others on the DyNeRF dataset. Our results are comparable to Gao's, but significantly outperform other methods.

Fig.13 and Fig.15 of the supplementary show more qualitative results on CoNeRF Controllable and DyNeRF datasets. As shown in Fig.13, compared to other methods, ours has reconstructed more details than the current SOTA methods. For DyNeRF dataset, as shown in Fig.15, our method also has good control over the motion regions. Results demonstrate that our method has certain advantages in real-world scenarios.

Moreover, we have reproduced Gao’s method since the code is not currently available. We compared it with ours on the #seq001, #seq004, and #seq015 subsets of Omnisim, as shown in the Tab.8 of the supplementary. From the results, it is evident that in scenarios containing both camera motion and object motion, the advantages of our method become apparent. By decomposing the optical flow, FreeGaussian isolates the flow generated solely by the object’s intrinsic motion, enhancing the rendering effect while enabling precise control over the object. In contrast, Gao’s method treats camera motion as object motion, resulting in suboptimal overall optimization that does not significantly surpass those of traditional 4D methods. Furthermore, Gao’s method is unable to achieve individual control over objects in multi-object reconstruction scenarios, highlighting another key limitation compared to our approach.

Q6. Does the datasets used in the paper contain videos of dynamic objects and scenes? How does the baseline work?

Exactly, all our datasets[2][3] involve capturing dynamic objects with complex camera motions. Please refer to anonymous link in Sec 9 of the supplementary for more demos.

In our experiments, we primarily compared three types of methods: static methods, 4D deformable methods, and controllable scene reconstruction methods. As for the static method like NeRF, the model is bound to fail and cannot complete the task, and we included them in the comparison to evaluate their reconstruction metrics for static backgrounds. Regarding the 4D methods, the model can control the scene in a temporal order along the time axis, but they cannot achieve control over individual objects. In contrast, the controllable methods require the model to not only model the static background accurately but also to control different objects independently. For a static scene, dynamic gaussian flows always become zeros, as illustrated in Fig.3(a) of the manuscript. In this case, Eq.(3) degenerates to $\mathbf{u} = \mathbf{u}^\text{Cam}$ , which is a simpler setting that not only preserves algorithmic accuracy but also enhances stability. This observation further validates our insight into highlighting dynamic regions with dynamic Gaussian flow.

We are deeply grateful for the reviewer's comments and hope that our response can clarify any misunderstandings regarding the motivation, novelty, and dataset, and look forward to a fair and objective evaluation.

Reference

[1] GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation

[2] CoNeRF: Controllable Neural Radiance Fields

[3] LiveScene: Language Embedding Interactive Radiance Fields for Physical Scene Rendering and Control

[4] Subspace methods for recovering rigid motion I: Algorithm and implementation

Q4. The technical novelty of flow derivation and assertion of neglecting rotation of 3D Gaussians.

Our dynamic Gaussian flow analysis introduces a novel formulation that models both camera motion and dynamic scene motion at the Gaussian ellipsoid level. Unlike Gao et al., who assume a static camera and focus solely on Gaussian motion, our approach derives the optical flow relationship among the observed optical flow $\mathbf{u}$, camera-induced flow $\mathbf{u}^\text{Cam}$, and dynamic Gaussian flow $\mathbf{u}^\text{GS}$. This comprehensive modeling allows us to decouple the optical flow into components caused by camera motion and by the object’s intrinsic motion, leading to a more accurate and generalizable solution. Our rigorous derivation under instantaneous motion assumptions is detailed in Eq.(9) to Eq.(16) of the manuscript and further elaborated in Sec 6 of the supplementary.

While Gao et al. primarily address translation, our method incorporates Gaussian rotation through Eq.(10). Since the self-rotation of isotropic Gaussians does not affect the projection of their centers onto the 2D plane, it is neglected in our formulation. The rotation and translation velocities are defined using Poisson’s equation, as shown in our derivations: $\mathbf{R}^{\top}(t)\dot{\mathbf{R}}(t) = [\boldsymbol{\omega}]_{\times}$, $\mathbf{R}^{\top}(t)\dot{\mathbf{T}}(t) = \boldsymbol{v}$, and $\mathbf{R}^{\top}(t)\dot{\mathbf{T}^\text{GS}}(t) = \boldsymbol{v}^\text{GS}$.

We must emphasize that, the main idea of Gao et al.[1] differs significantly from our proposed dynamic Gaussian flow analysis in terms of motion formulation, generality, and motivation. Below are additional explanations in task and method that contribute to the novelty of our approach.

• Gao et al. consider a static camera and do not account for camera motion in their flow derivation. In contrast, our method explicitly models both camera rotation and translation, as well as the rotation and translation of 3D Gaussians. This allows our approach to handle scenarios with significant camera movement, making it more versatile in practical applications.
• Our approach remains effective in dynamic scenes with camera motion, whereas Gao et al.’s method is limited to static camera setups. In scenarios without camera motion, their method can be viewed as a special case of our more general framework.
• By modeling the complete motion process at the Gaussian ellipsoid level and considering both camera and object motions, our method achieves higher accuracy in optical flow estimation. The inclusion of the residual term $\Delta$ in Eq.(9) and our ablation studies (Fig.6) demonstrate the improved performance over methods that do not consider these factors.
• The proposed loss function in Eq.(7) is just one component of our innovative contributions. As discussed in Lines 235 and Line 242 of the manuscript, our method leverages dynamic Gaussian flow to explore dynamic Gaussians of interactive objects and extract their trajectories for joint optimization. This enables the restoration of scene interactivity and provides continuous motion constraints for dynamic Gaussian optimization.

We hope this detailed explanation clarifies the technical novelties and key contributions of our work compared to Gao et al. Our method’s ability to model both camera and object motions, consider Gaussian rotations, and automatically discover dynamic objects represents significant advancements in the field. We kindly encourage you to review the relevant sections of our manuscript and supplementary materials for a comprehensive understanding. Thank you for your consideration.

Q1. The purpose of factorizing 2D optical flow and it’s benefit compared to Gao et al.[1].

We must emphasize that our task is to reconstruct a multi-object controllable scene and control the objects within the scene. All evaluation sequences contain both camera motion and object motion simultaneously, thus the 2D optical flow consists of two components: one caused by camera motion and the other by object motion itself, as illustrated in Fig.3 of the manuscript. Decomposing the optical flow allows us to obtain the changes solely caused by object motion, which serves as the input for subsequent control. In contrast, setting of Gao et al., is relatively simple, with a fixed camera and only a single moving object in the scene, which is suitable for traditional 4D reconstruction. It would fail to control the scene when camera motion is involved, as it mistakenly treats camera motion as object motion. Therefore, the Gaussian flow in Gao et al 2024 is actually a special case of ours. By correctly decomposing camera and object motion, FreeGaussian is able to reconstruct controllable scene and achieve independent object control. Please refer to Q4 for more novelty comparisons.

Q2. Details about DBSCAN algorithm and the usefulness of dynamic Gaussian flow (2D scene flow).

We employ the Euclidean distance between center of 3D Gaussians as the distance metric in DBSCAN algorithm. as mentioned in Line 304. More detailed implementation could be found in Sec 4 Implementation details of the manuscript.

As previously explained, dynamic Gaussian flow solely captures the object motions. We leverage this flow to identify the 3D Gaussians associated with the objects. Specifically, after decoupling the dynamic Gaussian flow, a flow map is constructed from it, see Fig.3(b) of the manuscript for details. Subsequently, through Gaussian rasterization, we establish a correspondence between the 3D Gaussians and pixels in the image. The flow map then serves as a mask, enabling us to determine the 3D Gaussians matched to the current dynamic Gaussian flow via a splatting process. Once we have identified which Gaussians in 3D space related to the current motion, we cluster the relevant Gaussian ellipsoids, thereby obtaining different cluster centers, i.e., different objects.

Q3. The definition of model $\Theta$ and $E$.

In Eq.(6) of the manuscript, $E$ is an encoder which encodes the cluster trajectory $\mathbf{v}_c^i$  as well as the coordinates of 3D Gaussians with learnable Hash Grids. The model $\Theta$ is an additional network that maps the 3D Gaussians and the trajectory to variation of control vector, which is used in later control process.

Thanks for your time and effort in reviewing our work. We want to kindly follow up on the review process and greatly appreciate your feedback. Look forward to your response.

Thank you for your thoughtful review. The goal is to manipulate individual objects independently within the dynamic scene, not just to perform motion decomposition and control.

Independent object control requires identifying the association between static background and dynamic objects for manipulation. Hence, we employ optical flow prior to establishing this association, which captures dynamically changing objects in the scene. By motion decomposition, optical flow is decoupled into object motion and camera ego motion, isolating the influence of static background Gaussians. The object motion derived from optical flow represents all moving objects in the scene and is further separated by clustering, ultimately achieving independent control through spherical vector controlling scheme in Sec 3.3.

The mentioned 4D reconstruction methods[1][2] model Gaussian ellipsoids in 3D space and replay the motions of all Gaussians over time. However, they fail to separate individual objects from the dynamic scene. Directly using 3D Gaussian motion for control fundamentally cannot achieve independent object control.

Besides, the expression “world space 2D motion = optical flow (which is camera space 2D motion) + projected 3D motion” is ambiguous. Instead, optical flow on the image plane is composed of the projected 3D motion and camera ego-motion, i.e., optical flow = projected 3D motion flow + camera ego-motion flow, which was first introduced in [3].

Therefore, 2D optical flow decomposition is essential to achieve independent control. We kindly suggest that the reviewer refer to Fig.9 and Fig.10 of the supplementary, as well as the project demo in Sec 9, for visual demonstrations of the task and the proposed method's effectiveness.

[1] MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds.

[2] Shape of Motion: 4D Reconstruction from a Single Video.

[3] Subspace methods for recovering rigid motion I: Algorithm and implementation

I've read the response provided by the authors.

If the goal is motion decomposition and control, we could do it in 3D, e.g.,world space 3D motion = camera space 3D motion + camera motion. This is part of existing dynamic 3D gaussian methods, where the 3D motion is typically represented in the world space (or the camera space of the first frame), given camera pose with regard to the scene is often known [A, B]. If I understand correctly, the introduced 2D motion decomposition, world space 2D motion = optical flow (which is camera space 2D motion) + projected 3D motion due to camera, is unnecessary and complicates the problem.

While the response cleared my concerns about the evaluation and writing, this issue is unresolved and therefore I'm leaning towards a reject rating.

[A] MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds.

[B] Shape of Motion: 4D Reconstruction from a Single Video.

Q4. The detailed design on the specific activation functions and output processing used of Eq.(6).

We have modified Eq.(6) of the manuscript to $\boldsymbol{f}_{\Theta}(\mathbf{E}(\mathbf{v}_c^j, \mathbf{X}_i)) \mapsto \left \langle \Delta\mathbf{X}_i, \Delta\mathbf{\Sigma}_i \right \rangle$, which describes the mapping relationship between the 3D spherical vector and the motion trajectory of the object's Gaussian ellipsoids. During the training phase, we track the dynamic Gaussian trajectory, and encode the trajectory $\mathbf{v}_c^i$ , and Gaussian coordinates using Hash encoding. The encoded features are then fed into a lightweight MLP $\Theta$, which predicts the offsets of the Gaussians. The hidden layers of the MLP utilize ReLU as an activation function. In the control phase, a manual 3D spherical vector as well as the 3D Gaussians are input to the network and the dynamics of the 3D Gaussian ellipsoids are predicted. Finally, based on the prediction, we query the original motion trajectory as the final output.

Q5. Does the inconsistency in the assumption of Gaussian isotropy between the two stages impact the subsequent performance?

By following existing works[1][2][3], we assume isotropy between two consecutive frames, solely to simplify the loss term, enhance numerical stability, and accelerate back-propagation. However, under instantaneous motion, the Gaussian ellipsoid changes minimally, resulting in a similarly small difference as $\Delta\mathbf{u}^\text{GS} = \| \mathbf{u}^\text{GS} - \mathbf{\tilde{u}}_i^\text{GS} \|$. Although providing a clear analysis from a numerical perspective is challenging, we have implemented the complete loss function in CUDA and set up several experiments to compare the actual effects. The experimental results show that using the complete loss function significantly increases the computation time, but does not yield a notable improvement in rendering quality. Therefore, it is reasonable to consider ignoring $\Delta\mathbf{u}^\text{GS} = \| \mathbf{u}^\text{GS} - \mathbf{\tilde{u}}_i^\text{GS} \|$. Furthermore, we agree that introducing the concept of isotropy is only partially accurate, and we plan to replace this isotropy assumption with a detailed experimental analysis.

Q6. Gradients backpropagation to the linear transformations A and B. More details about the CUDA implementation.

Please recall the pipeline in Line 115 and CUDA implementation in Line 787, we aim to optimize the Gaussian and neural network through back-propagation using $\mathcal{L}_\text{uGS}$. Note that in Eq.(3) and Eq.(4) of the manuscript, although the linear transformations A and B are already present in the loss function, they do not directly participate in the optimization. Instead, we decouple the Gaussian flow via Eq.(3), which serves as the ground truth in the loss function calculation. Furthermore, the challenge in CUDA optimization does not lie in computing the Jacobians of A and B. Rather, it is to establish a mapping relationship between image pixels and 3D Gaussians, enabling the retrieval of all relevant Gaussian ellipsoids through pixel coordinates, thereby ensuring that the optical flow loss can be computed and propagated.

We sincerely thank the reviewer, one of the few who have carefully read and thoughtfully commented on our paper. We look forward to an improved and objective score, free from misunderstandings.

Reference

[1] SplaTAM: Splat Track & Map 3D Gaussians for Dense RGB-D SLAM

[2] Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models

[3] GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation

Q1. Could there be instances where the perceived region is incomplete? Could this impact the method’s effectiveness?

In all our evaluation sequences, although certain objects may be occluded from specific viewpoints, the moving regions are always complete when considering the entire video sequence. To investigate our model's performance under incomplete moving region conditions, we conduct a new experiment.

As illustrated in Fig.16(a) of the supplementary, from the illustrated camera view, the microwave is occluded by the cabinet. In this case, the perception of the microwave through the decomposed dynamic Gaussian flow is relatively weak, leading to poor clustering results, as shown in Fig.16(b), where only a part of the microwave is obtained. When we move the camera to a viewpoint where the microwave is visible, the rendering result is poor, with only partial regions undergoing motion, see Fig.16(d).

However, when the training sequences include both incomplete (Fig.16(a)) and complete (Fig.16(e)) viewpoints, a good clustering result for the microwave could be obtained, as shown in Fig.16(g), where the overall geometric structure of the microwave is relatively clear. The rendering result is satisfactory in novel view synthesis (Fig.16(h)).

This demonstrates that FreeGaussian can correctly control and render in novel views even when the training includes incomplete moving regions, showcasing its robustness and superiority over current Nerf-based and Splat-based methods.

Q2. Could DBSCAN result in inaccurate clustering when objects are closely spaced?

Yes. In the case of extremely close objects, the clustering effect may not be ideal. Please refer to Fig.5 of the manuscript, the microwave and top cabinet in the scene are very close to each other, resulting in unclear clustering boundaries between these two objects. Nevertheless, clustering centers still fall within the dynamic range of the corresponding objects, and FreeGaussian achieves satisfying rendering results in the control stage. The robustness of FreeGaussian comes from its polling operation, which weights all Gaussian trajectories within a control region, rather than treating them as an independent control variable. The rendering metrics in Tab.5 of the manuscript support its effectiveness.

We conduct an additional experiment to illustrate the the parameter setting and the effectiveness of DBSCAN, as shown in Fig.19 of the supplementary. The experiment is conducted on #seq001 of the OmniSim dataset, with a cabinet and microwave close to each other in a small-scale kitchen. Fig.19 Line 1371 shows the clustering results of DBSCAN under different hyperparameters, as well as comparisons with KMeans and MeanShift. In the experiment, a fixed ratio (typically 10%) of the total points is set to limit the cluster size, which ensures the clusters are of manageable and adaptive size. The results in Fig.19 (e)-(h) demonstrate that, DBSCAN exhibits remarkable stability and accuracy, even when the hyperparameters fluctuate from (eps = 0.01, min_samples=5) to (eps = 0.1, min_samples=25). DBSCAN effectively captures the geometry of objects and obtains accurate outliers, with clear contours for the dishwasher and fridge.

Q3. The optimization for pre-trained Gaussians in the first stage lacks sufficient geometric priors, potential impact DBSCAN accuracy and the final result.

Notably, Gaussian Pre-training aims to perceive the dynamic regions in the scene, i.e., the controllable objects in the scene. For dynamic scene reconstruction, 4DGS methods still achieve high-quality reconstruction without prior knowledge, such as depth or point cloud with geometric shapes, making it satisfying for FreeGaussian’s first-stage training. For more complex motions, we introduce the Dynamic Gaussian Flow Loss in Sec 3.4 of the manuscript, which utilizes optical flow and camera flow as priors to achieve more robust and accurate modeling capabilities. Dynamic Gaussian Flow Loss helps to smooth the underlying 3DGS optimization process and further enhances the 4D scene reconstruction quality of 4DGS. The ablation in Sec 4.4 supports this.

Besides, in the second stage, FreeGaussian weights all Gaussian trajectories within a control region, rather than treating them as an independent control variable, which improves robustness, and mitigates DBSCAN clustering inaccuracies. Finally, we must emphasize that joint training in the second phase optimizes not only 3D spherical vectors but also 3D Gaussians and Hash Features, alleviating the problem of training in the first stage.

Thanks for your time and effort in reviewing our work. We want to kindly follow up on the review process and greatly appreciate your feedback. Look forward to your response.