Splatter a Video: Video Gaussian Representation for Versatile Processing

We appreciate the reviewer's valuable feedback and address the concerns raised.

Training Speed: While our method's training time relative to video duration is lengthy, it demonstrates significantly faster training speed compared to other methods, as shown in Table R1. We also report superior GPU memory and rendering efficiency (FPS). It's worth noting that once trained, these models can support various applications. Moreover, the speed may also potentially benefit from hardware advancements or parallel processing techniques.

Overestimation of Gaussians?: This query touches on the different frequencies between RGB and feature images. While features with higher semantic content may skip low-level signals, our current approach didn't process these signals specially but focused on a universal video processing representation, which might lead to a over-parameterized Gaussians for features. Although a finer Gaussian representation might seem excessive, it prevents information loss and maintains sharp segmentation contours, ideal for handling the dense information in RGB images.

About SAM Feature: The SAM feature appears smoother because it's interpolated from a lower-resolution tensor to full image resolution, whereas our rendering feature maintains the original resolution. Any high-frequency details smoothed during this process do not affect the final mask quality, as shown explicitly in Fig. R4 of the attached pdf file. Although SAM is more generalized, our method ensures better object integrity and segmentation quality across video frames, with more efficient processing time by skipping the image encoding step. The comparison is reported below:

Sensitive to learning rate?: All the videos share the same hyperparameters including loss weights and learning rate. We didn’t tune the parameters for different videos. Actually, most of the learning rates in Appendix Table 2 are derived from 3DGS.

Failure cases: We identify two scenarios where our method may underperform compared to dynamic GS methods or exhibit suboptimal performance, illustrated in Fig R7.

• The first scenario involves videos characterized by significant camera motion with few moving objects (e.g. elephant and cow in Table 1 of the paper). In these instances, the scene resembles a static environment where tools like COLMAP can effectively estimate precise camera poses. Dynamic GS methods model background motion using cameras extrinsic, whereas our method learns background trajectory during optimization. This approach imposes additional complexity on our training process, as our method must handle the bundle adjustment typically managed by COLMAP. In contrast, dynamic GS methods can fully use the camera prior, which is simpler for them to process.
• The second scenario involves videos with transient moving objects that appear only briefly, without continuous motion. In these cases, the estimated flow tends to be inaccurate, making it challenging for our method to learn pixel correspondence solely through the photometric consistency of Gaussian rendering.

Other Questions

Resolution / FPS of dataset: We follow the Tap-Vid DAVIS benchmark (480p). The resolution of each video is 910 x 480 or 854 × 480, with a frame rate of 24 FPS.

Storage space: Our model size differs scene by scene, typically less than 100 MB, comparable with 4DGS, and much less than CoDeF (around 500M).

The trade-off between temporal smoothness & spatial details: Interesting observation. We feel there is no inherent trade-off between temporal smoothness and spatial details. This is due to the fact that natural motion patterns are often temporally smooth. Temporal smoothness helps prevent overfitting to the training data, avoiding fitting noise and thus it can even help improve spatial qualities.

Quantitative results on downstream tasks: We have evaluated the tracking results on the Tap-Vid DAVIS benchmark. Our method outperforms most existing methods, except for Omnimotion. This is because Omnimotion is specifically designed for tracking tasks and performs poorly in terms of reconstruction and other metrics, and it does not support other editing tasks. In contrast, our method supports versatile video processing tasks, with higher computational efficiency and training efficiency. We also evaluate the segmentation results with our rasterized SAM feature on the gold-fish video, compared with SAM and reported in the above table.

Difficulty in tuning hyperparameters: We have not extensively fine-tuned hyperparameters such as learning rates. Our method shows robust performance and achieves the highest performance compared with existing methods with the same hyperparameters.

Extent of novel view synthesis: Our method effectively manages view changes in the application of generating stereoscopic videos. The specific boundary of view changes is determined by the accuracy of monocular estimation. With adequate depth supervision, our method can accommodate significant view warping. This important role of depth prior plays is also highlighted in the concurrent work "Shape of Motion: 4D Reconstruction from a Single Video."

Inference speed: The inference speed of our method is a little slower than 3DGS but still fast enough for real-time applications, approaching over 150 FPS on a resolution of 854×480 on a single NVIDIA RTX3090 GPU.

I have read the new discussion and appreciate the authors' verification. I am considering increasing my score from weak accept to accept, but I can't promise anything yet. Please give me some time to reconsider the contribution part.

After reconsideration, I have decided to increase my post-rebuttal score to accept. Just to remind you, my pre-rebuttal score was borderline accept. I believe the paper deserves to be shared at NeurIPS. While I trust that the authors have the ability to reorganize the paper with the new results, I will leave the revision issue to the AC.

Thank you so much for your support in our work! We are delighted to answer the concerns raised.

Compare with 4DGS based methods: Thank you for your suggestions. We have incorporated a comparison with 4DGS [1] on Tap-Vid DAVIS benchmarks for reconstruction and tracking tasks respectively. The results are reported in Table R1 above. We also visualize qualitative results in Fig. R1 and Fig.R2 of the attached pdf file. Note that due to the inaccurate camera pose for the wild scene, the performance of 4DGS is very limited.

Regarding [2], as the method does not employ 'deformation-based' modeling but instead uses static Gaussians in 4D space, it can not be used to establish correspondences. We acknowledge both [3] and [4] as significant concurrent works and will include them in the related work section.

More ablation: We have performed an ablation study on the design of VGR by varying the hyperparameters (n and l). It is important to note that when n=0 or l=0, our representation simplifies to Fourier-only and polynomial-only, respectively. We report the results below and visualize the comparison in Fig. R6

Thank you for the suggestion and we will include a discussion on this topic to enhance our paper.

Loss weight design: The loss weights for render, depth, flow, motion regularization, and label are set to $\lambda_{render}$ = 5.0,$\lambda_{depth}$ =1.0, $\lambda_{flow}$ = 2.0, $\lambda_{arap}$ =0.1, and $\lambda_{label}$ =1.0 respectively. We have also included more ablations regarding the camera model design and each loss in Table R2 and Fig. R3 of the attached pdf file. We will add more experimental details in the final version. All of our experiments are performed with the same hyperparameters, which demonstrate the robustness of our approach.

Thank you for your valuable advice! We’d like to emphasize that using 3D representations to model and process casually captured in the wild videos, different from dynamic NeRF/GS, is a much ill-posed problem. Here, we design a system that integrates representations and regularization: 1) using a fixed orthographic camera, alleviating the cumbersome camera pose estimation. 2) distilling monocular 2D priors for regularizing learning from such ill-posed scenarios with different regularizers from mono-depth and optical flow. We have illustrated some cases in which our model perfectly handles while dynamic NeRF/GS failed in Fig.R1 of the attached pdf file.

Note that our approach doesn’t rely on the specific motion module. We adopt Fouier and polynomial bases due to their good trade-off between fitting capability and complexity. Indeed, we have experimented with other motion representations in our exploration, which also achieved good results. We will report more results and analysis on different motion representations for this task.

Moreover, the main purpose of our work is not for 4D reconstruction, the primary objective for GS, but rather to perform versatile video processing tasks in the pseudo-3D space, which is wildly demonstrated in the main paper and Fig. R5.

Predominantly qualitative experiments: As our primary intention is to show the versatility of the approach on a bunch of video processing tasks which mostly focus on visual appearance, we focus on qualitative comparisons. Here, to more thoroughly evaluate our approach quantitatively, we add more experiments on the DAVIS test videos following the TAP-Vid benchmark(480p) and report both the reconstruction and tracking metric in Table R1. It can be seen that our approach significantly outperforms other methods in reconstruction.

Unfair comparison: CoDeF is a 2D image-based video representation, where the depth prior cannot be used. In terms of Omnimotion, we observed that adding our relative depth loss results in lower-quality final results, as detailed in the table below. This is due to the depth maps in Omnimotion do not correspond to physical depth, as stated in their own paper.

Comparison with Omnimotion with Depth Prior on camel dataset.

Inadequate ablation: We conducted additional ablation studies focusing on the selection of coefficients n and l. The results are detailed below and Fig.R6. Our contribution extends beyond trajectory representation, and we have added more ablation studies regarding the camera model selection, different types of depth loss in Table.R2 and Fig.R6.

We also conducted ablation studies on the selection of camera models and depth loss formats as shown in Table R2 and Fig.R3. Using a predefined pinhole camera intrinsic led to unstable optimization, resulting in artifacts in both geometry and appearance. This instability likely stems from the rasterization process, where the denominators of Gaussians’ screen coordinates UV include depth, complicating the gradients. Replacing the shift- and scale-invariant depth loss with absolute L2 loss degrades performance, as monocular depth cues are ambiguous regarding scale and shift.

About camera motion and intrinsic: Our method employs a stationary camera, effectively merging the intrinsic motion of points with their motion relative to the camera. Consequently, all points in the scene are in motion and adhere to the trajectory formulation described in Equation 2. As stated in lines #458-461 of the paper, our method utilizes an orthographic projection camera. Orthographic projection decouples the screen coordinates of points from their depth coordinates, simplifying the optimization difficulty, as demonstrated in Fig. R6 and Table 2.

Multi-object Videos: With multi-object mask labels, our approach can be extended directly by increasing the dimension of mask feature. We evaluated the reconstruction quality of our model under two different conditions: merging the masks of different objects into a single-channel mask and increasing the mask dimensions to fit multiple objects. We conducted evaluations in two different scenarios, judo, and goldfish, as shown in the table below. The results indicate that the outcomes are essentially consistent across both settings. Additionally, we visualized examples of multi-object scene editing in Fig. R5.

We thank the reviewer for your valuable time and the acknowledgment of our writing and illustrations. We are here to respond to the listed concerns.

1. Novelty:

We would like to emphasize that our work is the first for dynamic GS without the camera dependency. Our novelty doesn’t lie in any specific designs, but in the exploration to effectively lift casual videos from pixel to a more intrinsic 3D representation.

• Comparisons with CoDeF: Our novelty lies in the exploration of representing casual videos by lifting to 3D space, which is fundamentally different from CoDeF’s 2D representation. To model a video, CoDeF uses a canonical image and a network estimating the backward flow to model a video, which limits its flexibility in representing complex videos (Fig.3 of the main paper and Table.R1) and is incapable of video processing tasks requiring 3D like novel view synthesis and stereoscopic video creation.
• Comparison with 4D-GS methods like Gaussian-Flow: Our key innovation is converting casual video pixels into intrinsic 3D representations without relying on camera pose estimation, which often fails in casually captured videos and is unachievable by current dynamic GS methods (refer to Fig.R1 and Table R1). Our approach can incorporate various motion-driving methods, though we primarily use Fourier and polynomial bases for their optimal balance of complexity and fitting ability, as discussed in our paper (Lines #135-143).
• Depth loss & 3D Motion Regularization: For depth loss, the purposes and effects of the referred tasks and ours differ significantly. Our primary insight is to use monocular 2D clues to regularize the learning of 3D representations from videos, rather than to enhance depth estimation as in MiDaS. In terms of motion regularization, we have cited related papers in lines #187-189 and did not emphasize this as a contribution. In summary, these losses themselves are not the focus; rather, it is the insights gained from employing such regularizers for learning in this ill-posed task that is crucial.

2. Method Description

• Optimization procedure missing: Basically, our optimization procedure includes the following steps: 1) randomly sample one frame from the video sequence in each training iteration to construct the loss (Equation 9), 2) the densification of Gaussians follows the same strategy as 3DGS, as stated in the line #212 of the main paper. We will release the code to the community for reproduction.
• Unexplained n/l & ablation: We chose n=l=8 in our experiments based on the ablation of varying combinations, as shown below. Please refer to the Fig.R6 of pdf for visual comparison.

3. Evaluation

• Insufficient Evaluation & Quantitative Evaluation on Dense Tracking: Thank you for your feedback. Additional experiments on the TAP-Vid DAVIS benchmark (480p) have been included, with both reconstruction and tracking metrics presented in Table.R1.

  Despite Omnimotion's specialization in tracking, our approach supports a wider array of video processing tasks with higher computational and training efficiencies. It achieves comparable results with better reconstruction quality using fewer resources. Tracking performance comparisons with similar-cost methods (CoDeF / 4DGS) are shown in Fig. R2, highlighting our superior outcomes.

4. Claim

Using the rasterized SAM feature with the SAM decoder has proven effective in image segmentation, as shown in Fig.R4. Our method ensures higher object integrity and also reduces the time needed for SAM encoding. The IoU and segmentation times, tested on an NVIDIA 3090GPU, are reported below.

5. Literature

Thank you for your kind reminder. We will recognize the pioneering contributions of the paper "Non-Rigid Neural Radiance Fields" and the SOTA NeRF-based method RoDynRF. Additionally, we will include comparison results with RoDynRF in our analysis.

6. Other Questions

• Training time & Model size: The training time is approximately 30 minutes, which is comparable to image-based methods, slightly shorter than 4DGS (about 40 minutes), and significantly less than NeRF-based methods (Omnimotion, RoDyF). We have also detailed the training times in the table above. The model size differs scene by scene, typically less than 100 MB, comparable with 4DGS and much less than CoDeF (500M).

• Loss weight: The loss weights for render, depth, flow, motion regularization, and label are set to $\lambda_{render}$ = 5.0, $\lambda_{depth}$ =1.0,$\lambda_{flow}$ = 2.0, $\lambda_{arap}$ =0.1, and $\lambda_{label}$ =1.0 respectively. We also ablate the importance of each loss and module of our method in Table R2. We will add more experimental details in the final version.

• Optimization hyperparameters: All of our experiments are performed with the same hyperparameters.

7. Limitations

• Hyperparameters impact:

  1. The impact of hyperparameters of the deformation model (n/l) has been evaluated in the Table above and Fig.R6.

  2. As for the hyperparameters of learning rate in Table 2 of the paper, most of them are derived from 3DGS. The learning rate of newly added features(dynamic, mask) is referred to original Gaussian features.

  3. The effect of each loss is also evaluated in Table R2 and Fig. R3

  All of our experiments are performed with the same hyperparameters.

• Potential failure cases: We have demonstrated some failure cases in Fig. R7. Our approach fails to handle large motions, like intense background rotation or objects that suddenly appear.