Dynamic Gaussian Splatting from Defocused and Motion-blurred Monocular Videos

(Q1) The current design resembles a standard 3DGS pipeline followed by a separate blur modeling network, rather than a deeply integrated solution.

Indeed, our framework is not highly integrated in architecture. This is mainly because we adopt kernel-based post-processing to achieve unified modeling of both motion and defocus blur. Though our method is not a deeply integrated solution, it still outperforms previous methods, demonstrating its effectiveness.

(Q2) Clarify how adding an intensity output to the blur kernel network alone prevents it from deforming jointly with the 3DGS representation.

By adopting an intensity output, we are able to identify the blurry and sharp regions in the blurry input image, with which we can enforce the blur kernel network to focus on the blurry regions and also use the sharp regions to constrain the Gaussian rendered image, thereby avoiding deforming jointly with the 3DGS representation.

(Q3) Why not consider using a pre-trained large model to simulate blur instead of training a network together with 3DGS, which might avoid the deformation issue altogether?

Thank you for this inspiring comment. We agree that this is a promising direction to avoid the deformation issue. However, existing pre-trained large models basically recover sharp images from blurry inputs in an end-to-end manner without explicitly modeling blur. Although we have tried our best to fine-tune some pre-trained models (Restormer [1] and DiffBIR [2]) to estimate the blur kernel, we failed to obtain satisfactory results within the rebuttal period. We will keep working on this direction in our future work.

(Q4) A straightforward idea would be to first apply a network to generate the blurred image, then feed the clear image to 3DGS for reconstruction. Could the authors elaborate on the main advantages of the proposed coupled approach compared to such a decoupled pipeline?

The main advantage of our approach is that our method can obtain spatiotemporally consistent scene representation (see Table 2 and Figure 4 in the paper, as well as Figure 2 in the supplementary material for comparison), which is hardly achieved by the decoupled pipeline. The reason is that the decoupled pipeline performs deblurring in 2D image space without considering the 3D information of the scene, and thus there usually exists noticeable spatiotemporal inconsistencies between the deblurred frames, which in turn cause spatiotemporal inconsistency in 3DGS reconstructed scene.

References

[1] Zamir, S. W., Arora, A., Khan, S., Hayat, M., Khan, F. S., & Yang, M. H. Restormer: Efficient Transformer for High-Resolution Image Restoration. CVPR 2022.

[2] Lin, X., He, J., Chen, Z., Lyu, Z., Dai, B., Yu, F., ... & Dong, C. DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior. ECCV 2024.

(Q1) Line 82, what the number 20 specifically represents in R20.

The number 20 denotes the total number of SE(3) motion bases. We will clarify this in the paper.

(Q2) How the dynamic Gaussian achieves scaling transformation in Formula 2.

Note, we do not perform scaling transformation of dynamic Gaussian in Formula 2, as we aim to achieve dynamic Gaussian deformation in Formula 2. Therefore, we only adjust the pose parameters (position and rotation).

(Q3) The meaning in Formula 3 is not explained.

The goal of Formula 3 is to obtain the center of the densified dynamic Gaussian by performing depth reprojection of each pixel in a frame to its corresponding 3D point location.

(Q4) How the Image, Depth, and Mask in the right part of Figure 2 obtained, and how the Blur-aware Sparsity Constrain performs backpropagation.

The mentioned image, depth, and mask are obtained by rasterizing both dynamic and static Gaussians. During training, the gradient of blur-aware sparsity constraint flows to BP-Net through the learnable blur kernel, thereby allowing backpropagation.

(Q5) No explicit justification is provided for the chosen blur kernel size.

In addition to the visual justification in Figure 7, we in the table below further perform quantitative evaluation on how different blur kernel sizes (denoted as K) affect the performance of our method. As shown, a larger blur kernel helps to obtain better results. However, this trend becomes less obvious when K is larger than 9. To balance the performance and the computational cost, we thus choose K=9 as our default choice. We will add the table to the supplementary material.

(Q6) The physical meaning of "blur intensity mx" is not defined.

The physical meaning of $m_x$ is the degree of blur of a pixel $x$, and a larger $m_x$ indicates a blurrier pixel.

(Q7) The paper fails to explain why blur-aware sparsity constraints can stabilize mx.

Thank you for careful review. In fact, what we intend to express in Line 162-163 is that we adopt the blur-aware sparsity constraint $L_{spa}$ for training only when the blur intensity $m_x$ has been trained for a number of $N_{spa}$ iterations to be stable, rather than claiming that the constraint can stabilize $m_x$. We will revise the paper to clarify this.

(Q8) How is "restrict the selection to unseen views close to the training views" implemented?

We implement this by generating two types of unseen views: (i) parallel-unseen views: generated by interpolating between adjacent training views along the camera trajectory, (ii) perpendicular-unseen views: generated by first computing a local perpendicular direction to the camera trajectory and then perturbing the training view’s camera center along this perpendicular direction by a distance of [0.5, 1] (normalized units). We will clarify this in the paper.

(Q9) Key hyperparameter values are not specified.

In fact, we give all implementation details in the supplementary material, including the hyperparameter values. We will move the implementation details to the paper.

(Q10) Incorrect citation of the dataset.

Thank you for careful checking. We will address the issue in the revised paper.

(Q11) Line 3: "diference" is misspelled (should be "difference").

We will correct this typo.

(Q12) Using dashed lines in Figure 2 to represent the Gradient Flow.

We will update Figure 2 according to your suggestion.

References

[1] Luo, X., Sun, H., Peng, J., & Cao, Z. Dynamic neural radiance field from defocused monocular video. ECCV 2024.

[2] Sun, H., Li, X., Shen, L., Ye, X., Xian, K., & Cao, Z. Dyblurf: Dynamic neural radiance fields from blurry monocular video. CVPR 2024.

Dear reviewer qGyt,

Please be reminded that you have to finish the Mandatory Acknowledgement after reading the rebuttals. Please also kindly provide additional feedbacks on whether your concerns are addressed by the rebuttals.

Thanks, AC

(Q1) Unclear clarification of technical contributions.

Our technical contributions are twofold: (i) we present the first unified framework for dynamic Gaussian Splatting from both defocused and motion-blurred monocular videos, (ii) we develop an effective blur prediction network and introduce a dynamic Gaussian densification as well as an unseen view combined scene optimization scheme. Note, we do not consider the off-the-shelf tools like SAM, Depth-Anything, and TAPIR as our contributions. We will clarify this in the paper to avoid possible misunderstandings.

(Q2) Implementation details are omitted in the main manuscript.

Thank you for pointing this out. As described in the Paper Checklist, we provide the full implementation details in the supplementary material. We will move the implementation details to the paper to facilitate reading and understanding.

(Q3) Ablation study of the BP-Net.

Thank you for the constructive comment. In the table below, we conduct ablation study on the BP-Net, where we compare our method with two alternatives that replace BP-Net with the motion blur modeling approach introduced in Deblur4DGS and the defocus blur modeling method introduced in D2RF, respectively. As can be seen, our method with the proposed BP-Net produces better results than the other two alternatives, demonstrating the effectiveness of the BP-Net. We will include the ablation study in the paper.

(Q4) Citations and references are overly abstract.

We will improve the citations and references according to the comment.

(Q5) Comparing generated and estimated kernels to examine the effectiveness of BP-Net in predicting different types of blur.

Thanks for the comment. In the table below, we conduct comparison between generated and estimated kernels. To this end, we construct two datasets with ground truth blur kernels, referred to as D2RF-v2 and DyBluRF-v2, by randomly sampling two global Gaussian and linear distribution blur kernels of size 9 x 9 and then respectively applying them to the ground truth sharp images in D2RF [1] and DyBluRF [2] to obtain the corresponding blurry images. With the two datasets, we quantitatively compare our estimated blur kernels and the ground truth blur kernels using PSNR and KL divergence as metrics. As shown below, our estimated blur kernels are highly similar to the ground truth blur kernels in numerical metrics, manifesting the effectiveness of the BP-Net in predicting different types of blur. We will add the table and also visual comparison of blur kernels to the supplementary material.

(Q6) What if the two types of blurs occur jointly? Can the proposed method also handle this type of degradation?

Thank you for the insightful comment. In the table below, we evaluate our method on two datasets (DyBluRF-v3 and D2RF-v3) with both motion and defocus blur occurring simultaneously. Note, we obtain the DyBluRF-v3 dataset by applying depth-of-field (DoF) rendering technique in Bokehme [3] to the original DyBluRF dataset [2] to simulate defocus blur, and create the D2RF-v3 dataset with motion blur by processing the original D2RF dataset [1] using the motion blur generation method in Davanet [4]. As shown, our method clearly outperforms all the compared methods on the two datasets, verifying the advantage of our method in handling cases with motion and defocus blur occurring jointly.

References

[1] Luo, X., Sun, H., Peng, J., & Cao, Z. Dynamic neural radiance field from defocused monocular video. ECCV 2024.

[2] Sun, H., Li, X., Shen, L., Ye, X., Xian, K., & Cao, Z. Dyblurf: Dynamic neural radiance fields from blurry monocular video. CVPR 2024.

[3] Peng, J., Cao, Z., Luo, X., Lu, H., Xian, K., & Zhang, J. Bokehme: When neural rendering meets classical rendering. CVPR 2022.

[4] Zhou, S., Zhang, J., Zuo, W., Xie, H., Pan, J., & Ren, J. S. Davanet: Stereo deblurring with view aggregation. CVPR 2019.

(Q1) The numerical increase in the quantitative evaluation seems small.

As shown in Table 1 in the paper, our numerical performance improvement over current SOTA method (Deblur4DGS) is evident on the PSNR$\uparrow$/SSIM$\uparrow$/LPIPS$\downarrow$ metrics (defocus blur: 29.39/0.859/0.078 vs. 28.49/0.791/0.154, motion blur: 27.01/0.876/0.056 vs. 26.62/0.871/0.059), especially considering that our method has significantly fewer parameters than Deblur4DGS (192.2M vs. 754.6M). Moreover, we would like to point out that, our work makes the first attempt towards developing a unified framework for dynamic Gaussian Splatting from both defocused and motion-blurred monocular videos, which we believe is more important than the performance improvement.

(Q2) Motion blur still exists in the provided videos.

Despite the advantage over the compared methods, some of our provided videos do suffer from weak motion blur, because it remains a challenge for our method as well as other SOTA methods to achieve perfect dynamic object reconstruction for complex cases, which is also illustrated in our limitation analysis in Figure 8 in the paper.

(Q3) More analysis on how errors from external preprocessing steps (depth estimation, 2D point tracking, and SAM) affect the results.

Thank you for the suggestion. In the table below, we quantitatively evaluate how errors from external preprocessing steps affect the robustness of our method. To simulate errors from depth estimation, we randomly scale and shift the estimated depth maps within the range of [0.8, 1.2] and [-20, 20], respectively. To simulate errors from 2D point tracking and SAM, we randomly shift the 2D tracking points within the range of [-30, 30], and randomly add or delete five 25 X 25 mask regions towards the mask predicted by SAM. As shown, our results produced with the artificially perturbed depth, tracking points, and motion mask are comparable to those produced with the originally estimated depth, tracking points, and motion mask, indicating that our method has some tolerance to errors from external preprocessing steps.

(Q4) More discussion on cases where defocus and motion blur may not be accurately captured by the same convolutional abstraction.

For complex cases that involve fast-moving objects in defocused background, the convolutional abstraction may struggle to accurately model defocus and motion blur. We will add some examples of such cases to the supplementary material.

(Q5) Will it be more effective and easier to train if the motion and defocus blur are estimated separately (e.g. deblurring 3d gaussian splatting, ECCV 2024)?

We have followed your suggestion to train our method by separately estimating motion and defocus blur as done in the mentioned ECCV 2024 paper [3]. As shown in the table below, adopting the separate modeling strategy results in clear performance drop on all the three metrics, manifesting the effectiveness of our unified modeling of motion and defocus blur. We will include the table in the paper.

(Q6) More evaluation on some static blurry datasets.

Thank you for the suggestion. In the table below, we compare our method with a recent method (Deblurring 3D Gaussian Splatting [3], ECCV 2024) tailored for static blurry images on the Deblur-NeRF dataset [4]. As shown, our method consistently outperforms the compared method (referred to as Deblurring-GS) on both motion and defocus blur, validating the effectiveness of our method in dealing with static blurry images. We will add the table to the paper.

(Q7) More analysis on computational cost and memory demands.

Our model requires an average of 6GB GPU memory for both training and inference. During training, the BP-Net dominates the computational cost with a ratio of 40%, while the dynamic Gaussian deformation and rasterization respectively occupy 25% and 15% of the computational cost, and the rest of 10% is for loss computation. During inference, the computational cost only comes from dynamic Gaussian deformation and rasterization, with the former at 70% and the latter at 30%.

(Q8) Comparison with MoBGS.

Thank you for the comment. Since the implementation of MoBGS [5] is not yet publicly available, we compare our method with MoBGS by training on the same motion blur dataset as MoBGS in terms of PSNR$\uparrow$/SSIM$\uparrow$/LPIPS$\downarrow$ on the full region. The comparative results between our method and MoBGS are 28.61/0.942/0.059 vs. 28.70/0.945/0.051 (the results of MoBGS are from their paper), showing that our method can produce comparable results to MoBGS on motion blur, even though MoBGS is specially designed for motion blur while our method can effectively handle both motion and defocus blur.

References

[1] Luo, X., Sun, H., Peng, J., & Cao, Z. Dynamic neural radiance field from defocused monocular video. ECCV 2024.

[2] Sun, H., Li, X., Shen, L., Ye, X., Xian, K., & Cao, Z. Dyblurf: Dynamic neural radiance fields from blurry monocular video. CVPR 2024.

[3] Lee, B., Lee, H., Sun, X., Ali, U., & Park, E. Deblurring 3d gaussian splatting. ECCV 2024.

[4] Ma, L., Li, X., Liao, J., Zhang, Q., Wang, X., Wang, J., & Sander, P.V. Deblur-nerf: Neural radiance fields from blurry images. CVPR 2022.

[5] Bui, M. Q. V., Park, J., Bello, J. L. G., Moon, J., Oh, J., & Kim, M. MoBGS: Motion Deblurring Dynamic 3D Gaussian Splatting for Blurry Monocular Video. arXiv 2025.