Controllable Blur Data Augmentation Using 3D-Aware Motion Estimation

References

[1] Nah et al., Deep multi-scale convolutional neural network for dynamic scene deblurring, CVPR 2017

[2] Rim et al., Real-world blur dataset for learning and benchmarking deblurring algorithms, ECCV 2020

[3] Rim et al., Realistic blur synthesis for learning image deblurring, ECCV 2022

[4] Zhong et al., Efficient spatio-temporal recurrent neural network for video deblurring, ECCV 2020

[5] Li et al., Real-World Deep Local Motion Deblurring, AAAI 2023

Dear Reviewer iVrH,

We sincerely thank the reviewer for your valuable comments and efforts. The insights have helped us clarify our contributions. We have revised our draft (highlighted in blue) in response to the insightful and valuable comments. We do our best to clarify each of the concerns carefully as follows:

---

[W1] Handle video sequences by our method.

We agree that video deblurring generally achieves higher performance than image deblurring due to the additional temporal information available. However, we wish to clarify that our primal goal is to build a controllable blur synthesizer that estimates motions from a single blur image. This enables our blur synthesizer to be directly applicable to various blur datasets (blur-sharp image pairs) such as GoPro [1], RealBlur [2], RSBlur [3], BSD [4], and ReLoBlur [5]. Therefore, extending our blur synthesizer to handle video sequences is beyond the scope of our work.

Nevertheless, to address the reviewer's concern, we compare blur trajectories estimated by our blur synthesizer with those obtained from video frames using an off-the-shelf optical flow model, e.g., RAFT [6]. Specifically, we extract the optical flow maps from the video frame images in GoPro [1] and convert the optical flow maps into vector fields to represent a real-like blur trajectory. The results are visualized in the revised draft. Please refer to Fig. 10 of the Appendix. We observe that the blur trajectories are nearly identical, confirming the alignment between our blur trajectories and the real ones. Despite estimating blur trajectories from single blur images, our method is comparable to those derived from video frames.

---

[W2] Emphasizing the term "physical" and handling occluded regions.

We agree that the term "physical" may cause confusion. Our intention was to emphasize the incorporation of 3D motion modeling, which accounts for more realistic blur patterns, as opposed to the other methods that estimate 2D non-uniform kernels. Furthermore, as discussed in Section 3.1, our method explicitly follows a camera exposure model to approximate the blurring process. Therefore, we used the term "physical" for these reasons and do not imply that our blur synthesizer is inherently more physically grounded than the synthesizer using video frames. We have revised the misleading term "physical" to "realistic" and "3D aspects" throughout our draft.

As our blur synthesizer relies on a single target image to generate a synthesized blur image, handling occluded regions within the target image remains a challenge. While the occlusion issue may arise in certain cases, we believe that our 3D-aware blur synthesizer handles the majority of motion patterns encountered in real-world scenarios. Despite using only a single blurred image, our GeoSyn demonstrates better generalization ability compared to ID-Blau [7] which leverages video frame images. Please see the Common Response 1 for detailed generalization results.

---

[W3] Blur synthesis results on object motion.

To address the reviewer's concerns, we provide some blur synthesis results on object motion using RSBlur [3] in the revised draft. Please refer to Fig. 17 of the Appendix.

---

[W4] Limited motion patterns and cross-dataset evaluation.

We appreciate the reviewer for highlighting this point. To respond to the statement "At best, the method can adapt to the motion patterns of the dataset it is trained on.", we would like to clarify that our method can leverage separate blur datasets for the blur synthesizer and the deblurring model. We chose to use the same dataset for both training to ensure a fair comparison with other methods, addressing any concerns about unfair comparisons arising from the use of additional datasets. Even though we use the same dataset in both training, our blur synthesizer can improve model generalization. Specifically, if a blur dataset contains $3,000$ blur images and the deblurring model is trained over $1,000$ epochs, it generates approximately $3,000,000$ different synthetic blur images by randomly varying blur intensities, directions, and scenes during the deblurring training. This extensive diversity enhances the generalization ability of the deblurring model, enabling it to handle a wide range of unseen blur scenarios. As suggested by the reviewer, we conducted a cross-dataset evaluation to demonstrate the generalization ability of our blur synthesizer. Please refer to Common Response $1$ for detailed generalization results.

Dear Reviewer RsRS,

We sincerely thank the reviewer for your valuable comments and efforts. The insights have helped us clarify our contributions. We have revised our draft (highlighted in blue) in response to the insightful and valuable comments. We do our best to clarify each of the concerns carefully as follows:

---

[W1] Limitation of motion trajectories.

Thank you for pointing this out. We agree that our blur trajectories are content-independent and occasionally lead to perceptually implausible blur results when the blur trajectories are applied to other scene images. Regarding non-rigid motion cases, our blur synthesizer does not fully represent non-rigid motion blur caused by shape changes (e.g., blinking eyes or folding fingers), as such motions cannot be adequately rendered by using the grid sampling based on a single target image. While these issues may arise in certain cases, we believe that our 3D-aware blur synthesizer handles the majority of motion patterns encountered in real-world scenarios. Despite using only a single blurred image, our GeoSyn demonstrates better generalization ability compared to ID-Blau [1] which leverages video frame images. Please see the Common Response 1 for detailed generalization results.

---

[W2] Computational costs.

We would like to clarify that our GeoSyn is only used for training. Therefore, it does not increase the computational cost in the evaluation stage. We have revised Section 4.1 of our draft to clarify this. To discuss it further, we provide the computational costs in the training phase, such as GMACs, model size, and training latency. The GMACs is calculated for the image size of $256 \times 256$ and the training latency is measured for one iteration with the batch size $32$ with V100 8 GPUs. As shown in the table below, utilizing GeoSyn with the network architecture of NAFNet-64 corresponds to an increase of $1.13$ GMACs, $1.0$ MB in model size, and $0.024$ seconds in training latency.

---

[Q1] Visual comparison on blur trajectories.

Thank you for the insightful comments. To address this, we utilize GoPro dataset [2], which provides video frame images that allow us to extract optical flow maps. Then, we convert these optical flow maps into vector fields to represent a real-like blur trajectory, which allows us to verify that our blur trajectory aligns well with real one. The results are visualized in the revised draft. Please refer to Fig. 10 of the Appendix. We observe that the blur trajectories are nearly identical, confirming the alignment between our blur trajectories and the real ones. Despite estimating blur trajectories from single blur images, our method is comparable to those derived from video frames.

---

[Q2] How to obtain the camera intrinsics?

The intrinsic parameters are not explicitly estimated. Instead, they are inherently embedded within the projected 3D residual vector, which is directly optimized by the neural network to estimate 3D motions. We discuss the relationship between the camera intrinsic and projected 3D residual vector in the revised draft. Please refer to Section A of the Appendix for further details.

---

[Q3] Quantitative comparison of reverted sharp and ground-truth sharp images.

Thank you for the feedback. We measure the PSNR and SSIM between the average of the reverted sharp images and the ground-truth sharp images in RealBlur-J. The results are PSNR of $41.07$ dB and SSIM of $0.9902$.

---

References

[1] Wu et al., Id-blau: Image deblurring by implicit diffusion-based reblurring augmentation, CVPR 2024

[2] Nah et al., Deep multi-scale convolutional neural network for dynamic scene deblurring, CVPR 2017

Dear Reviewer RsRS,

Thank you again for your time and efforts in reviewing our paper.

As the discussion period draws close, we kindly remind you that two days remain for further comments or questions. We would appreciate the opportunity to address any additional concerns you may have before the discussion phase ends.

Thank you very much!

Authors

[W5] Discussion on generalization performance of ours.

Thank you for pointing this out. To ensure a fair comparison with other methods, we intended to utilize separate datasets for our blur synthesizer, i.e., using the same dataset in the training of the blur synthesis model and deblurring model, addressing any concerns about unfair comparisons arising from the use of additional datasets. We would like to clarify that our method can improve model generalization even though we use the same dataset in both trainings. Specifically, if a blur dataset contains $3,000$ blur images and the deblurring model is trained over $1,000$ epochs, it generates approximately $3,000,000$ different synthetic blur images by randomly varying blur intensities, directions, and scenes during the deblurring training. This extensive diversity enhances the generalization ability of the deblurring model, enabling it to handle a wide range of unseen blur scenarios. Meanwhile, the existing method (ID-Blau [1]) produces and uses $10,000$ synthetic blur images in advance, as it relies on a diffusion model, which is computationally expensive and challenging to adapt for online data augmentation. As suggested by the reviewer, we conducted the cross-dataset evaluations. Please refer to the Common Response $1$ for detailed generalization results.

---

[W6] More comparisons of real-world blurred images.

To address the reviewer's concerns, we have added more results on real-world blur images captured by us in the revised draft. Please refer to Fig. 11 of the Appendix.

---

References

[1] Wu et al., Id-blau: Image deblurring by implicit diffusion-based reblurring augmentation, CVPR 2024

Dear Reviewer BF8H,

We sincerely thank the reviewer for your valuable comments and efforts. The insights have helped us clarify our contributions. We have revised our draft (highlighted in blue) in response to the insightful and valuable comments. We do our best to clarify each of the concerns carefully as follows:

---

[W1] Regarding the adjustment of overall blur intensity.

As discussed in Section 3.4, we introduce an amplitude control parameter $\alpha$ to control the overall blur intensity. Note that the same value of $\alpha$ is applied to all vector fields generated from a single blur image to adjust the overall blur intensity. Additionally, the value of $\alpha$ is randomly determined for each blur image. This allows us to simulate variations in blur intensity, e.g., exposure time, for the same motion trajectory. For example, when $\alpha$ is set to $2$, the amplitude of the overall vector field results in doubling the blur intensity. Meanwhile, scaling it by 0.5 reduces the overall blur intensity by half. We have already provided the results of these adjustments as shown in Fig. 1, Fig. 15, and Fig. 16. Also, we have revised Section 3.4 of our draft to provide further clarification.

---

[W2] How to derive the projected 3D residual vector.

For further details on deriving the projected 3D residual vector, please refer to Section A of Appendix in the revised draft.

---

[W3] Blur diversity and its relationship to M.

Thank you for the insightful comment. When $M$ is small (e.g., $M=4$), the expressive power of complex motion using only four camera exposures becomes inherently constrained. Namely, complex motion trajectory is represented by a simplified form, leading to a lack of blur diversity. In contrast, increasing $M$ to $16$ provides the capability to capture intricate motion patterns, leading to better blur diversity. We have provided multiple examples in the revised draft. Please refer to Fig. 9 of the Appendix. We observe that the larger number of $M$, i.e., $M=16$ yields more accurate motion results, ultimately promoting a wider range of blur patterns. Therefore, it leads to better final deblurring results as discussed in Section 4.5.

---

[W4] Performance sensitivity and its relationship to vector fields.

We appreciate the reviewer’s insightful comment regarding the correlation between deblurring performance and the estimated vector field. To address this concern, we conducted the suggested experiment by introducing random perturbations to the vector fields. We believe that the robustness of the final deblurring performance is closely tied to the geometric consistency of the vector field. As shown in the table below, without geometric consistency ($\lambda_2=0.0$), the vector field lacks geometric coherence, and thus even no perturbation leads to performance degradation ($32.52$ dB) compared to the baseline performance, i.e., no data augmentation ($32.56$ dB). In contrast, with strong geometric consistency ($\lambda_2=1.0$), the deblurring model remains robust to the perturbations of the vector field, consistently outperforming the baseline (indicated in bold). For mid-level geometric consistency ($\lambda_2=0.5$), the vector field is generally robust to small perturbations (indicated in bold), but large perturbations disrupt its geometric coherence and adversely affect performance ($32.51$ dB). These observations highlight the critical role of our geometric consistency regularization in mitigating the performance sensitivity to the accuracy of the vector field. In other words, blur data augmentation with inaccurate vector fields reduces performance gain but does not corrupt the final deblurring performance, e.g., baseline deblurring performance, as long as our motion estimator is trained under strong geometric consistency.

Dear Reviewer BF8H,

Thank you again for your time and efforts in reviewing our paper.

As the discussion period draws close, we kindly remind you that two days remain for further comments or questions. We would appreciate the opportunity to address any additional concerns you may have before the discussion phase ends.

Thank you very much!

Authors