Adaptive Window Pruning for Efficient Local Motion Deblurring

The number of shifting pixels and the necessity. In each AdaWPT block, we shift 4 tokens, which is half of the window size. The number of shifting pixels is proportional to the token size of each block. For example, for AdaWPT 1 and 9, whose resolution is $512\times512$, the number of shifting pixels is $4\times4$. Similarly, for AdaWPT 2 and 8, the number of shifting pixels is $8\times8$... For AdaWPT 5, the number of shifting pixels is $64\times64$. As the block's resolution decreases, the number of shifting pixels increases, effectively expanding the receptive field. This shifting operation aids in enlarging the receptive field of the local neighborhood, thereby enhancing the deblurring performance. Noticeably, we have trained our proposed network without the Feature Shift/Reverse layer, and the absence of these layers results in a PSNR drop of approximately 0.1dB.

The prediction accuracy on the validation set. We calculate the prediction accuracy by $(TP + TN) / N$, where $TP$ refers to the number of pixels that our model correctly predicts to be blurry, $TN$ refers to the number of pixels our model correctly predicts to be sharp, and $N$ is the total number of pixels. The mask prediction accuracies vary in different AdaWPT blocks. The highest is 94.51%. Figure 12 in the new version of our Appendix reports the accuracy of each AdaWPT block. Most AdaWPT blocks exhibit high prediction accuracies, indicating the effective recognition of local blurs by our proposed model. We have incorporated this discussion into the new version of Appendix E.6.

About the blur mask borders. We subtract the sharp and blurred images to observe the differences in local regions. This difference map helps us to determine the approximate position of the blurriness and the border. To prevent leakage, we extend the blurry region by 5 pixels, ensuring that all pixels within the annotated mask are considered blurry. Regarding pixels between blurry and sharp areas, we adhere to a principle: a diffuse spot encompassing more than 5 pixels is categorized as a blurry region, while conversely, it is classified as a sharp region. We add this discussion in the new version of Appendix C.

How abandoned patches are processed. Because the abandoned patches are sharp, they do not go through Transformer layers. They directly go to the Window Compound layer during inference. The Window Compound layer is incorporated \textcolor{red}{to integrate the abandoned windows and selected windows into a unified feature map in inference. This is explained in the new version of Section 2.2, highlighted in red.

Results of special cases (masks are all-ones/all-zeros). I guess you refer to the results of inputting globally blurred images and sharp images. When inputting globally blurred images, the masks are all one, and the decision maps are nearly in all-white color, as is shown in Appendix D and Figure 7. When inputting globally blurred images, the masks are all zero, and the decision maps are nearly in all-black color. The results of these extreme experiments demonstrate the model’s ability to effectively distinguish between blurred and sharp images, showcasing its robustness. We add the results and discussion in the new version of Appendix E.7, highlighted in red.

Results under different image resolutions. In the initial version of our paper, we presented results based on large images due to the ReLoBlur testing data exclusively comprising images with substantial resolutions. In this revision, we augment our findings by including results for various resolutions in the subsequent chart. Employing the nearest interpolation, we down-sample the ReLoBlur testing data and conduct comparisons between our proposed model and baselines using both middle-resolution and small-resolution images. The results depicted in the following chart demonstrate that our proposed LMD-ViT outperforms other methods across all resolutions, delivering rapid and efficient inference. This discussion has been incorporated into the updated version of Appendix E.4.

[1] Zamir et al, Restormer, CVPR 2022

[2] Wang et al, Uformer, CVPR 2022

About the organization and writing. Thank you for your advice. To improve the writing of our paper, we have made several modifications:

1. We have removed the second paragraph from Section 1 and incorporated discussions about our novelty in the same section.

2. We have consolidated the explanations for AdaWPT-F and AdaWPT-P under a unified term, AdaWPT, in Section 2. Accordingly, we've updated Figure 2 to reflect this change without differentiating between AdaWPT-F and AdaWPT-P.

3. We add more discussions in the new version of our Appendix. Specifically, the blur mask annotation details in Appendix C, the effectiveness of the joint training technique in Appendix E.3, deblurring results under different resolutions in Appendix E.4, the effectiveness of our backbone in Appendix E.5, mask prediction accuracy in Appendix E.6, results of special cases in Appendix E.7, as suggested by the reviewers.

4. To maintain the consistency of citing LBAG [1], we replace "LBFMG" [1] with "LBAG" [1] in the new version of our paper. (Notably, LBFMG [1] is a blur mask generation method in paper LBAG. In the original version of our paper, we wrote LBFMG [1] when discussing blur mask annotations.)

All the modifications are highlighted in red in the new version of our paper.

Why we use the GoPro [2] and ReLoBlur dataset [1] for joint training. We train with the GoPro dataset [2] and ReLoBlur dataset [1] together mainly because of two reasons. Firstly, joint training could prevent our model from over-fitting and improve the model's robustness. Secondly, we expect our proposed LMD-ViT to deblur both globally and locally. Training with the GoPro [2] and ReLoBlur [1] datasets together improves both the local and global deblurring performances. We compare models trained solely on the ReLoBlur dataset [1] and jointly trained with the two datasets in the Table below. The results show improvements (+0.13dB in local deblurring and +0.42dB in global deblurring) when we add the GoPro dataset [2] to train. We've added this discussion to the new version of Appendix E.3.

[1] Li et al, Real-World Deep Local Motion Deblurring, AAAI 2023

[2] Nah et al, Deep multi-scale convolutional neural network for dynamic scene deblurring, CVPR 2017

Dear Reviewer ePoq:

Thank you for taking the time to provide valuable feedback and constructive comments. As we are nearing the end of the discussion period, we would like to inquire if our rebuttal has successfully addressed your concerns.

If you have any additional questions, please feel free to respond to our responses. We are more than willing to address any remaining concerns you may have. Once again, thank you for all the helpful comments provided.

Best Wishes,

Authors of paper "Adaptive Window Pruning for Efficient Local Motion Deblurring"

Dear Reviewer ePoq,

Thank you for your inquiry.

Firstly, all the baselines, including Restormer [3] and Uformer-B [4] in Table 1 and Table 9 were trained on the same dataset as our proposed method.

For your second concern whether the performance improvement of the proposed method is due to the extra training data or the extra label, we think the performances are improved for both of the two reasons. We've done experiments before with controlled variables. Firstly, we train our method with the GoPro dataset [2] and the ReLoBlur dataset [1] together with our newly annotated masks, like the results presented in response to your second question. In our previous response regarding "Why we use the GoPro [2] and ReLoBlur dataset [1] for joint training", experiments No.1 and No.2 share the same training settings, differing only in the inclusion of the GoPro dataset [2]. Therefore, the results in our previous response substantiate that joint training can enhance deblurring performance. Secondly, the verify the influences of blur masks, we have included the results of training with masks annotated by the LBFMG [1] method in the updated table below. The results show that our newly annotated blur masks also contribute to the enhancement. Moreover, the updated table could further strengthen the assertion that training with both the ReLoBlur dataset [1] and the GoPro dataset [2] is a superior choice compared to training solely with the ReLoBlur dataset [1]. We have incorporated these updated findings in Table 7 of the new version.

For your third concern of fairness and global deblurring performances, we ensure that all experiments are conducted fairly and traceably. We provided the results of training with baselines and extra data (blur mask) in the previous response and illustrated that with the same information, our model achieves the best local deblurring performance.

Our module design is focused on local deblurring, that is, how to adaptively remove some areas that do not participate in the calculation while ensuring that there is a local performance improvement. From Table 1 in our main paper, we can see that our proposed method LMD-ViT has achieved great advantages while reducing a lot of computation. Table 4 shows the comparable performance of global deblurring and makes perfect sense, but global deblurring is not the focus of our design module concerns.

If our response addresses your concerns adequately, could you please update the rankings of our paper? Thank you for your time and interest in our work!

Best wishes,

Authors of paper "Adaptive Window Pruning for Efficient Local Motion Deblurring"

[1] Li et al, Real-World Deep Local Motion Deblurring, AAAI 2023

[2] Nah et al, Deep multi-scale convolutional neural network for dynamic scene deblurring, CVPR 2017

[3] Wang et al, Uformer: A general u-shaped transformer for image restoration, CVPR 2022

[4] Zamir, Restormer: Efficient transformer for high-resolution image restoration, CVPR 2022

Whether LMD-ViT requires blur masks in inference. In the inference phase, the input consists solely of locally blurred images without accompanying blur masks. During training, blur mask annotations are utilized to guide the prediction of blurriness confidence. However, in the inference phase, our network is capable of autonomously predicting the locations of blurred regions without requiring additional blur mask annotations. We add this explanation in the second paragraph of Section 3.1, highlighted in red.

The equivalence of training and testing. The difference between the training and inferring process lies in the decision layer. In inference, we use Softmax with a threshold to sparse the tokens. However, this technique is not suitable for training because setting a threshold to all images during training increases the instability of parallel training. Therefore, we use Gumbel-Softmax in training to overcome the non-differentiable problem of sampling from a distribution. These two methods have the same effect. Softmax produces a probability distribution over classes, while Gumbel-Softmax introduces stochasticity into the process of selecting discrete values in a differentiable way. Both of them sparse the tokens and distinguish between sharp and blurry tokens.

The details of acquiring real-world locally blurred images in the user study. In the user study experiment, we acquired locally blurred images using a static Sony industrial camera and a static Fuji XT20 SLR camera. The camera sensors and shooting environments in this experiment differ from those in the ReLoBlur dataset and the GoPro dataset. Consequently, the blur patterns observed in the user study may exhibit variations compared to the training data. The visual results presented in Figure 4 demonstrate the robustness of our proposed model in effectively addressing real-world instances of local motion blur. We add the details in the third paragraph of Section 3.2, highlighted in red.

About the references. Thank you for your advice. We have added the references of `"Window-based multi-head self-attention" on page 2, and "LeFF” in Section 2.4.1. The modifications are highlighted in red in the new version of our paper.

Dear Reviewer SKAv:

Thank you for taking the time to provide valuable feedback and constructive comments. Your advice has greatly helped us enhance the quality of our paper and proposed method. As we are nearing the end of the discussion period, we would like to inquire if our rebuttal has successfully addressed your concerns. If you have any additional questions, please feel free to respond to our responses. We are willing to address any remaining concerns you may have.

Best Wishes,

Authors of paper "Adaptive Window Pruning for Efficient Local Motion Deblurring"