User-Instructed Disparity-aware Defocus Control

1.Clarity and Notation.

Thank you for the feedback. We will revise the data flow and notation in the final version to improve readability.

2.Rationale of Method.

Actually, our network still adheres to the PSF-based framework. The refocusing process does not employ an additional decoder; instead, it follows the workflow indicated by the black arrows and orange arrows in Fig. 2.

black arrows: Given a deblurred image and a user-specified mask of the region of interest, the modified disparity is first estimated for blur kernel retrieval.

orange arrows: Then, blur kernel is used to achieve spatially-variant defocus control using inverse mapping of our invertible network.

3.Architectural and Component-level Explanation.

Thank you for your suggestion. Here, we clarify the design motivation behind our architecture and core components. Architectural Explanation: In essence, our goal is to design a framework for refocusing, a process can be typically decomposed into two steps: (1) first deblurring to obtain a sharp image, and (2) then reblurring specific regions according to user needs to achieve the desired defocus effect.

Mathematically, deblurring process can be modeled as $I_{s}$ = $I_{b}$ $\circledast$ $K$, where $I_{b}$ is the blur image, and $I_{s}$ is the restored image. According to the convolutional theorm, we have $\mathcal{F}(I_{s})$ = $\mathcal{F}(I_{b})$ $\odot$ $\mathcal{F}(K)$. Its inverse process can be represented as a reblurring process, $I_{b}$ = $I_{s}$ $\circledast$ $\mathcal{F}^{-1}(\frac{1}{\mathcal{F}(K)})$ = $I_{s}$ $\circledast$ $K^{-1}$.

We observe that the blur kernels $K$ and $K^{-1}$ used in these two processes satisfy a reversibility property. This insight motivates us to learn an invertible network with reversible kernels. Intuitively, an effective blur kernel should be capable of both restoring an image (deblurring) and blurring it (reblurring). Therefore, we supervise the network's joint learning using both the reblurring loss and the deblurring loss. By co-training these two tasks within a single network, we achieve refocusing efficiently—reducing training time and lowering the model parameter count.

Component-level Explanation: As illustrated in Fig. 2, our model follows a common architecture comprising an encoder ($F_{e}$), a refine component, and a decoder ($F_{d}$). Our primary contribution lies within the refine module, which consists of:

(i) Disparity-aware Feature Convolution: This component retrieves the corresponding kernel $K$ from the kernel dictionary using the estimated disparity. The forward process (deblurring) uses kernel $K$ for modulation, while the inverse process (reblurring) employs its corresponding inverse kernel $K^{-1}$ for modulation.

(ii) Invertible Blocks: To further enhance the representational capacity of network features and strengthen the network's inherent reversibility, we design invertible blocks, denoted as $Inv(·)$ and $Inv^{-1}(·)$, which has been elaborated in Tab.1 in the appendix. This design ensures strict invertibility of the network (enabling lossless, reversible information flow [3]), where the Jacobian matrix of the output with respect to the input is invertible.

4.Absence of a Limitations Section.

Thank you for the suggestion. We will include a limitation section in the final version.

5.Citation.

Thank you. We will add the missing citations in the table.

6.Experimental Comparison.

Regarding this point, our experimental table setup follows the conventions established in most prior works [1, 2]. We will revise the formatting according to your recommendations.

6. Could the authors please provide a revised version of Figure 2? The current diagram is difficult to parse due to notational inconsistencies and an unclear depiction of the framework's data flow.

Please refer to Response 1.

7.What is the specific rationale for using a learned decoder over a more interpretable, physics-based Point Spread Function (PSF) model? Could the authors demonstrate or elaborate on the distinct advantages of their chosen approach?

Please refer to Response 2.

8.Could the authors elaborate on the design rationale for their architectural components? For instance, what motivated the choice of specific modules over other common alternatives in the literature?

Please refer to Response 3.

9.How robust is the framework to challenging inputs? We ask the authors to provide analysis on failure cases, specifically regarding (a) inaccurate segmentations from SAM, and (b) images with very large amounts of defocus blur. Could the authors add the "Limitations" section as indicated in their submission checklist? This section should detail the method's specific failure modes and boundary conditions.

(a) Inaccurate segmentations from SAM: The language-based SAM, when guided by specially designed prompts, can generate sufficiently robust masks even on deblurred images. Given that the original SAM model was trained with diverse data augmentation, it exhibits inherent robustness to mild noise or blur. To further improve mask quality in our implementation:

• For region of interest, we use language-style prompts: number + object color + object name + spatial position (e.g., a brown dog on the chair).
• For those still imperfect masks, we further apply post-processing techniques such as dilation or erosion operations for refinement.

To evaluate the mask quality from our adopted SAM, we manually annotate two objects for each image of our collected images as the groundtruth. We compare their average IOU metrics,

We would provide the failure visualization case in our limitation section in final version.

(b) Images with very large amounts of defocus blur. We empirically observe that our used SAM could functions well when handling the image with large defocus blur (i.e., $f$-1.8). We sample two groups of sample from DP5K-test and, each of which has 5 cases with different $f$-number, 1.8, 2.8, 4.0, and 5.6. We use the language-based SAM to generate the mask, and manually annotate the region with interest as the ground-truth.

We calculate their respective IOU metric.

We observe mariginal mask degradation. We would include above analysis, and provide more visualization in our final version. Thanks.

10. Could the authors provide the missing details on computational resources (GPU model, training/inference times) as stated in the checklist? This information is essential for reproducibility.

Our model is trained on one A100 80G GPU, and test on GTX A6000.

11. Could the author update the checklist for "Open access to data and code".

Yes, I will update it. Thanks.

Reference:

[1] Restormer: Efficient Transformer for High-Resolution Image Restoration, CVPR 22

[2] K3DN: Disparity-aware Kernel Estimation for Dual-Pixel Defocus Deblurring, CVPR 23

[3] Density estimation using real NVP, ICLR 2017

Dear Authors,

Thank you for your detailed and thoughtful rebuttal. Your responses have successfully addressed the vast majority of my concerns and have significantly clarified the technical contributions and positioning of your work.

Confirmation of Resolved Issues

I would like to confirm that the following points have been effectively addressed and have greatly strengthened the manuscript:

• Methodology & Rationale: Your explanation of the invertible network and how it functions within a PSF-based paradigm was a crucial clarification. This resolved my primary misunderstanding regarding the technical foundation of your method. It is now clear that the core contribution is the insightful synthesis of these techniques, motivated by the inverse relationship between deblurring and reblurring. To further strengthen the paper, I encourage you to ensure this well-articulated rationale is clearly presented in the final manuscript.

• Robustness & Reproducibility: The new quantitative experiments on SAM's performance are excellent and provide convincing evidence of the framework's robustness. Furthermore, providing the specific computational details has fully addressed my concerns about reproducibility.

• Experimental Comparisons: Thank you for the clarification regarding the experimental setup. My initial suggestion was aimed at improving the focus of the results by de-emphasizing the Single-Image baselines. I acknowledge your commitment to reformat the results based on my suggestion, and I now consider this point resolved.

• Promised Revisions: I acknowledge your commitment to revise the paper's clarity, add the missing citations, include a dedicated limitations section, and update the submission checklist. I trust these changes will be incorporated into the final version.

---

Overall, you have provided an excellent rebuttal that successfully addresses the critical issues. Based on these significant improvements, I will update my score to borderline accept and look forward to seeing the final version of the manuscript.

---

1.Confined to Dual-Pixel imagery

Our method is primarily designed for defocus control in static scenes. It is worth noting that our focus is not limited to deblurring and reblurring separately, but rather unifying both processes to serve the purpose of refocusing. The works you mentioned, such as [1, 2], primarily address motion deblurring, which is indeed a different task from ours. It is universally known that the crux of refocusing lies in accurately estimating the target disparity. In theory, our method can be extended from dual-pixel to stereo matching, as the disparity estimation network ($G$ network, mentioned in line 155) in our approach essentially functions as a stereo matching network. This network shares the similar architecture as those in articles [3, 4] without pretraining, and this structural prior theoretically would enable it to cover the larger disparities generated by stereo data. Considering that currently there is no dataset specifically designed for stereo image refocusing, we will consider optimizing this aspect in our future work.

2.Mobile Efficiency

Thank you for your question. Currently, we mainly focus on algorithm design, and have not yet supported deployment on mobile devices. We recorded the average inference time on a single A6000 GPU using our self-collected dual-pixel image pair dataset, with an image resolution of 1152 × 1792. The average time is SAM + refocusing = 0.45s + 0.13s = 0.58s.

3.Evaluation on SAM

The language-based SAM, when guided by specially designed prompts, can generate sufficiently robust masks even on deblurred images. Given that the original SAM model was trained with diverse data augmentation, it exhibits inherent robustness to mild noise or blur. To further improve mask quality in our implementation:

• For region of interest, we use language-style prompts: number + object color + object name + spatial position (e.g., a brown dog on the chair).
• For those still imperfect masks, we further apply post-processing techniques such as dilation or erosion operations for refinement.

To evaluate the mask quality from our adopted SAM, we manually annotate two objects for each image of our collected images as the groundtruth. We compare their average IOU metrics,

4. Limitation Section

Thanks. We would add the limitation section and potential improvements of diffusion framework in our final version.

Reference

[1] Joint Stereo Video Deblurring, Scene Flow Estimation and Moving Object Segmentation, TIP 2019

[2] Stereo Video Deblurring, ECCV 2016

[3] Hierarchical Neural Architecture Search for Deep Stereo Matching, NeurIPS 20

[4] Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration, CVPR 21

We sincerely thank your comments, and we give point-to-point below.

1. Consideration on Monocular or Stereo-only Inputs

Thanks. As the adopted disparity estimation network $G$ in our approach shares similar architecture with [1, 2] (stereo setting), therefore its structural prior theorectically could handle the stereo-only inputs that shares similar pattern with DP inputs. Regarding monocular setting that hard to estimate depth or disparity, I think a possible solution is to leverage a series of powerful pretrained model for depth or disparity, such as DepthAnything [3]. This can be further discussed and explored in our future work.

2. Evaluation on SAM

The language-based SAM, when guided by specially designed prompts, can generate sufficiently robust masks even on deblurred images. Given that the original SAM model was trained with diverse data augmentation, it exhibits inherent robustness to mild noise or blur. To further improve mask quality in our implementation:

• For region of interest, we use language-style prompts: number + object color + object name + spatial position (e.g., a brown dog on the chair).
• For those still imperfect masks, we further apply post-processing techniques such as dilation or erosion operations for refinement.

To evaluate the mask quality from our adopted SAM, we manually annotate two objects for each image of our collected images as the groundtruth. We compare their average IOU metrics,

We would provide the failure visualization case in our final version.

3.User Experience Studies.

Thanks for your suggestion. We would include this consideration in our final version.

4. What happend if the mask produced by SAM is imprecise or unclear? Does the system generate invalid defocus regions or does it degrade gracefully?

It would degrade gracefully.

5. How robust is your method to different DP hardware calibrations or disparities with high noise?

Our method leverages the structural prior of the stereo matching network $G$ to automatically estimate disparity from the DP image pair (Eq. 7), guided by both deblurring and reblurring losses. This dual supervision encourages the model to focus on the downstream task, thereby enhancing its robustness to inherent noise.

6. What is the need for invertibility? When compared to standard encoding-decoding, can you measure the improvement in results?

The network's invertibility ensures that reblurring and deblurring can be jointly trained while sharing the blur kernel $K$, better establishing one-to-one correspondence between disparity feature and blur kernel $K$. Intuitively, an effective disparity-aware blur kernel should be capable of both restoring an image (deblurring) and blurring it (reblurring). This paves a crucial path for the subsequent refocusing, enabling accurate kernel retrieval for defocus control by modifying the disparity (refer to Fig. 7 for illustration). When we empiricially adopt a traditional encoder-decoder architecture, the correspondence between the kernel and disparity becomes difficult to learn effectively, leading to unintended control effects (if possible, we would provide results in the rolling stage.).

7. Is the method fragile to segmentation boundary errors or small masks?

Segmentation boundary errors would affect the edges of the refocused image, but the results generally follow reasonableness; Our model exhibits good robustness to small masks.

8. Is performance degraded with imprecise or noisy user prompts?

Yes. SAM may generate imprecise masks in response to noisy user prompts, which could have a certain impact on our refocusing process.

Reference:

[1] Hierarchical Neural Architecture Search for Deep Stereo Matching, NeurIPS 20

[2] Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration, CVPR 21

[3] Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data, CVPR 24.

We sincerely thank your comments, and we give point-to-point below.

---

1. Novelty Highlight

Please kindly note lines 56–68. The core innovation of our work lies in the adoption of a unified invertible network that decomposes the defocus control task into reversible deblurring and reblurring subtasks. Since deblurring and reblurring are inverse processes, we establish an essential connection between them by constraining these two subtasks to share the same blur kernel $K$ but with different form (as formulated in Eq.10). Specifically:

• The forward process (deblurring) uses disparity feature to retrieve kernel $K$ for convolution.
• The reverse process (reblurring) uses disparity feature to retrieve kernel $K$, and adopt its inverse kernel $K^{-1} = \mathcal{F}^{-1}(\frac{1}{\mathcal{F}(K)})$ for convolution, where $\mathcal{F}$ and $\mathcal{F}^{-1}$ denote the fourier transformation and its inverse transformation, respectively.

We also employ the invertible block (elaborated in suppl Tab.1) to better refine the representation. These two processes are trained collaboratively using loss in Eq.(13) .

Compared to existing deblur-and-reblur frameworks (e.g., RefocusGAN, $\text{DC}^2$), our key improvements are:

1. Reversible Block with Shared Kernel: We establish an invertible connection between deblurring and reblurring with shared kernel learning, which strictly follows the mathematical formuation (Eq.10).
2. Unified Network Architecture: We reuse one network parameter to achieve refocusing task (deblur-and-reblur), significantly reducing the total number of parameters and improving training efficiency.

---

2. Off-the-Shelf SAM

1. Note that our method does not require masks during training. Masks are only provided at test time, where the user inputs a binary mask (single-channel, $H$ $\times$ $W$, the same size with input image) to specify the region of interest for refocusing. Therefore, no task-specific modification would not affect the dynamics and robustness of our model training (train only once).
2. The language-based SAM, when guided by specially designed prompts, can generate sufficiently robust masks even on deblurred images. Given that the original SAM model was trained with diverse data augmentation, it exhibits inherent robustness to mild noise or blur. To further improve mask quality in our implementation:
   • For region of interest, we use language-style prompts: number + object color + object name + spatial position (e.g., a brown dog on the chair).
   • For those still imperfect masks, we further apply post-processing techniques such as dilation or erosion operations for refinement.

To evaluate the mask quality from our adopted SAM, we manually annotate two objects for each image of our collected images as the groundtruth. We compare their average IOU metrics,

We acknowledge that the quality of refocusing results may be influenced by the quality input mask, we would provide the failure case in our final version.

---

3. Implementation Details for SAM

For language-based mask generation:

• The LLM tokenizer converts user textual prompts into tokens.
• The global token [seg] computes similarity with image features to generate the segmentation mask.

For click-based interactive mask generation (point or box inputs):

• Each click coordinate is transformed into a heatmap (the region with click has large score).
• This heatmap is fused with the previously generated mask to iteratively refine the segmentation result. More click would further iteratively improve the quality of the generated mask.

We will provide further technical details in the appendix of the final submission.

---

4. Detailed Descriptions of Self-Collected DP Dataset

Please refer to Appendix. Additionally, our self-collected DP dataset consists of images with a resolution of 6720 × 4480 pixels. Additional characteristics will be added in the supplementary material of final version.

---

5. Hyperparameter Settings

Thanks for your suggestion. Considering the reversible nature of deblurring and reblurring, we assume equal importance to the deblurring loss $L_{\text{deb}}$ and reblurring loss $L_{\text{reb}}$. The gradient regularization term $\lambda_{grad}$ is a widely used auxiliary loss in image restoration tasks [1], and is not a primary contribution of our work; hence, we did not emphasize its analysis initially.

In response to your suggestion, we conduct additional ablation studies on the impact of different hyperparameters. Results on PSNR (dB) are summarized below:

Ablation on $\lambda_{\text{grad}}$:

Ablation on $\lambda_{coc}$:

We will include these ablation studies in the experimental section of the final manuscript.

---

6. Formatting Errors

We appreciate your feedback and will carefully revise all formatting errors in the manuscript accordingly.

Reference:

[1] Structure-Preserving Super Resolution with Gradient Guidance, CVPR 2020