PhoCoLens: Photorealistic and Consistent Reconstruction in Lensless Imaging

1. Real-world experiments

The DiffuserCam and PhlatCam datasets utilized in our study are collected from real-world lensless camera prototypes. These datasets are widely recognized and extensively used in lensless imaging literature, providing robust and reliable benchmarks for evaluation. Thus, while our study focuses on standard datasets, they effectively represent real-world scenarios and ensure a comprehensive evaluation of our proposed methods.

2. Discussion from the optics field

We appreciate the reviewer's suggestion to incorporate deeper insights from optics. We agree that a more thorough optical analysis can strengthen our model's foundation. In the revised manuscript, we will provide a detailed derivation of the imaging model mismatch from an optical perspective to enhance the theoretical basis of our approach.

3. Mitigating vignetting in WinnerDeconv outputs

The appearance of vignetting is primarily due to the limitations of the shift-invariant convolution model which motivates us to propose a spatial-varying formuation. Our method effectively addresses vignetting through two stages. First, the proposed SVDeconv component in the first stage corrects for the inaccuracies of the shift-invariant model, mitigating vignetting artifacts. Second, the subsequent neural network stage learns a natural image distribution that is typically devoid of vignetting. This observation is supported by the fact that even a trained U-Net architecture, as employed in FlatNet, exhibits the ability to reduce vignetting.

4. Revision on Fig. 4

We appreciate the reviewer's suggestion. Figure 4(a) has been revised to accurately depict the light propagation as suggested. The updated figure is presented in Fig. a of the rebuttal PDF.

5. Other comments

Thanks for the suggestions on adding references. We will incorporate the recommended references to enhance the discussion on the theoretical foundations and applications of spatially varying PSFs in the revised manuscript.

1. Details of the training process

In the first stage, SVDeconv is trained using range-space content derived from ground truth images. SVDeconv consists of two main parameter components: a learnable deconvolution kernel initialized with known PSFs, and a U-Net initialized with standard weights without pretraining. Once trained, SVDeconv processes input lensless measurements to estimate the range-space content of training samples. Subsequently, we use this estimated range-space content as input conditions for fine-tuning via null-space diffusion. During diffusion fine-tuning, we utilize a pre-trained diffusion model with frozen weights. We only train the supplementary conditioning modules like StableSR [35], to guide the reconstruction process effectively.

2. Variation of spatially-varying PSF related to the depth

In our lensless camera setup, the scene-to-camera distance significantly exceeds the sensor size (less than 1cm), typically around 40cm, which is practical for everyday capture scenarios. This distance effectively disregards spatial variance along the depth axis (from 30cm to infinity), treating the PSF at such distances as equivalent to that from an infinitely distant point source. Our simulations validate this by showing a 0.995 similarity score between the PSF of a light point at 30cm and 100cm distances. Such scenarios are widely accepted in the relevant literature [15,17,23,43] and align with the lensless imaging datasets used in our study.

Consideration of depth-dependent spatial variance in the PSF becomes critical only when the scene-to-camera distance is comparable to the camera size, typically between 1cm and 5cm. However, these scenarios require capturing scenes very close to our camera, which is beyond the scope of this paper. Future work will explore these 3D spatially varying PSF effects.

3. Comparison of reconstruction using accurate SV-PSF and our method

We conducted an additional experiment below, which shows that our method achieves comparable performance to SV-deconvolution methods when accurate PSFs are provided.

In real-world lensless camera datasets like the PhlatCam and DiffuserCam, accurately calibrated Spatially-Varying Point Spread Functions (SV-PSFs) for different incident angles are typically unavailable. To validate the effectiveness of our proposed method, we simulated a dataset using the simulated lensless camera discussed in our paper, incorporating known SV-PSFs. This dataset comprises 2000 images with 20dB noise. We evaluate four methods for SV-deconvolution: 1) spatially varying FISTA [a], 2) MultiWinnerNet [b] using a 3x3 grid of known PSFs, 3) our method using a single known PSF, and 4) our method using a 3x3 grid of known PSFs. Results in the following table demonstrate that our method achieves comparable performance to SV-deconv methods utilizing accurately calibrated Spatially-Varying Point Spread Functions (SV-PSFs).

4. Data fidelity in the reconstructed range-space contents

For data fidelity in the reconstructed range-space contents, the results in Tab.2 of the original paper compare various deconvolution methods, showing that our approach achieves superior fidelity quantitatively. Reviewers can also refer to the qualitative examples in Fig. b in the rebuttal PDF for further confirmation.

5. Other comments

Thank you for pointing out the $𝑁$×$𝑁$ typo, we will revise it.

[a]. Yanny, Kyrollos, et al. "Miniscope3D: optimized single-shot miniature 3D fluorescence microscopy." Light: Science & Applications 9.1 (2020): 171.

[b]. Yanny, Kyrollos, et al. "Deep learning for fast spatially varying deconvolution." Optica 9.1 (2022): 96-99.

1. Comparison with DDNM

Our paper presents notable advancements compared to DDNM, in both the reconstruction quality and inference speed. Training-free methods like DDNM depend on an accurate imaging model to recover null space, but acquiring such a model for lensless imaging is difficult. In contrast, our null space diffusion learns also utilizes real-world training data pairs, improving its performance on real captures where the imaging formulation model is imperfect. As demonstrated in Tab. 3 and Fig. 9 in the original paper, our approach consistently outperforms DDNM on real captures. Additionally, DDNM requires 3x more time than ours for the reconstruction Under lensless imaging setting. Unlike DDNM, which requires complex calculations at each sampling step to maintain consistency with the original measurement, our feed-forward model minimizes additional computational overhead while delivering superior results. Moreover, we introduce a novel imaging model that enhances the accuracy of estimating range-space content, crucially improving the fidelity of final reconstruction.

2. Effectiveness of range-null space decomposition

In Tab. 4 and Fig. 10 of the original paper, we conduct a comparative analysis of our diffusion model under various conditions. Specifically, for the SVD-OC method, we utilize a similar deconvolution approach to recover the original content, contrasting with our method that incorporates range-null space decomposition. The results consistently demonstrate our method's superiority over SVD-OC, highlighting the clear advantage of integrating range-null space decomposition in enhancing lensless image restoration.

3. Clarification about deconvolution kernel size

The deconvolution kernel size is as large as the PSF size of the lensless camera. For example, in the PhlatCam dataset (1280 × 1480, same as the lensless measurement), and the DiffuserCam dataset (540 × 960, larger than the original lensless measurement). For DiffuserCam, we employ replicate padding to align the PSF size with the padded measurements, as detailed in our paper. This ensures our method effectively handles the large PSF sizes typical in lensless imaging, maintaining accuracy in image restoration.

4. The function of the simulated lensless camera and PSF

The simulation of the lensless PSF is solely used to demonstrate the mismatch in widely used spatial invariant convolution models. We do not use the simulation in our lensless imaging experiments. For lensless datasets captured in the real world we used, like the DiffuserCam and PhlatCam datasets, we only have a calibrated PSF in the center (zero angle of incidence light). Therefore, we use the calibrated PSF to initialize all the learnable deconvolution kernels.

5. Relation between the incident angle and the FoV center

In a lensless setup with a 2-dimensional imaging plane, the Huygens-Fresnel principle [9] establishes a direct relationship between the incident angles ($\theta$, $\phi$) and the center shift of the PSF — known as the focus center. Specifically, if the PSF center for zero angles of incidence light is located at ($c_x$, $c_y$), then for incident angles ($\theta$, $\phi$), the PSF center approximately shifts to ($c_x - d \sin \theta$, $c_y - d \sin \phi$) according to the Fresnel propagation approximation [9], where $d$ denotes the mask-sensor distance. For detailed derivation, please refer to the Phlatcam paper [5], which we will further elaborate upon in the revised version. Reviewers can also refer to Fig. a in the rebuttal PDF for an illustration. This approach allows us to model the PSF variation across different focus centers on the imaging plane, reflecting the variation due to incident angles.

In our formulation, the center of the learnable kernel corresponds to a specific incident angle as well as coordinates on the imaging plane. We define this specific incident angle as the FoV center of the learnable kernel and the coordinates as the focus center of the kernel. Therefore, the weights can be determined by the distance between the pixel coordinates (u, v) and the focus center.

6. Robustness of the two-stage Framework

We appreciate the concern about the potential robustness issues of a two-stage framework. Actually, our training scheme is designed to mitigate errors introduced by inaccurate image formation in the first stage. Our range-space reconstruction in the first stage is a deterministic process, focused on recovering information directly observable from the lensless measurement. This reduces the risk of introducing artifacts or errors that would hinder the subsequent diffusion process. Additionally, our SVDencov component is specifically designed to handle the challenges of lensless image reconstruction, improving the accuracy of the first stage. We conducted ablation studies (Tab. 4 and Fig. 10) comparing different intermediate outputs, demonstrating that using the full estimated image reconstruction as input to the diffusion model can indeed amplify first-stage errors. But our range-space-based approach alleviates this issue. This is because recovering range-space content is easier than the original content in the first stage, therefore we achieve better data fidelity and fewer artifacts. Fig. b in the rebuttal PDF also shows the data fidelity of our first stage.

Moreover, iterative frameworks that project the output onto the lensless measurement space generally rely on precise imaging models, which are difficult to establish accurately in the context of lensless imaging. However, inaccurate modeling may introduce additional errors in the reconstruction process. Moreover, iterative methods often require more computational resources, such as DDNM and similar approaches.

Given these considerations, our method achieves a balanced trade-off between performance and efficiency.

7. Choice of the number of deconvolution kernels

Please refer to the reply to Reviewer FAra (R1).

1. Choice of the number of deconvolution kernels

The rationale for using 3x3 PSF kernels is based on the assumption that the central PSF effectively represents the region of the field of view (FoV) where PSFs change smoothly, which is typical in lensless settings. The 3x3 choice balances performance and cost: by discretely sampling a small number of the central region and their corresponding PSFs, we approximate the continuous PSF across the entire FoV while minimizing computational and memory costs.

Furthermore, opting for an odd number of PSFs (in both width and height dimensions) is recommended when employing a single calibrated PSF at the center of the FoV ($\theta$ = 0). This choice ensures that the initialized deconvolution kernel at the FoV center aligns accurately with the calibrated PSF, establishing a reliable starting point for the deconvolution process. In contrast, using an even number of kernels may result in inaccurate initializations across all deconvolution kernels, potentially compromising the effectiveness of the deconvolution. With an odd number, at least one correct initialization can be assured.

2. Comparison of the computational complexity

We evaluated the computational efficiency of the proposed method on a machine equipped with Intel(R) Xeon(R) Platinum 8352V CPU @ 2.10GHz and RTX 4090 GPU. The above table presents the Inference time results. Future work will explore optimal trade-offs between computational speed and performance for various application scenarios.

3. Usage of WinnerDeconv

Thanks for the reminder! We agree that we should use the correct term 'Wiener Deconvolution' in the main context to ensure clarity and accuracy and we apologize for any confusion caused.

4. Phase modulation masks v.s. amplitude modulation masks

Our model is applicable to all lensless cameras using a convolution imaging model, whether they employ phase modulation masks or amplitude modulation masks. However, for amplitude modulation lensless cameras like Flatcam [3], which utilize a separate imaging model, further exploration is needed to assess the effectiveness of the proposed method.

5. Other comments

In the revised version, we will include a citation to the paper 'Recent Advances in Lensless Imaging,' which contributes to our research by offering a comprehensive review of the definition and evolution of lensless imaging.