Enhancing lensless imaging via Explicit Learning of Model Mismatch

We deeply appreciate your insightful comments and recognition of our work, and have provided detailed responses to each point to ensure clarity and address your concerns comprehensively.

Questions:

Q1:About the derivation from Eq. (5) to (6). We appreciate your detailed feedback about Eqs. (5) and (6). Below is the step-by-step derivation from Eq. (5) to Eq. (6). Restating Eq. (5):

we apply Bayes’ rule to $p(x|A,\hat{x})$:

Since $x$ and $A$ are independent, we have $p(A|x)=p(A)$, which implies that

Substituting this back into Eq. (5), we get:

Here, $p(\hat{x}|A)=\int p(\hat{x}|A,x)p(x)dx$ serves as a normalization constant, ensuring the conditional probability is normalized. It does not affect the optimization of $\log p(A,x|\hat{x})$ and can therefore be omitted when focusing on proportionality. Simplifying, we express Eq. (5) as:

Finally, taking the logarithm of both sides leads directly to Eq. (6).

Q2:the derivation from Eq. (7) to Eq. (8a). We sincerely appreciate your thoughtful question regarding the derivation from Eq. (7) to Eq. (8a). In Eq. (7), we have the joint log-likelihood:

To optimize it efficiently, we adopt a decomposition strategy, as outlined in Sec. 3.2 of reference [1], splitting Eq. (7) into two subproblems (Eq. (8a) and Eq. (8b)) for sequential optimization. The transition to Eq. (8a) involves maximizing the log-likelihood over A while treating $x$ as fixed. Specifically, the terms $\log p(\hat{x}|A, x)$ and $\log p(x)$ are treated as constants with respect to $A$ during this step, simplifying the optimization to: $A^\* = \arg \max_A \log p(A|\hat{x})$.

This assumption is justified because, at this stage, $x$ is not a variable in the optimization over $A$; instead, it serves as a fixed input derived from the previous iteration. Such a decomposition aligns with similar approaches in joint optimization problems, enabling tractable solutions while preserving the overall consistency of the framework.

[1] Zhao Q, Wang H, Yue Z, Meng D. A deep variational Bayesian framework for blind image deblurring. Knowledge-Based Systems. 5, 2022, 249:109008.

Q3:the derivation from Eq. (7) to Eq. (8b). We sincerely appreciate your insightful question regarding the derivation from Eq. (7) to Eq. (8b). In this step, the goal is to optimize for $x$ under the assumption that A is fixed. This treatment is consistent with the decomposition strategy detailed in Sec. 3.2 of reference [1], which employs conditional optimization for tractability. The original optimization problem in Eq. (7) is:

Combining with statement in Q2, when optimizing with respect to $x$, the term $\log p(A|\hat{x})$ becomes independent of $x$ because it depends only on $A$ and $\hat{x}$. Since $A$ is fixed in this step, $\log p(A|\hat{x})$ does not affect the optimization for $x$ and can be treated as a constant. This simplification reduces the problem to:

This decomposition allows us to decouple the joint optimization problem into two manageable subproblems: Eq. (8a) for $A$ and Eq. (8b) for $x$. By isolating these processes, we ensure more efficient optimization and targeted parameter learning for each variable. This stepwise method effectively balances model simplicity and performance, enabling robust parameter estimation.

Q4: About the Eq. (16). We appreciate your detailed feedback about the Eq. (16). In Eq. (16), the integral is used to represent a continuous approximation of the discrete model for $w_i$. While $w_i$ is inherently discrete for each pixel, the integral is applied to model the prior distribution of $w_i$ as a continuous random variable. This is a standard method to incorporate uncertainty in $w_i$, allowing for a more flexible and tractable representation of the prior distribution. The use of the integral instead of a discrete sum simplifies the optimization and captures the variability of $w_i$ efficiently, making it a reasonable approximation for modeling continuous scale parameters.

Q4:The color highlight in Table.1 is confusing. We sincerely appreciate your attention to the details in our Table 1. As noted in the caption, "The best three results are shown in red, green, and blue," indicating that the first, second, and third best performances are highlighted in red, green, and blue, respectively. We apologize for not clarifying this more explicitly in the manuscript, which may have caused confusion. We have updated the text to include this explanation.

Q5:Some most recent related works are not mentioned and compared. Thank you for recommending additional comparison methods. Following your advice, we have incorporated the methods you suggested ([1], [2]) into our evaluation, and the comparative results are now presented in the in Appendix A.8 and Fig. 18 of revised manuscript. These results demonstrate that our method outperforms these state-of-the-art methods. Additionally, we had already included evaluations of recent algorithms, such as MWDNS and MM-FISTA-Net, in Appendix A.4 (Page 16) and Fig. 13 (Page 17), where our method also demonstrates superior performance.

Reference: [1]Lensless Imaging Based on Dual-Input Physics-Driven Neural Network[J], Advanced Photonics Research, 5, 11, 2024. [2]DeepLIR: Attention-Based method for Mask-Based Lensless Image Reconstruction[C], Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024: 431-439.

We sincerely appreciate your valuable comments and recognition, and have provided detailed responses to address each point clearly and comprehensively.

Weaknesses:

Q1:It is good to focus on the M2E, but I think the design of the M2ER is simple. Thank you for your insightful comment. The design of $M^2ER$ is intentionally kept simple, aligning with the structure of Eq. (13) to ensure mathematical consistency as described in Eq. (4). We chose a simple Gaussian distribution to model ($\Delta_{\hat{O}}$) because it effectively captures the randomness and uncertainty caused by misalignment, OSD variations, and system noise. This choice allows for efficient modeling of both the feature (mean) and uncertainty (variance) in the latent space. While the design is simple, it strikes a balance between robustness in $M^2E$ estimation and practical efficiency, making it scalable and effective for real-world applications.

Q2:Comparison with some recent works. Thank you for your valuable feedback. Following your advice, we have added more SOTA methods (DPNN (2024) and DeepLIR (2024)) into our evaluation. The comparative results are now presented in Appendix A.8 and Fig. 18 of revised manuscript. As illustrated, our method continues to demonstrate superior performance.

Q3:About the ablation study. We appreciate your comments regarding the ablation study. To clarify, the ablation study is organized as follows:

(1) Component Ablation:

• TFR: This is the baseline model, where only the TFR module is applied, without any additional components.
• TFR+MSRN: In this step, we introduce the MSRN to the TFR model, enhancing feature extraction.
• TFR+MSRN(w./o. DPMB)+$M^2ER$:** Here, we evaluate the effect of adding MSRN (without DPMB) along with $M^2ER$.
• Full Model: The full model integrates all component-TFR, MSRN, and $M^2ER$-representing the complete $M^2LNet$ framework.

(2) Loss Function Ablation: We investigate the impact of varying coefficients for the Lp and Lkl loss terms to understand their relative contributions to the overall model performance.

To prevent any confusion, we have revised the corresponding description accordingly in the revised manuscript.

Questions:

Q1:1. It is good to focus on the M2E in lensless imaging, but I think the design of the reconstruction model is simple. Thanks for your insightful comment. The design of the reconstruction model is based on a joint MAP estimation framework, where both the $M^2E$ and the underlying scene are co-estimated. This is incorporated into the $M^2LNet$, which integrates a learned LSM prior and the estimated $M^2E$ within a multi-stage reconstruction architecture. While the model may appear simple, its design is deliberate, ensuring mathematical consistency and computational efficiency. By focusing on the joint estimation of $M^2E$ and scene reconstruction, the model effectively leverages both learned priors and $M^2E$ information to enhance reconstruction accuracy without unnecessary complexity. This design strikes a balance between performance and computational feasibility, making it suitable for real-world lensless imaging tasks.

Q2:what is the differences of ‘TFR + MSRN + M2ER’ and ‘Full model’ in your ablation study in Table. 3? We greatly appreciate your concern regarding our ablation experiments. The difference between "TFR + MSRN (w./o. DPMB) + $M^2ER$" and the "Full model" in our ablation study is that "Full model" includes the DPMB, while "TFR + MSRN (w./o. DPMB) + $M^2ER$" excludes it. "w./o. DPMB" stands for "without DPMB". We agree that the improvement from DPMB should have been more explicitly highlighted and we have revised the manuscript (Sec. 4.6) to clarify the impact of this module.

Q3:Why do you choose to use ‘0.01 * Lp ’in the following tests? The choice of "0.01*Lp" in the tests was based on its lowest LPIPS value, which indicates better alignment with human visual perception. While "1.0*Lp" achieves a slight improvements in PSNR and SSIM, it results in a higher LPIPS, indicating poorer visual quality compared to "0.01*Lp". Considering the primary goal of reconstruction is to enhance perceptual quality, we opted for "0.01*Lp". We apologize for the lack of explanation in the manuscript and have clarified this choice in Sec. 4.6 of the revised manuscript.

We sincerely appreciate your valuable comments and recognition, and have provided detailed responses to address each point clearly and comprehensively.

Weaknesses:

Q1:About Contribution, Methodological Distinctiveness and Potential Applications. Thanks for your insightful comment. We provide a summary of our reply in points:

(1)Broader Contributions to Computational Imaging:

• Learnable Forward Model Estimation: Unlike conventional deep unfolding methods, our framework introduces a model mismatch error $(M^2E)$ estimation mechanism for PSF modeling. This innovation addresses real-world challenges, such as misalignment, lateral shifts, and object-to-sensor distance variations. The method is generalizable and applicable to imaging systems with dynamic or imprecise hardware parameters, including computational holography and imaging through scattering media.

• Generalizable Framework for Inverse Problems: The proposed framework incorporates a joint MAP estimator with a learned Laplacian Scale Mixture (LSM) prior. This methodology extends beyond lensless imaging and can be applied to other inverse problems in computational imaging, such as compressed sensing and phase retrieval. This contributes to the broader goal of enhancing the efficiency and quality of reconstruction across diverse imaging modalities.

• Application to Minimal Hardware Systems: Lensless imaging exemplifies the potential for computational methods to replace or augment physical hardware. Our method highlights how advanced computational techniques can reduce dependence on costly or complex optical setups. This principle aligns with the broader goals of computational imaging, emphasizing efficiency and accessibility.

(2)Distinction from Existing Deep Unfolding Methods: While joint forward-model calibration and image reconstruction using deep unfolding networks (DUNs) have been explored, our method stands out by directly addressing the “model mismatch error $(M^2E)$ ”, a crucial factor often overlooked in existing lensless imaging methodes. Conventional DUNs typically rely on approximated point spread function (PSF), which result in reconstruction inaccuracies. In contrast, we introduce a novel "latent space representation" for $(M^2E)$, explicitly modeling and correcting mismatches, improving robustness against PSF errors.

Additionally, our method integrates a "Laplacian Scale Mixture (LSM) prior" in a joint MAP framework, enabling more precise and adaptive reconstruction compared to traditional PSF estimation techniques. This enhances reconstruction quality and generalization across varying imaging conditions, addressing limitations in current DUNs for lensless imaging.

Our main contribution lies in the explicit modeling of $(M^2E)$ and the use of learned priors, leading to significant improvements in lensless imaging. We have revised the manuscript to better highlight these distinctions and the unique advantages of our method (in Related Works ).

Q2: About Dataset and Fairness Assessment Thanks for your comment. We would like to clarify that the DCD-PHlatCam dataset, which was used to evaluate our $(M^2LNet)$, is in fact a publicly available dataset, as described in [1]. It is widely accessible for independent verification and comparison with other methods tested on public datasets. Therefore, the evaluation results on the DCD-PHlatCam can be considered fair and generalizable in the broader context. While the DCD-FinCam dataset is proprietary, it was designed to capture more diverse and complex imaging conditions, showcasing the robustness of our method in real-world scenarios. Nonetheless, we recognize the importance of benchmarking with publicly available datasets, therefore we have revised to highlight this point more clearly in Sec. 4.1 of revised manuscript, ensuring that the use of the DCD-PHlatCam dataset is emphasized.

Reference:

[1]Khan Salman, Siddique, Sundar Varun, Boominathan Vivek, Veeraraghavan Ashok, and Mitra Kaushik. FlatNet: Towards photorealistic scene reconstruction from lensless measurements. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(4):1934-1948, Oct 2022.

Thank you sincerely for these valuable comments and acknowledgment of our work. To ensure clarity and address your concerns comprehensively, we respond to each comment individually. Weaknesses:

Q1: There is no evidence the proposed method generalizes to real-world 3D scenes.

We appreciate this comment regarding 3D natural scene reconstruction. In Appendix A.5 (Page 16) and Fig. 14 (Page 18), we present the reconstruction results for objects in real-world natural scenes using our FinCam lensless imaging system. The provided results demonstrate that our method generalizes effectively to natural 3D environments.

Q2: Many (generally minor) mathematical errors.

We appreciate your detailed feedback about the mathematical issues. Here is our response in points:

• About biased PSF. In real-world systems, the measurement inaccuracies of the PSF arise not only from noise but also from various physical factors such as misalignment, lateral shift, and variations in object-to-sensor distance (OSD), all of which manifest to varying degrees. Hence, we cautiously refer to the PSF affected by these factors as the "biased PSF". Furthermore, regarding Fig. 7, it is important to note that the PSF shown in the figure is based on a simulated experiment. In real-world scenarios, distinguishing between a "biased PSF" and the "true PSF" is inherently challenging due to the complexities of measurement and system dynamics. As such, the term "biased PSF" in this context refers to the simulated measurement of the PSF, which takes into account various imperfections that would typically affect a real-world system.

• The process of deriving from Eq. (5) to Eq. (6). Indeed, the process follows Bayes' rule, and there is a normalization constant that can be ignored during optimization. To clarify the derivation from Eq. (5) to Eq. (6), we first apply Bayes' rule:

  $$p(x|A,\hat{x}) = \frac{p(\hat{x}|A,x) p(x)}{p(\hat{x}|A)}.$$

  Substituting this into Eq. (5) results in:

  $$p(A,x| \hat{x}) = p(A|\hat{x}) \cdot \frac{p(\hat{x}|A,x)p(x)}{p(\hat{x}|A)},$$

  where

  $$p(\hat{x}|A)=\int p(\hat{x}|A,x)p(x)dx$$

  is a normalization constant, which ensures the proper normalization of the conditional probability. However, since this constant does not affect the relative proportionality of the joint distribution of $A$ and $x$, it can be safely ignored when optimizing $\log p(A, x|\hat{x})$. Therefore, Eq. (5) can be further expressed as:

  $$p(A, x|\hat{x}) \propto p(A|\hat{x}) p(x|A, \hat{x}) p(x).$$

  Taking the logarithm of both sides of the Eq. (5) yields Eq. (6). We have updated the manuscript to clarify this process and mention the normalization constant.

• About Eq. (8a) and Eq.(8b). In Eq. (7), we have the joint log-likelihood:

  $$(A*, x*) = \arg \max_{A,x} \log p(A|\hat{x}) + \log p(\hat{x}|A,x) + \log p(x).$$

  To optimize this efficiently, we decompose the problem into two subproblems (Eq. (8a) and Eq. (8b)) by applying a decomposition strategy, as discussed in Sec. 3.2 of reference [1]. The transition from Eq. (7) to Eq. (8a) involves maximizing the log-likelihood over $A$, while treating terms involving $x$ as constants. Specifically, we consider $\log p(\hat{x}|A, x)$ and $\log p(x)$ as constants with respect to A because we are not optimizing over $x$ in this step. In this context, $x$ is treated as fixed, simplifying the optimization for $A$ to:

  $$A^\* = \arg \max_A \log p(A|\hat{x}).$$

  In the transition from Eq. (7) to Eq. (8b), we focus on solving $x$ under the given $A$. Since the optimization in Eq.(7) is a joint maximization problem, the term $\log p(A|\hat{x})$ becomes independent of $x$ during this step because it depends only on $A$ with the given $\hat{x}$. Since $A$ is fixed in this step, $\log p(A|\hat{x})$ does not affect the optimization for $x$ and is thus treated as a constant. Therefore, the optimization for $x$ simplifies to:

  $$x* = \arg \max_x \log p(\hat{x}|A,x) + \log p(x).$$

• About the Eq. (8b) and {A*, x*}. As outlined above, Eq. (8b) focuses on solving for x under the assumption that $A$ is given. This treatment aligns with the methodology detailed in Sec. 3.2 of reference [1], where a similar conditional optimization method is employed. The decomposition of Eq. (7) into Eqs. (8a) and (8b) serves a clear purpose: to decouple the optimization of the two variables, $x$ and $A$, which are relatively independent. By isolating their optimization processes, this formulation ensures more efficient and targeted parameter learning for each variable, ultimately improving the overall model performance. $A^{\*}$ and $x^{\*}$ denote the corresponding expected solution results.

Reference:

[1] Zhao Q, Wang H, Yue Z, Meng D. A deep variational Bayesian framework for blind image deblurring. Knowledge-Based Systems. 5, 2022, 249:109008.

Q3: Presentation

We sincerely thank you for your thorough feedback on the presentation. We address each point one by one:

• About the equation in Line 318. This equation represents a simplified expression of Eq.(3) after being reformulated in a networked form, allowing for better compatibility with deep learning models.
• About that the paper seems to be citing the latest paper to mention a result, rather than the paper where it was first introduced. Thanks for your comment. Lensless imaging has emerged as a key focus in computational imaging, with convolutional models serving as a widely adopted framework. Our work extends this paradigm while addressing a critical gap: the model mismatch error ($M^2E$). While previous studies (as discussed in the introduction) have acknowledged $M^2E$, they have not explicitly modeled or mitigated its impact. We are the first to introduce a model for $M^2E$, significantly improving reconstruction accuracy. Experimental results on various lensless imaging prototypes (PHlatCam，FinCam，even DiffuserCam) demonstrate the enhanced generalization and robustness of our method, validating its effectiveness across different setups.
• About the meaning of mine. We sincerely appreciate your question regarding the term "mine". In the context of our work, "mine" refers to the process of "learning" or "extracting" meaningful patterns or information. Specifically, in our M²ER framework, it denotes the process of learning the model mismatch error by leveraging the designed architecture and training strategy. This terminology aligns with common usage in machine learning, where "mining" often refers to uncovering useful insights or features.

Questions: Q1:About image Reconstruction in natural scene and proposed method working on the webcam captures and other real-world scenes found in the phlatcam and flatnet papers. We greatly appreciate your concerns regarding the practical applicability of our method. As shown in Fig. 14 (Page 18) , we have provided the reconstruction results of real-world natural scenes captured by our FinCam. These results demonstrate the ability of our method to generalize effectively to complex 3D environments.

Furthermore, we agree that the same experiments should be conducted with both PHlatCam and FlatNet for a fair comparison, while due to the absence of the corresponding original data provided by the authors, we are unable to verify it. Nevertheless, we acknowledge the importance of such validation and plan to incorporate it in future work to comprehensively assess our method's adaptability to various real-world scenarios, including webcam captures.

Q2:Would the proposed method work with diffusercams? We sincerely appreciate your interest in exploring the adaptability of our method. Based on our experiments, our approach is indeed compatible with DiffuserCam systems. Using the dataset from reference [2] and adhering to its data configuration protocols, we conducted a comparative analysis with existing methods, including MMCN [3], FlatNet [4], MWDNs [5], and MDGAN [6]. The results highlight our method's ability to recover detailed scene information effectively, demonstrating its applicability to DiffuserCam setups.

To provide comprehensive evidence, we have included these experiments in the revised manuscript ( Appendix A.7 and Fig. 17 ). We remain committed to ensuring our method’s applicability across diverse optical imaging systems.

Reference:

[2] K. Monakhova, J. Yurtsever, G. Kuo, N. Antipa, K. Yanny, and L. Waller. Learned reconstructions for practical mask-based lensless imaging. Opt. Express, vol. 27, no. 20, pp. 28075-28090, 2019.

[3] Tianjiao Zeng and Edmund Y. Lam. Robust reconstruction with deep learning to handle model mismatch in lensless imaging. IEEE Transactions on Computational Imaging, vol.7, 1080–1092, 2021

[4] Khan Salman, Siddique, Sundar Varun, Boominathan Vivek,Veeraraghavan Ashok, and Mitra Kaushik. FlatNet: Towards photorealistic scene reconstruction from lensless measurements. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no.4, pp.1934-1948, 2022.

[5] Ying Li, Zhengdai Li, Kaiyu Chen, Youming Guo, and Changhui Rao. MWDNs: reconstruction in multi-scale feature spaces for lensless imaging. Opt. Express, vol. 31, no.23, pp. 39088-39101, 2023.

[6] Cong Ni, Chen Yang, Xinye Zhang, Yusen Li, Wenwen Zhang, Yusheng Zhai, Weiji He, and Qian Chen. Address model mismatch and defocus in FZA lensless imaging via model-driven cyclegan. Opt. Lett., vol. 49, no. 15, pp. 4170-4173, 2024.