FDTDNet: Privacy-Preserving Lensless Object Segmentation via Feature Demultiplexing and Task Decoupling

We sincerely thank you for your valuable feedback and acknowledgment of our work. To ensure clarity and comprehensively address your concerns, we have systematically provided detailed responses to each comment.

Weaknesses:

Q1：The performance of the network is not analyzed, such as the number of parameters, number of floating-point operations, inference time, etc.

We sincerely appreciate your concern regarding the computational complexity of our network. As Ablation study on the weight setting of loss functionsnoted in Appendix A.7 , we have provided the number of parameters and the number of floating-point operations for each method. Our method effectively balances segmentation performance with computational complexity. In response to your suggestion, we incorporate the inference time in the revised manuscript (Also in Appendix A.7) to further evaluate the performance of our method. Experimental results demonstrate that our method achieves a frame rate of 35.9 FPS, ranking just behind BDG-Net and ZoomNet, thereby fulfilling the essential real-time processing requirements.

Q2: Lack of explanation and verification of the weight setting of the hybrid loss function.

We sincerely appreciate your valuable feedback regarding the weight setting of the hybrid loss function. According to your suggestion, we have included additional experiments and clarifications on the explanation and verification of the coefficients for the loss function in Appendix A.5 . Here, we provide a brief explanation. The weight setting in the hybrid loss function is designed to balance the contributions of each component (BCE loss and IoU loss) for different tasks (CDM, BDM, and segmentation). Ablation experiments for these tasks have been presented in Sec. 4.3 (Ablation Studies on Tasks). In this response, we focus specifically on the weighting of the BCE loss and IoU loss in the Appendix A.5. The results suggest that variations in performance across different weight configurations are marginal. As a result, we empirically adopted a 1:1 weight ratio. Here we provide the results in following table.

Table 1 Ablation study on the weight setting of loss functions.

Thanks for your question. In our work, we only used data provided by FlatCam, which is a separable system. On the other hand, PHlatCam is not a separable system, so its collected data cannot be directly applied to our method. However, we can adapt the data from PHlatCam by adjusting the representation of the OFD to a Wiener filter-based formulation, as :$X_\theta = \mathcal {F}^{-1}\left(\frac{ \left(\mathcal{F}(A_\theta)\right)^\top}{K_\theta+|\mathcal{F}(A_\theta)|^2} \odot \mathcal {F}(Y_\theta)\right)$, where $A_\theta$ is the PSF, $\mathcal {F}$ and $\mathcal {F}^{-1}$ are the Fourier Transform and its inverse. $K_\theta$ is the regularization parameter.

We sincerely appreciate your insightful comments and recognition of our work. To ensure clarity and thoroughly address your concerns, we have provided detailed responses to each comment individually.

Weaknesses:

Q1:does the linearity between the original image and the measurement mean that the semantic features of the original image and the measurement are also linear?

We greatly appreciate your attention to Eq. (1) in our paper. While the linearity of the semantic features between the original image and the measurements may not always hold perfectly, it serves as an effective approximation in our model. This simplification enables computational efficiency while still providing accurate feature recovery. Our hypothesis builds on the foundational work in Eq. (1) and Eq. (2) of reference [1], which extends the model from the image domain to the feature domain, achieving improved performance. Following this rationale, we have incorporated a similar assumption into our framework, as demonstrated in Eq. (1). The experimental results further validate the effectiveness of this design.

[1]Dong, Jiangxin, Stefan Roth, and Bernt Schiele. "DWDN: Deep Wiener deconvolution network for non-blind image deblurring." IEEE Transactions on Pattern Analysis and Machine Intelligence, 44.12 (2021): 9960-9976.

Questions:

Q1:Why not do the same for AL/AR and Y? What is the author's consideration?

We greatly appreciate your insightful comments and concerns regarding the design of our OFD. The distinct treatment of $Y$, $A_L$ and $A_R$ is attributed to their differing roles in the system. $Y$ encapsulates rich semantic information related to the scene, while $A_L$ and $A_R$ are primarily associated with the system's transfer function and do not inherently contain scene-specific information, limiting their semantic content. Consequently, $Y$ requires a more sophisticated feature extractor to effectively capture its semantic characteristics, whereas $A_L$ and $A_R$ can be processed with simpler convolutional operations, reducing network complexity while maintaining efficiency. This design choice strikes a balance between computational demands and the need for accurate feature extraction.

We sincerely thank you for your valuable comments and recognition, and have provided individual responses to address each concern clearly and thoroughly.

Weaknesses:

Q1：The clarity of our manuscript.

We appreciate your valuable feedback regarding the clarity of our manuscript. To address your concerns thoroughly, we have responded to each point individually:

• About the mathematical presentation. We appreciate your point regarding the complexity of the mathematical presentation, particularly in the OFD mechanism. While our intention was to provide a rigorous and detailed explanation, we recognize that the density of Eqs.(3)-(11) may hinder clarity. In the revised manuscript, we have simplified the mathematical formulation by merging the steps and providing more intuitive explanations for key concepts, as shown in Appendix A.1 and Sec. 3.2. Our goal is to make the equations more accessible while maintaining the technical precision of the proposed method.
• This section mainly stacks equations without sufficient explanation, making it difficult for readers to grasp the underlying principles. We agree that the section would benefit from clearer explanations. In the revision, we include more intuitive and conceptual descriptions alongside the Eqs.(3)-(11) to better highlight the underlying principles and enhance accessibility for readers, as shown in Appendix A.1 and Sec. 3.2.

-Labeling elements of Figure 2 to indicate which parts correspond to specific equations could greatly improve clarity. Thanks for your helpful suggestion about Fig.2. We have labeled the elements of the figure to clearly correspond with the relevant equations, aiming to improve the clarity of the presentation. Additionally, we have simplified the equations where possible and provided clearer explanations to enhance readability while maintaining technical accuracy.

Q2: The analysis of proposed method.

We greatly appreciate your concerns regarding the analysis of our experiments. We respond to each point as follows:

• The paper could benefit from a more in-depth discussion of its limitations. Thanks for your feedback. While we have addressed limitations in Appendix A.5, we agree that this discussion could be expanded. We have included more detailed textual descriptions of the limitations in Sec. 4.4 of the revised manuscript.
• Although some failure cases are illustrated in Figure 12 on page 16 (Appendix), it would be helpful to place these directly in the main text and discuss potential solutions more explicitly. We agree that placing the failure cases directly in the main text, along with a more explicit discussion of potential solutions, will enhance clarity. In the revision, we have integrated the failure cases from Fig. 12 into the main text and provide a detailed analysis of these cases, including possible solutions and improvements, as illustrated in Sec. 4.4.
• Discussion about addressing these limitations directly within the main body is suggested. We agree and have incorporated a detailed discussion of the limitations and potential solutions directly into the main text for better clarity, as illutrasted in Sec. 4.4 of the revised manuscript.
• However, this is a minor suggestion. My main concern is the first point about clarity. Thanks for your feedback. We have prioritized improving clarity in the revised manuscript to address your main concern.

Thank you for your detailed response. However, I find that my concerns regarding the innovations and contributions of your work remain unresolved. My concerns can be summarized into three main points:

1. Privacy Protection: From a high-level perspective, I do not see any unique or superior advantages in privacy protection offered by your approach compared to other existing methods. Even without lensless imaging, other imaging techniques, such as single-pixel imaging and minimalist cameras, can achieve similar levels of privacy protection. As a result, your work does not appear to make any distinct or original contributions in this regard.

2. Direct High-Level CV Tasks on Measurements: If your main contribution is the ability to perform high-level CV tasks, such as segmentation, directly on measurements, this also does not seem novel. For example, RecSegNet [1] is designed to perform both reconstruction and segmentation simultaneously rather than sequentially, so its method can also work without reconstructing the original visual data. In your response to the AC, you stated that “the OFD operates at the feature level, rather than on visual images.” However, after reading the RecSegNet paper, I found that its segmentation also operates at the feature level rather than on visual images. Additionally, other works in computational imaging have already explored performing CV tasks directly on measurements, such as [2].

Therefore, whether from the perspective of privacy protection or from performing CV tasks directly on measurements, your work does not stand out as innovative or unique.

3. Scope and Contribution Level: Your work seems more like a targeted solution within the specific application domain of lensless imaging, addressing how to perform segmentation directly on measurements in that context. In my opinion, this type of work does not reach the level of innovation and contribution typically expected for a conference like ICLR. Furthermore, after reviewing the references in your paper and confirming related work, I noticed that most similar research is published in optics or computational imaging journals rather than at machine learning conferences like ICLR. This suggests that your work might be somewhat out of scope for ICLR.

To justify its importance and relevance to ICLR, you need to make a much stronger case for the significance of your work and its connection to the broader machine learning community.

[1] Yin, Xiangjun, et al. "A Multi-Task Deep Learning Framework Integrating Segmentation and Reconstruction for Lensless Imaging." IEEE Transactions on Emerging Topics in Computational Intelligence (2024).

[2] Zhang, Zhihong, et al. "From compressive sampling to compressive tasking: retrieving semantics in compressed domain with low bandwidth." PhotoniX 3.1 (2022): 19.

Thank you very much for your thoughtful feedback and for recognizing the clarifications and modifications we made. We appreciate your insightful comments regarding the mathematical equations and understand your concern about distinguishing our contributions. We provide detailed responses to each of your points.

About the Theoretical Innovations. Thank you for your comment. The point we would like to emphasize is that although the equations in our paper are based on established techniques (Tikhonov Least Squares), our key innovation lies in extending the Tikhonov Least Squares method—traditionally used for image reconstruction—into the semantic feature domain. By validating the assumptions in Eq. (1), we shift the application of this technique from reconstruction to direct end-to-end inference. This change overcomes the performance bottleneck often associated with the traditional "reconstruction + inference" paradigm and provides a more efficient framework for low-quality scene inference tasks. We believe this represents a significant advancement in the field, and we will emphasize this distinction more clearly in the revised manuscript.

About Privacy Protection. As mentioned in my response to AC, in our work, the lensless measurements are directly input into the network, which only performs semantic inversion rather than visual reconstruction. This means that the information transmitted through our network is not in the form of visual data, thereby mitigating the risk of sensitive information leakage during the network's operation. we also have ensured to highlight these innovations more explicitly in the revised version of the manuscript to make the contribution clearer for readers.

Regarding privacy protection, it is not the primary focus of our work but rather an additional benefit of our approach. Our main objective is to emphasize the potential of using lensless devices for high-level inference tasks, which holds significant value for expanding the applications of lensless imaging technology.

As for single-pixel imaging or minimalist cameras, to the best of our knowledge, these fields are not directly related to lensless imaging technology. The privacy protection effectiveness associated with these technologies has not yet been discussed in high-level peer-reviewed journals. To maintain the rigor of our work, we have refrained from addressing this aspect. However, we acknowledge the importance of exploring the performance of these imaging mechanisms, including downstream task performance and privacy protection, in our future research.

About Real-world Experiments. As mentioned in our response to AC's third comment, for the real-world experiments conducted under varying conditions, we utilized scene data from FlatCam, such as the DIRC dataset. This dataset includes lensless imaging measurements captured under different illumination scenarios, which partially capture the complexity of real-world environments. However, we acknowledge that the availability of publicly accessible datasets for lensless imaging remains limited, presenting challenges in validating our method across more complex scenarios. We fully agree that incorporating a broader range of datasets would provide a more robust evaluation of our approach. As lensless imaging is an emerging field, we anticipate that the availability of diverse datasets will naturally increase with its development. In the meantime, we have made every effort to include the most comprehensive data and experiments available at present, and we kindly ask for your understanding in this regard.

In terms of experimental setup, while it is true that FlatCam captures are typically conducted in controlled environments, our experiments were not limited to isolated objects. For example, the DISC-Test dataset includes scenarios with multiple objects. In the revised manuscript (Appendix Fig. 16), we present multi-object segmentation results that highlight the strong performance of our method even in the presence of multiple objects. These results validate the potential of our approach for handling dense scenes and complex segmentation tasks. As such, we believe our findings demonstrate the significant applicability of our method to real-world environments, dense scenes, and multi-object scenarios.

We hope that this response addresses your concerns and provides clarity on the scope and contributions of our work. Thank you once again for your valuable input.

We greatly value your profound insights and acknowledgment, and have provided meticulous, point-by-point responses to address each concern with precision and clarity.

Weaknesses:

Q1:The paper adds an unnecessary "privacy preserving".

Thank you for highlighting concerns regarding the "privacy-preserving" claim in the title. Based on your valuable feedback, we have removed this term from the title and revised the manuscript to better align with the central contribution of our work—a segmentation approach for lensless cameras.

Our initial mention of "privacy-preserving" was motivated by the inherent properties of lensless imaging, where lensless measurements lack directly interpretable high-resolution details, which can reduce the risk of immediate information leakage. This aspect is particularly relevant for applications where safeguarding sensitive details is essential. However, as you rightly pointed out, positioning the paper around privacy-preserving could detract from its primary focus.

We appreciate your thoughtful suggestion and have revised the discussion in the introduction to clarify this point and avoid any perception of overclaiming. Thank you again for your insightful feedback, which has helped us strengthen the clarity and focus of the manuscript.

Q2:The ODM + CDM approach could be explained a bit better, and especially discussed more with related work. Has this division into subtasks been tried before? How does this relate to CDMNet?.

Thanks for your insightful question. Our method of decoupling the segmentation task into CDM and BDM addresses the inherent imbalance in pixel distributions during segmentation. While task decomposition, such as edge-based guidance, has been explored in works like CDMNet, our division into CDM and BDM offers a novel method tailored for lensless imaging, where spatial boundaries are often ambiguous or absent. Our method extends this concept by introducing the contour-body interaction (CBI) module, which models the correlations between CDM and BDM, enabling mutual learning and enhancing segmentation accuracy. In contrast, CDMNet focuses primarily on edge-based cues and does not utilize a dual-branch interaction for more comprehensive segmentation. We appreciate your suggestion to further discuss and ensure that these connections are more clearly articulated in Sec. 4.2 of revised manuscript.

Q3:Minor point, but the paper should make the experimental results section a bit more self contained and describe the content of the two benchmark datasets.

Thanks for your valuable suggestion. We agree that providing more context on the benchmark datasets and ensuring the experimental results section is more self-contained would enhance the clarity of the paper. In the revised manuscript, we have expanded on the characteristics and content of the DIRC and DISC datasets, detailing their specific relevance to our method and how their role in facilitating the evaluation of lensless object segmentation. This additional information will help readers better understand the experimental setup and ensure that the results are more comprehensively presented. The additional details about the two benchmark datasets can be found in Appendix A.2 (Page 14) .

Thanks for your concerns regarding privacy protection. As mentioned in my response to AC, in our work, the lensless measurements are directly input into the network, which only performs semantic inversion rather than visual reconstruction. This means that the information transmitted through our network is not in the form of visual data, thereby mitigating the risk of sensitive information leakage during the network's operation. Thank you once again for your recognition of our work.

Q5:Is there any fundamental innovation in the proposed work over RecSegNet or LOINet?

Thanks for your insightful question. To clarify the fundamental innovation of our proposed work over existing methods such as RecSegNet and LOINet, the key distinction lies in the elimination of the visual reconstruction step, providing the unified framework, and design of task decoupling and interaction , which is typically employed in these methods.

1. No Visual Reconstruction: Both RecSegNet and LOINet rely on a simple visual inversion setup, which inevitably introduces reconstruction errors that limit the performance of downstream tasks. These errors not only degrade task performance but also create a bottleneck in the optimization process, as the initial reconstruction and downstream tasks are somewhat isolated due to the nature of the reconstruction framework.

   In contrast, our method avoids the need for visual reconstruction altogether. By directly using lensless imaging measurements in the computational framework for downstream tasks, we bypass the error accumulation introduced by reconstruction and eliminate the bottleneck caused by separating the reconstruction and task optimization processes. This method streamlines the process and facilitates more effective optimization, which results in better overall performance.

2. Unified Framework: We extend the framework paradigm for downstream tasks from traditional imaging domain to computational imaging domain, enabling a more cohesive method. Specifically, our design offers insights into performing downstream tasks in challenging imaging environments, such as low light, blurriness, noise, and extreme weather conditions. For these, we simply need to adapt the configuration of the OFD for effective application in these scenarios.

3. Task Decoupling and Information Interaction: We also design a task decoupling mechanism that improves information interaction between different components. This design enhances the segmentation performance by ensuring that task-specific information is more effectively shared and optimized across the network, which further strengthens the robustness of our method in real-world conditions.

Unlike the methods in RecSegNet and LOINet primarily focus on the task of segmentation with initial reconstruction, our method offers a fundamentally different framework by eliminating the need for reconstruction and addressing the challenges inherent in lensless imaging for downstream tasks. This shift in paradigm provides a significant contribution, rather than simply adding new modules to existing systems.

Q3:One serious concern related to the lack of real experiments and the specific dataset used in this paper.

Thanks for your thoughtful and constructive feedback. We appreciate the opportunity to address your concerns regarding the datasets and experiments in our work.

For real experiments conducted under varying conditions, we utilized real-world scene data provided by FlatCam, such as the DIRC dataset. This dataset includes lensless imaging measurements captured under different illumination conditions, which partially reflect the complexity of real-world scenarios. However, we acknowledge that the availability of publicly accessible datasets for lensless imaging remains limited, making it challenging to verify our method under more complex scenarios. We fully agree that incorporating more diverse datasets would better validate the robustness of our approach. As lensless imaging is still an emerging field, we believe the availability of datasets will naturally grow as the field develops. We have made every effort to include the most comprehensive data types and experiments currently available in our manuscript and kindly request your understanding on this matter.

Regarding the source and naming of datasets, they primarily encompass the display capture dataset and the direct capture dataset. Lensless imaging measurements and paired ground-truth scenes were selected from the publicly available FlatCam dataset (5.9k samples for the display capture dataset and 30 samples for the direct capture dataset), with corresponding segmentation labels derived from [1]. To ensure consistency, we adopted the dataset naming conventions introduced in [1] (i.e, Direct Capture (DIRC) dataset and the Display Capture (DISC) dataset) and cited this reference accordingly.

For the experimental setup, while it is true that FlatCam captures are often conducted in controlled environments, our experiments were not limited to isolated objects. For instance, the DISC-Test dataset includes scenarios with multiple objects. In the revised manuscript ( Appendix Fig. 16 ), we present multi-object segmentation results that demonstrate the strong performance of our method even in the presence of multiple objects. These results validate the potential of our approach for dense scenes and more complex segmentation tasks. Consequently, we believe our findings highlight the significant applicability of the proposed method to real-world environments, dense scenes, and multi-object scenarios.

We hope this response addresses your concerns and clarifies the scope and contributions of our work. Thank you again for your valuable input.

Reference

[1] Xiangjun Yin, Huanjing Yue, Huihui Yue, Mengxi Zhang, Kun Li, and Jingyu Yang. A multi-task deep learning framework integrating segmentation and reconstruction for lensless imaging. IEEE Transactions on Emerging Topics in Computational Intelligence, 2024.

Q4:The results for LOINet and RecSegNet reported in this paper (Fig 6) do not match the results in (e.g., Fig 10 in ) RecSegNet paper. Why is this discrepancy?

Thanks for raising this important question. I would like to clarify the reasons behind the observed discrepancy between the results for LOINet and RecSegNet reported in this paper (Fig. 6) and those in the RecSegNet paper (e.g., Fig. 10):

1. In our manuscript, we have introduced additional comparison methods that were not included in the RecSegNet paper, such as CDMNet, OCENet, LL_T, Raw3dNet, and EyeCoD. To ensure a fair and unbiased comparison across all methods (including those overlapping with the RecSegNet paper and those not included in it), we re-trained all models under a same experimental conditions. Our goal was to provide a comprehensive comparison of a broader set of methods, rather than directly replicate the results reported in the RecSegNet paper. The inclusion of these additional methods and the emphasis on maintaining consistent evaluation conditions naturally led to some differences in the results.

2. As the models were re-trained from scratch, the random initialization of weights and biases can influence the convergence behavior of the networks, which in turn affects the final results. This randomness is inherent to the training process and can contribute to variations in performance.

3. Furthermore, the multi-threaded parallel computation used in codes introduces another source of randomness. For instance, in multi-threaded mode, data partitioning and loading, as well as optimizations in the underlying algorithms, may introduce minor differences in the results.

We would like to emphasize that the results we report are intended to reflect the performance under our experimental setup, ensuring the reliability of the results in the context of our study. We hope this explanation clarifies the reasons behind the observed discrepancies and provides a better understanding of our experimental method.

Thanks for your thoughtful feedback and positive discussion on our manuscript. We have carefully considered your questions and made revisions to the manuscript accordingly. Below, we provide detailed responses to each of your points.

Q1: I would like to hear some clarification on what do you mean by "the OFD mechanism facilitates back-end tasks without visual reconstruction, mitigating sensitive privacy leakage." in L202-203?

We sincerely appreciate your insightful question regarding the statement in L202-203: "the OFD mechanism facilitates back-end tasks without visual reconstruction, mitigating sensitive privacy leakage." This provides an opportunity to further clarify our work.

Traditional methods to downstream tasks typically start by reconstructing visual images from lensless measurements before extracting task-relevant features. While effective, this practice poses a significant risk in privacy-sensitive scenarios, such as medical imaging or surveillance, as the reconstructed images may inadvertently reveal sensitive information—even if such details are irrelevant to the downstream task.

To address this issue, our proposed OFD eliminates the need for visual image reconstruction altogether. Specifically, the OFD is designed to abtain high-level semantic features ($X_\theta$) associated with the underlying scene from high-level semantic features ($Y_\theta$) associated with lensless measurements by feature-level inversion. These features are task-relevant abstractions rather than direct visual data. That is the OFD operates at the feature level, rather than on visual images. Therefore, by operating solely on these abstract features, the OFD avoids reconstructing or extracting any visual details of the underlying scene, thereby significantly mitigating the risk of sensitive information leakage.

To further clarify, we have provided examples about the outputs of the OFD in Appendix Fig. 9 . These results clearly demonstrate that the outputs of the OFD are composed entirely of abstract semantic features, such as object contours, which are effective for downstream task performance while remaining devoid of sensitive visual details. This reinforces the privacy-preserving nature of our method, as it circumvents the reconstruction of original visual data.

We hope this clarification, along with the additional context provided in Appendix Fig. 9 , enhances your understanding of how the OFD mechanism operates and safeguards privacy.

Q2: I have another question about the so-called OFD-based extractor module. How is this new or a significant contribution that has a subsection and an appendix? It is a well-known Tikhonov Least Squares solution for a separable system, an identical version was proposed in the original FlatCam paper and subsequently used in other follow up papers.

Thanks for your insightful question. While the Tikhonov Least Squares is a well-established technique, and was effectively applied in the original FlatCam paper, our work introduces meaningful advancements. In traditional methods, Tikhonov Least Squares is typically used for image reconstruction. However, our method extends this technique to the semantic feature level, shifting its application from the image level to the feature level. This innovation not only eliminates reconstruction errors that typically hinder task performance in traditional methods but also significantly enhances the efficacy of downstream tasks. Consequently, the Tikhonov Least Squares solution discussed in our paper—especially in the dedicated subsection and appendix—is applied specifically at the feature level.

Furthermore, in the OFD module, unlike the traditional Tikhonov Least Squares solution that integrates $A_L$ and $A_R$ in a fixed manner, our method incorporates a feature extraction module to derive learnable $A_{L,\theta}$ and $A_{R,\theta}$. This enables them to adapt to the specific needs of downstream tasks, making the application of Tikhonov Least Squares more flexible and efficient within our framework.

Finally, the multi-level architecture of the OFD provides a richer semantic features for applying Tikhonov Least Squares at the feature level. By applying Tikhonov Least Squares across multiple scales, we effectively integrate complementary information, enhancing robustness and reducing errors, which further improves performance.

To clearly explain the differences and innovations of our method compared to traditional Tikhonov Least Squares, we used a subsection and an appendix, where we provide a detailed explanation of our method’s unique contributions. We believe these improvements represent a meaningful advancement over existing methods.

After carefully reviewing and reflecting on your discussion and verifying the related literature, I would like to raise a concern regarding the novelty of the authors' contribution. Specifically, I found that RecSegNet [1] is designed to perform both reconstruction and segmentation simultaneously rather than sequentially, meaning its method can also work without reconstructing the original visual data.

The authors stated that their innovation and contribution lie in their OFD "operating at the feature level, rather than on visual images.” However, after reading the RecSegNet paper [1], I found that its segmentation also operates at the feature level rather than on the original visual images.

Given these findings, I believe the authors need to more clearly articulate how their work demonstrates sufficient innovation and contribution compared to prior approaches.

[1] Yin, Xiangjun, et al. "A Multi-Task Deep Learning Framework Integrating Segmentation and Reconstruction for Lensless Imaging." IEEE Transactions on Emerging Topics in Computational Intelligence (2024).