Generalizable Non-Line-of-Sight Imaging with Learnable Physical Priors

Thank you for your valuable time and comments. The main concerns are addressed below.

Q1: The relation between the Eq. (1) and Eq. (4).

• In Eq. (1), H represents the transient measurement under ideal conditions, whereas H' in Eq. (4) incorporates noise (as claimed in lines 205--206 of the manuscript). During network training and testing, H in Eq. (1) is replaced by H' to better reflect real-world conditions. We hope this explanation clarifies your concerns.

Q2: Explanations of the notations in Eq. (3).

• Thanks for your reminder. The coordinates in $I$ are intended to represent point coordinates in the 3D image, which have three dimensions. However, upon review, the coordinates notation in Eq. (3) is indeed imprecise, which should be revised to $x_v=(x,y,z)$ in alignment with [1,2]. $I(x_v)$ represents the 3D image, reflecting the shape of the hidden object. The relationship between $x_s$ and $x_v$ is described by the RSD integral. For details and derivations of the RSD integral, please refer to [1].

Q3: Intuitive explanation of the function $\Phi$ in equation 3.

• Thanks for your suggestion. Since we use RSD as the physical prior for our method, the function $\Phi$ here can be represented as the RSD integral. Based on the explanation of the function by Liu et al. [1,2], the RSD operator is used to model the propagation in intensity-modulated light scenes. For more details, please refer to [1].

Q4: Synthetic performance concern.

• Compared to previous CNN-based methods, our approach achieves a notable improvement of at least 0.55 dB, which is significant relative to the marginal improvements typically observed among other CNN-based models. Additionally, our method requires only half the memory consumption and achieves faster inference compared to the transformer-based method NLOST. This balance between performance and resource efficiency highlights the advantages of our method, making it a potential solution for real-world applications.

Q5: Regarding the naming of the rendering module.

• Our approach is designed based on the LFE framework, as claimed in lines 214--215. The output of the wave propagation module is a three-dimensional feature volume, which is then rendered by the Rendering Net to produce the 3D albedo. This rendering process can be understood as a decoding procedure for the feature information. Thanks for your comments, we will include additional explanations in the revised paper.

Reference:

[1] X. Liu et al., “Non-line-of-sight imaging using phasor-field virtual wave optics,” Nature, vol. 572, no. 7771, pp. 620–623, Aug. 2019.

[2] X. Liu, S. Bauer, and A. Velten, “Phasor field diffraction based reconstruction for fast non-line-of-sight imaging systems,” Nat. Commun., vol. 11, no. 1, p. 1645, Apr. 2020.

Q8: The limited improvement compared with NLOST.

• It needs to be noted that NLOST is based on the transformer architecture and the resolution of the input measurement is limited due to computational resource restriction. Relying on a more lightweight CNN structure, our approach achieves improved results but only requires half the memory consumption and less inference time compared to NLOST, which greatly faciliates the reconstruction of higher-resolution scenes. This balance between performance and efficiency highlights the advantages of our approach.

Reference:

[1] M. O’Toole, D. B. Lindell, and G. Wetzstein, “Confocal non-line-of-sight imaging based on the light-cone transform,” Nature, vol. 555, no. 7696, pp. 338–341, Mar. 2018.

[2] X. Liu et al., “Non-line-of-sight imaging using phasor-field virtual wave optics,” Nature, vol. 572, no. 7771, pp. 620–623, Aug. 2019.

[3] X. Liu, S. Bauer, and A. Velten, “Phasor field diffraction based reconstruction for fast non-line-of-sight imaging systems,” Nat. Commun., vol. 11, no. 1, p. 1645, Apr. 2020.

[4] Q. Hernandez, D. Gutierrez, and A. Jarabo, “A Computational Model of a Single-Photon Avalanche Diode Sensor for Transient Imaging,” Feb. 23, 2017, arXiv: arXiv:1703.02635.

[5] W. Chen, F. Wei, K. N. Kutulakos, S. Rusinkiewicz, and F. Heide, “Learned feature embeddings for non-line-of-sight imaging and recognition,” ACM Trans. Graph., vol. 39, no. 6, pp. 1–18, Dec. 2020.

[6] I. Cho, H. Shim, and S. J. Kim, “Learning to Enhance Aperture Phasor Field for Non-Line-of-Sight Imaging,” Jul. 28, 2024, arXiv: arXiv:2407.18574.

Thank you for your valuable time and comments. The main concerns are addressed below.

Q1: Selection of Gaussian-shaped function as light source function is inaccurate.

• There might be some misunderstanding here. We do NOT claim that the illumination function must be a Gaussian-shaped function. In this paper, we aim at active NLOS imaging based on a SPAD camera and thus we select the commonly used Gaussian-shaped function as the illumination function. This is also consistent with Table S.2 of Liu’s paper [2], where different illumination functions are coupled with different imaging systems.

Q2: Learning a Gaussian window with a changeable spectrum band is inaccurate.

• We respectfully cannot agree. As highlighted in the concurrent work [6], the illumination function with the Gaussian shape works as a band-pass filter in the frequency domain, and only a certain range of the spectrum is necessary for recovering hidden objects. This is highly consistent with our motivation, as discussed in lines 255--260 of our manuscript.

  Since different scenes exhibit varying spectrum under different noise conditions, we introduce the APF to learn the standard deviation for relative band selection and adaptive denoising.

  Additionally, we visualize the output of the APF at:

  [Anonymous Link]https://picx.zhimg.com/80/v2-15039835e627e26b62ef2ceb7f5f0e92.png

  which includes evaluation with 10 transient measurements in the Seen test set under distinct SNR conditions. The results show that the standard deviation decreases as the SNR decreases, indicating that APF learns an adaptively Gaussian window and applies stronger high-frequency noise suppression under lower SNR conditions.

Q3: Explanation of the $I(x, y)$.

• We sincerely thank the reviewer for pointing out the imprecise description regarding $I(x_v)$. $I(x_v)$ should represent the 3D image of the hidden scene. The intended meaning of lines 202--203 in the manuscript is that the wave propagation function propagates $x_s$ to points in $I(x_v)$, which reflect the shape of the hidden objects (point coordinates denoted as $x_v$, consistent with [2,3]). Following the reviewer's suggestion, we will make the following revisions to ensure the theoretical accuracy of the manuscript:

  • Modify Eq. (3) to $I(x_v)= \Phi \left ( \mathcal{P}(x_s, t) \right )$,
  • Revise lines 202–-203 to redefine $I(x_v)$.

  These changes will not affect the experimental conclusions in the manuscript.

Q4: Question of the FFT for light source function.

• Firstly, Eq. (2) defines the frequency-domain representation of the illumination function, illustrating that the illumination phasor field, while Gaussian-shaped, performs a filtering operation centered at $\Omega_C$. Secondly, we do not perform an FFT on the illumination function itself. The purpose is to enable the network to learn data characteristics in the frequency domain and predict the optimal standard deviation.

Q5: Concerns for the lines 78--79.

• Hernandez et al. model the noise inherent in NLOS imaging [4], accounting for the effects of ambient light and dark counts. Subsequently, Chen et al. synthesize datasets based on this mathematical model for training the LFE algorithm [5]. To maintain consistency with prior work, we continue using the dataset simulation tools employed by [5], which is why we refer to ambient light and dark counts as the noise sources of interest in our paper. The motivation of our paper is not to model noise but to discuss how to improve the approach's generalization performance from the perspective of noise present in the measurements. Section 4.5 validates the proposed approach's ability to suppress noise effectively.

Q6: Reasons of choosing 3 path compensation weights.

• For quadratic and quartic compensations, please refer to the description of "Validating radiometric falloff" in the supplementary materials of [1], Page 2. Additionally, to enhance the stability of network training, we include an extra linear compensation. Thanks for pointing this out, we will include additional explanations in the revised paper.

Q7: The LFE results in Fig.7 is different from the LFE (baseline) in Fig.9.

• In Fig. 9 of the manuscript, the LFE (baseline) is configured with the same physical prior (RSD) as our proposed method, which helps eliminate the influence of the physical prior on the conclusions of the ablation. In contrast, the LFE shown in Fig. 7 employs FK as the physical prior, consistent with the official codebase. Thanks for pointing this out, we will include additional explanations in the revised paper.

Dear reviewer iqtw,

We sincerely thank you for your valuable time and suggestions. As today is the final day for manuscript revisions, we aim to improve the manuscript as much as possible before the deadline. Should you have any further questions, we are happy to provide additional clarification and resolve them.

We look forward to your positive feedback if our previous responses and revised manuscript adequately address your major concerns.

Thanks once again.

Sincerely,

The Authors

Thank you for your valuable time and comments. The main concerns are addressed below.

Major:

Q1: Visualization of the compensation coefficients from LPC module.

• We provide visualizations of the compensation coefficients predicted by LPC at:

  [Anonymous Link]https://pic1.zhimg.com/80/v2-78bc54bc3453588342fd05b3d4b5cf36.png and [Anonymous Link]https://pic1.zhimg.com/80/v2-4c49999ca18b5d8a4df94131ad5bb759.png

  In the visualization, the 'Reconstruction' denotes the results of our method, while the 'Coeffs map' denotes the predicted compensation coefficients for the corresponding scenes. It is important to note that the SPAD captures scattered photons from the entire scene at each sampling point, the predicted coefficients are globally optimized values.

  It is worth mentioning that, since the SPAD captures scattered photons from the entire scene at each sampling point, the predicted coefficients are globally optimized values, which may not exhibit pixel-wise correspondence.

  The figure from the first link demonstrates that LPC assigns larger compensation coefficients to the object farther away (i.e., the statue on the left) in the scene, compared to the closer object (i.e., the deer on the right). For single-object scenes from the figure in the second link, LPC assigns compensation coefficients corresponding to the material properties of the objects (with the 'man' scene containing diffuse material, and the 'ladder' scene containing retro-reflective material) in the areas where the objects are placed. Since an end-to-end network tends to learn and converge with the goal of optimizing the final output (reconstructed image and depth), the intermediate results in visualizations may not perfectly align with the theoretical values (quadratic or quartic). However, we can observe that the compensation coefficients indeed vary according to different materials.

Q2: Visualization of the standard deviations from APF module.

• The visualization of standard deviation predictions can be assessed at:

  [Anonymous Link]https://picx.zhimg.com/80/v2-15039835e627e26b62ef2ceb7f5f0e92.png

  For this evaluation, we randomly select 10 transient measurements from the Seen test set for evaluation. Different colored lines in the figure illustrate how the predicted standard deviation varies under different SNR conditions. A lower standard deviation of the Gaussian window in the frequency domain typically indicates a narrower frequency response, leading to stronger high-frequency noise suppression. As shown in the figure, the standard deviation exhibits a decreasing trend as the SNR decreases, which means APF applies a 'tighter Gaussian' for spectrum filtering in lower SNR conditions as the reviewer expected.

Minor:

Q1: Inverse Fourier Transform in Eq.(8).

• Eq. (8) is implemented as a time-domain convolution in the actual computation process, without NO Inverse Fourier transformation. The Fourier equation in Eq. (8) is provided solely to explain APF from a spectral perspective further. Thanks for pointing this out, we will include additional explanations in the manuscript.

Q2: The way of capture the real data at varying SNR levels.

• The qualitative results, which are from the same scene under different exposure times, are presented in Fig.10 and Fig.11 of Appendix A. For the SNR conditions of real-world transient measurements, different exposure times result in different SNR levels, as the reviewer expected. It can be seen that our method achieves the best performance under different exposure times, highlighting its superior generalization capability.

Thank you for your valuable time and comments. The main concerns are addressed below.

Q1: Quantitative evaluation on how different surface materials at different distances are compensated by LPC.

• As suggested, we add visualizations of the predicted compensation coefficients at:

  [Anonymous Link]https://pic1.zhimg.com/80/v2-78bc54bc3453588342fd05b3d4b5cf36.png and [Anonymous Link]https://pic1.zhimg.com/80/v2-4c49999ca18b5d8a4df94131ad5bb759.png

  The figure from the first link shows that LPC assigns larger compensation coefficients to the object farther away (i.e., the statue on the left) in the scene, compared to the closer object (i.e., the deer on the right). For single-object scenes from the figure in the second link, LPC assigns compensation coefficients corresponding to the material properties of the objects (with the 'man' scene containing diffuse material, and the 'ladder' scene containing retro-reflective material). It is worth mentioning that, since the SPAD captures scattered photons from the entire scene at each sampling point, the predicted coefficients are globally optimized values, which may not exhibit pixel-wise correspondence.

Q2: Contributions of the initial compensated feature in LPC pipeline.

• To investigate the contribution of the initial compensation to LPC, we remove the initial compensation coefficients, as expressed by:

  $F_{C}^{ini} =$ (G_{Z}), (G_{Z}), (G_{Z}) $\otimes F_{E}^{'}$.

  The modified approach is denoted as 'Ours-w/o IC'. The quantitative results under the same testing conditions as Tab. 1 are shown below:

  Method	PSNR	SSIM	RMSE	MAD
  Ours-w/o IC	23.86	0.8636	0.0903	0.0398
  Ours	23.99	0.8703	0.0874	0.0312

  These results demonstrate that incorporating initial compensation effectively enhances performance. To further exhibit the contribution in real-world data, we have provided a qualitative comparison of reconstructed results, which can be assessed in:

  [Anonymous Link]https://pic1.zhimg.com/80/v2-f2e5a87ba02a3bd1240266d730239a8f.png

  It can be observed that our approach reconstructs richer scene details compared to 'Ours-w/o IC' such as the plaster statue in the rear right of the 'teaser'.

Q3: Motivation for using average pooling instead of max pooling.

• Thanks for your valuable comment. Average pooling was originally chosen to capture the overall feature distribution, ensuring the method considers the broader context. As suggested, we have tested max pooling and observed a slight improvement on performance. We greatly appreciate this insight and plan to incorporate this update in the revised paper.

Q4: Contributions of the 3D CNN block in LPC pipeline.

• The 3D CNN block is placed within the LPC module to learn compensation features, and removing this block would make the LPC module non-functional. However, it is important to emphasize that the 3D CNN block is not the core contribution of the LPC module. While modifications to this block could potentially enhance network performance, such exploration falls beyond the scope of this paper.

Q5: The architecture of spectrum convolution.

• Firstly, the features of the transient measurements are transformed into the frequency domain using a Fourier transform. Subsequently, a series of 3D convolutional layers with a stride of (1, 2, 2) is applied to reduce the spatial dimensions of the feature vectors, with each convolutional layer followed by a ReLU activation layer. The extracted features are then processed using a 1D convolutional layer, normalized with LayerNorm, and passed through a fully connected layer to fuse the spectrum dimension into a scalar value. Finally, a sigmoid activation computes the desired standard deviation. We will include the detailed descriptions of the network architecture in the revised paper.

Q6: Materials used in the captured scene.

• As shown in Fig. 7 and Fig. 8 of the manuscript, the material types of the various objects can be divided into two groups: retro-reflective type (e.g., a bookshelf, a dragon model, and the letters 'NYLT' covered with reflective tape) and diffuse type (e.g., the number '2' covered with white printer paper, a cardboard printed with '123XYZ', and the plaster statues).