Virtual Scanning: Unsupervised Non-line-of-sight Imaging from Irregularly Undersampled Transients

We are highly encouraged by the positive recommendation and comments from the reviewer on our experiment, method and presentation. Furthermore, the comments and suggestions are inspiring, helpful, and valuable. We address the main issues as follows.

Q1: Irregularity of scanning patterns.

Reply: We tested our method on publicly available real datasets with more irregular relay surface. These irregular relay surfaces do not contain simple repetitive or symmetrical shapes, such as the signage of NIPS 2024, window papercut with various shapes and even random graffiti. As shown in Figure 3, our method still achieves the best quality for various sampling patterns. In addition, as shown in Fig. 5 and 6 in the manuscript, the relay surface ''random points'' and ''regular 32$\times$32'' are more different from our training patterns, but our method can also handle them well. This is because our method's generalization is guaranteed by the theoretically grounded framework in learning beyond the range space.

Q2: The motivation of using different shutter patterns for training.

Reply: In the early development of our pipeline, we have tried to use random irregular relay surfaces for training, hoping that it would enhance our algorithm's robustness. However, this setting resulted in dramatic loss fluctuations and unstable training. We found that shutter-like patterns yield stable training. By adjusting the rotation angles and intervals of these patterns, the network generalizes well to a wide range of irregular sampling patterns (see Ablation studies in Sec. C.2).

We are highly encouraged by the positive recommendation and comments from the reviewer. We address the main questions as follows.

Q1: The lack of explanations for some key symbols and schematics.

Reply: Explanations: As stated in line 90-92, the forward operator $H$ is highly related to the scannable region on the relay surface. For the relay surface numbered $g$, $H_g$ is the forward operator using this relay surface, $M_g$ is the 3D version mask of the sampling pattern, constructed by repeating the 2D sampling pattern $t$ times along the time-dimension. Similarly, $M_k$ is another 3D version mask different from $M_g$.

Schematics: Figure 3(c) (main text) builds on Figure 3(b) (main text), which shows the training case with only the measurement consistency (MC) loss, corresponding to Figure 2(b) (main text). If only the MC constraint is imposed, the network cannot uniquely recover $\rho$ because multiple outputs can satisfy the MC constraint, as depicted by the blue dashed line in Figure 3(b) (main text). In Figure 3(c) (main text), when using the virtual scanning, $\rho^{(1)}$ is projected onto the $H_2$'s range-space $R_{H_2}$, and we can recover the range-space component $D_r(\rho^{(1)})$ of $\rho^{(1)}$ using the pseduo-inverse operator (LCT). Given $\rho^{(2)}=D_r(\rho^{(1)})+F_\theta(D_r(\rho^{(1)}))$ and $\rho^{(1)}=D_r(\rho^{(1)})+D_n(\rho^{(1)})$, the proposed VS loss $\rho^{(1)}=\rho^{(2)}$ is equivalent to $F_\theta(D_r(\rho^{(1)}))=D_n(\rho^{(1)})$. Therefore, virtual scanning ensures that the network $F_\theta$ learns a mapping from $D_r(\rho)$ to $D_n(\rho)$, ensuring the ability of our algorithm to achieve high-quality reconstruction.

Q2: The quantitative evaluation results for VSRnet and SURE-denoiser.

Reply: We simulated a dataset of 1,000 transients by rendering objects (chairs, clocks, guitars, sofas, motorcycles) with random scaling and positioning. We tested our method and its variants on the simualted dataset sampling with 16 various relay patterns. The quantitative results for VSRnet and SURE-based denoiser are as follows:

As evident from the results, the SURE-denoiser provided an average performance improvement of 1.83 dB and the virtual scanning provided an average improvement of 0.89 dB. The SURE-denoiser's quantitative performance improvement is more significant because background noise affects the entire reconstruction volume, while aliasing artifacts caused by irregular undersampling mainly disrupt the primary structure.

Q3: Is the choice of an unsupervised framework a necessary outcome for addressing the reconstruction problem of irregularly undersampled transients?

Reply: This is a very insightful question. We believe that the unsupervised framework is not only an effective approach for NLOS imaging from IUT but also a promising paradigm for deep learning in NLOS imaging research.

In recent years, learning-based algorithms have gained wide attention in the NLOS field. Their powerful function fitting and prior learning capabilities allow them to achieve better performance upper bound compared to model-based algorithms. However, most of these methods rely on supervised learning with simulated data, which creates a gap between simulated and real-world data, limiting their real-world performance. Furthermore, supervised learning requires paired datasets, which makes the model be prone to overfitting to the training domain, leading to poor generalization.

In contrast, unsupervised learning does not rely on paired data, allowing models to generalize better to new and unseen scenes. This approach is crucial for NLOS imaging tasks where recovering diverse hidden scenes is necessary. We believe the robust generalization capability of the unsupervised paradigm has been demonstrated by our extensive experiments with various real data and irregular patterns. Additionally, the unsupervised paradigm reduces the costs and time associated with acquiring labeled datasets, accelerating research progress. Therefore, the choice of an unsupervised framework is crucial for effectively addressing NLOS imaging. We will also open-source our code to support further research in the NLOS community.

Q4: Would introducing more irregularly varied masks further enhance the robustness of the network?

Reply: In our experiment, we found that introducing irregular masks led to unstable training and further reduced network performance. Due to the light attenuation effect, only a portion of the transient typically contain useful scene information. Therefore, using irregular masks for training often miss a significant part of the information in fully-sampled transients, leading to anomalous irregularly undersampled transient. Since our simulated dataset includes objects of various sizes and poses, these anomalies frequently occur during training, making it difficult for the network to converge and reducing its robustness.

Q5: How the authors constructed different forward propagation operators H based on the varying irregularly relay surfaces.

Reply: We constructed different forward operators $H$ using an operator decoupling method, as described in line 149-152.

For the full-sampled relay pattern, the forward operator is denoted as $H_r$, and the corresponding full-sampled transient is $u_r=H_r\rho$, where $\rho$ is the hidden volume. Similarly, for a relay surface numbered $g$, the observed transient $u_g$ is given by $u_g=H_g\rho$. By replacing through expere $u_g=M_g\odot u_r$, we can obtain expression that $M_g\odot H_r\rho = H_g\rho$. So we composition $M_g\odot H_r$ to constructed forward operator $H_g$. In practice, the composition can be efficiently computed through Hadamard product and the fast Fourier transform.

We thank the reviewer for valuable comments. We address the main questions as follows.

Q1: Connection and novelty of our SURE-based denoiser.

Reply: In the irregularly undersampling, the quality and stability of reconstruction could be severely affected by noise, necessitating a robust denoiser to rescue. As shown in Fig. 2 (see experiment setup in the global rebuttal), the PSNR improvement of our method with SURE denoiser over the one without denoiser is up to more than 7 dB at sampling rates between 1.0% $\sim$ 4%. With the SURE denoiser, the operating range of sampling rates is broadened to low rate end, which significantly extend the applicability of our method.

Regarding the originality, the SURE denoiser is specifically tailored for NLOS imaging. First, we incorporated the noise model of time-resolved detectors (see Sec. 4.3), addressing the dark noise $b$ in transients, and derived the SURE loss, which is not a straightforward modification of an existing one. Second, we proposed a neural network combining partial convolution and instance normalization (IN) layers (see Sec. A). Partial convolution contributes to suppress the loss of IUT information during feature propagation through the vanilla convolution and IN layers make network suitable to various distributions of IUTs with different sampling rates.

Q2: The assumption of relay surfaces.

Reply: Presently, NLOS imaging research community commonly assumes that relay surfaces exhibit uniform diffuse reflectance. However, to our knowledge, none previous methods have taken into account relay surfaces with varying BRDFs. We thank the reviewer poses a challenging task for achieving more practical NLOS imaging and warrants in-depth investigation in future studies.

Q3: The padding (interpolation) mode for unobserved points.

Reply: We further evaluate two interpolation methods, the bilinear method and nearest-neighbor method, as the pre-processing of reconstruction from IUTs with four baseline methods (LCT, FK, RSD and USM). As shown in Figure 4, the performance of interpolators on the final reconstruction is really dependent on the quality of interpolated transient signals, and essentially on the scanning patterns. For scanning patterns with relatively uniform scanning points (small holes on the relay surface, such as Window 3 and 5), the bilinear interpolater can help significantly suppress background reconstruction artifacts. However, scanning patterns with skewed distributed scanning points (large missing areas on the relay surface, such as Window 7), the bilinear interpolater can even degrade the reconstruction quality due to the unreliable interpolated transients in large missing areas. For all cases, our method still outperforms than other four baselines.

Since there is not an all-conquering one, we felt that using the simple zero padding is a reasonable comparison setup. Another choice is to adaptively select interpolation methods according to the scanning pattern, e.g. bilinear interpolation for patterns with more uniform sampling and zero padding with skewed sampling. We will discuss more above the padding mode in the final version.

Q4: Null-space learning of the virtual scanning.

Reply:

1. This could be a misunderstanding. Our goal is trying to recover the null space components of the reconstruction target (line 159). For this, We provide an intuitive illustration in Fig. 3 of the manuscript and some theoretical explanations in lines 159-171. For the special task of NLOS imaging from IUT, some portions of information about the hidden scene may not be captured due to light attenuation. So there is no guarantee that the proposed virtual scanning could complement the entire null space. However, adding various operators for training can effectively improve the generalization of our method. We will discuss more about this in our final version.

2. The phrase ''randomly sampled from the surface base'' (line 138) indicates that during training, we select one sampling pattern different from $M_g$ from the group of 200 sampling patterns during training to learn the null space components.

3. Indeed, the sampling rate for each sampling pattern should not be too low for stable and acceptable reconstruction. We conducted a quantitative experiment, which is detailed in the global rebuttal. Figure 2 illustrates a significant drop in reconstruction quality when the sampling rate falls below 6%. Notably, all sampling rates for patterns in the surface base exceed this lower bound.

4. There are two main reasons for the design of virtual scanning: First, more iterations in virtual scanning would accumulates errors, causing severe fluctuations in losses and hindering network convergence, which affects final the performance of network. This would also increase required computation. Second, in practice, we iterate through each pair of sampling patterns from the surface base to make network learn the null space of the associated operator. This training strategy successfully accelerates network convergence, stabilizes parameter updates, and improves reconstruction quality.

Q5: Some minor comments.

Reply: The notation $\tilde{}$ refers to the noisy version of transients, and $\hat{}$ refers to their denoised version. $l$ refers the spatial dimension of recovered volume and $G$ is the total number of the operators group. $k\neq g$ means that $k$ is a different number from $g$ and hence their corresponding sampling patterns are also different.

Thank you for pointing out these issues. We will add an additional section to clarify all symbols and variables. We will add more descriptions on the obtaining process of $\rho^{(1)}$ and $\rho^{(2)}$, and updates other presentation issues in the camera-ready version.

We are highly encouraged by the positive recommendation and comments from the reviewer on our experiment, method and presentation. Furthermore, the raised questions are both central and valuable. We address the main questions as follows.

Q1: The significance of NLOS imaging from irregularly undersampled transients.

Reply: The problem of irregularly undersampled transients in NLOS imaging is critical for practical applications. In real-world scenarios, relay surfaces are often discrete and irregularly shaped, such as fences, window shutter and window frame. Most current algorithms rely on dense and continuous transients, limiting their practical application. Developing algorithms for IUTs is essential to extend NLOS imaging to these common environments. Our method addresses this by efficiently using sparse transient information, significantly broadening the applicability of NLOS imaging.

Q2: The importance of the discussion of the null space.

Reply: The discussion in lines 100-114 is crucial for understanding the foundation of our proposed unsupervised learning algorithm, as it explains why our virtual scanning strategy is necessary and effective.

Specifically, our method is inspired by the range-null decomposition [1] and null space learning [2][3][4]. We observed that using only measurement consistency loss (MC loss) is insufficient for high-quality reconstruction. This is because only using MC loss for training cannot recover the null-space component of the target. To this end, we propose the virtual scanning strategy to recover the missing null-space component for high quality reconstructions, as shown in Figure 3(c) of the manuscript.

It establishes the theoretical foundation for our approach, elucidating the rationale behind our method and emphasizing how we address the limitations of prior techniques. Without this theoretical context, readers might question why the pipeline performs exceptionally well.

Q3: The effects of different patterns on the reconstruction and some limits we can expect.

Reply: Thank you for this very insightful observation and suggestion. In Figure 1(b) (see global rebuttal), we illustrate how the missing pattern affect the reconstruction quality. In general, a relay surface with a large missing area often results in poor reconstruction quality due to the loss of information. Even with the same sampling rate, more uniform scanning point distribution yields better results (e.g., the fifth column vs. the second).

As shown in Figure 1(a) (global rebuttal), light attenuation term $1/r^4$ causes that a histogram ($1\times1\times T$) captured from scanning point $p$ often only contain information from hidden areas close to the scanning point $p$, such as the green oval area shown. This leads to poor reconstruction if large parts of the relay surface are missing. However, the acceptable size of the missing area for reconstruction is hard to derive theoretically. As it depends on multiple factors such as the relative depth between the scene and the relay surface, the scene's reflectivity, normals, detector efficiency, and laser power.

Nevertheless, we tried to obtain a general trend through quantitative experiments. We simulated a dataset of 1,000 transients by rendering objects (chairs, clocks, guitars, sofas, motorcycles) with random scaling and positioning. The relay surface consisted of random points with sampling rates ranging from 0.1% to 100%, forming 30 groups. As shown by the red curve in Figure 2, the PSNR of the reconstructed intensity map starts to decline sharply below the sampling rate of 6%. Therefore, we recommend that the sampling rates for patterns should larger than 6%. We will add these discussion in the final version.

Q4: Presentations about Figure 2 and others.

Reply: Thanks for your suggestions on the presentation. We will include additional descriptions of the components in Figure 2, such as the networks $F_\theta$ and $F_\phi$, in the camera-ready version. Additionally, we will repair the hyperlinks in the citations and move the limitations section to the main text.

[1] Schwab, Johannes, Stephan Antholzer, and Markus Haltmeier. "Deep null space learning for inverse problems: convergence analysis and rates." Inverse Problems 35.2 (2019): 025008.

[2] Sønderby, Casper Kaae, et al. "Amortised MAP Inference for Image Super-resolution." International Conference on Learning Representations. 2022.

[3] Wang, Yinhuai, et al. "Gan prior based null space learning for consistent super-resolution." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 3. 2023.

[4] Wang, Yinhuai, Jiwen Yu, and Jian Zhang. "Zero-Shot Image Restoration Using Denoising Diffusion null space Model." The Eleventh International Conference on Learning Representations.

Thank you very much to the authors for their responses.

I'm not sure my comments on the motivation for the paper have been adequately addressed. But I'm not sure this can be addressed. Nonetheless, I appreciate the general rebuttal, especially Figure 1, I think it's quite interesting to see the effects of the reconstructions with the patterns the authors show, and I would suggest the authors add this discussion to the paper.

I'm not yet sure I will increase my score, I will be following the author's discussions with the other reviewers to set my final score, thanks!