Untrained Neural Nets for Snapshot Compressive Imaging: Theory and Algorithms

We thank the reviewer for carefully reading our paper and providing constructive feedback.

 

Q-1: Formatting in Figure 2 can be improved. The text near smaller cubes is not readable.

A-1: We thanks the reviewer for the suggestion. In the revised version, we modify Figure 2 as follows: i) We increase the font size to make the figure more readable; ii) We add a detailed caption to explain the various components of the figure, including the meaning of different colored lines and dots; iii) We separate the 2D measurement $\bf y$ and the 3D binary mask $H$ to avoid confusion; iv) We change some of the solid lines to dashed lines to make the figure easier to interpret; v) We improve the layout of the figure to make it easier to follow. The updated Figure 2 is included in the one-page PDF response under the general rebuttal.

We thank the reviewer for carefully reading our paper and providing constructive feedback.

Q-1: Application of DIP hypothesis to all untrained networks and the relevance of the Lipschitz condition.

A-1: As mentioned by the reviewer, not all untrained neural nets (UNNs) satisfy the DIP hypothesis and this is not the assumption in our paper. The assumption is that for any class of signals, there exist UNN structures that satisfy the DIP hypothesis. More specifically, consider a class of signals denoted by $\mathcal{Q}\subset\mathbb{R}^n$. A UNN, modeled as $g_{\theta}(\bf u)$, satisfies the DIP-hypothesis if for any randomly generated $\bf u\in\mathbb{R}^N$ (generated according to some preset distribution), and any $\bf x\in\mathcal{Q}$, $\min_{\theta} ||g_{\theta}(\bf u) - \bf x||_2 \leq \delta,$, almost surely. Existence of such UNN structures for different class of signals $\mathcal{Q}$ is shown empirically. For example, deep image prior (DIP) and deep decoder (DD) are well-known UNN structures satisfying this property for natural images.

Regarding Lipschitz continuity, note that our UNN architecture comprises multiple hidden layers, each employing a linear convolution operator followed by a ReLU nonlinearity. The final layer consists of a linear operation and a sigmoid activation function. Given the compositional nature of these layers, it is evident that our UNN satisfies the Lipschitz condition. Notably, our theorem is applicable to any Lipschitz constant $L$.

 

Q-2: Questions about the contributions of the paper.

A-2: The focus of this work is on the application of UNNs in solving SCI problem and developing i) a theoretical understanding of the problem, and ii) a robust UNN-based SCI solution. Specifically, here are our key contributions:

1. Theoretically characterizing the performance of SCI solutions that employ a generic UNN for recovery. The key implication of our theoretical characterizations is that it enables us to optimize the parameters of SCI systems' hardware, under both noise-free and noisy setups.

2. Proposing a novel unsupervised SCI solution for video SCI. We establish the effectiveness of our proposed solution through extensive simulations.

 

Q-3: Comparison between this work and the GAP-Net paper [1] and the potential overlap between the two papers.

A-3: In this paper, we theoretically analyze the performance of DIP-SCI optimization defined as

$

\hat{\bf x}=\arg\min ||\bf y-\bf H\bf c||_2,

$

subject to $\bf c=g_{\theta}(\bf u),\theta\in[0,1]^k$

[1] does not include any theoretical analysis of this optimization or any of its variants. Instead, [1] focuses on the convergence behavior of projected gradient descent and its variant, GAP, in solving a related compression-based optimization. Also note that in [1], the entries of the masks are assumed to be i.i.d.~ Gaussian, which is inconsistent with masks used in practice. Instead, in our work, we consider an i.i.d.~Bern$(p)$ distribution for masks.

 

Q-4: Conditions of DIP hypothesis compared to the original DIP paper and connections to pre-trained networks.

Pre-trained networks and UNNs differ significantly in their operational basis. While pre-trained networks leverage extensive datasets to acquire knowledge, UNNs operate without requiring any training data. This fundamental distinction grants UNNs a clear advantage in scenarios with limited data, making them more adaptable to various applications. Consequently, although pre-trained networks can be integrated into our framework, the unique strength of UNNs in data-scarce environments underscores their important role in our methodology.

Finally, we would like to ask the reviewer to explicitly identify which DIP conditions they believe are overlooked in our paper. This helps us better address the concerns.

 

Q-5: Comparison algorithms included in the paper.

A-5: In Section 5, we compare the performance of our proposed method against several other methods, listed in the section labeled `Datasets and baselines'. To the best of our knowledge, we cover all existing UNN-based video SCI solutions in our comparisons. We also extended one UNN-based method, originally proposed for hyperspectral SCI, and compared its performance against our method.

To better clarify and distinguish the methods compared to our proposed method, we will update Table 1 in our main paper by organizing the comparisons into four categories:

1. GAP-TV [9], a traditional optimization-based algorithm.

2. PnP-FFDnet [17] and PnP-FastDVDnet [18], pre-trained deep denoisers + PnP algorithm.

3. Existing UNN-based solutions for video SCI [30], [31].

4. Our main proposed method (SCI-BDVP) and two simpler variants for comparison: 1) SCI-DVP, simple E2E DVP, 2) SCI-BDVP, bagged E2E DVP, and 3) SCI-BDVP, our main proposed method.

The last two categories are UNN-based methods, we consider our model to be state-of-the-art among this category.

 

Q-6: Comparison with some mentioned recent papers.

A-6: Methods such as SCI3D and EfficientSCI are supervised approaches that require extensive training data. A key limitation of these supervised methods is that they fix the sensing matrix during training. This makes it challenging to infer the impact of masks on performance and to separate this effect from the optimization performance. Additionally, applying the trained network to different sensing matrices and simulation settings typically results in performance degradation.

In this paper, one of our main goals is to develop unsupervised solutions that do not require training data or rely on prior knowledge of the sensing matrix and also to provide a fundamental understanding of the role of masks in performance. Therefore, we have not included supervised, mask-dependent methods in our comparisons.

References can be found in our main paper.

We thank the reviewer for carefully reading our paper and providing constructive feedback.

 

Q-1: Discussion about computational complexity and runtime.

A-1: We have made the following changes to the paper: In the main body of the paper, in Section 5.1, we have added the following explanation on the computational complexity of our proposed method. "The proposed SCI-BDVP method relies on the bagging of multiple DIP projections. These DIP projections, which vary depending on the patch size, involve different levels of computational complexity. Table 2 shows the average time required to perform each DIP projection for each patch size. As observed, the time increases considerably as the patch size decreases. This is expected because the number of networks that need to be trained grows significantly. (Refer to Figure 1 for a pictorial representation.) Additional computational complexity analysis of our proposed method and its comparisons with other methods is included in Appendix B.3."

Furthermore, we have added Section B.3 in the Appendix, which provides detailed information about the computational and time complexity of our proposed method. It also includes a comparison between our method and other UNN-based approaches. For completeness, we have copied the newly added section here as well.

"Implementing SCI-BDVP involves outer loop iterations (described in Algorithm 1 in appendix B.1) and also inner loop iterations for training DIPs. Table 3 presents average number of inner loop iterations used for different patch sizes (64, 128, 256) of various videos, and the number of outer loop iterations. Detailed time consumption for each patch level computation is recorded in Table 2. A comparison across different UNN-based methods is provided in Table 1. All comparisons are performed on a single NVIDIA RTX 4090. It is important to note that training a bagged DIP requires training multiple separate DIPs. This process can be readily parallelized, which is expected to significantly speed up the algorithm. We plan to explore this direction to optimize the algorithm's efficiency in future work. Lastly, making a direct comparison among all methods is challenging because, for supervised methods, the main time is spent in training, whereas, for unsupervised methods, the main time is spent on training the UNNs. This is an expected trade-off for requiring no training data and achieving a robust solution."

Table 1: Time complexity of different methods on one 8-frame benchmark video.

Table 2: Time complexity of our proposed SCI-BDVP was evaluated on various patch sizes (64, 128, 256) of video blocks, using a standard 1000 DVP iterations for training.

 

Q-2: Clarity of notations and figure presentation.

A-2: In the revised version, we modify Figure 2 as follows: i) increase the font size; ii) add a detailed caption; iii) separate the 2D measurement $\bf y$ and the 3D binary mask $H$; iv) change some of the solid lines to dashed lines; v) improve the layout of the figure to make it easier to follow. The updated Figure 2 is included in the one-page PDF response under the general rebuttal.

Also, we have reviewed the paper to ensure that all notations are clearly defined and consistently used.

 

Q-3: Potential application of our proposed method in hyperspectral imaging.

A-3: We expect our method to be applicable to hyperspectral imaging as well. In fact, one of the papers we have cited and used in our simulations for comparison (reference [27] in the paper) utilizes DIP for hyperspectral imaging (HI). Here are two key advantages of our proposed framework:

1. Our proposed theoretical framework is applicable to hyperspectral imaging and provides a solid theoretical foundation to analytically explore other aspects of HI, such as the effect of shifted masks.

2. A well-known challenge in using UNNs in reconstruction algorithms is their tendency to suffer from overfitting. However, our proposed method, based on bagging, is expected to perform well in HI and overcome these issues.

To highlight this direction and its potential, we add this paragraph in the conclusion section of the revised version: "An important application of SCI is hyperspectral snapshot imaging (HSI). Our results in this paper provide a theoretical foundation to understand HSI systems and optimize their hardware. Additionally, the developed theoretical framework can be used to explore aspects specific to HSI, such as masks being shifted versions of each other. We also expect our algorithm to effectively address overfitting in HSI tasks, enhancing reconstruction performance. We plan to explore these aspects further in our future research."

 

Q-4: Different noise models beyond additive Gaussian noise?

A-4: The exploration of noise models beyond additive white Gaussian (AWGN) noise is a valuable and important direction for future work. While AWGN is a commonly-adopted model in many imaging solutions, including most SCI systems, there are situations, such as some newly-proposed coherence imaging SCI methods, that other types of noise such as speckle noise are dominant. We believe that our UNN-based approach to model the source has the potential to address these alternative noise scenarios. To accommodate non-AWGN noise models, the loss function $||\bf y-\bf H\bf c||_2$ would require modification depending on the noise model. We consider this an important avenue for future investigation.

We thank the reviewer for carefully reading our paper and providing constructive feedback.

 

Q-1: Discussion about Computational complexity.

A-1: We have made the following changes to the paper: In the main body of the paper, in Section 5.1, we have added the following explanation on the computational complexity of our proposed method. "The proposed SCI-BDVP method relies on the bagging of multiple DIP projections. These DIP projections, which vary depending on the patch size, involve different levels of computational complexity. Table 2 shows the average time required to perform each DIP projection for each patch size. As observed, the time increases considerably as the patch size decreases. This is expected because the number of networks that need to be trained grows significantly. (Refer to Figure 1 for a pictorial representation.) Additional computational complexity analysis of our proposed method and its comparisons with other methods is included in Appendix B.3."

Furthermore, we have added Section B.3 in the Appendix, which provides detailed information about the computational and time complexity of our proposed method. It also includes a comparison between our method and other UNN-based approaches. For completeness, we have copied the newly added section here as well.

"Implementing SCI-BDVP involves outer loop iterations (described in Algorithm 1 in appendix B.1) and also inner loop iterations for training DIPs. Table 3 presents average number of inner loop iterations used for different patch sizes (64, 128, 256) of various videos, and the number of outer loop iterations. Detailed time consumption for each patch level computation is recorded in Table 2. A comparison across different UNN-based methods is provided in Table 1. All comparisons are performed on a single NVIDIA RTX 4090. It is important to note that training a bagged DIP requires training multiple separate DIPs. This process can be readily parallelized, which is expected to significantly speed up the algorithm. We plan to explore this direction to optimize the algorithm's efficiency in future work. Lastly, making a direct comparison among all methods is challenging because, for supervised methods, the main time is spent in training, whereas, for unsupervised methods, the main time is spent on training the UNNs. This is an expected trade-off for requiring no training data and achieving a robust solution."

Table 1: Time complexity of different methods on one 8-frame benchmark video.

Table 2: Time complexity of our proposed SCI-BDVP was evaluated on various patch sizes (64, 128, 256) of video blocks, using a standard 1000 DVP iterations for training.

 

Q-2: Proofs of the theoretical results.

A-2: The main pdf file (29 pages) includes detailed proofs of all the results. More specifically, in Appendix A, we have provided detailed proofs in the following order: proof of Theorem 3.1 in A.2, proof for Corollary 3.3 in A.3, proof for Theorem 3.4 in A.4 and finally proof for Theorem 3.5 in A.5.

 

Q-3: Theorem 3.1: Are parameters $p$ in $u[0,1]^p$ and Bern($p$) for $D_{i,j}$ the same?

A-3: We had inadvertently overloaded the symbol $p$. We will substitute $p$ in $u[0,1]^p$ with $N$.

 

Q-4: Theorem 3.1. reconstruction error bound - $\rho$ not defined.

A-4: Thanks for pointing this out. $\rho$ denotes an upper bound on the $\ell$-infinity norm of the signals in $\mathcal{Q}$. We added the description in the revised version.